2024-01-16T14:43:02.629 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T14:43:02.629 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --archive /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191 --verbose --name rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/orig.config.yaml 2024-01-16T14:43:02.677 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T14:43:02.764 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7518191' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2668 sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi125.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9/3G22L2LTGLeUmjReflqE9a8+S32eThZv9VfVV2VqP8LRTiHfJ25gh1NiimO9Lj/a+n3nB8QQhzq3wH1G6n0= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvVVxBXj5PaYLksTxs/EpEzdXB4QGafjrHYSGvsfdMHzNM22qCplleZggqUAVamp+v+Yeyed3RnOuoQ4eGOA+M= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_14:37:50 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T14:43:02.764 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-16T14:43:02.765 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-16T14:43:02.765 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T14:43:02.767 INFO:teuthology.task.internal:Checking packages... 2024-01-16T14:43:02.811 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '73a4532fc4d97adee0b985d4d42874ea351a1c86' 2024-01-16T14:43:02.811 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T14:43:02.811 INFO:teuthology.packaging:ref: None 2024-01-16T14:43:02.811 INFO:teuthology.packaging:tag: None 2024-01-16T14:43:02.811 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-16T14:43:02.811 INFO:teuthology.packaging:sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:43:02.812 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-01-16T14:43:03.183 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g73a4532f 2024-01-16T14:43:03.184 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T14:43:03.191 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T14:43:03.191 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T14:43:03.211 INFO:teuthology.task.internal:Saving configuration 2024-01-16T14:43:03.227 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T14:43:03.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T14:43:03.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:00.677787', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9/3G22L2LTGLeUmjReflqE9a8+S32eThZv9VfVV2VqP8LRTiHfJ25gh1NiimO9Lj/a+n3nB8QQhzq3wH1G6n0='} 2024-01-16T14:43:03.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:00.680535', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvVVxBXj5PaYLksTxs/EpEzdXB4QGafjrHYSGvsfdMHzNM22qCplleZggqUAVamp+v+Yeyed3RnOuoQ4eGOA+M='} 2024-01-16T14:43:03.280 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T14:43:03.287 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-16T14:43:03.287 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-16T14:43:03.287 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T14:43:03.349 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6e1c077160>, signals=[15]) 2024-01-16T14:43:03.350 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T14:43:03.355 INFO:teuthology.task.internal:Opening connections... 2024-01-16T14:43:03.356 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:43:03.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:03.447 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:43:03.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:03.546 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T14:43:03.553 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2024-01-16T14:43:03.571 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2024-01-16T14:43:03.571 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2024-01-16T14:43:03.628 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Stream" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:03.629 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:03.630 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2024-01-16T14:43:03.661 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-01-16T14:43:03.678 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-01-16T14:43:03.679 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-01-16T14:43:03.735 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-01-16T14:43:03.735 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:03.736 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:03.737 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-01-16T14:43:03.761 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T14:43:03.770 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T14:43:03.775 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T14:43:03.776 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:03.778 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:03.793 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T14:43:03.799 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T14:43:03.800 DEBUG:teuthology.orchestra.run.smithi125:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:03.834 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:03.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T14:43:03.989 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T14:43:04.003 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-16T14:43:04.003 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-16T14:43:04.004 DEBUG:teuthology.orchestra.run.smithi125:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:04.004 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:04.020 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.020 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:04.021 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-16T14:43:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:04.022 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.022 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:04.022 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2024-01-16T14:43:04.077 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:04.077 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.077 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:04.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.081 INFO:teuthology.orchestra.run.smithi125.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:04.081 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.081 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:04.081 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:04.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:04.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:06.962 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-9 - AppStream 6.2 MB/s | 6.4 MB 00:01 2024-01-16T14:43:07.024 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 5.8 MB/s | 6.4 MB 00:01 2024-01-16T14:43:08.540 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-01-16T14:43:08.586 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-01-16T14:43:10.377 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-01-16T14:43:10.573 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-01-16T14:43:14.503 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-01-16T14:43:18.156 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-01-16T14:43:19.915 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.9 MB 00:01 2024-01-16T14:43:21.403 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2024-01-16T14:43:24.864 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 5.9 MB 00:03 2024-01-16T14:43:25.880 INFO:teuthology.orchestra.run.smithi125.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2024-01-16T14:43:26.415 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-01-16T14:43:30.671 INFO:teuthology.orchestra.run.smithi125.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-01-16T14:43:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2024-01-16T14:43:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2024-01-16T14:43:32.932 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2024-01-16T14:43:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2024-01-16T14:43:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrading: 2024-01-16T14:43:32.932 INFO:teuthology.orchestra.run.smithi125.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrade 1 Package 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 375 M 2024-01-16T14:43:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2024-01-16T14:43:37.622 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 20 MB 00:11 2024-01-16T14:43:42.423 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-16T14:43:44.757 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-16T14:43:44.768 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-16T14:43:44.769 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 1 Package 2024-01-16T14:43:44.769 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:43:44.769 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 375 M 2024-01-16T14:43:44.769 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-16T14:44:01.147 INFO:teuthology.orchestra.run.smithi137.stdout:linux-firmware-20240109-142.el9.noarch.rpm 23 MB/s | 375 MB 00:16 2024-01-16T14:44:01.148 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:01.148 INFO:teuthology.orchestra.run.smithi137.stdout:Total 23 MB/s | 375 MB 00:16 2024-01-16T14:44:01.148 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-16T14:44:01.235 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-16T14:44:01.236 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-16T14:44:03.771 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-16T14:44:03.772 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-16T14:44:05.908 INFO:teuthology.orchestra.run.smithi125.stdout:linux-firmware-20240109-142.el9.noarch.rpm 11 MB/s | 375 MB 00:32 2024-01-16T14:44:05.909 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:05.910 INFO:teuthology.orchestra.run.smithi125.stdout:Total 11 MB/s | 375 MB 00:32 2024-01-16T14:44:05.910 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2024-01-16T14:44:06.000 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2024-01-16T14:44:06.000 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2024-01-16T14:44:08.417 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2024-01-16T14:44:08.417 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2024-01-16T14:44:09.947 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-16T14:44:10.219 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:10.311 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:11.510 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:11.510 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:44:11.874 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-16T14:44:12.173 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:44:12.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:12.546 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:12.546 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:12.549 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi137.front.sepia.ceph.com to complete... 2024-01-16T14:44:14.787 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2024-01-16T14:44:15.014 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:15.097 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:16.045 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:16.045 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout:Upgraded: 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:44:16.438 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2024-01-16T14:44:16.718 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:44:16.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:17.039 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:17.039 DEBUG:teuthology.orchestra.run.smithi125:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:17.042 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi125.front.sepia.ceph.com to complete... 2024-01-16T14:44:44.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16 14:44:44 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:44:44.406 INFO:teuthology.orchestra.run.smithi137.stderr:FINISHED --2024-01-16 14:44:44-- 2024-01-16T14:44:44.406 INFO:teuthology.orchestra.run.smithi137.stderr:Total wall clock time: 32s 2024-01-16T14:44:44.406 INFO:teuthology.orchestra.run.smithi137.stderr:Downloaded: 1 files, 315M in 32s (10.0 MB/s) 2024-01-16T14:44:44.409 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:44:44.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:44:46.303 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying... ######################################## 2024-01-16T14:44:46.369 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing... ######################################## 2024-01-16T14:44:46.447 INFO:teuthology.orchestra.run.smithi137.stdout:Updating / installing... 2024-01-16T14:44:51.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16 14:44:51 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:44:51.408 INFO:teuthology.orchestra.run.smithi125.stderr:FINISHED --2024-01-16 14:44:51-- 2024-01-16T14:44:51.409 INFO:teuthology.orchestra.run.smithi125.stderr:Total wall clock time: 34s 2024-01-16T14:44:51.409 INFO:teuthology.orchestra.run.smithi125.stderr:Downloaded: 1 files, 315M in 34s (9.24 MB/s) 2024-01-16T14:44:51.412 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:44:51.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:44:53.313 INFO:teuthology.orchestra.run.smithi125.stdout:Verifying... ######################################## 2024-01-16T14:44:53.781 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing... ######################################## 2024-01-16T14:44:53.785 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:44:53.861 INFO:teuthology.orchestra.run.smithi125.stdout:Updating / installing... 2024-01-16T14:45:01.217 INFO:teuthology.orchestra.run.smithi125.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:45:07.508 INFO:teuthology.orchestra.run.smithi137.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:45:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:45:29.500 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:45:29.500 DEBUG:teuthology.orchestra.run.smithi137:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:45:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:45:29.569 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:45:29.570 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:45:29.571 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:45:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:45:29.573 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:45:29.574 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:45:29.575 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:45:29.576 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:45:29.577 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:45:29.578 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:45:29.579 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:45:29.580 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:45:29.581 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:45:29.582 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:45:29.583 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:45:29.584 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:45:29.585 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:45:29.586 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:45:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:45:29.588 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:45:29.589 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:45:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:45:29.591 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:45:29.592 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:45:29.593 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:45:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:45:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:45:29.596 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:45:29.597 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:45:29.598 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:45:29.599 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:45:29.600 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:45:29.601 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:45:29.602 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:45:29.603 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:45:29.604 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:45:29.605 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:45:29.606 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:45:29.607 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:45:29.608 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:45:29.609 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:45:29.610 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:45:29.611 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:45:29.612 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:45:29.613 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:45:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:45:29.615 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:45:29.616 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:45:29.617 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:45:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:45:29.619 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:45:29.620 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:45:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:45:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:45:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:45:29.624 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:45:29.625 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:45:29.626 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:45:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:45:29.628 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:45:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:45:29.630 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:45:29.631 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:45:29.632 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:45:29.633 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:45:29.634 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:45:29.635 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:45:29.636 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:45:29.637 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:45:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:45:29.639 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:45:29.640 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:45:29.641 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:45:29.642 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:45:29.643 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:45:29.644 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:45:29.645 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:45:29.646 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:45:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:45:29.648 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:45:29.649 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:45:29.650 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:45:29.651 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:45:29.652 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:45:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:45:29.654 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:45:29.655 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:45:29.656 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:45:29.657 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:45:29.658 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:45:29.659 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:45:29.660 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:45:29.661 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:45:29.662 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:45:29.663 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:45:29.664 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:45:29.665 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:45:29.666 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:45:29.667 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:45:29.668 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:45:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:45:29.670 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:45:29.671 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:45:29.672 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:45:29.673 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:45:29.674 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:45:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:45:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:45:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:45:29.678 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:45:29.679 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:45:29.680 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:45:29.681 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:45:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:45:29.683 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:45:29.684 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:45:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:45:29.687 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:45:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:45:29.689 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:45:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:45:29.691 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:45:29.692 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:45:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:45:29.694 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:45:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:45:29.696 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:45:29.697 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:45:29.698 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:45:29.698 DEBUG:teuthology.orchestra.run.smithi137:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-16T14:45:29.720 INFO:teuthology.orchestra.run.smithi137.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:45:29.721 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-16T14:45:29.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-01-16T14:45:29.810 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-01-16T14:45:29.811 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:45:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:45:29.813 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:45:29.813 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 6.7.0-g2668949e0971 2024-01-16T14:45:29.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:45:30.418 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-01-16T14:45:31.825 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:45:31.860 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-01-16T14:45:31.862 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-16T14:45:31.877 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.EQyQ7fpAdm 2024-01-16T14:45:31.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EQyQ7fpAdm 2024-01-16T14:45:31.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.EQyQ7fpAdm 2024-01-16T14:45:32.087 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.EQyQ7fpAdm is 6KB 2024-01-16T14:45:32.134 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.EQyQ7fpAdm 2024-01-16T14:45:32.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:45:32.221 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:45:32.221 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:45:32.221 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:45:32.221 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:45:32.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:45:32.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-01-16T14:45:37.262 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi125.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:45:37.262 DEBUG:teuthology.orchestra.run.smithi125:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:37.343 INFO:teuthology.orchestra.run.smithi125.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:37.343 INFO:teuthology.orchestra.run.smithi125.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:45:37.343 INFO:teuthology.orchestra.run.smithi125.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:45:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:45:37.345 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:45:37.346 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:45:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:45:37.348 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:45:37.349 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:45:37.350 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:45:37.351 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:45:37.352 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:45:37.353 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:45:37.354 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:45:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:45:37.356 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:45:37.357 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:45:37.358 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:45:37.359 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:45:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:45:37.361 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:45:37.362 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:45:37.363 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:45:37.364 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:45:37.365 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:45:37.366 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:45:37.367 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:45:37.368 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:45:37.369 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:45:37.370 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:45:37.371 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:45:37.372 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:45:37.373 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:45:37.374 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:45:37.375 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:45:37.376 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:45:37.377 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:45:37.378 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:45:37.379 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:45:37.380 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:45:37.381 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:45:37.382 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:45:37.383 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:45:37.384 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:45:37.385 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:45:37.386 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:45:37.387 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:45:37.388 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:45:37.389 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:45:37.390 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:45:37.391 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:45:37.392 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:45:37.393 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:45:37.394 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:45:37.395 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:45:37.396 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:45:37.397 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:45:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:45:37.399 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:45:37.400 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:45:37.401 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:45:37.402 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:45:37.403 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:45:37.404 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:45:37.405 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:45:37.406 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:45:37.407 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:45:37.408 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:45:37.409 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:45:37.410 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:45:37.411 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:45:37.412 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:45:37.413 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:45:37.414 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:45:37.415 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:45:37.416 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:45:37.417 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:45:37.418 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:45:37.419 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:45:37.420 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:45:37.421 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:45:37.422 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:45:37.423 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:45:37.424 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:45:37.425 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:45:37.426 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:45:37.427 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:45:37.428 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:45:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:45:37.430 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:45:37.431 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:45:37.432 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:45:37.433 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:45:37.434 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:45:37.435 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:45:37.436 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:45:37.437 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:45:37.438 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:45:37.439 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:45:37.440 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:45:37.441 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:45:37.442 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:45:37.443 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:45:37.444 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:45:37.445 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:45:37.446 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:45:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:45:37.448 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:45:37.449 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:45:37.450 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:45:37.451 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:45:37.452 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:45:37.453 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:45:37.454 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:45:37.455 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:45:37.456 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:45:37.457 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:45:37.458 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:45:37.459 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:45:37.460 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:45:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:45:37.462 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:45:37.463 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:45:37.464 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:45:37.465 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:45:37.466 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:45:37.467 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:45:37.468 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:45:37.470 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:45:37.471 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:45:37.472 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:45:37.473 INFO:teuthology.orchestra.run.smithi125.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:45:37.473 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:45:37.473 DEBUG:teuthology.orchestra.run.smithi125:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:37.495 INFO:teuthology.orchestra.run.smithi125.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:45:37.495 INFO:teuthology.orchestra.run.smithi125.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:fi 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:fi 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:37.496 INFO:teuthology.orchestra.run.smithi125.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:45:37.497 INFO:teuthology.orchestra.run.smithi125.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:45:37.497 INFO:teuthology.orchestra.run.smithi125.stdout:fi 2024-01-16T14:45:37.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo rpm -qi grub2-tools 2024-01-16T14:45:37.571 INFO:teuthology.orchestra.run.smithi125.stdout:Name : grub2-tools 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Epoch : 1 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Version : 2.06 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Release : 68.el9 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Architecture: x86_64 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Group : Unspecified 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Size : 8264254 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:License : GPLv3+ 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:45:37.572 INFO:teuthology.orchestra.run.smithi125.stdout:Packager : builder@centos.org 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:Vendor : CentOS 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:Summary : Support tools for GRUB. 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:Description : 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:hardware devices. 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:45:37.573 INFO:teuthology.orchestra.run.smithi125.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:45:37.574 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:45:37.574 INFO:teuthology.task.kernel:Updating grub on smithi125 to boot 6.7.0-g2668949e0971 2024-01-16T14:45:37.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:45:38.227 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2024-01-16T14:45:39.664 INFO:teuthology.orchestra.run.smithi125.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:45:39.699 INFO:teuthology.orchestra.run.smithi125.stderr:done 2024-01-16T14:45:39.706 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2024-01-16T14:45:39.762 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.iRiVvwn4Fn 2024-01-16T14:45:39.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iRiVvwn4Fn 2024-01-16T14:45:39.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.iRiVvwn4Fn 2024-01-16T14:45:39.984 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.iRiVvwn4Fn is 6KB 2024-01-16T14:45:40.033 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.iRiVvwn4Fn 2024-01-16T14:45:40.046 DEBUG:teuthology.orchestra.run.smithi125:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:45:40.112 INFO:teuthology.orchestra.run.smithi125.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:45:40.112 INFO:teuthology.orchestra.run.smithi125.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:45:40.112 INFO:teuthology.orchestra.run.smithi125.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:45:40.112 INFO:teuthology.orchestra.run.smithi125.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:45:40.114 DEBUG:teuthology.orchestra.run.smithi125:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:45:40.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo shutdown -r now 2024-01-16T14:46:02.385 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-16T14:46:02.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:46:02.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:10.266 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T14:46:10.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:46:10.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:20.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-16T14:46:28.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2024-01-16T14:46:29.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:46:29.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:32.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-16T14:46:37.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:46:37.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:40.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2024-01-16T14:46:44.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:46:44.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:52.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:46:52.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:11.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2024-01-16T14:47:26.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:47:26.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:28.653 DEBUG:teuthology.orchestra.run.smithi125:> true 2024-01-16T14:47:29.291 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:47:29.291 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-16T14:47:29.292 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2024-01-16T14:47:29.327 INFO:teuthology.orchestra.run.smithi125.stdout:6.7.0-g2668949e0971 2024-01-16T14:47:29.328 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:47:29.328 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:47:29.328 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:47:30.329 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T14:47:30.329 DEBUG:teuthology.orchestra.run.smithi125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:47:30.474 INFO:teuthology.orchestra.run.smithi125.stdout:ttyS1 2024-01-16T14:47:30.530 DEBUG:teuthology.parallel:result is None 2024-01-16T14:47:44.802 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T14:47:59.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:47:59.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:48:00.202 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-16T14:48:00.919 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:48:00.919 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-16T14:48:00.919 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-16T14:48:00.970 INFO:teuthology.orchestra.run.smithi137.stdout:6.7.0-g2668949e0971 2024-01-16T14:48:00.970 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:48:00.970 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:48:00.970 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:48:01.972 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-16T14:48:01.972 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:48:02.107 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-01-16T14:48:02.151 DEBUG:teuthology.parallel:result is None 2024-01-16T14:48:02.152 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T14:48:02.163 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T14:48:02.163 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:48:02.172 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:48:02.226 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T14:48:02.232 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T14:48:02.239 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T14:48:02.239 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:48:02.246 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:48:02.324 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T14:48:02.330 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T14:48:02.331 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:48:02.337 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:48:02.444 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:02.459 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:02.484 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:02.503 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:02.507 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T14:48:02.515 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T14:48:02.515 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:48:02.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:48:02.618 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T14:48:02.627 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T14:48:02.627 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:48:02.635 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:48:02.690 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:02.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:02.879 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:02.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:03.058 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:48:03.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:48:03.148 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:03.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:03.338 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:03.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:03.516 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:48:03.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:48:03.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo service rsyslog restart 2024-01-16T14:48:03.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-01-16T14:48:03.737 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:48:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:48:04.060 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T14:48:04.069 INFO:teuthology.task.internal:Starting timer... 2024-01-16T14:48:04.069 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T14:48:04.079 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T14:48:04.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo service auditd rotate 2024-01-16T14:48:04.225 INFO:teuthology.orchestra.run.smithi125.stdout:Rotating logs: 2024-01-16T14:48:04.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-01-16T14:48:04.367 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-01-16T14:48:04.372 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T14:48:04.373 DEBUG:teuthology.orchestra.run.smithi125:> /usr/sbin/getenforce 2024-01-16T14:48:04.429 INFO:teuthology.orchestra.run.smithi125.stdout:Permissive 2024-01-16T14:48:04.430 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-01-16T14:48:04.486 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-01-16T14:48:04.487 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi125.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-01-16T14:48:04.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:48:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:48:04.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:48:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:48:04.669 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T14:48:04.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:48:04.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:48:04.867 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T14:48:04.877 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-16T14:48:04.883 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-16T14:48:05.010 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T14:48:05.027 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T14:48:05.028 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi125.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T14:53:40.770 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi125.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-01-16T14:53:40.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:53:40.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:53:40.921 DEBUG:teuthology.orchestra.run.smithi125:> true 2024-01-16T14:53:41.074 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2024-01-16T14:53:41.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:53:41.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:53:41.212 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-16T14:53:41.368 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-16T14:53:41.369 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T14:53:41.379 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T14:53:41.379 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:53:41.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:53:41.388 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:53:41.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:53:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:53:41.541 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:53:41.632 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: ntpd: command not found 2024-01-16T14:53:41.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-01-16T14:53:41.672 INFO:teuthology.orchestra.run.smithi125.stdout:506 Cannot talk to daemon 2024-01-16T14:53:41.675 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-01-16T14:53:41.722 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:53:41.727 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:53:41.772 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:53:41.782 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi125.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi125.stdout:=============================================================================== 2024-01-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:53:41.915 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:53:41.915 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-01-16T14:53:41.915 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:41.917 INFO:teuthology.run_tasks:Running task install... 2024-01-16T14:53:41.925 DEBUG:teuthology.task.install:project ceph 2024-01-16T14:53:41.925 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}} 2024-01-16T14:53:41.925 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'} 2024-01-16T14:53:41.926 INFO:teuthology.task.install:Using flavor: default 2024-01-16T14:53:41.931 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T14:53:41.931 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-16T14:53:41.932 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:53:41.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:53:41.934 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:53:41.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:53:42.184 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:53:42.184 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:53:42.185 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:53:42.185 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:53:42.386 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:53:42.386 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:53:42.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:53:42.389 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:53:42.389 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:53:42.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:53:42.487 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:53:42.488 DEBUG:teuthology.orchestra.run.smithi125:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:53:42.495 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:53:42.495 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:53:42.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:53:42.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:53:42.749 DEBUG:teuthology.orchestra.run.smithi125:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:53:42.765 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:53:42.838 INFO:teuthology.orchestra.run.smithi125.stdout:check_obsoletes = 1 2024-01-16T14:53:42.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum clean all 2024-01-16T14:53:42.855 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-01-16T14:53:42.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-01-16T14:53:43.201 INFO:teuthology.orchestra.run.smithi125.stdout:73 files removed 2024-01-16T14:53:43.223 INFO:teuthology.orchestra.run.smithi137.stdout:73 files removed 2024-01-16T14:53:43.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:53:43.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:53:44.213 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 174 kB/s | 78 kB 00:00 2024-01-16T14:53:44.254 INFO:teuthology.orchestra.run.smithi125.stdout:ceph packages for x86_64 156 kB/s | 78 kB 00:00 2024-01-16T14:53:44.664 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 26 kB/s | 9.5 kB 00:00 2024-01-16T14:53:44.963 INFO:teuthology.orchestra.run.smithi125.stdout:ceph noarch packages 15 kB/s | 9.5 kB 00:00 2024-01-16T14:53:45.348 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-01-16T14:53:45.563 INFO:teuthology.orchestra.run.smithi125.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-01-16T14:53:46.260 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.4 MB 00:00 2024-01-16T14:53:46.433 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.4 MB 00:00 2024-01-16T14:53:48.061 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-01-16T14:53:48.160 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-01-16T14:53:49.151 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-01-16T14:53:50.097 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-01-16T14:53:53.421 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-01-16T14:53:53.973 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-01-16T14:53:57.883 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 5.9 MB 00:00 2024-01-16T14:53:59.248 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.9 MB 00:01 2024-01-16T14:53:59.717 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-01-16T14:54:00.790 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2024-01-16T14:54:02.166 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 20 MB 00:02 2024-01-16T14:54:02.292 INFO:teuthology.orchestra.run.smithi125.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-01-16T14:54:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-01-16T14:54:07.884 INFO:teuthology.orchestra.run.smithi125.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-16T14:54:10.464 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.466 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.487 INFO:teuthology.orchestra.run.smithi137.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:54:10.489 INFO:teuthology.orchestra.run.smithi137.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.490 INFO:teuthology.orchestra.run.smithi137.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:54:10.492 INFO:teuthology.orchestra.run.smithi137.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.494 INFO:teuthology.orchestra.run.smithi137.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.498 INFO:teuthology.orchestra.run.smithi137.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.499 INFO:teuthology.orchestra.run.smithi137.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:54:10.581 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-16T14:54:10.585 INFO:teuthology.orchestra.run.smithi125.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.587 INFO:teuthology.orchestra.run.smithi125.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.607 INFO:teuthology.orchestra.run.smithi125.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:54:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:54:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:54:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:54:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:54:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:54:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:54:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:54:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:54:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:54:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:54:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-16T14:54:10.619 INFO:teuthology.orchestra.run.smithi137.stdout:Install 135 Packages 2024-01-16T14:54:10.620 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 16 Packages 2024-01-16T14:54:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:54:10.620 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 204 M 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:54:10.621 INFO:teuthology.orchestra.run.smithi125.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:54:10.698 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================ 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================ 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:54:10.726 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:54:10.727 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:54:10.728 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrading: 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:54:10.729 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:54:10.730 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:54:10.731 INFO:teuthology.orchestra.run.smithi125.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:54:10.732 INFO:teuthology.orchestra.run.smithi125.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:54:10.733 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:54:10.734 INFO:teuthology.orchestra.run.smithi125.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:54:10.735 INFO:teuthology.orchestra.run.smithi125.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:54:10.736 INFO:teuthology.orchestra.run.smithi125.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================ 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout:Install 135 Packages 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrade 16 Packages 2024-01-16T14:54:10.737 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:54:10.738 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 204 M 2024-01-16T14:54:10.738 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2024-01-16T14:54:11.486 INFO:teuthology.orchestra.run.smithi137.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-01-16T14:54:11.677 INFO:teuthology.orchestra.run.smithi125.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-01-16T14:54:11.812 INFO:teuthology.orchestra.run.smithi137.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 2.6 MB/s | 861 kB 00:00 2024-01-16T14:54:12.143 INFO:teuthology.orchestra.run.smithi125.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 1.8 MB/s | 861 kB 00:00 2024-01-16T14:54:12.175 INFO:teuthology.orchestra.run.smithi137.stdout:(3/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 5.1 MB/s | 5.2 MB 00:01 2024-01-16T14:54:12.212 INFO:teuthology.orchestra.run.smithi137.stdout:(4/151): ceph-immutable-object-cache-18.0.0-762 353 kB/s | 141 kB 00:00 2024-01-16T14:54:12.218 INFO:teuthology.orchestra.run.smithi125.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.9 MB/s | 141 kB 00:00 2024-01-16T14:54:12.421 INFO:teuthology.orchestra.run.smithi137.stdout:(5/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 8.5 MB/s | 2.1 MB 00:00 2024-01-16T14:54:12.830 INFO:teuthology.orchestra.run.smithi137.stdout:(6/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-01-16T14:54:12.946 INFO:teuthology.orchestra.run.smithi137.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 8.4 MB/s | 4.4 MB 00:00 2024-01-16T14:54:13.069 INFO:teuthology.orchestra.run.smithi125.stdout:(4/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 2.5 MB/s | 2.1 MB 00:00 2024-01-16T14:54:13.611 INFO:teuthology.orchestra.run.smithi125.stdout:(5/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-01-16T14:54:13.688 INFO:teuthology.orchestra.run.smithi137.stdout:(8/151): ceph-common-18.0.0-7629.g73a4532f.el9. 7.5 MB/s | 19 MB 00:02 2024-01-16T14:54:13.872 INFO:teuthology.orchestra.run.smithi137.stdout:(9/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 9.3 MB/s | 8.6 MB 00:00 2024-01-16T14:54:13.916 INFO:teuthology.orchestra.run.smithi137.stdout:(10/151): ceph-selinux-18.0.0-7629.g73a4532f.el 111 kB/s | 25 kB 00:00 2024-01-16T14:54:13.944 INFO:teuthology.orchestra.run.smithi125.stdout:(6/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 2.1 MB/s | 5.2 MB 00:02 2024-01-16T14:54:14.153 INFO:teuthology.orchestra.run.smithi125.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 8.1 MB/s | 4.4 MB 00:00 2024-01-16T14:54:14.205 INFO:teuthology.orchestra.run.smithi137.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 754 kB/s | 217 kB 00:00 2024-01-16T14:54:14.369 INFO:teuthology.orchestra.run.smithi137.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 192 kB/s | 31 kB 00:00 2024-01-16T14:54:14.706 INFO:teuthology.orchestra.run.smithi137.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 2.1 MB/s | 719 kB 00:00 2024-01-16T14:54:14.831 INFO:teuthology.orchestra.run.smithi137.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 1.3 MB/s | 163 kB 00:00 2024-01-16T14:54:14.940 INFO:teuthology.orchestra.run.smithi137.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 1.2 MB/s | 127 kB 00:00 2024-01-16T14:54:15.173 INFO:teuthology.orchestra.run.smithi137.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 2.0 MB/s | 476 kB 00:00 2024-01-16T14:54:15.419 INFO:teuthology.orchestra.run.smithi125.stdout:(8/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 6.8 MB/s | 8.6 MB 00:01 2024-01-16T14:54:15.503 INFO:teuthology.orchestra.run.smithi125.stdout:(9/151): ceph-selinux-18.0.0-7629.g73a4532f.el9 305 kB/s | 25 kB 00:00 2024-01-16T14:54:16.041 INFO:teuthology.orchestra.run.smithi137.stdout:(17/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 5.5 MB/s | 4.8 MB 00:00 2024-01-16T14:54:16.174 INFO:teuthology.orchestra.run.smithi137.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 341 kB/s | 45 kB 00:00 2024-01-16T14:54:16.333 INFO:teuthology.orchestra.run.smithi137.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 819 kB/s | 129 kB 00:00 2024-01-16T14:54:16.449 INFO:teuthology.orchestra.run.smithi137.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 1.4 MB/s | 161 kB 00:00 2024-01-16T14:54:16.608 INFO:teuthology.orchestra.run.smithi137.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 2.0 MB/s | 321 kB 00:00 2024-01-16T14:54:16.742 INFO:teuthology.orchestra.run.smithi137.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 2.2 MB/s | 298 kB 00:00 2024-01-16T14:54:16.892 INFO:teuthology.orchestra.run.smithi137.stdout:(23/151): python3-rgw-18.0.0-7629.g73a4532f.el9 665 kB/s | 99 kB 00:00 2024-01-16T14:54:17.034 INFO:teuthology.orchestra.run.smithi137.stdout:(24/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 605 kB/s | 85 kB 00:00 2024-01-16T14:54:17.309 INFO:teuthology.orchestra.run.smithi137.stdout:(25/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 4.0 MB/s | 18 MB 00:04 2024-01-16T14:54:17.868 INFO:teuthology.orchestra.run.smithi137.stdout:(26/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 10 MB/s | 41 MB 00:03 2024-01-16T14:54:17.918 INFO:teuthology.orchestra.run.smithi137.stdout:(27/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 281 kB/s | 171 kB 00:00 2024-01-16T14:54:17.952 INFO:teuthology.orchestra.run.smithi137.stdout:(28/151): ceph-grafana-dashboards-18.0.0-7629.g 340 kB/s | 28 kB 00:00 2024-01-16T14:54:18.035 INFO:teuthology.orchestra.run.smithi137.stdout:(29/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 3.0 MB/s | 3.0 MB 00:01 2024-01-16T14:54:18.061 INFO:teuthology.orchestra.run.smithi137.stdout:(30/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 938 kB/s | 133 kB 00:00 2024-01-16T14:54:18.203 INFO:teuthology.orchestra.run.smithi137.stdout:(31/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 7.0 MB/s | 1.8 MB 00:00 2024-01-16T14:54:18.278 INFO:teuthology.orchestra.run.smithi137.stdout:(32/151): ceph-mgr-modules-core-18.0.0-7629.g73 1.1 MB/s | 246 kB 00:00 2024-01-16T14:54:18.303 INFO:teuthology.orchestra.run.smithi137.stdout:(33/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 505 kB/s | 50 kB 00:00 2024-01-16T14:54:18.370 INFO:teuthology.orchestra.run.smithi137.stdout:(34/151): ceph-prometheus-alerts-18.0.0-7629.g7 166 kB/s | 15 kB 00:00 2024-01-16T14:54:18.421 INFO:teuthology.orchestra.run.smithi137.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 2.2 MB/s | 262 kB 00:00 2024-01-16T14:54:18.521 INFO:teuthology.orchestra.run.smithi137.stdout:(36/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 184 kB/s | 18 kB 00:00 2024-01-16T14:54:18.697 INFO:teuthology.orchestra.run.smithi137.stdout:(37/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 2.3 MB/s | 778 kB 00:00 2024-01-16T14:54:18.889 INFO:teuthology.orchestra.run.smithi137.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-01-16T14:54:18.930 INFO:teuthology.orchestra.run.smithi137.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-01-16T14:54:18.955 INFO:teuthology.orchestra.run.smithi137.stdout:(40/151): boost-program-options-1.75.0-8.el9.x8 1.7 MB/s | 106 kB 00:00 2024-01-16T14:54:18.972 INFO:teuthology.orchestra.run.smithi137.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 508 kB/s | 20 kB 00:00 2024-01-16T14:54:19.023 INFO:teuthology.orchestra.run.smithi137.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-01-16T14:54:19.040 INFO:teuthology.orchestra.run.smithi137.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 476 kB/s | 32 kB 00:00 2024-01-16T14:54:19.149 INFO:teuthology.orchestra.run.smithi137.stdout:(44/151): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-01-16T14:54:19.257 INFO:teuthology.orchestra.run.smithi137.stdout:(45/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-01-16T14:54:19.261 INFO:teuthology.orchestra.run.smithi125.stdout:(10/151): ceph-common-18.0.0-7629.g73a4532f.el9 2.4 MB/s | 19 MB 00:07 2024-01-16T14:54:19.274 INFO:teuthology.orchestra.run.smithi137.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 84 kB/s | 10 kB 00:00 2024-01-16T14:54:19.408 INFO:teuthology.orchestra.run.smithi137.stdout:(47/151): ceph-mgr-diskprediction-local-18.0.0- 5.4 MB/s | 7.4 MB 00:01 2024-01-16T14:54:19.450 INFO:teuthology.orchestra.run.smithi137.stdout:(48/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 831 kB/s | 159 kB 00:00 2024-01-16T14:54:19.467 INFO:teuthology.orchestra.run.smithi137.stdout:(49/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 230 kB/s | 44 kB 00:00 2024-01-16T14:54:19.508 INFO:teuthology.orchestra.run.smithi137.stdout:(50/151): librdkafka-1.6.1-102.el9.x86_64.rpm 6.6 MB/s | 661 kB 00:00 2024-01-16T14:54:19.526 INFO:teuthology.orchestra.run.smithi137.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 463 kB/s | 24 kB 00:00 2024-01-16T14:54:19.584 INFO:teuthology.orchestra.run.smithi137.stdout:(52/151): libtool-2.4.6-45.el9.x86_64.rpm 5.1 MB/s | 578 kB 00:00 2024-01-16T14:54:19.601 INFO:teuthology.orchestra.run.smithi137.stdout:(53/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 506 kB/s | 46 kB 00:00 2024-01-16T14:54:19.626 INFO:teuthology.orchestra.run.smithi137.stdout:(54/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-01-16T14:54:19.676 INFO:teuthology.orchestra.run.smithi137.stdout:(55/151): lua-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 187 kB 00:00 2024-01-16T14:54:19.702 INFO:teuthology.orchestra.run.smithi137.stdout:(56/151): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 34 kB 00:00 2024-01-16T14:54:19.753 INFO:teuthology.orchestra.run.smithi125.stdout:(11/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 9.7 MB/s | 41 MB 00:04 2024-01-16T14:54:19.787 INFO:teuthology.orchestra.run.smithi125.stdout:(12/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 414 kB/s | 217 kB 00:00 2024-01-16T14:54:19.812 INFO:teuthology.orchestra.run.smithi125.stdout:(13/151): libcephfs-devel-18.0.0-7629.g73a4532f 539 kB/s | 31 kB 00:00 2024-01-16T14:54:19.879 INFO:teuthology.orchestra.run.smithi125.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 2.4 MB/s | 163 kB 00:00 2024-01-16T14:54:19.929 INFO:teuthology.orchestra.run.smithi125.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 2.5 MB/s | 127 kB 00:00 2024-01-16T14:54:19.963 INFO:teuthology.orchestra.run.smithi125.stdout:(16/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 4.0 MB/s | 719 kB 00:00 2024-01-16T14:54:19.977 INFO:teuthology.orchestra.run.smithi137.stdout:(57/151): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-01-16T14:54:20.022 INFO:teuthology.orchestra.run.smithi125.stdout:(17/151): libradosstriper1-18.0.0-7629.g73a4532 5.1 MB/s | 476 kB 00:00 2024-01-16T14:54:20.072 INFO:teuthology.orchestra.run.smithi125.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 913 kB/s | 45 kB 00:00 2024-01-16T14:54:20.122 INFO:teuthology.orchestra.run.smithi125.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 2.6 MB/s | 129 kB 00:00 2024-01-16T14:54:20.127 INFO:teuthology.orchestra.run.smithi137.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 3.6 MB/s | 1.5 MB 00:00 2024-01-16T14:54:20.164 INFO:teuthology.orchestra.run.smithi125.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 3.9 MB/s | 161 kB 00:00 2024-01-16T14:54:20.203 INFO:teuthology.orchestra.run.smithi137.stdout:(59/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 5.7 MB/s | 3.0 MB 00:00 2024-01-16T14:54:20.223 INFO:teuthology.orchestra.run.smithi125.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 5.4 MB/s | 321 kB 00:00 2024-01-16T14:54:20.236 INFO:teuthology.orchestra.run.smithi137.stdout:(60/151): postgresql-contrib-13.11-1.el9.x86_64 3.2 MB/s | 811 kB 00:00 2024-01-16T14:54:20.253 INFO:teuthology.orchestra.run.smithi137.stdout:(61/151): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-01-16T14:54:20.282 INFO:teuthology.orchestra.run.smithi125.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 5.0 MB/s | 298 kB 00:00 2024-01-16T14:54:20.524 INFO:teuthology.orchestra.run.smithi125.stdout:(23/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 2.7 MB/s | 18 MB 00:06 2024-01-16T14:54:20.557 INFO:teuthology.orchestra.run.smithi125.stdout:(24/151): python3-rgw-18.0.0-7629.g73a4532f.el9 362 kB/s | 99 kB 00:00 2024-01-16T14:54:20.582 INFO:teuthology.orchestra.run.smithi125.stdout:(25/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 1.4 MB/s | 85 kB 00:00 2024-01-16T14:54:20.611 INFO:teuthology.orchestra.run.smithi137.stdout:(62/151): python3-jinja2-2.11.3-4.el9.noarch.rp 641 kB/s | 228 kB 00:00 2024-01-16T14:54:20.658 INFO:teuthology.orchestra.run.smithi125.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.2 MB/s | 171 kB 00:00 2024-01-16T14:54:20.700 INFO:teuthology.orchestra.run.smithi125.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 687 kB/s | 28 kB 00:00 2024-01-16T14:54:20.737 INFO:teuthology.orchestra.run.smithi137.stdout:(63/151): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-01-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi125.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 3.2 MB/s | 133 kB 00:00 2024-01-16T14:54:20.850 INFO:teuthology.orchestra.run.smithi125.stdout:(29/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 5.4 MB/s | 4.8 MB 00:00 2024-01-16T14:54:20.870 INFO:teuthology.orchestra.run.smithi137.stdout:(64/151): postgresql-server-13.11-1.el9.x86_64. 9.0 MB/s | 5.7 MB 00:00 2024-01-16T14:54:20.895 INFO:teuthology.orchestra.run.smithi137.stdout:(65/151): python3-jsonpatch-1.21-16.el9.noarch. 87 kB/s | 24 kB 00:00 2024-01-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi137.stdout:(66/151): python3-jsonpointer-2.0-4.el9.noarch. 106 kB/s | 18 kB 00:00 2024-01-16T14:54:20.934 INFO:teuthology.orchestra.run.smithi125.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 9.2 MB/s | 1.8 MB 00:00 2024-01-16T14:54:20.993 INFO:teuthology.orchestra.run.smithi125.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 4.2 MB/s | 246 kB 00:00 2024-01-16T14:54:21.004 INFO:teuthology.orchestra.run.smithi137.stdout:(67/151): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-01-16T14:54:21.034 INFO:teuthology.orchestra.run.smithi125.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 1.2 MB/s | 50 kB 00:00 2024-01-16T14:54:21.046 INFO:teuthology.orchestra.run.smithi137.stdout:(68/151): python3-markupsafe-1.1.1-12.el9.x86_6 255 kB/s | 32 kB 00:00 2024-01-16T14:54:21.076 INFO:teuthology.orchestra.run.smithi125.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g7 371 kB/s | 15 kB 00:00 2024-01-16T14:54:21.152 INFO:teuthology.orchestra.run.smithi125.stdout:(34/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 5.1 MB/s | 3.0 MB 00:00 2024-01-16T14:54:21.185 INFO:teuthology.orchestra.run.smithi125.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 2.4 MB/s | 262 kB 00:00 2024-01-16T14:54:21.196 INFO:teuthology.orchestra.run.smithi137.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-01-16T14:54:21.230 INFO:teuthology.orchestra.run.smithi137.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 986 kB/s | 217 kB 00:00 2024-01-16T14:54:21.235 INFO:teuthology.orchestra.run.smithi125.stdout:(36/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 372 kB/s | 18 kB 00:00 2024-01-16T14:54:21.255 INFO:teuthology.orchestra.run.smithi137.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 882 kB/s | 174 kB 00:00 2024-01-16T14:54:21.280 INFO:teuthology.orchestra.run.smithi137.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 514 kB/s | 41 kB 00:00 2024-01-16T14:54:21.314 INFO:teuthology.orchestra.run.smithi137.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-01-16T14:54:21.339 INFO:teuthology.orchestra.run.smithi137.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-01-16T14:54:21.344 INFO:teuthology.orchestra.run.smithi125.stdout:(37/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 4.0 MB/s | 778 kB 00:00 2024-01-16T14:54:21.364 INFO:teuthology.orchestra.run.smithi137.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 570 kB/s | 47 kB 00:00 2024-01-16T14:54:21.422 INFO:teuthology.orchestra.run.smithi137.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 528 kB/s | 43 kB 00:00 2024-01-16T14:54:21.648 INFO:teuthology.orchestra.run.smithi137.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 139 kB/s | 39 kB 00:00 2024-01-16T14:54:21.682 INFO:teuthology.orchestra.run.smithi137.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-01-16T14:54:21.882 INFO:teuthology.orchestra.run.smithi137.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 245 kB/s | 56 kB 00:00 2024-01-16T14:54:21.928 INFO:teuthology.orchestra.run.smithi125.stdout:(38/151): ceph-mgr-diskprediction-local-18.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-01-16T14:54:22.036 INFO:teuthology.orchestra.run.smithi125.stdout:(39/151): automake-1.16.2-8.el9.noarch.rpm 828 kB/s | 662 kB 00:00 2024-01-16T14:54:22.078 INFO:teuthology.orchestra.run.smithi125.stdout:(40/151): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 920 kB 00:00 2024-01-16T14:54:22.099 INFO:teuthology.orchestra.run.smithi137.stdout:(80/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-01-16T14:54:22.103 INFO:teuthology.orchestra.run.smithi125.stdout:(41/151): boost-program-options-1.75.0-8.el9.x8 615 kB/s | 106 kB 00:00 2024-01-16T14:54:22.124 INFO:teuthology.orchestra.run.smithi137.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 203 kB/s | 87 kB 00:00 2024-01-16T14:54:22.129 INFO:teuthology.orchestra.run.smithi125.stdout:(42/151): elfutils-libelf-devel-0.190-2.el9.x86 226 kB/s | 20 kB 00:00 2024-01-16T14:54:22.150 INFO:teuthology.orchestra.run.smithi137.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 242 kB/s | 63 kB 00:00 2024-01-16T14:54:22.171 INFO:teuthology.orchestra.run.smithi125.stdout:(43/151): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-01-16T14:54:22.175 INFO:teuthology.orchestra.run.smithi137.stdout:(83/151): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-01-16T14:54:22.196 INFO:teuthology.orchestra.run.smithi125.stdout:(44/151): flexiblas-3.0.4-8.el9.x86_64.rpm 496 kB/s | 32 kB 00:00 2024-01-16T14:54:22.200 INFO:teuthology.orchestra.run.smithi137.stdout:(84/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 594 kB/s | 44 kB 00:00 2024-01-16T14:54:22.225 INFO:teuthology.orchestra.run.smithi137.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.5 MB/s | 186 kB 00:00 2024-01-16T14:54:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:(86/151): libgfortran-11.4.1-3.el9.x86_64.rpm 8.8 MB/s | 802 kB 00:00 2024-01-16T14:54:22.304 INFO:teuthology.orchestra.run.smithi125.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 131 kB/s | 17 kB 00:00 2024-01-16T14:54:22.321 INFO:teuthology.orchestra.run.smithi137.stdout:(87/151): libquadmath-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 194 kB 00:00 2024-01-16T14:54:22.330 INFO:teuthology.orchestra.run.smithi125.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 76 kB/s | 10 kB 00:00 2024-01-16T14:54:22.351 INFO:teuthology.orchestra.run.smithi137.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-01-16T14:54:22.385 INFO:teuthology.orchestra.run.smithi137.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 241 kB 00:00 2024-01-16T14:54:22.410 INFO:teuthology.orchestra.run.smithi137.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-01-16T14:54:22.460 INFO:teuthology.orchestra.run.smithi137.stdout:(91/151): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-01-16T14:54:22.480 INFO:teuthology.orchestra.run.smithi125.stdout:(47/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.6 MB/s | 3.0 MB 00:00 2024-01-16T14:54:22.486 INFO:teuthology.orchestra.run.smithi137.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-01-16T14:54:22.511 INFO:teuthology.orchestra.run.smithi137.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 1.2 MB/s | 113 kB 00:00 2024-01-16T14:54:22.514 INFO:teuthology.orchestra.run.smithi125.stdout:(48/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 772 kB/s | 159 kB 00:00 2024-01-16T14:54:22.539 INFO:teuthology.orchestra.run.smithi125.stdout:(49/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 216 kB/s | 44 kB 00:00 2024-01-16T14:54:22.544 INFO:teuthology.orchestra.run.smithi137.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-16T14:54:22.581 INFO:teuthology.orchestra.run.smithi125.stdout:(50/151): librdkafka-1.6.1-102.el9.x86_64.rpm 6.6 MB/s | 661 kB 00:00 2024-01-16T14:54:22.615 INFO:teuthology.orchestra.run.smithi125.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 357 kB/s | 24 kB 00:00 2024-01-16T14:54:22.648 INFO:teuthology.orchestra.run.smithi125.stdout:(52/151): libtool-2.4.6-45.el9.x86_64.rpm 5.5 MB/s | 578 kB 00:00 2024-01-16T14:54:22.699 INFO:teuthology.orchestra.run.smithi125.stdout:(53/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 406 kB/s | 46 kB 00:00 2024-01-16T14:54:22.732 INFO:teuthology.orchestra.run.smithi125.stdout:(54/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-01-16T14:54:22.766 INFO:teuthology.orchestra.run.smithi125.stdout:(55/151): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-01-16T14:54:22.791 INFO:teuthology.orchestra.run.smithi125.stdout:(56/151): openblas-0.3.21-2.el9.x86_64.rpm 386 kB/s | 34 kB 00:00 2024-01-16T14:54:23.025 INFO:teuthology.orchestra.run.smithi125.stdout:(57/151): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-01-16T14:54:23.045 INFO:teuthology.orchestra.run.smithi137.stdout:(95/151): python3-devel-3.9.17-2.el9.x86_64.rpm 441 kB/s | 245 kB 00:00 2024-01-16T14:54:23.099 INFO:teuthology.orchestra.run.smithi125.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 5.1 MB/s | 1.5 MB 00:00 2024-01-16T14:54:23.112 INFO:teuthology.orchestra.run.smithi137.stdout:(96/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 545 kB/s | 309 kB 00:00 2024-01-16T14:54:23.183 INFO:teuthology.orchestra.run.smithi125.stdout:(59/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-01-16T14:54:23.217 INFO:teuthology.orchestra.run.smithi125.stdout:(60/151): postgresql-contrib-13.11-1.el9.x86_64 4.3 MB/s | 811 kB 00:00 2024-01-16T14:54:23.237 INFO:teuthology.orchestra.run.smithi137.stdout:(97/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-01-16T14:54:23.242 INFO:teuthology.orchestra.run.smithi125.stdout:(61/151): postgresql-private-libs-13.11-1.el9.x 1.0 MB/s | 134 kB 00:00 2024-01-16T14:54:23.262 INFO:teuthology.orchestra.run.smithi137.stdout:(98/151): lua-devel-5.4.4-4.el9.x86_64.rpm 30 kB/s | 22 kB 00:00 2024-01-16T14:54:23.354 INFO:teuthology.orchestra.run.smithi137.stdout:(99/151): liboath-2.6.7-2.el9.x86_64.rpm 424 kB/s | 49 kB 00:00 2024-01-16T14:54:23.496 INFO:teuthology.orchestra.run.smithi137.stdout:(100/151): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-01-16T14:54:23.534 INFO:teuthology.orchestra.run.smithi125.stdout:(62/151): postgresql-server-13.11-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-01-16T14:54:23.688 INFO:teuthology.orchestra.run.smithi137.stdout:(101/151): libunwind-1.6.2-1.el9.x86_64.rpm 158 kB/s | 67 kB 00:00 2024-01-16T14:54:23.726 INFO:teuthology.orchestra.run.smithi125.stdout:(63/151): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-01-16T14:54:23.751 INFO:teuthology.orchestra.run.smithi125.stdout:(64/151): python3-jinja2-2.11.3-4.el9.noarch.rp 461 kB/s | 228 kB 00:00 2024-01-16T14:54:23.788 INFO:teuthology.orchestra.run.smithi137.stdout:(102/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-01-16T14:54:23.818 INFO:teuthology.orchestra.run.smithi125.stdout:(65/151): python3-jsonpatch-1.21-16.el9.noarch. 374 kB/s | 24 kB 00:00 2024-01-16T14:54:23.843 INFO:teuthology.orchestra.run.smithi125.stdout:(66/151): python3-jsonpointer-2.0-4.el9.noarch. 200 kB/s | 18 kB 00:00 2024-01-16T14:54:23.860 INFO:teuthology.orchestra.run.smithi125.stdout:(67/151): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-01-16T14:54:23.910 INFO:teuthology.orchestra.run.smithi125.stdout:(68/151): python3-markupsafe-1.1.1-12.el9.x86_6 486 kB/s | 32 kB 00:00 2024-01-16T14:54:23.922 INFO:teuthology.orchestra.run.smithi137.stdout:(103/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 328 kB/s | 43 kB 00:00 2024-01-16T14:54:24.064 INFO:teuthology.orchestra.run.smithi137.stdout:(104/151): python3-cachetools-4.2.4-1.el9.noarc 228 kB/s | 32 kB 00:00 2024-01-16T14:54:24.169 INFO:teuthology.orchestra.run.smithi125.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-01-16T14:54:24.203 INFO:teuthology.orchestra.run.smithi125.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 636 kB/s | 217 kB 00:00 2024-01-16T14:54:24.228 INFO:teuthology.orchestra.run.smithi125.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 574 kB/s | 174 kB 00:00 2024-01-16T14:54:24.247 INFO:teuthology.orchestra.run.smithi137.stdout:(105/151): libarrow-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-01-16T14:54:24.253 INFO:teuthology.orchestra.run.smithi125.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 517 kB/s | 41 kB 00:00 2024-01-16T14:54:24.273 INFO:teuthology.orchestra.run.smithi137.stdout:(106/151): python3-certifi-2021.10.8-2.el9.noar 72 kB/s | 15 kB 00:00 2024-01-16T14:54:24.287 INFO:teuthology.orchestra.run.smithi125.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-01-16T14:54:24.306 INFO:teuthology.orchestra.run.smithi137.stdout:(107/151): python3-asyncssh-2.13.2-3.el9.noarch 887 kB/s | 548 kB 00:00 2024-01-16T14:54:24.320 INFO:teuthology.orchestra.run.smithi125.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-01-16T14:54:24.345 INFO:teuthology.orchestra.run.smithi125.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 516 kB/s | 47 kB 00:00 2024-01-16T14:54:24.356 INFO:teuthology.orchestra.run.smithi137.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 1.6 MB/s | 172 kB 00:00 2024-01-16T14:54:24.371 INFO:teuthology.orchestra.run.smithi125.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 530 kB/s | 43 kB 00:00 2024-01-16T14:54:24.390 INFO:teuthology.orchestra.run.smithi137.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-01-16T14:54:24.396 INFO:teuthology.orchestra.run.smithi125.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 793 kB/s | 39 kB 00:00 2024-01-16T14:54:24.465 INFO:teuthology.orchestra.run.smithi137.stdout:(110/151): python3-google-auth-2.26.0-1.el9.noa 1.2 MB/s | 202 kB 00:00 2024-01-16T14:54:24.491 INFO:teuthology.orchestra.run.smithi137.stdout:(111/151): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-01-16T14:54:24.616 INFO:teuthology.orchestra.run.smithi137.stdout:(112/151): python3-jaraco-collections-3.0.0-8.e 154 kB/s | 23 kB 00:00 2024-01-16T14:54:24.650 INFO:teuthology.orchestra.run.smithi137.stdout:(113/151): python3-jaraco-functools-3.5.0-2.el9 123 kB/s | 19 kB 00:00 2024-01-16T14:54:24.675 INFO:teuthology.orchestra.run.smithi137.stdout:(114/151): python3-jaraco-classes-3.2.1-5.el9.n 62 kB/s | 18 kB 00:00 2024-01-16T14:54:24.734 INFO:teuthology.orchestra.run.smithi137.stdout:(115/151): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-01-16T14:54:24.754 INFO:teuthology.orchestra.run.smithi125.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 785 kB/s | 300 kB 00:00 2024-01-16T14:54:24.759 INFO:teuthology.orchestra.run.smithi137.stdout:(116/151): python3-jwt+crypto-2.4.0-1.el9.noarc 84 kB/s | 9.0 kB 00:00 2024-01-16T14:54:24.784 INFO:teuthology.orchestra.run.smithi137.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 378 kB/s | 41 kB 00:00 2024-01-16T14:54:24.788 INFO:teuthology.orchestra.run.smithi125.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 149 kB/s | 56 kB 00:00 2024-01-16T14:54:24.868 INFO:teuthology.orchestra.run.smithi137.stdout:(118/151): python3-logutils-0.3.5-21.el9.noarch 429 kB/s | 46 kB 00:00 2024-01-16T14:54:24.893 INFO:teuthology.orchestra.run.smithi137.stdout:(119/151): python3-more-itertools-8.12.0-2.el9. 729 kB/s | 79 kB 00:00 2024-01-16T14:54:24.943 INFO:teuthology.orchestra.run.smithi137.stdout:(120/151): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-01-16T14:54:25.002 INFO:teuthology.orchestra.run.smithi137.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 434 kB/s | 58 kB 00:00 2024-01-16T14:54:25.005 INFO:teuthology.orchestra.run.smithi125.stdout:(80/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-01-16T14:54:25.030 INFO:teuthology.orchestra.run.smithi125.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 316 kB/s | 87 kB 00:00 2024-01-16T14:54:25.052 INFO:teuthology.orchestra.run.smithi137.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-01-16T14:54:25.055 INFO:teuthology.orchestra.run.smithi125.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 236 kB/s | 63 kB 00:00 2024-01-16T14:54:25.077 INFO:teuthology.orchestra.run.smithi137.stdout:(123/151): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-01-16T14:54:25.094 INFO:teuthology.orchestra.run.smithi137.stdout:(124/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 982 kB/s | 90 kB 00:00 2024-01-16T14:54:25.098 INFO:teuthology.orchestra.run.smithi125.stdout:(83/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-01-16T14:54:25.123 INFO:teuthology.orchestra.run.smithi125.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-01-16T14:54:25.148 INFO:teuthology.orchestra.run.smithi125.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-01-16T14:54:25.161 INFO:teuthology.orchestra.run.smithi137.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-01-16T14:54:25.190 INFO:teuthology.orchestra.run.smithi125.stdout:(86/151): libgfortran-11.4.1-3.el9.x86_64.rpm 8.8 MB/s | 802 kB 00:00 2024-01-16T14:54:25.195 INFO:teuthology.orchestra.run.smithi137.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-01-16T14:54:25.220 INFO:teuthology.orchestra.run.smithi137.stdout:(127/151): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-01-16T14:54:25.224 INFO:teuthology.orchestra.run.smithi125.stdout:(87/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-01-16T14:54:25.257 INFO:teuthology.orchestra.run.smithi125.stdout:(88/151): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 194 kB 00:00 2024-01-16T14:54:25.290 INFO:teuthology.orchestra.run.smithi125.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-01-16T14:54:25.304 INFO:teuthology.orchestra.run.smithi137.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-01-16T14:54:25.337 INFO:teuthology.orchestra.run.smithi137.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 360 kB/s | 51 kB 00:00 2024-01-16T14:54:25.399 INFO:teuthology.orchestra.run.smithi125.stdout:(90/151): python3-cryptography-36.0.1-4.el9.x86 6.6 MB/s | 1.1 MB 00:00 2024-01-16T14:54:25.424 INFO:teuthology.orchestra.run.smithi125.stdout:(91/151): python3-ply-3.11-14.el9.noarch.rpm 625 kB/s | 103 kB 00:00 2024-01-16T14:54:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:(130/151): python3-websocket-client-1.2.3-2.el9 674 kB/s | 90 kB 00:00 2024-01-16T14:54:25.449 INFO:teuthology.orchestra.run.smithi125.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 819 kB/s | 124 kB 00:00 2024-01-16T14:54:25.463 INFO:teuthology.orchestra.run.smithi137.stdout:(131/151): python3-webob-1.8.7-6.el9.noarch.rpm 949 kB/s | 230 kB 00:00 2024-01-16T14:54:25.492 INFO:teuthology.orchestra.run.smithi125.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 2.7 MB/s | 113 kB 00:00 2024-01-16T14:54:25.496 INFO:teuthology.orchestra.run.smithi137.stdout:(132/151): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-01-16T14:54:25.525 INFO:teuthology.orchestra.run.smithi125.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-01-16T14:54:25.538 INFO:teuthology.orchestra.run.smithi137.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-01-16T14:54:25.588 INFO:teuthology.orchestra.run.smithi137.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-01-16T14:54:25.681 INFO:teuthology.orchestra.run.smithi137.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-01-16T14:54:25.723 INFO:teuthology.orchestra.run.smithi137.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-16T14:54:25.725 INFO:teuthology.orchestra.run.smithi125.stdout:(95/151): lua-devel-5.4.4-4.el9.x86_64.rpm 96 kB/s | 22 kB 00:00 2024-01-16T14:54:25.931 INFO:teuthology.orchestra.run.smithi137.stdout:(137/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 8.8 MB/s | 3.0 MB 00:00 2024-01-16T14:54:25.990 INFO:teuthology.orchestra.run.smithi137.stdout:(138/151): NetworkManager-1.45.9-1.el9.x86_64.r 8.5 MB/s | 2.2 MB 00:00 2024-01-16T14:54:26.090 INFO:teuthology.orchestra.run.smithi137.stdout:(139/151): librados2-18.0.0-7629.g73a4532f.el9. 6.0 MB/s | 3.3 MB 00:00 2024-01-16T14:54:26.157 INFO:teuthology.orchestra.run.smithi137.stdout:(140/151): NetworkManager-libnm-1.45.9-1.el9.x8 7.9 MB/s | 1.8 MB 00:00 2024-01-16T14:54:26.182 INFO:teuthology.orchestra.run.smithi137.stdout:(141/151): NetworkManager-team-1.45.9-1.el9.x86 228 kB/s | 42 kB 00:00 2024-01-16T14:54:26.192 INFO:teuthology.orchestra.run.smithi125.stdout:(96/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 464 kB/s | 309 kB 00:00 2024-01-16T14:54:26.218 INFO:teuthology.orchestra.run.smithi125.stdout:(97/151): python3-devel-3.9.17-2.el9.x86_64.rpm 320 kB/s | 245 kB 00:00 2024-01-16T14:54:26.224 INFO:teuthology.orchestra.run.smithi137.stdout:(142/151): NetworkManager-tui-1.45.9-1.el9.x86_ 2.0 MB/s | 246 kB 00:00 2024-01-16T14:54:26.258 INFO:teuthology.orchestra.run.smithi137.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 490 kB/s | 35 kB 00:00 2024-01-16T14:54:26.310 INFO:teuthology.orchestra.run.smithi125.stdout:(98/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 224 kB/s | 26 kB 00:00 2024-01-16T14:54:26.316 INFO:teuthology.orchestra.run.smithi137.stdout:(144/151): acl-2.3.1-4.el9.x86_64.rpm 525 kB/s | 69 kB 00:00 2024-01-16T14:54:26.342 INFO:teuthology.orchestra.run.smithi137.stdout:(145/151): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 191 kB 00:00 2024-01-16T14:54:26.375 INFO:teuthology.orchestra.run.smithi137.stdout:(146/151): elfutils-libs-0.190-2.el9.x86_64.rpm 2.2 MB/s | 254 kB 00:00 2024-01-16T14:54:26.400 INFO:teuthology.orchestra.run.smithi137.stdout:(147/151): libacl-2.3.1-4.el9.x86_64.rpm 396 kB/s | 22 kB 00:00 2024-01-16T14:54:26.468 INFO:teuthology.orchestra.run.smithi125.stdout:(99/151): libunwind-1.6.2-1.el9.x86_64.rpm 427 kB/s | 67 kB 00:00 2024-01-16T14:54:26.559 INFO:teuthology.orchestra.run.smithi137.stdout:(148/151): openssl-3.0.7-25.el9.x86_64.rpm 5.4 MB/s | 1.2 MB 00:00 2024-01-16T14:54:26.610 INFO:teuthology.orchestra.run.smithi125.stdout:(100/151): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-01-16T14:54:26.626 INFO:teuthology.orchestra.run.smithi137.stdout:(149/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 8.6 MB/s | 2.1 MB 00:00 2024-01-16T14:54:26.635 INFO:teuthology.orchestra.run.smithi125.stdout:(101/151): liboath-2.6.7-2.el9.x86_64.rpm 116 kB/s | 49 kB 00:00 2024-01-16T14:54:26.651 INFO:teuthology.orchestra.run.smithi137.stdout:(150/151): xfsdump-3.1.12-4.el9.x86_64.rpm 1.3 MB/s | 321 kB 00:00 2024-01-16T14:54:26.701 INFO:teuthology.orchestra.run.smithi137.stdout:(151/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-01-16T14:54:26.710 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:54:26.710 INFO:teuthology.orchestra.run.smithi137.stdout:Total 13 MB/s | 204 MB 00:16 2024-01-16T14:54:26.801 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-16T14:54:26.869 INFO:teuthology.orchestra.run.smithi125.stdout:(102/151): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-01-16T14:54:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:(103/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-01-16T14:54:26.970 INFO:teuthology.orchestra.run.smithi125.stdout:(104/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 437 kB/s | 43 kB 00:00 2024-01-16T14:54:27.028 INFO:teuthology.orchestra.run.smithi125.stdout:(105/151): python3-cachetools-4.2.4-1.el9.noarc 323 kB/s | 32 kB 00:00 2024-01-16T14:54:27.070 INFO:teuthology.orchestra.run.smithi125.stdout:(106/151): python3-certifi-2021.10.8-2.el9.noar 151 kB/s | 15 kB 00:00 2024-01-16T14:54:27.103 INFO:teuthology.orchestra.run.smithi125.stdout:(107/151): python3-asyncssh-2.13.2-3.el9.noarch 1.1 MB/s | 548 kB 00:00 2024-01-16T14:54:27.137 INFO:teuthology.orchestra.run.smithi125.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-01-16T14:54:27.148 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-16T14:54:27.148 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-16T14:54:27.179 INFO:teuthology.orchestra.run.smithi125.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-01-16T14:54:27.221 INFO:teuthology.orchestra.run.smithi125.stdout:(110/151): python3-google-auth-2.26.0-1.el9.noa 1.7 MB/s | 202 kB 00:00 2024-01-16T14:54:27.247 INFO:teuthology.orchestra.run.smithi125.stdout:(111/151): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-01-16T14:54:27.280 INFO:teuthology.orchestra.run.smithi125.stdout:(112/151): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-01-16T14:54:27.322 INFO:teuthology.orchestra.run.smithi125.stdout:(113/151): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-01-16T14:54:27.347 INFO:teuthology.orchestra.run.smithi125.stdout:(114/151): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-01-16T14:54:27.381 INFO:teuthology.orchestra.run.smithi125.stdout:(115/151): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-01-16T14:54:27.423 INFO:teuthology.orchestra.run.smithi125.stdout:(116/151): python3-jwt+crypto-2.4.0-1.el9.noarc 91 kB/s | 9.0 kB 00:00 2024-01-16T14:54:27.448 INFO:teuthology.orchestra.run.smithi125.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 409 kB/s | 41 kB 00:00 2024-01-16T14:54:27.506 INFO:teuthology.orchestra.run.smithi125.stdout:(118/151): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-01-16T14:54:27.532 INFO:teuthology.orchestra.run.smithi125.stdout:(119/151): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-01-16T14:54:27.557 INFO:teuthology.orchestra.run.smithi125.stdout:(120/151): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-01-16T14:54:27.608 INFO:teuthology.orchestra.run.smithi125.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-01-16T14:54:27.658 INFO:teuthology.orchestra.run.smithi125.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-01-16T14:54:27.683 INFO:teuthology.orchestra.run.smithi125.stdout:(123/151): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-01-16T14:54:27.700 INFO:teuthology.orchestra.run.smithi125.stdout:(124/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 981 kB/s | 90 kB 00:00 2024-01-16T14:54:27.750 INFO:teuthology.orchestra.run.smithi125.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-01-16T14:54:27.792 INFO:teuthology.orchestra.run.smithi125.stdout:(126/151): python3-rsa-4.9-2.el9.noarch.rpm 647 kB/s | 59 kB 00:00 2024-01-16T14:54:27.817 INFO:teuthology.orchestra.run.smithi125.stdout:(127/151): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-01-16T14:54:27.843 INFO:teuthology.orchestra.run.smithi125.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-01-16T14:54:27.885 INFO:teuthology.orchestra.run.smithi125.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 557 kB/s | 51 kB 00:00 2024-01-16T14:54:27.952 INFO:teuthology.orchestra.run.smithi125.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-01-16T14:54:27.977 INFO:teuthology.orchestra.run.smithi125.stdout:(131/151): python3-websocket-client-1.2.3-2.el9 672 kB/s | 90 kB 00:00 2024-01-16T14:54:28.011 INFO:teuthology.orchestra.run.smithi125.stdout:(132/151): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-01-16T14:54:28.052 INFO:teuthology.orchestra.run.smithi125.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-01-16T14:54:28.086 INFO:teuthology.orchestra.run.smithi125.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-01-16T14:54:28.153 INFO:teuthology.orchestra.run.smithi125.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-01-16T14:54:28.195 INFO:teuthology.orchestra.run.smithi125.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-16T14:54:28.312 INFO:teuthology.orchestra.run.smithi125.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 19 MB/s | 2.2 MB 00:00 2024-01-16T14:54:28.404 INFO:teuthology.orchestra.run.smithi125.stdout:(138/151): NetworkManager-libnm-1.45.9-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-01-16T14:54:28.479 INFO:teuthology.orchestra.run.smithi125.stdout:(139/151): librados2-18.0.0-7629.g73a4532f.el9. 7.8 MB/s | 3.3 MB 00:00 2024-01-16T14:54:28.504 INFO:teuthology.orchestra.run.smithi125.stdout:(140/151): NetworkManager-team-1.45.9-1.el9.x86 426 kB/s | 42 kB 00:00 2024-01-16T14:54:28.546 INFO:teuthology.orchestra.run.smithi125.stdout:(141/151): NetworkManager-tui-1.45.9-1.el9.x86_ 3.7 MB/s | 246 kB 00:00 2024-01-16T14:54:28.580 INFO:teuthology.orchestra.run.smithi125.stdout:(142/151): acl-2.3.1-4.el9.x86_64.rpm 920 kB/s | 69 kB 00:00 2024-01-16T14:54:28.622 INFO:teuthology.orchestra.run.smithi125.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 472 kB/s | 35 kB 00:00 2024-01-16T14:54:28.664 INFO:teuthology.orchestra.run.smithi125.stdout:(144/151): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 191 kB 00:00 2024-01-16T14:54:28.706 INFO:teuthology.orchestra.run.smithi125.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 254 kB 00:00 2024-01-16T14:54:28.740 INFO:teuthology.orchestra.run.smithi125.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 294 kB/s | 22 kB 00:00 2024-01-16T14:54:28.907 INFO:teuthology.orchestra.run.smithi125.stdout:(147/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 3.7 MB/s | 3.0 MB 00:00 2024-01-16T14:54:28.982 INFO:teuthology.orchestra.run.smithi125.stdout:(148/151): openssl-3.0.7-25.el9.x86_64.rpm 4.2 MB/s | 1.2 MB 00:00 2024-01-16T14:54:29.057 INFO:teuthology.orchestra.run.smithi125.stdout:(149/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 6.8 MB/s | 2.1 MB 00:00 2024-01-16T14:54:29.082 INFO:teuthology.orchestra.run.smithi125.stdout:(150/151): xfsdump-3.1.12-4.el9.x86_64.rpm 2.0 MB/s | 321 kB 00:00 2024-01-16T14:54:29.133 INFO:teuthology.orchestra.run.smithi125.stdout:(151/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.5 MB/s | 1.1 MB 00:00 2024-01-16T14:54:29.143 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:54:29.143 INFO:teuthology.orchestra.run.smithi125.stdout:Total 11 MB/s | 204 MB 00:18 2024-01-16T14:54:29.231 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2024-01-16T14:54:29.527 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-16T14:54:29.528 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-16T14:54:29.562 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2024-01-16T14:54:29.562 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2024-01-16T14:54:31.682 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2024-01-16T14:54:31.683 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2024-01-16T14:54:35.939 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-16T14:54:36.066 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:54:36.108 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:54:36.178 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:54:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:54:36.267 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:54:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:54:36.775 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:54:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:37.191 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:37.227 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:37.343 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:37.374 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:54:37.505 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:54:37.646 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:54:37.969 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:54:37.988 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2024-01-16T14:54:38.026 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:54:38.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:38.171 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:54:38.206 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:54:38.210 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:38.243 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:38.266 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:54:38.324 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:54:38.327 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:38.357 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:54:38.449 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:54:38.687 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:54:38.834 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:54:38.866 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:38.896 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:39.046 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:39.072 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:39.155 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:39.194 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:39.218 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:39.232 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:54:39.367 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:54:39.462 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:54:39.657 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:39.707 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:54:39.732 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:54:39.796 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:54:39.821 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:54:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:54:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:54:39.964 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:39.986 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:54:39.997 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:40.037 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:54:40.080 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:40.141 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:54:40.211 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:54:40.258 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:54:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:54:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:54:40.441 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:54:40.461 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:54:40.523 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:54:40.559 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:54:40.615 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:40.660 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:54:40.713 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:40.754 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:54:40.907 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:40.920 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:54:40.965 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:40.978 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:54:41.556 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:54:41.661 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:54:41.718 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:54:41.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:54:41.820 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:54:41.843 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:54:41.860 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:54:41.891 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:54:41.921 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:54:42.002 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:54:42.102 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:54:42.152 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:54:42.154 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:54:42.197 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:54:42.237 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:54:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:54:42.353 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:54:42.391 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:54:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:54:42.443 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:54:42.475 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:54:42.481 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:54:42.541 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:54:42.599 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:54:42.627 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:54:42.684 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:54:42.693 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:54:42.723 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:54:42.791 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:54:42.848 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:54:42.875 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:54:42.909 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:54:42.943 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:54:43.097 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:54:43.158 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:54:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:54:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:54:43.382 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:54:43.430 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:54:43.465 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:54:43.617 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:54:43.628 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:54:43.669 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:54:43.742 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:54:43.772 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:54:43.821 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:54:43.826 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:54:43.945 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:54:44.071 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:54:44.136 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:54:44.149 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:54:44.336 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:54:44.343 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:54:44.500 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:54:44.539 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:54:44.607 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:54:44.667 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:54:44.697 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:54:44.700 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:54:44.739 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:54:44.754 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:54:44.778 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:54:44.825 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:54:44.844 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:54:44.892 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:54:44.910 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:54:44.958 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:54:44.972 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:54:45.004 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:54:45.055 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:54:45.137 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:54:45.160 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:54:45.202 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:54:45.204 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:54:45.248 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:54:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:54:45.375 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:54:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:54:45.411 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:54:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:54:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:54:45.460 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:54:45.502 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:54:45.553 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:54:45.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:54:45.653 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:54:45.707 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:54:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:54:45.776 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:54:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:54:45.859 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:54:45.933 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:54:46.031 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:54:46.141 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:54:46.204 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:54:46.374 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:54:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:54:46.465 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:54:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:54:46.568 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:54:46.634 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:54:46.693 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:54:46.739 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:54:46.790 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:54:46.845 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:54:46.912 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:54:46.972 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:54:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:54:47.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:54:47.096 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:54:47.121 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:54:47.129 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:54:47.173 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:54:47.218 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:54:47.269 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:54:47.317 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:54:47.372 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:54:47.401 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:54:47.458 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:54:47.509 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:54:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:54:47.665 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:54:47.738 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:54:48.303 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:54:48.444 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:54:48.670 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:54:48.738 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:54:48.772 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:54:48.805 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:54:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:54:49.084 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:54:49.184 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:54:49.939 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:54:49.983 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:54:50.023 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:54:50.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:54:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:54:50.110 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:54:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:54:50.486 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:54:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:54:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:54:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:54:51.183 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:54:51.214 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:54:51.315 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:54:51.341 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:54:51.660 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:54:51.739 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:54:51.784 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:54:51.818 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:54:52.173 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:54:52.202 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:54:52.325 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:54:52.358 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:54:52.419 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:54:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:54:52.964 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:54:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:54:53.082 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:54:53.814 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:53.854 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:54.510 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:54.552 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:54:54.733 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:54:54.733 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:54:54.734 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:54:54.734 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:54:54.811 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:54:55.503 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:55.554 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:56.196 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:54:56.240 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:54:56.469 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:54:56.470 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:54:56.470 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:54:56.470 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:54:56.516 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:06.966 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-01-16T14:55:06.966 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-01-16T14:55:06.966 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-01-16T14:55:06.967 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-01-16T14:55:06.967 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-01-16T14:55:06.967 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-01-16T14:55:06.967 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-01-16T14:55:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:07.174 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:07.223 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:07.223 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:07.223 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:55:07.223 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /sys 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /proc 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /mnt 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /var/tmp 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /home 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /root 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /tmp 2024-01-16T14:55:08.758 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:08.893 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:08.964 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:08.965 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:08.965 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:55:08.965 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:08.965 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:08.965 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:55:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:09.176 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:09.242 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:09.735 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:55:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:09.946 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:55:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:10.641 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:55:10.692 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:10.771 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:10.775 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:10.991 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:11.060 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:11.060 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:11.060 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:55:11.061 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:11.061 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:11.061 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:11.090 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:11.135 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:11.542 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:11.609 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:11.727 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:55:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:12.344 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:12.386 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:12.430 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:55:12.434 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:12.525 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:55:12.565 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:12.609 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:55:12.637 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:12.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:12.788 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:12.825 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:13.793 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:55:13.856 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:13.952 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:14.007 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:14.032 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:14.066 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:14.105 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:14.105 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:14.105 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:55:14.105 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:14.183 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:14.207 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:14.291 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:14.381 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:55:14.410 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:14.479 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:14.479 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:14.479 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:55:14.479 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:14.480 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:14.480 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:14.535 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:14.573 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:14.660 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:55:14.683 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:14.769 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:14.769 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:14.769 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:55:14.769 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:14.769 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:14.770 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:15.502 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:55:15.538 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:15.604 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:15.605 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:55:15.605 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:15.605 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:15.605 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:15.739 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:15.740 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:55:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:15.823 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:15.900 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:15.900 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:15.900 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:55:15.900 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:15.900 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:15.901 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:16.273 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:16.351 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:18.907 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:55:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:55:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:55:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:55:19.346 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:55:19.462 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:55:19.508 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:55:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:55:19.664 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:55:19.784 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:55:20.427 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:55:20.469 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:55:20.515 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:55:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:55:20.800 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:55:20.805 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:20.850 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:20.919 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:55:20.950 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:20.956 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:20.989 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:55:21.029 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:55:21.029 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:55:21.065 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:55:21.181 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:21.268 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:55:21.309 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:55:22.234 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:22.281 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:22.361 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:22.388 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:55:22.425 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:55:22.473 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:55:22.529 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:55:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:55:22.717 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:55:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:55:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:55:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:22.959 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:23.010 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:55:23.065 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:55:23.129 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:55:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:23.598 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:23.722 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:23.757 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:55:23.766 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:55:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:23.837 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:55:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:55:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:55:24.067 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:55:24.106 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:24.106 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:55:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:55:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:24.249 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:24.293 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:55:24.315 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:24.332 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:55:24.366 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:55:24.377 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:55:24.412 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:55:24.445 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:55:24.478 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:24.478 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:24.498 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:24.761 INFO:teuthology.orchestra.run.smithi137.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:55:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:25.024 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:25.054 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:55:25.102 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:25.102 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:25.555 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:25.599 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:55:25.638 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:55:25.683 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:55:25.811 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:26.070 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:26.071 INFO:teuthology.orchestra.run.smithi125.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:55:26.071 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:55:29.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:55:29.067 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:55:29.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:55:29.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:55:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:55:29.083 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:55:29.084 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:55:29.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:55:29.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:55:29.087 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:55:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:55:29.089 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:55:29.090 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:55:30.323 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:55:30.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:55:30.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:55:30.340 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:55:30.340 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:55:30.340 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:55:30.341 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:55:30.342 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:55:30.343 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:55:30.344 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:55:30.345 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:55:30.346 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:55:30.347 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:55:30.348 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:55:30.349 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:55:30.350 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:55:31.377 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:55:31.378 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.379 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:55:31.380 INFO:teuthology.orchestra.run.smithi137.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:55:31.381 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:55:31.382 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:55:31.383 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:55:31.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:55:31.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:55:31.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:55:31.387 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:55:31.388 INFO:teuthology.orchestra.run.smithi137.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:55:31.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:31.388 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-16T14:55:31.914 DEBUG:teuthology.parallel:result is None 2024-01-16T14:55:32.521 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout:Upgraded: 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:55:32.522 INFO:teuthology.orchestra.run.smithi125.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.523 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.524 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:55:32.525 INFO:teuthology.orchestra.run.smithi125.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:55:32.526 INFO:teuthology.orchestra.run.smithi125.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:55:32.527 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:55:32.528 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:55:32.529 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:55:32.530 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:55:32.531 INFO:teuthology.orchestra.run.smithi125.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:55:32.532 INFO:teuthology.orchestra.run.smithi125.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:55:32.533 INFO:teuthology.orchestra.run.smithi125.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:55:32.533 INFO:teuthology.orchestra.run.smithi125.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:55:32.533 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:32.533 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2024-01-16T14:55:33.086 DEBUG:teuthology.parallel:result is None 2024-01-16T14:55:33.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:55:33.239 DEBUG:teuthology.orchestra.run.smithi125:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:33.300 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:33.301 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:55:33.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:55:33.439 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:55:33.505 INFO:teuthology.orchestra.run.smithi137.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:33.505 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:33.505 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:55:33.507 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T14:55:33.507 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:33.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:55:33.593 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:33.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:55:33.694 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T14:55:33.694 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:33.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:55:33.769 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:55:33.879 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:33.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:55:33.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:55:34.078 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T14:55:34.079 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:34.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:55:34.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:55:34.266 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:34.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:55:34.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:55:34.461 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T14:55:34.462 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:34.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:55:34.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:55:34.679 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:34.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:55:34.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:55:34.861 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T14:55:34.948 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T14:55:34.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /var/log/ceph 2024-01-16T14:55:34.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-01-16T14:55:35.028 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T14:55:35.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:55:35.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:55:35.119 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T14:55:35.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:55:35.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:55:35.207 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-16T14:55:35.207 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-16T14:55:35.207 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-16T14:55:35.207 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:35.208 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:55:35.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:55:35.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:55:35.528 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:35.528 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:55:35.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:55:35.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:55:35.847 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T14:55:35.847 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T14:55:35.847 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7518191', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2668, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi125.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9/3G22L2LTGLeUmjReflqE9a8+S32eThZv9VfVV2VqP8LRTiHfJ25gh1NiimO9Lj/a+n3nB8QQhzq3wH1G6n0=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvVVxBXj5PaYLksTxs/EpEzdXB4QGafjrHYSGvsfdMHzNM22qCplleZggqUAVamp+v+Yeyed3RnOuoQ4eGOA+M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-16_14:37:50', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T14:55:35.848 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:55:35.853 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:55:35.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:55:35.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:55:36.002 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:36.002 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:55:36.041 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:55:36.041 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_1 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2024-01-16 14:55:26.526692267 +0000 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2024-01-16 14:53:08.581189670 +0000 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2024-01-16 14:53:08.581189670 +0000 2024-01-16T14:55:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2024-01-16 14:53:08.581189670 +0000 2024-01-16T14:55:36.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:55:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2024-01-16T14:55:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2024-01-16T14:55:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000367054 s, 1.4 MB/s 2024-01-16T14:55:36.238 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:55:36.279 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_2 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2024-01-16 14:55:26.526692267 +0000 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2024-01-16 14:53:09.134180152 +0000 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2024-01-16 14:53:09.134180152 +0000 2024-01-16T14:55:36.365 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2024-01-16 14:53:09.134180152 +0000 2024-01-16T14:55:36.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:55:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2024-01-16T14:55:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2024-01-16T14:55:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000280386 s, 1.8 MB/s 2024-01-16T14:55:36.477 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:55:36.516 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_3 2024-01-16T14:55:36.598 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2024-01-16 14:55:26.527692249 +0000 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2024-01-16 14:53:09.706170307 +0000 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2024-01-16 14:53:09.706170307 +0000 2024-01-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2024-01-16 14:53:09.706170307 +0000 2024-01-16T14:55:36.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:55:36.703 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2024-01-16T14:55:36.703 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2024-01-16T14:55:36.703 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000286042 s, 1.8 MB/s 2024-01-16T14:55:36.706 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:55:36.746 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_4 2024-01-16T14:55:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2024-01-16 14:55:26.528692232 +0000 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2024-01-16 14:53:10.235161202 +0000 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2024-01-16 14:53:10.235161202 +0000 2024-01-16T14:55:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2024-01-16 14:53:10.235161202 +0000 2024-01-16T14:55:36.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:55:36.929 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2024-01-16T14:55:36.929 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2024-01-16T14:55:36.929 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000310448 s, 1.6 MB/s 2024-01-16T14:55:36.932 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:55:36.971 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:36.971 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:36.971 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:55:37.014 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:55:37.015 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-16 14:55:25.242858477 +0000 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-16 14:53:08.823560443 +0000 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-16 14:53:08.823560443 +0000 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-16 14:53:08.823560443 +0000 2024-01-16T14:55:37.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:55:37.216 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-16T14:55:37.216 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-16T14:55:37.217 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000413612 s, 1.2 MB/s 2024-01-16T14:55:37.221 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:55:37.274 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-01-16T14:55:37.322 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:55:37.322 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:37.322 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-01-16T14:55:37.322 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:37.323 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:37.323 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-16 14:55:25.243858457 +0000 2024-01-16T14:55:37.323 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-16 14:53:09.372549480 +0000 2024-01-16T14:55:37.323 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-16 14:53:09.372549480 +0000 2024-01-16T14:55:37.323 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-16 14:53:09.372549480 +0000 2024-01-16T14:55:37.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:55:37.410 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-16T14:55:37.410 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-16T14:55:37.410 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000442885 s, 1.2 MB/s 2024-01-16T14:55:37.414 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:55:37.470 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-01-16T14:55:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-16 14:55:25.244858437 +0000 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-16 14:53:09.953537878 +0000 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-16 14:53:09.953537878 +0000 2024-01-16T14:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-16 14:53:09.953537878 +0000 2024-01-16T14:55:37.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:55:37.605 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-16T14:55:37.605 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-16T14:55:37.605 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000389742 s, 1.3 MB/s 2024-01-16T14:55:37.609 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:55:37.661 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-16 14:55:25.245858417 +0000 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-16 14:53:10.522526516 +0000 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-16 14:53:10.522526516 +0000 2024-01-16T14:55:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-16 14:53:10.522526516 +0000 2024-01-16T14:55:37.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:55:37.791 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-16T14:55:37.792 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-16T14:55:37.792 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000366696 s, 1.4 MB/s 2024-01-16T14:55:37.796 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:55:37.848 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:37.849 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:55:37.849 INFO:tasks.ceph:Generating config... 2024-01-16T14:55:37.850 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-16T14:55:37.850 INFO:tasks.ceph:[client] debug client = 20 2024-01-16T14:55:37.850 INFO:tasks.ceph:[client] debug ms = 1 2024-01-16T14:55:37.850 INFO:tasks.ceph:[client] pid file = 2024-01-16T14:55:37.851 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-16T14:55:37.851 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-16T14:55:37.851 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T14:55:37.852 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T14:55:37.853 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T14:55:37.853 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T14:55:37.853 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-16T14:55:37.854 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-16T14:55:37.854 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T14:55:37.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T14:55:37.987 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T14:55:37.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T14:55:38.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:55:38.220 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.125'), ('mon.b', '172.21.15.137'), ('mon.c', '[v2:172.21.15.137:3301,v1:172.21.15.137:6790]')] 2024-01-16T14:55:38.220 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.125,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-16T14:55:38.221 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:38.221 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:55:38.271 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.125 --add b 172.21.15.137 --addv c '[v2:172.21.15.137:3301,v1:172.21.15.137:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:38.417 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:38.417 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: generated fsid 3c5ae53a-13b6-44c3-9861-4f92782fd96b 2024-01-16T14:55:38.417 INFO:teuthology.orchestra.run.smithi125.stdout:setting min_mon_release = pacific 2024-01-16T14:55:38.417 INFO:teuthology.orchestra.run.smithi125.stdout:epoch 0 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:fsid 3c5ae53a-13b6-44c3-9861-4f92782fd96b 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:last_changed 2024-01-16T14:55:38.435498+0000 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:created 2024-01-16T14:55:38.435498+0000 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:min_mon_release 16 (pacific) 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:election_strategy: 1 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:0: [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] mon.a 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:1: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.b 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:2: [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] mon.c 2024-01-16T14:55:38.418 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-16T14:55:38.420 DEBUG:teuthology.orchestra.run.smithi125:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:55:38.471 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3c5ae53a-13b6-44c3-9861-4f92782fd96b... 2024-01-16T14:55:38.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:55:38.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:55:38.636 INFO:teuthology.orchestra.run.smithi125.stdout:[global] 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: chdir = "" 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: auth supported = cephx 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: filestore xattr use omap = true 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: mon clock drift allowed = 1.000 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: osd crush chooseleaf type = 0 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: auth debug = true 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on old message = true 2024-01-16T14:55:38.637 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on bug = true 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon pg warn max object skew = 0 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default size = 2 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow primary affinity = true 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow pg remap = true 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:55:38.638 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on no sortbitwise = false 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on too few osds = false 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon_allow_pool_size_one = true 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: osd default data pool replay window = 5 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon allow pool delete = true 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon cluster log file level = debug 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: debug asserts on shutdown = true 2024-01-16T14:55:38.639 INFO:teuthology.orchestra.run.smithi125.stdout: mon health detail to clog = false 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: mon host = "172.21.15.125,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]" 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: fsid = 3c5ae53a-13b6-44c3-9861-4f92782fd96b 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout:[osd] 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd journal size = 100 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub load threshold = 5.0 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub max interval = 600 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd recover clone overlap = true 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd recovery max chunk = 1048576 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug shutdown = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug op order = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug verify stray on activate = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug trim objects = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd open classes on start = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug pg log writeout = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd map max advance = 10 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: journal zero on create = true 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.641 INFO:teuthology.orchestra.run.smithi125.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: filestore apply finisher threads = 3 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bdev debug aio = true 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug misdirected ops = true 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bdev async discard = True 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bdev enable discard = True 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore allocator = bitmap 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore block size = 96636764160 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore fsck on mount = True 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluefs = 1/20 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluestore = 1/20 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug osd = 20 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug rocksdb = 4/10 2024-01-16T14:55:38.642 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd full ratio = 0.9 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: osd objectstore = bluestore 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr] 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug mgr = 20 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: mgr/telemetry/nag = false 2024-01-16T14:55:38.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: debug finisher = 20 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout:[mon] 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: debug paxos = 20 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon data avail warn = 5 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon mgr mkfs grace = 240 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd reporter subtree level = osd 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd prime pg temp = true 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.644 INFO:teuthology.orchestra.run.smithi125.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: # 1m isn't quite enough 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: mon op complaint time = 120 2024-01-16T14:55:38.645 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout:[client] 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: rgw cache enabled = true 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable ops log = true 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable usage log = true 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: client mount timeout = 600 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = "" 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.a] 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.b] 2024-01-16T14:55:38.646 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.c] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mds] 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds = 20 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds balancer = 20 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug frag = True 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug scatterstat = True 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: mds op complaint time = 180 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: mds verify scatter = True 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2024-01-16T14:55:38.647 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2024-01-16T14:55:38.653 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-01-16T14:55:38.653 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-01-16T14:55:38.653 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:55:38.653 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-01-16T14:55:38.653 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:55:38.654 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.655 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "172.21.15.125,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]" 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 3c5ae53a-13b6-44c3-9861-4f92782fd96b 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-01-16T14:55:38.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: bdev async discard = True 2024-01-16T14:55:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: bdev enable discard = True 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore allocator = bitmap 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 1/20 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 1/20 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 4/10 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-01-16T14:55:38.659 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug finisher = 20 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-16T14:55:38.660 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:55:38.661 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: mon op complaint time = 120 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: client mount timeout = 600 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2024-01-16T14:55:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = "" 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 900 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 900 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.c] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.d] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout:[mds] 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 20 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds balancer = 20 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug frag = True 2024-01-16T14:55:38.663 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug scatterstat = True 2024-01-16T14:55:38.664 INFO:teuthology.orchestra.run.smithi137.stdout: mds op complaint time = 180 2024-01-16T14:55:38.664 INFO:teuthology.orchestra.run.smithi137.stdout: mds verify scatter = True 2024-01-16T14:55:38.664 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2024-01-16T14:55:38.664 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 900 2024-01-16T14:55:38.664 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 900 2024-01-16T14:55:38.698 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T14:55:38.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T14:55:38.847 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T14:55:38.848 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:38.848 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T14:55:38.896 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:38.897 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T14:55:38.943 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:55:38.943 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:38.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:55:38.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:55:39.121 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:55:39.122 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:39.218 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:55:39.218 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:39.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:55:39.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:55:39.371 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:55:39.371 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:39.467 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T14:55:39.467 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T14:55:39.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:55:39.646 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:55:39.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:55:39.839 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:55:39.848 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T14:55:39.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:55:40.032 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:55:40.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-16T14:55:40.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:55:40.306 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:55:40.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-16T14:55:40.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:55:40.573 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:55:40.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-16T14:55:40.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:55:40.809 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:55:40.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-16T14:55:40.898 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T14:55:40.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T14:55:41.023 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T14:55:41.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-16T14:55:41.204 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-16T14:55:41.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-16T14:55:41.370 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-16T14:55:41.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-16T14:55:41.547 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-16T14:55:41.599 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T14:55:41.599 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:55:41.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T14:55:41.685 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:41.685 INFO:tasks.ceph:role: osd.0 2024-01-16T14:55:41.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:55:41.685 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:55:41.763 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:41.764 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:55:42.265 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2024-01-16T14:55:42.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2024-01-16T14:55:42.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T14:55:42.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T14:55:42.531 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T14:55:42.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:42.609 INFO:tasks.ceph:role: osd.1 2024-01-16T14:55:42.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:55:42.609 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:42.691 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:42.692 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:42.692 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:55:43.190 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2024-01-16T14:55:43.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2024-01-16T14:55:43.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T14:55:43.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T14:55:43.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T14:55:43.510 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:43.510 INFO:tasks.ceph:role: osd.2 2024-01-16T14:55:43.510 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:55:43.511 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:55:43.606 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:43.606 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:43.607 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:43.607 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:43.607 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:55:44.114 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2024-01-16T14:55:44.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2024-01-16T14:55:44.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T14:55:44.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T14:55:44.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-16T14:55:44.451 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:44.451 INFO:tasks.ceph:role: osd.3 2024-01-16T14:55:44.451 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:55:44.451 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:55:44.547 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:44.547 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:44.547 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:44.547 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:44.548 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:55:45.053 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2024-01-16T14:55:45.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2024-01-16T14:55:45.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-16T14:55:45.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-16T14:55:45.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.449+0000 7f9123e20500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T14:55:45.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.455+0000 7f9123e20500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T14:55:45.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.455+0000 7f9123e20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:55:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.455+0000 7f9123e20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:55:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.455+0000 7f9123e20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:55:45.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:45.466+0000 7f9123e20500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T14:55:48.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T14:55:48.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.876+0000 7f5fd4412500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T14:55:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.877+0000 7f5fd4412500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T14:55:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.877+0000 7f5fd4412500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:55:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.877+0000 7f5fd4412500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:55:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.877+0000 7f5fd4412500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:55:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:48.879+0000 7f5fd4412500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T14:55:52.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T14:55:52.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:52.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.286+0000 7f2fdd010500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T14:55:52.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.287+0000 7f2fdd010500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T14:55:52.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.287+0000 7f2fdd010500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:55:52.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.287+0000 7f2fdd010500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:55:52.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.287+0000 7f2fdd010500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:55:52.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:52.289+0000 7f2fdd010500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T14:55:55.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T14:55:55.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:55:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.456+0000 7fa68e412500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-16T14:55:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.457+0000 7fa68e412500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-16T14:55:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.457+0000 7fa68e412500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:55:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.457+0000 7fa68e412500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:55:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.457+0000 7fa68e412500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:55:55.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:55:55.460+0000 7fa68e412500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-16T14:55:58.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-16T14:55:58.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-16T14:55:58.793 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:58.793 INFO:tasks.ceph:role: osd.4 2024-01-16T14:55:58.793 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:55:58.793 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:55:58.890 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:58.890 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:58.890 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:58.890 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:58.890 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:58.891 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:58.891 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:58.891 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:58.891 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:58.891 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:55:59.389 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-16T14:55:59.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-16T14:55:59.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-16T14:55:59.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-16T14:55:59.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-16T14:55:59.748 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:55:59.748 INFO:tasks.ceph:role: osd.5 2024-01-16T14:55:59.748 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:55:59.749 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:55:59.842 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:55:59.842 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:55:59.842 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:55:59.842 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:55:59.842 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:55:59.843 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:55:59.843 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:55:59.843 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:55:59.843 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:55:59.843 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:00.303 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-16T14:56:00.308 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-16T14:56:00.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:00.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:00.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:00.613 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:00.614 INFO:tasks.ceph:role: osd.6 2024-01-16T14:56:00.614 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:56:00.614 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:01.220 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-16T14:56:01.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-16T14:56:01.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:01.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:01.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:01.608 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:01.608 INFO:tasks.ceph:role: osd.7 2024-01-16T14:56:01.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:56:01.608 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:56:01.706 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:01.707 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:02.157 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-16T14:56:02.165 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-16T14:56:02.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:02.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:02.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:02.541 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.558+0000 7f516621e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-16T14:56:02.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.559+0000 7f516621e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-16T14:56:02.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.559+0000 7f516621e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:02.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.559+0000 7f516621e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:02.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.559+0000 7f516621e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:02.551 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:02.569+0000 7f516621e500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-16T14:56:05.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-16T14:56:05.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:05.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.742+0000 7f4a50e0c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-16T14:56:05.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.742+0000 7f4a50e0c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-16T14:56:05.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.742+0000 7f4a50e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:05.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.743+0000 7f4a50e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:05.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.743+0000 7f4a50e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:05.727 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:05.745+0000 7f4a50e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-16T14:56:05.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:05.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:08.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:08.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:09.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.147+0000 7f524f20c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-16T14:56:09.132 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.148+0000 7f524f20c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-16T14:56:09.132 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.148+0000 7f524f20c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:09.132 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.148+0000 7f524f20c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:09.132 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.148+0000 7f524f20c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:09.133 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:09.150+0000 7f524f20c500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-16T14:56:12.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:12.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:12.294 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.311+0000 7fc60500c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-16T14:56:12.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.312+0000 7fc60500c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-16T14:56:12.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.312+0000 7fc60500c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:12.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.312+0000 7fc60500c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:12.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.312+0000 7fc60500c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:12.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:12.314+0000 7fc60500c500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-16T14:56:15.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:15.309 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T14:56:15.309 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-16T14:56:15.405 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-16T14:56:15.498 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.498 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-16T14:56:15.592 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T14:56:15.687 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T14:56:15.817 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T14:56:15.903 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:15.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-16T14:56:15.992 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:15.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T14:56:16.090 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-16T14:56:16.184 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-16T14:56:16.281 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-16T14:56:16.373 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-16T14:56:16.469 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-16T14:56:16.560 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-16T14:56:16.651 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:16.651 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T14:56:16.707 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:16.707 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-16T14:56:16.758 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.758 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-16T14:56:16.814 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:16.814 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-16T14:56:16.865 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T14:56:16.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:56:16.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:56:16.955 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.y] 2024-01-16T14:56:16.955 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.0] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.1] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.2] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q== 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.3] 2024-01-16T14:56:16.956 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ== 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.x] 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw== 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA== 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw== 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.4] 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ== 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.5] 2024-01-16T14:56:16.957 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.6] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.7] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[client.2] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[client.1] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg== 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout:[client.3] 2024-01-16T14:56:16.958 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw== 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.y] 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg== 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A== 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.c] 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw== 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-01-16T14:56:16.959 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ== 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg== 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q== 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ== 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw== 2024-01-16T14:56:16.960 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.d] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.4] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.5] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.6] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.7] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA== 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-01-16T14:56:16.961 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:[client.2] 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ== 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg== 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:[client.3] 2024-01-16T14:56:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw== 2024-01-16T14:56:16.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:17.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:17.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:17.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:17.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:17.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:17.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:17.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:18.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:18.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:18.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:18.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:18.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:18.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:18.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:19.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.360 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:19.653 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T14:56:19.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T14:56:19.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:19.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T14:56:19.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-16T14:56:20.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:20.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-16T14:56:20.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-16T14:56:20.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:20.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-16T14:56:20.645 DEBUG:teuthology.orchestra.run.smithi125:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:20.652 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:20.702 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T14:56:20.702 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T14:56:20.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T14:56:20.708 INFO:tasks.ceph.mon.a:Started 2024-01-16T14:56:20.708 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-16T14:56:20.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-16T14:56:20.715 INFO:tasks.ceph.mon.b:Started 2024-01-16T14:56:20.715 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-16T14:56:20.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-16T14:56:20.722 INFO:tasks.ceph.mon.c:Started 2024-01-16T14:56:20.722 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T14:56:20.722 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-16T14:56:20.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-16T14:56:20.727 INFO:tasks.ceph.mgr.y:Started 2024-01-16T14:56:20.727 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T14:56:20.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T14:56:20.732 INFO:tasks.ceph.mgr.x:Started 2024-01-16T14:56:20.732 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T14:56:20.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T14:56:20.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.926+0000 7ff14c600640 1 Processor -- start 2024-01-16T14:56:20.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.926+0000 7ff14c600640 1 -- start start 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.927+0000 7ff14c600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 0x7ff1440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.927+0000 7ff14c600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.927+0000 7ff14c600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1440cfc30 con 0x7ff1440cf7d0 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.927+0000 7ff14c600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1440b1ee0 con 0x7ff14415e3b0 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.927+0000 7ff14c600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1440b2780 con 0x7ff144143ae0 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff149c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 0x7ff1440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43948/0 (socket says 172.21.15.125:43948) 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 learned_addr learned my addr 172.21.15.125:0/3077600411 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 msgr2=0x7ff1440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-16T14:56:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 msgr2=0x7ff1440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff149c00640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 msgr2=0x7ff1440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff149c00640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 msgr2=0x7ff1440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff149c00640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 0x7ff1440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff149c00640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 0x7ff1440cf290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:56:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.928+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:56:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.938+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946808074 0 0) 0x7ff1440cfc30 con 0x7ff1440cf7d0 2024-01-16T14:56:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.938+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124003520 con 0x7ff1440cf7d0 2024-01-16T14:56:20.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1865432999 0 0) 0x7ff124003520 con 0x7ff1440cf7d0 2024-01-16T14:56:20.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 msgr2=0x7ff1440cf290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:56:20.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.137:3300/0 conn(0x7ff144143ae0 0x7ff1440cf290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 msgr2=0x7ff1440702c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:56:20.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 --2- 172.21.15.125:0/3077600411 >> v2:172.21.15.125:3300/0 conn(0x7ff14415e3b0 0x7ff1440702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14415e990 con 0x7ff1440cf7d0 2024-01-16T14:56:20.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff12c001670 con 0x7ff1440cf7d0 2024-01-16T14:56:20.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 >> v1:172.21.15.137:6789/0 conn(0x7ff1440cf7d0 legacy=0x7ff14415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1240039c0 0x7ff124003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1240047c0 0x7ff124004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 0x7ff12400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440b1ee0 con 0x7ff124005180 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440b2780 con 0x7ff1240039c0 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.943+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12400c0d0 con 0x7ff1240047c0 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff149c00640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1240039c0 0x7ff124003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff149200640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1240047c0 0x7ff124004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 0x7ff12400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1240047c0 msgr2=0x7ff124004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:20.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1240047c0 0x7ff124004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1240039c0 msgr2=0x7ff124003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:20.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1240039c0 0x7ff124003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff124003520 con 0x7ff124005180 2024-01-16T14:56:20.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.944+0000 7ff149c00640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1240039c0 0x7ff124003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:20.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.945+0000 7ff14a600640 1 -- 172.21.15.125:0/3077600411 reap_dead start 2024-01-16T14:56:20.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.945+0000 7ff14a600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 0x7ff12400b980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009cc0 tx=0x7ff1240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee2d473604ae1ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.945+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c02cb10 con 0x7ff124005180 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c030070 con 0x7ff124005180 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff143e00640 1 -- 172.21.15.125:0/3077600411 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c036410 con 0x7ff124005180 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 -- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 msgr2=0x7ff12400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 0x7ff12400b980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009cc0 tx=0x7ff1240057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 -- 172.21.15.125:0/3077600411 shutdown_connections 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 --2- 172.21.15.125:0/3077600411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff124005180 0x7ff12400b980 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 -- 172.21.15.125:0/3077600411 >> 172.21.15.125:0/3077600411 conn(0x7ff14406a800 msgr2=0x7ff14406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 -- 172.21.15.125:0/3077600411 shutdown_connections 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.946+0000 7ff14c600640 1 -- 172.21.15.125:0/3077600411 wait complete. 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.948+0000 7ff14c600640 1 Processor -- start 2024-01-16T14:56:20.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.948+0000 7ff14c600640 1 -- start start 2024-01-16T14:56:20.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1440ab440 0x7ff144155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 0x7ff1440b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440ce8a0 con 0x7ff1440ae180 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440a8560 con 0x7ff124005180 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14c600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440a8840 con 0x7ff1440ab440 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff14a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 0x7ff1440b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.949+0000 7ff149c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff149c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:57186/0 (socket says 172.21.15.125:57186) 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff149c00640 1 -- 172.21.15.125:0/3977106286 learned_addr learned my addr 172.21.15.125:0/3977106286 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff149200640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1440ab440 0x7ff144155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 -- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1440ab440 msgr2=0x7ff144155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1440ab440 0x7ff144155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 -- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 msgr2=0x7ff1440ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:20.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 -- 172.21.15.125:0/3977106286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1440a8b20 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff149c00640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14a600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 0x7ff1440b0bc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff144071f10 tx=0x7ff13c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c036a10 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c036b90 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c03e420 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.950+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1440ce580 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.951+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff144156220 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.951+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff13c03e880 con 0x7ff1440ae180 2024-01-16T14:56:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.951+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff13c0407e0 con 0x7ff1440ae180 2024-01-16T14:56:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.952+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1440675d0 con 0x7ff1440ae180 2024-01-16T14:56:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:20.955+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff13c045070 con 0x7ff1440ae180 2024-01-16T14:56:21.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.026+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff1440aff70 con 0x7ff1440ae180 2024-01-16T14:56:21.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.029+0000 7ff141600640 1 -- 172.21.15.125:0/3977106286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff13400b5e0 con 0x7ff1440ae180 2024-01-16T14:56:21.012 INFO:teuthology.orchestra.run.smithi125.stderr:adjusted tunables profile to default 2024-01-16T14:56:21.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 msgr2=0x7ff1440b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:21.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 0x7ff1440b0bc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff144071f10 tx=0x7ff13c004ab0 comp rx=0 tx=0).stop 2024-01-16T14:56:21.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 shutdown_connections 2024-01-16T14:56:21.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1440ae180 0x7ff1440b0bc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1440ab440 0x7ff144155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 --2- 172.21.15.125:0/3977106286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff124005180 0x7ff1440ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 >> 172.21.15.125:0/3977106286 conn(0x7ff14406a800 msgr2=0x7ff14415c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 shutdown_connections 2024-01-16T14:56:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:21.030+0000 7ff14c600640 1 -- 172.21.15.125:0/3977106286 wait complete. 2024-01-16T14:56:21.031 INFO:tasks.ceph:check_enable_crimson: False 2024-01-16T14:56:21.031 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T14:56:21.032 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:21.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T14:56:21.106 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:21.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T14:56:21.223 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:21.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T14:56:21.349 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:56:21.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-16T14:56:21.477 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:21.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-16T14:56:21.568 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:21.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-16T14:56:21.642 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:21.659+0000 7f0185032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:56:21.656 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:21.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-16T14:56:21.668 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:21.685+0000 7fe19ec32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:56:21.707 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:21.724+0000 7f0185032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:56:21.733 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:21.750+0000 7fe19ec32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:56:21.751 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:56:21.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-16T14:56:21.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new f237602d-0b37-40be-8dae-8c522579f1c7 0 2024-01-16T14:56:22.066 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.083+0000 7fa5e9200640 1 Processor -- start 2024-01-16T14:56:22.066 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.084+0000 7fa5e9200640 1 -- start start 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.085+0000 7fa5e9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e4161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.085+0000 7fa5e9200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5e40ce8f0 0x7fa5e40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e9200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e41618a0 con 0x7fa5e40cfbe0 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e9200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e40d0de0 con 0x7fa5e40ce8f0 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40d10c0 con 0x7fa5e4160f80 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e4161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e2800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa5e40cfbe0 0x7fa5e40d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:36628/0 (socket says 172.21.15.137:36628) 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e2800640 1 -- 172.21.15.137:0/540342067 learned_addr learned my addr 172.21.15.137:0/540342067 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5dbe00640 1 --2- 172.21.15.137:0/540342067 >> v2:172.21.15.125:3300/0 conn(0x7fa5e40ce8f0 0x7fa5e40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1e00640 1 -- 172.21.15.137:0/540342067 >> v2:172.21.15.125:3300/0 conn(0x7fa5e40ce8f0 msgr2=0x7fa5e40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:22.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1e00640 1 --2- 172.21.15.137:0/540342067 >> v2:172.21.15.125:3300/0 conn(0x7fa5e40ce8f0 0x7fa5e40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1e00640 1 -- 172.21.15.137:0/540342067 >> v1:172.21.15.137:6789/0 conn(0x7fa5e40cfbe0 legacy=0x7fa5e40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1e00640 1 -- 172.21.15.137:0/540342067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e40d13a0 con 0x7fa5e4160f80 2024-01-16T14:56:22.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.086+0000 7fa5e1400640 1 -- 172.21.15.137:0/540342067 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339896006 0 0) 0x7fa5e41618a0 con 0x7fa5e40cfbe0 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.087+0000 7fa5e1e00640 1 --2- 172.21.15.137:0/540342067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e4161360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40097a0 tx=0x7fa5d402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb09ab9a03cb57b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e1400640 1 -- 172.21.15.137:0/540342067 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4031830 con 0x7fa5e4160f80 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e1400640 1 -- 172.21.15.137:0/540342067 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d4031990 con 0x7fa5e4160f80 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e1400640 1 -- 172.21.15.137:0/540342067 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4031d20 con 0x7fa5e4160f80 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 -- 172.21.15.137:0/540342067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 msgr2=0x7fa5e4161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 --2- 172.21.15.137:0/540342067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e4161360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40097a0 tx=0x7fa5d402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 -- 172.21.15.137:0/540342067 shutdown_connections 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 --2- 172.21.15.137:0/540342067 >> v2:172.21.15.125:3300/0 conn(0x7fa5e40ce8f0 0x7fa5e40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 --2- 172.21.15.137:0/540342067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e4161360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 -- 172.21.15.137:0/540342067 >> 172.21.15.137:0/540342067 conn(0x7fa5e409c550 msgr2=0x7fa5e40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 -- 172.21.15.137:0/540342067 shutdown_connections 2024-01-16T14:56:22.072 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.088+0000 7fa5e9200640 1 -- 172.21.15.137:0/540342067 wait complete. 2024-01-16T14:56:22.073 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.091+0000 7fa5e9200640 1 Processor -- start 2024-01-16T14:56:22.073 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.091+0000 7fa5e9200640 1 -- start start 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5e40ce8f0 0x7fa5e415c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5e40cfbe0 0x7fa5e40b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e415e710 con 0x7fa5e40ce8f0 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e415e960 con 0x7fa5e40cfbe0 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e415ec40 con 0x7fa5e4160f80 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5dbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5e40cfbe0 0x7fa5e40b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:59052/0 (socket says 172.21.15.137:59052) 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 -- 172.21.15.137:0/1735587799 learned_addr learned my addr 172.21.15.137:0/1735587799 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e1e00640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5e40ce8f0 0x7fa5e415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 -- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5e40cfbe0 msgr2=0x7fa5e40b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5e40cfbe0 0x7fa5e40b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 -- 172.21.15.137:0/1735587799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5e40ce8f0 msgr2=0x7fa5e415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5e40ce8f0 0x7fa5e415c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.093+0000 7fa5e2800640 1 -- 172.21.15.137:0/1735587799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4009400 con 0x7fa5e4160f80 2024-01-16T14:56:22.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5e2800640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc00c7e0 tx=0x7fa5dc00cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc00a640 con 0x7fa5e4160f80 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc00a7c0 con 0x7fa5e4160f80 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc00ab70 con 0x7fa5e4160f80 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e415ef20 con 0x7fa5e4160f80 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e415f4e0 con 0x7fa5e4160f80 2024-01-16T14:56:22.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa5dc0070d0 con 0x7fa5e4160f80 2024-01-16T14:56:22.079 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.094+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa5dc0126c0 con 0x7fa5e4160f80 2024-01-16T14:56:22.079 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.095+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e415f700 con 0x7fa5e4160f80 2024-01-16T14:56:22.080 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.098+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa5dc021060 con 0x7fa5e4160f80 2024-01-16T14:56:22.158 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.176+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f237602d-0b37-40be-8dae-8c522579f1c7", "id": 0} v 0) v1 -- 0x7fa5e40675d0 con 0x7fa5e4160f80 2024-01-16T14:56:22.165 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.183+0000 7fa5da000640 1 -- 172.21.15.137:0/1735587799 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f237602d-0b37-40be-8dae-8c522579f1c7", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa5dc018070 con 0x7fa5e4160f80 2024-01-16T14:56:22.166 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 msgr2=0x7fa5e40b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc00c7e0 tx=0x7fa5dc00cca0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 shutdown_connections 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5e4160f80 0x7fa5e40b41c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5e40cfbe0 0x7fa5e40b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 --2- 172.21.15.137:0/1735587799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5e40ce8f0 0x7fa5e415c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 >> 172.21.15.137:0/1735587799 conn(0x7fa5e409c550 msgr2=0x7fa5e40cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 shutdown_connections 2024-01-16T14:56:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.184+0000 7fa5e9200640 1 -- 172.21.15.137:0/1735587799 wait complete. 2024-01-16T14:56:22.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new c57d8db5-afc1-4ae9-8e25-5dbc99da5467 1 2024-01-16T14:56:22.299 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.316+0000 7f0185032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:56:22.360 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.377+0000 7f0185032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:56:22.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.378+0000 7f5bdfe00640 1 Processor -- start 2024-01-16T14:56:22.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.378+0000 7f5bdfe00640 1 -- start start 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd80c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdfe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5bd815ef70 0x7f5bd815f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd806b950 con 0x7f5bd815d940 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdfe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd806d1d0 con 0x7f5bd815ef70 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80c7290 con 0x7f5bd806b4f0 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5bd815ef70 0x7f5bd815f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdd400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f5bd815d940 0x7f5bd815dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:36640/0 (socket says 172.21.15.137:36640) 2024-01-16T14:56:22.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdd400640 1 -- 172.21.15.137:0/2824168911 learned_addr learned my addr 172.21.15.137:0/2824168911 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-16T14:56:22.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.380+0000 7f5bdca00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd80c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bd7e00640 1 -- 172.21.15.137:0/2824168911 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892607189 0 0) 0x7f5bd806b950 con 0x7f5bd815d940 2024-01-16T14:56:22.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bd7e00640 1 -- 172.21.15.137:0/2824168911 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8003520 con 0x7f5bd815d940 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bdde00640 1 -- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 msgr2=0x7f5bd80c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bdde00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd80c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bdde00640 1 -- 172.21.15.137:0/2824168911 >> v1:172.21.15.137:6789/0 conn(0x7f5bd815d940 legacy=0x7f5bd815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bdde00640 1 -- 172.21.15.137:0/2824168911 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd806bb80 con 0x7f5bd815ef70 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.381+0000 7f5bdca00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd80c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.382+0000 7f5bdde00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815ef70 0x7f5bd815f400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009d90 tx=0x7f5bd002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=898631176bee207 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.382+0000 7f5bd7e00640 1 -- 172.21.15.137:0/2824168911 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd002cbd0 con 0x7f5bd815ef70 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.382+0000 7f5bd7e00640 1 -- 172.21.15.137:0/2824168911 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd002cd30 con 0x7f5bd815ef70 2024-01-16T14:56:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.382+0000 7f5bd7e00640 1 -- 172.21.15.137:0/2824168911 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd002cea0 con 0x7f5bd815ef70 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815ef70 msgr2=0x7f5bd815f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815ef70 0x7f5bd815f400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009d90 tx=0x7f5bd002be20 comp rx=0 tx=0).stop 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2824168911 shutdown_connections 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815ef70 0x7f5bd815f400 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2824168911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd80c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2824168911 >> 172.21.15.137:0/2824168911 conn(0x7f5bd80b2ce0 msgr2=0x7f5bd80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2824168911 shutdown_connections 2024-01-16T14:56:22.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.383+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2824168911 wait complete. 2024-01-16T14:56:22.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.385+0000 7f5bdfe00640 1 Processor -- start 2024-01-16T14:56:22.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.385+0000 7f5bdfe00640 1 -- start start 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd8158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 0x7f5bd80b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80c8be0 con 0x7f5bd815ef70 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80cd660 con 0x7f5bd815d940 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80cd940 con 0x7f5bd806b4f0 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd8158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55206/0 (socket says 172.21.15.137:55206) 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 -- 172.21.15.137:0/2997284142 learned_addr learned my addr 172.21.15.137:0/2997284142 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdde00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 0x7f5bd80b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 -- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 msgr2=0x7f5bd8158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd8158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 -- 172.21.15.137:0/2997284142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 msgr2=0x7f5bd80b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 0x7f5bd80b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdca00640 1 -- 172.21.15.137:0/2997284142 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0009400 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.386+0000 7f5bdde00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 0x7f5bd80b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bdca00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000ebe0 tx=0x7f5bc000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0009760 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc000cc30 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc000eee0 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd0009a70 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd80cdf00 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5bc0005c90 con 0x7f5bd815d940 2024-01-16T14:56:22.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.387+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5bc0013930 con 0x7f5bd815d940 2024-01-16T14:56:22.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.388+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd80c8d50 con 0x7f5bd815d940 2024-01-16T14:56:22.376 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.394+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5bd80c8d50 con 0x7f5bd815d940 2024-01-16T14:56:22.381 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.399+0000 7fe19ec32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:56:22.410 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.427+0000 7f0185032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:56:22.445 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.462+0000 7fe19ec32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:56:22.454 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.471+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c57d8db5-afc1-4ae9-8e25-5dbc99da5467", "id": 1} v 0) v1 -- 0x7f5bd80675d0 con 0x7f5bd815d940 2024-01-16T14:56:22.461 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.478+0000 7f5bd5600640 1 -- 172.21.15.137:0/2997284142 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c57d8db5-afc1-4ae9-8e25-5dbc99da5467", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5bc0029020 con 0x7f5bd815d940 2024-01-16T14:56:22.461 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 msgr2=0x7f5bd80b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000ebe0 tx=0x7f5bc000c520 comp rx=0 tx=0).stop 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 shutdown_connections 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd815ef70 0x7f5bd80b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd815d940 0x7f5bd80b3f20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 --2- 172.21.15.137:0/2997284142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bd806b4f0 0x7f5bd8158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 >> 172.21.15.137:0/2997284142 conn(0x7f5bd80b2ce0 msgr2=0x7f5bd80cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 shutdown_connections 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.480+0000 7f5bdfe00640 1 -- 172.21.15.137:0/2997284142 wait complete. 2024-01-16T14:56:22.472 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.489+0000 7f0185032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:56:22.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 6ba40f64-5a36-4738-9543-ef1f8b4f4bf7 2 2024-01-16T14:56:22.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.517+0000 7fe19ec32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:56:22.520 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.537+0000 7f0185032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:56:22.579 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.597+0000 7fe19ec32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:56:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.649+0000 7fe19ec32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:56:22.634 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.651+0000 7f0185032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:56:22.649 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.667+0000 7fb60ca00640 1 Processor -- start 2024-01-16T14:56:22.649 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.667+0000 7fb60ca00640 1 -- start start 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.668+0000 7fb60ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.668+0000 7fb60ca00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6080d1e00 con 0x7fb608160e00 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb60ca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6080b38a0 con 0x7fb6080b3b20 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb60ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6080d3c40 con 0x7fb608160a20 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:59078/0 (socket says 172.21.15.137:59078) 2024-01-16T14:56:22.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 -- 172.21.15.137:0/1694033773 learned_addr learned my addr 172.21.15.137:0/1694033773 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:22.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 -- 172.21.15.137:0/1694033773 >> v1:172.21.15.137:6789/0 conn(0x7fb6080b3b20 legacy=0x7fb6080b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 -- 172.21.15.137:0/1694033773 >> v1:172.21.15.125:6789/0 conn(0x7fb608160e00 legacy=0x7fb6080b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.669+0000 7fb606800640 1 -- 172.21.15.137:0/1694033773 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6080d3e50 con 0x7fb608160a20 2024-01-16T14:56:22.653 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb606800640 1 --2- 172.21.15.137:0/1694033773 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80098a0 tx=0x7fb5f802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac6702428c236e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.653 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb605400640 1 -- 172.21.15.137:0/1694033773 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f802f830 con 0x7fb608160a20 2024-01-16T14:56:22.653 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb605400640 1 -- 172.21.15.137:0/1694033773 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f802f990 con 0x7fb608160a20 2024-01-16T14:56:22.653 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb605400640 1 -- 172.21.15.137:0/1694033773 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f802fd20 con 0x7fb608160a20 2024-01-16T14:56:22.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb60ca00640 1 -- 172.21.15.137:0/1694033773 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 msgr2=0x7fb6080cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.671+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1694033773 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80098a0 tx=0x7fb5f802ba10 comp rx=0 tx=0).stop 2024-01-16T14:56:22.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.672+0000 7fb60ca00640 1 -- 172.21.15.137:0/1694033773 shutdown_connections 2024-01-16T14:56:22.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.672+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1694033773 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080cf850 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.672+0000 7fb60ca00640 1 -- 172.21.15.137:0/1694033773 >> 172.21.15.137:0/1694033773 conn(0x7fb6080d3120 msgr2=0x7fb6080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.672+0000 7fb60ca00640 1 -- 172.21.15.137:0/1694033773 shutdown_connections 2024-01-16T14:56:22.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.672+0000 7fb60ca00640 1 -- 172.21.15.137:0/1694033773 wait complete. 2024-01-16T14:56:22.656 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.673+0000 7fb60ca00640 1 Processor -- start 2024-01-16T14:56:22.656 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.674+0000 7fb60ca00640 1 -- start start 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.674+0000 7fb60ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.674+0000 7fb60ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.674+0000 7fb60ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb608160e00 0x7fb6080b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb60ca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60815aa60 con 0x7fb608160e00 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb60ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60815a130 con 0x7fb6080b3b20 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb60ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60815a410 con 0x7fb608160a20 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb606800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb606800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55214/0 (socket says 172.21.15.137:55214) 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb606800640 1 -- 172.21.15.137:0/1267266220 learned_addr learned my addr 172.21.15.137:0/1267266220 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb607200640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb608160e00 0x7fb6080b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 -- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 msgr2=0x7fb608157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.659 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 -- 172.21.15.137:0/1267266220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb608160e00 msgr2=0x7fb6080b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb608160e00 0x7fb6080b5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb605e00640 1 -- 172.21.15.137:0/1267266220 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80094d0 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.675+0000 7fb606800640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb605e00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080ca230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb6081611e0 tx=0x7fb5f000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00052d0 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f0005430 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0003810 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60815a6f0 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6080b8cb0 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb5f0003c70 con 0x7fb608160a20 2024-01-16T14:56:22.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.676+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb5f0012840 con 0x7fb608160a20 2024-01-16T14:56:22.661 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.677+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60815abd0 con 0x7fb608160a20 2024-01-16T14:56:22.664 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.681+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb5f0017070 con 0x7fb608160a20 2024-01-16T14:56:22.714 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.731+0000 7f0185032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:22.741 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.759+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6ba40f64-5a36-4738-9543-ef1f8b4f4bf7", "id": 2} v 0) v1 -- 0x7fb608067b60 con 0x7fb608160a20 2024-01-16T14:56:22.749 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.766+0000 7fb5fea00640 1 -- 172.21.15.137:0/1267266220 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ba40f64-5a36-4738-9543-ef1f8b4f4bf7", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb5f00124e0 con 0x7fb608160a20 2024-01-16T14:56:22.749 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-01-16T14:56:22.750 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 msgr2=0x7fb6080ca230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080ca230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb6081611e0 tx=0x7fb5f000c820 comp rx=0 tx=0).stop 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 shutdown_connections 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb608160e00 0x7fb6080b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb608160a20 0x7fb6080ca230 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 --2- 172.21.15.137:0/1267266220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6080b3b20 0x7fb608157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 >> 172.21.15.137:0/1267266220 conn(0x7fb6080d3120 msgr2=0x7fb60806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 shutdown_connections 2024-01-16T14:56:22.751 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:22.768+0000 7fb60ca00640 1 -- 172.21.15.137:0/1267266220 wait complete. 2024-01-16T14:56:22.754 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.772+0000 7fe19ec32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:56:22.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 41d61215-6fd0-43de-a7b5-c4ac99339e19 3 2024-01-16T14:56:22.768 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.785+0000 7f0185032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:56:22.828 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.845+0000 7f0185032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:56:22.838 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.855+0000 7fe19ec32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:22.882 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:22.899+0000 7f0185032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:56:22.894 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.911+0000 7fe19ec32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:56:22.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:22.975+0000 7fe19ec32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:56:23.000 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.018+0000 7f8af1800640 1 Processor -- start 2024-01-16T14:56:23.000 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.018+0000 7f8af1800640 1 -- start start 2024-01-16T14:56:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.019+0000 7f8af1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 0x7f8aec0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.019+0000 7f8af1800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8aec0b31d0 0x7f8aec0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.019+0000 7f8af1800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0d0320 con 0x7f8aec15d4b0 2024-01-16T14:56:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.019+0000 7f8af1800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0d1c60 con 0x7f8aec0b31d0 2024-01-16T14:56:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.019+0000 7f8af1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b2730 con 0x7f8aec0cf9a0 2024-01-16T14:56:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8aec0b31d0 0x7f8aec0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8aeb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 0x7f8aec0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8aec0b31d0 0x7f8aec0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55228/0 (socket says 172.21.15.137:55228) 2024-01-16T14:56:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 -- 172.21.15.137:0/3627362358 learned_addr learned my addr 172.21.15.137:0/3627362358 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 -- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 msgr2=0x7f8aec0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 0x7f8aec0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 -- 172.21.15.137:0/3627362358 >> v1:172.21.15.125:6789/0 conn(0x7f8aec15d4b0 legacy=0x7f8aec0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8af0e00640 1 -- 172.21.15.137:0/3627362358 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0d1ec0 con 0x7f8aec0b31d0 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.020+0000 7f8aeb400640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 0x7f8aec0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.021+0000 7f8af0e00640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec0b31d0 0x7f8aec0b3660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4009e80 tx=0x7f8ae402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5015de2057396c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8aeaa00640 1 -- 172.21.15.137:0/3627362358 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae402cae0 con 0x7f8aec0b31d0 2024-01-16T14:56:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8aeaa00640 1 -- 172.21.15.137:0/3627362358 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae402cc40 con 0x7f8aec0b31d0 2024-01-16T14:56:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8aeaa00640 1 -- 172.21.15.137:0/3627362358 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae40343f0 con 0x7f8aec0b31d0 2024-01-16T14:56:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 -- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec0b31d0 msgr2=0x7f8aec0b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec0b31d0 0x7f8aec0b3660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4009e80 tx=0x7f8ae402bf40 comp rx=0 tx=0).stop 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 -- 172.21.15.137:0/3627362358 shutdown_connections 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec0b31d0 0x7f8aec0b3660 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 --2- 172.21.15.137:0/3627362358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0cf9a0 0x7f8aec0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.022+0000 7f8af1800640 1 -- 172.21.15.137:0/3627362358 >> 172.21.15.137:0/3627362358 conn(0x7f8aec0d3000 msgr2=0x7f8aec0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.023+0000 7f8af1800640 1 -- 172.21.15.137:0/3627362358 shutdown_connections 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.023+0000 7f8af1800640 1 -- 172.21.15.137:0/3627362358 wait complete. 2024-01-16T14:56:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.024+0000 7f8af1800640 1 Processor -- start 2024-01-16T14:56:23.007 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.024+0000 7f8af1800640 1 -- start start 2024-01-16T14:56:23.009 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8aec0cf9a0 0x7f8aec1f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 0x7f8aec151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0cd3e0 con 0x7f8aec0cf9a0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec15f4d0 con 0x7f8aec15d4b0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.025+0000 7f8af1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec15f7b0 con 0x7f8aec0b31d0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8af0e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 0x7f8aec151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:59100/0 (socket says 172.21.15.137:59100) 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 -- 172.21.15.137:0/407770400 learned_addr learned my addr 172.21.15.137:0/407770400 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aeb400640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8aec0cf9a0 0x7f8aec1f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 -- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 msgr2=0x7f8aec151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 0x7f8aec151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 -- 172.21.15.137:0/407770400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8aec0cf9a0 msgr2=0x7f8aec1f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8aec0cf9a0 0x7f8aec1f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 -- 172.21.15.137:0/407770400 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad4009400 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8af0e00640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 0x7f8aec151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.026+0000 7f8aebe00640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c400 tx=0x7f8adc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0052d0 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc005430 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc003830 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae4009b30 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec15fd70 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8adc003c90 con 0x7f8aec0b31d0 2024-01-16T14:56:23.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.027+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8adc011a20 con 0x7f8aec0b31d0 2024-01-16T14:56:23.012 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.028+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec15ff80 con 0x7f8aec0b31d0 2024-01-16T14:56:23.013 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.031+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8adc017070 con 0x7f8aec0b31d0 2024-01-16T14:56:23.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.034+0000 7fe19ec32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:56:23.091 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.108+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "41d61215-6fd0-43de-a7b5-c4ac99339e19", "id": 3} v 0) v1 -- 0x7f8aec067450 con 0x7f8aec0b31d0 2024-01-16T14:56:23.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.117+0000 7f8ae3e00640 1 -- 172.21.15.137:0/407770400 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41d61215-6fd0-43de-a7b5-c4ac99339e19", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8adc011590 con 0x7f8aec0b31d0 2024-01-16T14:56:23.100 INFO:teuthology.orchestra.run.smithi137.stdout:3 2024-01-16T14:56:23.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.118+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 msgr2=0x7f8aec0ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.118+0000 7f8af1800640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c400 tx=0x7f8adc00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 shutdown_connections 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aec15d4b0 0x7f8aec151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8aec0cf9a0 0x7f8aec1f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 --2- 172.21.15.137:0/407770400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aec0b31d0 0x7f8aec0ca0d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 >> 172.21.15.137:0/407770400 conn(0x7f8aec0d3000 msgr2=0x7f8aec0b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 shutdown_connections 2024-01-16T14:56:23.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.119+0000 7f8af1800640 1 -- 172.21.15.137:0/407770400 wait complete. 2024-01-16T14:56:23.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 9394c828-c510-4367-be90-fa386b887393 4 2024-01-16T14:56:23.329 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.346+0000 7f0185032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:56:23.360 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.377+0000 7f5775800640 1 Processor -- start 2024-01-16T14:56:23.360 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.377+0000 7f5775800640 1 -- start start 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f5775800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700cfcf0 0x7f5770160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f5775800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57700d00d0 0x7f5770160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f5775800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700d4160 con 0x7f57700d3910 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f5775800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57700ceff0 con 0x7f57700d00d0 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f5775800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700cf410 con 0x7f57700cfcf0 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f576f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57700d00d0 0x7f5770160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f576f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57700d00d0 0x7f5770160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55256/0 (socket says 172.21.15.137:55256) 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f576f400640 1 -- 172.21.15.137:0/1709925086 learned_addr learned my addr 172.21.15.137:0/1709925086 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:23.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.379+0000 7f576ea00640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700cfcf0 0x7f5770160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.380+0000 7f576f400640 1 -- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700cfcf0 msgr2=0x7f5770160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:23.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.380+0000 7f576f400640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700cfcf0 0x7f5770160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.380+0000 7f576f400640 1 -- 172.21.15.137:0/1709925086 >> v1:172.21.15.125:6789/0 conn(0x7f57700d3910 legacy=0x7f57700b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.380+0000 7f576f400640 1 -- 172.21.15.137:0/1709925086 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700b2420 con 0x7f57700d00d0 2024-01-16T14:56:23.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.381+0000 7f576f400640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700d00d0 0x7f5770160c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5768009e80 tx=0x7f576802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6619496073dd485e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.381+0000 7f576e000640 1 -- 172.21.15.137:0/1709925086 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576802ca50 con 0x7f57700d00d0 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.381+0000 7f576e000640 1 -- 172.21.15.137:0/1709925086 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576802cbb0 con 0x7f57700d00d0 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.381+0000 7f576e000640 1 -- 172.21.15.137:0/1709925086 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680343f0 con 0x7f57700d00d0 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 -- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700d00d0 msgr2=0x7f5770160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700d00d0 0x7f5770160c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5768009e80 tx=0x7f576802beb0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 -- 172.21.15.137:0/1709925086 shutdown_connections 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700d00d0 0x7f5770160c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 --2- 172.21.15.137:0/1709925086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700cfcf0 0x7f5770160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 -- 172.21.15.137:0/1709925086 >> 172.21.15.137:0/1709925086 conn(0x7f577009c530 msgr2=0x7f57700d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 -- 172.21.15.137:0/1709925086 shutdown_connections 2024-01-16T14:56:23.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.382+0000 7f5775800640 1 -- 172.21.15.137:0/1709925086 wait complete. 2024-01-16T14:56:23.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.383+0000 7f5775800640 1 Processor -- start 2024-01-16T14:56:23.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.384+0000 7f5775800640 1 -- start start 2024-01-16T14:56:23.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.384+0000 7f5775800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.384+0000 7f5775800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700d00d0 0x7f577015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f5775800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57700d3910 0x7f577015d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f5775800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770159c70 con 0x7f57700d3910 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f5775800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57700cb0a0 con 0x7f57700cfcf0 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f5775800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700cc4f0 con 0x7f57700d00d0 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f5767e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700d00d0 0x7f577015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55258/0 (socket says 172.21.15.137:55258) 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 -- 172.21.15.137:0/5944168 learned_addr learned my addr 172.21.15.137:0/5944168 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576f400640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57700d3910 0x7f577015d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 -- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700d00d0 msgr2=0x7f577015caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700d00d0 0x7f577015caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 -- 172.21.15.137:0/5944168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57700d3910 msgr2=0x7f577015d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57700d3910 0x7f577015d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 -- 172.21.15.137:0/5944168 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768009b30 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.385+0000 7f576ea00640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f576000e8a0 tx=0x7f576000eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760009510 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576000c940 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000ccf0 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57700cc790 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57700ccd50 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5760005c90 con 0x7f57700cfcf0 2024-01-16T14:56:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.386+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5760006840 con 0x7f57700cfcf0 2024-01-16T14:56:23.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.387+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577006ae00 con 0x7f57700cfcf0 2024-01-16T14:56:23.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.390+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5760021060 con 0x7f57700cfcf0 2024-01-16T14:56:23.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.467+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9394c828-c510-4367-be90-fa386b887393", "id": 4} v 0) v1 -- 0x7f57700675d0 con 0x7f57700cfcf0 2024-01-16T14:56:23.458 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.475+0000 7f5766a00640 1 -- 172.21.15.137:0/5944168 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9394c828-c510-4367-be90-fa386b887393", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5760029020 con 0x7f57700cfcf0 2024-01-16T14:56:23.458 INFO:teuthology.orchestra.run.smithi137.stdout:4 2024-01-16T14:56:23.459 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 msgr2=0x7f57701592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.459 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f576000e8a0 tx=0x7f576000eca0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 shutdown_connections 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57700d3910 0x7f577015d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57700d00d0 0x7f577015caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 --2- 172.21.15.137:0/5944168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57700cfcf0 0x7f57701592b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 >> 172.21.15.137:0/5944168 conn(0x7f577009c530 msgr2=0x7f57700b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 shutdown_connections 2024-01-16T14:56:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.477+0000 7f5775800640 1 -- 172.21.15.137:0/5944168 wait complete. 2024-01-16T14:56:23.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new ae8630f8-7c9f-43fe-ba8b-47066f53b1a9 5 2024-01-16T14:56:23.488 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.505+0000 7f0185032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:56:23.489 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.507+0000 7fe19ec32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:56:23.628 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.645+0000 7f0185032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:56:23.657 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.674+0000 7fe19ec32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:56:23.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.683+0000 7f1b33e00640 1 Processor -- start 2024-01-16T14:56:23.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.683+0000 7f1b33e00640 1 -- start start 2024-01-16T14:56:23.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b33e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1b2c161740 0x7f1b2c168380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b33e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c0cf260 con 0x7f1b2c0d0560 2024-01-16T14:56:23.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b33e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c0cf4c0 con 0x7f1b2c0c9870 2024-01-16T14:56:23.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b33e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c0d2ef0 con 0x7f1b2c161740 2024-01-16T14:56:23.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b31400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f1b2c0c9870 0x7f1b2c0ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:36664/0 (socket says 172.21.15.137:36664) 2024-01-16T14:56:23.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b31400640 1 -- 172.21.15.137:0/1154906178 learned_addr learned my addr 172.21.15.137:0/1154906178 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b31e00640 1 --2- 172.21.15.137:0/1154906178 >> v2:172.21.15.125:3300/0 conn(0x7f1b2c161740 0x7f1b2c168380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.685+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148267725 0 0) 0x7f1b2c0cf4c0 con 0x7f1b2c0c9870 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c003520 con 0x7f1b2c0c9870 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632608895 0 0) 0x7f1b0c003520 con 0x7f1b2c0c9870 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 >> v2:172.21.15.125:3300/0 conn(0x7f1b2c161740 msgr2=0x7f1b2c168380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 --2- 172.21.15.137:0/1154906178 >> v2:172.21.15.125:3300/0 conn(0x7f1b2c161740 0x7f1b2c168380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 >> v1:172.21.15.125:6789/0 conn(0x7f1b2c0d0560 legacy=0x7f1b2c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0cf7a0 con 0x7f1b2c0c9870 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1b1c001690 con 0x7f1b2c0c9870 2024-01-16T14:56:23.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 >> v1:172.21.15.137:6789/0 conn(0x7f1b2c0c9870 legacy=0x7f1b2c0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b0c003970 0x7f1b0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 0x7f1b0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c005180 0x7f1b0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0d2ef0 con 0x7f1b0c005180 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c00c020 con 0x7f1b0c003970 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c00c1e0 con 0x7f1b0c0047c0 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.686+0000 7f1b30a00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 0x7f1b0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b31e00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c005180 0x7f1b0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b31400640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b0c003970 0x7f1b0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b30a00640 1 -- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b0c003970 msgr2=0x7f1b0c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b30a00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b0c003970 0x7f1b0c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b30a00640 1 -- 172.21.15.137:0/1154906178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c005180 msgr2=0x7f1b0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b30a00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c005180 0x7f1b0c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b30a00640 1 -- 172.21.15.137:0/1154906178 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c003520 con 0x7f1b0c0047c0 2024-01-16T14:56:23.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b31e00640 1 -- 172.21.15.137:0/1154906178 reap_dead start 2024-01-16T14:56:23.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.687+0000 7f1b31400640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b0c003970 0x7f1b0c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:23.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.688+0000 7f1b30a00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 0x7f1b0c004c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1b14001750 tx=0x7f1b0c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7f85b4de486ac28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.688+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b140052d0 con 0x7f1b0c0047c0 2024-01-16T14:56:23.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.688+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1402e050 con 0x7f1b0c0047c0 2024-01-16T14:56:23.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.688+0000 7f1b2be00640 1 -- 172.21.15.137:0/1154906178 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003830 con 0x7f1b0c0047c0 2024-01-16T14:56:23.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 -- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 msgr2=0x7f1b0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 0x7f1b0c004c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1b14001750 tx=0x7f1b0c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 -- 172.21.15.137:0/1154906178 shutdown_connections 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 --2- 172.21.15.137:0/1154906178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b0c0047c0 0x7f1b0c004c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 -- 172.21.15.137:0/1154906178 >> 172.21.15.137:0/1154906178 conn(0x7f1b2c09c560 msgr2=0x7f1b2c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 -- 172.21.15.137:0/1154906178 shutdown_connections 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.689+0000 7f1b33e00640 1 -- 172.21.15.137:0/1154906178 wait complete. 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.691+0000 7f1b33e00640 1 Processor -- start 2024-01-16T14:56:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.691+0000 7f1b33e00640 1 -- start start 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 0x7f1b2c157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2c145530 0x7f1b2c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c1459b0 con 0x7f1b0c0047c0 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0c66e0 con 0x7f1b2c1599b0 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b33e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0cbfb0 con 0x7f1b2c145530 2024-01-16T14:56:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55274/0 (socket says 172.21.15.137:55274) 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b30a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2c145530 0x7f1b2c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 -- 172.21.15.137:0/4197681530 learned_addr learned my addr 172.21.15.137:0/4197681530 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31400640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 0x7f1b2c157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 -- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2c145530 msgr2=0x7f1b2c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2c145530 0x7f1b2c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 -- 172.21.15.137:0/4197681530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 msgr2=0x7f1b2c157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 0x7f1b2c157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31e00640 1 -- 172.21.15.137:0/4197681530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c15c740 con 0x7f1b2c1599b0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.692+0000 7f1b31400640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 0x7f1b2c157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b31e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c08c690 tx=0x7f1b2400f1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400e040 con 0x7f1b2c1599b0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2400f040 con 0x7f1b2c1599b0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c15c9c0 con 0x7f1b2c1599b0 2024-01-16T14:56:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24015430 con 0x7f1b2c1599b0 2024-01-16T14:56:23.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c0cb960 con 0x7f1b2c1599b0 2024-01-16T14:56:23.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.693+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1b24015890 con 0x7f1b2c1599b0 2024-01-16T14:56:23.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.694+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f1b2401d400 con 0x7f1b2c1599b0 2024-01-16T14:56:23.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.694+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c0748c0 con 0x7f1b2c1599b0 2024-01-16T14:56:23.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.697+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1b24015b40 con 0x7f1b2c1599b0 2024-01-16T14:56:23.682 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.699+0000 7f0185032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:56:23.757 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.775+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ae8630f8-7c9f-43fe-ba8b-47066f53b1a9", "id": 5} v 0) v1 -- 0x7f1b2c0675d0 con 0x7f1b2c1599b0 2024-01-16T14:56:23.767 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.782+0000 7f1b29600640 1 -- 172.21.15.137:0/4197681530 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae8630f8-7c9f-43fe-ba8b-47066f53b1a9", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1b24015ce0 con 0x7f1b2c1599b0 2024-01-16T14:56:23.767 INFO:teuthology.orchestra.run.smithi137.stdout:5 2024-01-16T14:56:23.768 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.783+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 msgr2=0x7f1b2c0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.768 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.783+0000 7f1b33e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c08c690 tx=0x7f1b2400f1b0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 shutdown_connections 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b2c1599b0 0x7f1b2c0cc440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2c145530 0x7f1b2c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 --2- 172.21.15.137:0/4197681530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0c0047c0 0x7f1b2c157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 >> 172.21.15.137:0/4197681530 conn(0x7f1b2c09c560 msgr2=0x7f1b2c15fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 shutdown_connections 2024-01-16T14:56:23.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:23.784+0000 7f1b33e00640 1 -- 172.21.15.137:0/4197681530 wait complete. 2024-01-16T14:56:23.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e 6 2024-01-16T14:56:23.792 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.809+0000 7f0185032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:23.806 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.824+0000 7fe19ec32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:56:23.862 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.880+0000 7fe19ec32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:56:23.901 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:23.918+0000 7f0185032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:56:23.976 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:23.993+0000 7fe19ec32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:23.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.012+0000 7fceb7200640 1 Processor -- start 2024-01-16T14:56:23.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.012+0000 7fceb7200640 1 -- start start 2024-01-16T14:56:23.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.013+0000 7fceb7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:23.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.013+0000 7fceb7200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80b2730 con 0x7fceb80cf9a0 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.013+0000 7fceb7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80d1ec0 con 0x7fceb80b41e0 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.013+0000 7fceb7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80b29d0 con 0x7fceb815d4b0 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb6800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fceb80b41e0 0x7fceb80ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:36672/0 (socket says 172.21.15.137:36672) 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb6800640 1 -- 172.21.15.137:0/1064998580 learned_addr learned my addr 172.21.15.137:0/1064998580 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb5e00640 1 -- 172.21.15.137:0/1064998580 >> v1:172.21.15.137:6789/0 conn(0x7fceb80b41e0 legacy=0x7fceb80ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb5400640 1 -- 172.21.15.137:0/1064998580 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504174845 0 0) 0x7fceb80d1ec0 con 0x7fceb80b41e0 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb5e00640 1 -- 172.21.15.137:0/1064998580 >> v1:172.21.15.125:6789/0 conn(0x7fceb80cf9a0 legacy=0x7fceb80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.014+0000 7fceb5e00640 1 -- 172.21.15.137:0/1064998580 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb80b2cb0 con 0x7fceb815d4b0 2024-01-16T14:56:23.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.015+0000 7fceb5e00640 1 --2- 172.21.15.137:0/1064998580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb80d1550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009800 tx=0x7fcea802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74868d270921820b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:23.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.015+0000 7fceb5400640 1 -- 172.21.15.137:0/1064998580 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031890 con 0x7fceb815d4b0 2024-01-16T14:56:23.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.015+0000 7fceb5400640 1 -- 172.21.15.137:0/1064998580 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea80319f0 con 0x7fceb815d4b0 2024-01-16T14:56:23.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.015+0000 7fceb5400640 1 -- 172.21.15.137:0/1064998580 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031d80 con 0x7fceb815d4b0 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 -- 172.21.15.137:0/1064998580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 msgr2=0x7fceb80d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 --2- 172.21.15.137:0/1064998580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb80d1550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009800 tx=0x7fcea802b930 comp rx=0 tx=0).stop 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 -- 172.21.15.137:0/1064998580 shutdown_connections 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 --2- 172.21.15.137:0/1064998580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb80d1550 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 -- 172.21.15.137:0/1064998580 >> 172.21.15.137:0/1064998580 conn(0x7fceb80d3000 msgr2=0x7fceb80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 -- 172.21.15.137:0/1064998580 shutdown_connections 2024-01-16T14:56:23.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.016+0000 7fceb7200640 1 -- 172.21.15.137:0/1064998580 wait complete. 2024-01-16T14:56:24.000 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.018+0000 7fceb7200640 1 Processor -- start 2024-01-16T14:56:24.000 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.018+0000 7fceb7200640 1 -- start start 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 0x7fceb8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb81f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb8151950 con 0x7fceb80b41e0 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb8160e60 con 0x7fceb80cf9a0 2024-01-16T14:56:24.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80cb740 con 0x7fceb815d4b0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceafe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceafe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55290/0 (socket says 172.21.15.137:55290) 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceafe00640 1 -- 172.21.15.137:0/811115257 learned_addr learned my addr 172.21.15.137:0/811115257 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb6800640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb81f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.019+0000 7fceb5e00640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 0x7fceb8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 -- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 msgr2=0x7fceb81f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb81f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 -- 172.21.15.137:0/811115257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 msgr2=0x7fceb8151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 0x7fceb8151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 -- 172.21.15.137:0/811115257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8009400 con 0x7fceb80cf9a0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceb5e00640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 0x7fceb8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceafe00640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcea000c450 tx=0x7fcea000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:24.004 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00052d0 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea0005430 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0012870 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb815da10 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.020+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb80cc2c0 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.021+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcea0012cd0 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.021+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fcea0011380 con 0x7fceb80cf9a0 2024-01-16T14:56:24.005 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.022+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb80cc450 con 0x7fceb80cf9a0 2024-01-16T14:56:24.007 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.024+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcea00035f0 con 0x7fceb80cf9a0 2024-01-16T14:56:24.062 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:24.078+0000 7f0185032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:24.084 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.102+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e", "id": 6} v 0) v1 -- 0x7fceb8067450 con 0x7fceb80cf9a0 2024-01-16T14:56:24.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:24.107+0000 7fe19ec32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:56:24.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.110+0000 7fceae000640 1 -- 172.21.15.137:0/811115257 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcea0003790 con 0x7fceb80cf9a0 2024-01-16T14:56:24.095 INFO:teuthology.orchestra.run.smithi137.stdout:6 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 msgr2=0x7fceb8160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcea000c450 tx=0x7fcea000c910 comp rx=0 tx=0).stop 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 shutdown_connections 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fceb815d4b0 0x7fceb81f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fceb80cf9a0 0x7fceb8160920 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 --2- 172.21.15.137:0/811115257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fceb80b41e0 0x7fceb8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.111+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 >> 172.21.15.137:0/811115257 conn(0x7fceb80d3000 msgr2=0x7fceb80cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:24.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.112+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 shutdown_connections 2024-01-16T14:56:24.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.112+0000 7fceb7200640 1 -- 172.21.15.137:0/811115257 wait complete. 2024-01-16T14:56:24.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 2defd956-d6b8-499c-a7cf-877615a2e0bc 7 2024-01-16T14:56:24.111 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:24.128+0000 7f0185032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:24.259 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:24.276+0000 7fe19ec32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:24.268 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:24.285+0000 7f0185032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:56:24.296 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.313+0000 7f7eeda00640 1 Processor -- start 2024-01-16T14:56:24.296 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.313+0000 7f7eeda00640 1 -- start start 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7eeda00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ee815d4b0 0x7f7ee80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7eeda00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80d0320 con 0x7f7ee80cddf0 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7eeda00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee80b2730 con 0x7f7ee815d4b0 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7eeda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80d1ec0 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7edfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7edfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:59166/0 (socket says 172.21.15.137:59166) 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.315+0000 7f7edfe00640 1 -- 172.21.15.137:0/1174834631 learned_addr learned my addr 172.21.15.137:0/1174834631 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.316+0000 7f7edfe00640 1 -- 172.21.15.137:0/1174834631 >> v2:172.21.15.125:3300/0 conn(0x7f7ee815d4b0 msgr2=0x7f7ee80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.316+0000 7f7edfe00640 1 --2- 172.21.15.137:0/1174834631 >> v2:172.21.15.125:3300/0 conn(0x7f7ee815d4b0 0x7f7ee80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.316+0000 7f7edfe00640 1 -- 172.21.15.137:0/1174834631 >> v1:172.21.15.125:6789/0 conn(0x7f7ee80cddf0 legacy=0x7f7ee80ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.316+0000 7f7edfe00640 1 -- 172.21.15.137:0/1174834631 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80b29d0 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.317+0000 7f7edfe00640 1 --2- 172.21.15.137:0/1174834631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098a0 tx=0x7f7ed002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10b22f9c01dee9df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:24.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.317+0000 7f7ee7400640 1 -- 172.21.15.137:0/1174834631 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0031640 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.317+0000 7f7ee7400640 1 -- 172.21.15.137:0/1174834631 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed00317a0 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.317+0000 7f7ee7400640 1 -- 172.21.15.137:0/1174834631 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0031b30 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 -- 172.21.15.137:0/1174834631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 msgr2=0x7f7ee80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 --2- 172.21.15.137:0/1174834631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098a0 tx=0x7f7ed002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 -- 172.21.15.137:0/1174834631 shutdown_connections 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 --2- 172.21.15.137:0/1174834631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cf9a0 0x7f7ee80cfde0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 --2- 172.21.15.137:0/1174834631 >> v2:172.21.15.125:3300/0 conn(0x7f7ee815d4b0 0x7f7ee80d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 -- 172.21.15.137:0/1174834631 >> 172.21.15.137:0/1174834631 conn(0x7f7ee80d3000 msgr2=0x7f7ee80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 -- 172.21.15.137:0/1174834631 shutdown_connections 2024-01-16T14:56:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.318+0000 7f7eeda00640 1 -- 172.21.15.137:0/1174834631 wait complete. 2024-01-16T14:56:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.320+0000 7f7eeda00640 1 Processor -- start 2024-01-16T14:56:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.320+0000 7f7eeda00640 1 -- start start 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ee80cf9a0 0x7f7ee815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ee815d4b0 0x7f7ee80b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815f420 con 0x7f7ee80cf9a0 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815ebb0 con 0x7f7ee815d4b0 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eeda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee81f30d0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7eed000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ee815d4b0 0x7f7ee80b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:59176/0 (socket says 172.21.15.137:59176) 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.321+0000 7f7ee7e00640 1 -- 172.21.15.137:0/653186067 learned_addr learned my addr 172.21.15.137:0/653186067 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7edfe00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ee80cf9a0 0x7f7ee815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 -- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ee815d4b0 msgr2=0x7f7ee80b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ee815d4b0 0x7f7ee80b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 -- 172.21.15.137:0/653186067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ee80cf9a0 msgr2=0x7f7ee815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ee80cf9a0 0x7f7ee815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 -- 172.21.15.137:0/653186067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00094d0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee7e00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b410 tx=0x7f7ed800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8011830 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed80119b0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8011d60 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee815da10 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.322+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee815e0e0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.323+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ed801a700 con 0x7f7ee80cddf0 2024-01-16T14:56:24.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.323+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7ed8019ce0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.324+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee815f180 con 0x7f7ee80cddf0 2024-01-16T14:56:24.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.326+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7ed8039ce0 con 0x7f7ee80cddf0 2024-01-16T14:56:24.312 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:24.329+0000 7fe19ec32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:24.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.404+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2defd956-d6b8-499c-a7cf-877615a2e0bc", "id": 7} v 0) v1 -- 0x7f7ee815f560 con 0x7f7ee80cddf0 2024-01-16T14:56:24.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.411+0000 7f7ee4c00640 1 -- 172.21.15.137:0/653186067 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2defd956-d6b8-499c-a7cf-877615a2e0bc", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ee815f560 con 0x7f7ee80cddf0 2024-01-16T14:56:24.393 INFO:teuthology.orchestra.run.smithi137.stdout:7 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 msgr2=0x7f7ee815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b410 tx=0x7f7ed800b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 shutdown_connections 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ee815d4b0 0x7f7ee80b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ee80cf9a0 0x7f7ee815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 --2- 172.21.15.137:0/653186067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ee80cddf0 0x7f7ee815aa60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 >> 172.21.15.137:0/653186067 conn(0x7f7ee80d3000 msgr2=0x7f7ee80b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 shutdown_connections 2024-01-16T14:56:24.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:56:24.412+0000 7f7eeda00640 1 -- 172.21.15.137:0/653186067 wait complete. 2024-01-16T14:56:24.410 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T14:56:24.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T14:56:24.416 INFO:tasks.ceph.osd.0:Started 2024-01-16T14:56:24.417 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T14:56:24.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T14:56:24.422 INFO:tasks.ceph.osd.1:Started 2024-01-16T14:56:24.423 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T14:56:24.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T14:56:24.428 INFO:tasks.ceph.osd.2:Started 2024-01-16T14:56:24.428 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-16T14:56:24.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-16T14:56:24.433 INFO:tasks.ceph.osd.3:Started 2024-01-16T14:56:24.433 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-16T14:56:24.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-16T14:56:24.441 INFO:tasks.ceph.osd.4:Started 2024-01-16T14:56:24.441 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-16T14:56:24.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-16T14:56:24.448 INFO:tasks.ceph.osd.5:Started 2024-01-16T14:56:24.448 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-16T14:56:24.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-16T14:56:24.454 INFO:tasks.ceph.osd.6:Started 2024-01-16T14:56:24.454 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-16T14:56:24.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-16T14:56:24.461 INFO:tasks.ceph.osd.7:Started 2024-01-16T14:56:24.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:24.484 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:24.501+0000 7fe19ec32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:56:24.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.680+0000 7fd5bba00640 1 Processor -- start 2024-01-16T14:56:24.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.680+0000 7fd5bba00640 1 -- start start 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.681+0000 7fd5bba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 0x7fd5b40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.681+0000 7fd5bba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 0x7fd5b406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.681+0000 7fd5bba00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b4071de0 con 0x7fd5b406a3e0 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.681+0000 7fd5bba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40af270 con 0x7fd5b40d0b60 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.681+0000 7fd5bba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4072030 con 0x7fd5b40aede0 2024-01-16T14:56:24.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd5b406a3e0 0x7fd5b406a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45868/0 (socket says 172.21.15.125:45868) 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9000640 1 -- 172.21.15.125:0/2622247682 learned_addr learned my addr 172.21.15.125:0/2622247682 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9a00640 1 --2- 172.21.15.125:0/2622247682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 0x7fd5b406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b3e00640 1 --2- 172.21.15.125:0/2622247682 >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 0x7fd5b40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b3400640 1 -- 172.21.15.125:0/2622247682 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608807849 0 0) 0x7fd5b4071de0 con 0x7fd5b406a3e0 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b3400640 1 -- 172.21.15.125:0/2622247682 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd594003520 con 0x7fd5b406a3e0 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9a00640 1 -- 172.21.15.125:0/2622247682 >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 msgr2=0x7fd5b40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9a00640 1 --2- 172.21.15.125:0/2622247682 >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 0x7fd5b40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9a00640 1 -- 172.21.15.125:0/2622247682 >> v1:172.21.15.125:6789/0 conn(0x7fd5b406a3e0 legacy=0x7fd5b406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b9a00640 1 -- 172.21.15.125:0/2622247682 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4072310 con 0x7fd5b40aede0 2024-01-16T14:56:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.682+0000 7fd5b3e00640 1 --2- 172.21.15.125:0/2622247682 >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 0x7fd5b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:24.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.683+0000 7fd5b9a00640 1 --2- 172.21.15.125:0/2622247682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 0x7fd5b406f080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9132db1eff0aa32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:24.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.683+0000 7fd5b3400640 1 -- 172.21.15.125:0/2622247682 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac02cc50 con 0x7fd5b40aede0 2024-01-16T14:56:24.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5b3400640 1 -- 172.21.15.125:0/2622247682 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac02cdb0 con 0x7fd5b40aede0 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5b3400640 1 -- 172.21.15.125:0/2622247682 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac034660 con 0x7fd5b40aede0 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 -- 172.21.15.125:0/2622247682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 msgr2=0x7fd5b406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2622247682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 0x7fd5b406f080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac02bea0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 -- 172.21.15.125:0/2622247682 shutdown_connections 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2622247682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b40aede0 0x7fd5b406f080 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2622247682 >> v2:172.21.15.137:3300/0 conn(0x7fd5b40d0b60 0x7fd5b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 -- 172.21.15.125:0/2622247682 >> 172.21.15.125:0/2622247682 conn(0x7fd5b40cfef0 msgr2=0x7fd5b40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 -- 172.21.15.125:0/2622247682 shutdown_connections 2024-01-16T14:56:24.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.684+0000 7fd5bba00640 1 -- 172.21.15.125:0/2622247682 wait complete. 2024-01-16T14:56:24.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.686+0000 7fd5bba00640 1 Processor -- start 2024-01-16T14:56:24.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.686+0000 7fd5bba00640 1 -- start start 2024-01-16T14:56:24.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b406a3e0 0x7fd5b4150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b40d0b60 0x7fd5b4159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:24.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4153b80 con 0x7fd5b40aede0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4153250 con 0x7fd5b40d0b60 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5bba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4153530 con 0x7fd5b406a3e0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44024/0 (socket says 172.21.15.125:44024) 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 -- 172.21.15.125:0/2885255484 learned_addr learned my addr 172.21.15.125:0/2885255484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b9000640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b406a3e0 0x7fd5b4150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b9a00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b40d0b60 0x7fd5b4159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 -- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b406a3e0 msgr2=0x7fd5b4150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b406a3e0 0x7fd5b4150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 -- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b40d0b60 msgr2=0x7fd5b4159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b40d0b60 0x7fd5b4159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.687+0000 7fd5b3e00640 1 -- 172.21.15.125:0/2885255484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c009400 con 0x7fd5b40aede0 2024-01-16T14:56:24.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b3e00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00e9b0 tx=0x7fd59c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00ca00 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac009a70 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4152f00 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c00cb60 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c013470 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd59c004320 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.688+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd59c005990 con 0x7fd5b40aede0 2024-01-16T14:56:24.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.689+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4153cf0 con 0x7fd5b40aede0 2024-01-16T14:56:24.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.692+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd59c021060 con 0x7fd5b40aede0 2024-01-16T14:56:24.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.754+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b40675d0 con 0x7fd5b40aede0 2024-01-16T14:56:24.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.756+0000 7fd5b0c00640 1 -- 172.21.15.125:0/2885255484 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd59c017070 con 0x7fd5b40aede0 2024-01-16T14:56:24.739 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:24.739 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":10,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:24.407089+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:24.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 msgr2=0x7fd5b4157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:24.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00e9b0 tx=0x7fd59c00ee70 comp rx=0 tx=0).stop 2024-01-16T14:56:24.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 shutdown_connections 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b40d0b60 0x7fd5b4159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5b40aede0 0x7fd5b4157000 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 --2- 172.21.15.125:0/2885255484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5b406a3e0 0x7fd5b4150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 >> 172.21.15.125:0/2885255484 conn(0x7fd5b40cfef0 msgr2=0x7fd5b40b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 shutdown_connections 2024-01-16T14:56:24.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:24.757+0000 7fd5bba00640 1 -- 172.21.15.125:0/2885255484 wait complete. 2024-01-16T14:56:24.756 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T14:56:24.756 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T14:56:25.128 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-16T14:56:25.146+0000 7fab7c467500 -1 Falling back to public interface 2024-01-16T14:56:25.144 INFO:tasks.ceph.osd.0.smithi125.stderr:2024-01-16T14:56:25.161+0000 7f09feb10500 -1 Falling back to public interface 2024-01-16T14:56:25.145 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-16T14:56:25.162+0000 7fe6de8b2500 -1 Falling back to public interface 2024-01-16T14:56:25.153 INFO:tasks.ceph.osd.1.smithi125.stderr:2024-01-16T14:56:25.170+0000 7f764360e500 -1 Falling back to public interface 2024-01-16T14:56:25.153 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:25.170+0000 7f146a422500 -1 Falling back to public interface 2024-01-16T14:56:25.154 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:25.171+0000 7f1468800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-01-16T14:56:25.158 INFO:tasks.ceph.osd.3.smithi125.stderr:2024-01-16T14:56:25.175+0000 7f5512707500 -1 Falling back to public interface 2024-01-16T14:56:25.166 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:25.183+0000 7f1468800640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6826/0: (98) Address already in use 2024-01-16T14:56:25.167 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-16T14:56:25.185+0000 7f74e2a10500 -1 Falling back to public interface 2024-01-16T14:56:25.167 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-16T14:56:25.185+0000 7fe203cb6500 -1 Falling back to public interface 2024-01-16T14:56:25.168 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-16T14:56:25.186+0000 7fe202800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2024-01-16T14:56:25.383 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:25.399+0000 7f0185032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:56:25.523 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:25.540+0000 7fe19ec32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:56:25.560 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:25.695 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:25.712+0000 7f0185032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:56:25.719 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.735+0000 7faf95e00640 1 Processor -- start 2024-01-16T14:56:25.719 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.736+0000 7faf95e00640 1 -- start start 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf95e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf9015d5d0 0x7faf900d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf95e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf900b21a0 0x7faf900d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf95e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf900afd50 con 0x7faf900d1a90 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf95e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf900d1f40 con 0x7faf900b21a0 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf95e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf900d11e0 con 0x7faf9015d5d0 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf8fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf900b21a0 0x7faf900d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf8fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf900b21a0 0x7faf900d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44026/0 (socket says 172.21.15.125:44026) 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.737+0000 7faf8fe00640 1 -- 172.21.15.125:0/1482215029 learned_addr learned my addr 172.21.15.125:0/1482215029 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.738+0000 7faf94a00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf9015d5d0 0x7faf900d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.738+0000 7faf8fe00640 1 -- 172.21.15.125:0/1482215029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf9015d5d0 msgr2=0x7faf900d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.738+0000 7faf8fe00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf9015d5d0 0x7faf900d3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.738+0000 7faf8fe00640 1 -- 172.21.15.125:0/1482215029 >> v1:172.21.15.137:6789/0 conn(0x7faf900d1a90 legacy=0x7faf900cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:25.721 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.738+0000 7faf8fe00640 1 -- 172.21.15.125:0/1482215029 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900cef90 con 0x7faf900b21a0 2024-01-16T14:56:25.722 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.739+0000 7faf8fe00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf900b21a0 0x7faf900d1550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faf780098a0 tx=0x7faf7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fb241a4614cb1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:25.722 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.739+0000 7faf8f400640 1 -- 172.21.15.125:0/1482215029 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78030840 con 0x7faf900b21a0 2024-01-16T14:56:25.722 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.739+0000 7faf8f400640 1 -- 172.21.15.125:0/1482215029 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf780309a0 con 0x7faf900b21a0 2024-01-16T14:56:25.722 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.739+0000 7faf8f400640 1 -- 172.21.15.125:0/1482215029 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78030d30 con 0x7faf900b21a0 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 -- 172.21.15.125:0/1482215029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf900b21a0 msgr2=0x7faf900d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf900b21a0 0x7faf900d1550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faf780098a0 tx=0x7faf7802b980 comp rx=0 tx=0).stop 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 -- 172.21.15.125:0/1482215029 shutdown_connections 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf900b21a0 0x7faf900d1550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 --2- 172.21.15.125:0/1482215029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf9015d5d0 0x7faf900d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 -- 172.21.15.125:0/1482215029 >> 172.21.15.125:0/1482215029 conn(0x7faf900d3090 msgr2=0x7faf900d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:25.723 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 -- 172.21.15.125:0/1482215029 shutdown_connections 2024-01-16T14:56:25.724 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.740+0000 7faf95e00640 1 -- 172.21.15.125:0/1482215029 wait complete. 2024-01-16T14:56:25.725 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.742+0000 7faf95e00640 1 Processor -- start 2024-01-16T14:56:25.725 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.742+0000 7faf95e00640 1 -- start start 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf900b21a0 0x7faf900ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf900d1a90 0x7faf900b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf90156910 con 0x7faf9015d5d0 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900b7240 con 0x7faf900b21a0 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf900b7520 con 0x7faf900d1a90 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44028/0 (socket says 172.21.15.125:44028) 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf95400640 1 -- 172.21.15.125:0/2230494364 learned_addr learned my addr 172.21.15.125:0/2230494364 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf8fe00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf900d1a90 0x7faf900b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:25.727 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.743+0000 7faf94a00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf900b21a0 0x7faf900ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 -- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf900d1a90 msgr2=0x7faf900b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf900d1a90 0x7faf900b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 -- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf900b21a0 msgr2=0x7faf900ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf900b21a0 0x7faf900ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 -- 172.21.15.125:0/2230494364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf780094d0 con 0x7faf9015d5d0 2024-01-16T14:56:25.728 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95400640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7a0 tx=0x7faf8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003b70 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88003cf0 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.744+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf900b7800 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.745+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003ea0 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.745+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf900b7dc0 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.745+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7faf880050d0 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.745+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7faf88012b40 con 0x7faf9015d5d0 2024-01-16T14:56:25.729 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.746+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf900b8fe0 con 0x7faf9015d5d0 2024-01-16T14:56:25.731 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.748+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7faf88012d90 con 0x7faf9015d5d0 2024-01-16T14:56:25.759 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:25.776+0000 7f0185032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:56:25.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.815+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf900b7fe0 con 0x7faf9015d5d0 2024-01-16T14:56:25.799 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.816+0000 7faf8cc00640 1 -- 172.21.15.125:0/2230494364 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7faf88005380 con 0x7faf9015d5d0 2024-01-16T14:56:25.799 INFO:teuthology.misc.health.smithi125.stdout: 2024-01-16T14:56:25.799 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":10,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:24.407089+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:25.800 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 msgr2=0x7faf90149570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:25.800 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7a0 tx=0x7faf8800cc60 comp rx=0 tx=0).stop 2024-01-16T14:56:25.800 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 shutdown_connections 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf9015d5d0 0x7faf90149570 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf900d1a90 0x7faf900b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 --2- 172.21.15.125:0/2230494364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf900b21a0 0x7faf900ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 >> 172.21.15.125:0/2230494364 conn(0x7faf900d3090 msgr2=0x7faf900b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 shutdown_connections 2024-01-16T14:56:25.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:25.817+0000 7faf95e00640 1 -- 172.21.15.125:0/2230494364 wait complete. 2024-01-16T14:56:25.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T14:56:25.826 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:25.844+0000 7fe19ec32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:56:25.875 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:25.892+0000 7f0185032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:56:25.885 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:25.903+0000 7fe19ec32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:56:26.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-16T14:56:26.023+0000 7fe19ec32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:56:26.933 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:26.950+0000 7f0125800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-16T14:56:26.933 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:56:26.950+0000 7f0125800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-16T14:56:28.485 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:28.502+0000 7f146a422500 -1 osd.2 0 log_to_monitors true 2024-01-16T14:56:28.673 INFO:tasks.ceph.osd.0.smithi125.stderr:2024-01-16T14:56:28.690+0000 7f09feb10500 -1 osd.0 0 log_to_monitors true 2024-01-16T14:56:28.677 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-16T14:56:28.694+0000 7fe6de8b2500 -1 osd.4 0 log_to_monitors true 2024-01-16T14:56:28.703 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-16T14:56:28.721+0000 7f74e2a10500 -1 osd.6 0 log_to_monitors true 2024-01-16T14:56:28.711 INFO:tasks.ceph.osd.3.smithi125.stderr:2024-01-16T14:56:28.729+0000 7f5512707500 -1 osd.3 0 log_to_monitors true 2024-01-16T14:56:28.725 INFO:tasks.ceph.osd.1.smithi125.stderr:2024-01-16T14:56:28.743+0000 7f764360e500 -1 osd.1 0 log_to_monitors true 2024-01-16T14:56:28.747 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-16T14:56:28.765+0000 7fe203cb6500 -1 osd.7 0 log_to_monitors true 2024-01-16T14:56:28.875 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-16T14:56:28.893+0000 7fab7c467500 -1 osd.5 0 log_to_monitors true 2024-01-16T14:56:32.176 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-16T14:56:32.193+0000 7f74dc800640 -1 osd.6 0 waiting for initial osdmap 2024-01-16T14:56:32.183 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-16T14:56:32.200+0000 7f74d3200640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.229 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-16T14:56:32.247+0000 7fe6dac00640 -1 osd.4 0 waiting for initial osdmap 2024-01-16T14:56:32.236 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-16T14:56:32.254+0000 7fe6cf800640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.308 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-16T14:56:32.326+0000 7fab76a00640 -1 osd.5 0 waiting for initial osdmap 2024-01-16T14:56:32.309 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-16T14:56:32.327+0000 7fe1fe200640 -1 osd.7 0 waiting for initial osdmap 2024-01-16T14:56:32.314 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-16T14:56:32.332+0000 7fab6d400640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.318 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-16T14:56:32.335+0000 7fe1f4c00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.384 INFO:tasks.ceph.osd.3.smithi125.stderr:2024-01-16T14:56:32.402+0000 7f550cc00640 -1 osd.3 0 waiting for initial osdmap 2024-01-16T14:56:32.386 INFO:tasks.ceph.osd.1.smithi125.stderr:2024-01-16T14:56:32.404+0000 7f763d400640 -1 osd.1 0 waiting for initial osdmap 2024-01-16T14:56:32.393 INFO:tasks.ceph.osd.1.smithi125.stderr:2024-01-16T14:56:32.411+0000 7f7633e00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.393 INFO:tasks.ceph.osd.3.smithi125.stderr:2024-01-16T14:56:32.411+0000 7f5503600640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.399 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:32.418+0000 7f1464200640 -1 osd.2 0 waiting for initial osdmap 2024-01-16T14:56:32.406 INFO:tasks.ceph.osd.2.smithi125.stderr:2024-01-16T14:56:32.424+0000 7f145ac00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.417 INFO:tasks.ceph.osd.0.smithi125.stderr:2024-01-16T14:56:32.436+0000 7f09f9000640 -1 osd.0 0 waiting for initial osdmap 2024-01-16T14:56:32.427 INFO:tasks.ceph.osd.0.smithi125.stderr:2024-01-16T14:56:32.445+0000 7f09efa00640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:32.619 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:32.793 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.811+0000 7f1067600640 1 Processor -- start 2024-01-16T14:56:32.793 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.811+0000 7f1067600640 1 -- start start 2024-01-16T14:56:32.795 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1067600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f106013d100 0x7f106015e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:32.795 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1067600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106015b5b0 con 0x7f106006c8f0 2024-01-16T14:56:32.795 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1067600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106015b2d0 con 0x7f106013cd20 2024-01-16T14:56:32.795 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1067600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106015dd30 con 0x7f106013d100 2024-01-16T14:56:32.795 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1065600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f106013d100 0x7f106015e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1065600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f106013d100 0x7f106015e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39088/0 (socket says 172.21.15.125:39088) 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f105fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f106006c8f0 0x7f106015a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33694/0 (socket says 172.21.15.125:33694) 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.813+0000 7f1065600640 1 -- 172.21.15.125:0/2025437715 learned_addr learned my addr 172.21.15.125:0/2025437715 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.814+0000 7f1065600640 1 -- 172.21.15.125:0/2025437715 >> v1:172.21.15.137:6789/0 conn(0x7f106013cd20 legacy=0x7f106006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.814+0000 7f1065600640 1 -- 172.21.15.125:0/2025437715 >> v1:172.21.15.125:6789/0 conn(0x7f106006c8f0 legacy=0x7f106015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.814+0000 7f1065600640 1 -- 172.21.15.125:0/2025437715 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106015e010 con 0x7f106013d100 2024-01-16T14:56:32.796 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.814+0000 7f105f400640 1 -- 172.21.15.125:0/2025437715 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504895875 0 0) 0x7f106015b5b0 con 0x7f106006c8f0 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.815+0000 7f1065600640 1 --2- 172.21.15.125:0/2025437715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f106015e810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1058009e50 tx=0x7f105802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd262a7a54c859d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.815+0000 7f105f400640 1 -- 172.21.15.125:0/2025437715 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105802ca80 con 0x7f106013d100 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.815+0000 7f105f400640 1 -- 172.21.15.125:0/2025437715 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105802cbe0 con 0x7f106013d100 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.815+0000 7f105f400640 1 -- 172.21.15.125:0/2025437715 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10580343f0 con 0x7f106013d100 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 -- 172.21.15.125:0/2025437715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 msgr2=0x7f106015e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 --2- 172.21.15.125:0/2025437715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f106015e810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1058009e50 tx=0x7f105802bee0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 -- 172.21.15.125:0/2025437715 shutdown_connections 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 --2- 172.21.15.125:0/2025437715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f106015e810 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 -- 172.21.15.125:0/2025437715 >> 172.21.15.125:0/2025437715 conn(0x7f106006dd50 msgr2=0x7f106006a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 -- 172.21.15.125:0/2025437715 shutdown_connections 2024-01-16T14:56:32.798 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.816+0000 7f1067600640 1 -- 172.21.15.125:0/2025437715 wait complete. 2024-01-16T14:56:32.799 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.817+0000 7f1067600640 1 Processor -- start 2024-01-16T14:56:32.799 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.818+0000 7f1067600640 1 -- start start 2024-01-16T14:56:32.801 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106006c8f0 0x7f10601590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106013cd20 0x7f1060159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b16b0 con 0x7f106013d100 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b42b0 con 0x7f106006c8f0 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1067600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1060151ea0 con 0x7f106013cd20 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39104/0 (socket says 172.21.15.125:39104) 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 -- 172.21.15.125:0/3591652793 learned_addr learned my addr 172.21.15.125:0/3591652793 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 -- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106013cd20 msgr2=0x7f1060159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106013cd20 0x7f1060159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 -- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106006c8f0 msgr2=0x7f10601590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106006c8f0 0x7f10601590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 -- 172.21.15.125:0/3591652793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058009b30 con 0x7f106013d100 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.819+0000 7f1065600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1058036040 tx=0x7f1058004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:32.802 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.820+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105803e450 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.820+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105803e5d0 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.820+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105803e980 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.820+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1060152180 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.820+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10600b1030 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.821+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10600675d0 con 0x7f106013d100 2024-01-16T14:56:32.803 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.822+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f105802cd50 con 0x7f106013d100 2024-01-16T14:56:32.804 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.822+0000 7f105cc00640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 0x7f103806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:32.804 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.822+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f10580ab9c0 con 0x7f106013d100 2024-01-16T14:56:32.804 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.823+0000 7f1064c00640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 0x7f103806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:32.805 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.823+0000 7f1064c00640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 0x7f103806d6f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1050003cc0 tx=0x7f10500079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:32.809 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:32.827+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1058077600 con 0x7f106013d100 2024-01-16T14:56:32.989 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.007+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10600b2050 con 0x7f106013d100 2024-01-16T14:56:32.990 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.008+0000 7f105cc00640 1 -- 172.21.15.125:0/3591652793 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1058043020 con 0x7f106013d100 2024-01-16T14:56:32.990 INFO:teuthology.misc.health.smithi125.stdout: 2024-01-16T14:56:32.990 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":12,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:29.962319+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.010+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 msgr2=0x7f103806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.010+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 0x7f103806d6f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1050003cc0 tx=0x7f10500079b0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 msgr2=0x7f10600b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1058036040 tx=0x7f1058004a40 comp rx=0 tx=0).stop 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 shutdown_connections 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f103806b230 0x7f103806d6f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.993 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106013d100 0x7f10600b2980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.994 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106013cd20 0x7f1060159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.994 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 --2- 172.21.15.125:0/3591652793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106006c8f0 0x7f10601590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:32.994 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 >> 172.21.15.125:0/3591652793 conn(0x7f106006dd50 msgr2=0x7f106015a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:32.994 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 shutdown_connections 2024-01-16T14:56:32.994 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:33.011+0000 7f1067600640 1 -- 172.21.15.125:0/3591652793 wait complete. 2024-01-16T14:56:33.005 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T14:56:36.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:36.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:39.812 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:39.995 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.013+0000 7f44cd600640 1 Processor -- start 2024-01-16T14:56:39.995 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.013+0000 7f44cd600640 1 -- start start 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44cd600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44c806c1e0 0x7f44c806c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44cd600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c8071040 con 0x7f44c8070380 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44cd600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c8071320 con 0x7f44c8072160 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44cd600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c806cbb0 con 0x7f44c806c1e0 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44c7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f44c8070380 0x7f44c80707d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37602/0 (socket says 172.21.15.125:37602) 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44bfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44c806c1e0 0x7f44c806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:39.997 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.015+0000 7f44c7400640 1 -- 172.21.15.125:0/3110086425 learned_addr learned my addr 172.21.15.125:0/3110086425 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:39.998 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.016+0000 7f44bfe00640 1 -- 172.21.15.125:0/3110086425 >> v1:172.21.15.137:6789/0 conn(0x7f44c8072160 legacy=0x7f44c80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:39.998 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.016+0000 7f44bfe00640 1 -- 172.21.15.125:0/3110086425 >> v1:172.21.15.125:6789/0 conn(0x7f44c8070380 legacy=0x7f44c80707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:39.998 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.016+0000 7f44bfe00640 1 -- 172.21.15.125:0/3110086425 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c80b33e0 con 0x7f44c806c1e0 2024-01-16T14:56:39.998 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.016+0000 7f44c6000640 1 -- 172.21.15.125:0/3110086425 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742049621 0 0) 0x7f44c8071040 con 0x7f44c8070380 2024-01-16T14:56:39.999 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.017+0000 7f44bfe00640 1 --2- 172.21.15.125:0/3110086425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c806c670 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44b00097a0 tx=0x7f44b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea2c33304f2ea30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:39.999 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.017+0000 7f44c6000640 1 -- 172.21.15.125:0/3110086425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0030610 con 0x7f44c806c1e0 2024-01-16T14:56:39.999 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.017+0000 7f44c6000640 1 -- 172.21.15.125:0/3110086425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b0030770 con 0x7f44c806c1e0 2024-01-16T14:56:39.999 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.017+0000 7f44c6000640 1 -- 172.21.15.125:0/3110086425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0030b00 con 0x7f44c806c1e0 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 -- 172.21.15.125:0/3110086425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 msgr2=0x7f44c806c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 --2- 172.21.15.125:0/3110086425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c806c670 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44b00097a0 tx=0x7f44b002b960 comp rx=0 tx=0).stop 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 -- 172.21.15.125:0/3110086425 shutdown_connections 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 --2- 172.21.15.125:0/3110086425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c806c670 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 -- 172.21.15.125:0/3110086425 >> 172.21.15.125:0/3110086425 conn(0x7f44c806dd50 msgr2=0x7f44c806a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 -- 172.21.15.125:0/3110086425 shutdown_connections 2024-01-16T14:56:40.000 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.018+0000 7f44cd600640 1 -- 172.21.15.125:0/3110086425 wait complete. 2024-01-16T14:56:40.001 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.020+0000 7f44cd600640 1 Processor -- start 2024-01-16T14:56:40.002 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.020+0000 7f44cd600640 1 -- start start 2024-01-16T14:56:40.004 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.004 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 0x7f44c81f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44c8072160 0x7f44c80c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81f7090 con 0x7f44c806c1e0 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81f2a30 con 0x7f44c8070380 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44cd600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80c4230 con 0x7f44c8072160 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44c6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44c6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33168/0 (socket says 172.21.15.125:33168) 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.021+0000 7f44c6a00640 1 -- 172.21.15.125:0/3565833943 learned_addr learned my addr 172.21.15.125:0/3565833943 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 -- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44c8072160 msgr2=0x7f44c80c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44bfe00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 0x7f44c81f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.005 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c7400640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44c8072160 0x7f44c80c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44c8072160 0x7f44c80c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 -- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 msgr2=0x7f44c81f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 0x7f44c81f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 -- 172.21.15.125:0/3565833943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0009400 con 0x7f44c806c1e0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44bfe00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 0x7f44c81f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44c6a00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c400 tx=0x7f44b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80052d0 con 0x7f44c806c1e0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8005430 con 0x7f44c806c1e0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8003830 con 0x7f44c806c1e0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c80c4510 con 0x7f44c806c1e0 2024-01-16T14:56:40.006 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.022+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c80c4ad0 con 0x7f44c806c1e0 2024-01-16T14:56:40.007 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.024+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c80675d0 con 0x7f44c806c1e0 2024-01-16T14:56:40.007 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.025+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f44b8003990 con 0x7f44c806c1e0 2024-01-16T14:56:40.007 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.025+0000 7f44bea00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 0x7f44a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.007 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.026+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44b8088e00 con 0x7f44c806c1e0 2024-01-16T14:56:40.008 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.026+0000 7f44bfe00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 0x7f44a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.008 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.026+0000 7f44bfe00640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 0x7f44a006d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44b0036000 tx=0x7f44b003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.012 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.031+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44b8053fc0 con 0x7f44c806c1e0 2024-01-16T14:56:40.207 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.226+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44c806c670 con 0x7f44c806c1e0 2024-01-16T14:56:40.209 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.228+0000 7f44bea00640 1 -- 172.21.15.125:0/3565833943 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12898 (secure 0 0 0) 0x7f44b8017090 con 0x7f44c806c1e0 2024-01-16T14:56:40.210 INFO:teuthology.misc.health.smithi125.stdout: 2024-01-16T14:56:40.210 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":16,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:36.253694+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 msgr2=0x7f44a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 0x7f44a006d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44b0036000 tx=0x7f44b003b040 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 msgr2=0x7f44c81f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c400 tx=0x7f44b800c8c0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 shutdown_connections 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44c8072160 0x7f44c80c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44c8070380 0x7f44c81f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44a006b2c0 0x7f44a006d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 --2- 172.21.15.125:0/3565833943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44c806c1e0 0x7f44c81f6b50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.231+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 >> 172.21.15.125:0/3565833943 conn(0x7f44c806dd50 msgr2=0x7f44c81462d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.232+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 shutdown_connections 2024-01-16T14:56:40.214 INFO:teuthology.misc.health.smithi125.stderr:2024-01-16T14:56:40.232+0000 7f44cd600640 1 -- 172.21.15.125:0/3565833943 wait complete. 2024-01-16T14:56:40.227 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-16T14:56:40.227 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T14:56:40.228 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:56:40.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:56:40.234 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:56:40.234 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:56:40.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:56:40.241 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:56:40.241 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:56:40.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:56:40.249 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:56:40.249 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:56:40.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:56:40.256 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:56:40.256 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-16T14:56:40.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:40.384 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T14:56:40.389 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T14:56:40.406 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T14:56:40.406 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T14:56:40.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.456+0000 7f1fbfe00640 1 Processor -- start 2024-01-16T14:56:40.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.456+0000 7f1fbfe00640 1 -- start start 2024-01-16T14:56:40.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.457+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 0x7f1fb8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.457+0000 7f1fbfe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80cfa70 con 0x7f1fb815e880 2024-01-16T14:56:40.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.457+0000 7f1fbfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb809a3e0 con 0x7f1fb806e7c0 2024-01-16T14:56:40.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.457+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80cff00 con 0x7f1fb8099a60 2024-01-16T14:56:40.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbd400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1fb815e880 0x7f1fb806ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37628/0 (socket says 172.21.15.125:37628) 2024-01-16T14:56:40.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbd400640 1 -- 172.21.15.125:0/1791489065 learned_addr learned my addr 172.21.15.125:0/1791489065 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:40.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbca00640 1 --2- 172.21.15.125:0/1791489065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 0x7f1fb8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306977923 0 0) 0x7f1fb80cfa70 con 0x7f1fb815e880 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f98003520 con 0x7f1fb815e880 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735326420 0 0) 0x7f1fb809a3e0 con 0x7f1fb806e7c0 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb80cfa70 con 0x7f1fb806e7c0 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbca00640 1 -- 172.21.15.125:0/1791489065 >> v1:172.21.15.137:6789/0 conn(0x7f1fb806e7c0 legacy=0x7f1fb806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbca00640 1 -- 172.21.15.125:0/1791489065 >> v1:172.21.15.125:6789/0 conn(0x7f1fb815e880 legacy=0x7f1fb806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.458+0000 7f1fbca00640 1 -- 172.21.15.125:0/1791489065 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb806fc70 con 0x7f1fb8099a60 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.459+0000 7f1fbca00640 1 --2- 172.21.15.125:0/1791489065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 0x7f1fb8099ea0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00097d0 tx=0x7f1fa002b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5471d7302a072117 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.459+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0030890 con 0x7f1fb8099a60 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa00309f0 con 0x7f1fb8099a60 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fb7e00640 1 -- 172.21.15.125:0/1791489065 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0030d80 con 0x7f1fb8099a60 2024-01-16T14:56:40.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1791489065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 msgr2=0x7f1fb8099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1791489065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 0x7f1fb8099ea0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00097d0 tx=0x7f1fa002b9d0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1791489065 shutdown_connections 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1791489065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb8099a60 0x7f1fb8099ea0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1791489065 >> 172.21.15.125:0/1791489065 conn(0x7f1fb80b2d00 msgr2=0x7f1fb80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1791489065 shutdown_connections 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.460+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1791489065 wait complete. 2024-01-16T14:56:40.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.462+0000 7f1fbfe00640 1 Processor -- start 2024-01-16T14:56:40.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.462+0000 7f1fbfe00640 1 -- start start 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 0x7f1fb8157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 0x7f1fb80b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80b4fb0 con 0x7f1fb8099a60 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80b51a0 con 0x7f1fb815e880 2024-01-16T14:56:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb806b590 con 0x7f1fb806e7c0 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33190/0 (socket says 172.21.15.125:33190) 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.463+0000 7f1fbca00640 1 -- 172.21.15.125:0/1887566347 learned_addr learned my addr 172.21.15.125:0/1887566347 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbd400640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 0x7f1fb8157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 -- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 msgr2=0x7f1fb8157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 0x7f1fb8157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbde00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 0x7f1fb80b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 -- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 msgr2=0x7f1fb80b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 0x7f1fb80b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 -- 172.21.15.125:0/1887566347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa0009400 con 0x7f1fb8099a60 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbd400640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 0x7f1fb8157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbde00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 0x7f1fb80b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbca00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098f0 tx=0x7f1fa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa003e040 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0039c50 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00443f0 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb806b870 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.464+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb80b5480 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.465+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb806bd90 con 0x7f1fb8099a60 2024-01-16T14:56:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.466+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1fa0044550 con 0x7f1fb8099a60 2024-01-16T14:56:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.468+0000 7f1fb5600640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 0x7f1f9006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.468+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1fa00abd40 con 0x7f1fb8099a60 2024-01-16T14:56:40.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.468+0000 7f1fbd400640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 0x7f1f9006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.468+0000 7f1fbd400640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 0x7f1f9006d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8007fd0 tx=0x7f1fa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.472+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fb806bd90 con 0x7f1fb8099a60 2024-01-16T14:56:40.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.664+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fb806bd90 con 0x7f1fb8099a60 2024-01-16T14:56:40.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.666+0000 7f1fb5600640 1 -- 172.21.15.125:0/1887566347 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12898 (secure 0 0 0) 0x7f1fb806bd90 con 0x7f1fb8099a60 2024-01-16T14:56:40.648 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:40.648 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":16,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:36.253694+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 msgr2=0x7f1f9006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 0x7f1f9006d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8007fd0 tx=0x7f1fa8009040 comp rx=0 tx=0).stop 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 msgr2=0x7f1fb80ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098f0 tx=0x7f1fa0004730 comp rx=0 tx=0).stop 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 shutdown_connections 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fb815e880 0x7f1fb80b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f9006b2c0 0x7f1f9006d780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fb8099a60 0x7f1fb80ce1c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 --2- 172.21.15.125:0/1887566347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fb806e7c0 0x7f1fb8157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 >> 172.21.15.125:0/1887566347 conn(0x7f1fb80b2d00 msgr2=0x7f1fb80c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:40.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 shutdown_connections 2024-01-16T14:56:40.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.669+0000 7f1fbfe00640 1 -- 172.21.15.125:0/1887566347 wait complete. 2024-01-16T14:56:40.668 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:56:40.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:56:40.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.891+0000 7f5fb6200640 1 Processor -- start 2024-01-16T14:56:40.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.891+0000 7f5fb6200640 1 -- start start 2024-01-16T14:56:40.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fb6200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5fb015d430 0x7f5fb00afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fb6200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b0730 con 0x7f5fb00d3870 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fb6200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b0a10 con 0x7f5fb00d0e80 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fb6200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb00b0cf0 con 0x7f5fb015d430 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fafe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5fb00d3870 0x7f5fb00b17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37644/0 (socket says 172.21.15.125:37644) 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5fafe00640 1 -- 172.21.15.125:0/186686447 learned_addr learned my addr 172.21.15.125:0/186686447 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.893+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089984108 0 0) 0x7f5fb00b0730 con 0x7f5fb00d3870 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faf400640 1 --2- 172.21.15.125:0/186686447 >> v2:172.21.15.137:3300/0 conn(0x7f5fb015d430 0x7f5fb00afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f90003520 con 0x7f5fb00d3870 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4277299085 0 0) 0x7f5f90003520 con 0x7f5fb00d3870 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 >> v2:172.21.15.137:3300/0 conn(0x7f5fb015d430 msgr2=0x7f5fb00afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 --2- 172.21.15.125:0/186686447 >> v2:172.21.15.137:3300/0 conn(0x7f5fb015d430 0x7f5fb00afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 >> v1:172.21.15.137:6789/0 conn(0x7f5fb00d0e80 legacy=0x7f5fb00d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00b0fd0 con 0x7f5fb00d3870 2024-01-16T14:56:40.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5fa80019f0 con 0x7f5fb00d3870 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 >> v1:172.21.15.125:6789/0 conn(0x7f5fb00d3870 legacy=0x7f5fb00b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f90003970 0x7f5f90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 0x7f5f90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f90005180 0x7f5f9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b0cf0 con 0x7f5f900047c0 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9000c0d0 con 0x7f5f90005180 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9000c290 con 0x7f5f90003970 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5faf400640 1 --2- 172.21.15.125:0/186686447 >> v2:172.21.15.137:3300/0 conn(0x7f5fb015d430 0x7f5fb00afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.894+0000 7f5fa6800640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 0x7f5f90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fa6800640 1 -- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f90003970 msgr2=0x7f5f90003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fafe00640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f90005180 0x7f5f9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fa6800640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f90003970 0x7f5f90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fa6800640 1 -- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f90005180 msgr2=0x7f5f9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fa6800640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f90005180 0x7f5f9000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fa6800640 1 -- 172.21.15.125:0/186686447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90003520 con 0x7f5f900047c0 2024-01-16T14:56:40.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.895+0000 7f5fafe00640 1 -- 172.21.15.125:0/186686447 reap_dead start 2024-01-16T14:56:40.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.896+0000 7f5fa6800640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 0x7f5f90004c40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f98001750 tx=0x7f5f900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acdff9f095b944a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.896+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980052d0 con 0x7f5f900047c0 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.896+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9802e050 con 0x7f5f900047c0 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.896+0000 7f5faea00640 1 -- 172.21.15.125:0/186686447 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98003830 con 0x7f5f900047c0 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 -- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 msgr2=0x7f5f90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 0x7f5f90004c40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f98001750 tx=0x7f5f900057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 -- 172.21.15.125:0/186686447 shutdown_connections 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 --2- 172.21.15.125:0/186686447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f900047c0 0x7f5f90004c40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 -- 172.21.15.125:0/186686447 >> 172.21.15.125:0/186686447 conn(0x7f5fb008ccd0 msgr2=0x7f5fb009c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 -- 172.21.15.125:0/186686447 shutdown_connections 2024-01-16T14:56:40.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.897+0000 7f5fb6200640 1 -- 172.21.15.125:0/186686447 wait complete. 2024-01-16T14:56:40.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.899+0000 7f5fb6200640 1 Processor -- start 2024-01-16T14:56:40.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.899+0000 7f5fb6200640 1 -- start start 2024-01-16T14:56:40.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f900047c0 0x7f5fb00cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fb015fcf0 0x7f5fb0160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01606c0 con 0x7f5fb015aa50 2024-01-16T14:56:40.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00ca380 con 0x7f5fb015fcf0 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fb6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb015aed0 con 0x7f5f900047c0 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33222/0 (socket says 172.21.15.125:33222) 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 -- 172.21.15.125:0/384991580 learned_addr learned my addr 172.21.15.125:0/384991580 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 -- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f900047c0 msgr2=0x7f5fb00cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f900047c0 0x7f5fb00cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 -- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fb015fcf0 msgr2=0x7f5fb0160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fb015fcf0 0x7f5fb0160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:40.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.900+0000 7f5fa6800640 1 -- 172.21.15.125:0/384991580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb015d990 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fa6800640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f9802bc00 tx=0x7f5f98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980076c0 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98007a30 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98006be0 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00ca5d0 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.901+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb00cab90 con 0x7f5fb015aa50 2024-01-16T14:56:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.902+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0067640 con 0x7f5fb015aa50 2024-01-16T14:56:40.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.903+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5f98006d40 con 0x7f5fb015aa50 2024-01-16T14:56:40.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.904+0000 7f5fa7e00640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 0x7f5f8806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:40.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.904+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f980ab490 con 0x7f5fb015aa50 2024-01-16T14:56:40.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.904+0000 7f5faf400640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 0x7f5f8806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:40.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.905+0000 7f5faf400640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 0x7f5f8806d7b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0006fd0 tx=0x7f5fa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:40.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:40.909+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f98076650 con 0x7f5fb015aa50 2024-01-16T14:56:41.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.099+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fb00d2310 con 0x7f5fb015aa50 2024-01-16T14:56:41.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.100+0000 7f5fa7e00640 1 -- 172.21.15.125:0/384991580 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f98079c70 con 0x7f5fb015aa50 2024-01-16T14:56:41.082 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:56:41.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 msgr2=0x7f5f8806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 0x7f5f8806d7b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0006fd0 tx=0x7f5fa0008040 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 msgr2=0x7f5fb00b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f9802bc00 tx=0x7f5f98001120 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 shutdown_connections 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fb015fcf0 0x7f5fb0160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8806b2f0 0x7f5f8806d7b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.102+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fb015aa50 0x7f5fb00b7590 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.103+0000 7f5fb6200640 1 --2- 172.21.15.125:0/384991580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f900047c0 0x7f5fb00cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.103+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 >> 172.21.15.125:0/384991580 conn(0x7f5fb008ccd0 msgr2=0x7f5fb00b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.103+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 shutdown_connections 2024-01-16T14:56:41.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.103+0000 7f5fb6200640 1 -- 172.21.15.125:0/384991580 wait complete. 2024-01-16T14:56:41.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:56:41.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.327+0000 7f251b200640 1 Processor -- start 2024-01-16T14:56:41.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.327+0000 7f251b200640 1 -- start start 2024-01-16T14:56:41.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25140ceea0 0x7f25140b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140b2c70 con 0x7f251415d3c0 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25140cf6d0 con 0x7f25140ceea0 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140cf3f0 con 0x7f25140d0150 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f2519e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f251415d3c0 0x7f25140cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37656/0 (socket says 172.21.15.125:37656) 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f2519e00640 1 -- 172.21.15.125:0/2654324732 learned_addr learned my addr 172.21.15.125:0/2654324732 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:41.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251a800640 1 --2- 172.21.15.125:0/2654324732 >> v2:172.21.15.125:3300/0 conn(0x7f25140ceea0 0x7f25140b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f2519400640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251a800640 1 -- 172.21.15.125:0/2654324732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 msgr2=0x7f25140d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f2518a00640 1 -- 172.21.15.125:0/2654324732 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541004686 0 0) 0x7f25140b2c70 con 0x7f251415d3c0 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251a800640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251a800640 1 -- 172.21.15.125:0/2654324732 >> v1:172.21.15.125:6789/0 conn(0x7f251415d3c0 legacy=0x7f25140cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.329+0000 7f251a800640 1 -- 172.21.15.125:0/2654324732 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25140b2620 con 0x7f25140ceea0 2024-01-16T14:56:41.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.330+0000 7f2519400640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:41.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251a800640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f25140b3100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9381fd0da8137140 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f2518a00640 1 -- 172.21.15.125:0/2654324732 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c02cc60 con 0x7f25140ceea0 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f2518a00640 1 -- 172.21.15.125:0/2654324732 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c02cdc0 con 0x7f25140ceea0 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f2518a00640 1 -- 172.21.15.125:0/2654324732 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c034660 con 0x7f25140ceea0 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 -- 172.21.15.125:0/2654324732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 msgr2=0x7f25140b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f25140b3100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c02beb0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 -- 172.21.15.125:0/2654324732 shutdown_connections 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f25140b3100 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 --2- 172.21.15.125:0/2654324732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.331+0000 7f251b200640 1 -- 172.21.15.125:0/2654324732 >> 172.21.15.125:0/2654324732 conn(0x7f25140d3050 msgr2=0x7f25140d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.332+0000 7f251b200640 1 -- 172.21.15.125:0/2654324732 shutdown_connections 2024-01-16T14:56:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.332+0000 7f251b200640 1 -- 172.21.15.125:0/2654324732 wait complete. 2024-01-16T14:56:41.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.333+0000 7f251b200640 1 Processor -- start 2024-01-16T14:56:41.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.333+0000 7f251b200640 1 -- start start 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 0x7f25140cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25140c7550 con 0x7f25140ceea0 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2514158f40 con 0x7f251415d3c0 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f251b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140b6cb0 con 0x7f25140d0150 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.334+0000 7f2519e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33226/0 (socket says 172.21.15.125:33226) 2024-01-16T14:56:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 -- 172.21.15.125:0/2241689404 learned_addr learned my addr 172.21.15.125:0/2241689404 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519400640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f251a800640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 0x7f25140cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 -- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 msgr2=0x7f25140cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 -- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 msgr2=0x7f25140cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 0x7f25140cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 -- 172.21.15.125:0/2241689404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009a70 con 0x7f25140ceea0 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f251a800640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 0x7f25140cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.335+0000 7f2519e00640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f250400c430 tx=0x7f250400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.336+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040074e0 con 0x7f25140ceea0 2024-01-16T14:56:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.336+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504007660 con 0x7f25140ceea0 2024-01-16T14:56:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.336+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504004a20 con 0x7f25140ceea0 2024-01-16T14:56:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.336+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25140c6c20 con 0x7f25140ceea0 2024-01-16T14:56:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.336+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25140c71e0 con 0x7f25140ceea0 2024-01-16T14:56:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.337+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251408ba80 con 0x7f25140ceea0 2024-01-16T14:56:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.338+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2504004b80 con 0x7f25140ceea0 2024-01-16T14:56:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.338+0000 7f2512000640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 0x7f24ec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.339+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2504089690 con 0x7f25140ceea0 2024-01-16T14:56:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.339+0000 7f2519400640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 0x7f24ec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.339+0000 7f2519400640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 0x7f24ec06d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25140b86d0 tx=0x7f24fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.343+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2504054850 con 0x7f25140ceea0 2024-01-16T14:56:41.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.572+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2514067450 con 0x7f25140ceea0 2024-01-16T14:56:41.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.574+0000 7f2512000640 1 -- 172.21.15.125:0/2241689404 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3706 (secure 0 0 0) 0x7f2504057e70 con 0x7f25140ceea0 2024-01-16T14:56:41.556 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:41.556 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-16T14:56:41.556 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 3 2024-01-16T14:56:41.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 msgr2=0x7f24ec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 0x7f24ec06d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25140b86d0 tx=0x7f24fc008040 comp rx=0 tx=0).stop 2024-01-16T14:56:41.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 msgr2=0x7f2514158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f250400c430 tx=0x7f250400c8f0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 shutdown_connections 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f251415d3c0 0x7f25140cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140d0150 0x7f25140cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f24ec06b2c0 0x7f24ec06d780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 --2- 172.21.15.125:0/2241689404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25140ceea0 0x7f2514158a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.576+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 >> 172.21.15.125:0/2241689404 conn(0x7f25140d3050 msgr2=0x7f25140b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.577+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 shutdown_connections 2024-01-16T14:56:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.577+0000 7f251b200640 1 -- 172.21.15.125:0/2241689404 wait complete. 2024-01-16T14:56:41.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.800+0000 7f26aba00640 1 Processor -- start 2024-01-16T14:56:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.800+0000 7f26aba00640 1 -- start start 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26aba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f26a409a170 0x7f26a40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26aba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f26a40cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26aba00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a406cea0 con 0x7f26a40cd240 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26aba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a406cbc0 con 0x7f26a409a170 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26aba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406f350 con 0x7f26a406c6b0 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26a9000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f26a40cd240 0x7f26a4099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37672/0 (socket says 172.21.15.125:37672) 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26a9000640 1 -- 172.21.15.125:0/2715567578 learned_addr learned my addr 172.21.15.125:0/2715567578 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.802+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785909800 0 0) 0x7f26a406cea0 con 0x7f26a40cd240 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2684003520 con 0x7f26a40cd240 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a9a00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f26a40cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3e00640 1 --2- 172.21.15.125:0/2715567578 >> v2:172.21.15.137:3300/0 conn(0x7f26a409a170 0x7f26a40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428262013 0 0) 0x7f2684003520 con 0x7f26a40cd240 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 msgr2=0x7f26a40cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f26a40cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 >> v2:172.21.15.137:3300/0 conn(0x7f26a409a170 msgr2=0x7f26a40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 --2- 172.21.15.125:0/2715567578 >> v2:172.21.15.137:3300/0 conn(0x7f26a409a170 0x7f26a40b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a406f630 con 0x7f26a40cd240 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f268c001670 con 0x7f26a40cd240 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 >> v1:172.21.15.125:6789/0 conn(0x7f26a40cd240 legacy=0x7f26a4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f2684003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2684003ff0 0x7f26840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2684004d00 0x7f2684005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406cbc0 con 0x7f2684004d00 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406f350 con 0x7f2684003ff0 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268400bae0 con 0x7f26a406c6b0 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.803+0000 7f26a9a00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2684004d00 0x7f2684005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9000640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f2684003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a3e00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2684003ff0 0x7f26840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9a00640 1 -- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 msgr2=0x7f2684003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9a00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26a406c6b0 0x7f2684003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9a00640 1 -- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2684003ff0 msgr2=0x7f26840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9a00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2684003ff0 0x7f26840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a9a00640 1 -- 172.21.15.125:0/2715567578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684003520 con 0x7f2684004d00 2024-01-16T14:56:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.804+0000 7f26a3e00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2684003ff0 0x7f26840047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26a9a00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2684004d00 0x7f2684005180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f269c007330 tx=0x7f26840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedef93ca601c825 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c02cca0 con 0x7f2684004d00 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c02ce00 con 0x7f2684004d00 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26a3400640 1 -- 172.21.15.125:0/2715567578 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c034680 con 0x7f2684004d00 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26aba00640 1 -- 172.21.15.125:0/2715567578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2684004d00 msgr2=0x7f2684005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26aba00640 1 --2- 172.21.15.125:0/2715567578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2684004d00 0x7f2684005180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f269c007330 tx=0x7f26840057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.805+0000 7f26a9a00640 1 -- 172.21.15.125:0/2715567578 reap_dead start 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.806+0000 7f26aba00640 1 -- 172.21.15.125:0/2715567578 shutdown_connections 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.806+0000 7f26aba00640 1 -- 172.21.15.125:0/2715567578 >> 172.21.15.125:0/2715567578 conn(0x7f26a40b2d00 msgr2=0x7f26a40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:41.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.806+0000 7f26aba00640 1 -- 172.21.15.125:0/2715567578 shutdown_connections 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.806+0000 7f26aba00640 1 -- 172.21.15.125:0/2715567578 wait complete. 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.807+0000 7f26aba00640 1 Processor -- start 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.807+0000 7f26aba00640 1 -- start start 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2684004d00 0x7f26a40c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26a41f6330 0x7f26a415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406beb0 con 0x7f26a406b4f0 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a41f2c50 con 0x7f26a41f6330 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26aba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40c7a20 con 0x7f2684004d00 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33242/0 (socket says 172.21.15.125:33242) 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 -- 172.21.15.125:0/3066260414 learned_addr learned my addr 172.21.15.125:0/3066260414 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 -- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2684004d00 msgr2=0x7f26a40c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a3e00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26a41f6330 0x7f26a415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9000640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2684004d00 0x7f26a40c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2684004d00 0x7f26a40c6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 -- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26a41f6330 msgr2=0x7f26a415ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26a41f6330 0x7f26a415ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.809+0000 7f26a9a00640 1 -- 172.21.15.125:0/3066260414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40c6ea0 con 0x7f26a406b4f0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26a9a00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f269c007450 tx=0x7f269c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c03e640 con 0x7f26a406b4f0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c03e7a0 con 0x7f26a406b4f0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c03eb30 con 0x7f26a406b4f0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a40c7180 con 0x7f26a406b4f0 2024-01-16T14:56:41.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.810+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a40c7740 con 0x7f26a406b4f0 2024-01-16T14:56:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.811+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a40b40f0 con 0x7f26a406b4f0 2024-01-16T14:56:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.812+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f269c03ec90 con 0x7f26a406b4f0 2024-01-16T14:56:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.813+0000 7f26a0c00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 0x7f267c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.813+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f269c0abcf0 con 0x7f26a406b4f0 2024-01-16T14:56:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.813+0000 7f26a9000640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 0x7f267c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:41.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.814+0000 7f26a9000640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 0x7f267c06d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f268c0058b0 tx=0x7f268c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:41.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:41.817+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26a40b40f0 con 0x7f26a406b4f0 2024-01-16T14:56:41.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.010+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26a40b40f0 con 0x7f26a406b4f0 2024-01-16T14:56:41.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.011+0000 7f26a0c00640 1 -- 172.21.15.125:0/3066260414 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12898 (secure 0 0 0) 0x7f26a40b40f0 con 0x7f26a406b4f0 2024-01-16T14:56:41.993 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:41.994 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":16,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:36.253694+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:41.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 msgr2=0x7f267c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 0x7f267c06d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f268c0058b0 tx=0x7f268c000ed0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 msgr2=0x7f26a406b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:41.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f269c007450 tx=0x7f269c004ab0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 shutdown_connections 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f267c06b2c0 0x7f267c06d780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26a406b4f0 0x7f26a406b970 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26a41f6330 0x7f26a415ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 --2- 172.21.15.125:0/3066260414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2684004d00 0x7f26a40c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 >> 172.21.15.125:0/3066260414 conn(0x7f26a40b2d00 msgr2=0x7f26a40cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 shutdown_connections 2024-01-16T14:56:41.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.014+0000 7f26aba00640 1 -- 172.21.15.125:0/3066260414 wait complete. 2024-01-16T14:56:42.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:56:42.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:56:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.247+0000 7fe9d8000640 1 Processor -- start 2024-01-16T14:56:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.247+0000 7fe9d8000640 1 -- start start 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 0x7fe9d00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d8000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d015c5f0 con 0x7fe9d015f9c0 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d8000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d006c470 con 0x7fe9d013df80 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006c750 con 0x7fe9d013e360 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d6000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe9d015f9c0 0x7fe9d00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37674/0 (socket says 172.21.15.125:37674) 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d6000640 1 -- 172.21.15.125:0/1568000117 learned_addr learned my addr 172.21.15.125:0/1568000117 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817318463 0 0) 0x7fe9d015c5f0 con 0x7fe9d015f9c0 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9d4c00640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 0x7fe9d00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.249+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003520 con 0x7fe9d015f9c0 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439759328 0 0) 0x7fe9b0003520 con 0x7fe9d015f9c0 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 msgr2=0x7fe9d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 0x7fe9d00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 >> v1:172.21.15.137:6789/0 conn(0x7fe9d013df80 legacy=0x7fe9d015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d006ca30 con 0x7fe9d015f9c0 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe9c80019f0 con 0x7fe9d015f9c0 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 >> v1:172.21.15.125:6789/0 conn(0x7fe9d015f9c0 legacy=0x7fe9d00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9b00039a0 0x7fe9b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 0x7fe9b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9b0004d00 0x7fe9b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006c750 con 0x7fe9b00039a0 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b000ba90 con 0x7fe9b0004d00 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b000bc50 con 0x7fe9d013e360 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9d5600640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9b00039a0 0x7fe9b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.250+0000 7fe9d5600640 1 -- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 msgr2=0x7fe9b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.251+0000 7fe9d5600640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9d013e360 0x7fe9b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.251+0000 7fe9d6000640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9b0004d00 0x7fe9b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.251+0000 7fe9d5600640 1 -- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9b0004d00 msgr2=0x7fe9b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.251+0000 7fe9d5600640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9b0004d00 0x7fe9b0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.251+0000 7fe9d5600640 1 -- 172.21.15.125:0/1568000117 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0003520 con 0x7fe9b00039a0 2024-01-16T14:56:42.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9d5600640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9b00039a0 0x7fe9b0003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0001720 tx=0x7fe9b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f7840bb61eba1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c00052d0 con 0x7fe9b00039a0 2024-01-16T14:56:42.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c002e050 con 0x7fe9b00039a0 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9cfe00640 1 -- 172.21.15.125:0/1568000117 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c00035f0 con 0x7fe9b00039a0 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9d8000640 1 -- 172.21.15.125:0/1568000117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9b00039a0 msgr2=0x7fe9b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9d8000640 1 --2- 172.21.15.125:0/1568000117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9b00039a0 0x7fe9b0003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0001720 tx=0x7fe9b00057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9d6000640 1 -- 172.21.15.125:0/1568000117 reap_dead start 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.252+0000 7fe9d8000640 1 -- 172.21.15.125:0/1568000117 shutdown_connections 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.253+0000 7fe9d8000640 1 -- 172.21.15.125:0/1568000117 >> 172.21.15.125:0/1568000117 conn(0x7fe9d006a800 msgr2=0x7fe9d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.253+0000 7fe9d8000640 1 -- 172.21.15.125:0/1568000117 shutdown_connections 2024-01-16T14:56:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.253+0000 7fe9d8000640 1 -- 172.21.15.125:0/1568000117 wait complete. 2024-01-16T14:56:42.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.254+0000 7fe9d8000640 1 Processor -- start 2024-01-16T14:56:42.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.254+0000 7fe9d8000640 1 -- start start 2024-01-16T14:56:42.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9b00039a0 0x7fe9d00cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9d0070cd0 0x7fe9d0150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d0151830 con 0x7fe9d0071130 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00716b0 con 0x7fe9d0070cd0 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.255+0000 7fe9d8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00b35b0 con 0x7fe9b00039a0 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33254/0 (socket says 172.21.15.125:33254) 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 -- 172.21.15.125:0/2723816164 learned_addr learned my addr 172.21.15.125:0/2723816164 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 -- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9b00039a0 msgr2=0x7fe9d00cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9b00039a0 0x7fe9d00cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 -- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9d0070cd0 msgr2=0x7fe9d0150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d4c00640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9d0070cd0 0x7fe9d0150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9d0070cd0 0x7fe9d0150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 -- 172.21.15.125:0/2723816164 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d00b3850 con 0x7fe9d0071130 2024-01-16T14:56:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.256+0000 7fe9d6000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8001370 tx=0x7fe9c800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.257+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800de90 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.257+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c800bb80 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.257+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8007110 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.257+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d00b3b30 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.257+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d0072120 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.258+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d00675d0 con 0x7fe9d0071130 2024-01-16T14:56:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.259+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe9c8007270 con 0x7fe9d0071130 2024-01-16T14:56:42.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.260+0000 7fe9cd600640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 0x7fe9a806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.260+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9c8089d30 con 0x7fe9d0071130 2024-01-16T14:56:42.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.260+0000 7fe9d5600640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 0x7fe9a806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.261+0000 7fe9d5600640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 0x7fe9a806d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0033000 tx=0x7fe9c002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.264+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9c8054ee0 con 0x7fe9d0071130 2024-01-16T14:56:42.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.457+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9d00cec50 con 0x7fe9d0071130 2024-01-16T14:56:42.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.458+0000 7fe9cd600640 1 -- 172.21.15.125:0/2723816164 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9c8058500 con 0x7fe9d0071130 2024-01-16T14:56:42.440 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.460+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 msgr2=0x7fe9a806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 0x7fe9a806d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0033000 tx=0x7fe9c002f000 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 msgr2=0x7fe9d01512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8001370 tx=0x7fe9c800b4a0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 shutdown_connections 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe9a806b2c0 0x7fe9a806d780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9d0071130 0x7fe9d01512f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9d0070cd0 0x7fe9d0150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 --2- 172.21.15.125:0/2723816164 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9b00039a0 0x7fe9d00cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 >> 172.21.15.125:0/2723816164 conn(0x7fe9d006a800 msgr2=0x7fe9d00aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 shutdown_connections 2024-01-16T14:56:42.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.461+0000 7fe9d8000640 1 -- 172.21.15.125:0/2723816164 wait complete. 2024-01-16T14:56:42.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:56:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.688+0000 7f4aa2800640 1 Processor -- start 2024-01-16T14:56:42.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.688+0000 7f4aa2800640 1 -- start start 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.689+0000 7f4aa2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.689+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.689+0000 7f4aa2800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c071690 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.689+0000 7f4aa2800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c14b6a0 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.689+0000 7f4aa2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c071970 con 0x7f4a9c070ea0 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a9b400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4a9c14b2c0 0x7f4a9c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37684/0 (socket says 172.21.15.125:37684) 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a9b400640 1 -- 172.21.15.125:0/1995423799 learned_addr learned my addr 172.21.15.125:0/1995423799 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a92800640 1 --2- 172.21.15.125:0/1995423799 >> v2:172.21.15.125:3300/0 conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a9be00640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a9aa00640 1 -- 172.21.15.125:0/1995423799 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472610228 0 0) 0x7f4a9c071690 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a9aa00640 1 -- 172.21.15.125:0/1995423799 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003520 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a92800640 1 -- 172.21.15.125:0/1995423799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c070ea0 msgr2=0x7f4a9c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a92800640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a92800640 1 -- 172.21.15.125:0/1995423799 >> v1:172.21.15.125:6789/0 conn(0x7f4a9c14b2c0 legacy=0x7f4a9c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.690+0000 7f4a92800640 1 -- 172.21.15.125:0/1995423799 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c071c50 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.691+0000 7f4a92800640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4a840097a0 tx=0x7f4a8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657d091dd1d06b5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4a9aa00640 1 -- 172.21.15.125:0/1995423799 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84031830 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4a9aa00640 1 -- 172.21.15.125:0/1995423799 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a84031990 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4a9aa00640 1 -- 172.21.15.125:0/1995423799 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84031d20 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 -- 172.21.15.125:0/1995423799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c0cf9a0 msgr2=0x7f4a9c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4a840097a0 tx=0x7f4a8402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 -- 172.21.15.125:0/1995423799 shutdown_connections 2024-01-16T14:56:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 --2- 172.21.15.125:0/1995423799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 -- 172.21.15.125:0/1995423799 >> 172.21.15.125:0/1995423799 conn(0x7f4a9c06a800 msgr2=0x7f4a9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 -- 172.21.15.125:0/1995423799 shutdown_connections 2024-01-16T14:56:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.692+0000 7f4aa2800640 1 -- 172.21.15.125:0/1995423799 wait complete. 2024-01-16T14:56:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.694+0000 7f4aa2800640 1 Processor -- start 2024-01-16T14:56:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.694+0000 7f4aa2800640 1 -- start start 2024-01-16T14:56:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c142830 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c1fb3e0 con 0x7f4a9c070ea0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4aa2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c163670 con 0x7f4a9c0cf9a0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.695+0000 7f4a9be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33276/0 (socket says 172.21.15.125:33276) 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 -- 172.21.15.125:0/102696067 learned_addr learned my addr 172.21.15.125:0/102696067 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9b400640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a92800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 -- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c0cf9a0 msgr2=0x7f4a9c1f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1f6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 -- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a9c070ea0 msgr2=0x7f4a9c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 -- 172.21.15.125:0/102696067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84009400 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a9be00640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a94007e40 tx=0x7f4a9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94014620 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9400ee90 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c142270 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.696+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940149e0 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.697+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c163910 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.698+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c0675d0 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.699+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4a94014b40 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.699+0000 7f4a93e00640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 0x7f4a7406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:42.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.700+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a9408a2a0 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.700+0000 7f4a9b400640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 0x7f4a7406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.700+0000 7f4a9b400640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 0x7f4a7406d780 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0011e0 tx=0x7f4a8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:42.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.704+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a94055460 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.896+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4a9c06db90 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.897+0000 7f4a93e00640 1 -- 172.21.15.125:0/102696067 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4a94058a80 con 0x7f4a9c14b2c0 2024-01-16T14:56:42.878 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:56:42.879 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:56:42.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 msgr2=0x7f4a7406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 0x7f4a7406d780 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0011e0 tx=0x7f4a8c009040 comp rx=0 tx=0).stop 2024-01-16T14:56:42.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 msgr2=0x7f4a9c1faea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a94007e40 tx=0x7f4a9400e4b0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 shutdown_connections 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a7406b2c0 0x7f4a7406d780 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a9c14b2c0 0x7f4a9c1faea0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 --2- 172.21.15.125:0/102696067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 >> 172.21.15.125:0/102696067 conn(0x7f4a9c06a800 msgr2=0x7f4a9c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.899+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 shutdown_connections 2024-01-16T14:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:42.900+0000 7f4aa2800640 1 -- 172.21.15.125:0/102696067 wait complete. 2024-01-16T14:56:42.899 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:56:42.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:56:43.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.127+0000 7f65d0e00640 1 Processor -- start 2024-01-16T14:56:43.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.128+0000 7f65d0e00640 1 -- start start 2024-01-16T14:56:43.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65d0e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65cc0cf9a0 0x7f65cc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65d0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc15e1e0 0x7f65cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65d0e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc0b1ee0 con 0x7f65cc143ae0 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65d0e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc0b2780 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65d0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc15e990 con 0x7f65cc15e1e0 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65c9400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65cc0cf9a0 0x7f65cc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65c9400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65cc0cf9a0 0x7f65cc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33284/0 (socket says 172.21.15.125:33284) 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65c9400640 1 -- 172.21.15.125:0/1358497597 learned_addr learned my addr 172.21.15.125:0/1358497597 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.129+0000 7f65c9e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f65cc143ae0 0x7f65cc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37696/0 (socket says 172.21.15.125:37696) 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65ca800640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc15e1e0 0x7f65cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65c9400640 1 -- 172.21.15.125:0/1358497597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc15e1e0 msgr2=0x7f65cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65c9400640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc15e1e0 0x7f65cc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65c9400640 1 -- 172.21.15.125:0/1358497597 >> v1:172.21.15.125:6789/0 conn(0x7f65cc143ae0 legacy=0x7f65cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65c9400640 1 -- 172.21.15.125:0/1358497597 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0b2130 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.130+0000 7f65c8a00640 1 -- 172.21.15.125:0/1358497597 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184815129 0 0) 0x7f65cc0b1ee0 con 0x7f65cc143ae0 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.131+0000 7f65c9400640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc0cf9a0 0x7f65cc15dca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2da2014f17b1bc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.131+0000 7f65c8a00640 1 -- 172.21.15.125:0/1358497597 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4030840 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.131+0000 7f65c8a00640 1 -- 172.21.15.125:0/1358497597 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b40309a0 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.131+0000 7f65c8a00640 1 -- 172.21.15.125:0/1358497597 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4030d30 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 -- 172.21.15.125:0/1358497597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc0cf9a0 msgr2=0x7f65cc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc0cf9a0 0x7f65cc15dca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b402b980 comp rx=0 tx=0).stop 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 -- 172.21.15.125:0/1358497597 shutdown_connections 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc15e1e0 0x7f65cc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 --2- 172.21.15.125:0/1358497597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc0cf9a0 0x7f65cc15dca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 -- 172.21.15.125:0/1358497597 >> 172.21.15.125:0/1358497597 conn(0x7f65cc06a800 msgr2=0x7f65cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:43.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 -- 172.21.15.125:0/1358497597 shutdown_connections 2024-01-16T14:56:43.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.132+0000 7f65d0e00640 1 -- 172.21.15.125:0/1358497597 wait complete. 2024-01-16T14:56:43.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.134+0000 7f65d0e00640 1 Processor -- start 2024-01-16T14:56:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.134+0000 7f65d0e00640 1 -- start start 2024-01-16T14:56:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.135+0000 7f65d0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc0cf9a0 0x7f65cc0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.135+0000 7f65d0e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65d0e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65cc15e1e0 0x7f65cc0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65d0e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0ce120 con 0x7f65cc143ae0 2024-01-16T14:56:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65d0e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0cde40 con 0x7f65cc15e1e0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65d0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc134580 con 0x7f65cc0cf9a0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33296/0 (socket says 172.21.15.125:33296) 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 -- 172.21.15.125:0/4088570276 learned_addr learned my addr 172.21.15.125:0/4088570276 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 -- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc0cf9a0 msgr2=0x7f65cc0cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc0cf9a0 0x7f65cc0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc0cf9a0 0x7f65cc0cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65ca800640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65cc15e1e0 0x7f65cc0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 -- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65cc15e1e0 msgr2=0x7f65cc0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65cc15e1e0 0x7f65cc0c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.136+0000 7f65c9400640 1 -- 172.21.15.125:0/4088570276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40094d0 con 0x7f65cc143ae0 2024-01-16T14:56:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65c9400640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4039c90 con 0x7f65cc143ae0 2024-01-16T14:56:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4039df0 con 0x7f65cc143ae0 2024-01-16T14:56:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4043640 con 0x7f65cc143ae0 2024-01-16T14:56:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc134860 con 0x7f65cc143ae0 2024-01-16T14:56:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.137+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0cd080 con 0x7f65cc143ae0 2024-01-16T14:56:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.138+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc134d80 con 0x7f65cc143ae0 2024-01-16T14:56:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.139+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f65b40437a0 con 0x7f65cc143ae0 2024-01-16T14:56:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.140+0000 7f65c2000640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 0x7f65a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.140+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65b403e070 con 0x7f65cc143ae0 2024-01-16T14:56:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.140+0000 7f65c9e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 0x7f65a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.141+0000 7f65c9e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 0x7f65a406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007fd0 tx=0x7f65bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:43.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.145+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65cc134d80 con 0x7f65cc143ae0 2024-01-16T14:56:43.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.337+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f65cc134d80 con 0x7f65cc143ae0 2024-01-16T14:56:43.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.958+0000 7f65c2000640 1 -- 172.21.15.125:0/4088570276 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65cc134d80 con 0x7f65cc143ae0 2024-01-16T14:56:43.940 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 msgr2=0x7f65a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 0x7f65a406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007fd0 tx=0x7f65bc009040 comp rx=0 tx=0).stop 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 msgr2=0x7f65cc0c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004730 comp rx=0 tx=0).stop 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 shutdown_connections 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65cc15e1e0 0x7f65cc0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f65a406b2c0 0x7f65a406d780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65cc143ae0 0x7f65cc0c6830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 --2- 172.21.15.125:0/4088570276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65cc0cf9a0 0x7f65cc0cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:43.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.963+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 >> 172.21.15.125:0/4088570276 conn(0x7f65cc06a800 msgr2=0x7f65cc070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:43.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.964+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 shutdown_connections 2024-01-16T14:56:43.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:43.964+0000 7f65d0e00640 1 -- 172.21.15.125:0/4088570276 wait complete. 2024-01-16T14:56:43.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:56:44.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.186+0000 7fe9b4a00640 1 Processor -- start 2024-01-16T14:56:44.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.186+0000 7fe9b4a00640 1 -- start start 2024-01-16T14:56:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe9ac160fe0 0x7fe9ac161470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9ac0b1f50 0x7fe9ac0b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b4a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac070bf0 con 0x7fe9ac0ce490 2024-01-16T14:56:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b4a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ac1619b0 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ac0702e0 con 0x7fe9ac160fe0 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b2000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe9ac0ce490 0x7fe9ac071080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37700/0 (socket says 172.21.15.125:37700) 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b2000640 1 -- 172.21.15.125:0/3770620734 learned_addr learned my addr 172.21.15.125:0/3770620734 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b2a00640 1 --2- 172.21.15.125:0/3770620734 >> v2:172.21.15.125:3300/0 conn(0x7fe9ac0b1f50 0x7fe9ac0b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.188+0000 7fe9b1600640 1 --2- 172.21.15.125:0/3770620734 >> v2:172.21.15.137:3300/0 conn(0x7fe9ac160fe0 0x7fe9ac161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.189+0000 7fe9b2a00640 1 -- 172.21.15.125:0/3770620734 >> v2:172.21.15.137:3300/0 conn(0x7fe9ac160fe0 msgr2=0x7fe9ac161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.189+0000 7fe9b2a00640 1 --2- 172.21.15.125:0/3770620734 >> v2:172.21.15.137:3300/0 conn(0x7fe9ac160fe0 0x7fe9ac161470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.189+0000 7fe9b2a00640 1 -- 172.21.15.125:0/3770620734 >> v1:172.21.15.125:6789/0 conn(0x7fe9ac0ce490 legacy=0x7fe9ac071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.189+0000 7fe9b2a00640 1 -- 172.21.15.125:0/3770620734 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac0b2cc0 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b2a00640 1 --2- 172.21.15.125:0/3770620734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0b1f50 0x7fe9ac0b23b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac06ee10 tx=0x7fe9a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76df6eedb25b9cfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b0c00640 1 -- 172.21.15.125:0/3770620734 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a402cc80 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b0c00640 1 -- 172.21.15.125:0/3770620734 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a402cde0 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b0c00640 1 -- 172.21.15.125:0/3770620734 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4034660 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b4a00640 1 -- 172.21.15.125:0/3770620734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0b1f50 msgr2=0x7fe9ac0b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.190+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/3770620734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0b1f50 0x7fe9ac0b23b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac06ee10 tx=0x7fe9a402bed0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 -- 172.21.15.125:0/3770620734 shutdown_connections 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/3770620734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0b1f50 0x7fe9ac0b23b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/3770620734 >> v2:172.21.15.137:3300/0 conn(0x7fe9ac160fe0 0x7fe9ac161470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 -- 172.21.15.125:0/3770620734 >> 172.21.15.125:0/3770620734 conn(0x7fe9ac0b0be0 msgr2=0x7fe9ac0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 -- 172.21.15.125:0/3770620734 shutdown_connections 2024-01-16T14:56:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.191+0000 7fe9b4a00640 1 -- 172.21.15.125:0/3770620734 wait complete. 2024-01-16T14:56:44.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.192+0000 7fe9b4a00640 1 Processor -- start 2024-01-16T14:56:44.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.193+0000 7fe9b4a00640 1 -- start start 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.193+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9ac0b1f50 0x7fe9ac1f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.193+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.193+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9ac160fe0 0x7fe9ac153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.193+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1538a0 con 0x7fe9ac0ce490 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac153b20 con 0x7fe9ac160fe0 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1554f0 con 0x7fe9ac0b1f50 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33308/0 (socket says 172.21.15.125:33308) 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 -- 172.21.15.125:0/2542457553 learned_addr learned my addr 172.21.15.125:0/2542457553 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b2000640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9ac0b1f50 0x7fe9ac1f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 -- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9ac0b1f50 msgr2=0x7fe9ac1f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9ac0b1f50 0x7fe9ac1f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 -- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9ac160fe0 msgr2=0x7fe9ac153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9ac160fe0 0x7fe9ac153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.194+0000 7fe9b1600640 1 -- 172.21.15.125:0/2542457553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4009a70 con 0x7fe9ac0ce490 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9b1600640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe99400d5f0 tx=0x7fe99400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940125f0 con 0x7fe9ac0ce490 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994012750 con 0x7fe9ac0ce490 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994012b00 con 0x7fe9ac0ce490 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac1557d0 con 0x7fe9ac0ce490 2024-01-16T14:56:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.195+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac153e00 con 0x7fe9ac0ce490 2024-01-16T14:56:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.196+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac155350 con 0x7fe9ac0ce490 2024-01-16T14:56:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.197+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe994012c60 con 0x7fe9ac0ce490 2024-01-16T14:56:44.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.198+0000 7fe9aa000640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 0x7fe98406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:44.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.199+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe994089ef0 con 0x7fe9ac0ce490 2024-01-16T14:56:44.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.199+0000 7fe9b2000640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 0x7fe98406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:44.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.199+0000 7fe9b2000640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 0x7fe98406d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005880 tx=0x7fe99c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.204+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9ac155350 con 0x7fe9ac0ce490 2024-01-16T14:56:44.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.396+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe9ac155350 con 0x7fe9ac0ce490 2024-01-16T14:56:44.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.964+0000 7fe9aa000640 1 -- 172.21.15.125:0/2542457553 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe9ac155350 con 0x7fe9ac0ce490 2024-01-16T14:56:44.947 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T14:56:44.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 msgr2=0x7fe98406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 0x7fe98406d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005880 tx=0x7fe99c009040 comp rx=0 tx=0).stop 2024-01-16T14:56:44.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 msgr2=0x7fe9ac155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:44.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe99400d5f0 tx=0x7fe99400dab0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 shutdown_connections 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9ac160fe0 0x7fe9ac153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe98406b2c0 0x7fe98406d780 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9ac0ce490 0x7fe9ac155fd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 --2- 172.21.15.125:0/2542457553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9ac0b1f50 0x7fe9ac1f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.968+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 >> 172.21.15.125:0/2542457553 conn(0x7fe9ac0b0be0 msgr2=0x7fe9ac0acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.969+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 shutdown_connections 2024-01-16T14:56:44.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:44.969+0000 7fe9b4a00640 1 -- 172.21.15.125:0/2542457553 wait complete. 2024-01-16T14:56:44.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:56:45.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.182+0000 7f172cc00640 1 Processor -- start 2024-01-16T14:56:45.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.183+0000 7f172cc00640 1 -- start start 2024-01-16T14:56:45.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f172cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f172815d4b0 0x7f17280d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f172cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17280cf9a0 0x7f17280cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f172cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280d0320 con 0x7f17280cddf0 2024-01-16T14:56:45.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f172cc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17280b2730 con 0x7f17280cf9a0 2024-01-16T14:56:45.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f172cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17280d1ec0 con 0x7f172815d4b0 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17280cf9a0 0x7f17280cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17280cf9a0 0x7f17280cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33320/0 (socket says 172.21.15.125:33320) 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 -- 172.21.15.125:0/452392906 learned_addr learned my addr 172.21.15.125:0/452392906 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 -- 172.21.15.125:0/452392906 >> v2:172.21.15.137:3300/0 conn(0x7f172815d4b0 msgr2=0x7f17280d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 --2- 172.21.15.125:0/452392906 >> v2:172.21.15.137:3300/0 conn(0x7f172815d4b0 0x7f17280d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 -- 172.21.15.125:0/452392906 >> v1:172.21.15.137:6789/0 conn(0x7f17280cddf0 legacy=0x7f17280ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:45.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.184+0000 7f1726a00640 1 -- 172.21.15.125:0/452392906 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280b29d0 con 0x7f17280cf9a0 2024-01-16T14:56:45.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.185+0000 7f1726a00640 1 --2- 172.21.15.125:0/452392906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cf9a0 0x7f17280cfde0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f172808b5b0 tx=0x7f171002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc926568ac43c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:45.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.186+0000 7f1726000640 1 -- 172.21.15.125:0/452392906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171002ca80 con 0x7f17280cf9a0 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.186+0000 7f1726000640 1 -- 172.21.15.125:0/452392906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171002cbe0 con 0x7f17280cf9a0 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.186+0000 7f1726000640 1 -- 172.21.15.125:0/452392906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17100343f0 con 0x7f17280cf9a0 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.186+0000 7f172cc00640 1 -- 172.21.15.125:0/452392906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cf9a0 msgr2=0x7f17280cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.186+0000 7f172cc00640 1 --2- 172.21.15.125:0/452392906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cf9a0 0x7f17280cfde0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f172808b5b0 tx=0x7f171002bee0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 -- 172.21.15.125:0/452392906 shutdown_connections 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 --2- 172.21.15.125:0/452392906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cf9a0 0x7f17280cfde0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 --2- 172.21.15.125:0/452392906 >> v2:172.21.15.137:3300/0 conn(0x7f172815d4b0 0x7f17280d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 -- 172.21.15.125:0/452392906 >> 172.21.15.125:0/452392906 conn(0x7f17280d3000 msgr2=0x7f17280d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 -- 172.21.15.125:0/452392906 shutdown_connections 2024-01-16T14:56:45.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.187+0000 7f172cc00640 1 -- 172.21.15.125:0/452392906 wait complete. 2024-01-16T14:56:45.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.188+0000 7f172cc00640 1 Processor -- start 2024-01-16T14:56:45.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.188+0000 7f172cc00640 1 -- start start 2024-01-16T14:56:45.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17280cf9a0 0x7f17281f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f172815d4b0 0x7f172814e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17281f7130 con 0x7f17280cddf0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17280cb230 con 0x7f17280cf9a0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f172cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17280cd0a0 con 0x7f172815d4b0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.189+0000 7f1727400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33328/0 (socket says 172.21.15.125:33328) 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 -- 172.21.15.125:0/3739353273 learned_addr learned my addr 172.21.15.125:0/3739353273 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1726a00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17280cf9a0 0x7f17281f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 -- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f172815d4b0 msgr2=0x7f172814e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f172815d4b0 0x7f172814e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 -- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17280cf9a0 msgr2=0x7f17281f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17280cf9a0 0x7f17281f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 -- 172.21.15.125:0/3739353273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1710009b30 con 0x7f17280cddf0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f1727400640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f171800b470 tx=0x7f171800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:45.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718011890 con 0x7f17280cddf0 2024-01-16T14:56:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1718011a10 con 0x7f17280cddf0 2024-01-16T14:56:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172815da10 con 0x7f17280cddf0 2024-01-16T14:56:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.190+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718011dc0 con 0x7f17280cddf0 2024-01-16T14:56:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.191+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17280ca620 con 0x7f17280cddf0 2024-01-16T14:56:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.192+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172804ed60 con 0x7f17280cddf0 2024-01-16T14:56:45.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.193+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f171801a400 con 0x7f17280cddf0 2024-01-16T14:56:45.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.194+0000 7f171f400640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 0x7f170006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:45.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.194+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f171808a430 con 0x7f17280cddf0 2024-01-16T14:56:45.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.194+0000 7f1726a00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 0x7f170006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:45.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.195+0000 7f1726a00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 0x7f170006d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1710035000 tx=0x7f1710030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:45.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.198+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17180553e0 con 0x7f17280cddf0 2024-01-16T14:56:45.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.413+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1728067450 con 0x7f17280cddf0 2024-01-16T14:56:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:45.998+0000 7f171f400640 1 -- 172.21.15.125:0/3739353273 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1718058a00 con 0x7f17280cddf0 2024-01-16T14:56:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 2 and data pool 3 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 msgr2=0x7f170006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 0x7f170006d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1710035000 tx=0x7f1710030000 comp rx=0 tx=0).stop 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 msgr2=0x7f17280cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f171800b470 tx=0x7f171800b930 comp rx=0 tx=0).stop 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 shutdown_connections 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f172815d4b0 0x7f172814e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17280cf9a0 0x7f17281f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f170006b2c0 0x7f170006d780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 --2- 172.21.15.125:0/3739353273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280cddf0 0x7f17280cb6c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:45.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 >> 172.21.15.125:0/3739353273 conn(0x7f17280d3000 msgr2=0x7f17280b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:45.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 shutdown_connections 2024-01-16T14:56:45.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.001+0000 7f172cc00640 1 -- 172.21.15.125:0/3739353273 wait complete. 2024-01-16T14:56:46.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:46.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.196+0000 7fc409800640 1 Processor -- start 2024-01-16T14:56:46.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.196+0000 7fc409800640 1 -- start start 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.197+0000 7fc409800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4040cf9a0 0x7fc404070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.197+0000 7fc409800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc404070ea0 0x7fc4040cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.197+0000 7fc409800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc404071690 con 0x7fc40414b2c0 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.197+0000 7fc409800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40414b6a0 con 0x7fc4040cf9a0 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.197+0000 7fc409800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc404071970 con 0x7fc404070ea0 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4040cf9a0 0x7fc404070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4040cf9a0 0x7fc404070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33334/0 (socket says 172.21.15.125:33334) 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 -- 172.21.15.125:0/1999408411 learned_addr learned my addr 172.21.15.125:0/1999408411 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc402a00640 1 --1- 172.21.15.125:0/1999408411 >> v1:172.21.15.125:6789/0 conn(0x7fc40414b2c0 0x7fc4040cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37708/0 (socket says 172.21.15.125:37708) 2024-01-16T14:56:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc403400640 1 --2- 172.21.15.125:0/1999408411 >> v2:172.21.15.137:3300/0 conn(0x7fc404070ea0 0x7fc4040cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 -- 172.21.15.125:0/1999408411 >> v2:172.21.15.137:3300/0 conn(0x7fc404070ea0 msgr2=0x7fc4040cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:46.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 --2- 172.21.15.125:0/1999408411 >> v2:172.21.15.137:3300/0 conn(0x7fc404070ea0 0x7fc4040cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 -- 172.21.15.125:0/1999408411 >> v1:172.21.15.125:6789/0 conn(0x7fc40414b2c0 legacy=0x7fc4040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.198+0000 7fc3f9e00640 1 -- 172.21.15.125:0/1999408411 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404071c50 con 0x7fc4040cf9a0 2024-01-16T14:56:46.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.199+0000 7fc3f9e00640 1 --2- 172.21.15.125:0/1999408411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4040cf9a0 0x7fc404070960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc404165330 tx=0x7fc3ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d2ffb2fdd6d9c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.199+0000 7fc402000640 1 -- 172.21.15.125:0/1999408411 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec030860 con 0x7fc4040cf9a0 2024-01-16T14:56:46.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.199+0000 7fc402000640 1 -- 172.21.15.125:0/1999408411 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec0309c0 con 0x7fc4040cf9a0 2024-01-16T14:56:46.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.199+0000 7fc402000640 1 -- 172.21.15.125:0/1999408411 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec030d50 con 0x7fc4040cf9a0 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 -- 172.21.15.125:0/1999408411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4040cf9a0 msgr2=0x7fc404070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 --2- 172.21.15.125:0/1999408411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4040cf9a0 0x7fc404070960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc404165330 tx=0x7fc3ec02b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 -- 172.21.15.125:0/1999408411 shutdown_connections 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 --2- 172.21.15.125:0/1999408411 >> v2:172.21.15.137:3300/0 conn(0x7fc404070ea0 0x7fc4040cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 --2- 172.21.15.125:0/1999408411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4040cf9a0 0x7fc404070960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 -- 172.21.15.125:0/1999408411 >> 172.21.15.125:0/1999408411 conn(0x7fc40406a800 msgr2=0x7fc40406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 -- 172.21.15.125:0/1999408411 shutdown_connections 2024-01-16T14:56:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.200+0000 7fc409800640 1 -- 172.21.15.125:0/1999408411 wait complete. 2024-01-16T14:56:46.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.202+0000 7fc409800640 1 Processor -- start 2024-01-16T14:56:46.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.202+0000 7fc409800640 1 -- start start 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4040cf9a0 0x7fc40415e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40414b2c0 0x7fc404140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041fac20 con 0x7fc404070ea0 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415fd80 con 0x7fc4040cf9a0 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc409800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415f200 con 0x7fc40414b2c0 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc402a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc402a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33348/0 (socket says 172.21.15.125:33348) 2024-01-16T14:56:46.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.203+0000 7fc402a00640 1 -- 172.21.15.125:0/700720589 learned_addr learned my addr 172.21.15.125:0/700720589 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 -- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40414b2c0 msgr2=0x7fc404140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc3f9e00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4040cf9a0 0x7fc40415e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40414b2c0 0x7fc404140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 -- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4040cf9a0 msgr2=0x7fc40415e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4040cf9a0 0x7fc40415e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 -- 172.21.15.125:0/700720589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0094d0 con 0x7fc404070ea0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc402a00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400c3e0 tx=0x7fc3f400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40074e0 con 0x7fc404070ea0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f4007640 con 0x7fc404070ea0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40049e0 con 0x7fc404070ea0 2024-01-16T14:56:46.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40415f4e0 con 0x7fc404070ea0 2024-01-16T14:56:46.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.204+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40415faa0 con 0x7fc404070ea0 2024-01-16T14:56:46.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.205+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4040675d0 con 0x7fc404070ea0 2024-01-16T14:56:46.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.206+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc3f4004b40 con 0x7fc404070ea0 2024-01-16T14:56:46.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.207+0000 7fc3fb400640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 0x7fc3dc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.207+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3f40895d0 con 0x7fc404070ea0 2024-01-16T14:56:46.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.207+0000 7fc3f9e00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 0x7fc3dc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.208+0000 7fc3f9e00640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 0x7fc3dc06d780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec036000 tx=0x7fc3ec03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.212+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3f4054430 con 0x7fc404070ea0 2024-01-16T14:56:46.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.386+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4040b20a0 con 0x7fc404070ea0 2024-01-16T14:56:46.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.390+0000 7fc3fb400640 1 -- 172.21.15.125:0/700720589 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fc3f4057a50 con 0x7fc404070ea0 2024-01-16T14:56:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:46.372 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":19,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:45.415154+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:46.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 msgr2=0x7fc3dc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 0x7fc3dc06d780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec036000 tx=0x7fc3ec03b040 comp rx=0 tx=0).stop 2024-01-16T14:56:46.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 msgr2=0x7fc4041fa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400c3e0 tx=0x7fc3f400c8a0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 shutdown_connections 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40414b2c0 0x7fc404140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4040cf9a0 0x7fc40415e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3dc06b2c0 0x7fc3dc06d780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 --2- 172.21.15.125:0/700720589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc404070ea0 0x7fc4041fa6e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 >> 172.21.15.125:0/700720589 conn(0x7fc40406a800 msgr2=0x7fc40406cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 shutdown_connections 2024-01-16T14:56:46.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.392+0000 7fc409800640 1 -- 172.21.15.125:0/700720589 wait complete. 2024-01-16T14:56:46.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.613+0000 7fbf4a800640 1 Processor -- start 2024-01-16T14:56:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.613+0000 7fbf4a800640 1 -- start start 2024-01-16T14:56:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf4a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf440ae5a0 0x7fbf4415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf4a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 0x7fbf440cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf4a800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf44070c40 con 0x7fbf440b3ae0 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf4a800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf44071450 con 0x7fbf440ae5a0 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf4a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440cfc60 con 0x7fbf440cf290 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf43400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf440ae5a0 0x7fbf4415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf3be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbf440b3ae0 0x7fbf4406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36434/0 (socket says 172.21.15.125:36434) 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf3be00640 1 -- 172.21.15.125:0/1894223719 learned_addr learned my addr 172.21.15.125:0/1894223719 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.615+0000 7fbf43e00640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 0x7fbf440cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.616+0000 7fbf43400640 1 -- 172.21.15.125:0/1894223719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 msgr2=0x7fbf440cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.616+0000 7fbf43400640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 0x7fbf440cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.616+0000 7fbf43400640 1 -- 172.21.15.125:0/1894223719 >> v1:172.21.15.125:6789/0 conn(0x7fbf440b3ae0 legacy=0x7fbf4406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.616+0000 7fbf43400640 1 -- 172.21.15.125:0/1894223719 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4406c470 con 0x7fbf440ae5a0 2024-01-16T14:56:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.616+0000 7fbf43e00640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 0x7fbf440cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.617+0000 7fbf43400640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440ae5a0 0x7fbf4415e430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf340097a0 tx=0x7fbf3402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eedf4c4550b0855 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.617+0000 7fbf42a00640 1 -- 172.21.15.125:0/1894223719 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34030800 con 0x7fbf440ae5a0 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.617+0000 7fbf42a00640 1 -- 172.21.15.125:0/1894223719 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34030960 con 0x7fbf440ae5a0 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.617+0000 7fbf42a00640 1 -- 172.21.15.125:0/1894223719 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34030cf0 con 0x7fbf440ae5a0 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 -- 172.21.15.125:0/1894223719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440ae5a0 msgr2=0x7fbf4415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440ae5a0 0x7fbf4415e430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf340097a0 tx=0x7fbf3402b940 comp rx=0 tx=0).stop 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 -- 172.21.15.125:0/1894223719 shutdown_connections 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440cf290 0x7fbf440cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 --2- 172.21.15.125:0/1894223719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440ae5a0 0x7fbf4415e430 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 -- 172.21.15.125:0/1894223719 >> 172.21.15.125:0/1894223719 conn(0x7fbf4406a800 msgr2=0x7fbf4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 -- 172.21.15.125:0/1894223719 shutdown_connections 2024-01-16T14:56:46.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.618+0000 7fbf4a800640 1 -- 172.21.15.125:0/1894223719 wait complete. 2024-01-16T14:56:46.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.620+0000 7fbf4a800640 1 Processor -- start 2024-01-16T14:56:46.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.620+0000 7fbf4a800640 1 -- start start 2024-01-16T14:56:46.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf440ae5a0 0x7fbf44150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440b3ae0 0x7fbf4414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44151660 con 0x7fbf440cf290 2024-01-16T14:56:46.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf441f63f0 con 0x7fbf440ae5a0 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf4a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf441f66d0 con 0x7fbf440b3ae0 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47576/0 (socket says 172.21.15.125:47576) 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 -- 172.21.15.125:0/414057480 learned_addr learned my addr 172.21.15.125:0/414057480 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf3be00640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440b3ae0 0x7fbf4414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43400640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf440ae5a0 0x7fbf44150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 -- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440b3ae0 msgr2=0x7fbf4414bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440b3ae0 0x7fbf4414bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 -- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf440ae5a0 msgr2=0x7fbf44150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf440ae5a0 0x7fbf44150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.621+0000 7fbf43e00640 1 -- 172.21.15.125:0/414057480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf34009400 con 0x7fbf440cf290 2024-01-16T14:56:46.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf43e00640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00cd50 tx=0x7fbf3c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00f680 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c00f800 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00fbb0 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf441f69b0 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.622+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf44151310 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.623+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf441f6bd0 con 0x7fbf440cf290 2024-01-16T14:56:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.624+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbf3c00b040 con 0x7fbf440cf290 2024-01-16T14:56:46.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.625+0000 7fbf3b400640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 0x7fbf1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:46.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.626+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf3c08a5e0 con 0x7fbf440cf290 2024-01-16T14:56:46.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.626+0000 7fbf43400640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 0x7fbf1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:46.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.626+0000 7fbf43400640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 0x7fbf1c06d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036000 tx=0x7fbf34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.630+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf3c055440 con 0x7fbf440cf290 2024-01-16T14:56:46.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.821+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf440675d0 con 0x7fbf440cf290 2024-01-16T14:56:46.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.824+0000 7fbf3b400640 1 -- 172.21.15.125:0/414057480 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fbf3c058a60 con 0x7fbf440cf290 2024-01-16T14:56:46.806 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:46.806 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":19,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:45.415154+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:46.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.826+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 msgr2=0x7fbf1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.826+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 0x7fbf1c06d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036000 tx=0x7fbf34008030 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.826+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 msgr2=0x7fbf441593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.826+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00cd50 tx=0x7fbf3c00a650 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 shutdown_connections 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbf1c06b2c0 0x7fbf1c06d780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf440cf290 0x7fbf441593b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf440b3ae0 0x7fbf4414bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 --2- 172.21.15.125:0/414057480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf440ae5a0 0x7fbf44150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 >> 172.21.15.125:0/414057480 conn(0x7fbf4406a800 msgr2=0x7fbf4415d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 shutdown_connections 2024-01-16T14:56:46.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:46.827+0000 7fbf4a800640 1 -- 172.21.15.125:0/414057480 wait complete. 2024-01-16T14:56:46.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:56:47.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.052+0000 7f456ce00640 1 Processor -- start 2024-01-16T14:56:47.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.052+0000 7f456ce00640 1 -- start start 2024-01-16T14:56:47.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f456ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45680cf9a0 0x7f456815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f456ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f456815e1e0 0x7f45680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f456ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45680b1ee0 con 0x7f4568143ae0 2024-01-16T14:56:47.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f456ce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45680b2780 con 0x7f45680cf9a0 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f456ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456815e990 con 0x7f456815e1e0 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45680cf9a0 0x7f456815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45680cf9a0 0x7f456815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47588/0 (socket says 172.21.15.125:47588) 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 -- 172.21.15.125:0/3210436740 learned_addr learned my addr 172.21.15.125:0/3210436740 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 -- 172.21.15.125:0/3210436740 >> v2:172.21.15.137:3300/0 conn(0x7f456815e1e0 msgr2=0x7f45680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 --2- 172.21.15.125:0/3210436740 >> v2:172.21.15.137:3300/0 conn(0x7f456815e1e0 0x7f45680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.054+0000 7f455fe00640 1 -- 172.21.15.125:0/3210436740 >> v1:172.21.15.137:6789/0 conn(0x7f4568143ae0 legacy=0x7f45680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:47.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.055+0000 7f455fe00640 1 -- 172.21.15.125:0/3210436740 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680b2130 con 0x7f45680cf9a0 2024-01-16T14:56:47.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.056+0000 7f455fe00640 1 --2- 172.21.15.125:0/3210436740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f456815dca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f45680ae9d0 tx=0x7f455002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4454ccfba5693fe3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:47.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.056+0000 7f4565400640 1 -- 172.21.15.125:0/3210436740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550031890 con 0x7f45680cf9a0 2024-01-16T14:56:47.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.056+0000 7f4565400640 1 -- 172.21.15.125:0/3210436740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45500319f0 con 0x7f45680cf9a0 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.056+0000 7f4565400640 1 -- 172.21.15.125:0/3210436740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550031d80 con 0x7f45680cf9a0 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 -- 172.21.15.125:0/3210436740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 msgr2=0x7f456815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 --2- 172.21.15.125:0/3210436740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f456815dca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f45680ae9d0 tx=0x7f455002b930 comp rx=0 tx=0).stop 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 -- 172.21.15.125:0/3210436740 shutdown_connections 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 --2- 172.21.15.125:0/3210436740 >> v2:172.21.15.137:3300/0 conn(0x7f456815e1e0 0x7f45680702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 --2- 172.21.15.125:0/3210436740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f456815dca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 -- 172.21.15.125:0/3210436740 >> 172.21.15.125:0/3210436740 conn(0x7f456806a800 msgr2=0x7f456806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 -- 172.21.15.125:0/3210436740 shutdown_connections 2024-01-16T14:56:47.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.057+0000 7f456ce00640 1 -- 172.21.15.125:0/3210436740 wait complete. 2024-01-16T14:56:47.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.058+0000 7f456ce00640 1 Processor -- start 2024-01-16T14:56:47.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.059+0000 7f456ce00640 1 -- start start 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.059+0000 7f456ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f456ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4568143ae0 0x7f45680a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f456ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f456815e1e0 0x7f456813dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f456ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4568159f40 con 0x7f45680cf9a0 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f456ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456813fbb0 con 0x7f4568143ae0 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f456ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456813ad20 con 0x7f456815e1e0 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47590/0 (socket says 172.21.15.125:47590) 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 -- 172.21.15.125:0/6042118 learned_addr learned my addr 172.21.15.125:0/6042118 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 -- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f456815e1e0 msgr2=0x7f456813dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f456815e1e0 0x7f456813dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 -- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4568143ae0 msgr2=0x7f45680a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4568143ae0 0x7f45680a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.060+0000 7f4565e00640 1 -- 172.21.15.125:0/6042118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45500094d0 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f4565e00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f455800c410 tx=0x7f455800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580035f0 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558003750 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558003b00 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456813bb80 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.061+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45680a8a00 con 0x7f45680cf9a0 2024-01-16T14:56:47.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.062+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45680675d0 con 0x7f45680cf9a0 2024-01-16T14:56:47.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.063+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4558003c60 con 0x7f45680cf9a0 2024-01-16T14:56:47.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.064+0000 7f455e000640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 0x7f454006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.064+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4558088e40 con 0x7f45680cf9a0 2024-01-16T14:56:47.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.064+0000 7f455fe00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 0x7f454006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.065+0000 7f455fe00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 0x7f454006d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4550036000 tx=0x7f4550030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:47.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.068+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4558053ca0 con 0x7f45680cf9a0 2024-01-16T14:56:47.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:47.294+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f456806b3f0 con 0x7f45680cf9a0 2024-01-16T14:56:48.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.018+0000 7f455e000640 1 -- 172.21.15.125:0/6042118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f45580572c0 con 0x7f45680cf9a0 2024-01-16T14:56:48.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 msgr2=0x7f454006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 0x7f454006d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4550036000 tx=0x7f4550030000 comp rx=0 tx=0).stop 2024-01-16T14:56:48.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 msgr2=0x7f4568159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f455800c410 tx=0x7f455800c8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 shutdown_connections 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f456815e1e0 0x7f456813dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4568143ae0 0x7f45680a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f454006b2c0 0x7f454006d780 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 --2- 172.21.15.125:0/6042118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45680cf9a0 0x7f4568159a00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.021+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 >> 172.21.15.125:0/6042118 conn(0x7f456806a800 msgr2=0x7f456806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.022+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 shutdown_connections 2024-01-16T14:56:48.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.022+0000 7f456ce00640 1 -- 172.21.15.125:0/6042118 wait complete. 2024-01-16T14:56:48.024 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:56:48.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:56:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.250+0000 7f9779200640 1 Processor -- start 2024-01-16T14:56:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.251+0000 7f9779200640 1 -- start start 2024-01-16T14:56:48.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9779200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97740d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9779200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740b2730 con 0x7f97740b41e0 2024-01-16T14:56:48.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9779200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740d1ec0 con 0x7f97740cf9a0 2024-01-16T14:56:48.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9779200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97740b29d0 con 0x7f977415d4b0 2024-01-16T14:56:48.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9773e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f97740b41e0 0x7f97740ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36448/0 (socket says 172.21.15.125:36448) 2024-01-16T14:56:48.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.252+0000 7f9773e00640 1 -- 172.21.15.125:0/2715946311 learned_addr learned my addr 172.21.15.125:0/2715946311 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:48.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9773400640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97740d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747326278 0 0) 0x7f97740b2730 con 0x7f97740b41e0 2024-01-16T14:56:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754003520 con 0x7f97740b41e0 2024-01-16T14:56:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472353480 0 0) 0x7f9754003520 con 0x7f97740b41e0 2024-01-16T14:56:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 msgr2=0x7f97740d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97740d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 >> v1:172.21.15.137:6789/0 conn(0x7f97740cf9a0 legacy=0x7f97740cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740b2cb0 con 0x7f97740b41e0 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f976c001670 con 0x7f97740b41e0 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 >> v1:172.21.15.125:6789/0 conn(0x7f97740b41e0 legacy=0x7f97740ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9754003f20 0x7f97540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.253+0000 7f9772000640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754004d00 0x7f9754005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97740b29d0 con 0x7f9754003f20 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975400ba60 con 0x7f9754004d00 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975400bc20 con 0x7f977415d4b0 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9754003f20 0x7f97540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9773400640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9773e00640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754004d00 0x7f9754005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 -- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 msgr2=0x7f97540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f977415d4b0 0x7f97540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 -- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754004d00 msgr2=0x7f9754005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754004d00 0x7f9754005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.254+0000 7f9772a00640 1 -- 172.21.15.125:0/2715946311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754003520 con 0x7f9754003f20 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.255+0000 7f9772a00640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9754003f20 0x7f97540047c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f975c001e50 tx=0x7f97540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f561a9d268259c2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.255+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c003b70 con 0x7f9754003f20 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.255+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c003cd0 con 0x7f9754003f20 2024-01-16T14:56:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.255+0000 7f9772000640 1 -- 172.21.15.125:0/2715946311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c005310 con 0x7f9754003f20 2024-01-16T14:56:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 -- 172.21.15.125:0/2715946311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9754003f20 msgr2=0x7f97540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 --2- 172.21.15.125:0/2715946311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9754003f20 0x7f97540047c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f975c001e50 tx=0x7f97540057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9773e00640 1 -- 172.21.15.125:0/2715946311 reap_dead start 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 -- 172.21.15.125:0/2715946311 shutdown_connections 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 -- 172.21.15.125:0/2715946311 >> 172.21.15.125:0/2715946311 conn(0x7f97740d3000 msgr2=0x7f97740d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 -- 172.21.15.125:0/2715946311 shutdown_connections 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.256+0000 7f9779200640 1 -- 172.21.15.125:0/2715946311 wait complete. 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.258+0000 7f9779200640 1 Processor -- start 2024-01-16T14:56:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.258+0000 7f9779200640 1 -- start start 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 0x7f977414d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97741602b0 0x7f97740c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9774160710 con 0x7f97740cbdd0 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97740cb940 con 0x7f9754003f20 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9779200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97740caab0 con 0x7f97741602b0 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9772a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9772a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47604/0 (socket says 172.21.15.125:47604) 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.259+0000 7f9772a00640 1 -- 172.21.15.125:0/259310698 learned_addr learned my addr 172.21.15.125:0/259310698 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9773e00640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97741602b0 0x7f97740c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9773400640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 0x7f977414d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 -- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97741602b0 msgr2=0x7f97740c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97741602b0 0x7f97740c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 -- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 msgr2=0x7f977414d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 0x7f977414d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 -- 172.21.15.125:0/259310698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977415da10 con 0x7f97740cbdd0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9773400640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 0x7f977414d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9772a00640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f975c001f70 tx=0x7f975c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c03e070 con 0x7f97740cbdd0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c006b50 con 0x7f97740cbdd0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97740cad30 con 0x7f97740cbdd0 2024-01-16T14:56:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.260+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c007030 con 0x7f97740cbdd0 2024-01-16T14:56:48.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.261+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97740cb2f0 con 0x7f97740cbdd0 2024-01-16T14:56:48.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.262+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97740cbad0 con 0x7f97740cbdd0 2024-01-16T14:56:48.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.263+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f975c007190 con 0x7f97740cbdd0 2024-01-16T14:56:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.264+0000 7f976b400640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 0x7f974c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.264+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f975c0ab7a0 con 0x7f97740cbdd0 2024-01-16T14:56:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.264+0000 7f9773400640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 0x7f974c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.265+0000 7f9773400640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 0x7f974c06d780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9764009a00 tx=0x7f9764008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:48.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.268+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f975c0765f0 con 0x7f97740cbdd0 2024-01-16T14:56:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:48.495+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9774067450 con 0x7f97740cbdd0 2024-01-16T14:56:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.031+0000 7f976b400640 1 -- 172.21.15.125:0/259310698 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f975c079c10 con 0x7f97740cbdd0 2024-01-16T14:56:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 msgr2=0x7f974c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 0x7f974c06d780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9764009a00 tx=0x7f9764008040 comp rx=0 tx=0).stop 2024-01-16T14:56:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 msgr2=0x7f977415fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f975c001f70 tx=0x7f975c0014a0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 shutdown_connections 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97741602b0 0x7f97740c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f974c06b2c0 0x7f974c06d780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97740cbdd0 0x7f977415fd70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 --2- 172.21.15.125:0/259310698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9754003f20 0x7f977414d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 >> 172.21.15.125:0/259310698 conn(0x7f97740d3000 msgr2=0x7f97740d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 shutdown_connections 2024-01-16T14:56:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.035+0000 7f9779200640 1 -- 172.21.15.125:0/259310698 wait complete. 2024-01-16T14:56:49.037 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:56:49.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:56:49.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.262+0000 7f8362800640 1 Processor -- start 2024-01-16T14:56:49.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.262+0000 7f8362800640 1 -- start start 2024-01-16T14:56:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.263+0000 7f8362800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f835c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.263+0000 7f8362800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0cfc30 con 0x7f835c15e3b0 2024-01-16T14:56:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.263+0000 7f8362800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0b1ee0 con 0x7f835c0cf7d0 2024-01-16T14:56:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.263+0000 7f8362800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c0b2780 con 0x7f835c143ae0 2024-01-16T14:56:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f835c15e3b0 0x7f835c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36454/0 (socket says 172.21.15.125:36454) 2024-01-16T14:56:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835be00640 1 -- 172.21.15.125:0/2549739467 learned_addr learned my addr 172.21.15.125:0/2549739467 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835b400640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f835c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152199733 0 0) 0x7f835c0cfc30 con 0x7f835c15e3b0 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c005730 con 0x7f835c15e3b0 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517622679 0 0) 0x7f833c005730 con 0x7f835c15e3b0 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 msgr2=0x7f835c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f835c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 >> v1:172.21.15.137:6789/0 conn(0x7f835c0cf7d0 legacy=0x7f835c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c15e990 con 0x7f835c15e3b0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f83540019f0 con 0x7f835c15e3b0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 >> v1:172.21.15.125:6789/0 conn(0x7f835c15e3b0 legacy=0x7f835c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f833c005bb0 0x7f833c005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.264+0000 7f835aa00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f833c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835aa00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f833c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c0b2780 con 0x7f833c0069d0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c00dcc0 con 0x7f833c005bb0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c00de80 con 0x7f835c143ae0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f833c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835b400640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f833c005bb0 0x7f833c005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835be00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f833c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 -- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 msgr2=0x7f833c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f833c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 -- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f833c005bb0 msgr2=0x7f833c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f833c005bb0 0x7f833c005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f8352800640 1 -- 172.21.15.125:0/2549739467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c005730 con 0x7f833c0069d0 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.265+0000 7f835be00640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c143ae0 0x7f833c0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.266+0000 7f8352800640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f833c006e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8344001750 tx=0x7f833c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b6e4283da0c043 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.266+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440052d0 con 0x7f833c0069d0 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.266+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834402e050 con 0x7f833c0069d0 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.266+0000 7f835aa00640 1 -- 172.21.15.125:0/2549739467 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440035f0 con 0x7f833c0069d0 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 -- 172.21.15.125:0/2549739467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 msgr2=0x7f833c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 --2- 172.21.15.125:0/2549739467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f833c006e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8344001750 tx=0x7f833c007a00 comp rx=0 tx=0).stop 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f835be00640 1 -- 172.21.15.125:0/2549739467 reap_dead start 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 -- 172.21.15.125:0/2549739467 shutdown_connections 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 -- 172.21.15.125:0/2549739467 >> 172.21.15.125:0/2549739467 conn(0x7f835c06a800 msgr2=0x7f835c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 -- 172.21.15.125:0/2549739467 shutdown_connections 2024-01-16T14:56:49.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.267+0000 7f8362800640 1 -- 172.21.15.125:0/2549739467 wait complete. 2024-01-16T14:56:49.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.269+0000 7f8362800640 1 Processor -- start 2024-01-16T14:56:49.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.269+0000 7f8362800640 1 -- start start 2024-01-16T14:56:49.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c1376b0 0x7f835c0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f835c137b30 0x7f835c0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c15a4e0 con 0x7f833c0069d0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c15bd80 con 0x7f835c137b30 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f8362800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c15c060 con 0x7f835c1376b0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f835b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f835b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47616/0 (socket says 172.21.15.125:47616) 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f835b400640 1 -- 172.21.15.125:0/2596546799 learned_addr learned my addr 172.21.15.125:0/2596546799 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.270+0000 7f835b400640 1 -- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c1376b0 msgr2=0x7f835c0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f835b400640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c1376b0 0x7f835c0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f835b400640 1 -- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f835c137b30 msgr2=0x7f835c0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f835b400640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f835c137b30 0x7f835c0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f835b400640 1 -- 172.21.15.125:0/2596546799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c15c340 con 0x7f833c0069d0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f835b400640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f834c00f580 tx=0x7f834c00fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c014870 con 0x7f833c0069d0 2024-01-16T14:56:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c0149d0 con 0x7f833c0069d0 2024-01-16T14:56:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c014d80 con 0x7f833c0069d0 2024-01-16T14:56:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c15a730 con 0x7f833c0069d0 2024-01-16T14:56:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.271+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c0ad840 con 0x7f833c0069d0 2024-01-16T14:56:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.272+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c0675d0 con 0x7f833c0069d0 2024-01-16T14:56:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.273+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f834c015410 con 0x7f833c0069d0 2024-01-16T14:56:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.274+0000 7f8353e00640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 0x7f833406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.274+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f834c08a610 con 0x7f833c0069d0 2024-01-16T14:56:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.274+0000 7f8352800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 0x7f833406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.275+0000 7f8352800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 0x7f833406d780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8344033000 tx=0x7f834402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.278+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f834c055490 con 0x7f833c0069d0 2024-01-16T14:56:49.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.505+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f835c141f20 con 0x7f833c0069d0 2024-01-16T14:56:49.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.507+0000 7f8353e00640 1 -- 172.21.15.125:0/2596546799 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f834c058ab0 con 0x7f833c0069d0 2024-01-16T14:56:49.489 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:49.489 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:56:49.489 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 msgr2=0x7f833406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 0x7f833406d780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8344033000 tx=0x7f834402f000 comp rx=0 tx=0).stop 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 msgr2=0x7f835c159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f834c00f580 tx=0x7f834c00fa40 comp rx=0 tx=0).stop 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 shutdown_connections 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f835c137b30 0x7f835c0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f835c1376b0 0x7f835c0ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f833406b2c0 0x7f833406d780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 --2- 172.21.15.125:0/2596546799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f833c0069d0 0x7f835c159a00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 >> 172.21.15.125:0/2596546799 conn(0x7f835c06a800 msgr2=0x7f835c0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 shutdown_connections 2024-01-16T14:56:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.509+0000 7f8362800640 1 -- 172.21.15.125:0/2596546799 wait complete. 2024-01-16T14:56:49.511 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:49.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.733+0000 7ff197c00640 1 Processor -- start 2024-01-16T14:56:49.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.733+0000 7ff197c00640 1 -- start start 2024-01-16T14:56:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff197c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff19013df80 0x7ff19015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff197c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1900cf9a0 0x7ff1900cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff197c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19006c470 con 0x7ff19013e360 2024-01-16T14:56:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff197c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19006c750 con 0x7ff19013df80 2024-01-16T14:56:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff197c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19006ca30 con 0x7ff1900cf9a0 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff19013df80 0x7ff19015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff19013df80 0x7ff19015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47632/0 (socket says 172.21.15.125:47632) 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff187c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff19013e360 0x7ff1900cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36464/0 (socket says 172.21.15.125:36464) 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 -- 172.21.15.125:0/2640054462 learned_addr learned my addr 172.21.15.125:0/2640054462 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 -- 172.21.15.125:0/2640054462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1900cf9a0 msgr2=0x7ff1900cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 --2- 172.21.15.125:0/2640054462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1900cf9a0 0x7ff1900cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 -- 172.21.15.125:0/2640054462 >> v1:172.21.15.125:6789/0 conn(0x7ff19013e360 legacy=0x7ff1900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff195200640 1 -- 172.21.15.125:0/2640054462 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19006cd10 con 0x7ff19013df80 2024-01-16T14:56:49.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.735+0000 7ff18fe00640 1 -- 172.21.15.125:0/2640054462 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480909956 0 0) 0x7ff19006c470 con 0x7ff19013e360 2024-01-16T14:56:49.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff195200640 1 --2- 172.21.15.125:0/2640054462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff19013df80 0x7ff19015bee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc09c6e7fba74a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff18fe00640 1 -- 172.21.15.125:0/2640054462 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180030840 con 0x7ff19013df80 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff18fe00640 1 -- 172.21.15.125:0/2640054462 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1800309a0 con 0x7ff19013df80 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff18fe00640 1 -- 172.21.15.125:0/2640054462 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180030d30 con 0x7ff19013df80 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff197c00640 1 -- 172.21.15.125:0/2640054462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff19013df80 msgr2=0x7ff19015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.737+0000 7ff197c00640 1 --2- 172.21.15.125:0/2640054462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff19013df80 0x7ff19015bee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002b980 comp rx=0 tx=0).stop 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 -- 172.21.15.125:0/2640054462 shutdown_connections 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 --2- 172.21.15.125:0/2640054462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1900cf9a0 0x7ff1900cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 --2- 172.21.15.125:0/2640054462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff19013df80 0x7ff19015bee0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 -- 172.21.15.125:0/2640054462 >> 172.21.15.125:0/2640054462 conn(0x7ff19006a800 msgr2=0x7ff19006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 -- 172.21.15.125:0/2640054462 shutdown_connections 2024-01-16T14:56:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.738+0000 7ff197c00640 1 -- 172.21.15.125:0/2640054462 wait complete. 2024-01-16T14:56:49.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.740+0000 7ff197c00640 1 Processor -- start 2024-01-16T14:56:49.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.740+0000 7ff197c00640 1 -- start start 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff19013df80 0x7ff19014ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 0x7ff19014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19013ddf0 con 0x7ff1900cf9a0 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900b38d0 con 0x7ff19013e360 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff197c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19014ca90 con 0x7ff19013df80 2024-01-16T14:56:49.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47642/0 (socket says 172.21.15.125:47642) 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 -- 172.21.15.125:0/2228798704 learned_addr learned my addr 172.21.15.125:0/2228798704 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 -- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff19013df80 msgr2=0x7ff19014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff19013df80 0x7ff19014ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 -- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 msgr2=0x7ff19014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 0x7ff19014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 0x7ff19014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.741+0000 7ff195200640 1 -- 172.21.15.125:0/2228798704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1800094d0 con 0x7ff1900cf9a0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff195c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 0x7ff19014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff195200640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff180000c00 tx=0x7ff180004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180039d60 con 0x7ff1900cf9a0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff180043400 con 0x7ff1900cf9a0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180043790 con 0x7ff1900cf9a0 2024-01-16T14:56:49.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19014e3b0 con 0x7ff1900cf9a0 2024-01-16T14:56:49.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.742+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19014c440 con 0x7ff1900cf9a0 2024-01-16T14:56:49.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.743+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1900675d0 con 0x7ff1900cf9a0 2024-01-16T14:56:49.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.744+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff1800438f0 con 0x7ff1900cf9a0 2024-01-16T14:56:49.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.745+0000 7ff18d600640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 0x7ff16806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.745+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff18003e070 con 0x7ff1900cf9a0 2024-01-16T14:56:49.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.745+0000 7ff187c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 0x7ff16806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.746+0000 7ff187c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 0x7ff16806d780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff178007fd0 tx=0x7ff178007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.749+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff180077d30 con 0x7ff1900cf9a0 2024-01-16T14:56:49.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.938+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff19006df00 con 0x7ff1900cf9a0 2024-01-16T14:56:49.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.941+0000 7ff18d600640 1 -- 172.21.15.125:0/2228798704 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7ff180034090 con 0x7ff1900cf9a0 2024-01-16T14:56:49.923 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:49.923 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 msgr2=0x7ff16806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 0x7ff16806d780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff178007fd0 tx=0x7ff178007920 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 msgr2=0x7ff19013d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff180000c00 tx=0x7ff180004730 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 shutdown_connections 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff19013e360 0x7ff19014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff19013df80 0x7ff19014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff16806b2c0 0x7ff16806d780 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 --2- 172.21.15.125:0/2228798704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1900cf9a0 0x7ff19013d8b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.943+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 >> 172.21.15.125:0/2228798704 conn(0x7ff19006a800 msgr2=0x7ff190153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.944+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 shutdown_connections 2024-01-16T14:56:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:49.944+0000 7ff197c00640 1 -- 172.21.15.125:0/2228798704 wait complete. 2024-01-16T14:56:49.944 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:56:49.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:56:50.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.151+0000 7ff43f600640 1 Processor -- start 2024-01-16T14:56:50.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.151+0000 7ff43f600640 1 -- start start 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff43f600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff43813df80 0x7ff43815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff43f600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43806c470 con 0x7ff43813e360 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff43f600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43806c750 con 0x7ff4380cf9a0 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff43f600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43806ca30 con 0x7ff43813df80 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff42fc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff43813e360 0x7ff4380cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36476/0 (socket says 172.21.15.125:36476) 2024-01-16T14:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.153+0000 7ff42fc00640 1 -- 172.21.15.125:0/2179721270 learned_addr learned my addr 172.21.15.125:0/2179721270 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837783757 0 0) 0x7ff43806c470 con 0x7ff43813e360 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff418003520 con 0x7ff43813e360 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2179721270 >> v2:172.21.15.137:3300/0 conn(0x7ff43813df80 0x7ff43815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274102663 0 0) 0x7ff418003520 con 0x7ff43813e360 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 >> v2:172.21.15.137:3300/0 conn(0x7ff43813df80 msgr2=0x7ff43815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 --2- 172.21.15.125:0/2179721270 >> v2:172.21.15.137:3300/0 conn(0x7ff43813df80 0x7ff43815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 >> v1:172.21.15.137:6789/0 conn(0x7ff4380cf9a0 legacy=0x7ff4380cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43806cd10 con 0x7ff43813e360 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff4200019f0 con 0x7ff43813e360 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 >> v1:172.21.15.125:6789/0 conn(0x7ff43813e360 legacy=0x7ff4380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 0x7ff418003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4180047c0 0x7ff418004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418005180 0x7ff41800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43806ca30 con 0x7ff418003970 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41800c0d0 con 0x7ff418005180 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41800c290 con 0x7ff4180047c0 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.154+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 0x7ff418003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43cc00640 1 -- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4180047c0 msgr2=0x7ff418004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4180047c0 0x7ff418004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43d600640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418005180 0x7ff41800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43cc00640 1 -- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418005180 msgr2=0x7ff41800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418005180 0x7ff41800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43cc00640 1 -- 172.21.15.125:0/2179721270 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff418003520 con 0x7ff418003970 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43d600640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418005180 0x7ff41800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.155+0000 7ff43d600640 1 -- 172.21.15.125:0/2179721270 reap_dead start 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.156+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 0x7ff418003d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff42800ba30 tx=0x7ff4180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62679163f1579e71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.156+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428032640 con 0x7ff418003970 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.156+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4280327a0 con 0x7ff418003970 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.156+0000 7ff437e00640 1 -- 172.21.15.125:0/2179721270 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428032b30 con 0x7ff418003970 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 -- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 msgr2=0x7ff418003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 0x7ff418003d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff42800ba30 tx=0x7ff4180057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 -- 172.21.15.125:0/2179721270 shutdown_connections 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 --2- 172.21.15.125:0/2179721270 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff418003970 0x7ff418003d90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 -- 172.21.15.125:0/2179721270 >> 172.21.15.125:0/2179721270 conn(0x7ff43806a800 msgr2=0x7ff43806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 -- 172.21.15.125:0/2179721270 shutdown_connections 2024-01-16T14:56:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.157+0000 7ff43f600640 1 -- 172.21.15.125:0/2179721270 wait complete. 2024-01-16T14:56:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.159+0000 7ff43f600640 1 Processor -- start 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.159+0000 7ff43f600640 1 -- start start 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418003970 0x7ff43814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 0x7ff4380b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b36c0 con 0x7ff4381e6b60 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43814bc10 con 0x7ff418003970 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43814bef0 con 0x7ff4380b2cf0 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47656/0 (socket says 172.21.15.125:47656) 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 -- 172.21.15.125:0/2008417992 learned_addr learned my addr 172.21.15.125:0/2008417992 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 -- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 msgr2=0x7ff4380b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418003970 0x7ff43814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43d600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 0x7ff4380b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 0x7ff4380b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 -- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418003970 msgr2=0x7ff43814b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418003970 0x7ff43814b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff42fc00640 1 -- 172.21.15.125:0/2008417992 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43814c1d0 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.160+0000 7ff43d600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 0x7ff4380b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff42fc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff42000ca50 tx=0x7ff42000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42000a7a0 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42000a920 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42000acd0 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43814c4b0 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.161+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43814b870 con 0x7ff4381e6b60 2024-01-16T14:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.162+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43814c6a0 con 0x7ff4381e6b60 2024-01-16T14:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.163+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff4200070d0 con 0x7ff4381e6b60 2024-01-16T14:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.164+0000 7ff435600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 0x7ff41006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.164+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff420089580 con 0x7ff4381e6b60 2024-01-16T14:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.165+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 0x7ff41006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.165+0000 7ff43cc00640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 0x7ff41006d780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff42800ba00 tx=0x7ff428030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.169+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4200542c0 con 0x7ff4381e6b60 2024-01-16T14:56:50.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.353+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4380b3f70 con 0x7ff41006b2c0 2024-01-16T14:56:50.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.377+0000 7ff435600640 1 -- 172.21.15.125:0/2008417992 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264507 (secure 0 0 0) 0x7ff4380b3f70 con 0x7ff41006b2c0 2024-01-16T14:56:50.363 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:50.365 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:56:50.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.383+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 msgr2=0x7ff41006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.383+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 0x7ff41006d780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff42800ba00 tx=0x7ff428030000 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.383+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 msgr2=0x7ff43813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.383+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff42000ca50 tx=0x7ff42000cf10 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 shutdown_connections 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4380b2cf0 0x7ff4380b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41006b2c0 0x7ff41006d780 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4381e6b60 0x7ff43813d3f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 --2- 172.21.15.125:0/2008417992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff418003970 0x7ff43814b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 >> 172.21.15.125:0/2008417992 conn(0x7ff43806a800 msgr2=0x7ff438154280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 shutdown_connections 2024-01-16T14:56:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.384+0000 7ff43f600640 1 -- 172.21.15.125:0/2008417992 wait complete. 2024-01-16T14:56:50.384 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:56:49.918706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2887680,"data_stored":1751817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958667"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956233+0000","last_change":"2024-01-16T14:56:45.980276+0000","last_active":"2024-01-16T14:56:46.956233+0000","last_peered":"2024-01-16T14:56:46.956233+0000","last_clean":"2024-01-16T14:56:46.956233+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956233+0000","last_undegraded":"2024-01-16T14:56:46.956233+0000","last_fullsized":"2024-01-16T14:56:46.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:53.535044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951915+0000","last_change":"2024-01-16T14:56:45.975609+0000","last_active":"2024-01-16T14:56:46.951915+0000","last_peered":"2024-01-16T14:56:46.951915+0000","last_clean":"2024-01-16T14:56:46.951915+0000","last_became_active":"2024-01-16T14:56:45.975291+0000","last_became_peered":"2024-01-16T14:56:45.975291+0000","last_unstale":"2024-01-16T14:56:46.951915+0000","last_undegraded":"2024-01-16T14:56:46.951915+0000","last_fullsized":"2024-01-16T14:56:46.951915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310992+0000","last_change":"2024-01-16T14:56:45.977545+0000","last_active":"2024-01-16T14:56:47.310992+0000","last_peered":"2024-01-16T14:56:47.310992+0000","last_clean":"2024-01-16T14:56:47.310992+0000","last_became_active":"2024-01-16T14:56:45.977253+0000","last_became_peered":"2024-01-16T14:56:45.977253+0000","last_unstale":"2024-01-16T14:56:47.310992+0000","last_undegraded":"2024-01-16T14:56:47.310992+0000","last_fullsized":"2024-01-16T14:56:47.310992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:34.727060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311691+0000","last_change":"2024-01-16T14:56:45.985746+0000","last_active":"2024-01-16T14:56:47.311691+0000","last_peered":"2024-01-16T14:56:47.311691+0000","last_clean":"2024-01-16T14:56:47.311691+0000","last_became_active":"2024-01-16T14:56:45.984925+0000","last_became_peered":"2024-01-16T14:56:45.984925+0000","last_unstale":"2024-01-16T14:56:47.311691+0000","last_undegraded":"2024-01-16T14:56:47.311691+0000","last_fullsized":"2024-01-16T14:56:47.311691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954569+0000","last_change":"2024-01-16T14:56:45.973905+0000","last_active":"2024-01-16T14:56:46.954569+0000","last_peered":"2024-01-16T14:56:46.954569+0000","last_clean":"2024-01-16T14:56:46.954569+0000","last_became_active":"2024-01-16T14:56:45.973591+0000","last_became_peered":"2024-01-16T14:56:45.973591+0000","last_unstale":"2024-01-16T14:56:46.954569+0000","last_undegraded":"2024-01-16T14:56:46.954569+0000","last_fullsized":"2024-01-16T14:56:46.954569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310746+0000","last_change":"2024-01-16T14:56:45.981879+0000","last_active":"2024-01-16T14:56:47.310746+0000","last_peered":"2024-01-16T14:56:47.310746+0000","last_clean":"2024-01-16T14:56:47.310746+0000","last_became_active":"2024-01-16T14:56:45.981506+0000","last_became_peered":"2024-01-16T14:56:45.981506+0000","last_unstale":"2024-01-16T14:56:47.310746+0000","last_undegraded":"2024-01-16T14:56:47.310746+0000","last_fullsized":"2024-01-16T14:56:47.310746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954278+0000","last_change":"2024-01-16T14:56:45.978039+0000","last_active":"2024-01-16T14:56:46.954278+0000","last_peered":"2024-01-16T14:56:46.954278+0000","last_clean":"2024-01-16T14:56:46.954278+0000","last_became_active":"2024-01-16T14:56:45.976116+0000","last_became_peered":"2024-01-16T14:56:45.976116+0000","last_unstale":"2024-01-16T14:56:46.954278+0000","last_undegraded":"2024-01-16T14:56:46.954278+0000","last_fullsized":"2024-01-16T14:56:46.954278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309521+0000","last_change":"2024-01-16T14:56:45.992939+0000","last_active":"2024-01-16T14:56:47.309521+0000","last_peered":"2024-01-16T14:56:47.309521+0000","last_clean":"2024-01-16T14:56:47.309521+0000","last_became_active":"2024-01-16T14:56:45.992563+0000","last_became_peered":"2024-01-16T14:56:45.992563+0000","last_unstale":"2024-01-16T14:56:47.309521+0000","last_undegraded":"2024-01-16T14:56:47.309521+0000","last_fullsized":"2024-01-16T14:56:47.309521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956108+0000","last_change":"2024-01-16T14:56:45.980818+0000","last_active":"2024-01-16T14:56:46.956108+0000","last_peered":"2024-01-16T14:56:46.956108+0000","last_clean":"2024-01-16T14:56:46.956108+0000","last_became_active":"2024-01-16T14:56:45.980565+0000","last_became_peered":"2024-01-16T14:56:45.980565+0000","last_unstale":"2024-01-16T14:56:46.956108+0000","last_undegraded":"2024-01-16T14:56:46.956108+0000","last_fullsized":"2024-01-16T14:56:46.956108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:05.361881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312563+0000","last_change":"2024-01-16T14:56:45.990281+0000","last_active":"2024-01-16T14:56:47.312563+0000","last_peered":"2024-01-16T14:56:47.312563+0000","last_clean":"2024-01-16T14:56:47.312563+0000","last_became_active":"2024-01-16T14:56:45.990014+0000","last_became_peered":"2024-01-16T14:56:45.990014+0000","last_unstale":"2024-01-16T14:56:47.312563+0000","last_undegraded":"2024-01-16T14:56:47.312563+0000","last_fullsized":"2024-01-16T14:56:47.312563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312223+0000","last_change":"2024-01-16T14:56:45.990395+0000","last_active":"2024-01-16T14:56:47.312223+0000","last_peered":"2024-01-16T14:56:47.312223+0000","last_clean":"2024-01-16T14:56:47.312223+0000","last_became_active":"2024-01-16T14:56:45.990100+0000","last_became_peered":"2024-01-16T14:56:45.990100+0000","last_unstale":"2024-01-16T14:56:47.312223+0000","last_undegraded":"2024-01-16T14:56:47.312223+0000","last_fullsized":"2024-01-16T14:56:47.312223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954100+0000","last_change":"2024-01-16T14:56:45.977544+0000","last_active":"2024-01-16T14:56:46.954100+0000","last_peered":"2024-01-16T14:56:46.954100+0000","last_clean":"2024-01-16T14:56:46.954100+0000","last_became_active":"2024-01-16T14:56:45.976912+0000","last_became_peered":"2024-01-16T14:56:45.976912+0000","last_unstale":"2024-01-16T14:56:46.954100+0000","last_undegraded":"2024-01-16T14:56:46.954100+0000","last_fullsized":"2024-01-16T14:56:46.954100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:04.924598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309263+0000","last_change":"2024-01-16T14:56:45.986451+0000","last_active":"2024-01-16T14:56:47.309263+0000","last_peered":"2024-01-16T14:56:47.309263+0000","last_clean":"2024-01-16T14:56:47.309263+0000","last_became_active":"2024-01-16T14:56:45.985694+0000","last_became_peered":"2024-01-16T14:56:45.985694+0000","last_unstale":"2024-01-16T14:56:47.309263+0000","last_undegraded":"2024-01-16T14:56:47.309263+0000","last_fullsized":"2024-01-16T14:56:47.309263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311052+0000","last_change":"2024-01-16T14:56:45.980372+0000","last_active":"2024-01-16T14:56:47.311052+0000","last_peered":"2024-01-16T14:56:47.311052+0000","last_clean":"2024-01-16T14:56:47.311052+0000","last_became_active":"2024-01-16T14:56:45.980114+0000","last_became_peered":"2024-01-16T14:56:45.980114+0000","last_unstale":"2024-01-16T14:56:47.311052+0000","last_undegraded":"2024-01-16T14:56:47.311052+0000","last_fullsized":"2024-01-16T14:56:47.311052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:10.565274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950877+0000","last_change":"2024-01-16T14:56:45.987106+0000","last_active":"2024-01-16T14:56:46.950877+0000","last_peered":"2024-01-16T14:56:46.950877+0000","last_clean":"2024-01-16T14:56:46.950877+0000","last_became_active":"2024-01-16T14:56:45.986126+0000","last_became_peered":"2024-01-16T14:56:45.986126+0000","last_unstale":"2024-01-16T14:56:46.950877+0000","last_undegraded":"2024-01-16T14:56:46.950877+0000","last_fullsized":"2024-01-16T14:56:46.950877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310669+0000","last_change":"2024-01-16T14:56:45.984242+0000","last_active":"2024-01-16T14:56:47.310669+0000","last_peered":"2024-01-16T14:56:47.310669+0000","last_clean":"2024-01-16T14:56:47.310669+0000","last_became_active":"2024-01-16T14:56:45.983936+0000","last_became_peered":"2024-01-16T14:56:45.983936+0000","last_unstale":"2024-01-16T14:56:47.310669+0000","last_undegraded":"2024-01-16T14:56:47.310669+0000","last_fullsized":"2024-01-16T14:56:47.310669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312880+0000","last_change":"2024-01-16T14:56:45.983089+0000","last_active":"2024-01-16T14:56:47.312880+0000","last_peered":"2024-01-16T14:56:47.312880+0000","last_clean":"2024-01-16T14:56:47.312880+0000","last_became_active":"2024-01-16T14:56:45.982784+0000","last_became_peered":"2024-01-16T14:56:45.982784+0000","last_unstale":"2024-01-16T14:56:47.312880+0000","last_undegraded":"2024-01-16T14:56:47.312880+0000","last_fullsized":"2024-01-16T14:56:47.312880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312245+0000","last_change":"2024-01-16T14:56:45.977860+0000","last_active":"2024-01-16T14:56:47.312245+0000","last_peered":"2024-01-16T14:56:47.312245+0000","last_clean":"2024-01-16T14:56:47.312245+0000","last_became_active":"2024-01-16T14:56:45.977563+0000","last_became_peered":"2024-01-16T14:56:45.977563+0000","last_unstale":"2024-01-16T14:56:47.312245+0000","last_undegraded":"2024-01-16T14:56:47.312245+0000","last_fullsized":"2024-01-16T14:56:47.312245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955821+0000","last_change":"2024-01-16T14:56:45.977287+0000","last_active":"2024-01-16T14:56:46.955821+0000","last_peered":"2024-01-16T14:56:46.955821+0000","last_clean":"2024-01-16T14:56:46.955821+0000","last_became_active":"2024-01-16T14:56:45.976639+0000","last_became_peered":"2024-01-16T14:56:45.976639+0000","last_unstale":"2024-01-16T14:56:46.955821+0000","last_undegraded":"2024-01-16T14:56:46.955821+0000","last_fullsized":"2024-01-16T14:56:46.955821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312028+0000","last_change":"2024-01-16T14:56:45.984360+0000","last_active":"2024-01-16T14:56:47.312028+0000","last_peered":"2024-01-16T14:56:47.312028+0000","last_clean":"2024-01-16T14:56:47.312028+0000","last_became_active":"2024-01-16T14:56:45.984102+0000","last_became_peered":"2024-01-16T14:56:45.984102+0000","last_unstale":"2024-01-16T14:56:47.312028+0000","last_undegraded":"2024-01-16T14:56:47.312028+0000","last_fullsized":"2024-01-16T14:56:47.312028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311119+0000","last_change":"2024-01-16T14:56:45.973908+0000","last_active":"2024-01-16T14:56:47.311119+0000","last_peered":"2024-01-16T14:56:47.311119+0000","last_clean":"2024-01-16T14:56:47.311119+0000","last_became_active":"2024-01-16T14:56:45.970417+0000","last_became_peered":"2024-01-16T14:56:45.970417+0000","last_unstale":"2024-01-16T14:56:47.311119+0000","last_undegraded":"2024-01-16T14:56:47.311119+0000","last_fullsized":"2024-01-16T14:56:47.311119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311312+0000","last_change":"2024-01-16T14:56:45.984704+0000","last_active":"2024-01-16T14:56:47.311312+0000","last_peered":"2024-01-16T14:56:47.311312+0000","last_clean":"2024-01-16T14:56:47.311312+0000","last_became_active":"2024-01-16T14:56:45.983436+0000","last_became_peered":"2024-01-16T14:56:45.983436+0000","last_unstale":"2024-01-16T14:56:47.311312+0000","last_undegraded":"2024-01-16T14:56:47.311312+0000","last_fullsized":"2024-01-16T14:56:47.311312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308745+0000","last_change":"2024-01-16T14:56:45.989345+0000","last_active":"2024-01-16T14:56:47.308745+0000","last_peered":"2024-01-16T14:56:47.308745+0000","last_clean":"2024-01-16T14:56:47.308745+0000","last_became_active":"2024-01-16T14:56:45.989090+0000","last_became_peered":"2024-01-16T14:56:45.989090+0000","last_unstale":"2024-01-16T14:56:47.308745+0000","last_undegraded":"2024-01-16T14:56:47.308745+0000","last_fullsized":"2024-01-16T14:56:47.308745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310832+0000","last_change":"2024-01-16T14:56:45.988656+0000","last_active":"2024-01-16T14:56:47.310832+0000","last_peered":"2024-01-16T14:56:47.310832+0000","last_clean":"2024-01-16T14:56:47.310832+0000","last_became_active":"2024-01-16T14:56:45.988350+0000","last_became_peered":"2024-01-16T14:56:45.988350+0000","last_unstale":"2024-01-16T14:56:47.310832+0000","last_undegraded":"2024-01-16T14:56:47.310832+0000","last_fullsized":"2024-01-16T14:56:47.310832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308691+0000","last_change":"2024-01-16T14:56:45.984838+0000","last_active":"2024-01-16T14:56:47.308691+0000","last_peered":"2024-01-16T14:56:47.308691+0000","last_clean":"2024-01-16T14:56:47.308691+0000","last_became_active":"2024-01-16T14:56:45.984542+0000","last_became_peered":"2024-01-16T14:56:45.984542+0000","last_unstale":"2024-01-16T14:56:47.308691+0000","last_undegraded":"2024-01-16T14:56:47.308691+0000","last_fullsized":"2024-01-16T14:56:47.308691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312755+0000","last_change":"2024-01-16T14:56:45.989694+0000","last_active":"2024-01-16T14:56:47.312755+0000","last_peered":"2024-01-16T14:56:47.312755+0000","last_clean":"2024-01-16T14:56:47.312755+0000","last_became_active":"2024-01-16T14:56:45.989384+0000","last_became_peered":"2024-01-16T14:56:45.989384+0000","last_unstale":"2024-01-16T14:56:47.312755+0000","last_undegraded":"2024-01-16T14:56:47.312755+0000","last_fullsized":"2024-01-16T14:56:47.312755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951404+0000","last_change":"2024-01-16T14:56:45.988721+0000","last_active":"2024-01-16T14:56:46.951404+0000","last_peered":"2024-01-16T14:56:46.951404+0000","last_clean":"2024-01-16T14:56:46.951404+0000","last_became_active":"2024-01-16T14:56:45.988428+0000","last_became_peered":"2024-01-16T14:56:45.988428+0000","last_unstale":"2024-01-16T14:56:46.951404+0000","last_undegraded":"2024-01-16T14:56:46.951404+0000","last_fullsized":"2024-01-16T14:56:46.951404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309579+0000","last_change":"2024-01-16T14:56:45.986196+0000","last_active":"2024-01-16T14:56:47.309579+0000","last_peered":"2024-01-16T14:56:47.309579+0000","last_clean":"2024-01-16T14:56:47.309579+0000","last_became_active":"2024-01-16T14:56:45.985927+0000","last_became_peered":"2024-01-16T14:56:45.985927+0000","last_unstale":"2024-01-16T14:56:47.309579+0000","last_undegraded":"2024-01-16T14:56:47.309579+0000","last_fullsized":"2024-01-16T14:56:47.309579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311237+0000","last_change":"2024-01-16T14:56:45.985321+0000","last_active":"2024-01-16T14:56:47.311237+0000","last_peered":"2024-01-16T14:56:47.311237+0000","last_clean":"2024-01-16T14:56:47.311237+0000","last_became_active":"2024-01-16T14:56:45.985015+0000","last_became_peered":"2024-01-16T14:56:45.985015+0000","last_unstale":"2024-01-16T14:56:47.311237+0000","last_undegraded":"2024-01-16T14:56:47.311237+0000","last_fullsized":"2024-01-16T14:56:47.311237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311646+0000","last_change":"2024-01-16T14:56:45.986171+0000","last_active":"2024-01-16T14:56:47.311646+0000","last_peered":"2024-01-16T14:56:47.311646+0000","last_clean":"2024-01-16T14:56:47.311646+0000","last_became_active":"2024-01-16T14:56:45.985890+0000","last_became_peered":"2024-01-16T14:56:45.985890+0000","last_unstale":"2024-01-16T14:56:47.311646+0000","last_undegraded":"2024-01-16T14:56:47.311646+0000","last_fullsized":"2024-01-16T14:56:47.311646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312240+0000","last_change":"2024-01-16T14:56:45.979713+0000","last_active":"2024-01-16T14:56:47.312240+0000","last_peered":"2024-01-16T14:56:47.312240+0000","last_clean":"2024-01-16T14:56:47.312240+0000","last_became_active":"2024-01-16T14:56:45.979379+0000","last_became_peered":"2024-01-16T14:56:45.979379+0000","last_unstale":"2024-01-16T14:56:47.312240+0000","last_undegraded":"2024-01-16T14:56:47.312240+0000","last_fullsized":"2024-01-16T14:56:47.312240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311228+0000","last_change":"2024-01-16T14:56:45.988420+0000","last_active":"2024-01-16T14:56:47.311228+0000","last_peered":"2024-01-16T14:56:47.311228+0000","last_clean":"2024-01-16T14:56:47.311228+0000","last_became_active":"2024-01-16T14:56:45.987162+0000","last_became_peered":"2024-01-16T14:56:45.987162+0000","last_unstale":"2024-01-16T14:56:47.311228+0000","last_undegraded":"2024-01-16T14:56:47.311228+0000","last_fullsized":"2024-01-16T14:56:47.311228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952781+0000","last_change":"2024-01-16T14:56:45.977184+0000","last_active":"2024-01-16T14:56:46.952781+0000","last_peered":"2024-01-16T14:56:46.952781+0000","last_clean":"2024-01-16T14:56:46.952781+0000","last_became_active":"2024-01-16T14:56:45.976825+0000","last_became_peered":"2024-01-16T14:56:45.976825+0000","last_unstale":"2024-01-16T14:56:46.952781+0000","last_undegraded":"2024-01-16T14:56:46.952781+0000","last_fullsized":"2024-01-16T14:56:46.952781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:33.458747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012001+0000","last_change":"2024-01-16T14:56:45.008639+0000","last_active":"2024-01-16T14:56:46.012001+0000","last_peered":"2024-01-16T14:56:46.012001+0000","last_clean":"2024-01-16T14:56:46.012001+0000","last_became_active":"2024-01-16T14:56:45.008234+0000","last_became_peered":"2024-01-16T14:56:45.008234+0000","last_unstale":"2024-01-16T14:56:46.012001+0000","last_undegraded":"2024-01-16T14:56:46.012001+0000","last_fullsized":"2024-01-16T14:56:46.012001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311455+0000","last_change":"2024-01-16T14:56:45.976366+0000","last_active":"2024-01-16T14:56:47.311455+0000","last_peered":"2024-01-16T14:56:47.311455+0000","last_clean":"2024-01-16T14:56:47.311455+0000","last_became_active":"2024-01-16T14:56:45.975839+0000","last_became_peered":"2024-01-16T14:56:45.975839+0000","last_unstale":"2024-01-16T14:56:47.311455+0000","last_undegraded":"2024-01-16T14:56:47.311455+0000","last_fullsized":"2024-01-16T14:56:47.311455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010576+0000","last_change":"2024-01-16T14:56:45.003662+0000","last_active":"2024-01-16T14:56:46.010576+0000","last_peered":"2024-01-16T14:56:46.010576+0000","last_clean":"2024-01-16T14:56:46.010576+0000","last_became_active":"2024-01-16T14:56:45.003318+0000","last_became_peered":"2024-01-16T14:56:45.003318+0000","last_unstale":"2024-01-16T14:56:46.010576+0000","last_undegraded":"2024-01-16T14:56:46.010576+0000","last_fullsized":"2024-01-16T14:56:46.010576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.982735+0000","last_change":"2024-01-16T14:56:45.013121+0000","last_active":"2024-01-16T14:56:45.982735+0000","last_peered":"2024-01-16T14:56:45.982735+0000","last_clean":"2024-01-16T14:56:45.982735+0000","last_became_active":"2024-01-16T14:56:45.012797+0000","last_became_peered":"2024-01-16T14:56:45.012797+0000","last_unstale":"2024-01-16T14:56:45.982735+0000","last_undegraded":"2024-01-16T14:56:45.982735+0000","last_fullsized":"2024-01-16T14:56:45.982735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308918+0000","last_change":"2024-01-16T14:56:45.986213+0000","last_active":"2024-01-16T14:56:47.308918+0000","last_peered":"2024-01-16T14:56:47.308918+0000","last_clean":"2024-01-16T14:56:47.308918+0000","last_became_active":"2024-01-16T14:56:45.985824+0000","last_became_peered":"2024-01-16T14:56:45.985824+0000","last_unstale":"2024-01-16T14:56:47.308918+0000","last_undegraded":"2024-01-16T14:56:47.308918+0000","last_fullsized":"2024-01-16T14:56:47.308918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.974634+0000","last_change":"2024-01-16T14:56:45.014687+0000","last_active":"2024-01-16T14:56:45.974634+0000","last_peered":"2024-01-16T14:56:45.974634+0000","last_clean":"2024-01-16T14:56:45.974634+0000","last_became_active":"2024-01-16T14:56:45.014356+0000","last_became_peered":"2024-01-16T14:56:45.014356+0000","last_unstale":"2024-01-16T14:56:45.974634+0000","last_undegraded":"2024-01-16T14:56:45.974634+0000","last_fullsized":"2024-01-16T14:56:45.974634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:34.974850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950693+0000","last_change":"2024-01-16T14:56:45.984146+0000","last_active":"2024-01-16T14:56:46.950693+0000","last_peered":"2024-01-16T14:56:46.950693+0000","last_clean":"2024-01-16T14:56:46.950693+0000","last_became_active":"2024-01-16T14:56:45.983841+0000","last_became_peered":"2024-01-16T14:56:45.983841+0000","last_unstale":"2024-01-16T14:56:46.950693+0000","last_undegraded":"2024-01-16T14:56:46.950693+0000","last_fullsized":"2024-01-16T14:56:46.950693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.971677+0000","last_change":"2024-01-16T14:56:45.011843+0000","last_active":"2024-01-16T14:56:45.971677+0000","last_peered":"2024-01-16T14:56:45.971677+0000","last_clean":"2024-01-16T14:56:45.971677+0000","last_became_active":"2024-01-16T14:56:45.011331+0000","last_became_peered":"2024-01-16T14:56:45.011331+0000","last_unstale":"2024-01-16T14:56:45.971677+0000","last_undegraded":"2024-01-16T14:56:45.971677+0000","last_fullsized":"2024-01-16T14:56:45.971677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313267+0000","last_change":"2024-01-16T14:56:45.986253+0000","last_active":"2024-01-16T14:56:47.313267+0000","last_peered":"2024-01-16T14:56:47.313267+0000","last_clean":"2024-01-16T14:56:47.313267+0000","last_became_active":"2024-01-16T14:56:45.985935+0000","last_became_peered":"2024-01-16T14:56:45.985935+0000","last_unstale":"2024-01-16T14:56:47.313267+0000","last_undegraded":"2024-01-16T14:56:47.313267+0000","last_fullsized":"2024-01-16T14:56:47.313267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.973280+0000","last_change":"2024-01-16T14:56:45.016005+0000","last_active":"2024-01-16T14:56:45.973280+0000","last_peered":"2024-01-16T14:56:45.973280+0000","last_clean":"2024-01-16T14:56:45.973280+0000","last_became_active":"2024-01-16T14:56:45.015680+0000","last_became_peered":"2024-01-16T14:56:45.015680+0000","last_unstale":"2024-01-16T14:56:45.973280+0000","last_undegraded":"2024-01-16T14:56:45.973280+0000","last_fullsized":"2024-01-16T14:56:45.973280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:19.042486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311306+0000","last_change":"2024-01-16T14:56:45.980174+0000","last_active":"2024-01-16T14:56:47.311306+0000","last_peered":"2024-01-16T14:56:47.311306+0000","last_clean":"2024-01-16T14:56:47.311306+0000","last_became_active":"2024-01-16T14:56:45.979743+0000","last_became_peered":"2024-01-16T14:56:45.979743+0000","last_unstale":"2024-01-16T14:56:47.311306+0000","last_undegraded":"2024-01-16T14:56:47.311306+0000","last_fullsized":"2024-01-16T14:56:47.311306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013898+0000","last_change":"2024-01-16T14:56:45.017298+0000","last_active":"2024-01-16T14:56:46.013898+0000","last_peered":"2024-01-16T14:56:46.013898+0000","last_clean":"2024-01-16T14:56:46.013898+0000","last_became_active":"2024-01-16T14:56:45.016921+0000","last_became_peered":"2024-01-16T14:56:45.016921+0000","last_unstale":"2024-01-16T14:56:46.013898+0000","last_undegraded":"2024-01-16T14:56:46.013898+0000","last_fullsized":"2024-01-16T14:56:46.013898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954466+0000","last_change":"2024-01-16T14:56:45.967606+0000","last_active":"2024-01-16T14:56:46.954466+0000","last_peered":"2024-01-16T14:56:46.954466+0000","last_clean":"2024-01-16T14:56:46.954466+0000","last_became_active":"2024-01-16T14:56:45.967036+0000","last_became_peered":"2024-01-16T14:56:45.967036+0000","last_unstale":"2024-01-16T14:56:46.954466+0000","last_undegraded":"2024-01-16T14:56:46.954466+0000","last_fullsized":"2024-01-16T14:56:46.954466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015870+0000","last_change":"2024-01-16T14:56:45.005985+0000","last_active":"2024-01-16T14:56:46.015870+0000","last_peered":"2024-01-16T14:56:46.015870+0000","last_clean":"2024-01-16T14:56:46.015870+0000","last_became_active":"2024-01-16T14:56:45.005530+0000","last_became_peered":"2024-01-16T14:56:45.005530+0000","last_unstale":"2024-01-16T14:56:46.015870+0000","last_undegraded":"2024-01-16T14:56:46.015870+0000","last_fullsized":"2024-01-16T14:56:46.015870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:17.631008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955160+0000","last_change":"2024-01-16T14:56:45.991157+0000","last_active":"2024-01-16T14:56:46.955160+0000","last_peered":"2024-01-16T14:56:46.955160+0000","last_clean":"2024-01-16T14:56:46.955160+0000","last_became_active":"2024-01-16T14:56:45.990848+0000","last_became_peered":"2024-01-16T14:56:45.990848+0000","last_unstale":"2024-01-16T14:56:46.955160+0000","last_undegraded":"2024-01-16T14:56:46.955160+0000","last_fullsized":"2024-01-16T14:56:46.955160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016578+0000","last_change":"2024-01-16T14:56:45.000374+0000","last_active":"2024-01-16T14:56:46.016578+0000","last_peered":"2024-01-16T14:56:46.016578+0000","last_clean":"2024-01-16T14:56:46.016578+0000","last_became_active":"2024-01-16T14:56:45.000028+0000","last_became_peered":"2024-01-16T14:56:45.000028+0000","last_unstale":"2024-01-16T14:56:46.016578+0000","last_undegraded":"2024-01-16T14:56:46.016578+0000","last_fullsized":"2024-01-16T14:56:46.016578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311398+0000","last_change":"2024-01-16T14:56:45.974961+0000","last_active":"2024-01-16T14:56:47.311398+0000","last_peered":"2024-01-16T14:56:47.311398+0000","last_clean":"2024-01-16T14:56:47.311398+0000","last_became_active":"2024-01-16T14:56:45.974581+0000","last_became_peered":"2024-01-16T14:56:45.974581+0000","last_unstale":"2024-01-16T14:56:47.311398+0000","last_undegraded":"2024-01-16T14:56:47.311398+0000","last_fullsized":"2024-01-16T14:56:47.311398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.972980+0000","last_change":"2024-01-16T14:56:45.017987+0000","last_active":"2024-01-16T14:56:45.972980+0000","last_peered":"2024-01-16T14:56:45.972980+0000","last_clean":"2024-01-16T14:56:45.972980+0000","last_became_active":"2024-01-16T14:56:45.017656+0000","last_became_peered":"2024-01-16T14:56:45.017656+0000","last_unstale":"2024-01-16T14:56:45.972980+0000","last_undegraded":"2024-01-16T14:56:45.972980+0000","last_fullsized":"2024-01-16T14:56:45.972980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954932+0000","last_change":"2024-01-16T14:56:45.976869+0000","last_active":"2024-01-16T14:56:46.954932+0000","last_peered":"2024-01-16T14:56:46.954932+0000","last_clean":"2024-01-16T14:56:46.954932+0000","last_became_active":"2024-01-16T14:56:45.976500+0000","last_became_peered":"2024-01-16T14:56:45.976500+0000","last_unstale":"2024-01-16T14:56:46.954932+0000","last_undegraded":"2024-01-16T14:56:46.954932+0000","last_fullsized":"2024-01-16T14:56:46.954932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.966318+0000","last_change":"2024-01-16T14:56:45.012069+0000","last_active":"2024-01-16T14:56:45.966318+0000","last_peered":"2024-01-16T14:56:45.966318+0000","last_clean":"2024-01-16T14:56:45.966318+0000","last_became_active":"2024-01-16T14:56:45.011711+0000","last_became_peered":"2024-01-16T14:56:45.011711+0000","last_unstale":"2024-01-16T14:56:45.966318+0000","last_undegraded":"2024-01-16T14:56:45.966318+0000","last_fullsized":"2024-01-16T14:56:45.966318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311361+0000","last_change":"2024-01-16T14:56:45.988281+0000","last_active":"2024-01-16T14:56:47.311361+0000","last_peered":"2024-01-16T14:56:47.311361+0000","last_clean":"2024-01-16T14:56:47.311361+0000","last_became_active":"2024-01-16T14:56:45.986521+0000","last_became_peered":"2024-01-16T14:56:45.986521+0000","last_unstale":"2024-01-16T14:56:47.311361+0000","last_undegraded":"2024-01-16T14:56:47.311361+0000","last_fullsized":"2024-01-16T14:56:47.311361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015745+0000","last_change":"2024-01-16T14:56:44.993188+0000","last_active":"2024-01-16T14:56:46.015745+0000","last_peered":"2024-01-16T14:56:46.015745+0000","last_clean":"2024-01-16T14:56:46.015745+0000","last_became_active":"2024-01-16T14:56:44.992904+0000","last_became_peered":"2024-01-16T14:56:44.992904+0000","last_unstale":"2024-01-16T14:56:46.015745+0000","last_undegraded":"2024-01-16T14:56:46.015745+0000","last_fullsized":"2024-01-16T14:56:46.015745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.524009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954660+0000","last_change":"2024-01-16T14:56:45.981562+0000","last_active":"2024-01-16T14:56:46.954660+0000","last_peered":"2024-01-16T14:56:46.954660+0000","last_clean":"2024-01-16T14:56:46.954660+0000","last_became_active":"2024-01-16T14:56:45.981272+0000","last_became_peered":"2024-01-16T14:56:45.981272+0000","last_unstale":"2024-01-16T14:56:46.954660+0000","last_undegraded":"2024-01-16T14:56:46.954660+0000","last_fullsized":"2024-01-16T14:56:46.954660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013126+0000","last_change":"2024-01-16T14:56:45.005288+0000","last_active":"2024-01-16T14:56:46.013126+0000","last_peered":"2024-01-16T14:56:46.013126+0000","last_clean":"2024-01-16T14:56:46.013126+0000","last_became_active":"2024-01-16T14:56:45.004988+0000","last_became_peered":"2024-01-16T14:56:45.004988+0000","last_unstale":"2024-01-16T14:56:46.013126+0000","last_undegraded":"2024-01-16T14:56:46.013126+0000","last_fullsized":"2024-01-16T14:56:46.013126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955886+0000","last_change":"2024-01-16T14:56:45.977479+0000","last_active":"2024-01-16T14:56:46.955886+0000","last_peered":"2024-01-16T14:56:46.955886+0000","last_clean":"2024-01-16T14:56:46.955886+0000","last_became_active":"2024-01-16T14:56:45.976447+0000","last_became_peered":"2024-01-16T14:56:45.976447+0000","last_unstale":"2024-01-16T14:56:46.955886+0000","last_undegraded":"2024-01-16T14:56:46.955886+0000","last_fullsized":"2024-01-16T14:56:46.955886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:09:28.671518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012387+0000","last_change":"2024-01-16T14:56:44.991027+0000","last_active":"2024-01-16T14:56:46.012387+0000","last_peered":"2024-01-16T14:56:46.012387+0000","last_clean":"2024-01-16T14:56:46.012387+0000","last_became_active":"2024-01-16T14:56:44.990777+0000","last_became_peered":"2024-01-16T14:56:44.990777+0000","last_unstale":"2024-01-16T14:56:46.012387+0000","last_undegraded":"2024-01-16T14:56:46.012387+0000","last_fullsized":"2024-01-16T14:56:46.012387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956303+0000","last_change":"2024-01-16T14:56:45.980059+0000","last_active":"2024-01-16T14:56:46.956303+0000","last_peered":"2024-01-16T14:56:46.956303+0000","last_clean":"2024-01-16T14:56:46.956303+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956303+0000","last_undegraded":"2024-01-16T14:56:46.956303+0000","last_fullsized":"2024-01-16T14:56:46.956303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:40.703250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010244+0000","last_change":"2024-01-16T14:56:45.001611+0000","last_active":"2024-01-16T14:56:46.010244+0000","last_peered":"2024-01-16T14:56:46.010244+0000","last_clean":"2024-01-16T14:56:46.010244+0000","last_became_active":"2024-01-16T14:56:45.001338+0000","last_became_peered":"2024-01-16T14:56:45.001338+0000","last_unstale":"2024-01-16T14:56:46.010244+0000","last_undegraded":"2024-01-16T14:56:46.010244+0000","last_fullsized":"2024-01-16T14:56:46.010244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312030+0000","last_change":"2024-01-16T14:56:45.986001+0000","last_active":"2024-01-16T14:56:47.312030+0000","last_peered":"2024-01-16T14:56:47.312030+0000","last_clean":"2024-01-16T14:56:47.312030+0000","last_became_active":"2024-01-16T14:56:45.984074+0000","last_became_peered":"2024-01-16T14:56:45.984074+0000","last_unstale":"2024-01-16T14:56:47.312030+0000","last_undegraded":"2024-01-16T14:56:47.312030+0000","last_fullsized":"2024-01-16T14:56:47.312030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.960950+0000","last_change":"2024-01-16T14:56:45.009683+0000","last_active":"2024-01-16T14:56:45.960950+0000","last_peered":"2024-01-16T14:56:45.960950+0000","last_clean":"2024-01-16T14:56:45.960950+0000","last_became_active":"2024-01-16T14:56:45.009325+0000","last_became_peered":"2024-01-16T14:56:45.009325+0000","last_unstale":"2024-01-16T14:56:45.960950+0000","last_undegraded":"2024-01-16T14:56:45.960950+0000","last_fullsized":"2024-01-16T14:56:45.960950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:27.159214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955489+0000","last_change":"2024-01-16T14:56:45.989743+0000","last_active":"2024-01-16T14:56:46.955489+0000","last_peered":"2024-01-16T14:56:46.955489+0000","last_clean":"2024-01-16T14:56:46.955489+0000","last_became_active":"2024-01-16T14:56:45.989084+0000","last_became_peered":"2024-01-16T14:56:45.989084+0000","last_unstale":"2024-01-16T14:56:46.955489+0000","last_undegraded":"2024-01-16T14:56:46.955489+0000","last_fullsized":"2024-01-16T14:56:46.955489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.955711+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:56:45.955711+0000","last_peered":"2024-01-16T14:56:45.955711+0000","last_clean":"2024-01-16T14:56:45.955711+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:56:45.955711+0000","last_undegraded":"2024-01-16T14:56:45.955711+0000","last_fullsized":"2024-01-16T14:56:45.955711+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963629+0000","last_change":"2024-01-16T14:56:45.009432+0000","last_active":"2024-01-16T14:56:45.963629+0000","last_peered":"2024-01-16T14:56:45.963629+0000","last_clean":"2024-01-16T14:56:45.963629+0000","last_became_active":"2024-01-16T14:56:45.008692+0000","last_became_peered":"2024-01-16T14:56:45.008692+0000","last_unstale":"2024-01-16T14:56:45.963629+0000","last_undegraded":"2024-01-16T14:56:45.963629+0000","last_fullsized":"2024-01-16T14:56:45.963629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308881+0000","last_change":"2024-01-16T14:56:45.990578+0000","last_active":"2024-01-16T14:56:47.308881+0000","last_peered":"2024-01-16T14:56:47.308881+0000","last_clean":"2024-01-16T14:56:47.308881+0000","last_became_active":"2024-01-16T14:56:45.990211+0000","last_became_peered":"2024-01-16T14:56:45.990211+0000","last_unstale":"2024-01-16T14:56:47.308881+0000","last_undegraded":"2024-01-16T14:56:47.308881+0000","last_fullsized":"2024-01-16T14:56:47.308881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.956468+0000","last_change":"2024-01-16T14:56:45.006632+0000","last_active":"2024-01-16T14:56:45.956468+0000","last_peered":"2024-01-16T14:56:45.956468+0000","last_clean":"2024-01-16T14:56:45.956468+0000","last_became_active":"2024-01-16T14:56:45.006340+0000","last_became_peered":"2024-01-16T14:56:45.006340+0000","last_unstale":"2024-01-16T14:56:45.956468+0000","last_undegraded":"2024-01-16T14:56:45.956468+0000","last_fullsized":"2024-01-16T14:56:45.956468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:41.754460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310892+0000","last_change":"2024-01-16T14:56:45.977532+0000","last_active":"2024-01-16T14:56:47.310892+0000","last_peered":"2024-01-16T14:56:47.310892+0000","last_clean":"2024-01-16T14:56:47.310892+0000","last_became_active":"2024-01-16T14:56:45.975102+0000","last_became_peered":"2024-01-16T14:56:45.975102+0000","last_unstale":"2024-01-16T14:56:47.310892+0000","last_undegraded":"2024-01-16T14:56:47.310892+0000","last_fullsized":"2024-01-16T14:56:47.310892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:18.643652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963071+0000","last_change":"2024-01-16T14:56:44.994705+0000","last_active":"2024-01-16T14:56:45.963071+0000","last_peered":"2024-01-16T14:56:45.963071+0000","last_clean":"2024-01-16T14:56:45.963071+0000","last_became_active":"2024-01-16T14:56:44.994387+0000","last_became_peered":"2024-01-16T14:56:44.994387+0000","last_unstale":"2024-01-16T14:56:45.963071+0000","last_undegraded":"2024-01-16T14:56:45.963071+0000","last_fullsized":"2024-01-16T14:56:45.963071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950691+0000","last_change":"2024-01-16T14:56:45.978166+0000","last_active":"2024-01-16T14:56:46.950691+0000","last_peered":"2024-01-16T14:56:46.950691+0000","last_clean":"2024-01-16T14:56:46.950691+0000","last_became_active":"2024-01-16T14:56:45.974628+0000","last_became_peered":"2024-01-16T14:56:45.974628+0000","last_unstale":"2024-01-16T14:56:46.950691+0000","last_undegraded":"2024-01-16T14:56:46.950691+0000","last_fullsized":"2024-01-16T14:56:46.950691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.980309+0000","last_change":"2024-01-16T14:56:45.017774+0000","last_active":"2024-01-16T14:56:45.980309+0000","last_peered":"2024-01-16T14:56:45.980309+0000","last_clean":"2024-01-16T14:56:45.980309+0000","last_became_active":"2024-01-16T14:56:45.017410+0000","last_became_peered":"2024-01-16T14:56:45.017410+0000","last_unstale":"2024-01-16T14:56:45.980309+0000","last_undegraded":"2024-01-16T14:56:45.980309+0000","last_fullsized":"2024-01-16T14:56:45.980309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:59.852603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954411+0000","last_change":"2024-01-16T14:56:45.970975+0000","last_active":"2024-01-16T14:56:46.954411+0000","last_peered":"2024-01-16T14:56:46.954411+0000","last_clean":"2024-01-16T14:56:46.954411+0000","last_became_active":"2024-01-16T14:56:45.970566+0000","last_became_peered":"2024-01-16T14:56:45.970566+0000","last_unstale":"2024-01-16T14:56:46.954411+0000","last_undegraded":"2024-01-16T14:56:46.954411+0000","last_fullsized":"2024-01-16T14:56:46.954411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.967326+0000","last_change":"2024-01-16T14:56:45.004864+0000","last_active":"2024-01-16T14:56:45.967326+0000","last_peered":"2024-01-16T14:56:45.967326+0000","last_clean":"2024-01-16T14:56:45.967326+0000","last_became_active":"2024-01-16T14:56:45.004057+0000","last_became_peered":"2024-01-16T14:56:45.004057+0000","last_unstale":"2024-01-16T14:56:45.967326+0000","last_undegraded":"2024-01-16T14:56:45.967326+0000","last_fullsized":"2024-01-16T14:56:45.967326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313581+0000","last_change":"2024-01-16T14:56:45.989780+0000","last_active":"2024-01-16T14:56:47.313581+0000","last_peered":"2024-01-16T14:56:47.313581+0000","last_clean":"2024-01-16T14:56:47.313581+0000","last_became_active":"2024-01-16T14:56:45.989519+0000","last_became_peered":"2024-01-16T14:56:45.989519+0000","last_unstale":"2024-01-16T14:56:47.313581+0000","last_undegraded":"2024-01-16T14:56:47.313581+0000","last_fullsized":"2024-01-16T14:56:47.313581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.959086+0000","last_change":"2024-01-16T14:56:45.009420+0000","last_active":"2024-01-16T14:56:45.959086+0000","last_peered":"2024-01-16T14:56:45.959086+0000","last_clean":"2024-01-16T14:56:45.959086+0000","last_became_active":"2024-01-16T14:56:45.009103+0000","last_became_peered":"2024-01-16T14:56:45.009103+0000","last_unstale":"2024-01-16T14:56:45.959086+0000","last_undegraded":"2024-01-16T14:56:45.959086+0000","last_fullsized":"2024-01-16T14:56:45.959086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:49.027396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951463+0000","last_change":"2024-01-16T14:56:45.976716+0000","last_active":"2024-01-16T14:56:46.951463+0000","last_peered":"2024-01-16T14:56:46.951463+0000","last_clean":"2024-01-16T14:56:46.951463+0000","last_became_active":"2024-01-16T14:56:45.976307+0000","last_became_peered":"2024-01-16T14:56:45.976307+0000","last_unstale":"2024-01-16T14:56:46.951463+0000","last_undegraded":"2024-01-16T14:56:46.951463+0000","last_fullsized":"2024-01-16T14:56:46.951463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.964691+0000","last_change":"2024-01-16T14:56:45.013169+0000","last_active":"2024-01-16T14:56:45.964691+0000","last_peered":"2024-01-16T14:56:45.964691+0000","last_clean":"2024-01-16T14:56:45.964691+0000","last_became_active":"2024-01-16T14:56:45.012872+0000","last_became_peered":"2024-01-16T14:56:45.012872+0000","last_unstale":"2024-01-16T14:56:45.964691+0000","last_undegraded":"2024-01-16T14:56:45.964691+0000","last_fullsized":"2024-01-16T14:56:45.964691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308843+0000","last_change":"2024-01-16T14:56:45.987343+0000","last_active":"2024-01-16T14:56:47.308843+0000","last_peered":"2024-01-16T14:56:47.308843+0000","last_clean":"2024-01-16T14:56:47.308843+0000","last_became_active":"2024-01-16T14:56:45.987065+0000","last_became_peered":"2024-01-16T14:56:45.987065+0000","last_unstale":"2024-01-16T14:56:47.308843+0000","last_undegraded":"2024-01-16T14:56:47.308843+0000","last_fullsized":"2024-01-16T14:56:47.308843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014224+0000","last_change":"2024-01-16T14:56:45.004437+0000","last_active":"2024-01-16T14:56:46.014224+0000","last_peered":"2024-01-16T14:56:46.014224+0000","last_clean":"2024-01-16T14:56:46.014224+0000","last_became_active":"2024-01-16T14:56:45.004165+0000","last_became_peered":"2024-01-16T14:56:45.004165+0000","last_unstale":"2024-01-16T14:56:46.014224+0000","last_undegraded":"2024-01-16T14:56:46.014224+0000","last_fullsized":"2024-01-16T14:56:46.014224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311396+0000","last_change":"2024-01-16T14:56:45.974262+0000","last_active":"2024-01-16T14:56:47.311396+0000","last_peered":"2024-01-16T14:56:47.311396+0000","last_clean":"2024-01-16T14:56:47.311396+0000","last_became_active":"2024-01-16T14:56:45.970037+0000","last_became_peered":"2024-01-16T14:56:45.970037+0000","last_unstale":"2024-01-16T14:56:47.311396+0000","last_undegraded":"2024-01-16T14:56:47.311396+0000","last_fullsized":"2024-01-16T14:56:47.311396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.952176+0000","last_change":"2024-01-16T14:56:45.009771+0000","last_active":"2024-01-16T14:56:45.952176+0000","last_peered":"2024-01-16T14:56:45.952176+0000","last_clean":"2024-01-16T14:56:45.952176+0000","last_became_active":"2024-01-16T14:56:45.009505+0000","last_became_peered":"2024-01-16T14:56:45.009505+0000","last_unstale":"2024-01-16T14:56:45.952176+0000","last_undegraded":"2024-01-16T14:56:45.952176+0000","last_fullsized":"2024-01-16T14:56:45.952176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952907+0000","last_change":"2024-01-16T14:56:45.970586+0000","last_active":"2024-01-16T14:56:46.952907+0000","last_peered":"2024-01-16T14:56:46.952907+0000","last_clean":"2024-01-16T14:56:46.952907+0000","last_became_active":"2024-01-16T14:56:45.970146+0000","last_became_peered":"2024-01-16T14:56:45.970146+0000","last_unstale":"2024-01-16T14:56:46.952907+0000","last_undegraded":"2024-01-16T14:56:46.952907+0000","last_fullsized":"2024-01-16T14:56:46.952907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:23.365045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.962308+0000","last_change":"2024-01-16T14:56:45.003139+0000","last_active":"2024-01-16T14:56:45.962308+0000","last_peered":"2024-01-16T14:56:45.962308+0000","last_clean":"2024-01-16T14:56:45.962308+0000","last_became_active":"2024-01-16T14:56:45.002787+0000","last_became_peered":"2024-01-16T14:56:45.002787+0000","last_unstale":"2024-01-16T14:56:45.962308+0000","last_undegraded":"2024-01-16T14:56:45.962308+0000","last_fullsized":"2024-01-16T14:56:45.962308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309294+0000","last_change":"2024-01-16T14:56:45.981803+0000","last_active":"2024-01-16T14:56:47.309294+0000","last_peered":"2024-01-16T14:56:47.309294+0000","last_clean":"2024-01-16T14:56:47.309294+0000","last_became_active":"2024-01-16T14:56:45.981480+0000","last_became_peered":"2024-01-16T14:56:45.981480+0000","last_unstale":"2024-01-16T14:56:47.309294+0000","last_undegraded":"2024-01-16T14:56:47.309294+0000","last_fullsized":"2024-01-16T14:56:47.309294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.968506+0000","last_change":"2024-01-16T14:56:45.006149+0000","last_active":"2024-01-16T14:56:45.968506+0000","last_peered":"2024-01-16T14:56:45.968506+0000","last_clean":"2024-01-16T14:56:45.968506+0000","last_became_active":"2024-01-16T14:56:45.005789+0000","last_became_peered":"2024-01-16T14:56:45.005789+0000","last_unstale":"2024-01-16T14:56:45.968506+0000","last_undegraded":"2024-01-16T14:56:45.968506+0000","last_fullsized":"2024-01-16T14:56:45.968506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309478+0000","last_change":"2024-01-16T14:56:45.985569+0000","last_active":"2024-01-16T14:56:47.309478+0000","last_peered":"2024-01-16T14:56:47.309478+0000","last_clean":"2024-01-16T14:56:47.309478+0000","last_became_active":"2024-01-16T14:56:45.983774+0000","last_became_peered":"2024-01-16T14:56:45.983774+0000","last_unstale":"2024-01-16T14:56:47.309478+0000","last_undegraded":"2024-01-16T14:56:47.309478+0000","last_fullsized":"2024-01-16T14:56:47.309478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013631+0000","last_change":"2024-01-16T14:56:45.010441+0000","last_active":"2024-01-16T14:56:46.013631+0000","last_peered":"2024-01-16T14:56:46.013631+0000","last_clean":"2024-01-16T14:56:46.013631+0000","last_became_active":"2024-01-16T14:56:45.007667+0000","last_became_peered":"2024-01-16T14:56:45.007667+0000","last_unstale":"2024-01-16T14:56:46.013631+0000","last_undegraded":"2024-01-16T14:56:46.013631+0000","last_fullsized":"2024-01-16T14:56:46.013631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:38.016001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311624+0000","last_change":"2024-01-16T14:56:45.984334+0000","last_active":"2024-01-16T14:56:47.311624+0000","last_peered":"2024-01-16T14:56:47.311624+0000","last_clean":"2024-01-16T14:56:47.311624+0000","last_became_active":"2024-01-16T14:56:45.984013+0000","last_became_peered":"2024-01-16T14:56:45.984013+0000","last_unstale":"2024-01-16T14:56:47.311624+0000","last_undegraded":"2024-01-16T14:56:47.311624+0000","last_fullsized":"2024-01-16T14:56:47.311624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313406+0000","last_change":"2024-01-16T14:56:45.985148+0000","last_active":"2024-01-16T14:56:47.313406+0000","last_peered":"2024-01-16T14:56:47.313406+0000","last_clean":"2024-01-16T14:56:47.313406+0000","last_became_active":"2024-01-16T14:56:45.983354+0000","last_became_peered":"2024-01-16T14:56:45.983354+0000","last_unstale":"2024-01-16T14:56:47.313406+0000","last_undegraded":"2024-01-16T14:56:47.313406+0000","last_fullsized":"2024-01-16T14:56:47.313406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011363+0000","last_change":"2024-01-16T14:56:45.017135+0000","last_active":"2024-01-16T14:56:46.011363+0000","last_peered":"2024-01-16T14:56:46.011363+0000","last_clean":"2024-01-16T14:56:46.011363+0000","last_became_active":"2024-01-16T14:56:45.016686+0000","last_became_peered":"2024-01-16T14:56:45.016686+0000","last_unstale":"2024-01-16T14:56:46.011363+0000","last_undegraded":"2024-01-16T14:56:46.011363+0000","last_fullsized":"2024-01-16T14:56:46.011363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311883+0000","last_change":"2024-01-16T14:56:45.983050+0000","last_active":"2024-01-16T14:56:47.311883+0000","last_peered":"2024-01-16T14:56:47.311883+0000","last_clean":"2024-01-16T14:56:47.311883+0000","last_became_active":"2024-01-16T14:56:45.982678+0000","last_became_peered":"2024-01-16T14:56:45.982678+0000","last_unstale":"2024-01-16T14:56:47.311883+0000","last_undegraded":"2024-01-16T14:56:47.311883+0000","last_fullsized":"2024-01-16T14:56:47.311883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011339+0000","last_change":"2024-01-16T14:56:45.014696+0000","last_active":"2024-01-16T14:56:46.011339+0000","last_peered":"2024-01-16T14:56:46.011339+0000","last_clean":"2024-01-16T14:56:46.011339+0000","last_became_active":"2024-01-16T14:56:45.014410+0000","last_became_peered":"2024-01-16T14:56:45.014410+0000","last_unstale":"2024-01-16T14:56:46.011339+0000","last_undegraded":"2024-01-16T14:56:46.011339+0000","last_fullsized":"2024-01-16T14:56:46.011339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311230+0000","last_change":"2024-01-16T14:56:45.983804+0000","last_active":"2024-01-16T14:56:47.311230+0000","last_peered":"2024-01-16T14:56:47.311230+0000","last_clean":"2024-01-16T14:56:47.311230+0000","last_became_active":"2024-01-16T14:56:45.983529+0000","last_became_peered":"2024-01-16T14:56:45.983529+0000","last_unstale":"2024-01-16T14:56:47.311230+0000","last_undegraded":"2024-01-16T14:56:47.311230+0000","last_fullsized":"2024-01-16T14:56:47.311230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014146+0000","last_change":"2024-01-16T14:56:45.005514+0000","last_active":"2024-01-16T14:56:46.014146+0000","last_peered":"2024-01-16T14:56:46.014146+0000","last_clean":"2024-01-16T14:56:46.014146+0000","last_became_active":"2024-01-16T14:56:45.004610+0000","last_became_peered":"2024-01-16T14:56:45.004610+0000","last_unstale":"2024-01-16T14:56:46.014146+0000","last_undegraded":"2024-01-16T14:56:46.014146+0000","last_fullsized":"2024-01-16T14:56:46.014146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308980+0000","last_change":"2024-01-16T14:56:45.992067+0000","last_active":"2024-01-16T14:56:47.308980+0000","last_peered":"2024-01-16T14:56:47.308980+0000","last_clean":"2024-01-16T14:56:47.308980+0000","last_became_active":"2024-01-16T14:56:45.991826+0000","last_became_peered":"2024-01-16T14:56:45.991826+0000","last_unstale":"2024-01-16T14:56:47.308980+0000","last_undegraded":"2024-01-16T14:56:47.308980+0000","last_fullsized":"2024-01-16T14:56:47.308980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016889+0000","last_change":"2024-01-16T14:56:45.003619+0000","last_active":"2024-01-16T14:56:46.016889+0000","last_peered":"2024-01-16T14:56:46.016889+0000","last_clean":"2024-01-16T14:56:46.016889+0000","last_became_active":"2024-01-16T14:56:45.002981+0000","last_became_peered":"2024-01-16T14:56:45.002981+0000","last_unstale":"2024-01-16T14:56:46.016889+0000","last_undegraded":"2024-01-16T14:56:46.016889+0000","last_fullsized":"2024-01-16T14:56:46.016889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":660897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0910000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1389999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:56:50.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:56:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.654+0000 7fbd43800640 1 Processor -- start 2024-01-16T14:56:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.654+0000 7fbd43800640 1 -- start start 2024-01-16T14:56:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.655+0000 7fbd43800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 0x7fbd3c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.655+0000 7fbd43800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbd3c13e360 0x7fbd3c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.655+0000 7fbd43800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd3c06c470 con 0x7fbd3c15f9c0 2024-01-16T14:56:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.655+0000 7fbd43800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3c06c750 con 0x7fbd3c13e360 2024-01-16T14:56:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.655+0000 7fbd43800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c06ca30 con 0x7fbd3c13df80 2024-01-16T14:56:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.656+0000 7fbd41800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fbd3c15f9c0 0x7fbd3c1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55208/0 (socket says 172.21.15.125:55208) 2024-01-16T14:56:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.656+0000 7fbd41800640 1 -- 172.21.15.125:0/4178574341 learned_addr learned my addr 172.21.15.125:0/4178574341 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.656+0000 7fbd40e00640 1 --2- 172.21.15.125:0/4178574341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 0x7fbd3c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.656+0000 7fbd3be00640 1 --2- 172.21.15.125:0/4178574341 >> v2:172.21.15.137:3300/0 conn(0x7fbd3c13e360 0x7fbd3c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.657+0000 7fbd40e00640 1 -- 172.21.15.125:0/4178574341 >> v2:172.21.15.137:3300/0 conn(0x7fbd3c13e360 msgr2=0x7fbd3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.657+0000 7fbd40e00640 1 --2- 172.21.15.125:0/4178574341 >> v2:172.21.15.137:3300/0 conn(0x7fbd3c13e360 0x7fbd3c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.657+0000 7fbd40e00640 1 -- 172.21.15.125:0/4178574341 >> v1:172.21.15.137:6789/0 conn(0x7fbd3c15f9c0 legacy=0x7fbd3c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.657+0000 7fbd40e00640 1 -- 172.21.15.125:0/4178574341 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c06cd10 con 0x7fbd3c13df80 2024-01-16T14:56:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.658+0000 7fbd40e00640 1 --2- 172.21.15.125:0/4178574341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 0x7fbd3c15bee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0097a0 tx=0x7fbd2c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ed53f5f6949e51 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.659+0000 7fbd3b400640 1 -- 172.21.15.125:0/4178574341 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c031830 con 0x7fbd3c13df80 2024-01-16T14:56:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.659+0000 7fbd3b400640 1 -- 172.21.15.125:0/4178574341 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c031990 con 0x7fbd3c13df80 2024-01-16T14:56:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.659+0000 7fbd3b400640 1 -- 172.21.15.125:0/4178574341 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c031d20 con 0x7fbd3c13df80 2024-01-16T14:56:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.659+0000 7fbd43800640 1 -- 172.21.15.125:0/4178574341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 msgr2=0x7fbd3c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.659+0000 7fbd43800640 1 --2- 172.21.15.125:0/4178574341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 0x7fbd3c15bee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0097a0 tx=0x7fbd2c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 -- 172.21.15.125:0/4178574341 shutdown_connections 2024-01-16T14:56:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 --2- 172.21.15.125:0/4178574341 >> v2:172.21.15.137:3300/0 conn(0x7fbd3c13e360 0x7fbd3c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 --2- 172.21.15.125:0/4178574341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c13df80 0x7fbd3c15bee0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 -- 172.21.15.125:0/4178574341 >> 172.21.15.125:0/4178574341 conn(0x7fbd3c06a800 msgr2=0x7fbd3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 -- 172.21.15.125:0/4178574341 shutdown_connections 2024-01-16T14:56:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.660+0000 7fbd43800640 1 -- 172.21.15.125:0/4178574341 wait complete. 2024-01-16T14:56:50.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.662+0000 7fbd43800640 1 Processor -- start 2024-01-16T14:56:50.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.662+0000 7fbd43800640 1 -- start start 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 0x7fbd3c14eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c15f9c0 0x7fbd3c14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c14f930 con 0x7fbd3c13df80 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c14fba0 con 0x7fbd3c13e360 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd43800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c14fe80 con 0x7fbd3c15f9c0 2024-01-16T14:56:50.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd40e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd40e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47668/0 (socket says 172.21.15.125:47668) 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.663+0000 7fbd40e00640 1 -- 172.21.15.125:0/248047112 learned_addr learned my addr 172.21.15.125:0/248047112 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd41800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c15f9c0 0x7fbd3c14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd3be00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 0x7fbd3c14eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 -- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c15f9c0 msgr2=0x7fbd3c14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c15f9c0 0x7fbd3c14f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 -- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 msgr2=0x7fbd3c14eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 0x7fbd3c14eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 -- 172.21.15.125:0/248047112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c009400 con 0x7fbd3c13df80 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd3be00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 0x7fbd3c14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd40e00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c034040 tx=0x7fbd2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c03a8f0 con 0x7fbd3c13df80 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c03aa70 con 0x7fbd3c13df80 2024-01-16T14:56:50.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c03ae20 con 0x7fbd3c13df80 2024-01-16T14:56:50.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.664+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3c150160 con 0x7fbd3c13df80 2024-01-16T14:56:50.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.665+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3c150720 con 0x7fbd3c13df80 2024-01-16T14:56:50.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.666+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3c14ffc0 con 0x7fbd3c13df80 2024-01-16T14:56:50.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.666+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbd2c04d440 con 0x7fbd3c13df80 2024-01-16T14:56:50.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.667+0000 7fbd38c00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 0x7fbd1406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.667+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd2c0abd30 con 0x7fbd3c13df80 2024-01-16T14:56:50.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.667+0000 7fbd3be00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 0x7fbd1406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.668+0000 7fbd3be00640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 0x7fbd1406d780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c15bb40 tx=0x7fbd24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.673+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd3c14ffc0 con 0x7fbd3c13df80 2024-01-16T14:56:50.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.859+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd3c0ce850 con 0x7fbd1406b2c0 2024-01-16T14:56:50.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.879+0000 7fbd38c00640 1 -- 172.21.15.125:0/248047112 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264507 (secure 0 0 0) 0x7fbd24004030 con 0x7fbd1406b2c0 2024-01-16T14:56:50.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:50.866 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.885+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 msgr2=0x7fbd1406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.885+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 0x7fbd1406d780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c15bb40 tx=0x7fbd24008040 comp rx=0 tx=0).stop 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.885+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 msgr2=0x7fbd3c1f4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.885+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c034040 tx=0x7fbd2c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 shutdown_connections 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd3c15f9c0 0x7fbd3c14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd3c13e360 0x7fbd3c14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd1406b2c0 0x7fbd1406d780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 --2- 172.21.15.125:0/248047112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd3c13df80 0x7fbd3c1f4660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 >> 172.21.15.125:0/248047112 conn(0x7fbd3c06a800 msgr2=0x7fbd3c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 shutdown_connections 2024-01-16T14:56:50.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:50.886+0000 7fbd43800640 1 -- 172.21.15.125:0/248047112 wait complete. 2024-01-16T14:56:50.887 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:56:49.918706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2887680,"data_stored":1751817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958667"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956233+0000","last_change":"2024-01-16T14:56:45.980276+0000","last_active":"2024-01-16T14:56:46.956233+0000","last_peered":"2024-01-16T14:56:46.956233+0000","last_clean":"2024-01-16T14:56:46.956233+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956233+0000","last_undegraded":"2024-01-16T14:56:46.956233+0000","last_fullsized":"2024-01-16T14:56:46.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:53.535044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951915+0000","last_change":"2024-01-16T14:56:45.975609+0000","last_active":"2024-01-16T14:56:46.951915+0000","last_peered":"2024-01-16T14:56:46.951915+0000","last_clean":"2024-01-16T14:56:46.951915+0000","last_became_active":"2024-01-16T14:56:45.975291+0000","last_became_peered":"2024-01-16T14:56:45.975291+0000","last_unstale":"2024-01-16T14:56:46.951915+0000","last_undegraded":"2024-01-16T14:56:46.951915+0000","last_fullsized":"2024-01-16T14:56:46.951915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310992+0000","last_change":"2024-01-16T14:56:45.977545+0000","last_active":"2024-01-16T14:56:47.310992+0000","last_peered":"2024-01-16T14:56:47.310992+0000","last_clean":"2024-01-16T14:56:47.310992+0000","last_became_active":"2024-01-16T14:56:45.977253+0000","last_became_peered":"2024-01-16T14:56:45.977253+0000","last_unstale":"2024-01-16T14:56:47.310992+0000","last_undegraded":"2024-01-16T14:56:47.310992+0000","last_fullsized":"2024-01-16T14:56:47.310992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:34.727060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311691+0000","last_change":"2024-01-16T14:56:45.985746+0000","last_active":"2024-01-16T14:56:47.311691+0000","last_peered":"2024-01-16T14:56:47.311691+0000","last_clean":"2024-01-16T14:56:47.311691+0000","last_became_active":"2024-01-16T14:56:45.984925+0000","last_became_peered":"2024-01-16T14:56:45.984925+0000","last_unstale":"2024-01-16T14:56:47.311691+0000","last_undegraded":"2024-01-16T14:56:47.311691+0000","last_fullsized":"2024-01-16T14:56:47.311691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954569+0000","last_change":"2024-01-16T14:56:45.973905+0000","last_active":"2024-01-16T14:56:46.954569+0000","last_peered":"2024-01-16T14:56:46.954569+0000","last_clean":"2024-01-16T14:56:46.954569+0000","last_became_active":"2024-01-16T14:56:45.973591+0000","last_became_peered":"2024-01-16T14:56:45.973591+0000","last_unstale":"2024-01-16T14:56:46.954569+0000","last_undegraded":"2024-01-16T14:56:46.954569+0000","last_fullsized":"2024-01-16T14:56:46.954569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310746+0000","last_change":"2024-01-16T14:56:45.981879+0000","last_active":"2024-01-16T14:56:47.310746+0000","last_peered":"2024-01-16T14:56:47.310746+0000","last_clean":"2024-01-16T14:56:47.310746+0000","last_became_active":"2024-01-16T14:56:45.981506+0000","last_became_peered":"2024-01-16T14:56:45.981506+0000","last_unstale":"2024-01-16T14:56:47.310746+0000","last_undegraded":"2024-01-16T14:56:47.310746+0000","last_fullsized":"2024-01-16T14:56:47.310746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954278+0000","last_change":"2024-01-16T14:56:45.978039+0000","last_active":"2024-01-16T14:56:46.954278+0000","last_peered":"2024-01-16T14:56:46.954278+0000","last_clean":"2024-01-16T14:56:46.954278+0000","last_became_active":"2024-01-16T14:56:45.976116+0000","last_became_peered":"2024-01-16T14:56:45.976116+0000","last_unstale":"2024-01-16T14:56:46.954278+0000","last_undegraded":"2024-01-16T14:56:46.954278+0000","last_fullsized":"2024-01-16T14:56:46.954278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309521+0000","last_change":"2024-01-16T14:56:45.992939+0000","last_active":"2024-01-16T14:56:47.309521+0000","last_peered":"2024-01-16T14:56:47.309521+0000","last_clean":"2024-01-16T14:56:47.309521+0000","last_became_active":"2024-01-16T14:56:45.992563+0000","last_became_peered":"2024-01-16T14:56:45.992563+0000","last_unstale":"2024-01-16T14:56:47.309521+0000","last_undegraded":"2024-01-16T14:56:47.309521+0000","last_fullsized":"2024-01-16T14:56:47.309521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956108+0000","last_change":"2024-01-16T14:56:45.980818+0000","last_active":"2024-01-16T14:56:46.956108+0000","last_peered":"2024-01-16T14:56:46.956108+0000","last_clean":"2024-01-16T14:56:46.956108+0000","last_became_active":"2024-01-16T14:56:45.980565+0000","last_became_peered":"2024-01-16T14:56:45.980565+0000","last_unstale":"2024-01-16T14:56:46.956108+0000","last_undegraded":"2024-01-16T14:56:46.956108+0000","last_fullsized":"2024-01-16T14:56:46.956108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:05.361881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312563+0000","last_change":"2024-01-16T14:56:45.990281+0000","last_active":"2024-01-16T14:56:47.312563+0000","last_peered":"2024-01-16T14:56:47.312563+0000","last_clean":"2024-01-16T14:56:47.312563+0000","last_became_active":"2024-01-16T14:56:45.990014+0000","last_became_peered":"2024-01-16T14:56:45.990014+0000","last_unstale":"2024-01-16T14:56:47.312563+0000","last_undegraded":"2024-01-16T14:56:47.312563+0000","last_fullsized":"2024-01-16T14:56:47.312563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312223+0000","last_change":"2024-01-16T14:56:45.990395+0000","last_active":"2024-01-16T14:56:47.312223+0000","last_peered":"2024-01-16T14:56:47.312223+0000","last_clean":"2024-01-16T14:56:47.312223+0000","last_became_active":"2024-01-16T14:56:45.990100+0000","last_became_peered":"2024-01-16T14:56:45.990100+0000","last_unstale":"2024-01-16T14:56:47.312223+0000","last_undegraded":"2024-01-16T14:56:47.312223+0000","last_fullsized":"2024-01-16T14:56:47.312223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954100+0000","last_change":"2024-01-16T14:56:45.977544+0000","last_active":"2024-01-16T14:56:46.954100+0000","last_peered":"2024-01-16T14:56:46.954100+0000","last_clean":"2024-01-16T14:56:46.954100+0000","last_became_active":"2024-01-16T14:56:45.976912+0000","last_became_peered":"2024-01-16T14:56:45.976912+0000","last_unstale":"2024-01-16T14:56:46.954100+0000","last_undegraded":"2024-01-16T14:56:46.954100+0000","last_fullsized":"2024-01-16T14:56:46.954100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:04.924598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309263+0000","last_change":"2024-01-16T14:56:45.986451+0000","last_active":"2024-01-16T14:56:47.309263+0000","last_peered":"2024-01-16T14:56:47.309263+0000","last_clean":"2024-01-16T14:56:47.309263+0000","last_became_active":"2024-01-16T14:56:45.985694+0000","last_became_peered":"2024-01-16T14:56:45.985694+0000","last_unstale":"2024-01-16T14:56:47.309263+0000","last_undegraded":"2024-01-16T14:56:47.309263+0000","last_fullsized":"2024-01-16T14:56:47.309263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311052+0000","last_change":"2024-01-16T14:56:45.980372+0000","last_active":"2024-01-16T14:56:47.311052+0000","last_peered":"2024-01-16T14:56:47.311052+0000","last_clean":"2024-01-16T14:56:47.311052+0000","last_became_active":"2024-01-16T14:56:45.980114+0000","last_became_peered":"2024-01-16T14:56:45.980114+0000","last_unstale":"2024-01-16T14:56:47.311052+0000","last_undegraded":"2024-01-16T14:56:47.311052+0000","last_fullsized":"2024-01-16T14:56:47.311052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:10.565274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950877+0000","last_change":"2024-01-16T14:56:45.987106+0000","last_active":"2024-01-16T14:56:46.950877+0000","last_peered":"2024-01-16T14:56:46.950877+0000","last_clean":"2024-01-16T14:56:46.950877+0000","last_became_active":"2024-01-16T14:56:45.986126+0000","last_became_peered":"2024-01-16T14:56:45.986126+0000","last_unstale":"2024-01-16T14:56:46.950877+0000","last_undegraded":"2024-01-16T14:56:46.950877+0000","last_fullsized":"2024-01-16T14:56:46.950877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310669+0000","last_change":"2024-01-16T14:56:45.984242+0000","last_active":"2024-01-16T14:56:47.310669+0000","last_peered":"2024-01-16T14:56:47.310669+0000","last_clean":"2024-01-16T14:56:47.310669+0000","last_became_active":"2024-01-16T14:56:45.983936+0000","last_became_peered":"2024-01-16T14:56:45.983936+0000","last_unstale":"2024-01-16T14:56:47.310669+0000","last_undegraded":"2024-01-16T14:56:47.310669+0000","last_fullsized":"2024-01-16T14:56:47.310669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312880+0000","last_change":"2024-01-16T14:56:45.983089+0000","last_active":"2024-01-16T14:56:47.312880+0000","last_peered":"2024-01-16T14:56:47.312880+0000","last_clean":"2024-01-16T14:56:47.312880+0000","last_became_active":"2024-01-16T14:56:45.982784+0000","last_became_peered":"2024-01-16T14:56:45.982784+0000","last_unstale":"2024-01-16T14:56:47.312880+0000","last_undegraded":"2024-01-16T14:56:47.312880+0000","last_fullsized":"2024-01-16T14:56:47.312880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312245+0000","last_change":"2024-01-16T14:56:45.977860+0000","last_active":"2024-01-16T14:56:47.312245+0000","last_peered":"2024-01-16T14:56:47.312245+0000","last_clean":"2024-01-16T14:56:47.312245+0000","last_became_active":"2024-01-16T14:56:45.977563+0000","last_became_peered":"2024-01-16T14:56:45.977563+0000","last_unstale":"2024-01-16T14:56:47.312245+0000","last_undegraded":"2024-01-16T14:56:47.312245+0000","last_fullsized":"2024-01-16T14:56:47.312245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955821+0000","last_change":"2024-01-16T14:56:45.977287+0000","last_active":"2024-01-16T14:56:46.955821+0000","last_peered":"2024-01-16T14:56:46.955821+0000","last_clean":"2024-01-16T14:56:46.955821+0000","last_became_active":"2024-01-16T14:56:45.976639+0000","last_became_peered":"2024-01-16T14:56:45.976639+0000","last_unstale":"2024-01-16T14:56:46.955821+0000","last_undegraded":"2024-01-16T14:56:46.955821+0000","last_fullsized":"2024-01-16T14:56:46.955821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312028+0000","last_change":"2024-01-16T14:56:45.984360+0000","last_active":"2024-01-16T14:56:47.312028+0000","last_peered":"2024-01-16T14:56:47.312028+0000","last_clean":"2024-01-16T14:56:47.312028+0000","last_became_active":"2024-01-16T14:56:45.984102+0000","last_became_peered":"2024-01-16T14:56:45.984102+0000","last_unstale":"2024-01-16T14:56:47.312028+0000","last_undegraded":"2024-01-16T14:56:47.312028+0000","last_fullsized":"2024-01-16T14:56:47.312028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311119+0000","last_change":"2024-01-16T14:56:45.973908+0000","last_active":"2024-01-16T14:56:47.311119+0000","last_peered":"2024-01-16T14:56:47.311119+0000","last_clean":"2024-01-16T14:56:47.311119+0000","last_became_active":"2024-01-16T14:56:45.970417+0000","last_became_peered":"2024-01-16T14:56:45.970417+0000","last_unstale":"2024-01-16T14:56:47.311119+0000","last_undegraded":"2024-01-16T14:56:47.311119+0000","last_fullsized":"2024-01-16T14:56:47.311119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311312+0000","last_change":"2024-01-16T14:56:45.984704+0000","last_active":"2024-01-16T14:56:47.311312+0000","last_peered":"2024-01-16T14:56:47.311312+0000","last_clean":"2024-01-16T14:56:47.311312+0000","last_became_active":"2024-01-16T14:56:45.983436+0000","last_became_peered":"2024-01-16T14:56:45.983436+0000","last_unstale":"2024-01-16T14:56:47.311312+0000","last_undegraded":"2024-01-16T14:56:47.311312+0000","last_fullsized":"2024-01-16T14:56:47.311312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308745+0000","last_change":"2024-01-16T14:56:45.989345+0000","last_active":"2024-01-16T14:56:47.308745+0000","last_peered":"2024-01-16T14:56:47.308745+0000","last_clean":"2024-01-16T14:56:47.308745+0000","last_became_active":"2024-01-16T14:56:45.989090+0000","last_became_peered":"2024-01-16T14:56:45.989090+0000","last_unstale":"2024-01-16T14:56:47.308745+0000","last_undegraded":"2024-01-16T14:56:47.308745+0000","last_fullsized":"2024-01-16T14:56:47.308745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310832+0000","last_change":"2024-01-16T14:56:45.988656+0000","last_active":"2024-01-16T14:56:47.310832+0000","last_peered":"2024-01-16T14:56:47.310832+0000","last_clean":"2024-01-16T14:56:47.310832+0000","last_became_active":"2024-01-16T14:56:45.988350+0000","last_became_peered":"2024-01-16T14:56:45.988350+0000","last_unstale":"2024-01-16T14:56:47.310832+0000","last_undegraded":"2024-01-16T14:56:47.310832+0000","last_fullsized":"2024-01-16T14:56:47.310832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308691+0000","last_change":"2024-01-16T14:56:45.984838+0000","last_active":"2024-01-16T14:56:47.308691+0000","last_peered":"2024-01-16T14:56:47.308691+0000","last_clean":"2024-01-16T14:56:47.308691+0000","last_became_active":"2024-01-16T14:56:45.984542+0000","last_became_peered":"2024-01-16T14:56:45.984542+0000","last_unstale":"2024-01-16T14:56:47.308691+0000","last_undegraded":"2024-01-16T14:56:47.308691+0000","last_fullsized":"2024-01-16T14:56:47.308691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312755+0000","last_change":"2024-01-16T14:56:45.989694+0000","last_active":"2024-01-16T14:56:47.312755+0000","last_peered":"2024-01-16T14:56:47.312755+0000","last_clean":"2024-01-16T14:56:47.312755+0000","last_became_active":"2024-01-16T14:56:45.989384+0000","last_became_peered":"2024-01-16T14:56:45.989384+0000","last_unstale":"2024-01-16T14:56:47.312755+0000","last_undegraded":"2024-01-16T14:56:47.312755+0000","last_fullsized":"2024-01-16T14:56:47.312755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951404+0000","last_change":"2024-01-16T14:56:45.988721+0000","last_active":"2024-01-16T14:56:46.951404+0000","last_peered":"2024-01-16T14:56:46.951404+0000","last_clean":"2024-01-16T14:56:46.951404+0000","last_became_active":"2024-01-16T14:56:45.988428+0000","last_became_peered":"2024-01-16T14:56:45.988428+0000","last_unstale":"2024-01-16T14:56:46.951404+0000","last_undegraded":"2024-01-16T14:56:46.951404+0000","last_fullsized":"2024-01-16T14:56:46.951404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309579+0000","last_change":"2024-01-16T14:56:45.986196+0000","last_active":"2024-01-16T14:56:47.309579+0000","last_peered":"2024-01-16T14:56:47.309579+0000","last_clean":"2024-01-16T14:56:47.309579+0000","last_became_active":"2024-01-16T14:56:45.985927+0000","last_became_peered":"2024-01-16T14:56:45.985927+0000","last_unstale":"2024-01-16T14:56:47.309579+0000","last_undegraded":"2024-01-16T14:56:47.309579+0000","last_fullsized":"2024-01-16T14:56:47.309579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311237+0000","last_change":"2024-01-16T14:56:45.985321+0000","last_active":"2024-01-16T14:56:47.311237+0000","last_peered":"2024-01-16T14:56:47.311237+0000","last_clean":"2024-01-16T14:56:47.311237+0000","last_became_active":"2024-01-16T14:56:45.985015+0000","last_became_peered":"2024-01-16T14:56:45.985015+0000","last_unstale":"2024-01-16T14:56:47.311237+0000","last_undegraded":"2024-01-16T14:56:47.311237+0000","last_fullsized":"2024-01-16T14:56:47.311237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311646+0000","last_change":"2024-01-16T14:56:45.986171+0000","last_active":"2024-01-16T14:56:47.311646+0000","last_peered":"2024-01-16T14:56:47.311646+0000","last_clean":"2024-01-16T14:56:47.311646+0000","last_became_active":"2024-01-16T14:56:45.985890+0000","last_became_peered":"2024-01-16T14:56:45.985890+0000","last_unstale":"2024-01-16T14:56:47.311646+0000","last_undegraded":"2024-01-16T14:56:47.311646+0000","last_fullsized":"2024-01-16T14:56:47.311646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312240+0000","last_change":"2024-01-16T14:56:45.979713+0000","last_active":"2024-01-16T14:56:47.312240+0000","last_peered":"2024-01-16T14:56:47.312240+0000","last_clean":"2024-01-16T14:56:47.312240+0000","last_became_active":"2024-01-16T14:56:45.979379+0000","last_became_peered":"2024-01-16T14:56:45.979379+0000","last_unstale":"2024-01-16T14:56:47.312240+0000","last_undegraded":"2024-01-16T14:56:47.312240+0000","last_fullsized":"2024-01-16T14:56:47.312240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311228+0000","last_change":"2024-01-16T14:56:45.988420+0000","last_active":"2024-01-16T14:56:47.311228+0000","last_peered":"2024-01-16T14:56:47.311228+0000","last_clean":"2024-01-16T14:56:47.311228+0000","last_became_active":"2024-01-16T14:56:45.987162+0000","last_became_peered":"2024-01-16T14:56:45.987162+0000","last_unstale":"2024-01-16T14:56:47.311228+0000","last_undegraded":"2024-01-16T14:56:47.311228+0000","last_fullsized":"2024-01-16T14:56:47.311228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952781+0000","last_change":"2024-01-16T14:56:45.977184+0000","last_active":"2024-01-16T14:56:46.952781+0000","last_peered":"2024-01-16T14:56:46.952781+0000","last_clean":"2024-01-16T14:56:46.952781+0000","last_became_active":"2024-01-16T14:56:45.976825+0000","last_became_peered":"2024-01-16T14:56:45.976825+0000","last_unstale":"2024-01-16T14:56:46.952781+0000","last_undegraded":"2024-01-16T14:56:46.952781+0000","last_fullsized":"2024-01-16T14:56:46.952781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:33.458747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012001+0000","last_change":"2024-01-16T14:56:45.008639+0000","last_active":"2024-01-16T14:56:46.012001+0000","last_peered":"2024-01-16T14:56:46.012001+0000","last_clean":"2024-01-16T14:56:46.012001+0000","last_became_active":"2024-01-16T14:56:45.008234+0000","last_became_peered":"2024-01-16T14:56:45.008234+0000","last_unstale":"2024-01-16T14:56:46.012001+0000","last_undegraded":"2024-01-16T14:56:46.012001+0000","last_fullsized":"2024-01-16T14:56:46.012001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311455+0000","last_change":"2024-01-16T14:56:45.976366+0000","last_active":"2024-01-16T14:56:47.311455+0000","last_peered":"2024-01-16T14:56:47.311455+0000","last_clean":"2024-01-16T14:56:47.311455+0000","last_became_active":"2024-01-16T14:56:45.975839+0000","last_became_peered":"2024-01-16T14:56:45.975839+0000","last_unstale":"2024-01-16T14:56:47.311455+0000","last_undegraded":"2024-01-16T14:56:47.311455+0000","last_fullsized":"2024-01-16T14:56:47.311455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010576+0000","last_change":"2024-01-16T14:56:45.003662+0000","last_active":"2024-01-16T14:56:46.010576+0000","last_peered":"2024-01-16T14:56:46.010576+0000","last_clean":"2024-01-16T14:56:46.010576+0000","last_became_active":"2024-01-16T14:56:45.003318+0000","last_became_peered":"2024-01-16T14:56:45.003318+0000","last_unstale":"2024-01-16T14:56:46.010576+0000","last_undegraded":"2024-01-16T14:56:46.010576+0000","last_fullsized":"2024-01-16T14:56:46.010576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.982735+0000","last_change":"2024-01-16T14:56:45.013121+0000","last_active":"2024-01-16T14:56:45.982735+0000","last_peered":"2024-01-16T14:56:45.982735+0000","last_clean":"2024-01-16T14:56:45.982735+0000","last_became_active":"2024-01-16T14:56:45.012797+0000","last_became_peered":"2024-01-16T14:56:45.012797+0000","last_unstale":"2024-01-16T14:56:45.982735+0000","last_undegraded":"2024-01-16T14:56:45.982735+0000","last_fullsized":"2024-01-16T14:56:45.982735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308918+0000","last_change":"2024-01-16T14:56:45.986213+0000","last_active":"2024-01-16T14:56:47.308918+0000","last_peered":"2024-01-16T14:56:47.308918+0000","last_clean":"2024-01-16T14:56:47.308918+0000","last_became_active":"2024-01-16T14:56:45.985824+0000","last_became_peered":"2024-01-16T14:56:45.985824+0000","last_unstale":"2024-01-16T14:56:47.308918+0000","last_undegraded":"2024-01-16T14:56:47.308918+0000","last_fullsized":"2024-01-16T14:56:47.308918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.974634+0000","last_change":"2024-01-16T14:56:45.014687+0000","last_active":"2024-01-16T14:56:45.974634+0000","last_peered":"2024-01-16T14:56:45.974634+0000","last_clean":"2024-01-16T14:56:45.974634+0000","last_became_active":"2024-01-16T14:56:45.014356+0000","last_became_peered":"2024-01-16T14:56:45.014356+0000","last_unstale":"2024-01-16T14:56:45.974634+0000","last_undegraded":"2024-01-16T14:56:45.974634+0000","last_fullsized":"2024-01-16T14:56:45.974634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:34.974850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950693+0000","last_change":"2024-01-16T14:56:45.984146+0000","last_active":"2024-01-16T14:56:46.950693+0000","last_peered":"2024-01-16T14:56:46.950693+0000","last_clean":"2024-01-16T14:56:46.950693+0000","last_became_active":"2024-01-16T14:56:45.983841+0000","last_became_peered":"2024-01-16T14:56:45.983841+0000","last_unstale":"2024-01-16T14:56:46.950693+0000","last_undegraded":"2024-01-16T14:56:46.950693+0000","last_fullsized":"2024-01-16T14:56:46.950693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.971677+0000","last_change":"2024-01-16T14:56:45.011843+0000","last_active":"2024-01-16T14:56:45.971677+0000","last_peered":"2024-01-16T14:56:45.971677+0000","last_clean":"2024-01-16T14:56:45.971677+0000","last_became_active":"2024-01-16T14:56:45.011331+0000","last_became_peered":"2024-01-16T14:56:45.011331+0000","last_unstale":"2024-01-16T14:56:45.971677+0000","last_undegraded":"2024-01-16T14:56:45.971677+0000","last_fullsized":"2024-01-16T14:56:45.971677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313267+0000","last_change":"2024-01-16T14:56:45.986253+0000","last_active":"2024-01-16T14:56:47.313267+0000","last_peered":"2024-01-16T14:56:47.313267+0000","last_clean":"2024-01-16T14:56:47.313267+0000","last_became_active":"2024-01-16T14:56:45.985935+0000","last_became_peered":"2024-01-16T14:56:45.985935+0000","last_unstale":"2024-01-16T14:56:47.313267+0000","last_undegraded":"2024-01-16T14:56:47.313267+0000","last_fullsized":"2024-01-16T14:56:47.313267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.973280+0000","last_change":"2024-01-16T14:56:45.016005+0000","last_active":"2024-01-16T14:56:45.973280+0000","last_peered":"2024-01-16T14:56:45.973280+0000","last_clean":"2024-01-16T14:56:45.973280+0000","last_became_active":"2024-01-16T14:56:45.015680+0000","last_became_peered":"2024-01-16T14:56:45.015680+0000","last_unstale":"2024-01-16T14:56:45.973280+0000","last_undegraded":"2024-01-16T14:56:45.973280+0000","last_fullsized":"2024-01-16T14:56:45.973280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:19.042486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311306+0000","last_change":"2024-01-16T14:56:45.980174+0000","last_active":"2024-01-16T14:56:47.311306+0000","last_peered":"2024-01-16T14:56:47.311306+0000","last_clean":"2024-01-16T14:56:47.311306+0000","last_became_active":"2024-01-16T14:56:45.979743+0000","last_became_peered":"2024-01-16T14:56:45.979743+0000","last_unstale":"2024-01-16T14:56:47.311306+0000","last_undegraded":"2024-01-16T14:56:47.311306+0000","last_fullsized":"2024-01-16T14:56:47.311306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013898+0000","last_change":"2024-01-16T14:56:45.017298+0000","last_active":"2024-01-16T14:56:46.013898+0000","last_peered":"2024-01-16T14:56:46.013898+0000","last_clean":"2024-01-16T14:56:46.013898+0000","last_became_active":"2024-01-16T14:56:45.016921+0000","last_became_peered":"2024-01-16T14:56:45.016921+0000","last_unstale":"2024-01-16T14:56:46.013898+0000","last_undegraded":"2024-01-16T14:56:46.013898+0000","last_fullsized":"2024-01-16T14:56:46.013898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954466+0000","last_change":"2024-01-16T14:56:45.967606+0000","last_active":"2024-01-16T14:56:46.954466+0000","last_peered":"2024-01-16T14:56:46.954466+0000","last_clean":"2024-01-16T14:56:46.954466+0000","last_became_active":"2024-01-16T14:56:45.967036+0000","last_became_peered":"2024-01-16T14:56:45.967036+0000","last_unstale":"2024-01-16T14:56:46.954466+0000","last_undegraded":"2024-01-16T14:56:46.954466+0000","last_fullsized":"2024-01-16T14:56:46.954466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015870+0000","last_change":"2024-01-16T14:56:45.005985+0000","last_active":"2024-01-16T14:56:46.015870+0000","last_peered":"2024-01-16T14:56:46.015870+0000","last_clean":"2024-01-16T14:56:46.015870+0000","last_became_active":"2024-01-16T14:56:45.005530+0000","last_became_peered":"2024-01-16T14:56:45.005530+0000","last_unstale":"2024-01-16T14:56:46.015870+0000","last_undegraded":"2024-01-16T14:56:46.015870+0000","last_fullsized":"2024-01-16T14:56:46.015870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:17.631008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955160+0000","last_change":"2024-01-16T14:56:45.991157+0000","last_active":"2024-01-16T14:56:46.955160+0000","last_peered":"2024-01-16T14:56:46.955160+0000","last_clean":"2024-01-16T14:56:46.955160+0000","last_became_active":"2024-01-16T14:56:45.990848+0000","last_became_peered":"2024-01-16T14:56:45.990848+0000","last_unstale":"2024-01-16T14:56:46.955160+0000","last_undegraded":"2024-01-16T14:56:46.955160+0000","last_fullsized":"2024-01-16T14:56:46.955160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016578+0000","last_change":"2024-01-16T14:56:45.000374+0000","last_active":"2024-01-16T14:56:46.016578+0000","last_peered":"2024-01-16T14:56:46.016578+0000","last_clean":"2024-01-16T14:56:46.016578+0000","last_became_active":"2024-01-16T14:56:45.000028+0000","last_became_peered":"2024-01-16T14:56:45.000028+0000","last_unstale":"2024-01-16T14:56:46.016578+0000","last_undegraded":"2024-01-16T14:56:46.016578+0000","last_fullsized":"2024-01-16T14:56:46.016578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311398+0000","last_change":"2024-01-16T14:56:45.974961+0000","last_active":"2024-01-16T14:56:47.311398+0000","last_peered":"2024-01-16T14:56:47.311398+0000","last_clean":"2024-01-16T14:56:47.311398+0000","last_became_active":"2024-01-16T14:56:45.974581+0000","last_became_peered":"2024-01-16T14:56:45.974581+0000","last_unstale":"2024-01-16T14:56:47.311398+0000","last_undegraded":"2024-01-16T14:56:47.311398+0000","last_fullsized":"2024-01-16T14:56:47.311398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.972980+0000","last_change":"2024-01-16T14:56:45.017987+0000","last_active":"2024-01-16T14:56:45.972980+0000","last_peered":"2024-01-16T14:56:45.972980+0000","last_clean":"2024-01-16T14:56:45.972980+0000","last_became_active":"2024-01-16T14:56:45.017656+0000","last_became_peered":"2024-01-16T14:56:45.017656+0000","last_unstale":"2024-01-16T14:56:45.972980+0000","last_undegraded":"2024-01-16T14:56:45.972980+0000","last_fullsized":"2024-01-16T14:56:45.972980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954932+0000","last_change":"2024-01-16T14:56:45.976869+0000","last_active":"2024-01-16T14:56:46.954932+0000","last_peered":"2024-01-16T14:56:46.954932+0000","last_clean":"2024-01-16T14:56:46.954932+0000","last_became_active":"2024-01-16T14:56:45.976500+0000","last_became_peered":"2024-01-16T14:56:45.976500+0000","last_unstale":"2024-01-16T14:56:46.954932+0000","last_undegraded":"2024-01-16T14:56:46.954932+0000","last_fullsized":"2024-01-16T14:56:46.954932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.966318+0000","last_change":"2024-01-16T14:56:45.012069+0000","last_active":"2024-01-16T14:56:45.966318+0000","last_peered":"2024-01-16T14:56:45.966318+0000","last_clean":"2024-01-16T14:56:45.966318+0000","last_became_active":"2024-01-16T14:56:45.011711+0000","last_became_peered":"2024-01-16T14:56:45.011711+0000","last_unstale":"2024-01-16T14:56:45.966318+0000","last_undegraded":"2024-01-16T14:56:45.966318+0000","last_fullsized":"2024-01-16T14:56:45.966318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311361+0000","last_change":"2024-01-16T14:56:45.988281+0000","last_active":"2024-01-16T14:56:47.311361+0000","last_peered":"2024-01-16T14:56:47.311361+0000","last_clean":"2024-01-16T14:56:47.311361+0000","last_became_active":"2024-01-16T14:56:45.986521+0000","last_became_peered":"2024-01-16T14:56:45.986521+0000","last_unstale":"2024-01-16T14:56:47.311361+0000","last_undegraded":"2024-01-16T14:56:47.311361+0000","last_fullsized":"2024-01-16T14:56:47.311361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015745+0000","last_change":"2024-01-16T14:56:44.993188+0000","last_active":"2024-01-16T14:56:46.015745+0000","last_peered":"2024-01-16T14:56:46.015745+0000","last_clean":"2024-01-16T14:56:46.015745+0000","last_became_active":"2024-01-16T14:56:44.992904+0000","last_became_peered":"2024-01-16T14:56:44.992904+0000","last_unstale":"2024-01-16T14:56:46.015745+0000","last_undegraded":"2024-01-16T14:56:46.015745+0000","last_fullsized":"2024-01-16T14:56:46.015745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.524009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954660+0000","last_change":"2024-01-16T14:56:45.981562+0000","last_active":"2024-01-16T14:56:46.954660+0000","last_peered":"2024-01-16T14:56:46.954660+0000","last_clean":"2024-01-16T14:56:46.954660+0000","last_became_active":"2024-01-16T14:56:45.981272+0000","last_became_peered":"2024-01-16T14:56:45.981272+0000","last_unstale":"2024-01-16T14:56:46.954660+0000","last_undegraded":"2024-01-16T14:56:46.954660+0000","last_fullsized":"2024-01-16T14:56:46.954660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013126+0000","last_change":"2024-01-16T14:56:45.005288+0000","last_active":"2024-01-16T14:56:46.013126+0000","last_peered":"2024-01-16T14:56:46.013126+0000","last_clean":"2024-01-16T14:56:46.013126+0000","last_became_active":"2024-01-16T14:56:45.004988+0000","last_became_peered":"2024-01-16T14:56:45.004988+0000","last_unstale":"2024-01-16T14:56:46.013126+0000","last_undegraded":"2024-01-16T14:56:46.013126+0000","last_fullsized":"2024-01-16T14:56:46.013126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955886+0000","last_change":"2024-01-16T14:56:45.977479+0000","last_active":"2024-01-16T14:56:46.955886+0000","last_peered":"2024-01-16T14:56:46.955886+0000","last_clean":"2024-01-16T14:56:46.955886+0000","last_became_active":"2024-01-16T14:56:45.976447+0000","last_became_peered":"2024-01-16T14:56:45.976447+0000","last_unstale":"2024-01-16T14:56:46.955886+0000","last_undegraded":"2024-01-16T14:56:46.955886+0000","last_fullsized":"2024-01-16T14:56:46.955886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:09:28.671518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012387+0000","last_change":"2024-01-16T14:56:44.991027+0000","last_active":"2024-01-16T14:56:46.012387+0000","last_peered":"2024-01-16T14:56:46.012387+0000","last_clean":"2024-01-16T14:56:46.012387+0000","last_became_active":"2024-01-16T14:56:44.990777+0000","last_became_peered":"2024-01-16T14:56:44.990777+0000","last_unstale":"2024-01-16T14:56:46.012387+0000","last_undegraded":"2024-01-16T14:56:46.012387+0000","last_fullsized":"2024-01-16T14:56:46.012387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956303+0000","last_change":"2024-01-16T14:56:45.980059+0000","last_active":"2024-01-16T14:56:46.956303+0000","last_peered":"2024-01-16T14:56:46.956303+0000","last_clean":"2024-01-16T14:56:46.956303+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956303+0000","last_undegraded":"2024-01-16T14:56:46.956303+0000","last_fullsized":"2024-01-16T14:56:46.956303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:40.703250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010244+0000","last_change":"2024-01-16T14:56:45.001611+0000","last_active":"2024-01-16T14:56:46.010244+0000","last_peered":"2024-01-16T14:56:46.010244+0000","last_clean":"2024-01-16T14:56:46.010244+0000","last_became_active":"2024-01-16T14:56:45.001338+0000","last_became_peered":"2024-01-16T14:56:45.001338+0000","last_unstale":"2024-01-16T14:56:46.010244+0000","last_undegraded":"2024-01-16T14:56:46.010244+0000","last_fullsized":"2024-01-16T14:56:46.010244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312030+0000","last_change":"2024-01-16T14:56:45.986001+0000","last_active":"2024-01-16T14:56:47.312030+0000","last_peered":"2024-01-16T14:56:47.312030+0000","last_clean":"2024-01-16T14:56:47.312030+0000","last_became_active":"2024-01-16T14:56:45.984074+0000","last_became_peered":"2024-01-16T14:56:45.984074+0000","last_unstale":"2024-01-16T14:56:47.312030+0000","last_undegraded":"2024-01-16T14:56:47.312030+0000","last_fullsized":"2024-01-16T14:56:47.312030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.960950+0000","last_change":"2024-01-16T14:56:45.009683+0000","last_active":"2024-01-16T14:56:45.960950+0000","last_peered":"2024-01-16T14:56:45.960950+0000","last_clean":"2024-01-16T14:56:45.960950+0000","last_became_active":"2024-01-16T14:56:45.009325+0000","last_became_peered":"2024-01-16T14:56:45.009325+0000","last_unstale":"2024-01-16T14:56:45.960950+0000","last_undegraded":"2024-01-16T14:56:45.960950+0000","last_fullsized":"2024-01-16T14:56:45.960950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:27.159214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955489+0000","last_change":"2024-01-16T14:56:45.989743+0000","last_active":"2024-01-16T14:56:46.955489+0000","last_peered":"2024-01-16T14:56:46.955489+0000","last_clean":"2024-01-16T14:56:46.955489+0000","last_became_active":"2024-01-16T14:56:45.989084+0000","last_became_peered":"2024-01-16T14:56:45.989084+0000","last_unstale":"2024-01-16T14:56:46.955489+0000","last_undegraded":"2024-01-16T14:56:46.955489+0000","last_fullsized":"2024-01-16T14:56:46.955489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.955711+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:56:45.955711+0000","last_peered":"2024-01-16T14:56:45.955711+0000","last_clean":"2024-01-16T14:56:45.955711+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:56:45.955711+0000","last_undegraded":"2024-01-16T14:56:45.955711+0000","last_fullsized":"2024-01-16T14:56:45.955711+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963629+0000","last_change":"2024-01-16T14:56:45.009432+0000","last_active":"2024-01-16T14:56:45.963629+0000","last_peered":"2024-01-16T14:56:45.963629+0000","last_clean":"2024-01-16T14:56:45.963629+0000","last_became_active":"2024-01-16T14:56:45.008692+0000","last_became_peered":"2024-01-16T14:56:45.008692+0000","last_unstale":"2024-01-16T14:56:45.963629+0000","last_undegraded":"2024-01-16T14:56:45.963629+0000","last_fullsized":"2024-01-16T14:56:45.963629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308881+0000","last_change":"2024-01-16T14:56:45.990578+0000","last_active":"2024-01-16T14:56:47.308881+0000","last_peered":"2024-01-16T14:56:47.308881+0000","last_clean":"2024-01-16T14:56:47.308881+0000","last_became_active":"2024-01-16T14:56:45.990211+0000","last_became_peered":"2024-01-16T14:56:45.990211+0000","last_unstale":"2024-01-16T14:56:47.308881+0000","last_undegraded":"2024-01-16T14:56:47.308881+0000","last_fullsized":"2024-01-16T14:56:47.308881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.956468+0000","last_change":"2024-01-16T14:56:45.006632+0000","last_active":"2024-01-16T14:56:45.956468+0000","last_peered":"2024-01-16T14:56:45.956468+0000","last_clean":"2024-01-16T14:56:45.956468+0000","last_became_active":"2024-01-16T14:56:45.006340+0000","last_became_peered":"2024-01-16T14:56:45.006340+0000","last_unstale":"2024-01-16T14:56:45.956468+0000","last_undegraded":"2024-01-16T14:56:45.956468+0000","last_fullsized":"2024-01-16T14:56:45.956468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:41.754460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310892+0000","last_change":"2024-01-16T14:56:45.977532+0000","last_active":"2024-01-16T14:56:47.310892+0000","last_peered":"2024-01-16T14:56:47.310892+0000","last_clean":"2024-01-16T14:56:47.310892+0000","last_became_active":"2024-01-16T14:56:45.975102+0000","last_became_peered":"2024-01-16T14:56:45.975102+0000","last_unstale":"2024-01-16T14:56:47.310892+0000","last_undegraded":"2024-01-16T14:56:47.310892+0000","last_fullsized":"2024-01-16T14:56:47.310892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:18.643652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963071+0000","last_change":"2024-01-16T14:56:44.994705+0000","last_active":"2024-01-16T14:56:45.963071+0000","last_peered":"2024-01-16T14:56:45.963071+0000","last_clean":"2024-01-16T14:56:45.963071+0000","last_became_active":"2024-01-16T14:56:44.994387+0000","last_became_peered":"2024-01-16T14:56:44.994387+0000","last_unstale":"2024-01-16T14:56:45.963071+0000","last_undegraded":"2024-01-16T14:56:45.963071+0000","last_fullsized":"2024-01-16T14:56:45.963071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950691+0000","last_change":"2024-01-16T14:56:45.978166+0000","last_active":"2024-01-16T14:56:46.950691+0000","last_peered":"2024-01-16T14:56:46.950691+0000","last_clean":"2024-01-16T14:56:46.950691+0000","last_became_active":"2024-01-16T14:56:45.974628+0000","last_became_peered":"2024-01-16T14:56:45.974628+0000","last_unstale":"2024-01-16T14:56:46.950691+0000","last_undegraded":"2024-01-16T14:56:46.950691+0000","last_fullsized":"2024-01-16T14:56:46.950691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.980309+0000","last_change":"2024-01-16T14:56:45.017774+0000","last_active":"2024-01-16T14:56:45.980309+0000","last_peered":"2024-01-16T14:56:45.980309+0000","last_clean":"2024-01-16T14:56:45.980309+0000","last_became_active":"2024-01-16T14:56:45.017410+0000","last_became_peered":"2024-01-16T14:56:45.017410+0000","last_unstale":"2024-01-16T14:56:45.980309+0000","last_undegraded":"2024-01-16T14:56:45.980309+0000","last_fullsized":"2024-01-16T14:56:45.980309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:59.852603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954411+0000","last_change":"2024-01-16T14:56:45.970975+0000","last_active":"2024-01-16T14:56:46.954411+0000","last_peered":"2024-01-16T14:56:46.954411+0000","last_clean":"2024-01-16T14:56:46.954411+0000","last_became_active":"2024-01-16T14:56:45.970566+0000","last_became_peered":"2024-01-16T14:56:45.970566+0000","last_unstale":"2024-01-16T14:56:46.954411+0000","last_undegraded":"2024-01-16T14:56:46.954411+0000","last_fullsized":"2024-01-16T14:56:46.954411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.967326+0000","last_change":"2024-01-16T14:56:45.004864+0000","last_active":"2024-01-16T14:56:45.967326+0000","last_peered":"2024-01-16T14:56:45.967326+0000","last_clean":"2024-01-16T14:56:45.967326+0000","last_became_active":"2024-01-16T14:56:45.004057+0000","last_became_peered":"2024-01-16T14:56:45.004057+0000","last_unstale":"2024-01-16T14:56:45.967326+0000","last_undegraded":"2024-01-16T14:56:45.967326+0000","last_fullsized":"2024-01-16T14:56:45.967326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313581+0000","last_change":"2024-01-16T14:56:45.989780+0000","last_active":"2024-01-16T14:56:47.313581+0000","last_peered":"2024-01-16T14:56:47.313581+0000","last_clean":"2024-01-16T14:56:47.313581+0000","last_became_active":"2024-01-16T14:56:45.989519+0000","last_became_peered":"2024-01-16T14:56:45.989519+0000","last_unstale":"2024-01-16T14:56:47.313581+0000","last_undegraded":"2024-01-16T14:56:47.313581+0000","last_fullsized":"2024-01-16T14:56:47.313581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.959086+0000","last_change":"2024-01-16T14:56:45.009420+0000","last_active":"2024-01-16T14:56:45.959086+0000","last_peered":"2024-01-16T14:56:45.959086+0000","last_clean":"2024-01-16T14:56:45.959086+0000","last_became_active":"2024-01-16T14:56:45.009103+0000","last_became_peered":"2024-01-16T14:56:45.009103+0000","last_unstale":"2024-01-16T14:56:45.959086+0000","last_undegraded":"2024-01-16T14:56:45.959086+0000","last_fullsized":"2024-01-16T14:56:45.959086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:49.027396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951463+0000","last_change":"2024-01-16T14:56:45.976716+0000","last_active":"2024-01-16T14:56:46.951463+0000","last_peered":"2024-01-16T14:56:46.951463+0000","last_clean":"2024-01-16T14:56:46.951463+0000","last_became_active":"2024-01-16T14:56:45.976307+0000","last_became_peered":"2024-01-16T14:56:45.976307+0000","last_unstale":"2024-01-16T14:56:46.951463+0000","last_undegraded":"2024-01-16T14:56:46.951463+0000","last_fullsized":"2024-01-16T14:56:46.951463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.964691+0000","last_change":"2024-01-16T14:56:45.013169+0000","last_active":"2024-01-16T14:56:45.964691+0000","last_peered":"2024-01-16T14:56:45.964691+0000","last_clean":"2024-01-16T14:56:45.964691+0000","last_became_active":"2024-01-16T14:56:45.012872+0000","last_became_peered":"2024-01-16T14:56:45.012872+0000","last_unstale":"2024-01-16T14:56:45.964691+0000","last_undegraded":"2024-01-16T14:56:45.964691+0000","last_fullsized":"2024-01-16T14:56:45.964691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308843+0000","last_change":"2024-01-16T14:56:45.987343+0000","last_active":"2024-01-16T14:56:47.308843+0000","last_peered":"2024-01-16T14:56:47.308843+0000","last_clean":"2024-01-16T14:56:47.308843+0000","last_became_active":"2024-01-16T14:56:45.987065+0000","last_became_peered":"2024-01-16T14:56:45.987065+0000","last_unstale":"2024-01-16T14:56:47.308843+0000","last_undegraded":"2024-01-16T14:56:47.308843+0000","last_fullsized":"2024-01-16T14:56:47.308843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014224+0000","last_change":"2024-01-16T14:56:45.004437+0000","last_active":"2024-01-16T14:56:46.014224+0000","last_peered":"2024-01-16T14:56:46.014224+0000","last_clean":"2024-01-16T14:56:46.014224+0000","last_became_active":"2024-01-16T14:56:45.004165+0000","last_became_peered":"2024-01-16T14:56:45.004165+0000","last_unstale":"2024-01-16T14:56:46.014224+0000","last_undegraded":"2024-01-16T14:56:46.014224+0000","last_fullsized":"2024-01-16T14:56:46.014224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311396+0000","last_change":"2024-01-16T14:56:45.974262+0000","last_active":"2024-01-16T14:56:47.311396+0000","last_peered":"2024-01-16T14:56:47.311396+0000","last_clean":"2024-01-16T14:56:47.311396+0000","last_became_active":"2024-01-16T14:56:45.970037+0000","last_became_peered":"2024-01-16T14:56:45.970037+0000","last_unstale":"2024-01-16T14:56:47.311396+0000","last_undegraded":"2024-01-16T14:56:47.311396+0000","last_fullsized":"2024-01-16T14:56:47.311396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.952176+0000","last_change":"2024-01-16T14:56:45.009771+0000","last_active":"2024-01-16T14:56:45.952176+0000","last_peered":"2024-01-16T14:56:45.952176+0000","last_clean":"2024-01-16T14:56:45.952176+0000","last_became_active":"2024-01-16T14:56:45.009505+0000","last_became_peered":"2024-01-16T14:56:45.009505+0000","last_unstale":"2024-01-16T14:56:45.952176+0000","last_undegraded":"2024-01-16T14:56:45.952176+0000","last_fullsized":"2024-01-16T14:56:45.952176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952907+0000","last_change":"2024-01-16T14:56:45.970586+0000","last_active":"2024-01-16T14:56:46.952907+0000","last_peered":"2024-01-16T14:56:46.952907+0000","last_clean":"2024-01-16T14:56:46.952907+0000","last_became_active":"2024-01-16T14:56:45.970146+0000","last_became_peered":"2024-01-16T14:56:45.970146+0000","last_unstale":"2024-01-16T14:56:46.952907+0000","last_undegraded":"2024-01-16T14:56:46.952907+0000","last_fullsized":"2024-01-16T14:56:46.952907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:23.365045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.962308+0000","last_change":"2024-01-16T14:56:45.003139+0000","last_active":"2024-01-16T14:56:45.962308+0000","last_peered":"2024-01-16T14:56:45.962308+0000","last_clean":"2024-01-16T14:56:45.962308+0000","last_became_active":"2024-01-16T14:56:45.002787+0000","last_became_peered":"2024-01-16T14:56:45.002787+0000","last_unstale":"2024-01-16T14:56:45.962308+0000","last_undegraded":"2024-01-16T14:56:45.962308+0000","last_fullsized":"2024-01-16T14:56:45.962308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309294+0000","last_change":"2024-01-16T14:56:45.981803+0000","last_active":"2024-01-16T14:56:47.309294+0000","last_peered":"2024-01-16T14:56:47.309294+0000","last_clean":"2024-01-16T14:56:47.309294+0000","last_became_active":"2024-01-16T14:56:45.981480+0000","last_became_peered":"2024-01-16T14:56:45.981480+0000","last_unstale":"2024-01-16T14:56:47.309294+0000","last_undegraded":"2024-01-16T14:56:47.309294+0000","last_fullsized":"2024-01-16T14:56:47.309294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.968506+0000","last_change":"2024-01-16T14:56:45.006149+0000","last_active":"2024-01-16T14:56:45.968506+0000","last_peered":"2024-01-16T14:56:45.968506+0000","last_clean":"2024-01-16T14:56:45.968506+0000","last_became_active":"2024-01-16T14:56:45.005789+0000","last_became_peered":"2024-01-16T14:56:45.005789+0000","last_unstale":"2024-01-16T14:56:45.968506+0000","last_undegraded":"2024-01-16T14:56:45.968506+0000","last_fullsized":"2024-01-16T14:56:45.968506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309478+0000","last_change":"2024-01-16T14:56:45.985569+0000","last_active":"2024-01-16T14:56:47.309478+0000","last_peered":"2024-01-16T14:56:47.309478+0000","last_clean":"2024-01-16T14:56:47.309478+0000","last_became_active":"2024-01-16T14:56:45.983774+0000","last_became_peered":"2024-01-16T14:56:45.983774+0000","last_unstale":"2024-01-16T14:56:47.309478+0000","last_undegraded":"2024-01-16T14:56:47.309478+0000","last_fullsized":"2024-01-16T14:56:47.309478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013631+0000","last_change":"2024-01-16T14:56:45.010441+0000","last_active":"2024-01-16T14:56:46.013631+0000","last_peered":"2024-01-16T14:56:46.013631+0000","last_clean":"2024-01-16T14:56:46.013631+0000","last_became_active":"2024-01-16T14:56:45.007667+0000","last_became_peered":"2024-01-16T14:56:45.007667+0000","last_unstale":"2024-01-16T14:56:46.013631+0000","last_undegraded":"2024-01-16T14:56:46.013631+0000","last_fullsized":"2024-01-16T14:56:46.013631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:38.016001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311624+0000","last_change":"2024-01-16T14:56:45.984334+0000","last_active":"2024-01-16T14:56:47.311624+0000","last_peered":"2024-01-16T14:56:47.311624+0000","last_clean":"2024-01-16T14:56:47.311624+0000","last_became_active":"2024-01-16T14:56:45.984013+0000","last_became_peered":"2024-01-16T14:56:45.984013+0000","last_unstale":"2024-01-16T14:56:47.311624+0000","last_undegraded":"2024-01-16T14:56:47.311624+0000","last_fullsized":"2024-01-16T14:56:47.311624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313406+0000","last_change":"2024-01-16T14:56:45.985148+0000","last_active":"2024-01-16T14:56:47.313406+0000","last_peered":"2024-01-16T14:56:47.313406+0000","last_clean":"2024-01-16T14:56:47.313406+0000","last_became_active":"2024-01-16T14:56:45.983354+0000","last_became_peered":"2024-01-16T14:56:45.983354+0000","last_unstale":"2024-01-16T14:56:47.313406+0000","last_undegraded":"2024-01-16T14:56:47.313406+0000","last_fullsized":"2024-01-16T14:56:47.313406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011363+0000","last_change":"2024-01-16T14:56:45.017135+0000","last_active":"2024-01-16T14:56:46.011363+0000","last_peered":"2024-01-16T14:56:46.011363+0000","last_clean":"2024-01-16T14:56:46.011363+0000","last_became_active":"2024-01-16T14:56:45.016686+0000","last_became_peered":"2024-01-16T14:56:45.016686+0000","last_unstale":"2024-01-16T14:56:46.011363+0000","last_undegraded":"2024-01-16T14:56:46.011363+0000","last_fullsized":"2024-01-16T14:56:46.011363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311883+0000","last_change":"2024-01-16T14:56:45.983050+0000","last_active":"2024-01-16T14:56:47.311883+0000","last_peered":"2024-01-16T14:56:47.311883+0000","last_clean":"2024-01-16T14:56:47.311883+0000","last_became_active":"2024-01-16T14:56:45.982678+0000","last_became_peered":"2024-01-16T14:56:45.982678+0000","last_unstale":"2024-01-16T14:56:47.311883+0000","last_undegraded":"2024-01-16T14:56:47.311883+0000","last_fullsized":"2024-01-16T14:56:47.311883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011339+0000","last_change":"2024-01-16T14:56:45.014696+0000","last_active":"2024-01-16T14:56:46.011339+0000","last_peered":"2024-01-16T14:56:46.011339+0000","last_clean":"2024-01-16T14:56:46.011339+0000","last_became_active":"2024-01-16T14:56:45.014410+0000","last_became_peered":"2024-01-16T14:56:45.014410+0000","last_unstale":"2024-01-16T14:56:46.011339+0000","last_undegraded":"2024-01-16T14:56:46.011339+0000","last_fullsized":"2024-01-16T14:56:46.011339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311230+0000","last_change":"2024-01-16T14:56:45.983804+0000","last_active":"2024-01-16T14:56:47.311230+0000","last_peered":"2024-01-16T14:56:47.311230+0000","last_clean":"2024-01-16T14:56:47.311230+0000","last_became_active":"2024-01-16T14:56:45.983529+0000","last_became_peered":"2024-01-16T14:56:45.983529+0000","last_unstale":"2024-01-16T14:56:47.311230+0000","last_undegraded":"2024-01-16T14:56:47.311230+0000","last_fullsized":"2024-01-16T14:56:47.311230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014146+0000","last_change":"2024-01-16T14:56:45.005514+0000","last_active":"2024-01-16T14:56:46.014146+0000","last_peered":"2024-01-16T14:56:46.014146+0000","last_clean":"2024-01-16T14:56:46.014146+0000","last_became_active":"2024-01-16T14:56:45.004610+0000","last_became_peered":"2024-01-16T14:56:45.004610+0000","last_unstale":"2024-01-16T14:56:46.014146+0000","last_undegraded":"2024-01-16T14:56:46.014146+0000","last_fullsized":"2024-01-16T14:56:46.014146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308980+0000","last_change":"2024-01-16T14:56:45.992067+0000","last_active":"2024-01-16T14:56:47.308980+0000","last_peered":"2024-01-16T14:56:47.308980+0000","last_clean":"2024-01-16T14:56:47.308980+0000","last_became_active":"2024-01-16T14:56:45.991826+0000","last_became_peered":"2024-01-16T14:56:45.991826+0000","last_unstale":"2024-01-16T14:56:47.308980+0000","last_undegraded":"2024-01-16T14:56:47.308980+0000","last_fullsized":"2024-01-16T14:56:47.308980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016889+0000","last_change":"2024-01-16T14:56:45.003619+0000","last_active":"2024-01-16T14:56:46.016889+0000","last_peered":"2024-01-16T14:56:46.016889+0000","last_clean":"2024-01-16T14:56:46.016889+0000","last_became_active":"2024-01-16T14:56:45.002981+0000","last_became_peered":"2024-01-16T14:56:45.002981+0000","last_unstale":"2024-01-16T14:56:46.016889+0000","last_undegraded":"2024-01-16T14:56:46.016889+0000","last_fullsized":"2024-01-16T14:56:46.016889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":660897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0910000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1389999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:56:50.890 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:56:50.891 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T14:56:50.891 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T14:56:50.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T14:56:50.891 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T14:56:51.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.161+0000 7fbae5a00640 1 Processor -- start 2024-01-16T14:56:51.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.161+0000 7fbae5a00640 1 -- start start 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbae5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00ae5a0 0x7fbae015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbae5a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbae00b3ae0 0x7fbae006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbae5a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae0070c40 con 0x7fbae006d120 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbae5a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae0071450 con 0x7fbae00b3ae0 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbae5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae006c470 con 0x7fbae00ae5a0 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbad7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbae00b3ae0 0x7fbae006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbadf400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbae006d120 0x7fbae00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36478/0 (socket says 172.21.15.125:36478) 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbadf400640 1 -- 172.21.15.125:0/3466496467 learned_addr learned my addr 172.21.15.125:0/3466496467 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbad7e00640 1 --2- 172.21.15.125:0/3466496467 >> v2:172.21.15.125:3300/0 conn(0x7fbae00b3ae0 0x7fbae006cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47684/0 (socket says 172.21.15.125:47684) 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.163+0000 7fbadea00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00ae5a0 0x7fbae015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.164+0000 7fbad7e00640 1 -- 172.21.15.125:0/3466496467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00ae5a0 msgr2=0x7fbae015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.164+0000 7fbad7e00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00ae5a0 0x7fbae015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.164+0000 7fbad7e00640 1 -- 172.21.15.125:0/3466496467 >> v1:172.21.15.125:6789/0 conn(0x7fbae006d120 legacy=0x7fbae00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.164+0000 7fbad7e00640 1 -- 172.21.15.125:0/3466496467 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae006c750 con 0x7fbae00b3ae0 2024-01-16T14:56:51.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.164+0000 7fbade000640 1 -- 172.21.15.125:0/3466496467 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079023523 0 0) 0x7fbae0070c40 con 0x7fbae006d120 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.165+0000 7fbad7e00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00b3ae0 0x7fbae006cbe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbac80097a0 tx=0x7fbac802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ef3ba5aab1027e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.165+0000 7fbade000640 1 -- 172.21.15.125:0/3466496467 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac802d040 con 0x7fbae00b3ae0 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.165+0000 7fbade000640 1 -- 172.21.15.125:0/3466496467 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8031810 con 0x7fbae00b3ae0 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.165+0000 7fbade000640 1 -- 172.21.15.125:0/3466496467 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8031ba0 con 0x7fbae00b3ae0 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 -- 172.21.15.125:0/3466496467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00b3ae0 msgr2=0x7fbae006cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00b3ae0 0x7fbae006cbe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbac80097a0 tx=0x7fbac802b8b0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 -- 172.21.15.125:0/3466496467 shutdown_connections 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00b3ae0 0x7fbae006cbe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 --2- 172.21.15.125:0/3466496467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00ae5a0 0x7fbae015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 -- 172.21.15.125:0/3466496467 >> 172.21.15.125:0/3466496467 conn(0x7fbae006a800 msgr2=0x7fbae006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 -- 172.21.15.125:0/3466496467 shutdown_connections 2024-01-16T14:56:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.166+0000 7fbae5a00640 1 -- 172.21.15.125:0/3466496467 wait complete. 2024-01-16T14:56:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.167+0000 7fbae5a00640 1 Processor -- start 2024-01-16T14:56:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 -- start start 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 0x7fbae00cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 0x7fbae01f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01f3eb0 con 0x7fbae00ae5a0 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00cb570 con 0x7fbae006d120 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.168+0000 7fbae5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae015cd00 con 0x7fbae00b3ae0 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47694/0 (socket says 172.21.15.125:47694) 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 -- 172.21.15.125:0/1076747313 learned_addr learned my addr 172.21.15.125:0/1076747313 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbadf400640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 0x7fbae01f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbadea00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 0x7fbae00cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 -- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 msgr2=0x7fbae01f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 0x7fbae01f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 -- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 msgr2=0x7fbae00cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 0x7fbae00cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 -- 172.21.15.125:0/1076747313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8009400 con 0x7fbae00ae5a0 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbadea00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 0x7fbae00cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbadf400640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 0x7fbae01f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:51.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.169+0000 7fbad7e00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbac8000c00 tx=0x7fbac80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.170+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac803ad60 con 0x7fbae00ae5a0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.170+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac80443f0 con 0x7fbae00ae5a0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.170+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac80447a0 con 0x7fbae00ae5a0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.170+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae0148dd0 con 0x7fbae00ae5a0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.170+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae015c460 con 0x7fbae00ae5a0 2024-01-16T14:56:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.171+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae0147840 con 0x7fbae00ae5a0 2024-01-16T14:56:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.172+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbac804d450 con 0x7fbae00ae5a0 2024-01-16T14:56:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.173+0000 7fbad6a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 0x7fbab806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.173+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbac80abf50 con 0x7fbae00ae5a0 2024-01-16T14:56:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.173+0000 7fbadea00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 0x7fbab806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.174+0000 7fbadea00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 0x7fbab806d780 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbad0006fd0 tx=0x7fbad0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.178+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbae0147840 con 0x7fbae00ae5a0 2024-01-16T14:56:51.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.416+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbae0147840 con 0x7fbae00ae5a0 2024-01-16T14:56:51.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.425+0000 7fbad6a00640 1 -- 172.21.15.125:0/1076747313 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171565 (secure 0 0 0) 0x7fbae0147840 con 0x7fbae00ae5a0 2024-01-16T14:56:51.407 INFO:tasks.ceph.mds.b.smithi137.stderr:2024-01-16T14:56:51.425+0000 7fa21c400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 msgr2=0x7fbab806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 0x7fbab806d780 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbad0006fd0 tx=0x7fbad0008040 comp rx=0 tx=0).stop 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 msgr2=0x7fbae00ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbac8000c00 tx=0x7fbac80046c0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 shutdown_connections 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbae00b3ae0 0x7fbae01f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbab806b2c0 0x7fbab806d780 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae00ae5a0 0x7fbae00ce550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 --2- 172.21.15.125:0/1076747313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbae006d120 0x7fbae00cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 >> 172.21.15.125:0/1076747313 conn(0x7fbae006a800 msgr2=0x7fbae006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 shutdown_connections 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.429+0000 7fbae5a00640 1 -- 172.21.15.125:0/1076747313 wait complete. 2024-01-16T14:56:51.428 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2670128691},{"type":"v1","addr":"172.21.15.125:6833","nonce":2670128691}]},"active_addr":"172.21.15.125:6833/2670128691","active_change":"2024-01-16T14:56:25.896577+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":3261756390}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":2499881277}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":1976106705}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":3836190261}]}]} 2024-01-16T14:56:51.431 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T14:56:51.432 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T14:56:51.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:51.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.652+0000 7f5708400640 1 Processor -- start 2024-01-16T14:56:51.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.652+0000 7f5708400640 1 -- start start 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.653+0000 7f5708400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f5700166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.653+0000 7f5708400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570015c5f0 con 0x7f570013df80 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.653+0000 7f5708400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570006c470 con 0x7f570013e360 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.653+0000 7f5708400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570006c750 con 0x7f57000cf9a0 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.653+0000 7f5705a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f570013df80 0x7f570015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36490/0 (socket says 172.21.15.125:36490) 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f5705a00640 1 -- 172.21.15.125:0/583377655 learned_addr learned my addr 172.21.15.125:0/583377655 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f5706400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f5700166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80895633 0 0) 0x7f570015c5f0 con 0x7f570013df80 2024-01-16T14:56:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e0003520 con 0x7f570013df80 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393542259 0 0) 0x7f56e0003520 con 0x7f570013df80 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 msgr2=0x7f5700166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f5700166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 >> v1:172.21.15.137:6789/0 conn(0x7f570013e360 legacy=0x7f57000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570006ca30 con 0x7f570013df80 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f56f00019f0 con 0x7f570013df80 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.654+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 >> v1:172.21.15.125:6789/0 conn(0x7f570013df80 legacy=0x7f570015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f56e00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e0003f20 0x7f56e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e0004d00 0x7f56e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570006c750 con 0x7f56e0004d00 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e000ba60 con 0x7f56e0003f20 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e000bc20 con 0x7f57000cf9a0 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e0004d00 0x7f56e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5705a00640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f56e00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5705000640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e0003f20 0x7f56e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 -- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 msgr2=0x7f56e00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57000cf9a0 0x7f56e00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 -- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e0003f20 msgr2=0x7f56e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e0003f20 0x7f56e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.655+0000 7f5706400640 1 -- 172.21.15.125:0/583377655 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e0003520 con 0x7f56e0004d00 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.656+0000 7f5706400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e0004d00 0x7f56e0005180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009a90 tx=0x7f56e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a82b7034d71776c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.656+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f802c680 con 0x7f56e0004d00 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f802c7e0 con 0x7f56e0004d00 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f56ffe00640 1 -- 172.21.15.125:0/583377655 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f802cb70 con 0x7f56e0004d00 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 -- 172.21.15.125:0/583377655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e0004d00 msgr2=0x7f56e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 --2- 172.21.15.125:0/583377655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e0004d00 0x7f56e0005180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009a90 tx=0x7f56e00057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5706400640 1 -- 172.21.15.125:0/583377655 reap_dead start 2024-01-16T14:56:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 -- 172.21.15.125:0/583377655 shutdown_connections 2024-01-16T14:56:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 -- 172.21.15.125:0/583377655 >> 172.21.15.125:0/583377655 conn(0x7f570006a800 msgr2=0x7f570006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 -- 172.21.15.125:0/583377655 shutdown_connections 2024-01-16T14:56:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.657+0000 7f5708400640 1 -- 172.21.15.125:0/583377655 wait complete. 2024-01-16T14:56:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.659+0000 7f5708400640 1 Processor -- start 2024-01-16T14:56:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.659+0000 7f5708400640 1 -- start start 2024-01-16T14:56:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e0004d00 0x7f570015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 0x7f570015c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570015c2e0 con 0x7f570015cb20 2024-01-16T14:56:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57001505a0 con 0x7f570015e750 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5708400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5700150b20 con 0x7f56e0004d00 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5706400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5706400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47722/0 (socket says 172.21.15.125:47722) 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.660+0000 7f5706400640 1 -- 172.21.15.125:0/3556231622 learned_addr learned my addr 172.21.15.125:0/3556231622 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5705a00640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e0004d00 0x7f570015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5705000640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 0x7f570015c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 -- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e0004d00 msgr2=0x7f570015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e0004d00 0x7f570015bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 -- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 msgr2=0x7f570015c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 0x7f570015c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 -- 172.21.15.125:0/3556231622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5700150840 con 0x7f570015cb20 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5705000640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 0x7f570015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:51.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5706400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009a90 tx=0x7f56f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f802c680 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8003830 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700151b40 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.661+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f802dea0 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.662+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57001525a0 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.663+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570015c450 con 0x7f570015cb20 2024-01-16T14:56:51.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.664+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f56f8003be0 con 0x7f570015cb20 2024-01-16T14:56:51.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.664+0000 7f56fd600640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 0x7f56d806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.664+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56f80ab430 con 0x7f570015cb20 2024-01-16T14:56:51.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.665+0000 7f5705a00640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 0x7f56d806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.665+0000 7f5705a00640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 0x7f56d806d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f56f0002010 tx=0x7f56f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.669+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56f8076290 con 0x7f570015cb20 2024-01-16T14:56:51.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.858+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5700151d60 con 0x7f570015cb20 2024-01-16T14:56:51.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.860+0000 7f56fd600640 1 -- 172.21.15.125:0/3556231622 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f56f80798b0 con 0x7f570015cb20 2024-01-16T14:56:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:51.843 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 msgr2=0x7f56d806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 0x7f56d806d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f56f0002010 tx=0x7f56f0009040 comp rx=0 tx=0).stop 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 msgr2=0x7f5700152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009a90 tx=0x7f56f8004730 comp rx=0 tx=0).stop 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 shutdown_connections 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56d806b2f0 0x7f56d806d7b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570015cb20 0x7f5700152060 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f570015e750 0x7f570015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 --2- 172.21.15.125:0/3556231622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e0004d00 0x7f570015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 >> 172.21.15.125:0/3556231622 conn(0x7f570006a800 msgr2=0x7f57000b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 shutdown_connections 2024-01-16T14:56:51.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:51.863+0000 7f5708400640 1 -- 172.21.15.125:0/3556231622 wait complete. 2024-01-16T14:56:51.863 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T14:56:51.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:52.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.085+0000 7f1f9e600640 1 Processor -- start 2024-01-16T14:56:52.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.085+0000 7f1f9e600640 1 -- start start 2024-01-16T14:56:52.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f9e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f9e600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1f9815e1e0 0x7f1f980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f9e600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f980b1ee0 con 0x7f1f98143ae0 2024-01-16T14:56:52.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f9e600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f980b2780 con 0x7f1f9815e1e0 2024-01-16T14:56:52.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f9e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9815e990 con 0x7f1f980cf9a0 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f96a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f96a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52272/0 (socket says 172.21.15.125:52272) 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f96a00640 1 -- 172.21.15.125:0/2102447373 learned_addr learned my addr 172.21.15.125:0/2102447373 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.087+0000 7f1f97e00640 1 --2- 172.21.15.125:0/2102447373 >> v2:172.21.15.137:3300/0 conn(0x7f1f9815e1e0 0x7f1f980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.088+0000 7f1f96a00640 1 -- 172.21.15.125:0/2102447373 >> v2:172.21.15.137:3300/0 conn(0x7f1f9815e1e0 msgr2=0x7f1f980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.088+0000 7f1f96a00640 1 --2- 172.21.15.125:0/2102447373 >> v2:172.21.15.137:3300/0 conn(0x7f1f9815e1e0 0x7f1f980702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.088+0000 7f1f96a00640 1 -- 172.21.15.125:0/2102447373 >> v1:172.21.15.137:6789/0 conn(0x7f1f98143ae0 legacy=0x7f1f980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.088+0000 7f1f96a00640 1 -- 172.21.15.125:0/2102447373 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980b2130 con 0x7f1f980cf9a0 2024-01-16T14:56:52.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.089+0000 7f1f96a00640 1 --2- 172.21.15.125:0/2102447373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009870 tx=0x7f1f8002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=314efc40abeaa205 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.090+0000 7f1f96000640 1 -- 172.21.15.125:0/2102447373 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80030840 con 0x7f1f980cf9a0 2024-01-16T14:56:52.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.090+0000 7f1f96000640 1 -- 172.21.15.125:0/2102447373 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f800309a0 con 0x7f1f980cf9a0 2024-01-16T14:56:52.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.090+0000 7f1f96000640 1 -- 172.21.15.125:0/2102447373 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80030d30 con 0x7f1f980cf9a0 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 -- 172.21.15.125:0/2102447373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 msgr2=0x7f1f9815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2102447373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009870 tx=0x7f1f8002b980 comp rx=0 tx=0).stop 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 -- 172.21.15.125:0/2102447373 shutdown_connections 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2102447373 >> v2:172.21.15.137:3300/0 conn(0x7f1f9815e1e0 0x7f1f980702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2102447373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f9815dca0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 -- 172.21.15.125:0/2102447373 >> 172.21.15.125:0/2102447373 conn(0x7f1f9806a800 msgr2=0x7f1f9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 -- 172.21.15.125:0/2102447373 shutdown_connections 2024-01-16T14:56:52.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.091+0000 7f1f9e600640 1 -- 172.21.15.125:0/2102447373 wait complete. 2024-01-16T14:56:52.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.093+0000 7f1f9e600640 1 Processor -- start 2024-01-16T14:56:52.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.093+0000 7f1f9e600640 1 -- start start 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f98159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 0x7f1f981f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980b06b0 con 0x7f1f98143ae0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980ac8d0 con 0x7f1f9815e1e0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f9e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f98159ed0 con 0x7f1f980cf9a0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47728/0 (socket says 172.21.15.125:47728) 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 -- 172.21.15.125:0/2400270361 learned_addr learned my addr 172.21.15.125:0/2400270361 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f97400640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f98159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f97e00640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 0x7f1f981f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 -- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 msgr2=0x7f1f98159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f98159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 -- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 msgr2=0x7f1f981f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 0x7f1f981f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f96a00640 1 -- 172.21.15.125:0/2400270361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800094d0 con 0x7f1f98143ae0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f97400640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f98159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.094+0000 7f1f97e00640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 0x7f1f981f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f96a00640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80039c90 con 0x7f1f98143ae0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f80039df0 con 0x7f1f98143ae0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80043640 con 0x7f1f98143ae0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f980afe10 con 0x7f1f98143ae0 2024-01-16T14:56:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.095+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f980b03d0 con 0x7f1f98143ae0 2024-01-16T14:56:52.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.096+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f980675d0 con 0x7f1f98143ae0 2024-01-16T14:56:52.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.097+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1f800437a0 con 0x7f1f98143ae0 2024-01-16T14:56:52.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.098+0000 7f1f8f400640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 0x7f1f7006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.098+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f8003e070 con 0x7f1f98143ae0 2024-01-16T14:56:52.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.098+0000 7f1f97400640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 0x7f1f7006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.099+0000 7f1f97400640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 0x7f1f7006d7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f88000f60 tx=0x7f1f88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.102+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f80077a60 con 0x7f1f98143ae0 2024-01-16T14:56:52.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.293+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f98140e20 con 0x7f1f98143ae0 2024-01-16T14:56:52.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.296+0000 7f1f8f400640 1 -- 172.21.15.125:0/2400270361 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f1f8007b080 con 0x7f1f98143ae0 2024-01-16T14:56:52.278 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:52.279 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 msgr2=0x7f1f7006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 0x7f1f7006d7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f88000f60 tx=0x7f1f88009040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 msgr2=0x7f1f980af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f80004730 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 shutdown_connections 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f9815e1e0 0x7f1f981f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f7006b2f0 0x7f1f7006d7b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f98143ae0 0x7f1f980af430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 --2- 172.21.15.125:0/2400270361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f980cf9a0 0x7f1f98159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 >> 172.21.15.125:0/2400270361 conn(0x7f1f9806a800 msgr2=0x7f1f9815c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 shutdown_connections 2024-01-16T14:56:52.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.299+0000 7f1f9e600640 1 -- 172.21.15.125:0/2400270361 wait complete. 2024-01-16T14:56:52.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T14:56:52.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T14:56:52.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T14:56:52.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-16T14:56:52.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-16T14:56:52.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-16T14:56:52.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-16T14:56:52.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-16T14:56:52.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.613+0000 7f9e23200640 1 Processor -- start 2024-01-16T14:56:52.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.613+0000 7f9e23200640 1 -- start start 2024-01-16T14:56:52.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.615+0000 7f9e23200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e1c03c070 0x7f9e1c0e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.615+0000 7f9e23200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c007b40 con 0x7f9e1c0e3130 2024-01-16T14:56:52.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.615+0000 7f9e23200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0587e0 con 0x7f9e1c053c80 2024-01-16T14:56:52.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.615+0000 7f9e23200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c058ac0 con 0x7f9e1c03c070 2024-01-16T14:56:52.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.616+0000 7f9e21e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9e1c0e3130 0x7f9e1c058020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36498/0 (socket says 172.21.15.125:36498) 2024-01-16T14:56:52.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.616+0000 7f9e21400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e1c03c070 0x7f9e1c0e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.616+0000 7f9e21400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e1c03c070 0x7f9e1c0e7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47742/0 (socket says 172.21.15.125:47742) 2024-01-16T14:56:52.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.616+0000 7f9e21400640 1 -- 172.21.15.125:0/1142819146 learned_addr learned my addr 172.21.15.125:0/1142819146 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.617+0000 7f9e21400640 1 -- 172.21.15.125:0/1142819146 >> v1:172.21.15.137:6789/0 conn(0x7f9e1c053c80 legacy=0x7f9e1c0540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.617+0000 7f9e21400640 1 -- 172.21.15.125:0/1142819146 >> v1:172.21.15.125:6789/0 conn(0x7f9e1c0e3130 legacy=0x7f9e1c058020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.618+0000 7f9e21400640 1 -- 172.21.15.125:0/1142819146 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c006330 con 0x7f9e1c03c070 2024-01-16T14:56:52.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.620+0000 7f9e21400640 1 --2- 172.21.15.125:0/1142819146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c0e7750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009870 tx=0x7f9e1402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779e47189223d986 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.620+0000 7f9e20a00640 1 -- 172.21.15.125:0/1142819146 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14030860 con 0x7f9e1c03c070 2024-01-16T14:56:52.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.621+0000 7f9e20a00640 1 -- 172.21.15.125:0/1142819146 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e140309c0 con 0x7f9e1c03c070 2024-01-16T14:56:52.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.621+0000 7f9e20a00640 1 -- 172.21.15.125:0/1142819146 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14030d50 con 0x7f9e1c03c070 2024-01-16T14:56:52.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7f9e23200640 1 -- 172.21.15.125:0/1142819146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 msgr2=0x7f9e1c0e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7f9e23200640 1 --2- 172.21.15.125:0/1142819146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c0e7750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009870 tx=0x7f9e1402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7f9e23200640 1 -- 172.21.15.125:0/1142819146 shutdown_connections 2024-01-16T14:56:52.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7f9e23200640 1 --2- 172.21.15.125:0/1142819146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c0e7750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7f9e23200640 1 -- 172.21.15.125:0/1142819146 >> 172.21.15.125:0/1142819146 conn(0x7f9e1c058eb0 msgr2=0x7f9e1c059290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.623+0000 7ff2fe600640 1 Processor -- start 2024-01-16T14:56:52.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7ff2fe600640 1 -- start start 2024-01-16T14:56:52.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7fafe6400640 1 Processor -- start 2024-01-16T14:56:52.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7f9e23200640 1 -- 172.21.15.125:0/1142819146 shutdown_connections 2024-01-16T14:56:52.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7fac5be00640 1 Processor -- start 2024-01-16T14:56:52.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7fafe6400640 1 -- start start 2024-01-16T14:56:52.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.625+0000 7fac5be00640 1 -- start start 2024-01-16T14:56:52.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.625+0000 7ff2fe600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff2f8153950 0x7ff2f80cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.624+0000 7f9e23200640 1 -- 172.21.15.125:0/1142819146 wait complete. 2024-01-16T14:56:52.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7ff2fe600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 0x7ff2f815d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7ff2fe600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f8153d30 con 0x7ff2f815ce90 2024-01-16T14:56:52.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7ff2fe600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f815d860 con 0x7ff2f8153950 2024-01-16T14:56:52.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7ff2fe600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80b2b70 con 0x7ff2f806d480 2024-01-16T14:56:52.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7fafe6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.626+0000 7fac5be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 0x7fac5c0d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac5be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac5c0ce9f0 0x7fac5c0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7ff2f7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff2f8153950 0x7ff2f80cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7f9e23200640 1 Processor -- start 2024-01-16T14:56:52.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fafe6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 0x7fafe00d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7ff2f7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff2f8153950 0x7ff2f80cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47746/0 (socket says 172.21.15.125:47746) 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7ff2f7400640 1 -- 172.21.15.125:0/2724005531 learned_addr learned my addr 172.21.15.125:0/2724005531 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7ff2f7e00640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 0x7ff2f815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac5be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c0b23c0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac5be00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0ce5d0 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac5be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0d0800 con 0x7fac5c0d0220 2024-01-16T14:56:52.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fafe6400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00afc60 con 0x7fafe00d1830 2024-01-16T14:56:52.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fafe6400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe00d3f70 con 0x7fafe00b1f10 2024-01-16T14:56:52.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fafe6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d1ca0 con 0x7fafe015d3b0 2024-01-16T14:56:52.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac53e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 0x7fac5c0d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac53e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 0x7fac5c0d0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45460/0 (socket says 172.21.15.125:45460) 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac53e00640 1 -- 172.21.15.125:0/1454949532 learned_addr learned my addr 172.21.15.125:0/1454949532 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.627+0000 7fac5b400640 1 --2- 172.21.15.125:0/1454949532 >> v2:172.21.15.125:3300/0 conn(0x7fac5c0ce9f0 0x7fac5c0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fafdf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fac5b400640 1 -- 172.21.15.125:0/1454949532 >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 msgr2=0x7fac5c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fac5b400640 1 --2- 172.21.15.125:0/1454949532 >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 0x7fac5c0d0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fac5b400640 1 -- 172.21.15.125:0/1454949532 >> v1:172.21.15.137:6789/0 conn(0x7fac5c15d3c0 legacy=0x7fac5c0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fafdf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52304/0 (socket says 172.21.15.125:52304) 2024-01-16T14:56:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fafdf400640 1 -- 172.21.15.125:0/303752538 learned_addr learned my addr 172.21.15.125:0/303752538 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7fac5b400640 1 -- 172.21.15.125:0/1454949532 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c0cdf10 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.629+0000 7fac5b400640 1 --2- 172.21.15.125:0/1454949532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0ce9f0 0x7fac5c0cee80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0a0fc0 tx=0x7fac5402bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f928e807f492fc06 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f7ae5200640 1 Processor -- start 2024-01-16T14:56:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7fac5a000640 1 -- 172.21.15.125:0/1454949532 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5402cb50 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7fafdea00640 1 --2- 172.21.15.125:0/303752538 >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 0x7fafe00d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7fac5a000640 1 -- 172.21.15.125:0/1454949532 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5402ccb0 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7fac5a000640 1 -- 172.21.15.125:0/1454949532 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54034660 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7f9e23200640 1 -- start start 2024-01-16T14:56:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c185560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e1c053c80 0x7f9e1c04ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e1c0e3130 0x7f9e1c0504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c052bf0 con 0x7f9e1c03c070 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c185aa0 con 0x7f9e1c053c80 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f9e23200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c050c30 con 0x7f9e1c0e3130 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7ff2f7400640 1 -- 172.21.15.125:0/2724005531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 msgr2=0x7ff2f815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7ff2f7400640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 0x7ff2f815d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7ff2f7400640 1 -- 172.21.15.125:0/2724005531 >> v1:172.21.15.137:6789/0 conn(0x7ff2f815ce90 legacy=0x7ff2f81598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.628+0000 7ff2f7400640 1 -- 172.21.15.125:0/2724005531 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80b2e50 con 0x7ff2f8153950 2024-01-16T14:56:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.629+0000 7ff2f7e00640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 0x7ff2f815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 -- 172.21.15.125:0/1454949532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0ce9f0 msgr2=0x7fac5c0cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 --2- 172.21.15.125:0/1454949532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0ce9f0 0x7fac5c0cee80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0a0fc0 tx=0x7fac5402bda0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fafde000640 1 -- 172.21.15.125:0/303752538 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811276334 0 0) 0x7fafe00afc60 con 0x7fafe00d1830 2024-01-16T14:56:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7ff2f7400640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f8153950 0x7ff2f80cb0d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00098a0 tx=0x7ff2e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7eee2f449343692 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7f9e21e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c185560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7f9e21e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c185560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47760/0 (socket says 172.21.15.125:47760) 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7f9e21e00640 1 -- 172.21.15.125:0/1529704530 learned_addr learned my addr 172.21.15.125:0/1529704530 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7f9e21400640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e1c053c80 0x7f9e1c04ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7f9e22800640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e1c0e3130 0x7f9e1c0504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 -- 172.21.15.125:0/1454949532 shutdown_connections 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 --2- 172.21.15.125:0/1454949532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0ce9f0 0x7fac5c0cee80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 --2- 172.21.15.125:0/1454949532 >> v2:172.21.15.137:3300/0 conn(0x7fac5c0d0220 0x7fac5c0d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 -- 172.21.15.125:0/1454949532 >> 172.21.15.125:0/1454949532 conn(0x7fac5c0d30b0 msgr2=0x7fac5c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 -- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e1c0e3130 msgr2=0x7f9e1c0504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e1c0e3130 0x7f9e1c0504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 -- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e1c053c80 msgr2=0x7f9e1c04ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e1c053c80 0x7f9e1c04ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140094d0 con 0x7f9e1c03c070 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7ff2f6000640 1 -- 172.21.15.125:0/2724005531 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0030840 con 0x7ff2f8153950 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7ff2f6000640 1 -- 172.21.15.125:0/2724005531 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e00309a0 con 0x7ff2f8153950 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7ff2f6000640 1 -- 172.21.15.125:0/2724005531 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0030840 con 0x7ff2f8153950 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 -- 172.21.15.125:0/1454949532 shutdown_connections 2024-01-16T14:56:52.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fac5be00640 1 -- 172.21.15.125:0/1454949532 wait complete. 2024-01-16T14:56:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fafdf400640 1 -- 172.21.15.125:0/303752538 >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 msgr2=0x7fafe00d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fafdf400640 1 --2- 172.21.15.125:0/303752538 >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 0x7fafe00d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fafdf400640 1 -- 172.21.15.125:0/303752538 >> v1:172.21.15.125:6789/0 conn(0x7fafe00d1830 legacy=0x7fafe00cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21400640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e1c053c80 0x7f9e1c04ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f9e21e00640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c185560 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c3e0 tx=0x7f9e0c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.630+0000 7f7ae5200640 1 -- start start 2024-01-16T14:56:52.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 0x7f7ae00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 0x7f7ae0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae014b6a0 con 0x7f7ae00cf7d0 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00cfbf0 con 0x7f7ae0070ea0 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.632+0000 7f7ae5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0071690 con 0x7f7ae014b2c0 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f7ad7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 0x7f7ae0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f7ade800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f7ade800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45488/0 (socket says 172.21.15.125:45488) 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f7ade800640 1 -- 172.21.15.125:0/2660076677 learned_addr learned my addr 172.21.15.125:0/2660076677 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7ff2fe600640 1 -- 172.21.15.125:0/2724005531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f8153950 msgr2=0x7ff2f80cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7ff2fe600640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f8153950 0x7ff2f80cb0d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00098a0 tx=0x7ff2e002b980 comp rx=0 tx=0).stop 2024-01-16T14:56:52.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f7adde00640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 0x7f7ae00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.633+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0074e0 con 0x7f9e1c03c070 2024-01-16T14:56:52.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7fac5be00640 1 Processor -- start 2024-01-16T14:56:52.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.631+0000 7fafdf400640 1 -- 172.21.15.125:0/303752538 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00ce7d0 con 0x7fafe015d3b0 2024-01-16T14:56:52.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7fac5be00640 1 -- start start 2024-01-16T14:56:52.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c007640 con 0x7f9e1c03c070 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0049e0 con 0x7f9e1c03c070 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c052910 con 0x7f9e1c03c070 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c0da5b0 con 0x7f9e1c03c070 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7fafdea00640 1 --2- 172.21.15.125:0/303752538 >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 0x7fafe00d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.634+0000 7fafdf400640 1 --2- 172.21.15.125:0/303752538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009870 tx=0x7fafd002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939021b768ed0873 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7f7ade800640 1 -- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 msgr2=0x7f7ae00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7f7ade800640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 0x7f7ae00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7f7ade800640 1 -- 172.21.15.125:0/2660076677 >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 msgr2=0x7f7ae0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7f7ade800640 1 --2- 172.21.15.125:0/2660076677 >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 0x7f7ae0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7f7ade800640 1 -- 172.21.15.125:0/2660076677 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae0071970 con 0x7f7ae0070ea0 2024-01-16T14:56:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fafde000640 1 -- 172.21.15.125:0/303752538 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0031870 con 0x7fafe015d3b0 2024-01-16T14:56:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fafde000640 1 -- 172.21.15.125:0/303752538 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd00319d0 con 0x7fafe015d3b0 2024-01-16T14:56:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fafde000640 1 -- 172.21.15.125:0/303752538 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0031d40 con 0x7fafe015d3b0 2024-01-16T14:56:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7ff2fe600640 1 -- 172.21.15.125:0/2724005531 shutdown_connections 2024-01-16T14:56:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7ff2fe600640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f806d480 0x7ff2f815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7ff2fe600640 1 --2- 172.21.15.125:0/2724005531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f8153950 0x7ff2f80cb0d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7ff2fe600640 1 -- 172.21.15.125:0/2724005531 >> 172.21.15.125:0/2724005531 conn(0x7ff2f80b0be0 msgr2=0x7ff2f80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e1c0e3510 con 0x7f9e1c03c070 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac5c0ce9f0 0x7fac5c1ff140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0d0220 0x7fac5c0b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 0x7fac5c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1ff680 con 0x7fac5c0d0220 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0b5cb0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fac5be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0cab40 con 0x7fac5c0ce9f0 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 0x7fac5c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 0x7fac5c0cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45500/0 (socket says 172.21.15.125:45500) 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 -- 172.21.15.125:0/3516626550 learned_addr learned my addr 172.21.15.125:0/3516626550 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 -- 172.21.15.125:0/303752538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 msgr2=0x7fafe00d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.635+0000 7fd7bdc00640 1 Processor -- start 2024-01-16T14:56:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 --2- 172.21.15.125:0/303752538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009870 tx=0x7fafd002b910 comp rx=0 tx=0).stop 2024-01-16T14:56:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7ff2fe600640 1 -- 172.21.15.125:0/2724005531 shutdown_connections 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac53e00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0d0220 0x7fac5c0b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 -- 172.21.15.125:0/3516626550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac5c0ce9f0 msgr2=0x7fac5c1ff140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac5c0ce9f0 0x7fac5c1ff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 -- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0d0220 msgr2=0x7fac5c0b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c0d0220 0x7fac5c0b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fac5b400640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54009a70 con 0x7fac5c15d3c0 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7f7adde00640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 0x7f7ae00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7f7ad7e00640 1 --2- 172.21.15.125:0/2660076677 >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 0x7f7ae0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 -- 172.21.15.125:0/303752538 shutdown_connections 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 --2- 172.21.15.125:0/303752538 >> v2:172.21.15.137:3300/0 conn(0x7fafe00b1f10 0x7fafe00d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 --2- 172.21.15.125:0/303752538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe015d3b0 0x7fafe00d3740 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fafe6400640 1 -- 172.21.15.125:0/303752538 >> 172.21.15.125:0/303752538 conn(0x7fafe009c370 msgr2=0x7fafe00d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7fd7bdc00640 1 -- start start 2024-01-16T14:56:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.637+0000 7f7ade800640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ae006e2a0 tx=0x7f7ad802bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9448e63c2ae5a4fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.637+0000 7fac5b400640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 0x7fac5c0cba90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac54009e80 tx=0x7fac54004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.637+0000 7f7add400640 1 -- 172.21.15.125:0/2660076677 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad802cac0 con 0x7f7ae0070ea0 2024-01-16T14:56:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.637+0000 7fafe6400640 1 -- 172.21.15.125:0/303752538 shutdown_connections 2024-01-16T14:56:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.637+0000 7fafe6400640 1 -- 172.21.15.125:0/303752538 wait complete. 2024-01-16T14:56:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.636+0000 7ff2fe600640 1 -- 172.21.15.125:0/2724005531 wait complete. 2024-01-16T14:56:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7f7add400640 1 -- 172.21.15.125:0/2660076677 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad802cc20 con 0x7f7ae0070ea0 2024-01-16T14:56:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7f7add400640 1 -- 172.21.15.125:0/2660076677 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad80343f0 con 0x7f7ae0070ea0 2024-01-16T14:56:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54040650 con 0x7fac5c15d3c0 2024-01-16T14:56:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac540407b0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54040b60 con 0x7fac5c15d3c0 2024-01-16T14:56:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c0cade0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c0cb3a0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fd7bdc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd7b80b4660 0x7fd7b80b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fd7bdc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 0x7fd7b80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fd7bdc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80ceb50 con 0x7fd7b8165560 2024-01-16T14:56:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fd7bdc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b80cf9b0 con 0x7fd7b80cfe40 2024-01-16T14:56:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.638+0000 7fd7bdc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b80d0830 con 0x7fd7b80b4660 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd7b80b4660 0x7fd7b80b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 0x7fd7b80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd7b80b4660 0x7fd7b80b60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45506/0 (socket says 172.21.15.125:45506) 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1116309902 learned_addr learned my addr 172.21.15.125:0/1116309902 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac2c000fc0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1116309902 >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 msgr2=0x7fd7b80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fafe6400640 1 Processor -- start 2024-01-16T14:56:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fafe6400640 1 -- start start 2024-01-16T14:56:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1116309902 >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 0x7fd7b80d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7ff2fe600640 1 Processor -- start 2024-01-16T14:56:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1116309902 >> v1:172.21.15.125:6789/0 conn(0x7fd7b8165560 legacy=0x7fd7b81659b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1116309902 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b80ceda0 con 0x7fd7b80b4660 2024-01-16T14:56:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7ff2fe600640 1 -- start start 2024-01-16T14:56:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.640+0000 7fd7b6000640 1 --2- 172.21.15.125:0/1116309902 >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 0x7fd7b80d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7f7ae5200640 1 -- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae0070ea0 msgr2=0x7f7ae00b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.639+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ae006e2a0 tx=0x7f7ad802bf20 comp rx=0 tx=0).stop 2024-01-16T14:56:52.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9e0c004b40 con 0x7f9e1c03c070 2024-01-16T14:56:52.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f7ae5200640 1 -- 172.21.15.125:0/2660076677 shutdown_connections 2024-01-16T14:56:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe00b1f10 0x7fafe0159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1116309902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80b4660 0x7fd7b80b60a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7b806a880 tx=0x7fd7b002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11613cf2484c0914 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafe00d1830 0x7fafe015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafe015d3b0 0x7fafe00b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe015a400 con 0x7fafe00d1830 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe015f020 con 0x7fafe015d3b0 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafe6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe015e450 con 0x7fafe00b1f10 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fd7b5600640 1 -- 172.21.15.125:0/1116309902 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b002cb10 con 0x7fd7b80b4660 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fd7b5600640 1 -- 172.21.15.125:0/1116309902 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b002cc70 con 0x7fd7b80b4660 2024-01-16T14:56:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fd7b5600640 1 -- 172.21.15.125:0/1116309902 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00343f0 con 0x7fd7b80b4660 2024-01-16T14:56:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafdf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe00b1f10 0x7fafe0159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafdfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafe015d3b0 0x7fafe00b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafdfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafe015d3b0 0x7fafe00b7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45508/0 (socket says 172.21.15.125:45508) 2024-01-16T14:56:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafdfe00640 1 -- 172.21.15.125:0/19261304 learned_addr learned my addr 172.21.15.125:0/19261304 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fafdea00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafe00d1830 0x7fafe015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7f9e1a000640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9dfc06b390 0x7f9dfc06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e0c089490 con 0x7f9e1c03c070 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7f9e1a000640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f9dfc0735b0 0x7f9dfc0759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9dfc076080 con 0x7f9dfc0735b0 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e0c08bab0 con 0x7f9e1c03c070 2024-01-16T14:56:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7ff2fe600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 0x7ff2f81f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.642+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac54040cc0 con 0x7fac5c15d3c0 2024-01-16T14:56:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1116309902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80b4660 msgr2=0x7fd7b80b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7ff2fe600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f8153950 0x7ff2f8152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7ff2fe600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 0x7ff2f81fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdea00640 1 -- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe00b1f10 msgr2=0x7fafe0159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdea00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe00b1f10 0x7fafe0159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdea00640 1 -- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafe015d3b0 msgr2=0x7fafe00b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdea00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafe015d3b0 0x7fafe00b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdea00640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00094d0 con 0x7fafe00d1830 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fafdf400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafe00b1f10 0x7fafe0159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafdea00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafe00d1830 0x7fafe015f4b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc8006fd0 tx=0x7fafc800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.643+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1116309902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80b4660 0x7fd7b80b60a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7b806a880 tx=0x7fd7b002bf70 comp rx=0 tx=0).stop 2024-01-16T14:56:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1116309902 shutdown_connections 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1116309902 >> v2:172.21.15.125:3300/0 conn(0x7fd7b80cfe40 0x7fd7b80d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1116309902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80b4660 0x7fd7b80b60a0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7b806a880 tx=0x7fd7b002bf70 comp rx=0 tx=0).stop 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1116309902 >> 172.21.15.125:0/1116309902 conn(0x7fd7b809c560 msgr2=0x7fd7b80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7f9e21400640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9dfc06b390 0x7f9dfc06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7f9e22800640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f9dfc0735b0 0x7f9dfc0759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fac52a00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac3406b2c0 0x7fac3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac540acf60 con 0x7fac5c15d3c0 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fac52a00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7fac340734e0 0x7fac34075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1116309902 shutdown_connections 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1116309902 wait complete. 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae0070ea0 0x7f7ae00b1ee0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2660076677 >> v2:172.21.15.125:3300/0 conn(0x7f7ae00cf7d0 0x7f7ae0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2660076677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae014b2c0 0x7f7ae00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.641+0000 7f7ae5200640 1 -- 172.21.15.125:0/2660076677 >> 172.21.15.125:0/2660076677 conn(0x7f7ae006a800 msgr2=0x7f7ae006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac34075fb0 con 0x7fac340734e0 2024-01-16T14:56:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac540ad310 con 0x7fac5c15d3c0 2024-01-16T14:56:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7f7ae5200640 1 -- 172.21.15.125:0/2660076677 shutdown_connections 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7fac5aa00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac3406b2c0 0x7fac3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800c6d0 con 0x7fafe00d1830 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc800c830 con 0x7fafe00d1830 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800cba0 con 0x7fafe00d1830 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe015e6f0 con 0x7fafe00d1830 2024-01-16T14:56:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe015ecb0 con 0x7fafe00d1830 2024-01-16T14:56:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7ff2f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 0x7ff2f81f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7fac53e00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7fac340734e0 0x7fac34075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.644+0000 7ff2f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 0x7ff2f81f7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47788/0 (socket says 172.21.15.125:47788) 2024-01-16T14:56:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7f9e22800640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f9dfc0735b0 0x7f9dfc0759d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 0x7ff2f81fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 0x7ff2f81fa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45520/0 (socket says 172.21.15.125:45520) 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2fe600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80cec30 con 0x7ff2f806d480 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2fe600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f8153180 con 0x7ff2f815ce90 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2fe600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80ce950 con 0x7ff2f8153950 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7ff2f7400640 1 -- 172.21.15.125:0/3485516978 learned_addr learned my addr 172.21.15.125:0/3485516978 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7ff2f7e00640 1 -- 172.21.15.125:0/3485516978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f8153950 msgr2=0x7ff2f8152c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7ff2f7e00640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff2f8153950 0x7ff2f8152c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7ff2f7e00640 1 -- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 msgr2=0x7ff2f81f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7fac5aa00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac3406b2c0 0x7fac3406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7ff2f7e00640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 0x7ff2f81f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7ff2f7e00640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0009a70 con 0x7ff2f815ce90 2024-01-16T14:56:52.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== osd.0 v2:172.21.15.125:6800/3892726325 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9dfc076080 con 0x7f9dfc0735b0 2024-01-16T14:56:52.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7ff2f7400640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f806d480 0x7ff2f81f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.645+0000 7f7ae5200640 1 -- 172.21.15.125:0/2660076677 wait complete. 2024-01-16T14:56:52.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7f7ae5200640 1 Processor -- start 2024-01-16T14:56:52.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.646+0000 7fac53e00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7fac340734e0 0x7fac34075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.647+0000 7fd7bdc00640 1 Processor -- start 2024-01-16T14:56:52.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2f7e00640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 0x7ff2f81fa280 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000ef90 tx=0x7ff2f000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== osd.2 v2:172.21.15.125:6810/1951203061 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fac34075fb0 con 0x7fac340734e0 2024-01-16T14:56:52.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7f7ae5200640 1 -- start start 2024-01-16T14:56:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7fd7bdc00640 1 -- start start 2024-01-16T14:56:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000cd30 con 0x7ff2f815ce90 2024-01-16T14:56:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f000f040 con 0x7ff2f815ce90 2024-01-16T14:56:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0003bd0 con 0x7ff2f815ce90 2024-01-16T14:56:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e00094d0 con 0x7ff2f815ce90 2024-01-16T14:56:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7f9e21400640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9dfc06b390 0x7f9dfc06d850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036000 tx=0x7f9e14008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.648+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f80ccc70 con 0x7ff2f815ce90 2024-01-16T14:56:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.649+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafe00d3c80 con 0x7fafe00d1830 2024-01-16T14:56:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2f80b36f0 con 0x7ff2f815ce90 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7f7ae5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ae0070ea0 0x7f7ae0137a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7f7ae5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae00cf7d0 0x7f7ae0164a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7f7ae5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae014b2c0 0x7f7ae00aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ae5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0131130 con 0x7f7ae0070ea0 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ae5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00a8b70 con 0x7f7ae014b2c0 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ae5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0131660 con 0x7f7ae00cf7d0 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ade800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae014b2c0 0x7f7ae00aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ad7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae00cf7d0 0x7f7ae0164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ad7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae00cf7d0 0x7f7ae0164a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52354/0 (socket says 172.21.15.125:52354) 2024-01-16T14:56:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7f7ad7e00640 1 -- 172.21.15.125:0/2168107704 learned_addr learned my addr 172.21.15.125:0/2168107704 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7adde00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ae0070ea0 0x7f7ae0137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7ade800640 1 -- 172.21.15.125:0/2168107704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae00cf7d0 msgr2=0x7f7ae0164a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7ade800640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ae00cf7d0 0x7f7ae0164a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7ade800640 1 -- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ae0070ea0 msgr2=0x7f7ae0137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fafc800cd00 con 0x7fafe00d1830 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fafd7400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafb806b2c0 0x7fafb806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafc80923a0 con 0x7fafe00d1830 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7fd7bdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7b80cfe40 0x7fd7b815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.650+0000 7fd7bdc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7b8165560 0x7fd7b80cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7bdc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80c9870 0x7fd7b80b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7bdc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80b6820 con 0x7fd7b8165560 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7bdc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b815ac10 con 0x7fd7b80c9870 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7b6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7b8165560 0x7fd7b80cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7bdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80b7360 con 0x7fd7b80cfe40 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7b6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7b8165560 0x7fd7b80cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47812/0 (socket says 172.21.15.125:47812) 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.651+0000 7fd7b6000640 1 -- 172.21.15.125:0/1270755110 learned_addr learned my addr 172.21.15.125:0/1270755110 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7b80cfe40 0x7fd7b815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fd7b7400640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80c9870 0x7fd7b80b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fd7c4a00640 1 Processor -- start 2024-01-16T14:56:52.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fd7c4a00640 1 -- start start 2024-01-16T14:56:52.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fe52d200640 1 Processor -- start 2024-01-16T14:56:52.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7fe52d200640 1 -- start start 2024-01-16T14:56:52.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7ade800640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ae0070ea0 0x7f7ae0137a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.652+0000 7f7ade800640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac8009400 con 0x7f7ae014b2c0 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7f7adde00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ae0070ea0 0x7f7ae0137a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7f7ade800640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae014b2c0 0x7f7ae00aff00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8032000 tx=0x7f7ad8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff2f0014480 con 0x7ff2f815ce90 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7ff2ef400640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff2d006b2c0 0x7ff2d006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2f008b1b0 con 0x7ff2f815ce90 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fafdf400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafb806b2c0 0x7fafb806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fafd7400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7fafb8073460 0x7fafb8075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafb8075f30 con 0x7fafb8073460 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafc8092720 con 0x7fafe00d1830 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fe52d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe5280ced30 0x7fe5280b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fe52d200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280cf1c0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fe52d200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280d0de0 con 0x7fe528160f80 2024-01-16T14:56:52.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fe52d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280d10c0 con 0x7fe5280ced30 2024-01-16T14:56:52.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad80094d0 con 0x7f7ae014b2c0 2024-01-16T14:56:52.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fd7c4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd7bc160f80 0x7fd7bc161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7ff2f7400640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff2d006b2c0 0x7ff2d006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad8009b30 con 0x7f7ae014b2c0 2024-01-16T14:56:52.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae0131380 con 0x7f7ae014b2c0 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fe525e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe528160f80 0x7fe528161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55276/0 (socket says 172.21.15.125:55276) 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fe525e00640 1 -- 172.21.15.125:0/1426547222 learned_addr learned my addr 172.21.15.125:0/1426547222 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c4a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc1618a0 con 0x7fd7bc0ce8f0 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0d0a00 con 0x7fd7bc0ced50 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7bc0cf1e0 con 0x7fd7bc160f80 2024-01-16T14:56:52.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fafdfe00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7fafb8073460 0x7fafb8075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fe526800640 1 --2- 172.21.15.125:0/1426547222 >> v2:172.21.15.137:3300/0 conn(0x7fe5280ced30 0x7fe5280b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c1600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd7bc0ce8f0 0x7fd7bc0cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36534/0 (socket says 172.21.15.125:36534) 2024-01-16T14:56:52.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c1600640 1 -- 172.21.15.125:0/2863404220 learned_addr learned my addr 172.21.15.125:0/2863404220 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:52.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7c2000640 1 --2- 172.21.15.125:0/2863404220 >> v2:172.21.15.137:3300/0 conn(0x7fd7bc160f80 0x7fd7bc161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fafdf400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafb806b2c0 0x7fafb806d780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fafd0036000 tx=0x7fafd0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80c9870 msgr2=0x7fd7b80b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7b80c9870 0x7fd7b80b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7b8165560 msgr2=0x7fd7b80cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7b8165560 0x7fd7b80cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.653+0000 7fd7b6a00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8009400 con 0x7fd7b80cfe40 2024-01-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fd7b6a00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7b80cfe40 0x7fd7b815a6d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0032000 tx=0x7fd7b0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b003e450 con 0x7fd7b80cfe40 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.654+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b0009b30 con 0x7fd7b80cfe40 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b80b6d10 con 0x7fd7b80cfe40 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b003e5d0 con 0x7fd7b80cfe40 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b003e980 con 0x7fd7b80cfe40 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.655+0000 7ff2ef400640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7ff2d0073460 0x7ff2d0075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2d0075f30 con 0x7ff2d0073460 2024-01-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2f008b560 con 0x7ff2f815ce90 2024-01-16T14:56:52.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad8004250 con 0x7f7ae014b2c0 2024-01-16T14:56:52.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ae012e690 con 0x7f7ae014b2c0 2024-01-16T14:56:52.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7ff2f7400640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff2d006b2c0 0x7ff2d006d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00097f0 tx=0x7ff2e003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8004640 con 0x7f7ae014b2c0 2024-01-16T14:56:52.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fd7c2000640 1 -- 172.21.15.125:0/2863404220 >> v1:172.21.15.137:6789/0 conn(0x7fd7bc0ced50 legacy=0x7fd7bc0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fd7c2000640 1 -- 172.21.15.125:0/2863404220 >> v1:172.21.15.125:6789/0 conn(0x7fd7bc0ce8f0 legacy=0x7fd7bc0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fd7c2000640 1 -- 172.21.15.125:0/2863404220 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0d0de0 con 0x7fd7bc160f80 2024-01-16T14:56:52.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fafdfe00640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7fafb8073460 0x7fafb8075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.657+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd788000fc0 con 0x7fd7b80cfe40 2024-01-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.657+0000 7ff2f6a00640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7ff2d0073460 0x7ff2d0075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.657+0000 7fe526800640 1 -- 172.21.15.125:0/1426547222 >> v1:172.21.15.137:6789/0 conn(0x7fe528160f80 legacy=0x7fe528161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.656+0000 7fe524a00640 1 -- 172.21.15.125:0/1426547222 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441966703 0 0) 0x7fe5280d0de0 con 0x7fe528160f80 2024-01-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.657+0000 7fe526800640 1 -- 172.21.15.125:0/1426547222 >> v1:172.21.15.125:6789/0 conn(0x7fe5280ce8f0 legacy=0x7fe5280b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.657+0000 7fe526800640 1 -- 172.21.15.125:0/1426547222 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280d13a0 con 0x7fe5280ced30 2024-01-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.658+0000 7ff2f6a00640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7ff2d0073460 0x7ff2d0075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.658+0000 7fd7c2000640 1 --2- 172.21.15.125:0/2863404220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc160f80 0x7fd7bc161360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ad3dc9d6b5a41 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.659+0000 7fd7c0c00640 1 -- 172.21.15.125:0/2863404220 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4031830 con 0x7fd7bc160f80 2024-01-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.659+0000 7fd7c0c00640 1 -- 172.21.15.125:0/2863404220 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a4031990 con 0x7fd7bc160f80 2024-01-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.659+0000 7fd7c0c00640 1 -- 172.21.15.125:0/2863404220 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4031d20 con 0x7fd7bc160f80 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.659+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== osd.1 v2:172.21.15.125:6808/2568561981 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff2d0075f30 con 0x7ff2d0073460 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.659+0000 7fe526800640 1 --2- 172.21.15.125:0/1426547222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280b3110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe520009d90 tx=0x7fe52002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc69473d96104f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 -- 172.21.15.125:0/2863404220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc160f80 msgr2=0x7fd7bc161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 --2- 172.21.15.125:0/2863404220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc160f80 0x7fd7bc161360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 -- 172.21.15.125:0/2863404220 shutdown_connections 2024-01-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 --2- 172.21.15.125:0/2863404220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc160f80 0x7fd7bc161360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fe524a00640 1 -- 172.21.15.125:0/1426547222 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52002cab0 con 0x7fe5280ced30 2024-01-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 -- 172.21.15.125:0/2863404220 >> 172.21.15.125:0/2863404220 conn(0x7fd7bc09c550 msgr2=0x7fd7bc0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fe524a00640 1 -- 172.21.15.125:0/1426547222 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52002cc10 con 0x7fe5280ced30 2024-01-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fe524a00640 1 -- 172.21.15.125:0/1426547222 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520034440 con 0x7fe5280ced30 2024-01-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7fd7c4a00640 1 -- 172.21.15.125:0/2863404220 shutdown_connections 2024-01-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.661+0000 7fd7c4a00640 1 -- 172.21.15.125:0/2863404220 wait complete. 2024-01-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.661+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd7b003eae0 con 0x7fd7b80cfe40 2024-01-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.660+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7ad803e450 con 0x7f7ae014b2c0 2024-01-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.662+0000 7f7ad6000640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ab806b2c0 0x7f7ab806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.662+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ad80acb00 con 0x7f7ae014b2c0 2024-01-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.662+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== osd.4 v2:172.21.15.137:6808/1439911452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fafb8075f30 con 0x7fafb8073460 2024-01-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.662+0000 7f7ad6000640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f7ab80734e0 0x7f7ab8075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.662+0000 7fd7c4a00640 1 Processor -- start 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.661+0000 7fe52d200640 1 -- 172.21.15.125:0/1426547222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 msgr2=0x7fe5280b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.661+0000 7fe52d200640 1 --2- 172.21.15.125:0/1426547222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280b3110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe520009d90 tx=0x7fe52002bd80 comp rx=0 tx=0).stop 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ab8075fb0 con 0x7f7ab80734e0 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7fe52d200640 1 -- 172.21.15.125:0/1426547222 shutdown_connections 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7fe52d200640 1 --2- 172.21.15.125:0/1426547222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280b3110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7fe52d200640 1 -- 172.21.15.125:0/1426547222 >> 172.21.15.125:0/1426547222 conn(0x7fe52809c550 msgr2=0x7fe5280d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7f7adde00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ab806b2c0 0x7f7ab806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ad80aceb0 con 0x7f7ae014b2c0 2024-01-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7f7ad7e00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f7ab80734e0 0x7f7ab8075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fe52d200640 1 -- 172.21.15.125:0/1426547222 shutdown_connections 2024-01-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fe52d200640 1 -- 172.21.15.125:0/1426547222 wait complete. 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.663+0000 7fd7c4a00640 1 -- start start 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7f7ad7e00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f7ab80734e0 0x7f7ab8075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc0ce8f0 0x7fd7bc0cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7bc0ced50 0x7fd7bc0c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7bc160f80 0x7fd7bc147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc15f820 con 0x7fd7bc160f80 2024-01-16T14:56:52.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0cc320 con 0x7fd7bc0ce8f0 2024-01-16T14:56:52.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.664+0000 7fd7c4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc15f160 con 0x7fd7bc0ced50 2024-01-16T14:56:52.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7f7adde00640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ab806b2c0 0x7f7ab806d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0006fd0 tx=0x7f7ad0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7fd7c2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc0ce8f0 0x7fd7bc0cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7fd7c2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc0ce8f0 0x7fd7bc0cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45564/0 (socket says 172.21.15.125:45564) 2024-01-16T14:56:52.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7fd7c2000640 1 -- 172.21.15.125:0/3126105157 learned_addr learned my addr 172.21.15.125:0/3126105157 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7fd7aea00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79006b2f0 0x7fd79006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.665+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7b00ad220 con 0x7fd7b80cfe40 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fd7aea00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7fd790073510 0x7fd790075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd790075fe0 con 0x7fd790073510 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b00ad5d0 con 0x7fd7b80cfe40 2024-01-16T14:56:52.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fd7c2a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7bc160f80 0x7fd7bc147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fd7c1600640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7bc0ced50 0x7fd7bc0c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== osd.3 v2:172.21.15.125:6820/397977967 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7ab8075fb0 con 0x7f7ab80734e0 2024-01-16T14:56:52.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.666+0000 7fe52d200640 1 Processor -- start 2024-01-16T14:56:52.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fe52d200640 1 -- start start 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 -- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7bc0ced50 msgr2=0x7fd7bc0c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7bc0ced50 0x7fd7bc0c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 -- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc0ce8f0 msgr2=0x7fd7bc0cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7bc0ce8f0 0x7fd7bc0cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4009400 con 0x7fd7bc160f80 2024-01-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c1600640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7bc0ced50 0x7fd7bc0c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7b6000640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79006b2f0 0x7fd79006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7c2a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7bc160f80 0x7fd7bc147e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c780 tx=0x7fd7b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.667+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400aa50 con 0x7fd7bc160f80 2024-01-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc15f440 con 0x7fd7bc160f80 2024-01-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400abb0 con 0x7fd7bc160f80 2024-01-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7b7400640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7fd790073510 0x7fd790075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc15ed80 con 0x7fd7bc160f80 2024-01-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400ce30 con 0x7fd7bc160f80 2024-01-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5280ce8f0 0x7fe5280b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe528160f80 0x7fe5280b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280c9a80 con 0x7fe528160f80 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280c9d60 con 0x7fe5280ced30 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fe52d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52815d6c0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.668+0000 7fd7b6000640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79006b2f0 0x7fd79006d7b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7b81fa850 tx=0x7fd7a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.669+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7bc0d1680 con 0x7fd7bc160f80 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.669+0000 7fe525400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.669+0000 7fe525400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45576/0 (socket says 172.21.15.125:45576) 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.669+0000 7fe525400640 1 -- 172.21.15.125:0/3709273521 learned_addr learned my addr 172.21.15.125:0/3709273521 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.669+0000 7fe525e00640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5280ce8f0 0x7fe5280b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe526800640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe528160f80 0x7fe5280b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fd7b7400640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7fd790073510 0x7fd790075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe525e00640 1 -- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 msgr2=0x7fe5280c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe525e00640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5280ced30 0x7fe5280c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe525e00640 1 -- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe528160f80 msgr2=0x7fe5280b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe525e00640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe528160f80 0x7fe5280b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.670+0000 7fe525e00640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520009a70 con 0x7fe5280ce8f0 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== osd.5 v2:172.21.15.137:6800/2884121150 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd790075fe0 con 0x7fd790073510 2024-01-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.671+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd7b4012420 con 0x7fd7bc160f80 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7ba000640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79406b2c0 0x7fd79406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7b408a4c0 con 0x7fd7bc160f80 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7ba000640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7fd794073460 0x7fd794075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd794075f30 con 0x7fd794073460 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b408a910 con 0x7fd7bc160f80 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.672+0000 7fe525e00640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5280ce8f0 0x7fe5280b8c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c3e0 tx=0x7fe51800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.673+0000 7fd7c2000640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79406b2c0 0x7fd79406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.673+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180074e0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.673+0000 7fd7c1600640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7fd794073460 0x7fd794075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.673+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518007640 con 0x7fe5280ce8f0 2024-01-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.673+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518004a20 con 0x7fe5280ce8f0 2024-01-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.674+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52815d9a0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.674+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5280c7fc0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.674+0000 7fd7c1600640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7fd794073460 0x7fd794075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.677+0000 7fd7c2000640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79406b2c0 0x7fd79406d780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd7a402b910 tx=0x7fd7a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.677+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== osd.6 v2:172.21.15.137:6816/1720012579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd794075f30 con 0x7fd794073460 2024-01-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.678+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe5280d2dc0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.678+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 --> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac2c002cb0 con 0x7fac340734e0 2024-01-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.679+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe518004b80 con 0x7fe5280ce8f0 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.679+0000 7fac52a00640 1 -- 172.21.15.125:0/3516626550 <== osd.2 v2:172.21.15.125:6810/1951203061 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac2c002cb0 con 0x7fac340734e0 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.680+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7fac340734e0 msgr2=0x7fac34075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.680+0000 7fac5be00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7fac340734e0 0x7fac34075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac3406b2c0 msgr2=0x7fac3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.680+0000 7fe51e000640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50006b2c0 0x7fe50006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.680+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe518089a40 con 0x7fe5280ce8f0 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fe51e000640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fe5000734e0 0x7fe500075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fe525400640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50006b2c0 0x7fe50006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe500075fb0 con 0x7fe5000734e0 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe51808e2a0 con 0x7fe5280ce8f0 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac3406b2c0 0x7fac3406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c006bc0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 msgr2=0x7fac5c0cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 --2- 172.21.15.125:0/3516626550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac5c15d3c0 0x7fac5c0cba90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac54009e80 tx=0x7fac54004a40 comp rx=0 tx=0).stop 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5b400640 1 -- 172.21.15.125:0/3516626550 reap_dead start 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 shutdown_connections 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 >> 172.21.15.125:0/3516626550 conn(0x7fac5c0d30b0 msgr2=0x7fac5c06c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 shutdown_connections 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.681+0000 7fac5be00640 1 -- 172.21.15.125:0/3516626550 wait complete. 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.682+0000 7fe526800640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fe5000734e0 0x7fe500075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.683+0000 7fe525400640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50006b2c0 0x7fe50006d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe52815d0d0 tx=0x7fe510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.683+0000 7fe526800640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fe5000734e0 0x7fe500075900 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.685+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== osd.7 v2:172.21.15.137:6817/1873161633 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe500075fb0 con 0x7fe5000734e0 2024-01-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.685+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 --> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e1c0da130 con 0x7f9dfc0735b0 2024-01-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.686+0000 7f9e1a000640 1 -- 172.21.15.125:0/1529704530 <== osd.0 v2:172.21.15.125:6800/3892726325 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e1c0da130 con 0x7f9dfc0735b0 2024-01-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f9dfc0735b0 msgr2=0x7f9dfc0759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f9dfc0735b0 0x7f9dfc0759d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9dfc06b390 msgr2=0x7f9dfc06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9dfc06b390 0x7f9dfc06d850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036000 tx=0x7f9e14008030 comp rx=0 tx=0).stop 2024-01-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 msgr2=0x7f9e1c185560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7f9e23200640 1 --2- 172.21.15.125:0/1529704530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e1c03c070 0x7f9e1c185560 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c3e0 tx=0x7f9e0c00c7e0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.689+0000 7f9e22800640 1 -- 172.21.15.125:0/1529704530 reap_dead start 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.689+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 shutdown_connections 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.689+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 >> 172.21.15.125:0/1529704530 conn(0x7f9e1c058eb0 msgr2=0x7f9e1c0e6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.689+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 shutdown_connections 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.689+0000 7f9e23200640 1 -- 172.21.15.125:0/1529704530 wait complete. 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.688+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 --> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2f80b3410 con 0x7ff2d0073460 2024-01-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.690+0000 7ff2ef400640 1 -- 172.21.15.125:0/3485516978 <== osd.1 v2:172.21.15.125:6808/2568561981 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2e8001120 con 0x7ff2d0073460 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7ff2d0073460 msgr2=0x7ff2d0075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7ff2d0073460 0x7ff2d0075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff2d006b2c0 msgr2=0x7ff2d006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff2d006b2c0 0x7ff2d006d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00097f0 tx=0x7ff2e003b040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 msgr2=0x7ff2f81fa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.691+0000 7ff2fe600640 1 --2- 172.21.15.125:0/3485516978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff2f815ce90 0x7ff2f81fa280 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000ef90 tx=0x7ff2f000c490 comp rx=0 tx=0).stop 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.692+0000 7ff2f7e00640 1 -- 172.21.15.125:0/3485516978 reap_dead start 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.692+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 shutdown_connections 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.692+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 >> 172.21.15.125:0/3485516978 conn(0x7ff2f80b0be0 msgr2=0x7ff2f814af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.692+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 shutdown_connections 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.692+0000 7ff2fe600640 1 -- 172.21.15.125:0/3485516978 wait complete. 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.697+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 --> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafe00ced90 con 0x7fafb8073460 2024-01-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.698+0000 7fafd7400640 1 -- 172.21.15.125:0/19261304 <== osd.4 v2:172.21.15.137:6808/1439911452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fafd8001bd0 con 0x7fafb8073460 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.698+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 --> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd788002cb0 con 0x7fd790073510 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.699+0000 7fd7aea00640 1 -- 172.21.15.125:0/1270755110 <== osd.5 v2:172.21.15.137:6800/2884121150 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd788002cb0 con 0x7fd790073510 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 --> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ae00cbc40 con 0x7f7ab80734e0 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7fafb8073460 msgr2=0x7fafb8075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7fafb8073460 0x7fafb8075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafb806b2c0 msgr2=0x7fafb806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafb806b2c0 0x7fafb806d780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fafd0036000 tx=0x7fafd0008030 comp rx=0 tx=0).stop 2024-01-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafe00d1830 msgr2=0x7fafe015f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.700+0000 7fafe6400640 1 --2- 172.21.15.125:0/19261304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafe00d1830 0x7fafe015f4b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc8006fd0 tx=0x7fafc800ec70 comp rx=0 tx=0).stop 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fafdfe00640 1 -- 172.21.15.125:0/19261304 reap_dead start 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 shutdown_connections 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 >> 172.21.15.125:0/19261304 conn(0x7fafe009c370 msgr2=0x7fafe015b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 shutdown_connections 2024-01-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fafe6400640 1 -- 172.21.15.125:0/19261304 wait complete. 2024-01-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7f7ad6000640 1 -- 172.21.15.125:0/2168107704 <== osd.3 v2:172.21.15.125:6820/397977967 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ac8009e80 con 0x7f7ab80734e0 2024-01-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7fd790073510 msgr2=0x7fd790075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.701+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7fd790073510 0x7fd790075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79006b2f0 msgr2=0x7fd79006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79006b2f0 0x7fd79006d7b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7b81fa850 tx=0x7fd7a8008040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f7ab80734e0 msgr2=0x7f7ab8075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7b80cfe40 msgr2=0x7fd7b815a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 --2- 172.21.15.125:0/1270755110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7b80cfe40 0x7fd7b815a6d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0032000 tx=0x7fd7b0004a40 comp rx=0 tx=0).stop 2024-01-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f7ab80734e0 0x7f7ab8075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7b7400640 1 -- 172.21.15.125:0/1270755110 reap_dead start 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 shutdown_connections 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 >> 172.21.15.125:0/1270755110 conn(0x7fd7b809c560 msgr2=0x7fd7b80c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 shutdown_connections 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.703+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ab806b2c0 msgr2=0x7f7ab806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.702+0000 7fd7bdc00640 1 -- 172.21.15.125:0/1270755110 wait complete. 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.703+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ab806b2c0 0x7f7ab806d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0006fd0 tx=0x7f7ad0008040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.703+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae014b2c0 msgr2=0x7f7ae00aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.703+0000 7f7ae5200640 1 --2- 172.21.15.125:0/2168107704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ae014b2c0 0x7f7ae00aff00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8032000 tx=0x7f7ad8004a40 comp rx=0 tx=0).stop 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.705+0000 7f7ade800640 1 -- 172.21.15.125:0/2168107704 reap_dead start 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.706+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 shutdown_connections 2024-01-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.706+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 >> 172.21.15.125:0/2168107704 conn(0x7f7ae006a800 msgr2=0x7f7ae00b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.706+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 shutdown_connections 2024-01-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.706+0000 7f7ae5200640 1 -- 172.21.15.125:0/2168107704 wait complete. 2024-01-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:52.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:56:52.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.711+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 --> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7bc0d13a0 con 0x7fd794073460 2024-01-16T14:56:52.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.712+0000 7fd7ba000640 1 -- 172.21.15.125:0/3126105157 <== osd.6 v2:172.21.15.137:6816/1720012579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd7ac001d60 con 0x7fd794073460 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7fd794073460 msgr2=0x7fd794075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7fd794073460 0x7fd794075880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79406b2c0 msgr2=0x7fd79406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd79406b2c0 0x7fd79406d780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd7a402b910 tx=0x7fd7a403c040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7bc160f80 msgr2=0x7fd7bc147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 --2- 172.21.15.125:0/3126105157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7bc160f80 0x7fd7bc147e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c780 tx=0x7fd7b400cc40 comp rx=0 tx=0).stop 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c2a00640 1 -- 172.21.15.125:0/3126105157 reap_dead start 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 shutdown_connections 2024-01-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.713+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 >> 172.21.15.125:0/3126105157 conn(0x7fd7bc09c550 msgr2=0x7fd7bc0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.714+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 --> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe5280d1c40 con 0x7fe5000734e0 2024-01-16T14:56:52.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.714+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 shutdown_connections 2024-01-16T14:56:52.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.714+0000 7fd7c4a00640 1 -- 172.21.15.125:0/3126105157 wait complete. 2024-01-16T14:56:52.714 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:52.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:56:52.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.715+0000 7fe51e000640 1 -- 172.21.15.125:0/3709273521 <== osd.7 v2:172.21.15.137:6817/1873161633 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe5200344c0 con 0x7fe5000734e0 2024-01-16T14:56:52.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fe5000734e0 msgr2=0x7fe500075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fe5000734e0 0x7fe500075900 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50006b2c0 msgr2=0x7fe50006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50006b2c0 0x7fe50006d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe52815d0d0 tx=0x7fe510009040 comp rx=0 tx=0).stop 2024-01-16T14:56:52.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5280ce8f0 msgr2=0x7fe5280b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe52d200640 1 --2- 172.21.15.125:0/3709273521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5280ce8f0 0x7fe5280b8c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c3e0 tx=0x7fe51800c8a0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.716+0000 7fe526800640 1 -- 172.21.15.125:0/3709273521 reap_dead start 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.717+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 shutdown_connections 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.717+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 >> 172.21.15.125:0/3709273521 conn(0x7fe52809c550 msgr2=0x7fe5280b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.717+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 shutdown_connections 2024-01-16T14:56:52.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.717+0000 7fe52d200640 1 -- 172.21.15.125:0/3709273521 wait complete. 2024-01-16T14:56:52.717 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:52.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:56:52.718 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:52.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:56:52.719 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:52.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:56:52.720 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:52.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:56:52.721 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:52.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:56:52.722 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:52.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:56:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.979+0000 7f78e0400640 1 Processor -- start 2024-01-16T14:56:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.979+0000 7f78e0400640 1 -- start start 2024-01-16T14:56:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.981+0000 7f78e0400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78d00e3120 0x7f78d00e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.981+0000 7f78e0400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 0x7f78d0058010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.981+0000 7f78e0400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d0058550 con 0x7f78d0053c70 2024-01-16T14:56:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.981+0000 7f78e0400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d0007b30 con 0x7f78d00e3120 2024-01-16T14:56:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.981+0000 7f78e0400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00587d0 con 0x7f78d003c060 2024-01-16T14:56:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.982+0000 7f78de400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f78d0053c70 0x7f78d00540c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55296/0 (socket says 172.21.15.125:55296) 2024-01-16T14:56:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.982+0000 7f78de400640 1 -- 172.21.15.125:0/1853857463 learned_addr learned my addr 172.21.15.125:0/1853857463 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.983+0000 7f78dda00640 1 --2- 172.21.15.125:0/1853857463 >> v2:172.21.15.125:3300/0 conn(0x7f78d00e3120 0x7f78d00e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.983+0000 7f78dd000640 1 --2- 172.21.15.125:0/1853857463 >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 0x7f78d0058010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.984+0000 7f78dda00640 1 -- 172.21.15.125:0/1853857463 >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 msgr2=0x7f78d0058010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.984+0000 7f78dda00640 1 --2- 172.21.15.125:0/1853857463 >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 0x7f78d0058010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.984+0000 7f78dda00640 1 -- 172.21.15.125:0/1853857463 >> v1:172.21.15.137:6789/0 conn(0x7f78d0053c70 legacy=0x7f78d00540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.984+0000 7f78dda00640 1 -- 172.21.15.125:0/1853857463 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0058ab0 con 0x7f78d00e3120 2024-01-16T14:56:52.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.985+0000 7f78dd000640 1 --2- 172.21.15.125:0/1853857463 >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 0x7f78d0058010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.985+0000 7f78dda00640 1 --2- 172.21.15.125:0/1853857463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d00e7740 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78d00536c0 tx=0x7f78c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaf6a7dfa339f72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.986+0000 7f78d7e00640 1 -- 172.21.15.125:0/1853857463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0031870 con 0x7f78d00e3120 2024-01-16T14:56:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.986+0000 7f78d7e00640 1 -- 172.21.15.125:0/1853857463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c00319d0 con 0x7f78d00e3120 2024-01-16T14:56:52.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.986+0000 7f78d7e00640 1 -- 172.21.15.125:0/1853857463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0031870 con 0x7f78d00e3120 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.987+0000 7f78e0400640 1 -- 172.21.15.125:0/1853857463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 msgr2=0x7f78d00e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.987+0000 7f78e0400640 1 --2- 172.21.15.125:0/1853857463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d00e7740 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78d00536c0 tx=0x7f78c002b910 comp rx=0 tx=0).stop 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 -- 172.21.15.125:0/1853857463 shutdown_connections 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 --2- 172.21.15.125:0/1853857463 >> v2:172.21.15.137:3300/0 conn(0x7f78d003c060 0x7f78d0058010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 --2- 172.21.15.125:0/1853857463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d00e7740 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 -- 172.21.15.125:0/1853857463 >> 172.21.15.125:0/1853857463 conn(0x7f78d0058ea0 msgr2=0x7f78d0059280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 -- 172.21.15.125:0/1853857463 shutdown_connections 2024-01-16T14:56:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.988+0000 7f78e0400640 1 -- 172.21.15.125:0/1853857463 wait complete. 2024-01-16T14:56:52.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.991+0000 7f78e0400640 1 Processor -- start 2024-01-16T14:56:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.991+0000 7f78e0400640 1 -- start start 2024-01-16T14:56:52.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.994+0000 7f78e0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 0x7f78d00e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.994+0000 7f78e0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.994+0000 7f78e0400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.995+0000 7f78dda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 0x7f78d00e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.995+0000 7f78dda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 0x7f78d00e3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45598/0 (socket says 172.21.15.125:45598) 2024-01-16T14:56:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.995+0000 7f78dd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.995+0000 7f78dd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52392/0 (socket says 172.21.15.125:52392) 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.996+0000 7f78de400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.996+0000 7f78de400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47842/0 (socket says 172.21.15.125:47842) 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.996+0000 7f78dda00640 1 -- 172.21.15.125:0/2261103241 learned_addr learned my addr 172.21.15.125:0/2261103241 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.996+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d004e340 con 0x7f78d00e3120 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.997+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d004e060 con 0x7f78d003c060 2024-01-16T14:56:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.997+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0051580 con 0x7f78d0053c70 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.998+0000 7f78de400640 1 -- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 msgr2=0x7f78d003d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.998+0000 7f78de400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.998+0000 7f78de400640 1 -- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 msgr2=0x7f78d00e3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.998+0000 7f78de400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 0x7f78d00e3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.998+0000 7f78de400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0009400 con 0x7f78d00e3120 2024-01-16T14:56:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.999+0000 7f78de400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f78d804fab0 tx=0x7f78d80748f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:52.999+0000 7f78dd000640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.000+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d808e070 con 0x7f78d00e3120 2024-01-16T14:56:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.000+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d806b9a0 con 0x7f78d00e3120 2024-01-16T14:56:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.000+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d0051860 con 0x7f78d00e3120 2024-01-16T14:56:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.000+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d003df20 con 0x7f78d00e3120 2024-01-16T14:56:52.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.004+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d806bd90 con 0x7f78d00e3120 2024-01-16T14:56:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.009+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f78d80721c0 con 0x7f78d00e3120 2024-01-16T14:56:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.009+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a8005250 con 0x7f78d00e3120 2024-01-16T14:56:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.011+0000 7f78d5600640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 0x7f78b006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.013+0000 7f78dda00640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 0x7f78b006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.013+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78d81031f0 con 0x7f78d00e3120 2024-01-16T14:56:53.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.022+0000 7f78dda00640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 0x7f78b006d7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78c00091d0 tx=0x7f78c003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.024+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78d80ce210 con 0x7f78d00e3120 2024-01-16T14:56:53.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.071+0000 7eff8d600640 1 Processor -- start 2024-01-16T14:56:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.071+0000 7eff8d600640 1 -- start start 2024-01-16T14:56:53.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.072+0000 7faf04c00640 1 Processor -- start 2024-01-16T14:56:53.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.072+0000 7faf04c00640 1 -- start start 2024-01-16T14:56:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.073+0000 7eff8d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 0x7eff88099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.073+0000 7eff8d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 0x7eff8806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.073+0000 7eff8d600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff880cfa70 con 0x7eff8815e880 2024-01-16T14:56:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.073+0000 7eff8d600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8809a3e0 con 0x7eff8806e7c0 2024-01-16T14:56:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.073+0000 7eff8d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff880cff00 con 0x7eff88099a60 2024-01-16T14:56:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7eff86a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7eff8815e880 0x7eff8806ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36536/0 (socket says 172.21.15.125:36536) 2024-01-16T14:56:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7eff87400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 0x7eff8806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7fb35f600640 1 Processor -- start 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faf04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf00070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faf04c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000706a0 con 0x7faf000b2e00 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faf04c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000708d0 con 0x7faf0006c730 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faf04c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf000b3290 con 0x7faf0006d220 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faeff400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7faf0006c730 0x7faf00071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55310/0 (socket says 172.21.15.125:55310) 2024-01-16T14:56:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7faeff400640 1 -- 172.21.15.125:0/2736672276 learned_addr learned my addr 172.21.15.125:0/2736672276 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faeffe00640 1 --2- 172.21.15.125:0/2736672276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf00070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faefe000640 1 -- 172.21.15.125:0/2736672276 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457295841 0 0) 0x7faf000708d0 con 0x7faf0006c730 2024-01-16T14:56:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7fb35f600640 1 -- start start 2024-01-16T14:56:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faefe000640 1 -- 172.21.15.125:0/2736672276 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee0003520 con 0x7faf0006c730 2024-01-16T14:56:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faeffe00640 1 -- 172.21.15.125:0/2736672276 >> v1:172.21.15.137:6789/0 conn(0x7faf0006c730 legacy=0x7faf00071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faeffe00640 1 -- 172.21.15.125:0/2736672276 >> v1:172.21.15.125:6789/0 conn(0x7faf000b2e00 legacy=0x7faf0006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.076+0000 7faeffe00640 1 -- 172.21.15.125:0/2736672276 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00070bb0 con 0x7faf0006d220 2024-01-16T14:56:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb35814b2c0 0x7fb3580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7eff87400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 0x7eff8806ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47852/0 (socket says 172.21.15.125:47852) 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7eff87400640 1 -- 172.21.15.125:0/3242995043 learned_addr learned my addr 172.21.15.125:0/3242995043 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.074+0000 7eff86000640 1 --2- 172.21.15.125:0/3242995043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 0x7eff88099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7eff86000640 1 -- 172.21.15.125:0/3242995043 >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 msgr2=0x7eff8806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7eff86000640 1 --2- 172.21.15.125:0/3242995043 >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 0x7eff8806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.075+0000 7eff86000640 1 -- 172.21.15.125:0/3242995043 >> v1:172.21.15.125:6789/0 conn(0x7eff8815e880 legacy=0x7eff8806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb3580cf7d0 0x7fb358070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35814b6a0 con 0x7fb35814b2c0 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580cfbf0 con 0x7fb3580cf7d0 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7fb35f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb358071690 con 0x7fb358070ea0 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.078+0000 7fb35d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7ff9e9600640 1 Processor -- start 2024-01-16T14:56:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.078+0000 7fb35d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52418/0 (socket says 172.21.15.125:52418) 2024-01-16T14:56:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.078+0000 7fb35d600640 1 -- 172.21.15.125:0/4176892517 learned_addr learned my addr 172.21.15.125:0/4176892517 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.078+0000 7fb35cc00640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.125:3300/0 conn(0x7fb35814b2c0 0x7fb3580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.078+0000 7ff9e9600640 1 -- start start 2024-01-16T14:56:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb357e00640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.137:3300/0 conn(0x7fb3580cf7d0 0x7fb358070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.077+0000 7eff86000640 1 -- 172.21.15.125:0/3242995043 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8806fc70 con 0x7eff88099a60 2024-01-16T14:56:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7faeffe00640 1 --2- 172.21.15.125:0/2736672276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf00070160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faef8009dc0 tx=0x7faef802bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7111aec0bf7c9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb35d600640 1 -- 172.21.15.125:0/4176892517 >> v2:172.21.15.137:3300/0 conn(0x7fb3580cf7d0 msgr2=0x7fb358070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7faefe000640 1 -- 172.21.15.125:0/2736672276 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef802ccc0 con 0x7faf0006d220 2024-01-16T14:56:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7eff86000640 1 --2- 172.21.15.125:0/3242995043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 0x7eff88099ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff700097a0 tx=0x7eff7002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d31125fcc9731d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7ff9e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb35d600640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.137:3300/0 conn(0x7fb3580cf7d0 0x7fb358070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7faefe000640 1 -- 172.21.15.125:0/2736672276 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef802ce20 con 0x7faf0006d220 2024-01-16T14:56:53.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7faefe000640 1 -- 172.21.15.125:0/2736672276 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8034640 con 0x7faf0006d220 2024-01-16T14:56:53.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb35d600640 1 -- 172.21.15.125:0/4176892517 >> v2:172.21.15.125:3300/0 conn(0x7fb35814b2c0 msgr2=0x7fb3580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e9600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 0x7ff9e40b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e9600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e40b6750 con 0x7ff9e40d00b0 2024-01-16T14:56:53.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e9600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e40ced20 con 0x7ff9e40ce8c0 2024-01-16T14:56:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40d0530 con 0x7ff9e415d590 2024-01-16T14:56:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb35d600640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.125:3300/0 conn(0x7fb35814b2c0 0x7fb3580cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.079+0000 7fb35d600640 1 -- 172.21.15.125:0/4176892517 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358071970 con 0x7fb358070ea0 2024-01-16T14:56:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52430/0 (socket says 172.21.15.125:52430) 2024-01-16T14:56:53.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7faf04c00640 1 -- 172.21.15.125:0/2736672276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 msgr2=0x7faf00070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7faf04c00640 1 --2- 172.21.15.125:0/2736672276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf00070160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faef8009dc0 tx=0x7faef802bf10 comp rx=0 tx=0).stop 2024-01-16T14:56:53.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.080+0000 7ff9e2a00640 1 -- 172.21.15.125:0/59733783 learned_addr learned my addr 172.21.15.125:0/59733783 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7ff9e3400640 1 --2- 172.21.15.125:0/59733783 >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 0x7ff9e40b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7f7c04400640 1 Processor -- start 2024-01-16T14:56:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7faf04c00640 1 -- 172.21.15.125:0/2736672276 shutdown_connections 2024-01-16T14:56:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7faf04c00640 1 --2- 172.21.15.125:0/2736672276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf00070160 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7faf04c00640 1 -- 172.21.15.125:0/2736672276 >> 172.21.15.125:0/2736672276 conn(0x7faf0006a7d0 msgr2=0x7faf0006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7eff85600640 1 -- 172.21.15.125:0/3242995043 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70031830 con 0x7eff88099a60 2024-01-16T14:56:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7ff9e2a00640 1 -- 172.21.15.125:0/59733783 >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 msgr2=0x7ff9e40b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7ff9e2a00640 1 --2- 172.21.15.125:0/59733783 >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 0x7ff9e40b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7eff85600640 1 -- 172.21.15.125:0/3242995043 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff70031990 con 0x7eff88099a60 2024-01-16T14:56:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7f2f07600640 1 Processor -- start 2024-01-16T14:56:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7f7c04400640 1 -- start start 2024-01-16T14:56:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7ff9e2a00640 1 -- 172.21.15.125:0/59733783 >> v1:172.21.15.125:6789/0 conn(0x7ff9e40d00b0 legacy=0x7ff9e40ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7ff9e2a00640 1 -- 172.21.15.125:0/59733783 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40cf1f0 con 0x7ff9e415d590 2024-01-16T14:56:53.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7fb35d600640 1 --2- 172.21.15.125:0/4176892517 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb3580cfea0 tx=0x7fb35002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cf15781ddc0463 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7ff9e3400640 1 --2- 172.21.15.125:0/59733783 >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 0x7ff9e40b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:53.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7faf04c00640 1 -- 172.21.15.125:0/2736672276 shutdown_connections 2024-01-16T14:56:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.081+0000 7eff85600640 1 -- 172.21.15.125:0/3242995043 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70031d20 con 0x7eff88099a60 2024-01-16T14:56:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 -- 172.21.15.125:0/3242995043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 msgr2=0x7eff88099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 --2- 172.21.15.125:0/3242995043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 0x7eff88099ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff700097a0 tx=0x7eff7002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7f2f07600640 1 -- start start 2024-01-16T14:56:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 -- 172.21.15.125:0/3242995043 shutdown_connections 2024-01-16T14:56:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 --2- 172.21.15.125:0/3242995043 >> v2:172.21.15.125:3300/0 conn(0x7eff8806e7c0 0x7eff8806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7fb357400640 1 -- 172.21.15.125:0/4176892517 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35002ca80 con 0x7fb358070ea0 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7fb357400640 1 -- 172.21.15.125:0/4176892517 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35002cbe0 con 0x7fb358070ea0 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7fb357400640 1 -- 172.21.15.125:0/4176892517 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500343f0 con 0x7fb358070ea0 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 --2- 172.21.15.125:0/3242995043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff88099a60 0x7eff88099ea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7f7c04400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7bfc0d0280 0x7f7bfc0ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7f7c04400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0b6750 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7f7c04400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0cf1f0 con 0x7f7bfc15d590 2024-01-16T14:56:53.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7f7c04400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0cef10 con 0x7f7bfc0d0280 2024-01-16T14:56:53.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7ff9e2a00640 1 --2- 172.21.15.125:0/59733783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009870 tx=0x7ff9d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9d64e0c4216ed1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7eff8d600640 1 -- 172.21.15.125:0/3242995043 >> 172.21.15.125:0/3242995043 conn(0x7eff880b2d00 msgr2=0x7eff880b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7eff8d600640 1 -- 172.21.15.125:0/3242995043 shutdown_connections 2024-01-16T14:56:53.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.082+0000 7faf04c00640 1 -- 172.21.15.125:0/2736672276 wait complete. 2024-01-16T14:56:53.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7eff8d600640 1 -- 172.21.15.125:0/3242995043 wait complete. 2024-01-16T14:56:53.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7fb35f600640 1 -- 172.21.15.125:0/4176892517 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 msgr2=0x7fb3580b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f7c02400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7bfc0ce6f0 0x7f7bfc0b6be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36552/0 (socket says 172.21.15.125:36552) 2024-01-16T14:56:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f7c02400640 1 -- 172.21.15.125:0/3552950881 learned_addr learned my addr 172.21.15.125:0/3552950881 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.083+0000 7ff9e1600640 1 -- 172.21.15.125:0/59733783 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4030840 con 0x7ff9e415d590 2024-01-16T14:56:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7ff9e1600640 1 -- 172.21.15.125:0/59733783 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d40309a0 con 0x7ff9e415d590 2024-01-16T14:56:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7ff9e1600640 1 -- 172.21.15.125:0/59733783 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4030d30 con 0x7ff9e415d590 2024-01-16T14:56:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f7c01000640 1 --2- 172.21.15.125:0/3552950881 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc0d0280 0x7f7bfc0ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7fb35f600640 1 --2- 172.21.15.125:0/4176892517 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb3580cfea0 tx=0x7fb35002bee0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f2f07600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f0014b2c0 0x7f2f000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7fb35f600640 1 -- 172.21.15.125:0/4176892517 shutdown_connections 2024-01-16T14:56:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f2f07600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000cfbf0 con 0x7f2f00071070 2024-01-16T14:56:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f2f07600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f00071690 con 0x7f2f000cf7d0 2024-01-16T14:56:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.084+0000 7f2f07600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0014b6a0 con 0x7f2f0014b2c0 2024-01-16T14:56:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f0014b2c0 0x7f2f000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7fb35f600640 1 --2- 172.21.15.125:0/4176892517 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb358070ea0 0x7fb3580b1ee0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7fb35f600640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.137:3300/0 conn(0x7fb3580cf7d0 0x7fb358070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f0014b2c0 0x7f2f000cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52446/0 (socket says 172.21.15.125:52446) 2024-01-16T14:56:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7f2f04c00640 1 -- 172.21.15.125:0/709495298 learned_addr learned my addr 172.21.15.125:0/709495298 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7faf04c00640 1 Processor -- start 2024-01-16T14:56:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7fb35f600640 1 --2- 172.21.15.125:0/4176892517 >> v2:172.21.15.125:3300/0 conn(0x7fb35814b2c0 0x7fb3580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7fb35f600640 1 -- 172.21.15.125:0/4176892517 >> 172.21.15.125:0/4176892517 conn(0x7fb35806a800 msgr2=0x7fb35806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7f7c01000640 1 -- 172.21.15.125:0/3552950881 >> v1:172.21.15.137:6789/0 conn(0x7f7bfc15d590 legacy=0x7f7bfc0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7f7c01000640 1 -- 172.21.15.125:0/3552950881 >> v1:172.21.15.125:6789/0 conn(0x7f7bfc0ce6f0 legacy=0x7f7bfc0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7f7c01000640 1 -- 172.21.15.125:0/3552950881 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0b6100 con 0x7f7bfc0d0280 2024-01-16T14:56:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.085+0000 7f7bfbe00640 1 -- 172.21.15.125:0/3552950881 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939152441 0 0) 0x7f7bfc0b6750 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7eff8d600640 1 Processor -- start 2024-01-16T14:56:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7fb35f600640 1 -- 172.21.15.125:0/4176892517 shutdown_connections 2024-01-16T14:56:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7fb35f600640 1 -- 172.21.15.125:0/4176892517 wait complete. 2024-01-16T14:56:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7eff8d600640 1 -- start start 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7ff9e9600640 1 -- 172.21.15.125:0/59733783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 msgr2=0x7ff9e40cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7ff9e9600640 1 --2- 172.21.15.125:0/59733783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009870 tx=0x7ff9d402b980 comp rx=0 tx=0).stop 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f9b49e00640 1 Processor -- start 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958621839 0 0) 0x7f2f000cfbf0 con 0x7f2f00071070 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0003520 con 0x7f2f00071070 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030059631 0 0) 0x7f2ee0003520 con 0x7f2f00071070 2024-01-16T14:56:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f0014b2c0 msgr2=0x7f2f000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f0014b2c0 0x7f2f000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 >> v1:172.21.15.137:6789/0 conn(0x7f2f000cf7d0 legacy=0x7f2f00070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00071970 con 0x7f2f00071070 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2ef80019f0 con 0x7f2f00071070 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 >> v1:172.21.15.125:6789/0 conn(0x7f2f00071070 legacy=0x7f2f000cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee00039a0 0x7f2ee0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee0005180 0x7f2ee000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0014b6a0 con 0x7f2ee0005180 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000bfa0 con 0x7f2ee00039a0 2024-01-16T14:56:53.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000c160 con 0x7f2ee00047c0 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7eff8d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff8d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 0x7eff88159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff8d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 0x7eff881452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff8d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8815a240 con 0x7eff8815e880 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff8d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff880b6df0 con 0x7eff88099a60 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.086+0000 7faf04c00640 1 -- start start 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0006c730 0x7faf00137b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 0x7faf000ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf000a9ee0 con 0x7faf000b2e00 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf000a9c00 con 0x7faf0006c730 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7faf04c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf000ab080 con 0x7faf0006d220 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff8d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8815b020 con 0x7eff8806e7c0 2024-01-16T14:56:53.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.087+0000 7f9b49e00640 1 -- start start 2024-01-16T14:56:53.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7f2f04c00640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee00039a0 0x7f2ee0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff86a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff86a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52458/0 (socket says 172.21.15.125:52458) 2024-01-16T14:56:53.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff86a00640 1 -- 172.21.15.125:0/2363895698 learned_addr learned my addr 172.21.15.125:0/2363895698 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7faefea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7f2f05600640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee0005180 0x7f2ee000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 -- 172.21.15.125:0/59733783 shutdown_connections 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 --2- 172.21.15.125:0/59733783 >> v2:172.21.15.125:3300/0 conn(0x7ff9e40ce8c0 0x7ff9e40b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 --2- 172.21.15.125:0/59733783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e415d590 0x7ff9e40cfb70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 -- 172.21.15.125:0/59733783 >> 172.21.15.125:0/59733783 conn(0x7ff9e409c540 msgr2=0x7ff9e40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 -- 172.21.15.125:0/59733783 shutdown_connections 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7faefea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52460/0 (socket says 172.21.15.125:52460) 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7faefea00640 1 -- 172.21.15.125:0/1727333461 learned_addr learned my addr 172.21.15.125:0/1727333461 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7eff87400640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 0x7eff881452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7faeff400640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0006c730 0x7faf00137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86000640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 0x7eff88159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f2f05600640 1 -- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee00047c0 msgr2=0x7f2ee0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7faeffe00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 0x7faf000ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86a00640 1 -- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 msgr2=0x7eff88159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86a00640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 0x7eff88159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f2f05600640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f2f05600640 1 -- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee00039a0 msgr2=0x7f2ee0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f2f05600640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee00039a0 0x7f2ee0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f2f05600640 1 -- 172.21.15.125:0/709495298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00071970 con 0x7f2ee0005180 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86a00640 1 -- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 msgr2=0x7eff881452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86a00640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 0x7eff881452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86a00640 1 -- 172.21.15.125:0/2363895698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70009400 con 0x7eff8806e7c0 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff87400640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 0x7eff881452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7eff86000640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 0x7eff88159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7f7c01000640 1 --2- 172.21.15.125:0/3552950881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0d0280 0x7f7bfc0ce1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009720 tx=0x7f7be402b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4d5bd5c85b1c882 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.088+0000 7ff9e9600640 1 -- 172.21.15.125:0/59733783 wait complete. 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.089+0000 7faefea00640 1 -- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0006c730 msgr2=0x7faf00137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faefea00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0006c730 0x7faf00137b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faefea00640 1 -- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 msgr2=0x7faf000ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faefea00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 0x7faf000ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faefea00640 1 -- 172.21.15.125:0/1727333461 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8009a70 con 0x7faf0006d220 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faeffe00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 0x7faf000ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b49e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9b4415d590 0x7f9b440cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b49e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440b6750 con 0x7f9b440ce8c0 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b49e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440cf1f0 con 0x7f9b440d00b0 2024-01-16T14:56:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b49e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b440d0530 con 0x7f9b4415d590 2024-01-16T14:56:53.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7eff86a00640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff8000ba50 tx=0x7eff8000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b43400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9b440ce8c0 0x7f9b440b6be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36562/0 (socket says 172.21.15.125:36562) 2024-01-16T14:56:53.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b43400640 1 -- 172.21.15.125:0/4125195657 learned_addr learned my addr 172.21.15.125:0/4125195657 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:53.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f7bfbe00640 1 -- 172.21.15.125:0/3552950881 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4030710 con 0x7f7bfc0d0280 2024-01-16T14:56:53.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f7bfbe00640 1 -- 172.21.15.125:0/3552950881 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be402e050 con 0x7f7bfc0d0280 2024-01-16T14:56:53.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f7bfbe00640 1 -- 172.21.15.125:0/3552950881 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4030a90 con 0x7f7bfc0d0280 2024-01-16T14:56:53.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7faefea00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faee800c390 tx=0x7faee800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7f9b42a00640 1 --2- 172.21.15.125:0/4125195657 >> v2:172.21.15.137:3300/0 conn(0x7f9b4415d590 0x7f9b440cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.090+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8000cbe0 con 0x7eff8806e7c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8815a4a0 con 0x7eff8806e7c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8815aa60 con 0x7eff8806e7c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8000cd80 con 0x7eff8806e7c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80014680 con 0x7eff8806e7c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994896170 0 0) 0x7f9b440b6750 con 0x7f9b440ce8c0 2024-01-16T14:56:53.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b24003520 con 0x7f9b440ce8c0 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7fb35f600640 1 Processor -- start 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7f2f05600640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee0005180 0x7f2ee000b980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800c430 tx=0x7f2ee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b2189b061431da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee80052d0 con 0x7faf0006d220 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee8005430 con 0x7faf0006d220 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.091+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8003830 con 0x7faf0006d220 2024-01-16T14:56:53.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf000ab360 con 0x7faf0006d220 2024-01-16T14:56:53.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370235833 0 0) 0x7f9b440cf1f0 con 0x7f9b440d00b0 2024-01-16T14:56:53.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b440b6750 con 0x7f9b440d00b0 2024-01-16T14:56:53.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f7c04400640 1 -- 172.21.15.125:0/3552950881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0d0280 msgr2=0x7f7bfc0ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f7c04400640 1 --2- 172.21.15.125:0/3552950881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0d0280 0x7f7bfc0ce1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009720 tx=0x7f7be402b730 comp rx=0 tx=0).stop 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff58005250 con 0x7eff8806e7c0 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f9b42a00640 1 -- 172.21.15.125:0/4125195657 >> v1:172.21.15.137:6789/0 conn(0x7f9b440d00b0 legacy=0x7f9b440ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f9b42a00640 1 -- 172.21.15.125:0/4125195657 >> v1:172.21.15.125:6789/0 conn(0x7f9b440ce8c0 legacy=0x7f9b440b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f9b42a00640 1 -- 172.21.15.125:0/4125195657 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440cef10 con 0x7f9b4415d590 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf000a8ed0 con 0x7faf0006d220 2024-01-16T14:56:53.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef802cd60 con 0x7f2ee0005180 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8034440 con 0x7f2ee0005180 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.092+0000 7f2eff400640 1 -- 172.21.15.125:0/709495298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8034850 con 0x7f2ee0005180 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7fb35f600640 1 -- start start 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f7c04400640 1 -- 172.21.15.125:0/3552950881 shutdown_connections 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f7c04400640 1 --2- 172.21.15.125:0/3552950881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0d0280 0x7f7bfc0ce1b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f7c04400640 1 -- 172.21.15.125:0/3552950881 >> 172.21.15.125:0/3552950881 conn(0x7f7bfc09c540 msgr2=0x7f7bfc0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f2f07600640 1 -- 172.21.15.125:0/709495298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee0005180 msgr2=0x7f2ee000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f2f07600640 1 --2- 172.21.15.125:0/709495298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee0005180 0x7f2ee000b980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800c430 tx=0x7f2ee00057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f2f05600640 1 -- 172.21.15.125:0/709495298 reap_dead start 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7ff9e9600640 1 Processor -- start 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f7c04400640 1 -- 172.21.15.125:0/3552950881 shutdown_connections 2024-01-16T14:56:53.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7ff9e9600640 1 -- start start 2024-01-16T14:56:53.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf000ab880 con 0x7faf0006d220 2024-01-16T14:56:53.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.093+0000 7f7c04400640 1 -- 172.21.15.125:0/3552950881 wait complete. 2024-01-16T14:56:53.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.094+0000 7f2f07600640 1 -- 172.21.15.125:0/709495298 shutdown_connections 2024-01-16T14:56:53.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.094+0000 7f2f07600640 1 -- 172.21.15.125:0/709495298 >> 172.21.15.125:0/709495298 conn(0x7f2f0006a800 msgr2=0x7f2f0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb358070ea0 0x7fb358148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7f2f07600640 1 -- 172.21.15.125:0/709495298 shutdown_connections 2024-01-16T14:56:53.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7f2f07600640 1 -- 172.21.15.125:0/709495298 wait complete. 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb35814b2c0 0x7fb358147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580aa530 con 0x7fb3580cf7d0 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35815f8d0 con 0x7fb358070ea0 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb358147cb0 con 0x7fb35814b2c0 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7f7c04400640 1 Processor -- start 2024-01-16T14:56:53.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7f7c04400640 1 -- start start 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e40ce8c0 0x7ff9e41598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e40d00b0 0x7ff9e40b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e4159de0 con 0x7ff9e415d590 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40cb360 con 0x7ff9e40d00b0 2024-01-16T14:56:53.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40cada0 con 0x7ff9e40ce8c0 2024-01-16T14:56:53.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff8001d3b0 con 0x7eff8806e7c0 2024-01-16T14:56:53.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7eff7ea00640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 0x7eff6006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7ff9e3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47912/0 (socket says 172.21.15.125:47912) 2024-01-16T14:56:53.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 -- 172.21.15.125:0/3948003850 learned_addr learned my addr 172.21.15.125:0/3948003850 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7f9b42a00640 1 --2- 172.21.15.125:0/4125195657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b440cfb70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b340097a0 tx=0x7f9b3402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fec95c66b089356 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb357e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb357e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47904/0 (socket says 172.21.15.125:47904) 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb357e00640 1 -- 172.21.15.125:0/1398764230 learned_addr learned my addr 172.21.15.125:0/1398764230 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.095+0000 7fb35d600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb35814b2c0 0x7fb358147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7fb357e00640 1 -- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb35814b2c0 msgr2=0x7fb358147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7fb357e00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb35814b2c0 0x7fb358147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7fb357e00640 1 -- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb358070ea0 msgr2=0x7fb358148d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7fb357e00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb358070ea0 0x7fb358148d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.096+0000 7fb357e00640 1 -- 172.21.15.125:0/1398764230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350009b30 con 0x7fb3580cf7d0 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7fb357e00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e870 tx=0x7fb34000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faee8003990 con 0x7faf0006d220 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000ee60 con 0x7fb3580cf7d0 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000c940 con 0x7fb3580cf7d0 2024-01-16T14:56:53.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000ccd0 con 0x7fb3580cf7d0 2024-01-16T14:56:53.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e2000640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e40d00b0 0x7ff9e40b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e2a00640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e40ce8c0 0x7ff9e41598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7f7c04400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7f2f07600640 1 Processor -- start 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb358147f90 con 0x7fb3580cf7d0 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb358148550 con 0x7fb3580cf7d0 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 -- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e40ce8c0 msgr2=0x7ff9e41598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e40ce8c0 0x7ff9e41598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 -- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e40d00b0 msgr2=0x7ff9e40b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e40d00b0 0x7ff9e40b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7ff9e3400640 1 -- 172.21.15.125:0/3948003850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40094d0 con 0x7ff9e415d590 2024-01-16T14:56:53.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f7c04400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0d0280 0x7f7bfc0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f2f07600640 1 -- start start 2024-01-16T14:56:53.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0019f0 con 0x7f9b4415d590 2024-01-16T14:56:53.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3402e050 con 0x7f9b4415d590 2024-01-16T14:56:53.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f9b42000640 1 -- 172.21.15.125:0/4125195657 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34030730 con 0x7f9b4415d590 2024-01-16T14:56:53.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f7c01a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f7c01a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45676/0 (socket says 172.21.15.125:45676) 2024-01-16T14:56:53.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f7c01a00640 1 -- 172.21.15.125:0/3489830456 learned_addr learned my addr 172.21.15.125:0/3489830456 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.097+0000 7eff86000640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 0x7eff6006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bfc15d590 0x7f7bfc0b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7ff9e3400640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00d380 tx=0x7ff9dc00d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0ca380 con 0x7f7bfc15d590 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f7c02400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bfc15d590 0x7f7bfc0b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0ca660 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f7c01000640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0d0280 0x7f7bfc0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0ca940 con 0x7f7bfc0d0280 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.098+0000 7faef7400640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 0x7faed806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc003b90 con 0x7ff9e415d590 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc00f040 con 0x7ff9e415d590 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc004030 con 0x7ff9e415d590 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee0005180 0x7f2f001614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f000adf90 0x7f2f001fa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001fae60 con 0x7f2f000adf90 2024-01-16T14:56:53.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00162db0 con 0x7f2ee0005180 2024-01-16T14:56:53.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7f2f07600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001fb090 con 0x7f2f00163520 2024-01-16T14:56:53.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faee8057fc0 con 0x7faf0006d220 2024-01-16T14:56:53.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e40cb080 con 0x7ff9e415d590 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7eff86000640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 0x7eff6006d7e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff70036000 tx=0x7eff7003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff80058ae0 con 0x7eff8806e7c0 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f2f05600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f9b49e00640 1 -- 172.21.15.125:0/4125195657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 msgr2=0x7f9b440cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f9b49e00640 1 --2- 172.21.15.125:0/4125195657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b440cfb70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b340097a0 tx=0x7f9b3402b710 comp rx=0 tx=0).stop 2024-01-16T14:56:53.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.099+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e40cb930 con 0x7ff9e415d590 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f7c01a00640 1 -- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0d0280 msgr2=0x7f7bfc0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f7c01a00640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0d0280 0x7f7bfc0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f7c01a00640 1 -- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bfc15d590 msgr2=0x7f7bfc0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f7c01a00640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bfc15d590 0x7f7bfc0b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f7c01a00640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be4009400 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7faeff400640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 0x7faed806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f9b49e00640 1 -- 172.21.15.125:0/4125195657 shutdown_connections 2024-01-16T14:56:53.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f9b49e00640 1 --2- 172.21.15.125:0/4125195657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b440cfb70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f9b49e00640 1 -- 172.21.15.125:0/4125195657 >> 172.21.15.125:0/4125195657 conn(0x7f9b4409c540 msgr2=0x7f9b440d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb328005250 con 0x7fb3580cf7d0 2024-01-16T14:56:53.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.101+0000 7f7c01a00640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c4d0 tx=0x7f7bec00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f2f05600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52504/0 (socket says 172.21.15.125:52504) 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.100+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee0005180 0x7f2f001614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.101+0000 7f2f05600640 1 -- 172.21.15.125:0/1732695635 learned_addr learned my addr 172.21.15.125:0/1732695635 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.101+0000 7f2effe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f000adf90 0x7f2f001fa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.101+0000 7faeff400640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 0x7faed806d7d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faef00011e0 tx=0x7faef0007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.101+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec005480 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec0035f0 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0039a0 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f2f05600640 1 -- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee0005180 msgr2=0x7f2f001614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f2f05600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee0005180 0x7f2f001614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f2f05600640 1 -- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f000adf90 msgr2=0x7f2f001fa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f2f05600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f000adf90 0x7f2f001fa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f2f05600640 1 -- 172.21.15.125:0/1732695635 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f001fb370 con 0x7f2f00163520 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e40cfb70 con 0x7ff9e415d590 2024-01-16T14:56:53.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc0cac20 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.102+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc15ae40 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.103+0000 7f9b49e00640 1 -- 172.21.15.125:0/4125195657 shutdown_connections 2024-01-16T14:56:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.103+0000 7f2f05600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ef802c7d0 tx=0x7f2ef8001360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.103+0000 7f9b49e00640 1 -- 172.21.15.125:0/4125195657 wait complete. 2024-01-16T14:56:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.103+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc0cb140 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.104+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb340013470 con 0x7fb3580cf7d0 2024-01-16T14:56:53.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.105+0000 7f9b49e00640 1 Processor -- start 2024-01-16T14:56:53.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.105+0000 7f9b49e00640 1 -- start start 2024-01-16T14:56:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.105+0000 7fb354c00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 0x7fb33006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.105+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb34008a320 con 0x7fb3580cf7d0 2024-01-16T14:56:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.106+0000 7fb35cc00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 0x7fb33006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7fb35cc00640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 0x7fb33006d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb3581646b0 tx=0x7fb348009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 0x7f9b441521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b441f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4409d790 con 0x7f9b440ce8c0 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440cc7f0 con 0x7f9b4415d590 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b49e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440a07c0 con 0x7f9b440d00b0 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.106+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7bec003b00 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 0x7f9b441521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.105+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff9dc004190 con 0x7ff9e415d590 2024-01-16T14:56:53.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.106+0000 7ff9daa00640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 0x7ff9bc06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.106+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9dc089f30 con 0x7ff9e415d590 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b3be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b3be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52508/0 (socket says 172.21.15.125:52508) 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b3be00640 1 -- 172.21.15.125:0/983059850 learned_addr learned my addr 172.21.15.125:0/983059850 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.107+0000 7f9b43400640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b441f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f7bf9600640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 0x7f7bd406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f9b3be00640 1 -- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 msgr2=0x7f9b441f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f9b3be00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b441f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f9b3be00640 1 -- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 msgr2=0x7f9b441521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7ff9e2a00640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 0x7ff9bc06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bec089fe0 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.108+0000 7f7c01000640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 0x7f7bd406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.109+0000 7f9b3be00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 0x7f9b441521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.109+0000 7f9b3be00640 1 -- 172.21.15.125:0/983059850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34009400 con 0x7f9b440d00b0 2024-01-16T14:56:53.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.109+0000 7f7c01000640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 0x7f7bd406d7d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7be4034000 tx=0x7f7be4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.109+0000 7f9b3be00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c4a0 tx=0x7f9b2c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.109+0000 7f9b42a00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 0x7f9b441521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:53.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.110+0000 7ff9e2a00640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 0x7ff9bc06d7d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4036000 tx=0x7ff9d4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.110+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3400552b0 con 0x7fb3580cf7d0 2024-01-16T14:56:53.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.113+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff800555b0 con 0x7eff8806e7c0 2024-01-16T14:56:53.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.119+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9dc054e10 con 0x7ff9e415d590 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.120+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bec054f40 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.126+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8006d50 con 0x7f2f00163520 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8001190 con 0x7f2f00163520 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80054c0 con 0x7f2f00163520 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001fb650 con 0x7f2f00163520 2024-01-16T14:56:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000a84c0 con 0x7f2f00163520 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0035f0 con 0x7f9b440d00b0 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c003750 con 0x7f9b440d00b0 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c003b20 con 0x7f9b440d00b0 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4409d4f0 con 0x7f9b440d00b0 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b440cbec0 con 0x7f9b440d00b0 2024-01-16T14:56:53.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.129+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f000675d0 con 0x7f2f00163520 2024-01-16T14:56:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.127+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faee8054a90 con 0x7faf0006d220 2024-01-16T14:56:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.131+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b14005250 con 0x7f9b440d00b0 2024-01-16T14:56:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.132+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ef803b040 con 0x7f2f00163520 2024-01-16T14:56:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.133+0000 7f2efcc00640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 0x7f2ed806d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.133+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ef80aad90 con 0x7f2f00163520 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.134+0000 7f2f04c00640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 0x7f2ed806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.133+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b2c003c80 con 0x7f9b440d00b0 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.135+0000 7f9b3b400640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 0x7f9b1c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.135+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b2c088fe0 con 0x7f9b440d00b0 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.136+0000 7f9b42a00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 0x7f9b1c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.135+0000 7f2f04c00640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 0x7f2ed806d880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0006cc0 tx=0x7f2ef0009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.137+0000 7f9b42a00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 0x7f9b1c06d7d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b34034000 tx=0x7f9b34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:53.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.147+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ef8075d20 con 0x7f2f00163520 2024-01-16T14:56:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.158+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b2c054010 con 0x7f9b440d00b0 2024-01-16T14:56:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.388+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f78a8005500 con 0x7f78d00e3120 2024-01-16T14:56:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.389+0000 7f78d5600640 1 -- 172.21.15.125:0/2261103241 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78d80ce210 con 0x7f78d00e3120 2024-01-16T14:56:53.371 INFO:teuthology.orchestra.run.smithi125.stdout:55834574854 2024-01-16T14:56:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.393+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 msgr2=0x7f78b006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.393+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 0x7f78b006d7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78c00091d0 tx=0x7f78c003c040 comp rx=0 tx=0).stop 2024-01-16T14:56:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.394+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 msgr2=0x7f78d004d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.394+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f78d804fab0 tx=0x7f78d80748f0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.394+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 shutdown_connections 2024-01-16T14:56:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.394+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f78b006b310 0x7f78b006d7d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78d00e3120 0x7f78d004d680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f78d0053c70 0x7f78d003d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 --2- 172.21.15.125:0/2261103241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78d003c060 0x7f78d00e3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 >> 172.21.15.125:0/2261103241 conn(0x7f78d0058ea0 msgr2=0x7f78d003a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 shutdown_connections 2024-01-16T14:56:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.395+0000 7f78e0400640 1 -- 172.21.15.125:0/2261103241 wait complete. 2024-01-16T14:56:53.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-01-16T14:56:53.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.481+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9b140057d0 con 0x7f9b440d00b0 2024-01-16T14:56:53.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.482+0000 7f9b3b400640 1 -- 172.21.15.125:0/983059850 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b2c057630 con 0x7f9b440d00b0 2024-01-16T14:56:53.463 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 msgr2=0x7f9b1c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 0x7f9b1c06d7d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b34034000 tx=0x7f9b34008030 comp rx=0 tx=0).stop 2024-01-16T14:56:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 msgr2=0x7f9b4409cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c4a0 tx=0x7f9b2c00c960 comp rx=0 tx=0).stop 2024-01-16T14:56:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 shutdown_connections 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b4415d590 0x7f9b441f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b440d00b0 0x7f9b4409cfb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9b1c06b310 0x7f9b1c06d7d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 --2- 172.21.15.125:0/983059850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b440ce8c0 0x7f9b441521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 >> 172.21.15.125:0/983059850 conn(0x7f9b4409c540 msgr2=0x7f9b4415fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.484+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 shutdown_connections 2024-01-16T14:56:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.485+0000 7f9b49e00640 1 -- 172.21.15.125:0/983059850 wait complete. 2024-01-16T14:56:53.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-01-16T14:56:53.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.507+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7eff580057d0 con 0x7eff8806e7c0 2024-01-16T14:56:53.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.508+0000 7eff7ea00640 1 -- 172.21.15.125:0/2363895698 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff80023070 con 0x7eff8806e7c0 2024-01-16T14:56:53.489 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 msgr2=0x7eff6006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 0x7eff6006d7e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff70036000 tx=0x7eff7003c040 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 msgr2=0x7eff881487b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff8000ba50 tx=0x7eff8000bf10 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 shutdown_connections 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff6006b320 0x7eff6006d7e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8815e880 0x7eff881452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff88099a60 0x7eff88159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 --2- 172.21.15.125:0/2363895698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff8806e7c0 0x7eff881487b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 >> 172.21.15.125:0/2363895698 conn(0x7eff880b2d00 msgr2=0x7eff880ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 shutdown_connections 2024-01-16T14:56:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.511+0000 7eff8d600640 1 -- 172.21.15.125:0/2363895698 wait complete. 2024-01-16T14:56:53.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.519+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb3280050c0 con 0x7fb3580cf7d0 2024-01-16T14:56:53.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.520+0000 7fb354c00640 1 -- 172.21.15.125:0/1398764230 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3400588d0 con 0x7fb3580cf7d0 2024-01-16T14:56:53.502 INFO:teuthology.orchestra.run.smithi125.stdout:55834574854 2024-01-16T14:56:53.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.520+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff9e40cbb50 con 0x7ff9e415d590 2024-01-16T14:56:53.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.520+0000 7ff9daa00640 1 -- 172.21.15.125:0/3948003850 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9dc01a020 con 0x7ff9e415d590 2024-01-16T14:56:53.502 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:53.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 msgr2=0x7fb33006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 0x7fb33006d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb3581646b0 tx=0x7fb348009190 comp rx=0 tx=0).stop 2024-01-16T14:56:53.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 msgr2=0x7fb35815f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e870 tx=0x7fb34000ed30 comp rx=0 tx=0).stop 2024-01-16T14:56:53.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 shutdown_connections 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb35814b2c0 0x7fb358147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb33006b390 0x7fb33006d850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3580cf7d0 0x7fb35815f390 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 --2- 172.21.15.125:0/1398764230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb358070ea0 0x7fb358148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.523+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 >> 172.21.15.125:0/1398764230 conn(0x7fb35806a800 msgr2=0x7fb35806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 shutdown_connections 2024-01-16T14:56:53.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7fb35f600640 1 -- 172.21.15.125:0/1398764230 wait complete. 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 msgr2=0x7ff9bc06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 0x7ff9bc06d7d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4036000 tx=0x7ff9d4008030 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 msgr2=0x7ff9e40ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00d380 tx=0x7ff9dc00d840 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 shutdown_connections 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff9bc06b310 0x7ff9bc06d7d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9e415d590 0x7ff9e40ca2e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e40d00b0 0x7ff9e40b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 --2- 172.21.15.125:0/3948003850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e40ce8c0 0x7ff9e41598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 >> 172.21.15.125:0/3948003850 conn(0x7ff9e409c540 msgr2=0x7ff9e40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.524+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 shutdown_connections 2024-01-16T14:56:53.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.525+0000 7ff9e9600640 1 -- 172.21.15.125:0/3948003850 wait complete. 2024-01-16T14:56:53.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-01-16T14:56:53.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.540+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7bfc0cb140 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.541+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7faf000ab880 con 0x7faf0006d220 2024-01-16T14:56:53.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.541+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2f00067ef0 con 0x7f2f00163520 2024-01-16T14:56:53.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.541+0000 7f7bf9600640 1 -- 172.21.15.125:0/3489830456 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7bfc0cb140 con 0x7f7bfc0ce6f0 2024-01-16T14:56:53.523 INFO:teuthology.orchestra.run.smithi125.stdout:55834574854 2024-01-16T14:56:53.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.541+0000 7f2efcc00640 1 -- 172.21.15.125:0/1732695635 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ef8079340 con 0x7f2f00163520 2024-01-16T14:56:53.524 INFO:teuthology.orchestra.run.smithi125.stdout:55834574854 2024-01-16T14:56:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.542+0000 7faef7400640 1 -- 172.21.15.125:0/1727333461 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faee8017090 con 0x7faf0006d220 2024-01-16T14:56:53.524 INFO:teuthology.orchestra.run.smithi125.stdout:55834574854 2024-01-16T14:56:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 msgr2=0x7f7bd406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 0x7f7bd406d7d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7be4034000 tx=0x7f7be4008030 comp rx=0 tx=0).stop 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 msgr2=0x7f7bfc15a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c4d0 tx=0x7f7bec00c990 comp rx=0 tx=0).stop 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 msgr2=0x7f2ed806d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 0x7f2ed806d880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0006cc0 tx=0x7f2ef0009240 comp rx=0 tx=0).stop 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 msgr2=0x7f2f001639a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ef802c7d0 tx=0x7f2ef8001360 comp rx=0 tx=0).stop 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 shutdown_connections 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bd406b310 0x7f7bd406d7d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bfc15d590 0x7f7bfc0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0d0280 0x7f7bfc0cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 --2- 172.21.15.125:0/3489830456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc0ce6f0 0x7f7bfc15a900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 >> 172.21.15.125:0/3489830456 conn(0x7f7bfc09c540 msgr2=0x7f7bfc06b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 shutdown_connections 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f00163520 0x7f2f001639a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ed806b3c0 0x7f2ed806d880 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f000adf90 0x7f2f001fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 --2- 172.21.15.125:0/1732695635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee0005180 0x7f2f001614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 >> 172.21.15.125:0/1732695635 conn(0x7f2f0006a800 msgr2=0x7f2f000aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 shutdown_connections 2024-01-16T14:56:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.544+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 shutdown_connections 2024-01-16T14:56:53.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.545+0000 7f7c04400640 1 -- 172.21.15.125:0/3489830456 wait complete. 2024-01-16T14:56:53.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.545+0000 7f2f07600640 1 -- 172.21.15.125:0/1732695635 wait complete. 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 msgr2=0x7faed806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 0x7faed806d7d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faef00011e0 tx=0x7faef0007d30 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 msgr2=0x7faf000ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faee800c390 tx=0x7faee800c850 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 shutdown_connections 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faed806b310 0x7faed806d7d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf000b2e00 0x7faf000ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf0006d220 0x7faf000ad910 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 --2- 172.21.15.125:0/1727333461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0006c730 0x7faf00137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 >> 172.21.15.125:0/1727333461 conn(0x7faf0006a7d0 msgr2=0x7faf000b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 shutdown_connections 2024-01-16T14:56:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:53.546+0000 7faf04c00640 1 -- 172.21.15.125:0/1727333461 wait complete. 2024-01-16T14:56:53.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-01-16T14:56:53.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-01-16T14:56:53.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-01-16T14:56:53.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-01-16T14:56:53.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-01-16T14:56:54.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:56:54.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:56:54.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:56:54.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:56:54.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:56:54.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:56:54.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:56:54.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:56:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.659+0000 7fd0da800640 1 Processor -- start 2024-01-16T14:56:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.659+0000 7fd0da800640 1 -- start start 2024-01-16T14:56:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.661+0000 7fd0da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d40cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.661+0000 7fd0da800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40b6750 con 0x7fd0d40ce8c0 2024-01-16T14:56:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.661+0000 7fd0da800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40cf1f0 con 0x7fd0d40d00b0 2024-01-16T14:56:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.661+0000 7fd0da800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40ced20 con 0x7fd0d415d590 2024-01-16T14:56:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.662+0000 7fd0d2a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fd0d40d00b0 0x7fd0d40ce1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55336/0 (socket says 172.21.15.125:55336) 2024-01-16T14:56:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.662+0000 7fd0d2a00640 1 -- 172.21.15.125:0/3541913498 learned_addr learned my addr 172.21.15.125:0/3541913498 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.663+0000 7fd0d3400640 1 --2- 172.21.15.125:0/3541913498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d40cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.663+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946487960 0 0) 0x7fd0d40cf1f0 con 0x7fd0d40d00b0 2024-01-16T14:56:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.664+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4003520 con 0x7fd0d40d00b0 2024-01-16T14:56:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.664+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939640921 0 0) 0x7fd0d40b6750 con 0x7fd0d40ce8c0 2024-01-16T14:56:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.664+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d40cf1f0 con 0x7fd0d40ce8c0 2024-01-16T14:56:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.665+0000 7fd0d3400640 1 -- 172.21.15.125:0/3541913498 >> v1:172.21.15.137:6789/0 conn(0x7fd0d40d00b0 legacy=0x7fd0d40ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.665+0000 7fd0d3400640 1 -- 172.21.15.125:0/3541913498 >> v1:172.21.15.125:6789/0 conn(0x7fd0d40ce8c0 legacy=0x7fd0d40b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.665+0000 7fd0d3400640 1 -- 172.21.15.125:0/3541913498 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40cef10 con 0x7fd0d415d590 2024-01-16T14:56:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.667+0000 7fd0d3400640 1 --2- 172.21.15.125:0/3541913498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d40cfb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009dc0 tx=0x7fd0cc02be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1f6474ef7b39fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.668+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc02cc30 con 0x7fd0d415d590 2024-01-16T14:56:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.668+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0cc02cd90 con 0x7fd0d415d590 2024-01-16T14:56:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.668+0000 7fd0d2000640 1 -- 172.21.15.125:0/3541913498 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc034660 con 0x7fd0d415d590 2024-01-16T14:56:54.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.669+0000 7fd0da800640 1 -- 172.21.15.125:0/3541913498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 msgr2=0x7fd0d40cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.669+0000 7fd0da800640 1 --2- 172.21.15.125:0/3541913498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d40cfb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009dc0 tx=0x7fd0cc02be80 comp rx=0 tx=0).stop 2024-01-16T14:56:54.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.669+0000 7fd0da800640 1 -- 172.21.15.125:0/3541913498 shutdown_connections 2024-01-16T14:56:54.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.669+0000 7fd0da800640 1 --2- 172.21.15.125:0/3541913498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d40cfb70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.669+0000 7fd0da800640 1 -- 172.21.15.125:0/3541913498 >> 172.21.15.125:0/3541913498 conn(0x7fd0d409c540 msgr2=0x7fd0d40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.670+0000 7fd0da800640 1 -- 172.21.15.125:0/3541913498 shutdown_connections 2024-01-16T14:56:54.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.670+0000 7fd0da800640 1 -- 172.21.15.125:0/3541913498 wait complete. 2024-01-16T14:56:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.673+0000 7fd0da800640 1 Processor -- start 2024-01-16T14:56:54.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.673+0000 7fd0da800640 1 -- start start 2024-01-16T14:56:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.675+0000 7fd0da800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 0x7fd0d40cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.676+0000 7fd0da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 0x7fd0d41f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.676+0000 7fd0da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d41f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0da800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4147e10 con 0x7fd0d40d00b0 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 0x7fd0d40cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 0x7fd0d40cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45684/0 (socket says 172.21.15.125:45684) 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0d3400640 1 -- 172.21.15.125:0/2860483726 learned_addr learned my addr 172.21.15.125:0/2860483726 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0d3e00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d41f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4148390 con 0x7fd0d40ce8c0 2024-01-16T14:56:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.677+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40cc7e0 con 0x7fd0d415d590 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d2a00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 0x7fd0d41f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d3e00640 1 -- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 msgr2=0x7fd0d40cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d3e00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 0x7fd0d40cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d3e00640 1 -- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 msgr2=0x7fd0d41f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d3e00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 0x7fd0d41f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.678+0000 7fd0d3e00640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc009a70 con 0x7fd0d415d590 2024-01-16T14:56:54.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.679+0000 7fd0d2a00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 0x7fd0d41f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:54.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.679+0000 7fd0d3e00640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d41f6440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c390 tx=0x7fd0c400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.680+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c40052d0 con 0x7fd0d415d590 2024-01-16T14:56:54.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.680+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c4005430 con 0x7fd0d415d590 2024-01-16T14:56:54.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.680+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d40ccac0 con 0x7fd0d415d590 2024-01-16T14:56:54.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.680+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d414db90 con 0x7fd0d415d590 2024-01-16T14:56:54.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.681+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4003830 con 0x7fd0d415d590 2024-01-16T14:56:54.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.682+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a4005250 con 0x7fd0d415d590 2024-01-16T14:56:54.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.685+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd0c4003990 con 0x7fd0d415d590 2024-01-16T14:56:54.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.687+0000 7fd0cb400640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 0x7fd0ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.687+0000 7fd0d3400640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 0x7fd0ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.688+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c4089060 con 0x7fd0d415d590 2024-01-16T14:56:54.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.689+0000 7fd0d3400640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 0x7fd0ac06d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc02bec0 tx=0x7fd0cc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.699+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0c4053ee0 con 0x7fd0d415d590 2024-01-16T14:56:54.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.747+0000 7f7390a00640 1 Processor -- start 2024-01-16T14:56:54.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.748+0000 7f7390a00640 1 -- start start 2024-01-16T14:56:54.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.750+0000 7f7390a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f73880cd170 0x7f7388071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.750+0000 7f7390a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7388071690 con 0x7f73880cf9a0 2024-01-16T14:56:54.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.750+0000 7f7390a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7388071970 con 0x7f738814b2c0 2024-01-16T14:56:54.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.750+0000 7f7390a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738814b6a0 con 0x7f73880cd170 2024-01-16T14:56:54.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.751+0000 7f738d600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f73880cf9a0 0x7f7388070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36576/0 (socket says 172.21.15.125:36576) 2024-01-16T14:56:54.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.751+0000 7f738d600640 1 -- 172.21.15.125:0/1723433093 learned_addr learned my addr 172.21.15.125:0/1723433093 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:54.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.751+0000 7f738ea00640 1 --2- 172.21.15.125:0/1723433093 >> v2:172.21.15.137:3300/0 conn(0x7f73880cd170 0x7f7388071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.753+0000 7f738ea00640 1 -- 172.21.15.125:0/1723433093 >> v1:172.21.15.137:6789/0 conn(0x7f738814b2c0 legacy=0x7f73880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.753+0000 7f738ea00640 1 -- 172.21.15.125:0/1723433093 >> v1:172.21.15.125:6789/0 conn(0x7f73880cf9a0 legacy=0x7f7388070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.753+0000 7f738ea00640 1 -- 172.21.15.125:0/1723433093 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388071c50 con 0x7f73880cd170 2024-01-16T14:56:54.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.753+0000 7f738cc00640 1 -- 172.21.15.125:0/1723433093 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846696953 0 0) 0x7f7388071970 con 0x7f738814b2c0 2024-01-16T14:56:54.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.755+0000 7f738ea00640 1 --2- 172.21.15.125:0/1723433093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f7388071070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7380009d90 tx=0x7f738002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=286de98c6e4914e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.756+0000 7f738cc00640 1 -- 172.21.15.125:0/1723433093 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738002cd30 con 0x7f73880cd170 2024-01-16T14:56:54.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.756+0000 7f738cc00640 1 -- 172.21.15.125:0/1723433093 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738002ce90 con 0x7f73880cd170 2024-01-16T14:56:54.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.756+0000 7f738cc00640 1 -- 172.21.15.125:0/1723433093 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380034660 con 0x7f73880cd170 2024-01-16T14:56:54.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.757+0000 7f7390a00640 1 -- 172.21.15.125:0/1723433093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 msgr2=0x7f7388071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.757+0000 7f7390a00640 1 --2- 172.21.15.125:0/1723433093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f7388071070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7380009d90 tx=0x7f738002bf80 comp rx=0 tx=0).stop 2024-01-16T14:56:54.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.758+0000 7f7390a00640 1 -- 172.21.15.125:0/1723433093 shutdown_connections 2024-01-16T14:56:54.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.758+0000 7f7390a00640 1 --2- 172.21.15.125:0/1723433093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f7388071070 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.758+0000 7f7390a00640 1 -- 172.21.15.125:0/1723433093 >> 172.21.15.125:0/1723433093 conn(0x7f738806a800 msgr2=0x7f738806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.758+0000 7f7390a00640 1 -- 172.21.15.125:0/1723433093 shutdown_connections 2024-01-16T14:56:54.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.759+0000 7f7390a00640 1 -- 172.21.15.125:0/1723433093 wait complete. 2024-01-16T14:56:54.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.762+0000 7f7390a00640 1 Processor -- start 2024-01-16T14:56:54.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.763+0000 7f7390a00640 1 -- start start 2024-01-16T14:56:54.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.765+0000 7f7390a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f738815d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.765+0000 7f7390a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.765+0000 7f7390a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 0x7f73880cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f738d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f738d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52540/0 (socket says 172.21.15.125:52540) 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f738d600640 1 -- 172.21.15.125:0/14252901 learned_addr learned my addr 172.21.15.125:0/14252901 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388160fb0 con 0x7f738814b2c0 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738815f500 con 0x7f73880cd170 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388164120 con 0x7f73880cf9a0 2024-01-16T14:56:54.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.766+0000 7f738e000640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f738815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.767+0000 7f738ea00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 0x7f73880cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738d600640 1 -- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 msgr2=0x7f738815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738d600640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f738815d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738d600640 1 -- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 msgr2=0x7f73880cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738d600640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 0x7f73880cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738d600640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380009a70 con 0x7f73880cf9a0 2024-01-16T14:56:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.768+0000 7f738ea00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 0x7f73880cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.769+0000 7f738d600640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f737000c4d0 tx=0x7f737000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.769+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370005ce0 con 0x7f73880cf9a0 2024-01-16T14:56:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.770+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7388164400 con 0x7f73880cf9a0 2024-01-16T14:56:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.770+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370005e40 con 0x7f73880cf9a0 2024-01-16T14:56:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.770+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370005ce0 con 0x7f73880cf9a0 2024-01-16T14:56:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.770+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738815d890 con 0x7f73880cf9a0 2024-01-16T14:56:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.772+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738815da20 con 0x7f73880cf9a0 2024-01-16T14:56:54.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.774+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7370007070 con 0x7f73880cf9a0 2024-01-16T14:56:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.776+0000 7f7386000640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 0x7f736006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.777+0000 7f738e000640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 0x7f736006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.778+0000 7f738e000640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 0x7f736006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7378000f60 tx=0x7f7378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.778+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7370057810 con 0x7f73880cf9a0 2024-01-16T14:56:54.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.791+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73700542e0 con 0x7f73880cf9a0 2024-01-16T14:56:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.809+0000 7f0903e00640 1 Processor -- start 2024-01-16T14:56:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.810+0000 7f0903e00640 1 -- start start 2024-01-16T14:56:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.812+0000 7f0902a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f090415d4b0 0x7f09040d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36592/0 (socket says 172.21.15.125:36592) 2024-01-16T14:56:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.812+0000 7f0903e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f09040cf9a0 0x7f09040cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.812+0000 7f0902a00640 1 -- 172.21.15.125:0/2239300723 learned_addr learned my addr 172.21.15.125:0/2239300723 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.812+0000 7f0902000640 1 --2- 172.21.15.125:0/2239300723 >> v2:172.21.15.125:3300/0 conn(0x7f09040cf9a0 0x7f09040cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.813+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040d0320 con 0x7f090415d4b0 2024-01-16T14:56:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.813+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040d1c60 con 0x7f09040b31d0 2024-01-16T14:56:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.813+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652085114 0 0) 0x7f09040d0320 con 0x7f090415d4b0 2024-01-16T14:56:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040b2730 con 0x7f09040cf9a0 2024-01-16T14:56:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e4003520 con 0x7f090415d4b0 2024-01-16T14:56:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.815+0000 7fe5b5200640 1 Processor -- start 2024-01-16T14:56:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447142587 0 0) 0x7f08e4003520 con 0x7f090415d4b0 2024-01-16T14:56:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0902000640 1 -- 172.21.15.125:0/2239300723 >> v1:172.21.15.137:6789/0 conn(0x7f09040b31d0 legacy=0x7f09040b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0902000640 1 -- 172.21.15.125:0/2239300723 >> v1:172.21.15.125:6789/0 conn(0x7f090415d4b0 legacy=0x7f09040d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.814+0000 7f0902000640 1 -- 172.21.15.125:0/2239300723 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040d1ec0 con 0x7f09040cf9a0 2024-01-16T14:56:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.815+0000 7fe5b5200640 1 -- start start 2024-01-16T14:56:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.816+0000 7f0902000640 1 --2- 172.21.15.125:0/2239300723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09040cf9a0 0x7f09040cfde0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0097a0 tx=0x7f08ec02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b872d75b15689bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec02f050 con 0x7f09040cf9a0 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5b015d590 0x7fe5b00cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 0x7fe5b00ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b00d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b00d0530 con 0x7fe5b015d590 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b00b6750 con 0x7fe5b00d00b0 2024-01-16T14:56:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7fe5b5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b00cf1f0 con 0x7fe5b00ce6f0 2024-01-16T14:56:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec031500 con 0x7f09040cf9a0 2024-01-16T14:56:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.817+0000 7f0901600640 1 -- 172.21.15.125:0/2239300723 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4001670 con 0x7f09040cf9a0 2024-01-16T14:56:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5ade00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5b015d590 0x7fe5b00cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5ae800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b00d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09040cf9a0 msgr2=0x7f09040cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7f0903e00640 1 --2- 172.21.15.125:0/2239300723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09040cf9a0 0x7f09040cfde0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0097a0 tx=0x7f08ec02b820 comp rx=0 tx=0).stop 2024-01-16T14:56:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5ae800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b00d0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52552/0 (socket says 172.21.15.125:52552) 2024-01-16T14:56:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5a7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 0x7fe5b00ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.819+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 shutdown_connections 2024-01-16T14:56:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5a7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 0x7fe5b00ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45708/0 (socket says 172.21.15.125:45708) 2024-01-16T14:56:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5a7e00640 1 -- 172.21.15.125:0/610441032 learned_addr learned my addr 172.21.15.125:0/610441032 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.819+0000 7f0903e00640 1 --2- 172.21.15.125:0/2239300723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09040cf9a0 0x7f09040cfde0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.819+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 >> 172.21.15.125:0/2239300723 conn(0x7f09040d3000 msgr2=0x7f09040d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.818+0000 7fe5ade00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5b015d590 0x7fe5b00cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47990/0 (socket says 172.21.15.125:47990) 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.819+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 shutdown_connections 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.819+0000 7f0903e00640 1 -- 172.21.15.125:0/2239300723 wait complete. 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.820+0000 7fe5ade00640 1 -- 172.21.15.125:0/610441032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 msgr2=0x7fe5b00d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.820+0000 7fe5ade00640 1 --2- 172.21.15.125:0/610441032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b00d0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.820+0000 7fe5ade00640 1 -- 172.21.15.125:0/610441032 >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 msgr2=0x7fe5b00ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.820+0000 7fe5ade00640 1 --2- 172.21.15.125:0/610441032 >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 0x7fe5b00ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.820+0000 7fe5ade00640 1 -- 172.21.15.125:0/610441032 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b00cef10 con 0x7fe5b015d590 2024-01-16T14:56:54.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.822+0000 7fe5ade00640 1 --2- 172.21.15.125:0/610441032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b00cfb70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5b008d970 tx=0x7fe59802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113b8e0574e4316b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.822+0000 7f0903e00640 1 Processor -- start 2024-01-16T14:56:54.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7f0903e00640 1 -- start start 2024-01-16T14:56:54.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7f4be8c00640 1 Processor -- start 2024-01-16T14:56:54.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7f4be8c00640 1 -- start start 2024-01-16T14:56:54.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f0903e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09040b31d0 0x7f090409d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 0x7f4be40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be8c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4be40b31d0 0x7f4be40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f0903e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7fe5ad400640 1 -- 172.21.15.125:0/610441032 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59802e070 con 0x7fe5b015d590 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7fe5ad400640 1 -- 172.21.15.125:0/610441032 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5980315a0 con 0x7fe5b015d590 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.823+0000 7fe5ad400640 1 -- 172.21.15.125:0/610441032 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598031930 con 0x7fe5b015d590 2024-01-16T14:56:54.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.824+0000 7fe5b5200640 1 -- 172.21.15.125:0/610441032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 msgr2=0x7fe5b00cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.824+0000 7fe5b5200640 1 --2- 172.21.15.125:0/610441032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b00cfb70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5b008d970 tx=0x7fe59802b840 comp rx=0 tx=0).stop 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 -- 172.21.15.125:0/610441032 shutdown_connections 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 --2- 172.21.15.125:0/610441032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b00d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 --2- 172.21.15.125:0/610441032 >> v2:172.21.15.137:3300/0 conn(0x7fe5b00d00b0 0x7fe5b00ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 --2- 172.21.15.125:0/610441032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b00cfb70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 -- 172.21.15.125:0/610441032 >> 172.21.15.125:0/610441032 conn(0x7fe5b009c540 msgr2=0x7fe5b00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f0903e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be8c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40d0320 con 0x7f4be415d4b0 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be8c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40d1c60 con 0x7f4be40b31d0 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40b2730 con 0x7f4be40cf9a0 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be3400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4be415d4b0 0x7f4be40d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36596/0 (socket says 172.21.15.125:36596) 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7f4be3400640 1 -- 172.21.15.125:0/1476485083 learned_addr learned my addr 172.21.15.125:0/1476485083 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 -- 172.21.15.125:0/610441032 shutdown_connections 2024-01-16T14:56:54.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.825+0000 7fe5b5200640 1 -- 172.21.15.125:0/610441032 wait complete. 2024-01-16T14:56:54.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f4be3e00640 1 --2- 172.21.15.125:0/1476485083 >> v2:172.21.15.137:3300/0 conn(0x7f4be40b31d0 0x7f4be40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f4be2a00640 1 --2- 172.21.15.125:0/1476485083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 0x7f4be40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0902000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0902000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52554/0 (socket says 172.21.15.125:52554) 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0903400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0903400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48002/0 (socket says 172.21.15.125:48002) 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0902000640 1 -- 172.21.15.125:0/3684032569 learned_addr learned my addr 172.21.15.125:0/3684032569 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0903e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0904151010 con 0x7f090415d4b0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0903e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090409f5c0 con 0x7f09040b31d0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.826+0000 7f0903e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090409db80 con 0x7f09040cf9a0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0903400640 1 -- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 msgr2=0x7f0904150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0903400640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0903400640 1 -- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09040b31d0 msgr2=0x7f090409d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0903400640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09040b31d0 0x7f090409d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0903400640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec009400 con 0x7f090415d4b0 2024-01-16T14:56:54.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f0902000640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:54.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.828+0000 7fe5b5200640 1 Processor -- start 2024-01-16T14:56:54.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f4be2a00640 1 -- 172.21.15.125:0/1476485083 >> v2:172.21.15.137:3300/0 conn(0x7f4be40b31d0 msgr2=0x7f4be40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f4be2a00640 1 --2- 172.21.15.125:0/1476485083 >> v2:172.21.15.137:3300/0 conn(0x7f4be40b31d0 0x7f4be40b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f4be2a00640 1 -- 172.21.15.125:0/1476485083 >> v1:172.21.15.125:6789/0 conn(0x7f4be415d4b0 legacy=0x7f4be40d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.827+0000 7f4be2a00640 1 -- 172.21.15.125:0/1476485083 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40d1ec0 con 0x7f4be40cf9a0 2024-01-16T14:56:54.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.828+0000 7f0903400640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00c760 tx=0x7f08fc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.828+0000 7f4be2a00640 1 --2- 172.21.15.125:0/1476485083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 0x7f4be40cfde0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009dc0 tx=0x7f4bdc02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9addbb643d6678 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc006d50 con 0x7f090415d4b0 2024-01-16T14:56:54.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f0903e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090409efb0 con 0x7f090415d4b0 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.828+0000 7fe5b5200640 1 -- start start 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b0143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b00d00b0 0x7fe5b006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b0141310 con 0x7fe5b015d590 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f4be2000640 1 -- 172.21.15.125:0/1476485083 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc02cbb0 con 0x7f4be40cf9a0 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f4be2000640 1 -- 172.21.15.125:0/1476485083 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc02cd10 con 0x7f4be40cf9a0 2024-01-16T14:56:54.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f4be2000640 1 -- 172.21.15.125:0/1476485083 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc034660 con 0x7f4be40cf9a0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b0141560 con 0x7fe5b00d00b0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5b5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b0141840 con 0x7fe5b00ce6f0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5ae800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7f0903e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0904150d30 con 0x7f090415d4b0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc00ce10 con 0x7f090415d4b0 2024-01-16T14:56:54.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc006d50 con 0x7f090415d4b0 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5ae800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48010/0 (socket says 172.21.15.125:48010) 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.829+0000 7fe5ae800640 1 -- 172.21.15.125:0/3956866187 learned_addr learned my addr 172.21.15.125:0/3956866187 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7fe5ae800640 1 -- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 msgr2=0x7fe5b0143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7fe5ae800640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b0143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7fe5ae800640 1 -- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b00d00b0 msgr2=0x7fe5b006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7fe5ae800640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b00d00b0 0x7fe5b006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7fe5ae800640 1 -- 172.21.15.125:0/3956866187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598009400 con 0x7fe5b015d590 2024-01-16T14:56:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.831+0000 7fe5ae800640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800efc0 tx=0x7fe5a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7f4be8c00640 1 -- 172.21.15.125:0/1476485083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 msgr2=0x7f4be40cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.830+0000 7f4be8c00640 1 --2- 172.21.15.125:0/1476485083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 0x7f4be40cfde0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009dc0 tx=0x7f4bdc02be00 comp rx=0 tx=0).stop 2024-01-16T14:56:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.831+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc005250 con 0x7f090415d4b0 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 -- 172.21.15.125:0/1476485083 shutdown_connections 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 --2- 172.21.15.125:0/1476485083 >> v2:172.21.15.137:3300/0 conn(0x7f4be40b31d0 0x7f4be40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 --2- 172.21.15.125:0/1476485083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be40cf9a0 0x7f4be40cfde0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 -- 172.21.15.125:0/1476485083 >> 172.21.15.125:0/1476485083 conn(0x7f4be40d3000 msgr2=0x7f4be40d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 -- 172.21.15.125:0/1476485083 shutdown_connections 2024-01-16T14:56:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7f4be8c00640 1 -- 172.21.15.125:0/1476485083 wait complete. 2024-01-16T14:56:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8014480 con 0x7fe5b015d590 2024-01-16T14:56:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a800f040 con 0x7fe5b015d590 2024-01-16T14:56:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8014800 con 0x7fe5b015d590 2024-01-16T14:56:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.832+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b0141b20 con 0x7fe5b015d590 2024-01-16T14:56:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.833+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b00ca640 con 0x7fe5b015d590 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.835+0000 7f4be8c00640 1 Processor -- start 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.835+0000 7f4be8c00640 1 -- start start 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.837+0000 7f4be8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.837+0000 7f4be8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be40cf9a0 0x7f4be40afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.837+0000 7f4be2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be40cf9a0 0x7f4be40afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.837+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b0142040 con 0x7fe5b015d590 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.837+0000 7f4be8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be415d4b0 0x7f4be40b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be8c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41f3740 con 0x7f4be40cf9a0 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be4160eb0 con 0x7f4be40b31d0 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be415ef40 con 0x7f4be415d4b0 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45738/0 (socket says 172.21.15.125:45738) 2024-01-16T14:56:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be3400640 1 -- 172.21.15.125:0/2773359283 learned_addr learned my addr 172.21.15.125:0/2773359283 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5a8014960 con 0x7fe5b015d590 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.838+0000 7f4be3e00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be415d4b0 0x7f4be40b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 -- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be415d4b0 msgr2=0x7f4be40b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be415d4b0 0x7f4be40b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 -- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be40cf9a0 msgr2=0x7f4be40afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be40cf9a0 0x7f4be40afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 -- 172.21.15.125:0/2773359283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc009a70 con 0x7f4be40b31d0 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.839+0000 7f4be3400640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c460 tx=0x7f4bcc00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7fe5a6000640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 0x7fe58806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5a808a8f0 con 0x7fe5b015d590 2024-01-16T14:56:54.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc005ce0 con 0x7f4be40b31d0 2024-01-16T14:56:54.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f08fc003970 con 0x7f090415d4b0 2024-01-16T14:56:54.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc005e60 con 0x7f4be40b31d0 2024-01-16T14:56:54.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.840+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0072b0 con 0x7f4be40b31d0 2024-01-16T14:56:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.841+0000 7f08faa00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 0x7f08dc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.841+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08fc0893d0 con 0x7f090415d4b0 2024-01-16T14:56:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.841+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be415ec60 con 0x7f4be40b31d0 2024-01-16T14:56:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.842+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be41f3360 con 0x7f4be40b31d0 2024-01-16T14:56:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.843+0000 7f0902a00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 0x7f08dc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.843+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be41f3880 con 0x7f4be40b31d0 2024-01-16T14:56:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.843+0000 7fe5ade00640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 0x7fe58806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.844+0000 7fe5ade00640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 0x7fe58806d7d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe598036000 tx=0x7fe598030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.846+0000 7f0902a00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 0x7f08dc06d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08f4001de0 tx=0x7f08f40010a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.846+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4bcc007410 con 0x7f4be40b31d0 2024-01-16T14:56:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.847+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08fc0541b0 con 0x7f090415d4b0 2024-01-16T14:56:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.847+0000 7f4bdb400640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 0x7f4bbc06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.847+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bcc08a250 con 0x7f4be40b31d0 2024-01-16T14:56:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.848+0000 7f4be2a00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 0x7f4bbc06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.850+0000 7f4be2a00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 0x7f4bbc06d7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc036000 tx=0x7f4bdc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.857+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5b0142040 con 0x7fe5b015d590 2024-01-16T14:56:54.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.859+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bcc055130 con 0x7f4be40b31d0 2024-01-16T14:56:54.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.894+0000 7fac47a00640 1 Processor -- start 2024-01-16T14:56:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.894+0000 7fac47a00640 1 -- start start 2024-01-16T14:56:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.896+0000 7fac47a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fac400b3eb0 0x7fac400b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac47a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400cc7b0 0x7fac400ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac47a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac400b4870 con 0x7fac4015a320 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac47a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4015a700 con 0x7fac400b3eb0 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac47a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4006cdc0 con 0x7fac400cc7b0 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac45000640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fac4015a320 0x7fac4006c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55378/0 (socket says 172.21.15.125:55378) 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac45000640 1 -- 172.21.15.125:0/2951871253 learned_addr learned my addr 172.21.15.125:0/2951871253 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.897+0000 7fac3fe00640 1 --2- 172.21.15.125:0/2951871253 >> v2:172.21.15.137:3300/0 conn(0x7fac400b3eb0 0x7fac400b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.898+0000 7fac45a00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400cc7b0 0x7fac400ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.898+0000 7fac3fe00640 1 -- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400cc7b0 msgr2=0x7fac400ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.898+0000 7fac3fe00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400cc7b0 0x7fac400ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.898+0000 7fac3fe00640 1 -- 172.21.15.125:0/2951871253 >> v1:172.21.15.137:6789/0 conn(0x7fac4015a320 legacy=0x7fac4006c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.898+0000 7fac3fe00640 1 -- 172.21.15.125:0/2951871253 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4006d0a0 con 0x7fac400b3eb0 2024-01-16T14:56:54.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.902+0000 7fac3fe00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400b3eb0 0x7fac400b4330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac2800a940 tx=0x7fac2802cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3558429b95998d77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.903+0000 7fac3f400640 1 -- 172.21.15.125:0/2951871253 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2802db30 con 0x7fac400b3eb0 2024-01-16T14:56:54.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.903+0000 7fac3f400640 1 -- 172.21.15.125:0/2951871253 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2802dc90 con 0x7fac400b3eb0 2024-01-16T14:56:54.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.903+0000 7fac3f400640 1 -- 172.21.15.125:0/2951871253 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28035440 con 0x7fac400b3eb0 2024-01-16T14:56:54.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.904+0000 7fac47a00640 1 -- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400b3eb0 msgr2=0x7fac400b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.904+0000 7fac47a00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400b3eb0 0x7fac400b4330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac2800a940 tx=0x7fac2802cd80 comp rx=0 tx=0).stop 2024-01-16T14:56:54.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.904+0000 7fac47a00640 1 -- 172.21.15.125:0/2951871253 shutdown_connections 2024-01-16T14:56:54.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.904+0000 7fac47a00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400cc7b0 0x7fac400ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.905+0000 7fac47a00640 1 --2- 172.21.15.125:0/2951871253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400b3eb0 0x7fac400b4330 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.905+0000 7fac47a00640 1 -- 172.21.15.125:0/2951871253 >> 172.21.15.125:0/2951871253 conn(0x7fac400aea00 msgr2=0x7fac400aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.905+0000 7fac47a00640 1 -- 172.21.15.125:0/2951871253 shutdown_connections 2024-01-16T14:56:54.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.905+0000 7fac47a00640 1 -- 172.21.15.125:0/2951871253 wait complete. 2024-01-16T14:56:54.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.906+0000 7f5fad200640 1 Processor -- start 2024-01-16T14:56:54.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.907+0000 7f5fad200640 1 -- start start 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7f5fad200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7f5fad200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5fa806c880 0x7f5fa80b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7f5fad200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa806cdc0 con 0x7f5fa80b43f0 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7f5fad200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa815a700 con 0x7f5fa806c880 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7f5fad200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80b48a0 con 0x7f5fa815a320 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.909+0000 7fac47a00640 1 Processor -- start 2024-01-16T14:56:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.910+0000 7fac47a00640 1 -- start start 2024-01-16T14:56:54.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.910+0000 7f5fa5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.911+0000 7f5fa5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52590/0 (socket says 172.21.15.125:52590) 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.911+0000 7f5fa5e00640 1 -- 172.21.15.125:0/3304954563 learned_addr learned my addr 172.21.15.125:0/3304954563 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.910+0000 7f5fa6800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5fa80b43f0 0x7f5fa815ccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36598/0 (socket says 172.21.15.125:36598) 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.910+0000 7f5fa5400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5fa806c880 0x7f5fa80b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.911+0000 7f5fa4a00640 1 -- 172.21.15.125:0/3304954563 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985556257 0 0) 0x7f5fa806cdc0 con 0x7f5fa80b43f0 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.911+0000 7f5fa4a00640 1 -- 172.21.15.125:0/3304954563 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88003520 con 0x7f5fa80b43f0 2024-01-16T14:56:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7f5fa5e00640 1 -- 172.21.15.125:0/3304954563 >> v2:172.21.15.137:3300/0 conn(0x7f5fa806c880 msgr2=0x7f5fa80b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7f5fa5e00640 1 --2- 172.21.15.125:0/3304954563 >> v2:172.21.15.137:3300/0 conn(0x7f5fa806c880 0x7f5fa80b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7f5fa5e00640 1 -- 172.21.15.125:0/3304954563 >> v1:172.21.15.125:6789/0 conn(0x7f5fa80b43f0 legacy=0x7f5fa815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7f5fa5e00640 1 -- 172.21.15.125:0/3304954563 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa806d0a0 con 0x7f5fa815a320 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7fac47a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400b3eb0 0x7fac4015c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7fac47a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400cc7b0 0x7fac400b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7fac47a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.912+0000 7fac47a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac400b39f0 con 0x7fac4015a320 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac47a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac400b3710 con 0x7fac400cc7b0 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac47a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac401f1e00 con 0x7fac400b3eb0 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac45000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400b3eb0 0x7fac4015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400cc7b0 0x7fac400b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac45a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac45a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48032/0 (socket says 172.21.15.125:48032) 2024-01-16T14:56:54.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7fac45a00640 1 -- 172.21.15.125:0/1862387188 learned_addr learned my addr 172.21.15.125:0/1862387188 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.913+0000 7f5fa5e00640 1 --2- 172.21.15.125:0/3304954563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0008870 tx=0x7f5fa002dfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c710d013280463 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 -- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400b3eb0 msgr2=0x7fac4015c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400b3eb0 0x7fac4015c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 -- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400cc7b0 msgr2=0x7fac400b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400cc7b0 0x7fac400b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 -- 172.21.15.125:0/1862387188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2800a5a0 con 0x7fac4015a320 2024-01-16T14:56:54.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7f5fa4a00640 1 -- 172.21.15.125:0/3304954563 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa002ed80 con 0x7f5fa815a320 2024-01-16T14:56:54.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7f5fa4a00640 1 -- 172.21.15.125:0/3304954563 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0032070 con 0x7f5fa815a320 2024-01-16T14:56:54.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7f5fa4a00640 1 -- 172.21.15.125:0/3304954563 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0038600 con 0x7f5fa815a320 2024-01-16T14:56:54.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.914+0000 7fac45a00640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac3800efc0 tx=0x7fac3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.915+0000 7f08f5000640 1 Processor -- start 2024-01-16T14:56:54.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.915+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38014480 con 0x7fac4015a320 2024-01-16T14:56:54.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.915+0000 7f08f5000640 1 -- start start 2024-01-16T14:56:54.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.915+0000 7fac47a00640 1 -- 172.21.15.125:0/1862387188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac401f20e0 con 0x7fac4015a320 2024-01-16T14:56:54.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.915+0000 7fac47a00640 1 -- 172.21.15.125:0/1862387188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4013fbf0 con 0x7fac4015a320 2024-01-16T14:56:54.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3800f040 con 0x7fac4015a320 2024-01-16T14:56:54.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38014480 con 0x7fac4015a320 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 -- 172.21.15.125:0/3304954563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 msgr2=0x7f5fa806c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 --2- 172.21.15.125:0/3304954563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0008870 tx=0x7f5fa002dfd0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 -- 172.21.15.125:0/3304954563 shutdown_connections 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 --2- 172.21.15.125:0/3304954563 >> v2:172.21.15.137:3300/0 conn(0x7f5fa806c880 0x7f5fa80b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.917+0000 7f08f5000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 0x7f08f00cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 --2- 172.21.15.125:0/3304954563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815a320 0x7f5fa806c340 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.916+0000 7f5fad200640 1 -- 172.21.15.125:0/3304954563 >> 172.21.15.125:0/3304954563 conn(0x7f5fa80aea00 msgr2=0x7f5fa80aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08f5000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08f00d00b0 0x7f08f00ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f5fad200640 1 -- 172.21.15.125:0/3304954563 shutdown_connections 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08f5000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f00b6750 con 0x7f08f00ce6f0 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08f5000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f00d0530 con 0x7f08f015d590 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08f5000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f00cf1f0 con 0x7f08f00d00b0 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08ed400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08f00d00b0 0x7f08f00ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08ed400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08f00d00b0 0x7f08f00ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45778/0 (socket says 172.21.15.125:45778) 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f08ed400640 1 -- 172.21.15.125:0/527664511 learned_addr learned my addr 172.21.15.125:0/527664511 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.919+0000 7f08ede00640 1 --2- 172.21.15.125:0/527664511 >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 0x7f08f00cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.919+0000 7f08ed400640 1 -- 172.21.15.125:0/527664511 >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 msgr2=0x7f08f00cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.919+0000 7f08ed400640 1 --2- 172.21.15.125:0/527664511 >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 0x7f08f00cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.919+0000 7f08ed400640 1 -- 172.21.15.125:0/527664511 >> v1:172.21.15.125:6789/0 conn(0x7f08f00ce6f0 legacy=0x7f08f00d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:54.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.919+0000 7f08ed400640 1 -- 172.21.15.125:0/527664511 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00cef10 con 0x7f08f00d00b0 2024-01-16T14:56:54.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.920+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac401f2600 con 0x7fac4015a320 2024-01-16T14:56:54.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.920+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac380145e0 con 0x7fac4015a320 2024-01-16T14:56:54.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.921+0000 7f08ede00640 1 --2- 172.21.15.125:0/527664511 >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 0x7f08f00cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:54.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.922+0000 7f08ed400640 1 --2- 172.21.15.125:0/527664511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00d00b0 0x7f08f00ce1b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08f008d970 tx=0x7f08e002b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ad0a55f04fa8ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.922+0000 7fac3cc00640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 0x7fac1806f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.922+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac3808a1b0 con 0x7fac4015a320 2024-01-16T14:56:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.923+0000 7f08eca00640 1 -- 172.21.15.125:0/527664511 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0030860 con 0x7f08f00d00b0 2024-01-16T14:56:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.923+0000 7f08eca00640 1 -- 172.21.15.125:0/527664511 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e00309c0 con 0x7f08f00d00b0 2024-01-16T14:56:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.923+0000 7f08eca00640 1 -- 172.21.15.125:0/527664511 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0030db0 con 0x7f08f00d00b0 2024-01-16T14:56:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.923+0000 7fac45000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 0x7fac1806f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7fac45000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 0x7fac1806f9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fac30006110 tx=0x7fac30005fc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.918+0000 7f5fad200640 1 -- 172.21.15.125:0/3304954563 wait complete. 2024-01-16T14:56:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.921+0000 7f5fad200640 1 Processor -- start 2024-01-16T14:56:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 -- 172.21.15.125:0/527664511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00d00b0 msgr2=0x7f08f00ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 --2- 172.21.15.125:0/527664511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00d00b0 0x7f08f00ce1b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08f008d970 tx=0x7f08e002b9c0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 -- 172.21.15.125:0/527664511 shutdown_connections 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 --2- 172.21.15.125:0/527664511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00d00b0 0x7f08f00ce1b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 --2- 172.21.15.125:0/527664511 >> v2:172.21.15.125:3300/0 conn(0x7f08f015d590 0x7f08f00cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 -- 172.21.15.125:0/527664511 >> 172.21.15.125:0/527664511 conn(0x7f08f009c540 msgr2=0x7f08f00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 -- 172.21.15.125:0/527664511 shutdown_connections 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.924+0000 7f08f5000640 1 -- 172.21.15.125:0/527664511 wait complete. 2024-01-16T14:56:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.925+0000 7f5fad200640 1 -- start start 2024-01-16T14:56:54.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806c880 0x7f5fa80ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 0x7f5fa80adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 0x7f5fa80b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ae3a0 con 0x7f5fa815a320 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8140b40 con 0x7f5fa80b43f0 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.926+0000 7f5fad200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8140250 con 0x7f5fa806c880 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.927+0000 7f08f5000640 1 Processor -- start 2024-01-16T14:56:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.927+0000 7f08f5000640 1 -- start start 2024-01-16T14:56:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.928+0000 7f5fa5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 0x7f5fa80adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.928+0000 7f5fa5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 0x7f5fa80adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45794/0 (socket says 172.21.15.125:45794) 2024-01-16T14:56:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.928+0000 7f5fa5400640 1 -- 172.21.15.125:0/271276550 learned_addr learned my addr 172.21.15.125:0/271276550 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.928+0000 7f5fa6800640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 0x7f5fa80b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.929+0000 7f08f5000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08f5000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08f5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 0x7f08f00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f5fa6800640 1 -- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806c880 msgr2=0x7f5fa80ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f5fa6800640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806c880 0x7f5fa80ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f5fa6800640 1 -- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 msgr2=0x7f5fa80adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f5fa6800640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 0x7f5fa80adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f5fa6800640 1 -- 172.21.15.125:0/271276550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0008550 con 0x7f5fa815a320 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08f5000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00cd390 con 0x7f08f00d00b0 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08f5000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00b2310 con 0x7f08f00ce6f0 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08ede00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08f5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00b1d50 con 0x7f08f015d590 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08ed400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08ede00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45806/0 (socket says 172.21.15.125:45806) 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08ed400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48062/0 (socket says 172.21.15.125:48062) 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.930+0000 7f08ede00640 1 -- 172.21.15.125:0/143909425 learned_addr learned my addr 172.21.15.125:0/143909425 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f08ee800640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 0x7f08f00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f5fa6800640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 0x7f5fa80b3390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c3c0 tx=0x7f5f9000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac401f2600 con 0x7fac4015a320 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f08ede00640 1 -- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 msgr2=0x7f08f00b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f08ede00640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 0x7f08f00b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f08ede00640 1 -- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 msgr2=0x7f08f00cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.931+0000 7f08ede00640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.932+0000 7f08ede00640 1 -- 172.21.15.125:0/143909425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8009400 con 0x7f08f00ce6f0 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.932+0000 7f08ee800640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 0x7f08f00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.932+0000 7f08ed400640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.932+0000 7f08ede00640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08d800ec10 tx=0x7f08d800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.932+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f900074e0 con 0x7f5fa815a320 2024-01-16T14:56:54.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.933+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90007660 con 0x7f5fa815a320 2024-01-16T14:56:54.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.933+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90004a20 con 0x7f5fa815a320 2024-01-16T14:56:54.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.933+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80ae0d0 con 0x7f5fa815a320 2024-01-16T14:56:54.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.934+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa81407d0 con 0x7f5fa815a320 2024-01-16T14:56:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.934+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d800cc30 con 0x7f08f00ce6f0 2024-01-16T14:56:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.934+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d800cd90 con 0x7f08f00ce6f0 2024-01-16T14:56:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.934+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e00094d0 con 0x7f08f00ce6f0 2024-01-16T14:56:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.935+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f00cc650 con 0x7f08f00ce6f0 2024-01-16T14:56:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.936+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8013670 con 0x7f08f00ce6f0 2024-01-16T14:56:54.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.936+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c0005250 con 0x7f08f00ce6f0 2024-01-16T14:56:54.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.939+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5f90004b80 con 0x7f5fa815a320 2024-01-16T14:56:54.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.939+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f08d8005990 con 0x7f08f00ce6f0 2024-01-16T14:56:54.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.939+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa81598e0 con 0x7f5fa815a320 2024-01-16T14:56:54.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.941+0000 7f08e6000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 0x7f08c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.941+0000 7f5f9e000640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 0x7f5f8006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:54.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.941+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f90089570 con 0x7f5fa815a320 2024-01-16T14:56:54.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.941+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08d808a0f0 con 0x7f08f00ce6f0 2024-01-16T14:56:54.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.942+0000 7f08ed400640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 0x7f08c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.942+0000 7f5fa5e00640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 0x7f5f8006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:54.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.943+0000 7f08ed400640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 0x7f08c806d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.949+0000 7f5fa5e00640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 0x7f5f8006d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0039000 tx=0x7f5fa0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:54.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.951+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08d8055180 con 0x7f08f00ce6f0 2024-01-16T14:56:54.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:54.952+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f90054340 con 0x7f5fa815a320 2024-01-16T14:56:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.072+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd0a4005500 con 0x7fd0d415d590 2024-01-16T14:56:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.073+0000 7fd0cb400640 1 -- 172.21.15.125:0/2860483726 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0c4017020 con 0x7fd0d415d590 2024-01-16T14:56:55.055 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.076+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 msgr2=0x7fd0ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 0x7fd0ac06d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc02bec0 tx=0x7fd0cc030000 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 msgr2=0x7fd0d41f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d41f6440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c390 tx=0x7fd0c400c850 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 shutdown_connections 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0d415d590 0x7fd0d41f6440 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd0ac06b2c0 0x7fd0ac06d780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d40d00b0 0x7fd0d41f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 --2- 172.21.15.125:0/2860483726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0d40ce8c0 0x7fd0d40cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.077+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 >> 172.21.15.125:0/2860483726 conn(0x7fd0d409c540 msgr2=0x7fd0d4160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.081+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 shutdown_connections 2024-01-16T14:56:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.081+0000 7fd0da800640 1 -- 172.21.15.125:0/2860483726 wait complete. 2024-01-16T14:56:55.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-01-16T14:56:55.090 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.193+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f738815da20 con 0x7f73880cf9a0 2024-01-16T14:56:55.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.194+0000 7f7386000640 1 -- 172.21.15.125:0/14252901 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7370017030 con 0x7f73880cf9a0 2024-01-16T14:56:55.175 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:55.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.197+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 msgr2=0x7f736006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.197+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 0x7f736006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7378000f60 tx=0x7f7378009040 comp rx=0 tx=0).stop 2024-01-16T14:56:55.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.197+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 msgr2=0x7f73880cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.197+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f737000c4d0 tx=0x7f737000c990 comp rx=0 tx=0).stop 2024-01-16T14:56:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 shutdown_connections 2024-01-16T14:56:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f736006b350 0x7f736006d810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f738814b2c0 0x7f73880cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73880cf9a0 0x7f73880cc060 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 --2- 172.21.15.125:0/14252901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73880cd170 0x7f738815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 >> 172.21.15.125:0/14252901 conn(0x7f738806a800 msgr2=0x7f738806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 shutdown_connections 2024-01-16T14:56:55.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.199+0000 7f7390a00640 1 -- 172.21.15.125:0/14252901 wait complete. 2024-01-16T14:56:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.216+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f08c00057d0 con 0x7f08f00ce6f0 2024-01-16T14:56:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.217+0000 7f08e6000640 1 -- 172.21.15.125:0/143909425 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08d8017090 con 0x7f08f00ce6f0 2024-01-16T14:56:55.199 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.219+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 msgr2=0x7f08c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.219+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 0x7f08c806d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e003b040 comp rx=0 tx=0).stop 2024-01-16T14:56:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 msgr2=0x7f08f01567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08d800ec10 tx=0x7f08d800c520 comp rx=0 tx=0).stop 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 shutdown_connections 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08f015d590 0x7f08f00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c806b2c0 0x7f08c806d780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f00d00b0 0x7f08f00cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 --2- 172.21.15.125:0/143909425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f00ce6f0 0x7f08f01567e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 >> 172.21.15.125:0/143909425 conn(0x7f08f009c540 msgr2=0x7f08f015c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 shutdown_connections 2024-01-16T14:56:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.220+0000 7f08f5000640 1 -- 172.21.15.125:0/143909425 wait complete. 2024-01-16T14:56:55.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-01-16T14:56:55.207 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-01-16T14:56:55.217 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.257+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4be4067450 con 0x7f4be40b31d0 2024-01-16T14:56:55.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.258+0000 7f4bdb400640 1 -- 172.21.15.125:0/2773359283 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4bcc058750 con 0x7f4be40b31d0 2024-01-16T14:56:55.240 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:55.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.261+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 msgr2=0x7f4bbc06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.261+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 0x7f4bbc06d7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc036000 tx=0x7f4bdc030000 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.261+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 msgr2=0x7f4be415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.261+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c460 tx=0x7f4bcc00c920 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 shutdown_connections 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4be415d4b0 0x7f4be40b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4bbc06b310 0x7f4bbc06d7d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be40cf9a0 0x7f4be40afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 --2- 172.21.15.125:0/2773359283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4be40b31d0 0x7f4be415e2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 >> 172.21.15.125:0/2773359283 conn(0x7f4be40d3000 msgr2=0x7f4be40d1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 shutdown_connections 2024-01-16T14:56:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.262+0000 7f4be8c00640 1 -- 172.21.15.125:0/2773359283 wait complete. 2024-01-16T14:56:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.272+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f08cc005500 con 0x7f090415d4b0 2024-01-16T14:56:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.272+0000 7f08faa00640 1 -- 172.21.15.125:0/3684032569 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08fc0577d0 con 0x7f090415d4b0 2024-01-16T14:56:55.254 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:55.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 msgr2=0x7f08dc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 0x7f08dc06d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08f4001de0 tx=0x7f08f40010a0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 msgr2=0x7f09040b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00c760 tx=0x7f08fc00cc20 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 shutdown_connections 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08dc06b350 0x7f08dc06d810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090415d4b0 0x7f09040b7c30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09040cf9a0 0x7f0904150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 --2- 172.21.15.125:0/3684032569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09040b31d0 0x7f090409d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 >> 172.21.15.125:0/3684032569 conn(0x7f09040d3000 msgr2=0x7f090406ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 shutdown_connections 2024-01-16T14:56:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.276+0000 7f08f3e00640 1 -- 172.21.15.125:0/3684032569 wait complete. 2024-01-16T14:56:55.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-01-16T14:56:55.266 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.285+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5fa8067450 con 0x7f5fa815a320 2024-01-16T14:56:55.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.286+0000 7f5f9e000640 1 -- 172.21.15.125:0/271276550 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f90054340 con 0x7f5fa815a320 2024-01-16T14:56:55.268 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:55.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.287+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe5b0142040 con 0x7fe5b015d590 2024-01-16T14:56:55.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.287+0000 7fe5a6000640 1 -- 172.21.15.125:0/3956866187 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5b0142040 con 0x7fe5b015d590 2024-01-16T14:56:55.269 INFO:teuthology.orchestra.run.smithi125.stdout:55834574856 2024-01-16T14:56:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 msgr2=0x7f5f8006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 0x7f5f8006d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0039000 tx=0x7f5fa0030000 comp rx=0 tx=0).stop 2024-01-16T14:56:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 msgr2=0x7f5fa80b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 0x7f5fa80b3390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c3c0 tx=0x7f5f9000c880 comp rx=0 tx=0).stop 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 shutdown_connections 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5f8006b2f0 0x7f5f8006d7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fa815a320 0x7f5fa80b3390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa80b43f0 0x7f5fa80adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 --2- 172.21.15.125:0/271276550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806c880 0x7f5fa80ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.288+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 >> 172.21.15.125:0/271276550 conn(0x7f5fa80aea00 msgr2=0x7f5fa815b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.289+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 shutdown_connections 2024-01-16T14:56:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.289+0000 7f5fad200640 1 -- 172.21.15.125:0/271276550 wait complete. 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 msgr2=0x7fe58806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 0x7fe58806d7d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe598036000 tx=0x7fe598030000 comp rx=0 tx=0).stop 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 msgr2=0x7fe5b0158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800efc0 tx=0x7fe5a800c490 comp rx=0 tx=0).stop 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 shutdown_connections 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe58806b310 0x7fe58806d7d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b015d590 0x7fe5b0158390 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b00d00b0 0x7fe5b006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 --2- 172.21.15.125:0/3956866187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b00ce6f0 0x7fe5b0143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 >> 172.21.15.125:0/3956866187 conn(0x7fe5b009c540 msgr2=0x7fe5b00b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 shutdown_connections 2024-01-16T14:56:55.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.290+0000 7fe5b5200640 1 -- 172.21.15.125:0/3956866187 wait complete. 2024-01-16T14:56:55.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-01-16T14:56:55.279 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-01-16T14:56:55.287 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-01-16T14:56:55.290 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.310+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fac401f2600 con 0x7fac4015a320 2024-01-16T14:56:55.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.310+0000 7fac3cc00640 1 -- 172.21.15.125:0/1862387188 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac401f2600 con 0x7fac4015a320 2024-01-16T14:56:55.292 INFO:teuthology.orchestra.run.smithi125.stdout:55834574855 2024-01-16T14:56:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.312+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 msgr2=0x7fac1806f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.312+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 0x7fac1806f9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fac30006110 tx=0x7fac30005fc0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.312+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 msgr2=0x7fac400b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.312+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac3800efc0 tx=0x7fac3800c490 comp rx=0 tx=0).stop 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 shutdown_connections 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac1806d530 0x7fac1806f9f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4015a320 0x7fac400b30e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac400cc7b0 0x7fac400b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 --2- 172.21.15.125:0/1862387188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac400b3eb0 0x7fac4015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 >> 172.21.15.125:0/1862387188 conn(0x7fac400aea00 msgr2=0x7fac4006c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 shutdown_connections 2024-01-16T14:56:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.313+0000 7fac36000640 1 -- 172.21.15.125:0/1862387188 wait complete. 2024-01-16T14:56:55.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-01-16T14:56:55.311 DEBUG:teuthology.parallel:result is None 2024-01-16T14:56:55.311 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T14:56:55.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:56:55.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.574+0000 7f2296e00640 1 Processor -- start 2024-01-16T14:56:55.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.574+0000 7f2296e00640 1 -- start start 2024-01-16T14:56:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.575+0000 7f2296e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f22900cf9a0 0x7f2290166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.575+0000 7f2296e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229015c5f0 con 0x7f229013df80 2024-01-16T14:56:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.575+0000 7f2296e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229006c470 con 0x7f229013e360 2024-01-16T14:56:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.575+0000 7f2296e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229006c750 con 0x7f22900cf9a0 2024-01-16T14:56:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.575+0000 7f228fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f229013df80 0x7f229015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36620/0 (socket says 172.21.15.125:36620) 2024-01-16T14:56:55.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228fe00640 1 -- 172.21.15.125:0/2421030766 learned_addr learned my addr 172.21.15.125:0/2421030766 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315303653 0 0) 0x7f229015c5f0 con 0x7f229013df80 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270003520 con 0x7f229013df80 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f2294e00640 1 --2- 172.21.15.125:0/2421030766 >> v2:172.21.15.137:3300/0 conn(0x7f22900cf9a0 0x7f2290166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275579902 0 0) 0x7f2270003520 con 0x7f229013df80 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 >> v2:172.21.15.137:3300/0 conn(0x7f22900cf9a0 msgr2=0x7f2290166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 --2- 172.21.15.125:0/2421030766 >> v2:172.21.15.137:3300/0 conn(0x7f22900cf9a0 0x7f2290166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 >> v1:172.21.15.137:6789/0 conn(0x7f229013e360 legacy=0x7f22900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229006ca30 con 0x7f229013df80 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f22800019f0 con 0x7f229013df80 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 >> v1:172.21.15.125:6789/0 conn(0x7f229013df80 legacy=0x7f229015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22700039c0 0x7f2270003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.576+0000 7f228ea00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22700047c0 0x7f2270004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f228ea00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 0x7f227000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229006c750 con 0x7f2270005180 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227000c080 con 0x7f22700039c0 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227000c240 con 0x7f22700047c0 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 0x7f227000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f228fe00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22700039c0 0x7f2270003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 -- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22700047c0 msgr2=0x7f2270004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22700047c0 0x7f2270004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 -- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22700039c0 msgr2=0x7f2270003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22700039c0 0x7f2270003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.577+0000 7f2294e00640 1 -- 172.21.15.125:0/2421030766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2270003520 con 0x7f2270005180 2024-01-16T14:56:55.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.578+0000 7f2294e00640 1 -- 172.21.15.125:0/2421030766 reap_dead start 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.578+0000 7f2294e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 0x7f227000b980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2288009a90 tx=0x7f22700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca56ff5581d1ef41 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.578+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288004020 con 0x7f2270005180 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.578+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288004180 con 0x7f2270005180 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.578+0000 7f228ea00640 1 -- 172.21.15.125:0/2421030766 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228802c6b0 con 0x7f2270005180 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 -- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 msgr2=0x7f227000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 0x7f227000b980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2288009a90 tx=0x7f22700057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 -- 172.21.15.125:0/2421030766 shutdown_connections 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 --2- 172.21.15.125:0/2421030766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2270005180 0x7f227000b980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 -- 172.21.15.125:0/2421030766 >> 172.21.15.125:0/2421030766 conn(0x7f229006a800 msgr2=0x7f229006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 -- 172.21.15.125:0/2421030766 shutdown_connections 2024-01-16T14:56:55.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.579+0000 7f2296e00640 1 -- 172.21.15.125:0/2421030766 wait complete. 2024-01-16T14:56:55.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.581+0000 7f2296e00640 1 Processor -- start 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.581+0000 7f2296e00640 1 -- start start 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.581+0000 7f2296e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2270005180 0x7f22901f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.581+0000 7f2296e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f2296e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22900cdfb0 0x7f22900ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f2296e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900ce970 con 0x7f2290149aa0 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f2296e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22901f2a50 con 0x7f2270005180 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f2296e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229014ad20 con 0x7f22900cdfb0 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48086/0 (socket says 172.21.15.125:48086) 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 -- 172.21.15.125:0/1451397344 learned_addr learned my addr 172.21.15.125:0/1451397344 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 -- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22900cdfb0 msgr2=0x7f22900ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22900cdfb0 0x7f22900ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 -- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2270005180 msgr2=0x7f22901f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2270005180 0x7f22901f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 -- 172.21.15.125:0/1451397344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229014d340 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f228f400640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f227800c490 tx=0x7f227800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.582+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22780052d0 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.583+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2278005450 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.583+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278007260 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.583+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229014d620 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.583+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229014cff0 con 0x7f2290149aa0 2024-01-16T14:56:55.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.584+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22900675d0 con 0x7f2290149aa0 2024-01-16T14:56:55.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.584+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f22780073c0 con 0x7f2290149aa0 2024-01-16T14:56:55.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.585+0000 7f2287e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 0x7f226806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:55.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.585+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2278088f90 con 0x7f2290149aa0 2024-01-16T14:56:55.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.586+0000 7f228fe00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 0x7f226806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:55.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.586+0000 7f228fe00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 0x7f226806d7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22800013d0 tx=0x7f22800012e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:55.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.590+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2278053d60 con 0x7f2290149aa0 2024-01-16T14:56:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.760+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f229006cff0 con 0x7f226806b2f0 2024-01-16T14:56:55.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.770+0000 7f2287e00640 1 -- 172.21.15.125:0/1451397344 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264450 (secure 0 0 0) 0x7f22800078b0 con 0x7f226806b2f0 2024-01-16T14:56:55.756 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:55.757 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 msgr2=0x7f226806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 0x7f226806d7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22800013d0 tx=0x7f22800012e0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 msgr2=0x7f229014a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f227800c490 tx=0x7f227800c950 comp rx=0 tx=0).stop 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 shutdown_connections 2024-01-16T14:56:55.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22900cdfb0 0x7f22900ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f226806b2f0 0x7f226806d7b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290149aa0 0x7f229014a7e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 --2- 172.21.15.125:0/1451397344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2270005180 0x7f22901f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 >> 172.21.15.125:0/1451397344 conn(0x7f229006a800 msgr2=0x7f2290154100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 shutdown_connections 2024-01-16T14:56:55.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:55.775+0000 7f2296e00640 1 -- 172.21.15.125:0/1451397344 wait complete. 2024-01-16T14:56:55.775 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-16T14:56:53.920105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2887680,"data_stored":1751817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003105"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956233+0000","last_change":"2024-01-16T14:56:45.980276+0000","last_active":"2024-01-16T14:56:46.956233+0000","last_peered":"2024-01-16T14:56:46.956233+0000","last_clean":"2024-01-16T14:56:46.956233+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956233+0000","last_undegraded":"2024-01-16T14:56:46.956233+0000","last_fullsized":"2024-01-16T14:56:46.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:53.535044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951915+0000","last_change":"2024-01-16T14:56:45.975609+0000","last_active":"2024-01-16T14:56:46.951915+0000","last_peered":"2024-01-16T14:56:46.951915+0000","last_clean":"2024-01-16T14:56:46.951915+0000","last_became_active":"2024-01-16T14:56:45.975291+0000","last_became_peered":"2024-01-16T14:56:45.975291+0000","last_unstale":"2024-01-16T14:56:46.951915+0000","last_undegraded":"2024-01-16T14:56:46.951915+0000","last_fullsized":"2024-01-16T14:56:46.951915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310992+0000","last_change":"2024-01-16T14:56:45.977545+0000","last_active":"2024-01-16T14:56:47.310992+0000","last_peered":"2024-01-16T14:56:47.310992+0000","last_clean":"2024-01-16T14:56:47.310992+0000","last_became_active":"2024-01-16T14:56:45.977253+0000","last_became_peered":"2024-01-16T14:56:45.977253+0000","last_unstale":"2024-01-16T14:56:47.310992+0000","last_undegraded":"2024-01-16T14:56:47.310992+0000","last_fullsized":"2024-01-16T14:56:47.310992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:34.727060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311691+0000","last_change":"2024-01-16T14:56:45.985746+0000","last_active":"2024-01-16T14:56:47.311691+0000","last_peered":"2024-01-16T14:56:47.311691+0000","last_clean":"2024-01-16T14:56:47.311691+0000","last_became_active":"2024-01-16T14:56:45.984925+0000","last_became_peered":"2024-01-16T14:56:45.984925+0000","last_unstale":"2024-01-16T14:56:47.311691+0000","last_undegraded":"2024-01-16T14:56:47.311691+0000","last_fullsized":"2024-01-16T14:56:47.311691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954569+0000","last_change":"2024-01-16T14:56:45.973905+0000","last_active":"2024-01-16T14:56:46.954569+0000","last_peered":"2024-01-16T14:56:46.954569+0000","last_clean":"2024-01-16T14:56:46.954569+0000","last_became_active":"2024-01-16T14:56:45.973591+0000","last_became_peered":"2024-01-16T14:56:45.973591+0000","last_unstale":"2024-01-16T14:56:46.954569+0000","last_undegraded":"2024-01-16T14:56:46.954569+0000","last_fullsized":"2024-01-16T14:56:46.954569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310746+0000","last_change":"2024-01-16T14:56:45.981879+0000","last_active":"2024-01-16T14:56:47.310746+0000","last_peered":"2024-01-16T14:56:47.310746+0000","last_clean":"2024-01-16T14:56:47.310746+0000","last_became_active":"2024-01-16T14:56:45.981506+0000","last_became_peered":"2024-01-16T14:56:45.981506+0000","last_unstale":"2024-01-16T14:56:47.310746+0000","last_undegraded":"2024-01-16T14:56:47.310746+0000","last_fullsized":"2024-01-16T14:56:47.310746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954278+0000","last_change":"2024-01-16T14:56:45.978039+0000","last_active":"2024-01-16T14:56:46.954278+0000","last_peered":"2024-01-16T14:56:46.954278+0000","last_clean":"2024-01-16T14:56:46.954278+0000","last_became_active":"2024-01-16T14:56:45.976116+0000","last_became_peered":"2024-01-16T14:56:45.976116+0000","last_unstale":"2024-01-16T14:56:46.954278+0000","last_undegraded":"2024-01-16T14:56:46.954278+0000","last_fullsized":"2024-01-16T14:56:46.954278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309521+0000","last_change":"2024-01-16T14:56:45.992939+0000","last_active":"2024-01-16T14:56:47.309521+0000","last_peered":"2024-01-16T14:56:47.309521+0000","last_clean":"2024-01-16T14:56:47.309521+0000","last_became_active":"2024-01-16T14:56:45.992563+0000","last_became_peered":"2024-01-16T14:56:45.992563+0000","last_unstale":"2024-01-16T14:56:47.309521+0000","last_undegraded":"2024-01-16T14:56:47.309521+0000","last_fullsized":"2024-01-16T14:56:47.309521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956108+0000","last_change":"2024-01-16T14:56:45.980818+0000","last_active":"2024-01-16T14:56:46.956108+0000","last_peered":"2024-01-16T14:56:46.956108+0000","last_clean":"2024-01-16T14:56:46.956108+0000","last_became_active":"2024-01-16T14:56:45.980565+0000","last_became_peered":"2024-01-16T14:56:45.980565+0000","last_unstale":"2024-01-16T14:56:46.956108+0000","last_undegraded":"2024-01-16T14:56:46.956108+0000","last_fullsized":"2024-01-16T14:56:46.956108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:05.361881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312563+0000","last_change":"2024-01-16T14:56:45.990281+0000","last_active":"2024-01-16T14:56:47.312563+0000","last_peered":"2024-01-16T14:56:47.312563+0000","last_clean":"2024-01-16T14:56:47.312563+0000","last_became_active":"2024-01-16T14:56:45.990014+0000","last_became_peered":"2024-01-16T14:56:45.990014+0000","last_unstale":"2024-01-16T14:56:47.312563+0000","last_undegraded":"2024-01-16T14:56:47.312563+0000","last_fullsized":"2024-01-16T14:56:47.312563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312223+0000","last_change":"2024-01-16T14:56:45.990395+0000","last_active":"2024-01-16T14:56:47.312223+0000","last_peered":"2024-01-16T14:56:47.312223+0000","last_clean":"2024-01-16T14:56:47.312223+0000","last_became_active":"2024-01-16T14:56:45.990100+0000","last_became_peered":"2024-01-16T14:56:45.990100+0000","last_unstale":"2024-01-16T14:56:47.312223+0000","last_undegraded":"2024-01-16T14:56:47.312223+0000","last_fullsized":"2024-01-16T14:56:47.312223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954100+0000","last_change":"2024-01-16T14:56:45.977544+0000","last_active":"2024-01-16T14:56:46.954100+0000","last_peered":"2024-01-16T14:56:46.954100+0000","last_clean":"2024-01-16T14:56:46.954100+0000","last_became_active":"2024-01-16T14:56:45.976912+0000","last_became_peered":"2024-01-16T14:56:45.976912+0000","last_unstale":"2024-01-16T14:56:46.954100+0000","last_undegraded":"2024-01-16T14:56:46.954100+0000","last_fullsized":"2024-01-16T14:56:46.954100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:04.924598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309263+0000","last_change":"2024-01-16T14:56:45.986451+0000","last_active":"2024-01-16T14:56:47.309263+0000","last_peered":"2024-01-16T14:56:47.309263+0000","last_clean":"2024-01-16T14:56:47.309263+0000","last_became_active":"2024-01-16T14:56:45.985694+0000","last_became_peered":"2024-01-16T14:56:45.985694+0000","last_unstale":"2024-01-16T14:56:47.309263+0000","last_undegraded":"2024-01-16T14:56:47.309263+0000","last_fullsized":"2024-01-16T14:56:47.309263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311052+0000","last_change":"2024-01-16T14:56:45.980372+0000","last_active":"2024-01-16T14:56:47.311052+0000","last_peered":"2024-01-16T14:56:47.311052+0000","last_clean":"2024-01-16T14:56:47.311052+0000","last_became_active":"2024-01-16T14:56:45.980114+0000","last_became_peered":"2024-01-16T14:56:45.980114+0000","last_unstale":"2024-01-16T14:56:47.311052+0000","last_undegraded":"2024-01-16T14:56:47.311052+0000","last_fullsized":"2024-01-16T14:56:47.311052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:10.565274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950877+0000","last_change":"2024-01-16T14:56:45.987106+0000","last_active":"2024-01-16T14:56:46.950877+0000","last_peered":"2024-01-16T14:56:46.950877+0000","last_clean":"2024-01-16T14:56:46.950877+0000","last_became_active":"2024-01-16T14:56:45.986126+0000","last_became_peered":"2024-01-16T14:56:45.986126+0000","last_unstale":"2024-01-16T14:56:46.950877+0000","last_undegraded":"2024-01-16T14:56:46.950877+0000","last_fullsized":"2024-01-16T14:56:46.950877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310669+0000","last_change":"2024-01-16T14:56:45.984242+0000","last_active":"2024-01-16T14:56:47.310669+0000","last_peered":"2024-01-16T14:56:47.310669+0000","last_clean":"2024-01-16T14:56:47.310669+0000","last_became_active":"2024-01-16T14:56:45.983936+0000","last_became_peered":"2024-01-16T14:56:45.983936+0000","last_unstale":"2024-01-16T14:56:47.310669+0000","last_undegraded":"2024-01-16T14:56:47.310669+0000","last_fullsized":"2024-01-16T14:56:47.310669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312880+0000","last_change":"2024-01-16T14:56:45.983089+0000","last_active":"2024-01-16T14:56:47.312880+0000","last_peered":"2024-01-16T14:56:47.312880+0000","last_clean":"2024-01-16T14:56:47.312880+0000","last_became_active":"2024-01-16T14:56:45.982784+0000","last_became_peered":"2024-01-16T14:56:45.982784+0000","last_unstale":"2024-01-16T14:56:47.312880+0000","last_undegraded":"2024-01-16T14:56:47.312880+0000","last_fullsized":"2024-01-16T14:56:47.312880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312245+0000","last_change":"2024-01-16T14:56:45.977860+0000","last_active":"2024-01-16T14:56:47.312245+0000","last_peered":"2024-01-16T14:56:47.312245+0000","last_clean":"2024-01-16T14:56:47.312245+0000","last_became_active":"2024-01-16T14:56:45.977563+0000","last_became_peered":"2024-01-16T14:56:45.977563+0000","last_unstale":"2024-01-16T14:56:47.312245+0000","last_undegraded":"2024-01-16T14:56:47.312245+0000","last_fullsized":"2024-01-16T14:56:47.312245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955821+0000","last_change":"2024-01-16T14:56:45.977287+0000","last_active":"2024-01-16T14:56:46.955821+0000","last_peered":"2024-01-16T14:56:46.955821+0000","last_clean":"2024-01-16T14:56:46.955821+0000","last_became_active":"2024-01-16T14:56:45.976639+0000","last_became_peered":"2024-01-16T14:56:45.976639+0000","last_unstale":"2024-01-16T14:56:46.955821+0000","last_undegraded":"2024-01-16T14:56:46.955821+0000","last_fullsized":"2024-01-16T14:56:46.955821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312028+0000","last_change":"2024-01-16T14:56:45.984360+0000","last_active":"2024-01-16T14:56:47.312028+0000","last_peered":"2024-01-16T14:56:47.312028+0000","last_clean":"2024-01-16T14:56:47.312028+0000","last_became_active":"2024-01-16T14:56:45.984102+0000","last_became_peered":"2024-01-16T14:56:45.984102+0000","last_unstale":"2024-01-16T14:56:47.312028+0000","last_undegraded":"2024-01-16T14:56:47.312028+0000","last_fullsized":"2024-01-16T14:56:47.312028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311119+0000","last_change":"2024-01-16T14:56:45.973908+0000","last_active":"2024-01-16T14:56:47.311119+0000","last_peered":"2024-01-16T14:56:47.311119+0000","last_clean":"2024-01-16T14:56:47.311119+0000","last_became_active":"2024-01-16T14:56:45.970417+0000","last_became_peered":"2024-01-16T14:56:45.970417+0000","last_unstale":"2024-01-16T14:56:47.311119+0000","last_undegraded":"2024-01-16T14:56:47.311119+0000","last_fullsized":"2024-01-16T14:56:47.311119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311312+0000","last_change":"2024-01-16T14:56:45.984704+0000","last_active":"2024-01-16T14:56:47.311312+0000","last_peered":"2024-01-16T14:56:47.311312+0000","last_clean":"2024-01-16T14:56:47.311312+0000","last_became_active":"2024-01-16T14:56:45.983436+0000","last_became_peered":"2024-01-16T14:56:45.983436+0000","last_unstale":"2024-01-16T14:56:47.311312+0000","last_undegraded":"2024-01-16T14:56:47.311312+0000","last_fullsized":"2024-01-16T14:56:47.311312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308745+0000","last_change":"2024-01-16T14:56:45.989345+0000","last_active":"2024-01-16T14:56:47.308745+0000","last_peered":"2024-01-16T14:56:47.308745+0000","last_clean":"2024-01-16T14:56:47.308745+0000","last_became_active":"2024-01-16T14:56:45.989090+0000","last_became_peered":"2024-01-16T14:56:45.989090+0000","last_unstale":"2024-01-16T14:56:47.308745+0000","last_undegraded":"2024-01-16T14:56:47.308745+0000","last_fullsized":"2024-01-16T14:56:47.308745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310832+0000","last_change":"2024-01-16T14:56:45.988656+0000","last_active":"2024-01-16T14:56:47.310832+0000","last_peered":"2024-01-16T14:56:47.310832+0000","last_clean":"2024-01-16T14:56:47.310832+0000","last_became_active":"2024-01-16T14:56:45.988350+0000","last_became_peered":"2024-01-16T14:56:45.988350+0000","last_unstale":"2024-01-16T14:56:47.310832+0000","last_undegraded":"2024-01-16T14:56:47.310832+0000","last_fullsized":"2024-01-16T14:56:47.310832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308691+0000","last_change":"2024-01-16T14:56:45.984838+0000","last_active":"2024-01-16T14:56:47.308691+0000","last_peered":"2024-01-16T14:56:47.308691+0000","last_clean":"2024-01-16T14:56:47.308691+0000","last_became_active":"2024-01-16T14:56:45.984542+0000","last_became_peered":"2024-01-16T14:56:45.984542+0000","last_unstale":"2024-01-16T14:56:47.308691+0000","last_undegraded":"2024-01-16T14:56:47.308691+0000","last_fullsized":"2024-01-16T14:56:47.308691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312755+0000","last_change":"2024-01-16T14:56:45.989694+0000","last_active":"2024-01-16T14:56:47.312755+0000","last_peered":"2024-01-16T14:56:47.312755+0000","last_clean":"2024-01-16T14:56:47.312755+0000","last_became_active":"2024-01-16T14:56:45.989384+0000","last_became_peered":"2024-01-16T14:56:45.989384+0000","last_unstale":"2024-01-16T14:56:47.312755+0000","last_undegraded":"2024-01-16T14:56:47.312755+0000","last_fullsized":"2024-01-16T14:56:47.312755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951404+0000","last_change":"2024-01-16T14:56:45.988721+0000","last_active":"2024-01-16T14:56:46.951404+0000","last_peered":"2024-01-16T14:56:46.951404+0000","last_clean":"2024-01-16T14:56:46.951404+0000","last_became_active":"2024-01-16T14:56:45.988428+0000","last_became_peered":"2024-01-16T14:56:45.988428+0000","last_unstale":"2024-01-16T14:56:46.951404+0000","last_undegraded":"2024-01-16T14:56:46.951404+0000","last_fullsized":"2024-01-16T14:56:46.951404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309579+0000","last_change":"2024-01-16T14:56:45.986196+0000","last_active":"2024-01-16T14:56:47.309579+0000","last_peered":"2024-01-16T14:56:47.309579+0000","last_clean":"2024-01-16T14:56:47.309579+0000","last_became_active":"2024-01-16T14:56:45.985927+0000","last_became_peered":"2024-01-16T14:56:45.985927+0000","last_unstale":"2024-01-16T14:56:47.309579+0000","last_undegraded":"2024-01-16T14:56:47.309579+0000","last_fullsized":"2024-01-16T14:56:47.309579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311237+0000","last_change":"2024-01-16T14:56:45.985321+0000","last_active":"2024-01-16T14:56:47.311237+0000","last_peered":"2024-01-16T14:56:47.311237+0000","last_clean":"2024-01-16T14:56:47.311237+0000","last_became_active":"2024-01-16T14:56:45.985015+0000","last_became_peered":"2024-01-16T14:56:45.985015+0000","last_unstale":"2024-01-16T14:56:47.311237+0000","last_undegraded":"2024-01-16T14:56:47.311237+0000","last_fullsized":"2024-01-16T14:56:47.311237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311646+0000","last_change":"2024-01-16T14:56:45.986171+0000","last_active":"2024-01-16T14:56:47.311646+0000","last_peered":"2024-01-16T14:56:47.311646+0000","last_clean":"2024-01-16T14:56:47.311646+0000","last_became_active":"2024-01-16T14:56:45.985890+0000","last_became_peered":"2024-01-16T14:56:45.985890+0000","last_unstale":"2024-01-16T14:56:47.311646+0000","last_undegraded":"2024-01-16T14:56:47.311646+0000","last_fullsized":"2024-01-16T14:56:47.311646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312240+0000","last_change":"2024-01-16T14:56:45.979713+0000","last_active":"2024-01-16T14:56:47.312240+0000","last_peered":"2024-01-16T14:56:47.312240+0000","last_clean":"2024-01-16T14:56:47.312240+0000","last_became_active":"2024-01-16T14:56:45.979379+0000","last_became_peered":"2024-01-16T14:56:45.979379+0000","last_unstale":"2024-01-16T14:56:47.312240+0000","last_undegraded":"2024-01-16T14:56:47.312240+0000","last_fullsized":"2024-01-16T14:56:47.312240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311228+0000","last_change":"2024-01-16T14:56:45.988420+0000","last_active":"2024-01-16T14:56:47.311228+0000","last_peered":"2024-01-16T14:56:47.311228+0000","last_clean":"2024-01-16T14:56:47.311228+0000","last_became_active":"2024-01-16T14:56:45.987162+0000","last_became_peered":"2024-01-16T14:56:45.987162+0000","last_unstale":"2024-01-16T14:56:47.311228+0000","last_undegraded":"2024-01-16T14:56:47.311228+0000","last_fullsized":"2024-01-16T14:56:47.311228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952781+0000","last_change":"2024-01-16T14:56:45.977184+0000","last_active":"2024-01-16T14:56:46.952781+0000","last_peered":"2024-01-16T14:56:46.952781+0000","last_clean":"2024-01-16T14:56:46.952781+0000","last_became_active":"2024-01-16T14:56:45.976825+0000","last_became_peered":"2024-01-16T14:56:45.976825+0000","last_unstale":"2024-01-16T14:56:46.952781+0000","last_undegraded":"2024-01-16T14:56:46.952781+0000","last_fullsized":"2024-01-16T14:56:46.952781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:33.458747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012001+0000","last_change":"2024-01-16T14:56:45.008639+0000","last_active":"2024-01-16T14:56:46.012001+0000","last_peered":"2024-01-16T14:56:46.012001+0000","last_clean":"2024-01-16T14:56:46.012001+0000","last_became_active":"2024-01-16T14:56:45.008234+0000","last_became_peered":"2024-01-16T14:56:45.008234+0000","last_unstale":"2024-01-16T14:56:46.012001+0000","last_undegraded":"2024-01-16T14:56:46.012001+0000","last_fullsized":"2024-01-16T14:56:46.012001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311455+0000","last_change":"2024-01-16T14:56:45.976366+0000","last_active":"2024-01-16T14:56:47.311455+0000","last_peered":"2024-01-16T14:56:47.311455+0000","last_clean":"2024-01-16T14:56:47.311455+0000","last_became_active":"2024-01-16T14:56:45.975839+0000","last_became_peered":"2024-01-16T14:56:45.975839+0000","last_unstale":"2024-01-16T14:56:47.311455+0000","last_undegraded":"2024-01-16T14:56:47.311455+0000","last_fullsized":"2024-01-16T14:56:47.311455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010576+0000","last_change":"2024-01-16T14:56:45.003662+0000","last_active":"2024-01-16T14:56:46.010576+0000","last_peered":"2024-01-16T14:56:46.010576+0000","last_clean":"2024-01-16T14:56:46.010576+0000","last_became_active":"2024-01-16T14:56:45.003318+0000","last_became_peered":"2024-01-16T14:56:45.003318+0000","last_unstale":"2024-01-16T14:56:46.010576+0000","last_undegraded":"2024-01-16T14:56:46.010576+0000","last_fullsized":"2024-01-16T14:56:46.010576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.982735+0000","last_change":"2024-01-16T14:56:45.013121+0000","last_active":"2024-01-16T14:56:45.982735+0000","last_peered":"2024-01-16T14:56:45.982735+0000","last_clean":"2024-01-16T14:56:45.982735+0000","last_became_active":"2024-01-16T14:56:45.012797+0000","last_became_peered":"2024-01-16T14:56:45.012797+0000","last_unstale":"2024-01-16T14:56:45.982735+0000","last_undegraded":"2024-01-16T14:56:45.982735+0000","last_fullsized":"2024-01-16T14:56:45.982735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308918+0000","last_change":"2024-01-16T14:56:45.986213+0000","last_active":"2024-01-16T14:56:47.308918+0000","last_peered":"2024-01-16T14:56:47.308918+0000","last_clean":"2024-01-16T14:56:47.308918+0000","last_became_active":"2024-01-16T14:56:45.985824+0000","last_became_peered":"2024-01-16T14:56:45.985824+0000","last_unstale":"2024-01-16T14:56:47.308918+0000","last_undegraded":"2024-01-16T14:56:47.308918+0000","last_fullsized":"2024-01-16T14:56:47.308918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.974634+0000","last_change":"2024-01-16T14:56:45.014687+0000","last_active":"2024-01-16T14:56:45.974634+0000","last_peered":"2024-01-16T14:56:45.974634+0000","last_clean":"2024-01-16T14:56:45.974634+0000","last_became_active":"2024-01-16T14:56:45.014356+0000","last_became_peered":"2024-01-16T14:56:45.014356+0000","last_unstale":"2024-01-16T14:56:45.974634+0000","last_undegraded":"2024-01-16T14:56:45.974634+0000","last_fullsized":"2024-01-16T14:56:45.974634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:34.974850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950693+0000","last_change":"2024-01-16T14:56:45.984146+0000","last_active":"2024-01-16T14:56:46.950693+0000","last_peered":"2024-01-16T14:56:46.950693+0000","last_clean":"2024-01-16T14:56:46.950693+0000","last_became_active":"2024-01-16T14:56:45.983841+0000","last_became_peered":"2024-01-16T14:56:45.983841+0000","last_unstale":"2024-01-16T14:56:46.950693+0000","last_undegraded":"2024-01-16T14:56:46.950693+0000","last_fullsized":"2024-01-16T14:56:46.950693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.971677+0000","last_change":"2024-01-16T14:56:45.011843+0000","last_active":"2024-01-16T14:56:45.971677+0000","last_peered":"2024-01-16T14:56:45.971677+0000","last_clean":"2024-01-16T14:56:45.971677+0000","last_became_active":"2024-01-16T14:56:45.011331+0000","last_became_peered":"2024-01-16T14:56:45.011331+0000","last_unstale":"2024-01-16T14:56:45.971677+0000","last_undegraded":"2024-01-16T14:56:45.971677+0000","last_fullsized":"2024-01-16T14:56:45.971677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313267+0000","last_change":"2024-01-16T14:56:45.986253+0000","last_active":"2024-01-16T14:56:47.313267+0000","last_peered":"2024-01-16T14:56:47.313267+0000","last_clean":"2024-01-16T14:56:47.313267+0000","last_became_active":"2024-01-16T14:56:45.985935+0000","last_became_peered":"2024-01-16T14:56:45.985935+0000","last_unstale":"2024-01-16T14:56:47.313267+0000","last_undegraded":"2024-01-16T14:56:47.313267+0000","last_fullsized":"2024-01-16T14:56:47.313267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.973280+0000","last_change":"2024-01-16T14:56:45.016005+0000","last_active":"2024-01-16T14:56:45.973280+0000","last_peered":"2024-01-16T14:56:45.973280+0000","last_clean":"2024-01-16T14:56:45.973280+0000","last_became_active":"2024-01-16T14:56:45.015680+0000","last_became_peered":"2024-01-16T14:56:45.015680+0000","last_unstale":"2024-01-16T14:56:45.973280+0000","last_undegraded":"2024-01-16T14:56:45.973280+0000","last_fullsized":"2024-01-16T14:56:45.973280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:19.042486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311306+0000","last_change":"2024-01-16T14:56:45.980174+0000","last_active":"2024-01-16T14:56:47.311306+0000","last_peered":"2024-01-16T14:56:47.311306+0000","last_clean":"2024-01-16T14:56:47.311306+0000","last_became_active":"2024-01-16T14:56:45.979743+0000","last_became_peered":"2024-01-16T14:56:45.979743+0000","last_unstale":"2024-01-16T14:56:47.311306+0000","last_undegraded":"2024-01-16T14:56:47.311306+0000","last_fullsized":"2024-01-16T14:56:47.311306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013898+0000","last_change":"2024-01-16T14:56:45.017298+0000","last_active":"2024-01-16T14:56:46.013898+0000","last_peered":"2024-01-16T14:56:46.013898+0000","last_clean":"2024-01-16T14:56:46.013898+0000","last_became_active":"2024-01-16T14:56:45.016921+0000","last_became_peered":"2024-01-16T14:56:45.016921+0000","last_unstale":"2024-01-16T14:56:46.013898+0000","last_undegraded":"2024-01-16T14:56:46.013898+0000","last_fullsized":"2024-01-16T14:56:46.013898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954466+0000","last_change":"2024-01-16T14:56:45.967606+0000","last_active":"2024-01-16T14:56:46.954466+0000","last_peered":"2024-01-16T14:56:46.954466+0000","last_clean":"2024-01-16T14:56:46.954466+0000","last_became_active":"2024-01-16T14:56:45.967036+0000","last_became_peered":"2024-01-16T14:56:45.967036+0000","last_unstale":"2024-01-16T14:56:46.954466+0000","last_undegraded":"2024-01-16T14:56:46.954466+0000","last_fullsized":"2024-01-16T14:56:46.954466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015870+0000","last_change":"2024-01-16T14:56:45.005985+0000","last_active":"2024-01-16T14:56:46.015870+0000","last_peered":"2024-01-16T14:56:46.015870+0000","last_clean":"2024-01-16T14:56:46.015870+0000","last_became_active":"2024-01-16T14:56:45.005530+0000","last_became_peered":"2024-01-16T14:56:45.005530+0000","last_unstale":"2024-01-16T14:56:46.015870+0000","last_undegraded":"2024-01-16T14:56:46.015870+0000","last_fullsized":"2024-01-16T14:56:46.015870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:17.631008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955160+0000","last_change":"2024-01-16T14:56:45.991157+0000","last_active":"2024-01-16T14:56:46.955160+0000","last_peered":"2024-01-16T14:56:46.955160+0000","last_clean":"2024-01-16T14:56:46.955160+0000","last_became_active":"2024-01-16T14:56:45.990848+0000","last_became_peered":"2024-01-16T14:56:45.990848+0000","last_unstale":"2024-01-16T14:56:46.955160+0000","last_undegraded":"2024-01-16T14:56:46.955160+0000","last_fullsized":"2024-01-16T14:56:46.955160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016578+0000","last_change":"2024-01-16T14:56:45.000374+0000","last_active":"2024-01-16T14:56:46.016578+0000","last_peered":"2024-01-16T14:56:46.016578+0000","last_clean":"2024-01-16T14:56:46.016578+0000","last_became_active":"2024-01-16T14:56:45.000028+0000","last_became_peered":"2024-01-16T14:56:45.000028+0000","last_unstale":"2024-01-16T14:56:46.016578+0000","last_undegraded":"2024-01-16T14:56:46.016578+0000","last_fullsized":"2024-01-16T14:56:46.016578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311398+0000","last_change":"2024-01-16T14:56:45.974961+0000","last_active":"2024-01-16T14:56:47.311398+0000","last_peered":"2024-01-16T14:56:47.311398+0000","last_clean":"2024-01-16T14:56:47.311398+0000","last_became_active":"2024-01-16T14:56:45.974581+0000","last_became_peered":"2024-01-16T14:56:45.974581+0000","last_unstale":"2024-01-16T14:56:47.311398+0000","last_undegraded":"2024-01-16T14:56:47.311398+0000","last_fullsized":"2024-01-16T14:56:47.311398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.972980+0000","last_change":"2024-01-16T14:56:45.017987+0000","last_active":"2024-01-16T14:56:45.972980+0000","last_peered":"2024-01-16T14:56:45.972980+0000","last_clean":"2024-01-16T14:56:45.972980+0000","last_became_active":"2024-01-16T14:56:45.017656+0000","last_became_peered":"2024-01-16T14:56:45.017656+0000","last_unstale":"2024-01-16T14:56:45.972980+0000","last_undegraded":"2024-01-16T14:56:45.972980+0000","last_fullsized":"2024-01-16T14:56:45.972980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954932+0000","last_change":"2024-01-16T14:56:45.976869+0000","last_active":"2024-01-16T14:56:46.954932+0000","last_peered":"2024-01-16T14:56:46.954932+0000","last_clean":"2024-01-16T14:56:46.954932+0000","last_became_active":"2024-01-16T14:56:45.976500+0000","last_became_peered":"2024-01-16T14:56:45.976500+0000","last_unstale":"2024-01-16T14:56:46.954932+0000","last_undegraded":"2024-01-16T14:56:46.954932+0000","last_fullsized":"2024-01-16T14:56:46.954932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.966318+0000","last_change":"2024-01-16T14:56:45.012069+0000","last_active":"2024-01-16T14:56:45.966318+0000","last_peered":"2024-01-16T14:56:45.966318+0000","last_clean":"2024-01-16T14:56:45.966318+0000","last_became_active":"2024-01-16T14:56:45.011711+0000","last_became_peered":"2024-01-16T14:56:45.011711+0000","last_unstale":"2024-01-16T14:56:45.966318+0000","last_undegraded":"2024-01-16T14:56:45.966318+0000","last_fullsized":"2024-01-16T14:56:45.966318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311361+0000","last_change":"2024-01-16T14:56:45.988281+0000","last_active":"2024-01-16T14:56:47.311361+0000","last_peered":"2024-01-16T14:56:47.311361+0000","last_clean":"2024-01-16T14:56:47.311361+0000","last_became_active":"2024-01-16T14:56:45.986521+0000","last_became_peered":"2024-01-16T14:56:45.986521+0000","last_unstale":"2024-01-16T14:56:47.311361+0000","last_undegraded":"2024-01-16T14:56:47.311361+0000","last_fullsized":"2024-01-16T14:56:47.311361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015745+0000","last_change":"2024-01-16T14:56:44.993188+0000","last_active":"2024-01-16T14:56:46.015745+0000","last_peered":"2024-01-16T14:56:46.015745+0000","last_clean":"2024-01-16T14:56:46.015745+0000","last_became_active":"2024-01-16T14:56:44.992904+0000","last_became_peered":"2024-01-16T14:56:44.992904+0000","last_unstale":"2024-01-16T14:56:46.015745+0000","last_undegraded":"2024-01-16T14:56:46.015745+0000","last_fullsized":"2024-01-16T14:56:46.015745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.524009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954660+0000","last_change":"2024-01-16T14:56:45.981562+0000","last_active":"2024-01-16T14:56:46.954660+0000","last_peered":"2024-01-16T14:56:46.954660+0000","last_clean":"2024-01-16T14:56:46.954660+0000","last_became_active":"2024-01-16T14:56:45.981272+0000","last_became_peered":"2024-01-16T14:56:45.981272+0000","last_unstale":"2024-01-16T14:56:46.954660+0000","last_undegraded":"2024-01-16T14:56:46.954660+0000","last_fullsized":"2024-01-16T14:56:46.954660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013126+0000","last_change":"2024-01-16T14:56:45.005288+0000","last_active":"2024-01-16T14:56:46.013126+0000","last_peered":"2024-01-16T14:56:46.013126+0000","last_clean":"2024-01-16T14:56:46.013126+0000","last_became_active":"2024-01-16T14:56:45.004988+0000","last_became_peered":"2024-01-16T14:56:45.004988+0000","last_unstale":"2024-01-16T14:56:46.013126+0000","last_undegraded":"2024-01-16T14:56:46.013126+0000","last_fullsized":"2024-01-16T14:56:46.013126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955886+0000","last_change":"2024-01-16T14:56:45.977479+0000","last_active":"2024-01-16T14:56:46.955886+0000","last_peered":"2024-01-16T14:56:46.955886+0000","last_clean":"2024-01-16T14:56:46.955886+0000","last_became_active":"2024-01-16T14:56:45.976447+0000","last_became_peered":"2024-01-16T14:56:45.976447+0000","last_unstale":"2024-01-16T14:56:46.955886+0000","last_undegraded":"2024-01-16T14:56:46.955886+0000","last_fullsized":"2024-01-16T14:56:46.955886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:09:28.671518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012387+0000","last_change":"2024-01-16T14:56:44.991027+0000","last_active":"2024-01-16T14:56:46.012387+0000","last_peered":"2024-01-16T14:56:46.012387+0000","last_clean":"2024-01-16T14:56:46.012387+0000","last_became_active":"2024-01-16T14:56:44.990777+0000","last_became_peered":"2024-01-16T14:56:44.990777+0000","last_unstale":"2024-01-16T14:56:46.012387+0000","last_undegraded":"2024-01-16T14:56:46.012387+0000","last_fullsized":"2024-01-16T14:56:46.012387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956303+0000","last_change":"2024-01-16T14:56:45.980059+0000","last_active":"2024-01-16T14:56:46.956303+0000","last_peered":"2024-01-16T14:56:46.956303+0000","last_clean":"2024-01-16T14:56:46.956303+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956303+0000","last_undegraded":"2024-01-16T14:56:46.956303+0000","last_fullsized":"2024-01-16T14:56:46.956303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:40.703250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010244+0000","last_change":"2024-01-16T14:56:45.001611+0000","last_active":"2024-01-16T14:56:46.010244+0000","last_peered":"2024-01-16T14:56:46.010244+0000","last_clean":"2024-01-16T14:56:46.010244+0000","last_became_active":"2024-01-16T14:56:45.001338+0000","last_became_peered":"2024-01-16T14:56:45.001338+0000","last_unstale":"2024-01-16T14:56:46.010244+0000","last_undegraded":"2024-01-16T14:56:46.010244+0000","last_fullsized":"2024-01-16T14:56:46.010244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312030+0000","last_change":"2024-01-16T14:56:45.986001+0000","last_active":"2024-01-16T14:56:47.312030+0000","last_peered":"2024-01-16T14:56:47.312030+0000","last_clean":"2024-01-16T14:56:47.312030+0000","last_became_active":"2024-01-16T14:56:45.984074+0000","last_became_peered":"2024-01-16T14:56:45.984074+0000","last_unstale":"2024-01-16T14:56:47.312030+0000","last_undegraded":"2024-01-16T14:56:47.312030+0000","last_fullsized":"2024-01-16T14:56:47.312030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.960950+0000","last_change":"2024-01-16T14:56:45.009683+0000","last_active":"2024-01-16T14:56:45.960950+0000","last_peered":"2024-01-16T14:56:45.960950+0000","last_clean":"2024-01-16T14:56:45.960950+0000","last_became_active":"2024-01-16T14:56:45.009325+0000","last_became_peered":"2024-01-16T14:56:45.009325+0000","last_unstale":"2024-01-16T14:56:45.960950+0000","last_undegraded":"2024-01-16T14:56:45.960950+0000","last_fullsized":"2024-01-16T14:56:45.960950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:27.159214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955489+0000","last_change":"2024-01-16T14:56:45.989743+0000","last_active":"2024-01-16T14:56:46.955489+0000","last_peered":"2024-01-16T14:56:46.955489+0000","last_clean":"2024-01-16T14:56:46.955489+0000","last_became_active":"2024-01-16T14:56:45.989084+0000","last_became_peered":"2024-01-16T14:56:45.989084+0000","last_unstale":"2024-01-16T14:56:46.955489+0000","last_undegraded":"2024-01-16T14:56:46.955489+0000","last_fullsized":"2024-01-16T14:56:46.955489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.955711+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:56:45.955711+0000","last_peered":"2024-01-16T14:56:45.955711+0000","last_clean":"2024-01-16T14:56:45.955711+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:56:45.955711+0000","last_undegraded":"2024-01-16T14:56:45.955711+0000","last_fullsized":"2024-01-16T14:56:45.955711+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963629+0000","last_change":"2024-01-16T14:56:45.009432+0000","last_active":"2024-01-16T14:56:45.963629+0000","last_peered":"2024-01-16T14:56:45.963629+0000","last_clean":"2024-01-16T14:56:45.963629+0000","last_became_active":"2024-01-16T14:56:45.008692+0000","last_became_peered":"2024-01-16T14:56:45.008692+0000","last_unstale":"2024-01-16T14:56:45.963629+0000","last_undegraded":"2024-01-16T14:56:45.963629+0000","last_fullsized":"2024-01-16T14:56:45.963629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308881+0000","last_change":"2024-01-16T14:56:45.990578+0000","last_active":"2024-01-16T14:56:47.308881+0000","last_peered":"2024-01-16T14:56:47.308881+0000","last_clean":"2024-01-16T14:56:47.308881+0000","last_became_active":"2024-01-16T14:56:45.990211+0000","last_became_peered":"2024-01-16T14:56:45.990211+0000","last_unstale":"2024-01-16T14:56:47.308881+0000","last_undegraded":"2024-01-16T14:56:47.308881+0000","last_fullsized":"2024-01-16T14:56:47.308881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.956468+0000","last_change":"2024-01-16T14:56:45.006632+0000","last_active":"2024-01-16T14:56:45.956468+0000","last_peered":"2024-01-16T14:56:45.956468+0000","last_clean":"2024-01-16T14:56:45.956468+0000","last_became_active":"2024-01-16T14:56:45.006340+0000","last_became_peered":"2024-01-16T14:56:45.006340+0000","last_unstale":"2024-01-16T14:56:45.956468+0000","last_undegraded":"2024-01-16T14:56:45.956468+0000","last_fullsized":"2024-01-16T14:56:45.956468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:41.754460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310892+0000","last_change":"2024-01-16T14:56:45.977532+0000","last_active":"2024-01-16T14:56:47.310892+0000","last_peered":"2024-01-16T14:56:47.310892+0000","last_clean":"2024-01-16T14:56:47.310892+0000","last_became_active":"2024-01-16T14:56:45.975102+0000","last_became_peered":"2024-01-16T14:56:45.975102+0000","last_unstale":"2024-01-16T14:56:47.310892+0000","last_undegraded":"2024-01-16T14:56:47.310892+0000","last_fullsized":"2024-01-16T14:56:47.310892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:18.643652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963071+0000","last_change":"2024-01-16T14:56:44.994705+0000","last_active":"2024-01-16T14:56:45.963071+0000","last_peered":"2024-01-16T14:56:45.963071+0000","last_clean":"2024-01-16T14:56:45.963071+0000","last_became_active":"2024-01-16T14:56:44.994387+0000","last_became_peered":"2024-01-16T14:56:44.994387+0000","last_unstale":"2024-01-16T14:56:45.963071+0000","last_undegraded":"2024-01-16T14:56:45.963071+0000","last_fullsized":"2024-01-16T14:56:45.963071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950691+0000","last_change":"2024-01-16T14:56:45.978166+0000","last_active":"2024-01-16T14:56:46.950691+0000","last_peered":"2024-01-16T14:56:46.950691+0000","last_clean":"2024-01-16T14:56:46.950691+0000","last_became_active":"2024-01-16T14:56:45.974628+0000","last_became_peered":"2024-01-16T14:56:45.974628+0000","last_unstale":"2024-01-16T14:56:46.950691+0000","last_undegraded":"2024-01-16T14:56:46.950691+0000","last_fullsized":"2024-01-16T14:56:46.950691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.980309+0000","last_change":"2024-01-16T14:56:45.017774+0000","last_active":"2024-01-16T14:56:45.980309+0000","last_peered":"2024-01-16T14:56:45.980309+0000","last_clean":"2024-01-16T14:56:45.980309+0000","last_became_active":"2024-01-16T14:56:45.017410+0000","last_became_peered":"2024-01-16T14:56:45.017410+0000","last_unstale":"2024-01-16T14:56:45.980309+0000","last_undegraded":"2024-01-16T14:56:45.980309+0000","last_fullsized":"2024-01-16T14:56:45.980309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:59.852603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954411+0000","last_change":"2024-01-16T14:56:45.970975+0000","last_active":"2024-01-16T14:56:46.954411+0000","last_peered":"2024-01-16T14:56:46.954411+0000","last_clean":"2024-01-16T14:56:46.954411+0000","last_became_active":"2024-01-16T14:56:45.970566+0000","last_became_peered":"2024-01-16T14:56:45.970566+0000","last_unstale":"2024-01-16T14:56:46.954411+0000","last_undegraded":"2024-01-16T14:56:46.954411+0000","last_fullsized":"2024-01-16T14:56:46.954411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.967326+0000","last_change":"2024-01-16T14:56:45.004864+0000","last_active":"2024-01-16T14:56:45.967326+0000","last_peered":"2024-01-16T14:56:45.967326+0000","last_clean":"2024-01-16T14:56:45.967326+0000","last_became_active":"2024-01-16T14:56:45.004057+0000","last_became_peered":"2024-01-16T14:56:45.004057+0000","last_unstale":"2024-01-16T14:56:45.967326+0000","last_undegraded":"2024-01-16T14:56:45.967326+0000","last_fullsized":"2024-01-16T14:56:45.967326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313581+0000","last_change":"2024-01-16T14:56:45.989780+0000","last_active":"2024-01-16T14:56:47.313581+0000","last_peered":"2024-01-16T14:56:47.313581+0000","last_clean":"2024-01-16T14:56:47.313581+0000","last_became_active":"2024-01-16T14:56:45.989519+0000","last_became_peered":"2024-01-16T14:56:45.989519+0000","last_unstale":"2024-01-16T14:56:47.313581+0000","last_undegraded":"2024-01-16T14:56:47.313581+0000","last_fullsized":"2024-01-16T14:56:47.313581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.959086+0000","last_change":"2024-01-16T14:56:45.009420+0000","last_active":"2024-01-16T14:56:45.959086+0000","last_peered":"2024-01-16T14:56:45.959086+0000","last_clean":"2024-01-16T14:56:45.959086+0000","last_became_active":"2024-01-16T14:56:45.009103+0000","last_became_peered":"2024-01-16T14:56:45.009103+0000","last_unstale":"2024-01-16T14:56:45.959086+0000","last_undegraded":"2024-01-16T14:56:45.959086+0000","last_fullsized":"2024-01-16T14:56:45.959086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:49.027396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951463+0000","last_change":"2024-01-16T14:56:45.976716+0000","last_active":"2024-01-16T14:56:46.951463+0000","last_peered":"2024-01-16T14:56:46.951463+0000","last_clean":"2024-01-16T14:56:46.951463+0000","last_became_active":"2024-01-16T14:56:45.976307+0000","last_became_peered":"2024-01-16T14:56:45.976307+0000","last_unstale":"2024-01-16T14:56:46.951463+0000","last_undegraded":"2024-01-16T14:56:46.951463+0000","last_fullsized":"2024-01-16T14:56:46.951463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952095+0000","last_change":"2024-01-16T14:56:45.013169+0000","last_active":"2024-01-16T14:56:46.952095+0000","last_peered":"2024-01-16T14:56:46.952095+0000","last_clean":"2024-01-16T14:56:46.952095+0000","last_became_active":"2024-01-16T14:56:45.012872+0000","last_became_peered":"2024-01-16T14:56:45.012872+0000","last_unstale":"2024-01-16T14:56:46.952095+0000","last_undegraded":"2024-01-16T14:56:46.952095+0000","last_fullsized":"2024-01-16T14:56:46.952095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308843+0000","last_change":"2024-01-16T14:56:45.987343+0000","last_active":"2024-01-16T14:56:47.308843+0000","last_peered":"2024-01-16T14:56:47.308843+0000","last_clean":"2024-01-16T14:56:47.308843+0000","last_became_active":"2024-01-16T14:56:45.987065+0000","last_became_peered":"2024-01-16T14:56:45.987065+0000","last_unstale":"2024-01-16T14:56:47.308843+0000","last_undegraded":"2024-01-16T14:56:47.308843+0000","last_fullsized":"2024-01-16T14:56:47.308843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014224+0000","last_change":"2024-01-16T14:56:45.004437+0000","last_active":"2024-01-16T14:56:46.014224+0000","last_peered":"2024-01-16T14:56:46.014224+0000","last_clean":"2024-01-16T14:56:46.014224+0000","last_became_active":"2024-01-16T14:56:45.004165+0000","last_became_peered":"2024-01-16T14:56:45.004165+0000","last_unstale":"2024-01-16T14:56:46.014224+0000","last_undegraded":"2024-01-16T14:56:46.014224+0000","last_fullsized":"2024-01-16T14:56:46.014224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311396+0000","last_change":"2024-01-16T14:56:45.974262+0000","last_active":"2024-01-16T14:56:47.311396+0000","last_peered":"2024-01-16T14:56:47.311396+0000","last_clean":"2024-01-16T14:56:47.311396+0000","last_became_active":"2024-01-16T14:56:45.970037+0000","last_became_peered":"2024-01-16T14:56:45.970037+0000","last_unstale":"2024-01-16T14:56:47.311396+0000","last_undegraded":"2024-01-16T14:56:47.311396+0000","last_fullsized":"2024-01-16T14:56:47.311396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.952176+0000","last_change":"2024-01-16T14:56:45.009771+0000","last_active":"2024-01-16T14:56:45.952176+0000","last_peered":"2024-01-16T14:56:45.952176+0000","last_clean":"2024-01-16T14:56:45.952176+0000","last_became_active":"2024-01-16T14:56:45.009505+0000","last_became_peered":"2024-01-16T14:56:45.009505+0000","last_unstale":"2024-01-16T14:56:45.952176+0000","last_undegraded":"2024-01-16T14:56:45.952176+0000","last_fullsized":"2024-01-16T14:56:45.952176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952907+0000","last_change":"2024-01-16T14:56:45.970586+0000","last_active":"2024-01-16T14:56:46.952907+0000","last_peered":"2024-01-16T14:56:46.952907+0000","last_clean":"2024-01-16T14:56:46.952907+0000","last_became_active":"2024-01-16T14:56:45.970146+0000","last_became_peered":"2024-01-16T14:56:45.970146+0000","last_unstale":"2024-01-16T14:56:46.952907+0000","last_undegraded":"2024-01-16T14:56:46.952907+0000","last_fullsized":"2024-01-16T14:56:46.952907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:23.365045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.962308+0000","last_change":"2024-01-16T14:56:45.003139+0000","last_active":"2024-01-16T14:56:45.962308+0000","last_peered":"2024-01-16T14:56:45.962308+0000","last_clean":"2024-01-16T14:56:45.962308+0000","last_became_active":"2024-01-16T14:56:45.002787+0000","last_became_peered":"2024-01-16T14:56:45.002787+0000","last_unstale":"2024-01-16T14:56:45.962308+0000","last_undegraded":"2024-01-16T14:56:45.962308+0000","last_fullsized":"2024-01-16T14:56:45.962308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309294+0000","last_change":"2024-01-16T14:56:45.981803+0000","last_active":"2024-01-16T14:56:47.309294+0000","last_peered":"2024-01-16T14:56:47.309294+0000","last_clean":"2024-01-16T14:56:47.309294+0000","last_became_active":"2024-01-16T14:56:45.981480+0000","last_became_peered":"2024-01-16T14:56:45.981480+0000","last_unstale":"2024-01-16T14:56:47.309294+0000","last_undegraded":"2024-01-16T14:56:47.309294+0000","last_fullsized":"2024-01-16T14:56:47.309294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.968506+0000","last_change":"2024-01-16T14:56:45.006149+0000","last_active":"2024-01-16T14:56:45.968506+0000","last_peered":"2024-01-16T14:56:45.968506+0000","last_clean":"2024-01-16T14:56:45.968506+0000","last_became_active":"2024-01-16T14:56:45.005789+0000","last_became_peered":"2024-01-16T14:56:45.005789+0000","last_unstale":"2024-01-16T14:56:45.968506+0000","last_undegraded":"2024-01-16T14:56:45.968506+0000","last_fullsized":"2024-01-16T14:56:45.968506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309478+0000","last_change":"2024-01-16T14:56:45.985569+0000","last_active":"2024-01-16T14:56:47.309478+0000","last_peered":"2024-01-16T14:56:47.309478+0000","last_clean":"2024-01-16T14:56:47.309478+0000","last_became_active":"2024-01-16T14:56:45.983774+0000","last_became_peered":"2024-01-16T14:56:45.983774+0000","last_unstale":"2024-01-16T14:56:47.309478+0000","last_undegraded":"2024-01-16T14:56:47.309478+0000","last_fullsized":"2024-01-16T14:56:47.309478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013631+0000","last_change":"2024-01-16T14:56:45.010441+0000","last_active":"2024-01-16T14:56:46.013631+0000","last_peered":"2024-01-16T14:56:46.013631+0000","last_clean":"2024-01-16T14:56:46.013631+0000","last_became_active":"2024-01-16T14:56:45.007667+0000","last_became_peered":"2024-01-16T14:56:45.007667+0000","last_unstale":"2024-01-16T14:56:46.013631+0000","last_undegraded":"2024-01-16T14:56:46.013631+0000","last_fullsized":"2024-01-16T14:56:46.013631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:38.016001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311624+0000","last_change":"2024-01-16T14:56:45.984334+0000","last_active":"2024-01-16T14:56:47.311624+0000","last_peered":"2024-01-16T14:56:47.311624+0000","last_clean":"2024-01-16T14:56:47.311624+0000","last_became_active":"2024-01-16T14:56:45.984013+0000","last_became_peered":"2024-01-16T14:56:45.984013+0000","last_unstale":"2024-01-16T14:56:47.311624+0000","last_undegraded":"2024-01-16T14:56:47.311624+0000","last_fullsized":"2024-01-16T14:56:47.311624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313406+0000","last_change":"2024-01-16T14:56:45.985148+0000","last_active":"2024-01-16T14:56:47.313406+0000","last_peered":"2024-01-16T14:56:47.313406+0000","last_clean":"2024-01-16T14:56:47.313406+0000","last_became_active":"2024-01-16T14:56:45.983354+0000","last_became_peered":"2024-01-16T14:56:45.983354+0000","last_unstale":"2024-01-16T14:56:47.313406+0000","last_undegraded":"2024-01-16T14:56:47.313406+0000","last_fullsized":"2024-01-16T14:56:47.313406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011363+0000","last_change":"2024-01-16T14:56:45.017135+0000","last_active":"2024-01-16T14:56:46.011363+0000","last_peered":"2024-01-16T14:56:46.011363+0000","last_clean":"2024-01-16T14:56:46.011363+0000","last_became_active":"2024-01-16T14:56:45.016686+0000","last_became_peered":"2024-01-16T14:56:45.016686+0000","last_unstale":"2024-01-16T14:56:46.011363+0000","last_undegraded":"2024-01-16T14:56:46.011363+0000","last_fullsized":"2024-01-16T14:56:46.011363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311883+0000","last_change":"2024-01-16T14:56:45.983050+0000","last_active":"2024-01-16T14:56:47.311883+0000","last_peered":"2024-01-16T14:56:47.311883+0000","last_clean":"2024-01-16T14:56:47.311883+0000","last_became_active":"2024-01-16T14:56:45.982678+0000","last_became_peered":"2024-01-16T14:56:45.982678+0000","last_unstale":"2024-01-16T14:56:47.311883+0000","last_undegraded":"2024-01-16T14:56:47.311883+0000","last_fullsized":"2024-01-16T14:56:47.311883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011339+0000","last_change":"2024-01-16T14:56:45.014696+0000","last_active":"2024-01-16T14:56:46.011339+0000","last_peered":"2024-01-16T14:56:46.011339+0000","last_clean":"2024-01-16T14:56:46.011339+0000","last_became_active":"2024-01-16T14:56:45.014410+0000","last_became_peered":"2024-01-16T14:56:45.014410+0000","last_unstale":"2024-01-16T14:56:46.011339+0000","last_undegraded":"2024-01-16T14:56:46.011339+0000","last_fullsized":"2024-01-16T14:56:46.011339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311230+0000","last_change":"2024-01-16T14:56:45.983804+0000","last_active":"2024-01-16T14:56:47.311230+0000","last_peered":"2024-01-16T14:56:47.311230+0000","last_clean":"2024-01-16T14:56:47.311230+0000","last_became_active":"2024-01-16T14:56:45.983529+0000","last_became_peered":"2024-01-16T14:56:45.983529+0000","last_unstale":"2024-01-16T14:56:47.311230+0000","last_undegraded":"2024-01-16T14:56:47.311230+0000","last_fullsized":"2024-01-16T14:56:47.311230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014146+0000","last_change":"2024-01-16T14:56:45.005514+0000","last_active":"2024-01-16T14:56:46.014146+0000","last_peered":"2024-01-16T14:56:46.014146+0000","last_clean":"2024-01-16T14:56:46.014146+0000","last_became_active":"2024-01-16T14:56:45.004610+0000","last_became_peered":"2024-01-16T14:56:45.004610+0000","last_unstale":"2024-01-16T14:56:46.014146+0000","last_undegraded":"2024-01-16T14:56:46.014146+0000","last_fullsized":"2024-01-16T14:56:46.014146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308980+0000","last_change":"2024-01-16T14:56:45.992067+0000","last_active":"2024-01-16T14:56:47.308980+0000","last_peered":"2024-01-16T14:56:47.308980+0000","last_clean":"2024-01-16T14:56:47.308980+0000","last_became_active":"2024-01-16T14:56:45.991826+0000","last_became_peered":"2024-01-16T14:56:45.991826+0000","last_unstale":"2024-01-16T14:56:47.308980+0000","last_undegraded":"2024-01-16T14:56:47.308980+0000","last_fullsized":"2024-01-16T14:56:47.308980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016889+0000","last_change":"2024-01-16T14:56:45.003619+0000","last_active":"2024-01-16T14:56:46.016889+0000","last_peered":"2024-01-16T14:56:46.016889+0000","last_clean":"2024-01-16T14:56:46.016889+0000","last_became_active":"2024-01-16T14:56:45.002981+0000","last_became_peered":"2024-01-16T14:56:45.002981+0000","last_unstale":"2024-01-16T14:56:46.016889+0000","last_undegraded":"2024-01-16T14:56:46.016889+0000","last_fullsized":"2024-01-16T14:56:46.016889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.782}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.972}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":660897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0910000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1389999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:56:55.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:56:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.039+0000 7fb6cba00640 1 Processor -- start 2024-01-16T14:56:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.039+0000 7fb6cba00640 1 -- start start 2024-01-16T14:56:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6cba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6cba00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6cba00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c406c470 con 0x7fb6c413e360 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6cba00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c406c750 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6cba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c406ca30 con 0x7fb6c413df80 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6c9a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6c3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb6c413e360 0x7fb6c40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36626/0 (socket says 172.21.15.125:36626) 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.041+0000 7fb6c3e00640 1 -- 172.21.15.125:0/2163287585 learned_addr learned my addr 172.21.15.125:0/2163287585 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c9000640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c9a00640 1 -- 172.21.15.125:0/2163287585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 msgr2=0x7fb6c415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c9a00640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c9a00640 1 -- 172.21.15.125:0/2163287585 >> v1:172.21.15.125:6789/0 conn(0x7fb6c413e360 legacy=0x7fb6c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c3400640 1 -- 172.21.15.125:0/2163287585 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245332904 0 0) 0x7fb6c406c470 con 0x7fb6c413e360 2024-01-16T14:56:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.042+0000 7fb6c9a00640 1 -- 172.21.15.125:0/2163287585 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c406cd10 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.043+0000 7fb6c9a00640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097a0 tx=0x7fb6b402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0428e6ee2236f9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.043+0000 7fb6c3400640 1 -- 172.21.15.125:0/2163287585 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b402d040 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.043+0000 7fb6c3400640 1 -- 172.21.15.125:0/2163287585 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b4031810 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.043+0000 7fb6c3400640 1 -- 172.21.15.125:0/2163287585 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4031ba0 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 -- 172.21.15.125:0/2163287585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6c40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097a0 tx=0x7fb6b402b8b0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 -- 172.21.15.125:0/2163287585 shutdown_connections 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2163287585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 -- 172.21.15.125:0/2163287585 >> 172.21.15.125:0/2163287585 conn(0x7fb6c406a800 msgr2=0x7fb6c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 -- 172.21.15.125:0/2163287585 shutdown_connections 2024-01-16T14:56:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.044+0000 7fb6cba00640 1 -- 172.21.15.125:0/2163287585 wait complete. 2024-01-16T14:56:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.045+0000 7fb6cba00640 1 Processor -- start 2024-01-16T14:56:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 -- start start 2024-01-16T14:56:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c40b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c413e360 0x7fb6c414f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c40b3ce0 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c414fe60 con 0x7fb6c413e360 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.046+0000 7fb6cba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c414eb60 con 0x7fb6c413df80 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48106/0 (socket says 172.21.15.125:48106) 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 -- 172.21.15.125:0/2332093843 learned_addr learned my addr 172.21.15.125:0/2332093843 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c3e00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c40b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 -- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 msgr2=0x7fb6c40b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c40b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 -- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c413e360 msgr2=0x7fb6c414f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c413e360 0x7fb6c414f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 -- 172.21.15.125:0/2332093843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4009400 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c9000640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d800 tx=0x7fb6bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00b6c0 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc00b840 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00bbf0 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.047+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c414ee40 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.048+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c414f400 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.049+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c414f620 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.049+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6bc00bd50 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.051+0000 7fb6c0c00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.051+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6bc08a320 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.051+0000 7fb6c3e00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.051+0000 7fb6c3e00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 0x7fb69c06d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb6c413df50 tx=0x7fb6ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.055+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6bc055060 con 0x7fb6c40cf9a0 2024-01-16T14:56:56.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.229+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6c414cb50 con 0x7fb69c06b2c0 2024-01-16T14:56:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.240+0000 7fb6c0c00640 1 -- 172.21.15.125:0/2332093843 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264400 (secure 0 0 0) 0x7fb6c414cb50 con 0x7fb69c06b2c0 2024-01-16T14:56:56.225 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:56.226 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.244+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 msgr2=0x7fb69c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.244+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 0x7fb69c06d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb6c413df50 tx=0x7fb6ac009040 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.244+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6c414cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.244+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d800 tx=0x7fb6bc00dcc0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.244+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 shutdown_connections 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c413e360 0x7fb6c414f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c413df80 0x7fb6c40b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 --2- 172.21.15.125:0/2332093843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c414cce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 >> 172.21.15.125:0/2332093843 conn(0x7fb6c406a800 msgr2=0x7fb6c40b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:56.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 shutdown_connections 2024-01-16T14:56:56.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.245+0000 7fb6cba00640 1 -- 172.21.15.125:0/2332093843 wait complete. 2024-01-16T14:56:56.243 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T14:56:55.921026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2887680,"data_stored":1751817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.975448"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956233+0000","last_change":"2024-01-16T14:56:45.980276+0000","last_active":"2024-01-16T14:56:46.956233+0000","last_peered":"2024-01-16T14:56:46.956233+0000","last_clean":"2024-01-16T14:56:46.956233+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956233+0000","last_undegraded":"2024-01-16T14:56:46.956233+0000","last_fullsized":"2024-01-16T14:56:46.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:53.535044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951915+0000","last_change":"2024-01-16T14:56:45.975609+0000","last_active":"2024-01-16T14:56:46.951915+0000","last_peered":"2024-01-16T14:56:46.951915+0000","last_clean":"2024-01-16T14:56:46.951915+0000","last_became_active":"2024-01-16T14:56:45.975291+0000","last_became_peered":"2024-01-16T14:56:45.975291+0000","last_unstale":"2024-01-16T14:56:46.951915+0000","last_undegraded":"2024-01-16T14:56:46.951915+0000","last_fullsized":"2024-01-16T14:56:46.951915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310992+0000","last_change":"2024-01-16T14:56:45.977545+0000","last_active":"2024-01-16T14:56:47.310992+0000","last_peered":"2024-01-16T14:56:47.310992+0000","last_clean":"2024-01-16T14:56:47.310992+0000","last_became_active":"2024-01-16T14:56:45.977253+0000","last_became_peered":"2024-01-16T14:56:45.977253+0000","last_unstale":"2024-01-16T14:56:47.310992+0000","last_undegraded":"2024-01-16T14:56:47.310992+0000","last_fullsized":"2024-01-16T14:56:47.310992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:34.727060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311691+0000","last_change":"2024-01-16T14:56:45.985746+0000","last_active":"2024-01-16T14:56:47.311691+0000","last_peered":"2024-01-16T14:56:47.311691+0000","last_clean":"2024-01-16T14:56:47.311691+0000","last_became_active":"2024-01-16T14:56:45.984925+0000","last_became_peered":"2024-01-16T14:56:45.984925+0000","last_unstale":"2024-01-16T14:56:47.311691+0000","last_undegraded":"2024-01-16T14:56:47.311691+0000","last_fullsized":"2024-01-16T14:56:47.311691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954569+0000","last_change":"2024-01-16T14:56:45.973905+0000","last_active":"2024-01-16T14:56:46.954569+0000","last_peered":"2024-01-16T14:56:46.954569+0000","last_clean":"2024-01-16T14:56:46.954569+0000","last_became_active":"2024-01-16T14:56:45.973591+0000","last_became_peered":"2024-01-16T14:56:45.973591+0000","last_unstale":"2024-01-16T14:56:46.954569+0000","last_undegraded":"2024-01-16T14:56:46.954569+0000","last_fullsized":"2024-01-16T14:56:46.954569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310746+0000","last_change":"2024-01-16T14:56:45.981879+0000","last_active":"2024-01-16T14:56:47.310746+0000","last_peered":"2024-01-16T14:56:47.310746+0000","last_clean":"2024-01-16T14:56:47.310746+0000","last_became_active":"2024-01-16T14:56:45.981506+0000","last_became_peered":"2024-01-16T14:56:45.981506+0000","last_unstale":"2024-01-16T14:56:47.310746+0000","last_undegraded":"2024-01-16T14:56:47.310746+0000","last_fullsized":"2024-01-16T14:56:47.310746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954278+0000","last_change":"2024-01-16T14:56:45.978039+0000","last_active":"2024-01-16T14:56:46.954278+0000","last_peered":"2024-01-16T14:56:46.954278+0000","last_clean":"2024-01-16T14:56:46.954278+0000","last_became_active":"2024-01-16T14:56:45.976116+0000","last_became_peered":"2024-01-16T14:56:45.976116+0000","last_unstale":"2024-01-16T14:56:46.954278+0000","last_undegraded":"2024-01-16T14:56:46.954278+0000","last_fullsized":"2024-01-16T14:56:46.954278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309521+0000","last_change":"2024-01-16T14:56:45.992939+0000","last_active":"2024-01-16T14:56:47.309521+0000","last_peered":"2024-01-16T14:56:47.309521+0000","last_clean":"2024-01-16T14:56:47.309521+0000","last_became_active":"2024-01-16T14:56:45.992563+0000","last_became_peered":"2024-01-16T14:56:45.992563+0000","last_unstale":"2024-01-16T14:56:47.309521+0000","last_undegraded":"2024-01-16T14:56:47.309521+0000","last_fullsized":"2024-01-16T14:56:47.309521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:00.111373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956108+0000","last_change":"2024-01-16T14:56:45.980818+0000","last_active":"2024-01-16T14:56:46.956108+0000","last_peered":"2024-01-16T14:56:46.956108+0000","last_clean":"2024-01-16T14:56:46.956108+0000","last_became_active":"2024-01-16T14:56:45.980565+0000","last_became_peered":"2024-01-16T14:56:45.980565+0000","last_unstale":"2024-01-16T14:56:46.956108+0000","last_undegraded":"2024-01-16T14:56:46.956108+0000","last_fullsized":"2024-01-16T14:56:46.956108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:05.361881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312563+0000","last_change":"2024-01-16T14:56:45.990281+0000","last_active":"2024-01-16T14:56:47.312563+0000","last_peered":"2024-01-16T14:56:47.312563+0000","last_clean":"2024-01-16T14:56:47.312563+0000","last_became_active":"2024-01-16T14:56:45.990014+0000","last_became_peered":"2024-01-16T14:56:45.990014+0000","last_unstale":"2024-01-16T14:56:47.312563+0000","last_undegraded":"2024-01-16T14:56:47.312563+0000","last_fullsized":"2024-01-16T14:56:47.312563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312223+0000","last_change":"2024-01-16T14:56:45.990395+0000","last_active":"2024-01-16T14:56:47.312223+0000","last_peered":"2024-01-16T14:56:47.312223+0000","last_clean":"2024-01-16T14:56:47.312223+0000","last_became_active":"2024-01-16T14:56:45.990100+0000","last_became_peered":"2024-01-16T14:56:45.990100+0000","last_unstale":"2024-01-16T14:56:47.312223+0000","last_undegraded":"2024-01-16T14:56:47.312223+0000","last_fullsized":"2024-01-16T14:56:47.312223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954100+0000","last_change":"2024-01-16T14:56:45.977544+0000","last_active":"2024-01-16T14:56:46.954100+0000","last_peered":"2024-01-16T14:56:46.954100+0000","last_clean":"2024-01-16T14:56:46.954100+0000","last_became_active":"2024-01-16T14:56:45.976912+0000","last_became_peered":"2024-01-16T14:56:45.976912+0000","last_unstale":"2024-01-16T14:56:46.954100+0000","last_undegraded":"2024-01-16T14:56:46.954100+0000","last_fullsized":"2024-01-16T14:56:46.954100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:04.924598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309263+0000","last_change":"2024-01-16T14:56:45.986451+0000","last_active":"2024-01-16T14:56:47.309263+0000","last_peered":"2024-01-16T14:56:47.309263+0000","last_clean":"2024-01-16T14:56:47.309263+0000","last_became_active":"2024-01-16T14:56:45.985694+0000","last_became_peered":"2024-01-16T14:56:45.985694+0000","last_unstale":"2024-01-16T14:56:47.309263+0000","last_undegraded":"2024-01-16T14:56:47.309263+0000","last_fullsized":"2024-01-16T14:56:47.309263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311052+0000","last_change":"2024-01-16T14:56:45.980372+0000","last_active":"2024-01-16T14:56:47.311052+0000","last_peered":"2024-01-16T14:56:47.311052+0000","last_clean":"2024-01-16T14:56:47.311052+0000","last_became_active":"2024-01-16T14:56:45.980114+0000","last_became_peered":"2024-01-16T14:56:45.980114+0000","last_unstale":"2024-01-16T14:56:47.311052+0000","last_undegraded":"2024-01-16T14:56:47.311052+0000","last_fullsized":"2024-01-16T14:56:47.311052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:10.565274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950877+0000","last_change":"2024-01-16T14:56:45.987106+0000","last_active":"2024-01-16T14:56:46.950877+0000","last_peered":"2024-01-16T14:56:46.950877+0000","last_clean":"2024-01-16T14:56:46.950877+0000","last_became_active":"2024-01-16T14:56:45.986126+0000","last_became_peered":"2024-01-16T14:56:45.986126+0000","last_unstale":"2024-01-16T14:56:46.950877+0000","last_undegraded":"2024-01-16T14:56:46.950877+0000","last_fullsized":"2024-01-16T14:56:46.950877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310669+0000","last_change":"2024-01-16T14:56:45.984242+0000","last_active":"2024-01-16T14:56:47.310669+0000","last_peered":"2024-01-16T14:56:47.310669+0000","last_clean":"2024-01-16T14:56:47.310669+0000","last_became_active":"2024-01-16T14:56:45.983936+0000","last_became_peered":"2024-01-16T14:56:45.983936+0000","last_unstale":"2024-01-16T14:56:47.310669+0000","last_undegraded":"2024-01-16T14:56:47.310669+0000","last_fullsized":"2024-01-16T14:56:47.310669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312880+0000","last_change":"2024-01-16T14:56:45.983089+0000","last_active":"2024-01-16T14:56:47.312880+0000","last_peered":"2024-01-16T14:56:47.312880+0000","last_clean":"2024-01-16T14:56:47.312880+0000","last_became_active":"2024-01-16T14:56:45.982784+0000","last_became_peered":"2024-01-16T14:56:45.982784+0000","last_unstale":"2024-01-16T14:56:47.312880+0000","last_undegraded":"2024-01-16T14:56:47.312880+0000","last_fullsized":"2024-01-16T14:56:47.312880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312245+0000","last_change":"2024-01-16T14:56:45.977860+0000","last_active":"2024-01-16T14:56:47.312245+0000","last_peered":"2024-01-16T14:56:47.312245+0000","last_clean":"2024-01-16T14:56:47.312245+0000","last_became_active":"2024-01-16T14:56:45.977563+0000","last_became_peered":"2024-01-16T14:56:45.977563+0000","last_unstale":"2024-01-16T14:56:47.312245+0000","last_undegraded":"2024-01-16T14:56:47.312245+0000","last_fullsized":"2024-01-16T14:56:47.312245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955821+0000","last_change":"2024-01-16T14:56:45.977287+0000","last_active":"2024-01-16T14:56:46.955821+0000","last_peered":"2024-01-16T14:56:46.955821+0000","last_clean":"2024-01-16T14:56:46.955821+0000","last_became_active":"2024-01-16T14:56:45.976639+0000","last_became_peered":"2024-01-16T14:56:45.976639+0000","last_unstale":"2024-01-16T14:56:46.955821+0000","last_undegraded":"2024-01-16T14:56:46.955821+0000","last_fullsized":"2024-01-16T14:56:46.955821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312028+0000","last_change":"2024-01-16T14:56:45.984360+0000","last_active":"2024-01-16T14:56:47.312028+0000","last_peered":"2024-01-16T14:56:47.312028+0000","last_clean":"2024-01-16T14:56:47.312028+0000","last_became_active":"2024-01-16T14:56:45.984102+0000","last_became_peered":"2024-01-16T14:56:45.984102+0000","last_unstale":"2024-01-16T14:56:47.312028+0000","last_undegraded":"2024-01-16T14:56:47.312028+0000","last_fullsized":"2024-01-16T14:56:47.312028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311119+0000","last_change":"2024-01-16T14:56:45.973908+0000","last_active":"2024-01-16T14:56:47.311119+0000","last_peered":"2024-01-16T14:56:47.311119+0000","last_clean":"2024-01-16T14:56:47.311119+0000","last_became_active":"2024-01-16T14:56:45.970417+0000","last_became_peered":"2024-01-16T14:56:45.970417+0000","last_unstale":"2024-01-16T14:56:47.311119+0000","last_undegraded":"2024-01-16T14:56:47.311119+0000","last_fullsized":"2024-01-16T14:56:47.311119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311312+0000","last_change":"2024-01-16T14:56:45.984704+0000","last_active":"2024-01-16T14:56:47.311312+0000","last_peered":"2024-01-16T14:56:47.311312+0000","last_clean":"2024-01-16T14:56:47.311312+0000","last_became_active":"2024-01-16T14:56:45.983436+0000","last_became_peered":"2024-01-16T14:56:45.983436+0000","last_unstale":"2024-01-16T14:56:47.311312+0000","last_undegraded":"2024-01-16T14:56:47.311312+0000","last_fullsized":"2024-01-16T14:56:47.311312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308745+0000","last_change":"2024-01-16T14:56:45.989345+0000","last_active":"2024-01-16T14:56:47.308745+0000","last_peered":"2024-01-16T14:56:47.308745+0000","last_clean":"2024-01-16T14:56:47.308745+0000","last_became_active":"2024-01-16T14:56:45.989090+0000","last_became_peered":"2024-01-16T14:56:45.989090+0000","last_unstale":"2024-01-16T14:56:47.308745+0000","last_undegraded":"2024-01-16T14:56:47.308745+0000","last_fullsized":"2024-01-16T14:56:47.308745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310832+0000","last_change":"2024-01-16T14:56:45.988656+0000","last_active":"2024-01-16T14:56:47.310832+0000","last_peered":"2024-01-16T14:56:47.310832+0000","last_clean":"2024-01-16T14:56:47.310832+0000","last_became_active":"2024-01-16T14:56:45.988350+0000","last_became_peered":"2024-01-16T14:56:45.988350+0000","last_unstale":"2024-01-16T14:56:47.310832+0000","last_undegraded":"2024-01-16T14:56:47.310832+0000","last_fullsized":"2024-01-16T14:56:47.310832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:39.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308691+0000","last_change":"2024-01-16T14:56:45.984838+0000","last_active":"2024-01-16T14:56:47.308691+0000","last_peered":"2024-01-16T14:56:47.308691+0000","last_clean":"2024-01-16T14:56:47.308691+0000","last_became_active":"2024-01-16T14:56:45.984542+0000","last_became_peered":"2024-01-16T14:56:45.984542+0000","last_unstale":"2024-01-16T14:56:47.308691+0000","last_undegraded":"2024-01-16T14:56:47.308691+0000","last_fullsized":"2024-01-16T14:56:47.308691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312755+0000","last_change":"2024-01-16T14:56:45.989694+0000","last_active":"2024-01-16T14:56:47.312755+0000","last_peered":"2024-01-16T14:56:47.312755+0000","last_clean":"2024-01-16T14:56:47.312755+0000","last_became_active":"2024-01-16T14:56:45.989384+0000","last_became_peered":"2024-01-16T14:56:45.989384+0000","last_unstale":"2024-01-16T14:56:47.312755+0000","last_undegraded":"2024-01-16T14:56:47.312755+0000","last_fullsized":"2024-01-16T14:56:47.312755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951404+0000","last_change":"2024-01-16T14:56:45.988721+0000","last_active":"2024-01-16T14:56:46.951404+0000","last_peered":"2024-01-16T14:56:46.951404+0000","last_clean":"2024-01-16T14:56:46.951404+0000","last_became_active":"2024-01-16T14:56:45.988428+0000","last_became_peered":"2024-01-16T14:56:45.988428+0000","last_unstale":"2024-01-16T14:56:46.951404+0000","last_undegraded":"2024-01-16T14:56:46.951404+0000","last_fullsized":"2024-01-16T14:56:46.951404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309579+0000","last_change":"2024-01-16T14:56:45.986196+0000","last_active":"2024-01-16T14:56:47.309579+0000","last_peered":"2024-01-16T14:56:47.309579+0000","last_clean":"2024-01-16T14:56:47.309579+0000","last_became_active":"2024-01-16T14:56:45.985927+0000","last_became_peered":"2024-01-16T14:56:45.985927+0000","last_unstale":"2024-01-16T14:56:47.309579+0000","last_undegraded":"2024-01-16T14:56:47.309579+0000","last_fullsized":"2024-01-16T14:56:47.309579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311237+0000","last_change":"2024-01-16T14:56:45.985321+0000","last_active":"2024-01-16T14:56:47.311237+0000","last_peered":"2024-01-16T14:56:47.311237+0000","last_clean":"2024-01-16T14:56:47.311237+0000","last_became_active":"2024-01-16T14:56:45.985015+0000","last_became_peered":"2024-01-16T14:56:45.985015+0000","last_unstale":"2024-01-16T14:56:47.311237+0000","last_undegraded":"2024-01-16T14:56:47.311237+0000","last_fullsized":"2024-01-16T14:56:47.311237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311646+0000","last_change":"2024-01-16T14:56:45.986171+0000","last_active":"2024-01-16T14:56:47.311646+0000","last_peered":"2024-01-16T14:56:47.311646+0000","last_clean":"2024-01-16T14:56:47.311646+0000","last_became_active":"2024-01-16T14:56:45.985890+0000","last_became_peered":"2024-01-16T14:56:45.985890+0000","last_unstale":"2024-01-16T14:56:47.311646+0000","last_undegraded":"2024-01-16T14:56:47.311646+0000","last_fullsized":"2024-01-16T14:56:47.311646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312240+0000","last_change":"2024-01-16T14:56:45.979713+0000","last_active":"2024-01-16T14:56:47.312240+0000","last_peered":"2024-01-16T14:56:47.312240+0000","last_clean":"2024-01-16T14:56:47.312240+0000","last_became_active":"2024-01-16T14:56:45.979379+0000","last_became_peered":"2024-01-16T14:56:45.979379+0000","last_unstale":"2024-01-16T14:56:47.312240+0000","last_undegraded":"2024-01-16T14:56:47.312240+0000","last_fullsized":"2024-01-16T14:56:47.312240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311228+0000","last_change":"2024-01-16T14:56:45.988420+0000","last_active":"2024-01-16T14:56:47.311228+0000","last_peered":"2024-01-16T14:56:47.311228+0000","last_clean":"2024-01-16T14:56:47.311228+0000","last_became_active":"2024-01-16T14:56:45.987162+0000","last_became_peered":"2024-01-16T14:56:45.987162+0000","last_unstale":"2024-01-16T14:56:47.311228+0000","last_undegraded":"2024-01-16T14:56:47.311228+0000","last_fullsized":"2024-01-16T14:56:47.311228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:52.822961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952781+0000","last_change":"2024-01-16T14:56:45.977184+0000","last_active":"2024-01-16T14:56:46.952781+0000","last_peered":"2024-01-16T14:56:46.952781+0000","last_clean":"2024-01-16T14:56:46.952781+0000","last_became_active":"2024-01-16T14:56:45.976825+0000","last_became_peered":"2024-01-16T14:56:45.976825+0000","last_unstale":"2024-01-16T14:56:46.952781+0000","last_undegraded":"2024-01-16T14:56:46.952781+0000","last_fullsized":"2024-01-16T14:56:46.952781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:33.458747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012001+0000","last_change":"2024-01-16T14:56:45.008639+0000","last_active":"2024-01-16T14:56:46.012001+0000","last_peered":"2024-01-16T14:56:46.012001+0000","last_clean":"2024-01-16T14:56:46.012001+0000","last_became_active":"2024-01-16T14:56:45.008234+0000","last_became_peered":"2024-01-16T14:56:45.008234+0000","last_unstale":"2024-01-16T14:56:46.012001+0000","last_undegraded":"2024-01-16T14:56:46.012001+0000","last_fullsized":"2024-01-16T14:56:46.012001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311455+0000","last_change":"2024-01-16T14:56:45.976366+0000","last_active":"2024-01-16T14:56:47.311455+0000","last_peered":"2024-01-16T14:56:47.311455+0000","last_clean":"2024-01-16T14:56:47.311455+0000","last_became_active":"2024-01-16T14:56:45.975839+0000","last_became_peered":"2024-01-16T14:56:45.975839+0000","last_unstale":"2024-01-16T14:56:47.311455+0000","last_undegraded":"2024-01-16T14:56:47.311455+0000","last_fullsized":"2024-01-16T14:56:47.311455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010576+0000","last_change":"2024-01-16T14:56:45.003662+0000","last_active":"2024-01-16T14:56:46.010576+0000","last_peered":"2024-01-16T14:56:46.010576+0000","last_clean":"2024-01-16T14:56:46.010576+0000","last_became_active":"2024-01-16T14:56:45.003318+0000","last_became_peered":"2024-01-16T14:56:45.003318+0000","last_unstale":"2024-01-16T14:56:46.010576+0000","last_undegraded":"2024-01-16T14:56:46.010576+0000","last_fullsized":"2024-01-16T14:56:46.010576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.982735+0000","last_change":"2024-01-16T14:56:45.013121+0000","last_active":"2024-01-16T14:56:45.982735+0000","last_peered":"2024-01-16T14:56:45.982735+0000","last_clean":"2024-01-16T14:56:45.982735+0000","last_became_active":"2024-01-16T14:56:45.012797+0000","last_became_peered":"2024-01-16T14:56:45.012797+0000","last_unstale":"2024-01-16T14:56:45.982735+0000","last_undegraded":"2024-01-16T14:56:45.982735+0000","last_fullsized":"2024-01-16T14:56:45.982735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308918+0000","last_change":"2024-01-16T14:56:45.986213+0000","last_active":"2024-01-16T14:56:47.308918+0000","last_peered":"2024-01-16T14:56:47.308918+0000","last_clean":"2024-01-16T14:56:47.308918+0000","last_became_active":"2024-01-16T14:56:45.985824+0000","last_became_peered":"2024-01-16T14:56:45.985824+0000","last_unstale":"2024-01-16T14:56:47.308918+0000","last_undegraded":"2024-01-16T14:56:47.308918+0000","last_fullsized":"2024-01-16T14:56:47.308918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.974634+0000","last_change":"2024-01-16T14:56:45.014687+0000","last_active":"2024-01-16T14:56:45.974634+0000","last_peered":"2024-01-16T14:56:45.974634+0000","last_clean":"2024-01-16T14:56:45.974634+0000","last_became_active":"2024-01-16T14:56:45.014356+0000","last_became_peered":"2024-01-16T14:56:45.014356+0000","last_unstale":"2024-01-16T14:56:45.974634+0000","last_undegraded":"2024-01-16T14:56:45.974634+0000","last_fullsized":"2024-01-16T14:56:45.974634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:34.974850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950693+0000","last_change":"2024-01-16T14:56:45.984146+0000","last_active":"2024-01-16T14:56:46.950693+0000","last_peered":"2024-01-16T14:56:46.950693+0000","last_clean":"2024-01-16T14:56:46.950693+0000","last_became_active":"2024-01-16T14:56:45.983841+0000","last_became_peered":"2024-01-16T14:56:45.983841+0000","last_unstale":"2024-01-16T14:56:46.950693+0000","last_undegraded":"2024-01-16T14:56:46.950693+0000","last_fullsized":"2024-01-16T14:56:46.950693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.971677+0000","last_change":"2024-01-16T14:56:45.011843+0000","last_active":"2024-01-16T14:56:45.971677+0000","last_peered":"2024-01-16T14:56:45.971677+0000","last_clean":"2024-01-16T14:56:45.971677+0000","last_became_active":"2024-01-16T14:56:45.011331+0000","last_became_peered":"2024-01-16T14:56:45.011331+0000","last_unstale":"2024-01-16T14:56:45.971677+0000","last_undegraded":"2024-01-16T14:56:45.971677+0000","last_fullsized":"2024-01-16T14:56:45.971677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:33.714416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313267+0000","last_change":"2024-01-16T14:56:45.986253+0000","last_active":"2024-01-16T14:56:47.313267+0000","last_peered":"2024-01-16T14:56:47.313267+0000","last_clean":"2024-01-16T14:56:47.313267+0000","last_became_active":"2024-01-16T14:56:45.985935+0000","last_became_peered":"2024-01-16T14:56:45.985935+0000","last_unstale":"2024-01-16T14:56:47.313267+0000","last_undegraded":"2024-01-16T14:56:47.313267+0000","last_fullsized":"2024-01-16T14:56:47.313267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.973280+0000","last_change":"2024-01-16T14:56:45.016005+0000","last_active":"2024-01-16T14:56:45.973280+0000","last_peered":"2024-01-16T14:56:45.973280+0000","last_clean":"2024-01-16T14:56:45.973280+0000","last_became_active":"2024-01-16T14:56:45.015680+0000","last_became_peered":"2024-01-16T14:56:45.015680+0000","last_unstale":"2024-01-16T14:56:45.973280+0000","last_undegraded":"2024-01-16T14:56:45.973280+0000","last_fullsized":"2024-01-16T14:56:45.973280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:19.042486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311306+0000","last_change":"2024-01-16T14:56:45.980174+0000","last_active":"2024-01-16T14:56:47.311306+0000","last_peered":"2024-01-16T14:56:47.311306+0000","last_clean":"2024-01-16T14:56:47.311306+0000","last_became_active":"2024-01-16T14:56:45.979743+0000","last_became_peered":"2024-01-16T14:56:45.979743+0000","last_unstale":"2024-01-16T14:56:47.311306+0000","last_undegraded":"2024-01-16T14:56:47.311306+0000","last_fullsized":"2024-01-16T14:56:47.311306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013898+0000","last_change":"2024-01-16T14:56:45.017298+0000","last_active":"2024-01-16T14:56:46.013898+0000","last_peered":"2024-01-16T14:56:46.013898+0000","last_clean":"2024-01-16T14:56:46.013898+0000","last_became_active":"2024-01-16T14:56:45.016921+0000","last_became_peered":"2024-01-16T14:56:45.016921+0000","last_unstale":"2024-01-16T14:56:46.013898+0000","last_undegraded":"2024-01-16T14:56:46.013898+0000","last_fullsized":"2024-01-16T14:56:46.013898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954466+0000","last_change":"2024-01-16T14:56:45.967606+0000","last_active":"2024-01-16T14:56:46.954466+0000","last_peered":"2024-01-16T14:56:46.954466+0000","last_clean":"2024-01-16T14:56:46.954466+0000","last_became_active":"2024-01-16T14:56:45.967036+0000","last_became_peered":"2024-01-16T14:56:45.967036+0000","last_unstale":"2024-01-16T14:56:46.954466+0000","last_undegraded":"2024-01-16T14:56:46.954466+0000","last_fullsized":"2024-01-16T14:56:46.954466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:09.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015870+0000","last_change":"2024-01-16T14:56:45.005985+0000","last_active":"2024-01-16T14:56:46.015870+0000","last_peered":"2024-01-16T14:56:46.015870+0000","last_clean":"2024-01-16T14:56:46.015870+0000","last_became_active":"2024-01-16T14:56:45.005530+0000","last_became_peered":"2024-01-16T14:56:45.005530+0000","last_unstale":"2024-01-16T14:56:46.015870+0000","last_undegraded":"2024-01-16T14:56:46.015870+0000","last_fullsized":"2024-01-16T14:56:46.015870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:17.631008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955160+0000","last_change":"2024-01-16T14:56:45.991157+0000","last_active":"2024-01-16T14:56:46.955160+0000","last_peered":"2024-01-16T14:56:46.955160+0000","last_clean":"2024-01-16T14:56:46.955160+0000","last_became_active":"2024-01-16T14:56:45.990848+0000","last_became_peered":"2024-01-16T14:56:45.990848+0000","last_unstale":"2024-01-16T14:56:46.955160+0000","last_undegraded":"2024-01-16T14:56:46.955160+0000","last_fullsized":"2024-01-16T14:56:46.955160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:20.859609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016578+0000","last_change":"2024-01-16T14:56:45.000374+0000","last_active":"2024-01-16T14:56:46.016578+0000","last_peered":"2024-01-16T14:56:46.016578+0000","last_clean":"2024-01-16T14:56:46.016578+0000","last_became_active":"2024-01-16T14:56:45.000028+0000","last_became_peered":"2024-01-16T14:56:45.000028+0000","last_unstale":"2024-01-16T14:56:46.016578+0000","last_undegraded":"2024-01-16T14:56:46.016578+0000","last_fullsized":"2024-01-16T14:56:46.016578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311398+0000","last_change":"2024-01-16T14:56:45.974961+0000","last_active":"2024-01-16T14:56:47.311398+0000","last_peered":"2024-01-16T14:56:47.311398+0000","last_clean":"2024-01-16T14:56:47.311398+0000","last_became_active":"2024-01-16T14:56:45.974581+0000","last_became_peered":"2024-01-16T14:56:45.974581+0000","last_unstale":"2024-01-16T14:56:47.311398+0000","last_undegraded":"2024-01-16T14:56:47.311398+0000","last_fullsized":"2024-01-16T14:56:47.311398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.972980+0000","last_change":"2024-01-16T14:56:45.017987+0000","last_active":"2024-01-16T14:56:45.972980+0000","last_peered":"2024-01-16T14:56:45.972980+0000","last_clean":"2024-01-16T14:56:45.972980+0000","last_became_active":"2024-01-16T14:56:45.017656+0000","last_became_peered":"2024-01-16T14:56:45.017656+0000","last_unstale":"2024-01-16T14:56:45.972980+0000","last_undegraded":"2024-01-16T14:56:45.972980+0000","last_fullsized":"2024-01-16T14:56:45.972980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954932+0000","last_change":"2024-01-16T14:56:45.976869+0000","last_active":"2024-01-16T14:56:46.954932+0000","last_peered":"2024-01-16T14:56:46.954932+0000","last_clean":"2024-01-16T14:56:46.954932+0000","last_became_active":"2024-01-16T14:56:45.976500+0000","last_became_peered":"2024-01-16T14:56:45.976500+0000","last_unstale":"2024-01-16T14:56:46.954932+0000","last_undegraded":"2024-01-16T14:56:46.954932+0000","last_fullsized":"2024-01-16T14:56:46.954932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:14.640548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.966318+0000","last_change":"2024-01-16T14:56:45.012069+0000","last_active":"2024-01-16T14:56:45.966318+0000","last_peered":"2024-01-16T14:56:45.966318+0000","last_clean":"2024-01-16T14:56:45.966318+0000","last_became_active":"2024-01-16T14:56:45.011711+0000","last_became_peered":"2024-01-16T14:56:45.011711+0000","last_unstale":"2024-01-16T14:56:45.966318+0000","last_undegraded":"2024-01-16T14:56:45.966318+0000","last_fullsized":"2024-01-16T14:56:45.966318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311361+0000","last_change":"2024-01-16T14:56:45.988281+0000","last_active":"2024-01-16T14:56:47.311361+0000","last_peered":"2024-01-16T14:56:47.311361+0000","last_clean":"2024-01-16T14:56:47.311361+0000","last_became_active":"2024-01-16T14:56:45.986521+0000","last_became_peered":"2024-01-16T14:56:45.986521+0000","last_unstale":"2024-01-16T14:56:47.311361+0000","last_undegraded":"2024-01-16T14:56:47.311361+0000","last_fullsized":"2024-01-16T14:56:47.311361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.015745+0000","last_change":"2024-01-16T14:56:44.993188+0000","last_active":"2024-01-16T14:56:46.015745+0000","last_peered":"2024-01-16T14:56:46.015745+0000","last_clean":"2024-01-16T14:56:46.015745+0000","last_became_active":"2024-01-16T14:56:44.992904+0000","last_became_peered":"2024-01-16T14:56:44.992904+0000","last_unstale":"2024-01-16T14:56:46.015745+0000","last_undegraded":"2024-01-16T14:56:46.015745+0000","last_fullsized":"2024-01-16T14:56:46.015745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.524009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954660+0000","last_change":"2024-01-16T14:56:45.981562+0000","last_active":"2024-01-16T14:56:46.954660+0000","last_peered":"2024-01-16T14:56:46.954660+0000","last_clean":"2024-01-16T14:56:46.954660+0000","last_became_active":"2024-01-16T14:56:45.981272+0000","last_became_peered":"2024-01-16T14:56:45.981272+0000","last_unstale":"2024-01-16T14:56:46.954660+0000","last_undegraded":"2024-01-16T14:56:46.954660+0000","last_fullsized":"2024-01-16T14:56:46.954660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013126+0000","last_change":"2024-01-16T14:56:45.005288+0000","last_active":"2024-01-16T14:56:46.013126+0000","last_peered":"2024-01-16T14:56:46.013126+0000","last_clean":"2024-01-16T14:56:46.013126+0000","last_became_active":"2024-01-16T14:56:45.004988+0000","last_became_peered":"2024-01-16T14:56:45.004988+0000","last_unstale":"2024-01-16T14:56:46.013126+0000","last_undegraded":"2024-01-16T14:56:46.013126+0000","last_fullsized":"2024-01-16T14:56:46.013126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955886+0000","last_change":"2024-01-16T14:56:45.977479+0000","last_active":"2024-01-16T14:56:46.955886+0000","last_peered":"2024-01-16T14:56:46.955886+0000","last_clean":"2024-01-16T14:56:46.955886+0000","last_became_active":"2024-01-16T14:56:45.976447+0000","last_became_peered":"2024-01-16T14:56:45.976447+0000","last_unstale":"2024-01-16T14:56:46.955886+0000","last_undegraded":"2024-01-16T14:56:46.955886+0000","last_fullsized":"2024-01-16T14:56:46.955886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:09:28.671518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.012387+0000","last_change":"2024-01-16T14:56:44.991027+0000","last_active":"2024-01-16T14:56:46.012387+0000","last_peered":"2024-01-16T14:56:46.012387+0000","last_clean":"2024-01-16T14:56:46.012387+0000","last_became_active":"2024-01-16T14:56:44.990777+0000","last_became_peered":"2024-01-16T14:56:44.990777+0000","last_unstale":"2024-01-16T14:56:46.012387+0000","last_undegraded":"2024-01-16T14:56:46.012387+0000","last_fullsized":"2024-01-16T14:56:46.012387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:22.352401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.956303+0000","last_change":"2024-01-16T14:56:45.980059+0000","last_active":"2024-01-16T14:56:46.956303+0000","last_peered":"2024-01-16T14:56:46.956303+0000","last_clean":"2024-01-16T14:56:46.956303+0000","last_became_active":"2024-01-16T14:56:45.979760+0000","last_became_peered":"2024-01-16T14:56:45.979760+0000","last_unstale":"2024-01-16T14:56:46.956303+0000","last_undegraded":"2024-01-16T14:56:46.956303+0000","last_fullsized":"2024-01-16T14:56:46.956303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:40.703250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.010244+0000","last_change":"2024-01-16T14:56:45.001611+0000","last_active":"2024-01-16T14:56:46.010244+0000","last_peered":"2024-01-16T14:56:46.010244+0000","last_clean":"2024-01-16T14:56:46.010244+0000","last_became_active":"2024-01-16T14:56:45.001338+0000","last_became_peered":"2024-01-16T14:56:45.001338+0000","last_unstale":"2024-01-16T14:56:46.010244+0000","last_undegraded":"2024-01-16T14:56:46.010244+0000","last_fullsized":"2024-01-16T14:56:46.010244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.312030+0000","last_change":"2024-01-16T14:56:45.986001+0000","last_active":"2024-01-16T14:56:47.312030+0000","last_peered":"2024-01-16T14:56:47.312030+0000","last_clean":"2024-01-16T14:56:47.312030+0000","last_became_active":"2024-01-16T14:56:45.984074+0000","last_became_peered":"2024-01-16T14:56:45.984074+0000","last_unstale":"2024-01-16T14:56:47.312030+0000","last_undegraded":"2024-01-16T14:56:47.312030+0000","last_fullsized":"2024-01-16T14:56:47.312030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:41.121667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.960950+0000","last_change":"2024-01-16T14:56:45.009683+0000","last_active":"2024-01-16T14:56:45.960950+0000","last_peered":"2024-01-16T14:56:45.960950+0000","last_clean":"2024-01-16T14:56:45.960950+0000","last_became_active":"2024-01-16T14:56:45.009325+0000","last_became_peered":"2024-01-16T14:56:45.009325+0000","last_unstale":"2024-01-16T14:56:45.960950+0000","last_undegraded":"2024-01-16T14:56:45.960950+0000","last_fullsized":"2024-01-16T14:56:45.960950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:27.159214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.955489+0000","last_change":"2024-01-16T14:56:45.989743+0000","last_active":"2024-01-16T14:56:46.955489+0000","last_peered":"2024-01-16T14:56:46.955489+0000","last_clean":"2024-01-16T14:56:46.955489+0000","last_became_active":"2024-01-16T14:56:45.989084+0000","last_became_peered":"2024-01-16T14:56:45.989084+0000","last_unstale":"2024-01-16T14:56:46.955489+0000","last_undegraded":"2024-01-16T14:56:46.955489+0000","last_fullsized":"2024-01-16T14:56:46.955489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.955711+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:56:45.955711+0000","last_peered":"2024-01-16T14:56:45.955711+0000","last_clean":"2024-01-16T14:56:45.955711+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:56:45.955711+0000","last_undegraded":"2024-01-16T14:56:45.955711+0000","last_fullsized":"2024-01-16T14:56:45.955711+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963629+0000","last_change":"2024-01-16T14:56:45.009432+0000","last_active":"2024-01-16T14:56:45.963629+0000","last_peered":"2024-01-16T14:56:45.963629+0000","last_clean":"2024-01-16T14:56:45.963629+0000","last_became_active":"2024-01-16T14:56:45.008692+0000","last_became_peered":"2024-01-16T14:56:45.008692+0000","last_unstale":"2024-01-16T14:56:45.963629+0000","last_undegraded":"2024-01-16T14:56:45.963629+0000","last_fullsized":"2024-01-16T14:56:45.963629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:55.977707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308881+0000","last_change":"2024-01-16T14:56:45.990578+0000","last_active":"2024-01-16T14:56:47.308881+0000","last_peered":"2024-01-16T14:56:47.308881+0000","last_clean":"2024-01-16T14:56:47.308881+0000","last_became_active":"2024-01-16T14:56:45.990211+0000","last_became_peered":"2024-01-16T14:56:45.990211+0000","last_unstale":"2024-01-16T14:56:47.308881+0000","last_undegraded":"2024-01-16T14:56:47.308881+0000","last_fullsized":"2024-01-16T14:56:47.308881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.956468+0000","last_change":"2024-01-16T14:56:45.006632+0000","last_active":"2024-01-16T14:56:45.956468+0000","last_peered":"2024-01-16T14:56:45.956468+0000","last_clean":"2024-01-16T14:56:45.956468+0000","last_became_active":"2024-01-16T14:56:45.006340+0000","last_became_peered":"2024-01-16T14:56:45.006340+0000","last_unstale":"2024-01-16T14:56:45.956468+0000","last_undegraded":"2024-01-16T14:56:45.956468+0000","last_fullsized":"2024-01-16T14:56:45.956468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:41.754460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.310892+0000","last_change":"2024-01-16T14:56:45.977532+0000","last_active":"2024-01-16T14:56:47.310892+0000","last_peered":"2024-01-16T14:56:47.310892+0000","last_clean":"2024-01-16T14:56:47.310892+0000","last_became_active":"2024-01-16T14:56:45.975102+0000","last_became_peered":"2024-01-16T14:56:45.975102+0000","last_unstale":"2024-01-16T14:56:47.310892+0000","last_undegraded":"2024-01-16T14:56:47.310892+0000","last_fullsized":"2024-01-16T14:56:47.310892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:18.643652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.963071+0000","last_change":"2024-01-16T14:56:44.994705+0000","last_active":"2024-01-16T14:56:45.963071+0000","last_peered":"2024-01-16T14:56:45.963071+0000","last_clean":"2024-01-16T14:56:45.963071+0000","last_became_active":"2024-01-16T14:56:44.994387+0000","last_became_peered":"2024-01-16T14:56:44.994387+0000","last_unstale":"2024-01-16T14:56:45.963071+0000","last_undegraded":"2024-01-16T14:56:45.963071+0000","last_fullsized":"2024-01-16T14:56:45.963071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:32.446103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.950691+0000","last_change":"2024-01-16T14:56:45.978166+0000","last_active":"2024-01-16T14:56:46.950691+0000","last_peered":"2024-01-16T14:56:46.950691+0000","last_clean":"2024-01-16T14:56:46.950691+0000","last_became_active":"2024-01-16T14:56:45.974628+0000","last_became_peered":"2024-01-16T14:56:45.974628+0000","last_unstale":"2024-01-16T14:56:46.950691+0000","last_undegraded":"2024-01-16T14:56:46.950691+0000","last_fullsized":"2024-01-16T14:56:46.950691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:52.566583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.980309+0000","last_change":"2024-01-16T14:56:45.017774+0000","last_active":"2024-01-16T14:56:45.980309+0000","last_peered":"2024-01-16T14:56:45.980309+0000","last_clean":"2024-01-16T14:56:45.980309+0000","last_became_active":"2024-01-16T14:56:45.017410+0000","last_became_peered":"2024-01-16T14:56:45.017410+0000","last_unstale":"2024-01-16T14:56:45.980309+0000","last_undegraded":"2024-01-16T14:56:45.980309+0000","last_fullsized":"2024-01-16T14:56:45.980309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:59.852603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.954411+0000","last_change":"2024-01-16T14:56:45.970975+0000","last_active":"2024-01-16T14:56:46.954411+0000","last_peered":"2024-01-16T14:56:46.954411+0000","last_clean":"2024-01-16T14:56:46.954411+0000","last_became_active":"2024-01-16T14:56:45.970566+0000","last_became_peered":"2024-01-16T14:56:45.970566+0000","last_unstale":"2024-01-16T14:56:46.954411+0000","last_undegraded":"2024-01-16T14:56:46.954411+0000","last_fullsized":"2024-01-16T14:56:46.954411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.967326+0000","last_change":"2024-01-16T14:56:45.004864+0000","last_active":"2024-01-16T14:56:45.967326+0000","last_peered":"2024-01-16T14:56:45.967326+0000","last_clean":"2024-01-16T14:56:45.967326+0000","last_became_active":"2024-01-16T14:56:45.004057+0000","last_became_peered":"2024-01-16T14:56:45.004057+0000","last_unstale":"2024-01-16T14:56:45.967326+0000","last_undegraded":"2024-01-16T14:56:45.967326+0000","last_fullsized":"2024-01-16T14:56:45.967326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313581+0000","last_change":"2024-01-16T14:56:45.989780+0000","last_active":"2024-01-16T14:56:47.313581+0000","last_peered":"2024-01-16T14:56:47.313581+0000","last_clean":"2024-01-16T14:56:47.313581+0000","last_became_active":"2024-01-16T14:56:45.989519+0000","last_became_peered":"2024-01-16T14:56:45.989519+0000","last_unstale":"2024-01-16T14:56:47.313581+0000","last_undegraded":"2024-01-16T14:56:47.313581+0000","last_fullsized":"2024-01-16T14:56:47.313581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:10.321784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.959086+0000","last_change":"2024-01-16T14:56:45.009420+0000","last_active":"2024-01-16T14:56:45.959086+0000","last_peered":"2024-01-16T14:56:45.959086+0000","last_clean":"2024-01-16T14:56:45.959086+0000","last_became_active":"2024-01-16T14:56:45.009103+0000","last_became_peered":"2024-01-16T14:56:45.009103+0000","last_unstale":"2024-01-16T14:56:45.959086+0000","last_undegraded":"2024-01-16T14:56:45.959086+0000","last_fullsized":"2024-01-16T14:56:45.959086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:49.027396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.951463+0000","last_change":"2024-01-16T14:56:45.976716+0000","last_active":"2024-01-16T14:56:46.951463+0000","last_peered":"2024-01-16T14:56:46.951463+0000","last_clean":"2024-01-16T14:56:46.951463+0000","last_became_active":"2024-01-16T14:56:45.976307+0000","last_became_peered":"2024-01-16T14:56:45.976307+0000","last_unstale":"2024-01-16T14:56:46.951463+0000","last_undegraded":"2024-01-16T14:56:46.951463+0000","last_fullsized":"2024-01-16T14:56:46.951463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:03.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952095+0000","last_change":"2024-01-16T14:56:45.013169+0000","last_active":"2024-01-16T14:56:46.952095+0000","last_peered":"2024-01-16T14:56:46.952095+0000","last_clean":"2024-01-16T14:56:46.952095+0000","last_became_active":"2024-01-16T14:56:45.012872+0000","last_became_peered":"2024-01-16T14:56:45.012872+0000","last_unstale":"2024-01-16T14:56:46.952095+0000","last_undegraded":"2024-01-16T14:56:46.952095+0000","last_fullsized":"2024-01-16T14:56:46.952095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308843+0000","last_change":"2024-01-16T14:56:45.987343+0000","last_active":"2024-01-16T14:56:47.308843+0000","last_peered":"2024-01-16T14:56:47.308843+0000","last_clean":"2024-01-16T14:56:47.308843+0000","last_became_active":"2024-01-16T14:56:45.987065+0000","last_became_peered":"2024-01-16T14:56:45.987065+0000","last_unstale":"2024-01-16T14:56:47.308843+0000","last_undegraded":"2024-01-16T14:56:47.308843+0000","last_fullsized":"2024-01-16T14:56:47.308843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:35.987495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014224+0000","last_change":"2024-01-16T14:56:45.004437+0000","last_active":"2024-01-16T14:56:46.014224+0000","last_peered":"2024-01-16T14:56:46.014224+0000","last_clean":"2024-01-16T14:56:46.014224+0000","last_became_active":"2024-01-16T14:56:45.004165+0000","last_became_peered":"2024-01-16T14:56:45.004165+0000","last_unstale":"2024-01-16T14:56:46.014224+0000","last_undegraded":"2024-01-16T14:56:46.014224+0000","last_fullsized":"2024-01-16T14:56:46.014224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:09.552630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311396+0000","last_change":"2024-01-16T14:56:45.974262+0000","last_active":"2024-01-16T14:56:47.311396+0000","last_peered":"2024-01-16T14:56:47.311396+0000","last_clean":"2024-01-16T14:56:47.311396+0000","last_became_active":"2024-01-16T14:56:45.970037+0000","last_became_peered":"2024-01-16T14:56:45.970037+0000","last_unstale":"2024-01-16T14:56:47.311396+0000","last_undegraded":"2024-01-16T14:56:47.311396+0000","last_fullsized":"2024-01-16T14:56:47.311396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:00.865248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.952176+0000","last_change":"2024-01-16T14:56:45.009771+0000","last_active":"2024-01-16T14:56:45.952176+0000","last_peered":"2024-01-16T14:56:45.952176+0000","last_clean":"2024-01-16T14:56:45.952176+0000","last_became_active":"2024-01-16T14:56:45.009505+0000","last_became_peered":"2024-01-16T14:56:45.009505+0000","last_unstale":"2024-01-16T14:56:45.952176+0000","last_undegraded":"2024-01-16T14:56:45.952176+0000","last_fullsized":"2024-01-16T14:56:45.952176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:03.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.952907+0000","last_change":"2024-01-16T14:56:45.970586+0000","last_active":"2024-01-16T14:56:46.952907+0000","last_peered":"2024-01-16T14:56:46.952907+0000","last_clean":"2024-01-16T14:56:46.952907+0000","last_became_active":"2024-01-16T14:56:45.970146+0000","last_became_peered":"2024-01-16T14:56:45.970146+0000","last_unstale":"2024-01-16T14:56:46.952907+0000","last_undegraded":"2024-01-16T14:56:46.952907+0000","last_fullsized":"2024-01-16T14:56:46.952907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:23.365045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.962308+0000","last_change":"2024-01-16T14:56:45.003139+0000","last_active":"2024-01-16T14:56:45.962308+0000","last_peered":"2024-01-16T14:56:45.962308+0000","last_clean":"2024-01-16T14:56:45.962308+0000","last_became_active":"2024-01-16T14:56:45.002787+0000","last_became_peered":"2024-01-16T14:56:45.002787+0000","last_unstale":"2024-01-16T14:56:45.962308+0000","last_undegraded":"2024-01-16T14:56:45.962308+0000","last_fullsized":"2024-01-16T14:56:45.962308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309294+0000","last_change":"2024-01-16T14:56:45.981803+0000","last_active":"2024-01-16T14:56:47.309294+0000","last_peered":"2024-01-16T14:56:47.309294+0000","last_clean":"2024-01-16T14:56:47.309294+0000","last_became_active":"2024-01-16T14:56:45.981480+0000","last_became_peered":"2024-01-16T14:56:45.981480+0000","last_unstale":"2024-01-16T14:56:47.309294+0000","last_undegraded":"2024-01-16T14:56:47.309294+0000","last_fullsized":"2024-01-16T14:56:47.309294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.040040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:45.968506+0000","last_change":"2024-01-16T14:56:45.006149+0000","last_active":"2024-01-16T14:56:45.968506+0000","last_peered":"2024-01-16T14:56:45.968506+0000","last_clean":"2024-01-16T14:56:45.968506+0000","last_became_active":"2024-01-16T14:56:45.005789+0000","last_became_peered":"2024-01-16T14:56:45.005789+0000","last_unstale":"2024-01-16T14:56:45.968506+0000","last_undegraded":"2024-01-16T14:56:45.968506+0000","last_fullsized":"2024-01-16T14:56:45.968506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.309478+0000","last_change":"2024-01-16T14:56:45.985569+0000","last_active":"2024-01-16T14:56:47.309478+0000","last_peered":"2024-01-16T14:56:47.309478+0000","last_clean":"2024-01-16T14:56:47.309478+0000","last_became_active":"2024-01-16T14:56:45.983774+0000","last_became_peered":"2024-01-16T14:56:45.983774+0000","last_unstale":"2024-01-16T14:56:47.309478+0000","last_undegraded":"2024-01-16T14:56:47.309478+0000","last_fullsized":"2024-01-16T14:56:47.309478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:29.874795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.013631+0000","last_change":"2024-01-16T14:56:45.010441+0000","last_active":"2024-01-16T14:56:46.013631+0000","last_peered":"2024-01-16T14:56:46.013631+0000","last_clean":"2024-01-16T14:56:46.013631+0000","last_became_active":"2024-01-16T14:56:45.007667+0000","last_became_peered":"2024-01-16T14:56:45.007667+0000","last_unstale":"2024-01-16T14:56:46.013631+0000","last_undegraded":"2024-01-16T14:56:46.013631+0000","last_fullsized":"2024-01-16T14:56:46.013631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:38.016001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311624+0000","last_change":"2024-01-16T14:56:45.984334+0000","last_active":"2024-01-16T14:56:47.311624+0000","last_peered":"2024-01-16T14:56:47.311624+0000","last_clean":"2024-01-16T14:56:47.311624+0000","last_became_active":"2024-01-16T14:56:45.984013+0000","last_became_peered":"2024-01-16T14:56:45.984013+0000","last_unstale":"2024-01-16T14:56:47.311624+0000","last_undegraded":"2024-01-16T14:56:47.311624+0000","last_fullsized":"2024-01-16T14:56:47.311624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.313406+0000","last_change":"2024-01-16T14:56:45.985148+0000","last_active":"2024-01-16T14:56:47.313406+0000","last_peered":"2024-01-16T14:56:47.313406+0000","last_clean":"2024-01-16T14:56:47.313406+0000","last_became_active":"2024-01-16T14:56:45.983354+0000","last_became_peered":"2024-01-16T14:56:45.983354+0000","last_unstale":"2024-01-16T14:56:47.313406+0000","last_undegraded":"2024-01-16T14:56:47.313406+0000","last_fullsized":"2024-01-16T14:56:47.313406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:37.783867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011363+0000","last_change":"2024-01-16T14:56:45.017135+0000","last_active":"2024-01-16T14:56:46.011363+0000","last_peered":"2024-01-16T14:56:46.011363+0000","last_clean":"2024-01-16T14:56:46.011363+0000","last_became_active":"2024-01-16T14:56:45.016686+0000","last_became_peered":"2024-01-16T14:56:45.016686+0000","last_unstale":"2024-01-16T14:56:46.011363+0000","last_undegraded":"2024-01-16T14:56:46.011363+0000","last_fullsized":"2024-01-16T14:56:46.011363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:13.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311883+0000","last_change":"2024-01-16T14:56:45.983050+0000","last_active":"2024-01-16T14:56:47.311883+0000","last_peered":"2024-01-16T14:56:47.311883+0000","last_clean":"2024-01-16T14:56:47.311883+0000","last_became_active":"2024-01-16T14:56:45.982678+0000","last_became_peered":"2024-01-16T14:56:45.982678+0000","last_unstale":"2024-01-16T14:56:47.311883+0000","last_undegraded":"2024-01-16T14:56:47.311883+0000","last_fullsized":"2024-01-16T14:56:47.311883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:40.901618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.011339+0000","last_change":"2024-01-16T14:56:45.014696+0000","last_active":"2024-01-16T14:56:46.011339+0000","last_peered":"2024-01-16T14:56:46.011339+0000","last_clean":"2024-01-16T14:56:46.011339+0000","last_became_active":"2024-01-16T14:56:45.014410+0000","last_became_peered":"2024-01-16T14:56:45.014410+0000","last_unstale":"2024-01-16T14:56:46.011339+0000","last_undegraded":"2024-01-16T14:56:46.011339+0000","last_fullsized":"2024-01-16T14:56:46.011339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:51.553939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.311230+0000","last_change":"2024-01-16T14:56:45.983804+0000","last_active":"2024-01-16T14:56:47.311230+0000","last_peered":"2024-01-16T14:56:47.311230+0000","last_clean":"2024-01-16T14:56:47.311230+0000","last_became_active":"2024-01-16T14:56:45.983529+0000","last_became_peered":"2024-01-16T14:56:45.983529+0000","last_unstale":"2024-01-16T14:56:47.311230+0000","last_undegraded":"2024-01-16T14:56:47.311230+0000","last_fullsized":"2024-01-16T14:56:47.311230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:20.055131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.014146+0000","last_change":"2024-01-16T14:56:45.005514+0000","last_active":"2024-01-16T14:56:46.014146+0000","last_peered":"2024-01-16T14:56:46.014146+0000","last_clean":"2024-01-16T14:56:46.014146+0000","last_became_active":"2024-01-16T14:56:45.004610+0000","last_became_peered":"2024-01-16T14:56:45.004610+0000","last_unstale":"2024-01-16T14:56:46.014146+0000","last_undegraded":"2024-01-16T14:56:46.014146+0000","last_fullsized":"2024-01-16T14:56:46.014146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:51.810316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T14:56:47.308980+0000","last_change":"2024-01-16T14:56:45.992067+0000","last_active":"2024-01-16T14:56:47.308980+0000","last_peered":"2024-01-16T14:56:47.308980+0000","last_clean":"2024-01-16T14:56:47.308980+0000","last_became_active":"2024-01-16T14:56:45.991826+0000","last_became_peered":"2024-01-16T14:56:45.991826+0000","last_unstale":"2024-01-16T14:56:47.308980+0000","last_undegraded":"2024-01-16T14:56:47.308980+0000","last_fullsized":"2024-01-16T14:56:47.308980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:44.958619+0000","last_clean_scrub_stamp":"2024-01-16T14:56:44.958619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:42.767105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-16T14:56:46.016889+0000","last_change":"2024-01-16T14:56:45.003619+0000","last_active":"2024-01-16T14:56:46.016889+0000","last_peered":"2024-01-16T14:56:46.016889+0000","last_clean":"2024-01-16T14:56:46.016889+0000","last_became_active":"2024-01-16T14:56:45.002981+0000","last_became_peered":"2024-01-16T14:56:45.002981+0000","last_unstale":"2024-01-16T14:56:46.016889+0000","last_undegraded":"2024-01-16T14:56:46.016889+0000","last_fullsized":"2024-01-16T14:56:46.016889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:43.945975+0000","last_clean_scrub_stamp":"2024-01-16T14:56:43.945975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:04.351721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.782}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.972}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1179999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.661}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":660897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:56:56.247 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T14:56:56.247 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T14:56:56.247 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T14:56:56.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T14:56:56.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.503+0000 7f7f1a200640 1 Processor -- start 2024-01-16T14:56:56.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.504+0000 7f7f1a200640 1 -- start start 2024-01-16T14:56:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f1a200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f140b3f30 0x7f7f140d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f1a200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140d0a10 con 0x7f7f14160920 2024-01-16T14:56:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f1a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140cf080 con 0x7f7f140d0100 2024-01-16T14:56:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f1a200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140c6be0 con 0x7f7f140b3f30 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f18e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7f14160920 0x7f7f140cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36630/0 (socket says 172.21.15.125:36630) 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f18e00640 1 -- 172.21.15.125:0/3908904382 learned_addr learned my addr 172.21.15.125:0/3908904382 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f19800640 1 --2- 172.21.15.125:0/3908904382 >> v2:172.21.15.125:3300/0 conn(0x7f7f140b3f30 0x7f7f140d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f19800640 1 -- 172.21.15.125:0/3908904382 >> v1:172.21.15.137:6789/0 conn(0x7f7f140d0100 legacy=0x7f7f140d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f19800640 1 -- 172.21.15.125:0/3908904382 >> v1:172.21.15.125:6789/0 conn(0x7f7f14160920 legacy=0x7f7f140cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.505+0000 7f7f19800640 1 -- 172.21.15.125:0/3908904382 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140ce7e0 con 0x7f7f140b3f30 2024-01-16T14:56:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.506+0000 7f7f19800640 1 --2- 172.21.15.125:0/3908904382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140b3f30 0x7f7f140d1650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7f040097a0 tx=0x7f7f0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f35b36180649db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f13400640 1 -- 172.21.15.125:0/3908904382 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04031830 con 0x7f7f140b3f30 2024-01-16T14:56:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f13400640 1 -- 172.21.15.125:0/3908904382 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f04031990 con 0x7f7f140b3f30 2024-01-16T14:56:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f13400640 1 -- 172.21.15.125:0/3908904382 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04031d20 con 0x7f7f140b3f30 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 -- 172.21.15.125:0/3908904382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140b3f30 msgr2=0x7f7f140d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 --2- 172.21.15.125:0/3908904382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140b3f30 0x7f7f140d1650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7f040097a0 tx=0x7f7f0402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 -- 172.21.15.125:0/3908904382 shutdown_connections 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 --2- 172.21.15.125:0/3908904382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140b3f30 0x7f7f140d1650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 -- 172.21.15.125:0/3908904382 >> 172.21.15.125:0/3908904382 conn(0x7f7f140d3090 msgr2=0x7f7f140d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 -- 172.21.15.125:0/3908904382 shutdown_connections 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.507+0000 7f7f1a200640 1 -- 172.21.15.125:0/3908904382 wait complete. 2024-01-16T14:56:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.509+0000 7f7f1a200640 1 Processor -- start 2024-01-16T14:56:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.509+0000 7f7f1a200640 1 -- start start 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 0x7f7f1414b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f140d0100 0x7f7f140c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140ca5f0 con 0x7f7f14160920 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140ca8d0 con 0x7f7f140b3f30 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f1a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cabb0 con 0x7f7f140d0100 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48116/0 (socket says 172.21.15.125:48116) 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 -- 172.21.15.125:0/2527785705 learned_addr learned my addr 172.21.15.125:0/2527785705 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f18e00640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 0x7f7f1414b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 -- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f140d0100 msgr2=0x7f7f140c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f140d0100 0x7f7f140c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.510+0000 7f7f19800640 1 -- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 msgr2=0x7f7f1414b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f19800640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 0x7f7f1414b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f19800640 1 -- 172.21.15.125:0/2527785705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f04009400 con 0x7f7f14160920 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f18e00640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 0x7f7f1414b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f19800640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f040098c0 tx=0x7f7f040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0403b9f0 con 0x7f7f14160920 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0403bb50 con 0x7f7f14160920 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f040453f0 con 0x7f7f14160920 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f140cae90 con 0x7f7f14160920 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.511+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f140c8fa0 con 0x7f7f14160920 2024-01-16T14:56:56.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.512+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f140cb3b0 con 0x7f7f14160920 2024-01-16T14:56:56.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.513+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f04045550 con 0x7f7f14160920 2024-01-16T14:56:56.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.514+0000 7f7f10c00640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 0x7f7eec06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.514+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f040abe50 con 0x7f7f14160920 2024-01-16T14:56:56.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.514+0000 7f7f18e00640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 0x7f7eec06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.515+0000 7f7f18e00640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 0x7f7eec06d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005880 tx=0x7f7f0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.518+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f140cb3b0 con 0x7f7f14160920 2024-01-16T14:56:56.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.748+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7f140cb3b0 con 0x7f7f14160920 2024-01-16T14:56:56.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.748+0000 7f7f10c00640 1 -- 172.21.15.125:0/2527785705 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7f140cb3b0 con 0x7f7f14160920 2024-01-16T14:56:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:56.731 INFO:teuthology.orchestra.run.smithi125.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T14:56:56.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 msgr2=0x7f7eec06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 0x7f7eec06d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005880 tx=0x7f7f0c009040 comp rx=0 tx=0).stop 2024-01-16T14:56:56.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 msgr2=0x7f7f1406bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f040098c0 tx=0x7f7f040046c0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 shutdown_connections 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7eec06b2f0 0x7f7eec06d7b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f14160920 0x7f7f1406bf00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f140d0100 0x7f7f140c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 --2- 172.21.15.125:0/2527785705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f140b3f30 0x7f7f1414b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 >> 172.21.15.125:0/2527785705 conn(0x7f7f140d3090 msgr2=0x7f7f1415e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.751+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 shutdown_connections 2024-01-16T14:56:56.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.752+0000 7f7f1a200640 1 -- 172.21.15.125:0/2527785705 wait complete. 2024-01-16T14:56:56.750 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T14:56:56.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:56.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.968+0000 7f7f24200640 1 Processor -- start 2024-01-16T14:56:56.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.968+0000 7f7f24200640 1 -- start start 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f24200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 0x7f7f1c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f24200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f24200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1c15c5f0 con 0x7f7f1c13df80 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f24200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1c06c470 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f24200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c06c750 con 0x7f7f1c13e360 2024-01-16T14:56:56.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60238/0 (socket says 172.21.15.125:60238) 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 -- 172.21.15.125:0/3498212579 learned_addr learned my addr 172.21.15.125:0/3498212579 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f20e00640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 0x7f7f1c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 -- 172.21.15.125:0/3498212579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 msgr2=0x7f7f1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 0x7f7f1c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 -- 172.21.15.125:0/3498212579 >> v1:172.21.15.137:6789/0 conn(0x7f7f1c13df80 legacy=0x7f7f1c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f22200640 1 -- 172.21.15.125:0/3498212579 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c06ca30 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.970+0000 7f7f20e00640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 0x7f7f1c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.971+0000 7f7f22200640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009870 tx=0x7f7f1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eaa2f0a734459a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f1be00640 1 -- 172.21.15.125:0/3498212579 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14031640 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f1be00640 1 -- 172.21.15.125:0/3498212579 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f140317a0 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f1be00640 1 -- 172.21.15.125:0/3498212579 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14031b30 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 -- 172.21.15.125:0/3498212579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c15f9c0 msgr2=0x7f7f1c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009870 tx=0x7f7f1402b8f0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 -- 172.21.15.125:0/3498212579 shutdown_connections 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c15f9c0 0x7f7f1c0cf7d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 --2- 172.21.15.125:0/3498212579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c13e360 0x7f7f1c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.972+0000 7f7f24200640 1 -- 172.21.15.125:0/3498212579 >> 172.21.15.125:0/3498212579 conn(0x7f7f1c06a800 msgr2=0x7f7f1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.973+0000 7f7f24200640 1 -- 172.21.15.125:0/3498212579 shutdown_connections 2024-01-16T14:56:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.973+0000 7f7f24200640 1 -- 172.21.15.125:0/3498212579 wait complete. 2024-01-16T14:56:56.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.974+0000 7f7f24200640 1 Processor -- start 2024-01-16T14:56:56.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.974+0000 7f7f24200640 1 -- start start 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f1c13e360 0x7f7f1c1f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c15f9c0 0x7f7f1c153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0cec90 con 0x7f7f1c13df80 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c134e40 con 0x7f7f1c13e360 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f24200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c1f1c10 con 0x7f7f1c15f9c0 2024-01-16T14:56:56.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f21800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f21800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60254/0 (socket says 172.21.15.125:60254) 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.975+0000 7f7f21800640 1 -- 172.21.15.125:0/2138953983 learned_addr learned my addr 172.21.15.125:0/2138953983 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f22200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c15f9c0 0x7f7f1c153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f20e00640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f1c13e360 0x7f7f1c1f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 -- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c15f9c0 msgr2=0x7f7f1c153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c15f9c0 0x7f7f1c153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 -- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f1c13e360 msgr2=0x7f7f1c1f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f1c13e360 0x7f7f1c1f16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 -- 172.21.15.125:0/2138953983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140094d0 con 0x7f7f1c13df80 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f21800640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ba80 tx=0x7f7f0c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00cc00 con 0x7f7f1c13df80 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c00cd60 con 0x7f7f1c13df80 2024-01-16T14:56:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c003bb0 con 0x7f7f1c13df80 2024-01-16T14:56:56.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.976+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c134880 con 0x7f7f1c13df80 2024-01-16T14:56:56.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.977+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c136590 con 0x7f7f1c13df80 2024-01-16T14:56:56.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.978+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c0675d0 con 0x7f7f1c13df80 2024-01-16T14:56:56.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.978+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f0c014440 con 0x7f7f1c13df80 2024-01-16T14:56:56.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.979+0000 7f7f19600640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 0x7f7ef406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:56.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.979+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f0c089a90 con 0x7f7f1c13df80 2024-01-16T14:56:56.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.980+0000 7f7f20e00640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 0x7f7ef406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:56.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.980+0000 7f7f20e00640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 0x7f7ef406d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c13a160 tx=0x7f7f04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:56.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:56.983+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f0c054860 con 0x7f7f1c13df80 2024-01-16T14:56:57.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.173+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f1c06df00 con 0x7f7f1c13df80 2024-01-16T14:56:57.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.175+0000 7f7f19600640 1 -- 172.21.15.125:0/2138953983 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f7f0c057e80 con 0x7f7f1c13df80 2024-01-16T14:56:57.157 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:57.157 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 msgr2=0x7f7ef406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 0x7f7ef406d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c13a160 tx=0x7f7f04008040 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 msgr2=0x7f7f1c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ba80 tx=0x7f7f0c00bf40 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 shutdown_connections 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f1c15f9c0 0x7f7f1c153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f1c13e360 0x7f7f1c1f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ef406b2c0 0x7f7ef406d780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 --2- 172.21.15.125:0/2138953983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f1c13df80 0x7f7f1c0ce630 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.177+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 >> 172.21.15.125:0/2138953983 conn(0x7f7f1c06a800 msgr2=0x7f7f1c15bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.178+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 shutdown_connections 2024-01-16T14:56:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.178+0000 7f7f24200640 1 -- 172.21.15.125:0/2138953983 wait complete. 2024-01-16T14:56:57.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:56:57.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:56:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.397+0000 7fd6e9600640 1 Processor -- start 2024-01-16T14:56:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.398+0000 7fd6e9600640 1 -- start start 2024-01-16T14:56:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e9600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6e40b2bd0 0x7fd6e40b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e4071190 con 0x7fd6e4150760 2024-01-16T14:56:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e9600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e4071470 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4071750 con 0x7fd6e415da10 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.399+0000 7fd6e3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6e40b2bd0 0x7fd6e40b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6e40b2bd0 0x7fd6e40b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60270/0 (socket says 172.21.15.125:60270) 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 -- 172.21.15.125:0/2445679631 learned_addr learned my addr 172.21.15.125:0/2445679631 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6d9e00640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 -- 172.21.15.125:0/2445679631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 msgr2=0x7fd6e4072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e4072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 -- 172.21.15.125:0/2445679631 >> v1:172.21.15.137:6789/0 conn(0x7fd6e4150760 legacy=0x7fd6e415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6e3400640 1 -- 172.21.15.125:0/2445679631 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e4071a30 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.400+0000 7fd6d9e00640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e4072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:57.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.401+0000 7fd6e3400640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40b3080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00bb00 tx=0x7fd6dc02dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888bb223748c005e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.401+0000 7fd6e2000640 1 -- 172.21.15.125:0/2445679631 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc02ec50 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.401+0000 7fd6e2000640 1 -- 172.21.15.125:0/2445679631 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc032070 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.401+0000 7fd6e2000640 1 -- 172.21.15.125:0/2445679631 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc038400 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 -- 172.21.15.125:0/2445679631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e40b2bd0 msgr2=0x7fd6e40b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40b3080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00bb00 tx=0x7fd6dc02dea0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 -- 172.21.15.125:0/2445679631 shutdown_connections 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40b3080 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 --2- 172.21.15.125:0/2445679631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e4072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 -- 172.21.15.125:0/2445679631 >> 172.21.15.125:0/2445679631 conn(0x7fd6e406a800 msgr2=0x7fd6e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 -- 172.21.15.125:0/2445679631 shutdown_connections 2024-01-16T14:56:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.402+0000 7fd6e9600640 1 -- 172.21.15.125:0/2445679631 wait complete. 2024-01-16T14:56:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.404+0000 7fd6e9600640 1 Processor -- start 2024-01-16T14:56:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.404+0000 7fd6e9600640 1 -- start start 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e41f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4155820 con 0x7fd6e4150760 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4144530 con 0x7fd6e40b2bd0 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e415a7b0 con 0x7fd6e415da10 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60272/0 (socket says 172.21.15.125:60272) 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 -- 172.21.15.125:0/1594815424 learned_addr learned my addr 172.21.15.125:0/1594815424 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 -- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 msgr2=0x7fd6e41f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e41f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 -- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e40b2bd0 msgr2=0x7fd6e40ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.405+0000 7fd6d9e00640 1 -- 172.21.15.125:0/1594815424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc00b760 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6d9e00640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c7f0 tx=0x7fd6cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc007820 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc007980 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e4155a90 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc007cf0 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.406+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e4156050 con 0x7fd6e4150760 2024-01-16T14:56:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.407+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e40675d0 con 0x7fd6e4150760 2024-01-16T14:56:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.408+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd6cc0113f0 con 0x7fd6e4150760 2024-01-16T14:56:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.409+0000 7fd6db400640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 0x7fd6bc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.409+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6cc089ec0 con 0x7fd6e4150760 2024-01-16T14:56:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.409+0000 7fd6e2a00640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 0x7fd6bc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.410+0000 7fd6e2a00640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 0x7fd6bc06d7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40b3490 tx=0x7fd6d4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.413+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6cc054d20 con 0x7fd6e4150760 2024-01-16T14:56:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.588+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6e415a4d0 con 0x7fd6e4150760 2024-01-16T14:56:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.590+0000 7fd6db400640 1 -- 172.21.15.125:0/1594815424 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6cc01f0a0 con 0x7fd6e4150760 2024-01-16T14:56:57.571 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 msgr2=0x7fd6bc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 0x7fd6bc06d7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40b3490 tx=0x7fd6d4007d30 comp rx=0 tx=0).stop 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 msgr2=0x7fd6e4159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c7f0 tx=0x7fd6cc00ccb0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 shutdown_connections 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e415da10 0x7fd6e41f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6bc06b2f0 0x7fd6bc06d7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6e4150760 0x7fd6e4159f90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 --2- 172.21.15.125:0/1594815424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e40b2bd0 0x7fd6e40ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 >> 172.21.15.125:0/1594815424 conn(0x7fd6e406a800 msgr2=0x7fd6e415ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:57.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 shutdown_connections 2024-01-16T14:56:57.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.592+0000 7fd6e9600640 1 -- 172.21.15.125:0/1594815424 wait complete. 2024-01-16T14:56:57.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:56:57.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.817+0000 7f03c2a00640 1 Processor -- start 2024-01-16T14:56:57.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.818+0000 7f03c2a00640 1 -- start start 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.819+0000 7f03c2a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03bc0ae5a0 0x7f03bc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.819+0000 7f03c2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03bc0cf290 0x7f03bc0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.819+0000 7f03c2a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc070c40 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.819+0000 7f03c2a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc071450 con 0x7f03bc0cf290 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.819+0000 7f03c2a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc0cfc60 con 0x7f03bc0ae5a0 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03bc0cf290 0x7f03bc0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03bc0cf290 0x7f03bc0cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60274/0 (socket says 172.21.15.125:60274) 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 -- 172.21.15.125:0/234782929 learned_addr learned my addr 172.21.15.125:0/234782929 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03bbe00640 1 --2- 172.21.15.125:0/234782929 >> v2:172.21.15.137:3300/0 conn(0x7f03bc0ae5a0 0x7f03bc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 -- 172.21.15.125:0/234782929 >> v2:172.21.15.137:3300/0 conn(0x7f03bc0ae5a0 msgr2=0x7f03bc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 --2- 172.21.15.125:0/234782929 >> v2:172.21.15.137:3300/0 conn(0x7f03bc0ae5a0 0x7f03bc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 -- 172.21.15.125:0/234782929 >> v1:172.21.15.125:6789/0 conn(0x7f03bc0b3ae0 legacy=0x7f03bc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.820+0000 7f03c0a00640 1 -- 172.21.15.125:0/234782929 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc06c470 con 0x7f03bc0cf290 2024-01-16T14:56:57.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.821+0000 7f03c0a00640 1 --2- 172.21.15.125:0/234782929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0cf290 0x7f03bc0cf720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0cfe20 tx=0x7f03b402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805c667f632aeceb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.821+0000 7f03baa00640 1 -- 172.21.15.125:0/234782929 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b402cbb0 con 0x7f03bc0cf290 2024-01-16T14:56:57.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.821+0000 7f03baa00640 1 -- 172.21.15.125:0/234782929 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b402cd10 con 0x7f03bc0cf290 2024-01-16T14:56:57.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.821+0000 7f03baa00640 1 -- 172.21.15.125:0/234782929 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4034610 con 0x7f03bc0cf290 2024-01-16T14:56:57.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 -- 172.21.15.125:0/234782929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0cf290 msgr2=0x7f03bc0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 --2- 172.21.15.125:0/234782929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0cf290 0x7f03bc0cf720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0cfe20 tx=0x7f03b402c010 comp rx=0 tx=0).stop 2024-01-16T14:56:57.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 -- 172.21.15.125:0/234782929 shutdown_connections 2024-01-16T14:56:57.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 --2- 172.21.15.125:0/234782929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0cf290 0x7f03bc0cf720 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 --2- 172.21.15.125:0/234782929 >> v2:172.21.15.137:3300/0 conn(0x7f03bc0ae5a0 0x7f03bc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 -- 172.21.15.125:0/234782929 >> 172.21.15.125:0/234782929 conn(0x7f03bc06a800 msgr2=0x7f03bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:57.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 -- 172.21.15.125:0/234782929 shutdown_connections 2024-01-16T14:56:57.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.822+0000 7f03c2a00640 1 -- 172.21.15.125:0/234782929 wait complete. 2024-01-16T14:56:57.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.824+0000 7f03c2a00640 1 Processor -- start 2024-01-16T14:56:57.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.824+0000 7f03c2a00640 1 -- start start 2024-01-16T14:56:57.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03bc0ae5a0 0x7f03bc14b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 0x7f03bc15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc158590 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0cbfc0 con 0x7f03bc0ae5a0 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03c2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc159830 con 0x7f03bc0cf290 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.825+0000 7f03bb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60280/0 (socket says 172.21.15.125:60280) 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 -- 172.21.15.125:0/1911409534 learned_addr learned my addr 172.21.15.125:0/1911409534 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bbe00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03bc0ae5a0 0x7f03bc14b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03c0a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 0x7f03bc15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 -- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 msgr2=0x7f03bc15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 0x7f03bc15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 -- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03bc0ae5a0 msgr2=0x7f03bc14b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:57.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03bc0ae5a0 0x7f03bc14b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 -- 172.21.15.125:0/1911409534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4009b30 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03c0a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 0x7f03bc15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03bb400640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c450 tx=0x7f03a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.826+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40052d0 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.827+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a4005430 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.827+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc159ab0 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.827+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4012870 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.827+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc159e40 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.828+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc0675d0 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.829+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f03a40129d0 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.829+0000 7f03b3e00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 0x7f039406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:57.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.830+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03a40896a0 con 0x7f03bc0b3ae0 2024-01-16T14:56:57.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.830+0000 7f03bbe00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 0x7f039406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:57.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.830+0000 7f03bbe00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 0x7f039406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03ac006fd0 tx=0x7f03ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:57.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:57.834+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03a4054580 con 0x7f03bc0b3ae0 2024-01-16T14:56:58.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.025+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f03bc15a410 con 0x7f03bc0b3ae0 2024-01-16T14:56:58.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.026+0000 7f03b3e00640 1 -- 172.21.15.125:0/1911409534 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f03a4057ba0 con 0x7f03bc0b3ae0 2024-01-16T14:56:58.008 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:56:58.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 msgr2=0x7f039406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 0x7f039406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03ac006fd0 tx=0x7f03ac008040 comp rx=0 tx=0).stop 2024-01-16T14:56:58.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 msgr2=0x7f03bc15c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c450 tx=0x7f03a400c910 comp rx=0 tx=0).stop 2024-01-16T14:56:58.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 shutdown_connections 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03bc0cf290 0x7f03bc15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f039406b2c0 0x7f039406d780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03bc0b3ae0 0x7f03bc15c720 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 --2- 172.21.15.125:0/1911409534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03bc0ae5a0 0x7f03bc14b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.028+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 >> 172.21.15.125:0/1911409534 conn(0x7f03bc06a800 msgr2=0x7f03bc0b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.029+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 shutdown_connections 2024-01-16T14:56:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.029+0000 7f03c2a00640 1 -- 172.21.15.125:0/1911409534 wait complete. 2024-01-16T14:56:58.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:56:58.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.244+0000 7efd60e00640 1 Processor -- start 2024-01-16T14:56:58.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.244+0000 7efd60e00640 1 -- start start 2024-01-16T14:56:58.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.245+0000 7efd60e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.245+0000 7efd60e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efd5c0b2bd0 0x7efd5c0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.245+0000 7efd60e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c071190 con 0x7efd5c150760 2024-01-16T14:56:58.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.245+0000 7efd60e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c071470 con 0x7efd5c0b2bd0 2024-01-16T14:56:58.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.245+0000 7efd60e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c071750 con 0x7efd5c15da10 2024-01-16T14:56:58.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.246+0000 7efd53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.246+0000 7efd53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:57776/0 (socket says 172.21.15.125:57776) 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.246+0000 7efd53e00640 1 -- 172.21.15.125:0/3832566507 learned_addr learned my addr 172.21.15.125:0/3832566507 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.246+0000 7efd5a800640 1 --2- 172.21.15.125:0/3832566507 >> v2:172.21.15.137:3300/0 conn(0x7efd5c0b2bd0 0x7efd5c0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd59400640 1 -- 172.21.15.125:0/3832566507 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839460414 0 0) 0x7efd5c071190 con 0x7efd5c150760 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd59400640 1 -- 172.21.15.125:0/3832566507 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c003520 con 0x7efd5c150760 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd53e00640 1 -- 172.21.15.125:0/3832566507 >> v2:172.21.15.137:3300/0 conn(0x7efd5c0b2bd0 msgr2=0x7efd5c0b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd53e00640 1 --2- 172.21.15.125:0/3832566507 >> v2:172.21.15.137:3300/0 conn(0x7efd5c0b2bd0 0x7efd5c0b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd53e00640 1 -- 172.21.15.125:0/3832566507 >> v1:172.21.15.137:6789/0 conn(0x7efd5c150760 legacy=0x7efd5c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.247+0000 7efd53e00640 1 -- 172.21.15.125:0/3832566507 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c071a30 con 0x7efd5c15da10 2024-01-16T14:56:58.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.248+0000 7efd53e00640 1 --2- 172.21.15.125:0/3832566507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd44009870 tx=0x7efd4402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9e6dab41ea905ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.248+0000 7efd59400640 1 -- 172.21.15.125:0/3832566507 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44030840 con 0x7efd5c15da10 2024-01-16T14:56:58.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.248+0000 7efd59400640 1 -- 172.21.15.125:0/3832566507 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd440309a0 con 0x7efd5c15da10 2024-01-16T14:56:58.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.248+0000 7efd59400640 1 -- 172.21.15.125:0/3832566507 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44030d10 con 0x7efd5c15da10 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 -- 172.21.15.125:0/3832566507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 msgr2=0x7efd5c072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 --2- 172.21.15.125:0/3832566507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd44009870 tx=0x7efd4402b980 comp rx=0 tx=0).stop 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 -- 172.21.15.125:0/3832566507 shutdown_connections 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 --2- 172.21.15.125:0/3832566507 >> v2:172.21.15.137:3300/0 conn(0x7efd5c0b2bd0 0x7efd5c0b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 --2- 172.21.15.125:0/3832566507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c15da10 0x7efd5c072230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 -- 172.21.15.125:0/3832566507 >> 172.21.15.125:0/3832566507 conn(0x7efd5c06a800 msgr2=0x7efd5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 -- 172.21.15.125:0/3832566507 shutdown_connections 2024-01-16T14:56:58.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.249+0000 7efd60e00640 1 -- 172.21.15.125:0/3832566507 wait complete. 2024-01-16T14:56:58.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.251+0000 7efd60e00640 1 Processor -- start 2024-01-16T14:56:58.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.251+0000 7efd60e00640 1 -- start start 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c0b2bd0 0x7efd5c14f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd5c150760 0x7efd5c1e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c1f0a50 con 0x7efd5c15da10 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0abff0 con 0x7efd5c150760 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd60e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0ac2d0 con 0x7efd5c0b2bd0 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60284/0 (socket says 172.21.15.125:60284) 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 -- 172.21.15.125:0/667406769 learned_addr learned my addr 172.21.15.125:0/667406769 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 -- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c0b2bd0 msgr2=0x7efd5c14f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c0b2bd0 0x7efd5c14f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 -- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd5c150760 msgr2=0x7efd5c1e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd5c150760 0x7efd5c1e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.252+0000 7efd5a800640 1 -- 172.21.15.125:0/667406769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd440094d0 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd5a800640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd5400efc0 tx=0x7efd5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54014480 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5400f040 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54014820 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c0ac5b0 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.253+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c0acb70 con 0x7efd5c15da10 2024-01-16T14:56:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.254+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c0675d0 con 0x7efd5c15da10 2024-01-16T14:56:58.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.255+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efd54014980 con 0x7efd5c15da10 2024-01-16T14:56:58.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.256+0000 7efd52000640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 0x7efd3406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.256+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd5408a6a0 con 0x7efd5c15da10 2024-01-16T14:56:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.256+0000 7efd59e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 0x7efd3406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.257+0000 7efd59e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 0x7efd3406d7b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0011e0 tx=0x7efd4c0078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.260+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd540555a0 con 0x7efd5c15da10 2024-01-16T14:56:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.448+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd5c0acd30 con 0x7efd5c15da10 2024-01-16T14:56:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.448+0000 7efd52000640 1 -- 172.21.15.125:0/667406769 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd54018020 con 0x7efd5c15da10 2024-01-16T14:56:58.430 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:56:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 msgr2=0x7efd3406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 0x7efd3406d7b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0011e0 tx=0x7efd4c0078f0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 msgr2=0x7efd5c0a8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd5400efc0 tx=0x7efd5400c490 comp rx=0 tx=0).stop 2024-01-16T14:56:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 shutdown_connections 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efd3406b2f0 0x7efd3406d7b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd5c15da10 0x7efd5c0a8fe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd5c150760 0x7efd5c1e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 --2- 172.21.15.125:0/667406769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd5c0b2bd0 0x7efd5c14f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 >> 172.21.15.125:0/667406769 conn(0x7efd5c06a800 msgr2=0x7efd5c15c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 shutdown_connections 2024-01-16T14:56:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.451+0000 7efd60e00640 1 -- 172.21.15.125:0/667406769 wait complete. 2024-01-16T14:56:58.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:56:58.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.663+0000 7f3f37e00640 1 Processor -- start 2024-01-16T14:56:58.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.663+0000 7f3f37e00640 1 -- start start 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.664+0000 7f3f37e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f300cf9a0 0x7f3f3015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.664+0000 7f3f37e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f3015e1e0 0x7f3f300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.664+0000 7f3f37e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300b1ee0 con 0x7f3f30143ae0 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.664+0000 7f3f37e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f300b2780 con 0x7f3f300cf9a0 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.664+0000 7f3f37e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3015e990 con 0x7f3f3015e1e0 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f300cf9a0 0x7f3f3015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f300cf9a0 0x7f3f3015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60292/0 (socket says 172.21.15.125:60292) 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 -- 172.21.15.125:0/1522353141 learned_addr learned my addr 172.21.15.125:0/1522353141 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f35400640 1 --1- 172.21.15.125:0/1522353141 >> v1:172.21.15.125:6789/0 conn(0x7f3f30143ae0 0x7f3f300cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45554/0 (socket says 172.21.15.125:45554) 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f35e00640 1 --2- 172.21.15.125:0/1522353141 >> v2:172.21.15.137:3300/0 conn(0x7f3f3015e1e0 0x7f3f300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 -- 172.21.15.125:0/1522353141 >> v2:172.21.15.137:3300/0 conn(0x7f3f3015e1e0 msgr2=0x7f3f300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:58.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 --2- 172.21.15.125:0/1522353141 >> v2:172.21.15.137:3300/0 conn(0x7f3f3015e1e0 0x7f3f300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 -- 172.21.15.125:0/1522353141 >> v1:172.21.15.125:6789/0 conn(0x7f3f30143ae0 legacy=0x7f3f300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f2fe00640 1 -- 172.21.15.125:0/1522353141 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180008558 0 0) 0x7f3f300b1ee0 con 0x7f3f30143ae0 2024-01-16T14:56:58.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.665+0000 7f3f34a00640 1 -- 172.21.15.125:0/1522353141 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f300b2130 con 0x7f3f300cf9a0 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.666+0000 7f3f34a00640 1 --2- 172.21.15.125:0/1522353141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f3015dca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3f300ae9d0 tx=0x7f3f1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31721ce3afc57a47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.666+0000 7f3f2fe00640 1 -- 172.21.15.125:0/1522353141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18030860 con 0x7f3f300cf9a0 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.666+0000 7f3f2fe00640 1 -- 172.21.15.125:0/1522353141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f180309c0 con 0x7f3f300cf9a0 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.666+0000 7f3f2fe00640 1 -- 172.21.15.125:0/1522353141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18030d50 con 0x7f3f300cf9a0 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 -- 172.21.15.125:0/1522353141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 msgr2=0x7f3f3015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 --2- 172.21.15.125:0/1522353141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f3015dca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3f300ae9d0 tx=0x7f3f1802b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 -- 172.21.15.125:0/1522353141 shutdown_connections 2024-01-16T14:56:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 --2- 172.21.15.125:0/1522353141 >> v2:172.21.15.137:3300/0 conn(0x7f3f3015e1e0 0x7f3f300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 --2- 172.21.15.125:0/1522353141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f3015dca0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 -- 172.21.15.125:0/1522353141 >> 172.21.15.125:0/1522353141 conn(0x7f3f3006a800 msgr2=0x7f3f3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 -- 172.21.15.125:0/1522353141 shutdown_connections 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.667+0000 7f3f37e00640 1 -- 172.21.15.125:0/1522353141 wait complete. 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.669+0000 7f3f37e00640 1 Processor -- start 2024-01-16T14:56:58.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.669+0000 7f3f37e00640 1 -- start start 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f30143ae0 0x7f3f300ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3015e1e0 0x7f3f300b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300c4b70 con 0x7f3f300cf9a0 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301f2960 con 0x7f3f30143ae0 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f37e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300c3ef0 con 0x7f3f3015e1e0 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60298/0 (socket says 172.21.15.125:60298) 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 -- 172.21.15.125:0/3787176082 learned_addr learned my addr 172.21.15.125:0/3787176082 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 -- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3015e1e0 msgr2=0x7f3f300b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3015e1e0 0x7f3f300b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 -- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f30143ae0 msgr2=0x7f3f300ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f30143ae0 0x7f3f300ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.670+0000 7f3f35400640 1 -- 172.21.15.125:0/3787176082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180094d0 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f35400640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000c4d0 tx=0x7f3f2000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f200074e0 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f20007640 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f20004a20 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f300c41d0 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.671+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f300c4790 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.672+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f300675d0 con 0x7f3f300cf9a0 2024-01-16T14:56:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.673+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3f20004b80 con 0x7f3f300cf9a0 2024-01-16T14:56:58.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.674+0000 7f3f2d600640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 0x7f3f0806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:58.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.674+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f20088e40 con 0x7f3f300cf9a0 2024-01-16T14:56:58.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.674+0000 7f3f34a00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 0x7f3f0806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:58.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.674+0000 7f3f34a00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 0x7f3f0806d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f18036000 tx=0x7f3f1803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:58.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.678+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f20053c10 con 0x7f3f300cf9a0 2024-01-16T14:56:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.906+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f3015dd10 con 0x7f3f300cf9a0 2024-01-16T14:56:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.907+0000 7f3f2d600640 1 -- 172.21.15.125:0/3787176082 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f3f20018020 con 0x7f3f300cf9a0 2024-01-16T14:56:58.889 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:58.889 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:56:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 msgr2=0x7f3f0806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 0x7f3f0806d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f18036000 tx=0x7f3f1803b040 comp rx=0 tx=0).stop 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 msgr2=0x7f3f301f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000c4d0 tx=0x7f3f2000c990 comp rx=0 tx=0).stop 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 shutdown_connections 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3015e1e0 0x7f3f300b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f30143ae0 0x7f3f300ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f0806b2c0 0x7f3f0806d780 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 --2- 172.21.15.125:0/3787176082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f300cf9a0 0x7f3f301f2420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 >> 172.21.15.125:0/3787176082 conn(0x7f3f3006a800 msgr2=0x7f3f3006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 shutdown_connections 2024-01-16T14:56:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:58.910+0000 7f3f37e00640 1 -- 172.21.15.125:0/3787176082 wait complete. 2024-01-16T14:56:58.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:59.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.137+0000 7fce26e00640 1 Processor -- start 2024-01-16T14:56:59.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.137+0000 7fce26e00640 1 -- start start 2024-01-16T14:56:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.138+0000 7fce26e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.138+0000 7fce26e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200cfc30 con 0x7fce2015e3b0 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.138+0000 7fce26e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b1ee0 con 0x7fce200cf7d0 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.138+0000 7fce26e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200b2780 con 0x7fce20143ae0 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.138+0000 7fce1fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.139+0000 7fce1fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60302/0 (socket says 172.21.15.125:60302) 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.139+0000 7fce1fe00640 1 -- 172.21.15.125:0/682478424 learned_addr learned my addr 172.21.15.125:0/682478424 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.139+0000 7fce1fe00640 1 -- 172.21.15.125:0/682478424 >> v1:172.21.15.137:6789/0 conn(0x7fce200cf7d0 legacy=0x7fce2015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.139+0000 7fce1fe00640 1 -- 172.21.15.125:0/682478424 >> v1:172.21.15.125:6789/0 conn(0x7fce2015e3b0 legacy=0x7fce200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.139+0000 7fce1fe00640 1 -- 172.21.15.125:0/682478424 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2015e990 con 0x7fce20143ae0 2024-01-16T14:56:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.140+0000 7fce1fe00640 1 --2- 172.21.15.125:0/682478424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce200cf290 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce10009870 tx=0x7fce1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a9c8736ca9c71c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.140+0000 7fce1f400640 1 -- 172.21.15.125:0/682478424 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10030860 con 0x7fce20143ae0 2024-01-16T14:56:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.140+0000 7fce1f400640 1 -- 172.21.15.125:0/682478424 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce100309c0 con 0x7fce20143ae0 2024-01-16T14:56:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.140+0000 7fce1f400640 1 -- 172.21.15.125:0/682478424 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10030d50 con 0x7fce20143ae0 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 -- 172.21.15.125:0/682478424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 msgr2=0x7fce200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 --2- 172.21.15.125:0/682478424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce200cf290 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce10009870 tx=0x7fce1002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 -- 172.21.15.125:0/682478424 shutdown_connections 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 --2- 172.21.15.125:0/682478424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 -- 172.21.15.125:0/682478424 >> 172.21.15.125:0/682478424 conn(0x7fce2006a800 msgr2=0x7fce2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 -- 172.21.15.125:0/682478424 shutdown_connections 2024-01-16T14:56:59.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.141+0000 7fce26e00640 1 -- 172.21.15.125:0/682478424 wait complete. 2024-01-16T14:56:59.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.143+0000 7fce26e00640 1 Processor -- start 2024-01-16T14:56:59.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.143+0000 7fce26e00640 1 -- start start 2024-01-16T14:56:59.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 0x7fce200cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 0x7fce200a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20159c40 con 0x7fce200cf7d0 2024-01-16T14:56:59.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200ccd50 con 0x7fce20143ae0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce26e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200cd030 con 0x7fce2015e3b0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce1fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce1fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60316/0 (socket says 172.21.15.125:60316) 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce1fe00640 1 -- 172.21.15.125:0/3655174313 learned_addr learned my addr 172.21.15.125:0/3655174313 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce17200640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 0x7fce200cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce24e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 0x7fce200a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.144+0000 7fce1fe00640 1 -- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 msgr2=0x7fce200a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1fe00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 0x7fce200a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1fe00640 1 -- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 msgr2=0x7fce200cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1fe00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 0x7fce200cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1fe00640 1 -- 172.21.15.125:0/3655174313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce100094d0 con 0x7fce200cf7d0 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce17200640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 0x7fce200cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:59.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce24e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 0x7fce200a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1fe00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce10009870 tx=0x7fce10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10030810 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce10043400 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce200cd310 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce100437b0 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.145+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce200cd8d0 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.146+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce200675d0 con 0x7fce200cf7d0 2024-01-16T14:56:59.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.148+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fce10043910 con 0x7fce200cf7d0 2024-01-16T14:56:59.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.149+0000 7fce1cc00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 0x7fcdf806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.149+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce1003e070 con 0x7fce200cf7d0 2024-01-16T14:56:59.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.149+0000 7fce17200640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 0x7fcdf806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.150+0000 7fce17200640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 0x7fcdf806d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce20142a80 tx=0x7fce08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.153+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce10077ef0 con 0x7fce200cf7d0 2024-01-16T14:56:59.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.349+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce2015d4d0 con 0x7fce200cf7d0 2024-01-16T14:56:59.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.351+0000 7fce1cc00640 1 -- 172.21.15.125:0/3655174313 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7fce100aa0c0 con 0x7fce200cf7d0 2024-01-16T14:56:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:56:59.334 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:59.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 msgr2=0x7fcdf806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 0x7fcdf806d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce20142a80 tx=0x7fce08009040 comp rx=0 tx=0).stop 2024-01-16T14:56:59.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 msgr2=0x7fce200cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce10009870 tx=0x7fce10004730 comp rx=0 tx=0).stop 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 shutdown_connections 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce2015e3b0 0x7fce200a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce20143ae0 0x7fce200cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806b2c0 0x7fcdf806d780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 --2- 172.21.15.125:0/3655174313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce200cf7d0 0x7fce200cddf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.354+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 >> 172.21.15.125:0/3655174313 conn(0x7fce2006a800 msgr2=0x7fce20142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.355+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 shutdown_connections 2024-01-16T14:56:59.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.355+0000 7fce26e00640 1 -- 172.21.15.125:0/3655174313 wait complete. 2024-01-16T14:56:59.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:56:59.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:56:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.583+0000 7f42e3a00640 1 Processor -- start 2024-01-16T14:56:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.583+0000 7f42e3a00640 1 -- start start 2024-01-16T14:56:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e3a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42dc15f9c0 0x7f42dc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e3a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc15c5f0 con 0x7f42dc13df80 2024-01-16T14:56:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e3a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42dc06c470 con 0x7f42dc15f9c0 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e3a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc06c750 con 0x7f42dc13e360 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e1a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42dc15f9c0 0x7f42dc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e1000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f42dc13df80 0x7f42dc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45560/0 (socket says 172.21.15.125:45560) 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42e1000640 1 -- 172.21.15.125:0/166181898 learned_addr learned my addr 172.21.15.125:0/166181898 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.585+0000 7f42dbe00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.586+0000 7f42e1a00640 1 -- 172.21.15.125:0/166181898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 msgr2=0x7f42dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.586+0000 7f42e1a00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.586+0000 7f42e1a00640 1 -- 172.21.15.125:0/166181898 >> v1:172.21.15.125:6789/0 conn(0x7f42dc13df80 legacy=0x7f42dc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.586+0000 7f42e1a00640 1 -- 172.21.15.125:0/166181898 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc06ca30 con 0x7f42dc15f9c0 2024-01-16T14:56:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.586+0000 7f42db400640 1 -- 172.21.15.125:0/166181898 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036416445 0 0) 0x7f42dc15c5f0 con 0x7f42dc13df80 2024-01-16T14:56:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42e1a00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc0cf7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a606f19c0ba76296 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42db400640 1 -- 172.21.15.125:0/166181898 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d402cc50 con 0x7f42dc15f9c0 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42db400640 1 -- 172.21.15.125:0/166181898 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d402cdb0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42db400640 1 -- 172.21.15.125:0/166181898 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4034660 con 0x7f42dc15f9c0 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42e3a00640 1 -- 172.21.15.125:0/166181898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 msgr2=0x7f42dc0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.587+0000 7f42e3a00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc0cf7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d402bea0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 -- 172.21.15.125:0/166181898 shutdown_connections 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc0cf7d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 --2- 172.21.15.125:0/166181898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 -- 172.21.15.125:0/166181898 >> 172.21.15.125:0/166181898 conn(0x7f42dc06a800 msgr2=0x7f42dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 -- 172.21.15.125:0/166181898 shutdown_connections 2024-01-16T14:56:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.588+0000 7f42e3a00640 1 -- 172.21.15.125:0/166181898 wait complete. 2024-01-16T14:56:59.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.589+0000 7f42e3a00640 1 Processor -- start 2024-01-16T14:56:59.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.589+0000 7f42e3a00640 1 -- start start 2024-01-16T14:56:59.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 0x7f42dc1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc1f4c20 con 0x7f42dc15f9c0 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc1580b0 con 0x7f42dc13df80 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.590+0000 7f42e3a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc158390 con 0x7f42dc13e360 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60338/0 (socket says 172.21.15.125:60338) 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 -- 172.21.15.125:0/2297939587 learned_addr learned my addr 172.21.15.125:0/2297939587 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1000640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 0x7f42dc1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42dbe00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 -- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 msgr2=0x7f42dc152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 -- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 msgr2=0x7f42dc1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 0x7f42dc1f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 -- 172.21.15.125:0/2297939587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4009a70 con 0x7f42dc15f9c0 2024-01-16T14:56:59.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1000640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 0x7f42dc1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.591+0000 7f42e1a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.592+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4040650 con 0x7f42dc15f9c0 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.592+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d40407d0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.592+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4040b80 con 0x7f42dc15f9c0 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.592+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc158670 con 0x7f42dc15f9c0 2024-01-16T14:56:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.592+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc158c30 con 0x7f42dc15f9c0 2024-01-16T14:56:59.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.593+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc158e20 con 0x7f42dc15f9c0 2024-01-16T14:56:59.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.594+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f42d4040ce0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.595+0000 7f42d8c00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 0x7f42b406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.595+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42d40aca30 con 0x7f42dc15f9c0 2024-01-16T14:56:59.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.595+0000 7f42e1000640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 0x7f42b406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.596+0000 7f42e1000640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 0x7f42b406d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000f60 tx=0x7f42cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.599+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42d40778a0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.786+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42dc0675d0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.787+0000 7f42d8c00640 1 -- 172.21.15.125:0/2297939587 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42d407aec0 con 0x7f42dc15f9c0 2024-01-16T14:56:59.769 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.789+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 msgr2=0x7f42b406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.789+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 0x7f42b406d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000f60 tx=0x7f42cc009040 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.789+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 msgr2=0x7f42dc153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.789+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d4004a40 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 shutdown_connections 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42b406b2c0 0x7f42b406d780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42dc15f9c0 0x7f42dc153ce0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42dc13e360 0x7f42dc152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 --2- 172.21.15.125:0/2297939587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42dc13df80 0x7f42dc1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 >> 172.21.15.125:0/2297939587 conn(0x7f42dc06a800 msgr2=0x7f42dc0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 shutdown_connections 2024-01-16T14:56:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:56:59.790+0000 7f42e3a00640 1 -- 172.21.15.125:0/2297939587 wait complete. 2024-01-16T14:56:59.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:00.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.018+0000 7f2482200640 1 Processor -- start 2024-01-16T14:57:00.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.018+0000 7f2482200640 1 -- start start 2024-01-16T14:57:00.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.019+0000 7f2482200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247c0b2e00 0x7f247c070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.019+0000 7f2482200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f247c06cb10 0x7f247c06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.019+0000 7f2482200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0706a0 con 0x7f247c06c730 2024-01-16T14:57:00.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.019+0000 7f2482200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c0b3290 con 0x7f247c0b2e00 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.019+0000 7f2482200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c0708d0 con 0x7f247c06cb10 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247c0b2e00 0x7f247c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247c0b2e00 0x7f247c070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60354/0 (socket says 172.21.15.125:60354) 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 -- 172.21.15.125:0/1819187644 learned_addr learned my addr 172.21.15.125:0/1819187644 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2481800640 1 --2- 172.21.15.125:0/1819187644 >> v2:172.21.15.137:3300/0 conn(0x7f247c06cb10 0x7f247c06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 -- 172.21.15.125:0/1819187644 >> v2:172.21.15.137:3300/0 conn(0x7f247c06cb10 msgr2=0x7f247c06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 --2- 172.21.15.125:0/1819187644 >> v2:172.21.15.137:3300/0 conn(0x7f247c06cb10 0x7f247c06cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.020+0000 7f2473c00640 1 -- 172.21.15.125:0/1819187644 >> v1:172.21.15.137:6789/0 conn(0x7f247c06c730 legacy=0x7f247c071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.021+0000 7f2473c00640 1 -- 172.21.15.125:0/1819187644 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c070bb0 con 0x7f247c0b2e00 2024-01-16T14:57:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.022+0000 7f2473c00640 1 --2- 172.21.15.125:0/1819187644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c0b2e00 0x7f247c070160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f247c06e530 tx=0x7f246402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6453fceffaf1d2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.022+0000 7f247be00640 1 -- 172.21.15.125:0/1819187644 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464030860 con 0x7f247c0b2e00 2024-01-16T14:57:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.022+0000 7f247be00640 1 -- 172.21.15.125:0/1819187644 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24640309c0 con 0x7f247c0b2e00 2024-01-16T14:57:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.022+0000 7f247be00640 1 -- 172.21.15.125:0/1819187644 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464030d50 con 0x7f247c0b2e00 2024-01-16T14:57:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 -- 172.21.15.125:0/1819187644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c0b2e00 msgr2=0x7f247c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 --2- 172.21.15.125:0/1819187644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c0b2e00 0x7f247c070160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f247c06e530 tx=0x7f246402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 -- 172.21.15.125:0/1819187644 shutdown_connections 2024-01-16T14:57:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 --2- 172.21.15.125:0/1819187644 >> v2:172.21.15.137:3300/0 conn(0x7f247c06cb10 0x7f247c06cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 --2- 172.21.15.125:0/1819187644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c0b2e00 0x7f247c070160 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 -- 172.21.15.125:0/1819187644 >> 172.21.15.125:0/1819187644 conn(0x7f247c06a7d0 msgr2=0x7f247c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 -- 172.21.15.125:0/1819187644 shutdown_connections 2024-01-16T14:57:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.023+0000 7f2482200640 1 -- 172.21.15.125:0/1819187644 wait complete. 2024-01-16T14:57:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.025+0000 7f2482200640 1 Processor -- start 2024-01-16T14:57:00.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.025+0000 7f2482200640 1 -- start start 2024-01-16T14:57:00.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247c06c730 0x7f247c1468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247c0b2e00 0x7f247c0adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c15cac0 con 0x7f247c06cb10 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c145560 con 0x7f247c06c730 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2482200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c145840 con 0x7f247c0b2e00 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60360/0 (socket says 172.21.15.125:60360) 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 -- 172.21.15.125:0/3459655082 learned_addr learned my addr 172.21.15.125:0/3459655082 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 -- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247c0b2e00 msgr2=0x7f247c0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247c0b2e00 0x7f247c0adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 -- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247c06c730 msgr2=0x7f247c1468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.026+0000 7f2473c00640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247c06c730 0x7f247c1468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2473c00640 1 -- 172.21.15.125:0/3459655082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640094d0 con 0x7f247c06cb10 2024-01-16T14:57:00.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2473c00640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f246402b9e0 tx=0x7f24640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246403e070 con 0x7f247c06cb10 2024-01-16T14:57:00.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464039a60 con 0x7f247c06cb10 2024-01-16T14:57:00.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464039df0 con 0x7f247c06cb10 2024-01-16T14:57:00.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c145b20 con 0x7f247c06cb10 2024-01-16T14:57:00.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.027+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c1460e0 con 0x7f247c06cb10 2024-01-16T14:57:00.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.028+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c067450 con 0x7f247c06cb10 2024-01-16T14:57:00.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.029+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f246404b400 con 0x7f247c06cb10 2024-01-16T14:57:00.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.030+0000 7f2479600640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 0x7f245406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.030+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24640ac310 con 0x7f247c06cb10 2024-01-16T14:57:00.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.030+0000 7f2480e00640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 0x7f245406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.031+0000 7f2480e00640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 0x7f245406d7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f246c000f60 tx=0x7f246c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.035+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2464077210 con 0x7f247c06cb10 2024-01-16T14:57:00.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.229+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f247c0b07c0 con 0x7f247c06cb10 2024-01-16T14:57:00.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.230+0000 7f2479600640 1 -- 172.21.15.125:0/3459655082 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f246407a830 con 0x7f247c06cb10 2024-01-16T14:57:00.212 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:00.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 msgr2=0x7f245406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 0x7f245406d7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f246c000f60 tx=0x7f246c0079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 msgr2=0x7f247c148190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f246402b9e0 tx=0x7f24640046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 shutdown_connections 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247c0b2e00 0x7f247c0adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f245406b310 0x7f245406d7d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247c06cb10 0x7f247c148190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 --2- 172.21.15.125:0/3459655082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247c06c730 0x7f247c1468e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 >> 172.21.15.125:0/3459655082 conn(0x7f247c06a7d0 msgr2=0x7f247c0aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 shutdown_connections 2024-01-16T14:57:00.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.233+0000 7f2482200640 1 -- 172.21.15.125:0/3459655082 wait complete. 2024-01-16T14:57:00.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.465+0000 7f4362800640 1 Processor -- start 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.465+0000 7f4362800640 1 -- start start 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13e360 0x7f435c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f435c15f9c0 0x7f435c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c06c470 con 0x7f435c15f9c0 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c06c750 con 0x7f435c13df80 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4362800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06ca30 con 0x7f435c13e360 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f435be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f435c15f9c0 0x7f435c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f435be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f435c15f9c0 0x7f435c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60376/0 (socket says 172.21.15.125:60376) 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f435be00640 1 -- 172.21.15.125:0/342002237 learned_addr learned my addr 172.21.15.125:0/342002237 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f435b400640 1 --2- 172.21.15.125:0/342002237 >> v2:172.21.15.137:3300/0 conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.467+0000 7f4353e00640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13e360 0x7f435c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.468+0000 7f435be00640 1 -- 172.21.15.125:0/342002237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13e360 msgr2=0x7f435c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.468+0000 7f435be00640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13e360 0x7f435c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.468+0000 7f435be00640 1 -- 172.21.15.125:0/342002237 >> v2:172.21.15.137:3300/0 conn(0x7f435c13df80 msgr2=0x7f435c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.468+0000 7f435be00640 1 --2- 172.21.15.125:0/342002237 >> v2:172.21.15.137:3300/0 conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.468+0000 7f435be00640 1 -- 172.21.15.125:0/342002237 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c06cd10 con 0x7f435c15f9c0 2024-01-16T14:57:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.469+0000 7f435be00640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c15f9c0 0x7f435c0cf7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f435c072770 tx=0x7f435402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b2bfe0e5dcfca5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.469+0000 7f435aa00640 1 -- 172.21.15.125:0/342002237 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435402ca70 con 0x7f435c15f9c0 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.469+0000 7f435aa00640 1 -- 172.21.15.125:0/342002237 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435402cbd0 con 0x7f435c15f9c0 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.469+0000 7f435aa00640 1 -- 172.21.15.125:0/342002237 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540343f0 con 0x7f435c15f9c0 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 -- 172.21.15.125:0/342002237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c15f9c0 msgr2=0x7f435c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c15f9c0 0x7f435c0cf7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f435c072770 tx=0x7f435402bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 -- 172.21.15.125:0/342002237 shutdown_connections 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c15f9c0 0x7f435c0cf7d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 --2- 172.21.15.125:0/342002237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13e360 0x7f435c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 --2- 172.21.15.125:0/342002237 >> v2:172.21.15.137:3300/0 conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 -- 172.21.15.125:0/342002237 >> 172.21.15.125:0/342002237 conn(0x7f435c06a800 msgr2=0x7f435c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 -- 172.21.15.125:0/342002237 shutdown_connections 2024-01-16T14:57:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.470+0000 7f4362800640 1 -- 172.21.15.125:0/342002237 wait complete. 2024-01-16T14:57:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.472+0000 7f4362800640 1 Processor -- start 2024-01-16T14:57:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.472+0000 7f4362800640 1 -- start start 2024-01-16T14:57:00.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13df80 0x7f435c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f435c15f9c0 0x7f435c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f2960 con 0x7f435c13e360 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c154b20 con 0x7f435c15f9c0 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4362800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1596a0 con 0x7f435c13df80 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.473+0000 7f4353e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60386/0 (socket says 172.21.15.125:60386) 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 -- 172.21.15.125:0/1935597037 learned_addr learned my addr 172.21.15.125:0/1935597037 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f435b400640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13df80 0x7f435c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f435be00640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f435c15f9c0 0x7f435c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 -- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13df80 msgr2=0x7f435c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13df80 0x7f435c13ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 -- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f435c15f9c0 msgr2=0x7f435c0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f435c15f9c0 0x7f435c0b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 -- 172.21.15.125:0/1935597037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354009b30 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353e00640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f434400d830 tx=0x7f434400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434400b6c0 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434400b820 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434400bbb0 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.474+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c1f1fc0 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.475+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c1f2580 con 0x7f435c13e360 2024-01-16T14:57:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.476+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0675d0 con 0x7f435c13e360 2024-01-16T14:57:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.477+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f434401e460 con 0x7f435c13e360 2024-01-16T14:57:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.478+0000 7f4353400640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 0x7f433406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.478+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f434408a230 con 0x7f435c13e360 2024-01-16T14:57:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.478+0000 7f435b400640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 0x7f433406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.479+0000 7f435b400640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 0x7f433406d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f434c006fd0 tx=0x7f434c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.482+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4344055130 con 0x7f435c13e360 2024-01-16T14:57:00.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.670+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f435c0ae300 con 0x7f435c13e360 2024-01-16T14:57:00.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.670+0000 7f4353400640 1 -- 172.21.15.125:0/1935597037 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4344058750 con 0x7f435c13e360 2024-01-16T14:57:00.652 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 msgr2=0x7f433406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 0x7f433406d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f434c006fd0 tx=0x7f434c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 msgr2=0x7f435c0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f434400d830 tx=0x7f434400dcf0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 shutdown_connections 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f435c15f9c0 0x7f435c0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f433406b2c0 0x7f433406d780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f435c13e360 0x7f435c0b39b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 --2- 172.21.15.125:0/1935597037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f435c13df80 0x7f435c13ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 >> 172.21.15.125:0/1935597037 conn(0x7f435c06a800 msgr2=0x7f435c13c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 shutdown_connections 2024-01-16T14:57:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.673+0000 7f4362800640 1 -- 172.21.15.125:0/1935597037 wait complete. 2024-01-16T14:57:00.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:00.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.899+0000 7f451f000640 1 Processor -- start 2024-01-16T14:57:00.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.899+0000 7f451f000640 1 -- start start 2024-01-16T14:57:00.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451f000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f45180ae5a0 0x7f451815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180cf290 0x7f45180cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451f000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4518070c40 con 0x7f45180b3ae0 2024-01-16T14:57:00.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451f000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4518071450 con 0x7f45180ae5a0 2024-01-16T14:57:00.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45180cfc60 con 0x7f45180cf290 2024-01-16T14:57:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f451d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180cf290 0x7f45180cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f450fe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f45180b3ae0 0x7f451806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:57156/0 (socket says 172.21.15.125:57156) 2024-01-16T14:57:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.901+0000 7f450fe00640 1 -- 172.21.15.125:0/1455346542 learned_addr learned my addr 172.21.15.125:0/1455346542 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517e00640 1 --2- 172.21.15.125:0/1455346542 >> v2:172.21.15.137:3300/0 conn(0x7f45180ae5a0 0x7f451815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517400640 1 -- 172.21.15.125:0/1455346542 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046484985 0 0) 0x7f4518070c40 con 0x7f45180b3ae0 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517400640 1 -- 172.21.15.125:0/1455346542 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8003520 con 0x7f45180b3ae0 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517e00640 1 -- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180cf290 msgr2=0x7f45180cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517e00640 1 --2- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180cf290 0x7f45180cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517e00640 1 -- 172.21.15.125:0/1455346542 >> v1:172.21.15.137:6789/0 conn(0x7f45180b3ae0 legacy=0x7f451806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.902+0000 7f4517e00640 1 -- 172.21.15.125:0/1455346542 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451806c470 con 0x7f45180ae5a0 2024-01-16T14:57:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.904+0000 7f4517e00640 1 --2- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180ae5a0 0x7f451815e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45080097a0 tx=0x7f450802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc67eb11e3579bc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.905+0000 7f4517400640 1 -- 172.21.15.125:0/1455346542 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508030610 con 0x7f45180ae5a0 2024-01-16T14:57:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.905+0000 7f4517400640 1 -- 172.21.15.125:0/1455346542 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508030770 con 0x7f45180ae5a0 2024-01-16T14:57:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.905+0000 7f4517400640 1 -- 172.21.15.125:0/1455346542 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508030b00 con 0x7f45180ae5a0 2024-01-16T14:57:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.905+0000 7f451f000640 1 -- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180ae5a0 msgr2=0x7f451815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.905+0000 7f451f000640 1 --2- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180ae5a0 0x7f451815e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45080097a0 tx=0x7f450802b960 comp rx=0 tx=0).stop 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 -- 172.21.15.125:0/1455346542 shutdown_connections 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 --2- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180cf290 0x7f45180cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 --2- 172.21.15.125:0/1455346542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180ae5a0 0x7f451815e430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 -- 172.21.15.125:0/1455346542 >> 172.21.15.125:0/1455346542 conn(0x7f451806a800 msgr2=0x7f451806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 -- 172.21.15.125:0/1455346542 shutdown_connections 2024-01-16T14:57:00.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.906+0000 7f451f000640 1 -- 172.21.15.125:0/1455346542 wait complete. 2024-01-16T14:57:00.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.908+0000 7f451f000640 1 Processor -- start 2024-01-16T14:57:00.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.908+0000 7f451f000640 1 -- start start 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180b3ae0 0x7f45181502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180cf290 0x7f451815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4518153140 con 0x7f45180ae5a0 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45181520a0 con 0x7f45180cf290 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f451f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4518152380 con 0x7f45180b3ae0 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f4517e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f4517e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60402/0 (socket says 172.21.15.125:60402) 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.909+0000 7f4517e00640 1 -- 172.21.15.125:0/3187080916 learned_addr learned my addr 172.21.15.125:0/3187080916 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f450fe00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180b3ae0 0x7f45181502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f451d000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180cf290 0x7f451815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 -- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180b3ae0 msgr2=0x7f45181502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180b3ae0 0x7f45181502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 -- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180cf290 msgr2=0x7f451815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180cf290 0x7f451815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 -- 172.21.15.125:0/3187080916 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508009400 con 0x7f45180ae5a0 2024-01-16T14:57:00.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4517e00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f450802bf70 tx=0x7f4508004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.910+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450803e040 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.911+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508039a40 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.911+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4518152660 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.911+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508039df0 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.911+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4518152c20 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.912+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45180675d0 con 0x7f45180ae5a0 2024-01-16T14:57:00.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.913+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f450804d3b0 con 0x7f45180ae5a0 2024-01-16T14:57:00.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.914+0000 7f4514c00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 0x7f44f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.914+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45080ac130 con 0x7f45180ae5a0 2024-01-16T14:57:00.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.915+0000 7f450fe00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 0x7f44f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.915+0000 7f450fe00640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 0x7f44f006d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f451815ca60 tx=0x7f4500009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:00.918+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4508076f00 con 0x7f45180ae5a0 2024-01-16T14:57:01.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.149+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4518161be0 con 0x7f45180ae5a0 2024-01-16T14:57:01.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.150+0000 7f4514c00640 1 -- 172.21.15.125:0/3187080916 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f450807a520 con 0x7f45180ae5a0 2024-01-16T14:57:01.132 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:01.132 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:01.132 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.152+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 msgr2=0x7f44f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.152+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 0x7f44f006d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f451815ca60 tx=0x7f4500009040 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 msgr2=0x7f45181f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f450802bf70 tx=0x7f4508004730 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 shutdown_connections 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45180cf290 0x7f451815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45180b3ae0 0x7f45181502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44f006b2c0 0x7f44f006d780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 --2- 172.21.15.125:0/3187080916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45180ae5a0 0x7f45181f25c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 >> 172.21.15.125:0/3187080916 conn(0x7f451806a800 msgr2=0x7f451815a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 shutdown_connections 2024-01-16T14:57:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.153+0000 7f451f000640 1 -- 172.21.15.125:0/3187080916 wait complete. 2024-01-16T14:57:01.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:01.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.371+0000 7fa3c8e00640 1 Processor -- start 2024-01-16T14:57:01.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.371+0000 7fa3c8e00640 1 -- start start 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c8e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa3c413fcb0 0x7fa3c4164d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c8e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c40cdc40 con 0x7fa3c415afb0 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c40cdec0 con 0x7fa3c413cfa0 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c8e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c40ce1a0 con 0x7fa3c413fcb0 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa3c415afb0 0x7fa3c413f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45574/0 (socket says 172.21.15.125:45574) 2024-01-16T14:57:01.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.372+0000 7fa3c2a00640 1 -- 172.21.15.125:0/2252817145 learned_addr learned my addr 172.21.15.125:0/2252817145 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:01.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2252817145 >> v2:172.21.15.137:3300/0 conn(0x7fa3c413fcb0 0x7fa3c4164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384348048 0 0) 0x7fa3c40cdc40 con 0x7fa3c415afb0 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a4005730 con 0x7fa3c415afb0 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537509185 0 0) 0x7fa3a4005730 con 0x7fa3c415afb0 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 >> v2:172.21.15.137:3300/0 conn(0x7fa3c413fcb0 msgr2=0x7fa3c4164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 --2- 172.21.15.125:0/2252817145 >> v2:172.21.15.137:3300/0 conn(0x7fa3c413fcb0 0x7fa3c4164d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 >> v1:172.21.15.137:6789/0 conn(0x7fa3c413cfa0 legacy=0x7fa3c40cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c40ce480 con 0x7fa3c415afb0 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa3ac001670 con 0x7fa3c415afb0 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 >> v1:172.21.15.125:6789/0 conn(0x7fa3c415afb0 legacy=0x7fa3c413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a4005b80 0x7fa3a4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a40069d0 0x7fa3a4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 0x7fa3a400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c40ce1a0 con 0x7fa3a4007390 2024-01-16T14:57:01.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a400e330 con 0x7fa3a40069d0 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.373+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a400e4f0 con 0x7fa3a4005b80 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 0x7fa3a400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c2a00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a40069d0 0x7fa3a4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 -- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a4005b80 msgr2=0x7fa3a4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3400640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a4005b80 0x7fa3a4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a4005b80 0x7fa3a4005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 -- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a40069d0 msgr2=0x7fa3a4006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a40069d0 0x7fa3a4006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c3e00640 1 -- 172.21.15.125:0/2252817145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4005730 con 0x7fa3a4007390 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.374+0000 7fa3c2a00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a40069d0 0x7fa3a4006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.375+0000 7fa3c3e00640 1 -- 172.21.15.125:0/2252817145 reap_dead start 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.375+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 0x7fa3a400db90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009f90 tx=0x7fa3a4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c3cde7e6f7e820 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.375+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc02cc20 con 0x7fa3a4007390 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.375+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc00f040 con 0x7fa3a4007390 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.375+0000 7fa3c2000640 1 -- 172.21.15.125:0/2252817145 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc034440 con 0x7fa3a4007390 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 msgr2=0x7fa3a400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 0x7fa3a400db90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009f90 tx=0x7fa3a4007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2252817145 shutdown_connections 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2252817145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a4007390 0x7fa3a400db90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2252817145 >> 172.21.15.125:0/2252817145 conn(0x7fa3c406a800 msgr2=0x7fa3c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2252817145 shutdown_connections 2024-01-16T14:57:01.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.376+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2252817145 wait complete. 2024-01-16T14:57:01.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.377+0000 7fa3c8e00640 1 Processor -- start 2024-01-16T14:57:01.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 -- start start 2024-01-16T14:57:01.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 0x7fa3c40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3c41f2600 0x7fa3c41f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4139e60 con 0x7fa3c40adfc0 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c40acf40 con 0x7fa3a4007390 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.378+0000 7fa3c8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c40ad220 con 0x7fa3c41f2600 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60416/0 (socket says 172.21.15.125:60416) 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 -- 172.21.15.125:0/2396044955 learned_addr learned my addr 172.21.15.125:0/2396044955 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c3400640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 0x7fa3c40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c3e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3c41f2600 0x7fa3c41f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 -- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3c41f2600 msgr2=0x7fa3c41f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3c41f2600 0x7fa3c41f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 -- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 msgr2=0x7fa3c40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 0x7fa3c40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 -- 172.21.15.125:0/2396044955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c40ad500 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c3400640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 0x7fa3c40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.379+0000 7fa3c2a00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac001200 tx=0x7fa3ac00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.380+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0074e0 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.380+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac007640 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.380+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0049e0 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.380+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c40ad7e0 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.380+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c41f72d0 con 0x7fa3c40adfc0 2024-01-16T14:57:01.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.381+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c4067450 con 0x7fa3c40adfc0 2024-01-16T14:57:01.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.382+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa3ac004b40 con 0x7fa3c40adfc0 2024-01-16T14:57:01.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.382+0000 7fa3bb400640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 0x7fa39c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.383+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3ac089590 con 0x7fa3c40adfc0 2024-01-16T14:57:01.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.383+0000 7fa3c3400640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 0x7fa39c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.383+0000 7fa3c3400640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 0x7fa39c06d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.387+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3ac0543f0 con 0x7fa3c40adfc0 2024-01-16T14:57:01.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.577+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c413f760 con 0x7fa3c40adfc0 2024-01-16T14:57:01.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.580+0000 7fa3bb400640 1 -- 172.21.15.125:0/2396044955 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7fa3ac057a10 con 0x7fa3c40adfc0 2024-01-16T14:57:01.563 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:01.563 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:01.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 msgr2=0x7fa39c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 0x7fa39c06d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:01.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 msgr2=0x7fa3c41f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac001200 tx=0x7fa3ac00dd40 comp rx=0 tx=0).stop 2024-01-16T14:57:01.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 shutdown_connections 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3c41f2600 0x7fa3c41f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa39c06b2c0 0x7fa39c06d780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3c40adfc0 0x7fa3c41f6d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 --2- 172.21.15.125:0/2396044955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a4007390 0x7fa3c40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 >> 172.21.15.125:0/2396044955 conn(0x7fa3c406a800 msgr2=0x7fa3c40aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 shutdown_connections 2024-01-16T14:57:01.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.583+0000 7fa3c8e00640 1 -- 172.21.15.125:0/2396044955 wait complete. 2024-01-16T14:57:01.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:01.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.802+0000 7f91b6200640 1 Processor -- start 2024-01-16T14:57:01.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.802+0000 7f91b6200640 1 -- start start 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91b6200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f91b015b030 0x7f91b00b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91b6200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91b00b31b0 0x7f91b00b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91b6200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00b3610 con 0x7f91b00b5380 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91b6200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00d0780 con 0x7f91b00b31b0 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91b6200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00cf8f0 con 0x7f91b015b030 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91aea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91b00b31b0 0x7f91b00b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91aea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91b00b31b0 0x7f91b00b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60426/0 (socket says 172.21.15.125:60426) 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91aea00640 1 -- 172.21.15.125:0/3039142261 learned_addr learned my addr 172.21.15.125:0/3039142261 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.804+0000 7f91afe00640 1 --1- 172.21.15.125:0/3039142261 >> v1:172.21.15.125:6789/0 conn(0x7f91b00b5380 0x7f91b015ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45578/0 (socket says 172.21.15.125:45578) 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.805+0000 7f91af400640 1 --2- 172.21.15.125:0/3039142261 >> v2:172.21.15.137:3300/0 conn(0x7f91b015b030 0x7f91b00b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.805+0000 7f91aea00640 1 -- 172.21.15.125:0/3039142261 >> v2:172.21.15.137:3300/0 conn(0x7f91b015b030 msgr2=0x7f91b00b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.805+0000 7f91aea00640 1 --2- 172.21.15.125:0/3039142261 >> v2:172.21.15.137:3300/0 conn(0x7f91b015b030 0x7f91b00b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.805+0000 7f91aea00640 1 -- 172.21.15.125:0/3039142261 >> v1:172.21.15.125:6789/0 conn(0x7f91b00b5380 legacy=0x7f91b015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.805+0000 7f91aea00640 1 -- 172.21.15.125:0/3039142261 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00b5810 con 0x7f91b00b31b0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91aea00640 1 --2- 172.21.15.125:0/3039142261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b00b31b0 0x7f91b00b4e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91b00b61d0 tx=0x7f919802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42e3f5ddfce833f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91ae000640 1 -- 172.21.15.125:0/3039142261 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919802cb10 con 0x7f91b00b31b0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91ae000640 1 -- 172.21.15.125:0/3039142261 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919802cc70 con 0x7f91b00b31b0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91ae000640 1 -- 172.21.15.125:0/3039142261 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980343f0 con 0x7f91b00b31b0 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91b6200640 1 -- 172.21.15.125:0/3039142261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b00b31b0 msgr2=0x7f91b00b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.806+0000 7f91b6200640 1 --2- 172.21.15.125:0/3039142261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b00b31b0 0x7f91b00b4e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91b00b61d0 tx=0x7f919802bf70 comp rx=0 tx=0).stop 2024-01-16T14:57:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 -- 172.21.15.125:0/3039142261 shutdown_connections 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 --2- 172.21.15.125:0/3039142261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b00b31b0 0x7f91b00b4e40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 --2- 172.21.15.125:0/3039142261 >> v2:172.21.15.137:3300/0 conn(0x7f91b015b030 0x7f91b00b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 -- 172.21.15.125:0/3039142261 >> 172.21.15.125:0/3039142261 conn(0x7f91b006a3e0 msgr2=0x7f91b006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 -- 172.21.15.125:0/3039142261 shutdown_connections 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.807+0000 7f91b6200640 1 -- 172.21.15.125:0/3039142261 wait complete. 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.808+0000 7f91b6200640 1 Processor -- start 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.808+0000 7f91b6200640 1 -- start start 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 0x7f91b00ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 0x7f91b00ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b015bc00 con 0x7f91b015b030 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00cd320 con 0x7f91b00b5380 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.809+0000 7f91b6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b015b5b0 con 0x7f91b00b31b0 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60438/0 (socket says 172.21.15.125:60438) 2024-01-16T14:57:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 -- 172.21.15.125:0/2164208730 learned_addr learned my addr 172.21.15.125:0/2164208730 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91af400640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 0x7f91b00ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91aea00640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 0x7f91b00ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 -- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 msgr2=0x7f91b00ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 0x7f91b00ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 -- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 msgr2=0x7f91b00ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 0x7f91b00ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 -- 172.21.15.125:0/2164208730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198009b30 con 0x7f91b015b030 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91af400640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 0x7f91b00ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91aea00640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 0x7f91b00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.810+0000 7f91afe00640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c400 tx=0x7f91a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.811+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80074e0 con 0x7f91b015b030 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.811+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8007640 con 0x7f91b015b030 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.811+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b015b890 con 0x7f91b015b030 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.811+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80049e0 con 0x7f91b015b030 2024-01-16T14:57:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.811+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b01f2900 con 0x7f91b015b030 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.812+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b015b410 con 0x7f91b015b030 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.813+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f91a8004b40 con 0x7f91b015b030 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.814+0000 7f91a7400640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 0x7f918806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.814+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91a8089670 con 0x7f91b015b030 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.814+0000 7f91af400640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 0x7f918806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.814+0000 7f91af400640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 0x7f918806d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f91a0006fd0 tx=0x7f91a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:01.818+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91b015b410 con 0x7f91b015b030 2024-01-16T14:57:02.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.037+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b015b410 con 0x7f91b015b030 2024-01-16T14:57:02.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.038+0000 7f91a7400640 1 -- 172.21.15.125:0/2164208730 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f91b015b410 con 0x7f91b015b030 2024-01-16T14:57:02.020 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:02.020 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:02.020 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:02.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 msgr2=0x7f918806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 0x7f918806d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f91a0006fd0 tx=0x7f91a0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 msgr2=0x7f91b01f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c400 tx=0x7f91a800c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 shutdown_connections 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f918806b2c0 0x7f918806d780 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91b015b030 0x7f91b01f23c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91b00b5380 0x7f91b00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 --2- 172.21.15.125:0/2164208730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91b00b31b0 0x7f91b00ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.040+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 >> 172.21.15.125:0/2164208730 conn(0x7f91b006a3e0 msgr2=0x7f91b006c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.041+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 shutdown_connections 2024-01-16T14:57:02.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.041+0000 7f91b6200640 1 -- 172.21.15.125:0/2164208730 wait complete. 2024-01-16T14:57:02.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:56:45.415990+0000', 'modified': '2024-01-16T14:56:48.497267+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.137:6835/3474295024', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.137:6833', 'nonce': 3474295024}, {'type': 'v1', 'addr': '172.21.15.137:6835', 'nonce': 3474295024}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T14:57:02.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T14:57:02.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-16T14:57:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.676+0000 7f6ee0a00640 1 Processor -- start 2024-01-16T14:57:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.676+0000 7f6ee0a00640 1 -- start start 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ee0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6edc143c40 0x7f6edc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ee0a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 0x7f6edc06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ee0a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc070e50 con 0x7f6edc0717d0 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ee0a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc0723c0 con 0x7f6edc143c40 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ee0a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc144020 con 0x7f6edc0ccc10 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6eda800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6edc143c40 0x7f6edc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6eda800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6edc143c40 0x7f6edc071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60442/0 (socket says 172.21.15.125:60442) 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6eda800640 1 -- 172.21.15.125:0/2600029047 learned_addr learned my addr 172.21.15.125:0/2600029047 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6eda800640 1 -- 172.21.15.125:0/2600029047 >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 msgr2=0x7f6edc06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6edb200640 1 --2- 172.21.15.125:0/2600029047 >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 0x7f6edc06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6eda800640 1 --2- 172.21.15.125:0/2600029047 >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 0x7f6edc06d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.678+0000 7f6ed9e00640 1 -- 172.21.15.125:0/2600029047 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440868062 0 0) 0x7f6edc070e50 con 0x7f6edc0717d0 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.679+0000 7f6eda800640 1 -- 172.21.15.125:0/2600029047 >> v1:172.21.15.125:6789/0 conn(0x7f6edc0717d0 legacy=0x7f6edc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.679+0000 7f6eda800640 1 -- 172.21.15.125:0/2600029047 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc071e00 con 0x7f6edc143c40 2024-01-16T14:57:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.679+0000 7f6edb200640 1 --2- 172.21.15.125:0/2600029047 >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 0x7f6edc06d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6eda800640 1 --2- 172.21.15.125:0/2600029047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc143c40 0x7f6edc071290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6edc070690 tx=0x7f6ecc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2668bb91b2425741 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6ed9e00640 1 -- 172.21.15.125:0/2600029047 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc031870 con 0x7f6edc143c40 2024-01-16T14:57:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6ed9e00640 1 -- 172.21.15.125:0/2600029047 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc0319d0 con 0x7f6edc143c40 2024-01-16T14:57:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6ed9e00640 1 -- 172.21.15.125:0/2600029047 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc031d60 con 0x7f6edc143c40 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6ee0a00640 1 -- 172.21.15.125:0/2600029047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc143c40 msgr2=0x7f6edc071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.680+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/2600029047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc143c40 0x7f6edc071290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6edc070690 tx=0x7f6ecc02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 -- 172.21.15.125:0/2600029047 shutdown_connections 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/2600029047 >> v2:172.21.15.137:3300/0 conn(0x7f6edc0ccc10 0x7f6edc06d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/2600029047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc143c40 0x7f6edc071290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 -- 172.21.15.125:0/2600029047 >> 172.21.15.125:0/2600029047 conn(0x7f6edc06a800 msgr2=0x7f6edc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 -- 172.21.15.125:0/2600029047 shutdown_connections 2024-01-16T14:57:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.681+0000 7f6ee0a00640 1 -- 172.21.15.125:0/2600029047 wait complete. 2024-01-16T14:57:02.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.682+0000 7f6ee0a00640 1 Processor -- start 2024-01-16T14:57:02.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 -- start start 2024-01-16T14:57:02.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6edc0ccc10 0x7f6edc15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6edc143c40 0x7f6edc0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc15af00 con 0x7f6edc0717d0 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc12f7d0 con 0x7f6edc0ccc10 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.683+0000 7f6ee0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc12fab0 con 0x7f6edc143c40 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60454/0 (socket says 172.21.15.125:60454) 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 -- 172.21.15.125:0/566226052 learned_addr learned my addr 172.21.15.125:0/566226052 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 -- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6edc143c40 msgr2=0x7f6edc0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6edc143c40 0x7f6edc0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 -- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6edc0ccc10 msgr2=0x7f6edc15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6edc0ccc10 0x7f6edc15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 -- 172.21.15.125:0/566226052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0094d0 con 0x7f6edc0717d0 2024-01-16T14:57:02.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.684+0000 7f6eda800640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc000c00 tx=0x7f6ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.685+0000 7f6ed2a00640 1 -- 172.21.15.125:0/566226052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc03bbf0 con 0x7f6edc0717d0 2024-01-16T14:57:02.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.685+0000 7f6ed2a00640 1 -- 172.21.15.125:0/566226052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc03bd50 con 0x7f6edc0717d0 2024-01-16T14:57:02.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.685+0000 7f6ed2a00640 1 -- 172.21.15.125:0/566226052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc004270 con 0x7f6edc0717d0 2024-01-16T14:57:02.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.685+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc12fd90 con 0x7f6edc0717d0 2024-01-16T14:57:02.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.685+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc130350 con 0x7f6edc0717d0 2024-01-16T14:57:02.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.687+0000 7f6ed2a00640 1 -- 172.21.15.125:0/566226052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6ecc043070 con 0x7f6edc0717d0 2024-01-16T14:57:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.688+0000 7f6ed2a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 0x7f6eb406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.689+0000 7f6ed2a00640 1 -- 172.21.15.125:0/566226052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ecc02d040 con 0x7f6edc0717d0 2024-01-16T14:57:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.689+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 0x7f6eb406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.689+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 0x7f6eb406d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0ae0e0 tx=0x7f6ec4007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.689+0000 7f6ee3f9e740 1 Processor -- start 2024-01-16T14:57:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 -- start start 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x55701867f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557018679d70 0x55701867e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570185dd880 con 0x55701867e7b0 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570185ecae0 con 0x557018679d70 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.690+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570185ea1d0 con 0x55701867f070 2024-01-16T14:57:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60456/0 (socket says 172.21.15.125:60456) 2024-01-16T14:57:02.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 -- 172.21.15.125:0/413089013 learned_addr learned my addr 172.21.15.125:0/413089013 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6eda800640 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x55701867f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557018679d70 0x55701867e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 -- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 msgr2=0x55701867f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x55701867f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 -- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557018679d70 msgr2=0x55701867e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557018679d70 0x55701867e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 -- 172.21.15.125:0/413089013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570185ad860 con 0x55701867e7b0 2024-01-16T14:57:02.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.691+0000 7f6edb200640 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4008f30 tx=0x7f6ed400d860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb733b2426969a1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ecb400640 1 -- 172.21.15.125:0/413089013 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed400e910 con 0x55701867e7b0 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ecb400640 1 -- 172.21.15.125:0/413089013 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed400ea70 con 0x55701867e7b0 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ecb400640 1 -- 172.21.15.125:0/413089013 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed400ee20 con 0x55701867e7b0 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/413089013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 msgr2=0x5570186c2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4008f30 tx=0x7f6ed400d860 comp rx=0 tx=0).stop 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/413089013 shutdown_connections 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186c2730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557018679d70 0x55701867e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/413089013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x55701867f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/413089013 >> 172.21.15.125:0/413089013 conn(0x557018640c70 msgr2=0x5570186c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/413089013 shutdown_connections 2024-01-16T14:57:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.692+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/413089013 wait complete. 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.693+0000 7f6ee3f9e740 1 -- start start 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.693+0000 7f6ee3f9e740 1 Processor -- start 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x55701869fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570185e5ac0 con 0x557018679d70 2024-01-16T14:57:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570185822f0 con 0x55701867e7b0 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55701859fc70 con 0x55701867f070 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6ee3f9e740 10 client.0 _ll_register_callbacks cb 0x5570185a43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60462/0 (socket says 172.21.15.125:60462) 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.694+0000 7f6eda800640 1 -- 172.21.15.125:0/2543949420 learned_addr learned my addr 172.21.15.125:0/2543949420 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 -- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 msgr2=0x5570186c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 -- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 msgr2=0x55701869fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x55701869fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 -- 172.21.15.125:0/2543949420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed4008b00 con 0x557018679d70 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6eda800640 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02b950 tx=0x7f6ecc07b360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ffd3db9b5f4663 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ec3e00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc082990 con 0x557018679d70 2024-01-16T14:57:02.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ecc082af0 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570185deaa0 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ecc0aa120 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ee3f9e740 10 client.4677 fetch_fsmap learned FSMap version 8 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5570185e4140 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.695+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc086070 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6ecc0920d0 con 0x557018679d70 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 10 client.4677 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 10 client.4677 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 20 client.4677 populate_metadata read hostname 'smithi125' 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701869f4a0 0x5570186c6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 4 client.4677 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 --> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570185de7c0 con 0x55701869f4a0 2024-01-16T14:57:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.696+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701869f4a0 0x5570186c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.697+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701869f4a0 0x5570186c6400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10b65e8827173e1b server_cookie=6fdcf67ee803f8b3 in_seq=0 out_seq=0 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.697+0000 7f6ec3e00640 10 client.4677 ms_handle_connect on v2:172.21.15.137:6833/3474295024 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2543949420 <== mds.0 v2:172.21.15.137:6833/3474295024 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5570185de7c0 con 0x55701869f4a0 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ec3e00640 10 client.4677 handle_command_reply: tid=0 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 client.4677 shutdown 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 msgr2=0x55701869c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02b950 tx=0x7f6ecc07b360 comp rx=0 tx=0).stop 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 shutdown_connections 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701869f4a0 0x5570186c6400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x55701869fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ec3e00640 0 client.4677 ms_handle_reset on v2:172.21.15.137:6833/3474295024 2024-01-16T14:57:02.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2543949420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x55701869c950 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.699+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 >> 172.21.15.125:0/2543949420 conn(0x557018640c70 msgr2=0x557018678dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.700+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 shutdown_connections 2024-01-16T14:57:02.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.700+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2543949420 wait complete. 2024-01-16T14:57:02.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.700+0000 7f6ee3f9e740 20 client.4677 trim_cache size 0 max 16384 2024-01-16T14:57:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.716+0000 7f6ee3f9e740 1 Processor -- start 2024-01-16T14:57:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.716+0000 7f6ee3f9e740 1 -- start start 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557018679d70 0x557018728c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867f070 0x55701867d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x557018348f20 con 0x55701867e7b0 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570186d7330 con 0x55701867f070 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x557018680410 con 0x557018679d70 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60472/0 (socket says 172.21.15.125:60472) 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 -- 172.21.15.125:0/746362870 learned_addr learned my addr 172.21.15.125:0/746362870 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 -- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557018679d70 msgr2=0x557018728c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557018679d70 0x557018728c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 -- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867f070 msgr2=0x55701867d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867f070 0x55701867d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.717+0000 7f6ed3e00640 1 -- 172.21.15.125:0/746362870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557018581b80 con 0x55701867e7b0 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6edc1f5e70 tx=0x7f6ec40010f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b78702a7250e30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ecbe00640 1 -- 172.21.15.125:0/746362870 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4003bf0 con 0x55701867e7b0 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ecbe00640 1 -- 172.21.15.125:0/746362870 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4003d50 con 0x55701867e7b0 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ecbe00640 1 -- 172.21.15.125:0/746362870 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40205d0 con 0x55701867e7b0 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/746362870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 msgr2=0x5570186754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6edc1f5e70 tx=0x7f6ec40010f0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/746362870 shutdown_connections 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867f070 0x55701867d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55701867e7b0 0x5570186754f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/746362870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557018679d70 0x557018728c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.718+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/746362870 >> 172.21.15.125:0/746362870 conn(0x557018640c70 msgr2=0x5570186a6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.719+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/746362870 shutdown_connections 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.719+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/746362870 wait complete. 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.719+0000 7f6ee3f9e740 1 -- start start 2024-01-16T14:57:02.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.719+0000 7f6ee3f9e740 1 Processor -- start 2024-01-16T14:57:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x5570186dda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570186a5f50 con 0x557018679d70 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570186aca10 con 0x55701867e7b0 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570186ccb90 con 0x55701867f070 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6ee3f9e740 10 client.0 _ll_register_callbacks cb 0x7f6ecc03bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.720+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60482/0 (socket says 172.21.15.125:60482) 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 -- 172.21.15.125:0/2801835827 learned_addr learned my addr 172.21.15.125:0/2801835827 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 -- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 msgr2=0x5570186c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 -- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 msgr2=0x5570186dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x5570186dda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 -- 172.21.15.125:0/2801835827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec40050d0 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6eda800640 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0090b0 tx=0x7f6ecc009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b42e114de12c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6ec3e00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc07a8f0 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ecc0920d0 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ed400ea70 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.721+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ecc076910 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 10 client.4683 fetch_fsmap learned FSMap version 8 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5570185a7170 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed400ea70 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6ecc076a70 con 0x557018679d70 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 10 client.4683 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 10 client.4683 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 20 client.4683 populate_metadata read hostname 'smithi125' 2024-01-16T14:57:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701867df40 0x5570186c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 4 client.4683 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T14:57:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 --> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5570182e66c0 con 0x55701867df40 2024-01-16T14:57:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.722+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701867df40 0x5570186c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.723+0000 7f6ed3e00640 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701867df40 0x5570186c50a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb0cf07930cf8346 server_cookie=6304b39b0745e99e in_seq=0 out_seq=0 2024-01-16T14:57:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.723+0000 7f6ec3e00640 10 client.4683 ms_handle_connect on v2:172.21.15.137:6833/3474295024 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ec3e00640 1 -- 172.21.15.125:0/2801835827 <== mds.0 v2:172.21.15.137:6833/3474295024 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5570182e66c0 con 0x55701867df40 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ec3e00640 10 client.4683 handle_command_reply: tid=0 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 client.4683 shutdown 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 msgr2=0x557018725ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0090b0 tx=0x7f6ecc009240 comp rx=0 tx=0).stop 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 shutdown_connections 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:6833/3474295024,v1:172.21.15.137:6835/3474295024] conn(0x55701867df40 0x5570186c50a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55701867f070 0x5570186c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55701867e7b0 0x5570186dda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ec3e00640 0 client.4683 ms_handle_reset on v2:172.21.15.137:6833/3474295024 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 --2- 172.21.15.125:0/2801835827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557018679d70 0x557018725ae0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 >> 172.21.15.125:0/2801835827 conn(0x557018640c70 msgr2=0x5570186e6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 shutdown_connections 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 1 -- 172.21.15.125:0/2801835827 wait complete. 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.724+0000 7f6ee3f9e740 20 client.4683 trim_cache size 0 max 16384 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:57:02.709 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4275, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 16.724446986, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 8, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 20, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 20, 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 22.295627939999999 2024-01-16T14:57:02.710 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 msgr2=0x7f6eb406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 0x7f6eb406d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0ae0e0 tx=0x7f6ec4007930 comp rx=0 tx=0).stop 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 msgr2=0x7f6edc15a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc000c00 tx=0x7f6ecc0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 shutdown_connections 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6edc143c40 0x7f6edc0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6edc0ccc10 0x7f6edc15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6eb406b2c0 0x7f6eb406d780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 --2- 172.21.15.125:0/566226052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6edc0717d0 0x7f6edc15a480 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 >> 172.21.15.125:0/566226052 conn(0x7f6edc06a800 msgr2=0x7f6edc0aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 shutdown_connections 2024-01-16T14:57:02.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:02.725+0000 7f6ee0a00640 1 -- 172.21.15.125:0/566226052 wait complete. 2024-01-16T14:57:02.722 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-16T14:57:02.762 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-16T14:57:02.762 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-16T14:57:02.762 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:02.763 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:02.763 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:02.763 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:02.763 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:02.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:02.763 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T14:57:02.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:02.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2024-01-16T14:57:02.910 INFO:teuthology.orchestra.run.smithi125.stderr:Cannot find device "ceph-brx" 2024-01-16T14:57:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:57:02.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:02.914 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T14:57:02.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:02.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-01-16T14:57:03.065 INFO:teuthology.orchestra.run.smithi137.stderr:Cannot find device "ceph-brx" 2024-01-16T14:57:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:57:03.069 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-16T14:57:03.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:03.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.305+0000 7f818ae00640 1 Processor -- start 2024-01-16T14:57:03.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.305+0000 7f818ae00640 1 -- start start 2024-01-16T14:57:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f818ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e360 0x7f81840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f818ae00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818415f9c0 0x7f81840cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f818ae00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818415c5f0 con 0x7f818413df80 2024-01-16T14:57:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f818ae00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f818406c470 con 0x7f818415f9c0 2024-01-16T14:57:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f818ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818406c750 con 0x7f818413e360 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f8188e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818415f9c0 0x7f81840cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f8188e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818415f9c0 0x7f81840cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60492/0 (socket says 172.21.15.125:60492) 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.307+0000 7f8188e00640 1 -- 172.21.15.125:0/3253374715 learned_addr learned my addr 172.21.15.125:0/3253374715 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.308+0000 7f8183400640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e360 0x7f81840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.308+0000 7f8188e00640 1 -- 172.21.15.125:0/3253374715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e360 msgr2=0x7f81840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.308+0000 7f8188e00640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e360 0x7f81840cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.308+0000 7f8188e00640 1 -- 172.21.15.125:0/3253374715 >> v1:172.21.15.137:6789/0 conn(0x7f818413df80 legacy=0x7f818415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.308+0000 7f8188e00640 1 -- 172.21.15.125:0/3253374715 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818406ca30 con 0x7f818415f9c0 2024-01-16T14:57:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.309+0000 7f8188e00640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f81840cf7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f817c009e50 tx=0x7f817c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de288cacf7c2e75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.309+0000 7f8182a00640 1 -- 172.21.15.125:0/3253374715 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c02ca70 con 0x7f818415f9c0 2024-01-16T14:57:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.309+0000 7f8182a00640 1 -- 172.21.15.125:0/3253374715 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c02cbd0 con 0x7f818415f9c0 2024-01-16T14:57:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.309+0000 7f8182a00640 1 -- 172.21.15.125:0/3253374715 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c0343f0 con 0x7f818415f9c0 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 -- 172.21.15.125:0/3253374715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 msgr2=0x7f81840cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f81840cf7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f817c009e50 tx=0x7f817c02bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 -- 172.21.15.125:0/3253374715 shutdown_connections 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f81840cf7d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 --2- 172.21.15.125:0/3253374715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e360 0x7f81840cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 -- 172.21.15.125:0/3253374715 >> 172.21.15.125:0/3253374715 conn(0x7f818406a800 msgr2=0x7f818406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 -- 172.21.15.125:0/3253374715 shutdown_connections 2024-01-16T14:57:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.310+0000 7f818ae00640 1 -- 172.21.15.125:0/3253374715 wait complete. 2024-01-16T14:57:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.312+0000 7f818ae00640 1 Processor -- start 2024-01-16T14:57:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.312+0000 7f818ae00640 1 -- start start 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413df80 0x7f81840b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818413e360 0x7f818413d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818413de40 con 0x7f818415f9c0 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818414bd60 con 0x7f818413e360 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f818ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818414c000 con 0x7f818413df80 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60494/0 (socket says 172.21.15.125:60494) 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 -- 172.21.15.125:0/3086129123 learned_addr learned my addr 172.21.15.125:0/3086129123 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:03.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 -- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413df80 msgr2=0x7f81840b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413df80 0x7f81840b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 -- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818413e360 msgr2=0x7f818413d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818413e360 0x7f818413d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.313+0000 7f8188e00640 1 -- 172.21.15.125:0/3086129123 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c009b30 con 0x7f818415f9c0 2024-01-16T14:57:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f8188e00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f817c036040 tx=0x7f817c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c03e450 con 0x7f818415f9c0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c03e5b0 con 0x7f818415f9c0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c03e960 con 0x7f818415f9c0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818414c2e0 con 0x7f818415f9c0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.314+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818414c8a0 con 0x7f818415f9c0 2024-01-16T14:57:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.316+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818414cac0 con 0x7f818415f9c0 2024-01-16T14:57:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.316+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f817c03eac0 con 0x7f818415f9c0 2024-01-16T14:57:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.317+0000 7f817be00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 0x7f815c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.317+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f817c0ac790 con 0x7f818415f9c0 2024-01-16T14:57:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.317+0000 7f8183e00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 0x7f815c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.318+0000 7f8183e00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 0x7f815c06d800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8174006fd0 tx=0x7f8174006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.322+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f817c0775e0 con 0x7f818415f9c0 2024-01-16T14:57:03.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.551+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f81840675d0 con 0x7f818415f9c0 2024-01-16T14:57:03.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.552+0000 7f817be00640 1 -- 172.21.15.125:0/3086129123 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f817c07ac00 con 0x7f818415f9c0 2024-01-16T14:57:03.534 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:03.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 msgr2=0x7f815c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 0x7f815c06d800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8174006fd0 tx=0x7f8174006bc0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 msgr2=0x7f818413d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f817c036040 tx=0x7f817c004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 shutdown_connections 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b340 0x7f815c06d800 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818415f9c0 0x7f818413d900 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818413e360 0x7f818413d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 --2- 172.21.15.125:0/3086129123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413df80 0x7f81840b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 >> 172.21.15.125:0/3086129123 conn(0x7f818406a800 msgr2=0x7f81840cdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 shutdown_connections 2024-01-16T14:57:03.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:03.555+0000 7f818ae00640 1 -- 172.21.15.125:0/3086129123 wait complete. 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:03.556 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:57:03.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:03.557 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:03.605 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42625sec preferred_lft 42625sec 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T14:57:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:03.607 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:57:03.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr flush dev ceph-brx 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set ceph-brx up 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:57:03.607 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:04.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:04.052 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:04.136 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T14:57:04.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:04.140 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T14:57:04.201 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T14:57:04.202 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:04.202 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:04.202 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:04.202 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:04.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:04.203 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:04.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:04.901 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T14:57:04.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:04.954 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list-id 2024-01-16T14:57:05.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:05.003 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:05.003 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:05.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:05.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-16T14:57:05.003 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:05.137 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-16T14:57:05.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-16T14:57:05.138 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:05.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:05.411 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:05.411 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:05.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T14:57:05.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:57:05.411 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:05.576 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T14:57:05.576 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:05.576 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:05.623 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:57:05.623 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:05.623 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:05.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:57:05.804 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:57:05.856 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:05.858 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:05.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:05.859 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:57:05.905 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:57:05.905 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T14:57:05.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:57:06.026 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:06.067 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:06.069 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:06.070 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:06.070 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:57:06.070 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:06.070 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:06.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:06.071 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:57:06.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T14:57:06.167+0000 7f408dc38480 -1 init, newargv = 0x5612c70e6c70 newargc=15 2024-01-16T14:57:06.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[44430]: starting ceph client 2024-01-16T14:57:06.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[44430]: starting fuse 2024-01-16T14:57:06.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:06.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:07.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:57:07.359 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:57:07.417 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:07.418 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:07.419 INFO:teuthology.orchestra.run.smithi125.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:07.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:07.420 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:57:07.471 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T14:57:07.471 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T14:57:07.472 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T14:57:07.473 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T14:57:07.644 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.44430.asok 2024-01-16T14:57:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:07.663 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44430.asok 2024-01-16T14:57:07.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:07.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44430.asok status 2024-01-16T14:57:07.897 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:57:07.897 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T14:57:07.897 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "44430", 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4695, 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 4695 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4043740612 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4043740612 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.4695 192.168.0.1:0/4043740612", 2024-01-16T14:57:07.899 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/4043740612", 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 8, 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 20, 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:57:07.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.141+0000 7eff73e00640 1 Processor -- start 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.141+0000 7eff73e00640 1 -- start start 2024-01-16T14:57:08.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7eff740cc500 0x7eff740cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff7406d160 0x7eff7406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff74070e50 con 0x7eff74143c40 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff740723c0 con 0x7eff7406d160 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff740cced0 con 0x7eff740cc500 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff72a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7eff74143c40 0x7eff74071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59758/0 (socket says 172.21.15.125:59758) 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff72a00640 1 -- 172.21.15.125:0/4066614440 learned_addr learned my addr 172.21.15.125:0/4066614440 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:08.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.143+0000 7eff73400640 1 --2- 172.21.15.125:0/4066614440 >> v2:172.21.15.125:3300/0 conn(0x7eff7406d160 0x7eff7406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff69e00640 1 --2- 172.21.15.125:0/4066614440 >> v2:172.21.15.137:3300/0 conn(0x7eff740cc500 0x7eff740cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff72000640 1 -- 172.21.15.125:0/4066614440 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776140317 0 0) 0x7eff74070e50 con 0x7eff74143c40 2024-01-16T14:57:08.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff72000640 1 -- 172.21.15.125:0/4066614440 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff54003520 con 0x7eff74143c40 2024-01-16T14:57:08.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff73400640 1 -- 172.21.15.125:0/4066614440 >> v2:172.21.15.137:3300/0 conn(0x7eff740cc500 msgr2=0x7eff740cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:08.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff73400640 1 --2- 172.21.15.125:0/4066614440 >> v2:172.21.15.137:3300/0 conn(0x7eff740cc500 0x7eff740cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff73400640 1 -- 172.21.15.125:0/4066614440 >> v1:172.21.15.125:6789/0 conn(0x7eff74143c40 legacy=0x7eff74071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.144+0000 7eff73400640 1 -- 172.21.15.125:0/4066614440 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff74144020 con 0x7eff7406d160 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.145+0000 7eff73400640 1 --2- 172.21.15.125:0/4066614440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff7406d160 0x7eff7406d560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff740b44a0 tx=0x7eff6c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0db0c1e769c420 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.145+0000 7eff72000640 1 -- 172.21.15.125:0/4066614440 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c02cbf0 con 0x7eff7406d160 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.145+0000 7eff72000640 1 -- 172.21.15.125:0/4066614440 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c02cd50 con 0x7eff7406d160 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.145+0000 7eff72000640 1 -- 172.21.15.125:0/4066614440 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c034660 con 0x7eff7406d160 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 -- 172.21.15.125:0/4066614440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff7406d160 msgr2=0x7eff7406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 --2- 172.21.15.125:0/4066614440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff7406d160 0x7eff7406d560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff740b44a0 tx=0x7eff6c02be40 comp rx=0 tx=0).stop 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 -- 172.21.15.125:0/4066614440 shutdown_connections 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 --2- 172.21.15.125:0/4066614440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff7406d160 0x7eff7406d560 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 --2- 172.21.15.125:0/4066614440 >> v2:172.21.15.137:3300/0 conn(0x7eff740cc500 0x7eff740cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 -- 172.21.15.125:0/4066614440 >> 172.21.15.125:0/4066614440 conn(0x7eff7406a800 msgr2=0x7eff7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 -- 172.21.15.125:0/4066614440 shutdown_connections 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.146+0000 7eff73e00640 1 -- 172.21.15.125:0/4066614440 wait complete. 2024-01-16T14:57:08.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.148+0000 7eff73e00640 1 Processor -- start 2024-01-16T14:57:08.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.148+0000 7eff73e00640 1 -- start start 2024-01-16T14:57:08.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7406d160 0x7eff740ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 0x7eff741406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740c3ef0 con 0x7eff74143c40 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740c41d0 con 0x7eff740cc500 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff740c44b0 con 0x7eff7406d160 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35868/0 (socket says 172.21.15.125:35868) 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 -- 172.21.15.125:0/3962310890 learned_addr learned my addr 172.21.15.125:0/3962310890 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff69e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 0x7eff741406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 -- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7406d160 msgr2=0x7eff740ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7406d160 0x7eff740ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 -- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 msgr2=0x7eff741406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.149+0000 7eff73400640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 0x7eff741406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff73400640 1 -- 172.21.15.125:0/3962310890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c009a70 con 0x7eff74143c40 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff69e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 0x7eff741406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff73400640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009e80 tx=0x7eff6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c040650 con 0x7eff74143c40 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c0407b0 con 0x7eff74143c40 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff740c4790 con 0x7eff74143c40 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c040b60 con 0x7eff74143c40 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.150+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff740ab5e0 con 0x7eff74143c40 2024-01-16T14:57:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.151+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff740c4cb0 con 0x7eff74143c40 2024-01-16T14:57:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.152+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff6c040cc0 con 0x7eff74143c40 2024-01-16T14:57:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.154+0000 7eff6b400640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 0x7eff4c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.154+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff6c0ac9e0 con 0x7eff74143c40 2024-01-16T14:57:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.154+0000 7eff72a00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 0x7eff4c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.155+0000 7eff72a00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 0x7eff4c06d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff64001570 tx=0x7eff64007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.159+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff6c0777b0 con 0x7eff74143c40 2024-01-16T14:57:08.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.386+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7eff74067450 con 0x7eff74143c40 2024-01-16T14:57:08.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.387+0000 7eff6b400640 1 -- 172.21.15.125:0/3962310890 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7eff6c039060 con 0x7eff74143c40 2024-01-16T14:57:08.369 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 msgr2=0x7eff4c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 0x7eff4c06d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff64001570 tx=0x7eff64007930 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 msgr2=0x7eff7406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009e80 tx=0x7eff6c004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 shutdown_connections 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7eff4c06b2c0 0x7eff4c06d780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff74143c40 0x7eff7406d540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff740cc500 0x7eff741406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 --2- 172.21.15.125:0/3962310890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7406d160 0x7eff740ab0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 >> 172.21.15.125:0/3962310890 conn(0x7eff7406a800 msgr2=0x7eff740b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:08.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 shutdown_connections 2024-01-16T14:57:08.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:08.389+0000 7eff73e00640 1 -- 172.21.15.125:0/3962310890 wait complete. 2024-01-16T14:57:08.390 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:08.390 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:08.391 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:57:08.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.391 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42620sec preferred_lft 42620sec 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T14:57:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:08.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.441 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:08.574 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T14:57:08.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.579 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T14:57:08.626 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T14:57:08.627 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:08.627 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:08.627 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:08.627 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:08.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:08.628 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:08.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.809 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T14:57:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T14:57:08.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.857 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list-id 2024-01-16T14:57:08.903 INFO:teuthology.orchestra.run.smithi125.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-16T14:57:08.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:08.904 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:08.904 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:08.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:08.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-16T14:57:08.904 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:09.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:09.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 86:19:14:05:de:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::8419:14ff:fe05:de37/64 scope link 2024-01-16T14:57:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:09.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 86:19:14:05:de:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::8419:14ff:fe05:de37/64 scope link 2024-01-16T14:57:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:09.271 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-16T14:57:09.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-16T14:57:09.271 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:09.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:09.518 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:57:09.519 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:57:09.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.1 up 2024-01-16T14:57:09.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:57:09.519 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:57:09.679 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-16T14:57:09.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:09.680 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:09.726 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:57:09.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:09.726 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:09.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:57:09.902 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:09.952 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:09.953 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:09.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:09.954 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:57:10.003 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T14:57:10.003 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-01-16T14:57:10.003 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-16T14:57:10.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:57:10.078 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:57:10.118 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:10.119 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:10.120 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:10.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:10.121 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:57:10.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi125.stderr:2024-01-16T14:57:10.203+0000 7f82bb038480 -1 init, newargv = 0x55eca2c4ac20 newargc=15 2024-01-16T14:57:10.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi125.stderr:ceph-fuse[45010]: starting ceph client 2024-01-16T14:57:10.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi125.stderr:ceph-fuse[45010]: starting fuse 2024-01-16T14:57:10.205 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T14:57:10.205 INFO:teuthology.orchestra.run.smithi125.stdout:42 2024-01-16T14:57:10.205 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 42] 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T14:57:10.206 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.2")) 2024-01-16T14:57:10.207 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T14:57:10.403 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.2.45010.asok 2024-01-16T14:57:10.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:10.420 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45010.asok 2024-01-16T14:57:10.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:10.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45010.asok status 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "2", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "45010", 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T14:57:10.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4707, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 4707 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1158443196 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:57:10.659 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1158443196 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.4707 192.168.0.2:0/1158443196", 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.2:0/1158443196", 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 8, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 20, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T14:57:10.660 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:57:10.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:10.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.895+0000 7f5c35e00640 1 Processor -- start 2024-01-16T14:57:10.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.895+0000 7f5c35e00640 1 -- start start 2024-01-16T14:57:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.896+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.896+0000 7f5c35e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c300cfc30 con 0x7f5c3015e3b0 2024-01-16T14:57:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.896+0000 7f5c35e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c300b1ee0 con 0x7f5c300cf7d0 2024-01-16T14:57:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.896+0000 7f5c35e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300b2780 con 0x7f5c30143ae0 2024-01-16T14:57:10.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.897+0000 7f5c2f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5c3015e3b0 0x7f5c300702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59762/0 (socket says 172.21.15.125:59762) 2024-01-16T14:57:10.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.897+0000 7f5c2f400640 1 -- 172.21.15.125:0/3182847216 learned_addr learned my addr 172.21.15.125:0/3182847216 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:10.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.897+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.897+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753931897 0 0) 0x7f5c300cfc30 con 0x7f5c3015e3b0 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.897+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c10003520 con 0x7f5c3015e3b0 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004817611 0 0) 0x7f5c10003520 con 0x7f5c3015e3b0 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 msgr2=0x7f5c300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 >> v1:172.21.15.137:6789/0 conn(0x7f5c300cf7d0 legacy=0x7f5c3015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3015e990 con 0x7f5c3015e3b0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5c280019f0 con 0x7f5c3015e3b0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 >> v1:172.21.15.125:6789/0 conn(0x7f5c3015e3b0 legacy=0x7f5c300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c100039a0 0x7f5c10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c10004d00 0x7f5c10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300b2780 con 0x7f5c100039a0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1000ba30 con 0x7f5c10004d00 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1000bbf0 con 0x7f5c30143ae0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c100039a0 0x7f5c10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2e000640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2f400640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c10004d00 0x7f5c10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 -- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 msgr2=0x7f5c100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 -- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c10004d00 msgr2=0x7f5c10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c10004d00 0x7f5c10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2ea00640 1 -- 172.21.15.125:0/3182847216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c10003520 con 0x7f5c100039a0 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2e000640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c30143ae0 0x7f5c100047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.898+0000 7f5c2f400640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c10004d00 0x7f5c10005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.899+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c100039a0 0x7f5c10003dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009a90 tx=0x7f5c100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f86d34f9b20882 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2002c680 con 0x7f5c100039a0 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2002c7e0 con 0x7f5c100039a0 2024-01-16T14:57:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c2d600640 1 -- 172.21.15.125:0/3182847216 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2002cb70 con 0x7f5c100039a0 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 -- 172.21.15.125:0/3182847216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c100039a0 msgr2=0x7f5c10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 --2- 172.21.15.125:0/3182847216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c100039a0 0x7f5c10003dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009a90 tx=0x7f5c100057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c2f400640 1 -- 172.21.15.125:0/3182847216 reap_dead start 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 -- 172.21.15.125:0/3182847216 shutdown_connections 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 -- 172.21.15.125:0/3182847216 >> 172.21.15.125:0/3182847216 conn(0x7f5c3006a800 msgr2=0x7f5c3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 -- 172.21.15.125:0/3182847216 shutdown_connections 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.900+0000 7f5c35e00640 1 -- 172.21.15.125:0/3182847216 wait complete. 2024-01-16T14:57:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.902+0000 7f5c35e00640 1 Processor -- start 2024-01-16T14:57:10.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.902+0000 7f5c35e00640 1 -- start start 2024-01-16T14:57:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c100039a0 0x7f5c300ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c300acd50 0x7f5c30159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300adf30 con 0x7f5c300ac8d0 2024-01-16T14:57:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300ae350 con 0x7f5c100039a0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c35e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300ad2b0 con 0x7f5c300acd50 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c2e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c2e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35892/0 (socket says 172.21.15.125:35892) 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.903+0000 7f5c2e000640 1 -- 172.21.15.125:0/2967447218 learned_addr learned my addr 172.21.15.125:0/2967447218 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2f400640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c300acd50 0x7f5c30159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 -- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c300acd50 msgr2=0x7f5c30159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c100039a0 0x7f5c300ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c300acd50 0x7f5c30159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 -- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c100039a0 msgr2=0x7f5c300ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c100039a0 0x7f5c300ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 -- 172.21.15.125:0/2967447218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300ad590 con 0x7f5c300ac8d0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c2e000640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800d7a0 tx=0x7f5c1800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18017070 con 0x7f5c300ac8d0 2024-01-16T14:57:10.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18005070 con 0x7f5c300ac8d0 2024-01-16T14:57:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18005420 con 0x7f5c300ac8d0 2024-01-16T14:57:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.904+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c300ad870 con 0x7f5c300ac8d0 2024-01-16T14:57:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.905+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c300ce630 con 0x7f5c300ac8d0 2024-01-16T14:57:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.906+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c300675d0 con 0x7f5c300ac8d0 2024-01-16T14:57:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.906+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5c18007030 con 0x7f5c300ac8d0 2024-01-16T14:57:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.907+0000 7f5c27400640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 0x7f5c0806d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.907+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c180893c0 con 0x7f5c300ac8d0 2024-01-16T14:57:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.908+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 0x7f5c0806d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.908+0000 7f5c2ea00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 0x7f5c0806d7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c20036000 tx=0x7f5c20030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:10.912+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c18054220 con 0x7f5c300ac8d0 2024-01-16T14:57:11.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.143+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5c3006d190 con 0x7f5c300ac8d0 2024-01-16T14:57:11.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.143+0000 7f5c27400640 1 -- 172.21.15.125:0/2967447218 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5c18057840 con 0x7f5c300ac8d0 2024-01-16T14:57:11.126 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:11.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 msgr2=0x7f5c0806d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 0x7f5c0806d7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c20036000 tx=0x7f5c20030000 comp rx=0 tx=0).stop 2024-01-16T14:57:11.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 msgr2=0x7f5c301594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800d7a0 tx=0x7f5c1800dc60 comp rx=0 tx=0).stop 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 shutdown_connections 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c300acd50 0x7f5c30159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c0806b2e0 0x7f5c0806d7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300ac8d0 0x7f5c301594f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 --2- 172.21.15.125:0/2967447218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c100039a0 0x7f5c300ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 >> 172.21.15.125:0/2967447218 conn(0x7f5c3006a800 msgr2=0x7f5c3015c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 shutdown_connections 2024-01-16T14:57:11.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:11.146+0000 7f5c35e00640 1 -- 172.21.15.125:0/2967447218 wait complete. 2024-01-16T14:57:11.147 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:11.148 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:57:11.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:11.148 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:11.205 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42607sec preferred_lft 42607sec 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T14:57:11.206 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:11.207 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:57:11.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:11.207 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:11.207 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:11.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:57:11.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr flush dev ceph-brx 2024-01-16T14:57:11.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set ceph-brx up 2024-01-16T14:57:11.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:57:11.208 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:11.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:11.654 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:11.739 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T14:57:11.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:11.743 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T14:57:11.799 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T14:57:11.799 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:11.799 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:11.800 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:11.800 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:11.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:11.800 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:12.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:12.493 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T14:57:12.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:12.548 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-01-16T14:57:12.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:12.598 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:12.598 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:12.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:12.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-16T14:57:12.598 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:12.736 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-16T14:57:12.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-16T14:57:12.736 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:13.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:13.012 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:13.012 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:13.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T14:57:13.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:57:13.012 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:13.168 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T14:57:13.169 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:13.169 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:13.217 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:57:13.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:13.217 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:13.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:57:13.401 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:57:13.452 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:57:13.453 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:13.454 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:13.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:13.455 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:57:13.493 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:57:13.493 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T14:57:13.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:57:13.618 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:57:13.658 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:13.659 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:13.660 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:13.660 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:13.660 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:57:13.660 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:13.660 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:13.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:13.661 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:57:13.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T14:57:13.755+0000 7ff15de3a480 -1 init, newargv = 0x55b9f8f54ac0 newargc=15 2024-01-16T14:57:13.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[41693]: starting ceph client 2024-01-16T14:57:13.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[41693]: starting fuse 2024-01-16T14:57:14.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:57:14.945 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:57:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:14.999 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:15.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:15.001 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:57:15.050 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T14:57:15.050 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T14:57:15.051 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T14:57:15.052 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T14:57:15.227 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.41693.asok 2024-01-16T14:57:15.231 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:57:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:15.245 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41693.asok 2024-01-16T14:57:15.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:15.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41693.asok status 2024-01-16T14:57:15.480 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T14:57:15.480 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T14:57:15.480 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:15.480 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "41693", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4531, 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4531 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:15.481 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1198467744 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1198467744 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4531 192.168.0.1:0/1198467744", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/1198467744", 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 8, 2024-01-16T14:57:15.482 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 20, 2024-01-16T14:57:15.483 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:15.483 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T14:57:15.483 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T14:57:15.483 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T14:57:15.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:15.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.733+0000 7fc00f200640 1 Processor -- start 2024-01-16T14:57:15.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.734+0000 7fc00f200640 1 -- start start 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc008143c40 0x7fc008071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080717d0 0x7fc0080cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc00815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0080723c0 con 0x7fc0080717d0 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc008070e50 con 0x7fc008143c40 2024-01-16T14:57:15.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080cce40 con 0x7fc0080cca40 2024-01-16T14:57:15.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.735+0000 7fc00d400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080717d0 0x7fc0080cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080717d0 0x7fc0080cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35896/0 (socket says 172.21.15.125:35896) 2024-01-16T14:57:15.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 -- 172.21.15.125:0/2608566622 learned_addr learned my addr 172.21.15.125:0/2608566622 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00de00640 1 --2- 172.21.15.125:0/2608566622 >> v2:172.21.15.137:3300/0 conn(0x7fc008143c40 0x7fc008071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00e800640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc00815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 -- 172.21.15.125:0/2608566622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 msgr2=0x7fc00815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc00815eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 -- 172.21.15.125:0/2608566622 >> v2:172.21.15.137:3300/0 conn(0x7fc008143c40 msgr2=0x7fc008071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:15.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 --2- 172.21.15.125:0/2608566622 >> v2:172.21.15.137:3300/0 conn(0x7fc008143c40 0x7fc008071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.736+0000 7fc00d400640 1 -- 172.21.15.125:0/2608566622 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc008144020 con 0x7fc0080717d0 2024-01-16T14:57:15.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.737+0000 7fc00d400640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080717d0 0x7fc0080cc500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc00806e410 tx=0x7fbff002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea8be7c964429f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:15.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00ca00640 1 -- 172.21.15.125:0/2608566622 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00317e0 con 0x7fc0080717d0 2024-01-16T14:57:15.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00ca00640 1 -- 172.21.15.125:0/2608566622 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0031940 con 0x7fc0080717d0 2024-01-16T14:57:15.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00ca00640 1 -- 172.21.15.125:0/2608566622 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0031cd0 con 0x7fc0080717d0 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 -- 172.21.15.125:0/2608566622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080717d0 msgr2=0x7fc0080cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080717d0 0x7fc0080cc500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc00806e410 tx=0x7fbff002b880 comp rx=0 tx=0).stop 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 -- 172.21.15.125:0/2608566622 shutdown_connections 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc00815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 --2- 172.21.15.125:0/2608566622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080717d0 0x7fc0080cc500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 --2- 172.21.15.125:0/2608566622 >> v2:172.21.15.137:3300/0 conn(0x7fc008143c40 0x7fc008071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.738+0000 7fc00f200640 1 -- 172.21.15.125:0/2608566622 >> 172.21.15.125:0/2608566622 conn(0x7fc00806a800 msgr2=0x7fc00806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.739+0000 7fc00f200640 1 -- 172.21.15.125:0/2608566622 shutdown_connections 2024-01-16T14:57:15.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.739+0000 7fc00f200640 1 -- 172.21.15.125:0/2608566622 wait complete. 2024-01-16T14:57:15.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.740+0000 7fc00f200640 1 Processor -- start 2024-01-16T14:57:15.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.740+0000 7fc00f200640 1 -- start start 2024-01-16T14:57:15.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 0x7fc00815aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc0080ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080cad10 con 0x7fc008143c40 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080c5c00 con 0x7fc0080717d0 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.741+0000 7fc00f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080ca750 con 0x7fc0080cca40 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35910/0 (socket says 172.21.15.125:35910) 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 -- 172.21.15.125:0/1324360649 learned_addr learned my addr 172.21.15.125:0/1324360649 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00de00640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 0x7fc00815aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00d400640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc0080ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 -- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 msgr2=0x7fc0080ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc0080ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 -- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 msgr2=0x7fc00815aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 0x7fc00815aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 -- 172.21.15.125:0/1324360649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00094d0 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00de00640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 0x7fc00815aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.742+0000 7fc00e800640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc000007e70 tx=0x7fc00000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.743+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000014620 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.743+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0000147a0 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.743+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000014b50 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.743+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0080caa30 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.743+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0080c4160 con 0x7fc008143c40 2024-01-16T14:57:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.744+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0080c4350 con 0x7fc008143c40 2024-01-16T14:57:15.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.745+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc000014cb0 con 0x7fc008143c40 2024-01-16T14:57:15.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.746+0000 7fc006000640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 0x7fbfe006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:15.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.747+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc00008a660 con 0x7fc008143c40 2024-01-16T14:57:15.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.747+0000 7fc00de00640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 0x7fbfe006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:15.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.747+0000 7fc00de00640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 0x7fbfe006d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006fd0 tx=0x7fbff8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:15.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.751+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc000055430 con 0x7fc008143c40 2024-01-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.980+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc008067450 con 0x7fc008143c40 2024-01-16T14:57:15.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.981+0000 7fc006000640 1 -- 172.21.15.125:0/1324360649 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc000058a50 con 0x7fc008143c40 2024-01-16T14:57:15.963 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 msgr2=0x7fbfe006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 0x7fbfe006d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006fd0 tx=0x7fbff8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 msgr2=0x7fc0080c56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc000007e70 tx=0x7fc00000e4b0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 shutdown_connections 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbfe006b2c0 0x7fbfe006d780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc008143c40 0x7fc0080c56c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0080cca40 0x7fc0080ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 --2- 172.21.15.125:0/1324360649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0080717d0 0x7fc00815aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.983+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 >> 172.21.15.125:0/1324360649 conn(0x7fc00806a800 msgr2=0x7fc0080acae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.984+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 shutdown_connections 2024-01-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:15.984+0000 7fc00f200640 1 -- 172.21.15.125:0/1324360649 wait complete. 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:15.986 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:57:15.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:15.987 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.044 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:16.045 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T14:57:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.047 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:16.132 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T14:57:16.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.137 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:16.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.189 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:16.190 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:16.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:16.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:16.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:16.190 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:16.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.367 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T14:57:16.417 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T14:57:16.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.418 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-01-16T14:57:16.466 INFO:teuthology.orchestra.run.smithi137.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-16T14:57:16.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.467 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:16.467 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:16.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:16.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-16T14:57:16.467 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:16.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether ca:99:13:5d:97:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::c899:13ff:fe5d:972d/64 scope link 2024-01-16T14:57:16.687 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:57:16.784 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:16.784 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:16.784 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:16.784 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether ca:99:13:5d:97:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::c899:13ff:fe5d:972d/64 scope link 2024-01-16T14:57:16.785 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:16.789 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-16T14:57:16.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-16T14:57:16.790 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:17.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:17.042 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:57:17.042 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:57:17.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.1 up 2024-01-16T14:57:17.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:57:17.043 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:57:17.210 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-16T14:57:17.210 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:17.210 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:17.257 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:57:17.257 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:17.257 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:17.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:57:17.441 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:57:17.495 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:17.496 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:17.497 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:17.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:17.498 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:57:17.549 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T14:57:17.549 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-16T14:57:17.549 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-16T14:57:17.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:57:17.630 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:17.672 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:57:17.673 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:17.674 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:17.675 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:17.675 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:17.675 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:17.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:17.676 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:57:17.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:2024-01-16T14:57:17.753+0000 7f93cde39480 -1 init, newargv = 0x55db4b70cc10 newargc=15 2024-01-16T14:57:17.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:ceph-fuse[42213]: starting ceph client 2024-01-16T14:57:17.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:ceph-fuse[42213]: starting fuse 2024-01-16T14:57:17.759 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T14:57:17.759 INFO:teuthology.orchestra.run.smithi137.stdout:42 2024-01-16T14:57:17.759 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-16T14:57:17.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:17.759 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T14:57:17.759 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T14:57:17.759 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T14:57:17.760 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.3")) 2024-01-16T14:57:17.761 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T14:57:17.955 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.3.42213.asok 2024-01-16T14:57:17.961 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:17.976 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.42213.asok 2024-01-16T14:57:17.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:17.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.42213.asok status 2024-01-16T14:57:18.209 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "3", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "42213", 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4540, 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T14:57:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4540 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 201046018 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 201046018 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4540 192.168.0.2:0/201046018", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.2:0/201046018", 2024-01-16T14:57:18.211 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 8, 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 20, 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T14:57:18.212 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T14:57:18.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.224 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T14:57:18.224 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:18.282 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T14:57:18.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:18.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:18.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.385 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T14:57:18.385 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:18.438 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T14:57:18.438 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:18.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:18.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.532 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T14:57:18.532 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:18.586 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T14:57:18.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:18.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:18.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.679 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T14:57:18.679 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:18.734 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T14:57:18.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:18.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:18.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:18.827 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-16T14:57:18.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:19.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.084+0000 7f8a4c000640 1 Processor -- start 2024-01-16T14:57:19.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.085+0000 7f8a4c000640 1 -- start start 2024-01-16T14:57:19.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a4c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a4415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a4c000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a44070c40 con 0x7f8a440b3ae0 2024-01-16T14:57:19.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a4c000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4406c470 con 0x7f8a440cf290 2024-01-16T14:57:19.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a4c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44071450 con 0x7f8a440ae5a0 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a48c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8a440b3ae0 0x7f8a4406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50672/0 (socket says 172.21.15.125:50672) 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.086+0000 7f8a48c00640 1 -- 172.21.15.125:0/3869336042 learned_addr learned my addr 172.21.15.125:0/3869336042 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a49600640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a4415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990257811 0 0) 0x7f8a44070c40 con 0x7f8a440b3ae0 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a24003520 con 0x7f8a440b3ae0 2024-01-16T14:57:19.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3322080960 0 0) 0x7f8a24003520 con 0x7f8a440b3ae0 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 msgr2=0x7f8a4415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a4415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 >> v1:172.21.15.137:6789/0 conn(0x7f8a440cf290 legacy=0x7f8a440b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4406c750 con 0x7f8a440b3ae0 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8a2c001670 con 0x7f8a440b3ae0 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.087+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 >> v1:172.21.15.125:6789/0 conn(0x7f8a440b3ae0 legacy=0x7f8a4406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a240039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a24003f20 0x7f8a240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24004d00 0x7f8a24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44071450 con 0x7f8a24003f20 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2400baf0 con 0x7f8a24004d00 2024-01-16T14:57:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2400bcb0 con 0x7f8a440ae5a0 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a24003f20 0x7f8a240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a49600640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a240039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a4a000640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24004d00 0x7f8a24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 -- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 msgr2=0x7f8a240039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a240039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 -- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24004d00 msgr2=0x7f8a24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24004d00 0x7f8a24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a48c00640 1 -- 172.21.15.125:0/3869336042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24003520 con 0x7f8a24003f20 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a49600640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a440ae5a0 0x7f8a240039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.088+0000 7f8a4a000640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24004d00 0x7f8a24005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:19.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.090+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a24003f20 0x7f8a240047c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00c210 tx=0x7f8a240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecc9533f78021a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.090+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c005ad0 con 0x7f8a24003f20 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.090+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c005c30 con 0x7f8a24003f20 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.090+0000 7f8a43e00640 1 -- 172.21.15.125:0/3869336042 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c02cad0 con 0x7f8a24003f20 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 -- 172.21.15.125:0/3869336042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a24003f20 msgr2=0x7f8a240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3869336042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a24003f20 0x7f8a240047c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00c210 tx=0x7f8a240057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4a000640 1 -- 172.21.15.125:0/3869336042 reap_dead start 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 -- 172.21.15.125:0/3869336042 shutdown_connections 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 -- 172.21.15.125:0/3869336042 >> 172.21.15.125:0/3869336042 conn(0x7f8a4406a800 msgr2=0x7f8a4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 -- 172.21.15.125:0/3869336042 shutdown_connections 2024-01-16T14:57:19.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.091+0000 7f8a4c000640 1 -- 172.21.15.125:0/3869336042 wait complete. 2024-01-16T14:57:19.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.093+0000 7f8a4c000640 1 Processor -- start 2024-01-16T14:57:19.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.093+0000 7f8a4c000640 1 -- start start 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24003f20 0x7f8a44156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 0x7f8a44160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441618f0 con 0x7f8a44161470 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441571f0 con 0x7f8a24003f20 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44157e90 con 0x7f8a44158f30 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52174/0 (socket says 172.21.15.125:52174) 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a4a000640 1 -- 172.21.15.125:0/3069717609 learned_addr learned my addr 172.21.15.125:0/3069717609 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.094+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 0x7f8a44160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 -- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 msgr2=0x7f8a44160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a49600640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24003f20 0x7f8a44156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 0x7f8a44160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 -- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24003f20 msgr2=0x7f8a44156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24003f20 0x7f8a44156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 -- 172.21.15.125:0/3069717609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a44158170 con 0x7f8a44161470 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a48c00640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 0x7f8a44160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4a000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000f80 tx=0x7f8a3c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c006d50 con 0x7f8a44161470 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c003970 con 0x7f8a44161470 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c003d60 con 0x7f8a44161470 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a44158450 con 0x7f8a44161470 2024-01-16T14:57:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.095+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a44158a10 con 0x7f8a44161470 2024-01-16T14:57:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.097+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a440675d0 con 0x7f8a44161470 2024-01-16T14:57:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.098+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a3c005070 con 0x7f8a44161470 2024-01-16T14:57:19.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.099+0000 7f8a41600640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 0x7f8a1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.099+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a3c089920 con 0x7f8a44161470 2024-01-16T14:57:19.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.099+0000 7f8a49600640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 0x7f8a1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.100+0000 7f8a49600640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 0x7f8a1c06d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a340099b0 tx=0x7f8a34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.103+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a3c054790 con 0x7f8a44161470 2024-01-16T14:57:19.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.294+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a4415de30 con 0x7f8a44161470 2024-01-16T14:57:19.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.297+0000 7f8a41600640 1 -- 172.21.15.125:0/3069717609 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f8a3c057db0 con 0x7f8a44161470 2024-01-16T14:57:19.280 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:19.280 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:19.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.299+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 msgr2=0x7f8a1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.299+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 0x7f8a1c06d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a340099b0 tx=0x7f8a34008040 comp rx=0 tx=0).stop 2024-01-16T14:57:19.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 msgr2=0x7f8a440cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000f80 tx=0x7f8a3c00dc00 comp rx=0 tx=0).stop 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 shutdown_connections 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a1c06b2c0 0x7f8a1c06d780 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a44161470 0x7f8a440cc9c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a44158f30 0x7f8a44160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 --2- 172.21.15.125:0/3069717609 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a24003f20 0x7f8a44156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 >> 172.21.15.125:0/3069717609 conn(0x7f8a4406a800 msgr2=0x7f8a440ad5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 shutdown_connections 2024-01-16T14:57:19.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.300+0000 7f8a4c000640 1 -- 172.21.15.125:0/3069717609 wait complete. 2024-01-16T14:57:19.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:19.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:19.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.535+0000 7f6b56400640 1 Processor -- start 2024-01-16T14:57:19.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.536+0000 7f6b56400640 1 -- start start 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b56400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 0x7f6b50164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b56400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500b1ee0 con 0x7f6b50143ae0 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b56400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500b2780 con 0x7f6b500cf9a0 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b56400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5015e990 con 0x7f6b5015e3b0 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b4f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6b50143ae0 0x7f6b500cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50676/0 (socket says 172.21.15.125:50676) 2024-01-16T14:57:19.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.537+0000 7f6b4f400640 1 -- 172.21.15.125:0/3603832381 learned_addr learned my addr 172.21.15.125:0/3603832381 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:19.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4fe00640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 0x7f6b50164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817381653 0 0) 0x7f6b500b1ee0 con 0x7f6b50143ae0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b30003520 con 0x7f6b50143ae0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611927613 0 0) 0x7f6b30003520 con 0x7f6b50143ae0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 msgr2=0x7f6b50164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 0x7f6b50164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 >> v1:172.21.15.137:6789/0 conn(0x7f6b500cf9a0 legacy=0x7f6b5015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b500b2130 con 0x7f6b50143ae0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6b40001670 con 0x7f6b50143ae0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 >> v1:172.21.15.125:6789/0 conn(0x7f6b50143ae0 legacy=0x7f6b500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b300039a0 0x7f6b30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 0x7f6b300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b30004d00 0x7f6b30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5015e990 con 0x7f6b300039a0 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.538+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3000ba60 con 0x7f6b30004d00 2024-01-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3000de30 con 0x7f6b5015e3b0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b300039a0 0x7f6b30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 -- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 msgr2=0x7f6b300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b5015e3b0 0x7f6b300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 -- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b30004d00 msgr2=0x7f6b30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b30004d00 0x7f6b30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.539+0000 7f6b4f400640 1 -- 172.21.15.125:0/3603832381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30003520 con 0x7f6b300039a0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.540+0000 7f6b4f400640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b300039a0 0x7f6b30003dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b40007fd0 tx=0x7f6b300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0e5e9b1492b606 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.540+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40005ad0 con 0x7f6b300039a0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.540+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40005c30 con 0x7f6b300039a0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.540+0000 7f6b4ea00640 1 -- 172.21.15.125:0/3603832381 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4002c870 con 0x7f6b300039a0 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 -- 172.21.15.125:0/3603832381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b300039a0 msgr2=0x7f6b30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 --2- 172.21.15.125:0/3603832381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b300039a0 0x7f6b30003dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b40007fd0 tx=0x7f6b300057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b4fe00640 1 -- 172.21.15.125:0/3603832381 reap_dead start 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 -- 172.21.15.125:0/3603832381 shutdown_connections 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 -- 172.21.15.125:0/3603832381 >> 172.21.15.125:0/3603832381 conn(0x7f6b5006a800 msgr2=0x7f6b5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 -- 172.21.15.125:0/3603832381 shutdown_connections 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.541+0000 7f6b56400640 1 -- 172.21.15.125:0/3603832381 wait complete. 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.543+0000 7f6b56400640 1 Processor -- start 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.543+0000 7f6b56400640 1 -- start start 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b300039a0 0x7f6b500ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 0x7f6b500cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b50137c90 con 0x7f6b50159560 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b50159f20 con 0x7f6b300039a0 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b56400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b50137e20 con 0x7f6b500af560 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52194/0 (socket says 172.21.15.125:52194) 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b4fe00640 1 -- 172.21.15.125:0/2347546425 learned_addr learned my addr 172.21.15.125:0/2347546425 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b47e00640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 0x7f6b500cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.544+0000 7f6b4f400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b300039a0 0x7f6b500ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 -- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 msgr2=0x7f6b500cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 0x7f6b500cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 -- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b300039a0 msgr2=0x7f6b500ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b300039a0 0x7f6b500ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 -- 172.21.15.125:0/2347546425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b500cd0e0 con 0x7f6b50159560 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b47e00640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 0x7f6b500cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b4fe00640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b500cc3c0 tx=0x7f6b4800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010040 con 0x7f6b50159560 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4800bb10 con 0x7f6b50159560 2024-01-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b500cd3c0 con 0x7f6b50159560 2024-01-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48014400 con 0x7f6b50159560 2024-01-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.545+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b500cd980 con 0x7f6b50159560 2024-01-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.546+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b50137fd0 con 0x7f6b50159560 2024-01-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.547+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6b48014560 con 0x7f6b50159560 2024-01-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.548+0000 7f6b47400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 0x7f6b2806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.548+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b48089fa0 con 0x7f6b50159560 2024-01-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.548+0000 7f6b4f400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 0x7f6b2806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.549+0000 7f6b4f400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 0x7f6b2806d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6b40034000 tx=0x7f6b40009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.552+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b50137fd0 con 0x7f6b50159560 2024-01-16T14:57:19.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.741+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b50137f90 con 0x7f6b50159560 2024-01-16T14:57:19.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.742+0000 7f6b47400640 1 -- 172.21.15.125:0/2347546425 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b48058770 con 0x7f6b50159560 2024-01-16T14:57:19.725 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 msgr2=0x7f6b2806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 0x7f6b2806d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6b40034000 tx=0x7f6b40009030 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 msgr2=0x7f6b501599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b500cc3c0 tx=0x7f6b4800b480 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 shutdown_connections 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b2806b2c0 0x7f6b2806d780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b50159560 0x7f6b501599e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b500af560 0x7f6b500cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 --2- 172.21.15.125:0/2347546425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b300039a0 0x7f6b500ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 >> 172.21.15.125:0/2347546425 conn(0x7f6b5006a800 msgr2=0x7f6b500cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 shutdown_connections 2024-01-16T14:57:19.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.745+0000 7f6b56400640 1 -- 172.21.15.125:0/2347546425 wait complete. 2024-01-16T14:57:19.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:19.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.973+0000 7f5b02000640 1 Processor -- start 2024-01-16T14:57:19.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.973+0000 7f5b02000640 1 -- start start 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5afc0adf20 0x7f5afc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5afc15bf10 0x7f5afc0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc0cc980 0x7f5afc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc15f970 con 0x7f5afc0adf20 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc15f4e0 con 0x7f5afc15bf10 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b02000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0cce10 con 0x7f5afc0cc980 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5afc0adf20 0x7f5afc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5afc0adf20 0x7f5afc070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52198/0 (socket says 172.21.15.125:52198) 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 -- 172.21.15.125:0/103310983 learned_addr learned my addr 172.21.15.125:0/103310983 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b01600640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc0cc980 0x7f5afc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5afbe00640 1 --2- 172.21.15.125:0/103310983 >> v2:172.21.15.137:3300/0 conn(0x7f5afc15bf10 0x7f5afc0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 -- 172.21.15.125:0/103310983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc0cc980 msgr2=0x7f5afc143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc0cc980 0x7f5afc143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 -- 172.21.15.125:0/103310983 >> v2:172.21.15.137:3300/0 conn(0x7f5afc15bf10 msgr2=0x7f5afc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.975+0000 7f5b00c00640 1 --2- 172.21.15.125:0/103310983 >> v2:172.21.15.137:3300/0 conn(0x7f5afc15bf10 0x7f5afc0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.976+0000 7f5b00c00640 1 -- 172.21.15.125:0/103310983 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc071130 con 0x7f5afc0adf20 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5b00c00640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0adf20 0x7f5afc070a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5afc15e9e0 tx=0x7f5ae402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec61fbfff84a9c9b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5afb400640 1 -- 172.21.15.125:0/103310983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40317e0 con 0x7f5afc0adf20 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5afb400640 1 -- 172.21.15.125:0/103310983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4031940 con 0x7f5afc0adf20 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5afb400640 1 -- 172.21.15.125:0/103310983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4031cd0 con 0x7f5afc0adf20 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5b02000640 1 -- 172.21.15.125:0/103310983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0adf20 msgr2=0x7f5afc070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.977+0000 7f5b02000640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0adf20 0x7f5afc070a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5afc15e9e0 tx=0x7f5ae402b880 comp rx=0 tx=0).stop 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 -- 172.21.15.125:0/103310983 shutdown_connections 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc0cc980 0x7f5afc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 --2- 172.21.15.125:0/103310983 >> v2:172.21.15.137:3300/0 conn(0x7f5afc15bf10 0x7f5afc0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 --2- 172.21.15.125:0/103310983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0adf20 0x7f5afc070a30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 -- 172.21.15.125:0/103310983 >> 172.21.15.125:0/103310983 conn(0x7f5afc06a7d0 msgr2=0x7f5afc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 -- 172.21.15.125:0/103310983 shutdown_connections 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.978+0000 7f5b02000640 1 -- 172.21.15.125:0/103310983 wait complete. 2024-01-16T14:57:19.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.979+0000 7f5b02000640 1 Processor -- start 2024-01-16T14:57:19.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.979+0000 7f5b02000640 1 -- start start 2024-01-16T14:57:19.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5afc0adf20 0x7f5afc0c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc15bf10 0x7f5afc0b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0ab4e0 con 0x7f5afc0cc980 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0aa4d0 con 0x7f5afc0adf20 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.980+0000 7f5b02000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0aa720 con 0x7f5afc15bf10 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52210/0 (socket says 172.21.15.125:52210) 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 -- 172.21.15.125:0/3904069416 learned_addr learned my addr 172.21.15.125:0/3904069416 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5b01600640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc15bf10 0x7f5afc0b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 -- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc15bf10 msgr2=0x7f5afc0b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5b00c00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5afc0adf20 0x7f5afc0c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc15bf10 0x7f5afc0b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 -- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5afc0adf20 msgr2=0x7f5afc0c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5afc0adf20 0x7f5afc0c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 -- 172.21.15.125:0/3904069416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40094d0 con 0x7f5afc0cc980 2024-01-16T14:57:19.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.981+0000 7f5afbe00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00c7f0 tx=0x7f5aec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.982+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec004220 con 0x7f5afc0cc980 2024-01-16T14:57:19.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.982+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec0043a0 con 0x7f5afc0cc980 2024-01-16T14:57:19.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.982+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec007860 con 0x7f5afc0cc980 2024-01-16T14:57:19.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.982+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc0aaa00 con 0x7f5afc0cc980 2024-01-16T14:57:19.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.982+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc0aafc0 con 0x7f5afc0cc980 2024-01-16T14:57:19.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.983+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0ab1e0 con 0x7f5afc0cc980 2024-01-16T14:57:19.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.984+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5aec007bd0 con 0x7f5afc0cc980 2024-01-16T14:57:19.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.985+0000 7f5af8c00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 0x7f5ad406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.985+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5aec089c50 con 0x7f5afc0cc980 2024-01-16T14:57:19.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.985+0000 7f5b00c00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 0x7f5ad406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.986+0000 7f5b00c00640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 0x7f5ad406d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4036000 tx=0x7f5ae403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:19.990+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5aec0549a0 con 0x7f5afc0cc980 2024-01-16T14:57:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.177+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5afc067450 con 0x7f5afc0cc980 2024-01-16T14:57:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.178+0000 7f5af8c00640 1 -- 172.21.15.125:0/3904069416 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5aec057fc0 con 0x7f5afc0cc980 2024-01-16T14:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:20.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 msgr2=0x7f5ad406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 0x7f5ad406d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4036000 tx=0x7f5ae403d040 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 msgr2=0x7f5afc0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00c7f0 tx=0x7f5aec00ccb0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 shutdown_connections 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5afc15bf10 0x7f5afc0b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ad406b2c0 0x7f5ad406d780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5afc0cc980 0x7f5afc0ab670 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 --2- 172.21.15.125:0/3904069416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5afc0adf20 0x7f5afc0c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 >> 172.21.15.125:0/3904069416 conn(0x7f5afc06a7d0 msgr2=0x7f5afc0aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 shutdown_connections 2024-01-16T14:57:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.181+0000 7f5b02000640 1 -- 172.21.15.125:0/3904069416 wait complete. 2024-01-16T14:57:20.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:20.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.409+0000 7f87ea800640 1 Processor -- start 2024-01-16T14:57:20.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.409+0000 7f87ea800640 1 -- start start 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.410+0000 7f87ea800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e40adf20 0x7f87e4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.410+0000 7f87ea800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87e40ccb50 0x7f87e4143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.410+0000 7f87ea800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e415f4e0 con 0x7f87e415bf10 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.410+0000 7f87ea800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e415f970 con 0x7f87e40ccb50 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.410+0000 7f87ea800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4071130 con 0x7f87e40adf20 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e8a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f87e415bf10 0x7f87e40cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50682/0 (socket says 172.21.15.125:50682) 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e8a00640 1 -- 172.21.15.125:0/3942420217 learned_addr learned my addr 172.21.15.125:0/3942420217 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9e00640 1 --2- 172.21.15.125:0/3942420217 >> v2:172.21.15.125:3300/0 conn(0x7f87e40ccb50 0x7f87e4143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9400640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e40adf20 0x7f87e4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9e00640 1 -- 172.21.15.125:0/3942420217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e40adf20 msgr2=0x7f87e4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9e00640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e40adf20 0x7f87e4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9e00640 1 -- 172.21.15.125:0/3942420217 >> v1:172.21.15.125:6789/0 conn(0x7f87e415bf10 legacy=0x7f87e40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e9e00640 1 -- 172.21.15.125:0/3942420217 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e415c390 con 0x7f87e40ccb50 2024-01-16T14:57:20.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.411+0000 7f87e3e00640 1 -- 172.21.15.125:0/3942420217 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931599301 0 0) 0x7f87e415f4e0 con 0x7f87e415bf10 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.412+0000 7f87e9e00640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e4143d30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009d90 tx=0x7f87dc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20078f40207dcca7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87e3e00640 1 -- 172.21.15.125:0/3942420217 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc02cbc0 con 0x7f87e40ccb50 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87e3e00640 1 -- 172.21.15.125:0/3942420217 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc02cd20 con 0x7f87e40ccb50 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87e3e00640 1 -- 172.21.15.125:0/3942420217 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc034660 con 0x7f87e40ccb50 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 -- 172.21.15.125:0/3942420217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 msgr2=0x7f87e4143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e4143d30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009d90 tx=0x7f87dc02be10 comp rx=0 tx=0).stop 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 -- 172.21.15.125:0/3942420217 shutdown_connections 2024-01-16T14:57:20.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e4143d30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 --2- 172.21.15.125:0/3942420217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e40adf20 0x7f87e4070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 -- 172.21.15.125:0/3942420217 >> 172.21.15.125:0/3942420217 conn(0x7f87e406a7d0 msgr2=0x7f87e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 -- 172.21.15.125:0/3942420217 shutdown_connections 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.413+0000 7f87ea800640 1 -- 172.21.15.125:0/3942420217 wait complete. 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.415+0000 7f87ea800640 1 Processor -- start 2024-01-16T14:57:20.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.415+0000 7f87ea800640 1 -- start start 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87e40adf20 0x7f87e41f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e415bf10 0x7f87e41f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e41f60c0 con 0x7f87e40ccb50 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e41f33b0 con 0x7f87e40adf20 2024-01-16T14:57:20.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87ea800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e41f3690 con 0x7f87e415bf10 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87e8a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.416+0000 7f87e8a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52226/0 (socket says 172.21.15.125:52226) 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 -- 172.21.15.125:0/2062481914 learned_addr learned my addr 172.21.15.125:0/2062481914 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e9400640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87e40adf20 0x7f87e41f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 -- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e415bf10 msgr2=0x7f87e41f68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e415bf10 0x7f87e41f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 -- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87e40adf20 msgr2=0x7f87e41f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87e40adf20 0x7f87e41f2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 -- 172.21.15.125:0/2062481914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc009a70 con 0x7f87e40ccb50 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e8a00640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00c430 tx=0x7f87cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0074e0 con 0x7f87e40ccb50 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc007660 con 0x7f87e40ccb50 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc004a20 con 0x7f87e40ccb50 2024-01-16T14:57:20.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.417+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e41f3970 con 0x7f87e40ccb50 2024-01-16T14:57:20.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.418+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e41f3030 con 0x7f87e40ccb50 2024-01-16T14:57:20.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.419+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e41f65c0 con 0x7f87e40ccb50 2024-01-16T14:57:20.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.420+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f87cc004b80 con 0x7f87e40ccb50 2024-01-16T14:57:20.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.420+0000 7f87e1600640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 0x7f87bc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.421+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87cc089a10 con 0x7f87e40ccb50 2024-01-16T14:57:20.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.421+0000 7f87e9400640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 0x7f87bc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.421+0000 7f87e9400640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 0x7f87bc06d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006fd0 tx=0x7f87d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.425+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87e41f65c0 con 0x7f87e40ccb50 2024-01-16T14:57:20.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.614+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87e41f65c0 con 0x7f87e40ccb50 2024-01-16T14:57:20.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.615+0000 7f87e1600640 1 -- 172.21.15.125:0/2062481914 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87e41f65c0 con 0x7f87e40ccb50 2024-01-16T14:57:20.597 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:20.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.617+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 msgr2=0x7f87bc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.617+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 0x7f87bc06d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006fd0 tx=0x7f87d4008040 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 msgr2=0x7f87e40af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00c430 tx=0x7f87cc00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 shutdown_connections 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87e415bf10 0x7f87e41f68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87bc06b2c0 0x7f87bc06d780 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e40ccb50 0x7f87e40af7b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 --2- 172.21.15.125:0/2062481914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87e40adf20 0x7f87e41f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 >> 172.21.15.125:0/2062481914 conn(0x7f87e406a7d0 msgr2=0x7f87e415b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 shutdown_connections 2024-01-16T14:57:20.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.618+0000 7f87ea800640 1 -- 172.21.15.125:0/2062481914 wait complete. 2024-01-16T14:57:20.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.836+0000 7f7400c00640 1 Processor -- start 2024-01-16T14:57:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.837+0000 7f7400c00640 1 -- start start 2024-01-16T14:57:20.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.838+0000 7f7400c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f73fc15afb0 0x7f73fc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.838+0000 7f7400c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 0x7f73fc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.838+0000 7f7400c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc13ff30 con 0x7f73fc13cfa0 2024-01-16T14:57:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.838+0000 7f7400c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc0cdc40 con 0x7f73fc15afb0 2024-01-16T14:57:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.838+0000 7f7400c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0cdec0 con 0x7f73fc13fae0 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fb400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f73fc13cfa0 0x7f73fc0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:45862/0 (socket says 172.21.15.125:45862) 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fb400640 1 -- 172.21.15.125:0/2850429701 learned_addr learned my addr 172.21.15.125:0/2850429701 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fbe00640 1 --2- 172.21.15.125:0/2850429701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 0x7f73fc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73faa00640 1 --2- 172.21.15.125:0/2850429701 >> v2:172.21.15.137:3300/0 conn(0x7f73fc15afb0 0x7f73fc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fa000640 1 -- 172.21.15.125:0/2850429701 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308167345 0 0) 0x7f73ec001110 con 0x7f73fc13cfa0 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fa000640 1 -- 172.21.15.125:0/2850429701 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73dc003520 con 0x7f73fc13cfa0 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fbe00640 1 -- 172.21.15.125:0/2850429701 >> v2:172.21.15.137:3300/0 conn(0x7f73fc15afb0 msgr2=0x7f73fc13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fbe00640 1 --2- 172.21.15.125:0/2850429701 >> v2:172.21.15.137:3300/0 conn(0x7f73fc15afb0 0x7f73fc13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fbe00640 1 -- 172.21.15.125:0/2850429701 >> v1:172.21.15.137:6789/0 conn(0x7f73fc13cfa0 legacy=0x7f73fc0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.839+0000 7f73fbe00640 1 -- 172.21.15.125:0/2850429701 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0ce1a0 con 0x7f73fc13fae0 2024-01-16T14:57:20.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.841+0000 7f73fbe00640 1 --2- 172.21.15.125:0/2850429701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 0x7f73fc0b2040 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009d90 tx=0x7f73f402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=626815c8353d50ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.841+0000 7f73fa000640 1 -- 172.21.15.125:0/2850429701 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f402cbc0 con 0x7f73fc13fae0 2024-01-16T14:57:20.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.841+0000 7f73fa000640 1 -- 172.21.15.125:0/2850429701 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f402cd20 con 0x7f73fc13fae0 2024-01-16T14:57:20.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.841+0000 7f73fa000640 1 -- 172.21.15.125:0/2850429701 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4034640 con 0x7f73fc13fae0 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 -- 172.21.15.125:0/2850429701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 msgr2=0x7f73fc0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 --2- 172.21.15.125:0/2850429701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 0x7f73fc0b2040 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009d90 tx=0x7f73f402be10 comp rx=0 tx=0).stop 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 -- 172.21.15.125:0/2850429701 shutdown_connections 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 --2- 172.21.15.125:0/2850429701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc13fae0 0x7f73fc0b2040 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 --2- 172.21.15.125:0/2850429701 >> v2:172.21.15.137:3300/0 conn(0x7f73fc15afb0 0x7f73fc13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 -- 172.21.15.125:0/2850429701 >> 172.21.15.125:0/2850429701 conn(0x7f73fc06a800 msgr2=0x7f73fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 -- 172.21.15.125:0/2850429701 shutdown_connections 2024-01-16T14:57:20.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.842+0000 7f7400c00640 1 -- 172.21.15.125:0/2850429701 wait complete. 2024-01-16T14:57:20.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.844+0000 7f7400c00640 1 Processor -- start 2024-01-16T14:57:20.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.844+0000 7f7400c00640 1 -- start start 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc13fae0 0x7f73fc1f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15afb0 0x7f73fc139fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc137090 con 0x7f73fc13cfa0 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc137960 con 0x7f73fc13fae0 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f7400c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc137bb0 con 0x7f73fc15afb0 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f73fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f73fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52242/0 (socket says 172.21.15.125:52242) 2024-01-16T14:57:20.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f73fb400640 1 -- 172.21.15.125:0/2902498654 learned_addr learned my addr 172.21.15.125:0/2902498654 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.845+0000 7f73fbe00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15afb0 0x7f73fc139fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73faa00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc13fae0 0x7f73fc1f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 -- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15afb0 msgr2=0x7f73fc139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15afb0 0x7f73fc139fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 -- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc13fae0 msgr2=0x7f73fc1f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc13fae0 0x7f73fc1f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 -- 172.21.15.125:0/2902498654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4009a70 con 0x7f73fc13cfa0 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73fb400640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00c730 tx=0x7f73ec00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec005ab0 con 0x7f73fc13cfa0 2024-01-16T14:57:20.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec005c10 con 0x7f73fc13cfa0 2024-01-16T14:57:20.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec00ce70 con 0x7f73fc13cfa0 2024-01-16T14:57:20.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc137e90 con 0x7f73fc13cfa0 2024-01-16T14:57:20.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.846+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc138450 con 0x7f73fc13cfa0 2024-01-16T14:57:20.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.848+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc067450 con 0x7f73fc13cfa0 2024-01-16T14:57:20.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.848+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f73ec007070 con 0x7f73fc13cfa0 2024-01-16T14:57:20.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.850+0000 7f73f3400640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 0x7f73d406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.850+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73ec088f80 con 0x7f73fc13cfa0 2024-01-16T14:57:20.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.850+0000 7f73faa00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 0x7f73d406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.851+0000 7f73faa00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 0x7f73d406d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f73fc15a950 tx=0x7f73e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:20.854+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73ec0bbe90 con 0x7f73fc13cfa0 2024-01-16T14:57:21.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.042+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73fc13ff30 con 0x7f73fc13cfa0 2024-01-16T14:57:21.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.046+0000 7f73f3400640 1 -- 172.21.15.125:0/2902498654 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f73fc13ff30 con 0x7f73fc13cfa0 2024-01-16T14:57:21.028 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:21.029 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 msgr2=0x7f73d406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 0x7f73d406d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f73fc15a950 tx=0x7f73e4008040 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 msgr2=0x7f73fc1f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00c730 tx=0x7f73ec00cb30 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 shutdown_connections 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15afb0 0x7f73fc139fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc13fae0 0x7f73fc1f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f73d406b2c0 0x7f73d406d780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 --2- 172.21.15.125:0/2902498654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73fc13cfa0 0x7f73fc1f7570 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 >> 172.21.15.125:0/2902498654 conn(0x7f73fc06a800 msgr2=0x7f73fc13c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 shutdown_connections 2024-01-16T14:57:21.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.048+0000 7f7400c00640 1 -- 172.21.15.125:0/2902498654 wait complete. 2024-01-16T14:57:21.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:21.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:21.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.279+0000 7f15b2a00640 1 Processor -- start 2024-01-16T14:57:21.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.279+0000 7f15b2a00640 1 -- start start 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f15ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15b2a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f15ac15f9c0 0x7f15ac0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15b2a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac15c5f0 con 0x7f15ac13df80 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15b2a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac06c470 con 0x7f15ac15f9c0 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac06c750 con 0x7f15ac13e360 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15abe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f15ac13df80 0x7f15ac15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50694/0 (socket says 172.21.15.125:50694) 2024-01-16T14:57:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15abe00640 1 -- 172.21.15.125:0/793038358 learned_addr learned my addr 172.21.15.125:0/793038358 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.281+0000 7f15ab400640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f15ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15b0a00640 1 --2- 172.21.15.125:0/793038358 >> v2:172.21.15.137:3300/0 conn(0x7f15ac15f9c0 0x7f15ac0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186440438 0 0) 0x7f15ac15c5f0 con 0x7f15ac13df80 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158c003520 con 0x7f15ac13df80 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507971194 0 0) 0x7f158c003520 con 0x7f15ac13df80 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 msgr2=0x7f15ac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f15ac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 >> v2:172.21.15.137:3300/0 conn(0x7f15ac15f9c0 msgr2=0x7f15ac0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 --2- 172.21.15.125:0/793038358 >> v2:172.21.15.137:3300/0 conn(0x7f15ac15f9c0 0x7f15ac0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac06ca30 con 0x7f15ac13df80 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f159c001670 con 0x7f15ac13df80 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 >> v1:172.21.15.125:6789/0 conn(0x7f15ac13df80 legacy=0x7f15ac15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f158c003970 0x7f158c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f158c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c004d00 0x7f158c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac06c470 con 0x7f158c003970 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac06c750 con 0x7f158c004d00 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.282+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c00b980 con 0x7f15ac13e360 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f158c003970 0x7f158c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15ab400640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f158c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 -- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 msgr2=0x7f158c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15b0a00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c004d00 0x7f158c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f158c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 -- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c004d00 msgr2=0x7f158c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c004d00 0x7f158c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15abe00640 1 -- 172.21.15.125:0/793038358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c003520 con 0x7f158c003970 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15b0a00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c004d00 0x7f158c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.283+0000 7f15ab400640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac13e360 0x7f158c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.284+0000 7f15abe00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f158c003970 0x7f158c003d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c2c0 tx=0x7f158c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7a878802c6263a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.284+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c005ad0 con 0x7f158c003970 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.284+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c005c50 con 0x7f158c003970 2024-01-16T14:57:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.284+0000 7f15aaa00640 1 -- 172.21.15.125:0/793038358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c02c8b0 con 0x7f158c003970 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 -- 172.21.15.125:0/793038358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f158c003970 msgr2=0x7f158c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 --2- 172.21.15.125:0/793038358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f158c003970 0x7f158c003d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c2c0 tx=0x7f158c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b0a00640 1 -- 172.21.15.125:0/793038358 reap_dead start 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 -- 172.21.15.125:0/793038358 shutdown_connections 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 -- 172.21.15.125:0/793038358 >> 172.21.15.125:0/793038358 conn(0x7f15ac06a800 msgr2=0x7f15ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 -- 172.21.15.125:0/793038358 shutdown_connections 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.285+0000 7f15b2a00640 1 -- 172.21.15.125:0/793038358 wait complete. 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.287+0000 7f15b2a00640 1 Processor -- start 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.287+0000 7f15b2a00640 1 -- start start 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 0x7f15ac0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac0a84c0 0x7f15ac0a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0a8e80 con 0x7f15ac12fbd0 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac130f30 con 0x7f158c003970 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac130050 con 0x7f15ac0a84c0 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52258/0 (socket says 172.21.15.125:52258) 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.288+0000 7f15ab400640 1 -- 172.21.15.125:0/1492074925 learned_addr learned my addr 172.21.15.125:0/1492074925 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:21.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 -- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac0a84c0 msgr2=0x7f15ac0a8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15b0a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac0a84c0 0x7f15ac0a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac0a84c0 0x7f15ac0a8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15abe00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 0x7f15ac0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 -- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 msgr2=0x7f15ac0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 0x7f15ac0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 -- 172.21.15.125:0/1492074925 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0ae2a0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15abe00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 0x7f15ac0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15ab400640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f15940108b0 tx=0x7f1594010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400eb60 con 0x7f15ac12fbd0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400ecc0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac130320 con 0x7f15ac12fbd0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.289+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159401c400 con 0x7f15ac12fbd0 2024-01-16T14:57:21.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.290+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac1308e0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.291+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac0675d0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.291+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f159401c560 con 0x7f15ac12fbd0 2024-01-16T14:57:21.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.293+0000 7f15a3e00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 0x7f158406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.293+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f159408a320 con 0x7f15ac12fbd0 2024-01-16T14:57:21.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.293+0000 7f15abe00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 0x7f158406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.294+0000 7f15abe00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 0x7f158406d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f159c034000 tx=0x7f159c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.298+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15940551a0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.487+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15ac06df00 con 0x7f15ac12fbd0 2024-01-16T14:57:21.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.487+0000 7f15a3e00640 1 -- 172.21.15.125:0/1492074925 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f15940587c0 con 0x7f15ac12fbd0 2024-01-16T14:57:21.470 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 msgr2=0x7f158406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 0x7f158406d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f159c034000 tx=0x7f159c007000 comp rx=0 tx=0).stop 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 msgr2=0x7f15ac0add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f15940108b0 tx=0x7f1594010d70 comp rx=0 tx=0).stop 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 shutdown_connections 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15ac0a84c0 0x7f15ac0a8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f158406b2c0 0x7f158406d780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15ac12fbd0 0x7f15ac0add60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 --2- 172.21.15.125:0/1492074925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f158c003970 0x7f15ac0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.490+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 >> 172.21.15.125:0/1492074925 conn(0x7f15ac06a800 msgr2=0x7f15ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.491+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 shutdown_connections 2024-01-16T14:57:21.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.491+0000 7f15b2a00640 1 -- 172.21.15.125:0/1492074925 wait complete. 2024-01-16T14:57:21.492 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:21.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.721+0000 7fc96da00640 1 Processor -- start 2024-01-16T14:57:21.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.721+0000 7fc96da00640 1 -- start start 2024-01-16T14:57:21.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.722+0000 7fc96da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813df80 0x7fc96815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.722+0000 7fc96da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc96813e360 0x7fc9680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc96da00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96806c470 con 0x7fc96815f9c0 2024-01-16T14:57:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc96da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96806c750 con 0x7fc96813e360 2024-01-16T14:57:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc96da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96806ca30 con 0x7fc96813df80 2024-01-16T14:57:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc95fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc96813e360 0x7fc9680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc95fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc96813e360 0x7fc9680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52270/0 (socket says 172.21.15.125:52270) 2024-01-16T14:57:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc95fe00640 1 -- 172.21.15.125:0/715868443 learned_addr learned my addr 172.21.15.125:0/715868443 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:21.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.723+0000 7fc966a00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813df80 0x7fc96815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.724+0000 7fc95fe00640 1 -- 172.21.15.125:0/715868443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813df80 msgr2=0x7fc96815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.724+0000 7fc95fe00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813df80 0x7fc96815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.724+0000 7fc95fe00640 1 -- 172.21.15.125:0/715868443 >> v1:172.21.15.137:6789/0 conn(0x7fc96815f9c0 legacy=0x7fc9681408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.724+0000 7fc95fe00640 1 -- 172.21.15.125:0/715868443 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96806cd10 con 0x7fc96813e360 2024-01-16T14:57:21.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.725+0000 7fc95fe00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813e360 0x7fc9680cf290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc950009870 tx=0x7fc95002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1047d3fa763cc843 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.725+0000 7fc966000640 1 -- 172.21.15.125:0/715868443 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950030840 con 0x7fc96813e360 2024-01-16T14:57:21.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.725+0000 7fc966000640 1 -- 172.21.15.125:0/715868443 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9500309a0 con 0x7fc96813e360 2024-01-16T14:57:21.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.725+0000 7fc966000640 1 -- 172.21.15.125:0/715868443 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950030d30 con 0x7fc96813e360 2024-01-16T14:57:21.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 -- 172.21.15.125:0/715868443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813e360 msgr2=0x7fc9680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813e360 0x7fc9680cf290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc950009870 tx=0x7fc95002b980 comp rx=0 tx=0).stop 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 -- 172.21.15.125:0/715868443 shutdown_connections 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813e360 0x7fc9680cf290 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 --2- 172.21.15.125:0/715868443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813df80 0x7fc96815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 -- 172.21.15.125:0/715868443 >> 172.21.15.125:0/715868443 conn(0x7fc96806a800 msgr2=0x7fc96806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 -- 172.21.15.125:0/715868443 shutdown_connections 2024-01-16T14:57:21.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.726+0000 7fc96da00640 1 -- 172.21.15.125:0/715868443 wait complete. 2024-01-16T14:57:21.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.728+0000 7fc96da00640 1 Processor -- start 2024-01-16T14:57:21.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.728+0000 7fc96da00640 1 -- start start 2024-01-16T14:57:21.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813e360 0x7fc9681495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc96815f9c0 0x7fc96814bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681405e0 con 0x7fc96813df80 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681408c0 con 0x7fc96815f9c0 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc96da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc968140ba0 con 0x7fc96813e360 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc966a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc966a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52282/0 (socket says 172.21.15.125:52282) 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc966a00640 1 -- 172.21.15.125:0/2026789210 learned_addr learned my addr 172.21.15.125:0/2026789210 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc95fe00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813e360 0x7fc9681495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.729+0000 7fc967400640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc96815f9c0 0x7fc96814bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 -- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813e360 msgr2=0x7fc9681495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813e360 0x7fc9681495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 -- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc96815f9c0 msgr2=0x7fc96814bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc96815f9c0 0x7fc96814bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 -- 172.21.15.125:0/2026789210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500094d0 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc966a00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc95800d5d0 tx=0x7fc95800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580125f0 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc958012770 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc968140e80 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958012b20 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.730+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96814d5b0 con 0x7fc96813df80 2024-01-16T14:57:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.731+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9681413a0 con 0x7fc96813df80 2024-01-16T14:57:21.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.733+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc958012c80 con 0x7fc96813df80 2024-01-16T14:57:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.734+0000 7fc95ea00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 0x7fc94006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.734+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc95808a1c0 con 0x7fc96813df80 2024-01-16T14:57:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.734+0000 7fc95fe00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 0x7fc94006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.735+0000 7fc95fe00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 0x7fc94006d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc950036000 tx=0x7fc950008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.738+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9681413a0 con 0x7fc96813df80 2024-01-16T14:57:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.927+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9681413a0 con 0x7fc96813df80 2024-01-16T14:57:21.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.927+0000 7fc95ea00640 1 -- 172.21.15.125:0/2026789210 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc958054f90 con 0x7fc96813df80 2024-01-16T14:57:21.910 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 msgr2=0x7fc94006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 0x7fc94006d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc950036000 tx=0x7fc950008030 comp rx=0 tx=0).stop 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 msgr2=0x7fc96814d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc95800d5d0 tx=0x7fc95800da90 comp rx=0 tx=0).stop 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 shutdown_connections 2024-01-16T14:57:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc96815f9c0 0x7fc96814bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc96813e360 0x7fc9681495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc94006b2c0 0x7fc94006d780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 --2- 172.21.15.125:0/2026789210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc96813df80 0x7fc96814d070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.930+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 >> 172.21.15.125:0/2026789210 conn(0x7fc96806a800 msgr2=0x7fc96814aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.931+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 shutdown_connections 2024-01-16T14:57:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:21.931+0000 7fc96da00640 1 -- 172.21.15.125:0/2026789210 wait complete. 2024-01-16T14:57:21.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:22.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.159+0000 7ff4a1000640 1 Processor -- start 2024-01-16T14:57:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.159+0000 7ff4a1000640 1 -- start start 2024-01-16T14:57:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff4a1000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0cf9a0 0x7ff49c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff4a1000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 0x7ff49c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff4a1000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0b1ee0 con 0x7ff49c143ae0 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff4a1000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c0b2780 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff4a1000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c15e990 con 0x7ff49c15e1e0 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff493e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0cf9a0 0x7ff49c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff499e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff49c143ae0 0x7ff49c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50696/0 (socket says 172.21.15.125:50696) 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff493e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0cf9a0 0x7ff49c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52298/0 (socket says 172.21.15.125:52298) 2024-01-16T14:57:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.161+0000 7ff499e00640 1 -- 172.21.15.125:0/24802224 learned_addr learned my addr 172.21.15.125:0/24802224 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff49a800640 1 --2- 172.21.15.125:0/24802224 >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 0x7ff49c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff493e00640 1 -- 172.21.15.125:0/24802224 >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 msgr2=0x7ff49c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff493e00640 1 --2- 172.21.15.125:0/24802224 >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 0x7ff49c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff493e00640 1 -- 172.21.15.125:0/24802224 >> v1:172.21.15.125:6789/0 conn(0x7ff49c143ae0 legacy=0x7ff49c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff493e00640 1 -- 172.21.15.125:0/24802224 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c0b2130 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.162+0000 7ff49a800640 1 --2- 172.21.15.125:0/24802224 >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 0x7ff49c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.163+0000 7ff493e00640 1 --2- 172.21.15.125:0/24802224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c15dca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0ae9d0 tx=0x7ff48402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab88f9dcad9d855 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.163+0000 7ff499400640 1 -- 172.21.15.125:0/24802224 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484031830 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.163+0000 7ff499400640 1 -- 172.21.15.125:0/24802224 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff484031990 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.163+0000 7ff499400640 1 -- 172.21.15.125:0/24802224 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484031d20 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 -- 172.21.15.125:0/24802224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0cf9a0 msgr2=0x7ff49c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 --2- 172.21.15.125:0/24802224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c15dca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0ae9d0 tx=0x7ff48402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 -- 172.21.15.125:0/24802224 shutdown_connections 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 --2- 172.21.15.125:0/24802224 >> v2:172.21.15.137:3300/0 conn(0x7ff49c15e1e0 0x7ff49c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 --2- 172.21.15.125:0/24802224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c15dca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 -- 172.21.15.125:0/24802224 >> 172.21.15.125:0/24802224 conn(0x7ff49c06a800 msgr2=0x7ff49c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 -- 172.21.15.125:0/24802224 shutdown_connections 2024-01-16T14:57:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.164+0000 7ff4a1000640 1 -- 172.21.15.125:0/24802224 wait complete. 2024-01-16T14:57:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.166+0000 7ff4a1000640 1 Processor -- start 2024-01-16T14:57:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.166+0000 7ff4a1000640 1 -- start start 2024-01-16T14:57:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff49c143ae0 0x7ff49c0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c142270 con 0x7ff49c15e1e0 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0ccce0 con 0x7ff49c0cf9a0 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff4a1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c141cb0 con 0x7ff49c143ae0 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52312/0 (socket says 172.21.15.125:52312) 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 -- 172.21.15.125:0/4033401385 learned_addr learned my addr 172.21.15.125:0/4033401385 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 -- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff49c143ae0 msgr2=0x7ff49c0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff49c143ae0 0x7ff49c0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 -- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff49c0cf9a0 msgr2=0x7ff49c141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.167+0000 7ff49a800640 1 -- 172.21.15.125:0/4033401385 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484009400 con 0x7ff49c15e1e0 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff49a800640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff49400cf20 tx=0x7ff49400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49400d840 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49400d9a0 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49400dd50 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c141f90 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.168+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c0add60 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.169+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c0675d0 con 0x7ff49c15e1e0 2024-01-16T14:57:22.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.170+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff49401e3b0 con 0x7ff49c15e1e0 2024-01-16T14:57:22.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.171+0000 7ff492000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 0x7ff47406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.171+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff49408a6b0 con 0x7ff49c15e1e0 2024-01-16T14:57:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.171+0000 7ff499e00640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 0x7ff47406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.172+0000 7ff499e00640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 0x7ff47406d780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0015f0 tx=0x7ff48c007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.176+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff494055510 con 0x7ff49c15e1e0 2024-01-16T14:57:22.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.364+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff49c0adf50 con 0x7ff49c15e1e0 2024-01-16T14:57:22.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.365+0000 7ff492000640 1 -- 172.21.15.125:0/4033401385 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff49401a020 con 0x7ff49c15e1e0 2024-01-16T14:57:22.348 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:22.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.367+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 msgr2=0x7ff47406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.367+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 0x7ff47406d780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0015f0 tx=0x7ff48c007a40 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.367+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 msgr2=0x7ff49c0ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.367+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff49400cf20 tx=0x7ff49400a7d0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 shutdown_connections 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff47406b2c0 0x7ff47406d780 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c15e1e0 0x7ff49c0ae280 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff49c143ae0 0x7ff49c0cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 --2- 172.21.15.125:0/4033401385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff49c0cf9a0 0x7ff49c141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 >> 172.21.15.125:0/4033401385 conn(0x7ff49c06a800 msgr2=0x7ff49c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 shutdown_connections 2024-01-16T14:57:22.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.368+0000 7ff4a1000640 1 -- 172.21.15.125:0/4033401385 wait complete. 2024-01-16T14:57:22.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:22.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.595+0000 7f0310c00640 1 Processor -- start 2024-01-16T14:57:22.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.595+0000 7f0310c00640 1 -- start start 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.596+0000 7f0310c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 0x7f030c164d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.596+0000 7f0310c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c0cdc40 con 0x7f030c15afb0 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.596+0000 7f0310c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c0cdec0 con 0x7f030c13cfa0 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.596+0000 7f0310c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0ce1a0 con 0x7f030c13fcb0 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f0302800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f030c15afb0 0x7f030c13f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50702/0 (socket says 172.21.15.125:50702) 2024-01-16T14:57:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f0302800640 1 -- 172.21.15.125:0/755458430 learned_addr learned my addr 172.21.15.125:0/755458430 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030be00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 0x7f030c164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448101166 0 0) 0x7f030c0cdc40 con 0x7f030c15afb0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec003520 con 0x7f030c15afb0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636818603 0 0) 0x7f02ec003520 con 0x7f030c15afb0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 msgr2=0x7f030c164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 0x7f030c164d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 >> v1:172.21.15.137:6789/0 conn(0x7f030c13cfa0 legacy=0x7f030c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c0ce480 con 0x7f030c15afb0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f02f4001670 con 0x7f030c15afb0 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 >> v1:172.21.15.125:6789/0 conn(0x7f030c15afb0 legacy=0x7f030c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 0x7f02ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.597+0000 7f030aa00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02ec003f20 0x7f02ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030aa00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f02ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0ce1a0 con 0x7f02ec004d00 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec00bb10 con 0x7f02ec003f20 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec00dee0 con 0x7f030c13fcb0 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f02ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f0302800640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02ec003f20 0x7f02ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 -- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 msgr2=0x7f02ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c13fcb0 0x7f02ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 -- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02ec003f20 msgr2=0x7f02ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02ec003f20 0x7f02ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.598+0000 7f030be00640 1 -- 172.21.15.125:0/755458430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec003520 con 0x7f02ec004d00 2024-01-16T14:57:22.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.599+0000 7f030be00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f02ec005180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0304009fc0 tx=0x7f02ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae66610d4d44b67c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030402cc60 con 0x7f02ec004d00 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030400f040 con 0x7f02ec004d00 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f030aa00640 1 -- 172.21.15.125:0/755458430 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304034440 con 0x7f02ec004d00 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 -- 172.21.15.125:0/755458430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 msgr2=0x7f02ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 --2- 172.21.15.125:0/755458430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f02ec005180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0304009fc0 tx=0x7f02ec0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f030be00640 1 -- 172.21.15.125:0/755458430 reap_dead start 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 -- 172.21.15.125:0/755458430 shutdown_connections 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 -- 172.21.15.125:0/755458430 >> 172.21.15.125:0/755458430 conn(0x7f030c06a800 msgr2=0x7f030c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 -- 172.21.15.125:0/755458430 shutdown_connections 2024-01-16T14:57:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.600+0000 7f0310c00640 1 -- 172.21.15.125:0/755458430 wait complete. 2024-01-16T14:57:22.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.602+0000 7f0310c00640 1 Processor -- start 2024-01-16T14:57:22.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.602+0000 7f0310c00640 1 -- start start 2024-01-16T14:57:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c154ac0 0x7f030c1461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f030c148d10 0x7f030c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c139500 con 0x7f02ec004d00 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c139d10 con 0x7f030c148d10 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f0310c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c139750 con 0x7f030c154ac0 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f030b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f030b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52322/0 (socket says 172.21.15.125:52322) 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.603+0000 7f030b400640 1 -- 172.21.15.125:0/1346638350 learned_addr learned my addr 172.21.15.125:0/1346638350 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f0302800640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c154ac0 0x7f030c1461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030be00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f030c148d10 0x7f030c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 -- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c154ac0 msgr2=0x7f030c1461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c154ac0 0x7f030c1461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 -- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f030c148d10 msgr2=0x7f030c0b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f030c148d10 0x7f030c0b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 -- 172.21.15.125:0/1346638350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c139a30 con 0x7f02ec004d00 2024-01-16T14:57:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f030b400640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c480 tx=0x7f02fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0052d0 con 0x7f02ec004d00 2024-01-16T14:57:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc005430 con 0x7f02ec004d00 2024-01-16T14:57:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc012870 con 0x7f02ec004d00 2024-01-16T14:57:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.604+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c13e530 con 0x7f02ec004d00 2024-01-16T14:57:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.605+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c0b2480 con 0x7f02ec004d00 2024-01-16T14:57:22.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.606+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c067450 con 0x7f02ec004d00 2024-01-16T14:57:22.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.607+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f02fc0129d0 con 0x7f02ec004d00 2024-01-16T14:57:22.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.607+0000 7f0303e00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 0x7f02e406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.607+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02fc0896e0 con 0x7f02ec004d00 2024-01-16T14:57:22.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.608+0000 7f0302800640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 0x7f02e406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.608+0000 7f0302800640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 0x7f02e406d7b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02f4001c90 tx=0x7f02f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.612+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02fc0545e0 con 0x7f02ec004d00 2024-01-16T14:57:22.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.801+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f030c0cbff0 con 0x7f02ec004d00 2024-01-16T14:57:22.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.803+0000 7f0303e00640 1 -- 172.21.15.125:0/1346638350 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f02fc057c00 con 0x7f02ec004d00 2024-01-16T14:57:22.786 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:22.786 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:22.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.805+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 msgr2=0x7f02e406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 0x7f02e406d7b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02f4001c90 tx=0x7f02f4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 msgr2=0x7f030c1378e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c480 tx=0x7f02fc00c940 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 shutdown_connections 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f030c148d10 0x7f030c0b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f030c154ac0 0x7f030c1461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f02e406b2f0 0x7f02e406d7b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 --2- 172.21.15.125:0/1346638350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02ec004d00 0x7f030c1378e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 >> 172.21.15.125:0/1346638350 conn(0x7f030c06a800 msgr2=0x7f030c15a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 shutdown_connections 2024-01-16T14:57:22.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:22.806+0000 7f0310c00640 1 -- 172.21.15.125:0/1346638350 wait complete. 2024-01-16T14:57:22.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:22.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:23.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.039+0000 7fd6f0400640 1 Processor -- start 2024-01-16T14:57:23.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.040+0000 7fd6f0400640 1 -- start start 2024-01-16T14:57:23.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.041+0000 7fd6f0400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd6e815e3b0 0x7fd6e8164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.041+0000 7fd6f0400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e80b1ee0 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.041+0000 7fd6f0400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e80b2780 con 0x7fd6e8143ae0 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.041+0000 7fd6f0400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e815e990 con 0x7fd6e815e3b0 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6ed000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd6e80cf9a0 0x7fd6e815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50716/0 (socket says 172.21.15.125:50716) 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6ed000640 1 -- 172.21.15.125:0/4194071331 learned_addr learned my addr 172.21.15.125:0/4194071331 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6ee400640 1 --2- 172.21.15.125:0/4194071331 >> v2:172.21.15.137:3300/0 conn(0x7fd6e815e3b0 0x7fd6e8164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294293369 0 0) 0x7fd6e80b1ee0 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c8003520 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179584898 0 0) 0x7fd6c8003520 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 >> v2:172.21.15.137:3300/0 conn(0x7fd6e815e3b0 msgr2=0x7fd6e8164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 --2- 172.21.15.125:0/4194071331 >> v2:172.21.15.137:3300/0 conn(0x7fd6e815e3b0 0x7fd6e8164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 >> v1:172.21.15.137:6789/0 conn(0x7fd6e8143ae0 legacy=0x7fd6e80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.042+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80b2130 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd6d0001670 con 0x7fd6e80cf9a0 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 >> v1:172.21.15.125:6789/0 conn(0x7fd6e80cf9a0 legacy=0x7fd6e815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6c8003970 0x7fd6c8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6c8005180 0x7fd6c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e815e990 con 0x7fd6c80047c0 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c800c000 con 0x7fd6c8005180 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c800c1c0 con 0x7fd6c8003970 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6ed000640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6ee400640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6c8005180 0x7fd6c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6eda00640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6c8003970 0x7fd6c8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6ed000640 1 -- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6c8003970 msgr2=0x7fd6c8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6ed000640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6c8003970 0x7fd6c8003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.043+0000 7fd6ed000640 1 -- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6c8005180 msgr2=0x7fd6c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.044+0000 7fd6ed000640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6c8005180 0x7fd6c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.044+0000 7fd6ed000640 1 -- 172.21.15.125:0/4194071331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8003520 con 0x7fd6c80047c0 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.044+0000 7fd6ee400640 1 -- 172.21.15.125:0/4194071331 reap_dead start 2024-01-16T14:57:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6ed000640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6c8004c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0007f90 tx=0x7fd6c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1acb09fba4fb1b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d002ec80 con 0x7fd6c80047c0 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d002ee00 con 0x7fd6c80047c0 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6e7e00640 1 -- 172.21.15.125:0/4194071331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0005b10 con 0x7fd6c80047c0 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6f0400640 1 -- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 msgr2=0x7fd6c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.045+0000 7fd6f0400640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6c8004c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0007f90 tx=0x7fd6c80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.046+0000 7fd6f0400640 1 -- 172.21.15.125:0/4194071331 shutdown_connections 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.046+0000 7fd6f0400640 1 --2- 172.21.15.125:0/4194071331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6c8004c40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.046+0000 7fd6f0400640 1 -- 172.21.15.125:0/4194071331 >> 172.21.15.125:0/4194071331 conn(0x7fd6e806a800 msgr2=0x7fd6e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.046+0000 7fd6f0400640 1 -- 172.21.15.125:0/4194071331 shutdown_connections 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.046+0000 7fd6f0400640 1 -- 172.21.15.125:0/4194071331 wait complete. 2024-01-16T14:57:23.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.047+0000 7fd6f0400640 1 Processor -- start 2024-01-16T14:57:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.048+0000 7fd6f0400640 1 -- start start 2024-01-16T14:57:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e80ad850 0x7fd6e80ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e80ab6c0 0x7fd6e80aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e8159c10 con 0x7fd6c80047c0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80ae380 con 0x7fd6e80ad850 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6f0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80addc0 con 0x7fd6e80ab6c0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52342/0 (socket says 172.21.15.125:52342) 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 -- 172.21.15.125:0/2367366395 learned_addr learned my addr 172.21.15.125:0/2367366395 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 -- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e80ab6c0 msgr2=0x7fd6e80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e80ab6c0 0x7fd6e80aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 -- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e80ad850 msgr2=0x7fd6e80ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e80ad850 0x7fd6e80ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.049+0000 7fd6eda00640 1 -- 172.21.15.125:0/2367366395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80ae0a0 con 0x7fd6c80047c0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6eda00640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800e870 tx=0x7fd6d800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d800c850 con 0x7fd6c80047c0 2024-01-16T14:57:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d800c9b0 con 0x7fd6c80047c0 2024-01-16T14:57:23.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e80a8560 con 0x7fd6c80047c0 2024-01-16T14:57:23.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d800cd60 con 0x7fd6c80047c0 2024-01-16T14:57:23.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.050+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e80b04a0 con 0x7fd6c80047c0 2024-01-16T14:57:23.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.051+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e80675d0 con 0x7fd6c80047c0 2024-01-16T14:57:23.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.052+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd6d801f460 con 0x7fd6c80047c0 2024-01-16T14:57:23.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.053+0000 7fd6e5600640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 0x7fd6c006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.053+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d808a330 con 0x7fd6c80047c0 2024-01-16T14:57:23.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.054+0000 7fd6ed000640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 0x7fd6c006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.054+0000 7fd6ed000640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 0x7fd6c006d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0036000 tx=0x7fd6d003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.058+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6d8055100 con 0x7fd6c80047c0 2024-01-16T14:57:23.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.246+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6e80b3660 con 0x7fd6c80047c0 2024-01-16T14:57:23.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.247+0000 7fd6e5600640 1 -- 172.21.15.125:0/2367366395 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6d8058720 con 0x7fd6c80047c0 2024-01-16T14:57:23.230 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:23.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 msgr2=0x7fd6c006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 0x7fd6c006d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0036000 tx=0x7fd6d003b040 comp rx=0 tx=0).stop 2024-01-16T14:57:23.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 msgr2=0x7fd6e81596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800e870 tx=0x7fd6d800ed30 comp rx=0 tx=0).stop 2024-01-16T14:57:23.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 shutdown_connections 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.250+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6e80ab6c0 0x7fd6e80aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6e80ad850 0x7fd6e80ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd6c006b2c0 0x7fd6c006d780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 --2- 172.21.15.125:0/2367366395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80047c0 0x7fd6e81596d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 >> 172.21.15.125:0/2367366395 conn(0x7fd6e806a800 msgr2=0x7fd6e8142960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 shutdown_connections 2024-01-16T14:57:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.251+0000 7fd6f0400640 1 -- 172.21.15.125:0/2367366395 wait complete. 2024-01-16T14:57:23.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:23.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.481+0000 7f2f83c00640 1 Processor -- start 2024-01-16T14:57:23.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.481+0000 7f2f83c00640 1 -- start start 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f83c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2f7c13df80 0x7f2f7c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f83c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f83c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7c06c470 con 0x7f2f7c13e360 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f83c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c06c750 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f83c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c06ca30 con 0x7f2f7c13df80 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f81c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f81c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52350/0 (socket says 172.21.15.125:52350) 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f81c00640 1 -- 172.21.15.125:0/3251696043 learned_addr learned my addr 172.21.15.125:0/3251696043 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.483+0000 7f2f81200640 1 --2- 172.21.15.125:0/3251696043 >> v2:172.21.15.137:3300/0 conn(0x7f2f7c13df80 0x7f2f7c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.484+0000 7f2f81c00640 1 -- 172.21.15.125:0/3251696043 >> v2:172.21.15.137:3300/0 conn(0x7f2f7c13df80 msgr2=0x7f2f7c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.484+0000 7f2f81c00640 1 --2- 172.21.15.125:0/3251696043 >> v2:172.21.15.137:3300/0 conn(0x7f2f7c13df80 0x7f2f7c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.484+0000 7f2f81c00640 1 -- 172.21.15.125:0/3251696043 >> v1:172.21.15.137:6789/0 conn(0x7f2f7c13e360 legacy=0x7f2f7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.484+0000 7f2f81c00640 1 -- 172.21.15.125:0/3251696043 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c06cd10 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.485+0000 7f2f81c00640 1 --2- 172.21.15.125:0/3251696043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c072770 tx=0x7f2f7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949cdd66baa618c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.485+0000 7f2f7b400640 1 -- 172.21.15.125:0/3251696043 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7402ca80 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.485+0000 7f2f7b400640 1 -- 172.21.15.125:0/3251696043 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7402cbe0 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.485+0000 7f2f7b400640 1 -- 172.21.15.125:0/3251696043 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f740343f0 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 -- 172.21.15.125:0/3251696043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c0cf9a0 msgr2=0x7f2f7c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 --2- 172.21.15.125:0/3251696043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c072770 tx=0x7f2f7402bee0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 -- 172.21.15.125:0/3251696043 shutdown_connections 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 --2- 172.21.15.125:0/3251696043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c0cf9a0 0x7f2f7c0cd840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 --2- 172.21.15.125:0/3251696043 >> v2:172.21.15.137:3300/0 conn(0x7f2f7c13df80 0x7f2f7c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 -- 172.21.15.125:0/3251696043 >> 172.21.15.125:0/3251696043 conn(0x7f2f7c06a800 msgr2=0x7f2f7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 -- 172.21.15.125:0/3251696043 shutdown_connections 2024-01-16T14:57:23.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.486+0000 7f2f83c00640 1 -- 172.21.15.125:0/3251696043 wait complete. 2024-01-16T14:57:23.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.488+0000 7f2f83c00640 1 Processor -- start 2024-01-16T14:57:23.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.488+0000 7f2f83c00640 1 -- start start 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f7c0cf9a0 0x7f2f7c12f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f7c13e360 0x7f2f7c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c130a70 con 0x7f2f7c13df80 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c0ae2f0 con 0x7f2f7c13e360 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f83c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c12fe60 con 0x7f2f7c0cf9a0 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52356/0 (socket says 172.21.15.125:52356) 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 -- 172.21.15.125:0/1589393401 learned_addr learned my addr 172.21.15.125:0/1589393401 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:23.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f81200640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f7c0cf9a0 0x7f2f7c12f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f81c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f7c13e360 0x7f2f7c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 -- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f7c0cf9a0 msgr2=0x7f2f7c12f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f7c0cf9a0 0x7f2f7c12f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 -- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f7c13e360 msgr2=0x7f2f7c15df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f7c13e360 0x7f2f7c15df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.489+0000 7f2f7be00640 1 -- 172.21.15.125:0/1589393401 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f74009b30 con 0x7f2f7c13df80 2024-01-16T14:57:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f7be00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c400 tx=0x7f2f6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64011790 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64011930 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c130140 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64011ce0 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.490+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c130700 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.491+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c1308c0 con 0x7f2f7c13df80 2024-01-16T14:57:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.492+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2f6401c460 con 0x7f2f7c13df80 2024-01-16T14:57:23.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.493+0000 7f2f78c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 0x7f2f5406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.493+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f64089ba0 con 0x7f2f7c13df80 2024-01-16T14:57:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.493+0000 7f2f81200640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 0x7f2f5406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.494+0000 7f2f81200640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 0x7f2f5406d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006fd0 tx=0x7f2f6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.497+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f64054970 con 0x7f2f7c13df80 2024-01-16T14:57:23.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.687+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f7c0675d0 con 0x7f2f7c13df80 2024-01-16T14:57:23.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.688+0000 7f2f78c00640 1 -- 172.21.15.125:0/1589393401 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f64057f90 con 0x7f2f7c13df80 2024-01-16T14:57:23.670 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 msgr2=0x7f2f5406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 0x7f2f5406d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006fd0 tx=0x7f2f6c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 msgr2=0x7f2f7c0addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c400 tx=0x7f2f6400c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 shutdown_connections 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f7c13e360 0x7f2f7c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f5406b2f0 0x7f2f5406d7b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7c13df80 0x7f2f7c0addb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 --2- 172.21.15.125:0/1589393401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f7c0cf9a0 0x7f2f7c12f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.690+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 >> 172.21.15.125:0/1589393401 conn(0x7f2f7c06a800 msgr2=0x7f2f7c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.691+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 shutdown_connections 2024-01-16T14:57:23.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.691+0000 7f2f83c00640 1 -- 172.21.15.125:0/1589393401 wait complete. 2024-01-16T14:57:23.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:23.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.924+0000 7fb080400640 1 Processor -- start 2024-01-16T14:57:23.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.924+0000 7fb080400640 1 -- start start 2024-01-16T14:57:23.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb07815b960 0x7fb07815eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 0x7fb07806a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 0x7fb07806a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0780725b0 con 0x7fb07815b960 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07815fe00 con 0x7fb0780715e0 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb080400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780cc9f0 con 0x7fb078071a40 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb07da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb07815b960 0x7fb07815eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb07da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb07815b960 0x7fb07815eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52370/0 (socket says 172.21.15.125:52370) 2024-01-16T14:57:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.926+0000 7fb07da00640 1 -- 172.21.15.125:0/3488989956 learned_addr learned my addr 172.21.15.125:0/3488989956 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:23.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07d000640 1 --2- 172.21.15.125:0/3488989956 >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 0x7fb07806a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07da00640 1 -- 172.21.15.125:0/3488989956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 msgr2=0x7fb07806a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07e400640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 0x7fb07806a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07da00640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 0x7fb07806a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07da00640 1 -- 172.21.15.125:0/3488989956 >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 msgr2=0x7fb07806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07da00640 1 --2- 172.21.15.125:0/3488989956 >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 0x7fb07806a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07da00640 1 -- 172.21.15.125:0/3488989956 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07815f560 con 0x7fb07815b960 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07d000640 1 --2- 172.21.15.125:0/3488989956 >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 0x7fb07806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.927+0000 7fb07e400640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 0x7fb07806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:23.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.928+0000 7fb07da00640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07815b960 0x7fb07815eee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb07806c4d0 tx=0x7fb06802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb436532fe481d33 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.928+0000 7fb077e00640 1 -- 172.21.15.125:0/3488989956 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06802ca70 con 0x7fb07815b960 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.928+0000 7fb077e00640 1 -- 172.21.15.125:0/3488989956 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06802cbd0 con 0x7fb07815b960 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.928+0000 7fb077e00640 1 -- 172.21.15.125:0/3488989956 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0680343f0 con 0x7fb07815b960 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 -- 172.21.15.125:0/3488989956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07815b960 msgr2=0x7fb07815eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07815b960 0x7fb07815eee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb07806c4d0 tx=0x7fb06802bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 -- 172.21.15.125:0/3488989956 shutdown_connections 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb078071a40 0x7fb07806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 --2- 172.21.15.125:0/3488989956 >> v2:172.21.15.137:3300/0 conn(0x7fb0780715e0 0x7fb07806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 --2- 172.21.15.125:0/3488989956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07815b960 0x7fb07815eee0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 -- 172.21.15.125:0/3488989956 >> 172.21.15.125:0/3488989956 conn(0x7fb0780cff90 msgr2=0x7fb0780d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 -- 172.21.15.125:0/3488989956 shutdown_connections 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.929+0000 7fb080400640 1 -- 172.21.15.125:0/3488989956 wait complete. 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.931+0000 7fb080400640 1 Processor -- start 2024-01-16T14:57:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.931+0000 7fb080400640 1 -- start start 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0780715e0 0x7fb07813d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 0x7fb0780c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb078156b80 con 0x7fb078071a40 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780ca5e0 con 0x7fb07815b960 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb080400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07815ebf0 con 0x7fb0780715e0 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb07d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb07d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52384/0 (socket says 172.21.15.125:52384) 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.932+0000 7fb07d000640 1 -- 172.21.15.125:0/3279113611 learned_addr learned my addr 172.21.15.125:0/3279113611 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07da00640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0780715e0 0x7fb07813d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07e400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 0x7fb0780c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 -- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0780715e0 msgr2=0x7fb07813d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0780715e0 0x7fb07813d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 -- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 msgr2=0x7fb0780c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 0x7fb0780c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 -- 172.21.15.125:0/3279113611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068009b30 con 0x7fb078071a40 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07e400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 0x7fb0780c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb07d000640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb06000c7f0 tx=0x7fb06000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06000cea0 con 0x7fb078071a40 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb060007a30 con 0x7fb078071a40 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060007dc0 con 0x7fb078071a40 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.933+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0780cae40 con 0x7fb078071a40 2024-01-16T14:57:23.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.934+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0780ca7f0 con 0x7fb078071a40 2024-01-16T14:57:23.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.935+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0780cafd0 con 0x7fb078071a40 2024-01-16T14:57:23.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.935+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb0600113f0 con 0x7fb078071a40 2024-01-16T14:57:23.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.937+0000 7fb075600640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 0x7fb05006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.937+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb060089ce0 con 0x7fb078071a40 2024-01-16T14:57:23.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.937+0000 7fb07da00640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 0x7fb05006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.938+0000 7fb07da00640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 0x7fb05006d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb068035000 tx=0x7fb068030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:23.941+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb060054b40 con 0x7fb078071a40 2024-01-16T14:57:24.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.135+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0780675d0 con 0x7fb078071a40 2024-01-16T14:57:24.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.136+0000 7fb075600640 1 -- 172.21.15.125:0/3279113611 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb060058160 con 0x7fb078071a40 2024-01-16T14:57:24.118 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:24.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.138+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 msgr2=0x7fb05006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.138+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 0x7fb05006d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb068035000 tx=0x7fb068030000 comp rx=0 tx=0).stop 2024-01-16T14:57:24.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.138+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 msgr2=0x7fb078157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.138+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb06000c7f0 tx=0x7fb06000ccb0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 shutdown_connections 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb07815b960 0x7fb0780c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006b2c0 0x7fb05006d780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb078071a40 0x7fb078157f70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 --2- 172.21.15.125:0/3279113611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0780715e0 0x7fb07813d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 >> 172.21.15.125:0/3279113611 conn(0x7fb0780cff90 msgr2=0x7fb07815b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 shutdown_connections 2024-01-16T14:57:24.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.139+0000 7fb080400640 1 -- 172.21.15.125:0/3279113611 wait complete. 2024-01-16T14:57:24.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:24.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.363+0000 7f6d49800640 1 Processor -- start 2024-01-16T14:57:24.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.363+0000 7f6d49800640 1 -- start start 2024-01-16T14:57:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d49800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d4415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d49800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d440cf290 0x7f6d440cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d49800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d44070c40 con 0x7f6d440b3ae0 2024-01-16T14:57:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d49800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d44071450 con 0x7f6d440cf290 2024-01-16T14:57:24.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d49800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d440cfc60 con 0x7f6d440ae5a0 2024-01-16T14:57:24.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d3be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6d440b3ae0 0x7f6d4406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50724/0 (socket says 172.21.15.125:50724) 2024-01-16T14:57:24.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d3be00640 1 -- 172.21.15.125:0/4255420406 learned_addr learned my addr 172.21.15.125:0/4255420406 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d42a00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d4415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d43400640 1 --2- 172.21.15.125:0/4255420406 >> v2:172.21.15.137:3300/0 conn(0x7f6d440cf290 0x7f6d440cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.365+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680950002 0 0) 0x7f6d44070c40 con 0x7f6d440b3ae0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d24003520 con 0x7f6d440b3ae0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009247435 0 0) 0x7f6d24003520 con 0x7f6d440b3ae0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 msgr2=0x7f6d4415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d4415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 >> v2:172.21.15.137:3300/0 conn(0x7f6d440cf290 msgr2=0x7f6d440cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 --2- 172.21.15.125:0/4255420406 >> v2:172.21.15.137:3300/0 conn(0x7f6d440cf290 0x7f6d440cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4406c470 con 0x7f6d440b3ae0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6d2c001670 con 0x7f6d440b3ae0 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 >> v1:172.21.15.125:6789/0 conn(0x7f6d440b3ae0 legacy=0x7f6d4406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d24003970 0x7f6d24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d24004d00 0x7f6d24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44071450 con 0x7f6d24003970 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d440cfc60 con 0x7f6d24004d00 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2400b9b0 con 0x7f6d440ae5a0 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.366+0000 7f6d42a00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d24003970 0x7f6d24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d3be00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d43400640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d24004d00 0x7f6d24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d42a00640 1 -- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 msgr2=0x7f6d240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d42a00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d440ae5a0 0x7f6d240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d42a00640 1 -- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d24004d00 msgr2=0x7f6d24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d42a00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d24004d00 0x7f6d24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.367+0000 7f6d42a00640 1 -- 172.21.15.125:0/4255420406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24003520 con 0x7f6d24003970 2024-01-16T14:57:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.368+0000 7f6d42a00640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d24003970 0x7f6d24003d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009ac0 tx=0x7f6d240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c68c8b4d51c7fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.368+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3402c6b0 con 0x7f6d24003970 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.368+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3402c810 con 0x7f6d24003970 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.368+0000 7f6d42000640 1 -- 172.21.15.125:0/4255420406 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3402cba0 con 0x7f6d24003970 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 -- 172.21.15.125:0/4255420406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d24003970 msgr2=0x7f6d24003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 --2- 172.21.15.125:0/4255420406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d24003970 0x7f6d24003d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009ac0 tx=0x7f6d240057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d43400640 1 -- 172.21.15.125:0/4255420406 reap_dead start 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 -- 172.21.15.125:0/4255420406 shutdown_connections 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 -- 172.21.15.125:0/4255420406 >> 172.21.15.125:0/4255420406 conn(0x7f6d4406a800 msgr2=0x7f6d4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 -- 172.21.15.125:0/4255420406 shutdown_connections 2024-01-16T14:57:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.369+0000 7f6d49800640 1 -- 172.21.15.125:0/4255420406 wait complete. 2024-01-16T14:57:24.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.371+0000 7f6d49800640 1 Processor -- start 2024-01-16T14:57:24.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.371+0000 7f6d49800640 1 -- start start 2024-01-16T14:57:24.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d24003970 0x7f6d4414ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 0x7f6d4415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44158630 con 0x7f6d440cd4a0 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d441579b0 con 0x7f6d440cd920 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d49800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44157c90 con 0x7f6d24003970 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52404/0 (socket says 172.21.15.125:52404) 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.372+0000 7f6d3be00640 1 -- 172.21.15.125:0/446066155 learned_addr learned my addr 172.21.15.125:0/446066155 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d42a00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d24003970 0x7f6d4414ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d43400640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 0x7f6d4415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 -- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d24003970 msgr2=0x7f6d4414ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d24003970 0x7f6d4414ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 -- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 msgr2=0x7f6d4415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 0x7f6d4415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 -- 172.21.15.125:0/446066155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44157f70 con 0x7f6d440cd4a0 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d43400640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 0x7f6d4415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:24.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3be00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00e700 tx=0x7f6d2c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c00ee10 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c00c8d0 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d44158250 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.373+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c00cc80 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.374+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d441f2db0 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.375+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d440675d0 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.375+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6d2c01d460 con 0x7f6d440cd4a0 2024-01-16T14:57:24.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.376+0000 7f6d3aa00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 0x7f6d1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.376+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d2c089b30 con 0x7f6d440cd4a0 2024-01-16T14:57:24.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.377+0000 7f6d42a00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 0x7f6d1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.377+0000 7f6d42a00640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 0x7f6d1c06d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009ac0 tx=0x7f6d34030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.381+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d2c058420 con 0x7f6d440cd4a0 2024-01-16T14:57:24.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.572+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d440b1670 con 0x7f6d440cd4a0 2024-01-16T14:57:24.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.574+0000 7f6d3aa00640 1 -- 172.21.15.125:0/446066155 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f6d2c001560 con 0x7f6d440cd4a0 2024-01-16T14:57:24.557 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:24.557 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":20,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:56:46.942901+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:24.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 msgr2=0x7f6d1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 0x7f6d1c06d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009ac0 tx=0x7f6d34030000 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 msgr2=0x7f6d4415c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00e700 tx=0x7f6d2c00ebc0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 shutdown_connections 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d440cd920 0x7f6d4415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1c06b2c0 0x7f6d1c06d780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d440cd4a0 0x7f6d4415c5f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 --2- 172.21.15.125:0/446066155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d24003970 0x7f6d4414ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 >> 172.21.15.125:0/446066155 conn(0x7f6d4406a800 msgr2=0x7f6d44160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 shutdown_connections 2024-01-16T14:57:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.577+0000 7f6d49800640 1 -- 172.21.15.125:0/446066155 wait complete. 2024-01-16T14:57:24.579 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:24.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:24.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.809+0000 7fe4d3400640 1 Processor -- start 2024-01-16T14:57:24.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.809+0000 7fe4d3400640 1 -- start start 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc0adf20 0x7fe4cc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4cc0ccb50 0x7fe4cc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d3400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc15f4e0 con 0x7fe4cc15bf10 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d3400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4cc15f970 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc071130 con 0x7fe4cc0adf20 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d1600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe4cc15bf10 0x7fe4cc0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50734/0 (socket says 172.21.15.125:50734) 2024-01-16T14:57:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.811+0000 7fe4d1600640 1 -- 172.21.15.125:0/2489143308 learned_addr learned my addr 172.21.15.125:0/2489143308 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2a00640 1 --2- 172.21.15.125:0/2489143308 >> v2:172.21.15.125:3300/0 conn(0x7fe4cc0ccb50 0x7fe4cc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2000640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc0adf20 0x7fe4cc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380349509 0 0) 0x7fe4cc15f4e0 con 0x7fe4cc15bf10 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac003520 con 0x7fe4cc15bf10 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2a00640 1 -- 172.21.15.125:0/2489143308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc0adf20 msgr2=0x7fe4cc070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2a00640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc0adf20 0x7fe4cc070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2a00640 1 -- 172.21.15.125:0/2489143308 >> v1:172.21.15.125:6789/0 conn(0x7fe4cc15bf10 legacy=0x7fe4cc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737154525 0 0) 0x7fe4ac003520 con 0x7fe4cc15bf10 2024-01-16T14:57:24.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.812+0000 7fe4d2a00640 1 -- 172.21.15.125:0/2489143308 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc15c390 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.813+0000 7fe4d2a00640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc143d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7f9f3aff608f2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0019f0 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b402e050 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d0c00640 1 -- 172.21.15.125:0/2489143308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4030730 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 -- 172.21.15.125:0/2489143308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0ccb50 msgr2=0x7fe4cc143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc143d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402b710 comp rx=0 tx=0).stop 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 -- 172.21.15.125:0/2489143308 shutdown_connections 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc143d30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 --2- 172.21.15.125:0/2489143308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc0adf20 0x7fe4cc070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 -- 172.21.15.125:0/2489143308 >> 172.21.15.125:0/2489143308 conn(0x7fe4cc06a7d0 msgr2=0x7fe4cc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 -- 172.21.15.125:0/2489143308 shutdown_connections 2024-01-16T14:57:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.814+0000 7fe4d3400640 1 -- 172.21.15.125:0/2489143308 wait complete. 2024-01-16T14:57:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.816+0000 7fe4d3400640 1 Processor -- start 2024-01-16T14:57:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.816+0000 7fe4d3400640 1 -- start start 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc15bf10 0x7fe4cc135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc0c5160 con 0x7fe4cc0adf20 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc0c4550 con 0x7fe4cc0ccb50 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc0c4830 con 0x7fe4cc15bf10 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52430/0 (socket says 172.21.15.125:52430) 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.817+0000 7fe4d2000640 1 -- 172.21.15.125:0/4121648801 learned_addr learned my addr 172.21.15.125:0/4121648801 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 -- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc15bf10 msgr2=0x7fe4cc135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc15bf10 0x7fe4cc135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 -- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4cc0ccb50 msgr2=0x7fe4cc15a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d1600640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc15a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 -- 172.21.15.125:0/4121648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b4009400 con 0x7fe4cc0adf20 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d2000640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400e9b0 tx=0x7fe4c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4009670 con 0x7fe4cc0adf20 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c400c9a0 con 0x7fe4cc0adf20 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc0c4b10 con 0x7fe4cc0adf20 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.818+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400cd50 con 0x7fe4cc0adf20 2024-01-16T14:57:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.819+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc0c4240 con 0x7fe4cc0adf20 2024-01-16T14:57:24.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.820+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc067450 con 0x7fe4cc0adf20 2024-01-16T14:57:24.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.820+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe4c4013470 con 0x7fe4cc0adf20 2024-01-16T14:57:24.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.821+0000 7fe4ca000640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 0x7fe4a406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.821+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4c408a250 con 0x7fe4cc0adf20 2024-01-16T14:57:24.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.822+0000 7fe4d1600640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 0x7fe4a406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.822+0000 7fe4d1600640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 0x7fe4a406d7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc002230 tx=0x7fe4bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:24.825+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4c4054f90 con 0x7fe4cc0adf20 2024-01-16T14:57:25.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.020+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4cc070600 con 0x7fe4cc0adf20 2024-01-16T14:57:25.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.021+0000 7fe4ca000640 1 -- 172.21.15.125:0/4121648801 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe4c40585b0 con 0x7fe4cc0adf20 2024-01-16T14:57:25.004 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:25.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 msgr2=0x7fe4a406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 0x7fe4a406d7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc002230 tx=0x7fe4bc009040 comp rx=0 tx=0).stop 2024-01-16T14:57:25.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 msgr2=0x7fe4cc0c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400e9b0 tx=0x7fe4c400ee70 comp rx=0 tx=0).stop 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 shutdown_connections 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4cc15bf10 0x7fe4cc135e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4cc0ccb50 0x7fe4cc15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4a406b2f0 0x7fe4a406d7b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 --2- 172.21.15.125:0/4121648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4cc0adf20 0x7fe4cc0c3d00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 >> 172.21.15.125:0/4121648801 conn(0x7fe4cc06a7d0 msgr2=0x7fe4cc06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.024+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 shutdown_connections 2024-01-16T14:57:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.025+0000 7fe4d3400640 1 -- 172.21.15.125:0/4121648801 wait complete. 2024-01-16T14:57:25.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:25.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.245+0000 7f12d4a00640 1 Processor -- start 2024-01-16T14:57:25.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.246+0000 7f12d4a00640 1 -- start start 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.247+0000 7f12d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 0x7f12cc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.247+0000 7f12d4a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc06c470 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.247+0000 7f12d4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc06c750 con 0x7f12cc13e360 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.247+0000 7f12d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc06ca30 con 0x7f12cc13df80 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f12cc0cf9a0 0x7f12cc0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50740/0 (socket says 172.21.15.125:50740) 2024-01-16T14:57:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d2a00640 1 -- 172.21.15.125:0/1527033844 learned_addr learned my addr 172.21.15.125:0/1527033844 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d2000640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 0x7f12cc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227737239 0 0) 0x7f12cc06c470 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ac003520 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047535926 0 0) 0x7f12ac003520 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 msgr2=0x7f12cc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 0x7f12cc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.248+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 >> v1:172.21.15.137:6789/0 conn(0x7f12cc13e360 legacy=0x7f12cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc06cd10 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f12bc0019f0 con 0x7f12cc0cf9a0 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 >> v1:172.21.15.125:6789/0 conn(0x7f12cc0cf9a0 legacy=0x7f12cc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 0x7f12ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12ac004d00 0x7f12ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc06ca30 con 0x7f12ac0039a0 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac00ba30 con 0x7f12ac004d00 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac00bbf0 con 0x7f12cc13df80 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 -- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 msgr2=0x7f12ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:25.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc13df80 0x7f12ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 -- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12ac004d00 msgr2=0x7f12ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12ac004d00 0x7f12ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.249+0000 7f12d2000640 1 -- 172.21.15.125:0/1527033844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac003520 con 0x7f12ac0039a0 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.250+0000 7f12d2000640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12ac003dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009a90 tx=0x7f12ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78e62cc6bbb04ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c402c6b0 con 0x7f12ac0039a0 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c402c810 con 0x7f12ac0039a0 2024-01-16T14:57:25.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d0c00640 1 -- 172.21.15.125:0/1527033844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c402cba0 con 0x7f12ac0039a0 2024-01-16T14:57:25.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 -- 172.21.15.125:0/1527033844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 msgr2=0x7f12ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 --2- 172.21.15.125:0/1527033844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12ac003dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009a90 tx=0x7f12ac0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d2a00640 1 -- 172.21.15.125:0/1527033844 reap_dead start 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 -- 172.21.15.125:0/1527033844 shutdown_connections 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 -- 172.21.15.125:0/1527033844 >> 172.21.15.125:0/1527033844 conn(0x7f12cc06a800 msgr2=0x7f12cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 -- 172.21.15.125:0/1527033844 shutdown_connections 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.251+0000 7f12d4a00640 1 -- 172.21.15.125:0/1527033844 wait complete. 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.253+0000 7f12d4a00640 1 Processor -- start 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.253+0000 7f12d4a00640 1 -- start start 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 0x7f12cc0b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12cc157450 0x7f12cc157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc158cc0 con 0x7f12ac0039a0 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc157e40 con 0x7f12cc157450 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc0b36b0 con 0x7f12cc0b2cf0 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52448/0 (socket says 172.21.15.125:52448) 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.254+0000 7f12d2000640 1 -- 172.21.15.125:0/3074174695 learned_addr learned my addr 172.21.15.125:0/3074174695 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d1600640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 0x7f12cc0b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 -- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 msgr2=0x7f12cc0b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12cc157450 0x7f12cc157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 0x7f12cc0b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 -- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12cc157450 msgr2=0x7f12cc157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12cc157450 0x7f12cc157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 -- 172.21.15.125:0/3074174695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc1566b0 con 0x7f12ac0039a0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d1600640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 0x7f12cc0b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d2000640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c250 tx=0x7f12c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4003810 con 0x7f12ac0039a0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4003970 con 0x7f12ac0039a0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4003d00 con 0x7f12ac0039a0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.255+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc1590e0 con 0x7f12ac0039a0 2024-01-16T14:57:25.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.256+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc158320 con 0x7f12ac0039a0 2024-01-16T14:57:25.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.257+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12cc0675d0 con 0x7f12ac0039a0 2024-01-16T14:57:25.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.258+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f12c4047450 con 0x7f12ac0039a0 2024-01-16T14:57:25.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.258+0000 7f12ca000640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 0x7f12a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.258+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12c40abd60 con 0x7f12ac0039a0 2024-01-16T14:57:25.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.259+0000 7f12d1600640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 0x7f12a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.259+0000 7f12d1600640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 0x7f12a406d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0aee20 tx=0x7f12b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.263+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12c4076c70 con 0x7f12ac0039a0 2024-01-16T14:57:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.451+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12cc06df00 con 0x7f12ac0039a0 2024-01-16T14:57:25.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.452+0000 7f12ca000640 1 -- 172.21.15.125:0/3074174695 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12c40044e0 con 0x7f12ac0039a0 2024-01-16T14:57:25.435 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 msgr2=0x7f12a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 0x7f12a406d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0aee20 tx=0x7f12b4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 msgr2=0x7f12cc13d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c250 tx=0x7f12c4004730 comp rx=0 tx=0).stop 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 shutdown_connections 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12cc157450 0x7f12cc157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12cc0b2cf0 0x7f12cc0b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12a406b2c0 0x7f12a406d780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.454+0000 7f12d4a00640 1 --2- 172.21.15.125:0/3074174695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12ac0039a0 0x7f12cc13d710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.455+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 >> 172.21.15.125:0/3074174695 conn(0x7f12cc06a800 msgr2=0x7f12cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.455+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 shutdown_connections 2024-01-16T14:57:25.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.455+0000 7f12d4a00640 1 -- 172.21.15.125:0/3074174695 wait complete. 2024-01-16T14:57:25.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:25.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.671+0000 7fbac5600640 1 Processor -- start 2024-01-16T14:57:25.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.672+0000 7fbac5600640 1 -- start start 2024-01-16T14:57:25.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbac5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 0x7fbac00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbac5600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbac006c6b0 0x7fbac0099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbac5600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac006cea0 con 0x7fbac0099fa0 2024-01-16T14:57:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbac5600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac009a430 con 0x7fbac006c6b0 2024-01-16T14:57:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbac5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac006cbc0 con 0x7fbac00cd240 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbabf400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbac0099fa0 0x7fbac006b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50756/0 (socket says 172.21.15.125:50756) 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.673+0000 7fbabf400640 1 -- 172.21.15.125:0/2410318820 learned_addr learned my addr 172.21.15.125:0/2410318820 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabea00640 1 --2- 172.21.15.125:0/2410318820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 0x7fbac00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabe000640 1 --2- 172.21.15.125:0/2410318820 >> v2:172.21.15.137:3300/0 conn(0x7fbac006c6b0 0x7fbac0099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabd600640 1 -- 172.21.15.125:0/2410318820 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930371635 0 0) 0x7fbac006cea0 con 0x7fbac0099fa0 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabd600640 1 -- 172.21.15.125:0/2410318820 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003520 con 0x7fbac0099fa0 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabea00640 1 -- 172.21.15.125:0/2410318820 >> v2:172.21.15.137:3300/0 conn(0x7fbac006c6b0 msgr2=0x7fbac0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabea00640 1 --2- 172.21.15.125:0/2410318820 >> v2:172.21.15.137:3300/0 conn(0x7fbac006c6b0 0x7fbac0099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabea00640 1 -- 172.21.15.125:0/2410318820 >> v1:172.21.15.125:6789/0 conn(0x7fbac0099fa0 legacy=0x7fbac006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.674+0000 7fbabea00640 1 -- 172.21.15.125:0/2410318820 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac006f350 con 0x7fbac00cd240 2024-01-16T14:57:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.676+0000 7fbabea00640 1 --2- 172.21.15.125:0/2410318820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 0x7fbac00b3820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbab00097a0 tx=0x7fbab002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6b6e65fc8d6e6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.676+0000 7fbabd600640 1 -- 172.21.15.125:0/2410318820 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0031830 con 0x7fbac00cd240 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.676+0000 7fbabd600640 1 -- 172.21.15.125:0/2410318820 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab0031990 con 0x7fbac00cd240 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.676+0000 7fbabd600640 1 -- 172.21.15.125:0/2410318820 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0031d20 con 0x7fbac00cd240 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 -- 172.21.15.125:0/2410318820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 msgr2=0x7fbac00b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 --2- 172.21.15.125:0/2410318820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 0x7fbac00b3820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbab00097a0 tx=0x7fbab002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 -- 172.21.15.125:0/2410318820 shutdown_connections 2024-01-16T14:57:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 --2- 172.21.15.125:0/2410318820 >> v2:172.21.15.137:3300/0 conn(0x7fbac006c6b0 0x7fbac0099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 --2- 172.21.15.125:0/2410318820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac00cd240 0x7fbac00b3820 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 -- 172.21.15.125:0/2410318820 >> 172.21.15.125:0/2410318820 conn(0x7fbac00b2d00 msgr2=0x7fbac00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 -- 172.21.15.125:0/2410318820 shutdown_connections 2024-01-16T14:57:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.677+0000 7fbac5600640 1 -- 172.21.15.125:0/2410318820 wait complete. 2024-01-16T14:57:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.678+0000 7fbac5600640 1 Processor -- start 2024-01-16T14:57:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.679+0000 7fbac5600640 1 -- start start 2024-01-16T14:57:25.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.679+0000 7fbac5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.679+0000 7fbac5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac0099fa0 0x7fbac014e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbac5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00cd240 0x7fbac014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbac5600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac014f370 con 0x7fbac006c6b0 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbac5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b7010 con 0x7fbac00cd240 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbac5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c7040 con 0x7fbac0099fa0 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52464/0 (socket says 172.21.15.125:52464) 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 -- 172.21.15.125:0/2769427223 learned_addr learned my addr 172.21.15.125:0/2769427223 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 -- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac0099fa0 msgr2=0x7fbac014e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac0099fa0 0x7fbac014e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 -- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00cd240 msgr2=0x7fbac014ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00cd240 0x7fbac014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 -- 172.21.15.125:0/2769427223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab0009400 con 0x7fbac006c6b0 2024-01-16T14:57:25.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.680+0000 7fbabea00640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbab002bee0 tx=0x7fbab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.681+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab003a630 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.681+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab003a7b0 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.681+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac00b6d30 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.681+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab003ab60 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.681+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac00c75c0 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.682+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac00675d0 con 0x7fbac006c6b0 2024-01-16T14:57:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.683+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbab003acc0 con 0x7fbac006c6b0 2024-01-16T14:57:25.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.684+0000 7fbab6a00640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 0x7fba9806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:25.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.684+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbab00ab890 con 0x7fbac006c6b0 2024-01-16T14:57:25.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.684+0000 7fbabe000640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 0x7fba9806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:25.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.685+0000 7fbabe000640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 0x7fba9806d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006a90 tx=0x7fbaa8006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:25.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.688+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbab0076780 con 0x7fbac006c6b0 2024-01-16T14:57:25.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.876+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbac00cc780 con 0x7fbac006c6b0 2024-01-16T14:57:25.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.876+0000 7fbab6a00640 1 -- 172.21.15.125:0/2769427223 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbab0079da0 con 0x7fbac006c6b0 2024-01-16T14:57:25.859 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 msgr2=0x7fba9806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 0x7fba9806d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006a90 tx=0x7fbaa8006a20 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 msgr2=0x7fbac00c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbab002bee0 tx=0x7fbab00046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 shutdown_connections 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00cd240 0x7fbac014ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac0099fa0 0x7fbac014e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba9806b2f0 0x7fba9806d7b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 --2- 172.21.15.125:0/2769427223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbac006c6b0 0x7fbac00c7ae0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 >> 172.21.15.125:0/2769427223 conn(0x7fbac00b2d00 msgr2=0x7fbac00cb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.879+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 shutdown_connections 2024-01-16T14:57:25.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:25.880+0000 7fbac5600640 1 -- 172.21.15.125:0/2769427223 wait complete. 2024-01-16T14:57:25.941 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-16T14:57:25.941 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:57:25.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:26.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.162+0000 7efcdce00640 1 Processor -- start 2024-01-16T14:57:26.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.162+0000 7efcdce00640 1 -- start start 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 0x7efcd815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efcd813e360 0x7efcd80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcd815f9c0 0x7efcd80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd806c470 con 0x7efcd815f9c0 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd806c750 con 0x7efcd813e360 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.163+0000 7efcdce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd806ca30 con 0x7efcd813df80 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcd815f9c0 0x7efcd80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcd815f9c0 0x7efcd80cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52466/0 (socket says 172.21.15.125:52466) 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 -- 172.21.15.125:0/2546312438 learned_addr learned my addr 172.21.15.125:0/2546312438 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efccd200640 1 --2- 172.21.15.125:0/2546312438 >> v2:172.21.15.137:3300/0 conn(0x7efcd813e360 0x7efcd80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd5e00640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 0x7efcd815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 -- 172.21.15.125:0/2546312438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 msgr2=0x7efcd815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:26.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 0x7efcd815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 -- 172.21.15.125:0/2546312438 >> v2:172.21.15.137:3300/0 conn(0x7efcd813e360 msgr2=0x7efcd80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:26.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 --2- 172.21.15.125:0/2546312438 >> v2:172.21.15.137:3300/0 conn(0x7efcd813e360 0x7efcd80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd6800640 1 -- 172.21.15.125:0/2546312438 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd806cd10 con 0x7efcd815f9c0 2024-01-16T14:57:26.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.164+0000 7efcd5e00640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 0x7efcd815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.165+0000 7efcd6800640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd815f9c0 0x7efcd80cf7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcd80cfed0 tx=0x7efcd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5f7b56f9baf1c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.165+0000 7efcd5400640 1 -- 172.21.15.125:0/2546312438 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd002ca70 con 0x7efcd815f9c0 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.165+0000 7efcd5400640 1 -- 172.21.15.125:0/2546312438 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd002cbd0 con 0x7efcd815f9c0 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.165+0000 7efcd5400640 1 -- 172.21.15.125:0/2546312438 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd00343f0 con 0x7efcd815f9c0 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 -- 172.21.15.125:0/2546312438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd815f9c0 msgr2=0x7efcd80cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd815f9c0 0x7efcd80cf7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcd80cfed0 tx=0x7efcd002bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 -- 172.21.15.125:0/2546312438 shutdown_connections 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd815f9c0 0x7efcd80cf7d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 --2- 172.21.15.125:0/2546312438 >> v2:172.21.15.137:3300/0 conn(0x7efcd813e360 0x7efcd80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 --2- 172.21.15.125:0/2546312438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd813df80 0x7efcd815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 -- 172.21.15.125:0/2546312438 >> 172.21.15.125:0/2546312438 conn(0x7efcd806a800 msgr2=0x7efcd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:26.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 -- 172.21.15.125:0/2546312438 shutdown_connections 2024-01-16T14:57:26.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.166+0000 7efcdce00640 1 -- 172.21.15.125:0/2546312438 wait complete. 2024-01-16T14:57:26.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.168+0000 7efcdce00640 1 Processor -- start 2024-01-16T14:57:26.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.168+0000 7efcdce00640 1 -- start start 2024-01-16T14:57:26.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcd813e360 0x7efcd81f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 0x7efcd81f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd81f6f00 con 0x7efcd813df80 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd81f2ac0 con 0x7efcd813e360 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcdce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8148590 con 0x7efcd815f9c0 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcd5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcd5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52468/0 (socket says 172.21.15.125:52468) 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efcd5e00640 1 -- 172.21.15.125:0/3415741611 learned_addr learned my addr 172.21.15.125:0/3415741611 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.169+0000 7efccd200640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcd813e360 0x7efcd81f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd6800640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 0x7efcd81f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 -- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 msgr2=0x7efcd81f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 0x7efcd81f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 -- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcd813e360 msgr2=0x7efcd81f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcd813e360 0x7efcd81f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 -- 172.21.15.125:0/3415741611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd0009b30 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd6800640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 0x7efcd81f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcd5e00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c820 tx=0x7efcc800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8007a30 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc8007b90 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc800ce10 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd8148870 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.170+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd8148e30 con 0x7efcd813df80 2024-01-16T14:57:26.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.171+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd80675d0 con 0x7efcd813df80 2024-01-16T14:57:26.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.172+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efcc80113f0 con 0x7efcd813df80 2024-01-16T14:57:26.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.173+0000 7efccea00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 0x7efcb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.173+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7efcc8089dd0 con 0x7efcd813df80 2024-01-16T14:57:26.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.173+0000 7efccd200640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 0x7efcb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.174+0000 7efccd200640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 0x7efcb006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efcd813d0f0 tx=0x7efcc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.178+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcc8054b10 con 0x7efcd813df80 2024-01-16T14:57:26.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.371+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcd806df00 con 0x7efcd813df80 2024-01-16T14:57:26.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.373+0000 7efccea00640 1 -- 172.21.15.125:0/3415741611 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16847 (secure 0 0 0) 0x7efcc801f090 con 0x7efcd813df80 2024-01-16T14:57:26.356 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:26.356 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":21,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:25.998247+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 msgr2=0x7efcb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 0x7efcb006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efcd813d0f0 tx=0x7efcc0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 msgr2=0x7efcd8147fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c820 tx=0x7efcc800cce0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 shutdown_connections 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcd815f9c0 0x7efcd81f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcd813e360 0x7efcd81f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcb006b2c0 0x7efcb006d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 --2- 172.21.15.125:0/3415741611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcd813df80 0x7efcd8147fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.375+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 >> 172.21.15.125:0/3415741611 conn(0x7efcd806a800 msgr2=0x7efcd80ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:26.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.376+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 shutdown_connections 2024-01-16T14:57:26.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.376+0000 7efcdce00640 1 -- 172.21.15.125:0/3415741611 wait complete. 2024-01-16T14:57:26.377 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:26.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:26.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.595+0000 7f7d4ce00640 1 Processor -- start 2024-01-16T14:57:26.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.595+0000 7f7d4ce00640 1 -- start start 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.596+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d48070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.596+0000 7f7d4ce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d48071460 con 0x7f7d48071030 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.596+0000 7f7d4ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4815deb0 con 0x7f7d48157c30 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.596+0000 7f7d4ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815ea30 con 0x7f7d480cd600 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d46800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7d48071030 0x7f7d4806dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33370/0 (socket says 172.21.15.125:33370) 2024-01-16T14:57:26.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d46800640 1 -- 172.21.15.125:0/2235167131 learned_addr learned my addr 172.21.15.125:0/2235167131 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d3d200640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d48070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167239275 0 0) 0x7f7d48071460 con 0x7f7d48071030 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d28003520 con 0x7f7d48071030 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531563864 0 0) 0x7f7d28003520 con 0x7f7d48071030 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 msgr2=0x7f7d48070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d48070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 >> v1:172.21.15.137:6789/0 conn(0x7f7d48157c30 legacy=0x7f7d481618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4815e190 con 0x7f7d48071030 2024-01-16T14:57:26.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7d40001a10 con 0x7f7d48071030 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.597+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 >> v1:172.21.15.125:6789/0 conn(0x7f7d48071030 legacy=0x7f7d4806dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d280039a0 0x7f7d28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d28004d00 0x7f7d28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815ea30 con 0x7f7d280039a0 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800ba60 con 0x7f7d28004d00 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800bc20 con 0x7f7d480cd600 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d280039a0 0x7f7d28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 -- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 msgr2=0x7f7d280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d3d200640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d480cd600 0x7f7d280047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 -- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d28004d00 msgr2=0x7f7d28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d28004d00 0x7f7d28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.598+0000 7f7d45e00640 1 -- 172.21.15.125:0/2235167131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28003520 con 0x7f7d280039a0 2024-01-16T14:57:26.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.599+0000 7f7d45e00640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d280039a0 0x7f7d28003dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7d30001720 tx=0x7f7d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad2e150ea42540d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.599+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300052d0 con 0x7f7d280039a0 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3002e050 con 0x7f7d280039a0 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d45400640 1 -- 172.21.15.125:0/2235167131 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300035f0 con 0x7f7d280039a0 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 -- 172.21.15.125:0/2235167131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d280039a0 msgr2=0x7f7d28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/2235167131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d280039a0 0x7f7d28003dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7d30001720 tx=0x7f7d280057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d46800640 1 -- 172.21.15.125:0/2235167131 reap_dead start 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 -- 172.21.15.125:0/2235167131 shutdown_connections 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 -- 172.21.15.125:0/2235167131 >> 172.21.15.125:0/2235167131 conn(0x7f7d480cf700 msgr2=0x7f7d480cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 -- 172.21.15.125:0/2235167131 shutdown_connections 2024-01-16T14:57:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.600+0000 7f7d4ce00640 1 -- 172.21.15.125:0/2235167131 wait complete. 2024-01-16T14:57:26.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.602+0000 7f7d4ce00640 1 Processor -- start 2024-01-16T14:57:26.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.602+0000 7f7d4ce00640 1 -- start start 2024-01-16T14:57:26.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 0x7f7d481554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d48152b20 0x7f7d48155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4813fa00 con 0x7f7d4813fe20 2024-01-16T14:57:26.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4813ee10 con 0x7f7d48152b20 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d4ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4813f060 con 0x7f7d280039a0 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.603+0000 7f7d46800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47996/0 (socket says 172.21.15.125:47996) 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 -- 172.21.15.125:0/1453662898 learned_addr learned my addr 172.21.15.125:0/1453662898 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d3d200640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d48152b20 0x7f7d48155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d45e00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 0x7f7d481554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 -- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 msgr2=0x7f7d481554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 0x7f7d481554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 -- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d48152b20 msgr2=0x7f7d48155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d48152b20 0x7f7d48155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 -- 172.21.15.125:0/1453662898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4813f340 con 0x7f7d4813fe20 2024-01-16T14:57:26.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d45e00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 0x7f7d481554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.604+0000 7f7d46800640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7d480724a0 tx=0x7f7d4000ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.605+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000a5b0 con 0x7f7d4813fe20 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.605+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4000a710 con 0x7f7d4813fe20 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.605+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000aa80 con 0x7f7d4813fe20 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.605+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4813f620 con 0x7f7d4813fe20 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.605+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d481409c0 con 0x7f7d4813fe20 2024-01-16T14:57:26.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.606+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4813fb40 con 0x7f7d4813fe20 2024-01-16T14:57:26.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.607+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d4000abe0 con 0x7f7d4813fe20 2024-01-16T14:57:26.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.608+0000 7f7d3ea00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 0x7f7d2006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:26.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.608+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7d40089480 con 0x7f7d4813fe20 2024-01-16T14:57:26.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.608+0000 7f7d45e00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 0x7f7d2006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:26.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.608+0000 7f7d45e00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 0x7f7d2006d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d30033000 tx=0x7f7d30009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:26.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.612+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d4813fb40 con 0x7f7d4813fe20 2024-01-16T14:57:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.803+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d4813fb40 con 0x7f7d4813fe20 2024-01-16T14:57:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.804+0000 7f7d3ea00640 1 -- 172.21.15.125:0/1453662898 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d4813fb40 con 0x7f7d4813fe20 2024-01-16T14:57:26.786 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.806+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 msgr2=0x7f7d2006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.806+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 0x7f7d2006d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d30033000 tx=0x7f7d30009030 comp rx=0 tx=0).stop 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.806+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 msgr2=0x7f7d481402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.806+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7d480724a0 tx=0x7f7d4000ce80 comp rx=0 tx=0).stop 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 shutdown_connections 2024-01-16T14:57:26.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d2006b2c0 0x7f7d2006d780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d4813fe20 0x7f7d481402a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d48152b20 0x7f7d48155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 --2- 172.21.15.125:0/1453662898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d280039a0 0x7f7d481554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 >> 172.21.15.125:0/1453662898 conn(0x7f7d480cf700 msgr2=0x7f7d4806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 shutdown_connections 2024-01-16T14:57:26.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:26.807+0000 7f7d4ce00640 1 -- 172.21.15.125:0/1453662898 wait complete. 2024-01-16T14:57:26.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.040+0000 7f1a10000640 1 Processor -- start 2024-01-16T14:57:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.040+0000 7f1a10000640 1 -- start start 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a10000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 0x7f1a080ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a10000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1a080d1ef0 0x7f1a080b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a10000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a080b3400 con 0x7f1a080ced60 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a10000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a080d1390 con 0x7f1a080d1ef0 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a10000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b27f0 con 0x7f1a0815d610 2024-01-16T14:57:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a0cc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1a080ced60 0x7f1a080d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33374/0 (socket says 172.21.15.125:33374) 2024-01-16T14:57:27.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.042+0000 7f1a0cc00640 1 -- 172.21.15.125:0/1793444852 learned_addr learned my addr 172.21.15.125:0/1793444852 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:27.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a0d600640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 0x7f1a080ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076682954 0 0) 0x7f1a080b3400 con 0x7f1a080ced60 2024-01-16T14:57:27.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e8003520 con 0x7f1a080ced60 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> v2:172.21.15.137:3300/0 conn(0x7f1a080d1ef0 0x7f1a080b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822117305 0 0) 0x7f19e8003520 con 0x7f1a080ced60 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 msgr2=0x7f1a080ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 0x7f1a080ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 >> v2:172.21.15.137:3300/0 conn(0x7f1a080d1ef0 msgr2=0x7f1a080b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 --2- 172.21.15.125:0/1793444852 >> v2:172.21.15.137:3300/0 conn(0x7f1a080d1ef0 0x7f1a080b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080b2ad0 con 0x7f1a080ced60 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f19f00019f0 con 0x7f1a080ced60 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> v2:172.21.15.137:3300/0 conn(0x7f1a080d1ef0 0x7f1a080b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 >> v1:172.21.15.125:6789/0 conn(0x7f1a080ced60 legacy=0x7f1a080d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8003970 0x7f19e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 0x7f19e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e8004d00 0x7f19e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080d1390 con 0x7f19e8004d00 2024-01-16T14:57:27.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b27f0 con 0x7f19e8003970 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.043+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e800ba00 con 0x7f1a0815d610 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e8004d00 0x7f19e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0d600640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8003970 0x7f19e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 -- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 msgr2=0x7f19e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a0815d610 0x7f19e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 -- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8003970 msgr2=0x7f19e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8003970 0x7f19e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.044+0000 7f1a0e000640 1 -- 172.21.15.125:0/1793444852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8003520 con 0x7f19e8004d00 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.045+0000 7f1a0e000640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e8004d00 0x7f19e8005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1a0808bb40 tx=0x7f19e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3622729948eb0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.045+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00004230 con 0x7f19e8004d00 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.045+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00004390 con 0x7f19e8004d00 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.045+0000 7f1a07e00640 1 -- 172.21.15.125:0/1793444852 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00003830 con 0x7f19e8004d00 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 -- 172.21.15.125:0/1793444852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e8004d00 msgr2=0x7f19e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 --2- 172.21.15.125:0/1793444852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e8004d00 0x7f19e8005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1a0808bb40 tx=0x7f19e80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a0e000640 1 -- 172.21.15.125:0/1793444852 reap_dead start 2024-01-16T14:57:27.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 -- 172.21.15.125:0/1793444852 shutdown_connections 2024-01-16T14:57:27.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 -- 172.21.15.125:0/1793444852 >> 172.21.15.125:0/1793444852 conn(0x7f1a0809c540 msgr2=0x7f1a080d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:27.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 -- 172.21.15.125:0/1793444852 shutdown_connections 2024-01-16T14:57:27.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.046+0000 7f1a10000640 1 -- 172.21.15.125:0/1793444852 wait complete. 2024-01-16T14:57:27.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.048+0000 7f1a10000640 1 Processor -- start 2024-01-16T14:57:27.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.048+0000 7f1a10000640 1 -- start start 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 0x7f1a080cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a0814cb90 0x7f1a0814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b8ff0 con 0x7f1a080ca2d0 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080cc950 con 0x7f1a0814cb90 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a10000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080ccc30 con 0x7f19e8004d00 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a0cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a0cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48006/0 (socket says 172.21.15.125:48006) 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.049+0000 7f1a0cc00640 1 -- 172.21.15.125:0/3652551330 learned_addr learned my addr 172.21.15.125:0/3652551330 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0d600640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 0x7f1a080cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0e000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a0814cb90 0x7f1a0814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 -- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 msgr2=0x7f1a080cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 0x7f1a080cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 -- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a0814cb90 msgr2=0x7f1a0814d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a0814cb90 0x7f1a0814d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 -- 172.21.15.125:0/3652551330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080ccf10 con 0x7f1a080ca2d0 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0d600640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 0x7f1a080cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:27.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a0cc00640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f19f000caa0 tx=0x7f19f000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f000a790 con 0x7f1a080ca2d0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f000a8f0 con 0x7f1a080ca2d0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a080cd1f0 con 0x7f1a080ca2d0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.050+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f000aca0 con 0x7f1a080ca2d0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.051+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a080ca7e0 con 0x7f1a080ca2d0 2024-01-16T14:57:27.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.052+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a080cd430 con 0x7f1a080ca2d0 2024-01-16T14:57:27.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.053+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19f0007150 con 0x7f1a080ca2d0 2024-01-16T14:57:27.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.053+0000 7f1a05600640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 0x7f19e006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.054+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f19f0089310 con 0x7f1a080ca2d0 2024-01-16T14:57:27.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.054+0000 7f1a0d600640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 0x7f19e006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.054+0000 7f1a0d600640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 0x7f19e006d7b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009980 tx=0x7f19f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.058+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a080cd430 con 0x7f1a080ca2d0 2024-01-16T14:57:27.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.249+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1a080cd430 con 0x7f1a080ca2d0 2024-01-16T14:57:27.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.250+0000 7f1a05600640 1 -- 172.21.15.125:0/3652551330 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1a080cd430 con 0x7f1a080ca2d0 2024-01-16T14:57:27.233 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:27.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.252+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 msgr2=0x7f19e006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.252+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 0x7f19e006d7b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009980 tx=0x7f19f8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:27.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.252+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 msgr2=0x7f1a0814c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.252+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f19f000caa0 tx=0x7f19f000cf60 comp rx=0 tx=0).stop 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 shutdown_connections 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a0814cb90 0x7f1a0814d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19e006b2f0 0x7f19e006d7b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a080ca2d0 0x7f1a0814c650 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 --2- 172.21.15.125:0/3652551330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e8004d00 0x7f1a080cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 >> 172.21.15.125:0/3652551330 conn(0x7f1a0809c540 msgr2=0x7f1a080b5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 shutdown_connections 2024-01-16T14:57:27.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.253+0000 7f1a10000640 1 -- 172.21.15.125:0/3652551330 wait complete. 2024-01-16T14:57:27.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:27.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.476+0000 7fadd8e00640 1 Processor -- start 2024-01-16T14:57:27.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.477+0000 7fadd8e00640 1 -- start start 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.478+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadd40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.478+0000 7fadd8e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40d0320 con 0x7fadd40b31d0 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.478+0000 7fadd8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40d1c60 con 0x7fadd415d4b0 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.478+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40b2730 con 0x7fadd40cf9a0 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fadd40b31d0 0x7fadd40b3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33378/0 (socket says 172.21.15.125:33378) 2024-01-16T14:57:27.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd3e00640 1 -- 172.21.15.125:0/4241420548 learned_addr learned my addr 172.21.15.125:0/4241420548 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:27.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2a00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadd40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781344604 0 0) 0x7fadd40d0320 con 0x7fadd40b31d0 2024-01-16T14:57:27.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb4003520 con 0x7fadd40b31d0 2024-01-16T14:57:27.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824326959 0 0) 0x7fadb4003520 con 0x7fadd40b31d0 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 msgr2=0x7fadd40cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadd40cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 >> v1:172.21.15.137:6789/0 conn(0x7fadd415d4b0 legacy=0x7fadd40d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40d1ec0 con 0x7fadd40b31d0 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fadcc0019f0 con 0x7fadd40b31d0 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.479+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 >> v1:172.21.15.125:6789/0 conn(0x7fadd40b31d0 legacy=0x7fadd40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadb40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadb4003f20 0x7fadb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4004d00 0x7fadb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40b2730 con 0x7fadb4003f20 2024-01-16T14:57:27.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb400ba60 con 0x7fadb4004d00 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb400bc20 con 0x7fadd40cf9a0 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadb4003f20 0x7fadb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd3400640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadb40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd3e00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4004d00 0x7fadb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 -- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 msgr2=0x7fadb40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40cf9a0 0x7fadb40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 -- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4004d00 msgr2=0x7fadb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4004d00 0x7fadb4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd2a00640 1 -- 172.21.15.125:0/4241420548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4003520 con 0x7fadb4003f20 2024-01-16T14:57:27.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.480+0000 7fadd3e00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4004d00 0x7fadb4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd2a00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadb4003f20 0x7fadb40047c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009ac0 tx=0x7fadb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf878b13b9929ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc02c6b0 con 0x7fadb4003f20 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc02c810 con 0x7fadb4003f20 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd2000640 1 -- 172.21.15.125:0/4241420548 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc02cba0 con 0x7fadb4003f20 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd8e00640 1 -- 172.21.15.125:0/4241420548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadb4003f20 msgr2=0x7fadb40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd8e00640 1 --2- 172.21.15.125:0/4241420548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadb4003f20 0x7fadb40047c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009ac0 tx=0x7fadb40057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.482+0000 7fadd3e00640 1 -- 172.21.15.125:0/4241420548 reap_dead start 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.483+0000 7fadd8e00640 1 -- 172.21.15.125:0/4241420548 shutdown_connections 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.483+0000 7fadd8e00640 1 -- 172.21.15.125:0/4241420548 >> 172.21.15.125:0/4241420548 conn(0x7fadd40d3000 msgr2=0x7fadd40d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.483+0000 7fadd8e00640 1 -- 172.21.15.125:0/4241420548 shutdown_connections 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.483+0000 7fadd8e00640 1 -- 172.21.15.125:0/4241420548 wait complete. 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.484+0000 7fadd8e00640 1 Processor -- start 2024-01-16T14:57:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.484+0000 7fadd8e00640 1 -- start start 2024-01-16T14:57:27.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4003f20 0x7fadd40cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40c8b40 0x7fadd415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40c6b30 con 0x7fadd40c77f0 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40cbe30 con 0x7fadb4003f20 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.485+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40cbb50 con 0x7fadd40c8b40 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48028/0 (socket says 172.21.15.125:48028) 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 -- 172.21.15.125:0/3059124494 learned_addr learned my addr 172.21.15.125:0/3059124494 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3400640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4003f20 0x7fadd40cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd2a00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40c8b40 0x7fadd415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 -- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40c8b40 msgr2=0x7fadd415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40c8b40 0x7fadd415b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 -- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4003f20 msgr2=0x7fadd40cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4003f20 0x7fadd40cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.486+0000 7fadd3e00640 1 -- 172.21.15.125:0/3059124494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40c6d10 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadd3e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fadd4072af0 tx=0x7fadcc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc004d60 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00bb20 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00bed0 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40c6ff0 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.487+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd40cb740 con 0x7fadd40c77f0 2024-01-16T14:57:27.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.488+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd40c7510 con 0x7fadd40c77f0 2024-01-16T14:57:27.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.489+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fadcc007070 con 0x7fadd40c77f0 2024-01-16T14:57:27.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.490+0000 7fadcb400640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 0x7fadac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.490+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fadcc089b70 con 0x7fadd40c77f0 2024-01-16T14:57:27.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.491+0000 7fadd3400640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 0x7fadac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.491+0000 7fadd3400640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 0x7fadac06d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.495+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadd40c7510 con 0x7fadd40c77f0 2024-01-16T14:57:27.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.690+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fadd40c7510 con 0x7fadd40c77f0 2024-01-16T14:57:27.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.691+0000 7fadcb400640 1 -- 172.21.15.125:0/3059124494 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadd40c7510 con 0x7fadd40c77f0 2024-01-16T14:57:27.673 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:27.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.693+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 msgr2=0x7fadac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.693+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 0x7fadac06d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 msgr2=0x7fadd415a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fadd4072af0 tx=0x7fadcc00b4a0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 shutdown_connections 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06b2c0 0x7fadac06d780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd40c77f0 0x7fadd415a330 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40c8b40 0x7fadd415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3059124494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadb4003f20 0x7fadd40cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 >> 172.21.15.125:0/3059124494 conn(0x7fadd40d3000 msgr2=0x7fadd40cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 shutdown_connections 2024-01-16T14:57:27.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.694+0000 7fadd8e00640 1 -- 172.21.15.125:0/3059124494 wait complete. 2024-01-16T14:57:27.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T14:57:27.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.916+0000 7f011f800640 1 Processor -- start 2024-01-16T14:57:27.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.916+0000 7f011f800640 1 -- start start 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011f800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 0x7f011815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011f800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01180cf290 0x7f01180cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011f800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0118070c40 con 0x7f01180ae5a0 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011f800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118071450 con 0x7f01180cf290 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011f800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01180cfc60 con 0x7f01180b3ae0 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01180cf290 0x7f01180cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01180cf290 0x7f01180cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48044/0 (socket says 172.21.15.125:48044) 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 -- 172.21.15.125:0/2072414548 learned_addr learned my addr 172.21.15.125:0/2072414548 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:27.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 -- 172.21.15.125:0/2072414548 >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 msgr2=0x7f011815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f0117e00640 1 --2- 172.21.15.125:0/2072414548 >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 0x7f011815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 --2- 172.21.15.125:0/2072414548 >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 0x7f011815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.918+0000 7f011d800640 1 -- 172.21.15.125:0/2072414548 >> v1:172.21.15.137:6789/0 conn(0x7f01180ae5a0 legacy=0x7f011806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:27.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.919+0000 7f011d800640 1 -- 172.21.15.125:0/2072414548 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011806c470 con 0x7f01180cf290 2024-01-16T14:57:27.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.919+0000 7f0117e00640 1 --2- 172.21.15.125:0/2072414548 >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 0x7f011815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.920+0000 7f011d800640 1 --2- 172.21.15.125:0/2072414548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f01180cf720 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01180cfea0 tx=0x7f011002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed30815a626694a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.920+0000 7f0117400640 1 -- 172.21.15.125:0/2072414548 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011002cb20 con 0x7f01180cf290 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.920+0000 7f0117400640 1 -- 172.21.15.125:0/2072414548 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011002cc80 con 0x7f01180cf290 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.920+0000 7f0117400640 1 -- 172.21.15.125:0/2072414548 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100343f0 con 0x7f01180cf290 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 -- 172.21.15.125:0/2072414548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 msgr2=0x7f01180cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 --2- 172.21.15.125:0/2072414548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f01180cf720 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01180cfea0 tx=0x7f011002bf80 comp rx=0 tx=0).stop 2024-01-16T14:57:27.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 -- 172.21.15.125:0/2072414548 shutdown_connections 2024-01-16T14:57:27.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 --2- 172.21.15.125:0/2072414548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f01180cf720 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 --2- 172.21.15.125:0/2072414548 >> v2:172.21.15.137:3300/0 conn(0x7f01180b3ae0 0x7f011815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 -- 172.21.15.125:0/2072414548 >> 172.21.15.125:0/2072414548 conn(0x7f011806a800 msgr2=0x7f011806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:27.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 -- 172.21.15.125:0/2072414548 shutdown_connections 2024-01-16T14:57:27.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.921+0000 7f011f800640 1 -- 172.21.15.125:0/2072414548 wait complete. 2024-01-16T14:57:27.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.922+0000 7f011f800640 1 Processor -- start 2024-01-16T14:57:27.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.923+0000 7f011f800640 1 -- start start 2024-01-16T14:57:27.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01180ae5a0 0x7f0118158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01180b3ae0 0x7f0118159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011813f1f0 con 0x7f01180cf290 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011813e570 con 0x7f01180ae5a0 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011f800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011813e850 con 0x7f01180b3ae0 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48052/0 (socket says 172.21.15.125:48052) 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 -- 172.21.15.125:0/3544013212 learned_addr learned my addr 172.21.15.125:0/3544013212 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011ce00640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01180ae5a0 0x7f0118158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 -- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01180b3ae0 msgr2=0x7f0118159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01180b3ae0 0x7f0118159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 -- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01180ae5a0 msgr2=0x7f0118158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01180ae5a0 0x7f0118158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.924+0000 7f011d800640 1 -- 172.21.15.125:0/3544013212 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110009b30 con 0x7f01180cf290 2024-01-16T14:57:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f011d800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0110009f40 tx=0x7f0110004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110007e70 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0110009510 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011003e690 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011813eb30 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.925+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0118158cd0 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.926+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0118158e60 con 0x7f01180cf290 2024-01-16T14:57:27.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.927+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f011003e7f0 con 0x7f01180cf290 2024-01-16T14:57:27.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.928+0000 7f0114c00640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 0x7f00f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:27.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.928+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f01100aca00 con 0x7f01180cf290 2024-01-16T14:57:27.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.928+0000 7f011ce00640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 0x7f00f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:27.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.929+0000 7f011ce00640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 0x7f00f006d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0108003d90 tx=0x7f0108009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:27.933+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0110077740 con 0x7f01180cf290 2024-01-16T14:57:28.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:28.194+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f011813f330 con 0x7f01180cf290 2024-01-16T14:57:29.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.035+0000 7f0114c00640 1 -- 172.21.15.125:0/3544013212 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v61) v1 ==== 155+0+0 (secure 0 0 0) 0x7f011813f330 con 0x7f01180cf290 2024-01-16T14:57:29.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 msgr2=0x7f00f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 0x7f00f006d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0108003d90 tx=0x7f0108009040 comp rx=0 tx=0).stop 2024-01-16T14:57:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 msgr2=0x7f011814d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0110009f40 tx=0x7f0110004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 shutdown_connections 2024-01-16T14:57:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00f006b2c0 0x7f00f006d780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.038+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01180cf290 0x7f011814d450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.039+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01180b3ae0 0x7f0118159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.039+0000 7f011f800640 1 --2- 172.21.15.125:0/3544013212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01180ae5a0 0x7f0118158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.039+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 >> 172.21.15.125:0/3544013212 conn(0x7f011806a800 msgr2=0x7f011815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.039+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 shutdown_connections 2024-01-16T14:57:29.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.039+0000 7f011f800640 1 -- 172.21.15.125:0/3544013212 wait complete. 2024-01-16T14:57:29.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T14:57:29.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.275+0000 7fe126200640 1 Processor -- start 2024-01-16T14:57:29.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.275+0000 7fe126200640 1 -- start start 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.276+0000 7fe126200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe120143ae0 0x7fe1200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe126200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe12015e3b0 0x7fe1200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe126200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200cfc30 con 0x7fe1200cf7d0 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe126200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200b1ee0 con 0x7fe120143ae0 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe126200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b2780 con 0x7fe12015e3b0 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe116800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe1200cf7d0 0x7fe12015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33388/0 (socket says 172.21.15.125:33388) 2024-01-16T14:57:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe116800640 1 -- 172.21.15.125:0/646385006 learned_addr learned my addr 172.21.15.125:0/646385006 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe11f400640 1 --2- 172.21.15.125:0/646385006 >> v2:172.21.15.137:3300/0 conn(0x7fe120143ae0 0x7fe1200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629808780 0 0) 0x7fe1200cfc30 con 0x7fe1200cf7d0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe100003520 con 0x7fe1200cf7d0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.277+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe12015e3b0 0x7fe1200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680291985 0 0) 0x7fe100003520 con 0x7fe1200cf7d0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe12015e3b0 msgr2=0x7fe1200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe12015e3b0 0x7fe1200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 >> v2:172.21.15.137:3300/0 conn(0x7fe120143ae0 msgr2=0x7fe1200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 --2- 172.21.15.125:0/646385006 >> v2:172.21.15.137:3300/0 conn(0x7fe120143ae0 0x7fe1200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12015e990 con 0x7fe1200cf7d0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe108001670 con 0x7fe1200cf7d0 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 >> v1:172.21.15.125:6789/0 conn(0x7fe1200cf7d0 legacy=0x7fe12015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe100003970 0x7fe100003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1000047c0 0x7fe100004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe10000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b1ee0 con 0x7fe100005180 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b2780 con 0x7fe100003970 2024-01-16T14:57:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10000bef0 con 0x7fe1000047c0 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe12015e3b0 0x7fe1200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.278+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe10000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11f400640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe100003970 0x7fe100003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe116800640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1000047c0 0x7fe100004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11fe00640 1 -- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1000047c0 msgr2=0x7fe100004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1000047c0 0x7fe100004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11fe00640 1 -- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe100003970 msgr2=0x7fe100003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe100003970 0x7fe100003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.279+0000 7fe11fe00640 1 -- 172.21.15.125:0/646385006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe100003520 con 0x7fe100005180 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe11fe00640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe10000b980 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe118009f00 tx=0x7fe1000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d44232c33ca9c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11802cc70 con 0x7fe100005180 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11802cdd0 con 0x7fe100005180 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe11ea00640 1 -- 172.21.15.125:0/646385006 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118003bb0 con 0x7fe100005180 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe126200640 1 -- 172.21.15.125:0/646385006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 msgr2=0x7fe10000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe126200640 1 --2- 172.21.15.125:0/646385006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe10000b980 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe118009f00 tx=0x7fe1000057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.280+0000 7fe11fe00640 1 -- 172.21.15.125:0/646385006 reap_dead start 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.281+0000 7fe126200640 1 -- 172.21.15.125:0/646385006 shutdown_connections 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.281+0000 7fe126200640 1 -- 172.21.15.125:0/646385006 >> 172.21.15.125:0/646385006 conn(0x7fe12006a800 msgr2=0x7fe12006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.281+0000 7fe126200640 1 -- 172.21.15.125:0/646385006 shutdown_connections 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.281+0000 7fe126200640 1 -- 172.21.15.125:0/646385006 wait complete. 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.283+0000 7fe126200640 1 Processor -- start 2024-01-16T14:57:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.283+0000 7fe126200640 1 -- start start 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1200cc650 0x7fe12013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1200cd3f0 0x7fe12015c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200a8ed0 con 0x7fe100005180 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12013de80 con 0x7fe1200cd3f0 2024-01-16T14:57:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe126200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b1420 con 0x7fe1200cc650 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48068/0 (socket says 172.21.15.125:48068) 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 -- 172.21.15.125:0/1902099183 learned_addr learned my addr 172.21.15.125:0/1902099183 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 -- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1200cc650 msgr2=0x7fe12013d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1200cc650 0x7fe12013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 -- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1200cd3f0 msgr2=0x7fe12015c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1200cd3f0 0x7fe12015c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 -- 172.21.15.125:0/1902099183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200ccc80 con 0x7fe100005180 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.284+0000 7fe11f400640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe110011600 tx=0x7fe110011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.285+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110015870 con 0x7fe100005180 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.285+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1100159d0 con 0x7fe100005180 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.285+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe120159490 con 0x7fe100005180 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.285+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110015d40 con 0x7fe100005180 2024-01-16T14:57:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.285+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200cced0 con 0x7fe100005180 2024-01-16T14:57:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.286+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1200a9320 con 0x7fe100005180 2024-01-16T14:57:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.287+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe110013430 con 0x7fe100005180 2024-01-16T14:57:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.288+0000 7fe117e00640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 0x7fe0f806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.288+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe11008a3f0 con 0x7fe100005180 2024-01-16T14:57:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.288+0000 7fe116800640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 0x7fe0f806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:29.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.289+0000 7fe116800640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 0x7fe0f806d710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe108001050 tx=0x7fe108000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:29.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.292+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe110055260 con 0x7fe100005180 2024-01-16T14:57:29.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.481+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe1200cca90 con 0x7fe100005180 2024-01-16T14:57:29.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.489+0000 7fe117e00640 1 -- 172.21.15.125:0/1902099183 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe110058880 con 0x7fe100005180 2024-01-16T14:57:29.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 msgr2=0x7fe0f806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 0x7fe0f806d710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe108001050 tx=0x7fe108000fe0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 msgr2=0x7fe120159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe110011600 tx=0x7fe110011ac0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 shutdown_connections 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1200cd3f0 0x7fe12015c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1200cc650 0x7fe12013d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0f806b250 0x7fe0f806d710 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 --2- 172.21.15.125:0/1902099183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe100005180 0x7fe120159770 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 >> 172.21.15.125:0/1902099183 conn(0x7fe12006a800 msgr2=0x7fe12006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 shutdown_connections 2024-01-16T14:57:29.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:29.492+0000 7fe126200640 1 -- 172.21.15.125:0/1902099183 wait complete. 2024-01-16T14:57:29.492 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T14:57:29.492 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:57:29.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:57:29.713 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T14:57:29.714 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:29.714 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:29.771 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:29.772 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:29.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:30.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.051+0000 7fe65ea00640 1 Processor -- start 2024-01-16T14:57:30.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.051+0000 7fe65ea00640 1 -- start start 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe658143ae0 0x7fe6580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6580cf7d0 0x7fe65815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 0x7fe65815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6580cfc30 con 0x7fe658143ae0 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6580b1ee0 con 0x7fe65815e1e0 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe65ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6580b2780 con 0x7fe6580cf7d0 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe657e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe658143ae0 0x7fe6580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe657e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe658143ae0 0x7fe6580cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48078/0 (socket says 172.21.15.125:48078) 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.053+0000 7fe657e00640 1 -- 172.21.15.125:0/743277677 learned_addr learned my addr 172.21.15.125:0/743277677 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:30.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe65ca00640 1 --2- 172.21.15.125:0/743277677 >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 0x7fe65815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe657e00640 1 -- 172.21.15.125:0/743277677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6580cf7d0 msgr2=0x7fe65815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe657e00640 1 --2- 172.21.15.125:0/743277677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6580cf7d0 0x7fe65815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe657e00640 1 -- 172.21.15.125:0/743277677 >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 msgr2=0x7fe65815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe657e00640 1 --2- 172.21.15.125:0/743277677 >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 0x7fe65815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe657e00640 1 -- 172.21.15.125:0/743277677 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65815e990 con 0x7fe658143ae0 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.054+0000 7fe65ca00640 1 --2- 172.21.15.125:0/743277677 >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 0x7fe65815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:30.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.055+0000 7fe657e00640 1 --2- 172.21.15.125:0/743277677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580cf290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe658072160 tx=0x7fe64802db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6fb68e6cb7799a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.055+0000 7fe657400640 1 -- 172.21.15.125:0/743277677 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64802f8e0 con 0x7fe658143ae0 2024-01-16T14:57:30.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.055+0000 7fe657400640 1 -- 172.21.15.125:0/743277677 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64802fa40 con 0x7fe658143ae0 2024-01-16T14:57:30.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.055+0000 7fe657400640 1 -- 172.21.15.125:0/743277677 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64802fdd0 con 0x7fe658143ae0 2024-01-16T14:57:30.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 -- 172.21.15.125:0/743277677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 msgr2=0x7fe6580cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 --2- 172.21.15.125:0/743277677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580cf290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe658072160 tx=0x7fe64802db20 comp rx=0 tx=0).stop 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 -- 172.21.15.125:0/743277677 shutdown_connections 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 --2- 172.21.15.125:0/743277677 >> v2:172.21.15.137:3300/0 conn(0x7fe65815e1e0 0x7fe65815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 --2- 172.21.15.125:0/743277677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6580cf7d0 0x7fe65815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 --2- 172.21.15.125:0/743277677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580cf290 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 -- 172.21.15.125:0/743277677 >> 172.21.15.125:0/743277677 conn(0x7fe65806a800 msgr2=0x7fe65806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 -- 172.21.15.125:0/743277677 shutdown_connections 2024-01-16T14:57:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.056+0000 7fe65ea00640 1 -- 172.21.15.125:0/743277677 wait complete. 2024-01-16T14:57:30.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.058+0000 7fe65ea00640 1 Processor -- start 2024-01-16T14:57:30.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.058+0000 7fe65ea00640 1 -- start start 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 0x7fe6581599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe65815e1e0 0x7fe6580b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe658133f70 con 0x7fe658143ae0 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe658159f20 con 0x7fe6580cf7d0 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe658133c90 con 0x7fe65815e1e0 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe64f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe64f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48094/0 (socket says 172.21.15.125:48094) 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe64f200640 1 -- 172.21.15.125:0/2359548075 learned_addr learned my addr 172.21.15.125:0/2359548075 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe657e00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 0x7fe6581599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.059+0000 7fe65ca00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe65815e1e0 0x7fe6580b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 -- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe65815e1e0 msgr2=0x7fe6580b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe65815e1e0 0x7fe6580b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 -- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 msgr2=0x7fe6581599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 0x7fe6581599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 -- 172.21.15.125:0/2359548075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480094d0 con 0x7fe658143ae0 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe657e00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 0x7fe6581599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe64f200640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe64000ba50 tx=0x7fe64000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64000cbe0 con 0x7fe658143ae0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64000cd60 con 0x7fe658143ae0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640014680 con 0x7fe658143ae0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6581310e0 con 0x7fe658143ae0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.060+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6581316a0 con 0x7fe658143ae0 2024-01-16T14:57:30.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.061+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6580b2bf0 con 0x7fe658143ae0 2024-01-16T14:57:30.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.062+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe64001d3b0 con 0x7fe658143ae0 2024-01-16T14:57:30.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.063+0000 7fe654c00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 0x7fe63006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.063+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe640089c60 con 0x7fe658143ae0 2024-01-16T14:57:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.063+0000 7fe657e00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 0x7fe63006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.064+0000 7fe657e00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 0x7fe63006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe648038000 tx=0x7fe648008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.067+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6580b2bf0 con 0x7fe658143ae0 2024-01-16T14:57:30.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.258+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6580b2bf0 con 0x7fe658143ae0 2024-01-16T14:57:30.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.260+0000 7fe654c00640 1 -- 172.21.15.125:0/2359548075 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7fe6580b2bf0 con 0x7fe658143ae0 2024-01-16T14:57:30.242 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:30.242 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:30.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.262+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 msgr2=0x7fe63006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.262+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 0x7fe63006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe648038000 tx=0x7fe648008030 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.262+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 msgr2=0x7fe6580b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.262+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe64000ba50 tx=0x7fe64000bf10 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 shutdown_connections 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe65815e1e0 0x7fe6580b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe63006b2c0 0x7fe63006d780 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe658143ae0 0x7fe6580b0bc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 --2- 172.21.15.125:0/2359548075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6580cf7d0 0x7fe6581599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 >> 172.21.15.125:0/2359548075 conn(0x7fe65806a800 msgr2=0x7fe65815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 shutdown_connections 2024-01-16T14:57:30.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.263+0000 7fe65ea00640 1 -- 172.21.15.125:0/2359548075 wait complete. 2024-01-16T14:57:30.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:30.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:30.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.494+0000 7f38c1000640 1 Processor -- start 2024-01-16T14:57:30.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.494+0000 7f38c1000640 1 -- start start 2024-01-16T14:57:30.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38c1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f38bc070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38c1000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc0706a0 con 0x7f38bc0b2e00 2024-01-16T14:57:30.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38c1000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc0708d0 con 0x7f38bc06c730 2024-01-16T14:57:30.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38c1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0b3290 con 0x7f38bc06d220 2024-01-16T14:57:30.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38baa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f38bc0b2e00 0x7f38bc06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33394/0 (socket says 172.21.15.125:33394) 2024-01-16T14:57:30.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.496+0000 7f38baa00640 1 -- 172.21.15.125:0/1522172894 learned_addr learned my addr 172.21.15.125:0/1522172894 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38bbe00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f38bc070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982593662 0 0) 0x7f38bc0706a0 con 0x7f38bc0b2e00 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c003520 con 0x7f38bc0b2e00 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263247135 0 0) 0x7f389c003520 con 0x7f38bc0b2e00 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 msgr2=0x7f38bc070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f38bc070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 >> v1:172.21.15.137:6789/0 conn(0x7f38bc06c730 legacy=0x7f38bc071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc070bb0 con 0x7f38bc0b2e00 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f38a4001a10 con 0x7f38bc0b2e00 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 >> v1:172.21.15.125:6789/0 conn(0x7f38bc0b2e00 legacy=0x7f38bc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f389c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f389c003f70 0x7f389c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f389c004d00 0x7f389c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c00ba90 con 0x7f389c003f70 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c00bc00 con 0x7f389c004d00 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.497+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c00bdc0 con 0x7f38bc06d220 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f389c003f70 0x7f389c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38bb400640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f389c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38bbe00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f389c004d00 0x7f389c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 -- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 msgr2=0x7f389c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38bc06d220 0x7f389c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 -- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f389c004d00 msgr2=0x7f389c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f389c004d00 0x7f389c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.498+0000 7f38baa00640 1 -- 172.21.15.125:0/1522172894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c003520 con 0x7f389c003f70 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.499+0000 7f38baa00640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f389c003f70 0x7f389c0047c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f38a400a780 tx=0x7f389c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b4c6d3b2005ea3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.499+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4034630 con 0x7f389c003f70 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.499+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a402f070 con 0x7f389c003f70 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.499+0000 7f38ba000640 1 -- 172.21.15.125:0/1522172894 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a40349d0 con 0x7f389c003f70 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 -- 172.21.15.125:0/1522172894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f389c003f70 msgr2=0x7f389c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 --2- 172.21.15.125:0/1522172894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f389c003f70 0x7f389c0047c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f38a400a780 tx=0x7f389c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38bbe00640 1 -- 172.21.15.125:0/1522172894 reap_dead start 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 -- 172.21.15.125:0/1522172894 shutdown_connections 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 -- 172.21.15.125:0/1522172894 >> 172.21.15.125:0/1522172894 conn(0x7f38bc06a7d0 msgr2=0x7f38bc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 -- 172.21.15.125:0/1522172894 shutdown_connections 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.500+0000 7f38c1000640 1 -- 172.21.15.125:0/1522172894 wait complete. 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.502+0000 7f38c1000640 1 Processor -- start 2024-01-16T14:57:30.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.502+0000 7f38c1000640 1 -- start start 2024-01-16T14:57:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f389c003f70 0x7f38bc0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38bc144160 0x7f38bc0aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc1f7c60 con 0x7f38bc1f77e0 2024-01-16T14:57:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0a8bd0 con 0x7f38bc144160 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38c1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0a8eb0 con 0x7f389c003f70 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38baa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.503+0000 7f38baa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48114/0 (socket says 172.21.15.125:48114) 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 -- 172.21.15.125:0/3213076880 learned_addr learned my addr 172.21.15.125:0/3213076880 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38bb400640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f389c003f70 0x7f38bc0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38bbe00640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38bc144160 0x7f38bc0aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 -- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f389c003f70 msgr2=0x7f38bc0a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f389c003f70 0x7f38bc0a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 -- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38bc144160 msgr2=0x7f38bc0aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38bc144160 0x7f38bc0aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 -- 172.21.15.125:0/3213076880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0a9190 con 0x7f38bc1f77e0 2024-01-16T14:57:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.504+0000 7f38baa00640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f38a400a780 tx=0x7f38a4001cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.505+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4034980 con 0x7f38bc1f77e0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.505+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4034ae0 con 0x7f38bc1f77e0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.505+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4034e50 con 0x7f38bc1f77e0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.505+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc0a9470 con 0x7f38bc1f77e0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.505+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc0a9c70 con 0x7f38bc1f77e0 2024-01-16T14:57:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.506+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc067450 con 0x7f38bc1f77e0 2024-01-16T14:57:30.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.507+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f38a4031030 con 0x7f38bc1f77e0 2024-01-16T14:57:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.508+0000 7f38b3400640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 0x7f389406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.508+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f38a40abfc0 con 0x7f38bc1f77e0 2024-01-16T14:57:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.508+0000 7f38bb400640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 0x7f389406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.509+0000 7f38bb400640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 0x7f389406d7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f38ac007fd0 tx=0x7f38ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.512+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38a4076e80 con 0x7f38bc1f77e0 2024-01-16T14:57:30.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.703+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38bc067450 con 0x7f38bc1f77e0 2024-01-16T14:57:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.704+0000 7f38b3400640 1 -- 172.21.15.125:0/3213076880 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38a4032380 con 0x7f38bc1f77e0 2024-01-16T14:57:30.686 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.706+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 msgr2=0x7f389406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.706+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 0x7f389406d7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f38ac007fd0 tx=0x7f38ac009040 comp rx=0 tx=0).stop 2024-01-16T14:57:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 msgr2=0x7f38bc143c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f38a400a780 tx=0x7f38a4001cd0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 shutdown_connections 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38bc144160 0x7f38bc0aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f389406b2f0 0x7f389406d7b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38bc1f77e0 0x7f38bc143c20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 --2- 172.21.15.125:0/3213076880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f389c003f70 0x7f38bc0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 >> 172.21.15.125:0/3213076880 conn(0x7f38bc06a7d0 msgr2=0x7f38bc0cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 shutdown_connections 2024-01-16T14:57:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.707+0000 7f38c1000640 1 -- 172.21.15.125:0/3213076880 wait complete. 2024-01-16T14:57:30.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:30.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.930+0000 7f4fba200640 1 Processor -- start 2024-01-16T14:57:30.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.930+0000 7f4fba200640 1 -- start start 2024-01-16T14:57:30.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.931+0000 7f4fba200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4fb40ce8f0 0x7f4fb40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.931+0000 7f4fba200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4fb40b2a00 0x7f4fb40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.931+0000 7f4fba200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb40d0de0 con 0x7f4fb4160f80 2024-01-16T14:57:30.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.931+0000 7f4fba200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb40d10c0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.931+0000 7f4fba200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb40d13a0 con 0x7f4fb40ce8f0 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4fb40b2a00 0x7f4fb40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4fb40b2a00 0x7f4fb40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48130/0 (socket says 172.21.15.125:48130) 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 -- 172.21.15.125:0/3016519361 learned_addr learned my addr 172.21.15.125:0/3016519361 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb2a00640 1 --2- 172.21.15.125:0/3016519361 >> v2:172.21.15.137:3300/0 conn(0x7f4fb40ce8f0 0x7f4fb40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 -- 172.21.15.125:0/3016519361 >> v2:172.21.15.137:3300/0 conn(0x7f4fb40ce8f0 msgr2=0x7f4fb40ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 --2- 172.21.15.125:0/3016519361 >> v2:172.21.15.137:3300/0 conn(0x7f4fb40ce8f0 0x7f4fb40ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 -- 172.21.15.125:0/3016519361 >> v1:172.21.15.137:6789/0 conn(0x7f4fb4160f80 legacy=0x7f4fb4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.932+0000 7f4fb3e00640 1 -- 172.21.15.125:0/3016519361 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb40d1680 con 0x7f4fb40b2a00 2024-01-16T14:57:30.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.933+0000 7f4fb3e00640 1 --2- 172.21.15.125:0/3016519361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb40b2e90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b5d10 tx=0x7f4fac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1feedddc76cd47a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fb2000640 1 -- 172.21.15.125:0/3016519361 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac031870 con 0x7f4fb40b2a00 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fb2000640 1 -- 172.21.15.125:0/3016519361 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac0319d0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fb2000640 1 -- 172.21.15.125:0/3016519361 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac031d60 con 0x7f4fb40b2a00 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 -- 172.21.15.125:0/3016519361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 msgr2=0x7f4fb40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 --2- 172.21.15.125:0/3016519361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb40b2e90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b5d10 tx=0x7f4fac02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 -- 172.21.15.125:0/3016519361 shutdown_connections 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 --2- 172.21.15.125:0/3016519361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb40b2e90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 --2- 172.21.15.125:0/3016519361 >> v2:172.21.15.137:3300/0 conn(0x7f4fb40ce8f0 0x7f4fb40ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 -- 172.21.15.125:0/3016519361 >> 172.21.15.125:0/3016519361 conn(0x7f4fb409c550 msgr2=0x7f4fb40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 -- 172.21.15.125:0/3016519361 shutdown_connections 2024-01-16T14:57:30.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.934+0000 7f4fba200640 1 -- 172.21.15.125:0/3016519361 wait complete. 2024-01-16T14:57:30.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.936+0000 7f4fba200640 1 Processor -- start 2024-01-16T14:57:30.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.936+0000 7f4fba200640 1 -- start start 2024-01-16T14:57:30.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fb40ce8f0 0x7f4fb414c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fb4160f80 0x7f4fb414c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40b5e30 con 0x7f4fb40b2a00 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4150630 con 0x7f4fb40ce8f0 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fba200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40b8be0 con 0x7f4fb4160f80 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fb3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fb3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48136/0 (socket says 172.21.15.125:48136) 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.937+0000 7f4fb3400640 1 -- 172.21.15.125:0/830482386 learned_addr learned my addr 172.21.15.125:0/830482386 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb2a00640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fb40ce8f0 0x7f4fb414c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3e00640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fb4160f80 0x7f4fb414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 -- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fb4160f80 msgr2=0x7f4fb414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fb4160f80 0x7f4fb414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 -- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fb40ce8f0 msgr2=0x7f4fb414c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fb40ce8f0 0x7f4fb414c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 -- 172.21.15.125:0/830482386 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0094d0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fb3400640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400ba80 tx=0x7f4fa400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400cc00 con 0x7f4fb40b2a00 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.938+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa400cd60 con 0x7f4fb40b2a00 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.939+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4003bb0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.939+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb40b6080 con 0x7f4fb40b2a00 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.939+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb40b6640 con 0x7f4fb40b2a00 2024-01-16T14:57:30.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.940+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb40675d0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.941+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4fa4014440 con 0x7f4fb40b2a00 2024-01-16T14:57:30.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.941+0000 7f4fab400640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 0x7f4f8c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:30.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.942+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4fa40899f0 con 0x7f4fb40b2a00 2024-01-16T14:57:30.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.942+0000 7f4fb2a00640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 0x7f4f8c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:30.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.942+0000 7f4fb2a00640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 0x7f4f8c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b5300 tx=0x7f4f9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:30.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:30.946+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fa4054860 con 0x7f4fb40b2a00 2024-01-16T14:57:31.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.138+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4fb40d3920 con 0x7f4fb40b2a00 2024-01-16T14:57:31.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.139+0000 7f4fab400640 1 -- 172.21.15.125:0/830482386 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4fa4057e80 con 0x7f4fb40b2a00 2024-01-16T14:57:31.120 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 msgr2=0x7f4f8c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 0x7f4f8c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b5300 tx=0x7f4f9c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 msgr2=0x7f4fb415fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400ba80 tx=0x7f4fa400bf40 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 shutdown_connections 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fb4160f80 0x7f4fb414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fb40ce8f0 0x7f4fb414c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f8c06b2c0 0x7f4f8c06d780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 --2- 172.21.15.125:0/830482386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fb40b2a00 0x7f4fb415fe90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 >> 172.21.15.125:0/830482386 conn(0x7f4fb409c550 msgr2=0x7f4fb406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 shutdown_connections 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.142+0000 7f4fba200640 1 -- 172.21.15.125:0/830482386 wait complete. 2024-01-16T14:57:31.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:31.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.361+0000 7fad73200640 1 Processor -- start 2024-01-16T14:57:31.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.361+0000 7fad73200640 1 -- start start 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.362+0000 7fad73200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.362+0000 7fad73200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad6c15bf10 0x7fad6c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.362+0000 7fad73200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c15f4e0 con 0x7fad6c0cc980 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad73200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c15f970 con 0x7fad6c15bf10 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad73200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0cce10 con 0x7fad6c0adf20 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad6c15bf10 0x7fad6c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad6c15bf10 0x7fad6c0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48146/0 (socket says 172.21.15.125:48146) 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 -- 172.21.15.125:0/1416990401 learned_addr learned my addr 172.21.15.125:0/1416990401 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71e00640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 -- 172.21.15.125:0/1416990401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 msgr2=0x7fad6c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 -- 172.21.15.125:0/1416990401 >> v1:172.21.15.137:6789/0 conn(0x7fad6c0cc980 legacy=0x7fad6c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:31.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71400640 1 -- 172.21.15.125:0/1416990401 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c071130 con 0x7fad6c15bf10 2024-01-16T14:57:31.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.363+0000 7fad71e00640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.364+0000 7fad71400640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c0cc440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1d2fc2a3e0f7e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad70a00640 1 -- 172.21.15.125:0/1416990401 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540317c0 con 0x7fad6c15bf10 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad70a00640 1 -- 172.21.15.125:0/1416990401 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54031920 con 0x7fad6c15bf10 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad70a00640 1 -- 172.21.15.125:0/1416990401 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54031cb0 con 0x7fad6c15bf10 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 -- 172.21.15.125:0/1416990401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 msgr2=0x7fad6c0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c0cc440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5402b860 comp rx=0 tx=0).stop 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 -- 172.21.15.125:0/1416990401 shutdown_connections 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c0cc440 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 --2- 172.21.15.125:0/1416990401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 -- 172.21.15.125:0/1416990401 >> 172.21.15.125:0/1416990401 conn(0x7fad6c06a7d0 msgr2=0x7fad6c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.365+0000 7fad73200640 1 -- 172.21.15.125:0/1416990401 shutdown_connections 2024-01-16T14:57:31.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.366+0000 7fad73200640 1 -- 172.21.15.125:0/1416990401 wait complete. 2024-01-16T14:57:31.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.367+0000 7fad73200640 1 Processor -- start 2024-01-16T14:57:31.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.367+0000 7fad73200640 1 -- start start 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c0cc980 0x7fad6c142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c140450 con 0x7fad6c15bf10 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c1f3600 con 0x7fad6c0cc980 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad73200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c1f38e0 con 0x7fad6c0adf20 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad72800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.368+0000 7fad72800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48160/0 (socket says 172.21.15.125:48160) 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 -- 172.21.15.125:0/1587568122 learned_addr learned my addr 172.21.15.125:0/1587568122 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad71400640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c0cc980 0x7fad6c142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad71e00640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 -- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 msgr2=0x7fad6c1f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c1f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 -- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c0cc980 msgr2=0x7fad6c142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c0cc980 0x7fad6c142820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 -- 172.21.15.125:0/1587568122 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540094d0 con 0x7fad6c15bf10 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad71e00640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c1f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad72800640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c780 tx=0x7fad6400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64003d80 con 0x7fad6c15bf10 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6400ce90 con 0x7fad6c15bf10 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.369+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad640052b0 con 0x7fad6c15bf10 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.370+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6c1f3bc0 con 0x7fad6c15bf10 2024-01-16T14:57:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.370+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c1f4230 con 0x7fad6c15bf10 2024-01-16T14:57:31.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.371+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c067450 con 0x7fad6c15bf10 2024-01-16T14:57:31.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.372+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fad64005410 con 0x7fad6c15bf10 2024-01-16T14:57:31.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.373+0000 7fad6a000640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 0x7fad4406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.373+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fad640890a0 con 0x7fad6c15bf10 2024-01-16T14:57:31.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.373+0000 7fad71e00640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 0x7fad4406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.374+0000 7fad71e00640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 0x7fad4406d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fad5c006fd0 tx=0x7fad5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.378+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad64053f10 con 0x7fad6c15bf10 2024-01-16T14:57:31.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.566+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad6c070220 con 0x7fad6c15bf10 2024-01-16T14:57:31.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.567+0000 7fad6a000640 1 -- 172.21.15.125:0/1587568122 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad640170a0 con 0x7fad6c15bf10 2024-01-16T14:57:31.548 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:31.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.569+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 msgr2=0x7fad4406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.569+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 0x7fad4406d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fad5c006fd0 tx=0x7fad5c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:31.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 msgr2=0x7fad6c15a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c780 tx=0x7fad6400cc40 comp rx=0 tx=0).stop 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 shutdown_connections 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406b2c0 0x7fad4406d780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c15bf10 0x7fad6c15a440 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c0cc980 0x7fad6c142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 --2- 172.21.15.125:0/1587568122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c1f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 >> 172.21.15.125:0/1587568122 conn(0x7fad6c06a7d0 msgr2=0x7fad6c0cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 shutdown_connections 2024-01-16T14:57:31.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.570+0000 7fad73200640 1 -- 172.21.15.125:0/1587568122 wait complete. 2024-01-16T14:57:31.570 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:31.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.796+0000 7f9a97200640 1 Processor -- start 2024-01-16T14:57:31.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.797+0000 7f9a97200640 1 -- start start 2024-01-16T14:57:31.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a97200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a900ae3f0 0x7f9a90071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a97200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a90070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a97200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a90072640 con 0x7f9a9015b3f0 2024-01-16T14:57:31.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a97200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a900ae8a0 con 0x7f9a900ae3f0 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a97200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9015fd40 con 0x7f9a90072210 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.798+0000 7f9a8f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a900ae3f0 0x7f9a90071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a900ae3f0 0x7f9a90071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48172/0 (socket says 172.21.15.125:48172) 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 -- 172.21.15.125:0/3660056100 learned_addr learned my addr 172.21.15.125:0/3660056100 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a95200640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a90070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 -- 172.21.15.125:0/3660056100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 msgr2=0x7f9a90070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a90070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 -- 172.21.15.125:0/3660056100 >> v1:172.21.15.137:6789/0 conn(0x7f9a9015b3f0 legacy=0x7f9a900adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:31.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.799+0000 7f9a8f400640 1 -- 172.21.15.125:0/3660056100 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9015ef80 con 0x7f9a900ae3f0 2024-01-16T14:57:31.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.800+0000 7f9a8f400640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a900ae3f0 0x7f9a90071cd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9a900aeb20 tx=0x7f9a7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf927b45f5a4ad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.800+0000 7f9a8ea00640 1 -- 172.21.15.125:0/3660056100 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031830 con 0x7f9a900ae3f0 2024-01-16T14:57:31.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.800+0000 7f9a8ea00640 1 -- 172.21.15.125:0/3660056100 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78031990 con 0x7f9a900ae3f0 2024-01-16T14:57:31.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.800+0000 7f9a8ea00640 1 -- 172.21.15.125:0/3660056100 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031d20 con 0x7f9a900ae3f0 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 -- 172.21.15.125:0/3660056100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a900ae3f0 msgr2=0x7f9a90071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a900ae3f0 0x7f9a90071cd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9a900aeb20 tx=0x7f9a7802b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 -- 172.21.15.125:0/3660056100 shutdown_connections 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a90070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 --2- 172.21.15.125:0/3660056100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a900ae3f0 0x7f9a90071cd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 -- 172.21.15.125:0/3660056100 >> 172.21.15.125:0/3660056100 conn(0x7f9a900b0c00 msgr2=0x7f9a900b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 -- 172.21.15.125:0/3660056100 shutdown_connections 2024-01-16T14:57:31.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.801+0000 7f9a97200640 1 -- 172.21.15.125:0/3660056100 wait complete. 2024-01-16T14:57:31.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.803+0000 7f9a97200640 1 Processor -- start 2024-01-16T14:57:31.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.803+0000 7f9a97200640 1 -- start start 2024-01-16T14:57:31.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a900c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a900ae3f0 0x7f9a9009a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900b2510 con 0x7f9a9015b3f0 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900ca070 con 0x7f9a900ae3f0 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a97200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900ca350 con 0x7f9a90072210 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a95200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a95200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48180/0 (socket says 172.21.15.125:48180) 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a95200640 1 -- 172.21.15.125:0/817629087 learned_addr learned my addr 172.21.15.125:0/817629087 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.804+0000 7f9a8fe00640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a900c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 -- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 msgr2=0x7f9a900c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a900c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a8f400640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a900ae3f0 0x7f9a9009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 -- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a900ae3f0 msgr2=0x7f9a9009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a900ae3f0 0x7f9a9009a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 -- 172.21.15.125:0/817629087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094d0 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a95200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9a88007e60 tx=0x7f9a8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88014420 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a880145a0 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88014950 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a900b21a0 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.805+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a901583b0 con 0x7f9a9015b3f0 2024-01-16T14:57:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.806+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a900b2000 con 0x7f9a9015b3f0 2024-01-16T14:57:31.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.807+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9a88014ab0 con 0x7f9a9015b3f0 2024-01-16T14:57:31.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.808+0000 7f9a87e00640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.808+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9a8808a650 con 0x7f9a9015b3f0 2024-01-16T14:57:31.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.809+0000 7f9a8fe00640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.809+0000 7f9a8fe00640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 0x7f9a6806d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9a80003d90 tx=0x7f9a80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:31.812+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a900b2000 con 0x7f9a9015b3f0 2024-01-16T14:57:31.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.016+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a900b2000 con 0x7f9a9015b3f0 2024-01-16T14:57:31.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.017+0000 7f9a87e00640 1 -- 172.21.15.125:0/817629087 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9a900b2000 con 0x7f9a9015b3f0 2024-01-16T14:57:31.998 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 msgr2=0x7f9a6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 0x7f9a6806d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9a80003d90 tx=0x7f9a80009040 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 msgr2=0x7f9a900c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9a88007e60 tx=0x7f9a8800e4b0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 shutdown_connections 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a9015b3f0 0x7f9a900c8ed0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a900ae3f0 0x7f9a9009a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 --2- 172.21.15.125:0/817629087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a90072210 0x7f9a900c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.019+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 >> 172.21.15.125:0/817629087 conn(0x7f9a900b0c00 msgr2=0x7f9a90159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.020+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 shutdown_connections 2024-01-16T14:57:32.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.020+0000 7f9a97200640 1 -- 172.21.15.125:0/817629087 wait complete. 2024-01-16T14:57:32.020 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:32.020 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:32.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:32.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.281+0000 7f37b3200640 1 Processor -- start 2024-01-16T14:57:32.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.281+0000 7f37b3200640 1 -- start start 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.282+0000 7f37b3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 0x7f37ac13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.282+0000 7f37b3200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37ac13fae0 0x7f37ac0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.282+0000 7f37b3200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac13ff00 con 0x7f37ac13cfa0 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.282+0000 7f37b3200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac0cdc40 con 0x7f37ac13fae0 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.282+0000 7f37b3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0cdec0 con 0x7f37ac15afb0 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f37ac13cfa0 0x7f37ac0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33410/0 (socket says 172.21.15.125:33410) 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b1e00640 1 -- 172.21.15.125:0/3053105602 learned_addr learned my addr 172.21.15.125:0/3053105602 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:32.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b2800640 1 --2- 172.21.15.125:0/3053105602 >> v2:172.21.15.125:3300/0 conn(0x7f37ac13fae0 0x7f37ac0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b1400640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 0x7f37ac13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b2800640 1 -- 172.21.15.125:0/3053105602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 msgr2=0x7f37ac13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:32.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b2800640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 0x7f37ac13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b2800640 1 -- 172.21.15.125:0/3053105602 >> v1:172.21.15.125:6789/0 conn(0x7f37ac13cfa0 legacy=0x7f37ac0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b2800640 1 -- 172.21.15.125:0/3053105602 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0ce1a0 con 0x7f37ac13fae0 2024-01-16T14:57:32.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.283+0000 7f37b1400640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 0x7f37ac13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:32.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.284+0000 7f37b2800640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13fae0 0x7f37ac0b2040 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009d90 tx=0x7f37a402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b01a13343f66127 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.284+0000 7f37b0a00640 1 -- 172.21.15.125:0/3053105602 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a402cbd0 con 0x7f37ac13fae0 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.284+0000 7f37b0a00640 1 -- 172.21.15.125:0/3053105602 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a402cd30 con 0x7f37ac13fae0 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.284+0000 7f37b0a00640 1 -- 172.21.15.125:0/3053105602 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a402cea0 con 0x7f37ac13fae0 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 -- 172.21.15.125:0/3053105602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13fae0 msgr2=0x7f37ac0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13fae0 0x7f37ac0b2040 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009d90 tx=0x7f37a402be20 comp rx=0 tx=0).stop 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 -- 172.21.15.125:0/3053105602 shutdown_connections 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13fae0 0x7f37ac0b2040 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 --2- 172.21.15.125:0/3053105602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac15afb0 0x7f37ac13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 -- 172.21.15.125:0/3053105602 >> 172.21.15.125:0/3053105602 conn(0x7f37ac06a800 msgr2=0x7f37ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 -- 172.21.15.125:0/3053105602 shutdown_connections 2024-01-16T14:57:32.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.285+0000 7f37b3200640 1 -- 172.21.15.125:0/3053105602 wait complete. 2024-01-16T14:57:32.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.287+0000 7f37b3200640 1 Processor -- start 2024-01-16T14:57:32.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.287+0000 7f37b3200640 1 -- start start 2024-01-16T14:57:32.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac13fae0 0x7f37ac1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37ac15afb0 0x7f37ac152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac138c40 con 0x7f37ac13cfa0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac138940 con 0x7f37ac15afb0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac1f3410 con 0x7f37ac13fae0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48206/0 (socket says 172.21.15.125:48206) 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 -- 172.21.15.125:0/3400454973 learned_addr learned my addr 172.21.15.125:0/3400454973 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1400640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac13fae0 0x7f37ac1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 -- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac13fae0 msgr2=0x7f37ac1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac13fae0 0x7f37ac1f2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 -- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37ac15afb0 msgr2=0x7f37ac152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.288+0000 7f37b1e00640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37ac15afb0 0x7f37ac152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37b1e00640 1 -- 172.21.15.125:0/3400454973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a4009a70 con 0x7f37ac13cfa0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37b1e00640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f379c00c500 tx=0x7f379c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c005ce0 con 0x7f37ac13cfa0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c005e40 con 0x7f37ac13cfa0 2024-01-16T14:57:32.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac15d100 con 0x7f37ac13cfa0 2024-01-16T14:57:32.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c0072f0 con 0x7f37ac13cfa0 2024-01-16T14:57:32.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.289+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac15d680 con 0x7f37ac13cfa0 2024-01-16T14:57:32.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.290+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac067450 con 0x7f37ac13cfa0 2024-01-16T14:57:32.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.291+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f379c007450 con 0x7f37ac13cfa0 2024-01-16T14:57:32.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.292+0000 7f37aa000640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 0x7f378406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.292+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f379c089120 con 0x7f37ac13cfa0 2024-01-16T14:57:32.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.292+0000 7f37b1400640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 0x7f378406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.293+0000 7f37b1400640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 0x7f378406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f37ac13acc0 tx=0x7f3794008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.296+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f379c053fa0 con 0x7f37ac13cfa0 2024-01-16T14:57:32.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.473+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37ac0cbff0 con 0x7f37ac13cfa0 2024-01-16T14:57:32.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.474+0000 7f37aa000640 1 -- 172.21.15.125:0/3400454973 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f379c0575c0 con 0x7f37ac13cfa0 2024-01-16T14:57:32.456 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:32.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 msgr2=0x7f378406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 0x7f378406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f37ac13acc0 tx=0x7f3794008040 comp rx=0 tx=0).stop 2024-01-16T14:57:32.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 msgr2=0x7f37ac13a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f379c00c500 tx=0x7f379c00c9c0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 shutdown_connections 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37ac15afb0 0x7f37ac152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37ac13fae0 0x7f37ac1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f378406b2c0 0x7f378406d780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 --2- 172.21.15.125:0/3400454973 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ac13cfa0 0x7f37ac13a780 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 >> 172.21.15.125:0/3400454973 conn(0x7f37ac06a800 msgr2=0x7f37ac0aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 shutdown_connections 2024-01-16T14:57:32.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.477+0000 7f37b3200640 1 -- 172.21.15.125:0/3400454973 wait complete. 2024-01-16T14:57:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:32.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:32.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.699+0000 7f0199e00640 1 Processor -- start 2024-01-16T14:57:32.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.700+0000 7f0199e00640 1 -- start start 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.701+0000 7f0199e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0194161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.701+0000 7f0199e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f01940ce8f0 0x7f01940ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.701+0000 7f0199e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01941618a0 con 0x7f01940cfbe0 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.701+0000 7f0199e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01940d0de0 con 0x7f01940ce8f0 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.701+0000 7f0199e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940d10c0 con 0x7f0194160f80 2024-01-16T14:57:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0193400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01940cfbe0 0x7f01940d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33416/0 (socket says 172.21.15.125:33416) 2024-01-16T14:57:32.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0193400640 1 -- 172.21.15.125:0/4152650881 learned_addr learned my addr 172.21.15.125:0/4152650881 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:32.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0192a00640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0194161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0192000640 1 --2- 172.21.15.125:0/4152650881 >> v2:172.21.15.137:3300/0 conn(0x7f01940ce8f0 0x7f01940ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950960784 0 0) 0x7f01941618a0 con 0x7f01940cfbe0 2024-01-16T14:57:32.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174003520 con 0x7f01940cfbe0 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622332927 0 0) 0x7f0174003520 con 0x7f01940cfbe0 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 msgr2=0x7f0194161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0194161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 >> v2:172.21.15.137:3300/0 conn(0x7f01940ce8f0 msgr2=0x7f01940ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 --2- 172.21.15.125:0/4152650881 >> v2:172.21.15.137:3300/0 conn(0x7f01940ce8f0 0x7f01940ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940d13a0 con 0x7f01940cfbe0 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f018c0019f0 con 0x7f01940cfbe0 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.702+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 >> v1:172.21.15.125:6789/0 conn(0x7f01940cfbe0 legacy=0x7f01940d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0174003970 0x7f0174003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f0174004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0174005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940d0de0 con 0x7f01740047c0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940d10c0 con 0x7f0174003970 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017400b980 con 0x7f0194160f80 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f0174004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192a00640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0174003970 0x7f0174003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0193400640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0174005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 -- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 msgr2=0x7f0174005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0194160f80 0x7f0174005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 -- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0174003970 msgr2=0x7f0174003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0174003970 0x7f0174003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192000640 1 -- 172.21.15.125:0/4152650881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174003520 con 0x7f01740047c0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.703+0000 7f0192a00640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0174003970 0x7f0174003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.704+0000 7f0192000640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f0174004c40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f017c009ac0 tx=0x7f01740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c89e9d01284092b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c02c6d0 con 0x7f01740047c0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c02c830 con 0x7f01740047c0 2024-01-16T14:57:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0191600640 1 -- 172.21.15.125:0/4152650881 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c02cbc0 con 0x7f01740047c0 2024-01-16T14:57:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 -- 172.21.15.125:0/4152650881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 msgr2=0x7f0174004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 --2- 172.21.15.125:0/4152650881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f0174004c40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f017c009ac0 tx=0x7f01740057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0193400640 1 -- 172.21.15.125:0/4152650881 reap_dead start 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 -- 172.21.15.125:0/4152650881 shutdown_connections 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 -- 172.21.15.125:0/4152650881 >> 172.21.15.125:0/4152650881 conn(0x7f019409c550 msgr2=0x7f01940d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 -- 172.21.15.125:0/4152650881 shutdown_connections 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.705+0000 7f0199e00640 1 -- 172.21.15.125:0/4152650881 wait complete. 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.707+0000 7f0199e00640 1 Processor -- start 2024-01-16T14:57:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.707+0000 7f0199e00640 1 -- start start 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01940ccc40 0x7f01940c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 0x7f0194151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cb760 con 0x7f01740047c0 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940ca460 con 0x7f01940ccc40 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0199e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940ca740 con 0x7f01940b1230 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0192a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0192a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48226/0 (socket says 172.21.15.125:48226) 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.708+0000 7f0192a00640 1 -- 172.21.15.125:0/3899985180 learned_addr learned my addr 172.21.15.125:0/3899985180 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0193400640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 0x7f0194151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 -- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 msgr2=0x7f0194151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 0x7f0194151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192000640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01940ccc40 0x7f01940c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 -- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01940ccc40 msgr2=0x7f01940c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01940ccc40 0x7f01940c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 -- 172.21.15.125:0/3899985180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940caa20 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0193400640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 0x7f0194151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0192a00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0184010890 tx=0x7f0184010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018400eb00 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018400ec60 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.709+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01940cad00 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.710+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018401c400 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.710+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01940b16b0 con 0x7f01740047c0 2024-01-16T14:57:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.711+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01940675d0 con 0x7f01740047c0 2024-01-16T14:57:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.712+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f018401c560 con 0x7f01740047c0 2024-01-16T14:57:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.713+0000 7f018b400640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 0x7f016c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.713+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f018408a020 con 0x7f01740047c0 2024-01-16T14:57:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.713+0000 7f0192000640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 0x7f016c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:32.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.714+0000 7f0192000640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 0x7f016c06d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:32.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.717+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0184054f40 con 0x7f01740047c0 2024-01-16T14:57:32.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.905+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01940cf9b0 con 0x7f01740047c0 2024-01-16T14:57:32.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.906+0000 7f018b400640 1 -- 172.21.15.125:0/3899985180 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0184006c30 con 0x7f01740047c0 2024-01-16T14:57:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 msgr2=0x7f016c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 0x7f016c06d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c030000 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 msgr2=0x7f01940cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0184010890 tx=0x7f0184010d50 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 shutdown_connections 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01940b1230 0x7f0194151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01940ccc40 0x7f01940c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f016c06b2c0 0x7f016c06d780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 --2- 172.21.15.125:0/3899985180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01740047c0 0x7f01940cb220 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 >> 172.21.15.125:0/3899985180 conn(0x7f019409c550 msgr2=0x7f01940d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 shutdown_connections 2024-01-16T14:57:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:32.908+0000 7f0199e00640 1 -- 172.21.15.125:0/3899985180 wait complete. 2024-01-16T14:57:32.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:33.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.131+0000 7f77a8800640 1 Processor -- start 2024-01-16T14:57:33.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.131+0000 7f77a8800640 1 -- start start 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a8800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77a013df80 0x7f77a015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a8800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77a013e360 0x7f77a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a8800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a006c470 con 0x7f77a015f9c0 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a8800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a006c750 con 0x7f77a013df80 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a8800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a006ca30 con 0x7f77a013e360 2024-01-16T14:57:33.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77a013df80 0x7f77a015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77a013df80 0x7f77a015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48232/0 (socket says 172.21.15.125:48232) 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a5e00640 1 -- 172.21.15.125:0/1811825950 learned_addr learned my addr 172.21.15.125:0/1811825950 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.133+0000 7f77a6800640 1 --1- 172.21.15.125:0/1811825950 >> v1:172.21.15.125:6789/0 conn(0x7f77a015f9c0 0x7f77a01408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33428/0 (socket says 172.21.15.125:33428) 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.134+0000 7f77a5400640 1 --2- 172.21.15.125:0/1811825950 >> v2:172.21.15.137:3300/0 conn(0x7f77a013e360 0x7f77a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.134+0000 7f77a5e00640 1 -- 172.21.15.125:0/1811825950 >> v2:172.21.15.137:3300/0 conn(0x7f77a013e360 msgr2=0x7f77a00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.134+0000 7f77a5e00640 1 --2- 172.21.15.125:0/1811825950 >> v2:172.21.15.137:3300/0 conn(0x7f77a013e360 0x7f77a00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.134+0000 7f77a5e00640 1 -- 172.21.15.125:0/1811825950 >> v1:172.21.15.125:6789/0 conn(0x7f77a015f9c0 legacy=0x7f77a01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.134+0000 7f77a5e00640 1 -- 172.21.15.125:0/1811825950 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a006cd10 con 0x7f77a013df80 2024-01-16T14:57:33.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.135+0000 7f77a5e00640 1 --2- 172.21.15.125:0/1811825950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a013df80 0x7f77a015bee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77a006ded0 tx=0x7f779002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5423b70018f942e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.136+0000 7f77a4a00640 1 -- 172.21.15.125:0/1811825950 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790030860 con 0x7f77a013df80 2024-01-16T14:57:33.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.136+0000 7f77a4a00640 1 -- 172.21.15.125:0/1811825950 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77900309c0 con 0x7f77a013df80 2024-01-16T14:57:33.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.136+0000 7f77a4a00640 1 -- 172.21.15.125:0/1811825950 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790030d50 con 0x7f77a013df80 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.136+0000 7f77a8800640 1 -- 172.21.15.125:0/1811825950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a013df80 msgr2=0x7f77a015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.136+0000 7f77a8800640 1 --2- 172.21.15.125:0/1811825950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a013df80 0x7f77a015bee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77a006ded0 tx=0x7f779002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 -- 172.21.15.125:0/1811825950 shutdown_connections 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 --2- 172.21.15.125:0/1811825950 >> v2:172.21.15.137:3300/0 conn(0x7f77a013e360 0x7f77a00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 --2- 172.21.15.125:0/1811825950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a013df80 0x7f77a015bee0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 -- 172.21.15.125:0/1811825950 >> 172.21.15.125:0/1811825950 conn(0x7f77a006a800 msgr2=0x7f77a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:33.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 -- 172.21.15.125:0/1811825950 shutdown_connections 2024-01-16T14:57:33.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.137+0000 7f77a8800640 1 -- 172.21.15.125:0/1811825950 wait complete. 2024-01-16T14:57:33.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.138+0000 7f77a8800640 1 Processor -- start 2024-01-16T14:57:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.138+0000 7f77a8800640 1 -- start start 2024-01-16T14:57:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 0x7f77a01f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 0x7f77a00b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01335a0 con 0x7f77a015f9c0 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01f2a00 con 0x7f77a013e360 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.139+0000 7f77a8800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0132640 con 0x7f77a013df80 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48238/0 (socket says 172.21.15.125:48238) 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 -- 172.21.15.125:0/2957068516 learned_addr learned my addr 172.21.15.125:0/2957068516 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a5400640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 0x7f77a00b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a5e00640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 0x7f77a01f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 -- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 msgr2=0x7f77a01f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 0x7f77a01f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 -- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 msgr2=0x7f77a00b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 0x7f77a00b3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 -- 172.21.15.125:0/2957068516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77900094d0 con 0x7f77a015f9c0 2024-01-16T14:57:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a5e00640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 0x7f77a01f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a5400640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 0x7f77a00b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.140+0000 7f77a6800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f779800c7a0 tx=0x7f779800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.141+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779800a640 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.141+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779800a7a0 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.141+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a0132920 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.141+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779800ab10 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.141+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0132ee0 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.142+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a01336e0 con 0x7f77a015f9c0 2024-01-16T14:57:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.143+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f779800ac70 con 0x7f77a015f9c0 2024-01-16T14:57:33.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.144+0000 7f779e000640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 0x7f777806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.144+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f77980892d0 con 0x7f77a015f9c0 2024-01-16T14:57:33.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.144+0000 7f77a5e00640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 0x7f777806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.145+0000 7f77a5e00640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 0x7f777806d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7790036000 tx=0x7f779003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.148+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77a01336e0 con 0x7f77a015f9c0 2024-01-16T14:57:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.336+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f77a01336e0 con 0x7f77a015f9c0 2024-01-16T14:57:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.337+0000 7f779e000640 1 -- 172.21.15.125:0/2957068516 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f77a01336e0 con 0x7f77a015f9c0 2024-01-16T14:57:33.319 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:33.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.339+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 msgr2=0x7f777806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.339+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 0x7f777806d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7790036000 tx=0x7f779003b040 comp rx=0 tx=0).stop 2024-01-16T14:57:33.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.339+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 msgr2=0x7f77a00b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.339+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f779800c7a0 tx=0x7f779800cc60 comp rx=0 tx=0).stop 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 shutdown_connections 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f777806b2c0 0x7f777806d780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a015f9c0 0x7f77a00b2cf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77a013e360 0x7f77a00b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 --2- 172.21.15.125:0/2957068516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77a013df80 0x7f77a01f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 >> 172.21.15.125:0/2957068516 conn(0x7f77a006a800 msgr2=0x7f77a015a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 shutdown_connections 2024-01-16T14:57:33.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.340+0000 7f77a8800640 1 -- 172.21.15.125:0/2957068516 wait complete. 2024-01-16T14:57:33.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:33.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.565+0000 7f72e7600640 1 Processor -- start 2024-01-16T14:57:33.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.565+0000 7f72e7600640 1 -- start start 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.566+0000 7f72e7600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72e013df80 0x7f72e015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e7600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e7600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e006c470 con 0x7f72e013e360 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e7600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e006c750 con 0x7f72e013df80 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e7600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e006ca30 con 0x7f72e00cf9a0 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72e013df80 0x7f72e015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72e013df80 0x7f72e015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48242/0 (socket says 172.21.15.125:48242) 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e4c00640 1 -- 172.21.15.125:0/2488650484 learned_addr learned my addr 172.21.15.125:0/2488650484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e5600640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.567+0000 7f72e4c00640 1 -- 172.21.15.125:0/2488650484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 msgr2=0x7f72e00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.568+0000 7f72e4c00640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e00cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.568+0000 7f72e4c00640 1 -- 172.21.15.125:0/2488650484 >> v1:172.21.15.137:6789/0 conn(0x7f72e013e360 legacy=0x7f72e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.568+0000 7f72e4c00640 1 -- 172.21.15.125:0/2488650484 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e006cd10 con 0x7f72e013df80 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.568+0000 7f72e5600640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.569+0000 7f72e4c00640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013df80 0x7f72e015bee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009e50 tx=0x7f72d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666ec6434b7b49ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.569+0000 7f72dfe00640 1 -- 172.21.15.125:0/2488650484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d002ca70 con 0x7f72e013df80 2024-01-16T14:57:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.569+0000 7f72dfe00640 1 -- 172.21.15.125:0/2488650484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d002cbd0 con 0x7f72e013df80 2024-01-16T14:57:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.569+0000 7f72dfe00640 1 -- 172.21.15.125:0/2488650484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d00343f0 con 0x7f72e013df80 2024-01-16T14:57:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 -- 172.21.15.125:0/2488650484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013df80 msgr2=0x7f72e015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013df80 0x7f72e015bee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009e50 tx=0x7f72d002bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 -- 172.21.15.125:0/2488650484 shutdown_connections 2024-01-16T14:57:33.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 --2- 172.21.15.125:0/2488650484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013df80 0x7f72e015bee0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 -- 172.21.15.125:0/2488650484 >> 172.21.15.125:0/2488650484 conn(0x7f72e006a800 msgr2=0x7f72e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:33.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 -- 172.21.15.125:0/2488650484 shutdown_connections 2024-01-16T14:57:33.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.570+0000 7f72e7600640 1 -- 172.21.15.125:0/2488650484 wait complete. 2024-01-16T14:57:33.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.571+0000 7f72e7600640 1 Processor -- start 2024-01-16T14:57:33.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.572+0000 7f72e7600640 1 -- start start 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e0130720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72e013df80 0x7f72e01504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e0130c60 con 0x7f72e013e360 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e0153310 con 0x7f72e013df80 2024-01-16T14:57:33.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e7600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e0152700 con 0x7f72e00cf9a0 2024-01-16T14:57:33.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48248/0 (socket says 172.21.15.125:48248) 2024-01-16T14:57:33.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 -- 172.21.15.125:0/3278841211 learned_addr learned my addr 172.21.15.125:0/3278841211 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72d7c00640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72e013df80 0x7f72e01504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 -- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 msgr2=0x7f72e0130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e0130720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 -- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72e013df80 msgr2=0x7f72e01504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72e013df80 0x7f72e01504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.573+0000 7f72e5600640 1 -- 172.21.15.125:0/3278841211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0009b30 con 0x7f72e013e360 2024-01-16T14:57:33.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72e5600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f72d800c5c0 tx=0x7f72d800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8011800 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8011980 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e01529e0 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8011d30 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.574+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e0152fa0 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.575+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e00675d0 con 0x7f72e013e360 2024-01-16T14:57:33.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.576+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f72d8022350 con 0x7f72e013e360 2024-01-16T14:57:33.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.577+0000 7f72dd600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 0x7f72b806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.577+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f72d8089db0 con 0x7f72e013e360 2024-01-16T14:57:33.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.578+0000 7f72e4c00640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 0x7f72b806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.578+0000 7f72e4c00640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 0x7f72b806d7b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72d0035000 tx=0x7f72d002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.582+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72d8054b10 con 0x7f72e013e360 2024-01-16T14:57:33.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.772+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72e0153480 con 0x7f72e013e360 2024-01-16T14:57:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.773+0000 7f72dd600640 1 -- 172.21.15.125:0/3278841211 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f72d8058130 con 0x7f72e013e360 2024-01-16T14:57:33.755 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 msgr2=0x7f72b806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 0x7f72b806d7b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72d0035000 tx=0x7f72d002e000 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 msgr2=0x7f72e01f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f72d800c5c0 tx=0x7f72d800ca80 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 shutdown_connections 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72b806b2f0 0x7f72b806d7b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72e013e360 0x7f72e01f3240 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72e013df80 0x7f72e01504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 --2- 172.21.15.125:0/3278841211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72e00cf9a0 0x7f72e0130720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 >> 172.21.15.125:0/3278841211 conn(0x7f72e006a800 msgr2=0x7f72e015a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.776+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 shutdown_connections 2024-01-16T14:57:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.777+0000 7f72e7600640 1 -- 172.21.15.125:0/3278841211 wait complete. 2024-01-16T14:57:33.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.998+0000 7f4ebfe00640 1 Processor -- start 2024-01-16T14:57:33.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:33.998+0000 7f4ebfe00640 1 -- start start 2024-01-16T14:57:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.000+0000 7f4ebfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.000+0000 7f4ebfe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.000+0000 7f4ebfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec00d0a10 con 0x7f4ec0161010 2024-01-16T14:57:33.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.000+0000 7f4ebfe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec00d03b0 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.000+0000 7f4ebfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00d1f50 con 0x7f4ec0160920 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:52628/0 (socket says 172.21.15.125:52628) 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4ebea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 -- 172.21.15.125:0/1212283076 learned_addr learned my addr 172.21.15.125:0/1212283076 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 -- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 msgr2=0x7f4ec00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 --2- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 -- 172.21.15.125:0/1212283076 >> v1:172.21.15.137:6789/0 conn(0x7f4ec0161010 legacy=0x7f4ec01614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.001+0000 7f4eb5e00640 1 -- 172.21.15.125:0/1212283076 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00cf080 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.003+0000 7f4eb5e00640 1 --2- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009870 tx=0x7f4ea802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ead62d3b7b1248b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.003+0000 7f4ebe000640 1 -- 172.21.15.125:0/1212283076 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8030840 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.003+0000 7f4ebe000640 1 -- 172.21.15.125:0/1212283076 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea80309a0 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.003+0000 7f4ebe000640 1 -- 172.21.15.125:0/1212283076 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8030d30 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00cf9f0 msgr2=0x7f4ec00cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009870 tx=0x7f4ea802b980 comp rx=0 tx=0).stop 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1212283076 shutdown_connections 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec00cfe70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1212283076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1212283076 >> 172.21.15.125:0/1212283076 conn(0x7f4ec00d3090 msgr2=0x7f4ec00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1212283076 shutdown_connections 2024-01-16T14:57:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.004+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1212283076 wait complete. 2024-01-16T14:57:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.006+0000 7f4ebfe00640 1 Processor -- start 2024-01-16T14:57:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.006+0000 7f4ebfe00640 1 -- start start 2024-01-16T14:57:33.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 0x7f4ec014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00b6500 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00cc410 con 0x7f4ec0161010 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00cc6f0 con 0x7f4ec0160920 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48252/0 (socket says 172.21.15.125:48252) 2024-01-16T14:57:33.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.007+0000 7f4ebea00640 1 -- 172.21.15.125:0/1500949387 learned_addr learned my addr 172.21.15.125:0/1500949387 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:33.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4eb5e00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebf400640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 0x7f4ec014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 -- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 msgr2=0x7f4ec00cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 -- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 msgr2=0x7f4ec014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 0x7f4ec014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 -- 172.21.15.125:0/1500949387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea80094d0 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebf400640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 0x7f4ec014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebea00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000efc0 tx=0x7f4eb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0014480 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb000f040 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec00cc9d0 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0014800 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.008+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec00cc100 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.009+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec0067450 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.010+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4eb0014960 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.011+0000 7f4eb7400640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 0x7f4e9806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:33.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.011+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4eb008a820 con 0x7f4ec00cf9f0 2024-01-16T14:57:33.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.011+0000 7f4eb5e00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 0x7f4e9806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:33.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.012+0000 7f4eb5e00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 0x7f4e9806d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:33.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.016+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4eb00555f0 con 0x7f4ec00cf9f0 2024-01-16T14:57:34.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.222+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4ec008c7e0 con 0x7f4ec00cf9f0 2024-01-16T14:57:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.223+0000 7f4eb7400640 1 -- 172.21.15.125:0/1500949387 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4eb0058c10 con 0x7f4ec00cf9f0 2024-01-16T14:57:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:34.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.225+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 msgr2=0x7f4e9806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.225+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 0x7f4e9806d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea803b040 comp rx=0 tx=0).stop 2024-01-16T14:57:34.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.225+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 msgr2=0x7f4ec015fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.225+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000efc0 tx=0x7f4eb000c490 comp rx=0 tx=0).stop 2024-01-16T14:57:34.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 shutdown_connections 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec0161010 0x7f4ec014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec0160920 0x7f4ec00cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e9806b2f0 0x7f4e9806d7b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 --2- 172.21.15.125:0/1500949387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec00cf9f0 0x7f4ec015fd20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 >> 172.21.15.125:0/1500949387 conn(0x7f4ec00d3090 msgr2=0x7f4ec00cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 shutdown_connections 2024-01-16T14:57:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.226+0000 7f4ebfe00640 1 -- 172.21.15.125:0/1500949387 wait complete. 2024-01-16T14:57:34.226 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T14:57:34.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:34.227 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:34.318 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:34.318 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:57:34.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:34.319 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T14:57:34.374 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:34.374 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:34.431 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:34.431 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:34.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:34.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.712+0000 7f48cd400640 1 Processor -- start 2024-01-16T14:57:34.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.713+0000 7f48cd400640 1 -- start start 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.714+0000 7f48cd400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48c80cf9a0 0x7f48c815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.714+0000 7f48cd400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c8070c40 con 0x7f48c80b3ae0 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.714+0000 7f48cd400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c806c470 con 0x7f48c80ae5a0 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.714+0000 7f48cd400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c8071450 con 0x7f48c80cf9a0 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c5e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f48c80b3ae0 0x7f48c80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33440/0 (socket says 172.21.15.125:33440) 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c5e00640 1 -- 172.21.15.125:0/796001381 learned_addr learned my addr 172.21.15.125:0/796001381 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c7200640 1 --2- 172.21.15.125:0/796001381 >> v2:172.21.15.125:3300/0 conn(0x7f48c80cf9a0 0x7f48c815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c7200640 1 -- 172.21.15.125:0/796001381 >> v1:172.21.15.137:6789/0 conn(0x7f48c80ae5a0 legacy=0x7f48c806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c7200640 1 -- 172.21.15.125:0/796001381 >> v1:172.21.15.125:6789/0 conn(0x7f48c80b3ae0 legacy=0x7f48c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.715+0000 7f48c7200640 1 -- 172.21.15.125:0/796001381 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c806c750 con 0x7f48c80cf9a0 2024-01-16T14:57:34.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.717+0000 7f48c7200640 1 --2- 172.21.15.125:0/796001381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80cf9a0 0x7f48c815e430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009d90 tx=0x7f48c002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bae0d82f13b4518 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:34.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.717+0000 7f48c5400640 1 -- 172.21.15.125:0/796001381 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c002cd80 con 0x7f48c80cf9a0 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.717+0000 7f48c5400640 1 -- 172.21.15.125:0/796001381 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c0034440 con 0x7f48c80cf9a0 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.717+0000 7f48c5400640 1 -- 172.21.15.125:0/796001381 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c00347d0 con 0x7f48c80cf9a0 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 -- 172.21.15.125:0/796001381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80cf9a0 msgr2=0x7f48c815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 --2- 172.21.15.125:0/796001381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80cf9a0 0x7f48c815e430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009d90 tx=0x7f48c002bfd0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 -- 172.21.15.125:0/796001381 shutdown_connections 2024-01-16T14:57:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 --2- 172.21.15.125:0/796001381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80cf9a0 0x7f48c815e430 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 -- 172.21.15.125:0/796001381 >> 172.21.15.125:0/796001381 conn(0x7f48c806a800 msgr2=0x7f48c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 -- 172.21.15.125:0/796001381 shutdown_connections 2024-01-16T14:57:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.718+0000 7f48cd400640 1 -- 172.21.15.125:0/796001381 wait complete. 2024-01-16T14:57:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.720+0000 7f48cd400640 1 Processor -- start 2024-01-16T14:57:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.720+0000 7f48cd400640 1 -- start start 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.721+0000 7f48cd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.721+0000 7f48cd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48c80b3ae0 0x7f48c8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.721+0000 7f48cd400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 0x7f48c8149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48cd400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80cb980 con 0x7f48c80ae5a0 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48cd400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80ce620 con 0x7f48c80cf9a0 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48cd400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8157b20 con 0x7f48c80b3ae0 2024-01-16T14:57:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48256/0 (socket says 172.21.15.125:48256) 2024-01-16T14:57:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 -- 172.21.15.125:0/4876730 learned_addr learned my addr 172.21.15.125:0/4876730 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c7200640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 0x7f48c8149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 -- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48c80b3ae0 msgr2=0x7f48c8158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48c80b3ae0 0x7f48c8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 -- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 msgr2=0x7f48c8149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 0x7f48c8149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c6800640 1 -- 172.21.15.125:0/4876730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0009a70 con 0x7f48c80ae5a0 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.722+0000 7f48c7200640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 0x7f48c8149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48c6800640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c450 tx=0x7f48b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b80052d0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8005430 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8012870 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8157e00 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.723+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c81583c0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.724+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c80cbac0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.725+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f48b80129d0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.726+0000 7f48bea00640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 0x7f48a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.726+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f48b80897a0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.726+0000 7f48c5e00640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 0x7f48a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.727+0000 7f48c5e00640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 0x7f48a006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48c81f6920 tx=0x7f48b0007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:34.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.730+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48c80cbac0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.922+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48c80cbac0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.924+0000 7f48bea00640 1 -- 172.21.15.125:0/4876730 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f48c80cbac0 con 0x7f48c80ae5a0 2024-01-16T14:57:34.906 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:34.906 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:34.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.926+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 msgr2=0x7f48a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.926+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 0x7f48a006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48c81f6920 tx=0x7f48b0007a40 comp rx=0 tx=0).stop 2024-01-16T14:57:34.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.926+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 msgr2=0x7f48c80cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:34.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.926+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c450 tx=0x7f48b800c910 comp rx=0 tx=0).stop 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 shutdown_connections 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48c80cf9a0 0x7f48c8149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48c80b3ae0 0x7f48c8158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48a006b2c0 0x7f48a006d780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 --2- 172.21.15.125:0/4876730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c80ae5a0 0x7f48c80cd9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 >> 172.21.15.125:0/4876730 conn(0x7f48c806a800 msgr2=0x7f48c80b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 shutdown_connections 2024-01-16T14:57:34.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:34.927+0000 7f48cd400640 1 -- 172.21.15.125:0/4876730 wait complete. 2024-01-16T14:57:34.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:34.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.155+0000 7f2602200640 1 Processor -- start 2024-01-16T14:57:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.155+0000 7f2602200640 1 -- start start 2024-01-16T14:57:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.156+0000 7f2602200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f25fc143ae0 0x7f25fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.156+0000 7f2602200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.156+0000 7f2602200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0cfc30 con 0x7f25fc15e1e0 2024-01-16T14:57:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f2602200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc0b1ee0 con 0x7f25fc143ae0 2024-01-16T14:57:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f2602200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0b2780 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f25f3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f25f3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:59052/0 (socket says 172.21.15.125:59052) 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f25f3e00640 1 -- 172.21.15.125:0/3423741213 learned_addr learned my addr 172.21.15.125:0/3423741213 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.157+0000 7f25fb400640 1 --2- 172.21.15.125:0/3423741213 >> v2:172.21.15.137:3300/0 conn(0x7f25fc143ae0 0x7f25fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.158+0000 7f25f3e00640 1 -- 172.21.15.125:0/3423741213 >> v2:172.21.15.137:3300/0 conn(0x7f25fc143ae0 msgr2=0x7f25fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.158+0000 7f25f3e00640 1 --2- 172.21.15.125:0/3423741213 >> v2:172.21.15.137:3300/0 conn(0x7f25fc143ae0 0x7f25fc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.158+0000 7f25f3e00640 1 -- 172.21.15.125:0/3423741213 >> v1:172.21.15.137:6789/0 conn(0x7f25fc15e1e0 legacy=0x7f25fc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.158+0000 7f25f3e00640 1 -- 172.21.15.125:0/3423741213 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc15e990 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.160+0000 7f25f3e00640 1 --2- 172.21.15.125:0/3423741213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009870 tx=0x7f25e402db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad8b65497f22a2f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.160+0000 7f25faa00640 1 -- 172.21.15.125:0/3423741213 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e402e940 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f25faa00640 1 -- 172.21.15.125:0/3423741213 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e402eaa0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f25faa00640 1 -- 172.21.15.125:0/3423741213 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e402ee30 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 -- 172.21.15.125:0/3423741213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 msgr2=0x7f25fc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 --2- 172.21.15.125:0/3423741213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009870 tx=0x7f25e402db90 comp rx=0 tx=0).stop 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 -- 172.21.15.125:0/3423741213 shutdown_connections 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 --2- 172.21.15.125:0/3423741213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc0cf7d0 0x7f25fc15dca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 --2- 172.21.15.125:0/3423741213 >> v2:172.21.15.137:3300/0 conn(0x7f25fc143ae0 0x7f25fc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 -- 172.21.15.125:0/3423741213 >> 172.21.15.125:0/3423741213 conn(0x7f25fc06a800 msgr2=0x7f25fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 -- 172.21.15.125:0/3423741213 shutdown_connections 2024-01-16T14:57:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.161+0000 7f2602200640 1 -- 172.21.15.125:0/3423741213 wait complete. 2024-01-16T14:57:35.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.163+0000 7f2602200640 1 Processor -- start 2024-01-16T14:57:35.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.163+0000 7f2602200640 1 -- start start 2024-01-16T14:57:35.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc143ae0 0x7f25fc13f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 0x7f25fc13f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc141980 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc140d90 con 0x7f25fc15e1e0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f2602200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc140fe0 con 0x7f25fc143ae0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f25fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f25fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48258/0 (socket says 172.21.15.125:48258) 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.164+0000 7f25fb400640 1 -- 172.21.15.125:0/1444592216 learned_addr learned my addr 172.21.15.125:0/1444592216 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fbe00640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 0x7f25fc13f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25f3e00640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc143ae0 0x7f25fc13f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 -- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc143ae0 msgr2=0x7f25fc13f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc143ae0 0x7f25fc13f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 -- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 msgr2=0x7f25fc13f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 0x7f25fc13f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 -- 172.21.15.125:0/1444592216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40094d0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fbe00640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 0x7f25fc13f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:35.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25fb400640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00e9b0 tx=0x7f25ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.165+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec009670 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.166+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec00c9a0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.166+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec00cd50 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.166+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc1412c0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.166+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc13fdb0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.167+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc0675d0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.168+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25ec013470 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.169+0000 7f25f3400640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 0x7f25d406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.169+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f25ec089f20 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.169+0000 7f25f3e00640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 0x7f25d406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.170+0000 7f25f3e00640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 0x7f25d406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f25e4038000 tx=0x7f25e4032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.173+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25ec054cf0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.365+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25fc0ae290 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.366+0000 7f25f3400640 1 -- 172.21.15.125:0/1444592216 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25ec0170a0 con 0x7f25fc0cf7d0 2024-01-16T14:57:35.347 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 msgr2=0x7f25d406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 0x7f25d406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f25e4038000 tx=0x7f25e4032000 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 msgr2=0x7f25fc141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00e9b0 tx=0x7f25ec00ee70 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 shutdown_connections 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25fc15e1e0 0x7f25fc13f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25fc143ae0 0x7f25fc13f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f25d406b2c0 0x7f25d406d780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 --2- 172.21.15.125:0/1444592216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25fc0cf7d0 0x7f25fc141da0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.368+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 >> 172.21.15.125:0/1444592216 conn(0x7f25fc06a800 msgr2=0x7f25fc15ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:35.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.369+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 shutdown_connections 2024-01-16T14:57:35.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.369+0000 7f2602200640 1 -- 172.21.15.125:0/1444592216 wait complete. 2024-01-16T14:57:35.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:35.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.600+0000 7ff433000640 1 Processor -- start 2024-01-16T14:57:35.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.600+0000 7ff433000640 1 -- start start 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff433000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 0x7ff42c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff433000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff42c070ea0 0x7ff42c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff433000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c071690 con 0x7ff42c14b2c0 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff433000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c14b6a0 con 0x7ff42c070ea0 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff433000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c071970 con 0x7ff42c0cf9a0 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff431000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff42c070ea0 0x7ff42c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff431000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff42c070ea0 0x7ff42c0cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48260/0 (socket says 172.21.15.125:48260) 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.602+0000 7ff431000640 1 -- 172.21.15.125:0/3462492394 learned_addr learned my addr 172.21.15.125:0/3462492394 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff42b400640 1 --2- 172.21.15.125:0/3462492394 >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 0x7ff42c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff431000640 1 -- 172.21.15.125:0/3462492394 >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 msgr2=0x7ff42c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff431000640 1 --2- 172.21.15.125:0/3462492394 >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 0x7ff42c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff431000640 1 -- 172.21.15.125:0/3462492394 >> v1:172.21.15.137:6789/0 conn(0x7ff42c14b2c0 legacy=0x7ff42c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff431000640 1 -- 172.21.15.125:0/3462492394 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c071c50 con 0x7ff42c070ea0 2024-01-16T14:57:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.603+0000 7ff42b400640 1 --2- 172.21.15.125:0/3462492394 >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 0x7ff42c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.604+0000 7ff431000640 1 --2- 172.21.15.125:0/3462492394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c070ea0 0x7ff42c0cc160 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0cfea0 tx=0x7ff42402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b020900f956f53 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.604+0000 7ff42aa00640 1 -- 172.21.15.125:0/3462492394 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42402ca80 con 0x7ff42c070ea0 2024-01-16T14:57:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.604+0000 7ff42aa00640 1 -- 172.21.15.125:0/3462492394 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42402cbe0 con 0x7ff42c070ea0 2024-01-16T14:57:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.604+0000 7ff42aa00640 1 -- 172.21.15.125:0/3462492394 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240343f0 con 0x7ff42c070ea0 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 -- 172.21.15.125:0/3462492394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c070ea0 msgr2=0x7ff42c0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 --2- 172.21.15.125:0/3462492394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c070ea0 0x7ff42c0cc160 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0cfea0 tx=0x7ff42402bee0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 -- 172.21.15.125:0/3462492394 shutdown_connections 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 --2- 172.21.15.125:0/3462492394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c070ea0 0x7ff42c0cc160 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 --2- 172.21.15.125:0/3462492394 >> v2:172.21.15.137:3300/0 conn(0x7ff42c0cf9a0 0x7ff42c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 -- 172.21.15.125:0/3462492394 >> 172.21.15.125:0/3462492394 conn(0x7ff42c06a800 msgr2=0x7ff42c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 -- 172.21.15.125:0/3462492394 shutdown_connections 2024-01-16T14:57:35.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.605+0000 7ff433000640 1 -- 172.21.15.125:0/3462492394 wait complete. 2024-01-16T14:57:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.607+0000 7ff433000640 1 Processor -- start 2024-01-16T14:57:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.607+0000 7ff433000640 1 -- start start 2024-01-16T14:57:35.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff42c070ea0 0x7ff42c1f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff42c0cf9a0 0x7ff42c161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c163290 con 0x7ff42c14b2c0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c161860 con 0x7ff42c0cf9a0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.608+0000 7ff433000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1623b0 con 0x7ff42c070ea0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48266/0 (socket says 172.21.15.125:48266) 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 -- 172.21.15.125:0/3680199557 learned_addr learned my addr 172.21.15.125:0/3680199557 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff42b400640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff42c0cf9a0 0x7ff42c161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff42be00640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff42c070ea0 0x7ff42c1f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 -- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff42c070ea0 msgr2=0x7ff42c1f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff42c070ea0 0x7ff42c1f8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 -- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff42c0cf9a0 msgr2=0x7ff42c161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff42c0cf9a0 0x7ff42c161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 -- 172.21.15.125:0/3680199557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff424009b30 con 0x7ff42c14b2c0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.609+0000 7ff431000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff424036040 tx=0x7ff424004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.610+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42403e450 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.610+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42403e5d0 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.610+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c162680 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.610+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42403e980 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.610+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c162c40 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.611+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c163440 con 0x7ff42c14b2c0 2024-01-16T14:57:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.612+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff42402cd50 con 0x7ff42c14b2c0 2024-01-16T14:57:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.613+0000 7ff423e00640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 0x7ff40406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.613+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff4240ac5c0 con 0x7ff42c14b2c0 2024-01-16T14:57:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.613+0000 7ff42be00640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 0x7ff40406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.614+0000 7ff42be00640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 0x7ff40406d7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00a750 tx=0x7ff41c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.617+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff42c163440 con 0x7ff42c14b2c0 2024-01-16T14:57:35.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.807+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff42c0675d0 con 0x7ff42c14b2c0 2024-01-16T14:57:35.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.808+0000 7ff423e00640 1 -- 172.21.15.125:0/3680199557 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff424077300 con 0x7ff42c14b2c0 2024-01-16T14:57:35.789 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 msgr2=0x7ff40406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 0x7ff40406d7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00a750 tx=0x7ff41c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 msgr2=0x7ff42c161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff424036040 tx=0x7ff424004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 shutdown_connections 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff40406b2f0 0x7ff40406d7b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff42c14b2c0 0x7ff42c161e70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff42c0cf9a0 0x7ff42c161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.810+0000 7ff433000640 1 --2- 172.21.15.125:0/3680199557 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff42c070ea0 0x7ff42c1f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.811+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 >> 172.21.15.125:0/3680199557 conn(0x7ff42c06a800 msgr2=0x7ff42c0af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.811+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 shutdown_connections 2024-01-16T14:57:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:35.811+0000 7ff433000640 1 -- 172.21.15.125:0/3680199557 wait complete. 2024-01-16T14:57:35.812 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:36.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.046+0000 7f055ac00640 1 Processor -- start 2024-01-16T14:57:36.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.046+0000 7f055ac00640 1 -- start start 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f055ac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 0x7f055415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f055ac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05540cf9a0 0x7f05540cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f055ac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055406c470 con 0x7f055413e360 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f055ac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055406c750 con 0x7f05540cf9a0 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f055ac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055406ca30 con 0x7f055413df80 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f0558c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05540cf9a0 0x7f05540cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f0558c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05540cf9a0 0x7f05540cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48272/0 (socket says 172.21.15.125:48272) 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.048+0000 7f0558c00640 1 -- 172.21.15.125:0/1760115359 learned_addr learned my addr 172.21.15.125:0/1760115359 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:36.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0553e00640 1 --2- 172.21.15.125:0/1760115359 >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 0x7f055415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0558c00640 1 -- 172.21.15.125:0/1760115359 >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 msgr2=0x7f055415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0558c00640 1 --2- 172.21.15.125:0/1760115359 >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 0x7f055415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0558c00640 1 -- 172.21.15.125:0/1760115359 >> v1:172.21.15.137:6789/0 conn(0x7f055413e360 legacy=0x7f05540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0558c00640 1 -- 172.21.15.125:0/1760115359 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055406cd10 con 0x7f05540cf9a0 2024-01-16T14:57:36.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.049+0000 7f0553e00640 1 --2- 172.21.15.125:0/1760115359 >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 0x7f055415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:36.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.050+0000 7f0558c00640 1 --2- 172.21.15.125:0/1760115359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05540cf9a0 0x7f05540cd840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0554072770 tx=0x7f054c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae81d1ad5fd524d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.050+0000 7f0552a00640 1 -- 172.21.15.125:0/1760115359 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c02ca80 con 0x7f05540cf9a0 2024-01-16T14:57:36.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.050+0000 7f0552a00640 1 -- 172.21.15.125:0/1760115359 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054c02cbe0 con 0x7f05540cf9a0 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.050+0000 7f0552a00640 1 -- 172.21.15.125:0/1760115359 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c0343f0 con 0x7f05540cf9a0 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 -- 172.21.15.125:0/1760115359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05540cf9a0 msgr2=0x7f05540cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 --2- 172.21.15.125:0/1760115359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05540cf9a0 0x7f05540cd840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0554072770 tx=0x7f054c02bee0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 -- 172.21.15.125:0/1760115359 shutdown_connections 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 --2- 172.21.15.125:0/1760115359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05540cf9a0 0x7f05540cd840 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 --2- 172.21.15.125:0/1760115359 >> v2:172.21.15.137:3300/0 conn(0x7f055413df80 0x7f055415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 -- 172.21.15.125:0/1760115359 >> 172.21.15.125:0/1760115359 conn(0x7f055406a800 msgr2=0x7f055406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 -- 172.21.15.125:0/1760115359 shutdown_connections 2024-01-16T14:57:36.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.051+0000 7f055ac00640 1 -- 172.21.15.125:0/1760115359 wait complete. 2024-01-16T14:57:36.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.053+0000 7f055ac00640 1 Processor -- start 2024-01-16T14:57:36.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.053+0000 7f055ac00640 1 -- start start 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05540cf9a0 0x7f055413d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f055413e360 0x7f05541531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055414c900 con 0x7f055413df80 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055414bad0 con 0x7f05540cf9a0 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f055ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055414bcc0 con 0x7f055413e360 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f0553400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f0553400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48274/0 (socket says 172.21.15.125:48274) 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.054+0000 7f0553400640 1 -- 172.21.15.125:0/819411803 learned_addr learned my addr 172.21.15.125:0/819411803 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553e00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05540cf9a0 0x7f055413d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0558c00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f055413e360 0x7f05541531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 -- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f055413e360 msgr2=0x7f05541531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f055413e360 0x7f05541531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 -- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05540cf9a0 msgr2=0x7f055413d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05540cf9a0 0x7f055413d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 -- 172.21.15.125:0/819411803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c009b30 con 0x7f055413df80 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f0553400640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f053c00c400 tx=0x7f053c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011790 con 0x7f055413df80 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c011930 con 0x7f055413df80 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055414cd20 con 0x7f055413df80 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011ce0 con 0x7f055413df80 2024-01-16T14:57:36.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.055+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055414c240 con 0x7f055413df80 2024-01-16T14:57:36.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.057+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05540675d0 con 0x7f055413df80 2024-01-16T14:57:36.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.057+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f053c01c460 con 0x7f055413df80 2024-01-16T14:57:36.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.058+0000 7f054be00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 0x7f052c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.058+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f053c089be0 con 0x7f055413df80 2024-01-16T14:57:36.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.059+0000 7f0553e00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 0x7f052c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.059+0000 7f0553e00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 0x7f052c06d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0544008850 tx=0x7f0544008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.063+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f053c0549c0 con 0x7f055413df80 2024-01-16T14:57:36.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.253+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05540cead0 con 0x7f055413df80 2024-01-16T14:57:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.253+0000 7f054be00640 1 -- 172.21.15.125:0/819411803 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f053c057fe0 con 0x7f055413df80 2024-01-16T14:57:36.235 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 msgr2=0x7f052c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 0x7f052c06d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0544008850 tx=0x7f0544008040 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 msgr2=0x7f0554152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f053c00c400 tx=0x7f053c00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 shutdown_connections 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f055413e360 0x7f05541531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f052c06b2c0 0x7f052c06d780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f055413df80 0x7f0554152c60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 --2- 172.21.15.125:0/819411803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05540cf9a0 0x7f055413d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 >> 172.21.15.125:0/819411803 conn(0x7f055406a800 msgr2=0x7f055415d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 shutdown_connections 2024-01-16T14:57:36.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.256+0000 7f055ac00640 1 -- 172.21.15.125:0/819411803 wait complete. 2024-01-16T14:57:36.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:36.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:36.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.472+0000 7f0279c00640 1 Processor -- start 2024-01-16T14:57:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.472+0000 7f0279c00640 1 -- start start 2024-01-16T14:57:36.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0279c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f027413df80 0x7f027415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0279c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 0x7f02740cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0279c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027406c470 con 0x7f027413e360 2024-01-16T14:57:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0279c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027406c750 con 0x7f027413df80 2024-01-16T14:57:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0279c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027406ca30 con 0x7f02740cf9a0 2024-01-16T14:57:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0272a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f027413df80 0x7f027415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0272a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f027413df80 0x7f027415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48290/0 (socket says 172.21.15.125:48290) 2024-01-16T14:57:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0272a00640 1 -- 172.21.15.125:0/1072876204 learned_addr learned my addr 172.21.15.125:0/1072876204 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.474+0000 7f0273400640 1 --2- 172.21.15.125:0/1072876204 >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 0x7f02740cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.475+0000 7f0272a00640 1 -- 172.21.15.125:0/1072876204 >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 msgr2=0x7f02740cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.475+0000 7f0272a00640 1 --2- 172.21.15.125:0/1072876204 >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 0x7f02740cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.475+0000 7f0272a00640 1 -- 172.21.15.125:0/1072876204 >> v1:172.21.15.137:6789/0 conn(0x7f027413e360 legacy=0x7f02740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.475+0000 7f0272a00640 1 -- 172.21.15.125:0/1072876204 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027406cd10 con 0x7f027413df80 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.475+0000 7f0273400640 1 --2- 172.21.15.125:0/1072876204 >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 0x7f02740cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.476+0000 7f0272a00640 1 --2- 172.21.15.125:0/1072876204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027415bee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f02740ccb10 tx=0x7f026402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e54460bc43578c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.476+0000 7f0272000640 1 -- 172.21.15.125:0/1072876204 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264031870 con 0x7f027413df80 2024-01-16T14:57:36.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0272000640 1 -- 172.21.15.125:0/1072876204 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02640319d0 con 0x7f027413df80 2024-01-16T14:57:36.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0272000640 1 -- 172.21.15.125:0/1072876204 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264031d60 con 0x7f027413df80 2024-01-16T14:57:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 -- 172.21.15.125:0/1072876204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 msgr2=0x7f027415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 --2- 172.21.15.125:0/1072876204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027415bee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f02740ccb10 tx=0x7f026402b910 comp rx=0 tx=0).stop 2024-01-16T14:57:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 -- 172.21.15.125:0/1072876204 shutdown_connections 2024-01-16T14:57:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 --2- 172.21.15.125:0/1072876204 >> v2:172.21.15.137:3300/0 conn(0x7f02740cf9a0 0x7f02740cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 --2- 172.21.15.125:0/1072876204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027415bee0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 -- 172.21.15.125:0/1072876204 >> 172.21.15.125:0/1072876204 conn(0x7f027406a800 msgr2=0x7f027406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.477+0000 7f0279c00640 1 -- 172.21.15.125:0/1072876204 shutdown_connections 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.478+0000 7f0279c00640 1 -- 172.21.15.125:0/1072876204 wait complete. 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.479+0000 7f0279c00640 1 Processor -- start 2024-01-16T14:57:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.479+0000 7f0279c00640 1 -- start start 2024-01-16T14:57:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02740cf9a0 0x7f02740a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 0x7f0274152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274153710 con 0x7f027413df80 2024-01-16T14:57:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274070410 con 0x7f027413e360 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f0279c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02740706f0 con 0x7f02740cf9a0 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.480+0000 7f026be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48292/0 (socket says 172.21.15.125:48292) 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 -- 172.21.15.125:0/1142436034 learned_addr learned my addr 172.21.15.125:0/1142436034 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 -- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02740cf9a0 msgr2=0x7f02740a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f0273400640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 0x7f0274152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02740cf9a0 0x7f02740a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 -- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 msgr2=0x7f0274152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 0x7f0274152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 -- 172.21.15.125:0/1142436034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640094d0 con 0x7f027413df80 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f0273400640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 0x7f0274152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026be00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c400 tx=0x7f025c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.481+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c011790 con 0x7f027413df80 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.482+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c011930 con 0x7f027413df80 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.482+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02740709d0 con 0x7f027413df80 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.482+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c011ce0 con 0x7f027413df80 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.482+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027413af90 con 0x7f027413df80 2024-01-16T14:57:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.483+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0274070ef0 con 0x7f027413df80 2024-01-16T14:57:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.484+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f025c01c460 con 0x7f027413df80 2024-01-16T14:57:36.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.485+0000 7f026aa00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 0x7f024c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.485+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f025c089bf0 con 0x7f027413df80 2024-01-16T14:57:36.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.485+0000 7f0272a00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 0x7f024c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.485+0000 7f0272a00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 0x7f024c06d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f0264030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.489+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0274070ef0 con 0x7f027413df80 2024-01-16T14:57:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.693+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0274070ef0 con 0x7f027413df80 2024-01-16T14:57:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.694+0000 7f026aa00640 1 -- 172.21.15.125:0/1142436034 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0274070ef0 con 0x7f027413df80 2024-01-16T14:57:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 msgr2=0x7f024c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 0x7f024c06d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f0264030000 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 msgr2=0x7f027413aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c400 tx=0x7f025c00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 shutdown_connections 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f027413e360 0x7f0274152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f024c06b2c0 0x7f024c06d780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f027413df80 0x7f027413aa50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 --2- 172.21.15.125:0/1142436034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02740cf9a0 0x7f02740a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 >> 172.21.15.125:0/1142436034 conn(0x7f027406a800 msgr2=0x7f027406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 shutdown_connections 2024-01-16T14:57:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.697+0000 7f0279c00640 1 -- 172.21.15.125:0/1142436034 wait complete. 2024-01-16T14:57:36.698 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:36.698 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:36.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:36.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.984+0000 7f14e2e00640 1 Processor -- start 2024-01-16T14:57:36.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.984+0000 7f14e2e00640 1 -- start start 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.985+0000 7f14e2e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14dc15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.985+0000 7f14e2e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc0cdec0 con 0x7f14dc13fcb0 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.985+0000 7f14e2e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc0ce1a0 con 0x7f14dc0cd530 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.985+0000 7f14e2e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc0ce480 con 0x7f14dc13cfa0 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.986+0000 7f14e2400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f14dc13fcb0 0x7f14dc0b2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46048/0 (socket says 172.21.15.125:46048) 2024-01-16T14:57:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.986+0000 7f14e2400640 1 -- 172.21.15.125:0/761874393 learned_addr learned my addr 172.21.15.125:0/761874393 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.986+0000 7f14e1a00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14dc15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.986+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29977644 0 0) 0x7f14dc0cdec0 con 0x7f14dc13fcb0 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.986+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc003520 con 0x7f14dc13fcb0 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860372233 0 0) 0x7f14bc003520 con 0x7f14dc13fcb0 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 msgr2=0x7f14dc15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14dc15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 >> v1:172.21.15.137:6789/0 conn(0x7f14dc0cd530 legacy=0x7f14dc13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc0ce760 con 0x7f14dc13fcb0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f14d4001670 con 0x7f14dc13fcb0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 >> v1:172.21.15.125:6789/0 conn(0x7f14dc13fcb0 legacy=0x7f14dc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14bc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14bc004d00 0x7f14bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc0ce480 con 0x7f14bc003f70 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc00bb70 con 0x7f14bc004d00 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc00bd30 con 0x7f14dc13cfa0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14e1000640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14e1a00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14bc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.987+0000 7f14e2400640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14bc004d00 0x7f14bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.988+0000 7f14e1000640 1 -- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 msgr2=0x7f14bc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.988+0000 7f14e1000640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13cfa0 0x7f14bc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.988+0000 7f14e1000640 1 -- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14bc004d00 msgr2=0x7f14bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.988+0000 7f14e1000640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14bc004d00 0x7f14bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.988+0000 7f14e1000640 1 -- 172.21.15.125:0/761874393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc003520 con 0x7f14bc003f70 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14e1000640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14bc0047c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f14c4001330 tx=0x7f14bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fba6b9910f30912 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c40052d0 con 0x7f14bc003f70 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c4005430 con 0x7f14bc003f70 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14dbe00640 1 -- 172.21.15.125:0/761874393 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4003830 con 0x7f14bc003f70 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14e2e00640 1 -- 172.21.15.125:0/761874393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 msgr2=0x7f14bc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14e2e00640 1 --2- 172.21.15.125:0/761874393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14bc0047c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f14c4001330 tx=0x7f14bc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.989+0000 7f14e2400640 1 -- 172.21.15.125:0/761874393 reap_dead start 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.990+0000 7f14e2e00640 1 -- 172.21.15.125:0/761874393 shutdown_connections 2024-01-16T14:57:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.990+0000 7f14e2e00640 1 -- 172.21.15.125:0/761874393 >> 172.21.15.125:0/761874393 conn(0x7f14dc06a800 msgr2=0x7f14dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.990+0000 7f14e2e00640 1 -- 172.21.15.125:0/761874393 shutdown_connections 2024-01-16T14:57:36.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.990+0000 7f14e2e00640 1 -- 172.21.15.125:0/761874393 wait complete. 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.991+0000 7f14e2e00640 1 Processor -- start 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.991+0000 7f14e2e00640 1 -- start start 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 0x7f14dc13ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13d410 0x7f14dc15c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc154f40 con 0x7f14bc003f70 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc13ee90 con 0x7f14dc154ac0 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.992+0000 7f14e2e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc13f0c0 con 0x7f14dc13d410 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47318/0 (socket says 172.21.15.125:47318) 2024-01-16T14:57:36.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 -- 172.21.15.125:0/3042734709 learned_addr learned my addr 172.21.15.125:0/3042734709 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 -- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13d410 msgr2=0x7f14dc15c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13d410 0x7f14dc15c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1000640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 0x7f14dc13ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 -- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 msgr2=0x7f14dc13ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 0x7f14dc13ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 -- 172.21.15.125:0/3042734709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc13f360 con 0x7f14bc003f70 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1000640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 0x7f14dc13ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.993+0000 7f14e1a00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f14cc007f60 tx=0x7f14cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.994+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00e030 con 0x7f14bc003f70 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.994+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc016470 con 0x7f14bc003f70 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.994+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc016820 con 0x7f14bc003f70 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.994+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc13f640 con 0x7f14bc003f70 2024-01-16T14:57:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.994+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc13d890 con 0x7f14bc003f70 2024-01-16T14:57:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.995+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc1582c0 con 0x7f14bc003f70 2024-01-16T14:57:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.996+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f14cc016b90 con 0x7f14bc003f70 2024-01-16T14:57:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.996+0000 7f14d9600640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 0x7f14b406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.997+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f14cc00f070 con 0x7f14bc003f70 2024-01-16T14:57:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.997+0000 7f14e1000640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 0x7f14b406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:36.997+0000 7f14e1000640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 0x7f14b406d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14c4036000 tx=0x7f14c4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:36.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.001+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14cc055c40 con 0x7f14bc003f70 2024-01-16T14:57:37.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.193+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14dc067450 con 0x7f14bc003f70 2024-01-16T14:57:37.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.197+0000 7f14d9600640 1 -- 172.21.15.125:0/3042734709 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f14cc059260 con 0x7f14bc003f70 2024-01-16T14:57:37.178 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:37.178 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 msgr2=0x7f14b406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 0x7f14b406d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14c4036000 tx=0x7f14c4009030 comp rx=0 tx=0).stop 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 msgr2=0x7f14dc159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f14cc007f60 tx=0x7f14cc007f90 comp rx=0 tx=0).stop 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 shutdown_connections 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.199+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14dc13d410 0x7f14dc15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14dc154ac0 0x7f14dc13ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14b406b2c0 0x7f14b406d780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 --2- 172.21.15.125:0/3042734709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14bc003f70 0x7f14dc159ec0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 >> 172.21.15.125:0/3042734709 conn(0x7f14dc06a800 msgr2=0x7f14dc071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:37.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 shutdown_connections 2024-01-16T14:57:37.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.200+0000 7f14e2e00640 1 -- 172.21.15.125:0/3042734709 wait complete. 2024-01-16T14:57:37.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:37.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.422+0000 7f5c57c00640 1 Processor -- start 2024-01-16T14:57:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.423+0000 7f5c57c00640 1 -- start start 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.424+0000 7f5c57c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 0x7f5c500aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.424+0000 7f5c57c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c500b3670 con 0x7f5c500b27c0 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.424+0000 7f5c57c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c500b1d20 con 0x7f5c5015d240 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.424+0000 7f5c57c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500b38d0 con 0x7f5c5006d7d0 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c55c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5c500b27c0 0x7f5c5015d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46054/0 (socket says 172.21.15.125:46054) 2024-01-16T14:57:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c55c00640 1 -- 172.21.15.125:0/3866301917 learned_addr learned my addr 172.21.15.125:0/3866301917 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4fe00640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 0x7f5c500aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093929917 0 0) 0x7f5c500b3670 con 0x7f5c500b27c0 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c30003520 con 0x7f5c500b27c0 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2511095236 0 0) 0x7f5c30003520 con 0x7f5c500b27c0 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 msgr2=0x7f5c500aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 0x7f5c500aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.425+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 >> v1:172.21.15.137:6789/0 conn(0x7f5c5015d240 legacy=0x7f5c500b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500b1fc0 con 0x7f5c500b27c0 2024-01-16T14:57:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5c48001a10 con 0x7f5c500b27c0 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 >> v1:172.21.15.125:6789/0 conn(0x7f5c500b27c0 legacy=0x7f5c5015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c30003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 0x7f5c300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c30004d00 0x7f5c30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500b38d0 con 0x7f5c300039a0 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3000ba90 con 0x7f5c30004d00 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3000bc50 con 0x7f5c5006d7d0 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c30003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55c00640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c30004d00 0x7f5c30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 -- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 msgr2=0x7f5c300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c5006d7d0 0x7f5c300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 -- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c30004d00 msgr2=0x7f5c30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c30004d00 0x7f5c30005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.426+0000 7f5c55200640 1 -- 172.21.15.125:0/3866301917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30003520 con 0x7f5c300039a0 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.427+0000 7f5c55200640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c30003d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001860 tx=0x7f5c300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82679d77b789fc59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40033630 con 0x7f5c300039a0 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400337b0 con 0x7f5c300039a0 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c4f400640 1 -- 172.21.15.125:0/3866301917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40033b60 con 0x7f5c300039a0 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c57c00640 1 -- 172.21.15.125:0/3866301917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 msgr2=0x7f5c30003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3866301917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c30003d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001860 tx=0x7f5c300057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c55c00640 1 -- 172.21.15.125:0/3866301917 reap_dead start 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c57c00640 1 -- 172.21.15.125:0/3866301917 shutdown_connections 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.428+0000 7f5c57c00640 1 -- 172.21.15.125:0/3866301917 >> 172.21.15.125:0/3866301917 conn(0x7f5c5006b4f0 msgr2=0x7f5c5006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.429+0000 7f5c57c00640 1 -- 172.21.15.125:0/3866301917 shutdown_connections 2024-01-16T14:57:37.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.429+0000 7f5c57c00640 1 -- 172.21.15.125:0/3866301917 wait complete. 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.430+0000 7f5c57c00640 1 Processor -- start 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.430+0000 7f5c57c00640 1 -- start start 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 0x7f5c501420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c50142610 0x7f5c500b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500afde0 con 0x7f5c300039a0 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c50152aa0 con 0x7f5c50152600 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.431+0000 7f5c57c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c501554e0 con 0x7f5c50142610 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47338/0 (socket says 172.21.15.125:47338) 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 -- 172.21.15.125:0/3453957865 learned_addr learned my addr 172.21.15.125:0/3453957865 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c4fe00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 0x7f5c501420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c50142610 0x7f5c500b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 -- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c50142610 msgr2=0x7f5c500b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c50142610 0x7f5c500b0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 -- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 msgr2=0x7f5c501420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 0x7f5c501420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 -- 172.21.15.125:0/3453957865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c501557c0 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c4fe00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 0x7f5c501420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.432+0000 7f5c55200640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5c4002bd20 tx=0x7f5c40001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.433+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40007050 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.433+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400071b0 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.433+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40007540 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.433+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c50151e30 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.433+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c500b0740 con 0x7f5c300039a0 2024-01-16T14:57:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.434+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c500675d0 con 0x7f5c300039a0 2024-01-16T14:57:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.435+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5c400067d0 con 0x7f5c300039a0 2024-01-16T14:57:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.436+0000 7f5c4cc00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 0x7f5c2806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.436+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5c400abf10 con 0x7f5c300039a0 2024-01-16T14:57:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.436+0000 7f5c4fe00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 0x7f5c2806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.437+0000 7f5c4fe00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 0x7f5c2806d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5c5015cc50 tx=0x7f5c38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.440+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c40076d60 con 0x7f5c300039a0 2024-01-16T14:57:37.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.631+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c500725d0 con 0x7f5c300039a0 2024-01-16T14:57:37.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.632+0000 7f5c4cc00640 1 -- 172.21.15.125:0/3453957865 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c4007a380 con 0x7f5c300039a0 2024-01-16T14:57:37.614 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 msgr2=0x7f5c2806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 0x7f5c2806d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5c5015cc50 tx=0x7f5c38008040 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 msgr2=0x7f5c50142b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5c4002bd20 tx=0x7f5c40001230 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 shutdown_connections 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c50142610 0x7f5c500b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c50152600 0x7f5c501420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c2806b2c0 0x7f5c2806d780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 --2- 172.21.15.125:0/3453957865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c300039a0 0x7f5c50142b00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 >> 172.21.15.125:0/3453957865 conn(0x7f5c5006b4f0 msgr2=0x7f5c500adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 shutdown_connections 2024-01-16T14:57:37.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.635+0000 7f5c57c00640 1 -- 172.21.15.125:0/3453957865 wait complete. 2024-01-16T14:57:37.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.852+0000 7f6f9b000640 1 Processor -- start 2024-01-16T14:57:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.852+0000 7f6f9b000640 1 -- start start 2024-01-16T14:57:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413df80 0x7f6f9415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 0x7f6f940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9415f9c0 0x7f6f940cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9406c470 con 0x7f6f9415f9c0 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9406c750 con 0x7f6f9413e360 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f9b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9406ca30 con 0x7f6f9413df80 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.854+0000 7f6f99000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9415f9c0 0x7f6f940cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9415f9c0 0x7f6f940cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47346/0 (socket says 172.21.15.125:47346) 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 -- 172.21.15.125:0/1190530846 learned_addr learned my addr 172.21.15.125:0/1190530846 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f8b200640 1 --2- 172.21.15.125:0/1190530846 >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 0x7f6f940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f93e00640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413df80 0x7f6f9415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 -- 172.21.15.125:0/1190530846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413df80 msgr2=0x7f6f9415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413df80 0x7f6f9415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 -- 172.21.15.125:0/1190530846 >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 msgr2=0x7f6f940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 --2- 172.21.15.125:0/1190530846 >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 0x7f6f940cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f99000640 1 -- 172.21.15.125:0/1190530846 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9406cd10 con 0x7f6f9415f9c0 2024-01-16T14:57:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.855+0000 7f6f8b200640 1 --2- 172.21.15.125:0/1190530846 >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 0x7f6f940cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.856+0000 7f6f99000640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f940cf7d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6f940cfed0 tx=0x7f6f8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d64f1e006f5e754 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.856+0000 7f6f93400640 1 -- 172.21.15.125:0/1190530846 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c02ca80 con 0x7f6f9415f9c0 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f93400640 1 -- 172.21.15.125:0/1190530846 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c02cbe0 con 0x7f6f9415f9c0 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f93400640 1 -- 172.21.15.125:0/1190530846 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0343f0 con 0x7f6f9415f9c0 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 -- 172.21.15.125:0/1190530846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 msgr2=0x7f6f940cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f940cf7d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6f940cfed0 tx=0x7f6f8c02bee0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 -- 172.21.15.125:0/1190530846 shutdown_connections 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f940cf7d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1190530846 >> v2:172.21.15.137:3300/0 conn(0x7f6f9413e360 0x7f6f940cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1190530846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413df80 0x7f6f9415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 -- 172.21.15.125:0/1190530846 >> 172.21.15.125:0/1190530846 conn(0x7f6f9406a800 msgr2=0x7f6f9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:37.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.857+0000 7f6f9b000640 1 -- 172.21.15.125:0/1190530846 shutdown_connections 2024-01-16T14:57:37.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.858+0000 7f6f9b000640 1 -- 172.21.15.125:0/1190530846 wait complete. 2024-01-16T14:57:37.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.859+0000 7f6f9b000640 1 Processor -- start 2024-01-16T14:57:37.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.859+0000 7f6f9b000640 1 -- start start 2024-01-16T14:57:37.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.860+0000 7f6f9b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f9413df80 0x7f6f940b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.860+0000 7f6f9b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 0x7f6f94159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.860+0000 7f6f9b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.860+0000 7f6f9b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941f1b90 con 0x7f6f9415f9c0 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f9b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941f1e70 con 0x7f6f9413df80 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f9b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941f2150 con 0x7f6f9413e360 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47358/0 (socket says 172.21.15.125:47358) 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 -- 172.21.15.125:0/1973327594 learned_addr learned my addr 172.21.15.125:0/1973327594 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f93e00640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f9413df80 0x7f6f940b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f8b200640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 0x7f6f94159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 -- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 msgr2=0x7f6f94159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 0x7f6f94159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 -- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f9413df80 msgr2=0x7f6f940b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f9413df80 0x7f6f940b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:37.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f99000640 1 -- 172.21.15.125:0/1973327594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c009b30 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.861+0000 7f6f8b200640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 0x7f6f94159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f99000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c036040 tx=0x7f6f8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c009510 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c03e450 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c03e800 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f941f2430 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.862+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f940b34a0 con 0x7f6f9415f9c0 2024-01-16T14:57:37.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.863+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f940b3630 con 0x7f6f9415f9c0 2024-01-16T14:57:37.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.864+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6f8c02cd50 con 0x7f6f9415f9c0 2024-01-16T14:57:37.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.865+0000 7f6f90c00640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 0x7f6f6c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:37.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.865+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6f8c0ac7b0 con 0x7f6f9415f9c0 2024-01-16T14:57:37.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.865+0000 7f6f93e00640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 0x7f6f6c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:37.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.866+0000 7f6f93e00640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 0x7f6f6c06d780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f84006fd0 tx=0x7f6f84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:37.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:37.869+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f8c077580 con 0x7f6f9415f9c0 2024-01-16T14:57:38.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.060+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f940675d0 con 0x7f6f9415f9c0 2024-01-16T14:57:38.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.061+0000 7f6f90c00640 1 -- 172.21.15.125:0/1973327594 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f8c03e960 con 0x7f6f9415f9c0 2024-01-16T14:57:38.043 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 msgr2=0x7f6f6c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 0x7f6f6c06d780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f84006fd0 tx=0x7f6f84008040 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 msgr2=0x7f6f9413aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c036040 tx=0x7f6f8c004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 shutdown_connections 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f6c06b2c0 0x7f6f6c06d780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9415f9c0 0x7f6f9413aaa0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f9413e360 0x7f6f94159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 --2- 172.21.15.125:0/1973327594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f9413df80 0x7f6f940b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 >> 172.21.15.125:0/1973327594 conn(0x7f6f9406a800 msgr2=0x7f6f9413e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 shutdown_connections 2024-01-16T14:57:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.064+0000 7f6f9b000640 1 -- 172.21.15.125:0/1973327594 wait complete. 2024-01-16T14:57:38.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:38.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.290+0000 7f3436800640 1 Processor -- start 2024-01-16T14:57:38.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.290+0000 7f3436800640 1 -- start start 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3436800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f34300ccb50 0x7f3430070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3436800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343015f4e0 con 0x7f34300adf20 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3436800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3430071130 con 0x7f343015bf10 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3436800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343015f970 con 0x7f34300ccb50 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3435400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f34300adf20 0x7f34300cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46058/0 (socket says 172.21.15.125:46058) 2024-01-16T14:57:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3435400640 1 -- 172.21.15.125:0/2013435390 learned_addr learned my addr 172.21.15.125:0/2013435390 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122064521 0 0) 0x7f343015f4e0 con 0x7f34300adf20 2024-01-16T14:57:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3410003520 con 0x7f34300adf20 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.292+0000 7f3434a00640 1 --2- 172.21.15.125:0/2013435390 >> v2:172.21.15.137:3300/0 conn(0x7f34300ccb50 0x7f3430070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437500022 0 0) 0x7f3410003520 con 0x7f34300adf20 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 >> v2:172.21.15.137:3300/0 conn(0x7f34300ccb50 msgr2=0x7f3430070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 --2- 172.21.15.125:0/2013435390 >> v2:172.21.15.137:3300/0 conn(0x7f34300ccb50 0x7f3430070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 >> v1:172.21.15.137:6789/0 conn(0x7f343015bf10 legacy=0x7f3430143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343015ebb0 con 0x7f34300adf20 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3420001670 con 0x7f34300adf20 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f3434a00640 1 --2- 172.21.15.125:0/2013435390 >> v2:172.21.15.137:3300/0 conn(0x7f34300ccb50 0x7f3430070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 >> v1:172.21.15.125:6789/0 conn(0x7f34300adf20 legacy=0x7f34300cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 0x7f3410003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34100047c0 0x7f3410004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100052a0 0x7f341000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343015f970 con 0x7f34100039c0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341000c1c0 con 0x7f34100047c0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341000c380 con 0x7f34100052a0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.293+0000 7f3435400640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 0x7f3410003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3434a00640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34100047c0 0x7f3410004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435400640 1 -- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100052a0 msgr2=0x7f341000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435e00640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100052a0 0x7f341000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435400640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100052a0 0x7f341000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435400640 1 -- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34100047c0 msgr2=0x7f3410004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435400640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34100047c0 0x7f3410004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435400640 1 -- 172.21.15.125:0/2013435390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3410003520 con 0x7f34100039c0 2024-01-16T14:57:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.294+0000 7f3435e00640 1 -- 172.21.15.125:0/2013435390 reap_dead start 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.295+0000 7f3435400640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 0x7f3410003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f342000c0e0 tx=0x7f34100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b137771d9bff49ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.295+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420005ad0 con 0x7f34100039c0 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.295+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420005c30 con 0x7f34100039c0 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.295+0000 7f342fe00640 1 -- 172.21.15.125:0/2013435390 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342002c7d0 con 0x7f34100039c0 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 -- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 msgr2=0x7f3410003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 0x7f3410003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f342000c0e0 tx=0x7f34100057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 -- 172.21.15.125:0/2013435390 shutdown_connections 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 --2- 172.21.15.125:0/2013435390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34100039c0 0x7f3410003de0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 -- 172.21.15.125:0/2013435390 >> 172.21.15.125:0/2013435390 conn(0x7f343006a7d0 msgr2=0x7f343006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 -- 172.21.15.125:0/2013435390 shutdown_connections 2024-01-16T14:57:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.296+0000 7f3436800640 1 -- 172.21.15.125:0/2013435390 wait complete. 2024-01-16T14:57:38.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.297+0000 7f3436800640 1 Processor -- start 2024-01-16T14:57:38.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.298+0000 7f3436800640 1 -- start start 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.298+0000 7f3436800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100039c0 0x7f3430152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.298+0000 7f3436800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34300b3770 0x7f34300ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3436800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3436800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343013f8e0 con 0x7f34300b3bf0 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3436800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343013e840 con 0x7f34300b3770 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3436800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343013eb20 con 0x7f34100039c0 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47368/0 (socket says 172.21.15.125:47368) 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 -- 172.21.15.125:0/51065363 learned_addr learned my addr 172.21.15.125:0/51065363 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 -- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100039c0 msgr2=0x7f3430152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435400640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100039c0 0x7f3430152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3434a00640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34300b3770 0x7f34300ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100039c0 0x7f3430152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 -- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34300b3770 msgr2=0x7f34300ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34300b3770 0x7f34300ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 -- 172.21.15.125:0/51065363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343013ee00 con 0x7f34300b3bf0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.299+0000 7f3435e00640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f343015c2f0 tx=0x7f342800d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.300+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428003d80 con 0x7f34300b3bf0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.300+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428005070 con 0x7f34300b3bf0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.300+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280053e0 con 0x7f34300b3bf0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.300+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343013f0e0 con 0x7f34300b3bf0 2024-01-16T14:57:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.300+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3430152ab0 con 0x7f34300b3bf0 2024-01-16T14:57:38.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.301+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3430067450 con 0x7f34300b3bf0 2024-01-16T14:57:38.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.302+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3428005540 con 0x7f34300b3bf0 2024-01-16T14:57:38.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.303+0000 7f342d600640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 0x7f340806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.303+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f34280893c0 con 0x7f34300b3bf0 2024-01-16T14:57:38.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.303+0000 7f3435400640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 0x7f340806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.303+0000 7f3435400640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 0x7f340806d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3420034000 tx=0x7f3420009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.307+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3428054240 con 0x7f34300b3bf0 2024-01-16T14:57:38.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.499+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f343006cf10 con 0x7f34300b3bf0 2024-01-16T14:57:38.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.500+0000 7f342d600640 1 -- 172.21.15.125:0/51065363 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34280015b0 con 0x7f34300b3bf0 2024-01-16T14:57:38.481 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 msgr2=0x7f340806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 0x7f340806d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3420034000 tx=0x7f3420009030 comp rx=0 tx=0).stop 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 msgr2=0x7f34300ace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f343015c2f0 tx=0x7f342800d970 comp rx=0 tx=0).stop 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 shutdown_connections 2024-01-16T14:57:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f340806b2c0 0x7f340806d780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34300b3bf0 0x7f34300ace10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34300b3770 0x7f34300ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.502+0000 7f3436800640 1 --2- 172.21.15.125:0/51065363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34100039c0 0x7f3430152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.503+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 >> 172.21.15.125:0/51065363 conn(0x7f343006a7d0 msgr2=0x7f343015dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.503+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 shutdown_connections 2024-01-16T14:57:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.503+0000 7f3436800640 1 -- 172.21.15.125:0/51065363 wait complete. 2024-01-16T14:57:38.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:38.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.728+0000 7f4f61a00640 1 Processor -- start 2024-01-16T14:57:38.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.728+0000 7f4f61a00640 1 -- start start 2024-01-16T14:57:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f61a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f5c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f61a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4f5c15e1e0 0x7f4f5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f61a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0b1ee0 con 0x7f4f5c143ae0 2024-01-16T14:57:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f61a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0b2780 con 0x7f4f5c15e1e0 2024-01-16T14:57:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f61a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c15e990 con 0x7f4f5c0cf9a0 2024-01-16T14:57:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f5aa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4f5c143ae0 0x7f4f5c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46072/0 (socket says 172.21.15.125:46072) 2024-01-16T14:57:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.730+0000 7f4f5aa00640 1 -- 172.21.15.125:0/3484509705 learned_addr learned my addr 172.21.15.125:0/3484509705 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5b400640 1 --2- 172.21.15.125:0/3484509705 >> v2:172.21.15.137:3300/0 conn(0x7f4f5c15e1e0 0x7f4f5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f51e00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f5c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358962261 0 0) 0x7f4f5c0b1ee0 con 0x7f4f5c143ae0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c003520 con 0x7f4f5c143ae0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1639523690 0 0) 0x7f4f3c003520 con 0x7f4f5c143ae0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 msgr2=0x7f4f5c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f5c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 >> v2:172.21.15.137:3300/0 conn(0x7f4f5c15e1e0 msgr2=0x7f4f5c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 --2- 172.21.15.125:0/3484509705 >> v2:172.21.15.137:3300/0 conn(0x7f4f5c15e1e0 0x7f4f5c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0b2130 con 0x7f4f5c143ae0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f4f4c001670 con 0x7f4f5c143ae0 2024-01-16T14:57:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 >> v1:172.21.15.125:6789/0 conn(0x7f4f5c143ae0 legacy=0x7f4f5c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c003970 0x7f4f3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f3c0047c0 0x7f4f3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0b2780 con 0x7f4f3c0047c0 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c15e990 con 0x7f4f3c003970 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.731+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c00b980 con 0x7f4f5c0cf9a0 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f3c0047c0 0x7f4f3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f5aa00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c003970 0x7f4f3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f5b400640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 -- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 msgr2=0x7f4f3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c0cf9a0 0x7f4f3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 -- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c003970 msgr2=0x7f4f3c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:38.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c003970 0x7f4f3c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.732+0000 7f4f51e00640 1 -- 172.21.15.125:0/3484509705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c003520 con 0x7f4f3c0047c0 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.733+0000 7f4f51e00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f3c0047c0 0x7f4f3c004c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009ac0 tx=0x7f4f3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f8f2b31c2a6ad8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4402c6b0 con 0x7f4f3c0047c0 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4402c810 con 0x7f4f3c0047c0 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f5a000640 1 -- 172.21.15.125:0/3484509705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4402cba0 con 0x7f4f3c0047c0 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 -- 172.21.15.125:0/3484509705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f3c0047c0 msgr2=0x7f4f3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 --2- 172.21.15.125:0/3484509705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f3c0047c0 0x7f4f3c004c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009ac0 tx=0x7f4f3c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f5b400640 1 -- 172.21.15.125:0/3484509705 reap_dead start 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 -- 172.21.15.125:0/3484509705 shutdown_connections 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 -- 172.21.15.125:0/3484509705 >> 172.21.15.125:0/3484509705 conn(0x7f4f5c06a800 msgr2=0x7f4f5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 -- 172.21.15.125:0/3484509705 shutdown_connections 2024-01-16T14:57:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.734+0000 7f4f61a00640 1 -- 172.21.15.125:0/3484509705 wait complete. 2024-01-16T14:57:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.736+0000 7f4f61a00640 1 Processor -- start 2024-01-16T14:57:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.736+0000 7f4f61a00640 1 -- start start 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c0047c0 0x7f4f5c141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c1f6c00 0x7f4f5c140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1411d0 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1422b0 con 0x7f4f3c0047c0 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.737+0000 7f4f61a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1f6120 con 0x7f4f5c1f6c00 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47372/0 (socket says 172.21.15.125:47372) 2024-01-16T14:57:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 -- 172.21.15.125:0/8945035 learned_addr learned my addr 172.21.15.125:0/8945035 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5aa00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c0047c0 0x7f4f5c141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f51e00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c1f6c00 0x7f4f5c140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 -- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c1f6c00 msgr2=0x7f4f5c140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c1f6c00 0x7f4f5c140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 -- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c0047c0 msgr2=0x7f4f5c141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c0047c0 0x7f4f5c141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 -- 172.21.15.125:0/8945035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c1f6400 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.738+0000 7f4f5b400640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0cc3c0 tx=0x7f4f540115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.739+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400e070 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.739+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f54011e10 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.739+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c1f66e0 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.739+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54003990 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.739+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c1f2880 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.740+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c1f68d0 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.741+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4f54003af0 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.742+0000 7f4f53400640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 0x7f4f3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.742+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4f5408a380 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.742+0000 7f4f5aa00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 0x7f4f3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:38.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.742+0000 7f4f5aa00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 0x7f4f3406d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c001eb0 tx=0x7f4f4c000fd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:38.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.746+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f54055150 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.948+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4f5c0675d0 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.948+0000 7f4f53400640 1 -- 172.21.15.125:0/8945035 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4f54058770 con 0x7f4f5c1f21f0 2024-01-16T14:57:38.930 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:38.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 msgr2=0x7f4f3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 0x7f4f3406d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c001eb0 tx=0x7f4f4c000fd0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 msgr2=0x7f4f5c0b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0cc3c0 tx=0x7f4f540115a0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 shutdown_connections 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4f3406b2c0 0x7f4f3406d780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f5c1f21f0 0x7f4f5c0b0010 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f5c1f6c00 0x7f4f5c140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 --2- 172.21.15.125:0/8945035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f3c0047c0 0x7f4f5c141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 >> 172.21.15.125:0/8945035 conn(0x7f4f5c06a800 msgr2=0x7f4f5c15b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:38.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.951+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 shutdown_connections 2024-01-16T14:57:38.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:38.952+0000 7f4f61a00640 1 -- 172.21.15.125:0/8945035 wait complete. 2024-01-16T14:57:38.950 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T14:57:38.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:39.052 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:39.052 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:57:39.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:39.053 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T14:57:39.116 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:39.117 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:39.171 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:39.171 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:39.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:39.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.481+0000 7f3d49000640 1 Processor -- start 2024-01-16T14:57:39.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.482+0000 7f3d49000640 1 -- start start 2024-01-16T14:57:39.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.483+0000 7f3d49000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.483+0000 7f3d49000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d4406e7c0 0x7f3d4406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.483+0000 7f3d49000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440cfa70 con 0x7f3d4415e880 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.483+0000 7f3d49000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4409a3e0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.483+0000 7f3d49000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440cff00 con 0x7f3d44099a60 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.484+0000 7f3d41400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.484+0000 7f3d41400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:52894/0 (socket says 172.21.15.125:52894) 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.484+0000 7f3d41400640 1 -- 172.21.15.125:0/3074588735 learned_addr learned my addr 172.21.15.125:0/3074588735 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:39.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.484+0000 7f3d42800640 1 --2- 172.21.15.125:0/3074588735 >> v2:172.21.15.137:3300/0 conn(0x7f3d4406e7c0 0x7f3d4406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d40a00640 1 -- 172.21.15.125:0/3074588735 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006626804 0 0) 0x7f3d440cfa70 con 0x7f3d4415e880 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d40a00640 1 -- 172.21.15.125:0/3074588735 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d24003520 con 0x7f3d4415e880 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d41400640 1 -- 172.21.15.125:0/3074588735 >> v2:172.21.15.137:3300/0 conn(0x7f3d4406e7c0 msgr2=0x7f3d4406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d41400640 1 --2- 172.21.15.125:0/3074588735 >> v2:172.21.15.137:3300/0 conn(0x7f3d4406e7c0 0x7f3d4406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d41400640 1 -- 172.21.15.125:0/3074588735 >> v1:172.21.15.137:6789/0 conn(0x7f3d4415e880 legacy=0x7f3d4406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.485+0000 7f3d41400640 1 -- 172.21.15.125:0/3074588735 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4406fc70 con 0x7f3d44099a60 2024-01-16T14:57:39.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.486+0000 7f3d41400640 1 --2- 172.21.15.125:0/3074588735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009870 tx=0x7f3d2c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fb1c95fa371be82 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.486+0000 7f3d40a00640 1 -- 172.21.15.125:0/3074588735 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c02f840 con 0x7f3d44099a60 2024-01-16T14:57:39.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d40a00640 1 -- 172.21.15.125:0/3074588735 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c02f9a0 con 0x7f3d44099a60 2024-01-16T14:57:39.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d40a00640 1 -- 172.21.15.125:0/3074588735 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c02fd10 con 0x7f3d44099a60 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 -- 172.21.15.125:0/3074588735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 msgr2=0x7f3d44099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 --2- 172.21.15.125:0/3074588735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009870 tx=0x7f3d2c02ba20 comp rx=0 tx=0).stop 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 -- 172.21.15.125:0/3074588735 shutdown_connections 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 --2- 172.21.15.125:0/3074588735 >> v2:172.21.15.137:3300/0 conn(0x7f3d4406e7c0 0x7f3d4406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 --2- 172.21.15.125:0/3074588735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44099ea0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 -- 172.21.15.125:0/3074588735 >> 172.21.15.125:0/3074588735 conn(0x7f3d440b2d00 msgr2=0x7f3d440b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 -- 172.21.15.125:0/3074588735 shutdown_connections 2024-01-16T14:57:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.487+0000 7f3d49000640 1 -- 172.21.15.125:0/3074588735 wait complete. 2024-01-16T14:57:39.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.489+0000 7f3d49000640 1 Processor -- start 2024-01-16T14:57:39.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.489+0000 7f3d49000640 1 -- start start 2024-01-16T14:57:39.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d4415e880 0x7f3d441572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d44156460 con 0x7f3d4406e7c0 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d441578f0 con 0x7f3d4415e880 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d49000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440b5080 con 0x7f3d44099a60 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47382/0 (socket says 172.21.15.125:47382) 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.490+0000 7f3d41e00640 1 -- 172.21.15.125:0/1464695649 learned_addr learned my addr 172.21.15.125:0/1464695649 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:39.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d42800640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d4415e880 0x7f3d441572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 -- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 msgr2=0x7f3d44156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41400640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 -- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d4415e880 msgr2=0x7f3d441572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d4415e880 0x7f3d441572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 -- 172.21.15.125:0/1464695649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0094d0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d41e00640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c430 tx=0x7f3d3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d340052d0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34005450 con 0x7f3d4406e7c0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34003830 con 0x7f3d4406e7c0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.491+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d440b5360 con 0x7f3d4406e7c0 2024-01-16T14:57:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.492+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d440b5920 con 0x7f3d4406e7c0 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.493+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d44099ea0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.493+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3d34003990 con 0x7f3d4406e7c0 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.494+0000 7f3d3a000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 0x7f3d1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.494+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3d340892f0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.495+0000 7f3d41400640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 0x7f3d1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.495+0000 7f3d41400640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 0x7f3d1c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c036000 tx=0x7f3d2c039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.499+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d340540d0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.690+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d440675d0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.693+0000 7f3d3a000640 1 -- 172.21.15.125:0/1464695649 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f3d340576f0 con 0x7f3d4406e7c0 2024-01-16T14:57:39.674 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:39.675 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 msgr2=0x7f3d1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 0x7f3d1c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c036000 tx=0x7f3d2c039040 comp rx=0 tx=0).stop 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 msgr2=0x7f3d44155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c430 tx=0x7f3d3400c8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 shutdown_connections 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d4415e880 0x7f3d441572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d44099a60 0x7f3d44156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d1c06b2c0 0x7f3d1c06d780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 --2- 172.21.15.125:0/1464695649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d4406e7c0 0x7f3d44155f20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.695+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 >> 172.21.15.125:0/1464695649 conn(0x7f3d440b2d00 msgr2=0x7f3d4406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.696+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 shutdown_connections 2024-01-16T14:57:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.696+0000 7f3d49000640 1 -- 172.21.15.125:0/1464695649 wait complete. 2024-01-16T14:57:39.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:39.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:39.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.929+0000 7f890dc00640 1 Processor -- start 2024-01-16T14:57:39.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.929+0000 7f890dc00640 1 -- start start 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f890dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f890813cf10 0x7f890813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f890dc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f890813fae0 0x7f89080cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f890dc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080ce440 con 0x7f89080cdc20 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f890dc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890813ff90 con 0x7f890813fae0 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f890dc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89080ce720 con 0x7f890813cf10 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f88ffe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f890813fae0 0x7f89080cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f88ffe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f890813fae0 0x7f89080cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47394/0 (socket says 172.21.15.125:47394) 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f88ffe00640 1 -- 172.21.15.125:0/4083178696 learned_addr learned my addr 172.21.15.125:0/4083178696 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.931+0000 7f8907e00640 1 --2- 172.21.15.125:0/4083178696 >> v2:172.21.15.137:3300/0 conn(0x7f890813cf10 0x7f890813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.932+0000 7f88ffe00640 1 -- 172.21.15.125:0/4083178696 >> v2:172.21.15.137:3300/0 conn(0x7f890813cf10 msgr2=0x7f890813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.932+0000 7f88ffe00640 1 --2- 172.21.15.125:0/4083178696 >> v2:172.21.15.137:3300/0 conn(0x7f890813cf10 0x7f890813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.932+0000 7f88ffe00640 1 -- 172.21.15.125:0/4083178696 >> v1:172.21.15.125:6789/0 conn(0x7f89080cdc20 legacy=0x7f890815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.932+0000 7f88ffe00640 1 -- 172.21.15.125:0/4083178696 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080cea00 con 0x7f890813fae0 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.933+0000 7f88ffe00640 1 --2- 172.21.15.125:0/4083178696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813fae0 0x7f89080cd6e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f890806e390 tx=0x7f88f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f682e498bcd42e39 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.933+0000 7f8907400640 1 -- 172.21.15.125:0/4083178696 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0030860 con 0x7f890813fae0 2024-01-16T14:57:39.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.933+0000 7f8907400640 1 -- 172.21.15.125:0/4083178696 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f00309c0 con 0x7f890813fae0 2024-01-16T14:57:39.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.933+0000 7f8907400640 1 -- 172.21.15.125:0/4083178696 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0030d50 con 0x7f890813fae0 2024-01-16T14:57:39.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 -- 172.21.15.125:0/4083178696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813fae0 msgr2=0x7f89080cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 --2- 172.21.15.125:0/4083178696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813fae0 0x7f89080cd6e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f890806e390 tx=0x7f88f002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 -- 172.21.15.125:0/4083178696 shutdown_connections 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 --2- 172.21.15.125:0/4083178696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813fae0 0x7f89080cd6e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 --2- 172.21.15.125:0/4083178696 >> v2:172.21.15.137:3300/0 conn(0x7f890813cf10 0x7f890813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 -- 172.21.15.125:0/4083178696 >> 172.21.15.125:0/4083178696 conn(0x7f890806a7d0 msgr2=0x7f890806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 -- 172.21.15.125:0/4083178696 shutdown_connections 2024-01-16T14:57:39.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.934+0000 7f890dc00640 1 -- 172.21.15.125:0/4083178696 wait complete. 2024-01-16T14:57:39.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.936+0000 7f890dc00640 1 Processor -- start 2024-01-16T14:57:39.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.936+0000 7f890dc00640 1 -- start start 2024-01-16T14:57:39.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89080cdc20 0x7f890815c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f890813fae0 0x7f8908071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890815cc40 con 0x7f890813cf10 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89081585c0 con 0x7f89080cdc20 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f890dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8908158860 con 0x7f890813fae0 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f88ffe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f88ffe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47410/0 (socket says 172.21.15.125:47410) 2024-01-16T14:57:39.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.937+0000 7f88ffe00640 1 -- 172.21.15.125:0/3564318177 learned_addr learned my addr 172.21.15.125:0/3564318177 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 -- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f890813fae0 msgr2=0x7f8908071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f890813fae0 0x7f8908071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 -- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89080cdc20 msgr2=0x7f890815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89080cdc20 0x7f890815c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 -- 172.21.15.125:0/3564318177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00094d0 con 0x7f890813cf10 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f88ffe00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88f002b9e0 tx=0x7f88f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f003e070 con 0x7f890813cf10 2024-01-16T14:57:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0039a60 con 0x7f890813cf10 2024-01-16T14:57:39.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0039df0 con 0x7f890813cf10 2024-01-16T14:57:39.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908158b40 con 0x7f890813cf10 2024-01-16T14:57:39.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.938+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890815c930 con 0x7f890813cf10 2024-01-16T14:57:39.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.940+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908159060 con 0x7f890813cf10 2024-01-16T14:57:39.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.941+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f88f004b400 con 0x7f890813cf10 2024-01-16T14:57:39.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.941+0000 7f8904c00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 0x7f88e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.941+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f88f00ac3f0 con 0x7f890813cf10 2024-01-16T14:57:39.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.942+0000 7f8907e00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 0x7f88e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.942+0000 7f8907e00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 0x7f88e006d780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f88f80098f0 tx=0x7f88f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:39.946+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8908159060 con 0x7f890813cf10 2024-01-16T14:57:40.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.142+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8908159060 con 0x7f890813cf10 2024-01-16T14:57:40.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.143+0000 7f8904c00640 1 -- 172.21.15.125:0/3564318177 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8908159060 con 0x7f890813cf10 2024-01-16T14:57:40.124 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 msgr2=0x7f88e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 0x7f88e006d780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f88f80098f0 tx=0x7f88f8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 msgr2=0x7f890813d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88f002b9e0 tx=0x7f88f00046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 shutdown_connections 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f890813fae0 0x7f8908071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f88e006b2c0 0x7f88e006d780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890813cf10 0x7f890813d2f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 --2- 172.21.15.125:0/3564318177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89080cdc20 0x7f890815c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 >> 172.21.15.125:0/3564318177 conn(0x7f890806a7d0 msgr2=0x7f89080b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 shutdown_connections 2024-01-16T14:57:40.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.146+0000 7f890dc00640 1 -- 172.21.15.125:0/3564318177 wait complete. 2024-01-16T14:57:40.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:40.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.377+0000 7f9de6200640 1 Processor -- start 2024-01-16T14:57:40.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.377+0000 7f9de6200640 1 -- start start 2024-01-16T14:57:40.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9de6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 0x7f9de0164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9de6200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de00b1ee0 con 0x7f9de00cf9a0 2024-01-16T14:57:40.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9de6200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de00b2780 con 0x7f9de0143ae0 2024-01-16T14:57:40.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9de6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de015e990 con 0x7f9de015e3b0 2024-01-16T14:57:40.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9ddea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9de00cf9a0 0x7f9de015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46084/0 (socket says 172.21.15.125:46084) 2024-01-16T14:57:40.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.379+0000 7f9ddea00640 1 -- 172.21.15.125:0/3013094111 learned_addr learned my addr 172.21.15.125:0/3013094111 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:40.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9ddfe00640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 0x7f9de0164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508246061 0 0) 0x7f9de00b1ee0 con 0x7f9de00cf9a0 2024-01-16T14:57:40.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0003520 con 0x7f9de00cf9a0 2024-01-16T14:57:40.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539721695 0 0) 0x7f9dc0003520 con 0x7f9de00cf9a0 2024-01-16T14:57:40.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 msgr2=0x7f9de0164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 0x7f9de0164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 >> v1:172.21.15.137:6789/0 conn(0x7f9de0143ae0 legacy=0x7f9de00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00b2130 con 0x7f9de00cf9a0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f9dc8001670 con 0x7f9de00cf9a0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 >> v1:172.21.15.125:6789/0 conn(0x7f9de00cf9a0 legacy=0x7f9de015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00039a0 0x7f9dc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dc00047c0 0x7f9dc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 0x7f9dc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de015e990 con 0x7f9dc00047c0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000ba60 con 0x7f9dc00039a0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000bc20 con 0x7f9de015e3b0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.380+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dc00047c0 0x7f9dc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddea00640 1 -- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 msgr2=0x7f9dc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de015e3b0 0x7f9dc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddea00640 1 -- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00039a0 msgr2=0x7f9dc0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddf400640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00039a0 0x7f9dc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00039a0 0x7f9dc0003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.381+0000 7f9ddea00640 1 -- 172.21.15.125:0/3013094111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0003520 con 0x7f9dc00047c0 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.382+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dc00047c0 0x7f9dc0004c40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8007fd0 tx=0x7f9dc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b841fccb077868c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.382+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8005ad0 con 0x7f9dc00047c0 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.382+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc8005c30 con 0x7f9dc00047c0 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.382+0000 7f9dde000640 1 -- 172.21.15.125:0/3013094111 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc802c850 con 0x7f9dc00047c0 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 -- 172.21.15.125:0/3013094111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dc00047c0 msgr2=0x7f9dc0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 --2- 172.21.15.125:0/3013094111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dc00047c0 0x7f9dc0004c40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8007fd0 tx=0x7f9dc00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9ddfe00640 1 -- 172.21.15.125:0/3013094111 reap_dead start 2024-01-16T14:57:40.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 -- 172.21.15.125:0/3013094111 shutdown_connections 2024-01-16T14:57:40.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 -- 172.21.15.125:0/3013094111 >> 172.21.15.125:0/3013094111 conn(0x7f9de006a800 msgr2=0x7f9de006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 -- 172.21.15.125:0/3013094111 shutdown_connections 2024-01-16T14:57:40.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.383+0000 7f9de6200640 1 -- 172.21.15.125:0/3013094111 wait complete. 2024-01-16T14:57:40.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.385+0000 7f9de6200640 1 Processor -- start 2024-01-16T14:57:40.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.385+0000 7f9de6200640 1 -- start start 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 0x7f9de013c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de013e1f0 0x7f9de013e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de013ebc0 con 0x7f9de00b13c0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de01f43d0 con 0x7f9dc00047c0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9de6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00b1860 con 0x7f9de013e1f0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47416/0 (socket says 172.21.15.125:47416) 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddea00640 1 -- 172.21.15.125:0/3974939191 learned_addr learned my addr 172.21.15.125:0/3974939191 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddf400640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 0x7f9de013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddfe00640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de013e1f0 0x7f9de013e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.386+0000 7f9ddea00640 1 -- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de013e1f0 msgr2=0x7f9de013e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de013e1f0 0x7f9de013e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddea00640 1 -- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 msgr2=0x7f9de013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 0x7f9de013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddea00640 1 -- 172.21.15.125:0/3974939191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de01f45e0 con 0x7f9de00b13c0 2024-01-16T14:57:40.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddf400640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 0x7f9de013c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9ddea00640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802c3f0 tx=0x7f9dc802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc803c040 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc802cd20 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8007290 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de01f48c0 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.387+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de0159a20 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.388+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0159c10 con 0x7f9de00b13c0 2024-01-16T14:57:40.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.389+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9dc8032070 con 0x7f9de00b13c0 2024-01-16T14:57:40.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.391+0000 7f9dd7400640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 0x7f9db806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.391+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9dc80ab150 con 0x7f9de00b13c0 2024-01-16T14:57:40.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.391+0000 7f9ddf400640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 0x7f9db806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.392+0000 7f9ddf400640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 0x7f9db806d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00015f0 tx=0x7f9dd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.395+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dc8075f30 con 0x7f9de00b13c0 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.584+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9de006e0b0 con 0x7f9de00b13c0 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.585+0000 7f9dd7400640 1 -- 172.21.15.125:0/3974939191 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9dc8079550 con 0x7f9de00b13c0 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 msgr2=0x7f9db806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 0x7f9db806d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00015f0 tx=0x7f9dd0009040 comp rx=0 tx=0).stop 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 msgr2=0x7f9de00aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802c3f0 tx=0x7f9dc802e070 comp rx=0 tx=0).stop 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 shutdown_connections 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9de013e1f0 0x7f9de013e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db806b2f0 0x7f9db806d7b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9de00b13c0 0x7f9de00aec10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 --2- 172.21.15.125:0/3974939191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dc00047c0 0x7f9de013c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.587+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 >> 172.21.15.125:0/3974939191 conn(0x7f9de006a800 msgr2=0x7f9de00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.588+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 shutdown_connections 2024-01-16T14:57:40.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.588+0000 7f9de6200640 1 -- 172.21.15.125:0/3974939191 wait complete. 2024-01-16T14:57:40.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:40.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.809+0000 7f90c7600640 1 Processor -- start 2024-01-16T14:57:40.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.809+0000 7f90c7600640 1 -- start start 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c7600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 0x7f90c00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c7600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90c015f9c0 0x7f90c00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c7600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c015c5f0 con 0x7f90c013df80 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c7600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c006c470 con 0x7f90c015f9c0 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c7600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c006c750 con 0x7f90c013e360 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90c015f9c0 0x7f90c00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90c015f9c0 0x7f90c00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47430/0 (socket says 172.21.15.125:47430) 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 -- 172.21.15.125:0/3899389687 learned_addr learned my addr 172.21.15.125:0/3899389687 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90bfe00640 1 --2- 172.21.15.125:0/3899389687 >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 0x7f90c00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 -- 172.21.15.125:0/3899389687 >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 msgr2=0x7f90c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 --2- 172.21.15.125:0/3899389687 >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 0x7f90c00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.811+0000 7f90c5600640 1 -- 172.21.15.125:0/3899389687 >> v1:172.21.15.137:6789/0 conn(0x7f90c013df80 legacy=0x7f90c015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.812+0000 7f90c5600640 1 -- 172.21.15.125:0/3899389687 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c006ca30 con 0x7f90c015f9c0 2024-01-16T14:57:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.812+0000 7f90bfe00640 1 --2- 172.21.15.125:0/3899389687 >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 0x7f90c00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.813+0000 7f90c5600640 1 --2- 172.21.15.125:0/3899389687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c015f9c0 0x7f90c00cf7d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90c00ccae0 tx=0x7f90b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be05b09dd708b7c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.813+0000 7f90bf400640 1 -- 172.21.15.125:0/3899389687 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b802ca50 con 0x7f90c015f9c0 2024-01-16T14:57:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.813+0000 7f90bf400640 1 -- 172.21.15.125:0/3899389687 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b802cbb0 con 0x7f90c015f9c0 2024-01-16T14:57:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.813+0000 7f90bf400640 1 -- 172.21.15.125:0/3899389687 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b80343f0 con 0x7f90c015f9c0 2024-01-16T14:57:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 -- 172.21.15.125:0/3899389687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c015f9c0 msgr2=0x7f90c00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 --2- 172.21.15.125:0/3899389687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c015f9c0 0x7f90c00cf7d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90c00ccae0 tx=0x7f90b802beb0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 -- 172.21.15.125:0/3899389687 shutdown_connections 2024-01-16T14:57:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 --2- 172.21.15.125:0/3899389687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c015f9c0 0x7f90c00cf7d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 --2- 172.21.15.125:0/3899389687 >> v2:172.21.15.137:3300/0 conn(0x7f90c013e360 0x7f90c00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 -- 172.21.15.125:0/3899389687 >> 172.21.15.125:0/3899389687 conn(0x7f90c006a800 msgr2=0x7f90c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 -- 172.21.15.125:0/3899389687 shutdown_connections 2024-01-16T14:57:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.814+0000 7f90c7600640 1 -- 172.21.15.125:0/3899389687 wait complete. 2024-01-16T14:57:40.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.816+0000 7f90c7600640 1 Processor -- start 2024-01-16T14:57:40.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.816+0000 7f90c7600640 1 -- start start 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90c013e360 0x7f90c00a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90c015f9c0 0x7f90c00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00a8f10 con 0x7f90c013df80 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00a91f0 con 0x7f90c015f9c0 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c7600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00a94d0 con 0x7f90c013e360 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.817+0000 7f90c4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47438/0 (socket says 172.21.15.125:47438) 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 -- 172.21.15.125:0/592820635 learned_addr learned my addr 172.21.15.125:0/592820635 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 -- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90c013e360 msgr2=0x7f90c00a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90c013e360 0x7f90c00a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 -- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90c015f9c0 msgr2=0x7f90c00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90c015f9c0 0x7f90c00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 -- 172.21.15.125:0/592820635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8009b30 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90c4c00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b3f0 tx=0x7f90b000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0011830 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b0011990 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.818+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0011d40 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.819+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c00a97b0 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.819+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c00adf50 con 0x7f90c013df80 2024-01-16T14:57:40.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.820+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c00a9cd0 con 0x7f90c013df80 2024-01-16T14:57:40.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.821+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f90b001a400 con 0x7f90c013df80 2024-01-16T14:57:40.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.821+0000 7f90bcc00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 0x7f909806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.822+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f90b0089b00 con 0x7f90c013df80 2024-01-16T14:57:40.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.822+0000 7f90bfe00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 0x7f909806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.822+0000 7f90bfe00640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 0x7f909806d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90a8006fd0 tx=0x7f90a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:40.826+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90c00a9cd0 con 0x7f90c013df80 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.019+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f90c00a9cd0 con 0x7f90c013df80 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.020+0000 7f90bcc00640 1 -- 172.21.15.125:0/592820635 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f90c00a9cd0 con 0x7f90c013df80 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 msgr2=0x7f909806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 0x7f909806d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90a8006fd0 tx=0x7f90a8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 msgr2=0x7f90c013d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b3f0 tx=0x7f90b000b8b0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 shutdown_connections 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90c015f9c0 0x7f90c00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90c013e360 0x7f90c00a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f909806b2c0 0x7f909806d780 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 --2- 172.21.15.125:0/592820635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90c013df80 0x7f90c013d6f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 >> 172.21.15.125:0/592820635 conn(0x7f90c006a800 msgr2=0x7f90c015a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 shutdown_connections 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.023+0000 7f90c7600640 1 -- 172.21.15.125:0/592820635 wait complete. 2024-01-16T14:57:41.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.246+0000 7f32b9600640 1 Processor -- start 2024-01-16T14:57:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.246+0000 7f32b9600640 1 -- start start 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f32b415afb0 0x7f32b413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b9600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b413ff30 con 0x7f32b413cfa0 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40cdc40 con 0x7f32b413fae0 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b40cdec0 con 0x7f32b415afb0 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f32b413cfa0 0x7f32b40cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46094/0 (socket says 172.21.15.125:46094) 2024-01-16T14:57:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b3e00640 1 -- 172.21.15.125:0/3330647870 learned_addr learned my addr 172.21.15.125:0/3330647870 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:41.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036117773 0 0) 0x7f32a4001490 con 0x7f32b413cfa0 2024-01-16T14:57:41.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.248+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294005730 con 0x7f32b413cfa0 2024-01-16T14:57:41.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32ab200640 1 --2- 172.21.15.125:0/3330647870 >> v2:172.21.15.137:3300/0 conn(0x7f32b415afb0 0x7f32b413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866026610 0 0) 0x7f3294005730 con 0x7f32b413cfa0 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 >> v2:172.21.15.137:3300/0 conn(0x7f32b415afb0 msgr2=0x7f32b413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 --2- 172.21.15.125:0/3330647870 >> v2:172.21.15.137:3300/0 conn(0x7f32b415afb0 0x7f32b413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 >> v1:172.21.15.137:6789/0 conn(0x7f32b413fae0 legacy=0x7f32b40b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b40ce1a0 con 0x7f32b413cfa0 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f32a4001b80 con 0x7f32b413cfa0 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 >> v1:172.21.15.125:6789/0 conn(0x7f32b413cfa0 legacy=0x7f32b40cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3294005b80 0x7f3294005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32940069d0 0x7f3294006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f329400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b40cdec0 con 0x7f32940074b0 2024-01-16T14:57:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329400e360 con 0x7f3294005b80 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329400e520 con 0x7f32940069d0 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b8c00640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f329400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.249+0000 7f32b3e00640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3294005b80 0x7f3294005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32ab200640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32940069d0 0x7f3294006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32b8c00640 1 -- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32940069d0 msgr2=0x7f3294006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32b8c00640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32940069d0 0x7f3294006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32b8c00640 1 -- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3294005b80 msgr2=0x7f3294005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32b8c00640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3294005b80 0x7f3294005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.250+0000 7f32b8c00640 1 -- 172.21.15.125:0/3330647870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3294005730 con 0x7f32940074b0 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.251+0000 7f32b8c00640 1 -- 172.21.15.125:0/3330647870 reap_dead start 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.251+0000 7f32b8c00640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f329400db90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32ac003d80 tx=0x7f3294007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b36a4d0f4244da2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.251+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac007220 con 0x7f32940074b0 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.251+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac02e050 con 0x7f32940074b0 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.251+0000 7f32b3400640 1 -- 172.21.15.125:0/3330647870 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0075e0 con 0x7f32940074b0 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 -- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 msgr2=0x7f329400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f329400db90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32ac003d80 tx=0x7f3294007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 -- 172.21.15.125:0/3330647870 shutdown_connections 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 --2- 172.21.15.125:0/3330647870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f329400db90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 -- 172.21.15.125:0/3330647870 >> 172.21.15.125:0/3330647870 conn(0x7f32b406a800 msgr2=0x7f32b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 -- 172.21.15.125:0/3330647870 shutdown_connections 2024-01-16T14:57:41.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.252+0000 7f32b9600640 1 -- 172.21.15.125:0/3330647870 wait complete. 2024-01-16T14:57:41.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.254+0000 7f32b9600640 1 Processor -- start 2024-01-16T14:57:41.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.254+0000 7f32b9600640 1 -- start start 2024-01-16T14:57:41.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32b4070ab0 0x7f32b415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32b4070f10 0x7f32b415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b415c110 con 0x7f32940074b0 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b415c3f0 con 0x7f32b4070f10 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b415c6d0 con 0x7f32b4070ab0 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47446/0 (socket says 172.21.15.125:47446) 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 -- 172.21.15.125:0/1278764911 learned_addr learned my addr 172.21.15.125:0/1278764911 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 -- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32b4070ab0 msgr2=0x7f32b415b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32b4070ab0 0x7f32b415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 -- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32b4070f10 msgr2=0x7f32b415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32b4070f10 0x7f32b415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.255+0000 7f32b3e00640 1 -- 172.21.15.125:0/1278764911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b415c9b0 con 0x7f32940074b0 2024-01-16T14:57:41.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b3e00640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32a4007a10 tx=0x7f32a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a400ee50 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a4013420 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b415cc90 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a40137d0 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.256+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b415d6f0 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.257+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b415ced0 con 0x7f32940074b0 2024-01-16T14:57:41.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.258+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f32a4013930 con 0x7f32940074b0 2024-01-16T14:57:41.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.259+0000 7f32b0c00640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 0x7f328c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.260+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f32a408a3a0 con 0x7f32940074b0 2024-01-16T14:57:41.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.260+0000 7f32ab200640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 0x7f328c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.260+0000 7f32ab200640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 0x7f328c06d7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f32b413bdc0 tx=0x7f329c00d280 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.264+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32b415ced0 con 0x7f32940074b0 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.467+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32b415ced0 con 0x7f32940074b0 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.468+0000 7f32b0c00640 1 -- 172.21.15.125:0/1278764911 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f32b415ced0 con 0x7f32940074b0 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:41.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 msgr2=0x7f328c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 0x7f328c06d7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f32b413bdc0 tx=0x7f329c00d280 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 msgr2=0x7f32b415d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32a4007a10 tx=0x7f32a400e530 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 shutdown_connections 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32b4070f10 0x7f32b415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32b4070ab0 0x7f32b415b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f328c06b2f0 0x7f328c06d7b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 --2- 172.21.15.125:0/1278764911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32940074b0 0x7f32b415d1b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 >> 172.21.15.125:0/1278764911 conn(0x7f32b406a800 msgr2=0x7f32b40b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 shutdown_connections 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.471+0000 7f32b9600640 1 -- 172.21.15.125:0/1278764911 wait complete. 2024-01-16T14:57:41.472 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:57:41.472 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:41.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:41.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.796+0000 7fe155200640 1 Processor -- start 2024-01-16T14:57:41.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.796+0000 7fe155200640 1 -- start start 2024-01-16T14:57:41.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.797+0000 7fe155200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe150143ae0 0x7fe1500cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.797+0000 7fe155200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500cfc30 con 0x7fe15015e3b0 2024-01-16T14:57:41.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.797+0000 7fe155200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b1ee0 con 0x7fe1500cf7d0 2024-01-16T14:57:41.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.797+0000 7fe155200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500b2780 con 0x7fe150143ae0 2024-01-16T14:57:41.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe15015e3b0 0x7fe1500702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46104/0 (socket says 172.21.15.125:46104) 2024-01-16T14:57:41.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14e800640 1 -- 172.21.15.125:0/2187114062 learned_addr learned my addr 172.21.15.125:0/2187114062 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230301053 0 0) 0x7fe1500cfc30 con 0x7fe15015e3b0 2024-01-16T14:57:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe130005730 con 0x7fe15015e3b0 2024-01-16T14:57:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14de00640 1 --2- 172.21.15.125:0/2187114062 >> v2:172.21.15.137:3300/0 conn(0x7fe150143ae0 0x7fe1500cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040356330 0 0) 0x7fe130005730 con 0x7fe15015e3b0 2024-01-16T14:57:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 >> v2:172.21.15.137:3300/0 conn(0x7fe150143ae0 msgr2=0x7fe1500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 --2- 172.21.15.125:0/2187114062 >> v2:172.21.15.137:3300/0 conn(0x7fe150143ae0 0x7fe1500cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 >> v1:172.21.15.137:6789/0 conn(0x7fe1500cf7d0 legacy=0x7fe15015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15015e990 con 0x7fe15015e3b0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.798+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe1480019f0 con 0x7fe15015e3b0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 >> v1:172.21.15.125:6789/0 conn(0x7fe15015e3b0 legacy=0x7fe1500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe130005bd0 0x7fe130005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 0x7fe130006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1300073e0 0x7fe13000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500b2780 con 0x7fe1300069d0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13000e330 con 0x7fe130005bd0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13000e4f0 con 0x7fe1300073e0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe147e00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 0x7fe130006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14de00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe130005bd0 0x7fe130005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe14e800640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1300073e0 0x7fe13000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe147e00640 1 -- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1300073e0 msgr2=0x7fe13000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe147e00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1300073e0 0x7fe13000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe147e00640 1 -- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe130005bd0 msgr2=0x7fe130005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.799+0000 7fe147e00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe130005bd0 0x7fe130005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.800+0000 7fe147e00640 1 -- 172.21.15.125:0/2187114062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe130005730 con 0x7fe1300069d0 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.800+0000 7fe14e800640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1300073e0 0x7fe13000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.800+0000 7fe14de00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe130005bd0 0x7fe130005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.800+0000 7fe14e800640 1 -- 172.21.15.125:0/2187114062 reap_dead start 2024-01-16T14:57:41.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe147e00640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 0x7fe130006e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe138001750 tx=0x7fe130007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc9c7b7d3a356e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1380052d0 con 0x7fe1300069d0 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13802e050 con 0x7fe1300069d0 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe14d400640 1 -- 172.21.15.125:0/2187114062 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138003830 con 0x7fe1300069d0 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe155200640 1 -- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 msgr2=0x7fe130006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe155200640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 0x7fe130006e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe138001750 tx=0x7fe130007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe155200640 1 -- 172.21.15.125:0/2187114062 shutdown_connections 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe155200640 1 --2- 172.21.15.125:0/2187114062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1300069d0 0x7fe130006e50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.801+0000 7fe155200640 1 -- 172.21.15.125:0/2187114062 >> 172.21.15.125:0/2187114062 conn(0x7fe15006a800 msgr2=0x7fe15006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.802+0000 7fe155200640 1 -- 172.21.15.125:0/2187114062 shutdown_connections 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.802+0000 7fe155200640 1 -- 172.21.15.125:0/2187114062 wait complete. 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.803+0000 7fe155200640 1 Processor -- start 2024-01-16T14:57:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.803+0000 7fe155200640 1 -- start start 2024-01-16T14:57:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1300069d0 0x7fe1501398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe150159620 0x7fe150158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500b1810 con 0x7fe1500b0e20 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15013a1b0 con 0x7fe1300069d0 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.804+0000 7fe155200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15013a470 con 0x7fe150159620 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47474/0 (socket says 172.21.15.125:47474) 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 -- 172.21.15.125:0/4205689043 learned_addr learned my addr 172.21.15.125:0/4205689043 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14de00640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1300069d0 0x7fe1501398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe147e00640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe150159620 0x7fe150158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 -- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe150159620 msgr2=0x7fe150158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe150159620 0x7fe150158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 -- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1300069d0 msgr2=0x7fe1501398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1300069d0 0x7fe1501398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.805+0000 7fe14e800640 1 -- 172.21.15.125:0/4205689043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15013a750 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe14e800640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe148001580 tx=0x7fe14800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148004d60 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14800bb00 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14800beb0 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15013aa30 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.806+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe150139e00 con 0x7fe1500b0e20 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.807+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15015b000 con 0x7fe1500b0e20 2024-01-16T14:57:41.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.808+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe148007070 con 0x7fe1500b0e20 2024-01-16T14:57:41.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.809+0000 7fe146000640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 0x7fe12806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.809+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe148089e70 con 0x7fe1500b0e20 2024-01-16T14:57:41.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.810+0000 7fe14de00640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 0x7fe12806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.810+0000 7fe14de00640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 0x7fe12806d7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe140006fd0 tx=0x7fe140008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:41.814+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe15015b000 con 0x7fe1500b0e20 2024-01-16T14:57:41.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.008+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe15015b000 con 0x7fe1500b0e20 2024-01-16T14:57:41.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.009+0000 7fe146000640 1 -- 172.21.15.125:0/4205689043 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7fe15015b000 con 0x7fe1500b0e20 2024-01-16T14:57:41.991 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:41.991 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 msgr2=0x7fe12806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 0x7fe12806d7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe140006fd0 tx=0x7fe140008040 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 msgr2=0x7fe1500b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe148001580 tx=0x7fe14800b4a0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 shutdown_connections 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe12806b2f0 0x7fe12806d7b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1500b0e20 0x7fe1500b12d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe150159620 0x7fe150158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 --2- 172.21.15.125:0/4205689043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1300069d0 0x7fe1501398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 >> 172.21.15.125:0/4205689043 conn(0x7fe15006a800 msgr2=0x7fe1500af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.012+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 shutdown_connections 2024-01-16T14:57:41.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.013+0000 7fe155200640 1 -- 172.21.15.125:0/4205689043 wait complete. 2024-01-16T14:57:42.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:42.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:42.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.237+0000 7f6d74600640 1 Processor -- start 2024-01-16T14:57:42.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.238+0000 7f6d74600640 1 -- start start 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.239+0000 7f6d74600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d6c1540b0 0x7f6d6c159f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.239+0000 7f6d74600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c154490 con 0x7f6d6c070af0 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.239+0000 7f6d74600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c15d550 con 0x7f6d6c06b690 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.239+0000 7f6d74600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c15e440 con 0x7f6d6c1540b0 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.239+0000 7f6d71c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d6c1540b0 0x7f6d6c159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d72600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6d6c070af0 0x7f6d6c0cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46108/0 (socket says 172.21.15.125:46108) 2024-01-16T14:57:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d72600640 1 -- 172.21.15.125:0/2470273598 learned_addr learned my addr 172.21.15.125:0/2470273598 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:42.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d71c00640 1 -- 172.21.15.125:0/2470273598 >> v1:172.21.15.137:6789/0 conn(0x7f6d6c06b690 legacy=0x7f6d6c15dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d71c00640 1 -- 172.21.15.125:0/2470273598 >> v1:172.21.15.125:6789/0 conn(0x7f6d6c070af0 legacy=0x7f6d6c0cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d71c00640 1 -- 172.21.15.125:0/2470273598 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c15a7b0 con 0x7f6d6c1540b0 2024-01-16T14:57:42.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.240+0000 7f6d6be00640 1 -- 172.21.15.125:0/2470273598 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773563495 0 0) 0x7f6d6c154490 con 0x7f6d6c070af0 2024-01-16T14:57:42.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.241+0000 7f6d71c00640 1 --2- 172.21.15.125:0/2470273598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c1540b0 0x7f6d6c159f60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0097a0 tx=0x7f6d5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcdb756c648a4c5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.241+0000 7f6d6be00640 1 -- 172.21.15.125:0/2470273598 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c031830 con 0x7f6d6c1540b0 2024-01-16T14:57:42.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.241+0000 7f6d6be00640 1 -- 172.21.15.125:0/2470273598 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c031990 con 0x7f6d6c1540b0 2024-01-16T14:57:42.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.241+0000 7f6d6be00640 1 -- 172.21.15.125:0/2470273598 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c031d20 con 0x7f6d6c1540b0 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 -- 172.21.15.125:0/2470273598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c1540b0 msgr2=0x7f6d6c159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 --2- 172.21.15.125:0/2470273598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c1540b0 0x7f6d6c159f60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0097a0 tx=0x7f6d5c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 -- 172.21.15.125:0/2470273598 shutdown_connections 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 --2- 172.21.15.125:0/2470273598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c1540b0 0x7f6d6c159f60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 -- 172.21.15.125:0/2470273598 >> 172.21.15.125:0/2470273598 conn(0x7f6d6c0cf700 msgr2=0x7f6d6c0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 -- 172.21.15.125:0/2470273598 shutdown_connections 2024-01-16T14:57:42.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.242+0000 7f6d74600640 1 -- 172.21.15.125:0/2470273598 wait complete. 2024-01-16T14:57:42.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.244+0000 7f6d74600640 1 Processor -- start 2024-01-16T14:57:42.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.244+0000 7f6d74600640 1 -- start start 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d6c070af0 0x7f6d6c14fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d6c1540b0 0x7f6d6c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c15cf30 con 0x7f6d6c06b690 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c13a610 con 0x7f6d6c070af0 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d74600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c15c2b0 con 0x7f6d6c1540b0 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d71c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d71c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47494/0 (socket says 172.21.15.125:47494) 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d71c00640 1 -- 172.21.15.125:0/1845569192 learned_addr learned my addr 172.21.15.125:0/1845569192 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.245+0000 7f6d71200640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d6c070af0 0x7f6d6c14fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d72600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d6c1540b0 0x7f6d6c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 -- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d6c1540b0 msgr2=0x7f6d6c1502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d6c1540b0 0x7f6d6c1502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 -- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d6c070af0 msgr2=0x7f6d6c14fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d6c070af0 0x7f6d6c14fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 -- 172.21.15.125:0/1845569192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c009400 con 0x7f6d6c06b690 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d71c00640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0098c0 tx=0x7f6d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c03ba70 con 0x7f6d6c06b690 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c03bbd0 con 0x7f6d6c06b690 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0453f0 con 0x7f6d6c06b690 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.246+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c15c590 con 0x7f6d6c06b690 2024-01-16T14:57:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.247+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c15cb50 con 0x7f6d6c06b690 2024-01-16T14:57:42.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.248+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c0675d0 con 0x7f6d6c06b690 2024-01-16T14:57:42.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.248+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6d5c045550 con 0x7f6d6c06b690 2024-01-16T14:57:42.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.249+0000 7f6d69600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 0x7f6d4406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.249+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6d5c0ac2a0 con 0x7f6d6c06b690 2024-01-16T14:57:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.250+0000 7f6d71200640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 0x7f6d4406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.250+0000 7f6d71200640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 0x7f6d4406d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d54006fd0 tx=0x7f6d54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.254+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d5c077070 con 0x7f6d6c06b690 2024-01-16T14:57:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.443+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6d6c0ce0b0 con 0x7f6d6c06b690 2024-01-16T14:57:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.444+0000 7f6d69600640 1 -- 172.21.15.125:0/1845569192 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6d5c07a690 con 0x7f6d6c06b690 2024-01-16T14:57:42.425 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 msgr2=0x7f6d4406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 0x7f6d4406d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d54006fd0 tx=0x7f6d54008040 comp rx=0 tx=0).stop 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 msgr2=0x7f6d6c13a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0098c0 tx=0x7f6d5c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 shutdown_connections 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d6c1540b0 0x7f6d6c1502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d6c070af0 0x7f6d6c14fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d4406b2c0 0x7f6d4406d780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 --2- 172.21.15.125:0/1845569192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d6c06b690 0x7f6d6c13a0d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.446+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 >> 172.21.15.125:0/1845569192 conn(0x7f6d6c0cf700 msgr2=0x7f6d6c0ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.447+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 shutdown_connections 2024-01-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.447+0000 7f6d74600640 1 -- 172.21.15.125:0/1845569192 wait complete. 2024-01-16T14:57:42.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:42.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.682+0000 7fc461400640 1 Processor -- start 2024-01-16T14:57:42.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.682+0000 7fc461400640 1 -- start start 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc461400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c0cf9a0 0x7fc45c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc461400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc45c15e1e0 0x7fc45c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc461400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b1ee0 con 0x7fc45c143ae0 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc461400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c0b2780 con 0x7fc45c15e1e0 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc461400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15e990 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc45a800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc45c143ae0 0x7fc45c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46116/0 (socket says 172.21.15.125:46116) 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc45a800640 1 -- 172.21.15.125:0/534622276 learned_addr learned my addr 172.21.15.125:0/534622276 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc45b200640 1 --2- 172.21.15.125:0/534622276 >> v2:172.21.15.125:3300/0 conn(0x7fc45c15e1e0 0x7fc45c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.684+0000 7fc459e00640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c0cf9a0 0x7fc45c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.685+0000 7fc45b200640 1 -- 172.21.15.125:0/534622276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c0cf9a0 msgr2=0x7fc45c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.685+0000 7fc45b200640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c0cf9a0 0x7fc45c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.685+0000 7fc45b200640 1 -- 172.21.15.125:0/534622276 >> v1:172.21.15.125:6789/0 conn(0x7fc45c143ae0 legacy=0x7fc45c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.685+0000 7fc459400640 1 -- 172.21.15.125:0/534622276 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231303432 0 0) 0x7fc45c0b1ee0 con 0x7fc45c143ae0 2024-01-16T14:57:42.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.685+0000 7fc45b200640 1 -- 172.21.15.125:0/534622276 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0b2130 con 0x7fc45c15e1e0 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc45b200640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0702c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc454009d90 tx=0x7fc45402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57594be871a94ae7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc459400640 1 -- 172.21.15.125:0/534622276 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45402cc50 con 0x7fc45c15e1e0 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc459400640 1 -- 172.21.15.125:0/534622276 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45402cdb0 con 0x7fc45c15e1e0 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc459400640 1 -- 172.21.15.125:0/534622276 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454034660 con 0x7fc45c15e1e0 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc461400640 1 -- 172.21.15.125:0/534622276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c15e1e0 msgr2=0x7fc45c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.686+0000 7fc461400640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0702c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc454009d90 tx=0x7fc45402bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 -- 172.21.15.125:0/534622276 shutdown_connections 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0702c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 --2- 172.21.15.125:0/534622276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c0cf9a0 0x7fc45c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 -- 172.21.15.125:0/534622276 >> 172.21.15.125:0/534622276 conn(0x7fc45c06a800 msgr2=0x7fc45c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 -- 172.21.15.125:0/534622276 shutdown_connections 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.687+0000 7fc461400640 1 -- 172.21.15.125:0/534622276 wait complete. 2024-01-16T14:57:42.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.688+0000 7fc461400640 1 Processor -- start 2024-01-16T14:57:42.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 -- start start 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 0x7fc45c0ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0adc80 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b07c0 con 0x7fc45c15e1e0 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.689+0000 7fc461400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0ad2e0 con 0x7fc45c143ae0 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47514/0 (socket says 172.21.15.125:47514) 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 -- 172.21.15.125:0/2907728641 learned_addr learned my addr 172.21.15.125:0/2907728641 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc459e00640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 0x7fc45c0ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45b200640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 -- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 msgr2=0x7fc45c0ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 0x7fc45c0ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 -- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc45c15e1e0 msgr2=0x7fc45c0cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 -- 172.21.15.125:0/2907728641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454009a70 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc459e00640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 0x7fc45c0ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.690+0000 7fc45a800640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c4d0 tx=0x7fc44c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.691+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c0074e0 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.691+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c007660 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.691+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c004a20 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.691+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c0ad5c0 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.691+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c0cac70 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.692+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c0675d0 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.693+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc44c004b80 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.694+0000 7fc452a00640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 0x7fc43406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.694+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc44c088e60 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.694+0000 7fc459e00640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 0x7fc43406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.695+0000 7fc459e00640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 0x7fc43406d780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0aecc0 tx=0x7fc444008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.698+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc44c053c30 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.888+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc45c070510 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.889+0000 7fc452a00640 1 -- 172.21.15.125:0/2907728641 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc44c0180a0 con 0x7fc45c0cf9a0 2024-01-16T14:57:42.871 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.891+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 msgr2=0x7fc43406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.891+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 0x7fc43406d780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0aecc0 tx=0x7fc444008040 comp rx=0 tx=0).stop 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.891+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 msgr2=0x7fc45c159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c4d0 tx=0x7fc44c00c990 comp rx=0 tx=0).stop 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 shutdown_connections 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc45c15e1e0 0x7fc45c0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc45c143ae0 0x7fc45c0ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc43406b2c0 0x7fc43406d780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 --2- 172.21.15.125:0/2907728641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0cf9a0 0x7fc45c159c90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 >> 172.21.15.125:0/2907728641 conn(0x7fc45c06a800 msgr2=0x7fc45c142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 shutdown_connections 2024-01-16T14:57:42.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:42.892+0000 7fc461400640 1 -- 172.21.15.125:0/2907728641 wait complete. 2024-01-16T14:57:42.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:43.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.109+0000 7f89a4200640 1 Processor -- start 2024-01-16T14:57:43.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.110+0000 7f89a4200640 1 -- start start 2024-01-16T14:57:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a4200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 0x7f899c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a4200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f899c15e3b0 0x7f899c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a4200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899c0cfc30 con 0x7f899c0cf7d0 2024-01-16T14:57:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a4200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f899c0b1ee0 con 0x7f899c15e3b0 2024-01-16T14:57:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a4200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899c0b2780 con 0x7f899c143ae0 2024-01-16T14:57:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a0e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f899c0cf7d0 0x7f899c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46118/0 (socket says 172.21.15.125:46118) 2024-01-16T14:57:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.111+0000 7f89a0e00640 1 -- 172.21.15.125:0/872846911 learned_addr learned my addr 172.21.15.125:0/872846911 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f89a1800640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 0x7f899c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f89a2200640 1 --2- 172.21.15.125:0/872846911 >> v2:172.21.15.137:3300/0 conn(0x7f899c15e3b0 0x7f899c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73643483 0 0) 0x7f899c0cfc30 con 0x7f899c0cf7d0 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f897c003520 con 0x7f899c0cf7d0 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988350078 0 0) 0x7f897c003520 con 0x7f899c0cf7d0 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 msgr2=0x7f899c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 0x7f899c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 >> v2:172.21.15.137:3300/0 conn(0x7f899c15e3b0 msgr2=0x7f899c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 --2- 172.21.15.125:0/872846911 >> v2:172.21.15.137:3300/0 conn(0x7f899c15e3b0 0x7f899c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c15e990 con 0x7f899c0cf7d0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f89840019f0 con 0x7f899c0cf7d0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 >> v1:172.21.15.125:6789/0 conn(0x7f899c0cf7d0 legacy=0x7f899c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f897c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.112+0000 7f899be00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f899be00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 0x7f897c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c0b1ee0 con 0x7f897c003970 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c0b2780 con 0x7f897c0047c0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c00b980 con 0x7f899c143ae0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f897c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a0e00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 -- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 msgr2=0x7f897c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c143ae0 0x7f897c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 -- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f897c0047c0 msgr2=0x7f897c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a1800640 1 -- 172.21.15.125:0/872846911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c003520 con 0x7f897c003970 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.113+0000 7f89a0e00640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.114+0000 7f89a1800640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f897c003d90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f898c009ac0 tx=0x7f897c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5487ae0e5d4b18ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.116+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c02c6d0 con 0x7f897c003970 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.116+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c02c830 con 0x7f897c003970 2024-01-16T14:57:43.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.116+0000 7f899be00640 1 -- 172.21.15.125:0/872846911 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c02cbc0 con 0x7f897c003970 2024-01-16T14:57:43.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 -- 172.21.15.125:0/872846911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 msgr2=0x7f897c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 --2- 172.21.15.125:0/872846911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f897c003d90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f898c009ac0 tx=0x7f897c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a2200640 1 -- 172.21.15.125:0/872846911 reap_dead start 2024-01-16T14:57:43.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 -- 172.21.15.125:0/872846911 shutdown_connections 2024-01-16T14:57:43.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 -- 172.21.15.125:0/872846911 >> 172.21.15.125:0/872846911 conn(0x7f899c06a800 msgr2=0x7f899c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 -- 172.21.15.125:0/872846911 shutdown_connections 2024-01-16T14:57:43.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.117+0000 7f89a4200640 1 -- 172.21.15.125:0/872846911 wait complete. 2024-01-16T14:57:43.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.119+0000 7f89a4200640 1 Processor -- start 2024-01-16T14:57:43.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.119+0000 7f89a4200640 1 -- start start 2024-01-16T14:57:43.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f899c1f0dc0 0x7f899c141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 0x7f899c142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c142590 con 0x7f897c003970 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c1f21b0 con 0x7f899c1f0dc0 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a4200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899c1f15a0 con 0x7f899c141bd0 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47536/0 (socket says 172.21.15.125:47536) 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 -- 172.21.15.125:0/1936780828 learned_addr learned my addr 172.21.15.125:0/1936780828 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a2200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 0x7f899c142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 -- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 msgr2=0x7f899c142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 0x7f899c142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a0e00640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f899c1f0dc0 0x7f899c141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 -- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f899c1f0dc0 msgr2=0x7f899c141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f899c1f0dc0 0x7f899c141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.120+0000 7f89a1800640 1 -- 172.21.15.125:0/1936780828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c1f1880 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f89a2200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 0x7f899c142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f89a1800640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f898c02c2d0 tx=0x7f898c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c003810 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c003970 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c003ce0 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c1f1b60 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.121+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c141970 con 0x7f897c003970 2024-01-16T14:57:43.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.122+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c0675d0 con 0x7f897c003970 2024-01-16T14:57:43.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.123+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f898c040660 con 0x7f897c003970 2024-01-16T14:57:43.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.124+0000 7f8999600640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 0x7f897406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.124+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f898c0abd10 con 0x7f897c003970 2024-01-16T14:57:43.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.125+0000 7f89a0e00640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 0x7f897406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.125+0000 7f89a0e00640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 0x7f897406d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8984001b20 tx=0x7f8984009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.129+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f898c0b1020 con 0x7f897c003970 2024-01-16T14:57:43.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.317+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f899c15c700 con 0x7f897c003970 2024-01-16T14:57:43.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.318+0000 7f8999600640 1 -- 172.21.15.125:0/1936780828 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f898c07a4e0 con 0x7f897c003970 2024-01-16T14:57:43.299 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:43.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 msgr2=0x7f897406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 0x7f897406d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8984001b20 tx=0x7f8984009040 comp rx=0 tx=0).stop 2024-01-16T14:57:43.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 msgr2=0x7f899c142b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f898c02c2d0 tx=0x7f898c004730 comp rx=0 tx=0).stop 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 shutdown_connections 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f899c141bd0 0x7f899c142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f899c1f0dc0 0x7f899c141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f897406b2f0 0x7f897406d7b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 --2- 172.21.15.125:0/1936780828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f897c003970 0x7f899c142b30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 >> 172.21.15.125:0/1936780828 conn(0x7f899c06a800 msgr2=0x7f899c15b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 shutdown_connections 2024-01-16T14:57:43.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.321+0000 7f89a4200640 1 -- 172.21.15.125:0/1936780828 wait complete. 2024-01-16T14:57:43.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:43.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.547+0000 7fd532400640 1 Processor -- start 2024-01-16T14:57:43.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.548+0000 7fd532400640 1 -- start start 2024-01-16T14:57:43.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd52c0cd240 0x7fd52c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c06c6b0 0x7fd52c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd52c099fa0 0x7fd52c06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c09a430 con 0x7fd52c099fa0 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c06cea0 con 0x7fd52c0cd240 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.549+0000 7fd532400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c06cbc0 con 0x7fd52c06c6b0 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd52c099fa0 0x7fd52c06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd52c099fa0 0x7fd52c06b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47552/0 (socket says 172.21.15.125:47552) 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd52c0cd240 0x7fd52c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 -- 172.21.15.125:0/4018064886 learned_addr learned my addr 172.21.15.125:0/4018064886 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52aa00640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c06c6b0 0x7fd52c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 -- 172.21.15.125:0/4018064886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c06c6b0 msgr2=0x7fd52c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c06c6b0 0x7fd52c099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 -- 172.21.15.125:0/4018064886 >> v2:172.21.15.137:3300/0 conn(0x7fd52c0cd240 msgr2=0x7fd52c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 --2- 172.21.15.125:0/4018064886 >> v2:172.21.15.137:3300/0 conn(0x7fd52c0cd240 0x7fd52c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.550+0000 7fd52be00640 1 -- 172.21.15.125:0/4018064886 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c06f350 con 0x7fd52c099fa0 2024-01-16T14:57:43.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.551+0000 7fd52be00640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c099fa0 0x7fd52c06b4f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd52c070a10 tx=0x7fd52402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c838cff5f2da5cfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd52a000640 1 -- 172.21.15.125:0/4018064886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524031870 con 0x7fd52c099fa0 2024-01-16T14:57:43.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd52a000640 1 -- 172.21.15.125:0/4018064886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5240319d0 con 0x7fd52c099fa0 2024-01-16T14:57:43.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd52a000640 1 -- 172.21.15.125:0/4018064886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524031d60 con 0x7fd52c099fa0 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 -- 172.21.15.125:0/4018064886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c099fa0 msgr2=0x7fd52c06b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c099fa0 0x7fd52c06b4f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd52c070a10 tx=0x7fd52402b910 comp rx=0 tx=0).stop 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 -- 172.21.15.125:0/4018064886 shutdown_connections 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c099fa0 0x7fd52c06b4f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 --2- 172.21.15.125:0/4018064886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c06c6b0 0x7fd52c099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 --2- 172.21.15.125:0/4018064886 >> v2:172.21.15.137:3300/0 conn(0x7fd52c0cd240 0x7fd52c0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 -- 172.21.15.125:0/4018064886 >> 172.21.15.125:0/4018064886 conn(0x7fd52c0b2d00 msgr2=0x7fd52c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 -- 172.21.15.125:0/4018064886 shutdown_connections 2024-01-16T14:57:43.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.552+0000 7fd532400640 1 -- 172.21.15.125:0/4018064886 wait complete. 2024-01-16T14:57:43.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.554+0000 7fd532400640 1 Processor -- start 2024-01-16T14:57:43.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.554+0000 7fd532400640 1 -- start start 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd52c099fa0 0x7fd52c0af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c0cd240 0x7fd52c0c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0c6410 con 0x7fd52c06c6b0 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0c5800 con 0x7fd52c099fa0 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.555+0000 7fd532400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0c5ae0 con 0x7fd52c0cd240 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47566/0 (socket says 172.21.15.125:47566) 2024-01-16T14:57:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 -- 172.21.15.125:0/3119219197 learned_addr learned my addr 172.21.15.125:0/3119219197 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52aa00640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd52c099fa0 0x7fd52c0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 -- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c0cd240 msgr2=0x7fd52c0c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c0cd240 0x7fd52c0c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 -- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd52c099fa0 msgr2=0x7fd52c0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd52c099fa0 0x7fd52c0af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 -- 172.21.15.125:0/3119219197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5240094d0 con 0x7fd52c06c6b0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.556+0000 7fd52b400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e9b0 tx=0x7fd51400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.557+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514009670 con 0x7fd52c06c6b0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.557+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51400c9a0 con 0x7fd52c06c6b0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.557+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c0c5dc0 con 0x7fd52c06c6b0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.557+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51400cd50 con 0x7fd52c06c6b0 2024-01-16T14:57:43.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.557+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c0cb090 con 0x7fd52c06c6b0 2024-01-16T14:57:43.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.558+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c0ceb20 con 0x7fd52c06c6b0 2024-01-16T14:57:43.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.559+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd514013470 con 0x7fd52c06c6b0 2024-01-16T14:57:43.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.560+0000 7fd523400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 0x7fd50406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.560+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd514089fa0 con 0x7fd52c06c6b0 2024-01-16T14:57:43.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.560+0000 7fd52aa00640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 0x7fd50406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.561+0000 7fd52aa00640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 0x7fd50406d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0b6810 tx=0x7fd51c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.564+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd514054c50 con 0x7fd52c06c6b0 2024-01-16T14:57:43.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.765+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd52c0b7390 con 0x7fd52c06c6b0 2024-01-16T14:57:43.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.766+0000 7fd523400640 1 -- 172.21.15.125:0/3119219197 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd514021080 con 0x7fd52c06c6b0 2024-01-16T14:57:43.748 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 msgr2=0x7fd50406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 0x7fd50406d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0b6810 tx=0x7fd51c00b040 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 msgr2=0x7fd52c0c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e9b0 tx=0x7fd51400ee70 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 shutdown_connections 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd52c0cd240 0x7fd52c0c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd52c099fa0 0x7fd52c0af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd50406b2c0 0x7fd50406d780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 --2- 172.21.15.125:0/3119219197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52c06c6b0 0x7fd52c0c68a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 >> 172.21.15.125:0/3119219197 conn(0x7fd52c0b2d00 msgr2=0x7fd52c0cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.769+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 shutdown_connections 2024-01-16T14:57:43.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:43.770+0000 7fd532400640 1 -- 172.21.15.125:0/3119219197 wait complete. 2024-01-16T14:57:43.768 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T14:57:43.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:43.769 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:43.851 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:43.851 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:57:43.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:43.851 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T14:57:43.901 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:43.901 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:43.957 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:43.957 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:44.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.249+0000 7f94a2c00640 1 Processor -- start 2024-01-16T14:57:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.249+0000 7f94a2c00640 1 -- start start 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a2c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f949c0cf9a0 0x7f949c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a2c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 0x7f949c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a2c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949c0b1ee0 con 0x7f949c143ae0 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a2c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949c0b2780 con 0x7f949c0cf9a0 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a2c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949c15e990 con 0x7f949c15e1e0 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f949c0cf9a0 0x7f949c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f949c0cf9a0 0x7f949c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47580/0 (socket says 172.21.15.125:47580) 2024-01-16T14:57:44.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 -- 172.21.15.125:0/1196741914 learned_addr learned my addr 172.21.15.125:0/1196741914 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a0c00640 1 --2- 172.21.15.125:0/1196741914 >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 0x7f949c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 -- 172.21.15.125:0/1196741914 >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 msgr2=0x7f949c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 --2- 172.21.15.125:0/1196741914 >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 0x7f949c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 -- 172.21.15.125:0/1196741914 >> v1:172.21.15.137:6789/0 conn(0x7f949c143ae0 legacy=0x7f949c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f949b400640 1 -- 172.21.15.125:0/1196741914 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c0b2130 con 0x7f949c0cf9a0 2024-01-16T14:57:44.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.251+0000 7f94a0c00640 1 --2- 172.21.15.125:0/1196741914 >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 0x7f949c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:44.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f949b400640 1 --2- 172.21.15.125:0/1196741914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c0cf9a0 0x7f949c15dca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f949c0ae9d0 tx=0x7f948402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fafde0e47784fe8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f949aa00640 1 -- 172.21.15.125:0/1196741914 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948402eea0 con 0x7f949c0cf9a0 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f949aa00640 1 -- 172.21.15.125:0/1196741914 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9484036440 con 0x7f949c0cf9a0 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f949aa00640 1 -- 172.21.15.125:0/1196741914 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840367d0 con 0x7f949c0cf9a0 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f94a2c00640 1 -- 172.21.15.125:0/1196741914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c0cf9a0 msgr2=0x7f949c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.253+0000 7f94a2c00640 1 --2- 172.21.15.125:0/1196741914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c0cf9a0 0x7f949c15dca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f949c0ae9d0 tx=0x7f948402e0f0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 -- 172.21.15.125:0/1196741914 shutdown_connections 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 --2- 172.21.15.125:0/1196741914 >> v2:172.21.15.137:3300/0 conn(0x7f949c15e1e0 0x7f949c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 --2- 172.21.15.125:0/1196741914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c0cf9a0 0x7f949c15dca0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 -- 172.21.15.125:0/1196741914 >> 172.21.15.125:0/1196741914 conn(0x7f949c06a800 msgr2=0x7f949c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 -- 172.21.15.125:0/1196741914 shutdown_connections 2024-01-16T14:57:44.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.254+0000 7f94a2c00640 1 -- 172.21.15.125:0/1196741914 wait complete. 2024-01-16T14:57:44.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.255+0000 7f94a2c00640 1 Processor -- start 2024-01-16T14:57:44.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.256+0000 7f94a2c00640 1 -- start start 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.256+0000 7f94a2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f949c0cf9a0 0x7f949c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.256+0000 7f94a2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 0x7f949c0cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a2c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a2c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949c1ee880 con 0x7f949c15e1e0 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949c0cb590 con 0x7f949c143ae0 2024-01-16T14:57:44.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949c1f2350 con 0x7f949c0cf9a0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47582/0 (socket says 172.21.15.125:47582) 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 -- 172.21.15.125:0/2276556 learned_addr learned my addr 172.21.15.125:0/2276556 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f949b400640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 0x7f949c0cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 -- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f949c0cf9a0 msgr2=0x7f949c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f949c0cf9a0 0x7f949c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 -- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 msgr2=0x7f949c0cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 0x7f949c0cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f94a0c00640 1 -- 172.21.15.125:0/2276556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948400b770 con 0x7f949c15e1e0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.257+0000 7f949b400640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 0x7f949c0cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f94a0c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f949400cef0 tx=0x7f949400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949400d840 con 0x7f949c15e1e0 2024-01-16T14:57:44.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949400d9c0 con 0x7f949c15e1e0 2024-01-16T14:57:44.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949400dd70 con 0x7f949c15e1e0 2024-01-16T14:57:44.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949c1f2630 con 0x7f949c15e1e0 2024-01-16T14:57:44.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.258+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949c1f2bf0 con 0x7f949c15e1e0 2024-01-16T14:57:44.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.259+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949c0719e0 con 0x7f949c15e1e0 2024-01-16T14:57:44.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.260+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f949401e3b0 con 0x7f949c15e1e0 2024-01-16T14:57:44.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.261+0000 7f9493e00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 0x7f947406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.261+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f949408a750 con 0x7f949c15e1e0 2024-01-16T14:57:44.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.261+0000 7f949be00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 0x7f947406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.262+0000 7f949be00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 0x7f947406d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f949c141660 tx=0x7f948c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.265+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94940555e0 con 0x7f949c15e1e0 2024-01-16T14:57:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.459+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f949c0675d0 con 0x7f949c15e1e0 2024-01-16T14:57:44.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.462+0000 7f9493e00640 1 -- 172.21.15.125:0/2276556 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f9494058c00 con 0x7f949c15e1e0 2024-01-16T14:57:44.443 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:44.444 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.464+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 msgr2=0x7f947406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.464+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 0x7f947406d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f949c141660 tx=0x7f948c0079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.464+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 msgr2=0x7f949c140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.464+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f949400cef0 tx=0x7f949400a710 comp rx=0 tx=0).stop 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 shutdown_connections 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f947406b2c0 0x7f947406d780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f949c15e1e0 0x7f949c140580 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f949c143ae0 0x7f949c0cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 --2- 172.21.15.125:0/2276556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f949c0cf9a0 0x7f949c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 >> 172.21.15.125:0/2276556 conn(0x7f949c06a800 msgr2=0x7f949c15b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 shutdown_connections 2024-01-16T14:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.465+0000 7f94a2c00640 1 -- 172.21.15.125:0/2276556 wait complete. 2024-01-16T14:57:44.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:44.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.687+0000 7fb98b000640 1 Processor -- start 2024-01-16T14:57:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.687+0000 7fb98b000640 1 -- start start 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb98b000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9840ae5a0 0x7fb98415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb98b000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb984070c40 con 0x7fb9840b3ae0 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb98b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98406c470 con 0x7fb9840cf290 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb98b000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb984071450 con 0x7fb9840ae5a0 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb983e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9840ae5a0 0x7fb98415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb983e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9840ae5a0 0x7fb98415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47596/0 (socket says 172.21.15.125:47596) 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb983e00640 1 -- 172.21.15.125:0/2331046650 learned_addr learned my addr 172.21.15.125:0/2331046650 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:44.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.689+0000 7fb97b200640 1 --1- 172.21.15.125:0/2331046650 >> v1:172.21.15.125:6789/0 conn(0x7fb9840b3ae0 0x7fb98406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46124/0 (socket says 172.21.15.125:46124) 2024-01-16T14:57:44.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.690+0000 7fb983e00640 1 -- 172.21.15.125:0/2331046650 >> v1:172.21.15.137:6789/0 conn(0x7fb9840cf290 legacy=0x7fb9840b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.690+0000 7fb983e00640 1 -- 172.21.15.125:0/2331046650 >> v1:172.21.15.125:6789/0 conn(0x7fb9840b3ae0 legacy=0x7fb98406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.690+0000 7fb983e00640 1 -- 172.21.15.125:0/2331046650 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98406c750 con 0x7fb9840ae5a0 2024-01-16T14:57:44.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.690+0000 7fb983400640 1 -- 172.21.15.125:0/2331046650 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213916713 0 0) 0x7fb984070c40 con 0x7fb9840b3ae0 2024-01-16T14:57:44.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.691+0000 7fb983e00640 1 --2- 172.21.15.125:0/2331046650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840ae5a0 0x7fb98415e430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb974009870 tx=0x7fb97402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c73461a36333897 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.691+0000 7fb983400640 1 -- 172.21.15.125:0/2331046650 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97402f850 con 0x7fb9840ae5a0 2024-01-16T14:57:44.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.691+0000 7fb983400640 1 -- 172.21.15.125:0/2331046650 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97402f9b0 con 0x7fb9840ae5a0 2024-01-16T14:57:44.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.691+0000 7fb983400640 1 -- 172.21.15.125:0/2331046650 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97402fd40 con 0x7fb9840ae5a0 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 -- 172.21.15.125:0/2331046650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840ae5a0 msgr2=0x7fb98415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 --2- 172.21.15.125:0/2331046650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840ae5a0 0x7fb98415e430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb974009870 tx=0x7fb97402ba30 comp rx=0 tx=0).stop 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 -- 172.21.15.125:0/2331046650 shutdown_connections 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 --2- 172.21.15.125:0/2331046650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840ae5a0 0x7fb98415e430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 -- 172.21.15.125:0/2331046650 >> 172.21.15.125:0/2331046650 conn(0x7fb98406a800 msgr2=0x7fb98406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 -- 172.21.15.125:0/2331046650 shutdown_connections 2024-01-16T14:57:44.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.692+0000 7fb98b000640 1 -- 172.21.15.125:0/2331046650 wait complete. 2024-01-16T14:57:44.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.694+0000 7fb98b000640 1 Processor -- start 2024-01-16T14:57:44.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.694+0000 7fb98b000640 1 -- start start 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9840ae5a0 0x7fb9840cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 0x7fb9840ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840cca90 con 0x7fb9840b3ae0 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840cc7b0 con 0x7fb9840cf290 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb98b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9841f6230 con 0x7fb9840ae5a0 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb97b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb97b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47608/0 (socket says 172.21.15.125:47608) 2024-01-16T14:57:44.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.695+0000 7fb97b200640 1 -- 172.21.15.125:0/3106181211 learned_addr learned my addr 172.21.15.125:0/3106181211 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb989000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 0x7fb9840ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 -- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9840ae5a0 msgr2=0x7fb9840cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9840ae5a0 0x7fb9840cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 -- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 msgr2=0x7fb9840ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 0x7fb9840ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 -- 172.21.15.125:0/3106181211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740094d0 con 0x7fb9840b3ae0 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb989000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 0x7fb9840ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:44.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb97b200640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c4d0 tx=0x7fb96c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c0074e0 con 0x7fb9840b3ae0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c007660 con 0x7fb9840b3ae0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9841f6510 con 0x7fb9840b3ae0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c004a20 con 0x7fb9840b3ae0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.696+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9840cbf00 con 0x7fb9840b3ae0 2024-01-16T14:57:44.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.697+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9841f6090 con 0x7fb9840b3ae0 2024-01-16T14:57:44.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.698+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb96c004b80 con 0x7fb9840b3ae0 2024-01-16T14:57:44.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.699+0000 7fb980c00640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 0x7fb95c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.699+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb96c089230 con 0x7fb9840b3ae0 2024-01-16T14:57:44.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.700+0000 7fb983e00640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 0x7fb95c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.700+0000 7fb983e00640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 0x7fb95c06d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb974036000 tx=0x7fb974039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.703+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9841f6090 con 0x7fb9840b3ae0 2024-01-16T14:57:44.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.894+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9841f6090 con 0x7fb9840b3ae0 2024-01-16T14:57:44.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.895+0000 7fb980c00640 1 -- 172.21.15.125:0/3106181211 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9841f6090 con 0x7fb9840b3ae0 2024-01-16T14:57:44.876 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:44.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.897+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 msgr2=0x7fb95c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 0x7fb95c06d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb974036000 tx=0x7fb974039040 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 msgr2=0x7fb984155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c4d0 tx=0x7fb96c00c990 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 shutdown_connections 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9840cf290 0x7fb9840ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb95c06b2c0 0x7fb95c06d780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9840b3ae0 0x7fb984155620 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 --2- 172.21.15.125:0/3106181211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9840ae5a0 0x7fb9840cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 >> 172.21.15.125:0/3106181211 conn(0x7fb98406a800 msgr2=0x7fb98406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 shutdown_connections 2024-01-16T14:57:44.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:44.898+0000 7fb98b000640 1 -- 172.21.15.125:0/3106181211 wait complete. 2024-01-16T14:57:44.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:45.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.127+0000 7fc2cee00640 1 Processor -- start 2024-01-16T14:57:45.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.128+0000 7fc2cee00640 1 -- start start 2024-01-16T14:57:45.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.129+0000 7fc2cee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2c815c3a0 0x7fc2c806a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.129+0000 7fc2cee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2c806f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.129+0000 7fc2cee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c806eb00 con 0x7fc2c80cd960 2024-01-16T14:57:45.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.129+0000 7fc2cee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c815c820 con 0x7fc2c815c3a0 2024-01-16T14:57:45.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2cee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806f0c0 con 0x7fc2c806f550 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2c7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2c80cd960 0x7fc2c80cdd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46140/0 (socket says 172.21.15.125:46140) 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2c7e00640 1 -- 172.21.15.125:0/1239079975 learned_addr learned my addr 172.21.15.125:0/1239079975 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2c7400640 1 --2- 172.21.15.125:0/1239079975 >> v2:172.21.15.137:3300/0 conn(0x7fc2c815c3a0 0x7fc2c806a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2cce00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2c806f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913418623 0 0) 0x7fc2c806eb00 con 0x7fc2c80cd960 2024-01-16T14:57:45.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.130+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a8003520 con 0x7fc2c80cd960 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1761832365 0 0) 0x7fc2a8003520 con 0x7fc2c80cd960 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 msgr2=0x7fc2c806f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2c806f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 >> v2:172.21.15.137:3300/0 conn(0x7fc2c815c3a0 msgr2=0x7fc2c806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 --2- 172.21.15.125:0/1239079975 >> v2:172.21.15.137:3300/0 conn(0x7fc2c815c3a0 0x7fc2c806a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c806a920 con 0x7fc2c80cd960 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc2b8001670 con 0x7fc2c80cd960 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 >> v1:172.21.15.125:6789/0 conn(0x7fc2c80cd960 legacy=0x7fc2c80cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a8003970 0x7fc2a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8004d00 0x7fc2a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c815c820 con 0x7fc2a8003970 2024-01-16T14:57:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806f0c0 con 0x7fc2a8004d00 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a800ba00 con 0x7fc2c806f550 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a8003970 0x7fc2a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.131+0000 7fc2cce00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8004d00 0x7fc2a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7400640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7e00640 1 -- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 msgr2=0x7fc2a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c806f550 0x7fc2a80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7e00640 1 -- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8004d00 msgr2=0x7fc2a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8004d00 0x7fc2a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.132+0000 7fc2c7e00640 1 -- 172.21.15.125:0/1239079975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8003520 con 0x7fc2a8003970 2024-01-16T14:57:45.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.133+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a8003970 0x7fc2a8003d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800c110 tx=0x7fc2a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b615582e8e4e10e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.133+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8005ad0 con 0x7fc2a8003970 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.133+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b8005c50 con 0x7fc2a8003970 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.133+0000 7fc2c6a00640 1 -- 172.21.15.125:0/1239079975 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b802c830 con 0x7fc2a8003970 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 -- 172.21.15.125:0/1239079975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a8003970 msgr2=0x7fc2a8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1239079975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a8003970 0x7fc2a8003d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800c110 tx=0x7fc2a80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cce00640 1 -- 172.21.15.125:0/1239079975 reap_dead start 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 -- 172.21.15.125:0/1239079975 shutdown_connections 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 -- 172.21.15.125:0/1239079975 >> 172.21.15.125:0/1239079975 conn(0x7fc2c80b1dd0 msgr2=0x7fc2c80b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 -- 172.21.15.125:0/1239079975 shutdown_connections 2024-01-16T14:57:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.134+0000 7fc2cee00640 1 -- 172.21.15.125:0/1239079975 wait complete. 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.136+0000 7fc2cee00640 1 Processor -- start 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.136+0000 7fc2cee00640 1 -- start start 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8003970 0x7fc2c81404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 0x7fc2c813cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8140040 con 0x7fc2c81394a0 2024-01-16T14:57:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8139920 con 0x7fc2a8003970 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2cee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8159060 con 0x7fc2c813cb60 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2c7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2c7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47618/0 (socket says 172.21.15.125:47618) 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2c7400640 1 -- 172.21.15.125:0/1461840375 learned_addr learned my addr 172.21.15.125:0/1461840375 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.137+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8003970 0x7fc2c81404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2cce00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 0x7fc2c813cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 -- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 msgr2=0x7fc2c813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 0x7fc2c813cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 -- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8003970 msgr2=0x7fc2c81404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8003970 0x7fc2c81404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 -- 172.21.15.125:0/1461840375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c813f430 con 0x7fc2c81394a0 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2cce00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 0x7fc2c813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:45.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2c7400640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0010910 tx=0x7fc2b0010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0009690 con 0x7fc2c81394a0 2024-01-16T14:57:45.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b000ec20 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b001c400 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c813f710 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.138+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c813fcd0 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.140+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c81401b0 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.140+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2b001c560 con 0x7fc2c81394a0 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.141+0000 7fc2bfe00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 0x7fc2a006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.141+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc2b008a0f0 con 0x7fc2c81394a0 2024-01-16T14:57:45.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.142+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 0x7fc2a006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.142+0000 7fc2c7e00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 0x7fc2a006d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8034000 tx=0x7fc2b8007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.146+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2b0054f80 con 0x7fc2c81394a0 2024-01-16T14:57:45.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.336+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2c80675d0 con 0x7fc2c81394a0 2024-01-16T14:57:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.337+0000 7fc2bfe00640 1 -- 172.21.15.125:0/1461840375 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2b00585a0 con 0x7fc2c81394a0 2024-01-16T14:57:45.318 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.339+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 msgr2=0x7fc2a006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.339+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 0x7fc2a006d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8034000 tx=0x7fc2b8007000 comp rx=0 tx=0).stop 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.339+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 msgr2=0x7fc2c813ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.339+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0010910 tx=0x7fc2b0010dd0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 shutdown_connections 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c813cb60 0x7fc2c813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc2a006b2f0 0x7fc2a006d7b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c81394a0 0x7fc2c813ed30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 --2- 172.21.15.125:0/1461840375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2a8003970 0x7fc2c81404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 >> 172.21.15.125:0/1461840375 conn(0x7fc2c80b1dd0 msgr2=0x7fc2c815b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 shutdown_connections 2024-01-16T14:57:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.340+0000 7fc2cee00640 1 -- 172.21.15.125:0/1461840375 wait complete. 2024-01-16T14:57:45.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.569+0000 7f93c0800640 1 Processor -- start 2024-01-16T14:57:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.569+0000 7f93c0800640 1 -- start start 2024-01-16T14:57:45.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.570+0000 7f93c0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b80cf9a0 0x7f93b815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.570+0000 7f93c0800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93b815e1e0 0x7f93b80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.570+0000 7f93c0800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b80b1ee0 con 0x7f93b8143ae0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.570+0000 7f93c0800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b80b2780 con 0x7f93b815e1e0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.570+0000 7f93c0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b815e990 con 0x7f93b80cf9a0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93bde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f93b8143ae0 0x7f93b80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46150/0 (socket says 172.21.15.125:46150) 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93bde00640 1 -- 172.21.15.125:0/2622680951 learned_addr learned my addr 172.21.15.125:0/2622680951 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93be800640 1 --2- 172.21.15.125:0/2622680951 >> v2:172.21.15.125:3300/0 conn(0x7f93b815e1e0 0x7f93b80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93bd400640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b80cf9a0 0x7f93b815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93be800640 1 -- 172.21.15.125:0/2622680951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b80cf9a0 msgr2=0x7f93b815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93be800640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b80cf9a0 0x7f93b815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93be800640 1 -- 172.21.15.125:0/2622680951 >> v1:172.21.15.125:6789/0 conn(0x7f93b8143ae0 legacy=0x7f93b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93be800640 1 -- 172.21.15.125:0/2622680951 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b80b2130 con 0x7f93b815e1e0 2024-01-16T14:57:45.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.571+0000 7f93bca00640 1 -- 172.21.15.125:0/2622680951 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535625000 0 0) 0x7f93b80b1ee0 con 0x7f93b8143ae0 2024-01-16T14:57:45.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.572+0000 7f93be800640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b815e1e0 0x7f93b80702c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d90 tx=0x7f93b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a4fccf5d11b2ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93bca00640 1 -- 172.21.15.125:0/2622680951 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b002cc50 con 0x7f93b815e1e0 2024-01-16T14:57:45.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93bca00640 1 -- 172.21.15.125:0/2622680951 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b002cdb0 con 0x7f93b815e1e0 2024-01-16T14:57:45.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93bca00640 1 -- 172.21.15.125:0/2622680951 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0034660 con 0x7f93b815e1e0 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 -- 172.21.15.125:0/2622680951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b815e1e0 msgr2=0x7f93b80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b815e1e0 0x7f93b80702c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d90 tx=0x7f93b002bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 -- 172.21.15.125:0/2622680951 shutdown_connections 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b815e1e0 0x7f93b80702c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 --2- 172.21.15.125:0/2622680951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b80cf9a0 0x7f93b815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 -- 172.21.15.125:0/2622680951 >> 172.21.15.125:0/2622680951 conn(0x7f93b806a800 msgr2=0x7f93b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.573+0000 7f93c0800640 1 -- 172.21.15.125:0/2622680951 shutdown_connections 2024-01-16T14:57:45.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.574+0000 7f93c0800640 1 -- 172.21.15.125:0/2622680951 wait complete. 2024-01-16T14:57:45.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.575+0000 7f93c0800640 1 Processor -- start 2024-01-16T14:57:45.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.575+0000 7f93c0800640 1 -- start start 2024-01-16T14:57:45.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f93b8143ae0 0x7f93b81f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 0x7f93b81ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b81ef630 con 0x7f93b80cf9a0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80a9fb0 con 0x7f93b8143ae0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.576+0000 7f93c0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b81f2b30 con 0x7f93b815e1e0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47648/0 (socket says 172.21.15.125:47648) 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 -- 172.21.15.125:0/3567080550 learned_addr learned my addr 172.21.15.125:0/3567080550 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bd400640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f93b8143ae0 0x7f93b81f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93be800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 0x7f93b81ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 -- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 msgr2=0x7f93b81ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 0x7f93b81ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 -- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f93b8143ae0 msgr2=0x7f93b81f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f93b8143ae0 0x7f93b81f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 -- 172.21.15.125:0/3567080550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b0009a70 con 0x7f93b80cf9a0 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93be800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 0x7f93b81ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:45.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.577+0000 7f93bde00640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c500 tx=0x7f93a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.578+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8007560 con 0x7f93b80cf9a0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.578+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a80047e0 con 0x7f93b80cf9a0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.578+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8004b90 con 0x7f93b80cf9a0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.578+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b81f2290 con 0x7f93b80cf9a0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.578+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b81f2850 con 0x7f93b80cf9a0 2024-01-16T14:57:45.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.579+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b80675d0 con 0x7f93b80cf9a0 2024-01-16T14:57:45.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.580+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f93a8004cf0 con 0x7f93b80cf9a0 2024-01-16T14:57:45.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.581+0000 7f93b6000640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 0x7f939006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.581+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f93a8088e30 con 0x7f93b80cf9a0 2024-01-16T14:57:45.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.581+0000 7f93bd400640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 0x7f939006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.582+0000 7f93bd400640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 0x7f939006d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93b80cf8f0 tx=0x7f93a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.585+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93a8053c00 con 0x7f93b80cf9a0 2024-01-16T14:57:45.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.776+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93b815ab40 con 0x7f93b80cf9a0 2024-01-16T14:57:45.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.777+0000 7f93b6000640 1 -- 172.21.15.125:0/3567080550 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93a8018020 con 0x7f93b80cf9a0 2024-01-16T14:57:45.759 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:45.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 msgr2=0x7f939006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 0x7f939006d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93b80cf8f0 tx=0x7f93a0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:45.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 msgr2=0x7f93b80a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c500 tx=0x7f93a800c9c0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 shutdown_connections 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93b815e1e0 0x7f93b81ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f93b8143ae0 0x7f93b81f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f939006b2f0 0x7f939006d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 --2- 172.21.15.125:0/3567080550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93b80cf9a0 0x7f93b80a84c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 >> 172.21.15.125:0/3567080550 conn(0x7f93b806a800 msgr2=0x7f93b80b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 shutdown_connections 2024-01-16T14:57:45.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:45.780+0000 7f93c0800640 1 -- 172.21.15.125:0/3567080550 wait complete. 2024-01-16T14:57:45.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:45.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.010+0000 7f5eee400640 1 Processor -- start 2024-01-16T14:57:45.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.011+0000 7f5eee400640 1 -- start start 2024-01-16T14:57:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5eee400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 0x7f5ee8161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5eee400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee80cf260 con 0x7f5ee80d2260 2024-01-16T14:57:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5eee400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee80cf4c0 con 0x7f5ee80c9870 2024-01-16T14:57:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5eee400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80d2ef0 con 0x7f5ee80d0560 2024-01-16T14:57:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5ee7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5ee80d2260 0x7f5ee80c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46166/0 (socket says 172.21.15.125:46166) 2024-01-16T14:57:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.012+0000 7f5ee7e00640 1 -- 172.21.15.125:0/2267212532 learned_addr learned my addr 172.21.15.125:0/2267212532 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ede800640 1 --2- 172.21.15.125:0/2267212532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 0x7f5ee8161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315196399 0 0) 0x7f5ee80cf260 con 0x7f5ee80d2260 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec8003520 con 0x7f5ee80d2260 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860355778 0 0) 0x7f5ee80cf4c0 con 0x7f5ee80c9870 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee80cf260 con 0x7f5ee80c9870 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ede800640 1 -- 172.21.15.125:0/2267212532 >> v1:172.21.15.137:6789/0 conn(0x7f5ee80c9870 legacy=0x7f5ee80ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ede800640 1 -- 172.21.15.125:0/2267212532 >> v1:172.21.15.125:6789/0 conn(0x7f5ee80d2260 legacy=0x7f5ee80c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893445925 0 0) 0x7f5ec8003520 con 0x7f5ee80d2260 2024-01-16T14:57:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.013+0000 7f5ede800640 1 -- 172.21.15.125:0/2267212532 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee80cf7a0 con 0x7f5ee80d0560 2024-01-16T14:57:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.015+0000 7f5ede800640 1 --2- 172.21.15.125:0/2267212532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 0x7f5ee8161030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00097d0 tx=0x7f5ed002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=323ee93d378a9b1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.015+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0030800 con 0x7f5ee80d0560 2024-01-16T14:57:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.015+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed0030960 con 0x7f5ee80d0560 2024-01-16T14:57:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.015+0000 7f5ee6a00640 1 -- 172.21.15.125:0/2267212532 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0030cf0 con 0x7f5ee80d0560 2024-01-16T14:57:45.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 -- 172.21.15.125:0/2267212532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 msgr2=0x7f5ee8161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 --2- 172.21.15.125:0/2267212532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 0x7f5ee8161030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00097d0 tx=0x7f5ed002b940 comp rx=0 tx=0).stop 2024-01-16T14:57:45.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 -- 172.21.15.125:0/2267212532 shutdown_connections 2024-01-16T14:57:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 --2- 172.21.15.125:0/2267212532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d0560 0x7f5ee8161030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 -- 172.21.15.125:0/2267212532 >> 172.21.15.125:0/2267212532 conn(0x7f5ee809c560 msgr2=0x7f5ee80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 -- 172.21.15.125:0/2267212532 shutdown_connections 2024-01-16T14:57:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.016+0000 7f5eee400640 1 -- 172.21.15.125:0/2267212532 wait complete. 2024-01-16T14:57:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.018+0000 7f5eee400640 1 Processor -- start 2024-01-16T14:57:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.018+0000 7f5eee400640 1 -- start start 2024-01-16T14:57:46.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 0x7f5ee814ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d2260 0x7f5ee80ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80cb630 con 0x7f5ee80d0560 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80cb910 con 0x7f5ee80c9870 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5eee400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80cbbf0 con 0x7f5ee80d2260 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5ede800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5ede800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47664/0 (socket says 172.21.15.125:47664) 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5ede800640 1 -- 172.21.15.125:0/662242616 learned_addr learned my addr 172.21.15.125:0/662242616 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5ee7e00640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d2260 0x7f5ee80ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.019+0000 7f5ee7400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 0x7f5ee814ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 -- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d2260 msgr2=0x7f5ee80ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d2260 0x7f5ee80ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 -- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 msgr2=0x7f5ee814ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 0x7f5ee814ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 -- 172.21.15.125:0/662242616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0009400 con 0x7f5ee80d0560 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ee7400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 0x7f5ee814ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5ede800640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00097d0 tx=0x7f5ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed003e040 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed0039c10 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0004030 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.020+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee80cbed0 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.021+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee80cb270 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.022+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee80675d0 con 0x7f5ee80d0560 2024-01-16T14:57:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.023+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5ed0004190 con 0x7f5ee80d0560 2024-01-16T14:57:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.023+0000 7f5edfe00640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 0x7f5ec006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.024+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5ed00abdc0 con 0x7f5ee80d0560 2024-01-16T14:57:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.024+0000 7f5ee7400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 0x7f5ec006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.024+0000 7f5ee7400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 0x7f5ec006d7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80011e0 tx=0x7f5ed8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.028+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ed0076c40 con 0x7f5ee80d0560 2024-01-16T14:57:46.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.233+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5ee80cb770 con 0x7f5ee80d0560 2024-01-16T14:57:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.234+0000 7f5edfe00640 1 -- 172.21.15.125:0/662242616 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5ee80cb770 con 0x7f5ee80d0560 2024-01-16T14:57:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:46.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.236+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 msgr2=0x7f5ec006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.236+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 0x7f5ec006d7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80011e0 tx=0x7f5ed8009040 comp rx=0 tx=0).stop 2024-01-16T14:57:46.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.236+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 msgr2=0x7f5ee80cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.236+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00097d0 tx=0x7f5ed0004730 comp rx=0 tx=0).stop 2024-01-16T14:57:46.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 shutdown_connections 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ee80d2260 0x7f5ee80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ec006b2f0 0x7f5ec006d7b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ee80d0560 0x7f5ee80cad30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 --2- 172.21.15.125:0/662242616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ee80c9870 0x7f5ee814ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 >> 172.21.15.125:0/662242616 conn(0x7f5ee809c560 msgr2=0x7f5ee80c7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 shutdown_connections 2024-01-16T14:57:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.237+0000 7f5eee400640 1 -- 172.21.15.125:0/662242616 wait complete. 2024-01-16T14:57:46.237 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:57:46.237 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:46.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:46.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.519+0000 7ff43b200640 1 Processor -- start 2024-01-16T14:57:46.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.520+0000 7ff43b200640 1 -- start start 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff43b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff43b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff43c13fcb0 0x7ff43c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff43b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c0cdec0 con 0x7ff43c0cd530 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff43b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0ce1a0 con 0x7ff43c13fcb0 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff43b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0ce480 con 0x7ff43c13cfa0 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff439e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff439e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:53120/0 (socket says 172.21.15.125:53120) 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.522+0000 7ff439e00640 1 -- 172.21.15.125:0/406337452 learned_addr learned my addr 172.21.15.125:0/406337452 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:46.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.523+0000 7ff43a800640 1 --2- 172.21.15.125:0/406337452 >> v2:172.21.15.137:3300/0 conn(0x7ff43c13fcb0 0x7ff43c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.523+0000 7ff439e00640 1 -- 172.21.15.125:0/406337452 >> v2:172.21.15.137:3300/0 conn(0x7ff43c13fcb0 msgr2=0x7ff43c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:46.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.523+0000 7ff439e00640 1 --2- 172.21.15.125:0/406337452 >> v2:172.21.15.137:3300/0 conn(0x7ff43c13fcb0 0x7ff43c0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.523+0000 7ff439e00640 1 -- 172.21.15.125:0/406337452 >> v1:172.21.15.137:6789/0 conn(0x7ff43c0cd530 legacy=0x7ff43c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.523+0000 7ff439e00640 1 -- 172.21.15.125:0/406337452 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0ce760 con 0x7ff43c13cfa0 2024-01-16T14:57:46.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.524+0000 7ff439e00640 1 --2- 172.21.15.125:0/406337452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009870 tx=0x7ff42c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9665183d34c9576 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.525+0000 7ff438a00640 1 -- 172.21.15.125:0/406337452 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c031640 con 0x7ff43c13cfa0 2024-01-16T14:57:46.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.525+0000 7ff438a00640 1 -- 172.21.15.125:0/406337452 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c0317a0 con 0x7ff43c13cfa0 2024-01-16T14:57:46.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.525+0000 7ff438a00640 1 -- 172.21.15.125:0/406337452 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c031b30 con 0x7ff43c13cfa0 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.525+0000 7ff43b200640 1 -- 172.21.15.125:0/406337452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 msgr2=0x7ff43c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.525+0000 7ff43b200640 1 --2- 172.21.15.125:0/406337452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009870 tx=0x7ff42c02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 -- 172.21.15.125:0/406337452 shutdown_connections 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 --2- 172.21.15.125:0/406337452 >> v2:172.21.15.137:3300/0 conn(0x7ff43c13fcb0 0x7ff43c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 --2- 172.21.15.125:0/406337452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c13cfa0 0x7ff43c15afb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 -- 172.21.15.125:0/406337452 >> 172.21.15.125:0/406337452 conn(0x7ff43c06a800 msgr2=0x7ff43c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 -- 172.21.15.125:0/406337452 shutdown_connections 2024-01-16T14:57:46.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.526+0000 7ff43b200640 1 -- 172.21.15.125:0/406337452 wait complete. 2024-01-16T14:57:46.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.528+0000 7ff43b200640 1 Processor -- start 2024-01-16T14:57:46.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.528+0000 7ff43b200640 1 -- start start 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c0cd530 0x7ff43c137550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff43c13cfa0 0x7ff43c13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c15c380 con 0x7ff43c13fcb0 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c15c660 con 0x7ff43c13cfa0 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.529+0000 7ff43b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c15c940 con 0x7ff43c0cd530 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47670/0 (socket says 172.21.15.125:47670) 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 -- 172.21.15.125:0/1251475145 learned_addr learned my addr 172.21.15.125:0/1251475145 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:46.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff439e00640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c0cd530 0x7ff43c137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff439400640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff43c13cfa0 0x7ff43c13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 -- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c0cd530 msgr2=0x7ff43c137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c0cd530 0x7ff43c137550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 -- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff43c13cfa0 msgr2=0x7ff43c13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff43c13cfa0 0x7ff43c13d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 -- 172.21.15.125:0/1251475145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0094d0 con 0x7ff43c13fcb0 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.530+0000 7ff43a800640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff434007e60 tx=0x7ff43400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.531+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434014420 con 0x7ff43c13fcb0 2024-01-16T14:57:46.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.531+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4340145a0 con 0x7ff43c13fcb0 2024-01-16T14:57:46.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.531+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434014950 con 0x7ff43c13fcb0 2024-01-16T14:57:46.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.531+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c15cc20 con 0x7ff43c13fcb0 2024-01-16T14:57:46.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.531+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c15d420 con 0x7ff43c13fcb0 2024-01-16T14:57:46.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.532+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c15d140 con 0x7ff43c13fcb0 2024-01-16T14:57:46.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.533+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff434014ab0 con 0x7ff43c13fcb0 2024-01-16T14:57:46.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.534+0000 7ff432000640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 0x7ff41406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.534+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff43408a4c0 con 0x7ff43c13fcb0 2024-01-16T14:57:46.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.534+0000 7ff439e00640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 0x7ff41406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.535+0000 7ff439e00640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 0x7ff41406d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff42c036000 tx=0x7ff42c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.540+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff43c15d140 con 0x7ff43c13fcb0 2024-01-16T14:57:46.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.734+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff43c15d140 con 0x7ff43c13fcb0 2024-01-16T14:57:46.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.736+0000 7ff432000640 1 -- 172.21.15.125:0/1251475145 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7ff43c15d140 con 0x7ff43c13fcb0 2024-01-16T14:57:46.718 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:46.718 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 msgr2=0x7ff41406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 0x7ff41406d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff42c036000 tx=0x7ff42c03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 msgr2=0x7ff43c13d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff434007e60 tx=0x7ff43400e4b0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 shutdown_connections 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff41406b2c0 0x7ff41406d780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43c13fcb0 0x7ff43c13d950 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff43c13cfa0 0x7ff43c13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 --2- 172.21.15.125:0/1251475145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff43c0cd530 0x7ff43c137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 >> 172.21.15.125:0/1251475145 conn(0x7ff43c06a800 msgr2=0x7ff43c13b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 shutdown_connections 2024-01-16T14:57:46.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.739+0000 7ff43b200640 1 -- 172.21.15.125:0/1251475145 wait complete. 2024-01-16T14:57:46.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:46.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.968+0000 7f0b3b200640 1 Processor -- start 2024-01-16T14:57:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.969+0000 7f0b3b200640 1 -- start start 2024-01-16T14:57:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.970+0000 7f0b3b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0b340cf9a0 0x7f0b34166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.970+0000 7f0b3b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3415c5f0 con 0x7f0b3413df80 2024-01-16T14:57:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.970+0000 7f0b3b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3406c470 con 0x7f0b3413e360 2024-01-16T14:57:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.970+0000 7f0b3b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3406c750 con 0x7f0b340cf9a0 2024-01-16T14:57:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b33e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0b3413df80 0x7f0b3415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48432/0 (socket says 172.21.15.125:48432) 2024-01-16T14:57:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b33e00640 1 -- 172.21.15.125:0/2569341870 learned_addr learned my addr 172.21.15.125:0/2569341870 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b39200640 1 --2- 172.21.15.125:0/2569341870 >> v2:172.21.15.125:3300/0 conn(0x7f0b340cf9a0 0x7f0b34166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b39200640 1 -- 172.21.15.125:0/2569341870 >> v1:172.21.15.137:6789/0 conn(0x7f0b3413e360 legacy=0x7f0b340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b39200640 1 -- 172.21.15.125:0/2569341870 >> v1:172.21.15.125:6789/0 conn(0x7f0b3413df80 legacy=0x7f0b3415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.971+0000 7f0b39200640 1 -- 172.21.15.125:0/2569341870 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3406ca30 con 0x7f0b340cf9a0 2024-01-16T14:57:46.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.972+0000 7f0b39200640 1 --2- 172.21.15.125:0/2569341870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b340cf9a0 0x7f0b34166030 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009d90 tx=0x7f0b2c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e13f39e276f8c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.972+0000 7f0b33400640 1 -- 172.21.15.125:0/2569341870 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c02cd00 con 0x7f0b340cf9a0 2024-01-16T14:57:46.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b33400640 1 -- 172.21.15.125:0/2569341870 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c02ce60 con 0x7f0b340cf9a0 2024-01-16T14:57:46.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b33400640 1 -- 172.21.15.125:0/2569341870 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c034660 con 0x7f0b340cf9a0 2024-01-16T14:57:46.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 -- 172.21.15.125:0/2569341870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b340cf9a0 msgr2=0x7f0b34166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2569341870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b340cf9a0 0x7f0b34166030 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009d90 tx=0x7f0b2c02bf50 comp rx=0 tx=0).stop 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 -- 172.21.15.125:0/2569341870 shutdown_connections 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2569341870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b340cf9a0 0x7f0b34166030 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 -- 172.21.15.125:0/2569341870 >> 172.21.15.125:0/2569341870 conn(0x7f0b3406a800 msgr2=0x7f0b3406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 -- 172.21.15.125:0/2569341870 shutdown_connections 2024-01-16T14:57:46.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.973+0000 7f0b3b200640 1 -- 172.21.15.125:0/2569341870 wait complete. 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.975+0000 7f0b3b200640 1 Processor -- start 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.975+0000 7f0b3b200640 1 -- start start 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b340cf9a0 0x7f0b341f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b3413e360 0x7f0b340adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b340a8580 con 0x7f0b3413df80 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341f6cc0 con 0x7f0b340cf9a0 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b3b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341f6fa0 con 0x7f0b3413e360 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b2b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b2b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55472/0 (socket says 172.21.15.125:55472) 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b2b200640 1 -- 172.21.15.125:0/2097892336 learned_addr learned my addr 172.21.15.125:0/2097892336 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b33e00640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b340cf9a0 0x7f0b341f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.976+0000 7f0b39200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b3413e360 0x7f0b340adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 -- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b3413e360 msgr2=0x7f0b340adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b3413e360 0x7f0b340adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 -- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b340cf9a0 msgr2=0x7f0b341f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b340cf9a0 0x7f0b341f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 -- 172.21.15.125:0/2097892336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c009a70 con 0x7f0b3413df80 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b2b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c400 tx=0x7f0b1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c0052d0 con 0x7f0b3413df80 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c005430 con 0x7f0b3413df80 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b341f7280 con 0x7f0b3413df80 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.977+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c003830 con 0x7f0b3413df80 2024-01-16T14:57:46.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.978+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b340ae1c0 con 0x7f0b3413df80 2024-01-16T14:57:46.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.979+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b340675d0 con 0x7f0b3413df80 2024-01-16T14:57:46.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.979+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0b1c003990 con 0x7f0b3413df80 2024-01-16T14:57:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.980+0000 7f0b30c00640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 0x7f0b0c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.980+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0b1c0891d0 con 0x7f0b3413df80 2024-01-16T14:57:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.981+0000 7f0b33e00640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 0x7f0b0c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.981+0000 7f0b33e00640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 0x7f0b0c06d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005880 tx=0x7f0b24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:46.985+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b1c054000 con 0x7f0b3413df80 2024-01-16T14:57:47.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.175+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b340cc480 con 0x7f0b3413df80 2024-01-16T14:57:47.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.176+0000 7f0b30c00640 1 -- 172.21.15.125:0/2097892336 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b1c057620 con 0x7f0b3413df80 2024-01-16T14:57:47.158 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 msgr2=0x7f0b0c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 0x7f0b0c06d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005880 tx=0x7f0b24009040 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 msgr2=0x7f0b3412f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c400 tx=0x7f0b1c00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 shutdown_connections 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b3413e360 0x7f0b340adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0b0c06b2c0 0x7f0b0c06d780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b3413df80 0x7f0b3412f6a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 --2- 172.21.15.125:0/2097892336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b340cf9a0 0x7f0b341f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.179+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 >> 172.21.15.125:0/2097892336 conn(0x7f0b3406a800 msgr2=0x7f0b340b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.180+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 shutdown_connections 2024-01-16T14:57:47.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.180+0000 7f0b3b200640 1 -- 172.21.15.125:0/2097892336 wait complete. 2024-01-16T14:57:47.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.408+0000 7f36b5800640 1 Processor -- start 2024-01-16T14:57:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.408+0000 7f36b5800640 1 -- start start 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36b5800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f36b013e360 0x7f36b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36b5800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b015c5f0 con 0x7f36b015f9c0 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36b5800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b006c470 con 0x7f36b013df80 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36b5800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b006c750 con 0x7f36b013e360 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36af400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f36b015f9c0 0x7f36b00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48440/0 (socket says 172.21.15.125:48440) 2024-01-16T14:57:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.410+0000 7f36af400640 1 -- 172.21.15.125:0/3940843416 learned_addr learned my addr 172.21.15.125:0/3940843416 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:47.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770271080 0 0) 0x7f36b015c5f0 con 0x7f36b015f9c0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3690003520 con 0x7f36b015f9c0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ae000640 1 --2- 172.21.15.125:0/3940843416 >> v2:172.21.15.137:3300/0 conn(0x7f36b013e360 0x7f36b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215835327 0 0) 0x7f3690003520 con 0x7f36b015f9c0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 >> v2:172.21.15.137:3300/0 conn(0x7f36b013e360 msgr2=0x7f36b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 --2- 172.21.15.125:0/3940843416 >> v2:172.21.15.137:3300/0 conn(0x7f36b013e360 0x7f36b00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 >> v1:172.21.15.137:6789/0 conn(0x7f36b013df80 legacy=0x7f36b015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b006ca30 con 0x7f36b015f9c0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f36a8001a10 con 0x7f36b015f9c0 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 >> v1:172.21.15.125:6789/0 conn(0x7f36b015f9c0 legacy=0x7f36b00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3690003970 0x7f3690003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 0x7f3690004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3690005180 0x7f369000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b006c750 con 0x7f36900047c0 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369000c0d0 con 0x7f3690003970 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369000c290 con 0x7f3690005180 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.411+0000 7f36ae000640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 0x7f3690004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36aea00640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3690003970 0x7f3690003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36af400640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3690005180 0x7f369000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36ae000640 1 -- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3690005180 msgr2=0x7f369000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36ae000640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3690005180 0x7f369000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36ae000640 1 -- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3690003970 msgr2=0x7f3690003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36ae000640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3690003970 0x7f3690003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36ae000640 1 -- 172.21.15.125:0/3940843416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3690003520 con 0x7f36900047c0 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36aea00640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3690003970 0x7f3690003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:47.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.412+0000 7f36af400640 1 -- 172.21.15.125:0/3940843416 reap_dead start 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.413+0000 7f36ae000640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 0x7f3690004c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f369800ba10 tx=0x7f36900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374ed25ed270287e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.413+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698032630 con 0x7f36900047c0 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.413+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698032790 con 0x7f36900047c0 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.413+0000 7f36ad600640 1 -- 172.21.15.125:0/3940843416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698032b20 con 0x7f36900047c0 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 -- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 msgr2=0x7f3690004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 0x7f3690004c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f369800ba10 tx=0x7f36900057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 -- 172.21.15.125:0/3940843416 shutdown_connections 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 --2- 172.21.15.125:0/3940843416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36900047c0 0x7f3690004c40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 -- 172.21.15.125:0/3940843416 >> 172.21.15.125:0/3940843416 conn(0x7f36b006a800 msgr2=0x7f36b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 -- 172.21.15.125:0/3940843416 shutdown_connections 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.414+0000 7f36b5800640 1 -- 172.21.15.125:0/3940843416 wait complete. 2024-01-16T14:57:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.416+0000 7f36b5800640 1 Processor -- start 2024-01-16T14:57:47.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.416+0000 7f36b5800640 1 -- start start 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 0x7f36b015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36b014cfa0 0x7f36b014d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b014d960 con 0x7f36b014eb80 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b01f3540 con 0x7f36900047c0 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36b5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b01f3820 con 0x7f36b014cfa0 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55484/0 (socket says 172.21.15.125:55484) 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 -- 172.21.15.125:0/2194755750 learned_addr learned my addr 172.21.15.125:0/2194755750 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36aea00640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 0x7f36b015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 -- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36b014cfa0 msgr2=0x7f36b014d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36b014cfa0 0x7f36b014d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 -- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 msgr2=0x7f36b015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 0x7f36b015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36af400640 1 -- 172.21.15.125:0/2194755750 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b01f3b00 con 0x7f36b014eb80 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.417+0000 7f36aea00640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 0x7f36b015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36af400640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f36b00cfea0 tx=0x7f36a800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8004d60 con 0x7f36b014eb80 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a800bb80 con 0x7f36b014eb80 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b01f3de0 con 0x7f36b014eb80 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8014430 con 0x7f36b014eb80 2024-01-16T14:57:47.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.418+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b014f550 con 0x7f36b014eb80 2024-01-16T14:57:47.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.419+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b01f4300 con 0x7f36b014eb80 2024-01-16T14:57:47.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.420+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f36a8014590 con 0x7f36b014eb80 2024-01-16T14:57:47.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.421+0000 7f36a6a00640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 0x7f368806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.421+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f36a808a130 con 0x7f36b014eb80 2024-01-16T14:57:47.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.421+0000 7f36aea00640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 0x7f368806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.422+0000 7f36aea00640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 0x7f368806d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36b006d8c0 tx=0x7f36a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.425+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36b01f4300 con 0x7f36b014eb80 2024-01-16T14:57:47.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.617+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36b01f4300 con 0x7f36b014eb80 2024-01-16T14:57:47.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.618+0000 7f36a6a00640 1 -- 172.21.15.125:0/2194755750 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f36b01f4300 con 0x7f36b014eb80 2024-01-16T14:57:47.599 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:47.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 msgr2=0x7f368806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 0x7f368806d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36b006d8c0 tx=0x7f36a0009040 comp rx=0 tx=0).stop 2024-01-16T14:57:47.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 msgr2=0x7f36b014f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f36b00cfea0 tx=0x7f36a800b4a0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 shutdown_connections 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f368806b2c0 0x7f368806d780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36b014eb80 0x7f36b014f010 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36b014cfa0 0x7f36b014d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 --2- 172.21.15.125:0/2194755750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36900047c0 0x7f36b015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.620+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 >> 172.21.15.125:0/2194755750 conn(0x7f36b006a800 msgr2=0x7f36b00b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.621+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 shutdown_connections 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.621+0000 7f36b5800640 1 -- 172.21.15.125:0/2194755750 wait complete. 2024-01-16T14:57:47.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.852+0000 7f7e4ea00640 1 Processor -- start 2024-01-16T14:57:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.852+0000 7f7e4ea00640 1 -- start start 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.853+0000 7f7e4ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7e48070a30 0x7f7e4815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.853+0000 7f7e4ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4815f4e0 con 0x7f7e480adf20 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.853+0000 7f7e4ea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e48070ea0 con 0x7f7e480ccb50 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.853+0000 7f7e4ea00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4815c450 con 0x7f7e48070a30 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.854+0000 7f7e4d600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7e480adf20 0x7f7e480cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48448/0 (socket says 172.21.15.125:48448) 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.854+0000 7f7e4d600640 1 -- 172.21.15.125:0/2679758499 learned_addr learned my addr 172.21.15.125:0/2679758499 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:47.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.854+0000 7f7e4e000640 1 --2- 172.21.15.125:0/2679758499 >> v2:172.21.15.125:3300/0 conn(0x7f7e48070a30 0x7f7e4815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.854+0000 7f7e47e00640 1 -- 172.21.15.125:0/2679758499 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090306322 0 0) 0x7f7e4815f4e0 con 0x7f7e480adf20 2024-01-16T14:57:47.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.854+0000 7f7e47e00640 1 -- 172.21.15.125:0/2679758499 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e28003520 con 0x7f7e480adf20 2024-01-16T14:57:47.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.855+0000 7f7e4e000640 1 -- 172.21.15.125:0/2679758499 >> v1:172.21.15.137:6789/0 conn(0x7f7e480ccb50 legacy=0x7f7e48143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.855+0000 7f7e4e000640 1 -- 172.21.15.125:0/2679758499 >> v1:172.21.15.125:6789/0 conn(0x7f7e480adf20 legacy=0x7f7e480cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.855+0000 7f7e4e000640 1 -- 172.21.15.125:0/2679758499 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48071130 con 0x7f7e48070a30 2024-01-16T14:57:47.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.856+0000 7f7e4e000640 1 --2- 172.21.15.125:0/2679758499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e48070a30 0x7f7e4815bf10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009d90 tx=0x7f7e4002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba021ceed98176b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.856+0000 7f7e47e00640 1 -- 172.21.15.125:0/2679758499 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4002cc60 con 0x7f7e48070a30 2024-01-16T14:57:47.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.856+0000 7f7e47e00640 1 -- 172.21.15.125:0/2679758499 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4002cdc0 con 0x7f7e48070a30 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.856+0000 7f7e47e00640 1 -- 172.21.15.125:0/2679758499 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40034660 con 0x7f7e48070a30 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 -- 172.21.15.125:0/2679758499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e48070a30 msgr2=0x7f7e4815bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/2679758499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e48070a30 0x7f7e4815bf10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009d90 tx=0x7f7e4002beb0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 -- 172.21.15.125:0/2679758499 shutdown_connections 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/2679758499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e48070a30 0x7f7e4815bf10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 -- 172.21.15.125:0/2679758499 >> 172.21.15.125:0/2679758499 conn(0x7f7e4806a7d0 msgr2=0x7f7e4806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 -- 172.21.15.125:0/2679758499 shutdown_connections 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.857+0000 7f7e4ea00640 1 -- 172.21.15.125:0/2679758499 wait complete. 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.858+0000 7f7e4ea00640 1 Processor -- start 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.859+0000 7f7e4ea00640 1 -- start start 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e48070a30 0x7f7e4813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e480adf20 0x7f7e480b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4813fd20 con 0x7f7e480ccb50 2024-01-16T14:57:47.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e480b31e0 con 0x7f7e48070a30 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4813e740 con 0x7f7e480adf20 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55496/0 (socket says 172.21.15.125:55496) 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 -- 172.21.15.125:0/3763543204 learned_addr learned my addr 172.21.15.125:0/3763543204 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4d600640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e48070a30 0x7f7e4813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 -- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e480adf20 msgr2=0x7f7e480b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e480adf20 0x7f7e480b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 -- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e48070a30 msgr2=0x7f7e4813f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e48070a30 0x7f7e4813f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.860+0000 7f7e4e000640 1 -- 172.21.15.125:0/3763543204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40009a70 con 0x7f7e480ccb50 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e4e000640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009d90 tx=0x7f7e40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40040650 con 0x7f7e480ccb50 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e400407b0 con 0x7f7e480ccb50 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40040b60 con 0x7f7e480ccb50 2024-01-16T14:57:47.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4813ea20 con 0x7f7e480ccb50 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.861+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4813efe0 con 0x7f7e480ccb50 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.863+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4815c270 con 0x7f7e480ccb50 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.863+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7e40040cc0 con 0x7f7e480ccb50 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.864+0000 7f7e45600640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 0x7f7e2006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.864+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7e400aca80 con 0x7f7e480ccb50 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.865+0000 7f7e4d600640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 0x7f7e2006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.865+0000 7f7e4d600640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 0x7f7e2006d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e380011e0 tx=0x7f7e38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:47.869+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e400777c0 con 0x7f7e480ccb50 2024-01-16T14:57:48.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.061+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e48067450 con 0x7f7e480ccb50 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.062+0000 7f7e45600640 1 -- 172.21.15.125:0/3763543204 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e40009450 con 0x7f7e480ccb50 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 msgr2=0x7f7e2006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 0x7f7e2006d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e380011e0 tx=0x7f7e38009040 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 msgr2=0x7f7e480ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009d90 tx=0x7f7e40004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 shutdown_connections 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e2006b2c0 0x7f7e2006d780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e480ccb50 0x7f7e480ae9e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e480adf20 0x7f7e480b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 --2- 172.21.15.125:0/3763543204 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e48070a30 0x7f7e4813f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.064+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 >> 172.21.15.125:0/3763543204 conn(0x7f7e4806a7d0 msgr2=0x7f7e4815dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.065+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 shutdown_connections 2024-01-16T14:57:48.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.065+0000 7f7e4ea00640 1 -- 172.21.15.125:0/3763543204 wait complete. 2024-01-16T14:57:48.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:48.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.286+0000 7f6955600640 1 Processor -- start 2024-01-16T14:57:48.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.286+0000 7f6955600640 1 -- start start 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6955600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f695013df80 0x7f695015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6955600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f69500cf9a0 0x7f69500cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6955600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695006c470 con 0x7f695013e360 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6955600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695006c750 con 0x7f695013df80 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6955600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695006ca30 con 0x7f69500cf9a0 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f694ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f695013df80 0x7f695015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6945e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f695013e360 0x7f69500cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48458/0 (socket says 172.21.15.125:48458) 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f6945e00640 1 -- 172.21.15.125:0/2005936630 learned_addr learned my addr 172.21.15.125:0/2005936630 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.288+0000 7f694f400640 1 --2- 172.21.15.125:0/2005936630 >> v2:172.21.15.137:3300/0 conn(0x7f69500cf9a0 0x7f69500cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.289+0000 7f694ea00640 1 -- 172.21.15.125:0/2005936630 >> v2:172.21.15.137:3300/0 conn(0x7f69500cf9a0 msgr2=0x7f69500cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:48.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.289+0000 7f694ea00640 1 --2- 172.21.15.125:0/2005936630 >> v2:172.21.15.137:3300/0 conn(0x7f69500cf9a0 0x7f69500cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.289+0000 7f694ea00640 1 -- 172.21.15.125:0/2005936630 >> v1:172.21.15.125:6789/0 conn(0x7f695013e360 legacy=0x7f69500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.289+0000 7f694ea00640 1 -- 172.21.15.125:0/2005936630 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695006cd10 con 0x7f695013df80 2024-01-16T14:57:48.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.289+0000 7f694e000640 1 -- 172.21.15.125:0/2005936630 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472109043 0 0) 0x7f695006c470 con 0x7f695013e360 2024-01-16T14:57:48.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.290+0000 7f694ea00640 1 --2- 172.21.15.125:0/2005936630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f695015bee0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69500ccb10 tx=0x7f694002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d03168d5797c0e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f694e000640 1 -- 172.21.15.125:0/2005936630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940031830 con 0x7f695013df80 2024-01-16T14:57:48.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f694e000640 1 -- 172.21.15.125:0/2005936630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940031990 con 0x7f695013df80 2024-01-16T14:57:48.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f694e000640 1 -- 172.21.15.125:0/2005936630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940031d20 con 0x7f695013df80 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 -- 172.21.15.125:0/2005936630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 msgr2=0x7f695015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 --2- 172.21.15.125:0/2005936630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f695015bee0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69500ccb10 tx=0x7f694002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 -- 172.21.15.125:0/2005936630 shutdown_connections 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 --2- 172.21.15.125:0/2005936630 >> v2:172.21.15.137:3300/0 conn(0x7f69500cf9a0 0x7f69500cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 --2- 172.21.15.125:0/2005936630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f695015bee0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.291+0000 7f6955600640 1 -- 172.21.15.125:0/2005936630 >> 172.21.15.125:0/2005936630 conn(0x7f695006a800 msgr2=0x7f695006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.292+0000 7f6955600640 1 -- 172.21.15.125:0/2005936630 shutdown_connections 2024-01-16T14:57:48.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.292+0000 7f6955600640 1 -- 172.21.15.125:0/2005936630 wait complete. 2024-01-16T14:57:48.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.293+0000 7f6955600640 1 Processor -- start 2024-01-16T14:57:48.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.293+0000 7f6955600640 1 -- start start 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69500cf9a0 0x7f695013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 0x7f695015a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695013de10 con 0x7f695013df80 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950152cb0 con 0x7f695013e360 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.294+0000 7f6955600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6950152380 con 0x7f69500cf9a0 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55520/0 (socket says 172.21.15.125:55520) 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 -- 172.21.15.125:0/1553580236 learned_addr learned my addr 172.21.15.125:0/1553580236 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:48.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f694ea00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69500cf9a0 0x7f695013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f694f400640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 0x7f695015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 -- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69500cf9a0 msgr2=0x7f695013d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69500cf9a0 0x7f695013d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 -- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 msgr2=0x7f695015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 0x7f695015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 -- 172.21.15.125:0/1553580236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6940009400 con 0x7f695013df80 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f694f400640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 0x7f695015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.295+0000 7f6945e00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f693800c820 tx=0x7f693800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.296+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693800a640 con 0x7f695013df80 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.296+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693800a7c0 con 0x7f695013df80 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.296+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6950152660 con 0x7f695013df80 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.296+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693800ab70 con 0x7f695013df80 2024-01-16T14:57:48.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.296+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69500a9870 con 0x7f695013df80 2024-01-16T14:57:48.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.297+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69500675d0 con 0x7f695013df80 2024-01-16T14:57:48.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.298+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f693800acd0 con 0x7f695013df80 2024-01-16T14:57:48.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.299+0000 7f6947400640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 0x7f692806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.299+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6938089460 con 0x7f695013df80 2024-01-16T14:57:48.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.299+0000 7f694ea00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 0x7f692806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.300+0000 7f694ea00640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 0x7f692806d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6940036000 tx=0x7f694003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.303+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69380542d0 con 0x7f695013df80 2024-01-16T14:57:48.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.509+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6950152df0 con 0x7f695013df80 2024-01-16T14:57:48.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.510+0000 7f6947400640 1 -- 172.21.15.125:0/1553580236 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f69380578f0 con 0x7f695013df80 2024-01-16T14:57:48.491 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.512+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 msgr2=0x7f692806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.512+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 0x7f692806d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6940036000 tx=0x7f694003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.512+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 msgr2=0x7f69500a8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.512+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f693800c820 tx=0x7f693800cce0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 shutdown_connections 2024-01-16T14:57:48.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f695013e360 0x7f695015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f692806b2c0 0x7f692806d780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695013df80 0x7f69500a8db0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 --2- 172.21.15.125:0/1553580236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69500cf9a0 0x7f695013d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 >> 172.21.15.125:0/1553580236 conn(0x7f695006a800 msgr2=0x7f69500cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 shutdown_connections 2024-01-16T14:57:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.513+0000 7f6955600640 1 -- 172.21.15.125:0/1553580236 wait complete. 2024-01-16T14:57:48.513 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T14:57:48.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:48.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:48.615 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:48.615 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:57:48.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:48.615 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T14:57:48.666 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T14:57:48.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.902+0000 7f96bce00640 1 Processor -- start 2024-01-16T14:57:48.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.902+0000 7f96bce00640 1 -- start start 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96bce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96b80cf9a0 0x7f96b8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96bce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b815c5f0 con 0x7f96b813e360 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96bce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b806c470 con 0x7f96b813df80 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96bce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b806c750 con 0x7f96b80cf9a0 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96b6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96b80cf9a0 0x7f96b8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96b6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96b80cf9a0 0x7f96b8166030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55524/0 (socket says 172.21.15.125:55524) 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96b6800640 1 -- 172.21.15.125:0/3056599653 learned_addr learned my addr 172.21.15.125:0/3056599653 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.904+0000 7f96ad200640 1 --1- 172.21.15.125:0/3056599653 >> v1:172.21.15.125:6789/0 conn(0x7f96b813e360 0x7f96b80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48466/0 (socket says 172.21.15.125:48466) 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.905+0000 7f96b6800640 1 -- 172.21.15.125:0/3056599653 >> v1:172.21.15.137:6789/0 conn(0x7f96b813df80 legacy=0x7f96b815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.905+0000 7f96b6800640 1 -- 172.21.15.125:0/3056599653 >> v1:172.21.15.125:6789/0 conn(0x7f96b813e360 legacy=0x7f96b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.905+0000 7f96b6800640 1 -- 172.21.15.125:0/3056599653 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b806ca30 con 0x7f96b80cf9a0 2024-01-16T14:57:48.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.906+0000 7f96b6800640 1 --2- 172.21.15.125:0/3056599653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b8166030 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009870 tx=0x7f96b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef672b6aff34ce9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.906+0000 7f96b5400640 1 -- 172.21.15.125:0/3056599653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0030860 con 0x7f96b80cf9a0 2024-01-16T14:57:48.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.906+0000 7f96b5400640 1 -- 172.21.15.125:0/3056599653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b00309c0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.906+0000 7f96b5400640 1 -- 172.21.15.125:0/3056599653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0030d50 con 0x7f96b80cf9a0 2024-01-16T14:57:48.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 -- 172.21.15.125:0/3056599653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 msgr2=0x7f96b8166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 --2- 172.21.15.125:0/3056599653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b8166030 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009870 tx=0x7f96b002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 -- 172.21.15.125:0/3056599653 shutdown_connections 2024-01-16T14:57:48.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 --2- 172.21.15.125:0/3056599653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b8166030 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 -- 172.21.15.125:0/3056599653 >> 172.21.15.125:0/3056599653 conn(0x7f96b806a800 msgr2=0x7f96b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 -- 172.21.15.125:0/3056599653 shutdown_connections 2024-01-16T14:57:48.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.907+0000 7f96bce00640 1 -- 172.21.15.125:0/3056599653 wait complete. 2024-01-16T14:57:48.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.908+0000 7f96bce00640 1 Processor -- start 2024-01-16T14:57:48.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.909+0000 7f96bce00640 1 -- start start 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.909+0000 7f96bce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.909+0000 7f96bce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96b813df80 0x7f96b8159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96bce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96b813e360 0x7f96b81f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96bce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8156e30 con 0x7f96b80cf9a0 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96bce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8153700 con 0x7f96b813e360 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96bce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8157010 con 0x7f96b813df80 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55532/0 (socket says 172.21.15.125:55532) 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 -- 172.21.15.125:0/1170515461 learned_addr learned my addr 172.21.15.125:0/1170515461 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b6800640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96b813e360 0x7f96b81f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 -- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96b813df80 msgr2=0x7f96b8159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96b813df80 0x7f96b8159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 -- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96b813e360 msgr2=0x7f96b81f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96b813e360 0x7f96b81f0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.910+0000 7f96b5e00640 1 -- 172.21.15.125:0/1170515461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00094d0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96b5e00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f96a800b3e0 tx=0x7f96a800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8011830 con 0x7f96b80cf9a0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a80119b0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8011d60 con 0x7f96b80cf9a0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b81572f0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.911+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b81578b0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.912+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b80675d0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.913+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f96a801a400 con 0x7f96b80cf9a0 2024-01-16T14:57:48.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.914+0000 7f96aea00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 0x7f969006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.914+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f96a8089cf0 con 0x7f96b80cf9a0 2024-01-16T14:57:48.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.914+0000 7f96ad200640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 0x7f969006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.915+0000 7f96ad200640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 0x7f969006d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96b8154f10 tx=0x7f96a0007d60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:48.918+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96a808e070 con 0x7f96b80cf9a0 2024-01-16T14:57:49.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.149+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96b806df00 con 0x7f96b80cf9a0 2024-01-16T14:57:49.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.150+0000 7f96aea00640 1 -- 172.21.15.125:0/1170515461 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f96a8015020 con 0x7f96b80cf9a0 2024-01-16T14:57:49.132 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:49.132 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:49.132 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:49.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 msgr2=0x7f969006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 0x7f969006d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96b8154f10 tx=0x7f96a0007d60 comp rx=0 tx=0).stop 2024-01-16T14:57:49.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 msgr2=0x7f96b81531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f96a800b3e0 tx=0x7f96a800b8a0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 shutdown_connections 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96b813e360 0x7f96b81f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96b813df80 0x7f96b8159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f969006b2c0 0x7f969006d780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 --2- 172.21.15.125:0/1170515461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96b80cf9a0 0x7f96b81531c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 >> 172.21.15.125:0/1170515461 conn(0x7f96b806a800 msgr2=0x7f96b80afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 shutdown_connections 2024-01-16T14:57:49.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.153+0000 7f96bce00640 1 -- 172.21.15.125:0/1170515461 wait complete. 2024-01-16T14:57:49.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:49.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.379+0000 7fc184400640 1 Processor -- start 2024-01-16T14:57:49.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.379+0000 7fc184400640 1 -- start start 2024-01-16T14:57:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc184400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17c14b2c0 0x7fc17c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc184400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc17c0cf7d0 0x7fc17c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc184400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c0cfbf0 con 0x7fc17c070ea0 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc184400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c14b6a0 con 0x7fc17c14b2c0 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc184400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c071690 con 0x7fc17c0cf7d0 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc181a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17c14b2c0 0x7fc17c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc181a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17c14b2c0 0x7fc17c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55538/0 (socket says 172.21.15.125:55538) 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.381+0000 7fc181a00640 1 -- 172.21.15.125:0/2627119143 learned_addr learned my addr 172.21.15.125:0/2627119143 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.382+0000 7fc181a00640 1 -- 172.21.15.125:0/2627119143 >> v2:172.21.15.137:3300/0 conn(0x7fc17c0cf7d0 msgr2=0x7fc17c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.382+0000 7fc181a00640 1 --2- 172.21.15.125:0/2627119143 >> v2:172.21.15.137:3300/0 conn(0x7fc17c0cf7d0 0x7fc17c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.382+0000 7fc181a00640 1 -- 172.21.15.125:0/2627119143 >> v1:172.21.15.137:6789/0 conn(0x7fc17c070ea0 legacy=0x7fc17c0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.382+0000 7fc181a00640 1 -- 172.21.15.125:0/2627119143 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c071970 con 0x7fc17c14b2c0 2024-01-16T14:57:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.383+0000 7fc181a00640 1 --2- 172.21.15.125:0/2627119143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c14b2c0 0x7fc17c0cf290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0cfea0 tx=0x7fc16c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c3bfc315b971f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.383+0000 7fc17be00640 1 -- 172.21.15.125:0/2627119143 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031870 con 0x7fc17c14b2c0 2024-01-16T14:57:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.383+0000 7fc17be00640 1 -- 172.21.15.125:0/2627119143 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c0319d0 con 0x7fc17c14b2c0 2024-01-16T14:57:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.383+0000 7fc17be00640 1 -- 172.21.15.125:0/2627119143 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031d60 con 0x7fc17c14b2c0 2024-01-16T14:57:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 -- 172.21.15.125:0/2627119143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c14b2c0 msgr2=0x7fc17c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 --2- 172.21.15.125:0/2627119143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c14b2c0 0x7fc17c0cf290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0cfea0 tx=0x7fc16c02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 -- 172.21.15.125:0/2627119143 shutdown_connections 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 --2- 172.21.15.125:0/2627119143 >> v2:172.21.15.137:3300/0 conn(0x7fc17c0cf7d0 0x7fc17c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 --2- 172.21.15.125:0/2627119143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c14b2c0 0x7fc17c0cf290 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 -- 172.21.15.125:0/2627119143 >> 172.21.15.125:0/2627119143 conn(0x7fc17c06a800 msgr2=0x7fc17c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 -- 172.21.15.125:0/2627119143 shutdown_connections 2024-01-16T14:57:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.384+0000 7fc184400640 1 -- 172.21.15.125:0/2627119143 wait complete. 2024-01-16T14:57:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.386+0000 7fc184400640 1 Processor -- start 2024-01-16T14:57:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.386+0000 7fc184400640 1 -- start start 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc17c0cf7d0 0x7fc17c1437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc17c14b2c0 0x7fc17c146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c144300 con 0x7fc17c070ea0 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c144020 con 0x7fc17c0cf7d0 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc184400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c162ff0 con 0x7fc17c14b2c0 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55542/0 (socket says 172.21.15.125:55542) 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 -- 172.21.15.125:0/2581312543 learned_addr learned my addr 172.21.15.125:0/2581312543 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 -- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc17c14b2c0 msgr2=0x7fc17c146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181000640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc17c0cf7d0 0x7fc17c1437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc17c14b2c0 0x7fc17c146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 -- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc17c0cf7d0 msgr2=0x7fc17c1437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc17c0cf7d0 0x7fc17c1437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.388+0000 7fc181a00640 1 -- 172.21.15.125:0/2581312543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0094d0 con 0x7fc17c070ea0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc181a00640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc16c000c00 tx=0x7fc16c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c03ad10 con 0x7fc17c070ea0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c03ae70 con 0x7fc17c070ea0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c044610 con 0x7fc17c070ea0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c1632d0 con 0x7fc17c070ea0 2024-01-16T14:57:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.389+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c143d10 con 0x7fc17c070ea0 2024-01-16T14:57:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.390+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c1637f0 con 0x7fc17c070ea0 2024-01-16T14:57:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.391+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc16c04d450 con 0x7fc17c070ea0 2024-01-16T14:57:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.392+0000 7fc179600640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 0x7fc15406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.392+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc16c02d040 con 0x7fc17c070ea0 2024-01-16T14:57:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.392+0000 7fc181000640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 0x7fc15406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.393+0000 7fc181000640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 0x7fc15406d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0b0040 tx=0x7fc164008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.396+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc16c077100 con 0x7fc17c070ea0 2024-01-16T14:57:49.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.588+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc17c0675d0 con 0x7fc17c070ea0 2024-01-16T14:57:49.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.591+0000 7fc179600640 1 -- 172.21.15.125:0/2581312543 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7fc16c07a720 con 0x7fc17c070ea0 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.593+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 msgr2=0x7fc15406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.593+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 0x7fc15406d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0b0040 tx=0x7fc164008040 comp rx=0 tx=0).stop 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.593+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 msgr2=0x7fc17c144d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.593+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc16c000c00 tx=0x7fc16c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.593+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 shutdown_connections 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc17c14b2c0 0x7fc17c146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc17c0cf7d0 0x7fc17c1437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc15406b2c0 0x7fc15406d780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 --2- 172.21.15.125:0/2581312543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17c070ea0 0x7fc17c144d10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 >> 172.21.15.125:0/2581312543 conn(0x7fc17c06a800 msgr2=0x7fc17c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 shutdown_connections 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.594+0000 7fc184400640 1 -- 172.21.15.125:0/2581312543 wait complete. 2024-01-16T14:57:49.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:56:43.384842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:56:44.446464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:49.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:49.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.820+0000 7efe3f200640 1 Processor -- start 2024-01-16T14:57:49.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.820+0000 7efe3f200640 1 -- start start 2024-01-16T14:57:49.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3f200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe380cfc30 con 0x7efe3815e3b0 2024-01-16T14:57:49.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3f200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe380b1ee0 con 0x7efe380cf7d0 2024-01-16T14:57:49.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe380b2780 con 0x7efe38143ae0 2024-01-16T14:57:49.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3d200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efe3815e3b0 0x7efe380702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48470/0 (socket says 172.21.15.125:48470) 2024-01-16T14:57:49.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.822+0000 7efe3d200640 1 -- 172.21.15.125:0/1745380592 learned_addr learned my addr 172.21.15.125:0/1745380592 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe37e00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894218821 0 0) 0x7efe380cfc30 con 0x7efe3815e3b0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe18005730 con 0x7efe3815e3b0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305509871 0 0) 0x7efe18005730 con 0x7efe3815e3b0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 msgr2=0x7efe380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 >> v1:172.21.15.137:6789/0 conn(0x7efe380cf7d0 legacy=0x7efe3815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3815e990 con 0x7efe3815e3b0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7efe300019f0 con 0x7efe3815e3b0 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 >> v1:172.21.15.125:6789/0 conn(0x7efe3815e3b0 legacy=0x7efe380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.823+0000 7efe36a00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe18005bb0 0x7efe18005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe36a00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe180069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe36a00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18006f10 0x7efe18007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe380b2780 con 0x7efe18005bb0 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1800dc40 con 0x7efe18006f10 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1800de00 con 0x7efe38143ae0 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe18005bb0 0x7efe18005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37400640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe180069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe3d200640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18006f10 0x7efe18007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 -- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 msgr2=0x7efe180069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe38143ae0 0x7efe180069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 -- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18006f10 msgr2=0x7efe18007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18006f10 0x7efe18007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.824+0000 7efe37e00640 1 -- 172.21.15.125:0/1745380592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18005730 con 0x7efe18005bb0 2024-01-16T14:57:49.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.825+0000 7efe37e00640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe18005bb0 0x7efe18005fd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efe28009a90 tx=0x7efe18007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f725236ef1e1dabb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2802c680 con 0x7efe18005bb0 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2802c7e0 con 0x7efe18005bb0 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe36a00640 1 -- 172.21.15.125:0/1745380592 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2802cb70 con 0x7efe18005bb0 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3f200640 1 -- 172.21.15.125:0/1745380592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe18005bb0 msgr2=0x7efe18005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3f200640 1 --2- 172.21.15.125:0/1745380592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe18005bb0 0x7efe18005fd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efe28009a90 tx=0x7efe18007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3d200640 1 -- 172.21.15.125:0/1745380592 reap_dead start 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3f200640 1 -- 172.21.15.125:0/1745380592 shutdown_connections 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3f200640 1 -- 172.21.15.125:0/1745380592 >> 172.21.15.125:0/1745380592 conn(0x7efe3806a800 msgr2=0x7efe3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.826+0000 7efe3f200640 1 -- 172.21.15.125:0/1745380592 shutdown_connections 2024-01-16T14:57:49.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.827+0000 7efe3f200640 1 -- 172.21.15.125:0/1745380592 wait complete. 2024-01-16T14:57:49.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.829+0000 7efe3f200640 1 Processor -- start 2024-01-16T14:57:49.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.829+0000 7efe3f200640 1 -- start start 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 0x7efe380aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe380b0020 0x7efe380b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38159ed0 con 0x7efe3813fd80 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe380b09e0 con 0x7efe18005bb0 2024-01-16T14:57:49.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe3f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe380afc80 con 0x7efe380b0020 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe37400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe37400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55556/0 (socket says 172.21.15.125:55556) 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.830+0000 7efe37400640 1 -- 172.21.15.125:0/2656518883 learned_addr learned my addr 172.21.15.125:0/2656518883 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37e00640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 0x7efe380aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe3d200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe380b0020 0x7efe380b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 -- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe380b0020 msgr2=0x7efe380b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe380b0020 0x7efe380b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 -- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 msgr2=0x7efe380aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 0x7efe380aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 -- 172.21.15.125:0/2656518883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe380af070 con 0x7efe3813fd80 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37e00640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 0x7efe380aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:49.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe37400640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d5a0 tx=0x7efe2000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20005480 con 0x7efe3813fd80 2024-01-16T14:57:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe20003610 con 0x7efe3813fd80 2024-01-16T14:57:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20003980 con 0x7efe3813fd80 2024-01-16T14:57:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.831+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe380af350 con 0x7efe3813fd80 2024-01-16T14:57:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.832+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe380af910 con 0x7efe3813fd80 2024-01-16T14:57:49.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.833+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe380675d0 con 0x7efe3813fd80 2024-01-16T14:57:49.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.834+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efe20003ae0 con 0x7efe3813fd80 2024-01-16T14:57:49.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.835+0000 7efe2fe00640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 0x7efe1006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.835+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7efe20089b70 con 0x7efe3813fd80 2024-01-16T14:57:49.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.835+0000 7efe37e00640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 0x7efe1006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.836+0000 7efe37e00640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 0x7efe1006d7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe28036000 tx=0x7efe28030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:49.839+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe20054940 con 0x7efe3813fd80 2024-01-16T14:57:50.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.033+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe38141010 con 0x7efe3813fd80 2024-01-16T14:57:50.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.034+0000 7efe2fe00640 1 -- 172.21.15.125:0/2656518883 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe20001230 con 0x7efe3813fd80 2024-01-16T14:57:50.015 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 msgr2=0x7efe1006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 0x7efe1006d7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe28036000 tx=0x7efe28030000 comp rx=0 tx=0).stop 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 msgr2=0x7efe38159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d5a0 tx=0x7efe2000da60 comp rx=0 tx=0).stop 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 shutdown_connections 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe380b0020 0x7efe380b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe1006b2f0 0x7efe1006d7b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3813fd80 0x7efe38159990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 --2- 172.21.15.125:0/2656518883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe18005bb0 0x7efe380aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.036+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 >> 172.21.15.125:0/2656518883 conn(0x7efe3806a800 msgr2=0x7efe3815bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.037+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 shutdown_connections 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.037+0000 7efe3f200640 1 -- 172.21.15.125:0/2656518883 wait complete. 2024-01-16T14:57:50.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:50.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.259+0000 7fa7be200640 1 Processor -- start 2024-01-16T14:57:50.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.259+0000 7fa7be200640 1 -- start start 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7be200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa7b8143ae0 0x7fa7b80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7be200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 0x7fa7b815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7be200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80cfc30 con 0x7fa7b815e1e0 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7be200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80b1ee0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7be200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80b2780 con 0x7fa7b80cf7d0 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7b7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa7b8143ae0 0x7fa7b80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7b7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa7b815e1e0 0x7fa7b815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48472/0 (socket says 172.21.15.125:48472) 2024-01-16T14:57:50.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7b7e00640 1 -- 172.21.15.125:0/2624436164 learned_addr learned my addr 172.21.15.125:0/2624436164 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.261+0000 7fa7b6a00640 1 --2- 172.21.15.125:0/2624436164 >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 0x7fa7b815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.262+0000 7fa7b7400640 1 -- 172.21.15.125:0/2624436164 >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 msgr2=0x7fa7b815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.262+0000 7fa7b7400640 1 --2- 172.21.15.125:0/2624436164 >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 0x7fa7b815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.262+0000 7fa7b7400640 1 -- 172.21.15.125:0/2624436164 >> v1:172.21.15.125:6789/0 conn(0x7fa7b815e1e0 legacy=0x7fa7b815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.262+0000 7fa7b7400640 1 -- 172.21.15.125:0/2624436164 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b815e990 con 0x7fa7b8143ae0 2024-01-16T14:57:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.262+0000 7fa7b6a00640 1 --2- 172.21.15.125:0/2624436164 >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 0x7fa7b815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.263+0000 7fa7b7400640 1 --2- 172.21.15.125:0/2624436164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b80cf290 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80cb830 tx=0x7fa7a802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4235e2ea36b2a027 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.263+0000 7fa7b6000640 1 -- 172.21.15.125:0/2624436164 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8030710 con 0x7fa7b8143ae0 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7b6000640 1 -- 172.21.15.125:0/2624436164 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a802e050 con 0x7fa7b8143ae0 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7b6000640 1 -- 172.21.15.125:0/2624436164 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8030a90 con 0x7fa7b8143ae0 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 -- 172.21.15.125:0/2624436164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 msgr2=0x7fa7b80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 --2- 172.21.15.125:0/2624436164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b80cf290 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80cb830 tx=0x7fa7a802b710 comp rx=0 tx=0).stop 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 -- 172.21.15.125:0/2624436164 shutdown_connections 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 --2- 172.21.15.125:0/2624436164 >> v2:172.21.15.137:3300/0 conn(0x7fa7b80cf7d0 0x7fa7b815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 --2- 172.21.15.125:0/2624436164 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b80cf290 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 -- 172.21.15.125:0/2624436164 >> 172.21.15.125:0/2624436164 conn(0x7fa7b806a800 msgr2=0x7fa7b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 -- 172.21.15.125:0/2624436164 shutdown_connections 2024-01-16T14:57:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.264+0000 7fa7be200640 1 -- 172.21.15.125:0/2624436164 wait complete. 2024-01-16T14:57:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.266+0000 7fa7be200640 1 Processor -- start 2024-01-16T14:57:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.266+0000 7fa7be200640 1 -- start start 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7b80cf7d0 0x7fa7b8151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7b815e1e0 0x7fa7b815c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b813a3b0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81f7090 con 0x7fa7b815e1e0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7be200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8152480 con 0x7fa7b80cf7d0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7b6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7b6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55574/0 (socket says 172.21.15.125:55574) 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.267+0000 7fa7b6a00640 1 -- 172.21.15.125:0/1562942317 learned_addr learned my addr 172.21.15.125:0/1562942317 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b7e00640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7b815e1e0 0x7fa7b815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b7400640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7b80cf7d0 0x7fa7b8151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 -- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7b80cf7d0 msgr2=0x7fa7b8151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7b80cf7d0 0x7fa7b8151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 -- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7b815e1e0 msgr2=0x7fa7b815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7b815e1e0 0x7fa7b815c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 -- 172.21.15.125:0/1562942317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8009400 con 0x7fa7b8143ae0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7b6a00640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d7d0 tx=0x7fa7a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0004030 con 0x7fa7b8143ae0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0004190 con 0x7fa7b8143ae0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b81f62d0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.268+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0004520 con 0x7fa7b8143ae0 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.269+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b81f6890 con 0x7fa7b8143ae0 2024-01-16T14:57:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.269+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b80b2bf0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.270+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa7a00096a0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.271+0000 7fa7af400640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 0x7fa79006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.271+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa7a0089eb0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.271+0000 7fa7b7400640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 0x7fa79006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.272+0000 7fa7b7400640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 0x7fa79006d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8034000 tx=0x7fa7a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.275+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7b80b2bf0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.453+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa7b80b2bf0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.454+0000 7fa7af400640 1 -- 172.21.15.125:0/1562942317 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7b80b2bf0 con 0x7fa7b8143ae0 2024-01-16T14:57:50.435 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 msgr2=0x7fa79006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 0x7fa79006d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8034000 tx=0x7fa7a8008030 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 msgr2=0x7fa7b8139a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d7d0 tx=0x7fa7a000dc90 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 shutdown_connections 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7b815e1e0 0x7fa7b815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa79006b2c0 0x7fa79006d780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7b8143ae0 0x7fa7b8139a60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 --2- 172.21.15.125:0/1562942317 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7b80cf7d0 0x7fa7b8151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 >> 172.21.15.125:0/1562942317 conn(0x7fa7b806a800 msgr2=0x7fa7b815ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 shutdown_connections 2024-01-16T14:57:50.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.456+0000 7fa7be200640 1 -- 172.21.15.125:0/1562942317 wait complete. 2024-01-16T14:57:50.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:50.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.675+0000 7f87c6c00640 1 Processor -- start 2024-01-16T14:57:50.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.675+0000 7f87c6c00640 1 -- start start 2024-01-16T14:57:50.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.676+0000 7f87c6c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f87c00cf9a0 0x7f87c015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.676+0000 7f87c6c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 0x7f87c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.676+0000 7f87c6c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c00b1ee0 con 0x7f87c0143ae0 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.676+0000 7f87c6c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c00b2780 con 0x7f87c00cf9a0 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.676+0000 7f87c6c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015e990 con 0x7f87c015e1e0 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f87c0143ae0 0x7f87c00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48486/0 (socket says 172.21.15.125:48486) 2024-01-16T14:57:50.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bfe00640 1 -- 172.21.15.125:0/1189514395 learned_addr learned my addr 172.21.15.125:0/1189514395 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585233994 0 0) 0x7f87c00b1ee0 con 0x7f87c0143ae0 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87c4c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 0x7f87c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87b7e00640 1 --2- 172.21.15.125:0/1189514395 >> v2:172.21.15.137:3300/0 conn(0x7f87c00cf9a0 0x7f87c015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a0003520 con 0x7f87c0143ae0 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3111105011 0 0) 0x7f87a0003520 con 0x7f87c0143ae0 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 msgr2=0x7f87c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 0x7f87c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 >> v2:172.21.15.137:3300/0 conn(0x7f87c00cf9a0 msgr2=0x7f87c015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 --2- 172.21.15.125:0/1189514395 >> v2:172.21.15.137:3300/0 conn(0x7f87c00cf9a0 0x7f87c015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00b2130 con 0x7f87c0143ae0 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.677+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f87b0001670 con 0x7f87c0143ae0 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 >> v1:172.21.15.125:6789/0 conn(0x7f87c0143ae0 legacy=0x7f87c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 0x7f87a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a0003f40 0x7f87a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00b2780 con 0x7f87a0004d00 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015e990 con 0x7f87a0003f40 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a000b980 con 0x7f87c015e1e0 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 -- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 msgr2=0x7f87a0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c015e1e0 0x7f87a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 -- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a0003f40 msgr2=0x7f87a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a0003f40 0x7f87a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.678+0000 7f87c4c00640 1 -- 172.21.15.125:0/1189514395 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0003520 con 0x7f87a0004d00 2024-01-16T14:57:50.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.679+0000 7f87c4c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87a0005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e170 tx=0x7f87a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593c33cb19e93bcf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.679+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b802cc20 con 0x7f87a0004d00 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b800f040 con 0x7f87a0004d00 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87bf400640 1 -- 172.21.15.125:0/1189514395 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8034440 con 0x7f87a0004d00 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 -- 172.21.15.125:0/1189514395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 msgr2=0x7f87a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 --2- 172.21.15.125:0/1189514395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87a0005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e170 tx=0x7f87a00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c4c00640 1 -- 172.21.15.125:0/1189514395 reap_dead start 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 -- 172.21.15.125:0/1189514395 shutdown_connections 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 -- 172.21.15.125:0/1189514395 >> 172.21.15.125:0/1189514395 conn(0x7f87c006a800 msgr2=0x7f87c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 -- 172.21.15.125:0/1189514395 shutdown_connections 2024-01-16T14:57:50.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.680+0000 7f87c6c00640 1 -- 172.21.15.125:0/1189514395 wait complete. 2024-01-16T14:57:50.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.682+0000 7f87c6c00640 1 Processor -- start 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.682+0000 7f87c6c00640 1 -- start start 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 0x7f87c00c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 0x7f87c0070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c0070c80 con 0x7f87a0004d00 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00ad120 con 0x7f87c00702c0 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87c6c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00ad400 con 0x7f87c00c4860 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87bfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87bfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55598/0 (socket says 172.21.15.125:55598) 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.683+0000 7f87bfe00640 1 -- 172.21.15.125:0/215376346 learned_addr learned my addr 172.21.15.125:0/215376346 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87b7e00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 0x7f87c00c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87c4c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 0x7f87c0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 -- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 msgr2=0x7f87c00c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 0x7f87c00c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 -- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 msgr2=0x7f87c0070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 0x7f87c0070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 -- 172.21.15.125:0/215376346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00ad6e0 con 0x7f87a0004d00 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87b7e00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 0x7f87c00c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87c4c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 0x7f87c0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bfe00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f87b000c7a0 tx=0x7f87b000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00074e0 con 0x7f87a0004d00 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b0007640 con 0x7f87a0004d00 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0004a00 con 0x7f87a0004d00 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c00ad9c0 con 0x7f87a0004d00 2024-01-16T14:57:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.684+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c00c5610 con 0x7f87a0004d00 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.685+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c00adee0 con 0x7f87a0004d00 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.686+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f87b0004b60 con 0x7f87a0004d00 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.687+0000 7f87bcc00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 0x7f879806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.687+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f87b0089050 con 0x7f87a0004d00 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.687+0000 7f87b7e00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 0x7f879806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.688+0000 7f87b7e00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 0x7f879806d710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87c00ad010 tx=0x7f87a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.692+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87c00adee0 con 0x7f87a0004d00 2024-01-16T14:57:50.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.866+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87c00adee0 con 0x7f87a0004d00 2024-01-16T14:57:50.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.868+0000 7f87bcc00640 1 -- 172.21.15.125:0/215376346 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87c00adee0 con 0x7f87a0004d00 2024-01-16T14:57:50.849 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 msgr2=0x7f879806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 0x7f879806d710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87c00ad010 tx=0x7f87a8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 msgr2=0x7f87c0141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f87b000c7a0 tx=0x7f87b000cc60 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 shutdown_connections 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87c00702c0 0x7f87c0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87c00c4860 0x7f87c00c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f879806b250 0x7f879806d710 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 --2- 172.21.15.125:0/215376346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a0004d00 0x7f87c0141f80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 >> 172.21.15.125:0/215376346 conn(0x7f87c006a800 msgr2=0x7f87c00ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 shutdown_connections 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:50.870+0000 7f87c6c00640 1 -- 172.21.15.125:0/215376346 wait complete. 2024-01-16T14:57:50.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.086+0000 7f8ad7400640 1 Processor -- start 2024-01-16T14:57:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.086+0000 7f8ad7400640 1 -- start start 2024-01-16T14:57:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ad00cf9a0 0x7f8ad0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad7400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad0071690 con 0x7f8ad014b2c0 2024-01-16T14:57:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad7400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad0071970 con 0x7f8ad0070ea0 2024-01-16T14:57:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad014b6a0 con 0x7f8ad00cf9a0 2024-01-16T14:57:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad4a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8ad014b2c0 0x7f8ad00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48488/0 (socket says 172.21.15.125:48488) 2024-01-16T14:57:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.088+0000 7f8ad4a00640 1 -- 172.21.15.125:0/2620215855 learned_addr learned my addr 172.21.15.125:0/2620215855 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8ac7c00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ad00cf9a0 0x7f8ad0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460643371 0 0) 0x7f8ad0071690 con 0x7f8ad014b2c0 2024-01-16T14:57:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0003520 con 0x7f8ad014b2c0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3785639873 0 0) 0x7f8ab0003520 con 0x7f8ad014b2c0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ad00cf9a0 msgr2=0x7f8ad0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ad00cf9a0 0x7f8ad0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 >> v1:172.21.15.137:6789/0 conn(0x7f8ad0070ea0 legacy=0x7f8ad00cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0071c50 con 0x7f8ad014b2c0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8ac0001670 con 0x7f8ad014b2c0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 >> v1:172.21.15.125:6789/0 conn(0x7f8ad014b2c0 legacy=0x7f8ad00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab00039a0 0x7f8ab0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ab00047c0 0x7f8ab0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab0005180 0x7f8ab000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad014b6a0 con 0x7f8ab0005180 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab000c050 con 0x7f8ab00047c0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab000c210 con 0x7f8ab00039a0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.089+0000 7f8ac7c00640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ad00cf9a0 0x7f8ad0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab0005180 0x7f8ab000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 -- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab00039a0 msgr2=0x7f8ab0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab00039a0 0x7f8ab0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 -- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ab00047c0 msgr2=0x7f8ab0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ab00047c0 0x7f8ab0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.090+0000 7f8ad5400640 1 -- 172.21.15.125:0/2620215855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab0003520 con 0x7f8ab0005180 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.091+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab0005180 0x7f8ab000b980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001e50 tx=0x7f8ab00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa3e1bda1e70139 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.091+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8003b70 con 0x7f8ab0005180 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.091+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8003cd0 con 0x7f8ab0005180 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.091+0000 7f8acfe00640 1 -- 172.21.15.125:0/2620215855 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8005310 con 0x7f8ab0005180 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 -- 172.21.15.125:0/2620215855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab0005180 msgr2=0x7f8ab000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2620215855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab0005180 0x7f8ab000b980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001e50 tx=0x7f8ab00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad5400640 1 -- 172.21.15.125:0/2620215855 reap_dead start 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 -- 172.21.15.125:0/2620215855 shutdown_connections 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 -- 172.21.15.125:0/2620215855 >> 172.21.15.125:0/2620215855 conn(0x7f8ad006a800 msgr2=0x7f8ad006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 -- 172.21.15.125:0/2620215855 shutdown_connections 2024-01-16T14:57:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.092+0000 7f8ad7400640 1 -- 172.21.15.125:0/2620215855 wait complete. 2024-01-16T14:57:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.094+0000 7f8ad7400640 1 Processor -- start 2024-01-16T14:57:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.094+0000 7f8ad7400640 1 -- start start 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab0005180 0x7f8ad01638b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ad0163df0 0x7f8ad01f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad01f8080 con 0x7f8ad01f7c00 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0161e90 con 0x7f8ad0163df0 2024-01-16T14:57:51.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0162170 con 0x7f8ab0005180 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.095+0000 7f8ad5400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55628/0 (socket says 172.21.15.125:55628) 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 -- 172.21.15.125:0/2251791681 learned_addr learned my addr 172.21.15.125:0/2251791681 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad4a00640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab0005180 0x7f8ad01638b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ac7c00640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ad0163df0 0x7f8ad01f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 -- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab0005180 msgr2=0x7f8ad01638b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab0005180 0x7f8ad01638b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 -- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ad0163df0 msgr2=0x7f8ad01f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ad0163df0 0x7f8ad01f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 -- 172.21.15.125:0/2251791681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0162450 con 0x7f8ad01f7c00 2024-01-16T14:57:51.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.096+0000 7f8ad5400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001f70 tx=0x7f8ac80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.097+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac803e070 con 0x7f8ad01f7c00 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.097+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8003e40 con 0x7f8ad01f7c00 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.097+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad0162730 con 0x7f8ad01f7c00 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.097+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8006d50 con 0x7f8ad01f7c00 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.097+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad0146b80 con 0x7f8ad01f7c00 2024-01-16T14:57:51.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.098+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad0162c50 con 0x7f8ad01f7c00 2024-01-16T14:57:51.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.099+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8ac8007030 con 0x7f8ad01f7c00 2024-01-16T14:57:51.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.100+0000 7f8acd600640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 0x7f8aa806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.100+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8ac80aae90 con 0x7f8ad01f7c00 2024-01-16T14:57:51.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.100+0000 7f8ad4a00640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 0x7f8aa806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.101+0000 7f8ad4a00640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 0x7f8aa806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00017a0 tx=0x7f8ac0001670 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.104+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ad0162c50 con 0x7f8ad01f7c00 2024-01-16T14:57:51.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.335+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ad0162c50 con 0x7f8ad01f7c00 2024-01-16T14:57:51.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.336+0000 7f8acd600640 1 -- 172.21.15.125:0/2251791681 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f8ad0162c50 con 0x7f8ad01f7c00 2024-01-16T14:57:51.317 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:51.318 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:51.318 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:51.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.338+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 msgr2=0x7f8aa806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.338+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 0x7f8aa806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00017a0 tx=0x7f8ac0001670 comp rx=0 tx=0).stop 2024-01-16T14:57:51.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 msgr2=0x7f8ad0146640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001f70 tx=0x7f8ac80014a0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 shutdown_connections 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8aa806b2c0 0x7f8aa806d780 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01f7c00 0x7f8ad0146640 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ad0163df0 0x7f8ad01f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 --2- 172.21.15.125:0/2251791681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ab0005180 0x7f8ad01638b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 >> 172.21.15.125:0/2251791681 conn(0x7f8ad006a800 msgr2=0x7f8ad00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 shutdown_connections 2024-01-16T14:57:51.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.339+0000 7f8ad7400640 1 -- 172.21.15.125:0/2251791681 wait complete. 2024-01-16T14:57:51.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.571+0000 7f5d11400640 1 Processor -- start 2024-01-16T14:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.571+0000 7f5d11400640 1 -- start start 2024-01-16T14:57:51.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.572+0000 7f5d11400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 0x7f5d0c06be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d11400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d11400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0cfb80 con 0x7f5d0c06c3b0 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d11400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c06c820 con 0x7f5d0c0cf730 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d11400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c0cfe30 con 0x7f5d0c15d560 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55630/0 (socket says 172.21.15.125:55630) 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 -- 172.21.15.125:0/3372356083 learned_addr learned my addr 172.21.15.125:0/3372356083 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d0be00640 1 --2- 172.21.15.125:0/3372356083 >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 0x7f5d0c06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 -- 172.21.15.125:0/3372356083 >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 msgr2=0x7f5d0c06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 --2- 172.21.15.125:0/3372356083 >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 0x7f5d0c06be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 -- 172.21.15.125:0/3372356083 >> v1:172.21.15.125:6789/0 conn(0x7f5d0c06c3b0 legacy=0x7f5d0c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d10a00640 1 -- 172.21.15.125:0/3372356083 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0d0110 con 0x7f5d0c0cf730 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.573+0000 7f5d0be00640 1 --2- 172.21.15.125:0/3372356083 >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 0x7f5d0c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d10a00640 1 --2- 172.21.15.125:0/3372356083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c070cb0 tx=0x7f5d0402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d5b7b5cacb00cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d0b400640 1 -- 172.21.15.125:0/3372356083 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0402cb10 con 0x7f5d0c0cf730 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d0b400640 1 -- 172.21.15.125:0/3372356083 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0402cc70 con 0x7f5d0c0cf730 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d0b400640 1 -- 172.21.15.125:0/3372356083 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040343f0 con 0x7f5d0c0cf730 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d11400640 1 -- 172.21.15.125:0/3372356083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c0cf730 msgr2=0x7f5d0c0b06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.575+0000 7f5d11400640 1 --2- 172.21.15.125:0/3372356083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c070cb0 tx=0x7f5d0402bf70 comp rx=0 tx=0).stop 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 -- 172.21.15.125:0/3372356083 shutdown_connections 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 --2- 172.21.15.125:0/3372356083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b06c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 --2- 172.21.15.125:0/3372356083 >> v2:172.21.15.137:3300/0 conn(0x7f5d0c15d560 0x7f5d0c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 -- 172.21.15.125:0/3372356083 >> 172.21.15.125:0/3372356083 conn(0x7f5d0c099a60 msgr2=0x7f5d0c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 -- 172.21.15.125:0/3372356083 shutdown_connections 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.576+0000 7f5d11400640 1 -- 172.21.15.125:0/3372356083 wait complete. 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.577+0000 7f5d11400640 1 Processor -- start 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.577+0000 7f5d11400640 1 -- start start 2024-01-16T14:57:51.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d0c06c3b0 0x7f5d0c14bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c1608a0 con 0x7f5d0c15d560 2024-01-16T14:57:51.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15fd20 con 0x7f5d0c0cf730 2024-01-16T14:57:51.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.578+0000 7f5d11400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c160000 con 0x7f5d0c06c3b0 2024-01-16T14:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55642/0 (socket says 172.21.15.125:55642) 2024-01-16T14:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 -- 172.21.15.125:0/1254842028 learned_addr learned my addr 172.21.15.125:0/1254842028 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d0be00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d0c06c3b0 0x7f5d0c14bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d03e00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 -- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d0c06c3b0 msgr2=0x7f5d0c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d0c06c3b0 0x7f5d0c14bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 -- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d0c0cf730 msgr2=0x7f5d0c0b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 -- 172.21.15.125:0/1254842028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04009b30 con 0x7f5d0c15d560 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.579+0000 7f5d10a00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5d04036040 tx=0x7f5d04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.580+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04009510 con 0x7f5d0c15d560 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.580+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0403e450 con 0x7f5d0c15d560 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.580+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0403e800 con 0x7f5d0c15d560 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.580+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c1602e0 con 0x7f5d0c15d560 2024-01-16T14:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.580+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c15fa20 con 0x7f5d0c15d560 2024-01-16T14:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.581+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c067b60 con 0x7f5d0c15d560 2024-01-16T14:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.582+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5d0403e960 con 0x7f5d0c15d560 2024-01-16T14:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.583+0000 7f5d08c00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 0x7f5ce406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.583+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5d040acbc0 con 0x7f5d0c15d560 2024-01-16T14:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.583+0000 7f5d0be00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 0x7f5ce406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.584+0000 7f5d0be00640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 0x7f5ce406d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00a950 tx=0x7f5cfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.587+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d040779b0 con 0x7f5d0c15d560 2024-01-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.780+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d0c0b6b30 con 0x7f5d0c15d560 2024-01-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.783+0000 7f5d08c00640 1 -- 172.21.15.125:0/1254842028 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f5d0407afd0 con 0x7f5d0c15d560 2024-01-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.785+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 msgr2=0x7f5ce406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.785+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 0x7f5ce406d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00a950 tx=0x7f5cfc008040 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.785+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 msgr2=0x7f5d0c147dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.785+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5d04036040 tx=0x7f5d04004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 shutdown_connections 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ce406b2c0 0x7f5ce406d780 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d0c15d560 0x7f5d0c147dc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d0c0cf730 0x7f5d0c0b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 --2- 172.21.15.125:0/1254842028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d0c06c3b0 0x7f5d0c14bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 >> 172.21.15.125:0/1254842028 conn(0x7f5d0c099a60 msgr2=0x7f5d0c06f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 shutdown_connections 2024-01-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:51.786+0000 7f5d11400640 1 -- 172.21.15.125:0/1254842028 wait complete. 2024-01-16T14:57:51.787 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T14:57:51.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:51.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.013+0000 7f5a6d400640 1 Processor -- start 2024-01-16T14:57:51.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.013+0000 7f5a6d400640 1 -- start start 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a6d400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a680ae5a0 0x7f5a6815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a6d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 0x7f5a680cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a6d400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a68070c40 con 0x7f5a680b3ae0 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a6d400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a68071450 con 0x7f5a680ae5a0 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a6d400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680cfc60 con 0x7f5a680cf290 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a66800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a680ae5a0 0x7f5a6815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a66800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a680ae5a0 0x7f5a6815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55656/0 (socket says 172.21.15.125:55656) 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a66800640 1 -- 172.21.15.125:0/3606415970 learned_addr learned my addr 172.21.15.125:0/3606415970 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a5fe00640 1 --1- 172.21.15.125:0/3606415970 >> v1:172.21.15.125:6789/0 conn(0x7f5a680b3ae0 0x7f5a6806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48504/0 (socket says 172.21.15.125:48504) 2024-01-16T14:57:51.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.015+0000 7f5a67200640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 0x7f5a680cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.016+0000 7f5a66800640 1 -- 172.21.15.125:0/3606415970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 msgr2=0x7f5a680cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.016+0000 7f5a66800640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 0x7f5a680cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.016+0000 7f5a66800640 1 -- 172.21.15.125:0/3606415970 >> v1:172.21.15.125:6789/0 conn(0x7f5a680b3ae0 legacy=0x7f5a6806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.016+0000 7f5a66800640 1 -- 172.21.15.125:0/3606415970 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6806c470 con 0x7f5a680ae5a0 2024-01-16T14:57:51.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.016+0000 7f5a67200640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 0x7f5a680cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:51.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.017+0000 7f5a66800640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680ae5a0 0x7f5a6815e430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009870 tx=0x7f5a5802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c554f0a901b766c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.017+0000 7f5a65e00640 1 -- 172.21.15.125:0/3606415970 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5802f830 con 0x7f5a680ae5a0 2024-01-16T14:57:51.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.017+0000 7f5a65e00640 1 -- 172.21.15.125:0/3606415970 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5802f990 con 0x7f5a680ae5a0 2024-01-16T14:57:51.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.017+0000 7f5a65e00640 1 -- 172.21.15.125:0/3606415970 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5802fd20 con 0x7f5a680ae5a0 2024-01-16T14:57:51.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 -- 172.21.15.125:0/3606415970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680ae5a0 msgr2=0x7f5a6815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680ae5a0 0x7f5a6815e430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009870 tx=0x7f5a5802ba10 comp rx=0 tx=0).stop 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 -- 172.21.15.125:0/3606415970 shutdown_connections 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680cf290 0x7f5a680cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3606415970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680ae5a0 0x7f5a6815e430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 -- 172.21.15.125:0/3606415970 >> 172.21.15.125:0/3606415970 conn(0x7f5a6806a800 msgr2=0x7f5a6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 -- 172.21.15.125:0/3606415970 shutdown_connections 2024-01-16T14:57:52.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.018+0000 7f5a6d400640 1 -- 172.21.15.125:0/3606415970 wait complete. 2024-01-16T14:57:52.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.020+0000 7f5a6d400640 1 Processor -- start 2024-01-16T14:57:52.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.020+0000 7f5a6d400640 1 -- start start 2024-01-16T14:57:52.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 0x7f5a681518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680b3ae0 0x7f5a68159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68153750 con 0x7f5a680cf290 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68153470 con 0x7f5a680ae5a0 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a6d400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6815ebf0 con 0x7f5a680b3ae0 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a67200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a67200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55666/0 (socket says 172.21.15.125:55666) 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a67200640 1 -- 172.21.15.125:0/3012251220 learned_addr learned my addr 172.21.15.125:0/3012251220 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:52.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.021+0000 7f5a66800640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 0x7f5a681518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 -- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680b3ae0 msgr2=0x7f5a68159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a5fe00640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680b3ae0 0x7f5a68159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680b3ae0 0x7f5a68159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 -- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 msgr2=0x7f5a681518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 0x7f5a681518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 -- 172.21.15.125:0/3012251220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580094d0 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a66800640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 0x7f5a681518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a67200640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cd50 tx=0x7f5a6000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000f680 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000f800 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000fbb0 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.022+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6815eed0 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.023+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6814fd50 con 0x7f5a680cf290 2024-01-16T14:57:52.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.024+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6815f3f0 con 0x7f5a680cf290 2024-01-16T14:57:52.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.025+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a6000b040 con 0x7f5a680cf290 2024-01-16T14:57:52.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.025+0000 7f5a5ea00640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 0x7f5a4006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.026+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5a6008a570 con 0x7f5a680cf290 2024-01-16T14:57:52.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.026+0000 7f5a66800640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 0x7f5a4006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.026+0000 7f5a66800640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 0x7f5a4006d780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a58036000 tx=0x7f5a58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.030+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a6815f3f0 con 0x7f5a680cf290 2024-01-16T14:57:52.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.262+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6815f3f0 con 0x7f5a680cf290 2024-01-16T14:57:52.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.263+0000 7f5a5ea00640 1 -- 172.21.15.125:0/3012251220 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f5a6815f3f0 con 0x7f5a680cf290 2024-01-16T14:57:52.244 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:52.244 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:52.244 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:52.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 msgr2=0x7f5a4006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 0x7f5a4006d780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a58036000 tx=0x7f5a58008030 comp rx=0 tx=0).stop 2024-01-16T14:57:52.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 msgr2=0x7f5a6814f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cd50 tx=0x7f5a6000a650 comp rx=0 tx=0).stop 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 shutdown_connections 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a4006b2c0 0x7f5a4006d780 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a680cf290 0x7f5a6814f810 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a680b3ae0 0x7f5a68159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 --2- 172.21.15.125:0/3012251220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a680ae5a0 0x7f5a681518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 >> 172.21.15.125:0/3012251220 conn(0x7f5a6806a800 msgr2=0x7f5a6815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 shutdown_connections 2024-01-16T14:57:52.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.265+0000 7f5a6d400640 1 -- 172.21.15.125:0/3012251220 wait complete. 2024-01-16T14:57:52.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:52.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.492+0000 7fb469000640 1 Processor -- start 2024-01-16T14:57:52.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.492+0000 7fb469000640 1 -- start start 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.493+0000 7fb469000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb4640cf9a0 0x7fb46415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.493+0000 7fb469000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb46415e1e0 0x7fb4640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.493+0000 7fb469000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4640b1ee0 con 0x7fb464143ae0 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.493+0000 7fb469000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4640b2780 con 0x7fb4640cf9a0 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.493+0000 7fb469000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46415e990 con 0x7fb46415e1e0 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb464143ae0 0x7fb4640cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48508/0 (socket says 172.21.15.125:48508) 2024-01-16T14:57:52.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461e00640 1 -- 172.21.15.125:0/3739681420 learned_addr learned my addr 172.21.15.125:0/3739681420 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:52.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb462800640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb46415e1e0 0x7fb4640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716805440 0 0) 0x7fb4640b1ee0 con 0x7fb464143ae0 2024-01-16T14:57:52.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb444003520 con 0x7fb464143ae0 2024-01-16T14:57:52.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb45be00640 1 --2- 172.21.15.125:0/3739681420 >> v2:172.21.15.137:3300/0 conn(0x7fb4640cf9a0 0x7fb46415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475705906 0 0) 0x7fb444003520 con 0x7fb464143ae0 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb46415e1e0 msgr2=0x7fb4640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb46415e1e0 0x7fb4640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 >> v2:172.21.15.137:3300/0 conn(0x7fb4640cf9a0 msgr2=0x7fb46415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 --2- 172.21.15.125:0/3739681420 >> v2:172.21.15.137:3300/0 conn(0x7fb4640cf9a0 0x7fb46415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4640b2130 con 0x7fb464143ae0 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.494+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb454001670 con 0x7fb464143ae0 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 >> v1:172.21.15.125:6789/0 conn(0x7fb464143ae0 legacy=0x7fb4640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4440039a0 0x7fb444003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4440047c0 0x7fb444004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb444005180 0x7fb44400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4640b2780 con 0x7fb4440047c0 2024-01-16T14:57:52.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46415e990 con 0x7fb4440039a0 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44400c020 con 0x7fb444005180 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb462800640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb46415e1e0 0x7fb4640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4440047c0 0x7fb444004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb461e00640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4440039a0 0x7fb444003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 -- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb444005180 msgr2=0x7fb44400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb444005180 0x7fb44400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 -- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4440039a0 msgr2=0x7fb444003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4440039a0 0x7fb444003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.495+0000 7fb45be00640 1 -- 172.21.15.125:0/3739681420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb444003520 con 0x7fb4440047c0 2024-01-16T14:57:52.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.497+0000 7fb45be00640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4440047c0 0x7fb444004c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba30 tx=0x7fb4440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ae8e75b55f6155 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.497+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c032680 con 0x7fb4440047c0 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.497+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c0327e0 con 0x7fb4440047c0 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.497+0000 7fb461400640 1 -- 172.21.15.125:0/3739681420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c032b70 con 0x7fb4440047c0 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 -- 172.21.15.125:0/3739681420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4440047c0 msgr2=0x7fb444004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 --2- 172.21.15.125:0/3739681420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4440047c0 0x7fb444004c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba30 tx=0x7fb4440057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb462800640 1 -- 172.21.15.125:0/3739681420 reap_dead start 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 -- 172.21.15.125:0/3739681420 shutdown_connections 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 -- 172.21.15.125:0/3739681420 >> 172.21.15.125:0/3739681420 conn(0x7fb46406a800 msgr2=0x7fb46406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 -- 172.21.15.125:0/3739681420 shutdown_connections 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.498+0000 7fb469000640 1 -- 172.21.15.125:0/3739681420 wait complete. 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.499+0000 7fb469000640 1 Processor -- start 2024-01-16T14:57:52.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.500+0000 7fb469000640 1 -- start start 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.500+0000 7fb469000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb464159a90 0x7fb4640ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb469000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb469000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4640aaa70 0x7fb4640aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb469000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb464142b30 con 0x7fb46415b7d0 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb469000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46415c190 con 0x7fb464159a90 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb469000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb464159e70 con 0x7fb4640aaa70 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55680/0 (socket says 172.21.15.125:55680) 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 -- 172.21.15.125:0/1524340072 learned_addr learned my addr 172.21.15.125:0/1524340072 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb462800640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4640aaa70 0x7fb4640aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb461e00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb464159a90 0x7fb4640ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 -- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4640aaa70 msgr2=0x7fb4640aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4640aaa70 0x7fb4640aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 -- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb464159a90 msgr2=0x7fb4640ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb464159a90 0x7fb4640ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.501+0000 7fb45be00640 1 -- 172.21.15.125:0/1524340072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464141fb0 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb45be00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00bb50 tx=0x7fb44c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0307e0 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c030960 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb464142290 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c030d10 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.502+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464142850 con 0x7fb46415b7d0 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.503+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46406e0b0 con 0x7fb46415b7d0 2024-01-16T14:57:52.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.504+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb44c03b420 con 0x7fb46415b7d0 2024-01-16T14:57:52.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.505+0000 7fb45a000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 0x7fb43c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.505+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb44c0ac070 con 0x7fb46415b7d0 2024-01-16T14:57:52.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.505+0000 7fb461e00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 0x7fb43c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.506+0000 7fb461e00640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 0x7fb43c06d7b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb454001050 tx=0x7fb454004800 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.509+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb44c076e50 con 0x7fb46415b7d0 2024-01-16T14:57:52.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.738+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4640675d0 con 0x7fb46415b7d0 2024-01-16T14:57:52.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.739+0000 7fb45a000640 1 -- 172.21.15.125:0/1524340072 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4919 (secure 0 0 0) 0x7fb44c07a470 con 0x7fb46415b7d0 2024-01-16T14:57:52.721 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:52.721 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3888287288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3888287288},{"type":"v1","addr":"172.21.15.125:6835","nonce":3888287288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1691320437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1691320437},{"type":"v1","addr":"172.21.15.125:6837","nonce":1691320437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2863352260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2863352260},{"type":"v1","addr":"172.21.15.137:6834","nonce":2863352260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:56:45.415990+0000","modified":"2024-01-16T14:56:48.497267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6835/3474295024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":3474295024},{"type":"v1","addr":"172.21.15.137:6835","nonce":3474295024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:52.721 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 8 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 msgr2=0x7fb43c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 0x7fb43c06d7b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb454001050 tx=0x7fb454004800 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 msgr2=0x7fb46415bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00bb50 tx=0x7fb44c004730 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 shutdown_connections 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4640aaa70 0x7fb4640aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb43c06b2f0 0x7fb43c06d7b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb46415b7d0 0x7fb46415bc50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 --2- 172.21.15.125:0/1524340072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb464159a90 0x7fb4640ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.742+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 >> 172.21.15.125:0/1524340072 conn(0x7fb46406a800 msgr2=0x7fb46406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.743+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 shutdown_connections 2024-01-16T14:57:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.743+0000 7fb469000640 1 -- 172.21.15.125:0/1524340072 wait complete. 2024-01-16T14:57:52.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:52.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.967+0000 7fa62ee00640 1 Processor -- start 2024-01-16T14:57:52.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.967+0000 7fa62ee00640 1 -- start start 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa62ee00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa62813df80 0x7fa62815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa62ee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa62813e360 0x7fa6280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa62ee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62806c470 con 0x7fa62815f9c0 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa62ee00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62806c750 con 0x7fa62813df80 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa62ee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62806ca30 con 0x7fa62813e360 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa627e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa62813df80 0x7fa62815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa627e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa62813df80 0x7fa62815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55686/0 (socket says 172.21.15.125:55686) 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.969+0000 7fa627e00640 1 -- 172.21.15.125:0/2835760180 learned_addr learned my addr 172.21.15.125:0/2835760180 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.970+0000 7fa627e00640 1 -- 172.21.15.125:0/2835760180 >> v2:172.21.15.137:3300/0 conn(0x7fa62813e360 msgr2=0x7fa6280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.970+0000 7fa627e00640 1 --2- 172.21.15.125:0/2835760180 >> v2:172.21.15.137:3300/0 conn(0x7fa62813e360 0x7fa6280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.970+0000 7fa627e00640 1 -- 172.21.15.125:0/2835760180 >> v1:172.21.15.137:6789/0 conn(0x7fa62815f9c0 legacy=0x7fa6281408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.970+0000 7fa627e00640 1 -- 172.21.15.125:0/2835760180 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62806cd10 con 0x7fa62813df80 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.971+0000 7fa627e00640 1 --2- 172.21.15.125:0/2835760180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813df80 0x7fa62815bee0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa62806ded0 tx=0x7fa61802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2c70e4b8a04c52 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.971+0000 7fa627400640 1 -- 172.21.15.125:0/2835760180 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618031870 con 0x7fa62813df80 2024-01-16T14:57:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.971+0000 7fa627400640 1 -- 172.21.15.125:0/2835760180 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6180319d0 con 0x7fa62813df80 2024-01-16T14:57:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.971+0000 7fa627400640 1 -- 172.21.15.125:0/2835760180 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618031d60 con 0x7fa62813df80 2024-01-16T14:57:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 -- 172.21.15.125:0/2835760180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813df80 msgr2=0x7fa62815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2835760180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813df80 0x7fa62815bee0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa62806ded0 tx=0x7fa61802b910 comp rx=0 tx=0).stop 2024-01-16T14:57:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 -- 172.21.15.125:0/2835760180 shutdown_connections 2024-01-16T14:57:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2835760180 >> v2:172.21.15.137:3300/0 conn(0x7fa62813e360 0x7fa6280cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2835760180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813df80 0x7fa62815bee0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 -- 172.21.15.125:0/2835760180 >> 172.21.15.125:0/2835760180 conn(0x7fa62806a800 msgr2=0x7fa62806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 -- 172.21.15.125:0/2835760180 shutdown_connections 2024-01-16T14:57:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.972+0000 7fa62ee00640 1 -- 172.21.15.125:0/2835760180 wait complete. 2024-01-16T14:57:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.974+0000 7fa62ee00640 1 Processor -- start 2024-01-16T14:57:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.974+0000 7fa62ee00640 1 -- start start 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa62813df80 0x7fa62814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa62815f9c0 0x7fa62814c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6280702c0 con 0x7fa62813e360 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6281f3390 con 0x7fa62815f9c0 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa628135890 con 0x7fa62813df80 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa61f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa61f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55692/0 (socket says 172.21.15.125:55692) 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa61f200640 1 -- 172.21.15.125:0/2453855675 learned_addr learned my addr 172.21.15.125:0/2453855675 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa61f200640 1 -- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa62813df80 msgr2=0x7fa62814df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa627e00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa62813df80 0x7fa62814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa61f200640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa62813df80 0x7fa62814df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.975+0000 7fa62ce00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa62815f9c0 0x7fa62814c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa61f200640 1 -- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa62815f9c0 msgr2=0x7fa62814c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa61f200640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa62815f9c0 0x7fa62814c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa61f200640 1 -- 172.21.15.125:0/2453855675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180094d0 con 0x7fa62813e360 2024-01-16T14:57:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa61f200640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b4b0 tx=0x7fa61000b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610010610 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610010770 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6280704d0 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610010b20 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.976+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa628070a90 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.977+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6280675d0 con 0x7fa62813e360 2024-01-16T14:57:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.978+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa610010c80 con 0x7fa62813e360 2024-01-16T14:57:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.979+0000 7fa624c00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.979+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa610089930 con 0x7fa62813e360 2024-01-16T14:57:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.979+0000 7fa627e00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.980+0000 7fa627e00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 0x7fa60006d780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa618036000 tx=0x7fa61803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:52.983+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa610058100 con 0x7fa62813e360 2024-01-16T14:57:53.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.174+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa628067f40 con 0x7fa62813e360 2024-01-16T14:57:53.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.177+0000 7fa624c00640 1 -- 172.21.15.125:0/2453855675 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7fa610014090 con 0x7fa62813e360 2024-01-16T14:57:53.159 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:57:53.159 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:57:27.010939+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:56:43.384842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:56:44.446464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:53.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 msgr2=0x7fa60006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 0x7fa60006d780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa618036000 tx=0x7fa61803c040 comp rx=0 tx=0).stop 2024-01-16T14:57:53.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 msgr2=0x7fa6281f2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b4b0 tx=0x7fa61000b970 comp rx=0 tx=0).stop 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 shutdown_connections 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa62815f9c0 0x7fa62814c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa62813e360 0x7fa6281f2e50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 --2- 172.21.15.125:0/2453855675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa62813df80 0x7fa62814df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.180+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 >> 172.21.15.125:0/2453855675 conn(0x7fa62806a800 msgr2=0x7fa62815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.181+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 shutdown_connections 2024-01-16T14:57:53.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.181+0000 7fa62ee00640 1 -- 172.21.15.125:0/2453855675 wait complete. 2024-01-16T14:57:53.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-16T14:57:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.403+0000 7fe001a00640 1 Processor -- start 2024-01-16T14:57:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.404+0000 7fe001a00640 1 -- start start 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fe001a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdffc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fe001a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0b2c70 con 0x7fdffc15d3c0 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fe001a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0cf6d0 con 0x7fdffc0ceea0 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fe001a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0cf3f0 con 0x7fdffc0d0150 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fdffbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdffc15d3c0 0x7fdffc0cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48514/0 (socket says 172.21.15.125:48514) 2024-01-16T14:57:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fdffbe00640 1 -- 172.21.15.125:0/1098953507 learned_addr learned my addr 172.21.15.125:0/1098953507 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fdff3200640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdffc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.405+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335768325 0 0) 0x7fdffc0b2c70 con 0x7fdffc15d3c0 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfdc003520 con 0x7fdffc15d3c0 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316035588 0 0) 0x7fdfdc003520 con 0x7fdffc15d3c0 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 msgr2=0x7fdffc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdffc0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 >> v1:172.21.15.137:6789/0 conn(0x7fdffc0ceea0 legacy=0x7fdffc0b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0b2620 con 0x7fdffc15d3c0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdfec0019f0 con 0x7fdffc15d3c0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 >> v1:172.21.15.125:6789/0 conn(0x7fdffc15d3c0 legacy=0x7fdffc0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfdc0039a0 0x7fdfdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfdc0047c0 0x7fdfdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdfdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0cf3f0 con 0x7fdfdc0039a0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc00ba60 con 0x7fdfdc0047c0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc00bc20 con 0x7fdffc0d0150 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.406+0000 7fdffbe00640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfdc0039a0 0x7fdfdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdff3200640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfdc0047c0 0x7fdfdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fe001000640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdfdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdffbe00640 1 -- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 msgr2=0x7fdfdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdffbe00640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc0d0150 0x7fdfdc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdffbe00640 1 -- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfdc0047c0 msgr2=0x7fdfdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdffbe00640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfdc0047c0 0x7fdfdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.407+0000 7fdffbe00640 1 -- 172.21.15.125:0/1098953507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc003520 con 0x7fdfdc0039a0 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.408+0000 7fdffbe00640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfdc0039a0 0x7fdfdc003dc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00a820 tx=0x7fdfdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f1c19957f3dc0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.408+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0072d0 con 0x7fdfdc0039a0 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.408+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec02f070 con 0x7fdfdc0039a0 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.408+0000 7fdffb400640 1 -- 172.21.15.125:0/1098953507 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec007670 con 0x7fdfdc0039a0 2024-01-16T14:57:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 -- 172.21.15.125:0/1098953507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfdc0039a0 msgr2=0x7fdfdc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 --2- 172.21.15.125:0/1098953507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfdc0039a0 0x7fdfdc003dc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00a820 tx=0x7fdfdc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001000640 1 -- 172.21.15.125:0/1098953507 reap_dead start 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 -- 172.21.15.125:0/1098953507 shutdown_connections 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 -- 172.21.15.125:0/1098953507 >> 172.21.15.125:0/1098953507 conn(0x7fdffc0d3050 msgr2=0x7fdffc0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 -- 172.21.15.125:0/1098953507 shutdown_connections 2024-01-16T14:57:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.409+0000 7fe001a00640 1 -- 172.21.15.125:0/1098953507 wait complete. 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.411+0000 7fe001a00640 1 Processor -- start 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.411+0000 7fe001a00640 1 -- start start 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdffc0cc0c0 0x7fdffc09ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc09d210 0x7fdffc09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc09d6c0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0c9e80 con 0x7fdffc0cc0c0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fe001a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0c75c0 con 0x7fdffc09d210 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55718/0 (socket says 172.21.15.125:55718) 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 -- 172.21.15.125:0/478644175 learned_addr learned my addr 172.21.15.125:0/478644175 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 -- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc09d210 msgr2=0x7fdffc09ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc09d210 0x7fdffc09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 -- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdffc0cc0c0 msgr2=0x7fdffc09ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdffc0cc0c0 0x7fdffc09ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.412+0000 7fdffbe00640 1 -- 172.21.15.125:0/478644175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0cc600 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fdffbe00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00a820 tx=0x7fdffc08b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0072e0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec007440 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0077d0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc0cc8e0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.413+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc0c7af0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.414+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc08ba80 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.415+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdfec031030 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.416+0000 7fdff8c00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 0x7fdfd406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.416+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fdfec0ab680 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.416+0000 7fdff3200640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 0x7fdfd406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.417+0000 7fdff3200640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 0x7fdfd406d7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdffc06c630 tx=0x7fdfe4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.420+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfec0763c0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.647+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdffc067450 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.630 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T14:57:53.649+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T14:57:53.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.653+0000 7fdff8c00640 1 -- 172.21.15.125:0/478644175 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdfec0799e0 con 0x7fdffc0c9aa0 2024-01-16T14:57:53.635 INFO:teuthology.orchestra.run.smithi125.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.656+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 msgr2=0x7fdfd406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.656+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 0x7fdfd406d7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdffc06c630 tx=0x7fdfe4008040 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.656+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 msgr2=0x7fdffc0cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.656+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00a820 tx=0x7fdffc08b0a0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 shutdown_connections 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdffc09d210 0x7fdffc09ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdffc0cc0c0 0x7fdffc09ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdfd406b2f0 0x7fdfd406d7b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 --2- 172.21.15.125:0/478644175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdffc0c9aa0 0x7fdffc0cbb80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 >> 172.21.15.125:0/478644175 conn(0x7fdffc0d3050 msgr2=0x7fdffc0cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 shutdown_connections 2024-01-16T14:57:53.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.657+0000 7fe001a00640 1 -- 172.21.15.125:0/478644175 wait complete. 2024-01-16T14:57:53.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-16T14:57:53.665 INFO:tasks.ceph.mds.b.smithi137.stderr: -3782> 2024-01-16T14:56:51.425+0000 7fa21c400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:57:53.751 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T14:57:53.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.878+0000 7f8210600640 1 Processor -- start 2024-01-16T14:57:53.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.879+0000 7f8210600640 1 -- start start 2024-01-16T14:57:53.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.880+0000 7f8210600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 0x7f82080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.880+0000 7f8210600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f82080cf7d0 0x7f8208070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.880+0000 7f8210600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080cfbf0 con 0x7f8208070ea0 2024-01-16T14:57:53.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.880+0000 7f8210600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820814b6a0 con 0x7f82080cf7d0 2024-01-16T14:57:53.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.880+0000 7f8210600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8208071690 con 0x7f820814b2c0 2024-01-16T14:57:53.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 0x7f82080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820e600640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8208070ea0 0x7f82080b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:53646/0 (socket says 172.21.15.125:53646) 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820e600640 1 -- 172.21.15.125:0/3158305401 learned_addr learned my addr 172.21.15.125:0/3158305401 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820d200640 1 --2- 172.21.15.125:0/3158305401 >> v2:172.21.15.137:3300/0 conn(0x7f82080cf7d0 0x7f8208070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820dc00640 1 -- 172.21.15.125:0/3158305401 >> v2:172.21.15.137:3300/0 conn(0x7f82080cf7d0 msgr2=0x7f8208070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820dc00640 1 --2- 172.21.15.125:0/3158305401 >> v2:172.21.15.137:3300/0 conn(0x7f82080cf7d0 0x7f8208070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820dc00640 1 -- 172.21.15.125:0/3158305401 >> v1:172.21.15.137:6789/0 conn(0x7f8208070ea0 legacy=0x7f82080b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.881+0000 7f820dc00640 1 -- 172.21.15.125:0/3158305401 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8208071970 con 0x7f820814b2c0 2024-01-16T14:57:53.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.883+0000 7f820dc00640 1 --2- 172.21.15.125:0/3158305401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 0x7f82080cf290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81f80097a0 tx=0x7f81f802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=967f4cb4250eadf2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.883+0000 7f8207e00640 1 -- 172.21.15.125:0/3158305401 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f802d040 con 0x7f820814b2c0 2024-01-16T14:57:53.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.883+0000 7f8207e00640 1 -- 172.21.15.125:0/3158305401 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f8031810 con 0x7f820814b2c0 2024-01-16T14:57:53.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.883+0000 7f8207e00640 1 -- 172.21.15.125:0/3158305401 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8031ba0 con 0x7f820814b2c0 2024-01-16T14:57:53.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 -- 172.21.15.125:0/3158305401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 msgr2=0x7f82080cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 --2- 172.21.15.125:0/3158305401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 0x7f82080cf290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81f80097a0 tx=0x7f81f802b8b0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 -- 172.21.15.125:0/3158305401 shutdown_connections 2024-01-16T14:57:53.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 --2- 172.21.15.125:0/3158305401 >> v2:172.21.15.137:3300/0 conn(0x7f82080cf7d0 0x7f8208070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 --2- 172.21.15.125:0/3158305401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820814b2c0 0x7f82080cf290 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 -- 172.21.15.125:0/3158305401 >> 172.21.15.125:0/3158305401 conn(0x7f820806a800 msgr2=0x7f820806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 -- 172.21.15.125:0/3158305401 shutdown_connections 2024-01-16T14:57:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.884+0000 7f8210600640 1 -- 172.21.15.125:0/3158305401 wait complete. 2024-01-16T14:57:53.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.886+0000 7f8210600640 1 Processor -- start 2024-01-16T14:57:53.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.886+0000 7f8210600640 1 -- start start 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8208070ea0 0x7f82080ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82080cf7d0 0x7f82080ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208162e60 con 0x7f820814b2c0 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82081622e0 con 0x7f82080cf7d0 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.887+0000 7f8210600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82081625c0 con 0x7f8208070ea0 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55732/0 (socket says 172.21.15.125:55732) 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 -- 172.21.15.125:0/3916325903 learned_addr learned my addr 172.21.15.125:0/3916325903 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820d200640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82080cf7d0 0x7f82080ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820dc00640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8208070ea0 0x7f82080ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 -- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8208070ea0 msgr2=0x7f82080ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8208070ea0 0x7f82080ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 -- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82080cf7d0 msgr2=0x7f82080ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82080cf7d0 0x7f82080ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 -- 172.21.15.125:0/3916325903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f8009400 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.888+0000 7f820e600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f820000cbc0 tx=0x7f820000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.889+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000ad40 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.889+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200012400 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.889+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000aea0 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.889+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82081628a0 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.889+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8208163930 con 0x7f820814b2c0 2024-01-16T14:57:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.890+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8208142e50 con 0x7f820814b2c0 2024-01-16T14:57:53.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.891+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8200007070 con 0x7f820814b2c0 2024-01-16T14:57:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.892+0000 7f8205600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 0x7f81e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.892+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f8200089b60 con 0x7f820814b2c0 2024-01-16T14:57:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.892+0000 7f820dc00640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 0x7f81e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.892+0000 7f820dc00640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 0x7f81e006d780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81f8036000 tx=0x7f81f803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:53.897+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8208142e50 con 0x7f820814b2c0 2024-01-16T14:57:54.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.125+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8208142e50 con 0x7f820814b2c0 2024-01-16T14:57:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.657+0000 7f8205600640 1 -- 172.21.15.125:0/3916325903 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8208142e50 con 0x7f820814b2c0 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.660+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 msgr2=0x7f81e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.660+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 0x7f81e006d780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81f8036000 tx=0x7f81f803c040 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.660+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 msgr2=0x7f82081f7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.660+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f820000cbc0 tx=0x7f820000a430 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 shutdown_connections 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e006b2c0 0x7f81e006d780 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820814b2c0 0x7f82081f7590 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82080cf7d0 0x7f82080ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 --2- 172.21.15.125:0/3916325903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8208070ea0 0x7f82080ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 >> 172.21.15.125:0/3916325903 conn(0x7f820806a800 msgr2=0x7f82080b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 shutdown_connections 2024-01-16T14:57:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.661+0000 7f8210600640 1 -- 172.21.15.125:0/3916325903 wait complete. 2024-01-16T14:57:54.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-16T14:57:54.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.885+0000 7fc1bbc00640 1 Processor -- start 2024-01-16T14:57:54.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.886+0000 7fc1bbc00640 1 -- start start 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc1b4143ae0 0x7fc1b40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b40cf7d0 0x7fc1b415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1b415e1e0 0x7fc1b415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40cfc30 con 0x7fc1b415e1e0 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40b1ee0 con 0x7fc1b4143ae0 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.887+0000 7fc1bbc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40b2780 con 0x7fc1b40cf7d0 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1b415e1e0 0x7fc1b415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1b415e1e0 0x7fc1b415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55740/0 (socket says 172.21.15.125:55740) 2024-01-16T14:57:54.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 -- 172.21.15.125:0/1540335725 learned_addr learned my addr 172.21.15.125:0/1540335725 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1abe00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b40cf7d0 0x7fc1b415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9200640 1 --2- 172.21.15.125:0/1540335725 >> v2:172.21.15.137:3300/0 conn(0x7fc1b4143ae0 0x7fc1b40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 -- 172.21.15.125:0/1540335725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b40cf7d0 msgr2=0x7fc1b415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b40cf7d0 0x7fc1b415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 -- 172.21.15.125:0/1540335725 >> v2:172.21.15.137:3300/0 conn(0x7fc1b4143ae0 msgr2=0x7fc1b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/1540335725 >> v2:172.21.15.137:3300/0 conn(0x7fc1b4143ae0 0x7fc1b40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.888+0000 7fc1b9c00640 1 -- 172.21.15.125:0/1540335725 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b415e990 con 0x7fc1b415e1e0 2024-01-16T14:57:54.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.890+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b415a670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40cfea0 tx=0x7fc1ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e697e4ae61f9596 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.890+0000 7fc1b3e00640 1 -- 172.21.15.125:0/1540335725 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac02ca70 con 0x7fc1b415e1e0 2024-01-16T14:57:54.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.890+0000 7fc1b3e00640 1 -- 172.21.15.125:0/1540335725 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac02cbd0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.890+0000 7fc1b3e00640 1 -- 172.21.15.125:0/1540335725 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac0343f0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 -- 172.21.15.125:0/1540335725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 msgr2=0x7fc1b415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b415a670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40cfea0 tx=0x7fc1ac02bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 -- 172.21.15.125:0/1540335725 shutdown_connections 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b415a670 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/1540335725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b40cf7d0 0x7fc1b415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/1540335725 >> v2:172.21.15.137:3300/0 conn(0x7fc1b4143ae0 0x7fc1b40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 -- 172.21.15.125:0/1540335725 >> 172.21.15.125:0/1540335725 conn(0x7fc1b406a800 msgr2=0x7fc1b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 -- 172.21.15.125:0/1540335725 shutdown_connections 2024-01-16T14:57:54.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.891+0000 7fc1bbc00640 1 -- 172.21.15.125:0/1540335725 wait complete. 2024-01-16T14:57:54.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.893+0000 7fc1bbc00640 1 Processor -- start 2024-01-16T14:57:54.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.893+0000 7fc1bbc00640 1 -- start start 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b40cf7d0 0x7fc1b40c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b4143ae0 0x7fc1b41f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41f3980 con 0x7fc1b415e1e0 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41f3680 con 0x7fc1b40cf7d0 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1bbc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b415c490 con 0x7fc1b4143ae0 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1b9c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1b9c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55748/0 (socket says 172.21.15.125:55748) 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.894+0000 7fc1b9c00640 1 -- 172.21.15.125:0/474791197 learned_addr learned my addr 172.21.15.125:0/474791197 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9200640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b40cf7d0 0x7fc1b40c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1abe00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b4143ae0 0x7fc1b41f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 -- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b4143ae0 msgr2=0x7fc1b41f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b4143ae0 0x7fc1b41f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 -- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b40cf7d0 msgr2=0x7fc1b40c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b40cf7d0 0x7fc1b40c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 -- 172.21.15.125:0/474791197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac009b30 con 0x7fc1b415e1e0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.895+0000 7fc1b9c00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009f40 tx=0x7fc1ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.896+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac040670 con 0x7fc1b415e1e0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.896+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac0407d0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.896+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b415c770 con 0x7fc1b415e1e0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.896+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac040b80 con 0x7fc1b415e1e0 2024-01-16T14:57:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.896+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b415cd30 con 0x7fc1b415e1e0 2024-01-16T14:57:54.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.897+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b41f3ac0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.899+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc1ac040ce0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.900+0000 7fc1b1600640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 0x7fc18c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.900+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fc1ac0acbf0 con 0x7fc1b415e1e0 2024-01-16T14:57:54.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.900+0000 7fc1b9200640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 0x7fc18c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.901+0000 7fc1b9200640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 0x7fc18c06d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:54.904+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1b41f3ac0 con 0x7fc1b415e1e0 2024-01-16T14:57:55.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.093+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc1b41f3ac0 con 0x7fc1b415e1e0 2024-01-16T14:57:55.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.667+0000 7fc1b1600640 1 -- 172.21.15.125:0/474791197 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc1b41f3ac0 con 0x7fc1b415e1e0 2024-01-16T14:57:55.649 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' removed 2024-01-16T14:57:55.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 msgr2=0x7fc18c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 0x7fc18c06d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4008040 comp rx=0 tx=0).stop 2024-01-16T14:57:55.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 msgr2=0x7fc1b41f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009f40 tx=0x7fc1ac004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 shutdown_connections 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc18c06b2c0 0x7fc18c06d780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b415e1e0 0x7fc1b41f3140 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b4143ae0 0x7fc1b41f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 --2- 172.21.15.125:0/474791197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b40cf7d0 0x7fc1b40c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.672+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 >> 172.21.15.125:0/474791197 conn(0x7fc1b406a800 msgr2=0x7fc1b40702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.673+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 shutdown_connections 2024-01-16T14:57:55.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.673+0000 7fc1bbc00640 1 -- 172.21.15.125:0/474791197 wait complete. 2024-01-16T14:57:55.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-16T14:57:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.905+0000 7f9211e00640 1 Processor -- start 2024-01-16T14:57:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.905+0000 7f9211e00640 1 -- start start 2024-01-16T14:57:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f9211e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f920c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f9211e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c0cfc30 con 0x7f920c15e3b0 2024-01-16T14:57:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f9211e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c0b1ee0 con 0x7f920c0cf7d0 2024-01-16T14:57:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f9211e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920c0b2780 con 0x7f920c143ae0 2024-01-16T14:57:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f920b400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f920c15e3b0 0x7f920c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48524/0 (socket says 172.21.15.125:48524) 2024-01-16T14:57:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.907+0000 7f920b400640 1 -- 172.21.15.125:0/2875733846 learned_addr learned my addr 172.21.15.125:0/2875733846 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:57:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f920aa00640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f920c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030730675 0 0) 0x7f920c0cfc30 con 0x7f920c15e3b0 2024-01-16T14:57:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec003520 con 0x7f920c15e3b0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444947351 0 0) 0x7f91ec003520 con 0x7f920c15e3b0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 msgr2=0x7f920c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f920c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 >> v1:172.21.15.137:6789/0 conn(0x7f920c0cf7d0 legacy=0x7f920c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c15e990 con 0x7f920c15e3b0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f92040019f0 con 0x7f920c15e3b0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 >> v1:172.21.15.125:6789/0 conn(0x7f920c15e3b0 legacy=0x7f920c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91ec0039a0 0x7f91ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91ec0047c0 0x7f91ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f91ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c0b2780 con 0x7f91ec0047c0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec00b980 con 0x7f91ec0039a0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.908+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec00bb40 con 0x7f920c143ae0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91ec0047c0 0x7f91ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920aa00640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91ec0039a0 0x7f91ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920b400640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f91ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 -- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 msgr2=0x7f91ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920c143ae0 0x7f91ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 -- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91ec0039a0 msgr2=0x7f91ec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91ec0039a0 0x7f91ec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920a000640 1 -- 172.21.15.125:0/2875733846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec003520 con 0x7f91ec0047c0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.909+0000 7f920aa00640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91ec0039a0 0x7f91ec003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.910+0000 7f920a000640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91ec0047c0 0x7f91ec004c40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f91f4001720 tx=0x7f91ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783a7901acd4c426 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.910+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f40052d0 con 0x7f91ec0047c0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.910+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f402e050 con 0x7f91ec0047c0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.910+0000 7f9209600640 1 -- 172.21.15.125:0/2875733846 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f40035f0 con 0x7f91ec0047c0 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 -- 172.21.15.125:0/2875733846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91ec0047c0 msgr2=0x7f91ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 --2- 172.21.15.125:0/2875733846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91ec0047c0 0x7f91ec004c40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f91f4001720 tx=0x7f91ec0057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f920b400640 1 -- 172.21.15.125:0/2875733846 reap_dead start 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 -- 172.21.15.125:0/2875733846 shutdown_connections 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 -- 172.21.15.125:0/2875733846 >> 172.21.15.125:0/2875733846 conn(0x7f920c06a800 msgr2=0x7f920c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 -- 172.21.15.125:0/2875733846 shutdown_connections 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.911+0000 7f9211e00640 1 -- 172.21.15.125:0/2875733846 wait complete. 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.913+0000 7f9211e00640 1 Processor -- start 2024-01-16T14:57:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.913+0000 7f9211e00640 1 -- start start 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91ec0047c0 0x7f920c0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f920c158060 0x7f920c13f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c13a4b0 con 0x7f920c137180 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c13a8d0 con 0x7f920c158060 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f9211e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920c1588d0 con 0x7f91ec0047c0 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f920a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f920a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55768/0 (socket says 172.21.15.125:55768) 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.914+0000 7f920a000640 1 -- 172.21.15.125:0/1975496489 learned_addr learned my addr 172.21.15.125:0/1975496489 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920b400640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f920c158060 0x7f920c13f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920aa00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91ec0047c0 0x7f920c0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 -- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91ec0047c0 msgr2=0x7f920c0b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91ec0047c0 0x7f920c0b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 -- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f920c158060 msgr2=0x7f920c13f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f920c158060 0x7f920c13f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 -- 172.21.15.125:0/1975496489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c139830 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f920a000640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f91f4008fd0 tx=0x7f91f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f40076c0 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f4007a30 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.915+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4006be0 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.916+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c139b10 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.916+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c13a0d0 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.917+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920c1493b0 con 0x7f920c137180 2024-01-16T14:57:55.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.918+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f91f4006d40 con 0x7f920c137180 2024-01-16T14:57:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.918+0000 7f9203400640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 0x7f91e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.918+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f91f40abda0 con 0x7f920c137180 2024-01-16T14:57:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.919+0000 7f920aa00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 0x7f91e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.919+0000 7f920aa00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 0x7f91e406d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f920c137630 tx=0x7f91fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:55.923+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91f4076df0 con 0x7f920c137180 2024-01-16T14:57:56.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.116+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f920c0675d0 con 0x7f920c137180 2024-01-16T14:57:56.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.969+0000 7f9203400640 1 -- 172.21.15.125:0/1975496489 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f91f404b0d0 con 0x7f920c137180 2024-01-16T14:57:56.950 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' removed 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.972+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 msgr2=0x7f91e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.972+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 0x7f91e406d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f920c137630 tx=0x7f91fc008040 comp rx=0 tx=0).stop 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.972+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 msgr2=0x7f920c13f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.972+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f91f4008fd0 tx=0x7f91f4001120 comp rx=0 tx=0).stop 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 shutdown_connections 2024-01-16T14:57:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f920c158060 0x7f920c13f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f91e406b2c0 0x7f91e406d780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920c137180 0x7f920c13f0d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 --2- 172.21.15.125:0/1975496489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91ec0047c0 0x7f920c0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 >> 172.21.15.125:0/1975496489 conn(0x7f920c06a800 msgr2=0x7f920c142b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 shutdown_connections 2024-01-16T14:57:56.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:57:56.973+0000 7f9211e00640 1 -- 172.21.15.125:0/1975496489 wait complete. 2024-01-16T14:57:56.984 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:57:56.984 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T14:57:56.984 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T14:57:56.985 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:56.985 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:57:56.985 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T14:57:56.985 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T14:57:56.985 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:56.985 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:57:56.985 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T14:57:56.985 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T14:57:56.986 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:56.986 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:57:56.986 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T14:57:56.986 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T14:57:56.986 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:57.004 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T14:57:57.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:57:57.005 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T14:57:57.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:57:57.013 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T14:57:57.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:57:57.050 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:57:57.051 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:57.055 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:57:57.055 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:57.060 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:57:57.060 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:57.199 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T14:57:57.201 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T14:57:57.237 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T14:57:59.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-16T14:58:03.013 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T14:58:03.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:58:03.021 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:58:03.021 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:03.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T14:58:03.186 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T14:58:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.231+0000 7f2ce6e00640 1 Processor -- start 2024-01-16T14:58:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.231+0000 7f2ce6e00640 1 -- start start 2024-01-16T14:58:03.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.232+0000 7f2ce6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2ce0166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.232+0000 7f2ce6e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce015c5f0 con 0x7f2ce013df80 2024-01-16T14:58:03.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.232+0000 7f2ce6e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce006c470 con 0x7f2ce013e360 2024-01-16T14:58:03.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.232+0000 7f2ce6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006c750 con 0x7f2ce00cf9a0 2024-01-16T14:58:03.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2ce013df80 0x7f2ce015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43810/0 (socket says 172.21.15.125:43810) 2024-01-16T14:58:03.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdfe00640 1 -- 172.21.15.125:0/1381249802 learned_addr learned my addr 172.21.15.125:0/1381249802 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:03.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343527480 0 0) 0x7f2ce015c5f0 con 0x7f2ce013df80 2024-01-16T14:58:03.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0003520 con 0x7f2ce013df80 2024-01-16T14:58:03.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2ce0166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219807282 0 0) 0x7f2cc0003520 con 0x7f2ce013df80 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 msgr2=0x7f2ce0166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2ce0166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 >> v1:172.21.15.137:6789/0 conn(0x7f2ce013e360 legacy=0x7f2ce00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce006ca30 con 0x7f2ce013df80 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2cd0001670 con 0x7f2ce013df80 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 >> v1:172.21.15.125:6789/0 conn(0x7f2ce013df80 legacy=0x7f2ce015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.233+0000 7f2cdea00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00039a0 0x7f2cc0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdea00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc00047c0 0x7f2cc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdea00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2cc00051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006c750 con 0x7f2cc00047c0 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc000bac0 con 0x7f2cc00039a0 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc000bc80 con 0x7f2ce00cf9a0 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc00047c0 0x7f2cc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdfe00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00039a0 0x7f2cc0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2cc00051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 -- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 msgr2=0x7f2cc00051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2cc00051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 -- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00039a0 msgr2=0x7f2cc0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00039a0 0x7f2cc0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2cdf400640 1 -- 172.21.15.125:0/1381249802 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0003520 con 0x7f2cc00047c0 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.234+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00cf9a0 0x7f2cc00051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:03.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.235+0000 7f2cdf400640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc00047c0 0x7f2cc0004c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003d80 tx=0x7f2cc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f027e667567358e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8032600 con 0x7f2cc00047c0 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc802e050 con 0x7f2cc00047c0 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2cdea00640 1 -- 172.21.15.125:0/1381249802 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc80329a0 con 0x7f2cc00047c0 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 -- 172.21.15.125:0/1381249802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc00047c0 msgr2=0x7f2cc0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/1381249802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc00047c0 0x7f2cc0004c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003d80 tx=0x7f2cc00057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce4e00640 1 -- 172.21.15.125:0/1381249802 reap_dead start 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 -- 172.21.15.125:0/1381249802 shutdown_connections 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 -- 172.21.15.125:0/1381249802 >> 172.21.15.125:0/1381249802 conn(0x7f2ce006a800 msgr2=0x7f2ce006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 -- 172.21.15.125:0/1381249802 shutdown_connections 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.236+0000 7f2ce6e00640 1 -- 172.21.15.125:0/1381249802 wait complete. 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.238+0000 7f2ce6e00640 1 Processor -- start 2024-01-16T14:58:03.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.238+0000 7f2ce6e00640 1 -- start start 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00047c0 0x7f2ce013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00afc40 0x7f2ce00b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01509f0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce00b0600 con 0x7f2cc00047c0 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01f5670 con 0x7f2ce00afc40 2024-01-16T14:58:03.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43196/0 (socket says 172.21.15.125:43196) 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.239+0000 7f2ce4e00640 1 -- 172.21.15.125:0/249070800 learned_addr learned my addr 172.21.15.125:0/249070800 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2cdf400640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00afc40 0x7f2ce00b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 -- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00afc40 msgr2=0x7f2ce00b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00afc40 0x7f2ce00b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 -- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00047c0 msgr2=0x7f2ce013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00047c0 0x7f2ce013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 -- 172.21.15.125:0/249070800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce01f5950 con 0x7f2ce01f48f0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce4e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00cfea0 tx=0x7f2cd8009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8007cf0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8007e70 con 0x7f2ce01f48f0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce01f5c30 con 0x7f2ce01f48f0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.240+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8014620 con 0x7f2ce01f48f0 2024-01-16T14:58:03.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.241+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce01f52b0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.242+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce00675d0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.243+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2cd801d3b0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.243+0000 7f2cd7e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 0x7f2cb806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.244+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2cd80895f0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.244+0000 7f2cdfe00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 0x7f2cb806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.245+0000 7f2cdfe00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 0x7f2cb806d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0005a10 tx=0x7f2cd00011a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.249+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cd80547f0 con 0x7f2ce01f48f0 2024-01-16T14:58:03.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:03.448+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2ce015ed30 con 0x7f2ce01f48f0 2024-01-16T14:58:05.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.021+0000 7f2cd7e00640 1 -- 172.21.15.125:0/249070800 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2ce015ed30 con 0x7f2ce01f48f0 2024-01-16T14:58:05.002 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T14:58:05.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.024+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 msgr2=0x7f2cb806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.024+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 0x7f2cb806d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0005a10 tx=0x7f2cd00011a0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.024+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 msgr2=0x7f2ce01f4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.024+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00cfea0 tx=0x7f2cd8009d30 comp rx=0 tx=0).stop 2024-01-16T14:58:05.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 shutdown_connections 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cb806b2c0 0x7f2cb806d780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce01f48f0 0x7f2ce01f4d70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce00afc40 0x7f2ce00b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 --2- 172.21.15.125:0/249070800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc00047c0 0x7f2ce013d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 >> 172.21.15.125:0/249070800 conn(0x7f2ce006a800 msgr2=0x7f2ce00aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 shutdown_connections 2024-01-16T14:58:05.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.025+0000 7f2ce6e00640 1 -- 172.21.15.125:0/249070800 wait complete. 2024-01-16T14:58:05.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T14:58:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.257+0000 7f8cca000640 1 Processor -- start 2024-01-16T14:58:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.257+0000 7f8cca000640 1 -- start start 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cca000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8cc4166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cca000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc415c5f0 con 0x7f8cc413e360 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cca000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc406c470 con 0x7f8cc413df80 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cca000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc406c750 con 0x7f8cc40cf9a0 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cbbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8cc413e360 0x7f8cc40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43826/0 (socket says 172.21.15.125:43826) 2024-01-16T14:58:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cbbe00640 1 -- 172.21.15.125:0/2123599034 learned_addr learned my addr 172.21.15.125:0/2123599034 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cc3e00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8cc4166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962277505 0 0) 0x7f8cc415c5f0 con 0x7f8cc413e360 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.259+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4003520 con 0x7f8cc413e360 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600744751 0 0) 0x7f8ca4003520 con 0x7f8cc413e360 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 msgr2=0x7f8cc4166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8cc4166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 >> v1:172.21.15.137:6789/0 conn(0x7f8cc413df80 legacy=0x7f8cc415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc406ca30 con 0x7f8cc413e360 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8cac001670 con 0x7f8cc413e360 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 >> v1:172.21.15.125:6789/0 conn(0x7f8cc413e360 legacy=0x7f8cc40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8ca40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca4004020 0x7f8ca40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8ca4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc406c750 con 0x7f8ca4004d00 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca400bbc0 con 0x7f8ca4004020 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca400bd80 con 0x7f8cc40cf9a0 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc3e00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8ca4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.260+0000 7f8cc3400640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8ca40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cbbe00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca4004020 0x7f8ca40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cc3e00640 1 -- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 msgr2=0x7f8ca40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cc3e00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc40cf9a0 0x7f8ca40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cc3e00640 1 -- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca4004020 msgr2=0x7f8ca40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cc3e00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca4004020 0x7f8ca40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.261+0000 7f8cc3e00640 1 -- 172.21.15.125:0/2123599034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4003520 con 0x7f8ca4004d00 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.262+0000 7f8cc3e00640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8ca4005180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc007330 tx=0x7f8ca40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a160ac2120cba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.262+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc02cca0 con 0x7f8ca4004d00 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.262+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc02ce00 con 0x7f8ca4004d00 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.262+0000 7f8cc2a00640 1 -- 172.21.15.125:0/2123599034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc034660 con 0x7f8ca4004d00 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 -- 172.21.15.125:0/2123599034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 msgr2=0x7f8ca4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 --2- 172.21.15.125:0/2123599034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8ca4005180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc007330 tx=0x7f8ca40057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cc3e00640 1 -- 172.21.15.125:0/2123599034 reap_dead start 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 -- 172.21.15.125:0/2123599034 shutdown_connections 2024-01-16T14:58:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 -- 172.21.15.125:0/2123599034 >> 172.21.15.125:0/2123599034 conn(0x7f8cc406a800 msgr2=0x7f8cc406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 -- 172.21.15.125:0/2123599034 shutdown_connections 2024-01-16T14:58:05.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.263+0000 7f8cca000640 1 -- 172.21.15.125:0/2123599034 wait complete. 2024-01-16T14:58:05.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.265+0000 7f8cca000640 1 Processor -- start 2024-01-16T14:58:05.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.265+0000 7f8cca000640 1 -- start start 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc4157210 0x7f8cc41500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cc415cc80 0x7f8cc415d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc415d640 con 0x7f8ca4004d00 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4158b30 con 0x7f8cc415cc80 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cca000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4150620 con 0x7f8cc4157210 2024-01-16T14:58:05.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43220/0 (socket says 172.21.15.125:43220) 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 -- 172.21.15.125:0/1546524982 learned_addr learned my addr 172.21.15.125:0/1546524982 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 -- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc4157210 msgr2=0x7f8cc41500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc4157210 0x7f8cc41500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 -- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cc415cc80 msgr2=0x7f8cc415d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cc415cc80 0x7f8cc415d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.266+0000 7f8cc3400640 1 -- 172.21.15.125:0/1546524982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc41372f0 con 0x7f8ca4004d00 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cc3400640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400e580 tx=0x7f8cb400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40052e0 con 0x7f8ca4004d00 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb4005440 con 0x7f8ca4004d00 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4003810 con 0x7f8ca4004d00 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc41510f0 con 0x7f8ca4004d00 2024-01-16T14:58:05.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.267+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4157690 con 0x7f8ca4004d00 2024-01-16T14:58:05.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.268+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc40675d0 con 0x7f8ca4004d00 2024-01-16T14:58:05.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.269+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8cb4003970 con 0x7f8ca4004d00 2024-01-16T14:58:05.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.270+0000 7f8cbb400640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 0x7f8c9c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.270+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8cb4089650 con 0x7f8ca4004d00 2024-01-16T14:58:05.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.270+0000 7f8cbbe00640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 0x7f8c9c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.271+0000 7f8cbbe00640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 0x7f8c9c06d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8cc41f4360 tx=0x7f8cac000ff0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.275+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cb4054780 con 0x7f8ca4004d00 2024-01-16T14:58:05.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.531+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8cc406df00 con 0x7f8ca4004d00 2024-01-16T14:58:05.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.533+0000 7f8cbb400640 1 -- 172.21.15.125:0/1546524982 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v3) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f8cb4057da0 con 0x7f8ca4004d00 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:58:05.515 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T14:58:05.516 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.518 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T14:58:05.520 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.521 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:58:05.522 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.523 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-mds", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZloVwoNxAAdZHX4ZytvCc9kjCtCMVaIQ==", 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.525 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-mds" 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-mgr", 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZlhWwoNxAAK9kc4TwSRW3OU+id6VuK/Q==", 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-osd", 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZlyHcoNxAAZATAVZRDv6cBLpVSyrJdAQ==", 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-osd" 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-rbd", 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZlhoAoNxAAgz7r928C/hn4A7JB2wHJFA==", 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZl84goNxAAecjybywwOFZtVDD9w6BpTA==", 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.bootstrap-rgw", 2024-01-16T14:58:05.527 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAUmaZl1pEoNxAAZNcpOyM4R+W80oFs91lMKA==", 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:05.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T14:58:05.529 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 msgr2=0x7f8c9c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 0x7f8c9c06d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8cc41f4360 tx=0x7f8cac000ff0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 msgr2=0x7f8cc415acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400e580 tx=0x7f8cb400ea40 comp rx=0 tx=0).stop 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 shutdown_connections 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cc415cc80 0x7f8cc415d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cc4157210 0x7f8cc41500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8c9c06b2c0 0x7f8c9c06d780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 --2- 172.21.15.125:0/1546524982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ca4004d00 0x7f8cc415acb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.535+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 >> 172.21.15.125:0/1546524982 conn(0x7f8cc406a800 msgr2=0x7f8cc40b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.536+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 shutdown_connections 2024-01-16T14:58:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.536+0000 7f8cca000640 1 -- 172.21.15.125:0/1546524982 wait complete. 2024-01-16T14:58:05.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-16T14:58:05.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.761+0000 7f9e5a400640 1 Processor -- start 2024-01-16T14:58:05.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.762+0000 7f9e5a400640 1 -- start start 2024-01-16T14:58:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.763+0000 7f9e5a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.763+0000 7f9e5a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9e540cf290 0x7f9e540cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.763+0000 7f9e5a400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e54070c40 con 0x7f9e540ae5a0 2024-01-16T14:58:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.763+0000 7f9e5a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e54071450 con 0x7f9e540cf290 2024-01-16T14:58:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.763+0000 7f9e5a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540cfc60 con 0x7f9e540b3ae0 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.764+0000 7f9e52a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.764+0000 7f9e52a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:54980/0 (socket says 172.21.15.125:54980) 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.764+0000 7f9e52a00640 1 -- 172.21.15.125:0/3800422751 learned_addr learned my addr 172.21.15.125:0/3800422751 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.764+0000 7f9e53e00640 1 --2- 172.21.15.125:0/3800422751 >> v2:172.21.15.137:3300/0 conn(0x7f9e540cf290 0x7f9e540cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e53e00640 1 -- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 msgr2=0x7f9e5415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e53e00640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e53e00640 1 -- 172.21.15.125:0/3800422751 >> v1:172.21.15.137:6789/0 conn(0x7f9e540ae5a0 legacy=0x7f9e5406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e52000640 1 -- 172.21.15.125:0/3800422751 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249470188 0 0) 0x7f9e54070c40 con 0x7f9e540ae5a0 2024-01-16T14:58:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e53e00640 1 -- 172.21.15.125:0/3800422751 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5406c470 con 0x7f9e540cf290 2024-01-16T14:58:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.765+0000 7f9e52a00640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.766+0000 7f9e53e00640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540cf290 0x7f9e540cf720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009d90 tx=0x7f9e4c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85a087a745d3ffe2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.767+0000 7f9e52000640 1 -- 172.21.15.125:0/3800422751 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c02cce0 con 0x7f9e540cf290 2024-01-16T14:58:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.767+0000 7f9e52000640 1 -- 172.21.15.125:0/3800422751 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c02ce40 con 0x7f9e540cf290 2024-01-16T14:58:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.767+0000 7f9e52000640 1 -- 172.21.15.125:0/3800422751 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c034660 con 0x7f9e540cf290 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 -- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540cf290 msgr2=0x7f9e540cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540cf290 0x7f9e540cf720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009d90 tx=0x7f9e4c02bf30 comp rx=0 tx=0).stop 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 -- 172.21.15.125:0/3800422751 shutdown_connections 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540cf290 0x7f9e540cf720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 --2- 172.21.15.125:0/3800422751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540b3ae0 0x7f9e5415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 -- 172.21.15.125:0/3800422751 >> 172.21.15.125:0/3800422751 conn(0x7f9e5406a800 msgr2=0x7f9e5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 -- 172.21.15.125:0/3800422751 shutdown_connections 2024-01-16T14:58:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.768+0000 7f9e5a400640 1 -- 172.21.15.125:0/3800422751 wait complete. 2024-01-16T14:58:05.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.770+0000 7f9e5a400640 1 Processor -- start 2024-01-16T14:58:05.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.770+0000 7f9e5a400640 1 -- start start 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540b3ae0 0x7f9e5415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540cf290 0x7f9e54146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54159230 con 0x7f9e540ae5a0 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54159510 con 0x7f9e540b3ae0 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.771+0000 7f9e5a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e541597f0 con 0x7f9e540cf290 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43222/0 (socket says 172.21.15.125:43222) 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 -- 172.21.15.125:0/4107298074 learned_addr learned my addr 172.21.15.125:0/4107298074 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 -- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540cf290 msgr2=0x7f9e54146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53e00640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540cf290 0x7f9e54146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e52a00640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540b3ae0 0x7f9e5415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540cf290 0x7f9e54146000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 -- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540b3ae0 msgr2=0x7f9e5415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540b3ae0 0x7f9e5415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.772+0000 7f9e53400640 1 -- 172.21.15.125:0/4107298074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c0094d0 con 0x7f9e540ae5a0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e53400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c4b0 tx=0x7f9e4400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44005480 con 0x7f9e540ae5a0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e440035f0 con 0x7f9e540ae5a0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c009a70 con 0x7f9e540ae5a0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e440039e0 con 0x7f9e540ae5a0 2024-01-16T14:58:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.773+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e54159db0 con 0x7f9e540ae5a0 2024-01-16T14:58:05.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.774+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e540675d0 con 0x7f9e540ae5a0 2024-01-16T14:58:05.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.775+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9e44003b40 con 0x7f9e540ae5a0 2024-01-16T14:58:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.776+0000 7f9e4b400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 0x7f9e2c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.776+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9e44088f00 con 0x7f9e540ae5a0 2024-01-16T14:58:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.776+0000 7f9e52a00640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 0x7f9e2c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.777+0000 7f9e52a00640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 0x7f9e2c06d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ce910 tx=0x7f9e3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:05.780+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e44054030 con 0x7f9e540ae5a0 2024-01-16T14:58:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.033+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f9e540008d0 con 0x7f9e540ae5a0 2024-01-16T14:58:06.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.047+0000 7f9e4b400640 1 -- 172.21.15.125:0/4107298074 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9e44017030 con 0x7f9e540ae5a0 2024-01-16T14:58:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 msgr2=0x7f9e2c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 0x7f9e2c06d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ce910 tx=0x7f9e3c008040 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 msgr2=0x7f9e5415fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c4b0 tx=0x7f9e4400c970 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 shutdown_connections 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e540cf290 0x7f9e54146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e540b3ae0 0x7f9e5415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e2c06b2c0 0x7f9e2c06d780 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 --2- 172.21.15.125:0/4107298074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540ae5a0 0x7f9e5415fa50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 >> 172.21.15.125:0/4107298074 conn(0x7f9e5406a800 msgr2=0x7f9e54158600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 shutdown_connections 2024-01-16T14:58:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.050+0000 7f9e5a400640 1 -- 172.21.15.125:0/4107298074 wait complete. 2024-01-16T14:58:06.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-16T14:58:06.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.273+0000 7fb82dc00640 1 Processor -- start 2024-01-16T14:58:06.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.273+0000 7fb82dc00640 1 -- start start 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.274+0000 7fb82dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb82dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb8280cf7d0 0x7fb82815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb82dc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280cfc30 con 0x7fb82815e1e0 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb82dc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8280b1ee0 con 0x7fb8280cf7d0 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb82dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280b2780 con 0x7fb828143ae0 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb826a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb826a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:54994/0 (socket says 172.21.15.125:54994) 2024-01-16T14:58:06.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.275+0000 7fb826a00640 1 -- 172.21.15.125:0/35998575 learned_addr learned my addr 172.21.15.125:0/35998575 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:06.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.276+0000 7fb81de00640 1 --2- 172.21.15.125:0/35998575 >> v2:172.21.15.137:3300/0 conn(0x7fb8280cf7d0 0x7fb82815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.276+0000 7fb826a00640 1 -- 172.21.15.125:0/35998575 >> v2:172.21.15.137:3300/0 conn(0x7fb8280cf7d0 msgr2=0x7fb82815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:06.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.276+0000 7fb826a00640 1 --2- 172.21.15.125:0/35998575 >> v2:172.21.15.137:3300/0 conn(0x7fb8280cf7d0 0x7fb82815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.276+0000 7fb826a00640 1 -- 172.21.15.125:0/35998575 >> v1:172.21.15.137:6789/0 conn(0x7fb82815e1e0 legacy=0x7fb82815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.276+0000 7fb826a00640 1 -- 172.21.15.125:0/35998575 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82815e990 con 0x7fb828143ae0 2024-01-16T14:58:06.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.277+0000 7fb826a00640 1 --2- 172.21.15.125:0/35998575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb818009e50 tx=0x7fb81802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=446a655bb8a06bb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb826000640 1 -- 172.21.15.125:0/35998575 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81802cb00 con 0x7fb828143ae0 2024-01-16T14:58:06.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb826000640 1 -- 172.21.15.125:0/35998575 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81802cc60 con 0x7fb828143ae0 2024-01-16T14:58:06.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb826000640 1 -- 172.21.15.125:0/35998575 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8180343f0 con 0x7fb828143ae0 2024-01-16T14:58:06.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 -- 172.21.15.125:0/35998575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 msgr2=0x7fb8280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 --2- 172.21.15.125:0/35998575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb818009e50 tx=0x7fb81802bf60 comp rx=0 tx=0).stop 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 -- 172.21.15.125:0/35998575 shutdown_connections 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 --2- 172.21.15.125:0/35998575 >> v2:172.21.15.137:3300/0 conn(0x7fb8280cf7d0 0x7fb82815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 --2- 172.21.15.125:0/35998575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb828143ae0 0x7fb8280cf290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.278+0000 7fb82dc00640 1 -- 172.21.15.125:0/35998575 >> 172.21.15.125:0/35998575 conn(0x7fb82806a800 msgr2=0x7fb82806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.279+0000 7fb82dc00640 1 -- 172.21.15.125:0/35998575 shutdown_connections 2024-01-16T14:58:06.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.279+0000 7fb82dc00640 1 -- 172.21.15.125:0/35998575 wait complete. 2024-01-16T14:58:06.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.280+0000 7fb82dc00640 1 Processor -- start 2024-01-16T14:58:06.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.280+0000 7fb82dc00640 1 -- start start 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 0x7fb828159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb82815e1e0 0x7fb82813fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82813b160 con 0x7fb828143ae0 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82813b400 con 0x7fb8280cf7d0 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.281+0000 7fb82dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82813b6e0 con 0x7fb82815e1e0 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43230/0 (socket says 172.21.15.125:43230) 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 -- 172.21.15.125:0/3065641776 learned_addr learned my addr 172.21.15.125:0/3065641776 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb826a00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 0x7fb828159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb827400640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb82815e1e0 0x7fb82813fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 -- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb82815e1e0 msgr2=0x7fb82813fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb82815e1e0 0x7fb82813fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 -- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 msgr2=0x7fb828159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 0x7fb828159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 -- 172.21.15.125:0/3065641776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818009b30 con 0x7fb828143ae0 2024-01-16T14:58:06.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb826a00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 0x7fb828159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.282+0000 7fb81de00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb81000e9b0 tx=0x7fb81000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.283+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810009670 con 0x7fb828143ae0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.283+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81000c9a0 con 0x7fb828143ae0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.283+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82813b9c0 con 0x7fb828143ae0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.283+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81000cd50 con 0x7fb828143ae0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.283+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb828159ca0 con 0x7fb828143ae0 2024-01-16T14:58:06.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.284+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8280b2bf0 con 0x7fb828143ae0 2024-01-16T14:58:06.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.285+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb810013470 con 0x7fb828143ae0 2024-01-16T14:58:06.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.286+0000 7fb81f400640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 0x7fb80006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.286+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb810089ae0 con 0x7fb828143ae0 2024-01-16T14:58:06.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.286+0000 7fb826a00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 0x7fb80006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.287+0000 7fb826a00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 0x7fb80006d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb818035000 tx=0x7fb818030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.291+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8280b2bf0 con 0x7fb828143ae0 2024-01-16T14:58:06.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-01-16T14:58:06.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:06.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:06.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.553+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fb8280b2bf0 con 0x7fb828143ae0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.557+0000 7fb81f400640 1 -- 172.21.15.125:0/3065641776 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb8280b2bf0 con 0x7fb828143ae0 2024-01-16T14:58:06.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.559+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 msgr2=0x7fb80006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.559+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 0x7fb80006d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb818035000 tx=0x7fb818030000 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 msgr2=0x7fb82813f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb81000e9b0 tx=0x7fb81000ee70 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 shutdown_connections 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb82815e1e0 0x7fb82813fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb80006b2c0 0x7fb80006d780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb828143ae0 0x7fb82813f6d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 --2- 172.21.15.125:0/3065641776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8280cf7d0 0x7fb828159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 >> 172.21.15.125:0/3065641776 conn(0x7fb82806a800 msgr2=0x7fb8280aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 shutdown_connections 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.560+0000 7fb82dc00640 1 -- 172.21.15.125:0/3065641776 wait complete. 2024-01-16T14:58:06.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-16T14:58:06.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.789+0000 7f6466a00640 1 Processor -- start 2024-01-16T14:58:06.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.789+0000 7f6466a00640 1 -- start start 2024-01-16T14:58:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f64601447f0 0x7f646015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646015c5c0 con 0x7f64600cccf0 2024-01-16T14:58:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646006ca10 con 0x7f646006c470 2024-01-16T14:58:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646006d690 con 0x7f64601447f0 2024-01-16T14:58:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6465600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f64601447f0 0x7f646015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64600cccf0 0x7f646015c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41982/0 (socket says 172.21.15.125:41982) 2024-01-16T14:58:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.791+0000 7f6466000640 1 -- 172.21.15.125:0/2651689268 learned_addr learned my addr 172.21.15.125:0/2651689268 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.792+0000 7f645fe00640 1 -- 172.21.15.125:0/2651689268 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653028375 0 0) 0x7f646015c5c0 con 0x7f64600cccf0 2024-01-16T14:58:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.792+0000 7f645fe00640 1 -- 172.21.15.125:0/2651689268 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003520 con 0x7f64600cccf0 2024-01-16T14:58:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.792+0000 7f6465600640 1 -- 172.21.15.125:0/2651689268 >> v1:172.21.15.137:6789/0 conn(0x7f646006c470 legacy=0x7f64600cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.792+0000 7f6465600640 1 -- 172.21.15.125:0/2651689268 >> v1:172.21.15.125:6789/0 conn(0x7f64600cccf0 legacy=0x7f646015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.792+0000 7f6465600640 1 -- 172.21.15.125:0/2651689268 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646006ccf0 con 0x7f64601447f0 2024-01-16T14:58:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.793+0000 7f6465600640 1 --2- 172.21.15.125:0/2651689268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64601447f0 0x7f646015c080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6450009d90 tx=0x7f645002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c4e568d8511c69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f645fe00640 1 -- 172.21.15.125:0/2651689268 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645002cc60 con 0x7f64601447f0 2024-01-16T14:58:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f645fe00640 1 -- 172.21.15.125:0/2651689268 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645002cdc0 con 0x7f64601447f0 2024-01-16T14:58:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f645fe00640 1 -- 172.21.15.125:0/2651689268 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450034660 con 0x7f64601447f0 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f6466a00640 1 -- 172.21.15.125:0/2651689268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64601447f0 msgr2=0x7f646015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f6466a00640 1 --2- 172.21.15.125:0/2651689268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64601447f0 0x7f646015c080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6450009d90 tx=0x7f645002beb0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f6466a00640 1 -- 172.21.15.125:0/2651689268 shutdown_connections 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f6466a00640 1 --2- 172.21.15.125:0/2651689268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64601447f0 0x7f646015c080 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.794+0000 7f6466a00640 1 -- 172.21.15.125:0/2651689268 >> 172.21.15.125:0/2651689268 conn(0x7f646006a3e0 msgr2=0x7f646006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.795+0000 7f6466a00640 1 -- 172.21.15.125:0/2651689268 shutdown_connections 2024-01-16T14:58:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.795+0000 7f6466a00640 1 -- 172.21.15.125:0/2651689268 wait complete. 2024-01-16T14:58:06.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.796+0000 7f6466a00640 1 Processor -- start 2024-01-16T14:58:06.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.796+0000 7f6466a00640 1 -- start start 2024-01-16T14:58:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 0x7f6460139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64601447f0 0x7f646013cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64600ae1f0 con 0x7f64600cccf0 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64601f2b10 con 0x7f646006c470 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.797+0000 7f6466a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64601f6e10 con 0x7f64601447f0 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57380/0 (socket says 172.21.15.125:57380) 2024-01-16T14:58:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 -- 172.21.15.125:0/947976264 learned_addr learned my addr 172.21.15.125:0/947976264 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6465600640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 0x7f6460139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6466000640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64601447f0 0x7f646013cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 -- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64601447f0 msgr2=0x7f646013cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64601447f0 0x7f646013cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 -- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 msgr2=0x7f6460139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 0x7f6460139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 -- 172.21.15.125:0/947976264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450009a70 con 0x7f64600cccf0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6465600640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 0x7f6460139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.798+0000 7f6464c00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f644800c450 tx=0x7f644800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.799+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64480052d0 con 0x7f64600cccf0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.799+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6448005430 con 0x7f64600cccf0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.799+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448012870 con 0x7f64600cccf0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.799+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64601f2db0 con 0x7f64600cccf0 2024-01-16T14:58:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.799+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64601f3370 con 0x7f64600cccf0 2024-01-16T14:58:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.800+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6460067450 con 0x7f64600cccf0 2024-01-16T14:58:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.801+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f64480129d0 con 0x7f64600cccf0 2024-01-16T14:58:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.802+0000 7f645d600640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 0x7f643806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.802+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64480893c0 con 0x7f64600cccf0 2024-01-16T14:58:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.802+0000 7f6465600640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 0x7f643806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.803+0000 7f6465600640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 0x7f643806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6450036000 tx=0x7f6450030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:06.806+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6448054570 con 0x7f64600cccf0 2024-01-16T14:58:07.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.067+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f64601f3530 con 0x7f64600cccf0 2024-01-16T14:58:07.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.071+0000 7f645d600640 1 -- 172.21.15.125:0/947976264 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6448057b90 con 0x7f64600cccf0 2024-01-16T14:58:07.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.074+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 msgr2=0x7f643806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.074+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 0x7f643806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6450036000 tx=0x7f6450030000 comp rx=0 tx=0).stop 2024-01-16T14:58:07.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.074+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 msgr2=0x7f646013c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.074+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f644800c450 tx=0x7f644800c910 comp rx=0 tx=0).stop 2024-01-16T14:58:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 shutdown_connections 2024-01-16T14:58:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64601447f0 0x7f646013cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806b2c0 0x7f643806d780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64600cccf0 0x7f646013c590 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 --2- 172.21.15.125:0/947976264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646006c470 0x7f6460139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 >> 172.21.15.125:0/947976264 conn(0x7f646006a3e0 msgr2=0x7f64600af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 shutdown_connections 2024-01-16T14:58:07.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.075+0000 7f6466a00640 1 -- 172.21.15.125:0/947976264 wait complete. 2024-01-16T14:58:07.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-16T14:58:07.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.291+0000 7ff0dfe00640 1 Processor -- start 2024-01-16T14:58:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.292+0000 7ff0dfe00640 1 -- start start 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0d80cf9a0 0x7ff0d8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dfe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d815c5f0 con 0x7ff0d813e360 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d806c470 con 0x7ff0d813df80 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dfe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d806c750 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0d80cf9a0 0x7ff0d8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dca00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff0d813e360 0x7ff0d80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41986/0 (socket says 172.21.15.125:41986) 2024-01-16T14:58:07.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.293+0000 7ff0dca00640 1 -- 172.21.15.125:0/492552617 learned_addr learned my addr 172.21.15.125:0/492552617 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:07.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.294+0000 7ff0dde00640 1 -- 172.21.15.125:0/492552617 >> v1:172.21.15.137:6789/0 conn(0x7ff0d813df80 legacy=0x7ff0d815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.294+0000 7ff0dde00640 1 -- 172.21.15.125:0/492552617 >> v1:172.21.15.125:6789/0 conn(0x7ff0d813e360 legacy=0x7ff0d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.294+0000 7ff0d7e00640 1 -- 172.21.15.125:0/492552617 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933045550 0 0) 0x7ff0d815c5f0 con 0x7ff0d813e360 2024-01-16T14:58:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.294+0000 7ff0dde00640 1 -- 172.21.15.125:0/492552617 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d806ca30 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.295+0000 7ff0dde00640 1 --2- 172.21.15.125:0/492552617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d8166030 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097a0 tx=0x7ff0d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a856fc7f2c811db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.295+0000 7ff0d7e00640 1 -- 172.21.15.125:0/492552617 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0030610 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0d7e00640 1 -- 172.21.15.125:0/492552617 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0030770 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0d7e00640 1 -- 172.21.15.125:0/492552617 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0030b00 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 -- 172.21.15.125:0/492552617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 msgr2=0x7ff0d8166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/492552617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d8166030 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097a0 tx=0x7ff0d002b960 comp rx=0 tx=0).stop 2024-01-16T14:58:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 -- 172.21.15.125:0/492552617 shutdown_connections 2024-01-16T14:58:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/492552617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d8166030 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 -- 172.21.15.125:0/492552617 >> 172.21.15.125:0/492552617 conn(0x7ff0d806a800 msgr2=0x7ff0d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 -- 172.21.15.125:0/492552617 shutdown_connections 2024-01-16T14:58:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.296+0000 7ff0dfe00640 1 -- 172.21.15.125:0/492552617 wait complete. 2024-01-16T14:58:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.298+0000 7ff0dfe00640 1 Processor -- start 2024-01-16T14:58:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.298+0000 7ff0dfe00640 1 -- start start 2024-01-16T14:58:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0d813df80 0x7ff0d8070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0d813e360 0x7ff0d80a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d80713d0 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d80716b0 con 0x7ff0d813e360 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8071990 con 0x7ff0d813df80 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57394/0 (socket says 172.21.15.125:57394) 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.299+0000 7ff0dd400640 1 -- 172.21.15.125:0/2910014634 learned_addr learned my addr 172.21.15.125:0/2910014634 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 -- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0d813df80 msgr2=0x7ff0d8070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dca00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0d813df80 0x7ff0d8070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dde00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0d813e360 0x7ff0d80a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0d813df80 0x7ff0d8070cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 -- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0d813e360 msgr2=0x7ff0d80a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0d813e360 0x7ff0d80a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 -- 172.21.15.125:0/2910014634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d0009400 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dd400640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c730 tx=0x7ff0c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003b70 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8003cf0 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003ea0 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.300+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d8071c70 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.301+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d8159f80 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.302+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d80675d0 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.302+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff0c800aa90 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.303+0000 7ff0d5600640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 0x7ff0b006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.303+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0c8089740 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.303+0000 7ff0dca00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 0x7ff0b006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.304+0000 7ff0dca00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 0x7ff0b006d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0007fd0 tx=0x7ff0c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.307+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0c8054870 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.548+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff0d806df00 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.552+0000 7ff0d5600640 1 -- 172.21.15.125:0/2910014634 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff0c8057e90 con 0x7ff0d80cf9a0 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.554+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 msgr2=0x7ff0b006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.554+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 0x7ff0b006d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0007fd0 tx=0x7ff0c0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.554+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 msgr2=0x7ff0d814b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.554+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c730 tx=0x7ff0c800cbf0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 shutdown_connections 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0d813e360 0x7ff0d80a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0d813df80 0x7ff0d8070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b006b2c0 0x7ff0b006d780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 --2- 172.21.15.125:0/2910014634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d80cf9a0 0x7ff0d814b0e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 >> 172.21.15.125:0/2910014634 conn(0x7ff0d806a800 msgr2=0x7ff0d806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 shutdown_connections 2024-01-16T14:58:07.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.555+0000 7ff0dfe00640 1 -- 172.21.15.125:0/2910014634 wait complete. 2024-01-16T14:58:07.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-16T14:58:07.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.776+0000 7f3cdce00640 1 Processor -- start 2024-01-16T14:58:07.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.776+0000 7f3cdce00640 1 -- start start 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cdce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cd80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cdce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3cd815e3b0 0x7f3cd80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cdce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd80cfc30 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cdce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd80b1ee0 con 0x7f3cd815e3b0 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cdce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b2780 con 0x7f3cd8143ae0 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3ccfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3cd80cf7d0 0x7f3cd815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41992/0 (socket says 172.21.15.125:41992) 2024-01-16T14:58:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3ccfe00640 1 -- 172.21.15.125:0/3788643919 learned_addr learned my addr 172.21.15.125:0/3788643919 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669685938 0 0) 0x7f3cd80cfc30 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb8003520 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cd80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.778+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> v2:172.21.15.137:3300/0 conn(0x7f3cd815e3b0 0x7f3cd80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469703908 0 0) 0x7f3cb8003520 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 msgr2=0x7f3cd80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cd80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 >> v2:172.21.15.137:3300/0 conn(0x7f3cd815e3b0 msgr2=0x7f3cd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 --2- 172.21.15.125:0/3788643919 >> v2:172.21.15.137:3300/0 conn(0x7f3cd815e3b0 0x7f3cd80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd815e990 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3cc0001670 con 0x7f3cd80cf7d0 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> v2:172.21.15.137:3300/0 conn(0x7f3cd815e3b0 0x7f3cd80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 >> v1:172.21.15.125:6789/0 conn(0x7f3cd80cf7d0 legacy=0x7f3cd815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb8003970 0x7f3cb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8004d00 0x7f3cb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b1ee0 con 0x7f3cb8004d00 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b2780 con 0x7f3cb8003970 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb800dcc0 con 0x7f3cd8143ae0 2024-01-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.779+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8004d00 0x7f3cb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3ccfe00640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb8003970 0x7f3cb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd6800640 1 -- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 msgr2=0x7f3cb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd8143ae0 0x7f3cb80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd6800640 1 -- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb8003970 msgr2=0x7f3cb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb8003970 0x7f3cb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd6800640 1 -- 172.21.15.125:0/3788643919 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8003520 con 0x7f3cb8004d00 2024-01-16T14:58:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.780+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb8003970 0x7f3cb8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:07.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.781+0000 7f3cd6800640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8004d00 0x7f3cb8005180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00073e0 tx=0x7f3cb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f63667d905024b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.781+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd002cdc0 con 0x7f3cb8004d00 2024-01-16T14:58:07.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.781+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0034440 con 0x7f3cb8004d00 2024-01-16T14:58:07.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.781+0000 7f3cd5400640 1 -- 172.21.15.125:0/3788643919 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd00347b0 con 0x7f3cb8004d00 2024-01-16T14:58:07.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 -- 172.21.15.125:0/3788643919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8004d00 msgr2=0x7f3cb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 --2- 172.21.15.125:0/3788643919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8004d00 0x7f3cb8005180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00073e0 tx=0x7f3cb80057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cd6800640 1 -- 172.21.15.125:0/3788643919 reap_dead start 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 -- 172.21.15.125:0/3788643919 shutdown_connections 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 -- 172.21.15.125:0/3788643919 >> 172.21.15.125:0/3788643919 conn(0x7f3cd806a800 msgr2=0x7f3cd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 -- 172.21.15.125:0/3788643919 shutdown_connections 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.782+0000 7f3cdce00640 1 -- 172.21.15.125:0/3788643919 wait complete. 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.783+0000 7f3cdce00640 1 Processor -- start 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.784+0000 7f3cdce00640 1 -- start start 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cb8004d00 0x7f3cd81f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd81f4f30 0x7f3cd81f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd81f4aa0 con 0x7f3cd81f3a00 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd81f3e90 con 0x7f3cd81f4f30 2024-01-16T14:58:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cdce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd81f4170 con 0x7f3cb8004d00 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57424/0 (socket says 172.21.15.125:57424) 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 -- 172.21.15.125:0/2310321066 learned_addr learned my addr 172.21.15.125:0/2310321066 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cb8004d00 0x7f3cd81f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 -- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cb8004d00 msgr2=0x7f3cd81f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cb8004d00 0x7f3cd81f0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 -- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd81f4f30 msgr2=0x7f3cd81f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd81f4f30 0x7f3cd81f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.785+0000 7f3ccfe00640 1 -- 172.21.15.125:0/2310321066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd81f4450 con 0x7f3cd81f3a00 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3ccfe00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000e700 tx=0x7f3cc000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000ee10 con 0x7f3cd81f3a00 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc000c850 con 0x7f3cd81f3a00 2024-01-16T14:58:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000cc00 con 0x7f3cd81f3a00 2024-01-16T14:58:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd81f4730 con 0x7f3cd81f3a00 2024-01-16T14:58:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.786+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd81f5900 con 0x7f3cd81f3a00 2024-01-16T14:58:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.787+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd81f4be0 con 0x7f3cd81f3a00 2024-01-16T14:58:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.788+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3cc001d460 con 0x7f3cd81f3a00 2024-01-16T14:58:07.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.789+0000 7f3cce000640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 0x7f3cb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.789+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3cc00896c0 con 0x7f3cd81f3a00 2024-01-16T14:58:07.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.790+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 0x7f3cb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.790+0000 7f3cd5e00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 0x7f3cb006d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80adf70 tx=0x7f3cc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:07.794+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cc0054910 con 0x7f3cd81f3a00 2024-01-16T14:58:08.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.037+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f3cd80675b0 con 0x7f3cd81f3a00 2024-01-16T14:58:08.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.041+0000 7f3cce000640 1 -- 172.21.15.125:0/2310321066 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3cc0057f30 con 0x7f3cd81f3a00 2024-01-16T14:58:08.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 msgr2=0x7f3cb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 0x7f3cb006d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80adf70 tx=0x7f3cc8008040 comp rx=0 tx=0).stop 2024-01-16T14:58:08.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 msgr2=0x7f3cd80aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000e700 tx=0x7f3cc000ebc0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 shutdown_connections 2024-01-16T14:58:08.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.044+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd81f4f30 0x7f3cd81f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cb006b2c0 0x7f3cb006d780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cd81f3a00 0x7f3cd80aff70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 --2- 172.21.15.125:0/2310321066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cb8004d00 0x7f3cd81f0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 >> 172.21.15.125:0/2310321066 conn(0x7f3cd806a800 msgr2=0x7f3cd815c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 shutdown_connections 2024-01-16T14:58:08.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.045+0000 7f3cdce00640 1 -- 172.21.15.125:0/2310321066 wait complete. 2024-01-16T14:58:08.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-16T14:58:08.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.251+0000 7f62c7200640 1 Processor -- start 2024-01-16T14:58:08.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.251+0000 7f62c7200640 1 -- start start 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.252+0000 7f62c7200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62c806c470 0x7f62c80cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62c7200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c806ca10 con 0x7f62c806dab0 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62c7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c806ccf0 con 0x7f62c81447f0 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62c7200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c806d690 con 0x7f62c806c470 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62c806c470 0x7f62c80cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62c806c470 0x7f62c80cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57434/0 (socket says 172.21.15.125:57434) 2024-01-16T14:58:08.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 -- 172.21.15.125:0/944643463 learned_addr learned my addr 172.21.15.125:0/944643463 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:08.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62c6800640 1 --1- 172.21.15.125:0/944643463 >> v1:172.21.15.125:6789/0 conn(0x7f62c806dab0 0x7f62c815c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42008/0 (socket says 172.21.15.125:42008) 2024-01-16T14:58:08.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 -- 172.21.15.125:0/944643463 >> v1:172.21.15.137:6789/0 conn(0x7f62c81447f0 legacy=0x7f62c815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 -- 172.21.15.125:0/944643463 >> v1:172.21.15.125:6789/0 conn(0x7f62c806dab0 legacy=0x7f62c815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.253+0000 7f62bfe00640 1 -- 172.21.15.125:0/944643463 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c806cfd0 con 0x7f62c806c470 2024-01-16T14:58:08.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.254+0000 7f62bfe00640 1 --2- 172.21.15.125:0/944643463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c806c470 0x7f62c80cc5e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009870 tx=0x7f62b002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6097360b4c96914 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c5400640 1 -- 172.21.15.125:0/944643463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0031860 con 0x7f62c806c470 2024-01-16T14:58:08.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c5400640 1 -- 172.21.15.125:0/944643463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00319c0 con 0x7f62c806c470 2024-01-16T14:58:08.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c5400640 1 -- 172.21.15.125:0/944643463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0031d50 con 0x7f62c806c470 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 -- 172.21.15.125:0/944643463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c806c470 msgr2=0x7f62c80cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 --2- 172.21.15.125:0/944643463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c806c470 0x7f62c80cc5e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009870 tx=0x7f62b002b900 comp rx=0 tx=0).stop 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 -- 172.21.15.125:0/944643463 shutdown_connections 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 --2- 172.21.15.125:0/944643463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c806c470 0x7f62c80cc5e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 -- 172.21.15.125:0/944643463 >> 172.21.15.125:0/944643463 conn(0x7f62c806a3e0 msgr2=0x7f62c806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 -- 172.21.15.125:0/944643463 shutdown_connections 2024-01-16T14:58:08.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.255+0000 7f62c7200640 1 -- 172.21.15.125:0/944643463 wait complete. 2024-01-16T14:58:08.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.257+0000 7f62c7200640 1 Processor -- start 2024-01-16T14:58:08.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.257+0000 7f62c7200640 1 -- start start 2024-01-16T14:58:08.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 0x7f62c8158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62c806dab0 0x7f62c80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80b0550 con 0x7f62c81447f0 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80c3f60 con 0x7f62c806dab0 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80c4240 con 0x7f62c806c470 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57446/0 (socket says 172.21.15.125:57446) 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.258+0000 7f62c6800640 1 -- 172.21.15.125:0/2999342666 learned_addr learned my addr 172.21.15.125:0/2999342666 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c5e00640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 0x7f62c8158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 -- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 msgr2=0x7f62c8158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62bfe00640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62c806dab0 0x7f62c80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 0x7f62c8158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 -- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62c806dab0 msgr2=0x7f62c80c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62c806dab0 0x7f62c80c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 -- 172.21.15.125:0/2999342666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b00094d0 con 0x7f62c81447f0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c5e00640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 0x7f62c8158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c6800640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c780 tx=0x7f62c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000ce90 con 0x7f62c81447f0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c000a640 con 0x7f62c81447f0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c80c4520 con 0x7f62c81447f0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000a9f0 con 0x7f62c81447f0 2024-01-16T14:58:08.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.259+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c80af790 con 0x7f62c81447f0 2024-01-16T14:58:08.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.261+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8067450 con 0x7f62c81447f0 2024-01-16T14:58:08.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.261+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f62c000ab50 con 0x7f62c81447f0 2024-01-16T14:58:08.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.262+0000 7f62be000640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 0x7f62a006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.262+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f62c0089070 con 0x7f62c81447f0 2024-01-16T14:58:08.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.263+0000 7f62c5e00640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 0x7f62a006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.263+0000 7f62c5e00640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 0x7f62a006d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f62b800a830 tx=0x7f62b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.266+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62c00bbe90 con 0x7f62c81447f0 2024-01-16T14:58:08.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.523+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f62c815c120 con 0x7f62c81447f0 2024-01-16T14:58:08.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.526+0000 7f62be000640 1 -- 172.21.15.125:0/2999342666 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f62c00170a0 con 0x7f62c81447f0 2024-01-16T14:58:08.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 msgr2=0x7f62a006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 0x7f62a006d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f62b800a830 tx=0x7f62b8008040 comp rx=0 tx=0).stop 2024-01-16T14:58:08.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 msgr2=0x7f62c80c69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c780 tx=0x7f62c000cc40 comp rx=0 tx=0).stop 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 shutdown_connections 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62a006b2f0 0x7f62a006d7b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62c81447f0 0x7f62c80c69b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62c806dab0 0x7f62c80c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 --2- 172.21.15.125:0/2999342666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62c806c470 0x7f62c8158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 >> 172.21.15.125:0/2999342666 conn(0x7f62c806a3e0 msgr2=0x7f62c80716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 shutdown_connections 2024-01-16T14:58:08.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.529+0000 7f62c7200640 1 -- 172.21.15.125:0/2999342666 wait complete. 2024-01-16T14:58:08.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.748+0000 7fb6c9200640 1 Processor -- start 2024-01-16T14:58:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.748+0000 7fb6c9200640 1 -- start start 2024-01-16T14:58:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6c415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c9200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c4070c40 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c9200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c406c470 con 0x7fb6c40b3ae0 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4071450 con 0x7fb6c40cf9a0 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb6c40ae5a0 0x7fb6c406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42016/0 (socket says 172.21.15.125:42016) 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.750+0000 7fb6c1e00640 1 -- 172.21.15.125:0/2026926891 learned_addr learned my addr 172.21.15.125:0/2026926891 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6c415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473335583 0 0) 0x7fb6c4070c40 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a4003520 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683610896 0 0) 0x7fb6a4003520 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6c415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6c415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 >> v1:172.21.15.137:6789/0 conn(0x7fb6c40b3ae0 legacy=0x7fb6c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c406c750 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb6b4001670 con 0x7fb6c40ae5a0 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 >> v1:172.21.15.125:6789/0 conn(0x7fb6c40ae5a0 legacy=0x7fb6c406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6a4003f20 0x7fb6a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a4004d00 0x7fb6a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4071450 con 0x7fb6a4004d00 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a400baf0 con 0x7fb6a4003f20 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.751+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a400bcb0 con 0x7fb6c40cf9a0 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a4004d00 0x7fb6a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c1e00640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6b9200640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6a4003f20 0x7fb6a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 -- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6c40cf9a0 0x7fb6a40039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 -- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6a4003f20 msgr2=0x7fb6a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6a4003f20 0x7fb6a40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.752+0000 7fb6c2800640 1 -- 172.21.15.125:0/2026926891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4003520 con 0x7fb6a4004d00 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.753+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a4004d00 0x7fb6a4005180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009fc0 tx=0x7fb6a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8450cbb11798c76e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.753+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc02cc20 con 0x7fb6a4004d00 2024-01-16T14:58:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.753+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc00f040 con 0x7fb6a4004d00 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.753+0000 7fb6c1400640 1 -- 172.21.15.125:0/2026926891 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc034440 con 0x7fb6a4004d00 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 -- 172.21.15.125:0/2026926891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a4004d00 msgr2=0x7fb6a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2026926891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a4004d00 0x7fb6a4005180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009fc0 tx=0x7fb6a40057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c2800640 1 -- 172.21.15.125:0/2026926891 reap_dead start 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 -- 172.21.15.125:0/2026926891 shutdown_connections 2024-01-16T14:58:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 -- 172.21.15.125:0/2026926891 >> 172.21.15.125:0/2026926891 conn(0x7fb6c406a800 msgr2=0x7fb6c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 -- 172.21.15.125:0/2026926891 shutdown_connections 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.754+0000 7fb6c9200640 1 -- 172.21.15.125:0/2026926891 wait complete. 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.755+0000 7fb6c9200640 1 Processor -- start 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 -- start start 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6a4004d00 0x7fb6c415eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c4160520 0x7fb6c4155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c415f060 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.756+0000 7fb6c9200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4155d80 con 0x7fb6c4160520 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c415f300 con 0x7fb6a4004d00 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57456/0 (socket says 172.21.15.125:57456) 2024-01-16T14:58:08.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 -- 172.21.15.125:0/2112648833 learned_addr learned my addr 172.21.15.125:0/2112648833 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c1e00640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6a4004d00 0x7fb6c415eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 -- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6a4004d00 msgr2=0x7fb6c415eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6a4004d00 0x7fb6c415eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 -- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c4160520 msgr2=0x7fb6c4155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c4160520 0x7fb6c4155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 -- 172.21.15.125:0/2112648833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c415f5e0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.757+0000 7fb6c2800640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc037040 tx=0x7fb6bc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.758+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc043070 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.758+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc034ad0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.758+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc034e40 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.758+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c415f8c0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.758+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c41f65a0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.759+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c40675d0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.760+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6bc03e440 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.761+0000 7fb6baa00640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 0x7fb69c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.761+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb6bc0ab250 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.761+0000 7fb6c1e00640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 0x7fb69c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.762+0000 7fb6c1e00640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 0x7fb69c06d7b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4000fb0 tx=0x7fb6b4001470 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.765+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6bc0764c0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.955+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6c4071ce0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.957+0000 7fb6baa00640 1 -- 172.21.15.125:0/2112648833 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12905 (secure 0 0 0) 0x7fb6bc079ae0 con 0x7fb6c41f6ac0 2024-01-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":30,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:58:05.018089+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 msgr2=0x7fb69c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 0x7fb69c06d7b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4000fb0 tx=0x7fb6b4001470 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 msgr2=0x7fb6c4159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc037040 tx=0x7fb6bc009750 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 shutdown_connections 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb69c06b2f0 0x7fb69c06d7b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6c41f6ac0 0x7fb6c4159330 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6c4160520 0x7fb6c4155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 --2- 172.21.15.125:0/2112648833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6a4004d00 0x7fb6c415eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 >> 172.21.15.125:0/2112648833 conn(0x7fb6c406a800 msgr2=0x7fb6c40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 shutdown_connections 2024-01-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:08.960+0000 7fb6c9200640 1 -- 172.21.15.125:0/2112648833 wait complete. 2024-01-16T14:58:08.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:08.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:09.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.178+0000 7fb6d9600640 1 Processor -- start 2024-01-16T14:58:09.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.178+0000 7fb6d9600640 1 -- start start 2024-01-16T14:58:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.180+0000 7fb6d9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.180+0000 7fb6d9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb6d415d770 0x7fb6d415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.180+0000 7fb6d9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d415e140 con 0x7fb6d4143810 2024-01-16T14:58:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.180+0000 7fb6d9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d4143bf0 con 0x7fb6d415d770 2024-01-16T14:58:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.180+0000 7fb6d9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40722c0 con 0x7fb6d4071e40 2024-01-16T14:58:09.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:34380/0 (socket says 172.21.15.125:34380) 2024-01-16T14:58:09.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 -- 172.21.15.125:0/2218014194 learned_addr learned my addr 172.21.15.125:0/2218014194 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:09.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d8c00640 1 --2- 172.21.15.125:0/2218014194 >> v2:172.21.15.137:3300/0 conn(0x7fb6d415d770 0x7fb6d415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 -- 172.21.15.125:0/2218014194 >> v2:172.21.15.137:3300/0 conn(0x7fb6d415d770 msgr2=0x7fb6d415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 --2- 172.21.15.125:0/2218014194 >> v2:172.21.15.137:3300/0 conn(0x7fb6d415d770 0x7fb6d415dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.181+0000 7fb6d3400640 1 -- 172.21.15.125:0/2218014194 >> v1:172.21.15.137:6789/0 conn(0x7fb6d4143810 legacy=0x7fb6d415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.182+0000 7fb6d3400640 1 -- 172.21.15.125:0/2218014194 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d406c730 con 0x7fb6d4071e40 2024-01-16T14:58:09.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.183+0000 7fb6d3400640 1 --2- 172.21.15.125:0/2218014194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009870 tx=0x7fb6bc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=addf6d9217aa21be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.184+0000 7fb6d2a00640 1 -- 172.21.15.125:0/2218014194 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0317c0 con 0x7fb6d4071e40 2024-01-16T14:58:09.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.184+0000 7fb6d2a00640 1 -- 172.21.15.125:0/2218014194 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc031920 con 0x7fb6d4071e40 2024-01-16T14:58:09.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.184+0000 7fb6d2a00640 1 -- 172.21.15.125:0/2218014194 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc031cb0 con 0x7fb6d4071e40 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 -- 172.21.15.125:0/2218014194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 msgr2=0x7fb6d40ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2218014194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009870 tx=0x7fb6bc02b860 comp rx=0 tx=0).stop 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 -- 172.21.15.125:0/2218014194 shutdown_connections 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2218014194 >> v2:172.21.15.137:3300/0 conn(0x7fb6d415d770 0x7fb6d415dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2218014194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4071e40 0x7fb6d40ae170 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 -- 172.21.15.125:0/2218014194 >> 172.21.15.125:0/2218014194 conn(0x7fb6d406a7d0 msgr2=0x7fb6d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 -- 172.21.15.125:0/2218014194 shutdown_connections 2024-01-16T14:58:09.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.185+0000 7fb6d9600640 1 -- 172.21.15.125:0/2218014194 wait complete. 2024-01-16T14:58:09.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.187+0000 7fb6d9600640 1 Processor -- start 2024-01-16T14:58:09.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.187+0000 7fb6d9600640 1 -- start start 2024-01-16T14:58:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4143810 0x7fb6d40ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6d415d770 0x7fb6d415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40af360 con 0x7fb6d4071e40 2024-01-16T14:58:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40af640 con 0x7fb6d415d770 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.188+0000 7fb6d9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40af920 con 0x7fb6d4143810 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57470/0 (socket says 172.21.15.125:57470) 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 -- 172.21.15.125:0/2961097323 learned_addr learned my addr 172.21.15.125:0/2961097323 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3400640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4143810 0x7fb6d40ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d8c00640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6d415d770 0x7fb6d415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 -- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4143810 msgr2=0x7fb6d40ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4143810 0x7fb6d40ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 -- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6d415d770 msgr2=0x7fb6d415cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6d415d770 0x7fb6d415cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 -- 172.21.15.125:0/2961097323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc0094d0 con 0x7fb6d4071e40 2024-01-16T14:58:09.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.189+0000 7fb6d3e00640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400c450 tx=0x7fb6c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.190+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c40052d0 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.190+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c4005430 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.190+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d40afc00 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.190+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4012870 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.190+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d4145330 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.191+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d40b0120 con 0x7fb6d4071e40 2024-01-16T14:58:09.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.192+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6c40129d0 con 0x7fb6d4071e40 2024-01-16T14:58:09.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.193+0000 7fb6cbe00640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 0x7fb6ac06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.193+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb6c40892f0 con 0x7fb6d4071e40 2024-01-16T14:58:09.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.193+0000 7fb6d3400640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 0x7fb6ac06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.194+0000 7fb6d3400640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 0x7fb6ac06d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc036000 tx=0x7fb6bc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.197+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6d40b0120 con 0x7fb6d4071e40 2024-01-16T14:58:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.387+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6d40b0120 con 0x7fb6d4071e40 2024-01-16T14:58:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.388+0000 7fb6cbe00640 1 -- 172.21.15.125:0/2961097323 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6d40b0120 con 0x7fb6d4071e40 2024-01-16T14:58:09.371 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:58:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.391+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 msgr2=0x7fb6ac06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.391+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 0x7fb6ac06d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc036000 tx=0x7fb6bc03d040 comp rx=0 tx=0).stop 2024-01-16T14:58:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 msgr2=0x7fb6d40b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400c450 tx=0x7fb6c400c910 comp rx=0 tx=0).stop 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 shutdown_connections 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6d415d770 0x7fb6d415cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6d4143810 0x7fb6d40ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6ac06b2f0 0x7fb6ac06d7b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 --2- 172.21.15.125:0/2961097323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6d4071e40 0x7fb6d40b0ea0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 >> 172.21.15.125:0/2961097323 conn(0x7fb6d406a7d0 msgr2=0x7fb6d406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 shutdown_connections 2024-01-16T14:58:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.392+0000 7fb6d9600640 1 -- 172.21.15.125:0/2961097323 wait complete. 2024-01-16T14:58:09.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:58:09.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.618+0000 7f6263e00640 1 Processor -- start 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.619+0000 7f6263e00640 1 -- start start 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6264160890 0x7f62640d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6264160c70 0x7f62640d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f62640c6be0 0x7f62640d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640d20f0 con 0x7f6264160c70 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640cf6a0 con 0x7f62640c6be0 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6263e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640d1430 con 0x7f6264160890 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6262000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6264160c70 0x7f62640d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6262000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6264160c70 0x7f62640d1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57482/0 (socket says 172.21.15.125:57482) 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.620+0000 7f6262000640 1 -- 172.21.15.125:0/3693072465 learned_addr learned my addr 172.21.15.125:0/3693072465 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262a00640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6264160890 0x7f62640d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6263400640 1 --2- 172.21.15.125:0/3693072465 >> v2:172.21.15.137:3300/0 conn(0x7f62640c6be0 0x7f62640d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262000640 1 -- 172.21.15.125:0/3693072465 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6264160890 msgr2=0x7f62640d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262000640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6264160890 0x7f62640d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262000640 1 -- 172.21.15.125:0/3693072465 >> v2:172.21.15.137:3300/0 conn(0x7f62640c6be0 msgr2=0x7f62640d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262000640 1 --2- 172.21.15.125:0/3693072465 >> v2:172.21.15.137:3300/0 conn(0x7f62640c6be0 0x7f62640d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.621+0000 7f6262000640 1 -- 172.21.15.125:0/3693072465 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640b4370 con 0x7f6264160c70 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.622+0000 7f6262000640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f62640d1670 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f62640cf2e0 tx=0x7f624c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfdaf1ffbf46cda server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.622+0000 7f6261600640 1 -- 172.21.15.125:0/3693072465 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c031750 con 0x7f6264160c70 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.622+0000 7f6261600640 1 -- 172.21.15.125:0/3693072465 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c0318b0 con 0x7f6264160c70 2024-01-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.622+0000 7f6261600640 1 -- 172.21.15.125:0/3693072465 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c031c40 con 0x7f6264160c70 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 -- 172.21.15.125:0/3693072465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 msgr2=0x7f62640d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f62640d1670 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f62640cf2e0 tx=0x7f624c02b7f0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 -- 172.21.15.125:0/3693072465 shutdown_connections 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 --2- 172.21.15.125:0/3693072465 >> v2:172.21.15.137:3300/0 conn(0x7f62640c6be0 0x7f62640d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f62640d1670 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 --2- 172.21.15.125:0/3693072465 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6264160890 0x7f62640d3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 -- 172.21.15.125:0/3693072465 >> 172.21.15.125:0/3693072465 conn(0x7f62640d3090 msgr2=0x7f62640d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 -- 172.21.15.125:0/3693072465 shutdown_connections 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.623+0000 7f6263e00640 1 -- 172.21.15.125:0/3693072465 wait complete. 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.625+0000 7f6263e00640 1 Processor -- start 2024-01-16T14:58:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.625+0000 7f6263e00640 1 -- start start 2024-01-16T14:58:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62640c6be0 0x7f6264155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 0x7f6264152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264152bd0 con 0x7f6264160c70 2024-01-16T14:58:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626415c300 con 0x7f6264160890 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6264153c50 con 0x7f62640c6be0 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57488/0 (socket says 172.21.15.125:57488) 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.626+0000 7f6263400640 1 -- 172.21.15.125:0/235335477 learned_addr learned my addr 172.21.15.125:0/235335477 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6262000640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 0x7f6264152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6262a00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62640c6be0 0x7f6264155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 -- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62640c6be0 msgr2=0x7f6264155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62640c6be0 0x7f6264155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 -- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 msgr2=0x7f6264152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 0x7f6264152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 -- 172.21.15.125:0/235335477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c0094d0 con 0x7f6264160c70 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6262000640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 0x7f6264152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263400640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f625c007e60 tx=0x7f625c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c014620 con 0x7f6264160c70 2024-01-16T14:58:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c00ee90 con 0x7f6264160c70 2024-01-16T14:58:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c0149e0 con 0x7f6264160c70 2024-01-16T14:58:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62640b7550 con 0x7f6264160c70 2024-01-16T14:58:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.627+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6264154160 con 0x7f6264160c70 2024-01-16T14:58:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.628+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62640b76e0 con 0x7f6264160c70 2024-01-16T14:58:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.629+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f625c014b40 con 0x7f6264160c70 2024-01-16T14:58:09.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.631+0000 7f625aa00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 0x7f623c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.631+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f625c08a230 con 0x7f6264160c70 2024-01-16T14:58:09.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.631+0000 7f6262a00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 0x7f623c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.632+0000 7f6262a00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 0x7f623c06d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6254006fd0 tx=0x7f6254008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.635+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f625c055350 con 0x7f6264160c70 2024-01-16T14:58:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.828+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f62640d0e40 con 0x7f6264160c70 2024-01-16T14:58:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.828+0000 7f625aa00640 1 -- 172.21.15.125:0/235335477 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f625c058970 con 0x7f6264160c70 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:09.811 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:58:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 msgr2=0x7f623c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 0x7f623c06d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6254006fd0 tx=0x7f6254008040 comp rx=0 tx=0).stop 2024-01-16T14:58:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 msgr2=0x7f6264153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f625c007e60 tx=0x7f625c00e4b0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 shutdown_connections 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f623c06b2c0 0x7f623c06d780 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6264160c70 0x7f6264153710 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6264160890 0x7f6264152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 --2- 172.21.15.125:0/235335477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62640c6be0 0x7f6264155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 >> 172.21.15.125:0/235335477 conn(0x7f62640d3090 msgr2=0x7f62640c7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 shutdown_connections 2024-01-16T14:58:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:09.831+0000 7f6263e00640 1 -- 172.21.15.125:0/235335477 wait complete. 2024-01-16T14:58:09.831 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:58:09.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:58:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.069+0000 7f2fdda00640 1 Processor -- start 2024-01-16T14:58:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.069+0000 7f2fdda00640 1 -- start start 2024-01-16T14:58:10.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2fd80b1970 0x7f2fd80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2fd80cf900 0x7f2fd80d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd80cfd90 con 0x7f2fd80b1970 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd80cb8f0 con 0x7f2fd80cf900 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.070+0000 7f2fdda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd815b650 con 0x7f2fd815b270 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2fd80b1970 0x7f2fd80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2fd80b1970 0x7f2fd80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57494/0 (socket says 172.21.15.125:57494) 2024-01-16T14:58:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 -- 172.21.15.125:0/3526327357 learned_addr learned my addr 172.21.15.125:0/3526327357 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 -- 172.21.15.125:0/3526327357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 msgr2=0x7f2fd80b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fdd000640 1 --2- 172.21.15.125:0/3526327357 >> v2:172.21.15.137:3300/0 conn(0x7f2fd80cf900 0x7f2fd80d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 -- 172.21.15.125:0/3526327357 >> v2:172.21.15.137:3300/0 conn(0x7f2fd80cf900 msgr2=0x7f2fd80d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 --2- 172.21.15.125:0/3526327357 >> v2:172.21.15.137:3300/0 conn(0x7f2fd80cf900 0x7f2fd80d0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7400640 1 -- 172.21.15.125:0/3526327357 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80ada50 con 0x7f2fd80b1970 2024-01-16T14:58:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.071+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:10.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.072+0000 7f2fd7400640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80b1970 0x7f2fd80cf3c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2fd806b320 tx=0x7f2fc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869b2e565193bc79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fd6a00640 1 -- 172.21.15.125:0/3526327357 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031870 con 0x7f2fd80b1970 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fd6a00640 1 -- 172.21.15.125:0/3526327357 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00319d0 con 0x7f2fd80b1970 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fd6a00640 1 -- 172.21.15.125:0/3526327357 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031d60 con 0x7f2fd80b1970 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 -- 172.21.15.125:0/3526327357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80b1970 msgr2=0x7f2fd80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80b1970 0x7f2fd80cf3c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2fd806b320 tx=0x7f2fc002b910 comp rx=0 tx=0).stop 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 -- 172.21.15.125:0/3526327357 shutdown_connections 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 --2- 172.21.15.125:0/3526327357 >> v2:172.21.15.137:3300/0 conn(0x7f2fd80cf900 0x7f2fd80d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80b1970 0x7f2fd80cf3c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 --2- 172.21.15.125:0/3526327357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 -- 172.21.15.125:0/3526327357 >> 172.21.15.125:0/3526327357 conn(0x7f2fd8099a60 msgr2=0x7f2fd8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.073+0000 7f2fdda00640 1 -- 172.21.15.125:0/3526327357 shutdown_connections 2024-01-16T14:58:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.074+0000 7f2fdda00640 1 -- 172.21.15.125:0/3526327357 wait complete. 2024-01-16T14:58:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.075+0000 7f2fdda00640 1 Processor -- start 2024-01-16T14:58:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.075+0000 7f2fdda00640 1 -- start start 2024-01-16T14:58:10.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 0x7f2fd8157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8157990 con 0x7f2fd80cf900 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c98f0 con 0x7f2fd80b1970 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.076+0000 7f2fdda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c9bd0 con 0x7f2fd815b270 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57504/0 (socket says 172.21.15.125:57504) 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 -- 172.21.15.125:0/2662738669 learned_addr learned my addr 172.21.15.125:0/2662738669 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fdd000640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 0x7f2fd8157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 -- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 msgr2=0x7f2fd80ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 -- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 msgr2=0x7f2fd8157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 0x7f2fd8157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7400640 1 -- 172.21.15.125:0/2662738669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00094d0 con 0x7f2fd80cf900 2024-01-16T14:58:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 0x7f2fd8157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.077+0000 7f2fdd000640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fd7400640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0000c00 tx=0x7f2fc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031840 con 0x7f2fd80cf900 2024-01-16T14:58:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00443f0 con 0x7f2fd80cf900 2024-01-16T14:58:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00447a0 con 0x7f2fd80cf900 2024-01-16T14:58:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd80c9eb0 con 0x7f2fd80cf900 2024-01-16T14:58:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.078+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd80ca470 con 0x7f2fd80cf900 2024-01-16T14:58:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.079+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd80ca660 con 0x7f2fd80cf900 2024-01-16T14:58:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.080+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2fc004d450 con 0x7f2fd80cf900 2024-01-16T14:58:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.081+0000 7f2fcfe00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 0x7f2fb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.081+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2fc002d040 con 0x7f2fd80cf900 2024-01-16T14:58:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.081+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 0x7f2fb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.082+0000 7f2fd7e00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 0x7f2fb006d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8006fd0 tx=0x7f2fc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.085+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fc0077180 con 0x7f2fd80cf900 2024-01-16T14:58:10.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:10.275+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2fd8067b60 con 0x7f2fd80cf900 2024-01-16T14:58:11.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.072+0000 7f2fcfe00640 1 -- 172.21.15.125:0/2662738669 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2fc00350b0 con 0x7f2fd80cf900 2024-01-16T14:58:11.054 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.075+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 msgr2=0x7f2fb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.075+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 0x7f2fb006d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8006fd0 tx=0x7f2fc8008040 comp rx=0 tx=0).stop 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.075+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 msgr2=0x7f2fd80cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.075+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0000c00 tx=0x7f2fc00046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 shutdown_connections 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd815b270 0x7f2fd80ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fb006b2c0 0x7f2fb006d780 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fd80cf900 0x7f2fd80cca80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 --2- 172.21.15.125:0/2662738669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd80b1970 0x7f2fd8157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 >> 172.21.15.125:0/2662738669 conn(0x7f2fd8099a60 msgr2=0x7f2fd806be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 shutdown_connections 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.076+0000 7f2fdda00640 1 -- 172.21.15.125:0/2662738669 wait complete. 2024-01-16T14:58:11.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:58:11.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.296+0000 7f3a94a00640 1 Processor -- start 2024-01-16T14:58:11.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.296+0000 7f3a94a00640 1 -- start start 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.297+0000 7f3a94a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3a90143c40 0x7f3a90071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a94a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a90070e50 con 0x7f3a900717d0 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a94a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a90144020 con 0x7f3a900ccc10 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a94a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a900723c0 con 0x7f3a90143c40 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a87e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3a900717d0 0x7f3a900cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42018/0 (socket says 172.21.15.125:42018) 2024-01-16T14:58:11.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a87e00640 1 -- 172.21.15.125:0/1900658132 learned_addr learned my addr 172.21.15.125:0/1900658132 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680288575 0 0) 0x7f3a90070e50 con 0x7f3a900717d0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a70003520 con 0x7f3a900717d0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.298+0000 7f3a8e800640 1 --2- 172.21.15.125:0/1900658132 >> v2:172.21.15.137:3300/0 conn(0x7f3a90143c40 0x7f3a90071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470566623 0 0) 0x7f3a70003520 con 0x7f3a900717d0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 >> v2:172.21.15.137:3300/0 conn(0x7f3a90143c40 msgr2=0x7f3a90071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 --2- 172.21.15.125:0/1900658132 >> v2:172.21.15.137:3300/0 conn(0x7f3a90143c40 0x7f3a90071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 >> v1:172.21.15.137:6789/0 conn(0x7f3a900ccc10 legacy=0x7f3a9006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90071e00 con 0x7f3a900717d0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3a78001670 con 0x7f3a900717d0 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 >> v1:172.21.15.125:6789/0 conn(0x7f3a900717d0 legacy=0x7f3a900cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a70003970 0x7f3a70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a700047c0 0x7f3a70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 0x7f3a7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900723c0 con 0x7f3a700052a0 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7000c170 con 0x7f3a70003970 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7000c330 con 0x7f3a700047c0 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8e800640 1 --2- 172.21.15.125:0/1900658132 >> v2:172.21.15.137:3300/0 conn(0x7f3a90143c40 0x7f3a90071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 0x7f3a7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 -- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a700047c0 msgr2=0x7f3a70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a700047c0 0x7f3a70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 -- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a70003970 msgr2=0x7f3a70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a70003970 0x7f3a70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.299+0000 7f3a8f200640 1 -- 172.21.15.125:0/1900658132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a70003520 con 0x7f3a700052a0 2024-01-16T14:58:11.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.300+0000 7f3a8f200640 1 -- 172.21.15.125:0/1900658132 reap_dead start 2024-01-16T14:58:11.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 0x7f3a7000b980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a88001e20 tx=0x7f3a700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2741be65fd9764f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88003b70 con 0x7f3a700052a0 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a88003cd0 con 0x7f3a700052a0 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a8de00640 1 -- 172.21.15.125:0/1900658132 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88005310 con 0x7f3a700052a0 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a94a00640 1 -- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 msgr2=0x7f3a7000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 0x7f3a7000b980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a88001e20 tx=0x7f3a700057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a94a00640 1 -- 172.21.15.125:0/1900658132 shutdown_connections 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1900658132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a700052a0 0x7f3a7000b980 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.301+0000 7f3a94a00640 1 -- 172.21.15.125:0/1900658132 >> 172.21.15.125:0/1900658132 conn(0x7f3a9006a800 msgr2=0x7f3a9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.302+0000 7f3a94a00640 1 -- 172.21.15.125:0/1900658132 shutdown_connections 2024-01-16T14:58:11.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.302+0000 7f3a94a00640 1 -- 172.21.15.125:0/1900658132 wait complete. 2024-01-16T14:58:11.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.303+0000 7f3a94a00640 1 Processor -- start 2024-01-16T14:58:11.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.303+0000 7f3a94a00640 1 -- start start 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a700052a0 0x7f3a9015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 0x7f3a90142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900ad560 con 0x7f3a900ae0c0 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9015f0a0 con 0x7f3a700052a0 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a94a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900ac4c0 con 0x7f3a901f04d0 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.304+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57526/0 (socket says 172.21.15.125:57526) 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 -- 172.21.15.125:0/1934477871 learned_addr learned my addr 172.21.15.125:0/1934477871 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a8e800640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a700052a0 0x7f3a9015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 0x7f3a90142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 -- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 msgr2=0x7f3a90142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 0x7f3a90142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 -- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a700052a0 msgr2=0x7f3a9015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a700052a0 0x7f3a9015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 -- 172.21.15.125:0/1934477871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900ac7a0 con 0x7f3a900ae0c0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a8f200640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 0x7f3a90142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.305+0000 7f3a87e00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3a78001eb0 tx=0x7f3a7800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.306+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a780074e0 con 0x7f3a900ae0c0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.306+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a78007640 con 0x7f3a900ae0c0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.306+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78004a00 con 0x7f3a900ae0c0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.306+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a900aca80 con 0x7f3a900ae0c0 2024-01-16T14:58:11.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.306+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a900ad040 con 0x7f3a900ae0c0 2024-01-16T14:58:11.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.307+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a90067450 con 0x7f3a900ae0c0 2024-01-16T14:58:11.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.308+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3a78004b60 con 0x7f3a900ae0c0 2024-01-16T14:58:11.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.309+0000 7f3a86a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 0x7f3a6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.309+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3a78088de0 con 0x7f3a900ae0c0 2024-01-16T14:58:11.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.309+0000 7f3a8e800640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 0x7f3a6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.310+0000 7f3a8e800640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 0x7f3a6806d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.314+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a78053e30 con 0x7f3a900ae0c0 2024-01-16T14:58:11.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:11.507+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3a9015f2b0 con 0x7f3a900ae0c0 2024-01-16T14:58:12.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.087+0000 7f3a86a00640 1 -- 172.21.15.125:0/1934477871 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3a78018020 con 0x7f3a900ae0c0 2024-01-16T14:58:12.070 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.091+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 msgr2=0x7f3a6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.091+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 0x7f3a6806d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80008040 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.091+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 msgr2=0x7f3a90142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.091+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3a78001eb0 tx=0x7f3a7800cc00 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 shutdown_connections 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a901f04d0 0x7f3a90142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a6806b2c0 0x7f3a6806d780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a900ae0c0 0x7f3a90142350 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 --2- 172.21.15.125:0/1934477871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a700052a0 0x7f3a9015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.092+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 >> 172.21.15.125:0/1934477871 conn(0x7f3a9006a800 msgr2=0x7f3a9006d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.093+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 shutdown_connections 2024-01-16T14:58:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.093+0000 7f3a94a00640 1 -- 172.21.15.125:0/1934477871 wait complete. 2024-01-16T14:58:12.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:58:12.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.304+0000 7f3ffee00640 1 Processor -- start 2024-01-16T14:58:12.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.305+0000 7f3ffee00640 1 -- start start 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ffee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3ff80cf9a0 0x7f3ff815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ffee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3ff80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ffee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff80b1ee0 con 0x7f3ff8143ae0 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ffee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff80b2780 con 0x7f3ff80cf9a0 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ffee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815e990 con 0x7f3ff815e1e0 2024-01-16T14:58:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ff7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3ff8143ae0 0x7f3ff80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42034/0 (socket says 172.21.15.125:42034) 2024-01-16T14:58:12.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.306+0000 7f3ff7e00640 1 -- 172.21.15.125:0/2752384315 learned_addr learned my addr 172.21.15.125:0/2752384315 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff7400640 1 --2- 172.21.15.125:0/2752384315 >> v2:172.21.15.137:3300/0 conn(0x7f3ff80cf9a0 0x7f3ff815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ffce00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3ff80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174875199 0 0) 0x7f3ff80b1ee0 con 0x7f3ff8143ae0 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd8003520 con 0x7f3ff8143ae0 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587146278 0 0) 0x7f3fd8003520 con 0x7f3ff8143ae0 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 msgr2=0x7f3ff80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3ff80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 >> v2:172.21.15.137:3300/0 conn(0x7f3ff80cf9a0 msgr2=0x7f3ff815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 --2- 172.21.15.125:0/2752384315 >> v2:172.21.15.137:3300/0 conn(0x7f3ff80cf9a0 0x7f3ff815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80b2130 con 0x7f3ff8143ae0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3fe8001670 con 0x7f3ff8143ae0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 >> v1:172.21.15.125:6789/0 conn(0x7f3ff8143ae0 legacy=0x7f3ff80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fd8003970 0x7f3fd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3fd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3fd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80b2780 con 0x7f3fd80047c0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815e990 con 0x7f3fd8003970 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.307+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd800db90 con 0x7f3ff815e1e0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3fd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7e00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fd8003970 0x7f3fd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 -- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 msgr2=0x7f3fd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ffce00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3fd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff815e1e0 0x7f3fd8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 -- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fd8003970 msgr2=0x7f3fd8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fd8003970 0x7f3fd8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7400640 1 -- 172.21.15.125:0/2752384315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8003520 con 0x7f3fd80047c0 2024-01-16T14:58:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.308+0000 7f3ff7e00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fd8003970 0x7f3fd8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.309+0000 7f3ff7400640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3fd8004c40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009fc0 tx=0x7f3fd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d729abb48775fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.309+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe002cc60 con 0x7f3fd80047c0 2024-01-16T14:58:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.309+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe000f040 con 0x7f3fd80047c0 2024-01-16T14:58:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.309+0000 7f3ff6a00640 1 -- 172.21.15.125:0/2752384315 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0034440 con 0x7f3fd80047c0 2024-01-16T14:58:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 -- 172.21.15.125:0/2752384315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 msgr2=0x7f3fd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 --2- 172.21.15.125:0/2752384315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3fd8004c40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009fc0 tx=0x7f3fd80057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffce00640 1 -- 172.21.15.125:0/2752384315 reap_dead start 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 -- 172.21.15.125:0/2752384315 shutdown_connections 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 -- 172.21.15.125:0/2752384315 >> 172.21.15.125:0/2752384315 conn(0x7f3ff806a800 msgr2=0x7f3ff806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 -- 172.21.15.125:0/2752384315 shutdown_connections 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.310+0000 7f3ffee00640 1 -- 172.21.15.125:0/2752384315 wait complete. 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.312+0000 7f3ffee00640 1 Processor -- start 2024-01-16T14:58:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.312+0000 7f3ffee00640 1 -- start start 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff8134550 0x7f3ff8159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ff8135bb0 0x7f3ff806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff8137530 con 0x7f3fd80047c0 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff8134b10 con 0x7f3ff8135bb0 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ffee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff8134df0 con 0x7f3ff8134550 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.313+0000 7f3ff7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57544/0 (socket says 172.21.15.125:57544) 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 -- 172.21.15.125:0/1302446467 learned_addr learned my addr 172.21.15.125:0/1302446467 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 -- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff8134550 msgr2=0x7f3ff8159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff8134550 0x7f3ff8159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 -- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ff8135bb0 msgr2=0x7f3ff806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ff8135bb0 0x7f3ff806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 -- 172.21.15.125:0/1302446467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff81350d0 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.314+0000 7f3ff7e00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800e820 tx=0x7f3fe800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.315+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800c7f0 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.315+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe800c950 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.315+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800ccc0 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.315+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff81353b0 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.315+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff8159db0 con 0x7f3fd80047c0 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.316+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff81358d0 con 0x7f3fd80047c0 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.317+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3fe801d460 con 0x7f3fd80047c0 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.318+0000 7f3fefe00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 0x7f3fd006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.319+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3fe8089ba0 con 0x7f3fd80047c0 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.319+0000 7f3ff7400640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 0x7f3fd006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.319+0000 7f3ff7400640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 0x7f3fd006d7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0036000 tx=0x7f3fe002e070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.324+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ff81358d0 con 0x7f3fd80047c0 2024-01-16T14:58:12.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:12.546+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3ff81358d0 con 0x7f3fd80047c0 2024-01-16T14:58:13.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.101+0000 7f3fefe00640 1 -- 172.21.15.125:0/1302446467 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3ff81358d0 con 0x7f3fd80047c0 2024-01-16T14:58:13.085 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 4 and data pool 5 2024-01-16T14:58:13.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.108+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 msgr2=0x7f3fd006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.108+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 0x7f3fd006d7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0036000 tx=0x7f3fe002e070 comp rx=0 tx=0).stop 2024-01-16T14:58:13.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.108+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 msgr2=0x7f3ff8136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.108+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800e820 tx=0x7f3fe800ece0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 shutdown_connections 2024-01-16T14:58:13.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ff8135bb0 0x7f3ff806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ff8134550 0x7f3ff8159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3fd006b2f0 0x7f3fd006d7b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 --2- 172.21.15.125:0/1302446467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fd80047c0 0x7f3ff8136ff0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.109+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 >> 172.21.15.125:0/1302446467 conn(0x7f3ff806a800 msgr2=0x7f3ff8142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.110+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 shutdown_connections 2024-01-16T14:58:13.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.110+0000 7f3ffee00640 1 -- 172.21.15.125:0/1302446467 wait complete. 2024-01-16T14:58:13.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.322+0000 7f0343600640 1 Processor -- start 2024-01-16T14:58:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.322+0000 7f0343600640 1 -- start start 2024-01-16T14:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0343600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0343600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c15c5f0 con 0x7f033c13df80 2024-01-16T14:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0343600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c06c470 con 0x7f033c15f9c0 2024-01-16T14:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0343600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033c06c750 con 0x7f033c13e360 2024-01-16T14:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f033be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0340c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f033c13df80 0x7f033c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42044/0 (socket says 172.21.15.125:42044) 2024-01-16T14:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.323+0000 7f0340c00640 1 -- 172.21.15.125:0/2714990849 learned_addr learned my addr 172.21.15.125:0/2714990849 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.324+0000 7f033be00640 1 -- 172.21.15.125:0/2714990849 >> v1:172.21.15.137:6789/0 conn(0x7f033c15f9c0 legacy=0x7f033c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.324+0000 7f033be00640 1 -- 172.21.15.125:0/2714990849 >> v1:172.21.15.125:6789/0 conn(0x7f033c13df80 legacy=0x7f033c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.324+0000 7f033be00640 1 -- 172.21.15.125:0/2714990849 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c06ca30 con 0x7f033c13e360 2024-01-16T14:58:13.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.324+0000 7f033b400640 1 -- 172.21.15.125:0/2714990849 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992316836 0 0) 0x7f033c15c5f0 con 0x7f033c13df80 2024-01-16T14:58:13.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.325+0000 7f033be00640 1 --2- 172.21.15.125:0/2714990849 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c0cf290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f03240097a0 tx=0x7f032402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba27e273b86b9dac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.325+0000 7f033b400640 1 -- 172.21.15.125:0/2714990849 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324031830 con 0x7f033c13e360 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.325+0000 7f033b400640 1 -- 172.21.15.125:0/2714990849 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0324031990 con 0x7f033c13e360 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.325+0000 7f033b400640 1 -- 172.21.15.125:0/2714990849 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324031d20 con 0x7f033c13e360 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 -- 172.21.15.125:0/2714990849 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 msgr2=0x7f033c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 --2- 172.21.15.125:0/2714990849 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c0cf290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f03240097a0 tx=0x7f032402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 -- 172.21.15.125:0/2714990849 shutdown_connections 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 --2- 172.21.15.125:0/2714990849 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 -- 172.21.15.125:0/2714990849 >> 172.21.15.125:0/2714990849 conn(0x7f033c06a800 msgr2=0x7f033c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 -- 172.21.15.125:0/2714990849 shutdown_connections 2024-01-16T14:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.326+0000 7f0343600640 1 -- 172.21.15.125:0/2714990849 wait complete. 2024-01-16T14:58:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.327+0000 7f0343600640 1 Processor -- start 2024-01-16T14:58:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.328+0000 7f0343600640 1 -- start start 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.328+0000 7f0343600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 0x7f033c13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.328+0000 7f0343600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0343600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f033c15f9c0 0x7f033c1e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0343600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c1e8c00 con 0x7f033c13e360 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0343600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c13dc50 con 0x7f033c13df80 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0343600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c1e8360 con 0x7f033c15f9c0 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57566/0 (socket says 172.21.15.125:57566) 2024-01-16T14:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 -- 172.21.15.125:0/3659314629 learned_addr learned my addr 172.21.15.125:0/3659314629 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:13.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0340c00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 0x7f033c13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 -- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f033c15f9c0 msgr2=0x7f033c1e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0341600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f033c15f9c0 0x7f033c1e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f033c15f9c0 0x7f033c1e7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 -- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 msgr2=0x7f033c13d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 0x7f033c13d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f033be00640 1 -- 172.21.15.125:0/3659314629 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324009400 con 0x7f033c13e360 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.329+0000 7f0340c00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 0x7f033c13d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f033be00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c00 tx=0x7f03240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324004230 con 0x7f033c13e360 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0324004390 con 0x7f033c13e360 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032403a940 con 0x7f033c13e360 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c1e8640 con 0x7f033c13e360 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.330+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c152710 con 0x7f033c13e360 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.331+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c152570 con 0x7f033c13e360 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.332+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f032403aaa0 con 0x7f033c13e360 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.333+0000 7f0338c00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 0x7f031406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.333+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03240ac390 con 0x7f033c13e360 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.333+0000 7f0340c00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 0x7f031406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.334+0000 7f0340c00640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 0x7f031406d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f60 tx=0x7f032c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.338+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f033c152570 con 0x7f033c13e360 2024-01-16T14:58:13.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.524+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f033c152570 con 0x7f033c13e360 2024-01-16T14:58:13.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.526+0000 7f0338c00640 1 -- 172.21.15.125:0/3659314629 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f033c152570 con 0x7f033c13e360 2024-01-16T14:58:13.509 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:13.509 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":33,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:58:12.549006+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:10.324911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:11.557252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:13.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 msgr2=0x7f031406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 0x7f031406d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f60 tx=0x7f032c009040 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 msgr2=0x7f033c1e77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c00 tx=0x7f03240046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 shutdown_connections 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f033c15f9c0 0x7f033c1e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f031406b2c0 0x7f031406d780 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f033c13e360 0x7f033c1e77a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 --2- 172.21.15.125:0/3659314629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f033c13df80 0x7f033c13d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 >> 172.21.15.125:0/3659314629 conn(0x7f033c06a800 msgr2=0x7f033c0aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 shutdown_connections 2024-01-16T14:58:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.529+0000 7f0343600640 1 -- 172.21.15.125:0/3659314629 wait complete. 2024-01-16T14:58:13.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:13.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.739+0000 7f4f20400640 1 Processor -- start 2024-01-16T14:58:13.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.739+0000 7f4f20400640 1 -- start start 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f20400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4f180ae5a0 0x7f4f1815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f20400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4f180cf290 0x7f4f180cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f20400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f18070c40 con 0x7f4f180b3ae0 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f20400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f18071450 con 0x7f4f180ae5a0 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f20400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180cfc60 con 0x7f4f180cf290 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f1da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4f180ae5a0 0x7f4f1815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f1da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4f180ae5a0 0x7f4f1815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57582/0 (socket says 172.21.15.125:57582) 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.741+0000 7f4f1da00640 1 -- 172.21.15.125:0/2920366278 learned_addr learned my addr 172.21.15.125:0/2920366278 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.742+0000 7f4f1da00640 1 -- 172.21.15.125:0/2920366278 >> v2:172.21.15.137:3300/0 conn(0x7f4f180cf290 msgr2=0x7f4f180cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:13.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.742+0000 7f4f1da00640 1 --2- 172.21.15.125:0/2920366278 >> v2:172.21.15.137:3300/0 conn(0x7f4f180cf290 0x7f4f180cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.742+0000 7f4f1da00640 1 -- 172.21.15.125:0/2920366278 >> v1:172.21.15.137:6789/0 conn(0x7f4f180b3ae0 legacy=0x7f4f1806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:13.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.742+0000 7f4f1da00640 1 -- 172.21.15.125:0/2920366278 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1806c470 con 0x7f4f180ae5a0 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.743+0000 7f4f1da00640 1 --2- 172.21.15.125:0/2920366278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180ae5a0 0x7f4f1815e430 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4f180b3f70 tx=0x7f4f0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f616241a7aa2e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.743+0000 7f4f17e00640 1 -- 172.21.15.125:0/2920366278 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08030860 con 0x7f4f180ae5a0 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.743+0000 7f4f17e00640 1 -- 172.21.15.125:0/2920366278 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f080309c0 con 0x7f4f180ae5a0 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.743+0000 7f4f17e00640 1 -- 172.21.15.125:0/2920366278 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08030d50 con 0x7f4f180ae5a0 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 -- 172.21.15.125:0/2920366278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180ae5a0 msgr2=0x7f4f1815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 --2- 172.21.15.125:0/2920366278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180ae5a0 0x7f4f1815e430 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4f180b3f70 tx=0x7f4f0802b9a0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 -- 172.21.15.125:0/2920366278 shutdown_connections 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 --2- 172.21.15.125:0/2920366278 >> v2:172.21.15.137:3300/0 conn(0x7f4f180cf290 0x7f4f180cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 --2- 172.21.15.125:0/2920366278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180ae5a0 0x7f4f1815e430 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 -- 172.21.15.125:0/2920366278 >> 172.21.15.125:0/2920366278 conn(0x7f4f1806a800 msgr2=0x7f4f1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 -- 172.21.15.125:0/2920366278 shutdown_connections 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.744+0000 7f4f20400640 1 -- 172.21.15.125:0/2920366278 wait complete. 2024-01-16T14:58:13.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.746+0000 7f4f20400640 1 Processor -- start 2024-01-16T14:58:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.746+0000 7f4f20400640 1 -- start start 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f180ae5a0 0x7f4f1813e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f180cf290 0x7f4f181447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18158d20 con 0x7f4f180b3ae0 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f181f2b30 con 0x7f4f180ae5a0 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f20400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180cdf50 con 0x7f4f180cf290 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57598/0 (socket says 172.21.15.125:57598) 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 -- 172.21.15.125:0/2467209625 learned_addr learned my addr 172.21.15.125:0/2467209625 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 -- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f180cf290 msgr2=0x7f4f181447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f180cf290 0x7f4f181447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 -- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f180ae5a0 msgr2=0x7f4f1813e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f180ae5a0 0x7f4f1813e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 -- 172.21.15.125:0/2467209625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080094d0 con 0x7f4f180b3ae0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.747+0000 7f4f1d000640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c410 tx=0x7f4f0000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.748+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f000035f0 con 0x7f4f180b3ae0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.748+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f00003750 con 0x7f4f180b3ae0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.748+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00003b40 con 0x7f4f180b3ae0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.748+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f180ce230 con 0x7f4f180b3ae0 2024-01-16T14:58:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.748+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f180ce7f0 con 0x7f4f180b3ae0 2024-01-16T14:58:13.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.749+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18140b60 con 0x7f4f180b3ae0 2024-01-16T14:58:13.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.750+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4f00003ca0 con 0x7f4f180b3ae0 2024-01-16T14:58:13.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.751+0000 7f4f15600640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 0x7f4ef006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.751+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f00088e80 con 0x7f4f180b3ae0 2024-01-16T14:58:13.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.751+0000 7f4f1da00640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 0x7f4ef006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.752+0000 7f4f1da00640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 0x7f4ef006d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f08036000 tx=0x7f4f0803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.756+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f18140b60 con 0x7f4f180b3ae0 2024-01-16T14:58:13.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.934+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f18140b60 con 0x7f4f180b3ae0 2024-01-16T14:58:13.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.937+0000 7f4f15600640 1 -- 172.21.15.125:0/2467209625 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f4f18140b60 con 0x7f4f180b3ae0 2024-01-16T14:58:13.920 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:13.920 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":33,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:58:12.549006+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:10.324911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:11.557252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:13.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 msgr2=0x7f4ef006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 0x7f4ef006d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f08036000 tx=0x7f4f0803b040 comp rx=0 tx=0).stop 2024-01-16T14:58:13.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 msgr2=0x7f4f1813ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c410 tx=0x7f4f0000c8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 shutdown_connections 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f180cf290 0x7f4f181447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ef006b2c0 0x7f4ef006d780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f180b3ae0 0x7f4f1813ead0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 --2- 172.21.15.125:0/2467209625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f180ae5a0 0x7f4f1813e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.940+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 >> 172.21.15.125:0/2467209625 conn(0x7f4f1806a800 msgr2=0x7f4f18160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.941+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 shutdown_connections 2024-01-16T14:58:13.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:13.941+0000 7f4f20400640 1 -- 172.21.15.125:0/2467209625 wait complete. 2024-01-16T14:58:13.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:58:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.169+0000 7f029e400640 1 Processor -- start 2024-01-16T14:58:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.169+0000 7f029e400640 1 -- start start 2024-01-16T14:58:14.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02980ccb50 0x7f0298070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f029815bf10 0x7f0298143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029e400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029815f4e0 con 0x7f02980adf20 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029e400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029815f970 con 0x7f02980ccb50 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029e400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0298071130 con 0x7f029815bf10 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f0297e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02980ccb50 0x7f0298070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029d000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f02980adf20 0x7f02980cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42056/0 (socket says 172.21.15.125:42056) 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f0297e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02980ccb50 0x7f0298070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57604/0 (socket says 172.21.15.125:57604) 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.171+0000 7f029d000640 1 -- 172.21.15.125:0/4219062875 learned_addr learned my addr 172.21.15.125:0/4219062875 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.172+0000 7f0297e00640 1 -- 172.21.15.125:0/4219062875 >> v2:172.21.15.137:3300/0 conn(0x7f029815bf10 msgr2=0x7f0298143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.172+0000 7f0297e00640 1 --2- 172.21.15.125:0/4219062875 >> v2:172.21.15.137:3300/0 conn(0x7f029815bf10 0x7f0298143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.172+0000 7f0297e00640 1 -- 172.21.15.125:0/4219062875 >> v1:172.21.15.125:6789/0 conn(0x7f02980adf20 legacy=0x7f02980cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.172+0000 7f0297e00640 1 -- 172.21.15.125:0/4219062875 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029815ebb0 con 0x7f02980ccb50 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f0297e00640 1 --2- 172.21.15.125:0/4219062875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02980ccb50 0x7f0298070a30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f029806ff20 tx=0x7f028002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dd5bd3cbfb786b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f0297400640 1 -- 172.21.15.125:0/4219062875 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028002e070 con 0x7f02980ccb50 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f0297400640 1 -- 172.21.15.125:0/4219062875 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02800315a0 con 0x7f02980ccb50 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f0297400640 1 -- 172.21.15.125:0/4219062875 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280031930 con 0x7f02980ccb50 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f029e400640 1 -- 172.21.15.125:0/4219062875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02980ccb50 msgr2=0x7f0298070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.173+0000 7f029e400640 1 --2- 172.21.15.125:0/4219062875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02980ccb50 0x7f0298070a30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f029806ff20 tx=0x7f028002b840 comp rx=0 tx=0).stop 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 -- 172.21.15.125:0/4219062875 shutdown_connections 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 --2- 172.21.15.125:0/4219062875 >> v2:172.21.15.137:3300/0 conn(0x7f029815bf10 0x7f0298143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 --2- 172.21.15.125:0/4219062875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02980ccb50 0x7f0298070a30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 -- 172.21.15.125:0/4219062875 >> 172.21.15.125:0/4219062875 conn(0x7f029806a7d0 msgr2=0x7f029806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 -- 172.21.15.125:0/4219062875 shutdown_connections 2024-01-16T14:58:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.174+0000 7f029e400640 1 -- 172.21.15.125:0/4219062875 wait complete. 2024-01-16T14:58:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.175+0000 7f029e400640 1 Processor -- start 2024-01-16T14:58:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 -- start start 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02980adf20 0x7f02980c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 0x7f029815a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02980c4840 con 0x7f029815bf10 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02980c51a0 con 0x7f02980ccb50 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.176+0000 7f029e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02980c3ea0 con 0x7f02980adf20 2024-01-16T14:58:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57608/0 (socket says 172.21.15.125:57608) 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 -- 172.21.15.125:0/1484766694 learned_addr learned my addr 172.21.15.125:0/1484766694 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f0297e00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 0x7f029815a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 -- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02980adf20 msgr2=0x7f02980c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029d000640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02980adf20 0x7f02980c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02980adf20 0x7f02980c4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 -- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 msgr2=0x7f029815a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 0x7f029815a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 -- 172.21.15.125:0/1484766694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0280009400 con 0x7f029815bf10 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f0297e00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 0x7f029815a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.177+0000 7f029da00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f029000b970 tx=0x7f029000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.178+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029000cb40 con 0x7f029815bf10 2024-01-16T14:58:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.178+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029000ccc0 con 0x7f029815bf10 2024-01-16T14:58:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.178+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0290014680 con 0x7f029815bf10 2024-01-16T14:58:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.178+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02980c4180 con 0x7f029815bf10 2024-01-16T14:58:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.178+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029815ac20 con 0x7f029815bf10 2024-01-16T14:58:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.179+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02980c4980 con 0x7f029815bf10 2024-01-16T14:58:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.180+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f029001d3b0 con 0x7f029815bf10 2024-01-16T14:58:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.181+0000 7f0294c00640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 0x7f027006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.181+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0290089af0 con 0x7f029815bf10 2024-01-16T14:58:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.181+0000 7f029d000640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 0x7f027006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.182+0000 7f029d000640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 0x7f027006d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0288005880 tx=0x7f0288009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.185+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02900549f0 con 0x7f029815bf10 2024-01-16T14:58:14.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:14.411+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0298067450 con 0x7f029815bf10 2024-01-16T14:58:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.126+0000 7f0294c00640 1 -- 172.21.15.125:0/1484766694 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0290019020 con 0x7f029815bf10 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.130+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 msgr2=0x7f027006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.130+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 0x7f027006d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0288005880 tx=0x7f0288009040 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.130+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 msgr2=0x7f0298071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.130+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f029000b970 tx=0x7f029000be30 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 shutdown_connections 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f027006b2c0 0x7f027006d780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f029815bf10 0x7f0298071660 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02980ccb50 0x7f029815a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 --2- 172.21.15.125:0/1484766694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02980adf20 0x7f02980c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 >> 172.21.15.125:0/1484766694 conn(0x7f029806a7d0 msgr2=0x7f029806cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:15.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 shutdown_connections 2024-01-16T14:58:15.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.131+0000 7f029e400640 1 -- 172.21.15.125:0/1484766694 wait complete. 2024-01-16T14:58:15.132 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:58:15.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:58:15.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.363+0000 7f3f73600640 1 Processor -- start 2024-01-16T14:58:15.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.363+0000 7f3f73600640 1 -- start start 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f73600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f6c06a3e0 0x7f3f6c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f73600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f6c0aede0 0x7f3f6c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f73600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c071de0 con 0x7f3f6c0d0b60 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f73600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0af270 con 0x7f3f6c0aede0 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f73600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c072030 con 0x7f3f6c06a3e0 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f71600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f6c0aede0 0x7f3f6c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f71600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f6c0aede0 0x7f3f6c06f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57616/0 (socket says 172.21.15.125:57616) 2024-01-16T14:58:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f71600640 1 -- 172.21.15.125:0/3443664102 learned_addr learned my addr 172.21.15.125:0/3443664102 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.365+0000 7f3f6be00640 1 --1- 172.21.15.125:0/3443664102 >> v1:172.21.15.125:6789/0 conn(0x7f3f6c0d0b60 0x7f3f6c06a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42064/0 (socket says 172.21.15.125:42064) 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f70c00640 1 --2- 172.21.15.125:0/3443664102 >> v2:172.21.15.137:3300/0 conn(0x7f3f6c06a3e0 0x7f3f6c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f71600640 1 -- 172.21.15.125:0/3443664102 >> v2:172.21.15.137:3300/0 conn(0x7f3f6c06a3e0 msgr2=0x7f3f6c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f71600640 1 --2- 172.21.15.125:0/3443664102 >> v2:172.21.15.137:3300/0 conn(0x7f3f6c06a3e0 0x7f3f6c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f71600640 1 -- 172.21.15.125:0/3443664102 >> v1:172.21.15.125:6789/0 conn(0x7f3f6c0d0b60 legacy=0x7f3f6c06a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f71600640 1 -- 172.21.15.125:0/3443664102 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c072310 con 0x7f3f6c0aede0 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.366+0000 7f3f6b400640 1 -- 172.21.15.125:0/3443664102 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166732500 0 0) 0x7f3f6c071de0 con 0x7f3f6c0d0b60 2024-01-16T14:58:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.367+0000 7f3f71600640 1 --2- 172.21.15.125:0/3443664102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c06f080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c073fe0 tx=0x7f3f6402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a83a71e0898683 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.367+0000 7f3f6b400640 1 -- 172.21.15.125:0/3443664102 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64030860 con 0x7f3f6c0aede0 2024-01-16T14:58:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.367+0000 7f3f6b400640 1 -- 172.21.15.125:0/3443664102 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f640309c0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.367+0000 7f3f6b400640 1 -- 172.21.15.125:0/3443664102 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64030d50 con 0x7f3f6c0aede0 2024-01-16T14:58:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 -- 172.21.15.125:0/3443664102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 msgr2=0x7f3f6c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 --2- 172.21.15.125:0/3443664102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c06f080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c073fe0 tx=0x7f3f6402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 -- 172.21.15.125:0/3443664102 shutdown_connections 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 --2- 172.21.15.125:0/3443664102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c06f080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 --2- 172.21.15.125:0/3443664102 >> v2:172.21.15.137:3300/0 conn(0x7f3f6c06a3e0 0x7f3f6c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 -- 172.21.15.125:0/3443664102 >> 172.21.15.125:0/3443664102 conn(0x7f3f6c0cfef0 msgr2=0x7f3f6c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 -- 172.21.15.125:0/3443664102 shutdown_connections 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.368+0000 7f3f73600640 1 -- 172.21.15.125:0/3443664102 wait complete. 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.370+0000 7f3f73600640 1 Processor -- start 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.370+0000 7f3f73600640 1 -- start start 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 0x7f3f6c14c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 0x7f3f6c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cd660 con 0x7f3f6c0aede0 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c1574a0 con 0x7f3f6c0d0b60 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f73600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cc9e0 con 0x7f3f6c06a3e0 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f6be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f6be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57624/0 (socket says 172.21.15.125:57624) 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.371+0000 7f3f6be00640 1 -- 172.21.15.125:0/121040353 learned_addr learned my addr 172.21.15.125:0/121040353 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f71600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 0x7f3f6c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f70c00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 0x7f3f6c14c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 -- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 msgr2=0x7f3f6c14c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 0x7f3f6c14c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 -- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 msgr2=0x7f3f6c156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 0x7f3f6c156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 -- 172.21.15.125:0/121040353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f640094d0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f71600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 0x7f3f6c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f70c00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 0x7f3f6c14c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f6be00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c820 tx=0x7f3f5400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400a640 con 0x7f3f6c0aede0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5400a7c0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ab70 con 0x7f3f6c0aede0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.372+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c0cccc0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.373+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c0cd280 con 0x7f3f6c0aede0 2024-01-16T14:58:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.374+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c0675d0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.375+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3f5400acd0 con 0x7f3f6c0aede0 2024-01-16T14:58:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.375+0000 7f3f68c00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 0x7f3f4406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.376+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f54089150 con 0x7f3f6c0aede0 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.376+0000 7f3f70c00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 0x7f3f4406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.376+0000 7f3f70c00640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 0x7f3f4406d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c006fd0 tx=0x7f3f5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.380+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f54054050 con 0x7f3f6c0aede0 2024-01-16T14:58:15.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:15.609+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3f6c0b3b50 con 0x7f3f6c0aede0 2024-01-16T14:58:16.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.136+0000 7f3f68c00640 1 -- 172.21.15.125:0/121040353 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3f54057670 con 0x7f3f6c0aede0 2024-01-16T14:58:16.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 msgr2=0x7f3f4406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 0x7f3f4406d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c006fd0 tx=0x7f3f5c008040 comp rx=0 tx=0).stop 2024-01-16T14:58:16.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 msgr2=0x7f3f6c156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c820 tx=0x7f3f5400cce0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 shutdown_connections 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0d0b60 0x7f3f6c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f4406b2c0 0x7f3f4406d780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f6c0aede0 0x7f3f6c156a20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 --2- 172.21.15.125:0/121040353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c06a3e0 0x7f3f6c14c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 >> 172.21.15.125:0/121040353 conn(0x7f3f6c0cfef0 msgr2=0x7f3f6c0adc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 shutdown_connections 2024-01-16T14:58:16.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.140+0000 7f3f73600640 1 -- 172.21.15.125:0/121040353 wait complete. 2024-01-16T14:58:16.143 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:58:16.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.377+0000 7f8293e00640 1 Processor -- start 2024-01-16T14:58:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.377+0000 7f8293e00640 1 -- start start 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8293e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f829413cfa0 0x7f829415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8293e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f829413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8293e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940cdec0 con 0x7f82940cd9b0 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8293e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82940ce1a0 con 0x7f829413cfa0 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8293e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82940ce480 con 0x7f82940cd530 2024-01-16T14:58:16.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f829413cfa0 0x7f829415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f829413cfa0 0x7f829415afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57638/0 (socket says 172.21.15.125:57638) 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 -- 172.21.15.125:0/1651783761 learned_addr learned my addr 172.21.15.125:0/1651783761 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 -- 172.21.15.125:0/1651783761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 msgr2=0x7f829413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 --2- 172.21.15.125:0/1651783761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f829413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 -- 172.21.15.125:0/1651783761 >> v1:172.21.15.137:6789/0 conn(0x7f82940cd9b0 legacy=0x7f829415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.379+0000 7f8292a00640 1 -- 172.21.15.125:0/1651783761 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940ce760 con 0x7f829413cfa0 2024-01-16T14:58:16.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.380+0000 7f8292a00640 1 --2- 172.21.15.125:0/1651783761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829415afb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8284009e50 tx=0x7f828402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1953725c05a47e39 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.381+0000 7f8292000640 1 -- 172.21.15.125:0/1651783761 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828402c9e0 con 0x7f829413cfa0 2024-01-16T14:58:16.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.381+0000 7f8292000640 1 -- 172.21.15.125:0/1651783761 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828402cb40 con 0x7f829413cfa0 2024-01-16T14:58:16.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.381+0000 7f8292000640 1 -- 172.21.15.125:0/1651783761 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840343f0 con 0x7f829413cfa0 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.381+0000 7f8293e00640 1 -- 172.21.15.125:0/1651783761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 msgr2=0x7f829415afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.381+0000 7f8293e00640 1 --2- 172.21.15.125:0/1651783761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829415afb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8284009e50 tx=0x7f828402be40 comp rx=0 tx=0).stop 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 -- 172.21.15.125:0/1651783761 shutdown_connections 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 --2- 172.21.15.125:0/1651783761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f829413f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 --2- 172.21.15.125:0/1651783761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829415afb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 -- 172.21.15.125:0/1651783761 >> 172.21.15.125:0/1651783761 conn(0x7f829406a800 msgr2=0x7f829406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 -- 172.21.15.125:0/1651783761 shutdown_connections 2024-01-16T14:58:16.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.382+0000 7f8293e00640 1 -- 172.21.15.125:0/1651783761 wait complete. 2024-01-16T14:58:16.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.383+0000 7f8293e00640 1 Processor -- start 2024-01-16T14:58:16.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.384+0000 7f8293e00640 1 -- start start 2024-01-16T14:58:16.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.384+0000 7f8293e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f8294165d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.384+0000 7f8293e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 0x7f82941536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8294135fc0 con 0x7f829413cfa0 2024-01-16T14:58:16.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8294167e50 con 0x7f82940cd9b0 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829414c9e0 con 0x7f82940cd530 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57644/0 (socket says 172.21.15.125:57644) 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 -- 172.21.15.125:0/2992082161 learned_addr learned my addr 172.21.15.125:0/2992082161 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8292a00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f8294165d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8289e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 0x7f82941536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 -- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 msgr2=0x7f8294165d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f8294165d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 -- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 msgr2=0x7f82941536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 0x7f82941536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8293400640 1 -- 172.21.15.125:0/2992082161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8284009b30 con 0x7f829413cfa0 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.385+0000 7f8289e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 0x7f82941536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:16.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f8293400640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f828c00c410 tx=0x7f828c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c011790 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c0118f0 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829414ccc0 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c011ca0 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.386+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8294135be0 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.387+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294067450 con 0x7f829413cfa0 2024-01-16T14:58:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.389+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f828c01c460 con 0x7f829413cfa0 2024-01-16T14:58:16.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.390+0000 7f828b400640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 0x7f826c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.390+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f828c089890 con 0x7f829413cfa0 2024-01-16T14:58:16.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.390+0000 7f8292a00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 0x7f826c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.391+0000 7f8292a00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 0x7f826c06d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8284035000 tx=0x7f8284030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.394+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f828c0546f0 con 0x7f829413cfa0 2024-01-16T14:58:16.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.623+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82940cbff0 con 0x7f829413cfa0 2024-01-16T14:58:16.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.624+0000 7f828b400640 1 -- 172.21.15.125:0/2992082161 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f828c057d10 con 0x7f829413cfa0 2024-01-16T14:58:16.606 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:16.606 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1608829453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1608829453},{"type":"v1","addr":"172.21.15.137:6835","nonce":1608829453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4798,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1479548002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1479548002},{"type":"v1","addr":"172.21.15.137:6833","nonce":1479548002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1004786444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1004786444},{"type":"v1","addr":"172.21.15.125:6835","nonce":1004786444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:58:12.549817+0000","modified":"2024-01-16T14:58:15.611534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6837/2265216525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2265216525},{"type":"v1","addr":"172.21.15.125:6837","nonce":2265216525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T14:58:16.606 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 18 2024-01-16T14:58:16.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 msgr2=0x7f826c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 0x7f826c06d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8284035000 tx=0x7f8284030000 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 msgr2=0x7f829413c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f828c00c410 tx=0x7f828c00c8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 shutdown_connections 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f826c06b2c0 0x7f826c06d780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f829413cfa0 0x7f829413c200 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82940cd9b0 0x7f82941536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 --2- 172.21.15.125:0/2992082161 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82940cd530 0x7f8294165d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.626+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 >> 172.21.15.125:0/2992082161 conn(0x7f829406a800 msgr2=0x7f829415cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.627+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 shutdown_connections 2024-01-16T14:58:16.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.627+0000 7f8293e00640 1 -- 172.21.15.125:0/2992082161 wait complete. 2024-01-16T14:58:16.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.851+0000 7fa179000640 1 Processor -- start 2024-01-16T14:58:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.851+0000 7fa179000640 1 -- start start 2024-01-16T14:58:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa179000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 0x7fa1740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa179000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 0x7fa1740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa179000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17415c5f0 con 0x7fa17413df80 2024-01-16T14:58:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa179000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17406c470 con 0x7fa17413e360 2024-01-16T14:58:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa179000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17406c750 con 0x7fa17415f9c0 2024-01-16T14:58:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa171e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa17413df80 0x7fa17415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:54560/0 (socket says 172.21.15.125:54560) 2024-01-16T14:58:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa171e00640 1 -- 172.21.15.125:0/4231375769 learned_addr learned my addr 172.21.15.125:0/4231375769 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.853+0000 7fa172800640 1 --2- 172.21.15.125:0/4231375769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 0x7fa1740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.854+0000 7fa171400640 1 --2- 172.21.15.125:0/4231375769 >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 0x7fa1740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.854+0000 7fa170a00640 1 -- 172.21.15.125:0/4231375769 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248208933 0 0) 0x7fa17415c5f0 con 0x7fa17413df80 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.854+0000 7fa170a00640 1 -- 172.21.15.125:0/4231375769 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154003520 con 0x7fa17413df80 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.854+0000 7fa172800640 1 -- 172.21.15.125:0/4231375769 >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 msgr2=0x7fa1740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.855+0000 7fa172800640 1 --2- 172.21.15.125:0/4231375769 >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 0x7fa1740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.855+0000 7fa172800640 1 -- 172.21.15.125:0/4231375769 >> v1:172.21.15.137:6789/0 conn(0x7fa17413df80 legacy=0x7fa17415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.855+0000 7fa172800640 1 -- 172.21.15.125:0/4231375769 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17406ca30 con 0x7fa17415f9c0 2024-01-16T14:58:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.855+0000 7fa171400640 1 --2- 172.21.15.125:0/4231375769 >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 0x7fa1740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.856+0000 7fa172800640 1 --2- 172.21.15.125:0/4231375769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 0x7fa1740cf7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0097a0 tx=0x7fa16c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f1ed138242f128 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.856+0000 7fa170a00640 1 -- 172.21.15.125:0/4231375769 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02d040 con 0x7fa17415f9c0 2024-01-16T14:58:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.856+0000 7fa170a00640 1 -- 172.21.15.125:0/4231375769 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c031810 con 0x7fa17415f9c0 2024-01-16T14:58:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.856+0000 7fa170a00640 1 -- 172.21.15.125:0/4231375769 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c031b80 con 0x7fa17415f9c0 2024-01-16T14:58:16.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 -- 172.21.15.125:0/4231375769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 msgr2=0x7fa1740cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 --2- 172.21.15.125:0/4231375769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 0x7fa1740cf7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0097a0 tx=0x7fa16c02b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 -- 172.21.15.125:0/4231375769 shutdown_connections 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 --2- 172.21.15.125:0/4231375769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17415f9c0 0x7fa1740cf7d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 --2- 172.21.15.125:0/4231375769 >> v2:172.21.15.137:3300/0 conn(0x7fa17413e360 0x7fa1740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 -- 172.21.15.125:0/4231375769 >> 172.21.15.125:0/4231375769 conn(0x7fa17406a800 msgr2=0x7fa17406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 -- 172.21.15.125:0/4231375769 shutdown_connections 2024-01-16T14:58:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.857+0000 7fa179000640 1 -- 172.21.15.125:0/4231375769 wait complete. 2024-01-16T14:58:16.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.859+0000 7fa179000640 1 Processor -- start 2024-01-16T14:58:16.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.859+0000 7fa179000640 1 -- start start 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa17413df80 0x7fa1741503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17413e360 0x7fa1741524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174153f60 con 0x7fa17415f9c0 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174152ec0 con 0x7fa17413df80 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa179000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741531a0 con 0x7fa17413e360 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa172800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa172800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57086/0 (socket says 172.21.15.125:57086) 2024-01-16T14:58:16.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.860+0000 7fa172800640 1 -- 172.21.15.125:0/2520751398 learned_addr learned my addr 172.21.15.125:0/2520751398 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa171e00640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa17413df80 0x7fa1741503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 -- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17413e360 msgr2=0x7fa1741524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17413e360 0x7fa1741524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 -- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa17413df80 msgr2=0x7fa1741503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa17413df80 0x7fa1741503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 -- 172.21.15.125:0/2520751398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c009a70 con 0x7fa17415f9c0 2024-01-16T14:58:16.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.861+0000 7fa172800640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0098c0 tx=0x7fa16c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.862+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0044c0 con 0x7fa17415f9c0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.862+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c03f070 con 0x7fa17415f9c0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.862+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c003a40 con 0x7fa17415f9c0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.862+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c009400 con 0x7fa17415f9c0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.862+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa174153760 con 0x7fa17415f9c0 2024-01-16T14:58:16.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.863+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1740675d0 con 0x7fa17415f9c0 2024-01-16T14:58:16.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.864+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa16c003ba0 con 0x7fa17415f9c0 2024-01-16T14:58:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.865+0000 7fa16a000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 0x7fa14c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.865+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa16c041070 con 0x7fa17415f9c0 2024-01-16T14:58:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.865+0000 7fa171e00640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 0x7fa14c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.866+0000 7fa171e00640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 0x7fa14c06d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa164007fd0 tx=0x7fa164009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:16.869+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa16c0dee90 con 0x7fa17415f9c0 2024-01-16T14:58:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.047+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1740b42a0 con 0x7fa17415f9c0 2024-01-16T14:58:17.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.050+0000 7fa16a000640 1 -- 172.21.15.125:0/2520751398 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fa16c0770a0 con 0x7fa17415f9c0 2024-01-16T14:58:17.032 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:17.033 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":34,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:58:14.102914+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:10.324911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:11.557252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 msgr2=0x7fa14c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 0x7fa14c06d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa164007fd0 tx=0x7fa164009040 comp rx=0 tx=0).stop 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 msgr2=0x7fa1740b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0098c0 tx=0x7fa16c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 shutdown_connections 2024-01-16T14:58:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa14c06b2c0 0x7fa14c06d780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17415f9c0 0x7fa1740b2cf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa17413e360 0x7fa1741524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 --2- 172.21.15.125:0/2520751398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa17413df80 0x7fa1741503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 >> 172.21.15.125:0/2520751398 conn(0x7fa17406a800 msgr2=0x7fa17406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 shutdown_connections 2024-01-16T14:58:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.053+0000 7fa179000640 1 -- 172.21.15.125:0/2520751398 wait complete. 2024-01-16T14:58:17.053 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:58:17.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.275+0000 7f347d200640 1 Processor -- start 2024-01-16T14:58:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.275+0000 7f347d200640 1 -- start start 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f347d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34780ae5a0 0x7f347815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f347d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f34780b3ae0 0x7f347806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f347d200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478070c40 con 0x7f347806d120 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f347d200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3478071450 con 0x7f34780ae5a0 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f347d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347806c470 con 0x7f34780b3ae0 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f3475e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34780ae5a0 0x7f347815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f3476800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f347806d120 0x7f34780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45762/0 (socket says 172.21.15.125:45762) 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.277+0000 7f3476800640 1 -- 172.21.15.125:0/1157185334 learned_addr learned my addr 172.21.15.125:0/1157185334 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f346d200640 1 --2- 172.21.15.125:0/1157185334 >> v2:172.21.15.137:3300/0 conn(0x7f34780b3ae0 0x7f347806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475400640 1 -- 172.21.15.125:0/1157185334 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577841007 0 0) 0x7f3478070c40 con 0x7f347806d120 2024-01-16T14:58:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475400640 1 -- 172.21.15.125:0/1157185334 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458003520 con 0x7f347806d120 2024-01-16T14:58:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475e00640 1 -- 172.21.15.125:0/1157185334 >> v2:172.21.15.137:3300/0 conn(0x7f34780b3ae0 msgr2=0x7f347806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475e00640 1 --2- 172.21.15.125:0/1157185334 >> v2:172.21.15.137:3300/0 conn(0x7f34780b3ae0 0x7f347806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475e00640 1 -- 172.21.15.125:0/1157185334 >> v1:172.21.15.125:6789/0 conn(0x7f347806d120 legacy=0x7f34780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.278+0000 7f3475e00640 1 -- 172.21.15.125:0/1157185334 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347806c750 con 0x7f34780ae5a0 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.279+0000 7f3475e00640 1 --2- 172.21.15.125:0/1157185334 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f347815e430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f34780cfea0 tx=0x7f346802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d451efa94de9e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.279+0000 7f3475400640 1 -- 172.21.15.125:0/1157185334 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468031830 con 0x7f34780ae5a0 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.279+0000 7f3475400640 1 -- 172.21.15.125:0/1157185334 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3468031990 con 0x7f34780ae5a0 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.279+0000 7f3475400640 1 -- 172.21.15.125:0/1157185334 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468031d00 con 0x7f34780ae5a0 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 -- 172.21.15.125:0/1157185334 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 msgr2=0x7f347815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 --2- 172.21.15.125:0/1157185334 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f347815e430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f34780cfea0 tx=0x7f346802b8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 -- 172.21.15.125:0/1157185334 shutdown_connections 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 --2- 172.21.15.125:0/1157185334 >> v2:172.21.15.137:3300/0 conn(0x7f34780b3ae0 0x7f347806cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 --2- 172.21.15.125:0/1157185334 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f347815e430 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 -- 172.21.15.125:0/1157185334 >> 172.21.15.125:0/1157185334 conn(0x7f347806a800 msgr2=0x7f347806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 -- 172.21.15.125:0/1157185334 shutdown_connections 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.280+0000 7f347d200640 1 -- 172.21.15.125:0/1157185334 wait complete. 2024-01-16T14:58:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.282+0000 7f347d200640 1 Processor -- start 2024-01-16T14:58:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.282+0000 7f347d200640 1 -- start start 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f347806d120 0x7f347814c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34780b3ae0 0x7f3478148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34781571a0 con 0x7f34780ae5a0 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3478157d20 con 0x7f347806d120 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f347d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34781573f0 con 0x7f34780b3ae0 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57100/0 (socket says 172.21.15.125:57100) 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 -- 172.21.15.125:0/1198215628 learned_addr learned my addr 172.21.15.125:0/1198215628 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 -- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34780b3ae0 msgr2=0x7f3478148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34780b3ae0 0x7f3478148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 -- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f347806d120 msgr2=0x7f347814c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f3475e00640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f347806d120 0x7f347814c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.283+0000 7f346d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f347806d120 0x7f347814c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f346d200640 1 -- 172.21.15.125:0/1198215628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468009400 con 0x7f34780ae5a0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f346d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f346000d600 tx=0x7f346000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600125f0 con 0x7f34780ae5a0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3460012770 con 0x7f34780ae5a0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460012b20 con 0x7f34780ae5a0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34781576d0 con 0x7f34780ae5a0 2024-01-16T14:58:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.284+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34781494f0 con 0x7f34780ae5a0 2024-01-16T14:58:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.285+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34780675d0 con 0x7f34780ae5a0 2024-01-16T14:58:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.286+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3460012c80 con 0x7f34780ae5a0 2024-01-16T14:58:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.287+0000 7f346ea00640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 0x7f345006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.287+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f346008a1b0 con 0x7f34780ae5a0 2024-01-16T14:58:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.287+0000 7f3475e00640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 0x7f345006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.288+0000 7f3475e00640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 0x7f345006d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3468036000 tx=0x7f3468008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.291+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3460054f80 con 0x7f34780ae5a0 2024-01-16T14:58:17.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.477+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3478070550 con 0x7f345006b2c0 2024-01-16T14:58:17.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.494+0000 7f346ea00640 1 -- 172.21.15.125:0/1198215628 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287205 (secure 0 0 0) 0x7f3468045400 con 0x7f345006b2c0 2024-01-16T14:58:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:17.482 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:58:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.499+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 msgr2=0x7f345006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.499+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 0x7f345006d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3468036000 tx=0x7f3468008030 comp rx=0 tx=0).stop 2024-01-16T14:58:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.499+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 msgr2=0x7f3478148240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.499+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f346000d600 tx=0x7f346000dac0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 shutdown_connections 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34780b3ae0 0x7f3478148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f345006b2c0 0x7f345006d780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34780ae5a0 0x7f3478148240 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 --2- 172.21.15.125:0/1198215628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f347806d120 0x7f347814c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 >> 172.21.15.125:0/1198215628 conn(0x7f347806a800 msgr2=0x7f3478160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 shutdown_connections 2024-01-16T14:58:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.500+0000 7f347d200640 1 -- 172.21.15.125:0/1198215628 wait complete. 2024-01-16T14:58:17.500 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-01-16T14:58:15.944862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220280,"kb_used_data":4112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754440,"statfs":{"total":773094113280,"available":772868546560,"internally_reserved":0,"allocated":4210688,"data_stored":2343298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.869916"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131614+0000","last_change":"2024-01-16T14:58:13.124277+0000","last_active":"2024-01-16T14:58:13.131614+0000","last_peered":"2024-01-16T14:58:13.131614+0000","last_clean":"2024-01-16T14:58:13.131614+0000","last_became_active":"2024-01-16T14:58:13.123951+0000","last_became_peered":"2024-01-16T14:58:13.123951+0000","last_unstale":"2024-01-16T14:58:13.131614+0000","last_undegraded":"2024-01-16T14:58:13.131614+0000","last_fullsized":"2024-01-16T14:58:13.131614+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135088+0000","last_change":"2024-01-16T14:58:13.123807+0000","last_active":"2024-01-16T14:58:13.135088+0000","last_peered":"2024-01-16T14:58:13.135088+0000","last_clean":"2024-01-16T14:58:13.135088+0000","last_became_active":"2024-01-16T14:58:13.123519+0000","last_became_peered":"2024-01-16T14:58:13.123519+0000","last_unstale":"2024-01-16T14:58:13.135088+0000","last_undegraded":"2024-01-16T14:58:13.135088+0000","last_fullsized":"2024-01-16T14:58:13.135088+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:06.578017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123476+0000","last_change":"2024-01-16T14:58:13.119873+0000","last_active":"2024-01-16T14:58:13.123476+0000","last_peered":"2024-01-16T14:58:13.123476+0000","last_clean":"2024-01-16T14:58:13.123476+0000","last_became_active":"2024-01-16T14:58:13.119611+0000","last_became_peered":"2024-01-16T14:58:13.119611+0000","last_unstale":"2024-01-16T14:58:13.123476+0000","last_undegraded":"2024-01-16T14:58:13.123476+0000","last_fullsized":"2024-01-16T14:58:13.123476+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.120612+0000","last_change":"2024-01-16T14:58:13.118612+0000","last_active":"2024-01-16T14:58:13.120612+0000","last_peered":"2024-01-16T14:58:13.120612+0000","last_clean":"2024-01-16T14:58:13.120612+0000","last_became_active":"2024-01-16T14:58:13.115549+0000","last_became_peered":"2024-01-16T14:58:13.115549+0000","last_unstale":"2024-01-16T14:58:13.120612+0000","last_undegraded":"2024-01-16T14:58:13.120612+0000","last_fullsized":"2024-01-16T14:58:13.120612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:53.759776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.140281+0000","last_change":"2024-01-16T14:58:13.134287+0000","last_active":"2024-01-16T14:58:13.140281+0000","last_peered":"2024-01-16T14:58:13.140281+0000","last_clean":"2024-01-16T14:58:13.140281+0000","last_became_active":"2024-01-16T14:58:13.133891+0000","last_became_peered":"2024-01-16T14:58:13.133891+0000","last_unstale":"2024-01-16T14:58:13.140281+0000","last_undegraded":"2024-01-16T14:58:13.140281+0000","last_fullsized":"2024-01-16T14:58:13.140281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:12.141938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133643+0000","last_change":"2024-01-16T14:58:13.122659+0000","last_active":"2024-01-16T14:58:13.133643+0000","last_peered":"2024-01-16T14:58:13.133643+0000","last_clean":"2024-01-16T14:58:13.133643+0000","last_became_active":"2024-01-16T14:58:13.122316+0000","last_became_peered":"2024-01-16T14:58:13.122316+0000","last_unstale":"2024-01-16T14:58:13.133643+0000","last_undegraded":"2024-01-16T14:58:13.133643+0000","last_fullsized":"2024-01-16T14:58:13.133643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125362+0000","last_change":"2024-01-16T14:58:13.120991+0000","last_active":"2024-01-16T14:58:13.125362+0000","last_peered":"2024-01-16T14:58:13.125362+0000","last_clean":"2024-01-16T14:58:13.125362+0000","last_became_active":"2024-01-16T14:58:13.120711+0000","last_became_peered":"2024-01-16T14:58:13.120711+0000","last_unstale":"2024-01-16T14:58:13.125362+0000","last_undegraded":"2024-01-16T14:58:13.125362+0000","last_fullsized":"2024-01-16T14:58:13.125362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:00.486870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133123+0000","last_change":"2024-01-16T14:58:13.126979+0000","last_active":"2024-01-16T14:58:13.133123+0000","last_peered":"2024-01-16T14:58:13.133123+0000","last_clean":"2024-01-16T14:58:13.133123+0000","last_became_active":"2024-01-16T14:58:13.126583+0000","last_became_peered":"2024-01-16T14:58:13.126583+0000","last_unstale":"2024-01-16T14:58:13.133123+0000","last_undegraded":"2024-01-16T14:58:13.133123+0000","last_fullsized":"2024-01-16T14:58:13.133123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125064+0000","last_change":"2024-01-16T14:58:13.113431+0000","last_active":"2024-01-16T14:58:13.125064+0000","last_peered":"2024-01-16T14:58:13.125064+0000","last_clean":"2024-01-16T14:58:13.125064+0000","last_became_active":"2024-01-16T14:58:13.113135+0000","last_became_peered":"2024-01-16T14:58:13.113135+0000","last_unstale":"2024-01-16T14:58:13.125064+0000","last_undegraded":"2024-01-16T14:58:13.125064+0000","last_fullsized":"2024-01-16T14:58:13.125064+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:08.843601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138168+0000","last_change":"2024-01-16T14:58:13.129831+0000","last_active":"2024-01-16T14:58:13.138168+0000","last_peered":"2024-01-16T14:58:13.138168+0000","last_clean":"2024-01-16T14:58:13.138168+0000","last_became_active":"2024-01-16T14:58:13.129463+0000","last_became_peered":"2024-01-16T14:58:13.129463+0000","last_unstale":"2024-01-16T14:58:13.138168+0000","last_undegraded":"2024-01-16T14:58:13.138168+0000","last_fullsized":"2024-01-16T14:58:13.138168+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.121897+0000","last_change":"2024-01-16T14:58:13.114216+0000","last_active":"2024-01-16T14:58:13.121897+0000","last_peered":"2024-01-16T14:58:13.121897+0000","last_clean":"2024-01-16T14:58:13.121897+0000","last_became_active":"2024-01-16T14:58:13.113901+0000","last_became_peered":"2024-01-16T14:58:13.113901+0000","last_unstale":"2024-01-16T14:58:13.121897+0000","last_undegraded":"2024-01-16T14:58:13.121897+0000","last_fullsized":"2024-01-16T14:58:13.121897+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:06.578017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133561+0000","last_change":"2024-01-16T14:58:13.129383+0000","last_active":"2024-01-16T14:58:13.133561+0000","last_peered":"2024-01-16T14:58:13.133561+0000","last_clean":"2024-01-16T14:58:13.133561+0000","last_became_active":"2024-01-16T14:58:13.129124+0000","last_became_peered":"2024-01-16T14:58:13.129124+0000","last_unstale":"2024-01-16T14:58:13.133561+0000","last_undegraded":"2024-01-16T14:58:13.133561+0000","last_fullsized":"2024-01-16T14:58:13.133561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137734+0000","last_change":"2024-01-16T14:58:13.129617+0000","last_active":"2024-01-16T14:58:13.137734+0000","last_peered":"2024-01-16T14:58:13.137734+0000","last_clean":"2024-01-16T14:58:13.137734+0000","last_became_active":"2024-01-16T14:58:13.127243+0000","last_became_peered":"2024-01-16T14:58:13.127243+0000","last_unstale":"2024-01-16T14:58:13.137734+0000","last_undegraded":"2024-01-16T14:58:13.137734+0000","last_fullsized":"2024-01-16T14:58:13.137734+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:53.475795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.143003+0000","last_change":"2024-01-16T14:58:13.140258+0000","last_active":"2024-01-16T14:58:13.143003+0000","last_peered":"2024-01-16T14:58:13.143003+0000","last_clean":"2024-01-16T14:58:13.143003+0000","last_became_active":"2024-01-16T14:58:13.138930+0000","last_became_peered":"2024-01-16T14:58:13.138930+0000","last_unstale":"2024-01-16T14:58:13.143003+0000","last_undegraded":"2024-01-16T14:58:13.143003+0000","last_fullsized":"2024-01-16T14:58:13.143003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123578+0000","last_change":"2024-01-16T14:58:13.121102+0000","last_active":"2024-01-16T14:58:13.123578+0000","last_peered":"2024-01-16T14:58:13.123578+0000","last_clean":"2024-01-16T14:58:13.123578+0000","last_became_active":"2024-01-16T14:58:13.120831+0000","last_became_peered":"2024-01-16T14:58:13.120831+0000","last_unstale":"2024-01-16T14:58:13.123578+0000","last_undegraded":"2024-01-16T14:58:13.123578+0000","last_fullsized":"2024-01-16T14:58:13.123578+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126888+0000","last_change":"2024-01-16T14:58:13.124979+0000","last_active":"2024-01-16T14:58:13.126888+0000","last_peered":"2024-01-16T14:58:13.126888+0000","last_clean":"2024-01-16T14:58:13.126888+0000","last_became_active":"2024-01-16T14:58:13.124684+0000","last_became_peered":"2024-01-16T14:58:13.124684+0000","last_unstale":"2024-01-16T14:58:13.126888+0000","last_undegraded":"2024-01-16T14:58:13.126888+0000","last_fullsized":"2024-01-16T14:58:13.126888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:25.176612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125165+0000","last_change":"2024-01-16T14:58:13.120735+0000","last_active":"2024-01-16T14:58:13.125165+0000","last_peered":"2024-01-16T14:58:13.125165+0000","last_clean":"2024-01-16T14:58:13.125165+0000","last_became_active":"2024-01-16T14:58:13.120230+0000","last_became_peered":"2024-01-16T14:58:13.120230+0000","last_unstale":"2024-01-16T14:58:13.125165+0000","last_undegraded":"2024-01-16T14:58:13.125165+0000","last_fullsized":"2024-01-16T14:58:13.125165+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135076+0000","last_change":"2024-01-16T14:58:13.123169+0000","last_active":"2024-01-16T14:58:13.135076+0000","last_peered":"2024-01-16T14:58:13.135076+0000","last_clean":"2024-01-16T14:58:13.135076+0000","last_became_active":"2024-01-16T14:58:13.122693+0000","last_became_peered":"2024-01-16T14:58:13.122693+0000","last_unstale":"2024-01-16T14:58:13.135076+0000","last_undegraded":"2024-01-16T14:58:13.135076+0000","last_fullsized":"2024-01-16T14:58:13.135076+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.128938+0000","last_change":"2024-01-16T14:58:13.120657+0000","last_active":"2024-01-16T14:58:13.128938+0000","last_peered":"2024-01-16T14:58:13.128938+0000","last_clean":"2024-01-16T14:58:13.128938+0000","last_became_active":"2024-01-16T14:58:13.120332+0000","last_became_peered":"2024-01-16T14:58:13.120332+0000","last_unstale":"2024-01-16T14:58:13.128938+0000","last_undegraded":"2024-01-16T14:58:13.128938+0000","last_fullsized":"2024-01-16T14:58:13.128938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:34.036848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133745+0000","last_change":"2024-01-16T14:58:13.125993+0000","last_active":"2024-01-16T14:58:13.133745+0000","last_peered":"2024-01-16T14:58:13.133745+0000","last_clean":"2024-01-16T14:58:13.133745+0000","last_became_active":"2024-01-16T14:58:13.125680+0000","last_became_peered":"2024-01-16T14:58:13.125680+0000","last_unstale":"2024-01-16T14:58:13.133745+0000","last_undegraded":"2024-01-16T14:58:13.133745+0000","last_fullsized":"2024-01-16T14:58:13.133745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.112508+0000","last_change":"2024-01-16T14:58:13.101390+0000","last_active":"2024-01-16T14:58:13.112508+0000","last_peered":"2024-01-16T14:58:13.112508+0000","last_clean":"2024-01-16T14:58:13.112508+0000","last_became_active":"2024-01-16T14:58:13.100325+0000","last_became_peered":"2024-01-16T14:58:13.100325+0000","last_unstale":"2024-01-16T14:58:13.112508+0000","last_undegraded":"2024-01-16T14:58:13.112508+0000","last_fullsized":"2024-01-16T14:58:13.112508+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126551+0000","last_change":"2024-01-16T14:58:13.122121+0000","last_active":"2024-01-16T14:58:13.126551+0000","last_peered":"2024-01-16T14:58:13.126551+0000","last_clean":"2024-01-16T14:58:13.126551+0000","last_became_active":"2024-01-16T14:58:13.121564+0000","last_became_peered":"2024-01-16T14:58:13.121564+0000","last_unstale":"2024-01-16T14:58:13.126551+0000","last_undegraded":"2024-01-16T14:58:13.126551+0000","last_fullsized":"2024-01-16T14:58:13.126551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.114637+0000","last_change":"2024-01-16T14:58:13.103203+0000","last_active":"2024-01-16T14:58:13.114637+0000","last_peered":"2024-01-16T14:58:13.114637+0000","last_clean":"2024-01-16T14:58:13.114637+0000","last_became_active":"2024-01-16T14:58:13.102474+0000","last_became_peered":"2024-01-16T14:58:13.102474+0000","last_unstale":"2024-01-16T14:58:13.114637+0000","last_undegraded":"2024-01-16T14:58:13.114637+0000","last_fullsized":"2024-01-16T14:58:13.114637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:17.620746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133539+0000","last_change":"2024-01-16T14:58:13.128584+0000","last_active":"2024-01-16T14:58:13.133539+0000","last_peered":"2024-01-16T14:58:13.133539+0000","last_clean":"2024-01-16T14:58:13.133539+0000","last_became_active":"2024-01-16T14:58:13.128008+0000","last_became_peered":"2024-01-16T14:58:13.128008+0000","last_unstale":"2024-01-16T14:58:13.133539+0000","last_undegraded":"2024-01-16T14:58:13.133539+0000","last_fullsized":"2024-01-16T14:58:13.133539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:42.748033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.114524+0000","last_change":"2024-01-16T14:58:13.111335+0000","last_active":"2024-01-16T14:58:13.114524+0000","last_peered":"2024-01-16T14:58:13.114524+0000","last_clean":"2024-01-16T14:58:13.114524+0000","last_became_active":"2024-01-16T14:58:13.111073+0000","last_became_peered":"2024-01-16T14:58:13.111073+0000","last_unstale":"2024-01-16T14:58:13.114524+0000","last_undegraded":"2024-01-16T14:58:13.114524+0000","last_fullsized":"2024-01-16T14:58:13.114524+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:17.620746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135098+0000","last_change":"2024-01-16T14:58:13.124443+0000","last_active":"2024-01-16T14:58:13.135098+0000","last_peered":"2024-01-16T14:58:13.135098+0000","last_clean":"2024-01-16T14:58:13.135098+0000","last_became_active":"2024-01-16T14:58:13.124106+0000","last_became_peered":"2024-01-16T14:58:13.124106+0000","last_unstale":"2024-01-16T14:58:13.135098+0000","last_undegraded":"2024-01-16T14:58:13.135098+0000","last_fullsized":"2024-01-16T14:58:13.135098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.120820+0000","last_change":"2024-01-16T14:58:13.116915+0000","last_active":"2024-01-16T14:58:13.120820+0000","last_peered":"2024-01-16T14:58:13.120820+0000","last_clean":"2024-01-16T14:58:13.120820+0000","last_became_active":"2024-01-16T14:58:13.116257+0000","last_became_peered":"2024-01-16T14:58:13.116257+0000","last_unstale":"2024-01-16T14:58:13.120820+0000","last_undegraded":"2024-01-16T14:58:13.120820+0000","last_fullsized":"2024-01-16T14:58:13.120820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:08.843601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.122252+0000","last_change":"2024-01-16T14:58:13.116110+0000","last_active":"2024-01-16T14:58:13.122252+0000","last_peered":"2024-01-16T14:58:13.122252+0000","last_clean":"2024-01-16T14:58:13.122252+0000","last_became_active":"2024-01-16T14:58:13.115700+0000","last_became_peered":"2024-01-16T14:58:13.115700+0000","last_unstale":"2024-01-16T14:58:13.122252+0000","last_undegraded":"2024-01-16T14:58:13.122252+0000","last_fullsized":"2024-01-16T14:58:13.122252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:28.383910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.132007+0000","last_change":"2024-01-16T14:58:13.128079+0000","last_active":"2024-01-16T14:58:13.132007+0000","last_peered":"2024-01-16T14:58:13.132007+0000","last_clean":"2024-01-16T14:58:13.132007+0000","last_became_active":"2024-01-16T14:58:13.127568+0000","last_became_peered":"2024-01-16T14:58:13.127568+0000","last_unstale":"2024-01-16T14:58:13.132007+0000","last_undegraded":"2024-01-16T14:58:13.132007+0000","last_fullsized":"2024-01-16T14:58:13.132007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.124382+0000","last_change":"2024-01-16T14:58:13.117151+0000","last_active":"2024-01-16T14:58:13.124382+0000","last_peered":"2024-01-16T14:58:13.124382+0000","last_clean":"2024-01-16T14:58:13.124382+0000","last_became_active":"2024-01-16T14:58:13.116817+0000","last_became_peered":"2024-01-16T14:58:13.116817+0000","last_unstale":"2024-01-16T14:58:13.124382+0000","last_undegraded":"2024-01-16T14:58:13.124382+0000","last_fullsized":"2024-01-16T14:58:13.124382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.130513+0000","last_change":"2024-01-16T14:58:13.126783+0000","last_active":"2024-01-16T14:58:13.130513+0000","last_peered":"2024-01-16T14:58:13.130513+0000","last_clean":"2024-01-16T14:58:13.130513+0000","last_became_active":"2024-01-16T14:58:13.126473+0000","last_became_peered":"2024-01-16T14:58:13.126473+0000","last_unstale":"2024-01-16T14:58:13.130513+0000","last_undegraded":"2024-01-16T14:58:13.130513+0000","last_fullsized":"2024-01-16T14:58:13.130513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123986+0000","last_change":"2024-01-16T14:58:13.116621+0000","last_active":"2024-01-16T14:58:13.123986+0000","last_peered":"2024-01-16T14:58:13.123986+0000","last_clean":"2024-01-16T14:58:13.123986+0000","last_became_active":"2024-01-16T14:58:13.116331+0000","last_became_peered":"2024-01-16T14:58:13.116331+0000","last_unstale":"2024-01-16T14:58:13.123986+0000","last_undegraded":"2024-01-16T14:58:13.123986+0000","last_fullsized":"2024-01-16T14:58:13.123986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:03.151295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.112302+0000","last_change":"2024-01-16T14:58:13.100907+0000","last_active":"2024-01-16T14:58:13.112302+0000","last_peered":"2024-01-16T14:58:13.112302+0000","last_clean":"2024-01-16T14:58:13.112302+0000","last_became_active":"2024-01-16T14:58:13.100149+0000","last_became_peered":"2024-01-16T14:58:13.100149+0000","last_unstale":"2024-01-16T14:58:13.112302+0000","last_undegraded":"2024-01-16T14:58:13.112302+0000","last_fullsized":"2024-01-16T14:58:13.112302+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.139129+0000","last_change":"2024-01-16T14:58:12.126242+0000","last_active":"2024-01-16T14:58:13.139129+0000","last_peered":"2024-01-16T14:58:13.139129+0000","last_clean":"2024-01-16T14:58:13.139129+0000","last_became_active":"2024-01-16T14:58:12.125733+0000","last_became_peered":"2024-01-16T14:58:12.125733+0000","last_unstale":"2024-01-16T14:58:13.139129+0000","last_undegraded":"2024-01-16T14:58:13.139129+0000","last_fullsized":"2024-01-16T14:58:13.139129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:17.174671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131637+0000","last_change":"2024-01-16T14:58:13.126298+0000","last_active":"2024-01-16T14:58:13.131637+0000","last_peered":"2024-01-16T14:58:13.131637+0000","last_clean":"2024-01-16T14:58:13.131637+0000","last_became_active":"2024-01-16T14:58:13.125956+0000","last_became_peered":"2024-01-16T14:58:13.125956+0000","last_unstale":"2024-01-16T14:58:13.131637+0000","last_undegraded":"2024-01-16T14:58:13.131637+0000","last_fullsized":"2024-01-16T14:58:13.131637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137278+0000","last_change":"2024-01-16T14:58:12.123554+0000","last_active":"2024-01-16T14:58:13.137278+0000","last_peered":"2024-01-16T14:58:13.137278+0000","last_clean":"2024-01-16T14:58:13.137278+0000","last_became_active":"2024-01-16T14:58:12.123224+0000","last_became_peered":"2024-01-16T14:58:12.123224+0000","last_unstale":"2024-01-16T14:58:13.137278+0000","last_undegraded":"2024-01-16T14:58:13.137278+0000","last_fullsized":"2024-01-16T14:58:13.137278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:11.527954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131846+0000","last_change":"2024-01-16T14:58:13.124444+0000","last_active":"2024-01-16T14:58:13.131846+0000","last_peered":"2024-01-16T14:58:13.131846+0000","last_clean":"2024-01-16T14:58:13.131846+0000","last_became_active":"2024-01-16T14:58:13.124149+0000","last_became_peered":"2024-01-16T14:58:13.124149+0000","last_unstale":"2024-01-16T14:58:13.131846+0000","last_undegraded":"2024-01-16T14:58:13.131846+0000","last_fullsized":"2024-01-16T14:58:13.131846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:12.539345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142268+0000","last_change":"2024-01-16T14:58:12.118834+0000","last_active":"2024-01-16T14:58:13.142268+0000","last_peered":"2024-01-16T14:58:13.142268+0000","last_clean":"2024-01-16T14:58:13.142268+0000","last_became_active":"2024-01-16T14:58:12.118535+0000","last_became_peered":"2024-01-16T14:58:12.118535+0000","last_unstale":"2024-01-16T14:58:13.142268+0000","last_undegraded":"2024-01-16T14:58:13.142268+0000","last_fullsized":"2024-01-16T14:58:13.142268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:59.475479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119440+0000","last_change":"2024-01-16T14:58:13.114187+0000","last_active":"2024-01-16T14:58:13.119440+0000","last_peered":"2024-01-16T14:58:13.119440+0000","last_clean":"2024-01-16T14:58:13.119440+0000","last_became_active":"2024-01-16T14:58:13.113937+0000","last_became_peered":"2024-01-16T14:58:13.113937+0000","last_unstale":"2024-01-16T14:58:13.119440+0000","last_undegraded":"2024-01-16T14:58:13.119440+0000","last_fullsized":"2024-01-16T14:58:13.119440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:12.539345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138918+0000","last_change":"2024-01-16T14:58:12.112457+0000","last_active":"2024-01-16T14:58:13.138918+0000","last_peered":"2024-01-16T14:58:13.138918+0000","last_clean":"2024-01-16T14:58:13.138918+0000","last_became_active":"2024-01-16T14:58:12.111173+0000","last_became_peered":"2024-01-16T14:58:12.111173+0000","last_unstale":"2024-01-16T14:58:13.138918+0000","last_undegraded":"2024-01-16T14:58:13.138918+0000","last_fullsized":"2024-01-16T14:58:13.138918+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:53.167551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123702+0000","last_change":"2024-01-16T14:58:13.117046+0000","last_active":"2024-01-16T14:58:13.123702+0000","last_peered":"2024-01-16T14:58:13.123702+0000","last_clean":"2024-01-16T14:58:13.123702+0000","last_became_active":"2024-01-16T14:58:13.115382+0000","last_became_peered":"2024-01-16T14:58:13.115382+0000","last_unstale":"2024-01-16T14:58:13.123702+0000","last_undegraded":"2024-01-16T14:58:13.123702+0000","last_fullsized":"2024-01-16T14:58:13.123702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142502+0000","last_change":"2024-01-16T14:58:12.117463+0000","last_active":"2024-01-16T14:58:13.142502+0000","last_peered":"2024-01-16T14:58:13.142502+0000","last_clean":"2024-01-16T14:58:13.142502+0000","last_became_active":"2024-01-16T14:58:12.117178+0000","last_became_peered":"2024-01-16T14:58:12.117178+0000","last_unstale":"2024-01-16T14:58:13.142502+0000","last_undegraded":"2024-01-16T14:58:13.142502+0000","last_fullsized":"2024-01-16T14:58:13.142502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:30:48.911413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123056+0000","last_change":"2024-01-16T14:58:13.116633+0000","last_active":"2024-01-16T14:58:13.123056+0000","last_peered":"2024-01-16T14:58:13.123056+0000","last_clean":"2024-01-16T14:58:13.123056+0000","last_became_active":"2024-01-16T14:58:13.116090+0000","last_became_peered":"2024-01-16T14:58:13.116090+0000","last_unstale":"2024-01-16T14:58:13.123056+0000","last_undegraded":"2024-01-16T14:58:13.123056+0000","last_fullsized":"2024-01-16T14:58:13.123056+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:03.151295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.141154+0000","last_change":"2024-01-16T14:58:12.116948+0000","last_active":"2024-01-16T14:58:13.141154+0000","last_peered":"2024-01-16T14:58:13.141154+0000","last_clean":"2024-01-16T14:58:13.141154+0000","last_became_active":"2024-01-16T14:58:12.116648+0000","last_became_peered":"2024-01-16T14:58:12.116648+0000","last_unstale":"2024-01-16T14:58:13.141154+0000","last_undegraded":"2024-01-16T14:58:13.141154+0000","last_fullsized":"2024-01-16T14:58:13.141154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:12.552354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.121854+0000","last_change":"2024-01-16T14:58:13.114884+0000","last_active":"2024-01-16T14:58:13.121854+0000","last_peered":"2024-01-16T14:58:13.121854+0000","last_clean":"2024-01-16T14:58:13.121854+0000","last_became_active":"2024-01-16T14:58:13.114192+0000","last_became_peered":"2024-01-16T14:58:13.114192+0000","last_unstale":"2024-01-16T14:58:13.121854+0000","last_undegraded":"2024-01-16T14:58:13.121854+0000","last_fullsized":"2024-01-16T14:58:13.121854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101197+0000","last_change":"2024-01-16T14:58:12.108522+0000","last_active":"2024-01-16T14:58:13.101197+0000","last_peered":"2024-01-16T14:58:13.101197+0000","last_clean":"2024-01-16T14:58:13.101197+0000","last_became_active":"2024-01-16T14:58:12.106159+0000","last_became_peered":"2024-01-16T14:58:12.106159+0000","last_unstale":"2024-01-16T14:58:13.101197+0000","last_undegraded":"2024-01-16T14:58:13.101197+0000","last_fullsized":"2024-01-16T14:58:13.101197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:27.372519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126102+0000","last_change":"2024-01-16T14:58:13.119476+0000","last_active":"2024-01-16T14:58:13.126102+0000","last_peered":"2024-01-16T14:58:13.126102+0000","last_clean":"2024-01-16T14:58:13.126102+0000","last_became_active":"2024-01-16T14:58:13.119227+0000","last_became_peered":"2024-01-16T14:58:13.119227+0000","last_unstale":"2024-01-16T14:58:13.126102+0000","last_undegraded":"2024-01-16T14:58:13.126102+0000","last_fullsized":"2024-01-16T14:58:13.126102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.094837+0000","last_change":"2024-01-16T14:58:12.115921+0000","last_active":"2024-01-16T14:58:13.094837+0000","last_peered":"2024-01-16T14:58:13.094837+0000","last_clean":"2024-01-16T14:58:13.094837+0000","last_became_active":"2024-01-16T14:58:12.114246+0000","last_became_peered":"2024-01-16T14:58:12.114246+0000","last_unstale":"2024-01-16T14:58:13.094837+0000","last_undegraded":"2024-01-16T14:58:13.094837+0000","last_fullsized":"2024-01-16T14:58:13.094837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:52.748385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133559+0000","last_change":"2024-01-16T14:58:13.128730+0000","last_active":"2024-01-16T14:58:13.133559+0000","last_peered":"2024-01-16T14:58:13.133559+0000","last_clean":"2024-01-16T14:58:13.133559+0000","last_became_active":"2024-01-16T14:58:13.128458+0000","last_became_peered":"2024-01-16T14:58:13.128458+0000","last_unstale":"2024-01-16T14:58:13.133559+0000","last_undegraded":"2024-01-16T14:58:13.133559+0000","last_fullsized":"2024-01-16T14:58:13.133559+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.110105+0000","last_change":"2024-01-16T14:58:12.109931+0000","last_active":"2024-01-16T14:58:13.110105+0000","last_peered":"2024-01-16T14:58:13.110105+0000","last_clean":"2024-01-16T14:58:13.110105+0000","last_became_active":"2024-01-16T14:58:12.109693+0000","last_became_peered":"2024-01-16T14:58:12.109693+0000","last_unstale":"2024-01-16T14:58:13.110105+0000","last_undegraded":"2024-01-16T14:58:13.110105+0000","last_fullsized":"2024-01-16T14:58:13.110105+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:24.165221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135451+0000","last_change":"2024-01-16T14:58:13.122057+0000","last_active":"2024-01-16T14:58:13.135451+0000","last_peered":"2024-01-16T14:58:13.135451+0000","last_clean":"2024-01-16T14:58:13.135451+0000","last_became_active":"2024-01-16T14:58:13.121786+0000","last_became_peered":"2024-01-16T14:58:13.121786+0000","last_unstale":"2024-01-16T14:58:13.135451+0000","last_undegraded":"2024-01-16T14:58:13.135451+0000","last_fullsized":"2024-01-16T14:58:13.135451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089403+0000","last_change":"2024-01-16T14:58:12.120831+0000","last_active":"2024-01-16T14:58:13.089403+0000","last_peered":"2024-01-16T14:58:13.089403+0000","last_clean":"2024-01-16T14:58:13.089403+0000","last_became_active":"2024-01-16T14:58:12.120541+0000","last_became_peered":"2024-01-16T14:58:12.120541+0000","last_unstale":"2024-01-16T14:58:13.089403+0000","last_undegraded":"2024-01-16T14:58:13.089403+0000","last_fullsized":"2024-01-16T14:58:13.089403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:52.748385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.127817+0000","last_change":"2024-01-16T14:58:13.118989+0000","last_active":"2024-01-16T14:58:13.127817+0000","last_peered":"2024-01-16T14:58:13.127817+0000","last_clean":"2024-01-16T14:58:13.127817+0000","last_became_active":"2024-01-16T14:58:13.118691+0000","last_became_peered":"2024-01-16T14:58:13.118691+0000","last_unstale":"2024-01-16T14:58:13.127817+0000","last_undegraded":"2024-01-16T14:58:13.127817+0000","last_fullsized":"2024-01-16T14:58:13.127817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.141748+0000","last_change":"2024-01-16T14:58:12.128472+0000","last_active":"2024-01-16T14:58:13.141748+0000","last_peered":"2024-01-16T14:58:13.141748+0000","last_clean":"2024-01-16T14:58:13.141748+0000","last_became_active":"2024-01-16T14:58:12.128197+0000","last_became_peered":"2024-01-16T14:58:12.128197+0000","last_unstale":"2024-01-16T14:58:13.141748+0000","last_undegraded":"2024-01-16T14:58:13.141748+0000","last_fullsized":"2024-01-16T14:58:13.141748+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:33.025457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123464+0000","last_change":"2024-01-16T14:58:13.112760+0000","last_active":"2024-01-16T14:58:13.123464+0000","last_peered":"2024-01-16T14:58:13.123464+0000","last_clean":"2024-01-16T14:58:13.123464+0000","last_became_active":"2024-01-16T14:58:13.110324+0000","last_became_peered":"2024-01-16T14:58:13.110324+0000","last_unstale":"2024-01-16T14:58:13.123464+0000","last_undegraded":"2024-01-16T14:58:13.123464+0000","last_fullsized":"2024-01-16T14:58:13.123464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.145569+0000","last_change":"2024-01-16T14:58:12.111745+0000","last_active":"2024-01-16T14:58:13.145569+0000","last_peered":"2024-01-16T14:58:13.145569+0000","last_clean":"2024-01-16T14:58:13.145569+0000","last_became_active":"2024-01-16T14:58:12.110894+0000","last_became_peered":"2024-01-16T14:58:12.110894+0000","last_unstale":"2024-01-16T14:58:13.145569+0000","last_undegraded":"2024-01-16T14:58:13.145569+0000","last_fullsized":"2024-01-16T14:58:13.145569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:49.347613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.116888+0000","last_change":"2024-01-16T14:58:13.111340+0000","last_active":"2024-01-16T14:58:13.116888+0000","last_peered":"2024-01-16T14:58:13.116888+0000","last_clean":"2024-01-16T14:58:13.116888+0000","last_became_active":"2024-01-16T14:58:13.111098+0000","last_became_peered":"2024-01-16T14:58:13.111098+0000","last_unstale":"2024-01-16T14:58:13.116888+0000","last_undegraded":"2024-01-16T14:58:13.116888+0000","last_fullsized":"2024-01-16T14:58:13.116888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138619+0000","last_change":"2024-01-16T14:58:12.122476+0000","last_active":"2024-01-16T14:58:13.138619+0000","last_peered":"2024-01-16T14:58:13.138619+0000","last_clean":"2024-01-16T14:58:13.138619+0000","last_became_active":"2024-01-16T14:58:12.122185+0000","last_became_peered":"2024-01-16T14:58:12.122185+0000","last_unstale":"2024-01-16T14:58:13.138619+0000","last_undegraded":"2024-01-16T14:58:13.138619+0000","last_fullsized":"2024-01-16T14:58:13.138619+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:02.009374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.134881+0000","last_change":"2024-01-16T14:58:13.124142+0000","last_active":"2024-01-16T14:58:13.134881+0000","last_peered":"2024-01-16T14:58:13.134881+0000","last_clean":"2024-01-16T14:58:13.134881+0000","last_became_active":"2024-01-16T14:58:13.123531+0000","last_became_peered":"2024-01-16T14:58:13.123531+0000","last_unstale":"2024-01-16T14:58:13.134881+0000","last_undegraded":"2024-01-16T14:58:13.134881+0000","last_fullsized":"2024-01-16T14:58:13.134881+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:03.020765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104740+0000","last_change":"2024-01-16T14:58:12.117665+0000","last_active":"2024-01-16T14:58:13.104740+0000","last_peered":"2024-01-16T14:58:13.104740+0000","last_clean":"2024-01-16T14:58:13.104740+0000","last_became_active":"2024-01-16T14:58:12.117392+0000","last_became_peered":"2024-01-16T14:58:12.117392+0000","last_unstale":"2024-01-16T14:58:13.104740+0000","last_undegraded":"2024-01-16T14:58:13.104740+0000","last_fullsized":"2024-01-16T14:58:13.104740+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125647+0000","last_change":"2024-01-16T14:58:13.117989+0000","last_active":"2024-01-16T14:58:13.125647+0000","last_peered":"2024-01-16T14:58:13.125647+0000","last_clean":"2024-01-16T14:58:13.125647+0000","last_became_active":"2024-01-16T14:58:13.117355+0000","last_became_peered":"2024-01-16T14:58:13.117355+0000","last_unstale":"2024-01-16T14:58:13.125647+0000","last_undegraded":"2024-01-16T14:58:13.125647+0000","last_fullsized":"2024-01-16T14:58:13.125647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:34.036848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101565+0000","last_change":"2024-01-16T14:58:12.112239+0000","last_active":"2024-01-16T14:58:13.101565+0000","last_peered":"2024-01-16T14:58:13.101565+0000","last_clean":"2024-01-16T14:58:13.101565+0000","last_became_active":"2024-01-16T14:58:12.111952+0000","last_became_peered":"2024-01-16T14:58:12.111952+0000","last_unstale":"2024-01-16T14:58:13.101565+0000","last_undegraded":"2024-01-16T14:58:13.101565+0000","last_fullsized":"2024-01-16T14:58:13.101565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:32.166222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.122890+0000","last_change":"2024-01-16T14:58:13.118164+0000","last_active":"2024-01-16T14:58:13.122890+0000","last_peered":"2024-01-16T14:58:13.122890+0000","last_clean":"2024-01-16T14:58:13.122890+0000","last_became_active":"2024-01-16T14:58:13.117808+0000","last_became_peered":"2024-01-16T14:58:13.117808+0000","last_unstale":"2024-01-16T14:58:13.122890+0000","last_undegraded":"2024-01-16T14:58:13.122890+0000","last_fullsized":"2024-01-16T14:58:13.122890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142239+0000","last_change":"2024-01-16T14:58:12.109940+0000","last_active":"2024-01-16T14:58:13.142239+0000","last_peered":"2024-01-16T14:58:13.142239+0000","last_clean":"2024-01-16T14:58:13.142239+0000","last_became_active":"2024-01-16T14:58:12.109213+0000","last_became_peered":"2024-01-16T14:58:12.109213+0000","last_unstale":"2024-01-16T14:58:13.142239+0000","last_undegraded":"2024-01-16T14:58:13.142239+0000","last_fullsized":"2024-01-16T14:58:13.142239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.736641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.130268+0000","last_change":"2024-01-16T14:58:13.122826+0000","last_active":"2024-01-16T14:58:13.130268+0000","last_peered":"2024-01-16T14:58:13.130268+0000","last_clean":"2024-01-16T14:58:13.130268+0000","last_became_active":"2024-01-16T14:58:13.121868+0000","last_became_peered":"2024-01-16T14:58:13.121868+0000","last_unstale":"2024-01-16T14:58:13.130268+0000","last_undegraded":"2024-01-16T14:58:13.130268+0000","last_fullsized":"2024-01-16T14:58:13.130268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142917+0000","last_change":"2024-01-16T14:58:12.117807+0000","last_active":"2024-01-16T14:58:13.142917+0000","last_peered":"2024-01-16T14:58:13.142917+0000","last_clean":"2024-01-16T14:58:13.142917+0000","last_became_active":"2024-01-16T14:58:12.117486+0000","last_became_peered":"2024-01-16T14:58:12.117486+0000","last_unstale":"2024-01-16T14:58:13.142917+0000","last_undegraded":"2024-01-16T14:58:13.142917+0000","last_fullsized":"2024-01-16T14:58:13.142917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:02.009374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123558+0000","last_change":"2024-01-16T14:58:13.120443+0000","last_active":"2024-01-16T14:58:13.123558+0000","last_peered":"2024-01-16T14:58:13.123558+0000","last_clean":"2024-01-16T14:58:13.123558+0000","last_became_active":"2024-01-16T14:58:13.118620+0000","last_became_peered":"2024-01-16T14:58:13.118620+0000","last_unstale":"2024-01-16T14:58:13.123558+0000","last_undegraded":"2024-01-16T14:58:13.123558+0000","last_fullsized":"2024-01-16T14:58:13.123558+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135477+0000","last_change":"2024-01-16T14:58:12.123306+0000","last_active":"2024-01-16T14:58:13.135477+0000","last_peered":"2024-01-16T14:58:13.135477+0000","last_clean":"2024-01-16T14:58:13.135477+0000","last_became_active":"2024-01-16T14:58:12.122837+0000","last_became_peered":"2024-01-16T14:58:12.122837+0000","last_unstale":"2024-01-16T14:58:13.135477+0000","last_undegraded":"2024-01-16T14:58:13.135477+0000","last_fullsized":"2024-01-16T14:58:13.135477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:27.372519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119904+0000","last_change":"2024-01-16T14:58:13.114235+0000","last_active":"2024-01-16T14:58:13.119904+0000","last_peered":"2024-01-16T14:58:13.119904+0000","last_clean":"2024-01-16T14:58:13.119904+0000","last_became_active":"2024-01-16T14:58:13.113201+0000","last_became_peered":"2024-01-16T14:58:13.113201+0000","last_unstale":"2024-01-16T14:58:13.119904+0000","last_undegraded":"2024-01-16T14:58:13.119904+0000","last_fullsized":"2024-01-16T14:58:13.119904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.140236+0000","last_change":"2024-01-16T14:58:12.124755+0000","last_active":"2024-01-16T14:58:13.140236+0000","last_peered":"2024-01-16T14:58:13.140236+0000","last_clean":"2024-01-16T14:58:13.140236+0000","last_became_active":"2024-01-16T14:58:12.124272+0000","last_became_peered":"2024-01-16T14:58:12.124272+0000","last_unstale":"2024-01-16T14:58:13.140236+0000","last_undegraded":"2024-01-16T14:58:13.140236+0000","last_fullsized":"2024-01-16T14:58:13.140236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.453323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.134537+0000","last_change":"2024-01-16T14:58:13.127657+0000","last_active":"2024-01-16T14:58:13.134537+0000","last_peered":"2024-01-16T14:58:13.134537+0000","last_clean":"2024-01-16T14:58:13.134537+0000","last_became_active":"2024-01-16T14:58:13.127376+0000","last_became_peered":"2024-01-16T14:58:13.127376+0000","last_unstale":"2024-01-16T14:58:13.134537+0000","last_undegraded":"2024-01-16T14:58:13.134537+0000","last_fullsized":"2024-01-16T14:58:13.134537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:19.082451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.102985+0000","last_change":"2024-01-16T14:58:12.124457+0000","last_active":"2024-01-16T14:58:13.102985+0000","last_peered":"2024-01-16T14:58:13.102985+0000","last_clean":"2024-01-16T14:58:13.102985+0000","last_became_active":"2024-01-16T14:58:12.124185+0000","last_became_peered":"2024-01-16T14:58:12.124185+0000","last_unstale":"2024-01-16T14:58:13.102985+0000","last_undegraded":"2024-01-16T14:58:13.102985+0000","last_fullsized":"2024-01-16T14:58:13.102985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.736641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138205+0000","last_change":"2024-01-16T14:58:13.129178+0000","last_active":"2024-01-16T14:58:13.138205+0000","last_peered":"2024-01-16T14:58:13.138205+0000","last_clean":"2024-01-16T14:58:13.138205+0000","last_became_active":"2024-01-16T14:58:13.125031+0000","last_became_peered":"2024-01-16T14:58:13.125031+0000","last_unstale":"2024-01-16T14:58:13.138205+0000","last_undegraded":"2024-01-16T14:58:13.138205+0000","last_fullsized":"2024-01-16T14:58:13.138205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:10.833062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138601+0000","last_change":"2024-01-16T14:58:12.116108+0000","last_active":"2024-01-16T14:58:13.138601+0000","last_peered":"2024-01-16T14:58:13.138601+0000","last_clean":"2024-01-16T14:58:13.138601+0000","last_became_active":"2024-01-16T14:58:12.115876+0000","last_became_peered":"2024-01-16T14:58:12.115876+0000","last_unstale":"2024-01-16T14:58:13.138601+0000","last_undegraded":"2024-01-16T14:58:13.138601+0000","last_fullsized":"2024-01-16T14:58:13.138601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:53.167551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137869+0000","last_change":"2024-01-16T14:58:13.131810+0000","last_active":"2024-01-16T14:58:13.137869+0000","last_peered":"2024-01-16T14:58:13.137869+0000","last_clean":"2024-01-16T14:58:13.137869+0000","last_became_active":"2024-01-16T14:58:13.131516+0000","last_became_peered":"2024-01-16T14:58:13.131516+0000","last_unstale":"2024-01-16T14:58:13.137869+0000","last_undegraded":"2024-01-16T14:58:13.137869+0000","last_fullsized":"2024-01-16T14:58:13.137869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:00.601483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.116021+0000","last_change":"2024-01-16T14:58:12.124856+0000","last_active":"2024-01-16T14:58:13.116021+0000","last_peered":"2024-01-16T14:58:13.116021+0000","last_clean":"2024-01-16T14:58:13.116021+0000","last_became_active":"2024-01-16T14:58:12.124533+0000","last_became_peered":"2024-01-16T14:58:12.124533+0000","last_unstale":"2024-01-16T14:58:13.116021+0000","last_undegraded":"2024-01-16T14:58:13.116021+0000","last_fullsized":"2024-01-16T14:58:13.116021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:11.527954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.129071+0000","last_change":"2024-01-16T14:58:13.124148+0000","last_active":"2024-01-16T14:58:13.129071+0000","last_peered":"2024-01-16T14:58:13.129071+0000","last_clean":"2024-01-16T14:58:13.129071+0000","last_became_active":"2024-01-16T14:58:13.123792+0000","last_became_peered":"2024-01-16T14:58:13.123792+0000","last_unstale":"2024-01-16T14:58:13.129071+0000","last_undegraded":"2024-01-16T14:58:13.129071+0000","last_fullsized":"2024-01-16T14:58:13.129071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:38.456905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.094747+0000","last_change":"2024-01-16T14:58:12.119102+0000","last_active":"2024-01-16T14:58:13.094747+0000","last_peered":"2024-01-16T14:58:13.094747+0000","last_clean":"2024-01-16T14:58:13.094747+0000","last_became_active":"2024-01-16T14:58:12.118801+0000","last_became_peered":"2024-01-16T14:58:12.118801+0000","last_unstale":"2024-01-16T14:58:13.094747+0000","last_undegraded":"2024-01-16T14:58:13.094747+0000","last_fullsized":"2024-01-16T14:58:13.094747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:16.609355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126657+0000","last_change":"2024-01-16T14:58:13.122617+0000","last_active":"2024-01-16T14:58:13.126657+0000","last_peered":"2024-01-16T14:58:13.126657+0000","last_clean":"2024-01-16T14:58:13.126657+0000","last_became_active":"2024-01-16T14:58:13.122100+0000","last_became_peered":"2024-01-16T14:58:13.122100+0000","last_unstale":"2024-01-16T14:58:13.126657+0000","last_undegraded":"2024-01-16T14:58:13.126657+0000","last_fullsized":"2024-01-16T14:58:13.126657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104636+0000","last_change":"2024-01-16T14:58:12.124162+0000","last_active":"2024-01-16T14:58:13.104636+0000","last_peered":"2024-01-16T14:58:13.104636+0000","last_clean":"2024-01-16T14:58:13.104636+0000","last_became_active":"2024-01-16T14:58:12.123869+0000","last_became_peered":"2024-01-16T14:58:12.123869+0000","last_unstale":"2024-01-16T14:58:13.104636+0000","last_undegraded":"2024-01-16T14:58:13.104636+0000","last_fullsized":"2024-01-16T14:58:13.104636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:32.166222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137784+0000","last_change":"2024-01-16T14:58:13.130023+0000","last_active":"2024-01-16T14:58:13.137784+0000","last_peered":"2024-01-16T14:58:13.137784+0000","last_clean":"2024-01-16T14:58:13.137784+0000","last_became_active":"2024-01-16T14:58:13.129149+0000","last_became_peered":"2024-01-16T14:58:13.129149+0000","last_unstale":"2024-01-16T14:58:13.137784+0000","last_undegraded":"2024-01-16T14:58:13.137784+0000","last_fullsized":"2024-01-16T14:58:13.137784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.139861+0000","last_change":"2024-01-16T14:58:12.122971+0000","last_active":"2024-01-16T14:58:13.139861+0000","last_peered":"2024-01-16T14:58:13.139861+0000","last_clean":"2024-01-16T14:58:13.139861+0000","last_became_active":"2024-01-16T14:58:12.122636+0000","last_became_peered":"2024-01-16T14:58:12.122636+0000","last_unstale":"2024-01-16T14:58:13.139861+0000","last_undegraded":"2024-01-16T14:58:13.139861+0000","last_fullsized":"2024-01-16T14:58:13.139861+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.453323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123537+0000","last_change":"2024-01-16T14:58:13.120078+0000","last_active":"2024-01-16T14:58:13.123537+0000","last_peered":"2024-01-16T14:58:13.123537+0000","last_clean":"2024-01-16T14:58:13.123537+0000","last_became_active":"2024-01-16T14:58:13.117991+0000","last_became_peered":"2024-01-16T14:58:13.117991+0000","last_unstale":"2024-01-16T14:58:13.123537+0000","last_undegraded":"2024-01-16T14:58:13.123537+0000","last_fullsized":"2024-01-16T14:58:13.123537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:25.176612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089712+0000","last_change":"2024-01-16T14:58:12.121746+0000","last_active":"2024-01-16T14:58:13.089712+0000","last_peered":"2024-01-16T14:58:13.089712+0000","last_clean":"2024-01-16T14:58:13.089712+0000","last_became_active":"2024-01-16T14:58:12.121318+0000","last_became_peered":"2024-01-16T14:58:12.121318+0000","last_unstale":"2024-01-16T14:58:13.089712+0000","last_undegraded":"2024-01-16T14:58:13.089712+0000","last_fullsized":"2024-01-16T14:58:13.089712+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.117325+0000","last_change":"2024-01-16T14:58:13.112000+0000","last_active":"2024-01-16T14:58:13.117325+0000","last_peered":"2024-01-16T14:58:13.117325+0000","last_clean":"2024-01-16T14:58:13.117325+0000","last_became_active":"2024-01-16T14:58:13.111730+0000","last_became_peered":"2024-01-16T14:58:13.111730+0000","last_unstale":"2024-01-16T14:58:13.117325+0000","last_undegraded":"2024-01-16T14:58:13.117325+0000","last_fullsized":"2024-01-16T14:58:13.117325+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.105268+0000","last_change":"2024-01-16T14:58:12.116991+0000","last_active":"2024-01-16T14:58:13.105268+0000","last_peered":"2024-01-16T14:58:13.105268+0000","last_clean":"2024-01-16T14:58:13.105268+0000","last_became_active":"2024-01-16T14:58:12.116733+0000","last_became_peered":"2024-01-16T14:58:12.116733+0000","last_unstale":"2024-01-16T14:58:13.105268+0000","last_undegraded":"2024-01-16T14:58:13.105268+0000","last_fullsized":"2024-01-16T14:58:13.105268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:30:48.911413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.129506+0000","last_change":"2024-01-16T14:58:13.121418+0000","last_active":"2024-01-16T14:58:13.129506+0000","last_peered":"2024-01-16T14:58:13.129506+0000","last_clean":"2024-01-16T14:58:13.129506+0000","last_became_active":"2024-01-16T14:58:13.121024+0000","last_became_peered":"2024-01-16T14:58:13.121024+0000","last_unstale":"2024-01-16T14:58:13.129506+0000","last_undegraded":"2024-01-16T14:58:13.129506+0000","last_fullsized":"2024-01-16T14:58:13.129506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089046+0000","last_change":"2024-01-16T14:58:12.122147+0000","last_active":"2024-01-16T14:58:13.089046+0000","last_peered":"2024-01-16T14:58:13.089046+0000","last_clean":"2024-01-16T14:58:13.089046+0000","last_became_active":"2024-01-16T14:58:12.121837+0000","last_became_peered":"2024-01-16T14:58:12.121837+0000","last_unstale":"2024-01-16T14:58:13.089046+0000","last_undegraded":"2024-01-16T14:58:13.089046+0000","last_fullsized":"2024-01-16T14:58:13.089046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:07.832210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119407+0000","last_change":"2024-01-16T14:58:13.116387+0000","last_active":"2024-01-16T14:58:13.119407+0000","last_peered":"2024-01-16T14:58:13.119407+0000","last_clean":"2024-01-16T14:58:13.119407+0000","last_became_active":"2024-01-16T14:58:13.116124+0000","last_became_peered":"2024-01-16T14:58:13.116124+0000","last_unstale":"2024-01-16T14:58:13.119407+0000","last_undegraded":"2024-01-16T14:58:13.119407+0000","last_fullsized":"2024-01-16T14:58:13.119407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104896+0000","last_change":"2024-01-16T14:58:12.122192+0000","last_active":"2024-01-16T14:58:13.104896+0000","last_peered":"2024-01-16T14:58:13.104896+0000","last_clean":"2024-01-16T14:58:13.104896+0000","last_became_active":"2024-01-16T14:58:12.120390+0000","last_became_peered":"2024-01-16T14:58:12.120390+0000","last_unstale":"2024-01-16T14:58:13.104896+0000","last_undegraded":"2024-01-16T14:58:13.104896+0000","last_fullsized":"2024-01-16T14:58:13.104896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:12.552354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.124636+0000","last_change":"2024-01-16T14:58:13.117548+0000","last_active":"2024-01-16T14:58:13.124636+0000","last_peered":"2024-01-16T14:58:13.124636+0000","last_clean":"2024-01-16T14:58:13.124636+0000","last_became_active":"2024-01-16T14:58:13.117288+0000","last_became_peered":"2024-01-16T14:58:13.117288+0000","last_unstale":"2024-01-16T14:58:13.124636+0000","last_undegraded":"2024-01-16T14:58:13.124636+0000","last_fullsized":"2024-01-16T14:58:13.124636+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:49.464714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.095773+0000","last_change":"2024-01-16T14:58:12.122568+0000","last_active":"2024-01-16T14:58:13.095773+0000","last_peered":"2024-01-16T14:58:13.095773+0000","last_clean":"2024-01-16T14:58:13.095773+0000","last_became_active":"2024-01-16T14:58:12.122229+0000","last_became_peered":"2024-01-16T14:58:12.122229+0000","last_unstale":"2024-01-16T14:58:13.095773+0000","last_undegraded":"2024-01-16T14:58:13.095773+0000","last_fullsized":"2024-01-16T14:58:13.095773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131933+0000","last_change":"2024-01-16T14:58:13.123092+0000","last_active":"2024-01-16T14:58:13.131933+0000","last_peered":"2024-01-16T14:58:13.131933+0000","last_clean":"2024-01-16T14:58:13.131933+0000","last_became_active":"2024-01-16T14:58:13.122683+0000","last_became_peered":"2024-01-16T14:58:13.122683+0000","last_unstale":"2024-01-16T14:58:13.131933+0000","last_undegraded":"2024-01-16T14:58:13.131933+0000","last_fullsized":"2024-01-16T14:58:13.131933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101560+0000","last_change":"2024-01-16T14:58:12.110101+0000","last_active":"2024-01-16T14:58:13.101560+0000","last_peered":"2024-01-16T14:58:13.101560+0000","last_clean":"2024-01-16T14:58:13.101560+0000","last_became_active":"2024-01-16T14:58:12.109615+0000","last_became_peered":"2024-01-16T14:58:12.109615+0000","last_unstale":"2024-01-16T14:58:13.101560+0000","last_undegraded":"2024-01-16T14:58:13.101560+0000","last_fullsized":"2024-01-16T14:58:13.101560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:02.139904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133177+0000","last_change":"2024-01-16T14:58:13.127178+0000","last_active":"2024-01-16T14:58:13.133177+0000","last_peered":"2024-01-16T14:58:13.133177+0000","last_clean":"2024-01-16T14:58:13.133177+0000","last_became_active":"2024-01-16T14:58:13.126936+0000","last_became_peered":"2024-01-16T14:58:13.126936+0000","last_unstale":"2024-01-16T14:58:13.133177+0000","last_undegraded":"2024-01-16T14:58:13.133177+0000","last_fullsized":"2024-01-16T14:58:13.133177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.117332+0000","last_change":"2024-01-16T14:58:12.122955+0000","last_active":"2024-01-16T14:58:13.117332+0000","last_peered":"2024-01-16T14:58:13.117332+0000","last_clean":"2024-01-16T14:58:13.117332+0000","last_became_active":"2024-01-16T14:58:12.122653+0000","last_became_peered":"2024-01-16T14:58:12.122653+0000","last_unstale":"2024-01-16T14:58:13.117332+0000","last_undegraded":"2024-01-16T14:58:13.117332+0000","last_fullsized":"2024-01-16T14:58:13.117332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:07.832210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T14:58:12.107253+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:58:12.107253+0000","last_peered":"2024-01-16T14:58:12.107253+0000","last_clean":"2024-01-16T14:58:12.107253+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:58:12.107253+0000","last_undegraded":"2024-01-16T14:58:12.107253+0000","last_fullsized":"2024-01-16T14:58:12.107253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.6080000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.843}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.8140000000000001,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.204,"15min":1.204},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.206}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.323},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.3700000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.929},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.7490000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.9850000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.7829999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.68899999999999995}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.198}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.2989999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.431},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.4750000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.599}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.589,"15min":1.589},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.526}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.96899999999999997}]},{"osd":1,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.353},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.0820000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1799999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.016}]},{"osd":7,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.615},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.2569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.46},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.417}]},{"osd":1,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.3220000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.75,"5min":2.75,"15min":2.75},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.603}]},{"osd":5,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.9609999999999999,"5min":2.9609999999999999,"15min":2.9609999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.93899999999999995}]},{"osd":6,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":0.999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.367}]},{"osd":7,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":1.1839999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.89200000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":3.226,"5min":3.226,"15min":3.226},"last":1.478},{"interface":"front","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.492,"15min":1.492},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":3.0379999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.258}]},{"osd":6,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9550000000000001,"5min":2.9550000000000001,"15min":2.9550000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.427}]},{"osd":7,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.4399999999999999,"5min":2.4399999999999999,"15min":2.4399999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":3.0979999999999999,"5min":3.0979999999999999,"15min":3.0979999999999999},"last":1.5449999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":145839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":1.302},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":5.9889999999999999,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.80300000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.1130000000000004,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":6.0259999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.91200000000000003}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":6.141,"5min":6.141,"15min":6.141},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.1799999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":5.2009999999999996,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.149},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.786}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.9199999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.349}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":5.024,"5min":5.024,"15min":5.024},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.8849999999999998,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.575}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":4.9930000000000003,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.9199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.97099999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.3160000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.113},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":1.2869999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":1.385},{"interface":"front","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.032},{"interface":"front","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.2569999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":735118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.026},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.202}]},{"osd":3,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.111}]},{"osd":4,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.997},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.23},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.1799999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.141},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:17.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.770+0000 7fc48de00640 1 Processor -- start 2024-01-16T14:58:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.770+0000 7fc48de00640 1 -- start start 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc488143ae0 0x7fc4880cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc4880cf7d0 0x7fc48815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 0x7fc48815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880cfc30 con 0x7fc488143ae0 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880b1ee0 con 0x7fc4880cf7d0 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc48de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880b2780 con 0x7fc48815e1e0 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc486a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc488143ae0 0x7fc4880cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc486a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc488143ae0 0x7fc4880cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57110/0 (socket says 172.21.15.125:57110) 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc486a00640 1 -- 172.21.15.125:0/1386981926 learned_addr learned my addr 172.21.15.125:0/1386981926 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc47fe00640 1 --2- 172.21.15.125:0/1386981926 >> v2:172.21.15.137:3300/0 conn(0x7fc4880cf7d0 0x7fc48815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.772+0000 7fc487400640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 0x7fc48815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc486a00640 1 -- 172.21.15.125:0/1386981926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 msgr2=0x7fc48815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc486a00640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 0x7fc48815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc486a00640 1 -- 172.21.15.125:0/1386981926 >> v2:172.21.15.137:3300/0 conn(0x7fc4880cf7d0 msgr2=0x7fc48815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc486a00640 1 --2- 172.21.15.125:0/1386981926 >> v2:172.21.15.137:3300/0 conn(0x7fc4880cf7d0 0x7fc48815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc486a00640 1 -- 172.21.15.125:0/1386981926 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48815e990 con 0x7fc488143ae0 2024-01-16T14:58:17.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.773+0000 7fc487400640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 0x7fc48815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.774+0000 7fc486a00640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc4880cf290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc4880cb830 tx=0x7fc47802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667bf97f90840eb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.774+0000 7fc486000640 1 -- 172.21.15.125:0/1386981926 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478031870 con 0x7fc488143ae0 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.774+0000 7fc486000640 1 -- 172.21.15.125:0/1386981926 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4780319d0 con 0x7fc488143ae0 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.774+0000 7fc486000640 1 -- 172.21.15.125:0/1386981926 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478031d60 con 0x7fc488143ae0 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 -- 172.21.15.125:0/1386981926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 msgr2=0x7fc4880cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc4880cf290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc4880cb830 tx=0x7fc47802b910 comp rx=0 tx=0).stop 2024-01-16T14:58:17.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 -- 172.21.15.125:0/1386981926 shutdown_connections 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc48815e1e0 0x7fc48815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 --2- 172.21.15.125:0/1386981926 >> v2:172.21.15.137:3300/0 conn(0x7fc4880cf7d0 0x7fc48815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 --2- 172.21.15.125:0/1386981926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc4880cf290 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 -- 172.21.15.125:0/1386981926 >> 172.21.15.125:0/1386981926 conn(0x7fc48806a800 msgr2=0x7fc48806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 -- 172.21.15.125:0/1386981926 shutdown_connections 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.775+0000 7fc48de00640 1 -- 172.21.15.125:0/1386981926 wait complete. 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.777+0000 7fc48de00640 1 Processor -- start 2024-01-16T14:58:17.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.777+0000 7fc48de00640 1 -- start start 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4880cf7d0 0x7fc4880cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc48815e1e0 0x7fc48815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc488134c10 con 0x7fc488143ae0 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48815a1d0 con 0x7fc48815e1e0 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc48de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48815cab0 con 0x7fc4880cf7d0 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57114/0 (socket says 172.21.15.125:57114) 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 -- 172.21.15.125:0/1733237855 learned_addr learned my addr 172.21.15.125:0/1733237855 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 -- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4880cf7d0 msgr2=0x7fc4880cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc486a00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4880cf7d0 0x7fc4880cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4880cf7d0 0x7fc4880cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 -- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc48815e1e0 msgr2=0x7fc48815c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.778+0000 7fc47fe00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc48815e1e0 0x7fc48815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc47fe00640 1 -- 172.21.15.125:0/1733237855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780094d0 con 0x7fc488143ae0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc47fe00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc47000d9c0 tx=0x7fc47000de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000ab80 con 0x7fc488143ae0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47000ace0 con 0x7fc488143ae0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470010690 con 0x7fc488143ae0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48815cd90 con 0x7fc488143ae0 2024-01-16T14:58:17.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.779+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc488134830 con 0x7fc488143ae0 2024-01-16T14:58:17.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.781+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4880b2bf0 con 0x7fc488143ae0 2024-01-16T14:58:17.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.781+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc4700107f0 con 0x7fc488143ae0 2024-01-16T14:58:17.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.782+0000 7fc47f400640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 0x7fc46006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.782+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc470089fc0 con 0x7fc488143ae0 2024-01-16T14:58:17.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.782+0000 7fc486a00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 0x7fc46006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.783+0000 7fc486a00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 0x7fc46006d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc478036000 tx=0x7fc47803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.786+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4700bce90 con 0x7fc488143ae0 2024-01-16T14:58:17.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.974+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4880b18f0 con 0x7fc46006b2c0 2024-01-16T14:58:17.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.991+0000 7fc47f400640 1 -- 172.21.15.125:0/1733237855 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287205 (secure 0 0 0) 0x7fc4880b18f0 con 0x7fc46006b2c0 2024-01-16T14:58:17.977 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:17.978 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:58:17.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 msgr2=0x7fc46006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 0x7fc46006d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc478036000 tx=0x7fc47803c040 comp rx=0 tx=0).stop 2024-01-16T14:58:17.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 msgr2=0x7fc48815bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc47000d9c0 tx=0x7fc47000de80 comp rx=0 tx=0).stop 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 shutdown_connections 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc48815e1e0 0x7fc48815c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc46006b2c0 0x7fc46006d780 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488143ae0 0x7fc48815bf40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 --2- 172.21.15.125:0/1733237855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4880cf7d0 0x7fc4880cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.996+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 >> 172.21.15.125:0/1733237855 conn(0x7fc48806a800 msgr2=0x7fc4880afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.997+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 shutdown_connections 2024-01-16T14:58:17.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:17.997+0000 7fc48de00640 1 -- 172.21.15.125:0/1733237855 wait complete. 2024-01-16T14:58:17.997 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-01-16T14:58:17.945430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220280,"kb_used_data":4112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754440,"statfs":{"total":773094113280,"available":772868546560,"internally_reserved":0,"allocated":4210688,"data_stored":2343298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001698"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131614+0000","last_change":"2024-01-16T14:58:13.124277+0000","last_active":"2024-01-16T14:58:13.131614+0000","last_peered":"2024-01-16T14:58:13.131614+0000","last_clean":"2024-01-16T14:58:13.131614+0000","last_became_active":"2024-01-16T14:58:13.123951+0000","last_became_peered":"2024-01-16T14:58:13.123951+0000","last_unstale":"2024-01-16T14:58:13.131614+0000","last_undegraded":"2024-01-16T14:58:13.131614+0000","last_fullsized":"2024-01-16T14:58:13.131614+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135088+0000","last_change":"2024-01-16T14:58:13.123807+0000","last_active":"2024-01-16T14:58:13.135088+0000","last_peered":"2024-01-16T14:58:13.135088+0000","last_clean":"2024-01-16T14:58:13.135088+0000","last_became_active":"2024-01-16T14:58:13.123519+0000","last_became_peered":"2024-01-16T14:58:13.123519+0000","last_unstale":"2024-01-16T14:58:13.135088+0000","last_undegraded":"2024-01-16T14:58:13.135088+0000","last_fullsized":"2024-01-16T14:58:13.135088+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:06.578017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123476+0000","last_change":"2024-01-16T14:58:13.119873+0000","last_active":"2024-01-16T14:58:13.123476+0000","last_peered":"2024-01-16T14:58:13.123476+0000","last_clean":"2024-01-16T14:58:13.123476+0000","last_became_active":"2024-01-16T14:58:13.119611+0000","last_became_peered":"2024-01-16T14:58:13.119611+0000","last_unstale":"2024-01-16T14:58:13.123476+0000","last_undegraded":"2024-01-16T14:58:13.123476+0000","last_fullsized":"2024-01-16T14:58:13.123476+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.120612+0000","last_change":"2024-01-16T14:58:13.118612+0000","last_active":"2024-01-16T14:58:13.120612+0000","last_peered":"2024-01-16T14:58:13.120612+0000","last_clean":"2024-01-16T14:58:13.120612+0000","last_became_active":"2024-01-16T14:58:13.115549+0000","last_became_peered":"2024-01-16T14:58:13.115549+0000","last_unstale":"2024-01-16T14:58:13.120612+0000","last_undegraded":"2024-01-16T14:58:13.120612+0000","last_fullsized":"2024-01-16T14:58:13.120612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:53.759776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.140281+0000","last_change":"2024-01-16T14:58:13.134287+0000","last_active":"2024-01-16T14:58:13.140281+0000","last_peered":"2024-01-16T14:58:13.140281+0000","last_clean":"2024-01-16T14:58:13.140281+0000","last_became_active":"2024-01-16T14:58:13.133891+0000","last_became_peered":"2024-01-16T14:58:13.133891+0000","last_unstale":"2024-01-16T14:58:13.140281+0000","last_undegraded":"2024-01-16T14:58:13.140281+0000","last_fullsized":"2024-01-16T14:58:13.140281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:12.141938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133643+0000","last_change":"2024-01-16T14:58:13.122659+0000","last_active":"2024-01-16T14:58:13.133643+0000","last_peered":"2024-01-16T14:58:13.133643+0000","last_clean":"2024-01-16T14:58:13.133643+0000","last_became_active":"2024-01-16T14:58:13.122316+0000","last_became_peered":"2024-01-16T14:58:13.122316+0000","last_unstale":"2024-01-16T14:58:13.133643+0000","last_undegraded":"2024-01-16T14:58:13.133643+0000","last_fullsized":"2024-01-16T14:58:13.133643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125362+0000","last_change":"2024-01-16T14:58:13.120991+0000","last_active":"2024-01-16T14:58:13.125362+0000","last_peered":"2024-01-16T14:58:13.125362+0000","last_clean":"2024-01-16T14:58:13.125362+0000","last_became_active":"2024-01-16T14:58:13.120711+0000","last_became_peered":"2024-01-16T14:58:13.120711+0000","last_unstale":"2024-01-16T14:58:13.125362+0000","last_undegraded":"2024-01-16T14:58:13.125362+0000","last_fullsized":"2024-01-16T14:58:13.125362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:00.486870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133123+0000","last_change":"2024-01-16T14:58:13.126979+0000","last_active":"2024-01-16T14:58:13.133123+0000","last_peered":"2024-01-16T14:58:13.133123+0000","last_clean":"2024-01-16T14:58:13.133123+0000","last_became_active":"2024-01-16T14:58:13.126583+0000","last_became_peered":"2024-01-16T14:58:13.126583+0000","last_unstale":"2024-01-16T14:58:13.133123+0000","last_undegraded":"2024-01-16T14:58:13.133123+0000","last_fullsized":"2024-01-16T14:58:13.133123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125064+0000","last_change":"2024-01-16T14:58:13.113431+0000","last_active":"2024-01-16T14:58:13.125064+0000","last_peered":"2024-01-16T14:58:13.125064+0000","last_clean":"2024-01-16T14:58:13.125064+0000","last_became_active":"2024-01-16T14:58:13.113135+0000","last_became_peered":"2024-01-16T14:58:13.113135+0000","last_unstale":"2024-01-16T14:58:13.125064+0000","last_undegraded":"2024-01-16T14:58:13.125064+0000","last_fullsized":"2024-01-16T14:58:13.125064+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:08.843601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138168+0000","last_change":"2024-01-16T14:58:13.129831+0000","last_active":"2024-01-16T14:58:13.138168+0000","last_peered":"2024-01-16T14:58:13.138168+0000","last_clean":"2024-01-16T14:58:13.138168+0000","last_became_active":"2024-01-16T14:58:13.129463+0000","last_became_peered":"2024-01-16T14:58:13.129463+0000","last_unstale":"2024-01-16T14:58:13.138168+0000","last_undegraded":"2024-01-16T14:58:13.138168+0000","last_fullsized":"2024-01-16T14:58:13.138168+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.121897+0000","last_change":"2024-01-16T14:58:13.114216+0000","last_active":"2024-01-16T14:58:13.121897+0000","last_peered":"2024-01-16T14:58:13.121897+0000","last_clean":"2024-01-16T14:58:13.121897+0000","last_became_active":"2024-01-16T14:58:13.113901+0000","last_became_peered":"2024-01-16T14:58:13.113901+0000","last_unstale":"2024-01-16T14:58:13.121897+0000","last_undegraded":"2024-01-16T14:58:13.121897+0000","last_fullsized":"2024-01-16T14:58:13.121897+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:06.578017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133561+0000","last_change":"2024-01-16T14:58:13.129383+0000","last_active":"2024-01-16T14:58:13.133561+0000","last_peered":"2024-01-16T14:58:13.133561+0000","last_clean":"2024-01-16T14:58:13.133561+0000","last_became_active":"2024-01-16T14:58:13.129124+0000","last_became_peered":"2024-01-16T14:58:13.129124+0000","last_unstale":"2024-01-16T14:58:13.133561+0000","last_undegraded":"2024-01-16T14:58:13.133561+0000","last_fullsized":"2024-01-16T14:58:13.133561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137734+0000","last_change":"2024-01-16T14:58:13.129617+0000","last_active":"2024-01-16T14:58:13.137734+0000","last_peered":"2024-01-16T14:58:13.137734+0000","last_clean":"2024-01-16T14:58:13.137734+0000","last_became_active":"2024-01-16T14:58:13.127243+0000","last_became_peered":"2024-01-16T14:58:13.127243+0000","last_unstale":"2024-01-16T14:58:13.137734+0000","last_undegraded":"2024-01-16T14:58:13.137734+0000","last_fullsized":"2024-01-16T14:58:13.137734+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:53.475795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.143003+0000","last_change":"2024-01-16T14:58:13.140258+0000","last_active":"2024-01-16T14:58:13.143003+0000","last_peered":"2024-01-16T14:58:13.143003+0000","last_clean":"2024-01-16T14:58:13.143003+0000","last_became_active":"2024-01-16T14:58:13.138930+0000","last_became_peered":"2024-01-16T14:58:13.138930+0000","last_unstale":"2024-01-16T14:58:13.143003+0000","last_undegraded":"2024-01-16T14:58:13.143003+0000","last_fullsized":"2024-01-16T14:58:13.143003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123578+0000","last_change":"2024-01-16T14:58:13.121102+0000","last_active":"2024-01-16T14:58:13.123578+0000","last_peered":"2024-01-16T14:58:13.123578+0000","last_clean":"2024-01-16T14:58:13.123578+0000","last_became_active":"2024-01-16T14:58:13.120831+0000","last_became_peered":"2024-01-16T14:58:13.120831+0000","last_unstale":"2024-01-16T14:58:13.123578+0000","last_undegraded":"2024-01-16T14:58:13.123578+0000","last_fullsized":"2024-01-16T14:58:13.123578+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:04.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126888+0000","last_change":"2024-01-16T14:58:13.124979+0000","last_active":"2024-01-16T14:58:13.126888+0000","last_peered":"2024-01-16T14:58:13.126888+0000","last_clean":"2024-01-16T14:58:13.126888+0000","last_became_active":"2024-01-16T14:58:13.124684+0000","last_became_peered":"2024-01-16T14:58:13.124684+0000","last_unstale":"2024-01-16T14:58:13.126888+0000","last_undegraded":"2024-01-16T14:58:13.126888+0000","last_fullsized":"2024-01-16T14:58:13.126888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:25.176612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125165+0000","last_change":"2024-01-16T14:58:13.120735+0000","last_active":"2024-01-16T14:58:13.125165+0000","last_peered":"2024-01-16T14:58:13.125165+0000","last_clean":"2024-01-16T14:58:13.125165+0000","last_became_active":"2024-01-16T14:58:13.120230+0000","last_became_peered":"2024-01-16T14:58:13.120230+0000","last_unstale":"2024-01-16T14:58:13.125165+0000","last_undegraded":"2024-01-16T14:58:13.125165+0000","last_fullsized":"2024-01-16T14:58:13.125165+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135076+0000","last_change":"2024-01-16T14:58:13.123169+0000","last_active":"2024-01-16T14:58:13.135076+0000","last_peered":"2024-01-16T14:58:13.135076+0000","last_clean":"2024-01-16T14:58:13.135076+0000","last_became_active":"2024-01-16T14:58:13.122693+0000","last_became_peered":"2024-01-16T14:58:13.122693+0000","last_unstale":"2024-01-16T14:58:13.135076+0000","last_undegraded":"2024-01-16T14:58:13.135076+0000","last_fullsized":"2024-01-16T14:58:13.135076+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:01.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.128938+0000","last_change":"2024-01-16T14:58:13.120657+0000","last_active":"2024-01-16T14:58:13.128938+0000","last_peered":"2024-01-16T14:58:13.128938+0000","last_clean":"2024-01-16T14:58:13.128938+0000","last_became_active":"2024-01-16T14:58:13.120332+0000","last_became_peered":"2024-01-16T14:58:13.120332+0000","last_unstale":"2024-01-16T14:58:13.128938+0000","last_undegraded":"2024-01-16T14:58:13.128938+0000","last_fullsized":"2024-01-16T14:58:13.128938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:34.036848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133745+0000","last_change":"2024-01-16T14:58:13.125993+0000","last_active":"2024-01-16T14:58:13.133745+0000","last_peered":"2024-01-16T14:58:13.133745+0000","last_clean":"2024-01-16T14:58:13.133745+0000","last_became_active":"2024-01-16T14:58:13.125680+0000","last_became_peered":"2024-01-16T14:58:13.125680+0000","last_unstale":"2024-01-16T14:58:13.133745+0000","last_undegraded":"2024-01-16T14:58:13.133745+0000","last_fullsized":"2024-01-16T14:58:13.133745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.112508+0000","last_change":"2024-01-16T14:58:13.101390+0000","last_active":"2024-01-16T14:58:13.112508+0000","last_peered":"2024-01-16T14:58:13.112508+0000","last_clean":"2024-01-16T14:58:13.112508+0000","last_became_active":"2024-01-16T14:58:13.100325+0000","last_became_peered":"2024-01-16T14:58:13.100325+0000","last_unstale":"2024-01-16T14:58:13.112508+0000","last_undegraded":"2024-01-16T14:58:13.112508+0000","last_fullsized":"2024-01-16T14:58:13.112508+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126551+0000","last_change":"2024-01-16T14:58:13.122121+0000","last_active":"2024-01-16T14:58:13.126551+0000","last_peered":"2024-01-16T14:58:13.126551+0000","last_clean":"2024-01-16T14:58:13.126551+0000","last_became_active":"2024-01-16T14:58:13.121564+0000","last_became_peered":"2024-01-16T14:58:13.121564+0000","last_unstale":"2024-01-16T14:58:13.126551+0000","last_undegraded":"2024-01-16T14:58:13.126551+0000","last_fullsized":"2024-01-16T14:58:13.126551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.114637+0000","last_change":"2024-01-16T14:58:13.103203+0000","last_active":"2024-01-16T14:58:13.114637+0000","last_peered":"2024-01-16T14:58:13.114637+0000","last_clean":"2024-01-16T14:58:13.114637+0000","last_became_active":"2024-01-16T14:58:13.102474+0000","last_became_peered":"2024-01-16T14:58:13.102474+0000","last_unstale":"2024-01-16T14:58:13.114637+0000","last_undegraded":"2024-01-16T14:58:13.114637+0000","last_fullsized":"2024-01-16T14:58:13.114637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:17.620746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133539+0000","last_change":"2024-01-16T14:58:13.128584+0000","last_active":"2024-01-16T14:58:13.133539+0000","last_peered":"2024-01-16T14:58:13.133539+0000","last_clean":"2024-01-16T14:58:13.133539+0000","last_became_active":"2024-01-16T14:58:13.128008+0000","last_became_peered":"2024-01-16T14:58:13.128008+0000","last_unstale":"2024-01-16T14:58:13.133539+0000","last_undegraded":"2024-01-16T14:58:13.133539+0000","last_fullsized":"2024-01-16T14:58:13.133539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:42.748033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.114524+0000","last_change":"2024-01-16T14:58:13.111335+0000","last_active":"2024-01-16T14:58:13.114524+0000","last_peered":"2024-01-16T14:58:13.114524+0000","last_clean":"2024-01-16T14:58:13.114524+0000","last_became_active":"2024-01-16T14:58:13.111073+0000","last_became_peered":"2024-01-16T14:58:13.111073+0000","last_unstale":"2024-01-16T14:58:13.114524+0000","last_undegraded":"2024-01-16T14:58:13.114524+0000","last_fullsized":"2024-01-16T14:58:13.114524+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:17.620746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135098+0000","last_change":"2024-01-16T14:58:13.124443+0000","last_active":"2024-01-16T14:58:13.135098+0000","last_peered":"2024-01-16T14:58:13.135098+0000","last_clean":"2024-01-16T14:58:13.135098+0000","last_became_active":"2024-01-16T14:58:13.124106+0000","last_became_peered":"2024-01-16T14:58:13.124106+0000","last_unstale":"2024-01-16T14:58:13.135098+0000","last_undegraded":"2024-01-16T14:58:13.135098+0000","last_fullsized":"2024-01-16T14:58:13.135098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.120820+0000","last_change":"2024-01-16T14:58:13.116915+0000","last_active":"2024-01-16T14:58:13.120820+0000","last_peered":"2024-01-16T14:58:13.120820+0000","last_clean":"2024-01-16T14:58:13.120820+0000","last_became_active":"2024-01-16T14:58:13.116257+0000","last_became_peered":"2024-01-16T14:58:13.116257+0000","last_unstale":"2024-01-16T14:58:13.120820+0000","last_undegraded":"2024-01-16T14:58:13.120820+0000","last_fullsized":"2024-01-16T14:58:13.120820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:08.843601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.122252+0000","last_change":"2024-01-16T14:58:13.116110+0000","last_active":"2024-01-16T14:58:13.122252+0000","last_peered":"2024-01-16T14:58:13.122252+0000","last_clean":"2024-01-16T14:58:13.122252+0000","last_became_active":"2024-01-16T14:58:13.115700+0000","last_became_peered":"2024-01-16T14:58:13.115700+0000","last_unstale":"2024-01-16T14:58:13.122252+0000","last_undegraded":"2024-01-16T14:58:13.122252+0000","last_fullsized":"2024-01-16T14:58:13.122252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:28.383910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.132007+0000","last_change":"2024-01-16T14:58:13.128079+0000","last_active":"2024-01-16T14:58:13.132007+0000","last_peered":"2024-01-16T14:58:13.132007+0000","last_clean":"2024-01-16T14:58:13.132007+0000","last_became_active":"2024-01-16T14:58:13.127568+0000","last_became_peered":"2024-01-16T14:58:13.127568+0000","last_unstale":"2024-01-16T14:58:13.132007+0000","last_undegraded":"2024-01-16T14:58:13.132007+0000","last_fullsized":"2024-01-16T14:58:13.132007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.124382+0000","last_change":"2024-01-16T14:58:13.117151+0000","last_active":"2024-01-16T14:58:13.124382+0000","last_peered":"2024-01-16T14:58:13.124382+0000","last_clean":"2024-01-16T14:58:13.124382+0000","last_became_active":"2024-01-16T14:58:13.116817+0000","last_became_peered":"2024-01-16T14:58:13.116817+0000","last_unstale":"2024-01-16T14:58:13.124382+0000","last_undegraded":"2024-01-16T14:58:13.124382+0000","last_fullsized":"2024-01-16T14:58:13.124382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.130513+0000","last_change":"2024-01-16T14:58:13.126783+0000","last_active":"2024-01-16T14:58:13.130513+0000","last_peered":"2024-01-16T14:58:13.130513+0000","last_clean":"2024-01-16T14:58:13.130513+0000","last_became_active":"2024-01-16T14:58:13.126473+0000","last_became_peered":"2024-01-16T14:58:13.126473+0000","last_unstale":"2024-01-16T14:58:13.130513+0000","last_undegraded":"2024-01-16T14:58:13.130513+0000","last_fullsized":"2024-01-16T14:58:13.130513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123986+0000","last_change":"2024-01-16T14:58:13.116621+0000","last_active":"2024-01-16T14:58:13.123986+0000","last_peered":"2024-01-16T14:58:13.123986+0000","last_clean":"2024-01-16T14:58:13.123986+0000","last_became_active":"2024-01-16T14:58:13.116331+0000","last_became_peered":"2024-01-16T14:58:13.116331+0000","last_unstale":"2024-01-16T14:58:13.123986+0000","last_undegraded":"2024-01-16T14:58:13.123986+0000","last_fullsized":"2024-01-16T14:58:13.123986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:03.151295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.112302+0000","last_change":"2024-01-16T14:58:13.100907+0000","last_active":"2024-01-16T14:58:13.112302+0000","last_peered":"2024-01-16T14:58:13.112302+0000","last_clean":"2024-01-16T14:58:13.112302+0000","last_became_active":"2024-01-16T14:58:13.100149+0000","last_became_peered":"2024-01-16T14:58:13.100149+0000","last_unstale":"2024-01-16T14:58:13.112302+0000","last_undegraded":"2024-01-16T14:58:13.112302+0000","last_fullsized":"2024-01-16T14:58:13.112302+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.139129+0000","last_change":"2024-01-16T14:58:12.126242+0000","last_active":"2024-01-16T14:58:13.139129+0000","last_peered":"2024-01-16T14:58:13.139129+0000","last_clean":"2024-01-16T14:58:13.139129+0000","last_became_active":"2024-01-16T14:58:12.125733+0000","last_became_peered":"2024-01-16T14:58:12.125733+0000","last_unstale":"2024-01-16T14:58:13.139129+0000","last_undegraded":"2024-01-16T14:58:13.139129+0000","last_fullsized":"2024-01-16T14:58:13.139129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:17.174671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131637+0000","last_change":"2024-01-16T14:58:13.126298+0000","last_active":"2024-01-16T14:58:13.131637+0000","last_peered":"2024-01-16T14:58:13.131637+0000","last_clean":"2024-01-16T14:58:13.131637+0000","last_became_active":"2024-01-16T14:58:13.125956+0000","last_became_peered":"2024-01-16T14:58:13.125956+0000","last_unstale":"2024-01-16T14:58:13.131637+0000","last_undegraded":"2024-01-16T14:58:13.131637+0000","last_fullsized":"2024-01-16T14:58:13.131637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137278+0000","last_change":"2024-01-16T14:58:12.123554+0000","last_active":"2024-01-16T14:58:13.137278+0000","last_peered":"2024-01-16T14:58:13.137278+0000","last_clean":"2024-01-16T14:58:13.137278+0000","last_became_active":"2024-01-16T14:58:12.123224+0000","last_became_peered":"2024-01-16T14:58:12.123224+0000","last_unstale":"2024-01-16T14:58:13.137278+0000","last_undegraded":"2024-01-16T14:58:13.137278+0000","last_fullsized":"2024-01-16T14:58:13.137278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:11.527954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131846+0000","last_change":"2024-01-16T14:58:13.124444+0000","last_active":"2024-01-16T14:58:13.131846+0000","last_peered":"2024-01-16T14:58:13.131846+0000","last_clean":"2024-01-16T14:58:13.131846+0000","last_became_active":"2024-01-16T14:58:13.124149+0000","last_became_peered":"2024-01-16T14:58:13.124149+0000","last_unstale":"2024-01-16T14:58:13.131846+0000","last_undegraded":"2024-01-16T14:58:13.131846+0000","last_fullsized":"2024-01-16T14:58:13.131846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:12.539345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142268+0000","last_change":"2024-01-16T14:58:12.118834+0000","last_active":"2024-01-16T14:58:13.142268+0000","last_peered":"2024-01-16T14:58:13.142268+0000","last_clean":"2024-01-16T14:58:13.142268+0000","last_became_active":"2024-01-16T14:58:12.118535+0000","last_became_peered":"2024-01-16T14:58:12.118535+0000","last_unstale":"2024-01-16T14:58:13.142268+0000","last_undegraded":"2024-01-16T14:58:13.142268+0000","last_fullsized":"2024-01-16T14:58:13.142268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:59.475479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119440+0000","last_change":"2024-01-16T14:58:13.114187+0000","last_active":"2024-01-16T14:58:13.119440+0000","last_peered":"2024-01-16T14:58:13.119440+0000","last_clean":"2024-01-16T14:58:13.119440+0000","last_became_active":"2024-01-16T14:58:13.113937+0000","last_became_peered":"2024-01-16T14:58:13.113937+0000","last_unstale":"2024-01-16T14:58:13.119440+0000","last_undegraded":"2024-01-16T14:58:13.119440+0000","last_fullsized":"2024-01-16T14:58:13.119440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:12.539345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138918+0000","last_change":"2024-01-16T14:58:12.112457+0000","last_active":"2024-01-16T14:58:13.138918+0000","last_peered":"2024-01-16T14:58:13.138918+0000","last_clean":"2024-01-16T14:58:13.138918+0000","last_became_active":"2024-01-16T14:58:12.111173+0000","last_became_peered":"2024-01-16T14:58:12.111173+0000","last_unstale":"2024-01-16T14:58:13.138918+0000","last_undegraded":"2024-01-16T14:58:13.138918+0000","last_fullsized":"2024-01-16T14:58:13.138918+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:53.167551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123702+0000","last_change":"2024-01-16T14:58:13.117046+0000","last_active":"2024-01-16T14:58:13.123702+0000","last_peered":"2024-01-16T14:58:13.123702+0000","last_clean":"2024-01-16T14:58:13.123702+0000","last_became_active":"2024-01-16T14:58:13.115382+0000","last_became_peered":"2024-01-16T14:58:13.115382+0000","last_unstale":"2024-01-16T14:58:13.123702+0000","last_undegraded":"2024-01-16T14:58:13.123702+0000","last_fullsized":"2024-01-16T14:58:13.123702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142502+0000","last_change":"2024-01-16T14:58:12.117463+0000","last_active":"2024-01-16T14:58:13.142502+0000","last_peered":"2024-01-16T14:58:13.142502+0000","last_clean":"2024-01-16T14:58:13.142502+0000","last_became_active":"2024-01-16T14:58:12.117178+0000","last_became_peered":"2024-01-16T14:58:12.117178+0000","last_unstale":"2024-01-16T14:58:13.142502+0000","last_undegraded":"2024-01-16T14:58:13.142502+0000","last_fullsized":"2024-01-16T14:58:13.142502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:30:48.911413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123056+0000","last_change":"2024-01-16T14:58:13.116633+0000","last_active":"2024-01-16T14:58:13.123056+0000","last_peered":"2024-01-16T14:58:13.123056+0000","last_clean":"2024-01-16T14:58:13.123056+0000","last_became_active":"2024-01-16T14:58:13.116090+0000","last_became_peered":"2024-01-16T14:58:13.116090+0000","last_unstale":"2024-01-16T14:58:13.123056+0000","last_undegraded":"2024-01-16T14:58:13.123056+0000","last_fullsized":"2024-01-16T14:58:13.123056+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:03.151295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.141154+0000","last_change":"2024-01-16T14:58:12.116948+0000","last_active":"2024-01-16T14:58:13.141154+0000","last_peered":"2024-01-16T14:58:13.141154+0000","last_clean":"2024-01-16T14:58:13.141154+0000","last_became_active":"2024-01-16T14:58:12.116648+0000","last_became_peered":"2024-01-16T14:58:12.116648+0000","last_unstale":"2024-01-16T14:58:13.141154+0000","last_undegraded":"2024-01-16T14:58:13.141154+0000","last_fullsized":"2024-01-16T14:58:13.141154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:12.552354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.121854+0000","last_change":"2024-01-16T14:58:13.114884+0000","last_active":"2024-01-16T14:58:13.121854+0000","last_peered":"2024-01-16T14:58:13.121854+0000","last_clean":"2024-01-16T14:58:13.121854+0000","last_became_active":"2024-01-16T14:58:13.114192+0000","last_became_peered":"2024-01-16T14:58:13.114192+0000","last_unstale":"2024-01-16T14:58:13.121854+0000","last_undegraded":"2024-01-16T14:58:13.121854+0000","last_fullsized":"2024-01-16T14:58:13.121854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101197+0000","last_change":"2024-01-16T14:58:12.108522+0000","last_active":"2024-01-16T14:58:13.101197+0000","last_peered":"2024-01-16T14:58:13.101197+0000","last_clean":"2024-01-16T14:58:13.101197+0000","last_became_active":"2024-01-16T14:58:12.106159+0000","last_became_peered":"2024-01-16T14:58:12.106159+0000","last_unstale":"2024-01-16T14:58:13.101197+0000","last_undegraded":"2024-01-16T14:58:13.101197+0000","last_fullsized":"2024-01-16T14:58:13.101197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:27.372519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126102+0000","last_change":"2024-01-16T14:58:13.119476+0000","last_active":"2024-01-16T14:58:13.126102+0000","last_peered":"2024-01-16T14:58:13.126102+0000","last_clean":"2024-01-16T14:58:13.126102+0000","last_became_active":"2024-01-16T14:58:13.119227+0000","last_became_peered":"2024-01-16T14:58:13.119227+0000","last_unstale":"2024-01-16T14:58:13.126102+0000","last_undegraded":"2024-01-16T14:58:13.126102+0000","last_fullsized":"2024-01-16T14:58:13.126102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:54.178942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.094837+0000","last_change":"2024-01-16T14:58:12.115921+0000","last_active":"2024-01-16T14:58:13.094837+0000","last_peered":"2024-01-16T14:58:13.094837+0000","last_clean":"2024-01-16T14:58:13.094837+0000","last_became_active":"2024-01-16T14:58:12.114246+0000","last_became_peered":"2024-01-16T14:58:12.114246+0000","last_unstale":"2024-01-16T14:58:13.094837+0000","last_undegraded":"2024-01-16T14:58:13.094837+0000","last_fullsized":"2024-01-16T14:58:13.094837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:52.748385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133559+0000","last_change":"2024-01-16T14:58:13.128730+0000","last_active":"2024-01-16T14:58:13.133559+0000","last_peered":"2024-01-16T14:58:13.133559+0000","last_clean":"2024-01-16T14:58:13.133559+0000","last_became_active":"2024-01-16T14:58:13.128458+0000","last_became_peered":"2024-01-16T14:58:13.128458+0000","last_unstale":"2024-01-16T14:58:13.133559+0000","last_undegraded":"2024-01-16T14:58:13.133559+0000","last_fullsized":"2024-01-16T14:58:13.133559+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.110105+0000","last_change":"2024-01-16T14:58:12.109931+0000","last_active":"2024-01-16T14:58:13.110105+0000","last_peered":"2024-01-16T14:58:13.110105+0000","last_clean":"2024-01-16T14:58:13.110105+0000","last_became_active":"2024-01-16T14:58:12.109693+0000","last_became_peered":"2024-01-16T14:58:12.109693+0000","last_unstale":"2024-01-16T14:58:13.110105+0000","last_undegraded":"2024-01-16T14:58:13.110105+0000","last_fullsized":"2024-01-16T14:58:13.110105+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:24.165221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135451+0000","last_change":"2024-01-16T14:58:13.122057+0000","last_active":"2024-01-16T14:58:13.135451+0000","last_peered":"2024-01-16T14:58:13.135451+0000","last_clean":"2024-01-16T14:58:13.135451+0000","last_became_active":"2024-01-16T14:58:13.121786+0000","last_became_peered":"2024-01-16T14:58:13.121786+0000","last_unstale":"2024-01-16T14:58:13.135451+0000","last_undegraded":"2024-01-16T14:58:13.135451+0000","last_fullsized":"2024-01-16T14:58:13.135451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.073281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089403+0000","last_change":"2024-01-16T14:58:12.120831+0000","last_active":"2024-01-16T14:58:13.089403+0000","last_peered":"2024-01-16T14:58:13.089403+0000","last_clean":"2024-01-16T14:58:13.089403+0000","last_became_active":"2024-01-16T14:58:12.120541+0000","last_became_peered":"2024-01-16T14:58:12.120541+0000","last_unstale":"2024-01-16T14:58:13.089403+0000","last_undegraded":"2024-01-16T14:58:13.089403+0000","last_fullsized":"2024-01-16T14:58:13.089403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:52.748385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.127817+0000","last_change":"2024-01-16T14:58:13.118989+0000","last_active":"2024-01-16T14:58:13.127817+0000","last_peered":"2024-01-16T14:58:13.127817+0000","last_clean":"2024-01-16T14:58:13.127817+0000","last_became_active":"2024-01-16T14:58:13.118691+0000","last_became_peered":"2024-01-16T14:58:13.118691+0000","last_unstale":"2024-01-16T14:58:13.127817+0000","last_undegraded":"2024-01-16T14:58:13.127817+0000","last_fullsized":"2024-01-16T14:58:13.127817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:28.922224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.141748+0000","last_change":"2024-01-16T14:58:12.128472+0000","last_active":"2024-01-16T14:58:13.141748+0000","last_peered":"2024-01-16T14:58:13.141748+0000","last_clean":"2024-01-16T14:58:13.141748+0000","last_became_active":"2024-01-16T14:58:12.128197+0000","last_became_peered":"2024-01-16T14:58:12.128197+0000","last_unstale":"2024-01-16T14:58:13.141748+0000","last_undegraded":"2024-01-16T14:58:13.141748+0000","last_fullsized":"2024-01-16T14:58:13.141748+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:33.025457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123464+0000","last_change":"2024-01-16T14:58:13.112760+0000","last_active":"2024-01-16T14:58:13.123464+0000","last_peered":"2024-01-16T14:58:13.123464+0000","last_clean":"2024-01-16T14:58:13.123464+0000","last_became_active":"2024-01-16T14:58:13.110324+0000","last_became_peered":"2024-01-16T14:58:13.110324+0000","last_unstale":"2024-01-16T14:58:13.123464+0000","last_undegraded":"2024-01-16T14:58:13.123464+0000","last_fullsized":"2024-01-16T14:58:13.123464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.145569+0000","last_change":"2024-01-16T14:58:12.111745+0000","last_active":"2024-01-16T14:58:13.145569+0000","last_peered":"2024-01-16T14:58:13.145569+0000","last_clean":"2024-01-16T14:58:13.145569+0000","last_became_active":"2024-01-16T14:58:12.110894+0000","last_became_peered":"2024-01-16T14:58:12.110894+0000","last_unstale":"2024-01-16T14:58:13.145569+0000","last_undegraded":"2024-01-16T14:58:13.145569+0000","last_fullsized":"2024-01-16T14:58:13.145569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:49.347613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.116888+0000","last_change":"2024-01-16T14:58:13.111340+0000","last_active":"2024-01-16T14:58:13.116888+0000","last_peered":"2024-01-16T14:58:13.116888+0000","last_clean":"2024-01-16T14:58:13.116888+0000","last_became_active":"2024-01-16T14:58:13.111098+0000","last_became_peered":"2024-01-16T14:58:13.111098+0000","last_unstale":"2024-01-16T14:58:13.116888+0000","last_undegraded":"2024-01-16T14:58:13.116888+0000","last_fullsized":"2024-01-16T14:58:13.116888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138619+0000","last_change":"2024-01-16T14:58:12.122476+0000","last_active":"2024-01-16T14:58:13.138619+0000","last_peered":"2024-01-16T14:58:13.138619+0000","last_clean":"2024-01-16T14:58:13.138619+0000","last_became_active":"2024-01-16T14:58:12.122185+0000","last_became_peered":"2024-01-16T14:58:12.122185+0000","last_unstale":"2024-01-16T14:58:13.138619+0000","last_undegraded":"2024-01-16T14:58:13.138619+0000","last_fullsized":"2024-01-16T14:58:13.138619+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:02.009374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.134881+0000","last_change":"2024-01-16T14:58:13.124142+0000","last_active":"2024-01-16T14:58:13.134881+0000","last_peered":"2024-01-16T14:58:13.134881+0000","last_clean":"2024-01-16T14:58:13.134881+0000","last_became_active":"2024-01-16T14:58:13.123531+0000","last_became_peered":"2024-01-16T14:58:13.123531+0000","last_unstale":"2024-01-16T14:58:13.134881+0000","last_undegraded":"2024-01-16T14:58:13.134881+0000","last_fullsized":"2024-01-16T14:58:13.134881+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:03.020765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104740+0000","last_change":"2024-01-16T14:58:12.117665+0000","last_active":"2024-01-16T14:58:13.104740+0000","last_peered":"2024-01-16T14:58:13.104740+0000","last_clean":"2024-01-16T14:58:13.104740+0000","last_became_active":"2024-01-16T14:58:12.117392+0000","last_became_peered":"2024-01-16T14:58:12.117392+0000","last_unstale":"2024-01-16T14:58:13.104740+0000","last_undegraded":"2024-01-16T14:58:13.104740+0000","last_fullsized":"2024-01-16T14:58:13.104740+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.125647+0000","last_change":"2024-01-16T14:58:13.117989+0000","last_active":"2024-01-16T14:58:13.125647+0000","last_peered":"2024-01-16T14:58:13.125647+0000","last_clean":"2024-01-16T14:58:13.125647+0000","last_became_active":"2024-01-16T14:58:13.117355+0000","last_became_peered":"2024-01-16T14:58:13.117355+0000","last_unstale":"2024-01-16T14:58:13.125647+0000","last_undegraded":"2024-01-16T14:58:13.125647+0000","last_fullsized":"2024-01-16T14:58:13.125647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:34.036848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101565+0000","last_change":"2024-01-16T14:58:12.112239+0000","last_active":"2024-01-16T14:58:13.101565+0000","last_peered":"2024-01-16T14:58:13.101565+0000","last_clean":"2024-01-16T14:58:13.101565+0000","last_became_active":"2024-01-16T14:58:12.111952+0000","last_became_peered":"2024-01-16T14:58:12.111952+0000","last_unstale":"2024-01-16T14:58:13.101565+0000","last_undegraded":"2024-01-16T14:58:13.101565+0000","last_fullsized":"2024-01-16T14:58:13.101565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:32.166222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.122890+0000","last_change":"2024-01-16T14:58:13.118164+0000","last_active":"2024-01-16T14:58:13.122890+0000","last_peered":"2024-01-16T14:58:13.122890+0000","last_clean":"2024-01-16T14:58:13.122890+0000","last_became_active":"2024-01-16T14:58:13.117808+0000","last_became_peered":"2024-01-16T14:58:13.117808+0000","last_unstale":"2024-01-16T14:58:13.122890+0000","last_undegraded":"2024-01-16T14:58:13.122890+0000","last_fullsized":"2024-01-16T14:58:13.122890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.209447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142239+0000","last_change":"2024-01-16T14:58:12.109940+0000","last_active":"2024-01-16T14:58:13.142239+0000","last_peered":"2024-01-16T14:58:13.142239+0000","last_clean":"2024-01-16T14:58:13.142239+0000","last_became_active":"2024-01-16T14:58:12.109213+0000","last_became_peered":"2024-01-16T14:58:12.109213+0000","last_unstale":"2024-01-16T14:58:13.142239+0000","last_undegraded":"2024-01-16T14:58:13.142239+0000","last_fullsized":"2024-01-16T14:58:13.142239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.736641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.130268+0000","last_change":"2024-01-16T14:58:13.122826+0000","last_active":"2024-01-16T14:58:13.130268+0000","last_peered":"2024-01-16T14:58:13.130268+0000","last_clean":"2024-01-16T14:58:13.130268+0000","last_became_active":"2024-01-16T14:58:13.121868+0000","last_became_peered":"2024-01-16T14:58:13.121868+0000","last_unstale":"2024-01-16T14:58:13.130268+0000","last_undegraded":"2024-01-16T14:58:13.130268+0000","last_fullsized":"2024-01-16T14:58:13.130268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.142917+0000","last_change":"2024-01-16T14:58:12.117807+0000","last_active":"2024-01-16T14:58:13.142917+0000","last_peered":"2024-01-16T14:58:13.142917+0000","last_clean":"2024-01-16T14:58:13.142917+0000","last_became_active":"2024-01-16T14:58:12.117486+0000","last_became_peered":"2024-01-16T14:58:12.117486+0000","last_unstale":"2024-01-16T14:58:13.142917+0000","last_undegraded":"2024-01-16T14:58:13.142917+0000","last_fullsized":"2024-01-16T14:58:13.142917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:02.009374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123558+0000","last_change":"2024-01-16T14:58:13.120443+0000","last_active":"2024-01-16T14:58:13.123558+0000","last_peered":"2024-01-16T14:58:13.123558+0000","last_clean":"2024-01-16T14:58:13.123558+0000","last_became_active":"2024-01-16T14:58:13.118620+0000","last_became_peered":"2024-01-16T14:58:13.118620+0000","last_unstale":"2024-01-16T14:58:13.123558+0000","last_undegraded":"2024-01-16T14:58:13.123558+0000","last_fullsized":"2024-01-16T14:58:13.123558+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.135477+0000","last_change":"2024-01-16T14:58:12.123306+0000","last_active":"2024-01-16T14:58:13.135477+0000","last_peered":"2024-01-16T14:58:13.135477+0000","last_clean":"2024-01-16T14:58:13.135477+0000","last_became_active":"2024-01-16T14:58:12.122837+0000","last_became_peered":"2024-01-16T14:58:12.122837+0000","last_unstale":"2024-01-16T14:58:13.135477+0000","last_undegraded":"2024-01-16T14:58:13.135477+0000","last_fullsized":"2024-01-16T14:58:13.135477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:27.372519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119904+0000","last_change":"2024-01-16T14:58:13.114235+0000","last_active":"2024-01-16T14:58:13.119904+0000","last_peered":"2024-01-16T14:58:13.119904+0000","last_clean":"2024-01-16T14:58:13.119904+0000","last_became_active":"2024-01-16T14:58:13.113201+0000","last_became_peered":"2024-01-16T14:58:13.113201+0000","last_unstale":"2024-01-16T14:58:13.119904+0000","last_undegraded":"2024-01-16T14:58:13.119904+0000","last_fullsized":"2024-01-16T14:58:13.119904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:24.116056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.140236+0000","last_change":"2024-01-16T14:58:12.124755+0000","last_active":"2024-01-16T14:58:13.140236+0000","last_peered":"2024-01-16T14:58:13.140236+0000","last_clean":"2024-01-16T14:58:13.140236+0000","last_became_active":"2024-01-16T14:58:12.124272+0000","last_became_peered":"2024-01-16T14:58:12.124272+0000","last_unstale":"2024-01-16T14:58:13.140236+0000","last_undegraded":"2024-01-16T14:58:13.140236+0000","last_fullsized":"2024-01-16T14:58:13.140236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.453323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.134537+0000","last_change":"2024-01-16T14:58:13.127657+0000","last_active":"2024-01-16T14:58:13.134537+0000","last_peered":"2024-01-16T14:58:13.134537+0000","last_clean":"2024-01-16T14:58:13.134537+0000","last_became_active":"2024-01-16T14:58:13.127376+0000","last_became_peered":"2024-01-16T14:58:13.127376+0000","last_unstale":"2024-01-16T14:58:13.134537+0000","last_undegraded":"2024-01-16T14:58:13.134537+0000","last_fullsized":"2024-01-16T14:58:13.134537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:19.082451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.102985+0000","last_change":"2024-01-16T14:58:12.124457+0000","last_active":"2024-01-16T14:58:13.102985+0000","last_peered":"2024-01-16T14:58:13.102985+0000","last_clean":"2024-01-16T14:58:13.102985+0000","last_became_active":"2024-01-16T14:58:12.124185+0000","last_became_peered":"2024-01-16T14:58:12.124185+0000","last_unstale":"2024-01-16T14:58:13.102985+0000","last_undegraded":"2024-01-16T14:58:13.102985+0000","last_fullsized":"2024-01-16T14:58:13.102985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.736641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138205+0000","last_change":"2024-01-16T14:58:13.129178+0000","last_active":"2024-01-16T14:58:13.138205+0000","last_peered":"2024-01-16T14:58:13.138205+0000","last_clean":"2024-01-16T14:58:13.138205+0000","last_became_active":"2024-01-16T14:58:13.125031+0000","last_became_peered":"2024-01-16T14:58:13.125031+0000","last_unstale":"2024-01-16T14:58:13.138205+0000","last_undegraded":"2024-01-16T14:58:13.138205+0000","last_fullsized":"2024-01-16T14:58:13.138205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:10.833062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.138601+0000","last_change":"2024-01-16T14:58:12.116108+0000","last_active":"2024-01-16T14:58:13.138601+0000","last_peered":"2024-01-16T14:58:13.138601+0000","last_clean":"2024-01-16T14:58:13.138601+0000","last_became_active":"2024-01-16T14:58:12.115876+0000","last_became_peered":"2024-01-16T14:58:12.115876+0000","last_unstale":"2024-01-16T14:58:13.138601+0000","last_undegraded":"2024-01-16T14:58:13.138601+0000","last_fullsized":"2024-01-16T14:58:13.138601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:53.167551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137869+0000","last_change":"2024-01-16T14:58:13.131810+0000","last_active":"2024-01-16T14:58:13.137869+0000","last_peered":"2024-01-16T14:58:13.137869+0000","last_clean":"2024-01-16T14:58:13.137869+0000","last_became_active":"2024-01-16T14:58:13.131516+0000","last_became_peered":"2024-01-16T14:58:13.131516+0000","last_unstale":"2024-01-16T14:58:13.137869+0000","last_undegraded":"2024-01-16T14:58:13.137869+0000","last_fullsized":"2024-01-16T14:58:13.137869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:00.601483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.116021+0000","last_change":"2024-01-16T14:58:12.124856+0000","last_active":"2024-01-16T14:58:13.116021+0000","last_peered":"2024-01-16T14:58:13.116021+0000","last_clean":"2024-01-16T14:58:13.116021+0000","last_became_active":"2024-01-16T14:58:12.124533+0000","last_became_peered":"2024-01-16T14:58:12.124533+0000","last_unstale":"2024-01-16T14:58:13.116021+0000","last_undegraded":"2024-01-16T14:58:13.116021+0000","last_fullsized":"2024-01-16T14:58:13.116021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:11.527954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.129071+0000","last_change":"2024-01-16T14:58:13.124148+0000","last_active":"2024-01-16T14:58:13.129071+0000","last_peered":"2024-01-16T14:58:13.129071+0000","last_clean":"2024-01-16T14:58:13.129071+0000","last_became_active":"2024-01-16T14:58:13.123792+0000","last_became_peered":"2024-01-16T14:58:13.123792+0000","last_unstale":"2024-01-16T14:58:13.129071+0000","last_undegraded":"2024-01-16T14:58:13.129071+0000","last_fullsized":"2024-01-16T14:58:13.129071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:38.456905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.094747+0000","last_change":"2024-01-16T14:58:12.119102+0000","last_active":"2024-01-16T14:58:13.094747+0000","last_peered":"2024-01-16T14:58:13.094747+0000","last_clean":"2024-01-16T14:58:13.094747+0000","last_became_active":"2024-01-16T14:58:12.118801+0000","last_became_peered":"2024-01-16T14:58:12.118801+0000","last_unstale":"2024-01-16T14:58:13.094747+0000","last_undegraded":"2024-01-16T14:58:13.094747+0000","last_fullsized":"2024-01-16T14:58:13.094747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:16.609355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.126657+0000","last_change":"2024-01-16T14:58:13.122617+0000","last_active":"2024-01-16T14:58:13.126657+0000","last_peered":"2024-01-16T14:58:13.126657+0000","last_clean":"2024-01-16T14:58:13.126657+0000","last_became_active":"2024-01-16T14:58:13.122100+0000","last_became_peered":"2024-01-16T14:58:13.122100+0000","last_unstale":"2024-01-16T14:58:13.126657+0000","last_undegraded":"2024-01-16T14:58:13.126657+0000","last_fullsized":"2024-01-16T14:58:13.126657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:36.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104636+0000","last_change":"2024-01-16T14:58:12.124162+0000","last_active":"2024-01-16T14:58:13.104636+0000","last_peered":"2024-01-16T14:58:13.104636+0000","last_clean":"2024-01-16T14:58:13.104636+0000","last_became_active":"2024-01-16T14:58:12.123869+0000","last_became_peered":"2024-01-16T14:58:12.123869+0000","last_unstale":"2024-01-16T14:58:13.104636+0000","last_undegraded":"2024-01-16T14:58:13.104636+0000","last_fullsized":"2024-01-16T14:58:13.104636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:32.166222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.137784+0000","last_change":"2024-01-16T14:58:13.130023+0000","last_active":"2024-01-16T14:58:13.137784+0000","last_peered":"2024-01-16T14:58:13.137784+0000","last_clean":"2024-01-16T14:58:13.137784+0000","last_became_active":"2024-01-16T14:58:13.129149+0000","last_became_peered":"2024-01-16T14:58:13.129149+0000","last_unstale":"2024-01-16T14:58:13.137784+0000","last_undegraded":"2024-01-16T14:58:13.137784+0000","last_fullsized":"2024-01-16T14:58:13.137784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.139861+0000","last_change":"2024-01-16T14:58:12.122971+0000","last_active":"2024-01-16T14:58:13.139861+0000","last_peered":"2024-01-16T14:58:13.139861+0000","last_clean":"2024-01-16T14:58:13.139861+0000","last_became_active":"2024-01-16T14:58:12.122636+0000","last_became_peered":"2024-01-16T14:58:12.122636+0000","last_unstale":"2024-01-16T14:58:13.139861+0000","last_undegraded":"2024-01-16T14:58:13.139861+0000","last_fullsized":"2024-01-16T14:58:13.139861+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.453323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.123537+0000","last_change":"2024-01-16T14:58:13.120078+0000","last_active":"2024-01-16T14:58:13.123537+0000","last_peered":"2024-01-16T14:58:13.123537+0000","last_clean":"2024-01-16T14:58:13.123537+0000","last_became_active":"2024-01-16T14:58:13.117991+0000","last_became_peered":"2024-01-16T14:58:13.117991+0000","last_unstale":"2024-01-16T14:58:13.123537+0000","last_undegraded":"2024-01-16T14:58:13.123537+0000","last_fullsized":"2024-01-16T14:58:13.123537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:25.176612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089712+0000","last_change":"2024-01-16T14:58:12.121746+0000","last_active":"2024-01-16T14:58:13.089712+0000","last_peered":"2024-01-16T14:58:13.089712+0000","last_clean":"2024-01-16T14:58:13.089712+0000","last_became_active":"2024-01-16T14:58:12.121318+0000","last_became_peered":"2024-01-16T14:58:12.121318+0000","last_unstale":"2024-01-16T14:58:13.089712+0000","last_undegraded":"2024-01-16T14:58:13.089712+0000","last_fullsized":"2024-01-16T14:58:13.089712+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.117325+0000","last_change":"2024-01-16T14:58:13.112000+0000","last_active":"2024-01-16T14:58:13.117325+0000","last_peered":"2024-01-16T14:58:13.117325+0000","last_clean":"2024-01-16T14:58:13.117325+0000","last_became_active":"2024-01-16T14:58:13.111730+0000","last_became_peered":"2024-01-16T14:58:13.111730+0000","last_unstale":"2024-01-16T14:58:13.117325+0000","last_undegraded":"2024-01-16T14:58:13.117325+0000","last_fullsized":"2024-01-16T14:58:13.117325+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:18.186062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.105268+0000","last_change":"2024-01-16T14:58:12.116991+0000","last_active":"2024-01-16T14:58:13.105268+0000","last_peered":"2024-01-16T14:58:13.105268+0000","last_clean":"2024-01-16T14:58:13.105268+0000","last_became_active":"2024-01-16T14:58:12.116733+0000","last_became_peered":"2024-01-16T14:58:12.116733+0000","last_unstale":"2024-01-16T14:58:13.105268+0000","last_undegraded":"2024-01-16T14:58:13.105268+0000","last_fullsized":"2024-01-16T14:58:13.105268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:30:48.911413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.129506+0000","last_change":"2024-01-16T14:58:13.121418+0000","last_active":"2024-01-16T14:58:13.129506+0000","last_peered":"2024-01-16T14:58:13.129506+0000","last_clean":"2024-01-16T14:58:13.129506+0000","last_became_active":"2024-01-16T14:58:13.121024+0000","last_became_peered":"2024-01-16T14:58:13.121024+0000","last_unstale":"2024-01-16T14:58:13.129506+0000","last_undegraded":"2024-01-16T14:58:13.129506+0000","last_fullsized":"2024-01-16T14:58:13.129506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.089046+0000","last_change":"2024-01-16T14:58:12.122147+0000","last_active":"2024-01-16T14:58:13.089046+0000","last_peered":"2024-01-16T14:58:13.089046+0000","last_clean":"2024-01-16T14:58:13.089046+0000","last_became_active":"2024-01-16T14:58:12.121837+0000","last_became_peered":"2024-01-16T14:58:12.121837+0000","last_unstale":"2024-01-16T14:58:13.089046+0000","last_undegraded":"2024-01-16T14:58:13.089046+0000","last_fullsized":"2024-01-16T14:58:13.089046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:07.832210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.119407+0000","last_change":"2024-01-16T14:58:13.116387+0000","last_active":"2024-01-16T14:58:13.119407+0000","last_peered":"2024-01-16T14:58:13.119407+0000","last_clean":"2024-01-16T14:58:13.119407+0000","last_became_active":"2024-01-16T14:58:13.116124+0000","last_became_peered":"2024-01-16T14:58:13.116124+0000","last_unstale":"2024-01-16T14:58:13.119407+0000","last_undegraded":"2024-01-16T14:58:13.119407+0000","last_fullsized":"2024-01-16T14:58:13.119407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:11.035135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.104896+0000","last_change":"2024-01-16T14:58:12.122192+0000","last_active":"2024-01-16T14:58:13.104896+0000","last_peered":"2024-01-16T14:58:13.104896+0000","last_clean":"2024-01-16T14:58:13.104896+0000","last_became_active":"2024-01-16T14:58:12.120390+0000","last_became_peered":"2024-01-16T14:58:12.120390+0000","last_unstale":"2024-01-16T14:58:13.104896+0000","last_undegraded":"2024-01-16T14:58:13.104896+0000","last_fullsized":"2024-01-16T14:58:13.104896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:12.552354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.124636+0000","last_change":"2024-01-16T14:58:13.117548+0000","last_active":"2024-01-16T14:58:13.124636+0000","last_peered":"2024-01-16T14:58:13.124636+0000","last_clean":"2024-01-16T14:58:13.124636+0000","last_became_active":"2024-01-16T14:58:13.117288+0000","last_became_peered":"2024-01-16T14:58:13.117288+0000","last_unstale":"2024-01-16T14:58:13.124636+0000","last_undegraded":"2024-01-16T14:58:13.124636+0000","last_fullsized":"2024-01-16T14:58:13.124636+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:49.464714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.095773+0000","last_change":"2024-01-16T14:58:12.122568+0000","last_active":"2024-01-16T14:58:13.095773+0000","last_peered":"2024-01-16T14:58:13.095773+0000","last_clean":"2024-01-16T14:58:13.095773+0000","last_became_active":"2024-01-16T14:58:12.122229+0000","last_became_peered":"2024-01-16T14:58:12.122229+0000","last_unstale":"2024-01-16T14:58:13.095773+0000","last_undegraded":"2024-01-16T14:58:13.095773+0000","last_fullsized":"2024-01-16T14:58:13.095773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:32.327445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.131933+0000","last_change":"2024-01-16T14:58:13.123092+0000","last_active":"2024-01-16T14:58:13.131933+0000","last_peered":"2024-01-16T14:58:13.131933+0000","last_clean":"2024-01-16T14:58:13.131933+0000","last_became_active":"2024-01-16T14:58:13.122683+0000","last_became_peered":"2024-01-16T14:58:13.122683+0000","last_unstale":"2024-01-16T14:58:13.131933+0000","last_undegraded":"2024-01-16T14:58:13.131933+0000","last_fullsized":"2024-01-16T14:58:13.131933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:20.006260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.101560+0000","last_change":"2024-01-16T14:58:12.110101+0000","last_active":"2024-01-16T14:58:13.101560+0000","last_peered":"2024-01-16T14:58:13.101560+0000","last_clean":"2024-01-16T14:58:13.101560+0000","last_became_active":"2024-01-16T14:58:12.109615+0000","last_became_peered":"2024-01-16T14:58:12.109615+0000","last_unstale":"2024-01-16T14:58:13.101560+0000","last_undegraded":"2024-01-16T14:58:13.101560+0000","last_fullsized":"2024-01-16T14:58:13.101560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:02.139904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.133177+0000","last_change":"2024-01-16T14:58:13.127178+0000","last_active":"2024-01-16T14:58:13.133177+0000","last_peered":"2024-01-16T14:58:13.133177+0000","last_clean":"2024-01-16T14:58:13.133177+0000","last_became_active":"2024-01-16T14:58:13.126936+0000","last_became_peered":"2024-01-16T14:58:13.126936+0000","last_unstale":"2024-01-16T14:58:13.133177+0000","last_undegraded":"2024-01-16T14:58:13.133177+0000","last_fullsized":"2024-01-16T14:58:13.133177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:12.074391+0000","last_clean_scrub_stamp":"2024-01-16T14:58:12.074391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:00.419343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:13.117332+0000","last_change":"2024-01-16T14:58:12.122955+0000","last_active":"2024-01-16T14:58:13.117332+0000","last_peered":"2024-01-16T14:58:13.117332+0000","last_clean":"2024-01-16T14:58:13.117332+0000","last_became_active":"2024-01-16T14:58:12.122653+0000","last_became_peered":"2024-01-16T14:58:12.122653+0000","last_unstale":"2024-01-16T14:58:13.117332+0000","last_undegraded":"2024-01-16T14:58:13.117332+0000","last_fullsized":"2024-01-16T14:58:13.117332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:11.062999+0000","last_clean_scrub_stamp":"2024-01-16T14:58:11.062999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:07.832210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T14:58:12.107253+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:58:12.107253+0000","last_peered":"2024-01-16T14:58:12.107253+0000","last_clean":"2024-01-16T14:58:12.107253+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:58:12.107253+0000","last_undegraded":"2024-01-16T14:58:12.107253+0000","last_fullsized":"2024-01-16T14:58:12.107253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.6080000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.843}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.8140000000000001,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.204,"15min":1.204},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.206}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.323},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.3700000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.929},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.7490000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.9850000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.7829999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.68899999999999995}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.198}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.2989999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.431},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.4750000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.599}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.589,"15min":1.589},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.526}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.96899999999999997}]},{"osd":1,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.353},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.0820000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1799999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.016}]},{"osd":7,"last update":"Tue Jan 16 14:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.615},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.2569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.46},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.417}]},{"osd":1,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.3220000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.75,"5min":2.75,"15min":2.75},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.603}]},{"osd":5,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.9609999999999999,"5min":2.9609999999999999,"15min":2.9609999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.93899999999999995}]},{"osd":6,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":0.999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.367}]},{"osd":7,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":1.1839999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.89200000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":3.226,"5min":3.226,"15min":3.226},"last":1.478},{"interface":"front","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.492,"15min":1.492},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":3.0379999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.258}]},{"osd":6,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9550000000000001,"5min":2.9550000000000001,"15min":2.9550000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.427}]},{"osd":7,"last update":"Tue Jan 16 14:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.4399999999999999,"5min":2.4399999999999999,"15min":2.4399999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":3.0979999999999999,"5min":3.0979999999999999,"15min":3.0979999999999999},"last":1.5449999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":145839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":1.302},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":5.9889999999999999,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.80300000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.1130000000000004,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":6.0259999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.91200000000000003}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":6.141,"5min":6.141,"15min":6.141},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.1799999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":5.2009999999999996,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.149},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.786}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.9199999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.349}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":5.024,"5min":5.024,"15min":5.024},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.8849999999999998,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.575}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":4.9930000000000003,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.9199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.97099999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.3160000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.113},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":1.2869999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":1.385},{"interface":"front","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.032},{"interface":"front","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.2569999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":735118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.026},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.202}]},{"osd":3,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.111}]},{"osd":4,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.997},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.23},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.1799999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.141},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:18.002 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:58:18.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T14:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.271+0000 7f2dba000640 1 Processor -- start 2024-01-16T14:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.271+0000 7f2dba000640 1 -- start start 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 0x7f2db4141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2db415f100 0x7f2db40cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 0x7f2db415b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db413e440 con 0x7f2db415f100 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db40ce1d0 con 0x7f2db40cdcc0 2024-01-16T14:58:18.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2dba000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40ce4b0 con 0x7f2db415ed20 2024-01-16T14:58:18.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2db2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2db415f100 0x7f2db40cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2db2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2db415f100 0x7f2db40cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57128/0 (socket says 172.21.15.125:57128) 2024-01-16T14:58:18.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.273+0000 7f2db2a00640 1 -- 172.21.15.125:0/4056511591 learned_addr learned my addr 172.21.15.125:0/4056511591 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db2a00640 1 -- 172.21.15.125:0/4056511591 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 msgr2=0x7f2db4141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db3400640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 0x7f2db4141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db3e00640 1 --2- 172.21.15.125:0/4056511591 >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 0x7f2db415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db2a00640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 0x7f2db4141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db2a00640 1 -- 172.21.15.125:0/4056511591 >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 msgr2=0x7f2db415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db2a00640 1 --2- 172.21.15.125:0/4056511591 >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 0x7f2db415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db2a00640 1 -- 172.21.15.125:0/4056511591 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db40ce790 con 0x7f2db415f100 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db3e00640 1 --2- 172.21.15.125:0/4056511591 >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 0x7f2db415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.274+0000 7f2db3400640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 0x7f2db4141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.275+0000 7f2db2a00640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40cd780 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2db4073dd0 tx=0x7f2d9c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637836fe44e58db2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.275+0000 7f2db2000640 1 -- 172.21.15.125:0/4056511591 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0317d0 con 0x7f2db415f100 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.275+0000 7f2db2000640 1 -- 172.21.15.125:0/4056511591 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c031930 con 0x7f2db415f100 2024-01-16T14:58:18.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.275+0000 7f2db2000640 1 -- 172.21.15.125:0/4056511591 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c031cc0 con 0x7f2db415f100 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 -- 172.21.15.125:0/4056511591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 msgr2=0x7f2db40cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40cd780 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2db4073dd0 tx=0x7f2d9c02b870 comp rx=0 tx=0).stop 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 -- 172.21.15.125:0/4056511591 shutdown_connections 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 --2- 172.21.15.125:0/4056511591 >> v2:172.21.15.137:3300/0 conn(0x7f2db40cdcc0 0x7f2db415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40cd780 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 --2- 172.21.15.125:0/4056511591 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db415ed20 0x7f2db4141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 -- 172.21.15.125:0/4056511591 >> 172.21.15.125:0/4056511591 conn(0x7f2db40b0be0 msgr2=0x7f2db40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 -- 172.21.15.125:0/4056511591 shutdown_connections 2024-01-16T14:58:18.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.276+0000 7f2dba000640 1 -- 172.21.15.125:0/4056511591 wait complete. 2024-01-16T14:58:18.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.278+0000 7f2dba000640 1 Processor -- start 2024-01-16T14:58:18.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.278+0000 7f2dba000640 1 -- start start 2024-01-16T14:58:18.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db40cdcc0 0x7f2db40aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 0x7f2db40a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40a9ef0 con 0x7f2db415f100 2024-01-16T14:58:18.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40ab610 con 0x7f2db415ed20 2024-01-16T14:58:18.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2dba000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40ab8f0 con 0x7f2db40cdcc0 2024-01-16T14:58:18.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2db3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2db3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57138/0 (socket says 172.21.15.125:57138) 2024-01-16T14:58:18.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.279+0000 7f2db3e00640 1 -- 172.21.15.125:0/3041276208 learned_addr learned my addr 172.21.15.125:0/3041276208 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3400640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db40cdcc0 0x7f2db40aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db2a00640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 0x7f2db40a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 -- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db40cdcc0 msgr2=0x7f2db40aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db40cdcc0 0x7f2db40aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 -- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 msgr2=0x7f2db40a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 0x7f2db40a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 -- 172.21.15.125:0/3041276208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c0094d0 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db2a00640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 0x7f2db40a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2db3e00640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00cef0 tx=0x7f2dac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac00d840 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac00d9a0 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac00dd50 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.280+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db415f6c0 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.281+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db40abeb0 con 0x7f2db415f100 2024-01-16T14:58:18.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.282+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db4067670 con 0x7f2db415f100 2024-01-16T14:58:18.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.283+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2dac01e3b0 con 0x7f2db415f100 2024-01-16T14:58:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.284+0000 7f2dab400640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 0x7f2d8c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.284+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dac08a620 con 0x7f2db415f100 2024-01-16T14:58:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.284+0000 7f2db3400640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 0x7f2d8c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.285+0000 7f2db3400640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 0x7f2d8c06d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2da4008850 tx=0x7f2da4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.288+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dac055490 con 0x7f2db415f100 2024-01-16T14:58:18.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.541+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2db406dd00 con 0x7f2db415f100 2024-01-16T14:58:18.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.545+0000 7f2dab400640 1 -- 172.21.15.125:0/3041276208 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2dac058ab0 con 0x7f2db415f100 2024-01-16T14:58:18.527 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T14:58:18.528 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T14:58:18.528 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T14:58:18.528 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T14:58:18.528 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T14:58:18.528 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T14:58:18.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 msgr2=0x7f2d8c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 0x7f2d8c06d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2da4008850 tx=0x7f2da4008040 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 msgr2=0x7f2db40ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00cef0 tx=0x7f2dac00a710 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 shutdown_connections 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d8c06b2c0 0x7f2d8c06d780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2db415f100 0x7f2db40ac6b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db415ed20 0x7f2db40a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 --2- 172.21.15.125:0/3041276208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2db40cdcc0 0x7f2db40aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 >> 172.21.15.125:0/3041276208 conn(0x7f2db40b0be0 msgr2=0x7f2db4151fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 shutdown_connections 2024-01-16T14:58:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.548+0000 7f2dba000640 1 -- 172.21.15.125:0/3041276208 wait complete. 2024-01-16T14:58:18.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:18.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.761+0000 7fdfab200640 1 Processor -- start 2024-01-16T14:58:18.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.761+0000 7fdfab200640 1 -- start start 2024-01-16T14:58:18.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfab200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdfa4143c40 0x7fdfa4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfab200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdfa40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfab200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa4070e50 con 0x7fdfa40cca40 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfab200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa40723c0 con 0x7fdfa4143c40 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfab200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40cce70 con 0x7fdfa40717d0 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfaa800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdfa40cca40 0x7fdfa415eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45772/0 (socket says 172.21.15.125:45772) 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfaa800640 1 -- 172.21.15.125:0/2945879883 learned_addr learned my addr 172.21.15.125:0/2945879883 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/2945879883 >> v2:172.21.15.137:3300/0 conn(0x7fdfa4143c40 0x7fdfa4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfa9400640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdfa40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.763+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598488832 0 0) 0x7fdfa4070e50 con 0x7fdfa40cca40 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf84003520 con 0x7fdfa40cca40 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940019748 0 0) 0x7fdf84003520 con 0x7fdfa40cca40 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 msgr2=0x7fdfa40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdfa40cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 >> v2:172.21.15.137:3300/0 conn(0x7fdfa4143c40 msgr2=0x7fdfa4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 --2- 172.21.15.125:0/2945879883 >> v2:172.21.15.137:3300/0 conn(0x7fdfa4143c40 0x7fdfa4071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa406e170 con 0x7fdfa40cca40 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdf9c0019f0 con 0x7fdfa40cca40 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 >> v1:172.21.15.125:6789/0 conn(0x7fdfa40cca40 legacy=0x7fdfa415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdf84003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84003f40 0x7fdf840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf84004d00 0x7fdf84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40723c0 con 0x7fdf84004d00 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40cce70 con 0x7fdf84003f40 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.764+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8400ba30 con 0x7fdfa40717d0 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf84004d00 0x7fdf84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdf84003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfa9400640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84003f40 0x7fdf840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 -- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 msgr2=0x7fdf84003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa40717d0 0x7fdf84003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 -- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84003f40 msgr2=0x7fdf840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84003f40 0x7fdf840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.765+0000 7fdfaa800640 1 -- 172.21.15.125:0/2945879883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84003520 con 0x7fdf84004d00 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.766+0000 7fdfaa800640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf84004d00 0x7fdf84005180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00a790 tx=0x7fdf840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8974e79b8e51ddfd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.766+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034630 con 0x7fdf84004d00 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.766+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c02f070 con 0x7fdf84004d00 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.766+0000 7fdfa8a00640 1 -- 172.21.15.125:0/2945879883 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034a30 con 0x7fdf84004d00 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 -- 172.21.15.125:0/2945879883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf84004d00 msgr2=0x7fdf84005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 --2- 172.21.15.125:0/2945879883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf84004d00 0x7fdf84005180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00a790 tx=0x7fdf840057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfaa800640 1 -- 172.21.15.125:0/2945879883 reap_dead start 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 -- 172.21.15.125:0/2945879883 shutdown_connections 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 -- 172.21.15.125:0/2945879883 >> 172.21.15.125:0/2945879883 conn(0x7fdfa406a800 msgr2=0x7fdfa406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 -- 172.21.15.125:0/2945879883 shutdown_connections 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.767+0000 7fdfab200640 1 -- 172.21.15.125:0/2945879883 wait complete. 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.769+0000 7fdfab200640 1 Processor -- start 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.769+0000 7fdfab200640 1 -- start start 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 0x7fdfa413b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 0x7fdfa4140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa413b1d0 con 0x7fdfa413a0c0 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa413a5c0 con 0x7fdf84004d00 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfab200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa413a8a0 con 0x7fdfa4142b60 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfa9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfa9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57164/0 (socket says 172.21.15.125:57164) 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.770+0000 7fdfa9400640 1 -- 172.21.15.125:0/1354785472 learned_addr learned my addr 172.21.15.125:0/1354785472 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfaa800640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 0x7fdfa4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 0x7fdfa413b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 -- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 msgr2=0x7fdfa4140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 0x7fdfa4140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 -- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 msgr2=0x7fdfa413b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 0x7fdfa413b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 -- 172.21.15.125:0/1354785472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa413ab80 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfaa800640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 0x7fdfa4140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 0x7fdfa413b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa9400640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c011600 tx=0x7fdf8c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00f920 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00fa80 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00fdf0 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa413ae60 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.771+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa4143010 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.773+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa413b340 con 0x7fdfa413a0c0 2024-01-16T14:58:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.773+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdf8c015460 con 0x7fdfa413a0c0 2024-01-16T14:58:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.774+0000 7fdfa2000640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 0x7fdf7c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.774+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf8c08a080 con 0x7fdfa413a0c0 2024-01-16T14:58:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.775+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 0x7fdf7c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.775+0000 7fdfa9e00640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 0x7fdf7c06d7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009a80 tx=0x7fdf94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:18.779+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf8c054ef0 con 0x7fdfa413a0c0 2024-01-16T14:58:18.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.006+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa4067450 con 0x7fdfa413a0c0 2024-01-16T14:58:18.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.007+0000 7fdfa2000640 1 -- 172.21.15.125:0/1354785472 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4925 (secure 0 0 0) 0x7fdf8c058510 con 0x7fdfa413a0c0 2024-01-16T14:58:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:58:18.989 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1608829453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1608829453},{"type":"v1","addr":"172.21.15.137:6835","nonce":1608829453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4798,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1479548002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1479548002},{"type":"v1","addr":"172.21.15.137:6833","nonce":1479548002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1004786444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1004786444},{"type":"v1","addr":"172.21.15.125:6835","nonce":1004786444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:58:12.549817+0000","modified":"2024-01-16T14:58:15.611534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6837/2265216525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2265216525},{"type":"v1","addr":"172.21.15.125:6837","nonce":2265216525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T14:58:18.989 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 18 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 msgr2=0x7fdf7c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 0x7fdf7c06d7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009a80 tx=0x7fdf94008040 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 msgr2=0x7fdfa4159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c011600 tx=0x7fdf8c011ac0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 shutdown_connections 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.009+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfa4142b60 0x7fdfa4140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf7c06b2f0 0x7fdf7c06d7b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa413a0c0 0x7fdfa4159890 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 --2- 172.21.15.125:0/1354785472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf84004d00 0x7fdfa413b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 >> 172.21.15.125:0/1354785472 conn(0x7fdfa406a800 msgr2=0x7fdfa40b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 shutdown_connections 2024-01-16T14:58:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.010+0000 7fdfab200640 1 -- 172.21.15.125:0/1354785472 wait complete. 2024-01-16T14:58:19.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:58:12.549817+0000', 'modified': '2024-01-16T14:58:15.611534+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5238}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5238': {'gid': 5238, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.125:6837/2265216525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6836', 'nonce': 2265216525}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 2265216525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T14:58:19.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T14:58:19.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T14:58:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.652+0000 7fe60a000640 1 Processor -- start 2024-01-16T14:58:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.652+0000 7fe60a000640 1 -- start start 2024-01-16T14:58:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe60a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe60413df80 0x7fe60415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe60a000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe60413e360 0x7fe6040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe60a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60406c470 con 0x7fe60415f9c0 2024-01-16T14:58:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe60a000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60406c750 con 0x7fe60413e360 2024-01-16T14:58:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe60a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60406ca30 con 0x7fe60413df80 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe5fa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe60413e360 0x7fe6040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe5fa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe60413e360 0x7fe6040cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57168/0 (socket says 172.21.15.125:57168) 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.654+0000 7fe5fa800640 1 -- 172.21.15.125:0/2953716293 learned_addr learned my addr 172.21.15.125:0/2953716293 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.655+0000 7fe5fa800640 1 -- 172.21.15.125:0/2953716293 >> v2:172.21.15.137:3300/0 conn(0x7fe60413df80 msgr2=0x7fe60415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.655+0000 7fe5fa800640 1 --2- 172.21.15.125:0/2953716293 >> v2:172.21.15.137:3300/0 conn(0x7fe60413df80 0x7fe60415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.655+0000 7fe5fa800640 1 -- 172.21.15.125:0/2953716293 >> v1:172.21.15.137:6789/0 conn(0x7fe60415f9c0 legacy=0x7fe6041408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.655+0000 7fe5fa800640 1 -- 172.21.15.125:0/2953716293 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60406cd10 con 0x7fe60413e360 2024-01-16T14:58:19.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.656+0000 7fe5fa800640 1 --2- 172.21.15.125:0/2953716293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe6040cf290 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe604072770 tx=0x7fe5ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7b6a034eedd2ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.656+0000 7fe602a00640 1 -- 172.21.15.125:0/2953716293 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec031870 con 0x7fe60413e360 2024-01-16T14:58:19.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.656+0000 7fe602a00640 1 -- 172.21.15.125:0/2953716293 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec0319d0 con 0x7fe60413e360 2024-01-16T14:58:19.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.656+0000 7fe602a00640 1 -- 172.21.15.125:0/2953716293 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec031d60 con 0x7fe60413e360 2024-01-16T14:58:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 -- 172.21.15.125:0/2953716293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 msgr2=0x7fe6040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 --2- 172.21.15.125:0/2953716293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe6040cf290 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe604072770 tx=0x7fe5ec02b910 comp rx=0 tx=0).stop 2024-01-16T14:58:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 -- 172.21.15.125:0/2953716293 shutdown_connections 2024-01-16T14:58:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 --2- 172.21.15.125:0/2953716293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe6040cf290 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 --2- 172.21.15.125:0/2953716293 >> v2:172.21.15.137:3300/0 conn(0x7fe60413df80 0x7fe60415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 -- 172.21.15.125:0/2953716293 >> 172.21.15.125:0/2953716293 conn(0x7fe60406a800 msgr2=0x7fe60406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 -- 172.21.15.125:0/2953716293 shutdown_connections 2024-01-16T14:58:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.657+0000 7fe60a000640 1 -- 172.21.15.125:0/2953716293 wait complete. 2024-01-16T14:58:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.659+0000 7fe60a000640 1 Processor -- start 2024-01-16T14:58:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.659+0000 7fe60a000640 1 -- start start 2024-01-16T14:58:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 0x7fe6041479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe60415f9c0 0x7fe604145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604147550 con 0x7fe60413e360 2024-01-16T14:58:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604146940 con 0x7fe60413df80 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe60a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe604146c20 con 0x7fe60415f9c0 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57178/0 (socket says 172.21.15.125:57178) 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe5fa800640 1 -- 172.21.15.125:0/2100600217 learned_addr learned my addr 172.21.15.125:0/2100600217 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.660+0000 7fe603400640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 0x7fe6041479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 -- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe60415f9c0 msgr2=0x7fe604145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe603e00640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe60415f9c0 0x7fe604145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe60415f9c0 0x7fe604145dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 -- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 msgr2=0x7fe6041479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 0x7fe6041479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 -- 172.21.15.125:0/2100600217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec0094d0 con 0x7fe60413e360 2024-01-16T14:58:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe603400640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 0x7fe6041479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fa800640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fbe00640 1 -- 172.21.15.125:0/2100600217 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec03adf0 con 0x7fe60413e360 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fbe00640 1 -- 172.21.15.125:0/2100600217 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec0443f0 con 0x7fe60413e360 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe604146f00 con 0x7fe60413e360 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.661+0000 7fe5fbe00640 1 -- 172.21.15.125:0/2100600217 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0447a0 con 0x7fe60413e360 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.662+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe604071ed0 con 0x7fe60413e360 2024-01-16T14:58:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.663+0000 7fe5fbe00640 1 -- 172.21.15.125:0/2100600217 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5ec04d450 con 0x7fe60413e360 2024-01-16T14:58:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.664+0000 7fe5fbe00640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 0x7fe5dc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.664+0000 7fe5fbe00640 1 -- 172.21.15.125:0/2100600217 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5ec02d040 con 0x7fe60413e360 2024-01-16T14:58:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.665+0000 7fe603400640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 0x7fe5dc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.665+0000 7fe603400640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 0x7fe5dc06d780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400a870 tx=0x7fe5f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.665+0000 7fe60cc32740 1 Processor -- start 2024-01-16T14:58:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.665+0000 7fe60cc32740 1 -- start start 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea60d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea651560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea56c880 con 0x555cea60dea0 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea57bae0 con 0x555cea608ba0 2024-01-16T14:58:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.666+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cea5791d0 con 0x555cea60d5e0 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57184/0 (socket says 172.21.15.125:57184) 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 -- 172.21.15.125:0/3090184457 learned_addr learned my addr 172.21.15.125:0/3090184457 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea60d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603e00640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea651560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 -- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 msgr2=0x555cea651560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea651560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 -- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 msgr2=0x555cea60d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea60d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 -- 172.21.15.125:0/3090184457 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cea53c860 con 0x555cea60dea0 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea60d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.667+0000 7fe603400640 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400cae0 tx=0x7fe5f400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26a154b5b928e86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe5f3e00640 1 -- 172.21.15.125:0/3090184457 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4013e70 con 0x555cea60dea0 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe5f3e00640 1 -- 172.21.15.125:0/3090184457 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f401e400 con 0x555cea60dea0 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe5f3e00640 1 -- 172.21.15.125:0/3090184457 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f401e770 con 0x555cea60dea0 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 -- 172.21.15.125:0/3090184457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 msgr2=0x555cea60e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400cae0 tx=0x7fe5f400f000 comp rx=0 tx=0).stop 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 -- 172.21.15.125:0/3090184457 shutdown_connections 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea651560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea60d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3090184457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60dea0 0x555cea60e280 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 -- 172.21.15.125:0/3090184457 >> 172.21.15.125:0/3090184457 conn(0x555cea5cfc70 msgr2=0x555cea653990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.668+0000 7fe60cc32740 1 -- 172.21.15.125:0/3090184457 shutdown_connections 2024-01-16T14:58:19.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.669+0000 7fe60cc32740 1 -- 172.21.15.125:0/3090184457 wait complete. 2024-01-16T14:58:19.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.669+0000 7fe60cc32740 1 -- start start 2024-01-16T14:58:19.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.670+0000 7fe60cc32740 1 Processor -- start 2024-01-16T14:58:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea663890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60dea0 0x555cea656f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea574ac0 con 0x555cea60d5e0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea5112f0 con 0x555cea608ba0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cea52ec70 con 0x555cea60dea0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe60cc32740 10 client.0 _ll_register_callbacks cb 0x555cea5333b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57190/0 (socket says 172.21.15.125:57190) 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe5fa800640 1 -- 172.21.15.125:0/3640187285 learned_addr learned my addr 172.21.15.125:0/3640187285 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe603400640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea663890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe5fa800640 1 -- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60dea0 msgr2=0x555cea656f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.671+0000 7fe603e00640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60dea0 0x555cea656f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60dea0 0x555cea656f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5fa800640 1 -- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 msgr2=0x555cea663890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea663890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5fa800640 1 -- 172.21.15.125:0/3640187285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f400c7f0 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe603400640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea663890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec07ab00 tx=0x7fe5ec07a7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107314e024f2fb46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec07d050 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5ec07ef50 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cea56daa0 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5ec083100 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe60cc32740 10 client.5403 fetch_fsmap learned FSMap version 18 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x555cea573140 con 0x555cea60d5e0 2024-01-16T14:58:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.672+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec096070 con 0x555cea60d5e0 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe5ec07a2f0 con 0x555cea60d5e0 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 10 client.5403 fetch_fsmap finished waiting for FSMap version 18 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 10 client.5403 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 20 client.5403 populate_metadata read hostname 'smithi125' 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea64b2f0 0x555cea6656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 4 client.5403 mds_command: new command op to 5238 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 --> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cea56d7c0 con 0x555cea64b2f0 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.673+0000 7fe603400640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea64b2f0 0x555cea6656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.674+0000 7fe603400640 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea64b2f0 0x555cea6656d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e18ef2f55c0f80 server_cookie=bab355b0bcb324dd in_seq=0 out_seq=0 2024-01-16T14:58:19.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.674+0000 7fe5f0c00640 10 client.5403 ms_handle_connect on v2:172.21.15.125:6836/2265216525 2024-01-16T14:58:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.674+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3640187285 <== mds.0 v2:172.21.15.125:6836/2265216525 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555cea56d7c0 con 0x555cea64b2f0 2024-01-16T14:58:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.674+0000 7fe5f0c00640 10 client.5403 handle_command_reply: tid=0 2024-01-16T14:58:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.674+0000 7fe60cc32740 1 client.5403 shutdown 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 msgr2=0x555cea6569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec07ab00 tx=0x7fe5ec07a7e0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 shutdown_connections 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea64b2f0 0x555cea6656d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60dea0 0x555cea656f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea60d5e0 0x555cea6569f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3640187285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea608ba0 0x555cea663890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe5f0c00640 0 client.5403 ms_handle_reset on v2:172.21.15.125:6836/2265216525 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 >> 172.21.15.125:0/3640187285 conn(0x555cea5cfc70 msgr2=0x555cea653990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 shutdown_connections 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 1 -- 172.21.15.125:0/3640187285 wait complete. 2024-01-16T14:58:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.675+0000 7fe60cc32740 20 client.5403 trim_cache size 0 max 16384 2024-01-16T14:58:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.692+0000 7fe60cc32740 1 Processor -- start 2024-01-16T14:58:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.692+0000 7fe60cc32740 1 -- start start 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea653990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea653ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea2d7f20 con 0x555cea608ba0 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea60eda0 con 0x555cea60dea0 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cea6090d0 con 0x555cea60d5e0 2024-01-16T14:58:19.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57198/0 (socket says 172.21.15.125:57198) 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.693+0000 7fe603400640 1 -- 172.21.15.125:0/521845160 learned_addr learned my addr 172.21.15.125:0/521845160 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe5fa800640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea653990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603e00640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea653ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 -- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 msgr2=0x555cea653990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea653990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 -- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 msgr2=0x555cea653ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea653ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 -- 172.21.15.125:0/521845160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cea510b80 con 0x555cea608ba0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603e00640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea653ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe603400640 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe5f401e000 tx=0x7fe5f401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c64f6469f374d5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe5f8a00640 1 -- 172.21.15.125:0/521845160 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f401e9b0 con 0x555cea608ba0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe5f8a00640 1 -- 172.21.15.125:0/521845160 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f401eb10 con 0x555cea608ba0 2024-01-16T14:58:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.694+0000 7fe5f8a00640 1 -- 172.21.15.125:0/521845160 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f401ee80 con 0x555cea608ba0 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 -- 172.21.15.125:0/521845160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 msgr2=0x555cea663830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe5f401e000 tx=0x7fe5f401b000 comp rx=0 tx=0).stop 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 -- 172.21.15.125:0/521845160 shutdown_connections 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea653ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea653990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 --2- 172.21.15.125:0/521845160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea663830 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 -- 172.21.15.125:0/521845160 >> 172.21.15.125:0/521845160 conn(0x555cea5cfc70 msgr2=0x555cea63c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 -- 172.21.15.125:0/521845160 shutdown_connections 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.695+0000 7fe60cc32740 1 -- 172.21.15.125:0/521845160 wait complete. 2024-01-16T14:58:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.696+0000 7fe60cc32740 1 -- start start 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.696+0000 7fe60cc32740 1 Processor -- start 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea6c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea662c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea6044d0 con 0x555cea608ba0 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cea6597b0 con 0x555cea60dea0 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cea6c65d0 con 0x555cea60d5e0 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe60cc32740 10 client.0 _ll_register_callbacks cb 0x7fe5ec02be30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe603400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57206/0 (socket says 172.21.15.125:57206) 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.697+0000 7fe603400640 1 -- 172.21.15.125:0/3225763848 learned_addr learned my addr 172.21.15.125:0/3225763848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 -- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 msgr2=0x555cea6c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea6c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 -- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 msgr2=0x555cea662c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea662c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 -- 172.21.15.125:0/3225763848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cea53c860 con 0x555cea608ba0 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe603400640 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400a160 tx=0x7fe5f4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1390bfa539e96c10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe5f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f402e320 con 0x555cea608ba0 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5f402ee40 con 0x555cea608ba0 2024-01-16T14:58:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5f401e400 con 0x555cea608ba0 2024-01-16T14:58:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5f402e480 con 0x555cea608ba0 2024-01-16T14:58:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe60cc32740 10 client.5409 fetch_fsmap learned FSMap version 18 2024-01-16T14:58:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.698+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x555cea536170 con 0x555cea608ba0 2024-01-16T14:58:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f401e400 con 0x555cea608ba0 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe5f4035070 con 0x555cea608ba0 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 10 client.5409 fetch_fsmap finished waiting for FSMap version 18 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 10 client.5409 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 20 client.5409 populate_metadata read hostname 'smithi125' 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea658c80 0x555cea659060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 4 client.5409 mds_command: new command op to 5238 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 --> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555cea2756c0 con 0x555cea658c80 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.699+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea658c80 0x555cea659060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe5fa800640 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea658c80 0x555cea659060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c7cf2b04c4fee66 server_cookie=f10eab30afdb16eb in_seq=0 out_seq=0 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe5f0c00640 10 client.5409 ms_handle_connect on v2:172.21.15.125:6836/2265216525 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe5f0c00640 1 -- 172.21.15.125:0/3225763848 <== mds.0 v2:172.21.15.125:6836/2265216525 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x555cea2756c0 con 0x555cea658c80 2024-01-16T14:58:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe5f0c00640 10 client.5409 handle_command_reply: tid=0 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 client.5409 shutdown 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 msgr2=0x555cea649ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400a160 tx=0x7fe5f4013000 comp rx=0 tx=0).stop 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 shutdown_connections 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:6836/2265216525,v1:172.21.15.125:6837/2265216525] conn(0x555cea658c80 0x555cea659060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x555cea60dea0 0x555cea662c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x555cea60d5e0 0x555cea6c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 --2- 172.21.15.125:0/3225763848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x555cea608ba0 0x555cea649ed0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 >> 172.21.15.125:0/3225763848 conn(0x555cea5cfc70 msgr2=0x555cea653990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe5f0c00640 0 client.5409 ms_handle_reset on v2:172.21.15.125:6836/2265216525 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 shutdown_connections 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 1 -- 172.21.15.125:0/3225763848 wait complete. 2024-01-16T14:58:19.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.700+0000 7fe60cc32740 20 client.5409 trim_cache size 0 max 16384 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5238, 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.5972866220000004, 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 18, 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 34, 2024-01-16T14:58:19.687 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 34, 2024-01-16T14:58:19.688 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 16.491504149000001 2024-01-16T14:58:19.688 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:58:19.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 msgr2=0x7fe5dc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 0x7fe5dc06d780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400a870 tx=0x7fe5f4008040 comp rx=0 tx=0).stop 2024-01-16T14:58:19.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 msgr2=0x7fe60414cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec0046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 shutdown_connections 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe60415f9c0 0x7fe604145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe5dc06b2c0 0x7fe5dc06d780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe60413e360 0x7fe60414cac0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 --2- 172.21.15.125:0/2100600217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe60413df80 0x7fe6041479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 >> 172.21.15.125:0/2100600217 conn(0x7fe60406a800 msgr2=0x7fe60415dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 shutdown_connections 2024-01-16T14:58:19.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.701+0000 7fe60a000640 1 -- 172.21.15.125:0/2100600217 wait complete. 2024-01-16T14:58:19.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:58:19.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.915+0000 7f062a600640 1 Processor -- start 2024-01-16T14:58:19.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.915+0000 7f062a600640 1 -- start start 2024-01-16T14:58:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.916+0000 7f062a600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f06240cf9a0 0x7f062415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.916+0000 7f062a600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240b1ee0 con 0x7f0624143ae0 2024-01-16T14:58:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.916+0000 7f062a600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240b2780 con 0x7f062415e1e0 2024-01-16T14:58:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.916+0000 7f062a600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f062415e990 con 0x7f06240cf9a0 2024-01-16T14:58:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0623400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0624143ae0 0x7f06240cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45776/0 (socket says 172.21.15.125:45776) 2024-01-16T14:58:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0623400640 1 -- 172.21.15.125:0/3999567452 learned_addr learned my addr 172.21.15.125:0/3999567452 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766440690 0 0) 0x7f06240b1ee0 con 0x7f0624143ae0 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604003520 con 0x7f0624143ae0 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f061a800640 1 --2- 172.21.15.125:0/3999567452 >> v2:172.21.15.137:3300/0 conn(0x7f06240cf9a0 0x7f062415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3731246486 0 0) 0x7f0604003520 con 0x7f0624143ae0 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 >> v2:172.21.15.137:3300/0 conn(0x7f06240cf9a0 msgr2=0x7f062415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 --2- 172.21.15.125:0/3999567452 >> v2:172.21.15.137:3300/0 conn(0x7f06240cf9a0 0x7f062415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 >> v1:172.21.15.137:6789/0 conn(0x7f062415e1e0 legacy=0x7f06240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240b2130 con 0x7f0624143ae0 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0614001670 con 0x7f0624143ae0 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.917+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 >> v1:172.21.15.125:6789/0 conn(0x7f0624143ae0 legacy=0x7f06240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 0x7f0604003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06040047c0 0x7f0604004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0604005180 0x7f060400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062415e990 con 0x7f0604003970 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060400c0d0 con 0x7f0604005180 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060400e4a0 con 0x7f06040047c0 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 0x7f0604003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 -- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06040047c0 msgr2=0x7f0604004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06040047c0 0x7f0604004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 -- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0604005180 msgr2=0x7f060400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623e00640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0604005180 0x7f060400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0604005180 0x7f060400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623400640 1 -- 172.21.15.125:0/3999567452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0604003520 con 0x7f0604003970 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623e00640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0604005180 0x7f060400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.918+0000 7f0623e00640 1 -- 172.21.15.125:0/3999567452 reap_dead start 2024-01-16T14:58:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.919+0000 7f0623400640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 0x7f0604003d90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f061400c240 tx=0x7f06040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f9802126b096ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.920+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614005ad0 con 0x7f0604003970 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.920+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0614005c30 con 0x7f0604003970 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.920+0000 7f0622a00640 1 -- 172.21.15.125:0/3999567452 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061402caf0 con 0x7f0604003970 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.920+0000 7f062a600640 1 -- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 msgr2=0x7f0604003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.920+0000 7f062a600640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 0x7f0604003d90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f061400c240 tx=0x7f06040057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.921+0000 7f062a600640 1 -- 172.21.15.125:0/3999567452 shutdown_connections 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.921+0000 7f062a600640 1 --2- 172.21.15.125:0/3999567452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0604003970 0x7f0604003d90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.921+0000 7f062a600640 1 -- 172.21.15.125:0/3999567452 >> 172.21.15.125:0/3999567452 conn(0x7f062406a800 msgr2=0x7f062406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.921+0000 7f062a600640 1 -- 172.21.15.125:0/3999567452 shutdown_connections 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.921+0000 7f062a600640 1 -- 172.21.15.125:0/3999567452 wait complete. 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.923+0000 7f062a600640 1 Processor -- start 2024-01-16T14:58:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.923+0000 7f062a600640 1 -- start start 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 0x7f062413daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f062415a4e0 0x7f062415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06240b0530 con 0x7f06240afb70 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062415bc50 con 0x7f062415a4e0 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f062a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062415bf30 con 0x7f0604003970 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f0623e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.924+0000 7f0623e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57216/0 (socket says 172.21.15.125:57216) 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 -- 172.21.15.125:0/2500498861 learned_addr learned my addr 172.21.15.125:0/2500498861 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623400640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 0x7f062413daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f061a800640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f062415a4e0 0x7f062415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 -- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 msgr2=0x7f062413daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 0x7f062413daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 -- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f062415a4e0 msgr2=0x7f062415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f062415a4e0 0x7f062415b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 -- 172.21.15.125:0/2500498861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062415c210 con 0x7f06240afb70 2024-01-16T14:58:19.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623400640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 0x7f062413daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f0623e00640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0624071f10 tx=0x7f061c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.925+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c003d80 con 0x7f06240afb70 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.926+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c005070 con 0x7f06240afb70 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.926+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c005420 con 0x7f06240afb70 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.926+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062415c4f0 con 0x7f06240afb70 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.926+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062415b8f0 con 0x7f06240afb70 2024-01-16T14:58:19.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.927+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06240675d0 con 0x7f06240afb70 2024-01-16T14:58:19.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.928+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f061c005580 con 0x7f06240afb70 2024-01-16T14:58:19.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.929+0000 7f061be00640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 0x7f05fc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.929+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f061c089510 con 0x7f06240afb70 2024-01-16T14:58:19.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.929+0000 7f0623400640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 0x7f05fc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.930+0000 7f0623400640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 0x7f05fc06d7b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0614034000 tx=0x7f0614009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:19.933+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f061c0542e0 con 0x7f06240afb70 2024-01-16T14:58:20.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.165+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06240aeda0 con 0x7f06240afb70 2024-01-16T14:58:20.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.166+0000 7f061be00640 1 -- 172.21.15.125:0/2500498861 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f061c018060 con 0x7f06240afb70 2024-01-16T14:58:20.148 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:58:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 msgr2=0x7f05fc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 0x7f05fc06d7b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0614034000 tx=0x7f0614009030 comp rx=0 tx=0).stop 2024-01-16T14:58:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 msgr2=0x7f06240afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0624071f10 tx=0x7f061c00d9d0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 shutdown_connections 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05fc06b2f0 0x7f05fc06d7b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06240afb70 0x7f06240afff0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f062415a4e0 0x7f062415b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 --2- 172.21.15.125:0/2500498861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0604003970 0x7f062413daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.168+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 >> 172.21.15.125:0/2500498861 conn(0x7f062406a800 msgr2=0x7f0624142b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.169+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 shutdown_connections 2024-01-16T14:58:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:20.169+0000 7f062a600640 1 -- 172.21.15.125:0/2500498861 wait complete. 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:20.170 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:58:20.171 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:58:20.171 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:58:20.171 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:20.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.171 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42548sec preferred_lft 42548sec 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:58:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T14:58:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:20.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.222 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:58:20.306 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T14:58:20.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.310 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T14:58:20.361 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T14:58:20.361 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:58:20.361 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:58:20.361 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:58:20.361 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:58:20.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.362 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T14:58:20.362 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T14:58:20.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:58:20.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:58:20.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:58:20.363 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T14:58:20.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.542 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T14:58:20.591 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T14:58:20.592 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T14:58:20.593 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:20.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T14:58:20.684 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T14:58:20.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:20.685 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:20.741 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:20.741 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:20.742 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:20.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:58:20.880 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:58:20.936 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:58:20.937 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:20.938 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:20.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:20.940 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:58:20.989 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:58:20.989 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T14:58:20.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T14:58:21.069 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:21.113 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T14:58:21.113 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T14:58:21.114 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:58:21.115 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T14:58:21.116 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:21.116 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:21.116 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:21.116 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:21.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:21.117 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T14:58:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T14:58:21.198+0000 7f0f032d4480 -1 init, newargv = 0x56381555f920 newargc=15 2024-01-16T14:58:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[50138]: starting ceph client 2024-01-16T14:58:21.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[50138]: starting fuse 2024-01-16T14:58:21.203 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T14:58:21.203 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T14:58:21.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T14:58:21.204 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T14:58:21.205 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T14:58:21.393 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.50138.asok 2024-01-16T14:58:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:58:21.409 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50138.asok 2024-01-16T14:58:21.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:21.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50138.asok status 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "50138", 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T14:58:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5421, 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 5421 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2457186316 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2457186316 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.5421 192.168.0.1:0/2457186316", 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/2457186316", 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 18, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 34, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T14:58:21.645 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:58:21.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:21.658 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T14:58:21.658 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:21.712 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T14:58:21.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:21.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:21.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:21.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:58:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.013+0000 7f7b97e00640 1 Processor -- start 2024-01-16T14:58:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.013+0000 7f7b97e00640 1 -- start start 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b98162430 0x7f7b980b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b98162810 0x7f7b98071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 0x7f7b98071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b98071cc0 con 0x7f7b98162810 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b980702e0 con 0x7f7b980b3650 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.014+0000 7f7b97e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980705c0 con 0x7f7b98162430 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b98162810 0x7f7b98071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b98162810 0x7f7b98071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57228/0 (socket says 172.21.15.125:57228) 2024-01-16T14:58:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 -- 172.21.15.125:0/1659785501 learned_addr learned my addr 172.21.15.125:0/1659785501 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b97400640 1 --2- 172.21.15.125:0/1659785501 >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 0x7f7b98071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96a00640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b98162430 0x7f7b980b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 -- 172.21.15.125:0/1659785501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b98162430 msgr2=0x7f7b980b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b98162430 0x7f7b980b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 -- 172.21.15.125:0/1659785501 >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 msgr2=0x7f7b98071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 --2- 172.21.15.125:0/1659785501 >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 0x7f7b98071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b96000640 1 -- 172.21.15.125:0/1659785501 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980708a0 con 0x7f7b98162810 2024-01-16T14:58:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.015+0000 7f7b97400640 1 --2- 172.21.15.125:0/1659785501 >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 0x7f7b98071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.017+0000 7f7b96000640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162810 0x7f7b98071240 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7b980b2bc0 tx=0x7f7b8002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdb80ff687760de server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.017+0000 7f7b95600640 1 -- 172.21.15.125:0/1659785501 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80031860 con 0x7f7b98162810 2024-01-16T14:58:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.017+0000 7f7b95600640 1 -- 172.21.15.125:0/1659785501 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b800319c0 con 0x7f7b98162810 2024-01-16T14:58:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.017+0000 7f7b95600640 1 -- 172.21.15.125:0/1659785501 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80031d50 con 0x7f7b98162810 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 -- 172.21.15.125:0/1659785501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162810 msgr2=0x7f7b98071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162810 0x7f7b98071240 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7b980b2bc0 tx=0x7f7b8002b900 comp rx=0 tx=0).stop 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 -- 172.21.15.125:0/1659785501 shutdown_connections 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 --2- 172.21.15.125:0/1659785501 >> v2:172.21.15.137:3300/0 conn(0x7f7b980b3650 0x7f7b98071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162810 0x7f7b98071240 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 --2- 172.21.15.125:0/1659785501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b98162430 0x7f7b980b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 -- 172.21.15.125:0/1659785501 >> 172.21.15.125:0/1659785501 conn(0x7f7b9806a800 msgr2=0x7f7b9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 -- 172.21.15.125:0/1659785501 shutdown_connections 2024-01-16T14:58:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.018+0000 7f7b97e00640 1 -- 172.21.15.125:0/1659785501 wait complete. 2024-01-16T14:58:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.020+0000 7f7b97e00640 1 Processor -- start 2024-01-16T14:58:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.020+0000 7f7b97e00640 1 -- start start 2024-01-16T14:58:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b980b3650 0x7f7b98152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b981544d0 0x7f7b98154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980b0a60 con 0x7f7b98162430 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980b0d40 con 0x7f7b981544d0 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b97e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980b1020 con 0x7f7b980b3650 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57234/0 (socket says 172.21.15.125:57234) 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 -- 172.21.15.125:0/3654365250 learned_addr learned my addr 172.21.15.125:0/3654365250 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 -- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b980b3650 msgr2=0x7f7b98152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b980b3650 0x7f7b98152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 -- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b981544d0 msgr2=0x7f7b98154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b981544d0 0x7f7b98154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.021+0000 7f7b96000640 1 -- 172.21.15.125:0/3654365250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b800094d0 con 0x7f7b98162430 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b96000640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7b8002b940 tx=0x7f7b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8003acb0 con 0x7f7b98162430 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8003ae10 con 0x7f7b98162430 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80044630 con 0x7f7b98162430 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b980b1300 con 0x7f7b98162430 2024-01-16T14:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.022+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98154ec0 con 0x7f7b98162430 2024-01-16T14:58:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.023+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b98067450 con 0x7f7b98162430 2024-01-16T14:58:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.025+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b8004d450 con 0x7f7b98162430 2024-01-16T14:58:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.025+0000 7f7b8ea00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 0x7f7b7006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.025+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b800ac210 con 0x7f7b98162430 2024-01-16T14:58:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.026+0000 7f7b96a00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 0x7f7b7006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.026+0000 7f7b96a00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 0x7f7b7006d780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009870 tx=0x7f7b88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.030+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b80076fe0 con 0x7f7b98162430 2024-01-16T14:58:22.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.259+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b980b2ef0 con 0x7f7b98162430 2024-01-16T14:58:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.259+0000 7f7b8ea00640 1 -- 172.21.15.125:0/3654365250 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7b8007a600 con 0x7f7b98162430 2024-01-16T14:58:22.242 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:58:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 msgr2=0x7f7b7006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 0x7f7b7006d780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009870 tx=0x7f7b88008040 comp rx=0 tx=0).stop 2024-01-16T14:58:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 msgr2=0x7f7b9815b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7b8002b940 tx=0x7f7b800046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 shutdown_connections 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b981544d0 0x7f7b98154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b7006b2c0 0x7f7b7006d780 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b98162430 0x7f7b9815b860 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 --2- 172.21.15.125:0/3654365250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b980b3650 0x7f7b98152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 >> 172.21.15.125:0/3654365250 conn(0x7f7b9806a800 msgr2=0x7f7b9806d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 shutdown_connections 2024-01-16T14:58:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:22.262+0000 7f7b97e00640 1 -- 172.21.15.125:0/3654365250 wait complete. 2024-01-16T14:58:22.263 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:58:22.264 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:22.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.264 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T14:58:22.321 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:58:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:58:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:58:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42535sec preferred_lft 42535sec 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T14:58:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T14:58:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:22.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.324 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:58:22.413 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T14:58:22.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.417 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T14:58:22.470 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T14:58:22.470 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:58:22.470 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:58:22.471 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:58:22.471 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:58:22.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:58:22.472 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T14:58:22.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.650 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T14:58:22.702 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T14:58:22.702 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T14:58:22.703 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:22.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:22.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T14:58:22.796 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T14:58:22.796 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:22.796 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:22.846 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:22.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:22.847 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:22.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:58:23.032 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:58:23.085 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:23.086 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:23.087 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:23.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:23.088 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:58:23.140 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:58:23.141 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T14:58:23.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T14:58:23.223 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T14:58:23.264 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:58:23.265 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:23.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:23.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:23.267 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T14:58:23.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T14:58:23.350+0000 7f1768d60480 -1 init, newargv = 0x558d9c725b20 newargc=15 2024-01-16T14:58:23.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42954]: starting ceph client 2024-01-16T14:58:23.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42954]: starting fuse 2024-01-16T14:58:23.352 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T14:58:23.353 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T14:58:23.354 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T14:58:23.355 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T14:58:23.549 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.42954.asok 2024-01-16T14:58:23.553 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T14:58:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:58:23.566 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42954.asok 2024-01-16T14:58:23.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:23.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42954.asok status 2024-01-16T14:58:23.796 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "42954", 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:58:23.797 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4891, 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4891 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:23.798 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1872654823 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1872654823 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4891 192.168.0.1:0/1872654823", 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/1872654823", 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T14:58:23.799 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 18, 2024-01-16T14:58:23.800 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 34, 2024-01-16T14:58:23.800 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:58:23.800 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T14:58:23.800 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T14:58:23.800 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T14:58:23.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:23.812 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T14:58:23.812 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:23.862 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T14:58:23.862 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:23.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:23.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:23.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T14:58:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.186+0000 7f68ea400640 1 Processor -- start 2024-01-16T14:58:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.186+0000 7f68ea400640 1 -- start start 2024-01-16T14:58:24.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68ea400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e413e360 0x7f68e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68ea400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68e415f9c0 0x7f68e40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68ea400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e415c5f0 con 0x7f68e413df80 2024-01-16T14:58:24.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68ea400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e406c470 con 0x7f68e415f9c0 2024-01-16T14:58:24.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68ea400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e406c750 con 0x7f68e413e360 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.188+0000 7f68e3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68e415f9c0 0x7f68e40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68e415f9c0 0x7f68e40cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57240/0 (socket says 172.21.15.125:57240) 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 -- 172.21.15.125:0/3795446860 learned_addr learned my addr 172.21.15.125:0/3795446860 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68da800640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e413e360 0x7f68e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 -- 172.21.15.125:0/3795446860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e413e360 msgr2=0x7f68e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:24.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e413e360 0x7f68e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 -- 172.21.15.125:0/3795446860 >> v1:172.21.15.137:6789/0 conn(0x7f68e413df80 legacy=0x7f68e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.189+0000 7f68e3e00640 1 -- 172.21.15.125:0/3795446860 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e406ca30 con 0x7f68e415f9c0 2024-01-16T14:58:24.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.191+0000 7f68e3e00640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e415f9c0 0x7f68e40cf7d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009870 tx=0x7f68dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4448fcc316722602 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.192+0000 7f68e2a00640 1 -- 172.21.15.125:0/3795446860 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc030840 con 0x7f68e415f9c0 2024-01-16T14:58:24.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68e2a00640 1 -- 172.21.15.125:0/3795446860 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc0309a0 con 0x7f68e415f9c0 2024-01-16T14:58:24.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68e2a00640 1 -- 172.21.15.125:0/3795446860 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc030d30 con 0x7f68e415f9c0 2024-01-16T14:58:24.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 -- 172.21.15.125:0/3795446860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e415f9c0 msgr2=0x7f68e40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e415f9c0 0x7f68e40cf7d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009870 tx=0x7f68dc02b980 comp rx=0 tx=0).stop 2024-01-16T14:58:24.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 -- 172.21.15.125:0/3795446860 shutdown_connections 2024-01-16T14:58:24.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e415f9c0 0x7f68e40cf7d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 --2- 172.21.15.125:0/3795446860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e413e360 0x7f68e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 -- 172.21.15.125:0/3795446860 >> 172.21.15.125:0/3795446860 conn(0x7f68e406a800 msgr2=0x7f68e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 -- 172.21.15.125:0/3795446860 shutdown_connections 2024-01-16T14:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.193+0000 7f68ea400640 1 -- 172.21.15.125:0/3795446860 wait complete. 2024-01-16T14:58:24.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.195+0000 7f68ea400640 1 Processor -- start 2024-01-16T14:58:24.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.195+0000 7f68ea400640 1 -- start start 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68e413e360 0x7f68e40aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e415f9c0 0x7f68e412f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e412fab0 con 0x7f68e413df80 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e412fd10 con 0x7f68e413e360 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.196+0000 7f68ea400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e412fff0 con 0x7f68e415f9c0 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57242/0 (socket says 172.21.15.125:57242) 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 -- 172.21.15.125:0/2910664202 learned_addr learned my addr 172.21.15.125:0/2910664202 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68da800640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68e413e360 0x7f68e40aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3e00640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e415f9c0 0x7f68e412f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 -- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e415f9c0 msgr2=0x7f68e412f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e415f9c0 0x7f68e412f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 -- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68e413e360 msgr2=0x7f68e40aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68e413e360 0x7f68e40aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 -- 172.21.15.125:0/2910664202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc0094d0 con 0x7f68e413df80 2024-01-16T14:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.197+0000 7f68e3400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f68d400c730 tx=0x7f68d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.198+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4003b70 con 0x7f68e413df80 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.198+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d4003cf0 con 0x7f68e413df80 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.198+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4003ea0 con 0x7f68e413df80 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.198+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e41302d0 con 0x7f68e413df80 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.198+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e4130890 con 0x7f68e413df80 2024-01-16T14:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.199+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e40675d0 con 0x7f68e413df80 2024-01-16T14:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.200+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f68d400aa90 con 0x7f68e413df80 2024-01-16T14:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.201+0000 7f68dbe00640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 0x7f68bc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.201+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68d4089a10 con 0x7f68e413df80 2024-01-16T14:58:24.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.201+0000 7f68da800640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 0x7f68bc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.202+0000 7f68da800640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 0x7f68bc06d780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68e41f6560 tx=0x7f68cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.205+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68d4054870 con 0x7f68e413df80 2024-01-16T14:58:24.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:24.436+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f68e4130a50 con 0x7f68e413df80 2024-01-16T14:58:25.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.202+0000 7f68dbe00640 1 -- 172.21.15.125:0/2910664202 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 136+0+0 (secure 0 0 0) 0x7f68d4057e90 con 0x7f68e413df80 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.205+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 msgr2=0x7f68bc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.205+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 0x7f68bc06d780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68e41f6560 tx=0x7f68cc008040 comp rx=0 tx=0).stop 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.205+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 msgr2=0x7f68e4132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.205+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f68d400c730 tx=0x7f68d400cbf0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 shutdown_connections 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68e415f9c0 0x7f68e412f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68e413e360 0x7f68e40aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f68bc06b2c0 0x7f68bc06d780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 --2- 172.21.15.125:0/2910664202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68e413df80 0x7f68e4132130 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 >> 172.21.15.125:0/2910664202 conn(0x7f68e406a800 msgr2=0x7f68e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 shutdown_connections 2024-01-16T14:58:25.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.206+0000 7f68ea400640 1 -- 172.21.15.125:0/2910664202 wait complete. 2024-01-16T14:58:25.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.439+0000 7fa653200640 1 Processor -- start 2024-01-16T14:58:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.440+0000 7fa653200640 1 -- start start 2024-01-16T14:58:25.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa653200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64c143c40 0x7fa64c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa653200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa653200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c070e50 con 0x7fa64c0cca40 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa653200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0723c0 con 0x7fa64c143c40 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa653200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0cce70 con 0x7fa64c0717d0 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa651e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64c143c40 0x7fa64c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.441+0000 7fa651e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64c143c40 0x7fa64c071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57244/0 (socket says 172.21.15.125:57244) 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651e00640 1 -- 172.21.15.125:0/1012296673 learned_addr learned my addr 172.21.15.125:0/1012296673 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651400640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651e00640 1 -- 172.21.15.125:0/1012296673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 msgr2=0x7fa64c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651e00640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651e00640 1 -- 172.21.15.125:0/1012296673 >> v1:172.21.15.137:6789/0 conn(0x7fa64c0cca40 legacy=0x7fa64c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.442+0000 7fa651e00640 1 -- 172.21.15.125:0/1012296673 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c06e170 con 0x7fa64c143c40 2024-01-16T14:58:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.444+0000 7fa651e00640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c143c40 0x7fa64c071290 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009870 tx=0x7fa63c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d210b57d45b716be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.444+0000 7fa650a00640 1 -- 172.21.15.125:0/1012296673 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c031640 con 0x7fa64c143c40 2024-01-16T14:58:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.444+0000 7fa650a00640 1 -- 172.21.15.125:0/1012296673 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c0317a0 con 0x7fa64c143c40 2024-01-16T14:58:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.444+0000 7fa650a00640 1 -- 172.21.15.125:0/1012296673 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c031b30 con 0x7fa64c143c40 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 -- 172.21.15.125:0/1012296673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c143c40 msgr2=0x7fa64c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c143c40 0x7fa64c071290 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009870 tx=0x7fa63c02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 -- 172.21.15.125:0/1012296673 shutdown_connections 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 --2- 172.21.15.125:0/1012296673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c143c40 0x7fa64c071290 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 -- 172.21.15.125:0/1012296673 >> 172.21.15.125:0/1012296673 conn(0x7fa64c06a800 msgr2=0x7fa64c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 -- 172.21.15.125:0/1012296673 shutdown_connections 2024-01-16T14:58:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.445+0000 7fa653200640 1 -- 172.21.15.125:0/1012296673 wait complete. 2024-01-16T14:58:25.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.447+0000 7fa653200640 1 Processor -- start 2024-01-16T14:58:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.447+0000 7fa653200640 1 -- start start 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c13bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 0x7fa64c159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c141630 con 0x7fa64c0cca40 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c1420c0 con 0x7fa64c143c40 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa653200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c15a0b0 con 0x7fa64c0717d0 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa651400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa651400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57260/0 (socket says 172.21.15.125:57260) 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa651400640 1 -- 172.21.15.125:0/2679876101 learned_addr learned my addr 172.21.15.125:0/2679876101 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.448+0000 7fa651e00640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c13bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa652800640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 0x7fa64c159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 -- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 msgr2=0x7fa64c13bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c13bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 -- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 msgr2=0x7fa64c159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 0x7fa64c159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 -- 172.21.15.125:0/2679876101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c0094d0 con 0x7fa64c0cca40 2024-01-16T14:58:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa652800640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 0x7fa64c159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa651400640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa63400d800 tx=0x7fa63400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63400b6c0 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63400b820 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63400bbb0 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.449+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c141820 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.450+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c141de0 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.451+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c066e70 con 0x7fa64c0cca40 2024-01-16T14:58:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.452+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa63400bd10 con 0x7fa64c0cca40 2024-01-16T14:58:25.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.453+0000 7fa64a000640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 0x7fa62406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.453+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa63408a210 con 0x7fa64c0cca40 2024-01-16T14:58:25.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.453+0000 7fa651e00640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 0x7fa62406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.454+0000 7fa651e00640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 0x7fa62406d780 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.457+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa634054fe0 con 0x7fa64c0cca40 2024-01-16T14:58:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.669+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa64c0720e0 con 0x7fa62406b2c0 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.672+0000 7fa64a000640 1 -- 172.21.15.125:0/2679876101 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa63c031910 con 0x7fa62406b2c0 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:58:25.654 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.674+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 msgr2=0x7fa62406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.674+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 0x7fa62406d780 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03c040 comp rx=0 tx=0).stop 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.674+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 msgr2=0x7fa64c1410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.674+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa63400d800 tx=0x7fa63400dcc0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 shutdown_connections 2024-01-16T14:58:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa64c143c40 0x7fa64c159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa62406b2c0 0x7fa62406d780 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64c0cca40 0x7fa64c1410f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 --2- 172.21.15.125:0/2679876101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa64c0717d0 0x7fa64c13bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 >> 172.21.15.125:0/2679876101 conn(0x7fa64c06a800 msgr2=0x7fa64c15a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 shutdown_connections 2024-01-16T14:58:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.675+0000 7fa653200640 1 -- 172.21.15.125:0/2679876101 wait complete. 2024-01-16T14:58:25.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T14:58:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.909+0000 7fc816a00640 1 Processor -- start 2024-01-16T14:58:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.909+0000 7fc816a00640 1 -- start start 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc816a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc81013e360 0x7fc8100cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc816a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc81015f9c0 0x7fc8100cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc816a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81015c5f0 con 0x7fc81013df80 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc816a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006c470 con 0x7fc81015f9c0 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc816a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006c750 con 0x7fc81013e360 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc80fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc81013df80 0x7fc81015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45784/0 (socket says 172.21.15.125:45784) 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc80fe00640 1 -- 172.21.15.125:0/2015759290 learned_addr learned my addr 172.21.15.125:0/2015759290 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc814a00640 1 --2- 172.21.15.125:0/2015759290 >> v2:172.21.15.125:3300/0 conn(0x7fc81015f9c0 0x7fc8100cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.911+0000 7fc807e00640 1 --2- 172.21.15.125:0/2015759290 >> v2:172.21.15.137:3300/0 conn(0x7fc81013e360 0x7fc8100cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.912+0000 7fc814a00640 1 -- 172.21.15.125:0/2015759290 >> v2:172.21.15.137:3300/0 conn(0x7fc81013e360 msgr2=0x7fc8100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.912+0000 7fc814a00640 1 --2- 172.21.15.125:0/2015759290 >> v2:172.21.15.137:3300/0 conn(0x7fc81013e360 0x7fc8100cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.912+0000 7fc814a00640 1 -- 172.21.15.125:0/2015759290 >> v1:172.21.15.125:6789/0 conn(0x7fc81013df80 legacy=0x7fc81015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.912+0000 7fc814a00640 1 -- 172.21.15.125:0/2015759290 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81006ca30 con 0x7fc81015f9c0 2024-01-16T14:58:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.913+0000 7fc814a00640 1 --2- 172.21.15.125:0/2015759290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100cf7d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc81006ded0 tx=0x7fc80802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73570eba63b56ff7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.913+0000 7fc80f400640 1 -- 172.21.15.125:0/2015759290 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80802cc70 con 0x7fc81015f9c0 2024-01-16T14:58:25.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.913+0000 7fc80f400640 1 -- 172.21.15.125:0/2015759290 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80802cdd0 con 0x7fc81015f9c0 2024-01-16T14:58:25.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.913+0000 7fc80f400640 1 -- 172.21.15.125:0/2015759290 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808034660 con 0x7fc81015f9c0 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 -- 172.21.15.125:0/2015759290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 msgr2=0x7fc8100cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 --2- 172.21.15.125:0/2015759290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100cf7d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc81006ded0 tx=0x7fc80802bec0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 -- 172.21.15.125:0/2015759290 shutdown_connections 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 --2- 172.21.15.125:0/2015759290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100cf7d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 --2- 172.21.15.125:0/2015759290 >> v2:172.21.15.137:3300/0 conn(0x7fc81013e360 0x7fc8100cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 -- 172.21.15.125:0/2015759290 >> 172.21.15.125:0/2015759290 conn(0x7fc81006a800 msgr2=0x7fc81006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 -- 172.21.15.125:0/2015759290 shutdown_connections 2024-01-16T14:58:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.914+0000 7fc816a00640 1 -- 172.21.15.125:0/2015759290 wait complete. 2024-01-16T14:58:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.916+0000 7fc816a00640 1 Processor -- start 2024-01-16T14:58:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.916+0000 7fc816a00640 1 -- start start 2024-01-16T14:58:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc81013df80 0x7fc81013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc81013e360 0x7fc81014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81014c3a0 con 0x7fc81015f9c0 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81013de10 con 0x7fc81013e360 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc816a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81014b070 con 0x7fc81013df80 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57276/0 (socket says 172.21.15.125:57276) 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 -- 172.21.15.125:0/3170593983 learned_addr learned my addr 172.21.15.125:0/3170593983 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc807e00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc81013e360 0x7fc81014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc80fe00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc81013df80 0x7fc81013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 -- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc81013df80 msgr2=0x7fc81013d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc81013df80 0x7fc81013d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 -- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc81013e360 msgr2=0x7fc81014f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc81013e360 0x7fc81014f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.917+0000 7fc814a00640 1 -- 172.21.15.125:0/3170593983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808009a70 con 0x7fc81015f9c0 2024-01-16T14:58:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc814a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc808009e80 tx=0x7fc808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808040650 con 0x7fc81015f9c0 2024-01-16T14:58:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8080407d0 con 0x7fc81015f9c0 2024-01-16T14:58:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808040b80 con 0x7fc81015f9c0 2024-01-16T14:58:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81014b350 con 0x7fc81015f9c0 2024-01-16T14:58:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.918+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81014b910 con 0x7fc81015f9c0 2024-01-16T14:58:25.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.919+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81014c4e0 con 0x7fc81015f9c0 2024-01-16T14:58:25.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.920+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc808040ce0 con 0x7fc81015f9c0 2024-01-16T14:58:25.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.921+0000 7fc80cc00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 0x7fc7e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.921+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8080ac8c0 con 0x7fc81015f9c0 2024-01-16T14:58:25.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.921+0000 7fc80fe00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 0x7fc7e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.922+0000 7fc80fe00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 0x7fc7e806d780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc800000f60 tx=0x7fc800009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:25.926+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc81014c4e0 con 0x7fc81015f9c0 2024-01-16T14:58:26.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.105+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc81014c4e0 con 0x7fc81015f9c0 2024-01-16T14:58:26.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.106+0000 7fc80cc00640 1 -- 172.21.15.125:0/3170593983 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc81014c4e0 con 0x7fc81015f9c0 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 msgr2=0x7fc7e806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 0x7fc7e806d780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc800000f60 tx=0x7fc800009040 comp rx=0 tx=0).stop 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 msgr2=0x7fc8100b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc808009e80 tx=0x7fc808004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 shutdown_connections 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e806b2c0 0x7fc7e806d780 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81015f9c0 0x7fc8100b2f50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc81013e360 0x7fc81014f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 --2- 172.21.15.125:0/3170593983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc81013df80 0x7fc81013d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 >> 172.21.15.125:0/3170593983 conn(0x7fc81006a800 msgr2=0x7fc81015ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 shutdown_connections 2024-01-16T14:58:26.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.109+0000 7fc816a00640 1 -- 172.21.15.125:0/3170593983 wait complete. 2024-01-16T14:58:26.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T14:58:26.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.340+0000 7fa4e7400640 1 Processor -- start 2024-01-16T14:58:26.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.340+0000 7fa4e7400640 1 -- start start 2024-01-16T14:58:26.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4e0143c40 0x7fa4e0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa4e00717d0 0x7fa4e00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e7400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e0070e50 con 0x7fa4e00cca40 2024-01-16T14:58:26.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e7400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e00723c0 con 0x7fa4e0143c40 2024-01-16T14:58:26.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e7400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e00cce40 con 0x7fa4e00717d0 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4e0143c40 0x7fa4e0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e6a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa4e00cca40 0x7fa4e015eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45790/0 (socket says 172.21.15.125:45790) 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.341+0000 7fa4e6a00640 1 -- 172.21.15.125:0/341661248 learned_addr learned my addr 172.21.15.125:0/341661248 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e5600640 1 --2- 172.21.15.125:0/341661248 >> v2:172.21.15.137:3300/0 conn(0x7fa4e00717d0 0x7fa4e00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e4c00640 1 -- 172.21.15.125:0/341661248 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914866561 0 0) 0x7fa4e0070e50 con 0x7fa4e00cca40 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e4c00640 1 -- 172.21.15.125:0/341661248 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c0003520 con 0x7fa4e00cca40 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e6000640 1 -- 172.21.15.125:0/341661248 >> v2:172.21.15.137:3300/0 conn(0x7fa4e00717d0 msgr2=0x7fa4e00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e6000640 1 --2- 172.21.15.125:0/341661248 >> v2:172.21.15.137:3300/0 conn(0x7fa4e00717d0 0x7fa4e00cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e6000640 1 -- 172.21.15.125:0/341661248 >> v1:172.21.15.125:6789/0 conn(0x7fa4e00cca40 legacy=0x7fa4e015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.342+0000 7fa4e6000640 1 -- 172.21.15.125:0/341661248 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e006e170 con 0x7fa4e0143c40 2024-01-16T14:58:26.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.343+0000 7fa4e6000640 1 --2- 172.21.15.125:0/341661248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e0143c40 0x7fa4e0071290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b44a0 tx=0x7fa4d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c96def277b2e893 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e4c00640 1 -- 172.21.15.125:0/341661248 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d002e070 con 0x7fa4e0143c40 2024-01-16T14:58:26.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e4c00640 1 -- 172.21.15.125:0/341661248 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d00315a0 con 0x7fa4e0143c40 2024-01-16T14:58:26.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e4c00640 1 -- 172.21.15.125:0/341661248 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0031910 con 0x7fa4e0143c40 2024-01-16T14:58:26.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 -- 172.21.15.125:0/341661248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e0143c40 msgr2=0x7fa4e0071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 --2- 172.21.15.125:0/341661248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e0143c40 0x7fa4e0071290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b44a0 tx=0x7fa4d002b840 comp rx=0 tx=0).stop 2024-01-16T14:58:26.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 -- 172.21.15.125:0/341661248 shutdown_connections 2024-01-16T14:58:26.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 --2- 172.21.15.125:0/341661248 >> v2:172.21.15.137:3300/0 conn(0x7fa4e00717d0 0x7fa4e00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 --2- 172.21.15.125:0/341661248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e0143c40 0x7fa4e0071290 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 -- 172.21.15.125:0/341661248 >> 172.21.15.125:0/341661248 conn(0x7fa4e006a800 msgr2=0x7fa4e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 -- 172.21.15.125:0/341661248 shutdown_connections 2024-01-16T14:58:26.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.344+0000 7fa4e7400640 1 -- 172.21.15.125:0/341661248 wait complete. 2024-01-16T14:58:26.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.346+0000 7fa4e7400640 1 Processor -- start 2024-01-16T14:58:26.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.346+0000 7fa4e7400640 1 -- start start 2024-01-16T14:58:26.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4e00717d0 0x7fa4e01e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 0x7fa4e00b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00b1f00 con 0x7fa4e00cca40 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01f3360 con 0x7fa4e00717d0 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.347+0000 7fa4e7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01f3640 con 0x7fa4e0143c40 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57300/0 (socket says 172.21.15.125:57300) 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 -- 172.21.15.125:0/4005887031 learned_addr learned my addr 172.21.15.125:0/4005887031 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e6000640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4e00717d0 0x7fa4e01e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e6a00640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 0x7fa4e00b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 -- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 msgr2=0x7fa4e00b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 0x7fa4e00b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 -- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4e00717d0 msgr2=0x7fa4e01e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4e00717d0 0x7fa4e01e1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 -- 172.21.15.125:0/4005887031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0009400 con 0x7fa4e00cca40 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e6a00640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 0x7fa4e00b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.348+0000 7fa4e5600640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d800 tx=0x7fa4c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.349+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800b6c0 con 0x7fa4e00cca40 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.349+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c800b820 con 0x7fa4e00cca40 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.349+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800bbb0 con 0x7fa4e00cca40 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.349+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01f3920 con 0x7fa4e00cca40 2024-01-16T14:58:26.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.349+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01e2070 con 0x7fa4e00cca40 2024-01-16T14:58:26.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.350+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0067450 con 0x7fa4e00cca40 2024-01-16T14:58:26.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.351+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa4c800bd10 con 0x7fa4e00cca40 2024-01-16T14:58:26.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.352+0000 7fa4de000640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 0x7fa4b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.352+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4c808a180 con 0x7fa4e00cca40 2024-01-16T14:58:26.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.352+0000 7fa4e6000640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 0x7fa4b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.353+0000 7fa4e6000640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 0x7fa4b806d780 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0036000 tx=0x7fa4d003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.357+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4c8054fe0 con 0x7fa4e00cca40 2024-01-16T14:58:26.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.573+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4e00720e0 con 0x7fa4b806b2c0 2024-01-16T14:58:26.566 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:58:26.584+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:26.566 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:58:26.584+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:26.566 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:58:26.584+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:26.566 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:58:26.584+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:26.566 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:58:26.584+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:26.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.626+0000 7fa4de000640 1 -- 172.21.15.125:0/4005887031 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa4d00319f0 con 0x7fa4b806b2c0 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.628+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 msgr2=0x7fa4b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.628+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 0x7fa4b806d780 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0036000 tx=0x7fa4d003d040 comp rx=0 tx=0).stop 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 msgr2=0x7fa4e00adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d800 tx=0x7fa4c800dcc0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 shutdown_connections 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4e0143c40 0x7fa4e00b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4b806b2c0 0x7fa4b806d780 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4e00cca40 0x7fa4e00adb90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 --2- 172.21.15.125:0/4005887031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4e00717d0 0x7fa4e01e1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 >> 172.21.15.125:0/4005887031 conn(0x7fa4e006a800 msgr2=0x7fa4e013cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 shutdown_connections 2024-01-16T14:58:26.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.629+0000 7fa4e7400640 1 -- 172.21.15.125:0/4005887031 wait complete. 2024-01-16T14:58:26.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T14:58:26.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.845+0000 7f87dfe00640 1 Processor -- start 2024-01-16T14:58:26.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.845+0000 7f87dfe00640 1 -- start start 2024-01-16T14:58:26.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d8143ae0 0x7f87d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87d80cf7d0 0x7f87d815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dfe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d80cfc30 con 0x7f87d815e1e0 2024-01-16T14:58:26.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dfe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d80b1ee0 con 0x7f87d80cf7d0 2024-01-16T14:58:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80b2780 con 0x7f87d8143ae0 2024-01-16T14:58:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87d80cf7d0 0x7f87d815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f87d815e1e0 0x7f87d815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50858/0 (socket says 172.21.15.125:50858) 2024-01-16T14:58:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dde00640 1 -- 172.21.15.125:0/2151854266 learned_addr learned my addr 172.21.15.125:0/2151854266 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:58:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dca00640 1 -- 172.21.15.125:0/2151854266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d8143ae0 msgr2=0x7f87d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dca00640 1 --2- 172.21.15.125:0/2151854266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d8143ae0 0x7f87d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dca00640 1 -- 172.21.15.125:0/2151854266 >> v1:172.21.15.125:6789/0 conn(0x7f87d815e1e0 legacy=0x7f87d815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87dca00640 1 -- 172.21.15.125:0/2151854266 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d815e990 con 0x7f87d80cf7d0 2024-01-16T14:58:26.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.848+0000 7f87d7e00640 1 -- 172.21.15.125:0/2151854266 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744635192 0 0) 0x7f87d80cfc30 con 0x7f87d815e1e0 2024-01-16T14:58:26.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.849+0000 7f87dca00640 1 --2- 172.21.15.125:0/2151854266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d80cf7d0 0x7f87d815dca0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f87c00097a0 tx=0x7f87c002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a39391dbcc5edc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.849+0000 7f87d7e00640 1 -- 172.21.15.125:0/2151854266 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c002d040 con 0x7f87d80cf7d0 2024-01-16T14:58:26.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87d7e00640 1 -- 172.21.15.125:0/2151854266 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c0031810 con 0x7f87d80cf7d0 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87d7e00640 1 -- 172.21.15.125:0/2151854266 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0031ba0 con 0x7f87d80cf7d0 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 -- 172.21.15.125:0/2151854266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d80cf7d0 msgr2=0x7f87d815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 --2- 172.21.15.125:0/2151854266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d80cf7d0 0x7f87d815dca0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f87c00097a0 tx=0x7f87c002b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 -- 172.21.15.125:0/2151854266 shutdown_connections 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 --2- 172.21.15.125:0/2151854266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d80cf7d0 0x7f87d815dca0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 --2- 172.21.15.125:0/2151854266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d8143ae0 0x7f87d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 -- 172.21.15.125:0/2151854266 >> 172.21.15.125:0/2151854266 conn(0x7f87d806a800 msgr2=0x7f87d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 -- 172.21.15.125:0/2151854266 shutdown_connections 2024-01-16T14:58:26.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.850+0000 7f87dfe00640 1 -- 172.21.15.125:0/2151854266 wait complete. 2024-01-16T14:58:26.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.852+0000 7f87dfe00640 1 Processor -- start 2024-01-16T14:58:26.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.852+0000 7f87dfe00640 1 -- start start 2024-01-16T14:58:26.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 0x7f87d8133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87d8143ae0 0x7f87d815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d813f7f0 con 0x7f87d815e1e0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8131d80 con 0x7f87d8143ae0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d813eea0 con 0x7f87d80cf7d0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49400/0 (socket says 172.21.15.125:49400) 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.853+0000 7f87dde00640 1 -- 172.21.15.125:0/3052934729 learned_addr learned my addr 172.21.15.125:0/3052934729 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dca00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87d8143ae0 0x7f87d815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dd400640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 0x7f87d8133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 -- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 msgr2=0x7f87d8133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 0x7f87d8133ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 -- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87d8143ae0 msgr2=0x7f87d815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87d8143ae0 0x7f87d815caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 -- 172.21.15.125:0/3052934729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0009400 con 0x7f87d815e1e0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dd400640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 0x7f87d8133ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dde00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001eb0 tx=0x7f87d000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d00074e0 con 0x7f87d815e1e0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d0007640 con 0x7f87d815e1e0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d0004a00 con 0x7f87d815e1e0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.854+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d80c45f0 con 0x7f87d815e1e0 2024-01-16T14:58:26.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.855+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d80c4310 con 0x7f87d815e1e0 2024-01-16T14:58:26.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.856+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d8131ef0 con 0x7f87d815e1e0 2024-01-16T14:58:26.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.857+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f87d0004b60 con 0x7f87d815e1e0 2024-01-16T14:58:26.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.857+0000 7f87d5600640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 0x7f87b006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.858+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87d0089a70 con 0x7f87d815e1e0 2024-01-16T14:58:26.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.858+0000 7f87dd400640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 0x7f87b006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.858+0000 7f87dd400640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 0x7f87b006d780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87c8007eb0 tx=0x7f87c80073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:26.862+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87d00582d0 con 0x7f87d815e1e0 2024-01-16T14:58:27.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.078+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f87d815e640 con 0x7f87b006b2c0 2024-01-16T14:58:27.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.085+0000 7f87d5600640 1 -- 172.21.15.125:0/3052934729 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f87c800cdd0 con 0x7f87b006b2c0 2024-01-16T14:58:27.068 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:58:27.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 msgr2=0x7f87b006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 0x7f87b006d780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87c8007eb0 tx=0x7f87c80073c0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 msgr2=0x7f87d80cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001eb0 tx=0x7f87d000c870 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 shutdown_connections 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b006b2c0 0x7f87b006d780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87d815e1e0 0x7f87d80cadd0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87d8143ae0 0x7f87d815caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 --2- 172.21.15.125:0/3052934729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87d80cf7d0 0x7f87d8133ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 >> 172.21.15.125:0/3052934729 conn(0x7f87d806a800 msgr2=0x7f87d815ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 shutdown_connections 2024-01-16T14:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:58:27.089+0000 7f87dfe00640 1 -- 172.21.15.125:0/3052934729 wait complete. 2024-01-16T14:58:27.089 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:58:27.089 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:58:27.089 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:27.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:27.135 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:58:36.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:36.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T14:58:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T14:58:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.7019 s, 73.0 MB/s 2024-01-16T14:58:41.953 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:58:41.953 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:42.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:42.006 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:58:53.534 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T14:58:53.535 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T14:58:53.535 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.4276 s, 94.0 MB/s 2024-01-16T14:58:53.536 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:58:53.536 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:53.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:53.589 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:59:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T14:59:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T14:59:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8886 s, 98.6 MB/s 2024-01-16T14:59:04.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T14:59:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.753+0000 7f3801400640 1 Processor -- start 2024-01-16T14:59:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.753+0000 7f3801400640 1 -- start start 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3801400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37fc13cf10 0x7f37fc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3801400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 0x7f37fc0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3801400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc0ce440 con 0x7f37fc0cdc20 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3801400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc13ff60 con 0x7f37fc13cf10 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3801400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0ce720 con 0x7f37fc13fae0 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f37fbe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37fc13cf10 0x7f37fc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3800a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f37fc0cdc20 0x7f37fc15d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49840/0 (socket says 172.21.15.125:49840) 2024-01-16T14:59:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.755+0000 7f3800a00640 1 -- 172.21.15.125:0/1984354869 learned_addr learned my addr 172.21.15.125:0/1984354869 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37f3200640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 0x7f37fc0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37fb400640 1 -- 172.21.15.125:0/1984354869 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491044343 0 0) 0x7f37fc0ce440 con 0x7f37fc0cdc20 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37fbe00640 1 -- 172.21.15.125:0/1984354869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 msgr2=0x7f37fc0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37fbe00640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 0x7f37fc0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37fbe00640 1 -- 172.21.15.125:0/1984354869 >> v1:172.21.15.125:6789/0 conn(0x7f37fc0cdc20 legacy=0x7f37fc15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37fbe00640 1 -- 172.21.15.125:0/1984354869 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc0cea00 con 0x7f37fc13cf10 2024-01-16T14:59:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.756+0000 7f37f3200640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 0x7f37fc0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.757+0000 7f37fbe00640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc13f5a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0097a0 tx=0x7f37ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d90f41118e5760 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.757+0000 7f37fb400640 1 -- 172.21.15.125:0/1984354869 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec02d040 con 0x7f37fc13cf10 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.757+0000 7f37fb400640 1 -- 172.21.15.125:0/1984354869 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec031810 con 0x7f37fc13cf10 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.757+0000 7f37fb400640 1 -- 172.21.15.125:0/1984354869 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec031ba0 con 0x7f37fc13cf10 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 -- 172.21.15.125:0/1984354869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 msgr2=0x7f37fc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc13f5a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0097a0 tx=0x7f37ec02b8b0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 -- 172.21.15.125:0/1984354869 shutdown_connections 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc13fae0 0x7f37fc0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 --2- 172.21.15.125:0/1984354869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc13f5a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 -- 172.21.15.125:0/1984354869 >> 172.21.15.125:0/1984354869 conn(0x7f37fc06a7d0 msgr2=0x7f37fc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 -- 172.21.15.125:0/1984354869 shutdown_connections 2024-01-16T14:59:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.758+0000 7f3801400640 1 -- 172.21.15.125:0/1984354869 wait complete. 2024-01-16T14:59:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.760+0000 7f3801400640 1 Processor -- start 2024-01-16T14:59:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.760+0000 7f3801400640 1 -- start start 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc0cdc20 0x7f37fc138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37fc13fae0 0x7f37fc15c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc13d6b0 con 0x7f37fc13cf10 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc138cc0 con 0x7f37fc13fae0 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f3801400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc13d2f0 con 0x7f37fc0cdc20 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f37f3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f37f3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57712/0 (socket says 172.21.15.125:57712) 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.761+0000 7f37f3200640 1 -- 172.21.15.125:0/1312130539 learned_addr learned my addr 172.21.15.125:0/1312130539 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37fbe00640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc0cdc20 0x7f37fc138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f3800a00640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37fc13fae0 0x7f37fc15c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 -- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc0cdc20 msgr2=0x7f37fc138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc0cdc20 0x7f37fc138780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 -- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37fc13fae0 msgr2=0x7f37fc15c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37fc13fae0 0x7f37fc15c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 -- 172.21.15.125:0/1312130539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec009400 con 0x7f37fc13cf10 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f3200640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c7f0 tx=0x7f37e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4007820 con 0x7f37fc13cf10 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e4007980 con 0x7f37fc13cf10 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4007cf0 con 0x7f37fc13cf10 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.762+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc157260 con 0x7f37fc13cf10 2024-01-16T14:59:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.763+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc157820 con 0x7f37fc13cf10 2024-01-16T14:59:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.764+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc13d7f0 con 0x7f37fc13cf10 2024-01-16T14:59:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.765+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f37e40113f0 con 0x7f37fc13cf10 2024-01-16T14:59:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.766+0000 7f37f8c00640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 0x7f37d406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.766+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37e408a0b0 con 0x7f37fc13cf10 2024-01-16T14:59:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.766+0000 7f37fbe00640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 0x7f37d406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.767+0000 7f37fbe00640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 0x7f37d406d810 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f37ec036000 tx=0x7f37ec03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:04.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.771+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37fc13d7f0 con 0x7f37fc13cf10 2024-01-16T14:59:04.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.964+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f37fc13d7f0 con 0x7f37fc13cf10 2024-01-16T14:59:04.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.971+0000 7f37f8c00640 1 -- 172.21.15.125:0/1312130539 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f37fc13d7f0 con 0x7f37fc13cf10 2024-01-16T14:59:04.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.974+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 msgr2=0x7f37d406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.974+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 0x7f37d406d810 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f37ec036000 tx=0x7f37ec03c040 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.974+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 msgr2=0x7f37fc15a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.974+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c7f0 tx=0x7f37e400ccb0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 shutdown_connections 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37fc13fae0 0x7f37fc15c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37d406b350 0x7f37d406d810 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37fc13cf10 0x7f37fc15a870 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 --2- 172.21.15.125:0/1312130539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37fc0cdc20 0x7f37fc138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 >> 172.21.15.125:0/1312130539 conn(0x7f37fc06a7d0 msgr2=0x7f37fc0716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 shutdown_connections 2024-01-16T14:59:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:04.975+0000 7f3801400640 1 -- 172.21.15.125:0/1312130539 wait complete. 2024-01-16T14:59:04.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T14:59:05.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.203+0000 7f6fc5200640 1 Processor -- start 2024-01-16T14:59:05.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.203+0000 7f6fc5200640 1 -- start start 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc013df80 0x7f6fc015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 0x7f6fc00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc015f9c0 0x7f6fc00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc006c470 con 0x7f6fc013df80 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc006c750 con 0x7f6fc013e360 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fc5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc006ca30 con 0x7f6fc015f9c0 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fbde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc013df80 0x7f6fc015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fbde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc013df80 0x7f6fc015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57728/0 (socket says 172.21.15.125:57728) 2024-01-16T14:59:05.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.205+0000 7f6fbde00640 1 -- 172.21.15.125:0/1221182734 learned_addr learned my addr 172.21.15.125:0/1221182734 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbd400640 1 --2- 172.21.15.125:0/1221182734 >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 0x7f6fc00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbe800640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc015f9c0 0x7f6fc00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbde00640 1 -- 172.21.15.125:0/1221182734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc015f9c0 msgr2=0x7f6fc00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbde00640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc015f9c0 0x7f6fc00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbde00640 1 -- 172.21.15.125:0/1221182734 >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 msgr2=0x7f6fc00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:05.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbde00640 1 --2- 172.21.15.125:0/1221182734 >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 0x7f6fc00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbde00640 1 -- 172.21.15.125:0/1221182734 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc006cd10 con 0x7f6fc013df80 2024-01-16T14:59:05.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.206+0000 7f6fbd400640 1 --2- 172.21.15.125:0/1221182734 >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 0x7f6fc00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:05.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.208+0000 7f6fbde00640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc015bee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6fc006ded0 tx=0x7f6fb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc970cd6cf3e89c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.208+0000 7f6fbca00640 1 -- 172.21.15.125:0/1221182734 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0031870 con 0x7f6fc013df80 2024-01-16T14:59:05.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.208+0000 7f6fbca00640 1 -- 172.21.15.125:0/1221182734 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00319d0 con 0x7f6fc013df80 2024-01-16T14:59:05.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.208+0000 7f6fbca00640 1 -- 172.21.15.125:0/1221182734 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0031d60 con 0x7f6fc013df80 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 -- 172.21.15.125:0/1221182734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 msgr2=0x7f6fc015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc015bee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6fc006ded0 tx=0x7f6fb002b910 comp rx=0 tx=0).stop 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 -- 172.21.15.125:0/1221182734 shutdown_connections 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc015f9c0 0x7f6fc00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 --2- 172.21.15.125:0/1221182734 >> v2:172.21.15.137:3300/0 conn(0x7f6fc013e360 0x7f6fc00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 --2- 172.21.15.125:0/1221182734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc015bee0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 -- 172.21.15.125:0/1221182734 >> 172.21.15.125:0/1221182734 conn(0x7f6fc006a800 msgr2=0x7f6fc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 -- 172.21.15.125:0/1221182734 shutdown_connections 2024-01-16T14:59:05.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.209+0000 7f6fc5200640 1 -- 172.21.15.125:0/1221182734 wait complete. 2024-01-16T14:59:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.211+0000 7f6fc5200640 1 Processor -- start 2024-01-16T14:59:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.211+0000 7f6fc5200640 1 -- start start 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc013e360 0x7f6fc013aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 0x7f6fc014c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00719c0 con 0x7f6fc013df80 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc013b010 con 0x7f6fc015f9c0 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fc5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc013be40 con 0x7f6fc013e360 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.212+0000 7f6fbde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57730/0 (socket says 172.21.15.125:57730) 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 -- 172.21.15.125:0/3590315449 learned_addr learned my addr 172.21.15.125:0/3590315449 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbe800640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 0x7f6fc014c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbd400640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc013e360 0x7f6fc013aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 -- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc013e360 msgr2=0x7f6fc013aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc013e360 0x7f6fc013aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 -- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 msgr2=0x7f6fc014c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 0x7f6fc014c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 -- 172.21.15.125:0/3590315449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00094d0 con 0x7f6fc013df80 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbe800640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 0x7f6fc014c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.213+0000 7f6fbde00640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0000c00 tx=0x7f6fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.214+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb003ad30 con 0x7f6fc013df80 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.214+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00443f0 con 0x7f6fc013df80 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.214+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00447a0 con 0x7f6fc013df80 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.214+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc013b230 con 0x7f6fc013df80 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.214+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc013b7f0 con 0x7f6fc013df80 2024-01-16T14:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.215+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc00675d0 con 0x7f6fc013df80 2024-01-16T14:59:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.216+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f6fb0044900 con 0x7f6fc013df80 2024-01-16T14:59:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.217+0000 7f6fb6000640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 0x7f6f9806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.217+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fb002d040 con 0x7f6fc013df80 2024-01-16T14:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.217+0000 7f6fbd400640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 0x7f6f9806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.218+0000 7f6fbd400640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 0x7f6f9806d810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8006fd0 tx=0x7f6fa8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.222+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fb0077070 con 0x7f6fc013df80 2024-01-16T14:59:05.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.445+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fc01408d0 con 0x7f6f9806b350 2024-01-16T14:59:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.470+0000 7f6fb6000640 1 -- 172.21.15.125:0/3590315449 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fa80095a0 con 0x7f6f9806b350 2024-01-16T14:59:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 msgr2=0x7f6f9806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 0x7f6f9806d810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8006fd0 tx=0x7f6fa8008040 comp rx=0 tx=0).stop 2024-01-16T14:59:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 msgr2=0x7f6fc0071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0000c00 tx=0x7f6fb00046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 shutdown_connections 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fc015f9c0 0x7f6fc014c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fc013e360 0x7f6fc013aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9806b350 0x7f6f9806d810 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.472+0000 7f6fc5200640 1 --2- 172.21.15.125:0/3590315449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc013df80 0x7f6fc0071480 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.473+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 >> 172.21.15.125:0/3590315449 conn(0x7f6fc006a800 msgr2=0x7f6fc01401a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.473+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 shutdown_connections 2024-01-16T14:59:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.473+0000 7f6fc5200640 1 -- 172.21.15.125:0/3590315449 wait complete. 2024-01-16T14:59:05.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T14:59:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.701+0000 7fa543e00640 1 Processor -- start 2024-01-16T14:59:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.701+0000 7fa543e00640 1 -- start start 2024-01-16T14:59:05.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa543e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa53c143ae0 0x7fa53c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa543e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa53c15e3b0 0x7fa53c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa543e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c0cfc30 con 0x7fa53c0cf7d0 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa543e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c0b1ee0 con 0x7fa53c143ae0 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa543e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c0b2780 con 0x7fa53c15e3b0 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa53c143ae0 0x7fa53c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa53c143ae0 0x7fa53c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57746/0 (socket says 172.21.15.125:57746) 2024-01-16T14:59:05.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 -- 172.21.15.125:0/514631918 learned_addr learned my addr 172.21.15.125:0/514631918 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 -- 172.21.15.125:0/514631918 >> v2:172.21.15.137:3300/0 conn(0x7fa53c15e3b0 msgr2=0x7fa53c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541e00640 1 --2- 172.21.15.125:0/514631918 >> v2:172.21.15.137:3300/0 conn(0x7fa53c15e3b0 0x7fa53c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 --2- 172.21.15.125:0/514631918 >> v2:172.21.15.137:3300/0 conn(0x7fa53c15e3b0 0x7fa53c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 -- 172.21.15.125:0/514631918 >> v1:172.21.15.137:6789/0 conn(0x7fa53c0cf7d0 legacy=0x7fa53c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.703+0000 7fa541400640 1 -- 172.21.15.125:0/514631918 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c15e990 con 0x7fa53c143ae0 2024-01-16T14:59:05.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.705+0000 7fa541400640 1 --2- 172.21.15.125:0/514631918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c143ae0 0x7fa53c0cf290 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06e4f0 tx=0x7fa52c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bb4f0741961706 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.705+0000 7fa53be00640 1 -- 172.21.15.125:0/514631918 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c02ca80 con 0x7fa53c143ae0 2024-01-16T14:59:05.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.705+0000 7fa53be00640 1 -- 172.21.15.125:0/514631918 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c02cbe0 con 0x7fa53c143ae0 2024-01-16T14:59:05.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.705+0000 7fa53be00640 1 -- 172.21.15.125:0/514631918 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c0343f0 con 0x7fa53c143ae0 2024-01-16T14:59:05.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 -- 172.21.15.125:0/514631918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c143ae0 msgr2=0x7fa53c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 --2- 172.21.15.125:0/514631918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c143ae0 0x7fa53c0cf290 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06e4f0 tx=0x7fa52c02bee0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 -- 172.21.15.125:0/514631918 shutdown_connections 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 --2- 172.21.15.125:0/514631918 >> v2:172.21.15.137:3300/0 conn(0x7fa53c15e3b0 0x7fa53c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 --2- 172.21.15.125:0/514631918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c143ae0 0x7fa53c0cf290 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 -- 172.21.15.125:0/514631918 >> 172.21.15.125:0/514631918 conn(0x7fa53c06a800 msgr2=0x7fa53c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 -- 172.21.15.125:0/514631918 shutdown_connections 2024-01-16T14:59:05.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.706+0000 7fa543e00640 1 -- 172.21.15.125:0/514631918 wait complete. 2024-01-16T14:59:05.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.708+0000 7fa543e00640 1 Processor -- start 2024-01-16T14:59:05.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.708+0000 7fa543e00640 1 -- start start 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa53c0cf7d0 0x7fa53c0afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 0x7fa53c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c0adaf0 con 0x7fa53c15e3b0 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c0add50 con 0x7fa53c0cf7d0 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa543e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c0ae030 con 0x7fa53c143ae0 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa541e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa541e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57760/0 (socket says 172.21.15.125:57760) 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa541e00640 1 -- 172.21.15.125:0/836162993 learned_addr learned my addr 172.21.15.125:0/836162993 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa541400640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa53c0cf7d0 0x7fa53c0afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.709+0000 7fa540a00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 0x7fa53c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 -- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 msgr2=0x7fa53c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 0x7fa53c0cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 -- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa53c0cf7d0 msgr2=0x7fa53c0afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa53c0cf7d0 0x7fa53c0afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 -- 172.21.15.125:0/836162993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c009b30 con 0x7fa53c15e3b0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa540a00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 0x7fa53c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa541e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa53400c550 tx=0x7fa53400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534004220 con 0x7fa53c15e3b0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa534004380 con 0x7fa53c15e3b0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534003830 con 0x7fa53c15e3b0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c0ae310 con 0x7fa53c15e3b0 2024-01-16T14:59:05.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.710+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c0aff40 con 0x7fa53c15e3b0 2024-01-16T14:59:05.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.712+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c06e100 con 0x7fa53c15e3b0 2024-01-16T14:59:05.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.712+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fa534003990 con 0x7fa53c15e3b0 2024-01-16T14:59:05.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.713+0000 7fa539600640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 0x7fa51406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:05.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.713+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa534089a70 con 0x7fa53c15e3b0 2024-01-16T14:59:05.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.713+0000 7fa541400640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 0x7fa51406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:05.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.714+0000 7fa541400640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 0x7fa51406d810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa52c035000 tx=0x7fa52c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:05.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.717+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa534054770 con 0x7fa53c15e3b0 2024-01-16T14:59:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.938+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa53c0ae1a0 con 0x7fa51406b350 2024-01-16T14:59:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.947+0000 7fa539600640 1 -- 172.21.15.125:0/836162993 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa53c0ae1a0 con 0x7fa51406b350 2024-01-16T14:59:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 msgr2=0x7fa51406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 0x7fa51406d810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa52c035000 tx=0x7fa52c030000 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 msgr2=0x7fa53c0cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa53400c550 tx=0x7fa53400ca10 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 shutdown_connections 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa51406b350 0x7fa51406d810 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa53c15e3b0 0x7fa53c0cd360 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa53c143ae0 0x7fa53c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 --2- 172.21.15.125:0/836162993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa53c0cf7d0 0x7fa53c0afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 >> 172.21.15.125:0/836162993 conn(0x7fa53c06a800 msgr2=0x7fa53c15bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:05.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 shutdown_connections 2024-01-16T14:59:05.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:05.950+0000 7fa543e00640 1 -- 172.21.15.125:0/836162993 wait complete. 2024-01-16T14:59:05.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:05.950 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55) 2024-01-16T14:59:06.008 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:59:06.008 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T14:59:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:06.009 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T14:59:06.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:06.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:08.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:08.011 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55) 2024-01-16T14:59:08.073 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:59:08.073 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T14:59:08.074 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:08.075 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T14:59:10.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:10.076 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55) 2024-01-16T14:59:10.139 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/5da1f25f-354f-4516-b8bf-33c7df91fd55 2024-01-16T14:59:10.139 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T14:59:10.139 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:10.140 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T14:59:10.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T14:59:10.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.378+0000 7f9a79e00640 1 Processor -- start 2024-01-16T14:59:10.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.379+0000 7f9a79e00640 1 -- start start 2024-01-16T14:59:10.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.380+0000 7f9a79e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a7413df80 0x7f9a7415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.380+0000 7f9a79e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 0x7f9a740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.380+0000 7f9a79e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7406c470 con 0x7f9a7415f9c0 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.380+0000 7f9a79e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406c750 con 0x7f9a7413df80 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.380+0000 7f9a79e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406ca30 con 0x7f9a7413e360 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a7413df80 0x7f9a7415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a7413df80 0x7f9a7415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54200/0 (socket says 172.21.15.125:54200) 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 -- 172.21.15.125:0/3938860480 learned_addr learned my addr 172.21.15.125:0/3938860480 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 -- 172.21.15.125:0/3938860480 >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 msgr2=0x7f9a740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a69e00640 1 --2- 172.21.15.125:0/3938860480 >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 0x7f9a740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 --2- 172.21.15.125:0/3938860480 >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 0x7f9a740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 -- 172.21.15.125:0/3938860480 >> v1:172.21.15.137:6789/0 conn(0x7f9a7415f9c0 legacy=0x7f9a741408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a72a00640 1 -- 172.21.15.125:0/3938860480 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7406cd10 con 0x7f9a7413df80 2024-01-16T14:59:10.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.381+0000 7f9a69e00640 1 --2- 172.21.15.125:0/3938860480 >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 0x7f9a740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:10.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.382+0000 7f9a72a00640 1 --2- 172.21.15.125:0/3938860480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7413df80 0x7f9a7415bee0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9a7406ded0 tx=0x7f9a6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b8ca904f13ea1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a72000640 1 -- 172.21.15.125:0/3938860480 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64031870 con 0x7f9a7413df80 2024-01-16T14:59:10.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a72000640 1 -- 172.21.15.125:0/3938860480 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a640319d0 con 0x7f9a7413df80 2024-01-16T14:59:10.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a72000640 1 -- 172.21.15.125:0/3938860480 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64031d60 con 0x7f9a7413df80 2024-01-16T14:59:10.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 -- 172.21.15.125:0/3938860480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7413df80 msgr2=0x7f9a7415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 --2- 172.21.15.125:0/3938860480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7413df80 0x7f9a7415bee0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9a7406ded0 tx=0x7f9a6402b910 comp rx=0 tx=0).stop 2024-01-16T14:59:10.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 -- 172.21.15.125:0/3938860480 shutdown_connections 2024-01-16T14:59:10.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 --2- 172.21.15.125:0/3938860480 >> v2:172.21.15.137:3300/0 conn(0x7f9a7413e360 0x7f9a740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 --2- 172.21.15.125:0/3938860480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7413df80 0x7f9a7415bee0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.383+0000 7f9a79e00640 1 -- 172.21.15.125:0/3938860480 >> 172.21.15.125:0/3938860480 conn(0x7f9a7406a800 msgr2=0x7f9a7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.384+0000 7f9a79e00640 1 -- 172.21.15.125:0/3938860480 shutdown_connections 2024-01-16T14:59:10.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.384+0000 7f9a79e00640 1 -- 172.21.15.125:0/3938860480 wait complete. 2024-01-16T14:59:10.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.385+0000 7f9a79e00640 1 Processor -- start 2024-01-16T14:59:10.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.385+0000 7f9a79e00640 1 -- start start 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 0x7f9a7413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 0x7f9a7415d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414cd00 con 0x7f9a7415f9c0 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414da90 con 0x7f9a7413e360 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.386+0000 7f9a79e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7415e340 con 0x7f9a7413df80 2024-01-16T14:59:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54208/0 (socket says 172.21.15.125:54208) 2024-01-16T14:59:10.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 -- 172.21.15.125:0/2156974162 learned_addr learned my addr 172.21.15.125:0/2156974162 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:10.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a72a00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 0x7f9a7413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a69e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 0x7f9a7415d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 -- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 msgr2=0x7f9a7413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 0x7f9a7413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 -- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 msgr2=0x7f9a7415d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 0x7f9a7415d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 -- 172.21.15.125:0/2156974162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640094d0 con 0x7f9a7415f9c0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a72a00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 0x7f9a7413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a69e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 0x7f9a7415d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.387+0000 7f9a73400640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c780 tx=0x7f9a6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.388+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c003d80 con 0x7f9a7415f9c0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.388+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00ce30 con 0x7f9a7415f9c0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.388+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c005290 con 0x7f9a7415f9c0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.388+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7414dce0 con 0x7f9a7415f9c0 2024-01-16T14:59:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.388+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7414e2a0 con 0x7f9a7415f9c0 2024-01-16T14:59:10.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.389+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7414e4c0 con 0x7f9a7415f9c0 2024-01-16T14:59:10.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.390+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f9a6c006b10 con 0x7f9a7415f9c0 2024-01-16T14:59:10.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.391+0000 7f9a6be00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.391+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a6c089080 con 0x7f9a7415f9c0 2024-01-16T14:59:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.391+0000 7f9a72a00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.392+0000 7f9a72a00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 0x7f9a4c06d810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a6403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.395+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a6c0bbe90 con 0x7f9a7415f9c0 2024-01-16T14:59:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.620+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a740ce850 con 0x7f9a4c06b350 2024-01-16T14:59:10.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.753+0000 7f9a6be00640 1 -- 172.21.15.125:0/2156974162 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9a64031dd0 con 0x7f9a4c06b350 2024-01-16T14:59:10.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 msgr2=0x7f9a4c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 0x7f9a4c06d810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a6403c040 comp rx=0 tx=0).stop 2024-01-16T14:59:10.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 msgr2=0x7f9a74150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c780 tx=0x7f9a6c00cc40 comp rx=0 tx=0).stop 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 shutdown_connections 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7415f9c0 0x7f9a74150410 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a7413e360 0x7f9a7415d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 --2- 172.21.15.125:0/2156974162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a7413df80 0x7f9a7413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.756+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 >> 172.21.15.125:0/2156974162 conn(0x7f9a7406a800 msgr2=0x7f9a740af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.757+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 shutdown_connections 2024-01-16T14:59:10.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.757+0000 7f9a79e00640 1 -- 172.21.15.125:0/2156974162 wait complete. 2024-01-16T14:59:10.742 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.760+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.742 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.760+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.742 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.760+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.742 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.760+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.742 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.760+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.763 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T14:59:10.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:10.764 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.781+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.764 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.781+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.764 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.781+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.764 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.781+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.765 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:10.781+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:10.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.973+0000 7fdde2200640 1 Processor -- start 2024-01-16T14:59:10.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.973+0000 7fdde2200640 1 -- start start 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde2200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdddc13cf10 0x7fdddc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 0x7fdddc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde2200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0ce440 con 0x7fdddc13fae0 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde2200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc13ff90 con 0x7fdddc13cf10 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0ce260 con 0x7fdddc0cddf0 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.974+0000 7fdde0e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdddc13cf10 0x7fdddc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdddc13cf10 0x7fdddc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54254/0 (socket says 172.21.15.125:54254) 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 -- 172.21.15.125:0/2819147687 learned_addr learned my addr 172.21.15.125:0/2819147687 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 -- 172.21.15.125:0/2819147687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 msgr2=0x7fdddc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde1800640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 0x7fdddc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 0x7fdddc0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 -- 172.21.15.125:0/2819147687 >> v1:172.21.15.137:6789/0 conn(0x7fdddc13fae0 legacy=0x7fdddc0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde0e00640 1 -- 172.21.15.125:0/2819147687 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0ce720 con 0x7fdddc13cf10 2024-01-16T14:59:10.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.975+0000 7fdde1800640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 0x7fdddc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:10.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.976+0000 7fdde0e00640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13cf10 0x7fdddc13f5a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009870 tx=0x7fddcc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56117e59d4b7f2e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdddbe00640 1 -- 172.21.15.125:0/2819147687 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc031640 con 0x7fdddc13cf10 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdddbe00640 1 -- 172.21.15.125:0/2819147687 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc0317a0 con 0x7fdddc13cf10 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdddbe00640 1 -- 172.21.15.125:0/2819147687 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc031b30 con 0x7fdddc13cf10 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 -- 172.21.15.125:0/2819147687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13cf10 msgr2=0x7fdddc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13cf10 0x7fdddc13f5a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009870 tx=0x7fddcc02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 -- 172.21.15.125:0/2819147687 shutdown_connections 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc0cddf0 0x7fdddc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 --2- 172.21.15.125:0/2819147687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13cf10 0x7fdddc13f5a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 -- 172.21.15.125:0/2819147687 >> 172.21.15.125:0/2819147687 conn(0x7fdddc06a7d0 msgr2=0x7fdddc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 -- 172.21.15.125:0/2819147687 shutdown_connections 2024-01-16T14:59:10.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.977+0000 7fdde2200640 1 -- 172.21.15.125:0/2819147687 wait complete. 2024-01-16T14:59:10.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.979+0000 7fdde2200640 1 Processor -- start 2024-01-16T14:59:10.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.979+0000 7fdde2200640 1 -- start start 2024-01-16T14:59:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdddc0cddf0 0x7fdddc15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc13cf10 0x7fdddc13dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc13e240 con 0x7fdddc13fae0 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc153650 con 0x7fdddc0cddf0 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc13e470 con 0x7fdddc13cf10 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.980+0000 7fdde1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54258/0 (socket says 172.21.15.125:54258) 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 -- 172.21.15.125:0/132672472 learned_addr learned my addr 172.21.15.125:0/132672472 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fddd3c00640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc13cf10 0x7fdddc13dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde0e00640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdddc0cddf0 0x7fdddc15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 -- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc13cf10 msgr2=0x7fdddc13dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc13cf10 0x7fdddc13dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 -- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdddc0cddf0 msgr2=0x7fdddc15cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdddc0cddf0 0x7fdddc15cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 -- 172.21.15.125:0/132672472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0094d0 con 0x7fdddc13fae0 2024-01-16T14:59:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fdde1800640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fddd400c7f0 tx=0x7fddd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.981+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4007820 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.982+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4007980 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.982+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4007d70 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.982+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc13e750 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.982+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc13ed10 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.983+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc067450 con 0x7fdddc13fae0 2024-01-16T14:59:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.984+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fddd40113f0 con 0x7fdddc13fae0 2024-01-16T14:59:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.984+0000 7fddd9600640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 0x7fddb406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.985+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddd408a150 con 0x7fdddc13fae0 2024-01-16T14:59:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.985+0000 7fdde0e00640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 0x7fddb406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.985+0000 7fdde0e00640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 0x7fddb406d810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fddcc036000 tx=0x7fddcc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:10.989+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddd4054e50 con 0x7fdddc13fae0 2024-01-16T14:59:11.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:11.219+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdddc0b2070 con 0x7fddb406b350 2024-01-16T14:59:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:12.125+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fddd4055220 con 0x7fdddc13fae0 2024-01-16T14:59:13.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.241+0000 7fddd9600640 1 -- 172.21.15.125:0/132672472 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fddcc0315e0 con 0x7fddb406b350 2024-01-16T14:59:13.223 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:13.223 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:13.223 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.224 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 msgr2=0x7fddb406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 0x7fddb406d810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fddcc036000 tx=0x7fddcc03c040 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 msgr2=0x7fdddc153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fddd400c7f0 tx=0x7fddd400ccb0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 shutdown_connections 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddb406b350 0x7fddb406d810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdddc13fae0 0x7fdddc153110 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdddc13cf10 0x7fdddc13dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 --2- 172.21.15.125:0/132672472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdddc0cddf0 0x7fdddc15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 >> 172.21.15.125:0/132672472 conn(0x7fdddc06a7d0 msgr2=0x7fdddc06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.245+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 shutdown_connections 2024-01-16T14:59:13.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.246+0000 7fdde2200640 1 -- 172.21.15.125:0/132672472 wait complete. 2024-01-16T14:59:13.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T14:59:13.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.459+0000 7fa6f2200640 1 Processor -- start 2024-01-16T14:59:13.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.460+0000 7fa6f2200640 1 -- start start 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.461+0000 7fa6f2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6ec072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.461+0000 7fa6f2200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa6ec0b2bd0 0x7fa6ec0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.461+0000 7fa6f2200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec071190 con 0x7fa6ec150760 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.461+0000 7fa6f2200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec071470 con 0x7fa6ec0b2bd0 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.461+0000 7fa6f2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec071750 con 0x7fa6ec15da10 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6eb400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa6ec150760 0x7fa6ec15d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44118/0 (socket says 172.21.15.125:44118) 2024-01-16T14:59:13.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6eb400640 1 -- 172.21.15.125:0/2575034685 learned_addr learned my addr 172.21.15.125:0/2575034685 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:13.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ebe00640 1 --2- 172.21.15.125:0/2575034685 >> v2:172.21.15.137:3300/0 conn(0x7fa6ec0b2bd0 0x7fa6ec0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6eaa00640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6ec072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861522412 0 0) 0x7fa6ec071190 con 0x7fa6ec150760 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc003520 con 0x7fa6ec150760 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974186904 0 0) 0x7fa6cc003520 con 0x7fa6ec150760 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 msgr2=0x7fa6ec072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6ec072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 >> v2:172.21.15.137:3300/0 conn(0x7fa6ec0b2bd0 msgr2=0x7fa6ec0b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 --2- 172.21.15.125:0/2575034685 >> v2:172.21.15.137:3300/0 conn(0x7fa6ec0b2bd0 0x7fa6ec0b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec071a30 con 0x7fa6ec150760 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa6dc001670 con 0x7fa6ec150760 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.462+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 >> v1:172.21.15.125:6789/0 conn(0x7fa6ec150760 legacy=0x7fa6ec15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6cc003970 0x7fa6cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6cc004d00 0x7fa6cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec071470 con 0x7fa6cc003970 2024-01-16T14:59:13.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec071750 con 0x7fa6cc004d00 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc00db90 con 0x7fa6ec15da10 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6cc003970 0x7fa6cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eaa00640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ebe00640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6cc004d00 0x7fa6cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 -- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 msgr2=0x7fa6cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ec15da10 0x7fa6cc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 -- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6cc004d00 msgr2=0x7fa6cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6cc004d00 0x7fa6cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6eb400640 1 -- 172.21.15.125:0/2575034685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc003520 con 0x7fa6cc003970 2024-01-16T14:59:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.463+0000 7fa6ebe00640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6cc004d00 0x7fa6cc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6eb400640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6cc003970 0x7fa6cc003d90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c190 tx=0x7fa6cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e09c30fada7cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc005ad0 con 0x7fa6cc003970 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc005c50 con 0x7fa6cc003970 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6ea000640 1 -- 172.21.15.125:0/2575034685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc02c810 con 0x7fa6cc003970 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6f2200640 1 -- 172.21.15.125:0/2575034685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6cc003970 msgr2=0x7fa6cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6f2200640 1 --2- 172.21.15.125:0/2575034685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6cc003970 0x7fa6cc003d90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c190 tx=0x7fa6cc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6ebe00640 1 -- 172.21.15.125:0/2575034685 reap_dead start 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6f2200640 1 -- 172.21.15.125:0/2575034685 shutdown_connections 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.465+0000 7fa6f2200640 1 -- 172.21.15.125:0/2575034685 >> 172.21.15.125:0/2575034685 conn(0x7fa6ec06a800 msgr2=0x7fa6ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.466+0000 7fa6f2200640 1 -- 172.21.15.125:0/2575034685 shutdown_connections 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.466+0000 7fa6f2200640 1 -- 172.21.15.125:0/2575034685 wait complete. 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.467+0000 7fa6f2200640 1 Processor -- start 2024-01-16T14:59:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.467+0000 7fa6f2200640 1 -- start start 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6cc003970 0x7fa6ec0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 0x7fa6ec159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1f3e70 con 0x7fa6ec0cc270 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec15a4b0 con 0x7fa6ec159ae0 2024-01-16T14:59:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.468+0000 7fa6f2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0ce0e0 con 0x7fa6cc003970 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54266/0 (socket says 172.21.15.125:54266) 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 -- 172.21.15.125:0/3245174977 learned_addr learned my addr 172.21.15.125:0/3245174977 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eb400640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6cc003970 0x7fa6ec0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6ebe00640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 0x7fa6ec159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 -- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6cc003970 msgr2=0x7fa6ec0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6cc003970 0x7fa6ec0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 -- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 msgr2=0x7fa6ec159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 0x7fa6ec159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 -- 172.21.15.125:0/3245174977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0cc790 con 0x7fa6ec0cc270 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6ebe00640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 0x7fa6ec159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.469+0000 7fa6eaa00640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40108c0 tx=0x7fa6d4010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.470+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d400eb60 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.470+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d400ecc0 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.470+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d401c400 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.470+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec0cca70 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.470+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec0cd030 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.471+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec0cd220 con 0x7fa6ec0cc270 2024-01-16T14:59:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.472+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fa6d401c560 con 0x7fa6ec0cc270 2024-01-16T14:59:13.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.473+0000 7fa6e3400640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 0x7fa6c406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.473+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6d408a4d0 con 0x7fa6ec0cc270 2024-01-16T14:59:13.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.474+0000 7fa6eb400640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 0x7fa6c406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.474+0000 7fa6eb400640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 0x7fa6c406d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc034000 tx=0x7fa6dc007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.478+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6d40550d0 con 0x7fa6ec0cc270 2024-01-16T14:59:13.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.737+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa6ec0675d0 con 0x7fa6ec0cc270 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.738+0000 7fa6e3400640 1 -- 172.21.15.125:0/3245174977 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2278 (secure 0 0 0) 0x7fa6d4021900 con 0x7fa6ec0cc270 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T14:59:13.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 172, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T14:59:13.721 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 34, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:13.722 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 793, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222027890, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6694076416, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766400036864, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 46121632, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T14:59:13.723 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 11 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 19, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 2, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 2, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "a", 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T14:59:13.724 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5238 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T14:59:13.725 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 7, 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T14:58:59.961350+0000", 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "services": { 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": { 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "daemons": { 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "summary": "", 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "a": { 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "start_epoch": 0, 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "start_stamp": "0.000000", 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 0, 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T14:59:13.726 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": {}, 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: "task_status": {} 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": {} 2024-01-16T14:59:13.727 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 msgr2=0x7fa6c406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 0x7fa6c406d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc034000 tx=0x7fa6dc007000 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 msgr2=0x7fa6ec14ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40108c0 tx=0x7fa6d4010d80 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 shutdown_connections 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ec159ae0 0x7fa6ec159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6c406b470 0x7fa6c406d930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ec0cc270 0x7fa6ec14ea70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 --2- 172.21.15.125:0/3245174977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6cc003970 0x7fa6ec0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 >> 172.21.15.125:0/3245174977 conn(0x7fa6ec06a800 msgr2=0x7fa6ec0b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 shutdown_connections 2024-01-16T14:59:13.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.741+0000 7fa6f2200640 1 -- 172.21.15.125:0/3245174977 wait complete. 2024-01-16T14:59:13.740 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T14:59:13.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.968+0000 7f474a400640 1 Processor -- start 2024-01-16T14:59:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.968+0000 7f474a400640 1 -- start start 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.969+0000 7f474a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f474415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.969+0000 7f474a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f474415e1e0 0x7f47440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.969+0000 7f474a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47440b1ee0 con 0x7f4744143ae0 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.969+0000 7f474a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440b2780 con 0x7f474415e1e0 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.969+0000 7f474a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474415e990 con 0x7f47440cf9a0 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4743400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4744143ae0 0x7f47440cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44120/0 (socket says 172.21.15.125:44120) 2024-01-16T14:59:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4743400640 1 -- 172.21.15.125:0/2467647779 learned_addr learned my addr 172.21.15.125:0/2467647779 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4743e00640 1 --2- 172.21.15.125:0/2467647779 >> v2:172.21.15.137:3300/0 conn(0x7f474415e1e0 0x7f47440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f473a800640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f474415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413148351 0 0) 0x7f47440b1ee0 con 0x7f4744143ae0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4724003520 con 0x7f4744143ae0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393060761 0 0) 0x7f4724003520 con 0x7f4744143ae0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 msgr2=0x7f474415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f474415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 >> v2:172.21.15.137:3300/0 conn(0x7f474415e1e0 msgr2=0x7f47440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.970+0000 7f4742a00640 1 --2- 172.21.15.125:0/2467647779 >> v2:172.21.15.137:3300/0 conn(0x7f474415e1e0 0x7f47440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440b2130 con 0x7f4744143ae0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f4734001670 con 0x7f4744143ae0 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 >> v1:172.21.15.125:6789/0 conn(0x7f4744143ae0 legacy=0x7f47440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f4724003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240047c0 0x7f4724004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f4724005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440b2780 con 0x7f4724003970 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474415e990 con 0x7f47240047c0 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472400db90 con 0x7f47440cf9a0 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f4724003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f473a800640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240047c0 0x7f4724004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743e00640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f4724005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 -- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 msgr2=0x7f4724005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440cf9a0 0x7f4724005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 -- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240047c0 msgr2=0x7f4724004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240047c0 0x7f4724004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f4743400640 1 -- 172.21.15.125:0/2467647779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724003520 con 0x7f4724003970 2024-01-16T14:59:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.971+0000 7f473a800640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240047c0 0x7f4724004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:13.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f4743400640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f4724003d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f473400c210 tx=0x7f47240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568fce479c21968e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473402cae0 con 0x7f4724003970 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473402cc40 con 0x7f4724003970 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f4742a00640 1 -- 172.21.15.125:0/2467647779 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47340058d0 con 0x7f4724003970 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f474a400640 1 -- 172.21.15.125:0/2467647779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 msgr2=0x7f4724003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.973+0000 7f474a400640 1 --2- 172.21.15.125:0/2467647779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f4724003d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f473400c210 tx=0x7f47240057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.974+0000 7f4743e00640 1 -- 172.21.15.125:0/2467647779 reap_dead start 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.974+0000 7f474a400640 1 -- 172.21.15.125:0/2467647779 shutdown_connections 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.974+0000 7f474a400640 1 -- 172.21.15.125:0/2467647779 >> 172.21.15.125:0/2467647779 conn(0x7f474406a800 msgr2=0x7f474406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.974+0000 7f474a400640 1 -- 172.21.15.125:0/2467647779 shutdown_connections 2024-01-16T14:59:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.974+0000 7f474a400640 1 -- 172.21.15.125:0/2467647779 wait complete. 2024-01-16T14:59:13.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.976+0000 7f474a400640 1 Processor -- start 2024-01-16T14:59:13.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.976+0000 7f474a400640 1 -- start start 2024-01-16T14:59:13.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f474413baf0 0x7f47440cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440ce540 0x7f47440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474413bf50 con 0x7f4724003970 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440ce0b0 con 0x7f474413baf0 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f474a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47440cd4a0 con 0x7f47440ce540 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54292/0 (socket says 172.21.15.125:54292) 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 -- 172.21.15.125:0/3348141610 learned_addr learned my addr 172.21.15.125:0/3348141610 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 -- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440ce540 msgr2=0x7f47440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440ce540 0x7f47440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 -- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f474413baf0 msgr2=0x7f47440cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f474413baf0 0x7f47440cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.977+0000 7f4743400640 1 -- 172.21.15.125:0/3348141610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440cd780 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f4743400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f473402c480 tx=0x7f47440b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734004bf0 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734004d50 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47440cda60 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734007270 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.978+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47440cb6d0 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.979+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47440675d0 con 0x7f4724003970 2024-01-16T14:59:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.980+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f4734032070 con 0x7f4724003970 2024-01-16T14:59:13.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.982+0000 7f473be00640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 0x7f471c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:13.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.982+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47340aae20 con 0x7f4724003970 2024-01-16T14:59:13.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.982+0000 7f473a800640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 0x7f471c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:13.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.982+0000 7f473a800640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 0x7f471c06d960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f47440713b0 tx=0x7f472c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:13.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:13.986+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47340759f0 con 0x7f4724003970 2024-01-16T14:59:14.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.217+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f474415e640 con 0x7f471c06b4a0 2024-01-16T14:59:14.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.229+0000 7f473be00640 1 -- 172.21.15.125:0/3348141610 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f472c009650 con 0x7f471c06b4a0 2024-01-16T14:59:14.210 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 0.0, 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "0 B/3 GB", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "0/3", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:14.211 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:14.212 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:14.212 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:14.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 msgr2=0x7f471c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:14.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 0x7f471c06d960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f47440713b0 tx=0x7f472c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:14.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 msgr2=0x7f47440ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:14.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f473402c480 tx=0x7f47440b26f0 comp rx=0 tx=0).stop 2024-01-16T14:59:14.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 shutdown_connections 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47440ce540 0x7f47440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f474413baf0 0x7f47440cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f471c06b4a0 0x7f471c06d960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 --2- 172.21.15.125:0/3348141610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4724003970 0x7f47440ccec0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.231+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 >> 172.21.15.125:0/3348141610 conn(0x7f474406a800 msgr2=0x7f474406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.232+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 shutdown_connections 2024-01-16T14:59:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:14.232+0000 7f474a400640 1 -- 172.21.15.125:0/3348141610 wait complete. 2024-01-16T14:59:14.231 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:15.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:15.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.495+0000 7fb546600640 1 Processor -- start 2024-01-16T14:59:15.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.495+0000 7fb546600640 1 -- start start 2024-01-16T14:59:15.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.497+0000 7fb546600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb540161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.497+0000 7fb546600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 0x7fb5400ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.497+0000 7fb546600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5401618a0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.497+0000 7fb546600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400d0de0 con 0x7fb5400ce8f0 2024-01-16T14:59:15.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.497+0000 7fb546600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400d10c0 con 0x7fb540160f80 2024-01-16T14:59:15.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.498+0000 7fb53ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 0x7fb5400ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.498+0000 7fb53f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb540161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.498+0000 7fb53ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 0x7fb5400ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:51964/0 (socket says 172.21.15.125:51964) 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.498+0000 7fb53ea00640 1 -- 172.21.15.125:0/2570711156 learned_addr learned my addr 172.21.15.125:0/2570711156 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.499+0000 7fb53f400640 1 -- 172.21.15.125:0/2570711156 >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 msgr2=0x7fb5400ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.499+0000 7fb53f400640 1 --2- 172.21.15.125:0/2570711156 >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 0x7fb5400ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.499+0000 7fb53f400640 1 -- 172.21.15.125:0/2570711156 >> v1:172.21.15.137:6789/0 conn(0x7fb5400cfbe0 legacy=0x7fb5400d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.499+0000 7fb53f400640 1 -- 172.21.15.125:0/2570711156 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400d13a0 con 0x7fb540160f80 2024-01-16T14:59:15.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.499+0000 7fb53e000640 1 -- 172.21.15.125:0/2570711156 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126946396 0 0) 0x7fb5401618a0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.500+0000 7fb53f400640 1 --2- 172.21.15.125:0/2570711156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb540161360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5300097a0 tx=0x7fb53002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2ff37dfb793c4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:15.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.501+0000 7fb53e000640 1 -- 172.21.15.125:0/2570711156 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53002d040 con 0x7fb540160f80 2024-01-16T14:59:15.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.502+0000 7fb53e000640 1 -- 172.21.15.125:0/2570711156 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530031810 con 0x7fb540160f80 2024-01-16T14:59:15.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.502+0000 7fb53e000640 1 -- 172.21.15.125:0/2570711156 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530031ba0 con 0x7fb540160f80 2024-01-16T14:59:15.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 -- 172.21.15.125:0/2570711156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 msgr2=0x7fb540161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 --2- 172.21.15.125:0/2570711156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb540161360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5300097a0 tx=0x7fb53002b8b0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 -- 172.21.15.125:0/2570711156 shutdown_connections 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 --2- 172.21.15.125:0/2570711156 >> v2:172.21.15.137:3300/0 conn(0x7fb5400ce8f0 0x7fb5400ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 --2- 172.21.15.125:0/2570711156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb540161360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 -- 172.21.15.125:0/2570711156 >> 172.21.15.125:0/2570711156 conn(0x7fb54009c550 msgr2=0x7fb5400d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.503+0000 7fb546600640 1 -- 172.21.15.125:0/2570711156 shutdown_connections 2024-01-16T14:59:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.504+0000 7fb546600640 1 -- 172.21.15.125:0/2570711156 wait complete. 2024-01-16T14:59:15.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.506+0000 7fb546600640 1 Processor -- start 2024-01-16T14:59:15.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.506+0000 7fb546600640 1 -- start start 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 0x7fb540150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb54015d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540160540 con 0x7fb5400cfbe0 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400c8ef0 con 0x7fb5400ce8f0 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.507+0000 7fb546600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400c8320 con 0x7fb540160f80 2024-01-16T14:59:15.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54298/0 (socket says 172.21.15.125:54298) 2024-01-16T14:59:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 -- 172.21.15.125:0/3744552439 learned_addr learned my addr 172.21.15.125:0/3744552439 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53fe00640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb54015d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53f400640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 0x7fb540150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 -- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 msgr2=0x7fb54015d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb54015d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.508+0000 7fb53ea00640 1 -- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 msgr2=0x7fb540150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.509+0000 7fb53ea00640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 0x7fb540150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.509+0000 7fb53ea00640 1 -- 172.21.15.125:0/3744552439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb530009400 con 0x7fb5400cfbe0 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.509+0000 7fb53fe00640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb54015d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.509+0000 7fb53f400640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 0x7fb540150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.509+0000 7fb53ea00640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e840 tx=0x7fb52800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.510+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52800c8e0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.510+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52800ca40 con 0x7fb5400cfbe0 2024-01-16T14:59:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.510+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52800cdf0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.510+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540150b80 con 0x7fb5400cfbe0 2024-01-16T14:59:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.510+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5400c88a0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.512+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5400c9080 con 0x7fb5400cfbe0 2024-01-16T14:59:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.513+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fb528013470 con 0x7fb5400cfbe0 2024-01-16T14:59:15.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.515+0000 7fb537400640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 0x7fb51806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:15.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.515+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb528089ff0 con 0x7fb5400cfbe0 2024-01-16T14:59:15.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.515+0000 7fb53f400640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 0x7fb51806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.517+0000 7fb53f400640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 0x7fb51806d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb530036000 tx=0x7fb53003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.520+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5280bce90 con 0x7fb5400cfbe0 2024-01-16T14:59:15.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.892+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5400d1c40 con 0x7fb51806b470 2024-01-16T14:59:15.894 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 0.0, 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "0 B/3 GB", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "0/3", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:15.895 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:15.896 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:15.896 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:15.896 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:15.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.910+0000 7fb537400640 1 -- 172.21.15.125:0/3744552439 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fb53003a420 con 0x7fb51806b470 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 msgr2=0x7fb51806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 0x7fb51806d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb530036000 tx=0x7fb53003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 msgr2=0x7fb5400c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e840 tx=0x7fb52800ed00 comp rx=0 tx=0).stop 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 shutdown_connections 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb540160f80 0x7fb54015d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb51806b470 0x7fb51806d930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5400cfbe0 0x7fb5400c9380 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 --2- 172.21.15.125:0/3744552439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5400ce8f0 0x7fb540150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:15.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 >> 172.21.15.125:0/3744552439 conn(0x7fb54009c550 msgr2=0x7fb5400ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:15.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.916+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 shutdown_connections 2024-01-16T14:59:15.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:15.917+0000 7fb546600640 1 -- 172.21.15.125:0/3744552439 wait complete. 2024-01-16T14:59:15.923 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:16.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.269+0000 7fca7d800640 1 Processor -- start 2024-01-16T14:59:17.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.269+0000 7fca7d800640 1 -- start start 2024-01-16T14:59:17.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.272+0000 7fca7d800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fca78162070 0x7fca78164460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:17.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.272+0000 7fca76000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fca780b1f00 0x7fca7806c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41472/0 (socket says 172.21.15.125:41472) 2024-01-16T14:59:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.273+0000 7fca76000640 1 -- 172.21.15.125:0/1572820256 learned_addr learned my addr 172.21.15.125:0/1572820256 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.273+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b3090 con 0x7fca780b1f00 2024-01-16T14:59:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.273+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78070df0 con 0x7fca781593e0 2024-01-16T14:59:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.273+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca780b42c0 con 0x7fca78162070 2024-01-16T14:59:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.274+0000 7fca77400640 1 --2- 172.21.15.125:0/1572820256 >> v2:172.21.15.125:3300/0 conn(0x7fca78162070 0x7fca78164460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.274+0000 7fca77400640 1 -- 172.21.15.125:0/1572820256 >> v1:172.21.15.137:6789/0 conn(0x7fca781593e0 legacy=0x7fca780b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.274+0000 7fca77400640 1 -- 172.21.15.125:0/1572820256 >> v1:172.21.15.125:6789/0 conn(0x7fca780b1f00 legacy=0x7fca7806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.274+0000 7fca77400640 1 -- 172.21.15.125:0/1572820256 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca780b2a80 con 0x7fca78162070 2024-01-16T14:59:17.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.276+0000 7fca77400640 1 --2- 172.21.15.125:0/1572820256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca78164460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fca70009d90 tx=0x7fca7002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c76fde15e518f99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:17.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.277+0000 7fca75600640 1 -- 172.21.15.125:0/1572820256 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7002ccf0 con 0x7fca78162070 2024-01-16T14:59:17.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.277+0000 7fca75600640 1 -- 172.21.15.125:0/1572820256 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7002ce50 con 0x7fca78162070 2024-01-16T14:59:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.278+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 msgr2=0x7fca78164460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.278+0000 7fca7d800640 1 --2- 172.21.15.125:0/1572820256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca78164460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fca70009d90 tx=0x7fca7002bf40 comp rx=0 tx=0).stop 2024-01-16T14:59:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.279+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 shutdown_connections 2024-01-16T14:59:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.279+0000 7fca7d800640 1 --2- 172.21.15.125:0/1572820256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca78164460 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.279+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 >> 172.21.15.125:0/1572820256 conn(0x7fca780b0c00 msgr2=0x7fca780b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.279+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 shutdown_connections 2024-01-16T14:59:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.280+0000 7fca7d800640 1 -- 172.21.15.125:0/1572820256 wait complete. 2024-01-16T14:59:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.283+0000 7fca7d800640 1 Processor -- start 2024-01-16T14:59:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.284+0000 7fca7d800640 1 -- start start 2024-01-16T14:59:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.287+0000 7fca7d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.288+0000 7fca7d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca781593e0 0x7fca7815b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.288+0000 7fca7d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca7815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.289+0000 7fca7d800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca781514e0 con 0x7fca78162070 2024-01-16T14:59:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.289+0000 7fca7d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780cd970 con 0x7fca780b1f00 2024-01-16T14:59:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.289+0000 7fca7d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca780cdc50 con 0x7fca781593e0 2024-01-16T14:59:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.290+0000 7fca76a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.290+0000 7fca76000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca781593e0 0x7fca7815b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.290+0000 7fca76a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:53478/0 (socket says 172.21.15.125:53478) 2024-01-16T14:59:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.290+0000 7fca76a00640 1 -- 172.21.15.125:0/1575611807 learned_addr learned my addr 172.21.15.125:0/1575611807 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca77400640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca7815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca76a00640 1 -- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca781593e0 msgr2=0x7fca7815b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca76a00640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca781593e0 0x7fca7815b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca76a00640 1 -- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 msgr2=0x7fca7815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca76a00640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca7815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.291+0000 7fca76a00640 1 -- 172.21.15.125:0/1575611807 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca70009a70 con 0x7fca780b1f00 2024-01-16T14:59:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.293+0000 7fca76a00640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c400 tx=0x7fca6800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:17.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.294+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680052d0 con 0x7fca780b1f00 2024-01-16T14:59:17.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.295+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca68005450 con 0x7fca780b1f00 2024-01-16T14:59:17.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.295+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68007260 con 0x7fca780b1f00 2024-01-16T14:59:17.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.295+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca780cdf30 con 0x7fca780b1f00 2024-01-16T14:59:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.295+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca780ce4f0 con 0x7fca780b1f00 2024-01-16T14:59:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.296+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca78070410 con 0x7fca780b1f00 2024-01-16T14:59:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.300+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fca680073c0 con 0x7fca780b1f00 2024-01-16T14:59:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.300+0000 7fca6ea00640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 0x7fca5006d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.300+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca68089a60 con 0x7fca780b1f00 2024-01-16T14:59:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.301+0000 7fca76000640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 0x7fca5006d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.301+0000 7fca76000640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 0x7fca5006d9b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fca7809cd90 tx=0x7fca60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:17.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.313+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca68054790 con 0x7fca780b1f00 2024-01-16T14:59:17.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.628+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fca780b2d20 con 0x7fca5006b4f0 2024-01-16T14:59:17.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.636+0000 7fca6ea00640 1 -- 172.21.15.125:0/1575611807 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fca60001580 con 0x7fca5006b4f0 2024-01-16T14:59:17.618 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:17.618 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:17.618 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:17.618 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:17.619 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:17.620 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:17.620 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.638+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 msgr2=0x7fca5006d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.638+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 0x7fca5006d9b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fca7809cd90 tx=0x7fca60009040 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 msgr2=0x7fca78150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c400 tx=0x7fca6800c8c0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 shutdown_connections 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca5006b4f0 0x7fca5006d9b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca78162070 0x7fca7815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca781593e0 0x7fca7815b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 --2- 172.21.15.125:0/1575611807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca780b1f00 0x7fca78150fa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 >> 172.21.15.125:0/1575611807 conn(0x7fca780b0c00 msgr2=0x7fca7815ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 shutdown_connections 2024-01-16T14:59:17.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:17.639+0000 7fca7d800640 1 -- 172.21.15.125:0/1575611807 wait complete. 2024-01-16T14:59:17.637 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:18.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:18.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.878+0000 7fbeca200640 1 Processor -- start 2024-01-16T14:59:18.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.878+0000 7fbeca200640 1 -- start start 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbeca200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec40cd240 0x7fbec40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbeca200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbec406c6b0 0x7fbec4099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbeca200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec406cea0 con 0x7fbec4099fa0 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbeca200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec409a430 con 0x7fbec406c6b0 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbeca200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec406cbc0 con 0x7fbec40cd240 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbec2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbec406c6b0 0x7fbec4099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbec2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbec406c6b0 0x7fbec4099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53548/0 (socket says 172.21.15.125:53548) 2024-01-16T14:59:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.880+0000 7fbec2a00640 1 -- 172.21.15.125:0/160002436 learned_addr learned my addr 172.21.15.125:0/160002436 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.881+0000 7fbec3400640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec40cd240 0x7fbec40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.881+0000 7fbec2a00640 1 -- 172.21.15.125:0/160002436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec40cd240 msgr2=0x7fbec40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.881+0000 7fbec2a00640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec40cd240 0x7fbec40b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.881+0000 7fbec2a00640 1 -- 172.21.15.125:0/160002436 >> v1:172.21.15.137:6789/0 conn(0x7fbec4099fa0 legacy=0x7fbec406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.881+0000 7fbec2a00640 1 -- 172.21.15.125:0/160002436 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec406f350 con 0x7fbec406c6b0 2024-01-16T14:59:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.882+0000 7fbec2a00640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec406c6b0 0x7fbec4099a60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009e50 tx=0x7fbeac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a626b8d7d7de2b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.882+0000 7fbec2000640 1 -- 172.21.15.125:0/160002436 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac02ca70 con 0x7fbec406c6b0 2024-01-16T14:59:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.882+0000 7fbec2000640 1 -- 172.21.15.125:0/160002436 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac02cbd0 con 0x7fbec406c6b0 2024-01-16T14:59:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.882+0000 7fbec2000640 1 -- 172.21.15.125:0/160002436 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0343f0 con 0x7fbec406c6b0 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 -- 172.21.15.125:0/160002436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec406c6b0 msgr2=0x7fbec4099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec406c6b0 0x7fbec4099a60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009e50 tx=0x7fbeac02bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 -- 172.21.15.125:0/160002436 shutdown_connections 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec406c6b0 0x7fbec4099a60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 --2- 172.21.15.125:0/160002436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec40cd240 0x7fbec40b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 -- 172.21.15.125:0/160002436 >> 172.21.15.125:0/160002436 conn(0x7fbec40b2d00 msgr2=0x7fbec40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 -- 172.21.15.125:0/160002436 shutdown_connections 2024-01-16T14:59:18.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.883+0000 7fbeca200640 1 -- 172.21.15.125:0/160002436 wait complete. 2024-01-16T14:59:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.885+0000 7fbeca200640 1 Processor -- start 2024-01-16T14:59:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.885+0000 7fbeca200640 1 -- start start 2024-01-16T14:59:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec406c6b0 0x7fbec414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbec40cd240 0x7fbec414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec415d470 con 0x7fbec4099fa0 2024-01-16T14:59:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec415aae0 con 0x7fbec40cd240 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbeca200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec415cb10 con 0x7fbec406c6b0 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53564/0 (socket says 172.21.15.125:53564) 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 -- 172.21.15.125:0/870422985 learned_addr learned my addr 172.21.15.125:0/870422985 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 -- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec406c6b0 msgr2=0x7fbec414ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec406c6b0 0x7fbec414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec3e00640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbec40cd240 0x7fbec414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 -- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbec40cd240 msgr2=0x7fbec414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbec40cd240 0x7fbec414b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.886+0000 7fbec2a00640 1 -- 172.21.15.125:0/870422985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac009b30 con 0x7fbec4099fa0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbec2a00640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000f80 tx=0x7fbeac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac03e450 con 0x7fbec4099fa0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac03e5d0 con 0x7fbec4099fa0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec414b360 con 0x7fbec4099fa0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac03e980 con 0x7fbec4099fa0 2024-01-16T14:59:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.887+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec415d090 con 0x7fbec4099fa0 2024-01-16T14:59:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.888+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec4073dc0 con 0x7fbec4099fa0 2024-01-16T14:59:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.889+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbeac03eae0 con 0x7fbec4099fa0 2024-01-16T14:59:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.890+0000 7fbebb400640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 0x7fbe9c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.890+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbeac0acdf0 con 0x7fbec4099fa0 2024-01-16T14:59:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.890+0000 7fbec3400640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 0x7fbe9c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.891+0000 7fbec3400640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 0x7fbe9c06d930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006fd0 tx=0x7fbeb4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:18.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:18.897+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbec4073dc0 con 0x7fbec4099fa0 2024-01-16T14:59:19.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.127+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbec40cd620 con 0x7fbe9c06b470 2024-01-16T14:59:19.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.139+0000 7fbebb400640 1 -- 172.21.15.125:0/870422985 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fbeb400ae20 con 0x7fbe9c06b470 2024-01-16T14:59:19.121 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:19.121 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:19.121 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:19.122 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:19.122 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:19.122 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:19.122 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:19.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:19.123 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:19.124 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:19.124 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:19.124 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:19.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 msgr2=0x7fbe9c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 0x7fbe9c06d930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006fd0 tx=0x7fbeb4008040 comp rx=0 tx=0).stop 2024-01-16T14:59:19.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 msgr2=0x7fbec414e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000f80 tx=0x7fbeac004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:19.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 shutdown_connections 2024-01-16T14:59:19.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbec40cd240 0x7fbec414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe9c06b470 0x7fbe9c06d930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec4099fa0 0x7fbec414e0b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 --2- 172.21.15.125:0/870422985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbec406c6b0 0x7fbec414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 >> 172.21.15.125:0/870422985 conn(0x7fbec40b2d00 msgr2=0x7fbec40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:19.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 shutdown_connections 2024-01-16T14:59:19.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:19.142+0000 7fbeca200640 1 -- 172.21.15.125:0/870422985 wait complete. 2024-01-16T14:59:19.141 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:20.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.502+0000 7fb191800640 1 Processor -- start 2024-01-16T14:59:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.503+0000 7fb191800640 1 -- start start 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb191800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb18c0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb191800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb18c0d14f0 0x7fb18c161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb191800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c146b10 con 0x7fb18c0d1090 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb191800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18c0cb6c0 con 0x7fb18c0d14f0 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb191800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c0c6860 con 0x7fb18c146730 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb18a000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb18c0d1090 0x7fb18c161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41474/0 (socket says 172.21.15.125:41474) 2024-01-16T14:59:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.504+0000 7fb18a000640 1 -- 172.21.15.125:0/4186466579 learned_addr learned my addr 172.21.15.125:0/4186466579 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.505+0000 7fb18b400640 1 --2- 172.21.15.125:0/4186466579 >> v2:172.21.15.137:3300/0 conn(0x7fb18c0d14f0 0x7fb18c161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.505+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093643544 0 0) 0x7fb18c146b10 con 0x7fb18c0d1090 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.505+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb18c0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.505+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb16c003520 con 0x7fb18c0d1090 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231674371 0 0) 0x7fb16c003520 con 0x7fb18c0d1090 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 msgr2=0x7fb18c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb18c0ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 >> v2:172.21.15.137:3300/0 conn(0x7fb18c0d14f0 msgr2=0x7fb18c161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 --2- 172.21.15.125:0/4186466579 >> v2:172.21.15.137:3300/0 conn(0x7fb18c0d14f0 0x7fb18c161770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0c5fc0 con 0x7fb18c0d1090 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb174001670 con 0x7fb18c0d1090 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 >> v1:172.21.15.125:6789/0 conn(0x7fb18c0d1090 legacy=0x7fb18c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb16c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb16c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb16c004d00 0x7fb16c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c00b980 con 0x7fb16c003970 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c00bac0 con 0x7fb16c004d00 2024-01-16T14:59:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.506+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c00bc00 con 0x7fb18c146730 2024-01-16T14:59:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18a000640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb16c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18b400640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb16c004d00 0x7fb16c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb16c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 -- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 msgr2=0x7fb16c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c146730 0x7fb16c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 -- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb16c004d00 msgr2=0x7fb16c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb16c004d00 0x7fb16c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.507+0000 7fb18aa00640 1 -- 172.21.15.125:0/4186466579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c003520 con 0x7fb16c003970 2024-01-16T14:59:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.510+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb16c003d90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009fc0 tx=0x7fb16c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee2789d5130ddf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.510+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c02cc00 con 0x7fb16c003970 2024-01-16T14:59:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.511+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c00f040 con 0x7fb16c003970 2024-01-16T14:59:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.511+0000 7fb189600640 1 -- 172.21.15.125:0/4186466579 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c034440 con 0x7fb16c003970 2024-01-16T14:59:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 -- 172.21.15.125:0/4186466579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 msgr2=0x7fb16c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 --2- 172.21.15.125:0/4186466579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb16c003d90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009fc0 tx=0x7fb16c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb18b400640 1 -- 172.21.15.125:0/4186466579 reap_dead start 2024-01-16T14:59:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 -- 172.21.15.125:0/4186466579 shutdown_connections 2024-01-16T14:59:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 -- 172.21.15.125:0/4186466579 >> 172.21.15.125:0/4186466579 conn(0x7fb18c066690 msgr2=0x7fb18c06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 -- 172.21.15.125:0/4186466579 shutdown_connections 2024-01-16T14:59:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.512+0000 7fb191800640 1 -- 172.21.15.125:0/4186466579 wait complete. 2024-01-16T14:59:20.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.517+0000 7fb191800640 1 Processor -- start 2024-01-16T14:59:20.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.520+0000 7fb191800640 1 -- start start 2024-01-16T14:59:20.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c0c8300 0x7fb18c1e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 0x7fb18c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c1e7480 con 0x7fb16c003970 2024-01-16T14:59:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c1e63e0 con 0x7fb18c1f6150 2024-01-16T14:59:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.522+0000 7fb191800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c1e66c0 con 0x7fb18c0c8300 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c0c8300 0x7fb18c1e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53586/0 (socket says 172.21.15.125:53586) 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 -- 172.21.15.125:0/4283792372 learned_addr learned my addr 172.21.15.125:0/4283792372 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18b400640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 0x7fb18c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 -- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c0c8300 msgr2=0x7fb18c1e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c0c8300 0x7fb18c1e5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 -- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 msgr2=0x7fb18c1f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.523+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 0x7fb18c1f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.524+0000 7fb18aa00640 1 -- 172.21.15.125:0/4283792372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c1e69a0 con 0x7fb16c003970 2024-01-16T14:59:20.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.524+0000 7fb18b400640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 0x7fb18c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:20.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.524+0000 7fb18aa00640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb17c02dfd0 tx=0x7fb17c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.525+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c034dd0 con 0x7fb16c003970 2024-01-16T14:59:20.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.525+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c1e6c80 con 0x7fb16c003970 2024-01-16T14:59:20.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.525+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c003990 con 0x7fb16c003970 2024-01-16T14:59:20.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.525+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c034dd0 con 0x7fb16c003970 2024-01-16T14:59:20.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.526+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c1fad50 con 0x7fb16c003970 2024-01-16T14:59:20.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.533+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c074a50 con 0x7fb16c003970 2024-01-16T14:59:20.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.536+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fb17c03b050 con 0x7fb16c003970 2024-01-16T14:59:20.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.539+0000 7fb182a00640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.540+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb17c0abb10 con 0x7fb16c003970 2024-01-16T14:59:20.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.547+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb17c0767c0 con 0x7fb16c003970 2024-01-16T14:59:20.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.551+0000 7fb18a000640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.552+0000 7fb18a000640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 0x7fb16406d9c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0ceb90 tx=0x7fb174009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.973+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb18c0b36f0 con 0x7fb16406b500 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.981+0000 7fb182a00640 1 -- 172.21.15.125:0/4283792372 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb174007770 con 0x7fb16406b500 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:20.963 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:20.964 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:20.964 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 msgr2=0x7fb16406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 0x7fb16406d9c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0ceb90 tx=0x7fb174009040 comp rx=0 tx=0).stop 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 msgr2=0x7fb18c1fa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb17c02dfd0 tx=0x7fb17c009750 comp rx=0 tx=0).stop 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 shutdown_connections 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18c1f6150 0x7fb18c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb18c0c8300 0x7fb18c1e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 --2- 172.21.15.125:0/4283792372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb16c003970 0x7fb18c1fa810 secure :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb17c02dfd0 tx=0x7fb17c009750 comp rx=0 tx=0).stop 2024-01-16T14:59:20.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.985+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 >> 172.21.15.125:0/4283792372 conn(0x7fb18c066690 msgr2=0x7fb18c1e7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.992+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 shutdown_connections 2024-01-16T14:59:20.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:20.994+0000 7fb191800640 1 -- 172.21.15.125:0/4283792372 wait complete. 2024-01-16T14:59:21.004 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:22.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:22.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.242+0000 7f8e0aa00640 1 Processor -- start 2024-01-16T14:59:22.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.242+0000 7f8e0aa00640 1 -- start start 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.243+0000 7f8e0aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e0413df80 0x7f8e0415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.243+0000 7f8e0aa00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0406c470 con 0x7f8e0413e360 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.243+0000 7f8e0aa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0406c750 con 0x7f8e040cf9a0 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.243+0000 7f8e0aa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0406ca30 con 0x7f8e0413df80 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.243+0000 7f8dfb200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8e0413e360 0x7f8e040cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41480/0 (socket says 172.21.15.125:41480) 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8dfb200640 1 -- 172.21.15.125:0/605478453 learned_addr learned my addr 172.21.15.125:0/605478453 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03e00640 1 --2- 172.21.15.125:0/605478453 >> v2:172.21.15.125:3300/0 conn(0x7f8e0413df80 0x7f8e0415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03400640 1 -- 172.21.15.125:0/605478453 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404172440 0 0) 0x7f8e0406c470 con 0x7f8e0413e360 2024-01-16T14:59:22.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03400640 1 -- 172.21.15.125:0/605478453 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4003520 con 0x7f8e0413e360 2024-01-16T14:59:22.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03e00640 1 -- 172.21.15.125:0/605478453 >> v1:172.21.15.137:6789/0 conn(0x7f8e040cf9a0 legacy=0x7f8e040cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:22.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03e00640 1 -- 172.21.15.125:0/605478453 >> v1:172.21.15.125:6789/0 conn(0x7f8e0413e360 legacy=0x7f8e040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.244+0000 7f8e03e00640 1 -- 172.21.15.125:0/605478453 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0406cd10 con 0x7f8e0413df80 2024-01-16T14:59:22.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.245+0000 7f8e03e00640 1 --2- 172.21.15.125:0/605478453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413df80 0x7f8e0415bee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8df40097a0 tx=0x7f8df402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cd79945aa308a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e03400640 1 -- 172.21.15.125:0/605478453 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4031830 con 0x7f8e0413df80 2024-01-16T14:59:22.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e03400640 1 -- 172.21.15.125:0/605478453 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df4031990 con 0x7f8e0413df80 2024-01-16T14:59:22.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e03400640 1 -- 172.21.15.125:0/605478453 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4031d20 con 0x7f8e0413df80 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 -- 172.21.15.125:0/605478453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413df80 msgr2=0x7f8e0415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/605478453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413df80 0x7f8e0415bee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8df40097a0 tx=0x7f8df402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 -- 172.21.15.125:0/605478453 shutdown_connections 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/605478453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413df80 0x7f8e0415bee0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 -- 172.21.15.125:0/605478453 >> 172.21.15.125:0/605478453 conn(0x7f8e0406a800 msgr2=0x7f8e0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 -- 172.21.15.125:0/605478453 shutdown_connections 2024-01-16T14:59:22.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.246+0000 7f8e0aa00640 1 -- 172.21.15.125:0/605478453 wait complete. 2024-01-16T14:59:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.248+0000 7f8e0aa00640 1 Processor -- start 2024-01-16T14:59:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.248+0000 7f8e0aa00640 1 -- start start 2024-01-16T14:59:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e040cf9a0 0x7f8e0413a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0413df80 0x7f8e041509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0414b3c0 con 0x7f8e0413e360 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0414bf00 con 0x7f8e0413df80 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e0aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0414b5d0 con 0x7f8e040cf9a0 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e08a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e08a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53610/0 (socket says 172.21.15.125:53610) 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.249+0000 7f8e08a00640 1 -- 172.21.15.125:0/11085450 learned_addr learned my addr 172.21.15.125:0/11085450 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e03e00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e040cf9a0 0x7f8e0413a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 -- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e040cf9a0 msgr2=0x7f8e0413a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8dfb200640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0413df80 0x7f8e041509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e040cf9a0 0x7f8e0413a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 -- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0413df80 msgr2=0x7f8e041509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0413df80 0x7f8e041509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 -- 172.21.15.125:0/11085450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4009400 con 0x7f8e0413e360 2024-01-16T14:59:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e08a00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b9e0 tx=0x7f8dfc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc003b70 con 0x7f8e0413e360 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc003cd0 con 0x7f8e0413e360 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00cb60 con 0x7f8e0413e360 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0414b8b0 con 0x7f8e0413e360 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.250+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e04150f20 con 0x7f8e0413e360 2024-01-16T14:59:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.252+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e04151140 con 0x7f8e0413e360 2024-01-16T14:59:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.252+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f8dfc00ccc0 con 0x7f8e0413e360 2024-01-16T14:59:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.253+0000 7f8e00c00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 0x7f8ddc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.253+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dfc089c50 con 0x7f8e0413e360 2024-01-16T14:59:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.254+0000 7f8e03e00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 0x7f8ddc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.254+0000 7f8e03e00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 0x7f8ddc06d930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8df4036000 tx=0x7f8df4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.258+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dfc054830 con 0x7f8e0413e360 2024-01-16T14:59:22.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.467+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e0415f9c0 con 0x7f8ddc06b470 2024-01-16T14:59:22.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.480+0000 7f8e00c00640 1 -- 172.21.15.125:0/11085450 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f8df4031630 con 0x7f8ddc06b470 2024-01-16T14:59:22.461 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:22.461 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:22.461 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:22.461 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 66.66666666666666, 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "2/3", 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:22.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:22.463 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 msgr2=0x7f8ddc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 0x7f8ddc06d930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8df4036000 tx=0x7f8df4008030 comp rx=0 tx=0).stop 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 msgr2=0x7f8e0414e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b9e0 tx=0x7f8dfc00bea0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 shutdown_connections 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ddc06b470 0x7f8ddc06d930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e0413e360 0x7f8e0414e620 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0413df80 0x7f8e041509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 --2- 172.21.15.125:0/11085450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e040cf9a0 0x7f8e0413a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 >> 172.21.15.125:0/11085450 conn(0x7f8e0406a800 msgr2=0x7f8e0415c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 shutdown_connections 2024-01-16T14:59:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:22.482+0000 7f8e0aa00640 1 -- 172.21.15.125:0/11085450 wait complete. 2024-01-16T14:59:22.480 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:23.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:23.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.718+0000 7fe3e2400640 1 Processor -- start 2024-01-16T14:59:23.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.718+0000 7fe3e2400640 1 -- start start 2024-01-16T14:59:23.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3e2400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe3dc0d3910 0x7fe3dc0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3e2400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0d4160 con 0x7fe3dc160e10 2024-01-16T14:59:23.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3e2400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0b2420 con 0x7fe3dc0cfcf0 2024-01-16T14:59:23.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3e2400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc0ceff0 con 0x7fe3dc0d3910 2024-01-16T14:59:23.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3dbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe3dc160e10 0x7fe3dc0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41492/0 (socket says 172.21.15.125:41492) 2024-01-16T14:59:23.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3dbe00640 1 -- 172.21.15.125:0/1245844292 learned_addr learned my addr 172.21.15.125:0/1245844292 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.721+0000 7fe3daa00640 1 --2- 172.21.15.125:0/1245844292 >> v2:172.21.15.137:3300/0 conn(0x7fe3dc0d3910 0x7fe3dc0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030215900 0 0) 0x7fe3dc0d4160 con 0x7fe3dc160e10 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc003520 con 0x7fe3dc160e10 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3daa00640 1 -- 172.21.15.125:0/1245844292 >> v1:172.21.15.137:6789/0 conn(0x7fe3dc0cfcf0 legacy=0x7fe3dc160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3daa00640 1 -- 172.21.15.125:0/1245844292 >> v1:172.21.15.125:6789/0 conn(0x7fe3dc160e10 legacy=0x7fe3dc0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1206360845 0 0) 0x7fe3bc003520 con 0x7fe3dc160e10 2024-01-16T14:59:23.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.722+0000 7fe3daa00640 1 -- 172.21.15.125:0/1245844292 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0b2140 con 0x7fe3dc0d3910 2024-01-16T14:59:23.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.723+0000 7fe3daa00640 1 --2- 172.21.15.125:0/1245844292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0d1100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40097a0 tx=0x7fe3c402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa6cceeaf190b45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.724+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d40019f0 con 0x7fe3dc0d3910 2024-01-16T14:59:23.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.724+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c402e050 con 0x7fe3dc0d3910 2024-01-16T14:59:23.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.724+0000 7fe3da000640 1 -- 172.21.15.125:0/1245844292 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4030730 con 0x7fe3dc0d3910 2024-01-16T14:59:23.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 -- 172.21.15.125:0/1245844292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0d3910 msgr2=0x7fe3dc0d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 --2- 172.21.15.125:0/1245844292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0d1100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40097a0 tx=0x7fe3c402b790 comp rx=0 tx=0).stop 2024-01-16T14:59:23.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 -- 172.21.15.125:0/1245844292 shutdown_connections 2024-01-16T14:59:23.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 --2- 172.21.15.125:0/1245844292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0d1100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 -- 172.21.15.125:0/1245844292 >> 172.21.15.125:0/1245844292 conn(0x7fe3dc09c530 msgr2=0x7fe3dc0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 -- 172.21.15.125:0/1245844292 shutdown_connections 2024-01-16T14:59:23.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.725+0000 7fe3e2400640 1 -- 172.21.15.125:0/1245844292 wait complete. 2024-01-16T14:59:23.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.727+0000 7fe3e2400640 1 Processor -- start 2024-01-16T14:59:23.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.727+0000 7fe3e2400640 1 -- start start 2024-01-16T14:59:23.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0cfcf0 0x7fe3dc154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3dc160e10 0x7fe3dc15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc1f4300 con 0x7fe3dc0d3910 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0b8300 con 0x7fe3dc0cfcf0 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3e2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0b79d0 con 0x7fe3dc160e10 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3daa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3daa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53624/0 (socket says 172.21.15.125:53624) 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3daa00640 1 -- 172.21.15.125:0/787884910 learned_addr learned my addr 172.21.15.125:0/787884910 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:23.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.729+0000 7fe3db400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0cfcf0 0x7fe3dc154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3dbe00640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3dc160e10 0x7fe3dc15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 -- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3dc160e10 msgr2=0x7fe3dc15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3dc160e10 0x7fe3dc15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 -- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0cfcf0 msgr2=0x7fe3dc154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0cfcf0 0x7fe3dc154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 -- 172.21.15.125:0/787884910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4009400 con 0x7fe3dc0d3910 2024-01-16T14:59:23.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.730+0000 7fe3daa00640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe3c402bda0 tx=0x7fe3c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.731+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4004230 con 0x7fe3dc0d3910 2024-01-16T14:59:23.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.731+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4004390 con 0x7fe3dc0d3910 2024-01-16T14:59:23.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.731+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c403d8d0 con 0x7fe3dc0d3910 2024-01-16T14:59:23.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.731+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc0b7cb0 con 0x7fe3dc0d3910 2024-01-16T14:59:23.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.732+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc154b60 con 0x7fe3dc0d3910 2024-01-16T14:59:23.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.733+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc154d20 con 0x7fe3dc0d3910 2024-01-16T14:59:23.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.736+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fe3c4039040 con 0x7fe3dc0d3910 2024-01-16T14:59:23.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.737+0000 7fe3d3400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 0x7fe3b406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.737+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3c40ac190 con 0x7fe3dc0d3910 2024-01-16T14:59:23.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.737+0000 7fe3db400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 0x7fe3b406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.738+0000 7fe3db400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 0x7fe3b406d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000f60 tx=0x7fe3cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:23.743+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3c4076d50 con 0x7fe3dc0d3910 2024-01-16T14:59:24.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.111+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3dc0cec10 con 0x7fe3b406b470 2024-01-16T14:59:24.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.130+0000 7fe3d3400640 1 -- 172.21.15.125:0/787884910 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fe3cc001360 con 0x7fe3b406b470 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T14:59:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 66.66666666666666, 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "2/3", 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:24.113 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:24.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 msgr2=0x7fe3b406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 0x7fe3b406d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000f60 tx=0x7fe3cc009040 comp rx=0 tx=0).stop 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 msgr2=0x7fe3dc0b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe3c402bda0 tx=0x7fe3c40046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 shutdown_connections 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3dc160e10 0x7fe3dc15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe3b406b470 0x7fe3b406d930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc0d3910 0x7fe3dc0b8790 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 --2- 172.21.15.125:0/787884910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3dc0cfcf0 0x7fe3dc154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.137+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 >> 172.21.15.125:0/787884910 conn(0x7fe3dc09c530 msgr2=0x7fe3dc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:24.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.138+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 shutdown_connections 2024-01-16T14:59:24.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:24.138+0000 7fe3e2400640 1 -- 172.21.15.125:0/787884910 wait complete. 2024-01-16T14:59:24.152 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:25.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.498+0000 7f56ec000640 1 Processor -- start 2024-01-16T14:59:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.498+0000 7f56ec000640 1 -- start start 2024-01-16T14:59:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f56e413df80 0x7f56e415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f56e415f9c0 0x7f56e40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e406c470 con 0x7f56e415f9c0 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e406c750 con 0x7f56e413df80 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.500+0000 7f56ec000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e406ca30 con 0x7f56e413e360 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56e8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f56e415f9c0 0x7f56e40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56e8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56994/0 (socket says 172.21.15.125:56994) 2024-01-16T14:59:25.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56e8c00640 1 -- 172.21.15.125:0/3979323864 learned_addr learned my addr 172.21.15.125:0/3979323864 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56e9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f56e413df80 0x7f56e415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 -- 172.21.15.125:0/3979323864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 msgr2=0x7f56e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 -- 172.21.15.125:0/3979323864 >> v2:172.21.15.137:3300/0 conn(0x7f56e413df80 msgr2=0x7f56e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 --2- 172.21.15.125:0/3979323864 >> v2:172.21.15.137:3300/0 conn(0x7f56e413df80 0x7f56e415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.501+0000 7f56ea000640 1 -- 172.21.15.125:0/3979323864 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e406cd10 con 0x7f56e415f9c0 2024-01-16T14:59:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.502+0000 7f56e8c00640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.503+0000 7f56ea000640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e415f9c0 0x7f56e40cf7d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f56e4072770 tx=0x7f56dc02c0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca07d9c26ae336a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.503+0000 7f56e3e00640 1 -- 172.21.15.125:0/3979323864 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc02ce60 con 0x7f56e415f9c0 2024-01-16T14:59:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.503+0000 7f56e3e00640 1 -- 172.21.15.125:0/3979323864 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc00d0d0 con 0x7f56e415f9c0 2024-01-16T14:59:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.504+0000 7f56e3e00640 1 -- 172.21.15.125:0/3979323864 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc00d480 con 0x7f56e415f9c0 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.504+0000 7f56ec000640 1 -- 172.21.15.125:0/3979323864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e415f9c0 msgr2=0x7f56e40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.504+0000 7f56ec000640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e415f9c0 0x7f56e40cf7d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f56e4072770 tx=0x7f56dc02c0b0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 -- 172.21.15.125:0/3979323864 shutdown_connections 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e415f9c0 0x7f56e40cf7d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 --2- 172.21.15.125:0/3979323864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e413e360 0x7f56e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 --2- 172.21.15.125:0/3979323864 >> v2:172.21.15.137:3300/0 conn(0x7f56e413df80 0x7f56e415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 -- 172.21.15.125:0/3979323864 >> 172.21.15.125:0/3979323864 conn(0x7f56e406a800 msgr2=0x7f56e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 -- 172.21.15.125:0/3979323864 shutdown_connections 2024-01-16T14:59:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.505+0000 7f56ec000640 1 -- 172.21.15.125:0/3979323864 wait complete. 2024-01-16T14:59:25.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.507+0000 7f56ec000640 1 Processor -- start 2024-01-16T14:59:25.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.508+0000 7f56ec000640 1 -- start start 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e413df80 0x7f56e41ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 0x7f56e4070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4159470 con 0x7f56e413e360 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41567d0 con 0x7f56e413df80 2024-01-16T14:59:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.509+0000 7f56ec000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4156ab0 con 0x7f56e415f9c0 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56e8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56ea000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 0x7f56e4070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56e8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53648/0 (socket says 172.21.15.125:53648) 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56e9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e413df80 0x7f56e41ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56e8c00640 1 -- 172.21.15.125:0/2858863402 learned_addr learned my addr 172.21.15.125:0/2858863402 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56ea000640 1 -- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e413df80 msgr2=0x7f56e41ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56ea000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e413df80 0x7f56e41ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.510+0000 7f56ea000640 1 -- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 msgr2=0x7f56e4155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.511+0000 7f56ea000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.511+0000 7f56ea000640 1 -- 172.21.15.125:0/2858863402 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0094d0 con 0x7f56e415f9c0 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.511+0000 7f56e8c00640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.511+0000 7f56ea000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 0x7f56e4070c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56dc032000 tx=0x7f56dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.512+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc00b740 con 0x7f56e415f9c0 2024-01-16T14:59:25.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.512+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc00b8a0 con 0x7f56e415f9c0 2024-01-16T14:59:25.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.512+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc00bc50 con 0x7f56e415f9c0 2024-01-16T14:59:25.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.512+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc008260 con 0x7f56e415f9c0 2024-01-16T14:59:25.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.513+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e4157070 con 0x7f56e415f9c0 2024-01-16T14:59:25.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.514+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e40675d0 con 0x7f56e415f9c0 2024-01-16T14:59:25.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.515+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f56dc004040 con 0x7f56e415f9c0 2024-01-16T14:59:25.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.516+0000 7f56e1600640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 0x7f56bc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:25.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.516+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56dc0ac990 con 0x7f56e415f9c0 2024-01-16T14:59:25.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.516+0000 7f56e9600640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 0x7f56bc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:25.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.518+0000 7f56e9600640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 0x7f56bc06d930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f56d4011fd0 tx=0x7f56d4013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:25.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.527+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56dc077750 con 0x7f56e415f9c0 2024-01-16T14:59:25.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:25.811+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f56e41408d0 con 0x7f56bc06b470 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.762+0000 7f56e1600640 1 -- 172.21.15.125:0/2858863402 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f56d4017040 con 0x7f56bc06b470 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:26.743 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 msgr2=0x7f56bc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 0x7f56bc06d930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f56d4011fd0 tx=0x7f56d4013040 comp rx=0 tx=0).stop 2024-01-16T14:59:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 msgr2=0x7f56e4070c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 0x7f56e4070c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56dc032000 tx=0x7f56dc004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 shutdown_connections 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56e415f9c0 0x7f56e4070c00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f56bc06b470 0x7f56bc06d930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56e413e360 0x7f56e4155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 --2- 172.21.15.125:0/2858863402 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56e413df80 0x7f56e41ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 >> 172.21.15.125:0/2858863402 conn(0x7f56e406a800 msgr2=0x7f56e4155400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 shutdown_connections 2024-01-16T14:59:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.766+0000 7f56ec000640 1 -- 172.21.15.125:0/2858863402 wait complete. 2024-01-16T14:59:26.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T14:59:26.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.994+0000 7f77fbe00640 1 Processor -- start 2024-01-16T14:59:26.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.995+0000 7f77fbe00640 1 -- start start 2024-01-16T14:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.996+0000 7f77fbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.996+0000 7f77fbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.996+0000 7f77fbe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc0cdec0 con 0x7f77fc0cd9b0 2024-01-16T14:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.996+0000 7f77fbe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc0ce1a0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.996+0000 7f77fbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0ce480 con 0x7f77fc0cd530 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.997+0000 7f77faa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.997+0000 7f77f1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.997+0000 7f77f1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:54442/0 (socket says 172.21.15.125:54442) 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.997+0000 7f77faa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:57168/0 (socket says 172.21.15.125:57168) 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.997+0000 7f77f1e00640 1 -- 172.21.15.125:0/3973270421 learned_addr learned my addr 172.21.15.125:0/3973270421 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.998+0000 7f77f1e00640 1 -- 172.21.15.125:0/3973270421 >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 msgr2=0x7f77fc15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.998+0000 7f77f1e00640 1 --2- 172.21.15.125:0/3973270421 >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.998+0000 7f77f1e00640 1 -- 172.21.15.125:0/3973270421 >> v1:172.21.15.137:6789/0 conn(0x7f77fc0cd9b0 legacy=0x7f77fc15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.998+0000 7f77f1e00640 1 -- 172.21.15.125:0/3973270421 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc0ce760 con 0x7f77fc0cd530 2024-01-16T14:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.998+0000 7f77faa00640 1 --2- 172.21.15.125:0/3973270421 >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.999+0000 7f77f1e00640 1 --2- 172.21.15.125:0/3973270421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009870 tx=0x7f77e402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=430f95f15c3cf60c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:26.999+0000 7f77fa000640 1 -- 172.21.15.125:0/3973270421 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4031640 con 0x7f77fc0cd530 2024-01-16T14:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fa000640 1 -- 172.21.15.125:0/3973270421 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e40317a0 con 0x7f77fc0cd530 2024-01-16T14:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fa000640 1 -- 172.21.15.125:0/3973270421 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4031b30 con 0x7f77fc0cd530 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 -- 172.21.15.125:0/3973270421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 msgr2=0x7f77fc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 --2- 172.21.15.125:0/3973270421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009870 tx=0x7f77e402b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 -- 172.21.15.125:0/3973270421 shutdown_connections 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 --2- 172.21.15.125:0/3973270421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd530 0x7f77fc13f5a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 --2- 172.21.15.125:0/3973270421 >> v2:172.21.15.137:3300/0 conn(0x7f77fc13cfa0 0x7f77fc15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 -- 172.21.15.125:0/3973270421 >> 172.21.15.125:0/3973270421 conn(0x7f77fc06a800 msgr2=0x7f77fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 -- 172.21.15.125:0/3973270421 shutdown_connections 2024-01-16T14:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.000+0000 7f77fbe00640 1 -- 172.21.15.125:0/3973270421 wait complete. 2024-01-16T14:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.002+0000 7f77fbe00640 1 Processor -- start 2024-01-16T14:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.002+0000 7f77fbe00640 1 -- start start 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77fc0cd530 0x7f77fc12c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd9b0 0x7f77fc1f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1f4ce0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1f47e0 con 0x7f77fc0cd530 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc12c600 con 0x7f77fc0cd9b0 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.003+0000 7f77fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52072/0 (socket says 172.21.15.125:52072) 2024-01-16T14:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 -- 172.21.15.125:0/686337624 learned_addr learned my addr 172.21.15.125:0/686337624 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77faa00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77fc0cd530 0x7f77fc12c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77f1e00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd9b0 0x7f77fc1f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 -- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd9b0 msgr2=0x7f77fc1f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd9b0 0x7f77fc1f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 -- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77fc0cd530 msgr2=0x7f77fc12c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77fc0cd530 0x7f77fc12c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 -- 172.21.15.125:0/686337624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec009400 con 0x7f77fc13cfa0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77fb400640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c7d0 tx=0x7f77f400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.004+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4003b70 con 0x7f77fc13cfa0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.005+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f4003cf0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.005+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e40094d0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.005+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400aa90 con 0x7f77fc13cfa0 2024-01-16T14:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.005+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc1f4220 con 0x7f77fc13cfa0 2024-01-16T14:59:26.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.006+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc1f4a40 con 0x7f77fc13cfa0 2024-01-16T14:59:26.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.007+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f77f400abf0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.008+0000 7f77f3e00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 0x7f77d406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.008+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77f4089dc0 con 0x7f77fc13cfa0 2024-01-16T14:59:26.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.008+0000 7f77faa00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 0x7f77d406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.008+0000 7f77faa00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 0x7f77d406d930 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009c60 tx=0x7f77ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.012+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77fc1f4a40 con 0x7f77fc13cfa0 2024-01-16T14:59:27.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.225+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f77fc1f4e20 con 0x7f77d406b470 2024-01-16T14:59:27.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.228+0000 7f77f3e00640 1 -- 172.21.15.125:0/686337624 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f77fc1f4e20 con 0x7f77d406b470 2024-01-16T14:59:27.208 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:59:27.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:27.209 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T14:59:27.209 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:27.209 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:59:27.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.230+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 msgr2=0x7f77d406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.230+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 0x7f77d406d930 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009c60 tx=0x7f77ec008040 comp rx=0 tx=0).stop 2024-01-16T14:59:27.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.230+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 msgr2=0x7f77fc14ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.230+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c7d0 tx=0x7f77f400cc90 comp rx=0 tx=0).stop 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 shutdown_connections 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77d406b470 0x7f77d406d930 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77fc13cfa0 0x7f77fc14ad90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77fc0cd9b0 0x7f77fc1f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 --2- 172.21.15.125:0/686337624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77fc0cd530 0x7f77fc12c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 >> 172.21.15.125:0/686337624 conn(0x7f77fc06a800 msgr2=0x7f77fc158820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 shutdown_connections 2024-01-16T14:59:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.231+0000 7f77fbe00640 1 -- 172.21.15.125:0/686337624 wait complete. 2024-01-16T14:59:27.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T14:59:27.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.454+0000 7f5256c00640 1 Processor -- start 2024-01-16T14:59:27.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.454+0000 7f5256c00640 1 -- start start 2024-01-16T14:59:27.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5256c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f52500ae5a0 0x7f525015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5256c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 0x7f52500cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5256c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5250070c40 con 0x7f52500b3ae0 2024-01-16T14:59:27.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5256c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5250071450 con 0x7f52500ae5a0 2024-01-16T14:59:27.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5256c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cfc60 con 0x7f52500cf290 2024-01-16T14:59:27.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f524f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f52500b3ae0 0x7f525006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34264/0 (socket says 172.21.15.125:34264) 2024-01-16T14:59:27.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f524f400640 1 -- 172.21.15.125:0/2557707462 learned_addr learned my addr 172.21.15.125:0/2557707462 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f5254c00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 0x7f52500cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.456+0000 7f524fe00640 1 --2- 172.21.15.125:0/2557707462 >> v2:172.21.15.137:3300/0 conn(0x7f52500ae5a0 0x7f525015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453478847 0 0) 0x7f5250070c40 con 0x7f52500b3ae0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5230003520 con 0x7f52500b3ae0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108079724 0 0) 0x7f5230003520 con 0x7f52500b3ae0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 msgr2=0x7f52500cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 0x7f52500cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 >> v2:172.21.15.137:3300/0 conn(0x7f52500ae5a0 msgr2=0x7f525015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 --2- 172.21.15.125:0/2557707462 >> v2:172.21.15.137:3300/0 conn(0x7f52500ae5a0 0x7f525015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525006c470 con 0x7f52500b3ae0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5238001670 con 0x7f52500b3ae0 2024-01-16T14:59:27.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 >> v1:172.21.15.125:6789/0 conn(0x7f52500b3ae0 legacy=0x7f525006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5230003970 0x7f5230003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f5230004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 0x7f5230005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250071450 con 0x7f52300047c0 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cfc60 con 0x7f5230003970 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523000ba00 con 0x7f52500cf290 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.457+0000 7f524f400640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f5230004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524f400640 1 -- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 msgr2=0x7f5230005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524f400640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52500cf290 0x7f5230005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524fe00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5230003970 0x7f5230003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524f400640 1 -- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5230003970 msgr2=0x7f5230003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524f400640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5230003970 0x7f5230003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.458+0000 7f524f400640 1 -- 172.21.15.125:0/2557707462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230003520 con 0x7f52300047c0 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.459+0000 7f524f400640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f5230004c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f523800c190 tx=0x7f52300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a823d1156e5b2a60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.459+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238005ad0 con 0x7f52300047c0 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.459+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5238005c50 con 0x7f52300047c0 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.459+0000 7f524ea00640 1 -- 172.21.15.125:0/2557707462 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523802c810 con 0x7f52300047c0 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 -- 172.21.15.125:0/2557707462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 msgr2=0x7f5230004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 --2- 172.21.15.125:0/2557707462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f5230004c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f523800c190 tx=0x7f52300057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5254c00640 1 -- 172.21.15.125:0/2557707462 reap_dead start 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 -- 172.21.15.125:0/2557707462 shutdown_connections 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 -- 172.21.15.125:0/2557707462 >> 172.21.15.125:0/2557707462 conn(0x7f525006a800 msgr2=0x7f525006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 -- 172.21.15.125:0/2557707462 shutdown_connections 2024-01-16T14:59:27.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.460+0000 7f5256c00640 1 -- 172.21.15.125:0/2557707462 wait complete. 2024-01-16T14:59:27.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.461+0000 7f5256c00640 1 Processor -- start 2024-01-16T14:59:27.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.462+0000 7f5256c00640 1 -- start start 2024-01-16T14:59:27.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.462+0000 7f5256c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.462+0000 7f5256c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52500cbf70 0x7f5250153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f5256c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f525015c5f0 0x7f525015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f5256c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525015cfb0 con 0x7f52300047c0 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f5256c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cc530 con 0x7f52500cbf70 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f5256c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cc810 con 0x7f525015c5f0 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52088/0 (socket says 172.21.15.125:52088) 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 -- 172.21.15.125:0/749681136 learned_addr learned my addr 172.21.15.125:0/749681136 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 -- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f525015c5f0 msgr2=0x7f525015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f525015c5f0 0x7f525015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 -- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52500cbf70 msgr2=0x7f5250153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52500cbf70 0x7f5250153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 -- 172.21.15.125:0/749681136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500ccaf0 con 0x7f52300047c0 2024-01-16T14:59:27.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.463+0000 7f524fe00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f52400108e0 tx=0x7f5240010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.464+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524000eae0 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.464+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524000ec40 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.464+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524001c400 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.464+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52500ccdd0 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.464+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52500b2910 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.465+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5250072130 con 0x7f52300047c0 2024-01-16T14:59:27.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.466+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f524001c560 con 0x7f52300047c0 2024-01-16T14:59:27.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.467+0000 7f5247e00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 0x7f522806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.467+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f524008a610 con 0x7f52300047c0 2024-01-16T14:59:27.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.467+0000 7f524f400640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 0x7f522806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.467+0000 7f524f400640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 0x7f522806d930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5238034000 tx=0x7f523802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.471+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5240055210 con 0x7f52300047c0 2024-01-16T14:59:27.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.691+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52500cc980 con 0x7f522806b470 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.698+0000 7f5247e00640 1 -- 172.21.15.125:0/749681136 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f52500cc980 con 0x7f522806b470 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1clone1" 2024-01-16T14:59:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:27.680 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 msgr2=0x7f522806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 0x7f522806d930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5238034000 tx=0x7f523802e000 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 msgr2=0x7f52501f2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f52400108e0 tx=0x7f5240010da0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 shutdown_connections 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f525015c5f0 0x7f525015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52500cbf70 0x7f5250153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f522806b470 0x7f522806d930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 --2- 172.21.15.125:0/749681136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52300047c0 0x7f52501f2a30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 >> 172.21.15.125:0/749681136 conn(0x7f525006a800 msgr2=0x7f525015a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 shutdown_connections 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.701+0000 7f5256c00640 1 -- 172.21.15.125:0/749681136 wait complete. 2024-01-16T14:59:27.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T14:59:27.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.929+0000 7fd612000640 1 Processor -- start 2024-01-16T14:59:27.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.929+0000 7fd612000640 1 -- start start 2024-01-16T14:59:27.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd612000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd60c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd612000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0b2c70 con 0x7fd60c0b4110 2024-01-16T14:59:27.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd612000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0cf6d0 con 0x7fd60c0cfa40 2024-01-16T14:59:27.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd612000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0cf3f0 con 0x7fd60c15d3c0 2024-01-16T14:59:27.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd611600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd60c0b4110 0x7fd60c0ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34270/0 (socket says 172.21.15.125:34270) 2024-01-16T14:59:27.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.931+0000 7fd611600640 1 -- 172.21.15.125:0/3003642014 learned_addr learned my addr 172.21.15.125:0/3003642014 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd610c00640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd60c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470705990 0 0) 0x7fd60c0b2c70 con 0x7fd60c0b4110 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5ec003520 con 0x7fd60c0b4110 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463069590 0 0) 0x7fd5ec003520 con 0x7fd60c0b4110 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 msgr2=0x7fd60c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd60c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 >> v1:172.21.15.137:6789/0 conn(0x7fd60c0cfa40 legacy=0x7fd60c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0b2620 con 0x7fd60c0b4110 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd604001670 con 0x7fd60c0b4110 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 >> v1:172.21.15.125:6789/0 conn(0x7fd60c0b4110 legacy=0x7fd60c0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd5ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5ec003f20 0x7fd5ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5ec004d00 0x7fd5ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0cf3f0 con 0x7fd5ec004d00 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec00ba60 con 0x7fd5ec003f20 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.932+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec00bc20 con 0x7fd60c15d3c0 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5ec004d00 0x7fd5ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd60be00640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5ec003f20 0x7fd5ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd610c00640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd5ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 -- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 msgr2=0x7fd5ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd60c15d3c0 0x7fd5ec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 -- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5ec003f20 msgr2=0x7fd5ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5ec003f20 0x7fd5ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.933+0000 7fd611600640 1 -- 172.21.15.125:0/3003642014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec003520 con 0x7fd5ec004d00 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.934+0000 7fd611600640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5ec004d00 0x7fd5ec005180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c190 tx=0x7fd5ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f0e9abc137eb65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.934+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd604005ad0 con 0x7fd5ec004d00 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.934+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604005c30 con 0x7fd5ec004d00 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.934+0000 7fd60b400640 1 -- 172.21.15.125:0/3003642014 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60402cae0 con 0x7fd5ec004d00 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 -- 172.21.15.125:0/3003642014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5ec004d00 msgr2=0x7fd5ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 --2- 172.21.15.125:0/3003642014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5ec004d00 0x7fd5ec005180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c190 tx=0x7fd5ec0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd611600640 1 -- 172.21.15.125:0/3003642014 reap_dead start 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 -- 172.21.15.125:0/3003642014 shutdown_connections 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 -- 172.21.15.125:0/3003642014 >> 172.21.15.125:0/3003642014 conn(0x7fd60c0d3050 msgr2=0x7fd60c0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 -- 172.21.15.125:0/3003642014 shutdown_connections 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.935+0000 7fd612000640 1 -- 172.21.15.125:0/3003642014 wait complete. 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.937+0000 7fd612000640 1 Processor -- start 2024-01-16T14:59:27.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.937+0000 7fd612000640 1 -- start start 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5ec004d00 0x7fd60c160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd60c0c7f20 0x7fd60c0b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0b6ad0 con 0x7fd60c15c0b0 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0b61a0 con 0x7fd60c0c7f20 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd612000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c15be90 con 0x7fd5ec004d00 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd60be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd60be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52112/0 (socket says 172.21.15.125:52112) 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.938+0000 7fd60be00640 1 -- 172.21.15.125:0/3178689273 learned_addr learned my addr 172.21.15.125:0/3178689273 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd611600640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd60c0c7f20 0x7fd60c0b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd610c00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5ec004d00 0x7fd60c160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 -- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5ec004d00 msgr2=0x7fd60c160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5ec004d00 0x7fd60c160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 -- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd60c0c7f20 msgr2=0x7fd60c0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd60c0c7f20 0x7fd60c0b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 -- 172.21.15.125:0/3178689273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0b6410 con 0x7fd60c15c0b0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd60be00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e580 tx=0x7fd5f400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40052e0 con 0x7fd60c15c0b0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f4005440 con 0x7fd60c15c0b0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4003810 con 0x7fd60c15c0b0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.939+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c0b66f0 con 0x7fd60c15c0b0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.940+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c14f8b0 con 0x7fd60c15c0b0 2024-01-16T14:59:27.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.941+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c14fad0 con 0x7fd60c15c0b0 2024-01-16T14:59:27.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.942+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd5f4003970 con 0x7fd60c15c0b0 2024-01-16T14:59:27.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.943+0000 7fd608c00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 0x7fd5e406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.943+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5f4089c80 con 0x7fd60c15c0b0 2024-01-16T14:59:27.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.943+0000 7fd610c00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 0x7fd5e406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.944+0000 7fd610c00640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 0x7fd5e406d960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc007520 tx=0x7fd5fc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:27.947+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5f40547e0 con 0x7fd60c15c0b0 2024-01-16T14:59:28.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.170+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd60c0d0f50 con 0x7fd5e406b4a0 2024-01-16T14:59:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.195+0000 7fd608c00640 1 -- 172.21.15.125:0/3178689273 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd5fc009db0 con 0x7fd5e406b4a0 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:28.176 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:59:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 msgr2=0x7fd5e406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 0x7fd5e406d960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc007520 tx=0x7fd5fc00b040 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 msgr2=0x7fd60c0c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e580 tx=0x7fd5f400ea40 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 shutdown_connections 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd60c0c7f20 0x7fd60c0b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd5e406b4a0 0x7fd5e406d960 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd60c15c0b0 0x7fd60c0c79e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 --2- 172.21.15.125:0/3178689273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5ec004d00 0x7fd60c160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 >> 172.21.15.125:0/3178689273 conn(0x7fd60c0d3050 msgr2=0x7fd60c0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 shutdown_connections 2024-01-16T14:59:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.198+0000 7fd612000640 1 -- 172.21.15.125:0/3178689273 wait complete. 2024-01-16T14:59:28.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T14:59:28.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.427+0000 7f2cc9800640 1 Processor -- start 2024-01-16T14:59:28.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.427+0000 7f2cc9800640 1 -- start start 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.428+0000 7f2cc9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cc40ce0c0 0x7f2cc4164df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.428+0000 7f2cc9800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4071b50 con 0x7f2cc413cda0 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.428+0000 7f2cc9800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40700f0 con 0x7f2cc415e170 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.428+0000 7f2cc9800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc40ce510 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2cc413cda0 0x7f2cc415da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34276/0 (socket says 172.21.15.125:34276) 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc3e00640 1 -- 172.21.15.125:0/3976541151 learned_addr learned my addr 172.21.15.125:0/3976541151 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc8e00640 1 --2- 172.21.15.125:0/3976541151 >> v2:172.21.15.125:3300/0 conn(0x7f2cc40ce0c0 0x7f2cc4164df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc2a00640 1 -- 172.21.15.125:0/3976541151 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548083550 0 0) 0x7f2cc4071b50 con 0x7f2cc413cda0 2024-01-16T14:59:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc2a00640 1 -- 172.21.15.125:0/3976541151 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4003520 con 0x7f2cc413cda0 2024-01-16T14:59:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc8e00640 1 -- 172.21.15.125:0/3976541151 >> v1:172.21.15.137:6789/0 conn(0x7f2cc415e170 legacy=0x7f2cc40cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc8e00640 1 -- 172.21.15.125:0/3976541151 >> v1:172.21.15.125:6789/0 conn(0x7f2cc413cda0 legacy=0x7f2cc415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.429+0000 7f2cc8e00640 1 -- 172.21.15.125:0/3976541151 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40ce720 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.430+0000 7f2cc8e00640 1 --2- 172.21.15.125:0/3976541151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc4164df0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a049d6b01e3f054 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc2a00640 1 -- 172.21.15.125:0/3976541151 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc02cbd0 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc2a00640 1 -- 172.21.15.125:0/3976541151 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc02cd30 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc2a00640 1 -- 172.21.15.125:0/3976541151 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc02cea0 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc9800640 1 -- 172.21.15.125:0/3976541151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40ce0c0 msgr2=0x7f2cc4164df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc9800640 1 --2- 172.21.15.125:0/3976541151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc4164df0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc02be20 comp rx=0 tx=0).stop 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc9800640 1 -- 172.21.15.125:0/3976541151 shutdown_connections 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc9800640 1 --2- 172.21.15.125:0/3976541151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc4164df0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.431+0000 7f2cc9800640 1 -- 172.21.15.125:0/3976541151 >> 172.21.15.125:0/3976541151 conn(0x7f2cc406a3e0 msgr2=0x7f2cc406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.432+0000 7f2cc9800640 1 -- 172.21.15.125:0/3976541151 shutdown_connections 2024-01-16T14:59:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.432+0000 7f2cc9800640 1 -- 172.21.15.125:0/3976541151 wait complete. 2024-01-16T14:59:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.433+0000 7f2cc9800640 1 Processor -- start 2024-01-16T14:59:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.433+0000 7f2cc9800640 1 -- start start 2024-01-16T14:59:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc40b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cc415e170 0x7f2cc413eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b0750 con 0x7f2cc413cda0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b3360 con 0x7f2cc40ce0c0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.434+0000 7f2cc9800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4159d10 con 0x7f2cc415e170 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52130/0 (socket says 172.21.15.125:52130) 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 -- 172.21.15.125:0/1297588397 learned_addr learned my addr 172.21.15.125:0/1297588397 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3e00640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc40b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc8e00640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cc415e170 0x7f2cc413eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 -- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cc415e170 msgr2=0x7f2cc413eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cc415e170 0x7f2cc413eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 -- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc40ce0c0 msgr2=0x7f2cc40b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc40b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.435+0000 7f2cc3400640 1 -- 172.21.15.125:0/1297588397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009a70 con 0x7f2cc413cda0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cc3400640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00c430 tx=0x7f2cac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac011790 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac0118f0 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac011ce0 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc4159ff0 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.436+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40b0420 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.437+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc40b2b40 con 0x7f2cc413cda0 2024-01-16T14:59:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.438+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2cac01c460 con 0x7f2cc413cda0 2024-01-16T14:59:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.439+0000 7f2cbbe00640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 0x7f2c9c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.439+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cac089e90 con 0x7f2cc413cda0 2024-01-16T14:59:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.439+0000 7f2cc3e00640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 0x7f2c9c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.440+0000 7f2cc3e00640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 0x7f2c9c06d930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005880 tx=0x7f2cb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.443+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cc40b2b40 con 0x7f2cc413cda0 2024-01-16T14:59:28.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.668+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cc4159e80 con 0x7f2c9c06b470 2024-01-16T14:59:28.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.726+0000 7f2cbbe00640 1 -- 172.21.15.125:0/1297588397 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2cc4159e80 con 0x7f2c9c06b470 2024-01-16T14:59:28.707 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:28.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.729+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 msgr2=0x7f2c9c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.729+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 0x7f2c9c06d930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005880 tx=0x7f2cb4009040 comp rx=0 tx=0).stop 2024-01-16T14:59:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.729+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 msgr2=0x7f2cc41591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.729+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00c430 tx=0x7f2cac00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 shutdown_connections 2024-01-16T14:59:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cc415e170 0x7f2cc413eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2c9c06b470 0x7f2c9c06d930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc413cda0 0x7f2cc41591e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 --2- 172.21.15.125:0/1297588397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cc40ce0c0 0x7f2cc40b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 >> 172.21.15.125:0/1297588397 conn(0x7f2cc406a3e0 msgr2=0x7f2cc40aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 shutdown_connections 2024-01-16T14:59:28.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.730+0000 7f2cc9800640 1 -- 172.21.15.125:0/1297588397 wait complete. 2024-01-16T14:59:28.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T14:59:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.947+0000 7f6f52200640 1 Processor -- start 2024-01-16T14:59:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.947+0000 7f6f52200640 1 -- start start 2024-01-16T14:59:28.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f52200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6f4c0b3ae0 0x7f6f4c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f52200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f4c0cf290 0x7f6f4c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f52200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c070c40 con 0x7f6f4c0ae5a0 2024-01-16T14:59:28.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f52200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c071450 con 0x7f6f4c0cf290 2024-01-16T14:59:28.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f52200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c0cfc60 con 0x7f6f4c0b3ae0 2024-01-16T14:59:28.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f4b400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6f4c0ae5a0 0x7f6f4c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34282/0 (socket says 172.21.15.125:34282) 2024-01-16T14:59:28.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f4b400640 1 -- 172.21.15.125:0/2248228137 learned_addr learned my addr 172.21.15.125:0/2248228137 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:28.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f4be00640 1 --2- 172.21.15.125:0/2248228137 >> v2:172.21.15.125:3300/0 conn(0x7f6f4c0cf290 0x7f6f4c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f42800640 1 --2- 172.21.15.125:0/2248228137 >> v2:172.21.15.137:3300/0 conn(0x7f6f4c0b3ae0 0x7f6f4c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.949+0000 7f6f4aa00640 1 -- 172.21.15.125:0/2248228137 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053962760 0 0) 0x7f6f4c070c40 con 0x7f6f4c0ae5a0 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.950+0000 7f6f4be00640 1 -- 172.21.15.125:0/2248228137 >> v2:172.21.15.137:3300/0 conn(0x7f6f4c0b3ae0 msgr2=0x7f6f4c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.950+0000 7f6f4be00640 1 --2- 172.21.15.125:0/2248228137 >> v2:172.21.15.137:3300/0 conn(0x7f6f4c0b3ae0 0x7f6f4c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.950+0000 7f6f4be00640 1 -- 172.21.15.125:0/2248228137 >> v1:172.21.15.125:6789/0 conn(0x7f6f4c0ae5a0 legacy=0x7f6f4c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.950+0000 7f6f4be00640 1 -- 172.21.15.125:0/2248228137 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c06c470 con 0x7f6f4c0cf290 2024-01-16T14:59:28.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f4be00640 1 --2- 172.21.15.125:0/2248228137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0cf720 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0cfea0 tx=0x7f6f4402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8fb2b535fce5dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f4aa00640 1 -- 172.21.15.125:0/2248228137 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4402cd00 con 0x7f6f4c0cf290 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f4aa00640 1 -- 172.21.15.125:0/2248228137 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4402ce60 con 0x7f6f4c0cf290 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f4aa00640 1 -- 172.21.15.125:0/2248228137 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44034660 con 0x7f6f4c0cf290 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f52200640 1 -- 172.21.15.125:0/2248228137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 msgr2=0x7f6f4c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.951+0000 7f6f52200640 1 --2- 172.21.15.125:0/2248228137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0cf720 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0cfea0 tx=0x7f6f4402bf50 comp rx=0 tx=0).stop 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 -- 172.21.15.125:0/2248228137 shutdown_connections 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 --2- 172.21.15.125:0/2248228137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0cf720 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 --2- 172.21.15.125:0/2248228137 >> v2:172.21.15.137:3300/0 conn(0x7f6f4c0b3ae0 0x7f6f4c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 -- 172.21.15.125:0/2248228137 >> 172.21.15.125:0/2248228137 conn(0x7f6f4c06a800 msgr2=0x7f6f4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 -- 172.21.15.125:0/2248228137 shutdown_connections 2024-01-16T14:59:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.952+0000 7f6f52200640 1 -- 172.21.15.125:0/2248228137 wait complete. 2024-01-16T14:59:28.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.953+0000 7f6f52200640 1 Processor -- start 2024-01-16T14:59:28.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.953+0000 7f6f52200640 1 -- start start 2024-01-16T14:59:28.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f4c0ae5a0 0x7f6f4c14ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f4c0b3ae0 0x7f6f4c0b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1570f0 con 0x7f6f4c0cf290 2024-01-16T14:59:28.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c152450 con 0x7f6f4c0b3ae0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.954+0000 7f6f52200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c152730 con 0x7f6f4c0ae5a0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52148/0 (socket says 172.21.15.125:52148) 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 -- 172.21.15.125:0/1104162573 learned_addr learned my addr 172.21.15.125:0/1104162573 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4b400640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f4c0ae5a0 0x7f6f4c14ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f42800640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f4c0b3ae0 0x7f6f4c0b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 -- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f4c0ae5a0 msgr2=0x7f6f4c14ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f4c0ae5a0 0x7f6f4c14ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 -- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f4c0b3ae0 msgr2=0x7f6f4c0b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f4c0b3ae0 0x7f6f4c0b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 -- 172.21.15.125:0/1104162573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44009a70 con 0x7f6f4c0cf290 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.955+0000 7f6f4be00640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009e80 tx=0x7f6f44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.956+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4403d600 con 0x7f6f4c0cf290 2024-01-16T14:59:28.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.956+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4403d780 con 0x7f6f4c0cf290 2024-01-16T14:59:28.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.956+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c152a10 con 0x7f6f4c0cf290 2024-01-16T14:59:28.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.956+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4403db30 con 0x7f6f4c0cf290 2024-01-16T14:59:28.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.956+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c152fd0 con 0x7f6f4c0cf290 2024-01-16T14:59:28.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.957+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c0675d0 con 0x7f6f4c0cf290 2024-01-16T14:59:28.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.958+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f6f4403b040 con 0x7f6f4c0cf290 2024-01-16T14:59:28.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.959+0000 7f6f43e00640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 0x7f6f2406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.959+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f440ac570 con 0x7f6f4c0cf290 2024-01-16T14:59:28.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.959+0000 7f6f4b400640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 0x7f6f2406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.960+0000 7f6f4b400640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 0x7f6f2406d930 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c000f60 tx=0x7f6f3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:28.963+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f440770c0 con 0x7f6f4c0cf290 2024-01-16T14:59:29.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.182+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f4c071170 con 0x7f6f2406b470 2024-01-16T14:59:29.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.220+0000 7f6f43e00640 1 -- 172.21.15.125:0/1104162573 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f3c001c70 con 0x7f6f2406b470 2024-01-16T14:59:29.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 msgr2=0x7f6f2406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 0x7f6f2406d930 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c000f60 tx=0x7f6f3c009040 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 msgr2=0x7f6f4c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009e80 tx=0x7f6f44004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 shutdown_connections 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f2406b470 0x7f6f2406d930 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f4c0cf290 0x7f6f4c0b21e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f4c0b3ae0 0x7f6f4c0b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 --2- 172.21.15.125:0/1104162573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f4c0ae5a0 0x7f6f4c14ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 >> 172.21.15.125:0/1104162573 conn(0x7f6f4c06a800 msgr2=0x7f6f4c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 shutdown_connections 2024-01-16T14:59:29.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.223+0000 7f6f52200640 1 -- 172.21.15.125:0/1104162573 wait complete. 2024-01-16T14:59:29.207 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.226+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.207 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.226+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.207 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.226+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.207 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.226+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.207 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.226+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T14:59:29.230 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.248+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.230 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.248+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.230 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.248+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.230 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.248+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.230 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T14:59:29.248+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:29.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.434+0000 7fc65ea00640 1 Processor -- start 2024-01-16T14:59:29.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.434+0000 7fc65ea00640 1 -- start start 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc658160920 0x7fc6580b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc6580cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc658161010 0x7fc6581614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6580d03b0 con 0x7fc658160920 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6580d0a10 con 0x7fc658161010 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6580cf080 con 0x7fc6580cf9f0 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.435+0000 7fc65d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc658160920 0x7fc6580b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc658160920 0x7fc6580b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52198/0 (socket says 172.21.15.125:52198) 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 -- 172.21.15.125:0/2033507873 learned_addr learned my addr 172.21.15.125:0/2033507873 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65cc00640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc6580cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 -- 172.21.15.125:0/2033507873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 msgr2=0x7fc6580cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc6580cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 -- 172.21.15.125:0/2033507873 >> v2:172.21.15.137:3300/0 conn(0x7fc658161010 msgr2=0x7fc6581614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65e000640 1 --2- 172.21.15.125:0/2033507873 >> v2:172.21.15.137:3300/0 conn(0x7fc658161010 0x7fc6581614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 --2- 172.21.15.125:0/2033507873 >> v2:172.21.15.137:3300/0 conn(0x7fc658161010 0x7fc6581614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.436+0000 7fc65d600640 1 -- 172.21.15.125:0/2033507873 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580c6be0 con 0x7fc658160920 2024-01-16T14:59:29.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.437+0000 7fc65d600640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc6580b3f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc6580c6930 tx=0x7fc64802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc04a028374dd342 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.437+0000 7fc657e00640 1 -- 172.21.15.125:0/2033507873 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648031860 con 0x7fc658160920 2024-01-16T14:59:29.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc657e00640 1 -- 172.21.15.125:0/2033507873 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6480319c0 con 0x7fc658160920 2024-01-16T14:59:29.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc657e00640 1 -- 172.21.15.125:0/2033507873 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648031d50 con 0x7fc658160920 2024-01-16T14:59:29.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 -- 172.21.15.125:0/2033507873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 msgr2=0x7fc6580b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc6580b3f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc6580c6930 tx=0x7fc64802b900 comp rx=0 tx=0).stop 2024-01-16T14:59:29.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 -- 172.21.15.125:0/2033507873 shutdown_connections 2024-01-16T14:59:29.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 --2- 172.21.15.125:0/2033507873 >> v2:172.21.15.137:3300/0 conn(0x7fc658161010 0x7fc6581614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc6580cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 --2- 172.21.15.125:0/2033507873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc6580b3f30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 -- 172.21.15.125:0/2033507873 >> 172.21.15.125:0/2033507873 conn(0x7fc6580d3090 msgr2=0x7fc6580d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 -- 172.21.15.125:0/2033507873 shutdown_connections 2024-01-16T14:59:29.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.438+0000 7fc65ea00640 1 -- 172.21.15.125:0/2033507873 wait complete. 2024-01-16T14:59:29.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.440+0000 7fc65ea00640 1 Processor -- start 2024-01-16T14:59:29.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.440+0000 7fc65ea00640 1 -- start start 2024-01-16T14:59:29.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc65806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc658161010 0x7fc6580b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc658152e80 con 0x7fc658160920 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6580cbc60 con 0x7fc658161010 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.441+0000 7fc65ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6580cbf00 con 0x7fc6580cf9f0 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52206/0 (socket says 172.21.15.125:52206) 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 -- 172.21.15.125:0/17713505 learned_addr learned my addr 172.21.15.125:0/17713505 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65d600640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc65806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65e000640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc658161010 0x7fc6580b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 -- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 msgr2=0x7fc65806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc65806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 -- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc658161010 msgr2=0x7fc6580b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc658161010 0x7fc6580b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 -- 172.21.15.125:0/17713505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6480094d0 con 0x7fc658160920 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.442+0000 7fc65cc00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d800 tx=0x7fc64000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.443+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400096a0 con 0x7fc658160920 2024-01-16T14:59:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.443+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640009800 con 0x7fc658160920 2024-01-16T14:59:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.443+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640009b90 con 0x7fc658160920 2024-01-16T14:59:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.443+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6580cc1e0 con 0x7fc658160920 2024-01-16T14:59:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.443+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6580cc7a0 con 0x7fc658160920 2024-01-16T14:59:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.444+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658067450 con 0x7fc658160920 2024-01-16T14:59:29.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.446+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fc640009cf0 con 0x7fc658160920 2024-01-16T14:59:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.447+0000 7fc655600640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 0x7fc63006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.447+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc64008a310 con 0x7fc658160920 2024-01-16T14:59:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.447+0000 7fc65d600640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 0x7fc63006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.448+0000 7fc65d600640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 0x7fc63006d930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc648036000 tx=0x7fc64803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.452+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc640054f90 con 0x7fc658160920 2024-01-16T14:59:29.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.670+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6580cc070 con 0x7fc63006b470 2024-01-16T14:59:29.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.689+0000 7fc655600640 1 -- 172.21.15.125:0/17713505 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc6580cc070 con 0x7fc63006b470 2024-01-16T14:59:29.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.691+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 msgr2=0x7fc63006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.691+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 0x7fc63006d930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc648036000 tx=0x7fc64803c040 comp rx=0 tx=0).stop 2024-01-16T14:59:29.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.691+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 msgr2=0x7fc658152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.691+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d800 tx=0x7fc64000dcc0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 shutdown_connections 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc658161010 0x7fc6580b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63006b470 0x7fc63006d930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc658160920 0x7fc658152940 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 --2- 172.21.15.125:0/17713505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6580cf9f0 0x7fc65806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 >> 172.21.15.125:0/17713505 conn(0x7fc6580d3090 msgr2=0x7fc6580b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 shutdown_connections 2024-01-16T14:59:29.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:29.692+0000 7fc65ea00640 1 -- 172.21.15.125:0/17713505 wait complete. 2024-01-16T14:59:29.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:29.693 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T14:59:29.745 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T14:59:29.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.009+0000 7f022f400640 1 Processor -- start 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.009+0000 7f022f400640 1 -- start start 2024-01-16T14:59:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.010+0000 7f022f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 0x7f022815a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.010+0000 7f022f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.010+0000 7f022f400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280b1e30 con 0x7f02281505b0 2024-01-16T14:59:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.010+0000 7f022f400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022815abd0 con 0x7f02280b3b20 2024-01-16T14:59:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.010+0000 7f022f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02280b20a0 con 0x7f022815d8a0 2024-01-16T14:59:29.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.011+0000 7f022ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.011+0000 7f022e000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 0x7f022815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.011+0000 7f022ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:54596/0 (socket says 172.21.15.125:54596) 2024-01-16T14:59:29.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.011+0000 7f022ea00640 1 -- 172.21.15.125:0/2136563676 learned_addr learned my addr 172.21.15.125:0/2136563676 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:29.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.012+0000 7f022ea00640 1 -- 172.21.15.125:0/2136563676 >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 msgr2=0x7f022815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.012+0000 7f022ea00640 1 --2- 172.21.15.125:0/2136563676 >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 0x7f022815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.012+0000 7f022ea00640 1 -- 172.21.15.125:0/2136563676 >> v1:172.21.15.137:6789/0 conn(0x7f02281505b0 legacy=0x7f022815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.012+0000 7f022ea00640 1 -- 172.21.15.125:0/2136563676 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02280b2380 con 0x7f022815d8a0 2024-01-16T14:59:29.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.012+0000 7f022e000640 1 --2- 172.21.15.125:0/2136563676 >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 0x7f022815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:29.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.013+0000 7f022ea00640 1 --2- 172.21.15.125:0/2136563676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0220009e50 tx=0x7f022002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab4e9d18bf59e5b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022cc00640 1 -- 172.21.15.125:0/2136563676 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022002cb00 con 0x7f022815d8a0 2024-01-16T14:59:29.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022cc00640 1 -- 172.21.15.125:0/2136563676 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022002cc60 con 0x7f022815d8a0 2024-01-16T14:59:29.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022cc00640 1 -- 172.21.15.125:0/2136563676 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02200343f0 con 0x7f022815d8a0 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 -- 172.21.15.125:0/2136563676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 msgr2=0x7f0228164520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 --2- 172.21.15.125:0/2136563676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0220009e50 tx=0x7f022002bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 -- 172.21.15.125:0/2136563676 shutdown_connections 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 --2- 172.21.15.125:0/2136563676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228164520 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 --2- 172.21.15.125:0/2136563676 >> v2:172.21.15.137:3300/0 conn(0x7f02280b3b20 0x7f022815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 -- 172.21.15.125:0/2136563676 >> 172.21.15.125:0/2136563676 conn(0x7f022806a7d0 msgr2=0x7f022806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 -- 172.21.15.125:0/2136563676 shutdown_connections 2024-01-16T14:59:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.014+0000 7f022f400640 1 -- 172.21.15.125:0/2136563676 wait complete. 2024-01-16T14:59:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.016+0000 7f022f400640 1 Processor -- start 2024-01-16T14:59:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.016+0000 7f022f400640 1 -- start start 2024-01-16T14:59:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02280b3b20 0x7f022814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228166f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02281434f0 con 0x7f02281505b0 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02280aee90 con 0x7f02280b3b20 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0228166a60 con 0x7f022815d8a0 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52218/0 (socket says 172.21.15.125:52218) 2024-01-16T14:59:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.017+0000 7f022d600640 1 -- 172.21.15.125:0/3279999129 learned_addr learned my addr 172.21.15.125:0/3279999129 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022ea00640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228166f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022e000640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02280b3b20 0x7f022814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 -- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 msgr2=0x7f0228166f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228166f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 -- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02280b3b20 msgr2=0x7f022814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02280b3b20 0x7f022814f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 -- 172.21.15.125:0/3279999129 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0218009400 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022d600640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f021000c400 tx=0x7f021000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02100052d0 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0210005430 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210003830 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0220009b30 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.018+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02280ae8d0 con 0x7f02281505b0 2024-01-16T14:59:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.019+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0228166c20 con 0x7f02281505b0 2024-01-16T14:59:30.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.020+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0210003990 con 0x7f02281505b0 2024-01-16T14:59:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.021+0000 7f0226000640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 0x7f020006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.021+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0210089500 con 0x7f02281505b0 2024-01-16T14:59:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.021+0000 7f022e000640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 0x7f020006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.022+0000 7f022e000640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 0x7f020006d930 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0218009ae0 tx=0x7f0218008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.026+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0210054050 con 0x7f02281505b0 2024-01-16T14:59:30.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.048+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f0210054420 con 0x7f02281505b0 2024-01-16T14:59:30.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:30.228+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f0228150a40 con 0x7f020006b470 2024-01-16T14:59:30.628 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T14:59:30.648+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T14:59:30.722 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T14:59:32.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:32.685+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mon.0 v2:172.21.15.125:3300/0 8 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0210057200 con 0x7f02281505b0 2024-01-16T14:59:33.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.684+0000 7f0226000640 1 -- 172.21.15.125:0/3279999129 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f0218010a20 con 0x7f020006b470 2024-01-16T14:59:33.666 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.688+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 msgr2=0x7f020006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.688+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 0x7f020006d930 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0218009ae0 tx=0x7f0218008040 comp rx=0 tx=0).stop 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 msgr2=0x7f0228143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f021000c400 tx=0x7f021000c8c0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 shutdown_connections 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f022815d8a0 0x7f0228166f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f020006b470 0x7f020006d930 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02281505b0 0x7f0228143fc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 --2- 172.21.15.125:0/3279999129 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02280b3b20 0x7f022814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 >> 172.21.15.125:0/3279999129 conn(0x7f022806a7d0 msgr2=0x7f022806cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 shutdown_connections 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:33.689+0000 7f022f400640 1 -- 172.21.15.125:0/3279999129 wait complete. 2024-01-16T14:59:33.699 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T14:59:33.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:33.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:33.948 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T14:59:33.948 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:59:33.948 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:34.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.282+0000 7f111c000640 1 Processor -- start 2024-01-16T14:59:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.283+0000 7f111c000640 1 -- start start 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111c000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1114143ae0 0x7f11140cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111c000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f111415e3b0 0x7f11140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111c000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11140cfc30 con 0x7f11140cf7d0 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111c000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11140b1ee0 con 0x7f111415e3b0 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111c000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11140b2780 con 0x7f1114143ae0 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f111a000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f111415e3b0 0x7f11140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f1118c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f11140cf7d0 0x7f111415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34302/0 (socket says 172.21.15.125:34302) 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.284+0000 7f1118c00640 1 -- 172.21.15.125:0/3876422053 learned_addr learned my addr 172.21.15.125:0/3876422053 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f1119600640 1 --2- 172.21.15.125:0/3876422053 >> v2:172.21.15.137:3300/0 conn(0x7f1114143ae0 0x7f11140cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f111a000640 1 -- 172.21.15.125:0/3876422053 >> v2:172.21.15.137:3300/0 conn(0x7f1114143ae0 msgr2=0x7f11140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f111a000640 1 --2- 172.21.15.125:0/3876422053 >> v2:172.21.15.137:3300/0 conn(0x7f1114143ae0 0x7f11140cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f111a000640 1 -- 172.21.15.125:0/3876422053 >> v1:172.21.15.125:6789/0 conn(0x7f11140cf7d0 legacy=0x7f111415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f111a000640 1 -- 172.21.15.125:0/3876422053 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111415e990 con 0x7f111415e3b0 2024-01-16T14:59:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.285+0000 7f1113e00640 1 -- 172.21.15.125:0/3876422053 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564978542 0 0) 0x7f11140cfc30 con 0x7f11140cf7d0 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.286+0000 7f111a000640 1 --2- 172.21.15.125:0/3876422053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f11140702c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f11140cfea0 tx=0x7f110c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326f53878bed32b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.286+0000 7f1113e00640 1 -- 172.21.15.125:0/3876422053 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c02cc80 con 0x7f111415e3b0 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.286+0000 7f1113e00640 1 -- 172.21.15.125:0/3876422053 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c02cde0 con 0x7f111415e3b0 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.286+0000 7f1113e00640 1 -- 172.21.15.125:0/3876422053 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c034660 con 0x7f111415e3b0 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 -- 172.21.15.125:0/3876422053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 msgr2=0x7f11140702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 --2- 172.21.15.125:0/3876422053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f11140702c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f11140cfea0 tx=0x7f110c02bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 -- 172.21.15.125:0/3876422053 shutdown_connections 2024-01-16T14:59:34.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 --2- 172.21.15.125:0/3876422053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f11140702c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 --2- 172.21.15.125:0/3876422053 >> v2:172.21.15.137:3300/0 conn(0x7f1114143ae0 0x7f11140cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 -- 172.21.15.125:0/3876422053 >> 172.21.15.125:0/3876422053 conn(0x7f111406a800 msgr2=0x7f111406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 -- 172.21.15.125:0/3876422053 shutdown_connections 2024-01-16T14:59:34.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.287+0000 7f111c000640 1 -- 172.21.15.125:0/3876422053 wait complete. 2024-01-16T14:59:34.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.288+0000 7f111c000640 1 Processor -- start 2024-01-16T14:59:34.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 -- start start 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11140cf7d0 0x7f111413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1114143ae0 0x7f1114165680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114141b60 con 0x7f111415e3b0 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114140310 con 0x7f11140cf7d0 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.289+0000 7f111c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1114140f50 con 0x7f1114143ae0 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52244/0 (socket says 172.21.15.125:52244) 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 -- 172.21.15.125:0/491582331 learned_addr learned my addr 172.21.15.125:0/491582331 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:34.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f1119600640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11140cf7d0 0x7f111413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 -- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1114143ae0 msgr2=0x7f1114165680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1114143ae0 0x7f1114165680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 -- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11140cf7d0 msgr2=0x7f111413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11140cf7d0 0x7f111413fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 -- 172.21.15.125:0/491582331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c009a70 con 0x7f111415e3b0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.290+0000 7f111a000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e80 tx=0x7f110c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.291+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c040650 con 0x7f111415e3b0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.291+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c0407d0 con 0x7f111415e3b0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.291+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c040b80 con 0x7f111415e3b0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.291+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1114141230 con 0x7f111415e3b0 2024-01-16T14:59:34.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.291+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11141417f0 con 0x7f111415e3b0 2024-01-16T14:59:34.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.292+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1114141d10 con 0x7f111415e3b0 2024-01-16T14:59:34.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.293+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f110c040ce0 con 0x7f111415e3b0 2024-01-16T14:59:34.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.293+0000 7f1111600640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 0x7f10ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.294+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f110c0acaa0 con 0x7f111415e3b0 2024-01-16T14:59:34.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.294+0000 7f1119600640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 0x7f10ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.294+0000 7f1119600640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 0x7f10ec06d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1104006fd0 tx=0x7f1104008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.298+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1114141d10 con 0x7f111415e3b0 2024-01-16T14:59:34.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.475+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11140cbf50 con 0x7f111415e3b0 2024-01-16T14:59:34.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.477+0000 7f1111600640 1 -- 172.21.15.125:0/491582331 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f110c0778b0 con 0x7f111415e3b0 2024-01-16T14:59:34.458 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:34.458 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:33.670394+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/2265216525":"2024-01-17T14:59:30.648400+0000","172.21.15.125:6836/2265216525":"2024-01-17T14:59:30.648400+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:34.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 msgr2=0x7f10ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 0x7f10ec06d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1104006fd0 tx=0x7f1104008040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 msgr2=0x7f1114165bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e80 tx=0x7f110c004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 shutdown_connections 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f10ec06b500 0x7f10ec06d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111415e3b0 0x7f1114165bc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1114143ae0 0x7f1114165680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 --2- 172.21.15.125:0/491582331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11140cf7d0 0x7f111413fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 >> 172.21.15.125:0/491582331 conn(0x7f111406a800 msgr2=0x7f111406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 shutdown_connections 2024-01-16T14:59:34.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.480+0000 7f111c000640 1 -- 172.21.15.125:0/491582331 wait complete. 2024-01-16T14:59:34.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:34.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.684+0000 7f6308a00640 1 Processor -- start 2024-01-16T14:59:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.684+0000 7f6308a00640 1 -- start start 2024-01-16T14:59:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6308a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 0x7f630406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6308a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040b1e30 con 0x7f630415a690 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6308a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630415ab20 con 0x7f63040b3b20 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6308a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b20a0 con 0x7f630406ce40 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6301e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f630415a690 0x7f630406c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34314/0 (socket says 172.21.15.125:34314) 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.686+0000 7f6301e00640 1 -- 172.21.15.125:0/1817175537 learned_addr learned my addr 172.21.15.125:0/1817175537 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.687+0000 7f6303200640 1 --2- 172.21.15.125:0/1817175537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 0x7f630406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.688+0000 7f6303200640 1 -- 172.21.15.125:0/1817175537 >> v1:172.21.15.137:6789/0 conn(0x7f63040b3b20 legacy=0x7f63041505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.688+0000 7f6303200640 1 -- 172.21.15.125:0/1817175537 >> v1:172.21.15.125:6789/0 conn(0x7f630415a690 legacy=0x7f630406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.688+0000 7f6303200640 1 -- 172.21.15.125:0/1817175537 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040b2380 con 0x7f630406ce40 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.689+0000 7f6303200640 1 --2- 172.21.15.125:0/1817175537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 0x7f630406d240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009dc0 tx=0x7f62fc02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c9ff17bfed09db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6301400640 1 -- 172.21.15.125:0/1817175537 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc02ccd0 con 0x7f630406ce40 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6301400640 1 -- 172.21.15.125:0/1817175537 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc02ce30 con 0x7f630406ce40 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6301400640 1 -- 172.21.15.125:0/1817175537 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc034660 con 0x7f630406ce40 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 -- 172.21.15.125:0/1817175537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 msgr2=0x7f630406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 --2- 172.21.15.125:0/1817175537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 0x7f630406d240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009dc0 tx=0x7f62fc02bf20 comp rx=0 tx=0).stop 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 -- 172.21.15.125:0/1817175537 shutdown_connections 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 --2- 172.21.15.125:0/1817175537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630406ce40 0x7f630406d240 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 -- 172.21.15.125:0/1817175537 >> 172.21.15.125:0/1817175537 conn(0x7f630406a7d0 msgr2=0x7f630406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.691+0000 7f6308a00640 1 -- 172.21.15.125:0/1817175537 shutdown_connections 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.692+0000 7f6308a00640 1 -- 172.21.15.125:0/1817175537 wait complete. 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.693+0000 7f6308a00640 1 Processor -- start 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.693+0000 7f6308a00640 1 -- start start 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 0x7f6304158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630415a690 0x7f6304158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304146d40 con 0x7f630406ce40 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304071c70 con 0x7f63040b3b20 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6308a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304070690 con 0x7f630415a690 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.694+0000 7f6302800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52260/0 (socket says 172.21.15.125:52260) 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 -- 172.21.15.125:0/1889840343 learned_addr learned my addr 172.21.15.125:0/1889840343 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6301e00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 0x7f6304158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6303200640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630415a690 0x7f6304158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 -- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630415a690 msgr2=0x7f6304158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630415a690 0x7f6304158bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 -- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 msgr2=0x7f6304158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 0x7f6304158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 -- 172.21.15.125:0/1889840343 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc009a70 con 0x7f630406ce40 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6301e00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 0x7f6304158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f6302800640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f62f400c450 tx=0x7f62f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.695+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40052d0 con 0x7f630406ce40 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.696+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4005430 con 0x7f630406ce40 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.696+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4012870 con 0x7f630406ce40 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.696+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304070970 con 0x7f630406ce40 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.696+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304070f30 con 0x7f630406ce40 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.697+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304067450 con 0x7f630406ce40 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.698+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f62f40129d0 con 0x7f630406ce40 2024-01-16T14:59:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.698+0000 7f62faa00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 0x7f62dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.698+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f62f4089830 con 0x7f630406ce40 2024-01-16T14:59:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.699+0000 7f6301e00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 0x7f62dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.699+0000 7f6301e00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 0x7f62dc06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6304153f30 tx=0x7f62ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.703+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62f4054640 con 0x7f630406ce40 2024-01-16T14:59:34.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.877+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63040b3660 con 0x7f630406ce40 2024-01-16T14:59:34.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.878+0000 7f62faa00640 1 -- 172.21.15.125:0/1889840343 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62f4057c60 con 0x7f630406ce40 2024-01-16T14:59:34.858 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:59:34.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 msgr2=0x7f62dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 0x7f62dc06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6304153f30 tx=0x7f62ec009040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 msgr2=0x7f6304071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f62f400c450 tx=0x7f62f400c910 comp rx=0 tx=0).stop 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 shutdown_connections 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630415a690 0x7f6304158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63040b3b20 0x7f6304158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f62dc06b500 0x7f62dc06d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 --2- 172.21.15.125:0/1889840343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f630406ce40 0x7f6304071730 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.880+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 >> 172.21.15.125:0/1889840343 conn(0x7f630406a7d0 msgr2=0x7f630406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.881+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 shutdown_connections 2024-01-16T14:59:34.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:34.881+0000 7f6308a00640 1 -- 172.21.15.125:0/1889840343 wait complete. 2024-01-16T14:59:34.879 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:59:35.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.099+0000 7f39f9400640 1 Processor -- start 2024-01-16T14:59:35.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.100+0000 7f39f9400640 1 -- start start 2024-01-16T14:59:35.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f39f409a170 0x7f39f40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 0x7f39f40cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f9400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f406cea0 con 0x7f39f40cd240 2024-01-16T14:59:35.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f9400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f406cbc0 con 0x7f39f409a170 2024-01-16T14:59:35.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f406f350 con 0x7f39f406c6b0 2024-01-16T14:59:35.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f2800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f39f40cd240 0x7f39f4099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34316/0 (socket says 172.21.15.125:34316) 2024-01-16T14:59:35.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.101+0000 7f39f2800640 1 -- 172.21.15.125:0/763008811 learned_addr learned my addr 172.21.15.125:0/763008811 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:35.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113486145 0 0) 0x7f39f406cea0 con 0x7f39f40cd240 2024-01-16T14:59:35.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d4003520 con 0x7f39f40cd240 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1e00640 1 --2- 172.21.15.125:0/763008811 >> v2:172.21.15.137:3300/0 conn(0x7f39f409a170 0x7f39f40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f3200640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 0x7f39f40cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1194981631 0 0) 0x7f39d4003520 con 0x7f39f40cd240 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 msgr2=0x7f39f40cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 0x7f39f40cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 >> v2:172.21.15.137:3300/0 conn(0x7f39f409a170 msgr2=0x7f39f40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 --2- 172.21.15.125:0/763008811 >> v2:172.21.15.137:3300/0 conn(0x7f39f409a170 0x7f39f40b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f406f630 con 0x7f39f40cd240 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f39e40019f0 con 0x7f39f40cd240 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 >> v1:172.21.15.125:6789/0 conn(0x7f39f40cd240 legacy=0x7f39f4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 0x7f39d4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d4004d00 0x7f39d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f406cbc0 con 0x7f39d4003fa0 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f406f350 con 0x7f39d4004d00 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.102+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d400ba80 con 0x7f39f406c6b0 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 -- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 msgr2=0x7f39d4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f406c6b0 0x7f39d4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 -- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d4004d00 msgr2=0x7f39d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d4004d00 0x7f39d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.103+0000 7f39f1e00640 1 -- 172.21.15.125:0/763008811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4003520 con 0x7f39d4003fa0 2024-01-16T14:59:35.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.104+0000 7f39f1e00640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39d40047c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ba30 tx=0x7f39d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8355fda6f846c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.104+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc032680 con 0x7f39d4003fa0 2024-01-16T14:59:35.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.104+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc0327e0 con 0x7f39d4003fa0 2024-01-16T14:59:35.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.104+0000 7f39f1400640 1 -- 172.21.15.125:0/763008811 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc032b70 con 0x7f39d4003fa0 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 -- 172.21.15.125:0/763008811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 msgr2=0x7f39d40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 --2- 172.21.15.125:0/763008811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39d40047c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ba30 tx=0x7f39d40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f3200640 1 -- 172.21.15.125:0/763008811 reap_dead start 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 -- 172.21.15.125:0/763008811 shutdown_connections 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 -- 172.21.15.125:0/763008811 >> 172.21.15.125:0/763008811 conn(0x7f39f40b2d00 msgr2=0x7f39f40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 -- 172.21.15.125:0/763008811 shutdown_connections 2024-01-16T14:59:35.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.105+0000 7f39f9400640 1 -- 172.21.15.125:0/763008811 wait complete. 2024-01-16T14:59:35.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.107+0000 7f39f9400640 1 Processor -- start 2024-01-16T14:59:35.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.107+0000 7f39f9400640 1 -- start start 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.107+0000 7f39f9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.107+0000 7f39f9400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39f40b50b0 0x7f39f41598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.107+0000 7f39f9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f40c84a0 0x7f39f40c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f9400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40b5530 con 0x7f39d4003fa0 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f9400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40c8e60 con 0x7f39f40b50b0 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f409f030 con 0x7f39f40c84a0 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52282/0 (socket says 172.21.15.125:52282) 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 -- 172.21.15.125:0/3688730281 learned_addr learned my addr 172.21.15.125:0/3688730281 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 -- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f40c84a0 msgr2=0x7f39f40c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f40c84a0 0x7f39f40c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 -- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39f40b50b0 msgr2=0x7f39f41598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39f40b50b0 0x7f39f41598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 -- 172.21.15.125:0/3688730281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f40b5740 con 0x7f39d4003fa0 2024-01-16T14:59:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.108+0000 7f39f2800640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cb10 tx=0x7f39e400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.109+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400a7a0 con 0x7f39d4003fa0 2024-01-16T14:59:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.109+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e400a920 con 0x7f39d4003fa0 2024-01-16T14:59:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.109+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400acd0 con 0x7f39d4003fa0 2024-01-16T14:59:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.109+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f40b5a20 con 0x7f39d4003fa0 2024-01-16T14:59:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.109+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f40b5fe0 con 0x7f39d4003fa0 2024-01-16T14:59:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.110+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f40675d0 con 0x7f39d4003fa0 2024-01-16T14:59:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.111+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f39e4007150 con 0x7f39d4003fa0 2024-01-16T14:59:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.112+0000 7f39eaa00640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 0x7f39cc06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.112+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f39e4089400 con 0x7f39d4003fa0 2024-01-16T14:59:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.112+0000 7f39f1e00640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 0x7f39cc06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.112+0000 7f39f1e00640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 0x7f39cc06d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ba00 tx=0x7f39dc032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.116+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39e40542a0 con 0x7f39d4003fa0 2024-01-16T14:59:35.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.315+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f39f40aff90 con 0x7f39d4003fa0 2024-01-16T14:59:35.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.316+0000 7f39eaa00640 1 -- 172.21.15.125:0/3688730281 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+124 (secure 0 0 0) 0x7f39e4018090 con 0x7f39d4003fa0 2024-01-16T14:59:35.296 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2265216525 2024-01-17T14:59:30.648400+0000 2024-01-16T14:59:35.296 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/2265216525 2024-01-17T14:59:30.648400+0000 2024-01-16T14:59:35.297 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 msgr2=0x7f39cc06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 0x7f39cc06d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ba00 tx=0x7f39dc032000 comp rx=0 tx=0).stop 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 msgr2=0x7f39f40ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cb10 tx=0x7f39e400a060 comp rx=0 tx=0).stop 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 shutdown_connections 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39f40c84a0 0x7f39f40c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39f40b50b0 0x7f39f41598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39cc06b530 0x7f39cc06d9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 --2- 172.21.15.125:0/3688730281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d4003fa0 0x7f39f40ca490 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.318+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 >> 172.21.15.125:0/3688730281 conn(0x7f39f40b2d00 msgr2=0x7f39f40cfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.319+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 shutdown_connections 2024-01-16T14:59:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.319+0000 7f39f9400640 1 -- 172.21.15.125:0/3688730281 wait complete. 2024-01-16T14:59:35.317 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T14:59:35.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:35.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:35.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[50138]: fuse finished with error 0 and tester_r 0 2024-01-16T14:59:35.408 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:59:35.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:35.409 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T14:59:35.631 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:59:35.631 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:35.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:35.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.923+0000 7ff496a00640 1 Processor -- start 2024-01-16T14:59:35.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.923+0000 7ff496a00640 1 -- start start 2024-01-16T14:59:35.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.924+0000 7ff496a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff490143ae0 0x7ff4900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.924+0000 7ff496a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.924+0000 7ff496a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4900cfc30 con 0x7ff49015e1e0 2024-01-16T14:59:35.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff496a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4900b1ee0 con 0x7ff490143ae0 2024-01-16T14:59:35.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff496a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900b2780 con 0x7ff4900cf7d0 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff48fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff490143ae0 0x7ff4900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff487e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff48fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff490143ae0 0x7ff4900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:57350/0 (socket says 172.21.15.125:57350) 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff48fe00640 1 -- 172.21.15.125:0/2899052986 learned_addr learned my addr 172.21.15.125:0/2899052986 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.925+0000 7ff494a00640 1 --1- 172.21.15.125:0/2899052986 >> v1:172.21.15.137:6789/0 conn(0x7ff49015e1e0 0x7ff49015a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:50418/0 (socket says 172.21.15.125:50418) 2024-01-16T14:59:35.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.926+0000 7ff48fe00640 1 -- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 msgr2=0x7ff49015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.926+0000 7ff48fe00640 1 --2- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.926+0000 7ff48fe00640 1 -- 172.21.15.125:0/2899052986 >> v1:172.21.15.137:6789/0 conn(0x7ff49015e1e0 legacy=0x7ff49015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.926+0000 7ff48fe00640 1 -- 172.21.15.125:0/2899052986 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49015e990 con 0x7ff490143ae0 2024-01-16T14:59:35.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.927+0000 7ff48fe00640 1 --2- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff4900cf290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48002db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93ebb1ebde2f957e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.928+0000 7ff48f400640 1 -- 172.21.15.125:0/2899052986 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48002e940 con 0x7ff490143ae0 2024-01-16T14:59:35.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.928+0000 7ff48f400640 1 -- 172.21.15.125:0/2899052986 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48002eaa0 con 0x7ff490143ae0 2024-01-16T14:59:35.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.928+0000 7ff48f400640 1 -- 172.21.15.125:0/2899052986 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48002ee30 con 0x7ff490143ae0 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 -- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 msgr2=0x7ff4900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 --2- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff4900cf290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48002db90 comp rx=0 tx=0).stop 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 -- 172.21.15.125:0/2899052986 shutdown_connections 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 --2- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 --2- 172.21.15.125:0/2899052986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff4900cf290 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 -- 172.21.15.125:0/2899052986 >> 172.21.15.125:0/2899052986 conn(0x7ff49006a800 msgr2=0x7ff49006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 -- 172.21.15.125:0/2899052986 shutdown_connections 2024-01-16T14:59:35.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.929+0000 7ff496a00640 1 -- 172.21.15.125:0/2899052986 wait complete. 2024-01-16T14:59:35.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.930+0000 7ff496a00640 1 Processor -- start 2024-01-16T14:59:35.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.931+0000 7ff496a00640 1 -- start start 2024-01-16T14:59:35.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.931+0000 7ff496a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49013e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.931+0000 7ff496a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff49015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff496a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff496a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900ab870 con 0x7ff49015e1e0 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff496a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900cd4d0 con 0x7ff490143ae0 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff496a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900cd7b0 con 0x7ff4900cf7d0 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52288/0 (socket says 172.21.15.125:52288) 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 -- 172.21.15.125:0/457249350 learned_addr learned my addr 172.21.15.125:0/457249350 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff48fe00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49013e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff487e00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff49015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 -- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 msgr2=0x7ff49013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49013e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 -- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 msgr2=0x7ff49015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff49015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.932+0000 7ff494a00640 1 -- 172.21.15.125:0/457249350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4800094d0 con 0x7ff49015e1e0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff494a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c730 tx=0x7ff48800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800ce40 con 0x7ff49015e1e0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48800a640 con 0x7ff49015e1e0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4900cda90 con 0x7ff49015e1e0 2024-01-16T14:59:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800a9f0 con 0x7ff49015e1e0 2024-01-16T14:59:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.933+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49013eeb0 con 0x7ff49015e1e0 2024-01-16T14:59:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.934+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4900ab640 con 0x7ff49015e1e0 2024-01-16T14:59:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.935+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff48800ab50 con 0x7ff49015e1e0 2024-01-16T14:59:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.936+0000 7ff48cc00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 0x7ff46806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.936+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff488089330 con 0x7ff49015e1e0 2024-01-16T14:59:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.936+0000 7ff48fe00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 0x7ff46806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.937+0000 7ff48fe00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 0x7ff46806d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff480038000 tx=0x7ff480032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:35.941+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4900ab640 con 0x7ff49015e1e0 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.133+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4900b2bf0 con 0x7ff49015e1e0 2024-01-16T14:59:36.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.135+0000 7ff48cc00640 1 -- 172.21.15.125:0/457249350 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13036 (secure 0 0 0) 0x7ff4900b2bf0 con 0x7ff49015e1e0 2024-01-16T14:59:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:36.116 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:33.670394+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/2265216525":"2024-01-17T14:59:30.648400+0000","172.21.15.125:6836/2265216525":"2024-01-17T14:59:30.648400+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:36.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.137+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 msgr2=0x7ff46806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 0x7ff46806d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff480038000 tx=0x7ff480032000 comp rx=0 tx=0).stop 2024-01-16T14:59:36.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 msgr2=0x7ff49015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c730 tx=0x7ff48800cbf0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 shutdown_connections 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff46806b530 0x7ff46806d9f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49015e1e0 0x7ff49015bee0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff490143ae0 0x7ff49015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 --2- 172.21.15.125:0/457249350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4900cf7d0 0x7ff49013e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 >> 172.21.15.125:0/457249350 conn(0x7ff49006a800 msgr2=0x7ff4900b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 shutdown_connections 2024-01-16T14:59:36.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.138+0000 7ff496a00640 1 -- 172.21.15.125:0/457249350 wait complete. 2024-01-16T14:59:36.138 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:36.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:36.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.367+0000 7f0e72e00640 1 Processor -- start 2024-01-16T14:59:36.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.367+0000 7f0e72e00640 1 -- start start 2024-01-16T14:59:36.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e6c0adf20 0x7f0e6c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0e6c15bf10 0x7f0e6c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0cc980 0x7f0e6c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c15f970 con 0x7f0e6c0adf20 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c15f4e0 con 0x7f0e6c15bf10 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0cce10 con 0x7f0e6c0cc980 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e71a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e6c0adf20 0x7f0e6c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e71a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e6c0adf20 0x7f0e6c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52300/0 (socket says 172.21.15.125:52300) 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e71a00640 1 -- 172.21.15.125:0/1624002759 learned_addr learned my addr 172.21.15.125:0/1624002759 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.369+0000 7f0e72400640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0cc980 0x7f0e6c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71000640 1 --2- 172.21.15.125:0/1624002759 >> v2:172.21.15.137:3300/0 conn(0x7f0e6c15bf10 0x7f0e6c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71a00640 1 -- 172.21.15.125:0/1624002759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0cc980 msgr2=0x7f0e6c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71a00640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0cc980 0x7f0e6c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71a00640 1 -- 172.21.15.125:0/1624002759 >> v2:172.21.15.137:3300/0 conn(0x7f0e6c15bf10 msgr2=0x7f0e6c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71a00640 1 --2- 172.21.15.125:0/1624002759 >> v2:172.21.15.137:3300/0 conn(0x7f0e6c15bf10 0x7f0e6c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.370+0000 7f0e71a00640 1 -- 172.21.15.125:0/1624002759 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c071130 con 0x7f0e6c0adf20 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e71a00640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0adf20 0x7f0e6c070a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c15e9e0 tx=0x7f0e5c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9276876dfd92d193 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e6be00640 1 -- 172.21.15.125:0/1624002759 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c0317d0 con 0x7f0e6c0adf20 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e6be00640 1 -- 172.21.15.125:0/1624002759 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c031930 con 0x7f0e6c0adf20 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e6be00640 1 -- 172.21.15.125:0/1624002759 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c031cc0 con 0x7f0e6c0adf20 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e72e00640 1 -- 172.21.15.125:0/1624002759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0adf20 msgr2=0x7f0e6c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.371+0000 7f0e72e00640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0adf20 0x7f0e6c070a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c15e9e0 tx=0x7f0e5c02b870 comp rx=0 tx=0).stop 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 -- 172.21.15.125:0/1624002759 shutdown_connections 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0cc980 0x7f0e6c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 --2- 172.21.15.125:0/1624002759 >> v2:172.21.15.137:3300/0 conn(0x7f0e6c15bf10 0x7f0e6c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 --2- 172.21.15.125:0/1624002759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0adf20 0x7f0e6c070a30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 -- 172.21.15.125:0/1624002759 >> 172.21.15.125:0/1624002759 conn(0x7f0e6c06a7d0 msgr2=0x7f0e6c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 -- 172.21.15.125:0/1624002759 shutdown_connections 2024-01-16T14:59:36.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.372+0000 7f0e72e00640 1 -- 172.21.15.125:0/1624002759 wait complete. 2024-01-16T14:59:36.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.373+0000 7f0e72e00640 1 Processor -- start 2024-01-16T14:59:36.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 -- start start 2024-01-16T14:59:36.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0adf20 0x7f0e6c0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e6c15bf10 0x7f0e6c15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c13a050 con 0x7f0e6c0cc980 2024-01-16T14:59:36.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c13a2c0 con 0x7f0e6c15bf10 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.374+0000 7f0e72e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c13a5a0 con 0x7f0e6c0adf20 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52304/0 (socket says 172.21.15.125:52304) 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 -- 172.21.15.125:0/3014561688 learned_addr learned my addr 172.21.15.125:0/3014561688 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 -- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0adf20 msgr2=0x7f0e6c0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0adf20 0x7f0e6c0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 -- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e6c15bf10 msgr2=0x7f0e6c15cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e6c15bf10 0x7f0e6c15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 -- 172.21.15.125:0/3014561688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0094d0 con 0x7f0e6c0cc980 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.375+0000 7f0e71000640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400d800 tx=0x7f0e5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.376+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400b6c0 con 0x7f0e6c0cc980 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.376+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5400b820 con 0x7f0e6c0cc980 2024-01-16T14:59:36.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.376+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400bbb0 con 0x7f0e6c0cc980 2024-01-16T14:59:36.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.376+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c13a880 con 0x7f0e6c0cc980 2024-01-16T14:59:36.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.376+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c15a530 con 0x7f0e6c0cc980 2024-01-16T14:59:36.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.377+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c067450 con 0x7f0e6c0cc980 2024-01-16T14:59:36.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.378+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0e5400bd10 con 0x7f0e6c0cc980 2024-01-16T14:59:36.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.379+0000 7f0e69600640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 0x7f0e4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.379+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0e5408a330 con 0x7f0e6c0cc980 2024-01-16T14:59:36.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.379+0000 7f0e71a00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 0x7f0e4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.380+0000 7f0e71a00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 0x7f0e4406d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c036000 tx=0x7f0e5c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.383+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e540550a0 con 0x7f0e6c0cc980 2024-01-16T14:59:36.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.573+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e6c144790 con 0x7f0e6c0cc980 2024-01-16T14:59:36.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.574+0000 7f0e69600640 1 -- 172.21.15.125:0/3014561688 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e540586c0 con 0x7f0e6c0cc980 2024-01-16T14:59:36.555 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:59:36.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 msgr2=0x7f0e4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 0x7f0e4406d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c036000 tx=0x7f0e5c030000 comp rx=0 tx=0).stop 2024-01-16T14:59:36.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 msgr2=0x7f0e6c15c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400d800 tx=0x7f0e5400dcc0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 shutdown_connections 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e6c15bf10 0x7f0e6c15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0e4406b500 0x7f0e4406d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e6c0cc980 0x7f0e6c15c5a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 --2- 172.21.15.125:0/3014561688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e6c0adf20 0x7f0e6c0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 >> 172.21.15.125:0/3014561688 conn(0x7f0e6c06a7d0 msgr2=0x7f0e6c1432c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 shutdown_connections 2024-01-16T14:59:36.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.577+0000 7f0e72e00640 1 -- 172.21.15.125:0/3014561688 wait complete. 2024-01-16T14:59:36.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.809+0000 7f129e600640 1 Processor -- start 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.809+0000 7f129e600640 1 -- start start 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f129e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f129806a3e0 0x7f12980ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f129e600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12980aede0 0x7f129806f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f129e600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1298071de0 con 0x7f12980d0b60 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f129e600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12980af270 con 0x7f12980aede0 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f129e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1298072030 con 0x7f129806a3e0 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f1296a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f12980d0b60 0x7f129806a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57404/0 (socket says 172.21.15.125:57404) 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.811+0000 7f1296a00640 1 -- 172.21.15.125:0/527747022 learned_addr learned my addr 172.21.15.125:0/527747022 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297e00640 1 --2- 172.21.15.125:0/527747022 >> v2:172.21.15.125:3300/0 conn(0x7f12980aede0 0x7f129806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297400640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f129806a3e0 0x7f12980ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1296000640 1 -- 172.21.15.125:0/527747022 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562831992 0 0) 0x7f1298071de0 con 0x7f12980d0b60 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1296000640 1 -- 172.21.15.125:0/527747022 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f12980d0b60 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297e00640 1 -- 172.21.15.125:0/527747022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f129806a3e0 msgr2=0x7f12980ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297e00640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f129806a3e0 0x7f12980ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297e00640 1 -- 172.21.15.125:0/527747022 >> v1:172.21.15.125:6789/0 conn(0x7f12980d0b60 legacy=0x7f129806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.812+0000 7f1297e00640 1 -- 172.21.15.125:0/527747022 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298072310 con 0x7f12980aede0 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.813+0000 7f1297e00640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12980aede0 0x7f129806f080 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f12900097a0 tx=0x7f129002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d55966866e13dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f1296000640 1 -- 172.21.15.125:0/527747022 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290031830 con 0x7f12980aede0 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f1296000640 1 -- 172.21.15.125:0/527747022 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290031990 con 0x7f12980aede0 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f1296000640 1 -- 172.21.15.125:0/527747022 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290031d20 con 0x7f12980aede0 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 -- 172.21.15.125:0/527747022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12980aede0 msgr2=0x7f129806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12980aede0 0x7f129806f080 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f12900097a0 tx=0x7f129002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 -- 172.21.15.125:0/527747022 shutdown_connections 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12980aede0 0x7f129806f080 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 --2- 172.21.15.125:0/527747022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f129806a3e0 0x7f12980ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 -- 172.21.15.125:0/527747022 >> 172.21.15.125:0/527747022 conn(0x7f12980cfef0 msgr2=0x7f12980d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 -- 172.21.15.125:0/527747022 shutdown_connections 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.814+0000 7f129e600640 1 -- 172.21.15.125:0/527747022 wait complete. 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.816+0000 7f129e600640 1 Processor -- start 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.816+0000 7f129e600640 1 -- start start 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12980aede0 0x7f12980cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12980d0b60 0x7f12980cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12980ce1f0 con 0x7f129806a3e0 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12980cd5e0 con 0x7f12980d0b60 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.817+0000 7f129e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12980cd8c0 con 0x7f12980aede0 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36216/0 (socket says 172.21.15.125:36216) 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 -- 172.21.15.125:0/2805777663 learned_addr learned my addr 172.21.15.125:0/2805777663 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1296a00640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12980aede0 0x7f12980cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 -- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12980aede0 msgr2=0x7f12980cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297e00640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12980d0b60 0x7f12980cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12980aede0 0x7f12980cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 -- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12980d0b60 msgr2=0x7f12980cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12980d0b60 0x7f12980cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 -- 172.21.15.125:0/2805777663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290009400 con 0x7f129806a3e0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.818+0000 7f1297400640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1288007e40 tx=0x7f128800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.819+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288014620 con 0x7f129806a3e0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.819+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128800ee90 con 0x7f129806a3e0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.819+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12980cdba0 con 0x7f129806a3e0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.819+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12880149c0 con 0x7f129806a3e0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.819+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1298150fd0 con 0x7f129806a3e0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.820+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1298152360 con 0x7f129806a3e0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.821+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1288014b20 con 0x7f129806a3e0 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.822+0000 7f128f400640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 0x7f127006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.822+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f128808a4f0 con 0x7f129806a3e0 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.822+0000 7f1296a00640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 0x7f127006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.823+0000 7f1296a00640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 0x7f127006d9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12981f6350 tx=0x7f1280009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:36.828+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12880553d0 con 0x7f129806a3e0 2024-01-16T14:59:36.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:36.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:37.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.035+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f12980ce330 con 0x7f129806a3e0 2024-01-16T14:59:37.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.036+0000 7f128f400640 1 -- 172.21.15.125:0/2805777663 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+124 (secure 0 0 0) 0x7f12880589f0 con 0x7f129806a3e0 2024-01-16T14:59:37.017 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2265216525 2024-01-17T14:59:30.648400+0000 2024-01-16T14:59:37.017 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/2265216525 2024-01-17T14:59:30.648400+0000 2024-01-16T14:59:37.017 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T14:59:37.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 msgr2=0x7f127006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 0x7f127006d9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12981f6350 tx=0x7f1280009040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 msgr2=0x7f1298150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1288007e40 tx=0x7f128800e4b0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 shutdown_connections 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12980d0b60 0x7f12980cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12980aede0 0x7f12980cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f127006b530 0x7f127006d9f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 --2- 172.21.15.125:0/2805777663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f129806a3e0 0x7f1298150a90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 >> 172.21.15.125:0/2805777663 conn(0x7f12980cfef0 msgr2=0x7f12981513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 shutdown_connections 2024-01-16T14:59:37.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.039+0000 7f129e600640 1 -- 172.21.15.125:0/2805777663 wait complete. 2024-01-16T14:59:37.038 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T14:59:37.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:37.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:37.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42954]: fuse finished with error 0 and tester_r 0 2024-01-16T14:59:37.137 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:59:37.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:37.137 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T14:59:37.327 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:59:37.327 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:37.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T14:59:37.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T14:59:37.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:37.386 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T14:59:37.437 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T14:59:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:37.438 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:59:37.438 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:37.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T14:59:37.495 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T14:59:37.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:37.495 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T14:59:37.547 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T14:59:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:37.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:37.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.784+0000 7f0ac5000640 1 Processor -- start 2024-01-16T14:59:37.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.785+0000 7f0ac5000640 1 -- start start 2024-01-16T14:59:37.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0ac5000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ac00ae5a0 0x7f0ac015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0ac5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 0x7f0ac006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0ac5000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac0070c40 con 0x7f0ac006d120 2024-01-16T14:59:37.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0ac5000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac0071450 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0ac5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac006c470 con 0x7f0ac00b3ae0 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0abde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ac00ae5a0 0x7f0ac015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0abe800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0ac006d120 0x7f0ac00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57416/0 (socket says 172.21.15.125:57416) 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0abde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ac00ae5a0 0x7f0ac015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36232/0 (socket says 172.21.15.125:36232) 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.786+0000 7f0abe800640 1 -- 172.21.15.125:0/607935616 learned_addr learned my addr 172.21.15.125:0/607935616 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0ab7e00640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 0x7f0ac006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0abde00640 1 -- 172.21.15.125:0/607935616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 msgr2=0x7f0ac006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0abde00640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 0x7f0ac006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0abde00640 1 -- 172.21.15.125:0/607935616 >> v1:172.21.15.125:6789/0 conn(0x7f0ac006d120 legacy=0x7f0ac00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0abde00640 1 -- 172.21.15.125:0/607935616 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac006c750 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0ab7e00640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 0x7f0ac006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:37.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.787+0000 7f0abd400640 1 -- 172.21.15.125:0/607935616 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303234084 0 0) 0x7f0ac0070c40 con 0x7f0ac006d120 2024-01-16T14:59:37.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.788+0000 7f0abde00640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac015e430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00097a0 tx=0x7f0ab002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01fe3e21ad736c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.788+0000 7f0abd400640 1 -- 172.21.15.125:0/607935616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab002d040 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.788+0000 7f0abd400640 1 -- 172.21.15.125:0/607935616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0031810 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.788+0000 7f0abd400640 1 -- 172.21.15.125:0/607935616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0031ba0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 -- 172.21.15.125:0/607935616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 msgr2=0x7f0ac015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac015e430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00097a0 tx=0x7f0ab002b8b0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 -- 172.21.15.125:0/607935616 shutdown_connections 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00b3ae0 0x7f0ac006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 --2- 172.21.15.125:0/607935616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac015e430 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 -- 172.21.15.125:0/607935616 >> 172.21.15.125:0/607935616 conn(0x7f0ac006a800 msgr2=0x7f0ac006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 -- 172.21.15.125:0/607935616 shutdown_connections 2024-01-16T14:59:37.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.789+0000 7f0ac5000640 1 -- 172.21.15.125:0/607935616 wait complete. 2024-01-16T14:59:37.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.791+0000 7f0ac5000640 1 Processor -- start 2024-01-16T14:59:37.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.791+0000 7f0ac5000640 1 -- start start 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac006d120 0x7f0ac01f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac00b3ae0 0x7f0ac00ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac015bed0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac015b2e0 con 0x7f0ac00b3ae0 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ac5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01472f0 con 0x7f0ac006d120 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36248/0 (socket says 172.21.15.125:36248) 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 -- 172.21.15.125:0/3152257039 learned_addr learned my addr 172.21.15.125:0/3152257039 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0abde00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac006d120 0x7f0ac01f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 -- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac006d120 msgr2=0x7f0ac01f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac006d120 0x7f0ac01f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 -- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac00b3ae0 msgr2=0x7f0ac00ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac00b3ae0 0x7f0ac00ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.792+0000 7f0ab7e00640 1 -- 172.21.15.125:0/3152257039 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0009400 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ab7e00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c7f0 tx=0x7f0aa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8007820 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8007980 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8007cf0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac015b530 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.793+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac015baf0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.794+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac006cbc0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.795+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0aa80113f0 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.796+0000 7f0ab6000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.796+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0aa808a080 con 0x7f0ac00ae5a0 2024-01-16T14:59:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.796+0000 7f0abde00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.797+0000 7f0abde00640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 0x7f0a9806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:37.800+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0aa8054e10 con 0x7f0ac00ae5a0 2024-01-16T14:59:38.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.027+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ac00675d0 con 0x7f0ac00ae5a0 2024-01-16T14:59:38.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.028+0000 7f0ab6000640 1 -- 172.21.15.125:0/3152257039 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f0aa801fe90 con 0x7f0ac00ae5a0 2024-01-16T14:59:38.009 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:38.009 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1608829453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1608829453},{"type":"v1","addr":"172.21.15.137:6835","nonce":1608829453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4798,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1479548002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1479548002},{"type":"v1","addr":"172.21.15.137:6833","nonce":1479548002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1004786444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1004786444},{"type":"v1","addr":"172.21.15.125:6835","nonce":1004786444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/181156614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":181156614},{"type":"v1","addr":"172.21.15.125:6837","nonce":181156614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-01-16T14:59:38.009 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 21 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 msgr2=0x7f0a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 0x7f0a9806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 msgr2=0x7f0ac0146db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c7f0 tx=0x7f0aa800ccb0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 shutdown_connections 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac00b3ae0 0x7f0ac00ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ac00ae5a0 0x7f0ac0146db0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 --2- 172.21.15.125:0/3152257039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac006d120 0x7f0ac01f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.031+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 >> 172.21.15.125:0/3152257039 conn(0x7f0ac006a800 msgr2=0x7f0ac01587c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.032+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 shutdown_connections 2024-01-16T14:59:38.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.032+0000 7f0ac5000640 1 -- 172.21.15.125:0/3152257039 wait complete. 2024-01-16T14:59:38.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T14:59:38.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.251+0000 7fe889800640 1 Processor -- start 2024-01-16T14:59:38.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.252+0000 7fe889800640 1 -- start start 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.253+0000 7fe889800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe88413df80 0x7fe88415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.253+0000 7fe889800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88406c470 con 0x7fe8840cf9a0 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.253+0000 7fe889800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88406c750 con 0x7fe88413e360 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.253+0000 7fe889800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88406ca30 con 0x7fe88413df80 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.253+0000 7fe882a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe88413df80 0x7fe88415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe883400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe8840cf9a0 0x7fe8840cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57424/0 (socket says 172.21.15.125:57424) 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe882a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe88413df80 0x7fe88415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36260/0 (socket says 172.21.15.125:36260) 2024-01-16T14:59:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe883400640 1 -- 172.21.15.125:0/277859393 learned_addr learned my addr 172.21.15.125:0/277859393 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe882a00640 1 -- 172.21.15.125:0/277859393 >> v1:172.21.15.137:6789/0 conn(0x7fe88413e360 legacy=0x7fe8840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe882a00640 1 -- 172.21.15.125:0/277859393 >> v1:172.21.15.125:6789/0 conn(0x7fe8840cf9a0 legacy=0x7fe8840cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe882a00640 1 -- 172.21.15.125:0/277859393 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88406cd10 con 0x7fe88413df80 2024-01-16T14:59:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.254+0000 7fe882000640 1 -- 172.21.15.125:0/277859393 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84684570 0 0) 0x7fe88406c470 con 0x7fe8840cf9a0 2024-01-16T14:59:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.255+0000 7fe882a00640 1 --2- 172.21.15.125:0/277859393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413df80 0x7fe88415bee0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097a0 tx=0x7fe87402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c24c1055166f0cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe882000640 1 -- 172.21.15.125:0/277859393 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874031830 con 0x7fe88413df80 2024-01-16T14:59:38.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe882000640 1 -- 172.21.15.125:0/277859393 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe874031990 con 0x7fe88413df80 2024-01-16T14:59:38.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe882000640 1 -- 172.21.15.125:0/277859393 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874031d20 con 0x7fe88413df80 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 -- 172.21.15.125:0/277859393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413df80 msgr2=0x7fe88415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 --2- 172.21.15.125:0/277859393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413df80 0x7fe88415bee0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097a0 tx=0x7fe87402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 -- 172.21.15.125:0/277859393 shutdown_connections 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 --2- 172.21.15.125:0/277859393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413df80 0x7fe88415bee0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 -- 172.21.15.125:0/277859393 >> 172.21.15.125:0/277859393 conn(0x7fe88406a800 msgr2=0x7fe88406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 -- 172.21.15.125:0/277859393 shutdown_connections 2024-01-16T14:59:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.256+0000 7fe889800640 1 -- 172.21.15.125:0/277859393 wait complete. 2024-01-16T14:59:38.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.258+0000 7fe889800640 1 Processor -- start 2024-01-16T14:59:38.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.258+0000 7fe889800640 1 -- start start 2024-01-16T14:59:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8840cf9a0 0x7fe8840b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 0x7fe8840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88414abe0 con 0x7fe88413e360 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88414ae30 con 0x7fe88413df80 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe889800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88414b110 con 0x7fe8840cf9a0 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe883400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe883400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36276/0 (socket says 172.21.15.125:36276) 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.259+0000 7fe883400640 1 -- 172.21.15.125:0/2946848927 learned_addr learned my addr 172.21.15.125:0/2946848927 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe882a00640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8840cf9a0 0x7fe8840b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe879e00640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 0x7fe8840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 -- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8840cf9a0 msgr2=0x7fe8840b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8840cf9a0 0x7fe8840b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 -- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 msgr2=0x7fe8840b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 0x7fe8840b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 -- 172.21.15.125:0/2946848927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe874009400 con 0x7fe88413e360 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe879e00640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 0x7fe8840b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe883400640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00c700 tx=0x7fe87c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.260+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00ce10 con 0x7fe88413e360 2024-01-16T14:59:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.261+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00a640 con 0x7fe88413e360 2024-01-16T14:59:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.261+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88414b3f0 con 0x7fe88413e360 2024-01-16T14:59:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.261+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00a9f0 con 0x7fe88413e360 2024-01-16T14:59:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.261+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88415a2e0 con 0x7fe88413e360 2024-01-16T14:59:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.262+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88414b910 con 0x7fe88413e360 2024-01-16T14:59:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.263+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe87c00ab50 con 0x7fe88413e360 2024-01-16T14:59:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.263+0000 7fe87b400640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 0x7fe85c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.264+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe87c0894a0 con 0x7fe88413e360 2024-01-16T14:59:38.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.264+0000 7fe882a00640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 0x7fe85c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.264+0000 7fe882a00640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 0x7fe85c06d9f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe874036000 tx=0x7fe87403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.268+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe88414b910 con 0x7fe88413e360 2024-01-16T14:59:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.458+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fe88414b910 con 0x7fe88413e360 2024-01-16T14:59:38.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.464+0000 7fe87b400640 1 -- 172.21.15.125:0/2946848927 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe88414b910 con 0x7fe88413e360 2024-01-16T14:59:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 msgr2=0x7fe85c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 0x7fe85c06d9f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe874036000 tx=0x7fe87403c040 comp rx=0 tx=0).stop 2024-01-16T14:59:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 msgr2=0x7fe88414bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00c700 tx=0x7fe87c00cbc0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 shutdown_connections 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe85c06b530 0x7fe85c06d9f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe88413e360 0x7fe88414bbf0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe88413df80 0x7fe8840b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.467+0000 7fe889800640 1 --2- 172.21.15.125:0/2946848927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8840cf9a0 0x7fe8840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.468+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 >> 172.21.15.125:0/2946848927 conn(0x7fe88406a800 msgr2=0x7fe8840aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.468+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 shutdown_connections 2024-01-16T14:59:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.468+0000 7fe889800640 1 -- 172.21.15.125:0/2946848927 wait complete. 2024-01-16T14:59:38.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T14:59:38.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.695+0000 7f2741000640 1 Processor -- start 2024-01-16T14:59:38.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.695+0000 7f2741000640 1 -- start start 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2741000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f273c0b3ae0 0x7f273c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2741000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0cf290 0x7f273c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2741000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c070c40 con 0x7f273c0ae5a0 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2741000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273c071450 con 0x7f273c0b3ae0 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2741000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0cfc60 con 0x7f273c0cf290 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2739400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f273c0b3ae0 0x7f273c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2739400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f273c0b3ae0 0x7f273c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36288/0 (socket says 172.21.15.125:36288) 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.697+0000 7f2739400640 1 -- 172.21.15.125:0/453844701 learned_addr learned my addr 172.21.15.125:0/453844701 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.698+0000 7f273a800640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0cf290 0x7f273c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.698+0000 7f2739400640 1 -- 172.21.15.125:0/453844701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0cf290 msgr2=0x7f273c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.698+0000 7f2739400640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0cf290 0x7f273c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.698+0000 7f2739400640 1 -- 172.21.15.125:0/453844701 >> v1:172.21.15.137:6789/0 conn(0x7f273c0ae5a0 legacy=0x7f273c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.698+0000 7f2739400640 1 -- 172.21.15.125:0/453844701 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c06c470 con 0x7f273c0b3ae0 2024-01-16T14:59:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.699+0000 7f2739400640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0b3ae0 0x7f273c15e430 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f272c009e50 tx=0x7f272c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636ecbbb7049434a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.699+0000 7f2738a00640 1 -- 172.21.15.125:0/453844701 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c02cb00 con 0x7f273c0b3ae0 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.699+0000 7f2738a00640 1 -- 172.21.15.125:0/453844701 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c02cc60 con 0x7f273c0b3ae0 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.699+0000 7f2738a00640 1 -- 172.21.15.125:0/453844701 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0343f0 con 0x7f273c0b3ae0 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 -- 172.21.15.125:0/453844701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0b3ae0 msgr2=0x7f273c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0b3ae0 0x7f273c15e430 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f272c009e50 tx=0x7f272c02bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 -- 172.21.15.125:0/453844701 shutdown_connections 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0cf290 0x7f273c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 --2- 172.21.15.125:0/453844701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0b3ae0 0x7f273c15e430 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 -- 172.21.15.125:0/453844701 >> 172.21.15.125:0/453844701 conn(0x7f273c06a800 msgr2=0x7f273c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 -- 172.21.15.125:0/453844701 shutdown_connections 2024-01-16T14:59:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.700+0000 7f2741000640 1 -- 172.21.15.125:0/453844701 wait complete. 2024-01-16T14:59:38.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.702+0000 7f2741000640 1 Processor -- start 2024-01-16T14:59:38.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.702+0000 7f2741000640 1 -- start start 2024-01-16T14:59:38.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0ae5a0 0x7f273c1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f273c0b3ae0 0x7f273c1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c1504f0 con 0x7f273c0cf290 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c1507d0 con 0x7f273c0b3ae0 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f2741000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c150ab0 con 0x7f273c0ae5a0 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.703+0000 7f273a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36300/0 (socket says 172.21.15.125:36300) 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 -- 172.21.15.125:0/3656615615 learned_addr learned my addr 172.21.15.125:0/3656615615 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 -- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0ae5a0 msgr2=0x7f273c1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0ae5a0 0x7f273c1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 -- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f273c0b3ae0 msgr2=0x7f273c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f273c0b3ae0 0x7f273c1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 -- 172.21.15.125:0/3656615615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c009b30 con 0x7f273c0cf290 2024-01-16T14:59:38.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.704+0000 7f273a800640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f273400d5d0 tx=0x7f273400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.705+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340125f0 con 0x7f273c0cf290 2024-01-16T14:59:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.705+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734012770 con 0x7f273c0cf290 2024-01-16T14:59:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.705+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734012b20 con 0x7f273c0cf290 2024-01-16T14:59:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.705+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c150d90 con 0x7f273c0cf290 2024-01-16T14:59:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.705+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c151350 con 0x7f273c0cf290 2024-01-16T14:59:38.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.706+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c0675d0 con 0x7f273c0cf290 2024-01-16T14:59:38.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.707+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2734012c80 con 0x7f273c0cf290 2024-01-16T14:59:38.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.708+0000 7f2732000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 0x7f271406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.708+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f273408a260 con 0x7f273c0cf290 2024-01-16T14:59:38.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.708+0000 7f2739e00640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 0x7f271406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.709+0000 7f2739e00640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 0x7f271406d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2724003cc0 tx=0x7f27240079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.713+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2734054fe0 con 0x7f273c0cf290 2024-01-16T14:59:38.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.902+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f273c0b37b0 con 0x7f273c0cf290 2024-01-16T14:59:38.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.910+0000 7f2732000640 1 -- 172.21.15.125:0/3656615615 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2734058600 con 0x7f273c0cf290 2024-01-16T14:59:38.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.912+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 msgr2=0x7f271406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.912+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 0x7f271406d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2724003cc0 tx=0x7f27240079b0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.912+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 msgr2=0x7f273c14e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.912+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f273400d5d0 tx=0x7f273400da90 comp rx=0 tx=0).stop 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 shutdown_connections 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f271406b500 0x7f271406d9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f273c0cf290 0x7f273c14e650 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f273c0b3ae0 0x7f273c1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 --2- 172.21.15.125:0/3656615615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f273c0ae5a0 0x7f273c1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 >> 172.21.15.125:0/3656615615 conn(0x7f273c06a800 msgr2=0x7f273c0b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 shutdown_connections 2024-01-16T14:59:38.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:38.913+0000 7f2741000640 1 -- 172.21.15.125:0/3656615615 wait complete. 2024-01-16T14:59:38.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T14:59:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.141+0000 7f8223800640 1 Processor -- start 2024-01-16T14:59:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.141+0000 7f8223800640 1 -- start start 2024-01-16T14:59:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.143+0000 7f8223800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.143+0000 7f8223800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 0x7f821c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.143+0000 7f8223800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c071de0 con 0x7f821c0d0b60 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.143+0000 7f8223800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c0af270 con 0x7f821c0aede0 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.143+0000 7f8223800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c072030 con 0x7f821c06a3e0 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.144+0000 7f8220e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.144+0000 7f8221800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 0x7f821c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.144+0000 7f8220e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44708/0 (socket says 172.21.15.125:44708) 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.144+0000 7f8220e00640 1 -- 172.21.15.125:0/3873423660 learned_addr learned my addr 172.21.15.125:0/3873423660 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.145+0000 7f8220e00640 1 -- 172.21.15.125:0/3873423660 >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 msgr2=0x7f821c06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.145+0000 7f8220e00640 1 --2- 172.21.15.125:0/3873423660 >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 0x7f821c06f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.145+0000 7f8220e00640 1 -- 172.21.15.125:0/3873423660 >> v1:172.21.15.137:6789/0 conn(0x7f821c0d0b60 legacy=0x7f821c06a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.145+0000 7f8220e00640 1 -- 172.21.15.125:0/3873423660 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c072310 con 0x7f821c06a3e0 2024-01-16T14:59:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.145+0000 7f8221800640 1 --2- 172.21.15.125:0/3873423660 >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 0x7f821c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.147+0000 7f8220e00640 1 --2- 172.21.15.125:0/3873423660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f820c009870 tx=0x7f820c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211ac49444553409 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.147+0000 7f821b400640 1 -- 172.21.15.125:0/3873423660 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c030840 con 0x7f821c06a3e0 2024-01-16T14:59:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.147+0000 7f821b400640 1 -- 172.21.15.125:0/3873423660 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c0309a0 con 0x7f821c06a3e0 2024-01-16T14:59:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.147+0000 7f821b400640 1 -- 172.21.15.125:0/3873423660 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c030d30 con 0x7f821c06a3e0 2024-01-16T14:59:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 -- 172.21.15.125:0/3873423660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 msgr2=0x7f821c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 --2- 172.21.15.125:0/3873423660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f820c009870 tx=0x7f820c02b980 comp rx=0 tx=0).stop 2024-01-16T14:59:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 -- 172.21.15.125:0/3873423660 shutdown_connections 2024-01-16T14:59:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 --2- 172.21.15.125:0/3873423660 >> v2:172.21.15.137:3300/0 conn(0x7f821c0aede0 0x7f821c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 --2- 172.21.15.125:0/3873423660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c06a3e0 0x7f821c0ac8d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 -- 172.21.15.125:0/3873423660 >> 172.21.15.125:0/3873423660 conn(0x7f821c0cfef0 msgr2=0x7f821c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 -- 172.21.15.125:0/3873423660 shutdown_connections 2024-01-16T14:59:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.148+0000 7f8223800640 1 -- 172.21.15.125:0/3873423660 wait complete. 2024-01-16T14:59:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.150+0000 7f8223800640 1 Processor -- start 2024-01-16T14:59:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.150+0000 7f8223800640 1 -- start start 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f821c06a3e0 0x7f821c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c0aede0 0x7f821c15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c1f72f0 con 0x7f821c0d0b60 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c160480 con 0x7f821c06a3e0 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8223800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c1f6840 con 0x7f821c0aede0 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36312/0 (socket says 172.21.15.125:36312) 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 -- 172.21.15.125:0/3317649573 learned_addr learned my addr 172.21.15.125:0/3317649573 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f821be00640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c0aede0 0x7f821c15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8220e00640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f821c06a3e0 0x7f821c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 -- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c0aede0 msgr2=0x7f821c15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c0aede0 0x7f821c15ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 -- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f821c06a3e0 msgr2=0x7f821c153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f821c06a3e0 0x7f821c153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.151+0000 7f8221800640 1 -- 172.21.15.125:0/3317649573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214009400 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8221800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f821400e930 tx=0x7f821400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214009700 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821400ca00 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c0094d0 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821400cd90 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.152+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1f6d30 con 0x7f821c0d0b60 2024-01-16T14:59:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.153+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c0675d0 con 0x7f821c0d0b60 2024-01-16T14:59:39.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.154+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f8214013470 con 0x7f821c0d0b60 2024-01-16T14:59:39.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.155+0000 7f8218c00640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 0x7f81f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.156+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82140898e0 con 0x7f821c0d0b60 2024-01-16T14:59:39.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.156+0000 7f8220e00640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 0x7f81f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.156+0000 7f8220e00640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 0x7f81f406d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f820c036000 tx=0x7f820c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.160+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82140547b0 con 0x7f821c0d0b60 2024-01-16T14:59:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:39.420+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f821c0ada50 con 0x7f821c0d0b60 2024-01-16T14:59:40.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.032+0000 7f8218c00640 1 -- 172.21.15.125:0/3317649573 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v165) v1 ==== 152+0+0 (secure 0 0 0) 0x7f8214057dd0 con 0x7f821c0d0b60 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 msgr2=0x7f81f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 0x7f81f406d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f820c036000 tx=0x7f820c03b040 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 msgr2=0x7f821c1f6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f821400e930 tx=0x7f821400edf0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 shutdown_connections 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81f406b500 0x7f81f406d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f821c0d0b60 0x7f821c1f6300 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f821c0aede0 0x7f821c15ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 --2- 172.21.15.125:0/3317649573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f821c06a3e0 0x7f821c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.036+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 >> 172.21.15.125:0/3317649573 conn(0x7f821c0cfef0 msgr2=0x7f821c15c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.037+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 shutdown_connections 2024-01-16T14:59:40.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.037+0000 7f8223800640 1 -- 172.21.15.125:0/3317649573 wait complete. 2024-01-16T14:59:40.037 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-16T14:59:40.038 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T14:59:40.038 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:59:40.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:40.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.256+0000 7f3080a00640 1 Processor -- start 2024-01-16T14:59:40.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.256+0000 7f3080a00640 1 -- start start 2024-01-16T14:59:40.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f3080a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f307c06a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f3080a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f307c06a920 0x7f307c070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f3080a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c06aea0 con 0x7f307c161890 2024-01-16T14:59:40.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f3080a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c0cd390 con 0x7f307c06a920 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f3080a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c06b180 con 0x7f307c0cef70 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f307b200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f307c161890 0x7f307c1539b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57426/0 (socket says 172.21.15.125:57426) 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.258+0000 7f307b200640 1 -- 172.21.15.125:0/2808199766 learned_addr learned my addr 172.21.15.125:0/2808199766 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3071c00640 1 --2- 172.21.15.125:0/2808199766 >> v2:172.21.15.137:3300/0 conn(0x7f307c06a920 0x7f307c070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435451709 0 0) 0x7f307c06aea0 con 0x7f307c161890 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f307a800640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f307c06a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f305c003520 con 0x7f307c161890 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212332036 0 0) 0x7f305c003520 con 0x7f307c161890 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 msgr2=0x7f307c06a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f307c06a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 >> v2:172.21.15.137:3300/0 conn(0x7f307c06a920 msgr2=0x7f307c070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 --2- 172.21.15.125:0/2808199766 >> v2:172.21.15.137:3300/0 conn(0x7f307c06a920 0x7f307c070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c160db0 con 0x7f307c161890 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f307a800640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f307c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3074001670 con 0x7f307c161890 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.259+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 >> v1:172.21.15.125:6789/0 conn(0x7f307c161890 legacy=0x7f307c1539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f305c003970 0x7f305c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f305c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c06b180 con 0x7f305c0047c0 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c00b980 con 0x7f305c003970 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c00bac0 con 0x7f307c0cef70 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f307a800640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f305c003970 0x7f305c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f307b200640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f305c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 -- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 msgr2=0x7f305c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0cef70 0x7f305c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 -- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f305c003970 msgr2=0x7f305c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f305c003970 0x7f305c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f3071c00640 1 -- 172.21.15.125:0/2808199766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c003520 con 0x7f305c0047c0 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.260+0000 7f307a800640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f305c003970 0x7f305c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.261+0000 7f3071c00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3064009ae0 tx=0x7f305c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef916518a6fef6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064005990 con 0x7f305c0047c0 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064005af0 con 0x7f305c0047c0 2024-01-16T14:59:40.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3079e00640 1 -- 172.21.15.125:0/2808199766 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064005ea0 con 0x7f305c0047c0 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 -- 172.21.15.125:0/2808199766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 msgr2=0x7f305c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 --2- 172.21.15.125:0/2808199766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3064009ae0 tx=0x7f305c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f307b200640 1 -- 172.21.15.125:0/2808199766 reap_dead start 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 -- 172.21.15.125:0/2808199766 shutdown_connections 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 -- 172.21.15.125:0/2808199766 >> 172.21.15.125:0/2808199766 conn(0x7f307c0cc1d0 msgr2=0x7f307c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 -- 172.21.15.125:0/2808199766 shutdown_connections 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.262+0000 7f3080a00640 1 -- 172.21.15.125:0/2808199766 wait complete. 2024-01-16T14:59:40.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.264+0000 7f3080a00640 1 Processor -- start 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.264+0000 7f3080a00640 1 -- start start 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0adcf0 0x7f307c0ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f307c1586b0 0x7f307c158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0ae690 con 0x7f305c0047c0 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c15e4f0 con 0x7f307c1586b0 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f3080a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c147810 con 0x7f307c0adcf0 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f307a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f307a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36326/0 (socket says 172.21.15.125:36326) 2024-01-16T14:59:40.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.265+0000 7f307a800640 1 -- 172.21.15.125:0/1379472818 learned_addr learned my addr 172.21.15.125:0/1379472818 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f3071c00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0adcf0 0x7f307c0ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307b200640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f307c1586b0 0x7f307c158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 -- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0adcf0 msgr2=0x7f307c0ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0adcf0 0x7f307c0ae150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 -- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f307c1586b0 msgr2=0x7f307c158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f307c1586b0 0x7f307c158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 -- 172.21.15.125:0/1379472818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c155080 con 0x7f305c0047c0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f307a800640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f306c009fc0 tx=0x7f306c00e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c015610 con 0x7f305c0047c0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c00f040 con 0x7f305c0047c0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c015970 con 0x7f305c0047c0 2024-01-16T14:59:40.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.266+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c0ae870 con 0x7f305c0047c0 2024-01-16T14:59:40.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.267+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c15e700 con 0x7f305c0047c0 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.268+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c0aed90 con 0x7f305c0047c0 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.269+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f306c015ad0 con 0x7f305c0047c0 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.269+0000 7f3073400640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 0x7f305406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.270+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f306c08ab70 con 0x7f305c0047c0 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.270+0000 7f3071c00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 0x7f305406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.270+0000 7f3071c00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 0x7f305406d9f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3064035000 tx=0x7f3064030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.274+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f306c055860 con 0x7f305c0047c0 2024-01-16T14:59:40.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.464+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f307c06bad0 con 0x7f305c0047c0 2024-01-16T14:59:40.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.466+0000 7f3073400640 1 -- 172.21.15.125:0/1379472818 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f306c058e80 con 0x7f305c0047c0 2024-01-16T14:59:40.446 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:40.447 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:33.670394+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/2265216525":"2024-01-17T14:59:30.648400+0000","172.21.15.125:6836/2265216525":"2024-01-17T14:59:30.648400+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 msgr2=0x7f305406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 0x7f305406d9f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3064035000 tx=0x7f3064030000 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 msgr2=0x7f307c1472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f306c009fc0 tx=0x7f306c00e5d0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 shutdown_connections 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f307c1586b0 0x7f307c158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f307c0adcf0 0x7f307c0ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f305406b530 0x7f305406d9f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 --2- 172.21.15.125:0/1379472818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f305c0047c0 0x7f307c1472d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.468+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 >> 172.21.15.125:0/1379472818 conn(0x7f307c0cc1d0 msgr2=0x7f307c0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.469+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 shutdown_connections 2024-01-16T14:59:40.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.469+0000 7f3080a00640 1 -- 172.21.15.125:0/1379472818 wait complete. 2024-01-16T14:59:40.469 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:40.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:40.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.703+0000 7fe839a00640 1 Processor -- start 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.704+0000 7fe839a00640 1 -- start start 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.705+0000 7fe839a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe8340cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.705+0000 7fe839a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340ce440 con 0x7fe8340cdc20 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.705+0000 7fe839a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340ce720 con 0x7fe83413cf10 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.705+0000 7fe839a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340cea00 con 0x7fe83413fcb0 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.706+0000 7fe839000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe8340cdc20 0x7fe8340b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57428/0 (socket says 172.21.15.125:57428) 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.706+0000 7fe839000640 1 -- 172.21.15.125:0/526507926 learned_addr learned my addr 172.21.15.125:0/526507926 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.706+0000 7fe82b200640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe8340cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.706+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850519842 0 0) 0x7fe8340ce440 con 0x7fe8340cdc20 2024-01-16T14:59:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.706+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814005730 con 0x7fe8340cdc20 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935104757 0 0) 0x7fe814005730 con 0x7fe8340cdc20 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 msgr2=0x7fe8340cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe8340cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 >> v1:172.21.15.137:6789/0 conn(0x7fe83413cf10 legacy=0x7fe83413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8340b1d90 con 0x7fe8340cdc20 2024-01-16T14:59:40.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe82c0019f0 con 0x7fe8340cdc20 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 >> v1:172.21.15.125:6789/0 conn(0x7fe8340cdc20 legacy=0x7fe8340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe814005bb0 0x7fe814005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8140069d0 0x7fe814006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe814007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340cea00 con 0x7fe8140069d0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400dc40 con 0x7fe814005bb0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400ddd0 con 0x7fe83413fcb0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8140069d0 0x7fe814006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833e00640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe814005bb0 0x7fe814005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe839000640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe814007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 -- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 msgr2=0x7fe814007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413fcb0 0x7fe814007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 -- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe814005bb0 msgr2=0x7fe814005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe814005bb0 0x7fe814005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe82b200640 1 -- 172.21.15.125:0/526507926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814005730 con 0x7fe8140069d0 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.707+0000 7fe833e00640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe814005bb0 0x7fe814005f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.708+0000 7fe82b200640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8140069d0 0x7fe814006e50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009a90 tx=0x7fe814007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55576b9b67570b96 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02c680 con 0x7fe8140069d0 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c02c7e0 con 0x7fe8140069d0 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe833400640 1 -- 172.21.15.125:0/526507926 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02cb70 con 0x7fe8140069d0 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 -- 172.21.15.125:0/526507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8140069d0 msgr2=0x7fe814006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 --2- 172.21.15.125:0/526507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8140069d0 0x7fe814006e50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009a90 tx=0x7fe814007a00 comp rx=0 tx=0).stop 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839000640 1 -- 172.21.15.125:0/526507926 reap_dead start 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 -- 172.21.15.125:0/526507926 shutdown_connections 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 -- 172.21.15.125:0/526507926 >> 172.21.15.125:0/526507926 conn(0x7fe83406a7d0 msgr2=0x7fe83406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 -- 172.21.15.125:0/526507926 shutdown_connections 2024-01-16T14:59:40.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.709+0000 7fe839a00640 1 -- 172.21.15.125:0/526507926 wait complete. 2024-01-16T14:59:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.711+0000 7fe839a00640 1 Processor -- start 2024-01-16T14:59:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.711+0000 7fe839a00640 1 -- start start 2024-01-16T14:59:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8140069d0 0x7fe8341397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413b7c0 0x7fe83413bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341532d0 con 0x7fe834138910 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83413c4d0 con 0x7fe8140069d0 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.712+0000 7fe839a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe834155b00 con 0x7fe83413b7c0 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36342/0 (socket says 172.21.15.125:36342) 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 -- 172.21.15.125:0/3858282404 learned_addr learned my addr 172.21.15.125:0/3858282404 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe833e00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8140069d0 0x7fe8341397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 -- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413b7c0 msgr2=0x7fe83413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413b7c0 0x7fe83413bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 -- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8140069d0 msgr2=0x7fe8341397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8140069d0 0x7fe8341397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 -- 172.21.15.125:0/3858282404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834155de0 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.713+0000 7fe82b200640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02c220 tx=0x7fe81c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.714+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c003810 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.714+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c003970 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.714+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8341560c0 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.714+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c003ce0 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.714+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83413c180 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.715+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe834067450 con 0x7fe834138910 2024-01-16T14:59:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.716+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe81c034440 con 0x7fe834138910 2024-01-16T14:59:40.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.717+0000 7fe830c00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 0x7fe80c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.717+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe81c0abbe0 con 0x7fe834138910 2024-01-16T14:59:40.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.717+0000 7fe833e00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 0x7fe80c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.718+0000 7fe833e00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 0x7fe80c06d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe824001780 tx=0x7fe824009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.721+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe81c076bc0 con 0x7fe834138910 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.912+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8340cbf10 con 0x7fe834138910 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.913+0000 7fe830c00640 1 -- 172.21.15.125:0/3858282404 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe81c0044e0 con 0x7fe834138910 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 msgr2=0x7fe80c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 0x7fe80c06d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe824001780 tx=0x7fe824009040 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 msgr2=0x7fe834139d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02c220 tx=0x7fe81c004730 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 shutdown_connections 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe83413b7c0 0x7fe83413bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe80c06b530 0x7fe80c06d9f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe834138910 0x7fe834139d30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 --2- 172.21.15.125:0/3858282404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8140069d0 0x7fe8341397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 >> 172.21.15.125:0/3858282404 conn(0x7fe83406a7d0 msgr2=0x7fe834153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 shutdown_connections 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:40.915+0000 7fe839a00640 1 -- 172.21.15.125:0/3858282404 wait complete. 2024-01-16T14:59:40.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T14:59:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.139+0000 7f313d000640 1 Processor -- start 2024-01-16T14:59:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.139+0000 7f313d000640 1 -- start start 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.140+0000 7f313d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31380cc990 0x7f313806d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.140+0000 7f313d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3138070e50 con 0x7f31380cc500 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.140+0000 7f313d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3138144020 con 0x7f3138143c40 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.140+0000 7f313d000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380723c0 con 0x7f31380cc990 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f3137e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31380cc990 0x7f313806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f312e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f31380cc500 0x7f313806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57438/0 (socket says 172.21.15.125:57438) 2024-01-16T14:59:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f312e800640 1 -- 172.21.15.125:0/2918476111 learned_addr learned my addr 172.21.15.125:0/2918476111 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f3137e00640 1 -- 172.21.15.125:0/2918476111 >> v1:172.21.15.137:6789/0 conn(0x7f3138143c40 legacy=0x7f3138071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f3137e00640 1 -- 172.21.15.125:0/2918476111 >> v1:172.21.15.125:6789/0 conn(0x7f31380cc500 legacy=0x7f313806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f3137e00640 1 -- 172.21.15.125:0/2918476111 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138071e00 con 0x7f31380cc990 2024-01-16T14:59:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.141+0000 7f3136a00640 1 -- 172.21.15.125:0/2918476111 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47490330 0 0) 0x7f3138070e50 con 0x7f31380cc500 2024-01-16T14:59:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.142+0000 7f3137e00640 1 --2- 172.21.15.125:0/2918476111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31380cc990 0x7f313806d870 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b1072c65d9dc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f3136a00640 1 -- 172.21.15.125:0/2918476111 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cbd0 con 0x7f31380cc990 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f3136a00640 1 -- 172.21.15.125:0/2918476111 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313002cd30 con 0x7f31380cc990 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f3136a00640 1 -- 172.21.15.125:0/2918476111 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cea0 con 0x7f31380cc990 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 -- 172.21.15.125:0/2918476111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31380cc990 msgr2=0x7f313806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 --2- 172.21.15.125:0/2918476111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31380cc990 0x7f313806d870 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be20 comp rx=0 tx=0).stop 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 -- 172.21.15.125:0/2918476111 shutdown_connections 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 --2- 172.21.15.125:0/2918476111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31380cc990 0x7f313806d870 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 -- 172.21.15.125:0/2918476111 >> 172.21.15.125:0/2918476111 conn(0x7f313806a800 msgr2=0x7f313806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 -- 172.21.15.125:0/2918476111 shutdown_connections 2024-01-16T14:59:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.143+0000 7f313d000640 1 -- 172.21.15.125:0/2918476111 wait complete. 2024-01-16T14:59:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.145+0000 7f313d000640 1 Processor -- start 2024-01-16T14:59:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.145+0000 7f313d000640 1 -- start start 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31380cc500 0x7f31381f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 0x7f313815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313815ac50 con 0x7f3138143c40 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31381e6640 con 0x7f31380cc990 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f313d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31381412d0 con 0x7f31380cc500 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f3137e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f3137e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36372/0 (socket says 172.21.15.125:36372) 2024-01-16T14:59:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.146+0000 7f3137e00640 1 -- 172.21.15.125:0/1963396868 learned_addr learned my addr 172.21.15.125:0/1963396868 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f312e800640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 0x7f313815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 -- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31380cc500 msgr2=0x7f31381f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31380cc500 0x7f31381f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 -- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 msgr2=0x7f313815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 0x7f313815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 -- 172.21.15.125:0/1963396868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130009a70 con 0x7f3138143c40 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f312e800640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 0x7f313815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f3137e00640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cea0 con 0x7f3138143c40 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313003e630 con 0x7f3138143c40 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313003e9e0 con 0x7f3138143c40 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31381415b0 con 0x7f3138143c40 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.147+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31381e6920 con 0x7f3138143c40 2024-01-16T14:59:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.148+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3138141ad0 con 0x7f3138143c40 2024-01-16T14:59:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.149+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f313003b050 con 0x7f3138143c40 2024-01-16T14:59:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.150+0000 7f312fe00640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 0x7f311006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.150+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f31300ac6d0 con 0x7f3138143c40 2024-01-16T14:59:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.150+0000 7f3137400640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 0x7f311006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.151+0000 7f3137400640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 0x7f311006d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3128006fd0 tx=0x7f3128006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.155+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3138141ad0 con 0x7f3138143c40 2024-01-16T14:59:41.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:41.412+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f3138141ad0 con 0x7f3138143c40 2024-01-16T14:59:42.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.038+0000 7f312fe00640 1 -- 172.21.15.125:0/1963396868 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v167) v1 ==== 156+0+0 (secure 0 0 0) 0x7f3138141ad0 con 0x7f3138143c40 2024-01-16T14:59:42.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.042+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 msgr2=0x7f311006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.042+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 0x7f311006d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3128006fd0 tx=0x7f3128006bc0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.042+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 msgr2=0x7f31381e7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.042+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 shutdown_connections 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f311006b500 0x7f311006d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3138143c40 0x7f31381e7180 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31380cc990 0x7f313815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 --2- 172.21.15.125:0/1963396868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31380cc500 0x7f31381f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 >> 172.21.15.125:0/1963396868 conn(0x7f313806a800 msgr2=0x7f313806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 shutdown_connections 2024-01-16T14:59:42.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.043+0000 7f313d000640 1 -- 172.21.15.125:0/1963396868 wait complete. 2024-01-16T14:59:42.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T14:59:42.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.268+0000 7f535f200640 1 Processor -- start 2024-01-16T14:59:42.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.269+0000 7f535f200640 1 -- start start 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.270+0000 7f535f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 0x7f53600715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.270+0000 7f535f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5360071b10 0x7f53600ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.270+0000 7f535f200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5360071f70 con 0x7f53600adb00 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.270+0000 7f535f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536006d480 con 0x7f5360071b10 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.270+0000 7f535f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53600b3490 con 0x7f5360146850 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f535e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f53600adb00 0x7f53600700f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57448/0 (socket says 172.21.15.125:57448) 2024-01-16T14:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f535e800640 1 -- 172.21.15.125:0/1185356050 learned_addr learned my addr 172.21.15.125:0/1185356050 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f5357e00640 1 --2- 172.21.15.125:0/1185356050 >> v2:172.21.15.137:3300/0 conn(0x7f5360071b10 0x7f53600ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f535de00640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 0x7f53600715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168795221 0 0) 0x7f5360071f70 con 0x7f53600adb00 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.271+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5340003520 con 0x7f53600adb00 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996481028 0 0) 0x7f5340003520 con 0x7f53600adb00 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 msgr2=0x7f53600715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 0x7f53600715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 >> v2:172.21.15.137:3300/0 conn(0x7f5360071b10 msgr2=0x7f53600ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 --2- 172.21.15.125:0/1185356050 >> v2:172.21.15.137:3300/0 conn(0x7f5360071b10 0x7f53600ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600b3770 con 0x7f53600adb00 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f53580019f0 con 0x7f53600adb00 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 >> v1:172.21.15.125:6789/0 conn(0x7f53600adb00 legacy=0x7f53600700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 0x7f5340003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340003ef0 0x7f53400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340004d00 0x7f5340005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536006d480 con 0x7f5340004d00 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53600b3490 con 0x7f5340003ef0 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534000b980 con 0x7f5360146850 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f5357e00640 1 --2- 172.21.15.125:0/1185356050 >> v2:172.21.15.137:3300/0 conn(0x7f5360071b10 0x7f53600ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340004d00 0x7f5340005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 -- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 msgr2=0x7f5340003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5360146850 0x7f5340003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 -- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340003ef0 msgr2=0x7f53400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340003ef0 0x7f53400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.272+0000 7f535e800640 1 -- 172.21.15.125:0/1185356050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340003520 con 0x7f5340004d00 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.273+0000 7f535e800640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340004d00 0x7f5340005180 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f535800a850 tx=0x7f53400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eeb6f682a127824 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358034630 con 0x7f5340004d00 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535802f070 con 0x7f5340004d00 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535d400640 1 -- 172.21.15.125:0/1185356050 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53580349f0 con 0x7f5340004d00 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 -- 172.21.15.125:0/1185356050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340004d00 msgr2=0x7f5340005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 --2- 172.21.15.125:0/1185356050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340004d00 0x7f5340005180 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f535800a850 tx=0x7f53400057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535e800640 1 -- 172.21.15.125:0/1185356050 reap_dead start 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 -- 172.21.15.125:0/1185356050 shutdown_connections 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 -- 172.21.15.125:0/1185356050 >> 172.21.15.125:0/1185356050 conn(0x7f536006a3e0 msgr2=0x7f536006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 -- 172.21.15.125:0/1185356050 shutdown_connections 2024-01-16T14:59:42.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.274+0000 7f535f200640 1 -- 172.21.15.125:0/1185356050 wait complete. 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.276+0000 7f535f200640 1 Processor -- start 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.276+0000 7f535f200640 1 -- start start 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340004d00 0x7f536015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536015c260 0x7f5360137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536015be40 con 0x7f53600c4010 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5360137bc0 con 0x7f5340004d00 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536015c6c0 con 0x7f536015c260 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36384/0 (socket says 172.21.15.125:36384) 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.277+0000 7f535e800640 1 -- 172.21.15.125:0/91840704 learned_addr learned my addr 172.21.15.125:0/91840704 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535de00640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340004d00 0x7f536015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 -- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536015c260 msgr2=0x7f5360137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536015c260 0x7f5360137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 -- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340004d00 msgr2=0x7f536015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340004d00 0x7f536015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 -- 172.21.15.125:0/91840704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536015b1c0 con 0x7f53600c4010 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535e800640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f535802cbf0 tx=0x7f5358001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358034840 con 0x7f53600c4010 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53580349a0 con 0x7f53600c4010 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536015b4a0 con 0x7f53600c4010 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.278+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358034d30 con 0x7f53600c4010 2024-01-16T14:59:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.279+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536015ba60 con 0x7f53600c4010 2024-01-16T14:59:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.280+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536015bf80 con 0x7f53600c4010 2024-01-16T14:59:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.281+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5358031030 con 0x7f53600c4010 2024-01-16T14:59:42.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.281+0000 7f5356000640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 0x7f533806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.282+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53580abc90 con 0x7f53600c4010 2024-01-16T14:59:42.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.282+0000 7f535de00640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 0x7f533806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.282+0000 7f535de00640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 0x7f533806d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5350004770 tx=0x7f5350008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.286+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f536015bf80 con 0x7f53600c4010 2024-01-16T14:59:42.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.477+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f536015bf80 con 0x7f53600c4010 2024-01-16T14:59:42.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.482+0000 7f5356000640 1 -- 172.21.15.125:0/91840704 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f536015bf80 con 0x7f53600c4010 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 msgr2=0x7f533806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 0x7f533806d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5350004770 tx=0x7f5350008040 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 msgr2=0x7f53600c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f535802cbf0 tx=0x7f5358001d80 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 shutdown_connections 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f533806b500 0x7f533806d9c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53600c4010 0x7f53600c4a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536015c260 0x7f5360137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 --2- 172.21.15.125:0/91840704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5340004d00 0x7f536015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 >> 172.21.15.125:0/91840704 conn(0x7f536006a3e0 msgr2=0x7f536015ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 shutdown_connections 2024-01-16T14:59:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:42.485+0000 7f535f200640 1 -- 172.21.15.125:0/91840704 wait complete. 2024-01-16T14:59:42.483 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T14:59:42.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:42.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:42.717 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T14:59:42.717 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:59:42.717 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:42.773 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T14:59:42.774 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:59:42.774 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:42.829 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T14:59:42.829 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T14:59:42.830 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:42.885 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T14:59:42.886 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T14:59:42.886 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:42.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T14:59:42.942 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T14:59:42.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:43.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.170+0000 7f8e0d000640 1 Processor -- start 2024-01-16T14:59:43.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.171+0000 7f8e0d000640 1 -- start start 2024-01-16T14:59:43.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e0d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8e080d0100 0x7f8e080b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e0d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080d0a10 con 0x7f8e08160920 2024-01-16T14:59:43.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e0d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080cf080 con 0x7f8e080d0f40 2024-01-16T14:59:43.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e0d000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e080c6be0 con 0x7f8e080d0100 2024-01-16T14:59:43.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e07400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8e08160920 0x7f8e080cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57454/0 (socket says 172.21.15.125:57454) 2024-01-16T14:59:43.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.172+0000 7f8e07400640 1 -- 172.21.15.125:0/2901973883 learned_addr learned my addr 172.21.15.125:0/2901973883 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395209013 0 0) 0x7f8e080d0a10 con 0x7f8e08160920 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8dfe800640 1 --2- 172.21.15.125:0/2901973883 >> v2:172.21.15.137:3300/0 conn(0x7f8e080d0100 0x7f8e080b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de8003520 con 0x7f8e08160920 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780332832 0 0) 0x7f8de8003520 con 0x7f8e08160920 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 >> v2:172.21.15.137:3300/0 conn(0x7f8e080d0100 msgr2=0x7f8e080b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 --2- 172.21.15.125:0/2901973883 >> v2:172.21.15.137:3300/0 conn(0x7f8e080d0100 0x7f8e080b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 >> v1:172.21.15.137:6789/0 conn(0x7f8e080d0f40 legacy=0x7f8e080d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080ce7e0 con 0x7f8e08160920 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8df80019f0 con 0x7f8e08160920 2024-01-16T14:59:43.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 >> v1:172.21.15.125:6789/0 conn(0x7f8e08160920 legacy=0x7f8e080cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8de8003970 0x7f8de8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8de80047c0 0x7f8de8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8de800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080c6be0 con 0x7f8de8005180 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de800c020 con 0x7f8de8003970 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.173+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de800c1e0 con 0x7f8de80047c0 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8de800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8dfe800640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8de80047c0 0x7f8de8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07400640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8de8003970 0x7f8de8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 -- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8de80047c0 msgr2=0x7f8de8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8de80047c0 0x7f8de8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 -- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8de8003970 msgr2=0x7f8de8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8de8003970 0x7f8de8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07e00640 1 -- 172.21.15.125:0/2901973883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8003520 con 0x7f8de8005180 2024-01-16T14:59:43.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8e07400640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8de8003970 0x7f8de8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:43.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.174+0000 7f8dfe800640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8de80047c0 0x7f8de8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.175+0000 7f8e07e00640 1 -- 172.21.15.125:0/2901973883 reap_dead start 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e07e00640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8de800b980 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001750 tx=0x7f8de80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d3a15092bb85f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000052d0 con 0x7f8de8005180 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0002e050 con 0x7f8de8005180 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e06a00640 1 -- 172.21.15.125:0/2901973883 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00003830 con 0x7f8de8005180 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e0d000640 1 -- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 msgr2=0x7f8de800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.176+0000 7f8e0d000640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8de800b980 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001750 tx=0x7f8de80057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.177+0000 7f8e0d000640 1 -- 172.21.15.125:0/2901973883 shutdown_connections 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.177+0000 7f8e0d000640 1 --2- 172.21.15.125:0/2901973883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8de800b980 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.177+0000 7f8e0d000640 1 -- 172.21.15.125:0/2901973883 >> 172.21.15.125:0/2901973883 conn(0x7f8e080d3090 msgr2=0x7f8e080d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.177+0000 7f8e0d000640 1 -- 172.21.15.125:0/2901973883 shutdown_connections 2024-01-16T14:59:43.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.177+0000 7f8e0d000640 1 -- 172.21.15.125:0/2901973883 wait complete. 2024-01-16T14:59:43.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.178+0000 7f8e0d000640 1 Processor -- start 2024-01-16T14:59:43.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.179+0000 7f8e0d000640 1 -- start start 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.179+0000 7f8e0d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e0d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e080b7920 0x7f8e080b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e0d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e080b72d0 0x7f8e080cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e0d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b5cf0 con 0x7f8de8005180 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e0d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b5fd0 con 0x7f8e080b7920 2024-01-16T14:59:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e0d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b62b0 con 0x7f8e080b72d0 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36394/0 (socket says 172.21.15.125:36394) 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 -- 172.21.15.125:0/665147498 learned_addr learned my addr 172.21.15.125:0/665147498 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8dfe800640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e080b7920 0x7f8e080b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 -- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e080b72d0 msgr2=0x7f8e080cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e080b72d0 0x7f8e080cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 -- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e080b7920 msgr2=0x7f8e080b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e080b7920 0x7f8e080b6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.180+0000 7f8e07400640 1 -- 172.21.15.125:0/665147498 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080b6590 con 0x7f8de8005180 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8e07400640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8df8001580 tx=0x7f8df800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8004d60 con 0x7f8de8005180 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df800be60 con 0x7f8de8005180 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e080b6870 con 0x7f8de8005180 2024-01-16T14:59:43.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df80072d0 con 0x7f8de8005180 2024-01-16T14:59:43.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.181+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e080cbd60 con 0x7f8de8005180 2024-01-16T14:59:43.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.182+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e080b7720 con 0x7f8de8005180 2024-01-16T14:59:43.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.183+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f8df8007430 con 0x7f8de8005180 2024-01-16T14:59:43.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.184+0000 7f8dffe00640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 0x7f8de006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.184+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8df808a160 con 0x7f8de8005180 2024-01-16T14:59:43.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.184+0000 7f8dfe800640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 0x7f8de006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.185+0000 7f8dfe800640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 0x7f8de006d9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8e080b8210 tx=0x7f8df0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.188+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8df8054e60 con 0x7f8de8005180 2024-01-16T14:59:43.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.415+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e08067450 con 0x7f8de8005180 2024-01-16T14:59:43.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.417+0000 7f8dffe00640 1 -- 172.21.15.125:0/665147498 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f8df8058480 con 0x7f8de8005180 2024-01-16T14:59:43.397 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:43.397 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1608829453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1608829453},{"type":"v1","addr":"172.21.15.137:6835","nonce":1608829453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4798,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1479548002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1479548002},{"type":"v1","addr":"172.21.15.137:6833","nonce":1479548002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1004786444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1004786444},{"type":"v1","addr":"172.21.15.125:6835","nonce":1004786444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/181156614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":181156614},{"type":"v1","addr":"172.21.15.125:6837","nonce":181156614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-01-16T14:59:43.397 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 21 2024-01-16T14:59:43.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 msgr2=0x7f8de006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 0x7f8de006d9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8e080b8210 tx=0x7f8df0008040 comp rx=0 tx=0).stop 2024-01-16T14:59:43.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 msgr2=0x7f8e080cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8df8001580 tx=0x7f8df800b4a0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 shutdown_connections 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e080b72d0 0x7f8e080cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e080b7920 0x7f8e080b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8de006b530 0x7f8de006d9f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 --2- 172.21.15.125:0/665147498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8de8005180 0x7f8e080cab70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 >> 172.21.15.125:0/665147498 conn(0x7f8e080d3090 msgr2=0x7f8e0806c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 shutdown_connections 2024-01-16T14:59:43.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.420+0000 7f8e0d000640 1 -- 172.21.15.125:0/665147498 wait complete. 2024-01-16T14:59:43.419 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:59:43.419 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T14:59:43.419 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T14:59:43.419 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:59:43.419 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:59:43.420 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T14:59:43.420 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T14:59:43.420 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:59:43.420 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:59:43.420 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T14:59:43.420 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T14:59:43.420 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:59:43.421 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:59:43.421 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T14:59:43.421 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T14:59:43.421 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:59:43.438 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T14:59:43.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:59:43.439 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T14:59:43.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:59:43.442 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T14:59:43.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:59:43.444 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T14:59:43.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:59:43.486 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:59:43.486 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:43.488 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:59:43.489 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:43.492 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:59:43.492 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:43.495 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:59:43.495 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:43.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T14:59:43.638 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T14:59:43.638 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T14:59:43.654 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T14:59:43.662 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T14:59:43.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.695+0000 7fe86b600640 1 Processor -- start 2024-01-16T14:59:43.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.695+0000 7fe86b600640 1 -- start start 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe86b600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe86413df80 0x7fe86415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe86b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8640cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe86b600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86406c470 con 0x7fe86413e360 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe86b600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86406c750 con 0x7fe86413df80 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe86b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86406ca30 con 0x7fe8640cf9a0 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe869600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8640cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe868c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe86413df80 0x7fe86415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe863e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe86413e360 0x7fe8640cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:37408/0 (socket says 172.21.15.125:37408) 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.697+0000 7fe863e00640 1 -- 172.21.15.125:0/685898554 learned_addr learned my addr 172.21.15.125:0/685898554 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe863400640 1 -- 172.21.15.125:0/685898554 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438475970 0 0) 0x7fe86406c470 con 0x7fe86413e360 2024-01-16T14:59:43.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe863400640 1 -- 172.21.15.125:0/685898554 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe844003520 con 0x7fe86413e360 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe869600640 1 -- 172.21.15.125:0/685898554 >> v2:172.21.15.137:3300/0 conn(0x7fe86413df80 msgr2=0x7fe86415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe869600640 1 --2- 172.21.15.125:0/685898554 >> v2:172.21.15.137:3300/0 conn(0x7fe86413df80 0x7fe86415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe869600640 1 -- 172.21.15.125:0/685898554 >> v1:172.21.15.137:6789/0 conn(0x7fe86413e360 legacy=0x7fe8640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.698+0000 7fe869600640 1 -- 172.21.15.125:0/685898554 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86406cd10 con 0x7fe8640cf9a0 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.699+0000 7fe869600640 1 --2- 172.21.15.125:0/685898554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8640cd840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0097a0 tx=0x7fe85c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76dfcc0562198c45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.699+0000 7fe863400640 1 -- 172.21.15.125:0/685898554 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c031830 con 0x7fe8640cf9a0 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.699+0000 7fe863400640 1 -- 172.21.15.125:0/685898554 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c031990 con 0x7fe8640cf9a0 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.699+0000 7fe863400640 1 -- 172.21.15.125:0/685898554 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c031d20 con 0x7fe8640cf9a0 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 -- 172.21.15.125:0/685898554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 msgr2=0x7fe8640cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 --2- 172.21.15.125:0/685898554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8640cd840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0097a0 tx=0x7fe85c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 -- 172.21.15.125:0/685898554 shutdown_connections 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 --2- 172.21.15.125:0/685898554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8640cd840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 --2- 172.21.15.125:0/685898554 >> v2:172.21.15.137:3300/0 conn(0x7fe86413df80 0x7fe86415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 -- 172.21.15.125:0/685898554 >> 172.21.15.125:0/685898554 conn(0x7fe86406a800 msgr2=0x7fe86406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 -- 172.21.15.125:0/685898554 shutdown_connections 2024-01-16T14:59:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.700+0000 7fe86b600640 1 -- 172.21.15.125:0/685898554 wait complete. 2024-01-16T14:59:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.701+0000 7fe86b600640 1 Processor -- start 2024-01-16T14:59:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 -- start start 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8641f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 0x7fe86413d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641e62e0 con 0x7fe86413df80 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864152620 con 0x7fe86413e360 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.702+0000 7fe86b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe864152900 con 0x7fe8640cf9a0 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36422/0 (socket says 172.21.15.125:36422) 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 -- 172.21.15.125:0/2432394760 learned_addr learned my addr 172.21.15.125:0/2432394760 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe868c00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8641f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe869600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 0x7fe86413d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 -- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 msgr2=0x7fe8641f78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8641f78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 -- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 msgr2=0x7fe86413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 0x7fe86413d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe863e00640 1 -- 172.21.15.125:0/2432394760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c009400 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.703+0000 7fe869600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 0x7fe86413d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe863e00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c4d0 tx=0x7fe84c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c005ce0 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c005e40 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c007270 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe864152be0 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.704+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641e5ff0 con 0x7fe86413df80 2024-01-16T14:59:43.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.706+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe864153100 con 0x7fe86413df80 2024-01-16T14:59:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.706+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe84c007490 con 0x7fe86413df80 2024-01-16T14:59:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.707+0000 7fe860c00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 0x7fe83c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.707+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe84c089130 con 0x7fe86413df80 2024-01-16T14:59:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.708+0000 7fe868c00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 0x7fe83c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.708+0000 7fe868c00640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 0x7fe83c06d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe8540113e0 tx=0x7fe85400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.712+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe864153100 con 0x7fe86413df80 2024-01-16T14:59:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:43.907+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe864153100 con 0x7fe86413df80 2024-01-16T14:59:44.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.066+0000 7fe860c00640 1 -- 172.21.15.125:0/2432394760 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe864153100 con 0x7fe86413df80 2024-01-16T14:59:44.047 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T14:59:44.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 msgr2=0x7fe83c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 0x7fe83c06d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe8540113e0 tx=0x7fe85400b040 comp rx=0 tx=0).stop 2024-01-16T14:59:44.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 msgr2=0x7fe8641e5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c4d0 tx=0x7fe84c00c990 comp rx=0 tx=0).stop 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 shutdown_connections 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe86413e360 0x7fe86413d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe83c06b500 0x7fe83c06d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe86413df80 0x7fe8641e5ab0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 --2- 172.21.15.125:0/2432394760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8640cf9a0 0x7fe8641f78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 >> 172.21.15.125:0/2432394760 conn(0x7fe86406a800 msgr2=0x7fe86415ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 shutdown_connections 2024-01-16T14:59:44.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.070+0000 7fe86b600640 1 -- 172.21.15.125:0/2432394760 wait complete. 2024-01-16T14:59:44.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T14:59:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.295+0000 7f5420e00640 1 Processor -- start 2024-01-16T14:59:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.296+0000 7f5420e00640 1 -- start start 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.297+0000 7f5420e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f541c144d00 0x7f541c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5420e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0cd180 con 0x7f541c0d0610 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5420e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0cd420 con 0x7f541c0d0ac0 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5420e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c0d0ed0 con 0x7f541c144d00 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5419400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f541c0d0610 0x7f541c0cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57472/0 (socket says 172.21.15.125:57472) 2024-01-16T14:59:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5419400640 1 -- 172.21.15.125:0/3446854738 learned_addr learned my addr 172.21.15.125:0/3446854738 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394804437 0 0) 0x7f541c0cd180 con 0x7f541c0d0610 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53fc003520 con 0x7f541c0d0610 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.298+0000 7f5419e00640 1 --2- 172.21.15.125:0/3446854738 >> v2:172.21.15.137:3300/0 conn(0x7f541c144d00 0x7f541c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335385455 0 0) 0x7f53fc003520 con 0x7f541c0d0610 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 >> v2:172.21.15.137:3300/0 conn(0x7f541c144d00 msgr2=0x7f541c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 --2- 172.21.15.125:0/3446854738 >> v2:172.21.15.137:3300/0 conn(0x7f541c144d00 0x7f541c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 >> v1:172.21.15.137:6789/0 conn(0x7f541c0d0ac0 legacy=0x7f541c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0cf020 con 0x7f541c0d0610 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f540c0019f0 con 0x7f541c0d0610 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 >> v1:172.21.15.125:6789/0 conn(0x7f541c0d0610 legacy=0x7f541c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f53fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53fc0047c0 0x7f53fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53fc005180 0x7f53fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0d0ed0 con 0x7f53fc003970 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc00c0d0 con 0x7f53fc005180 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc00c290 con 0x7f53fc0047c0 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.299+0000 7f5419e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f53fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f541a800640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53fc005180 0x7f53fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f5419e00640 1 -- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53fc0047c0 msgr2=0x7f53fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f5419e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53fc0047c0 0x7f53fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f5419e00640 1 -- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53fc005180 msgr2=0x7f53fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f5419e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53fc005180 0x7f53fc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f5419e00640 1 -- 172.21.15.125:0/3446854738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc003520 con 0x7f53fc003970 2024-01-16T14:59:44.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f541a800640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53fc005180 0x7f53fc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:44.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.300+0000 7f541a800640 1 -- 172.21.15.125:0/3446854738 reap_dead start 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.301+0000 7f5419e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f53fc003d90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f540400ba30 tx=0x7f53fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30cb89144239a01 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.301+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404032640 con 0x7f53fc003970 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.301+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54040327a0 con 0x7f53fc003970 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.301+0000 7f5418a00640 1 -- 172.21.15.125:0/3446854738 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404032b30 con 0x7f53fc003970 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.301+0000 7f5420e00640 1 -- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 msgr2=0x7f53fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f53fc003d90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f540400ba30 tx=0x7f53fc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 -- 172.21.15.125:0/3446854738 shutdown_connections 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 --2- 172.21.15.125:0/3446854738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f53fc003d90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 -- 172.21.15.125:0/3446854738 >> 172.21.15.125:0/3446854738 conn(0x7f541c066690 msgr2=0x7f541c06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 -- 172.21.15.125:0/3446854738 shutdown_connections 2024-01-16T14:59:44.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.302+0000 7f5420e00640 1 -- 172.21.15.125:0/3446854738 wait complete. 2024-01-16T14:59:44.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.303+0000 7f5420e00640 1 Processor -- start 2024-01-16T14:59:44.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 -- start start 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541c0b49c0 0x7f541c154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541c15b920 0x7f541c15c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b5a60 con 0x7f53fc003970 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b6430 con 0x7f541c0b49c0 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.304+0000 7f5420e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c155390 con 0x7f541c15b920 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36432/0 (socket says 172.21.15.125:36432) 2024-01-16T14:59:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 -- 172.21.15.125:0/1811500277 learned_addr learned my addr 172.21.15.125:0/1811500277 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:44.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f541a800640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541c15b920 0x7f541c15c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 -- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541c15b920 msgr2=0x7f541c15c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541c15b920 0x7f541c15c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 -- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541c0b49c0 msgr2=0x7f541c154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541c0b49c0 0x7f541c154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 -- 172.21.15.125:0/1811500277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0b4e50 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.305+0000 7f5419e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f540402bec0 tx=0x7f5404004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.306+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540403b9f0 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.306+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540403bb50 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.306+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404030410 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.306+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0b5130 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.306+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c0b56f0 con 0x7f53fc003970 2024-01-16T14:59:44.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.307+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c0675d0 con 0x7f53fc003970 2024-01-16T14:59:44.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.308+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5404030570 con 0x7f53fc003970 2024-01-16T14:59:44.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.309+0000 7f5412000640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 0x7f53f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.309+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54040ac180 con 0x7f53fc003970 2024-01-16T14:59:44.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.309+0000 7f5419400640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 0x7f53f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.309+0000 7f5419400640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 0x7f53f406d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f540c001560 tx=0x7f540c0018a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.313+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5404076f60 con 0x7f53fc003970 2024-01-16T14:59:44.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.572+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f541c15aa50 con 0x7f53fc003970 2024-01-16T14:59:44.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.573+0000 7f5412000640 1 -- 172.21.15.125:0/1811500277 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f540407a580 con 0x7f53fc003970 2024-01-16T14:59:44.553 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.554 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:59:44.555 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.557 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.558 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.559 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.560 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T14:59:44.561 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:59:44.562 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T14:59:44.563 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:59:44.564 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:44.565 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T14:59:44.566 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.575+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 msgr2=0x7f53f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.575+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 0x7f53f406d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f540c001560 tx=0x7f540c0018a0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.575+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 msgr2=0x7f541c0b5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.575+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f540402bec0 tx=0x7f5404004730 comp rx=0 tx=0).stop 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 shutdown_connections 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541c15b920 0x7f541c15c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541c0b49c0 0x7f541c154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53f406b500 0x7f53f406d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 --2- 172.21.15.125:0/1811500277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53fc003970 0x7f541c0b5ef0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 >> 172.21.15.125:0/1811500277 conn(0x7f541c066690 msgr2=0x7f541c0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 shutdown_connections 2024-01-16T14:59:44.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.576+0000 7f5420e00640 1 -- 172.21.15.125:0/1811500277 wait complete. 2024-01-16T14:59:44.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:44.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.806+0000 7f6217200640 1 Processor -- start 2024-01-16T14:59:44.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.806+0000 7f6217200640 1 -- start start 2024-01-16T14:59:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6217200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 0x7f621815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6217200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f62180cd530 0x7f621813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6217200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180cdec0 con 0x7f62180cd9b0 2024-01-16T14:59:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6217200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62180ce1a0 con 0x7f62180cd530 2024-01-16T14:59:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6217200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62180ce480 con 0x7f621813cfa0 2024-01-16T14:59:44.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6216800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f62180cd9b0 0x7f621815d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57474/0 (socket says 172.21.15.125:57474) 2024-01-16T14:59:44.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6216800640 1 -- 172.21.15.125:0/627116676 learned_addr learned my addr 172.21.15.125:0/627116676 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:44.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.808+0000 7f6215e00640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 0x7f621815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f620fe00640 1 --2- 172.21.15.125:0/627116676 >> v2:172.21.15.137:3300/0 conn(0x7f62180cd530 0x7f621813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476719345 0 0) 0x7f62180cdec0 con 0x7f62180cd9b0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8003520 con 0x7f62180cd9b0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585092358 0 0) 0x7f61f8003520 con 0x7f62180cd9b0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 msgr2=0x7f621815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 0x7f621815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 >> v2:172.21.15.137:3300/0 conn(0x7f62180cd530 msgr2=0x7f621813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 --2- 172.21.15.125:0/627116676 >> v2:172.21.15.137:3300/0 conn(0x7f62180cd530 0x7f621813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180ce760 con 0x7f62180cd9b0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6210001a10 con 0x7f62180cd9b0 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 >> v1:172.21.15.125:6789/0 conn(0x7f62180cd9b0 legacy=0x7f621815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61f8003970 0x7f61f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f80047c0 0x7f61f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 0x7f61f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180ce1a0 con 0x7f61f8003970 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180ce480 con 0x7f61f80047c0 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f800db90 con 0x7f621813cfa0 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.809+0000 7f6215e00640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61f8003970 0x7f61f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.810+0000 7f6215e00640 1 -- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 msgr2=0x7f61f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.810+0000 7f6215e00640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f621813cfa0 0x7f61f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.810+0000 7f6215e00640 1 -- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f80047c0 msgr2=0x7f61f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.810+0000 7f6215e00640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f80047c0 0x7f61f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.810+0000 7f6215e00640 1 -- 172.21.15.125:0/627116676 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8003520 con 0x7f61f8003970 2024-01-16T14:59:44.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.811+0000 7f6215e00640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61f8003970 0x7f61f8003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6208009af0 tx=0x7f61f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25874a341414b8d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.811+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620802c710 con 0x7f61f8003970 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.811+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620802c870 con 0x7f61f8003970 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.811+0000 7f6215400640 1 -- 172.21.15.125:0/627116676 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620802cc00 con 0x7f61f8003970 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 -- 172.21.15.125:0/627116676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61f8003970 msgr2=0x7f61f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 --2- 172.21.15.125:0/627116676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61f8003970 0x7f61f8003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6208009af0 tx=0x7f61f80057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6216800640 1 -- 172.21.15.125:0/627116676 reap_dead start 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 -- 172.21.15.125:0/627116676 shutdown_connections 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 -- 172.21.15.125:0/627116676 >> 172.21.15.125:0/627116676 conn(0x7f621806a800 msgr2=0x7f621806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 -- 172.21.15.125:0/627116676 shutdown_connections 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.812+0000 7f6217200640 1 -- 172.21.15.125:0/627116676 wait complete. 2024-01-16T14:59:44.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.814+0000 7f6217200640 1 Processor -- start 2024-01-16T14:59:44.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.814+0000 7f6217200640 1 -- start start 2024-01-16T14:59:44.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f8003970 0x7f6218151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6218070160 0x7f62180705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218070b20 con 0x7f6218071320 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180ca1f0 con 0x7f61f8003970 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f6217200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621813c9f0 con 0x7f6218070160 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36446/0 (socket says 172.21.15.125:36446) 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 -- 172.21.15.125:0/3101790491 learned_addr learned my addr 172.21.15.125:0/3101790491 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 -- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6218070160 msgr2=0x7f62180705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6218070160 0x7f62180705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 -- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f8003970 msgr2=0x7f6218151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f8003970 0x7f6218151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.815+0000 7f620fe00640 1 -- 172.21.15.125:0/3101790491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180cb5c0 con 0x7f6218071320 2024-01-16T14:59:44.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f620fe00640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f620000e890 tx=0x7f620000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000c8e0 con 0x7f6218071320 2024-01-16T14:59:44.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620000ca60 con 0x7f6218071320 2024-01-16T14:59:44.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000ce10 con 0x7f6218071320 2024-01-16T14:59:44.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62180cb8a0 con 0x7f6218071320 2024-01-16T14:59:44.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.816+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180c9e80 con 0x7f6218071320 2024-01-16T14:59:44.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.817+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62180ca040 con 0x7f6218071320 2024-01-16T14:59:44.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.818+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6200013470 con 0x7f6218071320 2024-01-16T14:59:44.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.819+0000 7f620e000640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 0x7f61f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.819+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6200089da0 con 0x7f6218071320 2024-01-16T14:59:44.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.819+0000 7f6215e00640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 0x7f61f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.820+0000 7f6215e00640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 0x7f61f006d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6208036000 tx=0x7f620802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:44.824+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6200054b60 con 0x7f6218071320 2024-01-16T14:59:44.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.015+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6218067450 con 0x7f6218071320 2024-01-16T14:59:44.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.017+0000 7f620e000640 1 -- 172.21.15.125:0/3101790491 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f6200017090 con 0x7f6218071320 2024-01-16T14:59:44.997 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:44.997 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":41,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:44.079640+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.019+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 msgr2=0x7f61f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.019+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 0x7f61f006d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6208036000 tx=0x7f620802e000 comp rx=0 tx=0).stop 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.019+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 msgr2=0x7f62180ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.019+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f620000e890 tx=0x7f620000ed50 comp rx=0 tx=0).stop 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 shutdown_connections 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6218070160 0x7f62180705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f006b500 0x7f61f006d9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6218071320 0x7f62180ca680 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 --2- 172.21.15.125:0/3101790491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61f8003970 0x7f6218151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 >> 172.21.15.125:0/3101790491 conn(0x7f621806a800 msgr2=0x7f621815b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 shutdown_connections 2024-01-16T14:59:45.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.020+0000 7f6217200640 1 -- 172.21.15.125:0/3101790491 wait complete. 2024-01-16T14:59:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:45.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.248+0000 7f5483000640 1 Processor -- start 2024-01-16T14:59:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.248+0000 7f5483000640 1 -- start start 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.249+0000 7f5483000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f547c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f5483000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c15c5f0 con 0x7f547c13df80 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f5483000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c06c470 con 0x7f547c15f9c0 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f5483000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c06c750 con 0x7f547c13e360 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f547be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f547c13df80 0x7f547c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57478/0 (socket says 172.21.15.125:57478) 2024-01-16T14:59:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f547be00640 1 -- 172.21.15.125:0/542427880 learned_addr learned my addr 172.21.15.125:0/542427880 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857909811 0 0) 0x7f547c15c5f0 con 0x7f547c13df80 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c003520 con 0x7f547c13df80 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.250+0000 7f5473200640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f547c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131401659 0 0) 0x7f545c003520 con 0x7f547c13df80 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 msgr2=0x7f547c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f547c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 >> v1:172.21.15.137:6789/0 conn(0x7f547c15f9c0 legacy=0x7f547c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c06ca30 con 0x7f547c13df80 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f546c001670 con 0x7f547c13df80 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 >> v1:172.21.15.125:6789/0 conn(0x7f547c13df80 legacy=0x7f547c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545c0039a0 0x7f545c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f545c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f545c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c06c750 con 0x7f545c004d00 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c00bb90 con 0x7f545c0039a0 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545c00bd50 con 0x7f547c13e360 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f5481000640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f545c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f547be00640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545c0039a0 0x7f545c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.251+0000 7f5473200640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f545c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.252+0000 7f5481000640 1 -- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 msgr2=0x7f545c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.252+0000 7f5481000640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13e360 0x7f545c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.252+0000 7f5481000640 1 -- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545c0039a0 msgr2=0x7f545c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.252+0000 7f5481000640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545c0039a0 0x7f545c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.252+0000 7f5481000640 1 -- 172.21.15.125:0/542427880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c003520 con 0x7f545c004d00 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.253+0000 7f5481000640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f545c005180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5474001e50 tx=0x7f545c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd9bc65425bd8de server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.253+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474003b70 con 0x7f545c004d00 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.253+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474003cd0 con 0x7f545c004d00 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.253+0000 7f547b400640 1 -- 172.21.15.125:0/542427880 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474005310 con 0x7f545c004d00 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 -- 172.21.15.125:0/542427880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 msgr2=0x7f545c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 --2- 172.21.15.125:0/542427880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f545c005180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5474001e50 tx=0x7f545c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5481000640 1 -- 172.21.15.125:0/542427880 reap_dead start 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 -- 172.21.15.125:0/542427880 shutdown_connections 2024-01-16T14:59:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 -- 172.21.15.125:0/542427880 >> 172.21.15.125:0/542427880 conn(0x7f547c06a800 msgr2=0x7f547c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 -- 172.21.15.125:0/542427880 shutdown_connections 2024-01-16T14:59:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.254+0000 7f5483000640 1 -- 172.21.15.125:0/542427880 wait complete. 2024-01-16T14:59:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.256+0000 7f5483000640 1 Processor -- start 2024-01-16T14:59:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.256+0000 7f5483000640 1 -- start start 2024-01-16T14:59:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f547c13d710 0x7f547c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13db70 0x7f547c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1517e0 con 0x7f545c004d00 2024-01-16T14:59:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c151c40 con 0x7f547c13d710 2024-01-16T14:59:45.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.257+0000 7f5483000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c151ee0 con 0x7f547c13db70 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36450/0 (socket says 172.21.15.125:36450) 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 -- 172.21.15.125:0/3573838670 learned_addr learned my addr 172.21.15.125:0/3573838670 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f5481000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13db70 0x7f547c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f5473200640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f547c13d710 0x7f547c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 -- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13db70 msgr2=0x7f547c14b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13db70 0x7f547c14b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 -- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f547c13d710 msgr2=0x7f547c14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f547c13d710 0x7f547c14cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 -- 172.21.15.125:0/3573838670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c1521c0 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.258+0000 7f547be00640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f546c0072c0 tx=0x7f546c0076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.259+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00f9c0 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.259+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00fb20 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.259+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00fed0 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.259+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c1524a0 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.259+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c152a60 con 0x7f545c004d00 2024-01-16T14:59:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.260+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c0675d0 con 0x7f545c004d00 2024-01-16T14:59:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.261+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f546c01d460 con 0x7f545c004d00 2024-01-16T14:59:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.262+0000 7f5478c00640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 0x7f545406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.262+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f546c089ff0 con 0x7f545c004d00 2024-01-16T14:59:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.262+0000 7f5473200640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 0x7f545406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.263+0000 7f5473200640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 0x7f545406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f547c0b0b40 tx=0x7f5464008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.266+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f546c054d40 con 0x7f545c004d00 2024-01-16T14:59:45.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.453+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f547c06df00 con 0x7f545c004d00 2024-01-16T14:59:45.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.454+0000 7f5478c00640 1 -- 172.21.15.125:0/3573838670 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f546c058360 con 0x7f545c004d00 2024-01-16T14:59:45.434 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 msgr2=0x7f545406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 0x7f545406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f547c0b0b40 tx=0x7f5464008040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 msgr2=0x7f547c152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f546c0072c0 tx=0x7f546c0076c0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 shutdown_connections 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f547c13db70 0x7f547c14b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f547c13d710 0x7f547c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f545406b500 0x7f545406d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 --2- 172.21.15.125:0/3573838670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f545c004d00 0x7f547c152f80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 >> 172.21.15.125:0/3573838670 conn(0x7f547c06a800 msgr2=0x7f547c1538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 shutdown_connections 2024-01-16T14:59:45.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.457+0000 7f5483000640 1 -- 172.21.15.125:0/3573838670 wait complete. 2024-01-16T14:59:45.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:59:45.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.677+0000 7fdf90a00640 1 Processor -- start 2024-01-16T14:59:45.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.677+0000 7fdf90a00640 1 -- start start 2024-01-16T14:59:45.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf90a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf880cf9a0 0x7fdf88070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf90a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88071690 con 0x7fdf8814b2c0 2024-01-16T14:59:45.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf90a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88071970 con 0x7fdf88070ea0 2024-01-16T14:59:45.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf90a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8814b6a0 con 0x7fdf880cf9a0 2024-01-16T14:59:45.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf8e000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdf8814b2c0 0x7fdf880cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57480/0 (socket says 172.21.15.125:57480) 2024-01-16T14:59:45.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.679+0000 7fdf8e000640 1 -- 172.21.15.125:0/1452351662 learned_addr learned my addr 172.21.15.125:0/1452351662 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8d600640 1 --2- 172.21.15.125:0/1452351662 >> v2:172.21.15.137:3300/0 conn(0x7fdf880cf9a0 0x7fdf88070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357042156 0 0) 0x7fdf88071690 con 0x7fdf8814b2c0 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf68003520 con 0x7fdf8814b2c0 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979726567 0 0) 0x7fdf68003520 con 0x7fdf8814b2c0 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 >> v2:172.21.15.137:3300/0 conn(0x7fdf880cf9a0 msgr2=0x7fdf88070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 --2- 172.21.15.125:0/1452351662 >> v2:172.21.15.137:3300/0 conn(0x7fdf880cf9a0 0x7fdf88070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 >> v1:172.21.15.137:6789/0 conn(0x7fdf88070ea0 legacy=0x7fdf880cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88071c50 con 0x7fdf8814b2c0 2024-01-16T14:59:45.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdf78001670 con 0x7fdf8814b2c0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 >> v1:172.21.15.125:6789/0 conn(0x7fdf8814b2c0 legacy=0x7fdf880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf680039c0 0x7fdf68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.680+0000 7fdf8cc00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 0x7fdf68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8cc00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf68005180 0x7fdf6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8814b6a0 con 0x7fdf680047c0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6800c080 con 0x7fdf680039c0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6800c240 con 0x7fdf68005180 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 0x7fdf68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8e000640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf680039c0 0x7fdf68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8ea00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf68005180 0x7fdf6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 -- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf68005180 msgr2=0x7fdf6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf68005180 0x7fdf6800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 -- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf680039c0 msgr2=0x7fdf68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf680039c0 0x7fdf68003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8d600640 1 -- 172.21.15.125:0/1452351662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf68003520 con 0x7fdf680047c0 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.681+0000 7fdf8ea00640 1 -- 172.21.15.125:0/1452351662 reap_dead start 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.682+0000 7fdf8d600640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 0x7fdf68004c40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009ac0 tx=0x7fdf680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e58362e6939d5b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7002c6b0 con 0x7fdf680047c0 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7002c810 con 0x7fdf680047c0 2024-01-16T14:59:45.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf8cc00640 1 -- 172.21.15.125:0/1452351662 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7002cba0 con 0x7fdf680047c0 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 -- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 msgr2=0x7fdf68004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 0x7fdf68004c40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009ac0 tx=0x7fdf680057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 -- 172.21.15.125:0/1452351662 shutdown_connections 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 --2- 172.21.15.125:0/1452351662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf680047c0 0x7fdf68004c40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 -- 172.21.15.125:0/1452351662 >> 172.21.15.125:0/1452351662 conn(0x7fdf8806a800 msgr2=0x7fdf8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 -- 172.21.15.125:0/1452351662 shutdown_connections 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.683+0000 7fdf90a00640 1 -- 172.21.15.125:0/1452351662 wait complete. 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.685+0000 7fdf90a00640 1 Processor -- start 2024-01-16T14:59:45.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.685+0000 7fdf90a00640 1 -- start start 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf680047c0 0x7fdf88142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 0x7fdf8815f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815ee20 con 0x7fdf8813fdc0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815df40 con 0x7fdf8815dac0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf90a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815f780 con 0x7fdf680047c0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36466/0 (socket says 172.21.15.125:36466) 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.686+0000 7fdf8ea00640 1 -- 172.21.15.125:0/417349389 learned_addr learned my addr 172.21.15.125:0/417349389 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8e000640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf680047c0 0x7fdf88142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8d600640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 0x7fdf8815f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 -- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf680047c0 msgr2=0x7fdf88142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf680047c0 0x7fdf88142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 -- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 msgr2=0x7fdf8815f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 0x7fdf8815f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 -- 172.21.15.125:0/417349389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8815e1a0 con 0x7fdf8813fdc0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8d600640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 0x7fdf8815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf8ea00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdf8806d650 tx=0x7fdf8000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.687+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80003b70 con 0x7fdf8813fdc0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.688+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80003cd0 con 0x7fdf8813fdc0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.688+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80005290 con 0x7fdf8813fdc0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.688+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8815e480 con 0x7fdf8813fdc0 2024-01-16T14:59:45.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.688+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8815ea40 con 0x7fdf8813fdc0 2024-01-16T14:59:45.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.689+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf880675d0 con 0x7fdf8813fdc0 2024-01-16T14:59:45.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.690+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdf800053f0 con 0x7fdf8813fdc0 2024-01-16T14:59:45.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.691+0000 7fdf86000640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.691+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdf80089640 con 0x7fdf8813fdc0 2024-01-16T14:59:45.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.691+0000 7fdf8e000640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.692+0000 7fdf8e000640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 0x7fdf6006d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf78000ff0 tx=0x7fdf78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.696+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf80054390 con 0x7fdf8813fdc0 2024-01-16T14:59:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.890+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdf8813fba0 con 0x7fdf8813fdc0 2024-01-16T14:59:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.891+0000 7fdf86000640 1 -- 172.21.15.125:0/417349389 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdf80018090 con 0x7fdf8813fdc0 2024-01-16T14:59:45.871 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T14:59:45.872 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 msgr2=0x7fdf6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 0x7fdf6006d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf78000ff0 tx=0x7fdf78009040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 msgr2=0x7fdf88140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdf8806d650 tx=0x7fdf8000dc00 comp rx=0 tx=0).stop 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 shutdown_connections 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf8813fdc0 0x7fdf88140240 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8815dac0 0x7fdf8815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 --2- 172.21.15.125:0/417349389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf680047c0 0x7fdf88142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.894+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 >> 172.21.15.125:0/417349389 conn(0x7fdf8806a800 msgr2=0x7fdf880aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.895+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 shutdown_connections 2024-01-16T14:59:45.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:45.895+0000 7fdf90a00640 1 -- 172.21.15.125:0/417349389 wait complete. 2024-01-16T14:59:45.893 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:59:45.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:59:46.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.129+0000 7faa21800640 1 Processor -- start 2024-01-16T14:59:46.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.129+0000 7faa21800640 1 -- start start 2024-01-16T14:59:46.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.131+0000 7faa21800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faa1c13df80 0x7faa1c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.131+0000 7faa21800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c06c470 con 0x7faa1c13e360 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.131+0000 7faa21800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c06c750 con 0x7faa1c0cf9a0 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.131+0000 7faa21800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c06ca30 con 0x7faa1c13df80 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa1aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faa1c13df80 0x7faa1c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa13e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faa1c13e360 0x7faa1c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57484/0 (socket says 172.21.15.125:57484) 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa13e00640 1 -- 172.21.15.125:0/4005202195 learned_addr learned my addr 172.21.15.125:0/4005202195 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:46.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa1aa00640 1 -- 172.21.15.125:0/4005202195 >> v1:172.21.15.137:6789/0 conn(0x7faa1c0cf9a0 legacy=0x7faa1c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa1aa00640 1 -- 172.21.15.125:0/4005202195 >> v1:172.21.15.125:6789/0 conn(0x7faa1c13e360 legacy=0x7faa1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa1aa00640 1 -- 172.21.15.125:0/4005202195 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c06cd10 con 0x7faa1c13df80 2024-01-16T14:59:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.132+0000 7faa1a000640 1 -- 172.21.15.125:0/4005202195 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085701917 0 0) 0x7faa1c06c470 con 0x7faa1c13e360 2024-01-16T14:59:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.133+0000 7faa1aa00640 1 --2- 172.21.15.125:0/4005202195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13df80 0x7faa1c15bee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097a0 tx=0x7faa0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daac446bdcd27c03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa1a000640 1 -- 172.21.15.125:0/4005202195 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c031830 con 0x7faa1c13df80 2024-01-16T14:59:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa1a000640 1 -- 172.21.15.125:0/4005202195 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c031990 con 0x7faa1c13df80 2024-01-16T14:59:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa1a000640 1 -- 172.21.15.125:0/4005202195 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c031d20 con 0x7faa1c13df80 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 -- 172.21.15.125:0/4005202195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13df80 msgr2=0x7faa1c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 --2- 172.21.15.125:0/4005202195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13df80 0x7faa1c15bee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097a0 tx=0x7faa0c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 -- 172.21.15.125:0/4005202195 shutdown_connections 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 --2- 172.21.15.125:0/4005202195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13df80 0x7faa1c15bee0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 -- 172.21.15.125:0/4005202195 >> 172.21.15.125:0/4005202195 conn(0x7faa1c06a800 msgr2=0x7faa1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 -- 172.21.15.125:0/4005202195 shutdown_connections 2024-01-16T14:59:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.134+0000 7faa21800640 1 -- 172.21.15.125:0/4005202195 wait complete. 2024-01-16T14:59:46.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.136+0000 7faa21800640 1 Processor -- start 2024-01-16T14:59:46.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.136+0000 7faa21800640 1 -- start start 2024-01-16T14:59:46.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa1c0cf9a0 0x7faa1c153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 0x7faa1c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1562a0 con 0x7faa1c13e360 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c156580 con 0x7faa1c13df80 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa21800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c156860 con 0x7faa1c0cf9a0 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.137+0000 7faa1b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36472/0 (socket says 172.21.15.125:36472) 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 -- 172.21.15.125:0/3645505452 learned_addr learned my addr 172.21.15.125:0/3645505452 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1aa00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa1c0cf9a0 0x7faa1c153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa13e00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 0x7faa1c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 -- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa1c0cf9a0 msgr2=0x7faa1c153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa1c0cf9a0 0x7faa1c153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 -- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 msgr2=0x7faa1c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 0x7faa1c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 -- 172.21.15.125:0/3645505452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c009400 con 0x7faa1c13e360 2024-01-16T14:59:46.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa13e00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 0x7faa1c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:46.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.138+0000 7faa1b400640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faa1400b9e0 tx=0x7faa1400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.139+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14003b70 con 0x7faa1c13e360 2024-01-16T14:59:46.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.139+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14003cd0 con 0x7faa1c13e360 2024-01-16T14:59:46.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.139+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400cb60 con 0x7faa1c13e360 2024-01-16T14:59:46.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.139+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c156b40 con 0x7faa1c13e360 2024-01-16T14:59:46.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.139+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c1536b0 con 0x7faa1c13e360 2024-01-16T14:59:46.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.140+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c157060 con 0x7faa1c13e360 2024-01-16T14:59:46.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.141+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7faa1400ccc0 con 0x7faa1c13e360 2024-01-16T14:59:46.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.142+0000 7faa12a00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 0x7fa9f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.142+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faa14089920 con 0x7faa1c13e360 2024-01-16T14:59:46.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.142+0000 7faa1aa00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 0x7fa9f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.143+0000 7faa1aa00640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 0x7fa9f406d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faa0c036000 tx=0x7faa0c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.147+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa1c157060 con 0x7faa1c13e360 2024-01-16T14:59:46.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:46.340+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faa1c157060 con 0x7faa1c13e360 2024-01-16T14:59:47.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.095+0000 7faa12a00640 1 -- 172.21.15.125:0/3645505452 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7faa1c157060 con 0x7faa1c13e360 2024-01-16T14:59:47.076 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 msgr2=0x7fa9f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 0x7fa9f406d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faa0c036000 tx=0x7faa0c03c040 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 msgr2=0x7faa1c157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faa1400b9e0 tx=0x7faa1400bea0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 shutdown_connections 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9f406b500 0x7fa9f406d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa1c13e360 0x7faa1c157340 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c13df80 0x7faa1c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 --2- 172.21.15.125:0/3645505452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa1c0cf9a0 0x7faa1c153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 >> 172.21.15.125:0/3645505452 conn(0x7faa1c06a800 msgr2=0x7faa1c153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.099+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 shutdown_connections 2024-01-16T14:59:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.100+0000 7faa21800640 1 -- 172.21.15.125:0/3645505452 wait complete. 2024-01-16T14:59:47.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:59:47.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.328+0000 7f3b3fa00640 1 Processor -- start 2024-01-16T14:59:47.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.329+0000 7f3b3fa00640 1 -- start start 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.330+0000 7f3b3fa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b3815dd40 0x7f3b3815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.330+0000 7f3b3fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815bed0 0x7f3b3815c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.330+0000 7f3b3fa00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3815e6e0 con 0x7f3b38157dc0 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.330+0000 7f3b3fa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b381581a0 con 0x7f3b3815dd40 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.330+0000 7f3b3fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3815ba40 con 0x7f3b3815bed0 2024-01-16T14:59:47.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b3d000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b38157dc0 0x7f3b38070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52818/0 (socket says 172.21.15.125:52818) 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b3d000640 1 -- 172.21.15.125:0/259732546 learned_addr learned my addr 172.21.15.125:0/259732546 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/259732546 >> v2:172.21.15.125:3300/0 conn(0x7f3b3815dd40 0x7f3b3815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b3da00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815bed0 0x7f3b3815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b37e00640 1 -- 172.21.15.125:0/259732546 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180923347 0 0) 0x7f3b3815e6e0 con 0x7f3b38157dc0 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b37e00640 1 -- 172.21.15.125:0/259732546 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003520 con 0x7f3b38157dc0 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b2fc00640 1 -- 172.21.15.125:0/259732546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815bed0 msgr2=0x7f3b3815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815bed0 0x7f3b3815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b2fc00640 1 -- 172.21.15.125:0/259732546 >> v1:172.21.15.125:6789/0 conn(0x7f3b38157dc0 legacy=0x7f3b38070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.331+0000 7f3b2fc00640 1 -- 172.21.15.125:0/259732546 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3815eb10 con 0x7f3b3815dd40 2024-01-16T14:59:47.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.332+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3815dd40 0x7f3b3815e1a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b200097a0 tx=0x7f3b2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e3c2af7479832b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b37e00640 1 -- 172.21.15.125:0/259732546 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031830 con 0x7f3b3815dd40 2024-01-16T14:59:47.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b37e00640 1 -- 172.21.15.125:0/259732546 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b20031990 con 0x7f3b3815dd40 2024-01-16T14:59:47.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b37e00640 1 -- 172.21.15.125:0/259732546 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031d00 con 0x7f3b3815dd40 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 -- 172.21.15.125:0/259732546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3815dd40 msgr2=0x7f3b3815e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3815dd40 0x7f3b3815e1a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b200097a0 tx=0x7f3b2002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 -- 172.21.15.125:0/259732546 shutdown_connections 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815bed0 0x7f3b3815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/259732546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3815dd40 0x7f3b3815e1a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 -- 172.21.15.125:0/259732546 >> 172.21.15.125:0/259732546 conn(0x7f3b380b0be0 msgr2=0x7f3b380b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 -- 172.21.15.125:0/259732546 shutdown_connections 2024-01-16T14:59:47.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.333+0000 7f3b3fa00640 1 -- 172.21.15.125:0/259732546 wait complete. 2024-01-16T14:59:47.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.335+0000 7f3b3fa00640 1 Processor -- start 2024-01-16T14:59:47.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.335+0000 7f3b3fa00640 1 -- start start 2024-01-16T14:59:47.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 0x7f3b380ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815dd40 0x7f3b380aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380adea0 con 0x7f3b38157dc0 2024-01-16T14:59:47.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380cd330 con 0x7f3b3815bed0 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380cd610 con 0x7f3b3815dd40 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.336+0000 7f3b3d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33184/0 (socket says 172.21.15.125:33184) 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 -- 172.21.15.125:0/3947183025 learned_addr learned my addr 172.21.15.125:0/3947183025 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3da00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815dd40 0x7f3b380aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 0x7f3b380ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 -- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815dd40 msgr2=0x7f3b380aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815dd40 0x7f3b380aca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 -- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 msgr2=0x7f3b380ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 0x7f3b380ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 -- 172.21.15.125:0/3947183025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20009400 con 0x7f3b38157dc0 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 0x7f3b380ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:47.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.337+0000 7f3b3d000640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c430 tx=0x7f3b2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.338+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28005ce0 con 0x7f3b38157dc0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.338+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b28005e40 con 0x7f3b38157dc0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.338+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b280072f0 con 0x7f3b38157dc0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.338+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b380cd8f0 con 0x7f3b38157dc0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.338+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b380cdeb0 con 0x7f3b38157dc0 2024-01-16T14:59:47.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.339+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b38067670 con 0x7f3b38157dc0 2024-01-16T14:59:47.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.340+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3b28007450 con 0x7f3b38157dc0 2024-01-16T14:59:47.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.341+0000 7f3b35600640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.341+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3b28089960 con 0x7f3b38157dc0 2024-01-16T14:59:47.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.341+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.342+0000 7f3b2fc00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036000 tx=0x7f3b2003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.345+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b28054630 con 0x7f3b38157dc0 2024-01-16T14:59:47.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:47.537+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3b380b1760 con 0x7f3b38157dc0 2024-01-16T14:59:48.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.102+0000 7f3b35600640 1 -- 172.21.15.125:0/3947183025 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3b280170d0 con 0x7f3b38157dc0 2024-01-16T14:59:48.083 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T14:59:48.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.106+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 msgr2=0x7f3b1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.106+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036000 tx=0x7f3b2003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:48.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 msgr2=0x7f3b380cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c430 tx=0x7f3b2800c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 shutdown_connections 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3815dd40 0x7f3b380aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b3815bed0 0x7f3b380ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 --2- 172.21.15.125:0/3947183025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b38157dc0 0x7f3b380cc750 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.107+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 >> 172.21.15.125:0/3947183025 conn(0x7f3b380b0be0 msgr2=0x7f3b38159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.108+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 shutdown_connections 2024-01-16T14:59:48.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.109+0000 7f3b3fa00640 1 -- 172.21.15.125:0/3947183025 wait complete. 2024-01-16T14:59:48.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:59:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.326+0000 7fa6fce00640 1 Processor -- start 2024-01-16T14:59:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.326+0000 7fa6fce00640 1 -- start start 2024-01-16T14:59:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6f8143ae0 0x7fa6f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa6f815e1e0 0x7fa6f815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f80cfc30 con 0x7fa6f8143ae0 2024-01-16T14:59:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f80b1ee0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6fce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80b2780 con 0x7fa6f80cf7d0 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6f5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6f8143ae0 0x7fa6f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6f5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6f8143ae0 0x7fa6f80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33196/0 (socket says 172.21.15.125:33196) 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.328+0000 7fa6f5e00640 1 -- 172.21.15.125:0/1507944303 learned_addr learned my addr 172.21.15.125:0/1507944303 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5400640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5e00640 1 -- 172.21.15.125:0/1507944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 msgr2=0x7fa6f815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f6800640 1 --2- 172.21.15.125:0/1507944303 >> v2:172.21.15.137:3300/0 conn(0x7fa6f815e1e0 0x7fa6f815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5e00640 1 -- 172.21.15.125:0/1507944303 >> v2:172.21.15.137:3300/0 conn(0x7fa6f815e1e0 msgr2=0x7fa6f815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/1507944303 >> v2:172.21.15.137:3300/0 conn(0x7fa6f815e1e0 0x7fa6f815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5e00640 1 -- 172.21.15.125:0/1507944303 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f815e990 con 0x7fa6f8143ae0 2024-01-16T14:59:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.329+0000 7fa6f5400640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.330+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f8143ae0 0x7fa6f80cf290 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8072160 tx=0x7fa6e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df59280b2f801c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.330+0000 7fa6f4a00640 1 -- 172.21.15.125:0/1507944303 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8031870 con 0x7fa6f8143ae0 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.330+0000 7fa6f4a00640 1 -- 172.21.15.125:0/1507944303 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e80319d0 con 0x7fa6f8143ae0 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.330+0000 7fa6f4a00640 1 -- 172.21.15.125:0/1507944303 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8031d60 con 0x7fa6f8143ae0 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 -- 172.21.15.125:0/1507944303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f8143ae0 msgr2=0x7fa6f80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f8143ae0 0x7fa6f80cf290 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8072160 tx=0x7fa6e802b910 comp rx=0 tx=0).stop 2024-01-16T14:59:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 -- 172.21.15.125:0/1507944303 shutdown_connections 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 --2- 172.21.15.125:0/1507944303 >> v2:172.21.15.137:3300/0 conn(0x7fa6f815e1e0 0x7fa6f815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 --2- 172.21.15.125:0/1507944303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f8143ae0 0x7fa6f80cf290 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 -- 172.21.15.125:0/1507944303 >> 172.21.15.125:0/1507944303 conn(0x7fa6f806a800 msgr2=0x7fa6f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 -- 172.21.15.125:0/1507944303 shutdown_connections 2024-01-16T14:59:48.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.331+0000 7fa6fce00640 1 -- 172.21.15.125:0/1507944303 wait complete. 2024-01-16T14:59:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.333+0000 7fa6fce00640 1 Processor -- start 2024-01-16T14:59:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.333+0000 7fa6fce00640 1 -- start start 2024-01-16T14:59:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f8142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6f8143ae0 0x7fa6f8164d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f81663d0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80ab4f0 con 0x7fa6f8143ae0 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6fce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8165850 con 0x7fa6f80cf7d0 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33198/0 (socket says 172.21.15.125:33198) 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6f6800640 1 -- 172.21.15.125:0/3406768016 learned_addr learned my addr 172.21.15.125:0/3406768016 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.334+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f8142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 -- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 msgr2=0x7fa6f8142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f8142d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 -- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6f8143ae0 msgr2=0x7fa6f8164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6f8143ae0 0x7fa6f8164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 -- 172.21.15.125:0/3406768016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80094d0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6f6800640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cc30 tx=0x7fa6f000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000ce40 con 0x7fa6f815e1e0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0003990 con 0x7fa6f815e1e0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0003d20 con 0x7fa6f815e1e0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f8165b30 con 0x7fa6f815e1e0 2024-01-16T14:59:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.335+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f81660f0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.336+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f80ab6b0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.338+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa6f0003e80 con 0x7fa6f815e1e0 2024-01-16T14:59:48.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.338+0000 7fa6ee000640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.339+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa6f008a470 con 0x7fa6f815e1e0 2024-01-16T14:59:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.339+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.339+0000 7fa6f5e00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 0x7fa6d006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.343+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6f0054ed0 con 0x7fa6f815e1e0 2024-01-16T14:59:48.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:48.570+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa6f80675d0 con 0x7fa6f815e1e0 2024-01-16T14:59:49.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.122+0000 7fa6ee000640 1 -- 172.21.15.125:0/3406768016 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v24) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa6f001d0c0 con 0x7fa6f815e1e0 2024-01-16T14:59:49.103 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 6 and data pool 7 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.127+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 msgr2=0x7fa6d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.127+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 0x7fa6d006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e803c040 comp rx=0 tx=0).stop 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.127+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 msgr2=0x7fa6f80b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.127+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cc30 tx=0x7fa6f000a5c0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 shutdown_connections 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6f815e1e0 0x7fa6f80b31b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6f8143ae0 0x7fa6f8164d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 --2- 172.21.15.125:0/3406768016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6f80cf7d0 0x7fa6f8142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 >> 172.21.15.125:0/3406768016 conn(0x7fa6f806a800 msgr2=0x7fa6f815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 shutdown_connections 2024-01-16T14:59:49.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.128+0000 7fa6fce00640 1 -- 172.21.15.125:0/3406768016 wait complete. 2024-01-16T14:59:49.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:49.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.345+0000 7f2f48800640 1 Processor -- start 2024-01-16T14:59:49.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.345+0000 7f2f48800640 1 -- start start 2024-01-16T14:59:49.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f48800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f4013df80 0x7f2f4015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f48800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4006c470 con 0x7f2f4013e360 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f48800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4006c750 con 0x7f2f400cf9a0 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f48800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4006ca30 con 0x7f2f4013df80 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f45e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f4013df80 0x7f2f4015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f45400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2f4013e360 0x7f2f400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52834/0 (socket says 172.21.15.125:52834) 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.346+0000 7f2f45400640 1 -- 172.21.15.125:0/3575349976 learned_addr learned my addr 172.21.15.125:0/3575349976 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.347+0000 7f2f45e00640 1 -- 172.21.15.125:0/3575349976 >> v1:172.21.15.137:6789/0 conn(0x7f2f400cf9a0 legacy=0x7f2f400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.347+0000 7f2f45e00640 1 -- 172.21.15.125:0/3575349976 >> v1:172.21.15.125:6789/0 conn(0x7f2f4013e360 legacy=0x7f2f400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.347+0000 7f2f45e00640 1 -- 172.21.15.125:0/3575349976 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4006cd10 con 0x7f2f4013df80 2024-01-16T14:59:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.347+0000 7f2f44a00640 1 -- 172.21.15.125:0/3575349976 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161015074 0 0) 0x7f2f4006c470 con 0x7f2f4013e360 2024-01-16T14:59:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.348+0000 7f2f45e00640 1 --2- 172.21.15.125:0/3575349976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013df80 0x7f2f4015bee0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad7185ef3a864e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.348+0000 7f2f44a00640 1 -- 172.21.15.125:0/3575349976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30031830 con 0x7f2f4013df80 2024-01-16T14:59:49.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.348+0000 7f2f44a00640 1 -- 172.21.15.125:0/3575349976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f30031990 con 0x7f2f4013df80 2024-01-16T14:59:49.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.348+0000 7f2f44a00640 1 -- 172.21.15.125:0/3575349976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30031d20 con 0x7f2f4013df80 2024-01-16T14:59:49.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 -- 172.21.15.125:0/3575349976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013df80 msgr2=0x7f2f4015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 --2- 172.21.15.125:0/3575349976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013df80 0x7f2f4015bee0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 -- 172.21.15.125:0/3575349976 shutdown_connections 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 --2- 172.21.15.125:0/3575349976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013df80 0x7f2f4015bee0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 -- 172.21.15.125:0/3575349976 >> 172.21.15.125:0/3575349976 conn(0x7f2f4006a800 msgr2=0x7f2f4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 -- 172.21.15.125:0/3575349976 shutdown_connections 2024-01-16T14:59:49.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.349+0000 7f2f48800640 1 -- 172.21.15.125:0/3575349976 wait complete. 2024-01-16T14:59:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.351+0000 7f2f48800640 1 Processor -- start 2024-01-16T14:59:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.351+0000 7f2f48800640 1 -- start start 2024-01-16T14:59:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f400cf9a0 0x7f2f400a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f4013df80 0x7f2f4015a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400702c0 con 0x7f2f4013e360 2024-01-16T14:59:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40070510 con 0x7f2f4013df80 2024-01-16T14:59:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f48800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400707f0 con 0x7f2f400cf9a0 2024-01-16T14:59:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f46800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f46800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33226/0 (socket says 172.21.15.125:33226) 2024-01-16T14:59:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.352+0000 7f2f46800640 1 -- 172.21.15.125:0/2665895838 learned_addr learned my addr 172.21.15.125:0/2665895838 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f45e00640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f400cf9a0 0x7f2f400a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f45400640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f4013df80 0x7f2f4015a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 -- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f400cf9a0 msgr2=0x7f2f400a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f400cf9a0 0x7f2f400a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 -- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f4013df80 msgr2=0x7f2f4015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f4013df80 0x7f2f4015a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 -- 172.21.15.125:0/2665895838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f30009400 con 0x7f2f4013e360 2024-01-16T14:59:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f46800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800b9e0 tx=0x7f2f3800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38003b70 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f38003cd0 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f40070ad0 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800cb60 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.353+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f400a8a00 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.354+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400675d0 con 0x7f2f4013e360 2024-01-16T14:59:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.355+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2f3800ccc0 con 0x7f2f4013e360 2024-01-16T14:59:49.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.356+0000 7f2f3e000640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 0x7f2f1806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.356+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f38089d60 con 0x7f2f4013e360 2024-01-16T14:59:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.356+0000 7f2f45e00640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 0x7f2f1806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.357+0000 7f2f45e00640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 0x7f2f1806d9f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036000 tx=0x7f2f30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.361+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f380547d0 con 0x7f2f4013e360 2024-01-16T14:59:49.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.545+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f4006df00 con 0x7f2f4013e360 2024-01-16T14:59:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.546+0000 7f2f3e000640 1 -- 172.21.15.125:0/2665895838 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16574 (secure 0 0 0) 0x7f2f38057df0 con 0x7f2f4013e360 2024-01-16T14:59:49.527 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:49.527 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:48.571125+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:46.393145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:47.589538+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:49.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 msgr2=0x7f2f1806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 0x7f2f1806d9f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036000 tx=0x7f2f30008030 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 msgr2=0x7f2f400a8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800b9e0 tx=0x7f2f3800bea0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 shutdown_connections 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f1806b530 0x7f2f1806d9f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4013e360 0x7f2f400a8e60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f4013df80 0x7f2f4015a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 --2- 172.21.15.125:0/2665895838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f400cf9a0 0x7f2f400a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 >> 172.21.15.125:0/2665895838 conn(0x7f2f4006a800 msgr2=0x7f2f40153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 shutdown_connections 2024-01-16T14:59:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.549+0000 7f2f48800640 1 -- 172.21.15.125:0/2665895838 wait complete. 2024-01-16T14:59:49.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.764+0000 7f94b3e00640 1 Processor -- start 2024-01-16T14:59:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.764+0000 7f94b3e00640 1 -- start start 2024-01-16T14:59:49.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.765+0000 7f94b3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94b4143c40 0x7f94b4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.765+0000 7f94b3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.765+0000 7f94b3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b4070e50 con 0x7f94b40cca40 2024-01-16T14:59:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.765+0000 7f94b3e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b40723c0 con 0x7f94b4143c40 2024-01-16T14:59:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.765+0000 7f94b3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40cce70 con 0x7f94b40717d0 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94b4143c40 0x7f94b4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94b4143c40 0x7f94b4071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33230/0 (socket says 172.21.15.125:33230) 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 -- 172.21.15.125:0/782397304 learned_addr learned my addr 172.21.15.125:0/782397304 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b3400640 1 --1- 172.21.15.125:0/782397304 >> v1:172.21.15.137:6789/0 conn(0x7f94b40cca40 0x7f94b415eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:36794/0 (socket says 172.21.15.125:36794) 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94abe00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 -- 172.21.15.125:0/782397304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 msgr2=0x7f94b40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 -- 172.21.15.125:0/782397304 >> v1:172.21.15.137:6789/0 conn(0x7f94b40cca40 legacy=0x7f94b415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.766+0000 7f94b2a00640 1 -- 172.21.15.125:0/782397304 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b406e170 con 0x7f94b4143c40 2024-01-16T14:59:49.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.767+0000 7f94b2a00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b4071290 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f1c975c7fcba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b2000640 1 -- 172.21.15.125:0/782397304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4031640 con 0x7f94b4143c40 2024-01-16T14:59:49.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b2000640 1 -- 172.21.15.125:0/782397304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a40317a0 con 0x7f94b4143c40 2024-01-16T14:59:49.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b2000640 1 -- 172.21.15.125:0/782397304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4031b30 con 0x7f94b4143c40 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 -- 172.21.15.125:0/782397304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 msgr2=0x7f94b4071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b4071290 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a402b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 -- 172.21.15.125:0/782397304 shutdown_connections 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 --2- 172.21.15.125:0/782397304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b4071290 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.768+0000 7f94b3e00640 1 -- 172.21.15.125:0/782397304 >> 172.21.15.125:0/782397304 conn(0x7f94b406a800 msgr2=0x7f94b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.769+0000 7f94b3e00640 1 -- 172.21.15.125:0/782397304 shutdown_connections 2024-01-16T14:59:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.769+0000 7f94b3e00640 1 -- 172.21.15.125:0/782397304 wait complete. 2024-01-16T14:59:49.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.770+0000 7f94b3e00640 1 Processor -- start 2024-01-16T14:59:49.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.770+0000 7f94b3e00640 1 -- start start 2024-01-16T14:59:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94b40cca40 0x7f94b413cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40b29d0 con 0x7f94b4143c40 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b4143110 con 0x7f94b40cca40 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.771+0000 7f94b3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b41433f0 con 0x7f94b40717d0 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33246/0 (socket says 172.21.15.125:33246) 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 -- 172.21.15.125:0/909421687 learned_addr learned my addr 172.21.15.125:0/909421687 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b2a00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94abe00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94b40cca40 0x7f94b413cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 -- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 msgr2=0x7f94b40b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 -- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94b40cca40 msgr2=0x7f94b413cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94b40cca40 0x7f94b413cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 -- 172.21.15.125:0/909421687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40094d0 con 0x7f94b4143c40 2024-01-16T14:59:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b2a00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.772+0000 7f94b3400640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00c780 tx=0x7f94ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.773+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00ce90 con 0x7f94b4143c40 2024-01-16T14:59:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.773+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac00a640 con 0x7f94b4143c40 2024-01-16T14:59:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.773+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00a9b0 con 0x7f94b4143c40 2024-01-16T14:59:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.773+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b40b2310 con 0x7f94b4143c40 2024-01-16T14:59:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.773+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b40ae010 con 0x7f94b4143c40 2024-01-16T14:59:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.774+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b40b2b10 con 0x7f94b4143c40 2024-01-16T14:59:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.775+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f94ac00ab10 con 0x7f94b4143c40 2024-01-16T14:59:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.776+0000 7f94aaa00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 0x7f948c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.776+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94ac089660 con 0x7f94b4143c40 2024-01-16T14:59:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.776+0000 7f94b2a00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 0x7f948c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.776+0000 7f94b2a00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 0x7f948c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94a4036000 tx=0x7f94a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.780+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94b40b2b10 con 0x7f94b4143c40 2024-01-16T14:59:49.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.956+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94b40b2b10 con 0x7f94b4143c40 2024-01-16T14:59:49.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.957+0000 7f94aaa00640 1 -- 172.21.15.125:0/909421687 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16574 (secure 0 0 0) 0x7f94b40b2b10 con 0x7f94b4143c40 2024-01-16T14:59:49.938 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:49.938 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:48.571125+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:46.393145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:47.589538+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:49.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 msgr2=0x7f948c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 0x7f948c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94a4036000 tx=0x7f94a403c040 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 msgr2=0x7f94b40adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00c780 tx=0x7f94ac00cc40 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 shutdown_connections 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948c06b500 0x7f948c06d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94b4143c40 0x7f94b40adad0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94b40cca40 0x7f94b413cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 --2- 172.21.15.125:0/909421687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94b40717d0 0x7f94b40b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 >> 172.21.15.125:0/909421687 conn(0x7f94b406a800 msgr2=0x7f94b4140190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 shutdown_connections 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:49.960+0000 7f94b3e00640 1 -- 172.21.15.125:0/909421687 wait complete. 2024-01-16T14:59:49.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:59:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.171+0000 7fb94c200640 1 Processor -- start 2024-01-16T14:59:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.171+0000 7fb94c200640 1 -- start start 2024-01-16T14:59:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.172+0000 7fb94c200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb9440d0b60 0x7fb9440ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.172+0000 7fb94c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.172+0000 7fb94c200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944071de0 con 0x7fb94406a3e0 2024-01-16T14:59:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.172+0000 7fb94c200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9440af270 con 0x7fb9440d0b60 2024-01-16T14:59:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.172+0000 7fb94c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb944072030 con 0x7fb9440aede0 2024-01-16T14:59:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.173+0000 7fb94a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.173+0000 7fb94a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:53430/0 (socket says 172.21.15.125:53430) 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.173+0000 7fb94a200640 1 -- 172.21.15.125:0/89495058 learned_addr learned my addr 172.21.15.125:0/89495058 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.173+0000 7fb948e00640 1 --2- 172.21.15.125:0/89495058 >> v2:172.21.15.137:3300/0 conn(0x7fb9440d0b60 0x7fb9440ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.174+0000 7fb94a200640 1 -- 172.21.15.125:0/89495058 >> v2:172.21.15.137:3300/0 conn(0x7fb9440d0b60 msgr2=0x7fb9440ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.174+0000 7fb94a200640 1 --2- 172.21.15.125:0/89495058 >> v2:172.21.15.137:3300/0 conn(0x7fb9440d0b60 0x7fb9440ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.174+0000 7fb94a200640 1 -- 172.21.15.125:0/89495058 >> v1:172.21.15.137:6789/0 conn(0x7fb94406a3e0 legacy=0x7fb94406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.174+0000 7fb94a200640 1 -- 172.21.15.125:0/89495058 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944072310 con 0x7fb9440aede0 2024-01-16T14:59:50.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.175+0000 7fb94a200640 1 --2- 172.21.15.125:0/89495058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009e50 tx=0x7fb93c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ebe9dee4ded1068 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.176+0000 7fb943e00640 1 -- 172.21.15.125:0/89495058 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c02cb00 con 0x7fb9440aede0 2024-01-16T14:59:50.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.176+0000 7fb943e00640 1 -- 172.21.15.125:0/89495058 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c02cc60 con 0x7fb9440aede0 2024-01-16T14:59:50.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.176+0000 7fb943e00640 1 -- 172.21.15.125:0/89495058 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0343f0 con 0x7fb9440aede0 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.176+0000 7fb94c200640 1 -- 172.21.15.125:0/89495058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 msgr2=0x7fb94406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.176+0000 7fb94c200640 1 --2- 172.21.15.125:0/89495058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009e50 tx=0x7fb93c02bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 -- 172.21.15.125:0/89495058 shutdown_connections 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 --2- 172.21.15.125:0/89495058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb9440aede0 0x7fb94406f080 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 --2- 172.21.15.125:0/89495058 >> v2:172.21.15.137:3300/0 conn(0x7fb9440d0b60 0x7fb9440ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 -- 172.21.15.125:0/89495058 >> 172.21.15.125:0/89495058 conn(0x7fb9440cfef0 msgr2=0x7fb9440d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:50.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 -- 172.21.15.125:0/89495058 shutdown_connections 2024-01-16T14:59:50.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.177+0000 7fb94c200640 1 -- 172.21.15.125:0/89495058 wait complete. 2024-01-16T14:59:50.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.178+0000 7fb94c200640 1 Processor -- start 2024-01-16T14:59:50.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.178+0000 7fb94c200640 1 -- start start 2024-01-16T14:59:50.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb94406a3e0 0x7fb944154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9440d0b60 0x7fb9440ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440cf880 con 0x7fb9440aede0 2024-01-16T14:59:50.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9441f2b70 con 0x7fb9440d0b60 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.179+0000 7fb94c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9441595c0 con 0x7fb94406a3e0 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33254/0 (socket says 172.21.15.125:33254) 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 -- 172.21.15.125:0/1339476918 learned_addr learned my addr 172.21.15.125:0/1339476918 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 -- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb94406a3e0 msgr2=0x7fb944154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb94406a3e0 0x7fb944154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 -- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9440d0b60 msgr2=0x7fb9440ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9440d0b60 0x7fb9440ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 -- 172.21.15.125:0/1339476918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c009b30 con 0x7fb9440aede0 2024-01-16T14:59:50.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb948e00640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e840 tx=0x7fb92c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.180+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c0094f0 con 0x7fb9440aede0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.181+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c00c940 con 0x7fb9440aede0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.181+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c00ccd0 con 0x7fb9440aede0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.181+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944159860 con 0x7fb9440aede0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.181+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944159e20 con 0x7fb9440aede0 2024-01-16T14:59:50.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.182+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb944073c00 con 0x7fb9440aede0 2024-01-16T14:59:50.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.182+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb92c013470 con 0x7fb9440aede0 2024-01-16T14:59:50.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.183+0000 7fb941600640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 0x7fb91c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.183+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb92c08a130 con 0x7fb9440aede0 2024-01-16T14:59:50.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.184+0000 7fb949800640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 0x7fb91c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.184+0000 7fb949800640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 0x7fb91c06d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb934000f60 tx=0x7fb9340079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.187+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb92c054ba0 con 0x7fb9440aede0 2024-01-16T14:59:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:50.408+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb94415a010 con 0x7fb9440aede0 2024-01-16T14:59:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.166+0000 7fb941600640 1 -- 172.21.15.125:0/1339476918 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb92c0581c0 con 0x7fb9440aede0 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 msgr2=0x7fb91c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 0x7fb91c06d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb934000f60 tx=0x7fb9340079b0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 msgr2=0x7fb944159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e840 tx=0x7fb92c00ed00 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 shutdown_connections 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9440d0b60 0x7fb9440ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb91c06b500 0x7fb91c06d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9440aede0 0x7fb944159080 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 --2- 172.21.15.125:0/1339476918 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb94406a3e0 0x7fb944154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 >> 172.21.15.125:0/1339476918 conn(0x7fb9440cfef0 msgr2=0x7fb94406d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 shutdown_connections 2024-01-16T14:59:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.169+0000 7fb94c200640 1 -- 172.21.15.125:0/1339476918 wait complete. 2024-01-16T14:59:51.169 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:59:51.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:59:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.403+0000 7fc905200640 1 Processor -- start 2024-01-16T14:59:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.403+0000 7fc905200640 1 -- start start 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.404+0000 7fc905200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc900150760 0x7fc900072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc905200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc9000b32e0 0x7fc90015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc905200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90015dc80 con 0x7fc90015d300 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc905200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000b3770 con 0x7fc9000b32e0 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc905200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900071190 con 0x7fc900150760 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8f7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc90015d300 0x7fc9000b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52844/0 (socket says 172.21.15.125:52844) 2024-01-16T14:59:51.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8f7e00640 1 -- 172.21.15.125:0/489863059 learned_addr learned my addr 172.21.15.125:0/489863059 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8fe800640 1 --2- 172.21.15.125:0/489863059 >> v2:172.21.15.137:3300/0 conn(0x7fc9000b32e0 0x7fc90015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312670315 0 0) 0x7fc90015dc80 con 0x7fc90015d300 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc900150760 0x7fc900072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.405+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e0003520 con 0x7fc90015d300 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3407094146 0 0) 0x7fc8e0003520 con 0x7fc90015d300 2024-01-16T14:59:51.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc900150760 msgr2=0x7fc900072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc900150760 0x7fc900072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 >> v2:172.21.15.137:3300/0 conn(0x7fc9000b32e0 msgr2=0x7fc90015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 --2- 172.21.15.125:0/489863059 >> v2:172.21.15.137:3300/0 conn(0x7fc9000b32e0 0x7fc90015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900071470 con 0x7fc90015d300 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc8e8001670 con 0x7fc90015d300 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 >> v1:172.21.15.125:6789/0 conn(0x7fc90015d300 legacy=0x7fc9000b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e0003970 0x7fc8e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e00047c0 0x7fc8e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0005180 0x7fc8e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000b3770 con 0x7fc8e0003970 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900071190 con 0x7fc8e00047c0 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e000bf70 con 0x7fc8e0005180 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc900150760 0x7fc900072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.406+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e0003970 0x7fc8e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8f7e00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e00047c0 0x7fc8e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fe800640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0005180 0x7fc8e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fde00640 1 -- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0005180 msgr2=0x7fc8e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0005180 0x7fc8e000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fde00640 1 -- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e00047c0 msgr2=0x7fc8e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e00047c0 0x7fc8e0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.407+0000 7fc8fde00640 1 -- 172.21.15.125:0/489863059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0003520 con 0x7fc8e0003970 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc8fde00640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e0003970 0x7fc8e0003d90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0006fd0 tx=0x7fc8e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f0cb4e27a778da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0033600 con 0x7fc8e0003970 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f002f050 con 0x7fc8e0003970 2024-01-16T14:59:51.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc8fd400640 1 -- 172.21.15.125:0/489863059 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0033980 con 0x7fc8e0003970 2024-01-16T14:59:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc905200640 1 -- 172.21.15.125:0/489863059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e0003970 msgr2=0x7fc8e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.408+0000 7fc905200640 1 --2- 172.21.15.125:0/489863059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e0003970 0x7fc8e0003d90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0006fd0 tx=0x7fc8e00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.409+0000 7fc8fe800640 1 -- 172.21.15.125:0/489863059 reap_dead start 2024-01-16T14:59:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.409+0000 7fc905200640 1 -- 172.21.15.125:0/489863059 shutdown_connections 2024-01-16T14:59:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.409+0000 7fc905200640 1 -- 172.21.15.125:0/489863059 >> 172.21.15.125:0/489863059 conn(0x7fc90006a800 msgr2=0x7fc90006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:51.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.409+0000 7fc905200640 1 -- 172.21.15.125:0/489863059 shutdown_connections 2024-01-16T14:59:51.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.409+0000 7fc905200640 1 -- 172.21.15.125:0/489863059 wait complete. 2024-01-16T14:59:51.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.410+0000 7fc905200640 1 Processor -- start 2024-01-16T14:59:51.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.411+0000 7fc905200640 1 -- start start 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.411+0000 7fc905200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0003970 0x7fc9001f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.411+0000 7fc905200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9001584c0 0x7fc90014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc905200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc905200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900158cd0 con 0x7fc90014f390 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc905200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001f3d10 con 0x7fc9001584c0 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc905200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90014fd50 con 0x7fc8e0003970 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33270/0 (socket says 172.21.15.125:33270) 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 -- 172.21.15.125:0/2037242719 learned_addr learned my addr 172.21.15.125:0/2037242719 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fde00640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0003970 0x7fc9001f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 -- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0003970 msgr2=0x7fc9001f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8f7e00640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9001584c0 0x7fc90014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0003970 0x7fc9001f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 -- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9001584c0 msgr2=0x7fc90014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9001584c0 0x7fc90014e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.412+0000 7fc8fe800640 1 -- 172.21.15.125:0/2037242719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900150fb0 con 0x7fc90014f390 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc8fe800640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc90006e090 tx=0x7fc8f800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8016620 con 0x7fc90014f390 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f80167a0 con 0x7fc90014f390 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9001f30e0 con 0x7fc90014f390 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8016b50 con 0x7fc90014f390 2024-01-16T14:59:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.413+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9001f3650 con 0x7fc90014f390 2024-01-16T14:59:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.414+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9000675d0 con 0x7fc90014f390 2024-01-16T14:59:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.415+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc8f8016cb0 con 0x7fc90014f390 2024-01-16T14:59:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.416+0000 7fc8f6000640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 0x7fc8d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.416+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8f808af40 con 0x7fc90014f390 2024-01-16T14:59:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.416+0000 7fc8fde00640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 0x7fc8d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:51.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.417+0000 7fc8fde00640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 0x7fc8d806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0036000 tx=0x7fc8f0031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.421+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8f8055950 con 0x7fc90014f390 2024-01-16T14:59:51.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:51.648+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc90015cab0 con 0x7fc90014f390 2024-01-16T14:59:52.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.177+0000 7fc8f6000640 1 -- 172.21.15.125:0/2037242719 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc8f8058f70 con 0x7fc90014f390 2024-01-16T14:59:52.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 msgr2=0x7fc8d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 0x7fc8d806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0036000 tx=0x7fc8f0031000 comp rx=0 tx=0).stop 2024-01-16T14:59:52.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 msgr2=0x7fc90014f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc90006e090 tx=0x7fc8f800f520 comp rx=0 tx=0).stop 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 shutdown_connections 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8d806b500 0x7fc8d806d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc90014f390 0x7fc90014f810 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9001584c0 0x7fc90014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.181+0000 7fc905200640 1 --2- 172.21.15.125:0/2037242719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e0003970 0x7fc9001f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.182+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 >> 172.21.15.125:0/2037242719 conn(0x7fc90006a800 msgr2=0x7fc900159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.182+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 shutdown_connections 2024-01-16T14:59:52.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.182+0000 7fc905200640 1 -- 172.21.15.125:0/2037242719 wait complete. 2024-01-16T14:59:52.184 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:59:52.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:52.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.410+0000 7fb4c5200640 1 Processor -- start 2024-01-16T14:59:52.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.410+0000 7fb4c5200640 1 -- start start 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.411+0000 7fb4c5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4c0166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.411+0000 7fb4c5200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c015c5f0 con 0x7fb4c013df80 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.411+0000 7fb4c5200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c006c470 con 0x7fb4c013e360 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.411+0000 7fb4c5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c006c750 con 0x7fb4c00cf9a0 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb4c013df80 0x7fb4c015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52846/0 (socket says 172.21.15.125:52846) 2024-01-16T14:59:52.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bde00640 1 -- 172.21.15.125:0/2183563460 learned_addr learned my addr 172.21.15.125:0/2183563460 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4be800640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4c0166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603468807 0 0) 0x7fb4c015c5f0 con 0x7fb4c013df80 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a0003520 con 0x7fb4c013df80 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620303631 0 0) 0x7fb4a0003520 con 0x7fb4c013df80 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 msgr2=0x7fb4c0166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4c0166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 >> v1:172.21.15.137:6789/0 conn(0x7fb4c013e360 legacy=0x7fb4c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c006ca30 con 0x7fb4c013df80 2024-01-16T14:59:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb4b0001670 con 0x7fb4c013df80 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.412+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 >> v1:172.21.15.125:6789/0 conn(0x7fb4c013df80 legacy=0x7fb4c015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4a00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4a0003fd0 0x7fb4a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c006c750 con 0x7fb4a0004d00 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000bb40 con 0x7fb4a0003fd0 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000bd00 con 0x7fb4c00cf9a0 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4bde00640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4a00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4b5200640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4a0003fd0 0x7fb4a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 -- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 msgr2=0x7fb4a00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c00cf9a0 0x7fb4a00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 -- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4a0003fd0 msgr2=0x7fb4a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4a0003fd0 0x7fb4a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.413+0000 7fb4be800640 1 -- 172.21.15.125:0/2183563460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0003520 con 0x7fb4a0004d00 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.414+0000 7fb4be800640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4a0005180 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8007330 tx=0x7fb4a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180caec471196c9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b802cca0 con 0x7fb4a0004d00 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b802ce00 con 0x7fb4a0004d00 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4bd400640 1 -- 172.21.15.125:0/2183563460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8034660 con 0x7fb4a0004d00 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 -- 172.21.15.125:0/2183563460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 msgr2=0x7fb4a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2183563460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4a0005180 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8007330 tx=0x7fb4a00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4be800640 1 -- 172.21.15.125:0/2183563460 reap_dead start 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 -- 172.21.15.125:0/2183563460 shutdown_connections 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 -- 172.21.15.125:0/2183563460 >> 172.21.15.125:0/2183563460 conn(0x7fb4c006a800 msgr2=0x7fb4c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 -- 172.21.15.125:0/2183563460 shutdown_connections 2024-01-16T14:59:52.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.415+0000 7fb4c5200640 1 -- 172.21.15.125:0/2183563460 wait complete. 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.417+0000 7fb4c5200640 1 Processor -- start 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.417+0000 7fb4c5200640 1 -- start start 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4c0159140 0x7fb4c00a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 0x7fb4c00a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c015abc0 con 0x7fb4a0004d00 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0159b20 con 0x7fb4c0159140 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4c5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0159e00 con 0x7fb4c01595c0 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33300/0 (socket says 172.21.15.125:33300) 2024-01-16T14:59:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.418+0000 7fb4bde00640 1 -- 172.21.15.125:0/2990417623 learned_addr learned my addr 172.21.15.125:0/2990417623 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4be800640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 0x7fb4c00a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4b5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4c0159140 0x7fb4c00a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 -- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 msgr2=0x7fb4c00a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 0x7fb4c00a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 -- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4c0159140 msgr2=0x7fb4c00a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4c0159140 0x7fb4c00a8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 -- 172.21.15.125:0/2990417623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c015a0e0 con 0x7fb4a0004d00 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4be800640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 0x7fb4c00a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4bde00640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000e8c0 tx=0x7fb4b000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000c870 con 0x7fb4a0004d00 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.419+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b000c9d0 con 0x7fb4a0004d00 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.420+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000cd80 con 0x7fb4a0004d00 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.420+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c015a3c0 con 0x7fb4a0004d00 2024-01-16T14:59:52.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.420+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c01ee820 con 0x7fb4a0004d00 2024-01-16T14:59:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.421+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c015a8e0 con 0x7fb4a0004d00 2024-01-16T14:59:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.422+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb4b001d460 con 0x7fb4a0004d00 2024-01-16T14:59:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.423+0000 7fb4b6a00640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 0x7fb49806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.423+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4b0089ff0 con 0x7fb4a0004d00 2024-01-16T14:59:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.423+0000 7fb4b5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 0x7fb49806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.424+0000 7fb4b5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 0x7fb49806d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.427+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4b00bce90 con 0x7fb4a0004d00 2024-01-16T14:59:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.654+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c00675d0 con 0x7fb4a0004d00 2024-01-16T14:59:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.655+0000 7fb4b6a00640 1 -- 172.21.15.125:0/2990417623 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fb4c015a8e0 con 0x7fb4a0004d00 2024-01-16T14:59:52.635 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:52.635 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/3050283375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3050283375},{"type":"v1","addr":"172.21.15.137:6833","nonce":3050283375}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4258955541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4258955541},{"type":"v1","addr":"172.21.15.137:6835","nonce":4258955541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5739,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1745104106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1745104106},{"type":"v1","addr":"172.21.15.125:6836","nonce":1745104106}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:48.571459+0000","modified":"2024-01-16T14:59:51.649467+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5742":{"gid":5742,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/351760567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":351760567},{"type":"v1","addr":"172.21.15.125:6837","nonce":351760567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T14:59:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 28 2024-01-16T14:59:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.657+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 msgr2=0x7fb49806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.657+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 0x7fb49806d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.657+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 msgr2=0x7fb4c01ee2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.657+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000e8c0 tx=0x7fb4b000ed80 comp rx=0 tx=0).stop 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 shutdown_connections 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb4c01595c0 0x7fb4c00a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4c0159140 0x7fb4c00a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb49806b500 0x7fb49806d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 --2- 172.21.15.125:0/2990417623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a0004d00 0x7fb4c01ee2e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 >> 172.21.15.125:0/2990417623 conn(0x7fb4c006a800 msgr2=0x7fb4c015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 shutdown_connections 2024-01-16T14:59:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.658+0000 7fb4c5200640 1 -- 172.21.15.125:0/2990417623 wait complete. 2024-01-16T14:59:52.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:52.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.886+0000 7f2f70e00640 1 Processor -- start 2024-01-16T14:59:52.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.886+0000 7f2f70e00640 1 -- start start 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 0x7f2f6c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cf7d0 0x7f2f6c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 0x7f2f6c0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c14b6a0 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0cfbf0 con 0x7f2f6c070ea0 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f70e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c071690 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cf7d0 0x7f2f6c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cf7d0 0x7f2f6c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33302/0 (socket says 172.21.15.125:33302) 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 -- 172.21.15.125:0/2531108075 learned_addr learned my addr 172.21.15.125:0/2531108075 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f69e00640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 0x7f2f6c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 -- 172.21.15.125:0/2531108075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 msgr2=0x7f2f6c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f6a800640 1 --2- 172.21.15.125:0/2531108075 >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 0x7f2f6c0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 0x7f2f6c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.888+0000 7f2f61200640 1 -- 172.21.15.125:0/2531108075 >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 msgr2=0x7f2f6c0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.889+0000 7f2f61200640 1 --2- 172.21.15.125:0/2531108075 >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 0x7f2f6c0b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.889+0000 7f2f61200640 1 -- 172.21.15.125:0/2531108075 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c071970 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.889+0000 7f2f69e00640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 0x7f2f6c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:52.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.889+0000 7f2f6a800640 1 --2- 172.21.15.125:0/2531108075 >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 0x7f2f6c0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:52.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.890+0000 7f2f61200640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c070960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06e2a0 tx=0x7f2f5402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f304d955bc0f0e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.890+0000 7f2f69400640 1 -- 172.21.15.125:0/2531108075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54031880 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.890+0000 7f2f69400640 1 -- 172.21.15.125:0/2531108075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f540319e0 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.890+0000 7f2f69400640 1 -- 172.21.15.125:0/2531108075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54031d70 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 -- 172.21.15.125:0/2531108075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c0cf7d0 msgr2=0x7f2f6c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c070960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06e2a0 tx=0x7f2f5402b920 comp rx=0 tx=0).stop 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 -- 172.21.15.125:0/2531108075 shutdown_connections 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 --2- 172.21.15.125:0/2531108075 >> v2:172.21.15.137:3300/0 conn(0x7f2f6c070ea0 0x7f2f6c0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c070960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 --2- 172.21.15.125:0/2531108075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c14b2c0 0x7f2f6c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 -- 172.21.15.125:0/2531108075 >> 172.21.15.125:0/2531108075 conn(0x7f2f6c06a800 msgr2=0x7f2f6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 -- 172.21.15.125:0/2531108075 shutdown_connections 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.891+0000 7f2f70e00640 1 -- 172.21.15.125:0/2531108075 wait complete. 2024-01-16T14:59:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.893+0000 7f2f70e00640 1 Processor -- start 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.893+0000 7f2f70e00640 1 -- start start 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c070ea0 0x7f2f6c161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c1f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1f4bc0 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1f59b0 con 0x7f2f6c0cf7d0 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f70e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1f4da0 con 0x7f2f6c070ea0 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f6a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f6a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33306/0 (socket says 172.21.15.125:33306) 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f6a800640 1 -- 172.21.15.125:0/1382655518 learned_addr learned my addr 172.21.15.125:0/1382655518 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f69e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c070ea0 0x7f2f6c161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f61200640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c1f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f6a800640 1 -- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c070ea0 msgr2=0x7f2f6c161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.894+0000 7f2f6a800640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c070ea0 0x7f2f6c161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f6a800640 1 -- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f6c0cf7d0 msgr2=0x7f2f6c1f7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f6a800640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c1f7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f6a800640 1 -- 172.21.15.125:0/1382655518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540094d0 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f6a800640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400cd10 tx=0x7f2f6400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400f680 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400f800 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400fbb0 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c1f5080 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.895+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c1f5640 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.897+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c0675d0 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.897+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2f6400b040 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.898+0000 7f2f62a00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 0x7f2f4406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.899+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f6408aa60 con 0x7f2f6c14b2c0 2024-01-16T14:59:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.899+0000 7f2f69e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 0x7f2f4406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.899+0000 7f2f69e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 0x7f2f4406d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ceab0 tx=0x7f2f5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:52.903+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f64055460 con 0x7f2f6c14b2c0 2024-01-16T14:59:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.093+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f6c0cb620 con 0x7f2f6c14b2c0 2024-01-16T14:59:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.094+0000 7f2f62a00640 1 -- 172.21.15.125:0/1382655518 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16561 (secure 0 0 0) 0x7f2f6401d090 con 0x7f2f6c14b2c0 2024-01-16T14:59:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:53.075 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T14:59:50.140708+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:46.393145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:47.589538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 msgr2=0x7f2f4406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 0x7f2f4406d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ceab0 tx=0x7f2f5c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 msgr2=0x7f2f6c0a8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400cd10 tx=0x7f2f6400a650 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 shutdown_connections 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2f4406b530 0x7f2f4406d9f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c14b2c0 0x7f2f6c0a8f70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c1f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 --2- 172.21.15.125:0/1382655518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f6c070ea0 0x7f2f6c161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 >> 172.21.15.125:0/1382655518 conn(0x7f2f6c06a800 msgr2=0x7f2f6c0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 shutdown_connections 2024-01-16T14:59:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.097+0000 7f2f70e00640 1 -- 172.21.15.125:0/1382655518 wait complete. 2024-01-16T14:59:53.098 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:59:53.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:53.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.319+0000 7f6fb2400640 1 Processor -- start 2024-01-16T14:59:53.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.319+0000 7f6fb2400640 1 -- start start 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fb2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6fac15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fb2400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fac0cf9a0 0x7f6fac0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fb2400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac06c470 con 0x7f6fac13e360 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fb2400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac06c750 con 0x7f6fac0cf9a0 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fb2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac06ca30 con 0x7f6fac13df80 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fa3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6fac13e360 0x7f6fac0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52852/0 (socket says 172.21.15.125:52852) 2024-01-16T14:59:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1948342222 learned_addr learned my addr 172.21.15.125:0/1948342222 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:53.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fab400640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6fac15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420918460 0 0) 0x7f6fac06c470 con 0x7f6fac13e360 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f8c003520 con 0x7f6fac13e360 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.321+0000 7f6fabe00640 1 --2- 172.21.15.125:0/1948342222 >> v2:172.21.15.137:3300/0 conn(0x7f6fac0cf9a0 0x7f6fac0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536460063 0 0) 0x7f6f8c003520 con 0x7f6fac13e360 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 msgr2=0x7f6fac15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6fac15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 >> v2:172.21.15.137:3300/0 conn(0x7f6fac0cf9a0 msgr2=0x7f6fac0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 --2- 172.21.15.125:0/1948342222 >> v2:172.21.15.137:3300/0 conn(0x7f6fac0cf9a0 0x7f6fac0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac06cd10 con 0x7f6fac13e360 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6f94001670 con 0x7f6fac13e360 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 >> v1:172.21.15.125:6789/0 conn(0x7f6fac13e360 legacy=0x7f6fac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6f8c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f8c003fa0 0x7f6f8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f8c004d00 0x7f6f8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac06c750 con 0x7f6f8c003fa0 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac06ca30 con 0x7f6f8c004d00 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c00ba80 con 0x7f6fac13df80 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.322+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f8c003fa0 0x7f6f8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fab400640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6f8c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fabe00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f8c004d00 0x7f6f8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 msgr2=0x7f6f8c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6f8c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f8c004d00 msgr2=0x7f6f8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f8c004d00 0x7f6f8c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1948342222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c003520 con 0x7f6f8c003fa0 2024-01-16T14:59:53.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.323+0000 7f6fab400640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fac13df80 0x7f6f8c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.324+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f8c003fa0 0x7f6f8c0047c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400c240 tx=0x7f6f8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4dc0234dcf369d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.324+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9402cb00 con 0x7f6f8c003fa0 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.324+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9402cc60 con 0x7f6f8c003fa0 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.324+0000 7f6faaa00640 1 -- 172.21.15.125:0/1948342222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94005af0 con 0x7f6f8c003fa0 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 -- 172.21.15.125:0/1948342222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f8c003fa0 msgr2=0x7f6f8c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1948342222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f8c003fa0 0x7f6f8c0047c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400c240 tx=0x7f6f8c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fabe00640 1 -- 172.21.15.125:0/1948342222 reap_dead start 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 -- 172.21.15.125:0/1948342222 shutdown_connections 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 -- 172.21.15.125:0/1948342222 >> 172.21.15.125:0/1948342222 conn(0x7f6fac06a800 msgr2=0x7f6fac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 -- 172.21.15.125:0/1948342222 shutdown_connections 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.325+0000 7f6fb2400640 1 -- 172.21.15.125:0/1948342222 wait complete. 2024-01-16T14:59:53.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.327+0000 7f6fb2400640 1 Processor -- start 2024-01-16T14:59:53.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.327+0000 7f6fb2400640 1 -- start start 2024-01-16T14:59:53.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f8c003fa0 0x7f6fac1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fac158160 0x7f6fac1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1f7470 con 0x7f6fac131d80 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac13da90 con 0x7f6fac158160 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fb2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac158b20 con 0x7f6f8c003fa0 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33334/0 (socket says 172.21.15.125:33334) 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1813429119 learned_addr learned my addr 172.21.15.125:0/1813429119 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fab400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f8c003fa0 0x7f6fac1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f8c003fa0 msgr2=0x7f6fac1f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f8c003fa0 0x7f6fac1f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fac158160 msgr2=0x7f6fac1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fac158160 0x7f6fac1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.328+0000 7f6fa3e00640 1 -- 172.21.15.125:0/1813429119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac1318f0 con 0x7f6fac131d80 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400c240 tx=0x7f6f94001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94039070 con 0x7f6fac131d80 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f94005d00 con 0x7f6fac131d80 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9403c040 con 0x7f6fac131d80 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac1f6bd0 con 0x7f6fac131d80 2024-01-16T14:59:53.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.329+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac1f7190 con 0x7f6fac131d80 2024-01-16T14:59:53.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.330+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac131a80 con 0x7f6fac131d80 2024-01-16T14:59:53.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.331+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6f94032070 con 0x7f6fac131d80 2024-01-16T14:59:53.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.332+0000 7f6fa3400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 0x7f6f8406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.332+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f940ab6e0 con 0x7f6fac131d80 2024-01-16T14:59:53.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.332+0000 7f6fab400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 0x7f6f8406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.333+0000 7f6fab400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 0x7f6f8406d9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006fd0 tx=0x7f6f9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.336+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f940761e0 con 0x7f6fac131d80 2024-01-16T14:59:53.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.523+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fac15c850 con 0x7f6f8406b530 2024-01-16T14:59:53.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.543+0000 7f6fa3400640 1 -- 172.21.15.125:0/1813429119 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283891 (secure 0 0 0) 0x7f6fac15c850 con 0x7f6f8406b530 2024-01-16T14:59:53.527 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:59:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 msgr2=0x7f6f8406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 0x7f6f8406d9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006fd0 tx=0x7f6f9c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 msgr2=0x7f6fac132200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400c240 tx=0x7f6f94001a00 comp rx=0 tx=0).stop 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 shutdown_connections 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fac158160 0x7f6fac1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8406b530 0x7f6f8406d9f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fac131d80 0x7f6fac132200 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 --2- 172.21.15.125:0/1813429119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f8c003fa0 0x7f6fac1f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 >> 172.21.15.125:0/1813429119 conn(0x7f6fac06a800 msgr2=0x7f6fac15cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.548+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 shutdown_connections 2024-01-16T14:59:53.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.549+0000 7f6fb2400640 1 -- 172.21.15.125:0/1813429119 wait complete. 2024-01-16T14:59:53.547 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-01-16T14:59:51.978757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590984,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":173,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254564,"kb_used_data":5092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720156,"statfs":{"total":773094113280,"available":772833439744,"internally_reserved":0,"allocated":5214208,"data_stored":2797205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.881273"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116216+0000","last_change":"2024-01-16T14:59:48.116216+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116216+0000","last_undegraded":"2024-01-16T14:59:48.116216+0000","last_fullsized":"2024-01-16T14:59:48.116216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.122243+0000","last_change":"2024-01-16T14:59:48.122243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.122243+0000","last_undegraded":"2024-01-16T14:59:48.122243+0000","last_fullsized":"2024-01-16T14:59:48.122243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157265+0000","last_change":"2024-01-16T14:59:49.157265+0000","last_active":"2024-01-16T14:59:49.157265+0000","last_peered":"2024-01-16T14:59:49.157265+0000","last_clean":"2024-01-16T14:59:49.157265+0000","last_became_active":"2024-01-16T14:59:49.156985+0000","last_became_peered":"2024-01-16T14:59:49.156985+0000","last_unstale":"2024-01-16T14:59:49.157265+0000","last_undegraded":"2024-01-16T14:59:49.157265+0000","last_fullsized":"2024-01-16T14:59:49.157265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.118153+0000","last_change":"2024-01-16T14:59:48.118153+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.118153+0000","last_undegraded":"2024-01-16T14:59:48.118153+0000","last_fullsized":"2024-01-16T14:59:48.118153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.121232+0000","last_change":"2024-01-16T14:59:48.121232+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.121232+0000","last_undegraded":"2024-01-16T14:59:48.121232+0000","last_fullsized":"2024-01-16T14:59:48.121232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164334+0000","last_change":"2024-01-16T14:59:49.160920+0000","last_active":"2024-01-16T14:59:49.164334+0000","last_peered":"2024-01-16T14:59:49.164334+0000","last_clean":"2024-01-16T14:59:49.164334+0000","last_became_active":"2024-01-16T14:59:49.158310+0000","last_became_peered":"2024-01-16T14:59:49.158310+0000","last_unstale":"2024-01-16T14:59:49.164334+0000","last_undegraded":"2024-01-16T14:59:49.164334+0000","last_fullsized":"2024-01-16T14:59:49.164334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.670502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116927+0000","last_change":"2024-01-16T14:59:48.116927+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116927+0000","last_undegraded":"2024-01-16T14:59:48.116927+0000","last_fullsized":"2024-01-16T14:59:48.116927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116447+0000","last_change":"2024-01-16T14:59:48.116447+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116447+0000","last_undegraded":"2024-01-16T14:59:48.116447+0000","last_fullsized":"2024-01-16T14:59:48.116447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114791+0000","last_change":"2024-01-16T14:59:48.114791+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114791+0000","last_undegraded":"2024-01-16T14:59:48.114791+0000","last_fullsized":"2024-01-16T14:59:48.114791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116913+0000","last_change":"2024-01-16T14:59:48.116913+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116913+0000","last_undegraded":"2024-01-16T14:59:48.116913+0000","last_fullsized":"2024-01-16T14:59:48.116913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115973+0000","last_change":"2024-01-16T14:59:48.115973+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115973+0000","last_undegraded":"2024-01-16T14:59:48.115973+0000","last_fullsized":"2024-01-16T14:59:48.115973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124245+0000","last_change":"2024-01-16T14:59:48.124245+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124245+0000","last_undegraded":"2024-01-16T14:59:48.124245+0000","last_fullsized":"2024-01-16T14:59:48.124245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114997+0000","last_change":"2024-01-16T14:59:48.114997+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114997+0000","last_undegraded":"2024-01-16T14:59:48.114997+0000","last_fullsized":"2024-01-16T14:59:48.114997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114309+0000","last_change":"2024-01-16T14:59:48.114309+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114309+0000","last_undegraded":"2024-01-16T14:59:48.114309+0000","last_fullsized":"2024-01-16T14:59:48.114309+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163054+0000","last_change":"2024-01-16T14:59:49.158516+0000","last_active":"2024-01-16T14:59:49.163054+0000","last_peered":"2024-01-16T14:59:49.163054+0000","last_clean":"2024-01-16T14:59:49.163054+0000","last_became_active":"2024-01-16T14:59:49.158217+0000","last_became_peered":"2024-01-16T14:59:49.158217+0000","last_unstale":"2024-01-16T14:59:49.163054+0000","last_undegraded":"2024-01-16T14:59:49.163054+0000","last_fullsized":"2024-01-16T14:59:49.163054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124125+0000","last_change":"2024-01-16T14:59:48.124125+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124125+0000","last_undegraded":"2024-01-16T14:59:48.124125+0000","last_fullsized":"2024-01-16T14:59:48.124125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.162728+0000","last_change":"2024-01-16T14:59:49.162728+0000","last_active":"2024-01-16T14:59:49.162728+0000","last_peered":"2024-01-16T14:59:49.162728+0000","last_clean":"2024-01-16T14:59:49.162728+0000","last_became_active":"2024-01-16T14:59:49.162473+0000","last_became_peered":"2024-01-16T14:59:49.162473+0000","last_unstale":"2024-01-16T14:59:49.162728+0000","last_undegraded":"2024-01-16T14:59:49.162728+0000","last_fullsized":"2024-01-16T14:59:49.162728+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:29.139287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159712+0000","last_change":"2024-01-16T14:59:49.146953+0000","last_active":"2024-01-16T14:59:49.159712+0000","last_peered":"2024-01-16T14:59:49.159712+0000","last_clean":"2024-01-16T14:59:49.159712+0000","last_became_active":"2024-01-16T14:59:49.146074+0000","last_became_peered":"2024-01-16T14:59:49.146074+0000","last_unstale":"2024-01-16T14:59:49.159712+0000","last_undegraded":"2024-01-16T14:59:49.159712+0000","last_fullsized":"2024-01-16T14:59:49.159712+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:06.294703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126633+0000","last_change":"2024-01-16T14:59:48.126633+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126633+0000","last_undegraded":"2024-01-16T14:59:48.126633+0000","last_fullsized":"2024-01-16T14:59:48.126633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115055+0000","last_change":"2024-01-16T14:59:48.115055+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115055+0000","last_undegraded":"2024-01-16T14:59:48.115055+0000","last_fullsized":"2024-01-16T14:59:48.115055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163749+0000","last_change":"2024-01-16T14:59:49.158956+0000","last_active":"2024-01-16T14:59:49.163749+0000","last_peered":"2024-01-16T14:59:49.163749+0000","last_clean":"2024-01-16T14:59:49.163749+0000","last_became_active":"2024-01-16T14:59:49.158665+0000","last_became_peered":"2024-01-16T14:59:49.158665+0000","last_unstale":"2024-01-16T14:59:49.163749+0000","last_undegraded":"2024-01-16T14:59:49.163749+0000","last_fullsized":"2024-01-16T14:59:49.163749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112923+0000","last_change":"2024-01-16T14:59:48.112923+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112923+0000","last_undegraded":"2024-01-16T14:59:48.112923+0000","last_fullsized":"2024-01-16T14:59:48.112923+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131707+0000","last_change":"2024-01-16T14:59:48.131707+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131707+0000","last_undegraded":"2024-01-16T14:59:48.131707+0000","last_fullsized":"2024-01-16T14:59:48.131707+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123440+0000","last_change":"2024-01-16T14:59:48.123440+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123440+0000","last_undegraded":"2024-01-16T14:59:48.123440+0000","last_fullsized":"2024-01-16T14:59:48.123440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131918+0000","last_change":"2024-01-16T14:59:48.131918+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131918+0000","last_undegraded":"2024-01-16T14:59:48.131918+0000","last_fullsized":"2024-01-16T14:59:48.131918+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113430+0000","last_change":"2024-01-16T14:59:48.113430+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113430+0000","last_undegraded":"2024-01-16T14:59:48.113430+0000","last_fullsized":"2024-01-16T14:59:48.113430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164250+0000","last_change":"2024-01-16T14:59:49.164250+0000","last_active":"2024-01-16T14:59:49.164250+0000","last_peered":"2024-01-16T14:59:49.164250+0000","last_clean":"2024-01-16T14:59:49.164250+0000","last_became_active":"2024-01-16T14:59:49.163931+0000","last_became_peered":"2024-01-16T14:59:49.163931+0000","last_unstale":"2024-01-16T14:59:49.164250+0000","last_undegraded":"2024-01-16T14:59:49.164250+0000","last_fullsized":"2024-01-16T14:59:49.164250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:48.014735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123754+0000","last_change":"2024-01-16T14:59:48.123754+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123754+0000","last_undegraded":"2024-01-16T14:59:48.123754+0000","last_fullsized":"2024-01-16T14:59:48.123754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126693+0000","last_change":"2024-01-16T14:59:48.126693+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126693+0000","last_undegraded":"2024-01-16T14:59:48.126693+0000","last_fullsized":"2024-01-16T14:59:48.126693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160966+0000","last_change":"2024-01-16T14:59:49.160966+0000","last_active":"2024-01-16T14:59:49.160966+0000","last_peered":"2024-01-16T14:59:49.160966+0000","last_clean":"2024-01-16T14:59:49.160966+0000","last_became_active":"2024-01-16T14:59:49.160612+0000","last_became_peered":"2024-01-16T14:59:49.160612+0000","last_unstale":"2024-01-16T14:59:49.160966+0000","last_undegraded":"2024-01-16T14:59:49.160966+0000","last_fullsized":"2024-01-16T14:59:49.160966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164208+0000","last_change":"2024-01-16T14:59:49.159653+0000","last_active":"2024-01-16T14:59:49.164208+0000","last_peered":"2024-01-16T14:59:49.164208+0000","last_clean":"2024-01-16T14:59:49.164208+0000","last_became_active":"2024-01-16T14:59:49.159375+0000","last_became_peered":"2024-01-16T14:59:49.159375+0000","last_unstale":"2024-01-16T14:59:49.164208+0000","last_undegraded":"2024-01-16T14:59:49.164208+0000","last_fullsized":"2024-01-16T14:59:49.164208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112726+0000","last_change":"2024-01-16T14:59:48.112726+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112726+0000","last_undegraded":"2024-01-16T14:59:48.112726+0000","last_fullsized":"2024-01-16T14:59:48.112726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165746+0000","last_change":"2024-01-16T14:59:49.156116+0000","last_active":"2024-01-16T14:59:49.165746+0000","last_peered":"2024-01-16T14:59:49.165746+0000","last_clean":"2024-01-16T14:59:49.165746+0000","last_became_active":"2024-01-16T14:59:49.155198+0000","last_became_peered":"2024-01-16T14:59:49.155198+0000","last_unstale":"2024-01-16T14:59:49.165746+0000","last_undegraded":"2024-01-16T14:59:49.165746+0000","last_fullsized":"2024-01-16T14:59:49.165746+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:12:21.166819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169542+0000","last_change":"2024-01-16T14:59:48.137330+0000","last_active":"2024-01-16T14:59:49.169542+0000","last_peered":"2024-01-16T14:59:49.169542+0000","last_clean":"2024-01-16T14:59:49.169542+0000","last_became_active":"2024-01-16T14:59:48.136959+0000","last_became_peered":"2024-01-16T14:59:48.136959+0000","last_unstale":"2024-01-16T14:59:49.169542+0000","last_undegraded":"2024-01-16T14:59:49.169542+0000","last_fullsized":"2024-01-16T14:59:49.169542+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:39.353399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.130678+0000","last_change":"2024-01-16T14:59:48.130678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.130678+0000","last_undegraded":"2024-01-16T14:59:48.130678+0000","last_fullsized":"2024-01-16T14:59:48.130678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160262+0000","last_change":"2024-01-16T14:59:48.137753+0000","last_active":"2024-01-16T14:59:49.160262+0000","last_peered":"2024-01-16T14:59:49.160262+0000","last_clean":"2024-01-16T14:59:49.160262+0000","last_became_active":"2024-01-16T14:59:48.137414+0000","last_became_peered":"2024-01-16T14:59:48.137414+0000","last_unstale":"2024-01-16T14:59:49.160262+0000","last_undegraded":"2024-01-16T14:59:49.160262+0000","last_fullsized":"2024-01-16T14:59:49.160262+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123243+0000","last_change":"2024-01-16T14:59:48.123243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123243+0000","last_undegraded":"2024-01-16T14:59:48.123243+0000","last_fullsized":"2024-01-16T14:59:48.123243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159457+0000","last_change":"2024-01-16T14:59:48.134911+0000","last_active":"2024-01-16T14:59:49.159457+0000","last_peered":"2024-01-16T14:59:49.159457+0000","last_clean":"2024-01-16T14:59:49.159457+0000","last_became_active":"2024-01-16T14:59:48.134259+0000","last_became_peered":"2024-01-16T14:59:48.134259+0000","last_unstale":"2024-01-16T14:59:49.159457+0000","last_undegraded":"2024-01-16T14:59:49.159457+0000","last_fullsized":"2024-01-16T14:59:49.159457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:28.124634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167122+0000","last_change":"2024-01-16T14:59:49.165223+0000","last_active":"2024-01-16T14:59:49.167122+0000","last_peered":"2024-01-16T14:59:49.167122+0000","last_clean":"2024-01-16T14:59:49.167122+0000","last_became_active":"2024-01-16T14:59:49.164935+0000","last_became_peered":"2024-01-16T14:59:49.164935+0000","last_unstale":"2024-01-16T14:59:49.167122+0000","last_undegraded":"2024-01-16T14:59:49.167122+0000","last_fullsized":"2024-01-16T14:59:49.167122+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.134046+0000","last_change":"2024-01-16T14:59:48.131434+0000","last_active":"2024-01-16T14:59:48.134046+0000","last_peered":"2024-01-16T14:59:48.134046+0000","last_clean":"2024-01-16T14:59:48.134046+0000","last_became_active":"2024-01-16T14:59:48.131165+0000","last_became_peered":"2024-01-16T14:59:48.131165+0000","last_unstale":"2024-01-16T14:59:48.134046+0000","last_undegraded":"2024-01-16T14:59:48.134046+0000","last_fullsized":"2024-01-16T14:59:48.134046+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159531+0000","last_change":"2024-01-16T14:59:49.159531+0000","last_active":"2024-01-16T14:59:49.159531+0000","last_peered":"2024-01-16T14:59:49.159531+0000","last_clean":"2024-01-16T14:59:49.159531+0000","last_became_active":"2024-01-16T14:59:49.159274+0000","last_became_peered":"2024-01-16T14:59:49.159274+0000","last_unstale":"2024-01-16T14:59:49.159531+0000","last_undegraded":"2024-01-16T14:59:49.159531+0000","last_fullsized":"2024-01-16T14:59:49.159531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168278+0000","last_change":"2024-01-16T14:59:48.128798+0000","last_active":"2024-01-16T14:59:49.168278+0000","last_peered":"2024-01-16T14:59:49.168278+0000","last_clean":"2024-01-16T14:59:49.168278+0000","last_became_active":"2024-01-16T14:59:48.128328+0000","last_became_peered":"2024-01-16T14:59:48.128328+0000","last_unstale":"2024-01-16T14:59:49.168278+0000","last_undegraded":"2024-01-16T14:59:49.168278+0000","last_fullsized":"2024-01-16T14:59:49.168278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:54.681593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113310+0000","last_change":"2024-01-16T14:59:48.113310+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113310+0000","last_undegraded":"2024-01-16T14:59:48.113310+0000","last_fullsized":"2024-01-16T14:59:48.113310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.161777+0000","last_change":"2024-01-16T14:59:48.131571+0000","last_active":"2024-01-16T14:59:49.161777+0000","last_peered":"2024-01-16T14:59:49.161777+0000","last_clean":"2024-01-16T14:59:49.161777+0000","last_became_active":"2024-01-16T14:59:48.131245+0000","last_became_peered":"2024-01-16T14:59:48.131245+0000","last_unstale":"2024-01-16T14:59:49.161777+0000","last_undegraded":"2024-01-16T14:59:49.161777+0000","last_fullsized":"2024-01-16T14:59:49.161777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113674+0000","last_change":"2024-01-16T14:59:48.113674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113674+0000","last_undegraded":"2024-01-16T14:59:48.113674+0000","last_fullsized":"2024-01-16T14:59:48.113674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.133471+0000","last_change":"2024-01-16T14:59:48.130388+0000","last_active":"2024-01-16T14:59:48.133471+0000","last_peered":"2024-01-16T14:59:48.133471+0000","last_clean":"2024-01-16T14:59:48.133471+0000","last_became_active":"2024-01-16T14:59:48.129990+0000","last_became_peered":"2024-01-16T14:59:48.129990+0000","last_unstale":"2024-01-16T14:59:48.133471+0000","last_undegraded":"2024-01-16T14:59:48.133471+0000","last_fullsized":"2024-01-16T14:59:48.133471+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:33.461419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.132738+0000","last_change":"2024-01-16T14:59:48.132738+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.132738+0000","last_undegraded":"2024-01-16T14:59:48.132738+0000","last_fullsized":"2024-01-16T14:59:48.132738+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.128590+0000","last_change":"2024-01-16T14:59:48.122577+0000","last_active":"2024-01-16T14:59:48.128590+0000","last_peered":"2024-01-16T14:59:48.128590+0000","last_clean":"2024-01-16T14:59:48.128590+0000","last_became_active":"2024-01-16T14:59:48.122280+0000","last_became_peered":"2024-01-16T14:59:48.122280+0000","last_unstale":"2024-01-16T14:59:48.128590+0000","last_undegraded":"2024-01-16T14:59:48.128590+0000","last_fullsized":"2024-01-16T14:59:48.128590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:53.539387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159679+0000","last_change":"2024-01-16T14:59:49.146563+0000","last_active":"2024-01-16T14:59:49.159679+0000","last_peered":"2024-01-16T14:59:49.159679+0000","last_clean":"2024-01-16T14:59:49.159679+0000","last_became_active":"2024-01-16T14:59:49.145334+0000","last_became_peered":"2024-01-16T14:59:49.145334+0000","last_unstale":"2024-01-16T14:59:49.159679+0000","last_undegraded":"2024-01-16T14:59:49.159679+0000","last_fullsized":"2024-01-16T14:59:49.159679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:13.595065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.140096+0000","last_change":"2024-01-16T14:59:48.127440+0000","last_active":"2024-01-16T14:59:49.140096+0000","last_peered":"2024-01-16T14:59:49.140096+0000","last_clean":"2024-01-16T14:59:49.140096+0000","last_became_active":"2024-01-16T14:59:48.126114+0000","last_became_peered":"2024-01-16T14:59:48.126114+0000","last_unstale":"2024-01-16T14:59:49.140096+0000","last_undegraded":"2024-01-16T14:59:49.140096+0000","last_fullsized":"2024-01-16T14:59:49.140096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114330+0000","last_change":"2024-01-16T14:59:48.114330+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114330+0000","last_undegraded":"2024-01-16T14:59:48.114330+0000","last_fullsized":"2024-01-16T14:59:48.114330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136234+0000","last_change":"2024-01-16T14:59:48.128532+0000","last_active":"2024-01-16T14:59:48.136234+0000","last_peered":"2024-01-16T14:59:48.136234+0000","last_clean":"2024-01-16T14:59:48.136234+0000","last_became_active":"2024-01-16T14:59:48.127045+0000","last_became_peered":"2024-01-16T14:59:48.127045+0000","last_unstale":"2024-01-16T14:59:48.136234+0000","last_undegraded":"2024-01-16T14:59:48.136234+0000","last_fullsized":"2024-01-16T14:59:48.136234+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:12.546318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165687+0000","last_change":"2024-01-16T14:59:49.157462+0000","last_active":"2024-01-16T14:59:49.165687+0000","last_peered":"2024-01-16T14:59:49.165687+0000","last_clean":"2024-01-16T14:59:49.165687+0000","last_became_active":"2024-01-16T14:59:49.157150+0000","last_became_peered":"2024-01-16T14:59:49.157150+0000","last_unstale":"2024-01-16T14:59:49.165687+0000","last_undegraded":"2024-01-16T14:59:49.165687+0000","last_fullsized":"2024-01-16T14:59:49.165687+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136759+0000","last_change":"2024-01-16T14:59:48.135944+0000","last_active":"2024-01-16T14:59:48.136759+0000","last_peered":"2024-01-16T14:59:48.136759+0000","last_clean":"2024-01-16T14:59:48.136759+0000","last_became_active":"2024-01-16T14:59:48.135643+0000","last_became_peered":"2024-01-16T14:59:48.135643+0000","last_unstale":"2024-01-16T14:59:48.136759+0000","last_undegraded":"2024-01-16T14:59:48.136759+0000","last_fullsized":"2024-01-16T14:59:48.136759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:39.871898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115155+0000","last_change":"2024-01-16T14:59:48.115155+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115155+0000","last_undegraded":"2024-01-16T14:59:48.115155+0000","last_fullsized":"2024-01-16T14:59:48.115155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132942+0000","last_change":"2024-01-16T14:59:48.129597+0000","last_active":"2024-01-16T14:59:48.132942+0000","last_peered":"2024-01-16T14:59:48.132942+0000","last_clean":"2024-01-16T14:59:48.132942+0000","last_became_active":"2024-01-16T14:59:48.128905+0000","last_became_peered":"2024-01-16T14:59:48.128905+0000","last_unstale":"2024-01-16T14:59:48.132942+0000","last_undegraded":"2024-01-16T14:59:48.132942+0000","last_fullsized":"2024-01-16T14:59:48.132942+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:15.588352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.125716+0000","last_change":"2024-01-16T14:59:48.125716+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.125716+0000","last_undegraded":"2024-01-16T14:59:48.125716+0000","last_fullsized":"2024-01-16T14:59:48.125716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169439+0000","last_change":"2024-01-16T14:59:48.128640+0000","last_active":"2024-01-16T14:59:49.169439+0000","last_peered":"2024-01-16T14:59:49.169439+0000","last_clean":"2024-01-16T14:59:49.169439+0000","last_became_active":"2024-01-16T14:59:48.128032+0000","last_became_peered":"2024-01-16T14:59:48.128032+0000","last_unstale":"2024-01-16T14:59:49.169439+0000","last_undegraded":"2024-01-16T14:59:49.169439+0000","last_fullsized":"2024-01-16T14:59:49.169439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:41.977214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112452+0000","last_change":"2024-01-16T14:59:48.112452+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112452+0000","last_undegraded":"2024-01-16T14:59:48.112452+0000","last_fullsized":"2024-01-16T14:59:48.112452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.130910+0000","last_change":"2024-01-16T14:59:48.126828+0000","last_active":"2024-01-16T14:59:48.130910+0000","last_peered":"2024-01-16T14:59:48.130910+0000","last_clean":"2024-01-16T14:59:48.130910+0000","last_became_active":"2024-01-16T14:59:48.126513+0000","last_became_peered":"2024-01-16T14:59:48.126513+0000","last_unstale":"2024-01-16T14:59:48.130910+0000","last_undegraded":"2024-01-16T14:59:48.130910+0000","last_fullsized":"2024-01-16T14:59:48.130910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153727+0000","last_change":"2024-01-16T14:59:49.150175+0000","last_active":"2024-01-16T14:59:49.153727+0000","last_peered":"2024-01-16T14:59:49.153727+0000","last_clean":"2024-01-16T14:59:49.153727+0000","last_became_active":"2024-01-16T14:59:49.149869+0000","last_became_peered":"2024-01-16T14:59:49.149869+0000","last_unstale":"2024-01-16T14:59:49.153727+0000","last_undegraded":"2024-01-16T14:59:49.153727+0000","last_fullsized":"2024-01-16T14:59:49.153727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138602+0000","last_change":"2024-01-16T14:59:48.129480+0000","last_active":"2024-01-16T14:59:49.138602+0000","last_peered":"2024-01-16T14:59:49.138602+0000","last_clean":"2024-01-16T14:59:49.138602+0000","last_became_active":"2024-01-16T14:59:48.129231+0000","last_became_peered":"2024-01-16T14:59:48.129231+0000","last_unstale":"2024-01-16T14:59:49.138602+0000","last_undegraded":"2024-01-16T14:59:49.138602+0000","last_fullsized":"2024-01-16T14:59:49.138602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114361+0000","last_change":"2024-01-16T14:59:48.114361+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114361+0000","last_undegraded":"2024-01-16T14:59:48.114361+0000","last_fullsized":"2024-01-16T14:59:48.114361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.137414+0000","last_change":"2024-01-16T14:59:48.133029+0000","last_active":"2024-01-16T14:59:48.137414+0000","last_peered":"2024-01-16T14:59:48.137414+0000","last_clean":"2024-01-16T14:59:48.137414+0000","last_became_active":"2024-01-16T14:59:48.132732+0000","last_became_peered":"2024-01-16T14:59:48.132732+0000","last_unstale":"2024-01-16T14:59:48.137414+0000","last_undegraded":"2024-01-16T14:59:48.137414+0000","last_fullsized":"2024-01-16T14:59:48.137414+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:36.691574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112029+0000","last_change":"2024-01-16T14:59:48.112029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112029+0000","last_undegraded":"2024-01-16T14:59:48.112029+0000","last_fullsized":"2024-01-16T14:59:48.112029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132892+0000","last_change":"2024-01-16T14:59:48.129568+0000","last_active":"2024-01-16T14:59:48.132892+0000","last_peered":"2024-01-16T14:59:48.132892+0000","last_clean":"2024-01-16T14:59:48.132892+0000","last_became_active":"2024-01-16T14:59:48.129222+0000","last_became_peered":"2024-01-16T14:59:48.129222+0000","last_unstale":"2024-01-16T14:59:48.132892+0000","last_undegraded":"2024-01-16T14:59:48.132892+0000","last_fullsized":"2024-01-16T14:59:48.132892+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:51.864931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114427+0000","last_change":"2024-01-16T14:59:48.114427+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114427+0000","last_undegraded":"2024-01-16T14:59:48.114427+0000","last_fullsized":"2024-01-16T14:59:48.114427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.166154+0000","last_change":"2024-01-16T14:59:48.128976+0000","last_active":"2024-01-16T14:59:49.166154+0000","last_peered":"2024-01-16T14:59:49.166154+0000","last_clean":"2024-01-16T14:59:49.166154+0000","last_became_active":"2024-01-16T14:59:48.128687+0000","last_became_peered":"2024-01-16T14:59:48.128687+0000","last_unstale":"2024-01-16T14:59:49.166154+0000","last_undegraded":"2024-01-16T14:59:49.166154+0000","last_fullsized":"2024-01-16T14:59:49.166154+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.129867+0000","last_change":"2024-01-16T14:59:48.129867+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.129867+0000","last_undegraded":"2024-01-16T14:59:48.129867+0000","last_fullsized":"2024-01-16T14:59:48.129867+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.138620+0000","last_change":"2024-01-16T14:59:48.134109+0000","last_active":"2024-01-16T14:59:48.138620+0000","last_peered":"2024-01-16T14:59:48.138620+0000","last_clean":"2024-01-16T14:59:48.138620+0000","last_became_active":"2024-01-16T14:59:48.133228+0000","last_became_peered":"2024-01-16T14:59:48.133228+0000","last_unstale":"2024-01-16T14:59:48.138620+0000","last_undegraded":"2024-01-16T14:59:48.138620+0000","last_fullsized":"2024-01-16T14:59:48.138620+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127674+0000","last_change":"2024-01-16T14:59:48.127674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127674+0000","last_undegraded":"2024-01-16T14:59:48.127674+0000","last_fullsized":"2024-01-16T14:59:48.127674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142152+0000","last_change":"2024-01-16T14:59:48.137888+0000","last_active":"2024-01-16T14:59:49.142152+0000","last_peered":"2024-01-16T14:59:49.142152+0000","last_clean":"2024-01-16T14:59:49.142152+0000","last_became_active":"2024-01-16T14:59:48.137587+0000","last_became_peered":"2024-01-16T14:59:48.137587+0000","last_unstale":"2024-01-16T14:59:49.142152+0000","last_undegraded":"2024-01-16T14:59:49.142152+0000","last_fullsized":"2024-01-16T14:59:49.142152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:53.615671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152916+0000","last_change":"2024-01-16T14:59:49.149041+0000","last_active":"2024-01-16T14:59:49.152916+0000","last_peered":"2024-01-16T14:59:49.152916+0000","last_clean":"2024-01-16T14:59:49.152916+0000","last_became_active":"2024-01-16T14:59:49.148799+0000","last_became_peered":"2024-01-16T14:59:49.148799+0000","last_unstale":"2024-01-16T14:59:49.152916+0000","last_undegraded":"2024-01-16T14:59:49.152916+0000","last_fullsized":"2024-01-16T14:59:49.152916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168417+0000","last_change":"2024-01-16T14:59:48.127494+0000","last_active":"2024-01-16T14:59:49.168417+0000","last_peered":"2024-01-16T14:59:49.168417+0000","last_clean":"2024-01-16T14:59:49.168417+0000","last_became_active":"2024-01-16T14:59:48.127232+0000","last_became_peered":"2024-01-16T14:59:48.127232+0000","last_unstale":"2024-01-16T14:59:49.168417+0000","last_undegraded":"2024-01-16T14:59:49.168417+0000","last_fullsized":"2024-01-16T14:59:49.168417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165904+0000","last_change":"2024-01-16T14:59:49.162106+0000","last_active":"2024-01-16T14:59:49.165904+0000","last_peered":"2024-01-16T14:59:49.165904+0000","last_clean":"2024-01-16T14:59:49.165904+0000","last_became_active":"2024-01-16T14:59:49.161715+0000","last_became_peered":"2024-01-16T14:59:49.161715+0000","last_unstale":"2024-01-16T14:59:49.165904+0000","last_undegraded":"2024-01-16T14:59:49.165904+0000","last_fullsized":"2024-01-16T14:59:49.165904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.140520+0000","last_change":"2024-01-16T14:59:48.139613+0000","last_active":"2024-01-16T14:59:48.140520+0000","last_peered":"2024-01-16T14:59:48.140520+0000","last_clean":"2024-01-16T14:59:48.140520+0000","last_became_active":"2024-01-16T14:59:48.139289+0000","last_became_peered":"2024-01-16T14:59:48.139289+0000","last_unstale":"2024-01-16T14:59:48.140520+0000","last_undegraded":"2024-01-16T14:59:48.140520+0000","last_fullsized":"2024-01-16T14:59:48.140520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112106+0000","last_change":"2024-01-16T14:59:48.112106+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112106+0000","last_undegraded":"2024-01-16T14:59:48.112106+0000","last_fullsized":"2024-01-16T14:59:48.112106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132124+0000","last_change":"2024-01-16T14:59:48.127338+0000","last_active":"2024-01-16T14:59:48.132124+0000","last_peered":"2024-01-16T14:59:48.132124+0000","last_clean":"2024-01-16T14:59:48.132124+0000","last_became_active":"2024-01-16T14:59:48.127065+0000","last_became_peered":"2024-01-16T14:59:48.127065+0000","last_unstale":"2024-01-16T14:59:48.132124+0000","last_undegraded":"2024-01-16T14:59:48.132124+0000","last_fullsized":"2024-01-16T14:59:48.132124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:20.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112846+0000","last_change":"2024-01-16T14:59:48.112846+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112846+0000","last_undegraded":"2024-01-16T14:59:48.112846+0000","last_fullsized":"2024-01-16T14:59:48.112846+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132404+0000","last_change":"2024-01-16T14:59:48.126842+0000","last_active":"2024-01-16T14:59:48.132404+0000","last_peered":"2024-01-16T14:59:48.132404+0000","last_clean":"2024-01-16T14:59:48.132404+0000","last_became_active":"2024-01-16T14:59:48.126505+0000","last_became_peered":"2024-01-16T14:59:48.126505+0000","last_unstale":"2024-01-16T14:59:48.132404+0000","last_undegraded":"2024-01-16T14:59:48.132404+0000","last_fullsized":"2024-01-16T14:59:48.132404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:30.105655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127029+0000","last_change":"2024-01-16T14:59:48.127029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127029+0000","last_undegraded":"2024-01-16T14:59:48.127029+0000","last_fullsized":"2024-01-16T14:59:48.127029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.141674+0000","last_change":"2024-01-16T14:59:48.130422+0000","last_active":"2024-01-16T14:59:49.141674+0000","last_peered":"2024-01-16T14:59:49.141674+0000","last_clean":"2024-01-16T14:59:49.141674+0000","last_became_active":"2024-01-16T14:59:48.130133+0000","last_became_peered":"2024-01-16T14:59:48.130133+0000","last_unstale":"2024-01-16T14:59:49.141674+0000","last_undegraded":"2024-01-16T14:59:49.141674+0000","last_fullsized":"2024-01-16T14:59:49.141674+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:55.956689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.117678+0000","last_change":"2024-01-16T14:59:48.117678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.117678+0000","last_undegraded":"2024-01-16T14:59:48.117678+0000","last_fullsized":"2024-01-16T14:59:48.117678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.139316+0000","last_change":"2024-01-16T14:59:48.138337+0000","last_active":"2024-01-16T14:59:48.139316+0000","last_peered":"2024-01-16T14:59:48.139316+0000","last_clean":"2024-01-16T14:59:48.139316+0000","last_became_active":"2024-01-16T14:59:48.137540+0000","last_became_peered":"2024-01-16T14:59:48.137540+0000","last_unstale":"2024-01-16T14:59:48.139316+0000","last_undegraded":"2024-01-16T14:59:48.139316+0000","last_fullsized":"2024-01-16T14:59:48.139316+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114300+0000","last_change":"2024-01-16T14:59:48.114300+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114300+0000","last_undegraded":"2024-01-16T14:59:48.114300+0000","last_fullsized":"2024-01-16T14:59:48.114300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.136941+0000","last_change":"2024-01-16T14:59:48.134570+0000","last_active":"2024-01-16T14:59:49.136941+0000","last_peered":"2024-01-16T14:59:49.136941+0000","last_clean":"2024-01-16T14:59:49.136941+0000","last_became_active":"2024-01-16T14:59:48.133797+0000","last_became_peered":"2024-01-16T14:59:48.133797+0000","last_unstale":"2024-01-16T14:59:49.136941+0000","last_undegraded":"2024-01-16T14:59:49.136941+0000","last_fullsized":"2024-01-16T14:59:49.136941+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:12.580412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152657+0000","last_change":"2024-01-16T14:59:49.147430+0000","last_active":"2024-01-16T14:59:49.152657+0000","last_peered":"2024-01-16T14:59:49.152657+0000","last_clean":"2024-01-16T14:59:49.152657+0000","last_became_active":"2024-01-16T14:59:49.147118+0000","last_became_peered":"2024-01-16T14:59:49.147118+0000","last_unstale":"2024-01-16T14:59:49.152657+0000","last_undegraded":"2024-01-16T14:59:49.152657+0000","last_fullsized":"2024-01-16T14:59:49.152657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136626+0000","last_change":"2024-01-16T14:59:48.134611+0000","last_active":"2024-01-16T14:59:48.136626+0000","last_peered":"2024-01-16T14:59:48.136626+0000","last_clean":"2024-01-16T14:59:48.136626+0000","last_became_active":"2024-01-16T14:59:48.134321+0000","last_became_peered":"2024-01-16T14:59:48.134321+0000","last_unstale":"2024-01-16T14:59:48.136626+0000","last_undegraded":"2024-01-16T14:59:48.136626+0000","last_fullsized":"2024-01-16T14:59:48.136626+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153730+0000","last_change":"2024-01-16T14:59:49.149860+0000","last_active":"2024-01-16T14:59:49.153730+0000","last_peered":"2024-01-16T14:59:49.153730+0000","last_clean":"2024-01-16T14:59:49.153730+0000","last_became_active":"2024-01-16T14:59:49.149557+0000","last_became_peered":"2024-01-16T14:59:49.149557+0000","last_unstale":"2024-01-16T14:59:49.153730+0000","last_undegraded":"2024-01-16T14:59:49.153730+0000","last_fullsized":"2024-01-16T14:59:49.153730+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.139001+0000","last_change":"2024-01-16T14:59:48.131348+0000","last_active":"2024-01-16T14:59:49.139001+0000","last_peered":"2024-01-16T14:59:49.139001+0000","last_clean":"2024-01-16T14:59:49.139001+0000","last_became_active":"2024-01-16T14:59:48.131045+0000","last_became_peered":"2024-01-16T14:59:48.131045+0000","last_unstale":"2024-01-16T14:59:49.139001+0000","last_undegraded":"2024-01-16T14:59:49.139001+0000","last_fullsized":"2024-01-16T14:59:49.139001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.119715+0000","last_change":"2024-01-16T14:59:48.119715+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.119715+0000","last_undegraded":"2024-01-16T14:59:48.119715+0000","last_fullsized":"2024-01-16T14:59:48.119715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138671+0000","last_change":"2024-01-16T14:59:48.136397+0000","last_active":"2024-01-16T14:59:49.138671+0000","last_peered":"2024-01-16T14:59:49.138671+0000","last_clean":"2024-01-16T14:59:49.138671+0000","last_became_active":"2024-01-16T14:59:48.136052+0000","last_became_peered":"2024-01-16T14:59:48.136052+0000","last_unstale":"2024-01-16T14:59:49.138671+0000","last_undegraded":"2024-01-16T14:59:49.138671+0000","last_fullsized":"2024-01-16T14:59:49.138671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:47.000083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157313+0000","last_change":"2024-01-16T14:59:49.157313+0000","last_active":"2024-01-16T14:59:49.157313+0000","last_peered":"2024-01-16T14:59:49.157313+0000","last_clean":"2024-01-16T14:59:49.157313+0000","last_became_active":"2024-01-16T14:59:49.156979+0000","last_became_peered":"2024-01-16T14:59:49.156979+0000","last_unstale":"2024-01-16T14:59:49.157313+0000","last_undegraded":"2024-01-16T14:59:49.157313+0000","last_fullsized":"2024-01-16T14:59:49.157313+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136547+0000","last_change":"2024-01-16T14:59:48.131399+0000","last_active":"2024-01-16T14:59:48.136547+0000","last_peered":"2024-01-16T14:59:48.136547+0000","last_clean":"2024-01-16T14:59:48.136547+0000","last_became_active":"2024-01-16T14:59:48.131079+0000","last_became_peered":"2024-01-16T14:59:48.131079+0000","last_unstale":"2024-01-16T14:59:48.136547+0000","last_undegraded":"2024-01-16T14:59:48.136547+0000","last_fullsized":"2024-01-16T14:59:48.136547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:33.060395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116250+0000","last_change":"2024-01-16T14:59:48.116250+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116250+0000","last_undegraded":"2024-01-16T14:59:48.116250+0000","last_fullsized":"2024-01-16T14:59:48.116250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136657+0000","last_change":"2024-01-16T14:59:48.132319+0000","last_active":"2024-01-16T14:59:48.136657+0000","last_peered":"2024-01-16T14:59:48.136657+0000","last_clean":"2024-01-16T14:59:48.136657+0000","last_became_active":"2024-01-16T14:59:48.132030+0000","last_became_peered":"2024-01-16T14:59:48.132030+0000","last_unstale":"2024-01-16T14:59:48.136657+0000","last_undegraded":"2024-01-16T14:59:48.136657+0000","last_fullsized":"2024-01-16T14:59:48.136657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-01-16T14:59:47.105463+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:59:47.105463+0000","last_peered":"2024-01-16T14:59:47.105463+0000","last_clean":"2024-01-16T14:59:47.105463+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:59:47.105463+0000","last_undegraded":"2024-01-16T14:59:47.105463+0000","last_fullsized":"2024-01-16T14:59:47.105463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0660000000000001,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.335,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.22,"15min":1.1319999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":1.044}]},{"osd":1,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1359999999999999,"15min":1.105},"min":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5629999999999999,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.196},{"interface":"front","average":{"1min":1.218,"5min":1.222,"15min":1.216},"min":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.746,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.4399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2050000000000001,"15min":1.1919999999999999},"min":{"1min":0.74299999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.278,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":1.413},{"interface":"front","average":{"1min":1.236,"5min":1.254,"15min":1.22},"min":{"1min":0.877,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4369999999999998,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2190000000000001,"15min":1.2230000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.419,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":5.7969999999999997,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.3169999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.425,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":3.234,"5min":1.7110000000000001,"15min":1.4870000000000001},"min":{"1min":0.82799999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":36.374000000000002,"5min":36.374000000000002,"15min":36.374000000000002},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.407,"15min":1.361},"min":{"1min":0.61299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.9889999999999999,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.431,"15min":1.4099999999999999},"min":{"1min":0.69699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.5139999999999998,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.5029999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"min":{"1min":0.879,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.4849999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.246},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.601},"min":{"1min":0.628,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.5230000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.087},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.254,"15min":1.24},"min":{"1min":0.75800000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.1220000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.081,"15min":1.0549999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.143,"15min":1.1339999999999999},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.643,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2230000000000001,"15min":1.2110000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":1.26},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.337}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.367,"15min":1.377},"min":{"1min":0.97399999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.526},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4159999999999999,"15min":1.4379999999999999},"min":{"1min":0.89300000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8640000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.4419999999999999,"15min":1.468},"min":{"1min":0.91500000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.3740000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0169999999999999,"5min":2.202,"15min":2.202},"last":1.171},{"interface":"front","average":{"1min":1.702,"5min":1.6000000000000001,"15min":1.579},"min":{"1min":1.1839999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.419}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.597,"15min":1.591},"min":{"1min":1.008,"5min":1,"15min":1},"max":{"1min":4.3550000000000004,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.468},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.234}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":1.0369999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.216,"15min":1.147},"min":{"1min":0.76100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.9780000000000002,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.3740000000000001,"15min":1.2490000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.6500000000000004,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1830000000000001,"15min":1.167},"min":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.272,"15min":1.3089999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.319,"15min":1.2669999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3480000000000001,"15min":1.3660000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3660000000000001,"15min":1.3520000000000001},"min":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4620000000000002,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.4810000000000001,"15min":1.496},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.339}]},{"osd":6,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.492,"15min":1.516},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.431,"15min":1.3859999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":10.015000000000001,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.018}]},{"osd":7,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.524,"15min":1.5660000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.583,"15min":1.5740000000000001},"min":{"1min":0.94699999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.357}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2330000000000001,"15min":1.1870000000000001},"min":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9550000000000001,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.383,"5min":1.27,"15min":1.252},"min":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.1870000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.141,"15min":1.089},"min":{"1min":0.76900000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.145,"15min":1.133},"min":{"1min":1.01,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.2789999999999999,"15min":1.2010000000000001},"min":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":8.0579999999999998,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.298,"15min":1.3},"min":{"1min":0.96699999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.0089999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2669999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2280000000000002,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.874},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":6.327,"5min":7.0650000000000004,"15min":7.0650000000000004},"last":0.96699999999999997}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.929,"5min":1.5720000000000001,"15min":1.5600000000000001},"min":{"1min":0.92900000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.289999999999999,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.089}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.173,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.395},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5660000000000001,"15min":1.6020000000000001},"min":{"1min":0.72299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.6259999999999999},"min":{"1min":1.02,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3580000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.349},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.7210000000000001,"15min":1.6950000000000001},"min":{"1min":0.92100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":3.8300000000000001,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.2310000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.91500000000000004,"15min":0.91900000000000004},"min":{"1min":0.57199999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.625,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.97799999999999998,"15min":0.93200000000000005},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.335,"5min":6.6669999999999998,"15min":6.6669999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.56100000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0700000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7370000000000001,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.7230000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.9930000000000001,"5min":6.734,"15min":6.734},"last":1.222},{"interface":"front","average":{"1min":1.115,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.67600000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.80800000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.3540000000000001},"min":{"1min":0.78100000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.6339999999999999,"5min":6.9640000000000004,"15min":6.9640000000000004},"last":1.113},{"interface":"front","average":{"1min":1.262,"5min":1.3200000000000001,"15min":1.3440000000000001},"min":{"1min":0.82499999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6319999999999999,"5min":2.762,"15min":2.762},"last":1.0720000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4179999999999999,"15min":1.466},"min":{"1min":0.88,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0659999999999998,"5min":3.169,"15min":3.169},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4359999999999999,"15min":1.4730000000000001},"min":{"1min":0.93700000000000006,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.956,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.829}]},{"osd":6,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4350000000000001,"15min":1.4099999999999999},"min":{"1min":0.95599999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.716,"5min":6.8460000000000001,"15min":6.8460000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.4379999999999999,"15min":1.4690000000000001},"min":{"1min":0.92800000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.835,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":2.6030000000000002}]},{"osd":7,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.482,"15min":1.448},"min":{"1min":0.92000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.865,"5min":6.9130000000000003,"15min":6.9130000000000003},"last":1.179},{"interface":"front","average":{"1min":1.615,"5min":1.5049999999999999,"15min":1.51},"min":{"1min":0.88400000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":2.3220000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.72099999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.3149999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.754},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.069,"15min":1.105},"min":{"1min":0.56499999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.5760000000000001,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.23},"min":{"1min":0.68200000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.718,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.27,"15min":1.3480000000000001},"min":{"1min":0.54600000000000004,"5min":0.373,"15min":0.373},"max":{"1min":2.6309999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.2509999999999999,"5min":6.141,"15min":6.141},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.331,"15min":1.397},"min":{"1min":0.88700000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.889,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0940000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9619999999999997,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4039999999999999,"15min":1.4399999999999999},"min":{"1min":0.71999999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.3679999999999999,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.4199999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.5009999999999999,"15min":1.5509999999999999},"min":{"1min":1.024,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.2069999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.1479999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.8290000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.278}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4550000000000001,"15min":1.4870000000000001},"min":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1150000000000002,"5min":5.024,"15min":5.024},"last":1.325},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6000000000000001,"15min":1.655},"min":{"1min":1.1140000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.3340000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.373}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6000000000000001,"15min":1.6559999999999999},"min":{"1min":1.196,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.7970000000000002,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5680000000000001,"15min":1.617},"min":{"1min":0.81399999999999995,"5min":0.754,"15min":0.754},"max":{"1min":2.7240000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.5009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.54500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.145,"5min":0.96899999999999997,"15min":0.88500000000000001},"min":{"1min":0.51300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.54,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.90700000000000003,"15min":0.91200000000000003},"min":{"1min":0.64400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.27,"5min":1.107,"15min":1.089},"min":{"1min":0.71599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.876}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2370000000000001,"15min":1.1870000000000001},"min":{"1min":0.753,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.1479999999999999,"15min":1.044},"min":{"1min":0.67500000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.3530000000000002,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.1559999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.4790000000000001,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.833,"5min":1.4239999999999999,"15min":1.3580000000000001},"min":{"1min":0.85499999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.9480000000000004,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.454,"15min":1.4039999999999999},"min":{"1min":1.1160000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":3.3250000000000002,"5min":4.8550000000000004,"15min":4.8550000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.411,"15min":1.421},"min":{"1min":1.0900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.53,"15min":1.492},"min":{"1min":1.0980000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":5.798,"5min":5.798,"15min":5.798},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.456,"15min":1.4039999999999999},"min":{"1min":0.95199999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.0250000000000004,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.534}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.534,"15min":1.504},"min":{"1min":0.90000000000000002,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.458,"15min":1.462},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":791535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.93500000000000005,"15min":0.93700000000000006},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0189999999999999,"15min":1.054},"min":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4690000000000001,"5min":1.587,"15min":1.587},"last":1.046},{"interface":"front","average":{"1min":1.075,"5min":1,"15min":1},"min":{"1min":0.53400000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.339}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.998,"15min":0.96799999999999997},"min":{"1min":0.45100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.145,"5min":1.198,"15min":1.228},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.397}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.607},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.24},"min":{"1min":0.72099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.177}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.4950000000000001},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4399999999999999,"15min":1.4350000000000001},"min":{"1min":0.89200000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.9429999999999996},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.488,"15min":1.5129999999999999},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.423,"15min":1.4610000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.496},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.468,"15min":1.466},"min":{"1min":0.98799999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.516}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:53.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.817+0000 7fbb43400640 1 Processor -- start 2024-01-16T14:59:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.817+0000 7fbb43400640 1 -- start start 2024-01-16T14:59:53.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb43400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb43400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb43400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c070c40 con 0x7fbb3c0b3ae0 2024-01-16T14:59:53.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb43400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3c071450 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb43400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0cfc60 con 0x7fbb3c0cf290 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb40a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb40a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33346/0 (socket says 172.21.15.125:33346) 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.819+0000 7fbb40a00640 1 -- 172.21.15.125:0/687801879 learned_addr learned my addr 172.21.15.125:0/687801879 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb40a00640 1 -- 172.21.15.125:0/687801879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 msgr2=0x7fbb3c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb41400640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb40a00640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb40a00640 1 -- 172.21.15.125:0/687801879 >> v1:172.21.15.137:6789/0 conn(0x7fbb3c0b3ae0 legacy=0x7fbb3c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb40a00640 1 -- 172.21.15.125:0/687801879 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c06c470 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.820+0000 7fbb41400640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.821+0000 7fbb40a00640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009870 tx=0x7fbb2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0a413058556a42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.821+0000 7fbb3be00640 1 -- 172.21.15.125:0/687801879 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c030840 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.821+0000 7fbb3be00640 1 -- 172.21.15.125:0/687801879 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c0309a0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.821+0000 7fbb3be00640 1 -- 172.21.15.125:0/687801879 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c030d30 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 -- 172.21.15.125:0/687801879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 msgr2=0x7fbb3c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009870 tx=0x7fbb2c02b980 comp rx=0 tx=0).stop 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 -- 172.21.15.125:0/687801879 shutdown_connections 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 --2- 172.21.15.125:0/687801879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c15e430 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 -- 172.21.15.125:0/687801879 >> 172.21.15.125:0/687801879 conn(0x7fbb3c06a800 msgr2=0x7fbb3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 -- 172.21.15.125:0/687801879 shutdown_connections 2024-01-16T14:59:53.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.822+0000 7fbb43400640 1 -- 172.21.15.125:0/687801879 wait complete. 2024-01-16T14:59:53.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.824+0000 7fbb43400640 1 Processor -- start 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.824+0000 7fbb43400640 1 -- start start 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0b3ae0 0x7fbb3c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c1591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c15fba0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c160690 con 0x7fbb3c0b3ae0 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb43400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c15fdf0 con 0x7fbb3c0cf290 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33354/0 (socket says 172.21.15.125:33354) 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 -- 172.21.15.125:0/3227641659 learned_addr learned my addr 172.21.15.125:0/3227641659 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 -- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 msgr2=0x7fbb3c1591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c1591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 -- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0b3ae0 msgr2=0x7fbb3c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0b3ae0 0x7fbb3c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.825+0000 7fbb40a00640 1 -- 172.21.15.125:0/3227641659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c0094d0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb40a00640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c000c00 tx=0x7fbb2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c030810 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c043400 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0437d0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c1600d0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.826+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c1571b0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.827+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c0675d0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.828+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fbb2c043930 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.829+0000 7fbb39600640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 0x7fbb1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.829+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb2c03e070 con 0x7fbb3c0ae5a0 2024-01-16T14:59:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.829+0000 7fbb33c00640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 0x7fbb1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.830+0000 7fbb33c00640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 0x7fbb1406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c161250 tx=0x7fbb240079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:53.834+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb2c077dd0 con 0x7fbb3c0ae5a0 2024-01-16T14:59:54.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.022+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb3c071170 con 0x7fbb1406b500 2024-01-16T14:59:54.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.031+0000 7fbb39600640 1 -- 172.21.15.125:0/3227641659 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283891 (secure 0 0 0) 0x7fbb24014400 con 0x7fbb1406b500 2024-01-16T14:59:54.018 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:54.020 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:59:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 msgr2=0x7fbb1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 0x7fbb1406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c161250 tx=0x7fbb240079b0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 msgr2=0x7fbb3c150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c000c00 tx=0x7fbb2c004730 comp rx=0 tx=0).stop 2024-01-16T14:59:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 shutdown_connections 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c0cf290 0x7fbb3c1591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0b3ae0 0x7fbb3c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb1406b500 0x7fbb1406d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 --2- 172.21.15.125:0/3227641659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb3c0ae5a0 0x7fbb3c150610 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 >> 172.21.15.125:0/3227641659 conn(0x7fbb3c06a800 msgr2=0x7fbb3c0b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 shutdown_connections 2024-01-16T14:59:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.038+0000 7fbb43400640 1 -- 172.21.15.125:0/3227641659 wait complete. 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-16T14:59:53.979363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590984,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":173,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254564,"kb_used_data":5092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720156,"statfs":{"total":773094113280,"available":772833439744,"internally_reserved":0,"allocated":5214208,"data_stored":2797205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116216+0000","last_change":"2024-01-16T14:59:48.116216+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116216+0000","last_undegraded":"2024-01-16T14:59:48.116216+0000","last_fullsized":"2024-01-16T14:59:48.116216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.122243+0000","last_change":"2024-01-16T14:59:48.122243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.122243+0000","last_undegraded":"2024-01-16T14:59:48.122243+0000","last_fullsized":"2024-01-16T14:59:48.122243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157265+0000","last_change":"2024-01-16T14:59:49.157265+0000","last_active":"2024-01-16T14:59:49.157265+0000","last_peered":"2024-01-16T14:59:49.157265+0000","last_clean":"2024-01-16T14:59:49.157265+0000","last_became_active":"2024-01-16T14:59:49.156985+0000","last_became_peered":"2024-01-16T14:59:49.156985+0000","last_unstale":"2024-01-16T14:59:49.157265+0000","last_undegraded":"2024-01-16T14:59:49.157265+0000","last_fullsized":"2024-01-16T14:59:49.157265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.118153+0000","last_change":"2024-01-16T14:59:48.118153+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.118153+0000","last_undegraded":"2024-01-16T14:59:48.118153+0000","last_fullsized":"2024-01-16T14:59:48.118153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.121232+0000","last_change":"2024-01-16T14:59:48.121232+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.121232+0000","last_undegraded":"2024-01-16T14:59:48.121232+0000","last_fullsized":"2024-01-16T14:59:48.121232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164334+0000","last_change":"2024-01-16T14:59:49.160920+0000","last_active":"2024-01-16T14:59:49.164334+0000","last_peered":"2024-01-16T14:59:49.164334+0000","last_clean":"2024-01-16T14:59:49.164334+0000","last_became_active":"2024-01-16T14:59:49.158310+0000","last_became_peered":"2024-01-16T14:59:49.158310+0000","last_unstale":"2024-01-16T14:59:49.164334+0000","last_undegraded":"2024-01-16T14:59:49.164334+0000","last_fullsized":"2024-01-16T14:59:49.164334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.670502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116927+0000","last_change":"2024-01-16T14:59:48.116927+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116927+0000","last_undegraded":"2024-01-16T14:59:48.116927+0000","last_fullsized":"2024-01-16T14:59:48.116927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116447+0000","last_change":"2024-01-16T14:59:48.116447+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116447+0000","last_undegraded":"2024-01-16T14:59:48.116447+0000","last_fullsized":"2024-01-16T14:59:48.116447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114791+0000","last_change":"2024-01-16T14:59:48.114791+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114791+0000","last_undegraded":"2024-01-16T14:59:48.114791+0000","last_fullsized":"2024-01-16T14:59:48.114791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116913+0000","last_change":"2024-01-16T14:59:48.116913+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116913+0000","last_undegraded":"2024-01-16T14:59:48.116913+0000","last_fullsized":"2024-01-16T14:59:48.116913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115973+0000","last_change":"2024-01-16T14:59:48.115973+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115973+0000","last_undegraded":"2024-01-16T14:59:48.115973+0000","last_fullsized":"2024-01-16T14:59:48.115973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124245+0000","last_change":"2024-01-16T14:59:48.124245+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124245+0000","last_undegraded":"2024-01-16T14:59:48.124245+0000","last_fullsized":"2024-01-16T14:59:48.124245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114997+0000","last_change":"2024-01-16T14:59:48.114997+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114997+0000","last_undegraded":"2024-01-16T14:59:48.114997+0000","last_fullsized":"2024-01-16T14:59:48.114997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114309+0000","last_change":"2024-01-16T14:59:48.114309+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114309+0000","last_undegraded":"2024-01-16T14:59:48.114309+0000","last_fullsized":"2024-01-16T14:59:48.114309+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163054+0000","last_change":"2024-01-16T14:59:49.158516+0000","last_active":"2024-01-16T14:59:49.163054+0000","last_peered":"2024-01-16T14:59:49.163054+0000","last_clean":"2024-01-16T14:59:49.163054+0000","last_became_active":"2024-01-16T14:59:49.158217+0000","last_became_peered":"2024-01-16T14:59:49.158217+0000","last_unstale":"2024-01-16T14:59:49.163054+0000","last_undegraded":"2024-01-16T14:59:49.163054+0000","last_fullsized":"2024-01-16T14:59:49.163054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124125+0000","last_change":"2024-01-16T14:59:48.124125+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124125+0000","last_undegraded":"2024-01-16T14:59:48.124125+0000","last_fullsized":"2024-01-16T14:59:48.124125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.162728+0000","last_change":"2024-01-16T14:59:49.162728+0000","last_active":"2024-01-16T14:59:49.162728+0000","last_peered":"2024-01-16T14:59:49.162728+0000","last_clean":"2024-01-16T14:59:49.162728+0000","last_became_active":"2024-01-16T14:59:49.162473+0000","last_became_peered":"2024-01-16T14:59:49.162473+0000","last_unstale":"2024-01-16T14:59:49.162728+0000","last_undegraded":"2024-01-16T14:59:49.162728+0000","last_fullsized":"2024-01-16T14:59:49.162728+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:29.139287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159712+0000","last_change":"2024-01-16T14:59:49.146953+0000","last_active":"2024-01-16T14:59:49.159712+0000","last_peered":"2024-01-16T14:59:49.159712+0000","last_clean":"2024-01-16T14:59:49.159712+0000","last_became_active":"2024-01-16T14:59:49.146074+0000","last_became_peered":"2024-01-16T14:59:49.146074+0000","last_unstale":"2024-01-16T14:59:49.159712+0000","last_undegraded":"2024-01-16T14:59:49.159712+0000","last_fullsized":"2024-01-16T14:59:49.159712+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:06.294703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126633+0000","last_change":"2024-01-16T14:59:48.126633+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126633+0000","last_undegraded":"2024-01-16T14:59:48.126633+0000","last_fullsized":"2024-01-16T14:59:48.126633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115055+0000","last_change":"2024-01-16T14:59:48.115055+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115055+0000","last_undegraded":"2024-01-16T14:59:48.115055+0000","last_fullsized":"2024-01-16T14:59:48.115055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163749+0000","last_change":"2024-01-16T14:59:49.158956+0000","last_active":"2024-01-16T14:59:49.163749+0000","last_peered":"2024-01-16T14:59:49.163749+0000","last_clean":"2024-01-16T14:59:49.163749+0000","last_became_active":"2024-01-16T14:59:49.158665+0000","last_became_peered":"2024-01-16T14:59:49.158665+0000","last_unstale":"2024-01-16T14:59:49.163749+0000","last_undegraded":"2024-01-16T14:59:49.163749+0000","last_fullsized":"2024-01-16T14:59:49.163749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112923+0000","last_change":"2024-01-16T14:59:48.112923+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112923+0000","last_undegraded":"2024-01-16T14:59:48.112923+0000","last_fullsized":"2024-01-16T14:59:48.112923+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131707+0000","last_change":"2024-01-16T14:59:48.131707+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131707+0000","last_undegraded":"2024-01-16T14:59:48.131707+0000","last_fullsized":"2024-01-16T14:59:48.131707+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123440+0000","last_change":"2024-01-16T14:59:48.123440+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123440+0000","last_undegraded":"2024-01-16T14:59:48.123440+0000","last_fullsized":"2024-01-16T14:59:48.123440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131918+0000","last_change":"2024-01-16T14:59:48.131918+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131918+0000","last_undegraded":"2024-01-16T14:59:48.131918+0000","last_fullsized":"2024-01-16T14:59:48.131918+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113430+0000","last_change":"2024-01-16T14:59:48.113430+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113430+0000","last_undegraded":"2024-01-16T14:59:48.113430+0000","last_fullsized":"2024-01-16T14:59:48.113430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164250+0000","last_change":"2024-01-16T14:59:49.164250+0000","last_active":"2024-01-16T14:59:49.164250+0000","last_peered":"2024-01-16T14:59:49.164250+0000","last_clean":"2024-01-16T14:59:49.164250+0000","last_became_active":"2024-01-16T14:59:49.163931+0000","last_became_peered":"2024-01-16T14:59:49.163931+0000","last_unstale":"2024-01-16T14:59:49.164250+0000","last_undegraded":"2024-01-16T14:59:49.164250+0000","last_fullsized":"2024-01-16T14:59:49.164250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:48.014735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123754+0000","last_change":"2024-01-16T14:59:48.123754+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123754+0000","last_undegraded":"2024-01-16T14:59:48.123754+0000","last_fullsized":"2024-01-16T14:59:48.123754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126693+0000","last_change":"2024-01-16T14:59:48.126693+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126693+0000","last_undegraded":"2024-01-16T14:59:48.126693+0000","last_fullsized":"2024-01-16T14:59:48.126693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160966+0000","last_change":"2024-01-16T14:59:49.160966+0000","last_active":"2024-01-16T14:59:49.160966+0000","last_peered":"2024-01-16T14:59:49.160966+0000","last_clean":"2024-01-16T14:59:49.160966+0000","last_became_active":"2024-01-16T14:59:49.160612+0000","last_became_peered":"2024-01-16T14:59:49.160612+0000","last_unstale":"2024-01-16T14:59:49.160966+0000","last_undegraded":"2024-01-16T14:59:49.160966+0000","last_fullsized":"2024-01-16T14:59:49.160966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164208+0000","last_change":"2024-01-16T14:59:49.159653+0000","last_active":"2024-01-16T14:59:49.164208+0000","last_peered":"2024-01-16T14:59:49.164208+0000","last_clean":"2024-01-16T14:59:49.164208+0000","last_became_active":"2024-01-16T14:59:49.159375+0000","last_became_peered":"2024-01-16T14:59:49.159375+0000","last_unstale":"2024-01-16T14:59:49.164208+0000","last_undegraded":"2024-01-16T14:59:49.164208+0000","last_fullsized":"2024-01-16T14:59:49.164208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112726+0000","last_change":"2024-01-16T14:59:48.112726+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112726+0000","last_undegraded":"2024-01-16T14:59:48.112726+0000","last_fullsized":"2024-01-16T14:59:48.112726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165746+0000","last_change":"2024-01-16T14:59:49.156116+0000","last_active":"2024-01-16T14:59:49.165746+0000","last_peered":"2024-01-16T14:59:49.165746+0000","last_clean":"2024-01-16T14:59:49.165746+0000","last_became_active":"2024-01-16T14:59:49.155198+0000","last_became_peered":"2024-01-16T14:59:49.155198+0000","last_unstale":"2024-01-16T14:59:49.165746+0000","last_undegraded":"2024-01-16T14:59:49.165746+0000","last_fullsized":"2024-01-16T14:59:49.165746+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:12:21.166819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169542+0000","last_change":"2024-01-16T14:59:48.137330+0000","last_active":"2024-01-16T14:59:49.169542+0000","last_peered":"2024-01-16T14:59:49.169542+0000","last_clean":"2024-01-16T14:59:49.169542+0000","last_became_active":"2024-01-16T14:59:48.136959+0000","last_became_peered":"2024-01-16T14:59:48.136959+0000","last_unstale":"2024-01-16T14:59:49.169542+0000","last_undegraded":"2024-01-16T14:59:49.169542+0000","last_fullsized":"2024-01-16T14:59:49.169542+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:39.353399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.130678+0000","last_change":"2024-01-16T14:59:48.130678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.130678+0000","last_undegraded":"2024-01-16T14:59:48.130678+0000","last_fullsized":"2024-01-16T14:59:48.130678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160262+0000","last_change":"2024-01-16T14:59:48.137753+0000","last_active":"2024-01-16T14:59:49.160262+0000","last_peered":"2024-01-16T14:59:49.160262+0000","last_clean":"2024-01-16T14:59:49.160262+0000","last_became_active":"2024-01-16T14:59:48.137414+0000","last_became_peered":"2024-01-16T14:59:48.137414+0000","last_unstale":"2024-01-16T14:59:49.160262+0000","last_undegraded":"2024-01-16T14:59:49.160262+0000","last_fullsized":"2024-01-16T14:59:49.160262+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123243+0000","last_change":"2024-01-16T14:59:48.123243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123243+0000","last_undegraded":"2024-01-16T14:59:48.123243+0000","last_fullsized":"2024-01-16T14:59:48.123243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159457+0000","last_change":"2024-01-16T14:59:48.134911+0000","last_active":"2024-01-16T14:59:49.159457+0000","last_peered":"2024-01-16T14:59:49.159457+0000","last_clean":"2024-01-16T14:59:49.159457+0000","last_became_active":"2024-01-16T14:59:48.134259+0000","last_became_peered":"2024-01-16T14:59:48.134259+0000","last_unstale":"2024-01-16T14:59:49.159457+0000","last_undegraded":"2024-01-16T14:59:49.159457+0000","last_fullsized":"2024-01-16T14:59:49.159457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:28.124634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167122+0000","last_change":"2024-01-16T14:59:49.165223+0000","last_active":"2024-01-16T14:59:49.167122+0000","last_peered":"2024-01-16T14:59:49.167122+0000","last_clean":"2024-01-16T14:59:49.167122+0000","last_became_active":"2024-01-16T14:59:49.164935+0000","last_became_peered":"2024-01-16T14:59:49.164935+0000","last_unstale":"2024-01-16T14:59:49.167122+0000","last_undegraded":"2024-01-16T14:59:49.167122+0000","last_fullsized":"2024-01-16T14:59:49.167122+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.134046+0000","last_change":"2024-01-16T14:59:48.131434+0000","last_active":"2024-01-16T14:59:48.134046+0000","last_peered":"2024-01-16T14:59:48.134046+0000","last_clean":"2024-01-16T14:59:48.134046+0000","last_became_active":"2024-01-16T14:59:48.131165+0000","last_became_peered":"2024-01-16T14:59:48.131165+0000","last_unstale":"2024-01-16T14:59:48.134046+0000","last_undegraded":"2024-01-16T14:59:48.134046+0000","last_fullsized":"2024-01-16T14:59:48.134046+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159531+0000","last_change":"2024-01-16T14:59:49.159531+0000","last_active":"2024-01-16T14:59:49.159531+0000","last_peered":"2024-01-16T14:59:49.159531+0000","last_clean":"2024-01-16T14:59:49.159531+0000","last_became_active":"2024-01-16T14:59:49.159274+0000","last_became_peered":"2024-01-16T14:59:49.159274+0000","last_unstale":"2024-01-16T14:59:49.159531+0000","last_undegraded":"2024-01-16T14:59:49.159531+0000","last_fullsized":"2024-01-16T14:59:49.159531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168278+0000","last_change":"2024-01-16T14:59:48.128798+0000","last_active":"2024-01-16T14:59:49.168278+0000","last_peered":"2024-01-16T14:59:49.168278+0000","last_clean":"2024-01-16T14:59:49.168278+0000","last_became_active":"2024-01-16T14:59:48.128328+0000","last_became_peered":"2024-01-16T14:59:48.128328+0000","last_unstale":"2024-01-16T14:59:49.168278+0000","last_undegraded":"2024-01-16T14:59:49.168278+0000","last_fullsized":"2024-01-16T14:59:49.168278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:54.681593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113310+0000","last_change":"2024-01-16T14:59:48.113310+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113310+0000","last_undegraded":"2024-01-16T14:59:48.113310+0000","last_fullsized":"2024-01-16T14:59:48.113310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.161777+0000","last_change":"2024-01-16T14:59:48.131571+0000","last_active":"2024-01-16T14:59:49.161777+0000","last_peered":"2024-01-16T14:59:49.161777+0000","last_clean":"2024-01-16T14:59:49.161777+0000","last_became_active":"2024-01-16T14:59:48.131245+0000","last_became_peered":"2024-01-16T14:59:48.131245+0000","last_unstale":"2024-01-16T14:59:49.161777+0000","last_undegraded":"2024-01-16T14:59:49.161777+0000","last_fullsized":"2024-01-16T14:59:49.161777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113674+0000","last_change":"2024-01-16T14:59:48.113674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113674+0000","last_undegraded":"2024-01-16T14:59:48.113674+0000","last_fullsized":"2024-01-16T14:59:48.113674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.133471+0000","last_change":"2024-01-16T14:59:48.130388+0000","last_active":"2024-01-16T14:59:48.133471+0000","last_peered":"2024-01-16T14:59:48.133471+0000","last_clean":"2024-01-16T14:59:48.133471+0000","last_became_active":"2024-01-16T14:59:48.129990+0000","last_became_peered":"2024-01-16T14:59:48.129990+0000","last_unstale":"2024-01-16T14:59:48.133471+0000","last_undegraded":"2024-01-16T14:59:48.133471+0000","last_fullsized":"2024-01-16T14:59:48.133471+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:33.461419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.132738+0000","last_change":"2024-01-16T14:59:48.132738+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.132738+0000","last_undegraded":"2024-01-16T14:59:48.132738+0000","last_fullsized":"2024-01-16T14:59:48.132738+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.128590+0000","last_change":"2024-01-16T14:59:48.122577+0000","last_active":"2024-01-16T14:59:48.128590+0000","last_peered":"2024-01-16T14:59:48.128590+0000","last_clean":"2024-01-16T14:59:48.128590+0000","last_became_active":"2024-01-16T14:59:48.122280+0000","last_became_peered":"2024-01-16T14:59:48.122280+0000","last_unstale":"2024-01-16T14:59:48.128590+0000","last_undegraded":"2024-01-16T14:59:48.128590+0000","last_fullsized":"2024-01-16T14:59:48.128590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:53.539387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159679+0000","last_change":"2024-01-16T14:59:49.146563+0000","last_active":"2024-01-16T14:59:49.159679+0000","last_peered":"2024-01-16T14:59:49.159679+0000","last_clean":"2024-01-16T14:59:49.159679+0000","last_became_active":"2024-01-16T14:59:49.145334+0000","last_became_peered":"2024-01-16T14:59:49.145334+0000","last_unstale":"2024-01-16T14:59:49.159679+0000","last_undegraded":"2024-01-16T14:59:49.159679+0000","last_fullsized":"2024-01-16T14:59:49.159679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:13.595065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.140096+0000","last_change":"2024-01-16T14:59:48.127440+0000","last_active":"2024-01-16T14:59:49.140096+0000","last_peered":"2024-01-16T14:59:49.140096+0000","last_clean":"2024-01-16T14:59:49.140096+0000","last_became_active":"2024-01-16T14:59:48.126114+0000","last_became_peered":"2024-01-16T14:59:48.126114+0000","last_unstale":"2024-01-16T14:59:49.140096+0000","last_undegraded":"2024-01-16T14:59:49.140096+0000","last_fullsized":"2024-01-16T14:59:49.140096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114330+0000","last_change":"2024-01-16T14:59:48.114330+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114330+0000","last_undegraded":"2024-01-16T14:59:48.114330+0000","last_fullsized":"2024-01-16T14:59:48.114330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136234+0000","last_change":"2024-01-16T14:59:48.128532+0000","last_active":"2024-01-16T14:59:48.136234+0000","last_peered":"2024-01-16T14:59:48.136234+0000","last_clean":"2024-01-16T14:59:48.136234+0000","last_became_active":"2024-01-16T14:59:48.127045+0000","last_became_peered":"2024-01-16T14:59:48.127045+0000","last_unstale":"2024-01-16T14:59:48.136234+0000","last_undegraded":"2024-01-16T14:59:48.136234+0000","last_fullsized":"2024-01-16T14:59:48.136234+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:12.546318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165687+0000","last_change":"2024-01-16T14:59:49.157462+0000","last_active":"2024-01-16T14:59:49.165687+0000","last_peered":"2024-01-16T14:59:49.165687+0000","last_clean":"2024-01-16T14:59:49.165687+0000","last_became_active":"2024-01-16T14:59:49.157150+0000","last_became_peered":"2024-01-16T14:59:49.157150+0000","last_unstale":"2024-01-16T14:59:49.165687+0000","last_undegraded":"2024-01-16T14:59:49.165687+0000","last_fullsized":"2024-01-16T14:59:49.165687+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136759+0000","last_change":"2024-01-16T14:59:48.135944+0000","last_active":"2024-01-16T14:59:48.136759+0000","last_peered":"2024-01-16T14:59:48.136759+0000","last_clean":"2024-01-16T14:59:48.136759+0000","last_became_active":"2024-01-16T14:59:48.135643+0000","last_became_peered":"2024-01-16T14:59:48.135643+0000","last_unstale":"2024-01-16T14:59:48.136759+0000","last_undegraded":"2024-01-16T14:59:48.136759+0000","last_fullsized":"2024-01-16T14:59:48.136759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:39.871898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115155+0000","last_change":"2024-01-16T14:59:48.115155+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115155+0000","last_undegraded":"2024-01-16T14:59:48.115155+0000","last_fullsized":"2024-01-16T14:59:48.115155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132942+0000","last_change":"2024-01-16T14:59:48.129597+0000","last_active":"2024-01-16T14:59:48.132942+0000","last_peered":"2024-01-16T14:59:48.132942+0000","last_clean":"2024-01-16T14:59:48.132942+0000","last_became_active":"2024-01-16T14:59:48.128905+0000","last_became_peered":"2024-01-16T14:59:48.128905+0000","last_unstale":"2024-01-16T14:59:48.132942+0000","last_undegraded":"2024-01-16T14:59:48.132942+0000","last_fullsized":"2024-01-16T14:59:48.132942+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:15.588352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.125716+0000","last_change":"2024-01-16T14:59:48.125716+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.125716+0000","last_undegraded":"2024-01-16T14:59:48.125716+0000","last_fullsized":"2024-01-16T14:59:48.125716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169439+0000","last_change":"2024-01-16T14:59:48.128640+0000","last_active":"2024-01-16T14:59:49.169439+0000","last_peered":"2024-01-16T14:59:49.169439+0000","last_clean":"2024-01-16T14:59:49.169439+0000","last_became_active":"2024-01-16T14:59:48.128032+0000","last_became_peered":"2024-01-16T14:59:48.128032+0000","last_unstale":"2024-01-16T14:59:49.169439+0000","last_undegraded":"2024-01-16T14:59:49.169439+0000","last_fullsized":"2024-01-16T14:59:49.169439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:41.977214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112452+0000","last_change":"2024-01-16T14:59:48.112452+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112452+0000","last_undegraded":"2024-01-16T14:59:48.112452+0000","last_fullsized":"2024-01-16T14:59:48.112452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.130910+0000","last_change":"2024-01-16T14:59:48.126828+0000","last_active":"2024-01-16T14:59:48.130910+0000","last_peered":"2024-01-16T14:59:48.130910+0000","last_clean":"2024-01-16T14:59:48.130910+0000","last_became_active":"2024-01-16T14:59:48.126513+0000","last_became_peered":"2024-01-16T14:59:48.126513+0000","last_unstale":"2024-01-16T14:59:48.130910+0000","last_undegraded":"2024-01-16T14:59:48.130910+0000","last_fullsized":"2024-01-16T14:59:48.130910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153727+0000","last_change":"2024-01-16T14:59:49.150175+0000","last_active":"2024-01-16T14:59:49.153727+0000","last_peered":"2024-01-16T14:59:49.153727+0000","last_clean":"2024-01-16T14:59:49.153727+0000","last_became_active":"2024-01-16T14:59:49.149869+0000","last_became_peered":"2024-01-16T14:59:49.149869+0000","last_unstale":"2024-01-16T14:59:49.153727+0000","last_undegraded":"2024-01-16T14:59:49.153727+0000","last_fullsized":"2024-01-16T14:59:49.153727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138602+0000","last_change":"2024-01-16T14:59:48.129480+0000","last_active":"2024-01-16T14:59:49.138602+0000","last_peered":"2024-01-16T14:59:49.138602+0000","last_clean":"2024-01-16T14:59:49.138602+0000","last_became_active":"2024-01-16T14:59:48.129231+0000","last_became_peered":"2024-01-16T14:59:48.129231+0000","last_unstale":"2024-01-16T14:59:49.138602+0000","last_undegraded":"2024-01-16T14:59:49.138602+0000","last_fullsized":"2024-01-16T14:59:49.138602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114361+0000","last_change":"2024-01-16T14:59:48.114361+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114361+0000","last_undegraded":"2024-01-16T14:59:48.114361+0000","last_fullsized":"2024-01-16T14:59:48.114361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.137414+0000","last_change":"2024-01-16T14:59:48.133029+0000","last_active":"2024-01-16T14:59:48.137414+0000","last_peered":"2024-01-16T14:59:48.137414+0000","last_clean":"2024-01-16T14:59:48.137414+0000","last_became_active":"2024-01-16T14:59:48.132732+0000","last_became_peered":"2024-01-16T14:59:48.132732+0000","last_unstale":"2024-01-16T14:59:48.137414+0000","last_undegraded":"2024-01-16T14:59:48.137414+0000","last_fullsized":"2024-01-16T14:59:48.137414+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:36.691574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112029+0000","last_change":"2024-01-16T14:59:48.112029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112029+0000","last_undegraded":"2024-01-16T14:59:48.112029+0000","last_fullsized":"2024-01-16T14:59:48.112029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132892+0000","last_change":"2024-01-16T14:59:48.129568+0000","last_active":"2024-01-16T14:59:48.132892+0000","last_peered":"2024-01-16T14:59:48.132892+0000","last_clean":"2024-01-16T14:59:48.132892+0000","last_became_active":"2024-01-16T14:59:48.129222+0000","last_became_peered":"2024-01-16T14:59:48.129222+0000","last_unstale":"2024-01-16T14:59:48.132892+0000","last_undegraded":"2024-01-16T14:59:48.132892+0000","last_fullsized":"2024-01-16T14:59:48.132892+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:51.864931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114427+0000","last_change":"2024-01-16T14:59:48.114427+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114427+0000","last_undegraded":"2024-01-16T14:59:48.114427+0000","last_fullsized":"2024-01-16T14:59:48.114427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.166154+0000","last_change":"2024-01-16T14:59:48.128976+0000","last_active":"2024-01-16T14:59:49.166154+0000","last_peered":"2024-01-16T14:59:49.166154+0000","last_clean":"2024-01-16T14:59:49.166154+0000","last_became_active":"2024-01-16T14:59:48.128687+0000","last_became_peered":"2024-01-16T14:59:48.128687+0000","last_unstale":"2024-01-16T14:59:49.166154+0000","last_undegraded":"2024-01-16T14:59:49.166154+0000","last_fullsized":"2024-01-16T14:59:49.166154+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.129867+0000","last_change":"2024-01-16T14:59:48.129867+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.129867+0000","last_undegraded":"2024-01-16T14:59:48.129867+0000","last_fullsized":"2024-01-16T14:59:48.129867+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.138620+0000","last_change":"2024-01-16T14:59:48.134109+0000","last_active":"2024-01-16T14:59:48.138620+0000","last_peered":"2024-01-16T14:59:48.138620+0000","last_clean":"2024-01-16T14:59:48.138620+0000","last_became_active":"2024-01-16T14:59:48.133228+0000","last_became_peered":"2024-01-16T14:59:48.133228+0000","last_unstale":"2024-01-16T14:59:48.138620+0000","last_undegraded":"2024-01-16T14:59:48.138620+0000","last_fullsized":"2024-01-16T14:59:48.138620+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127674+0000","last_change":"2024-01-16T14:59:48.127674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127674+0000","last_undegraded":"2024-01-16T14:59:48.127674+0000","last_fullsized":"2024-01-16T14:59:48.127674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142152+0000","last_change":"2024-01-16T14:59:48.137888+0000","last_active":"2024-01-16T14:59:49.142152+0000","last_peered":"2024-01-16T14:59:49.142152+0000","last_clean":"2024-01-16T14:59:49.142152+0000","last_became_active":"2024-01-16T14:59:48.137587+0000","last_became_peered":"2024-01-16T14:59:48.137587+0000","last_unstale":"2024-01-16T14:59:49.142152+0000","last_undegraded":"2024-01-16T14:59:49.142152+0000","last_fullsized":"2024-01-16T14:59:49.142152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:53.615671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152916+0000","last_change":"2024-01-16T14:59:49.149041+0000","last_active":"2024-01-16T14:59:49.152916+0000","last_peered":"2024-01-16T14:59:49.152916+0000","last_clean":"2024-01-16T14:59:49.152916+0000","last_became_active":"2024-01-16T14:59:49.148799+0000","last_became_peered":"2024-01-16T14:59:49.148799+0000","last_unstale":"2024-01-16T14:59:49.152916+0000","last_undegraded":"2024-01-16T14:59:49.152916+0000","last_fullsized":"2024-01-16T14:59:49.152916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168417+0000","last_change":"2024-01-16T14:59:48.127494+0000","last_active":"2024-01-16T14:59:49.168417+0000","last_peered":"2024-01-16T14:59:49.168417+0000","last_clean":"2024-01-16T14:59:49.168417+0000","last_became_active":"2024-01-16T14:59:48.127232+0000","last_became_peered":"2024-01-16T14:59:48.127232+0000","last_unstale":"2024-01-16T14:59:49.168417+0000","last_undegraded":"2024-01-16T14:59:49.168417+0000","last_fullsized":"2024-01-16T14:59:49.168417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165904+0000","last_change":"2024-01-16T14:59:49.162106+0000","last_active":"2024-01-16T14:59:49.165904+0000","last_peered":"2024-01-16T14:59:49.165904+0000","last_clean":"2024-01-16T14:59:49.165904+0000","last_became_active":"2024-01-16T14:59:49.161715+0000","last_became_peered":"2024-01-16T14:59:49.161715+0000","last_unstale":"2024-01-16T14:59:49.165904+0000","last_undegraded":"2024-01-16T14:59:49.165904+0000","last_fullsized":"2024-01-16T14:59:49.165904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.140520+0000","last_change":"2024-01-16T14:59:48.139613+0000","last_active":"2024-01-16T14:59:48.140520+0000","last_peered":"2024-01-16T14:59:48.140520+0000","last_clean":"2024-01-16T14:59:48.140520+0000","last_became_active":"2024-01-16T14:59:48.139289+0000","last_became_peered":"2024-01-16T14:59:48.139289+0000","last_unstale":"2024-01-16T14:59:48.140520+0000","last_undegraded":"2024-01-16T14:59:48.140520+0000","last_fullsized":"2024-01-16T14:59:48.140520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112106+0000","last_change":"2024-01-16T14:59:48.112106+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112106+0000","last_undegraded":"2024-01-16T14:59:48.112106+0000","last_fullsized":"2024-01-16T14:59:48.112106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132124+0000","last_change":"2024-01-16T14:59:48.127338+0000","last_active":"2024-01-16T14:59:48.132124+0000","last_peered":"2024-01-16T14:59:48.132124+0000","last_clean":"2024-01-16T14:59:48.132124+0000","last_became_active":"2024-01-16T14:59:48.127065+0000","last_became_peered":"2024-01-16T14:59:48.127065+0000","last_unstale":"2024-01-16T14:59:48.132124+0000","last_undegraded":"2024-01-16T14:59:48.132124+0000","last_fullsized":"2024-01-16T14:59:48.132124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:20.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112846+0000","last_change":"2024-01-16T14:59:48.112846+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112846+0000","last_undegraded":"2024-01-16T14:59:48.112846+0000","last_fullsized":"2024-01-16T14:59:48.112846+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132404+0000","last_change":"2024-01-16T14:59:48.126842+0000","last_active":"2024-01-16T14:59:48.132404+0000","last_peered":"2024-01-16T14:59:48.132404+0000","last_clean":"2024-01-16T14:59:48.132404+0000","last_became_active":"2024-01-16T14:59:48.126505+0000","last_became_peered":"2024-01-16T14:59:48.126505+0000","last_unstale":"2024-01-16T14:59:48.132404+0000","last_undegraded":"2024-01-16T14:59:48.132404+0000","last_fullsized":"2024-01-16T14:59:48.132404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:30.105655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127029+0000","last_change":"2024-01-16T14:59:48.127029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127029+0000","last_undegraded":"2024-01-16T14:59:48.127029+0000","last_fullsized":"2024-01-16T14:59:48.127029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.141674+0000","last_change":"2024-01-16T14:59:48.130422+0000","last_active":"2024-01-16T14:59:49.141674+0000","last_peered":"2024-01-16T14:59:49.141674+0000","last_clean":"2024-01-16T14:59:49.141674+0000","last_became_active":"2024-01-16T14:59:48.130133+0000","last_became_peered":"2024-01-16T14:59:48.130133+0000","last_unstale":"2024-01-16T14:59:49.141674+0000","last_undegraded":"2024-01-16T14:59:49.141674+0000","last_fullsized":"2024-01-16T14:59:49.141674+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:55.956689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.117678+0000","last_change":"2024-01-16T14:59:48.117678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.117678+0000","last_undegraded":"2024-01-16T14:59:48.117678+0000","last_fullsized":"2024-01-16T14:59:48.117678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.139316+0000","last_change":"2024-01-16T14:59:48.138337+0000","last_active":"2024-01-16T14:59:48.139316+0000","last_peered":"2024-01-16T14:59:48.139316+0000","last_clean":"2024-01-16T14:59:48.139316+0000","last_became_active":"2024-01-16T14:59:48.137540+0000","last_became_peered":"2024-01-16T14:59:48.137540+0000","last_unstale":"2024-01-16T14:59:48.139316+0000","last_undegraded":"2024-01-16T14:59:48.139316+0000","last_fullsized":"2024-01-16T14:59:48.139316+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114300+0000","last_change":"2024-01-16T14:59:48.114300+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114300+0000","last_undegraded":"2024-01-16T14:59:48.114300+0000","last_fullsized":"2024-01-16T14:59:48.114300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.136941+0000","last_change":"2024-01-16T14:59:48.134570+0000","last_active":"2024-01-16T14:59:49.136941+0000","last_peered":"2024-01-16T14:59:49.136941+0000","last_clean":"2024-01-16T14:59:49.136941+0000","last_became_active":"2024-01-16T14:59:48.133797+0000","last_became_peered":"2024-01-16T14:59:48.133797+0000","last_unstale":"2024-01-16T14:59:49.136941+0000","last_undegraded":"2024-01-16T14:59:49.136941+0000","last_fullsized":"2024-01-16T14:59:49.136941+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:12.580412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152657+0000","last_change":"2024-01-16T14:59:49.147430+0000","last_active":"2024-01-16T14:59:49.152657+0000","last_peered":"2024-01-16T14:59:49.152657+0000","last_clean":"2024-01-16T14:59:49.152657+0000","last_became_active":"2024-01-16T14:59:49.147118+0000","last_became_peered":"2024-01-16T14:59:49.147118+0000","last_unstale":"2024-01-16T14:59:49.152657+0000","last_undegraded":"2024-01-16T14:59:49.152657+0000","last_fullsized":"2024-01-16T14:59:49.152657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136626+0000","last_change":"2024-01-16T14:59:48.134611+0000","last_active":"2024-01-16T14:59:48.136626+0000","last_peered":"2024-01-16T14:59:48.136626+0000","last_clean":"2024-01-16T14:59:48.136626+0000","last_became_active":"2024-01-16T14:59:48.134321+0000","last_became_peered":"2024-01-16T14:59:48.134321+0000","last_unstale":"2024-01-16T14:59:48.136626+0000","last_undegraded":"2024-01-16T14:59:48.136626+0000","last_fullsized":"2024-01-16T14:59:48.136626+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153730+0000","last_change":"2024-01-16T14:59:49.149860+0000","last_active":"2024-01-16T14:59:49.153730+0000","last_peered":"2024-01-16T14:59:49.153730+0000","last_clean":"2024-01-16T14:59:49.153730+0000","last_became_active":"2024-01-16T14:59:49.149557+0000","last_became_peered":"2024-01-16T14:59:49.149557+0000","last_unstale":"2024-01-16T14:59:49.153730+0000","last_undegraded":"2024-01-16T14:59:49.153730+0000","last_fullsized":"2024-01-16T14:59:49.153730+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.139001+0000","last_change":"2024-01-16T14:59:48.131348+0000","last_active":"2024-01-16T14:59:49.139001+0000","last_peered":"2024-01-16T14:59:49.139001+0000","last_clean":"2024-01-16T14:59:49.139001+0000","last_became_active":"2024-01-16T14:59:48.131045+0000","last_became_peered":"2024-01-16T14:59:48.131045+0000","last_unstale":"2024-01-16T14:59:49.139001+0000","last_undegraded":"2024-01-16T14:59:49.139001+0000","last_fullsized":"2024-01-16T14:59:49.139001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.119715+0000","last_change":"2024-01-16T14:59:48.119715+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.119715+0000","last_undegraded":"2024-01-16T14:59:48.119715+0000","last_fullsized":"2024-01-16T14:59:48.119715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138671+0000","last_change":"2024-01-16T14:59:48.136397+0000","last_active":"2024-01-16T14:59:49.138671+0000","last_peered":"2024-01-16T14:59:49.138671+0000","last_clean":"2024-01-16T14:59:49.138671+0000","last_became_active":"2024-01-16T14:59:48.136052+0000","last_became_peered":"2024-01-16T14:59:48.136052+0000","last_unstale":"2024-01-16T14:59:49.138671+0000","last_undegraded":"2024-01-16T14:59:49.138671+0000","last_fullsized":"2024-01-16T14:59:49.138671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:47.000083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157313+0000","last_change":"2024-01-16T14:59:49.157313+0000","last_active":"2024-01-16T14:59:49.157313+0000","last_peered":"2024-01-16T14:59:49.157313+0000","last_clean":"2024-01-16T14:59:49.157313+0000","last_became_active":"2024-01-16T14:59:49.156979+0000","last_became_peered":"2024-01-16T14:59:49.156979+0000","last_unstale":"2024-01-16T14:59:49.157313+0000","last_undegraded":"2024-01-16T14:59:49.157313+0000","last_fullsized":"2024-01-16T14:59:49.157313+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136547+0000","last_change":"2024-01-16T14:59:48.131399+0000","last_active":"2024-01-16T14:59:48.136547+0000","last_peered":"2024-01-16T14:59:48.136547+0000","last_clean":"2024-01-16T14:59:48.136547+0000","last_became_active":"2024-01-16T14:59:48.131079+0000","last_became_peered":"2024-01-16T14:59:48.131079+0000","last_unstale":"2024-01-16T14:59:48.136547+0000","last_undegraded":"2024-01-16T14:59:48.136547+0000","last_fullsized":"2024-01-16T14:59:48.136547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:33.060395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116250+0000","last_change":"2024-01-16T14:59:48.116250+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116250+0000","last_undegraded":"2024-01-16T14:59:48.116250+0000","last_fullsized":"2024-01-16T14:59:48.116250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136657+0000","last_change":"2024-01-16T14:59:48.132319+0000","last_active":"2024-01-16T14:59:48.136657+0000","last_peered":"2024-01-16T14:59:48.136657+0000","last_clean":"2024-01-16T14:59:48.136657+0000","last_became_active":"2024-01-16T14:59:48.132030+0000","last_became_peered":"2024-01-16T14:59:48.132030+0000","last_unstale":"2024-01-16T14:59:48.136657+0000","last_undegraded":"2024-01-16T14:59:48.136657+0000","last_fullsized":"2024-01-16T14:59:48.136657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-01-16T14:59:47.105463+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:59:47.105463+0000","last_peered":"2024-01-16T14:59:47.105463+0000","last_clean":"2024-01-16T14:59:47.105463+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:59:47.105463+0000","last_undegraded":"2024-01-16T14:59:47.105463+0000","last_fullsized":"2024-01-16T14:59:47.105463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0660000000000001,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.335,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.22,"15min":1.1319999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":1.044}]},{"osd":1,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1359999999999999,"15min":1.105},"min":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5629999999999999,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.196},{"interface":"front","average":{"1min":1.218,"5min":1.222,"15min":1.216},"min":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.746,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.4399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2050000000000001,"15min":1.1919999999999999},"min":{"1min":0.74299999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.278,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":1.413},{"interface":"front","average":{"1min":1.236,"5min":1.254,"15min":1.22},"min":{"1min":0.877,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4369999999999998,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2190000000000001,"15min":1.2230000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.419,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":5.7969999999999997,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.3169999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.425,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":3.234,"5min":1.7110000000000001,"15min":1.4870000000000001},"min":{"1min":0.82799999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":36.374000000000002,"5min":36.374000000000002,"15min":36.374000000000002},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.407,"15min":1.361},"min":{"1min":0.61299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.9889999999999999,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.431,"15min":1.4099999999999999},"min":{"1min":0.69699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.5139999999999998,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.5029999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"min":{"1min":0.879,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.4849999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.246},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.601},"min":{"1min":0.628,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.5230000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.087},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.254,"15min":1.24},"min":{"1min":0.75800000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.1220000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.081,"15min":1.0549999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.143,"15min":1.1339999999999999},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.643,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2230000000000001,"15min":1.2110000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":1.26},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.337}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.367,"15min":1.377},"min":{"1min":0.97399999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.526},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4159999999999999,"15min":1.4379999999999999},"min":{"1min":0.89300000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8640000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.4419999999999999,"15min":1.468},"min":{"1min":0.91500000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.3740000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0169999999999999,"5min":2.202,"15min":2.202},"last":1.171},{"interface":"front","average":{"1min":1.702,"5min":1.6000000000000001,"15min":1.579},"min":{"1min":1.1839999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.419}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.597,"15min":1.591},"min":{"1min":1.008,"5min":1,"15min":1},"max":{"1min":4.3550000000000004,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.468},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.234}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":1.0369999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.216,"15min":1.147},"min":{"1min":0.76100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.9780000000000002,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.3740000000000001,"15min":1.2490000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.6500000000000004,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1830000000000001,"15min":1.167},"min":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.272,"15min":1.3089999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.319,"15min":1.2669999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3480000000000001,"15min":1.3660000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3660000000000001,"15min":1.3520000000000001},"min":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4620000000000002,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.4810000000000001,"15min":1.496},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.339}]},{"osd":6,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.492,"15min":1.516},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.431,"15min":1.3859999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":10.015000000000001,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.018}]},{"osd":7,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.524,"15min":1.5660000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.583,"15min":1.5740000000000001},"min":{"1min":0.94699999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.357}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2330000000000001,"15min":1.1870000000000001},"min":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9550000000000001,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.383,"5min":1.27,"15min":1.252},"min":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.1870000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.141,"15min":1.089},"min":{"1min":0.76900000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.145,"15min":1.133},"min":{"1min":1.01,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.2789999999999999,"15min":1.2010000000000001},"min":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":8.0579999999999998,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.298,"15min":1.3},"min":{"1min":0.96699999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.0089999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2669999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2280000000000002,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.874},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":6.327,"5min":7.0650000000000004,"15min":7.0650000000000004},"last":0.96699999999999997}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.929,"5min":1.5720000000000001,"15min":1.5600000000000001},"min":{"1min":0.92900000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.289999999999999,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.089}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.173,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.395},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5660000000000001,"15min":1.6020000000000001},"min":{"1min":0.72299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.6259999999999999},"min":{"1min":1.02,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3580000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.349},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.7210000000000001,"15min":1.6950000000000001},"min":{"1min":0.92100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":3.8300000000000001,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.2310000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.91500000000000004,"15min":0.91900000000000004},"min":{"1min":0.57199999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.625,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.97799999999999998,"15min":0.93200000000000005},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.335,"5min":6.6669999999999998,"15min":6.6669999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.56100000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0700000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7370000000000001,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.7230000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.9930000000000001,"5min":6.734,"15min":6.734},"last":1.222},{"interface":"front","average":{"1min":1.115,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.67600000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.80800000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.3540000000000001},"min":{"1min":0.78100000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.6339999999999999,"5min":6.9640000000000004,"15min":6.9640000000000004},"last":1.113},{"interface":"front","average":{"1min":1.262,"5min":1.3200000000000001,"15min":1.3440000000000001},"min":{"1min":0.82499999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6319999999999999,"5min":2.762,"15min":2.762},"last":1.0720000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4179999999999999,"15min":1.466},"min":{"1min":0.88,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0659999999999998,"5min":3.169,"15min":3.169},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4359999999999999,"15min":1.4730000000000001},"min":{"1min":0.93700000000000006,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.956,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.829}]},{"osd":6,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4350000000000001,"15min":1.4099999999999999},"min":{"1min":0.95599999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.716,"5min":6.8460000000000001,"15min":6.8460000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.4379999999999999,"15min":1.4690000000000001},"min":{"1min":0.92800000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.835,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":2.6030000000000002}]},{"osd":7,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.482,"15min":1.448},"min":{"1min":0.92000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.865,"5min":6.9130000000000003,"15min":6.9130000000000003},"last":1.179},{"interface":"front","average":{"1min":1.615,"5min":1.5049999999999999,"15min":1.51},"min":{"1min":0.88400000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":2.3220000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.72099999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.3149999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.754},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.069,"15min":1.105},"min":{"1min":0.56499999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.5760000000000001,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.23},"min":{"1min":0.68200000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.718,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.27,"15min":1.3480000000000001},"min":{"1min":0.54600000000000004,"5min":0.373,"15min":0.373},"max":{"1min":2.6309999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.2509999999999999,"5min":6.141,"15min":6.141},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.331,"15min":1.397},"min":{"1min":0.88700000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.889,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0940000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9619999999999997,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4039999999999999,"15min":1.4399999999999999},"min":{"1min":0.71999999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.3679999999999999,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.4199999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.5009999999999999,"15min":1.5509999999999999},"min":{"1min":1.024,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.2069999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.1479999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.8290000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.278}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4550000000000001,"15min":1.4870000000000001},"min":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1150000000000002,"5min":5.024,"15min":5.024},"last":1.325},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6000000000000001,"15min":1.655},"min":{"1min":1.1140000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.3340000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.373}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6000000000000001,"15min":1.6559999999999999},"min":{"1min":1.196,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.7970000000000002,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5680000000000001,"15min":1.617},"min":{"1min":0.81399999999999995,"5min":0.754,"15min":0.754},"max":{"1min":2.7240000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.5009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.54500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.145,"5min":0.96899999999999997,"15min":0.88500000000000001},"min":{"1min":0.51300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.54,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.90700000000000003,"15min":0.91200000000000003},"min":{"1min":0.64400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.27,"5min":1.107,"15min":1.089},"min":{"1min":0.71599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.876}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2370000000000001,"15min":1.1870000000000001},"min":{"1min":0.753,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.1479999999999999,"15min":1.044},"min":{"1min":0.67500000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.3530000000000002,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.1559999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.4790000000000001,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.833,"5min":1.4239999999999999,"15min":1.3580000000000001},"min":{"1min":0.85499999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.9480000000000004,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.454,"15min":1.4039999999999999},"min":{"1min":1.1160000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":3.3250000000000002,"5min":4.8550000000000004,"15min":4.8550000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.411,"15min":1.421},"min":{"1min":1.0900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.53,"15min":1.492},"min":{"1min":1.0980000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":5.798,"5min":5.798,"15min":5.798},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.456,"15min":1.4039999999999999},"min":{"1min":0.95199999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.0250000000000004,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.534}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.534,"15min":1.504},"min":{"1min":0.90000000000000002,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.458,"15min":1.462},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":791535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.93500000000000005,"15min":0.93700000000000006},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0189999999999999,"15min":1.054},"min":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4690000000000001,"5min":1.587,"15min":1.587},"last":1.046},{"interface":"front","average":{"1min":1.075,"5min":1,"15min":1},"min":{"1min":0.53400000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.339}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.998,"15min":0.96799999999999997},"min":{"1min":0.45100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.145,"5min":1.198,"15min":1.228},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.397}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.607},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.24},"min":{"1min":0.72099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.177}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.4950000000000001},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4399999999999999,"15min":1.4350000000000001},"min":{"1min":0.89200000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.9429999999999996},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.488,"15min":1.5129999999999999},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.423,"15min":1.4610000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.496},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.468,"15min":1.466},"min":{"1min":0.98799999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.516}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:54.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.310+0000 7f2decc00640 1 Processor -- start 2024-01-16T14:59:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.310+0000 7f2decc00640 1 -- start start 2024-01-16T14:59:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2decc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 0x7f2de815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2decc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2de813e360 0x7f2de80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2decc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de806c470 con 0x7f2de815f9c0 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2decc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de806c750 con 0x7f2de813e360 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2decc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de806ca30 con 0x7f2de813df80 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2ddd200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2de813e360 0x7f2de80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2ddd200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2de813e360 0x7f2de80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33368/0 (socket says 172.21.15.125:33368) 2024-01-16T14:59:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.312+0000 7f2ddd200640 1 -- 172.21.15.125:0/1160984091 learned_addr learned my addr 172.21.15.125:0/1160984091 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2de5e00640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 0x7f2de815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2ddd200640 1 -- 172.21.15.125:0/1160984091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 msgr2=0x7f2de815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2ddd200640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 0x7f2de815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2ddd200640 1 -- 172.21.15.125:0/1160984091 >> v1:172.21.15.137:6789/0 conn(0x7f2de815f9c0 legacy=0x7f2de81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2ddd200640 1 -- 172.21.15.125:0/1160984091 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de806cd10 con 0x7f2de813e360 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.313+0000 7f2de5e00640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 0x7f2de815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.314+0000 7f2ddd200640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813e360 0x7f2de80cf290 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009870 tx=0x7f2dd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b2c4898e40e817 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.314+0000 7f2de5400640 1 -- 172.21.15.125:0/1160984091 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0031640 con 0x7f2de813e360 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.314+0000 7f2de5400640 1 -- 172.21.15.125:0/1160984091 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd00317a0 con 0x7f2de813e360 2024-01-16T14:59:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.314+0000 7f2de5400640 1 -- 172.21.15.125:0/1160984091 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0031b30 con 0x7f2de813e360 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 -- 172.21.15.125:0/1160984091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813e360 msgr2=0x7f2de80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813e360 0x7f2de80cf290 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009870 tx=0x7f2dd002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 -- 172.21.15.125:0/1160984091 shutdown_connections 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813e360 0x7f2de80cf290 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 --2- 172.21.15.125:0/1160984091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de813df80 0x7f2de815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 -- 172.21.15.125:0/1160984091 >> 172.21.15.125:0/1160984091 conn(0x7f2de806a800 msgr2=0x7f2de806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 -- 172.21.15.125:0/1160984091 shutdown_connections 2024-01-16T14:59:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.315+0000 7f2decc00640 1 -- 172.21.15.125:0/1160984091 wait complete. 2024-01-16T14:59:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.317+0000 7f2decc00640 1 Processor -- start 2024-01-16T14:59:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.317+0000 7f2decc00640 1 -- start start 2024-01-16T14:59:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 0x7f2de80716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de815f9c0 0x7f2de8071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de813dd60 con 0x7f2de813df80 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80b3270 con 0x7f2de813e360 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2decc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80b3550 con 0x7f2de815f9c0 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33370/0 (socket says 172.21.15.125:33370) 2024-01-16T14:59:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.318+0000 7f2de5e00640 1 -- 172.21.15.125:0/736404091 learned_addr learned my addr 172.21.15.125:0/736404091 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:54.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2ddd200640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 0x7f2de80716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de6800640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de815f9c0 0x7f2de8071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 -- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de815f9c0 msgr2=0x7f2de8071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de815f9c0 0x7f2de8071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 -- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 msgr2=0x7f2de80716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 0x7f2de80716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 -- 172.21.15.125:0/736404091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00094d0 con 0x7f2de813df80 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2ddd200640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 0x7f2de80716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2de5e00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800c7f0 tx=0x7f2dd800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8007820 con 0x7f2de813df80 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd8007980 con 0x7f2de813df80 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8007cf0 con 0x7f2de813df80 2024-01-16T14:59:54.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.319+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de80b3830 con 0x7f2de813df80 2024-01-16T14:59:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.320+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de80b3df0 con 0x7f2de813df80 2024-01-16T14:59:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.321+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de80b4310 con 0x7f2de813df80 2024-01-16T14:59:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.322+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2dd80113f0 con 0x7f2de813df80 2024-01-16T14:59:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.323+0000 7f2ddea00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 0x7f2dc006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.323+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dd808a550 con 0x7f2de813df80 2024-01-16T14:59:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.323+0000 7f2ddd200640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 0x7f2dc006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.324+0000 7f2ddd200640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 0x7f2dc006d9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0036000 tx=0x7f2dd003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.327+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dd8054e20 con 0x7f2de813df80 2024-01-16T14:59:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.515+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2de80ce850 con 0x7f2dc006b530 2024-01-16T14:59:54.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.537+0000 7f2ddea00640 1 -- 172.21.15.125:0/736404091 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283891 (secure 0 0 0) 0x7f2dd0031a90 con 0x7f2dc006b530 2024-01-16T14:59:54.522 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:54.524 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:59:54.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 msgr2=0x7f2dc006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 0x7f2dc006d9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0036000 tx=0x7f2dd003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 msgr2=0x7f2de813d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800c7f0 tx=0x7f2dd800ccb0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 shutdown_connections 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2de815f9c0 0x7f2de8071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2de813e360 0x7f2de80716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dc006b530 0x7f2dc006d9f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 --2- 172.21.15.125:0/736404091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2de813df80 0x7f2de813d820 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 >> 172.21.15.125:0/736404091 conn(0x7f2de806a800 msgr2=0x7f2de80b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 shutdown_connections 2024-01-16T14:59:54.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.544+0000 7f2decc00640 1 -- 172.21.15.125:0/736404091 wait complete. 2024-01-16T14:59:54.542 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-16T14:59:53.979363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590984,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":173,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254564,"kb_used_data":5092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720156,"statfs":{"total":773094113280,"available":772833439744,"internally_reserved":0,"allocated":5214208,"data_stored":2797205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116216+0000","last_change":"2024-01-16T14:59:48.116216+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116216+0000","last_undegraded":"2024-01-16T14:59:48.116216+0000","last_fullsized":"2024-01-16T14:59:48.116216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.122243+0000","last_change":"2024-01-16T14:59:48.122243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.122243+0000","last_undegraded":"2024-01-16T14:59:48.122243+0000","last_fullsized":"2024-01-16T14:59:48.122243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157265+0000","last_change":"2024-01-16T14:59:49.157265+0000","last_active":"2024-01-16T14:59:49.157265+0000","last_peered":"2024-01-16T14:59:49.157265+0000","last_clean":"2024-01-16T14:59:49.157265+0000","last_became_active":"2024-01-16T14:59:49.156985+0000","last_became_peered":"2024-01-16T14:59:49.156985+0000","last_unstale":"2024-01-16T14:59:49.157265+0000","last_undegraded":"2024-01-16T14:59:49.157265+0000","last_fullsized":"2024-01-16T14:59:49.157265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.118153+0000","last_change":"2024-01-16T14:59:48.118153+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.118153+0000","last_undegraded":"2024-01-16T14:59:48.118153+0000","last_fullsized":"2024-01-16T14:59:48.118153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.121232+0000","last_change":"2024-01-16T14:59:48.121232+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.121232+0000","last_undegraded":"2024-01-16T14:59:48.121232+0000","last_fullsized":"2024-01-16T14:59:48.121232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164334+0000","last_change":"2024-01-16T14:59:49.160920+0000","last_active":"2024-01-16T14:59:49.164334+0000","last_peered":"2024-01-16T14:59:49.164334+0000","last_clean":"2024-01-16T14:59:49.164334+0000","last_became_active":"2024-01-16T14:59:49.158310+0000","last_became_peered":"2024-01-16T14:59:49.158310+0000","last_unstale":"2024-01-16T14:59:49.164334+0000","last_undegraded":"2024-01-16T14:59:49.164334+0000","last_fullsized":"2024-01-16T14:59:49.164334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.670502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116927+0000","last_change":"2024-01-16T14:59:48.116927+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116927+0000","last_undegraded":"2024-01-16T14:59:48.116927+0000","last_fullsized":"2024-01-16T14:59:48.116927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116447+0000","last_change":"2024-01-16T14:59:48.116447+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116447+0000","last_undegraded":"2024-01-16T14:59:48.116447+0000","last_fullsized":"2024-01-16T14:59:48.116447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114791+0000","last_change":"2024-01-16T14:59:48.114791+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114791+0000","last_undegraded":"2024-01-16T14:59:48.114791+0000","last_fullsized":"2024-01-16T14:59:48.114791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116913+0000","last_change":"2024-01-16T14:59:48.116913+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116913+0000","last_undegraded":"2024-01-16T14:59:48.116913+0000","last_fullsized":"2024-01-16T14:59:48.116913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115973+0000","last_change":"2024-01-16T14:59:48.115973+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115973+0000","last_undegraded":"2024-01-16T14:59:48.115973+0000","last_fullsized":"2024-01-16T14:59:48.115973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124245+0000","last_change":"2024-01-16T14:59:48.124245+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124245+0000","last_undegraded":"2024-01-16T14:59:48.124245+0000","last_fullsized":"2024-01-16T14:59:48.124245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114997+0000","last_change":"2024-01-16T14:59:48.114997+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114997+0000","last_undegraded":"2024-01-16T14:59:48.114997+0000","last_fullsized":"2024-01-16T14:59:48.114997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114309+0000","last_change":"2024-01-16T14:59:48.114309+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114309+0000","last_undegraded":"2024-01-16T14:59:48.114309+0000","last_fullsized":"2024-01-16T14:59:48.114309+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163054+0000","last_change":"2024-01-16T14:59:49.158516+0000","last_active":"2024-01-16T14:59:49.163054+0000","last_peered":"2024-01-16T14:59:49.163054+0000","last_clean":"2024-01-16T14:59:49.163054+0000","last_became_active":"2024-01-16T14:59:49.158217+0000","last_became_peered":"2024-01-16T14:59:49.158217+0000","last_unstale":"2024-01-16T14:59:49.163054+0000","last_undegraded":"2024-01-16T14:59:49.163054+0000","last_fullsized":"2024-01-16T14:59:49.163054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124125+0000","last_change":"2024-01-16T14:59:48.124125+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124125+0000","last_undegraded":"2024-01-16T14:59:48.124125+0000","last_fullsized":"2024-01-16T14:59:48.124125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.162728+0000","last_change":"2024-01-16T14:59:49.162728+0000","last_active":"2024-01-16T14:59:49.162728+0000","last_peered":"2024-01-16T14:59:49.162728+0000","last_clean":"2024-01-16T14:59:49.162728+0000","last_became_active":"2024-01-16T14:59:49.162473+0000","last_became_peered":"2024-01-16T14:59:49.162473+0000","last_unstale":"2024-01-16T14:59:49.162728+0000","last_undegraded":"2024-01-16T14:59:49.162728+0000","last_fullsized":"2024-01-16T14:59:49.162728+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:29.139287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159712+0000","last_change":"2024-01-16T14:59:49.146953+0000","last_active":"2024-01-16T14:59:49.159712+0000","last_peered":"2024-01-16T14:59:49.159712+0000","last_clean":"2024-01-16T14:59:49.159712+0000","last_became_active":"2024-01-16T14:59:49.146074+0000","last_became_peered":"2024-01-16T14:59:49.146074+0000","last_unstale":"2024-01-16T14:59:49.159712+0000","last_undegraded":"2024-01-16T14:59:49.159712+0000","last_fullsized":"2024-01-16T14:59:49.159712+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:06.294703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126633+0000","last_change":"2024-01-16T14:59:48.126633+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126633+0000","last_undegraded":"2024-01-16T14:59:48.126633+0000","last_fullsized":"2024-01-16T14:59:48.126633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115055+0000","last_change":"2024-01-16T14:59:48.115055+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115055+0000","last_undegraded":"2024-01-16T14:59:48.115055+0000","last_fullsized":"2024-01-16T14:59:48.115055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163749+0000","last_change":"2024-01-16T14:59:49.158956+0000","last_active":"2024-01-16T14:59:49.163749+0000","last_peered":"2024-01-16T14:59:49.163749+0000","last_clean":"2024-01-16T14:59:49.163749+0000","last_became_active":"2024-01-16T14:59:49.158665+0000","last_became_peered":"2024-01-16T14:59:49.158665+0000","last_unstale":"2024-01-16T14:59:49.163749+0000","last_undegraded":"2024-01-16T14:59:49.163749+0000","last_fullsized":"2024-01-16T14:59:49.163749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112923+0000","last_change":"2024-01-16T14:59:48.112923+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112923+0000","last_undegraded":"2024-01-16T14:59:48.112923+0000","last_fullsized":"2024-01-16T14:59:48.112923+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131707+0000","last_change":"2024-01-16T14:59:48.131707+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131707+0000","last_undegraded":"2024-01-16T14:59:48.131707+0000","last_fullsized":"2024-01-16T14:59:48.131707+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123440+0000","last_change":"2024-01-16T14:59:48.123440+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123440+0000","last_undegraded":"2024-01-16T14:59:48.123440+0000","last_fullsized":"2024-01-16T14:59:48.123440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131918+0000","last_change":"2024-01-16T14:59:48.131918+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131918+0000","last_undegraded":"2024-01-16T14:59:48.131918+0000","last_fullsized":"2024-01-16T14:59:48.131918+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113430+0000","last_change":"2024-01-16T14:59:48.113430+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113430+0000","last_undegraded":"2024-01-16T14:59:48.113430+0000","last_fullsized":"2024-01-16T14:59:48.113430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164250+0000","last_change":"2024-01-16T14:59:49.164250+0000","last_active":"2024-01-16T14:59:49.164250+0000","last_peered":"2024-01-16T14:59:49.164250+0000","last_clean":"2024-01-16T14:59:49.164250+0000","last_became_active":"2024-01-16T14:59:49.163931+0000","last_became_peered":"2024-01-16T14:59:49.163931+0000","last_unstale":"2024-01-16T14:59:49.164250+0000","last_undegraded":"2024-01-16T14:59:49.164250+0000","last_fullsized":"2024-01-16T14:59:49.164250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:48.014735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123754+0000","last_change":"2024-01-16T14:59:48.123754+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123754+0000","last_undegraded":"2024-01-16T14:59:48.123754+0000","last_fullsized":"2024-01-16T14:59:48.123754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126693+0000","last_change":"2024-01-16T14:59:48.126693+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126693+0000","last_undegraded":"2024-01-16T14:59:48.126693+0000","last_fullsized":"2024-01-16T14:59:48.126693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160966+0000","last_change":"2024-01-16T14:59:49.160966+0000","last_active":"2024-01-16T14:59:49.160966+0000","last_peered":"2024-01-16T14:59:49.160966+0000","last_clean":"2024-01-16T14:59:49.160966+0000","last_became_active":"2024-01-16T14:59:49.160612+0000","last_became_peered":"2024-01-16T14:59:49.160612+0000","last_unstale":"2024-01-16T14:59:49.160966+0000","last_undegraded":"2024-01-16T14:59:49.160966+0000","last_fullsized":"2024-01-16T14:59:49.160966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164208+0000","last_change":"2024-01-16T14:59:49.159653+0000","last_active":"2024-01-16T14:59:49.164208+0000","last_peered":"2024-01-16T14:59:49.164208+0000","last_clean":"2024-01-16T14:59:49.164208+0000","last_became_active":"2024-01-16T14:59:49.159375+0000","last_became_peered":"2024-01-16T14:59:49.159375+0000","last_unstale":"2024-01-16T14:59:49.164208+0000","last_undegraded":"2024-01-16T14:59:49.164208+0000","last_fullsized":"2024-01-16T14:59:49.164208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112726+0000","last_change":"2024-01-16T14:59:48.112726+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112726+0000","last_undegraded":"2024-01-16T14:59:48.112726+0000","last_fullsized":"2024-01-16T14:59:48.112726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165746+0000","last_change":"2024-01-16T14:59:49.156116+0000","last_active":"2024-01-16T14:59:49.165746+0000","last_peered":"2024-01-16T14:59:49.165746+0000","last_clean":"2024-01-16T14:59:49.165746+0000","last_became_active":"2024-01-16T14:59:49.155198+0000","last_became_peered":"2024-01-16T14:59:49.155198+0000","last_unstale":"2024-01-16T14:59:49.165746+0000","last_undegraded":"2024-01-16T14:59:49.165746+0000","last_fullsized":"2024-01-16T14:59:49.165746+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:12:21.166819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169542+0000","last_change":"2024-01-16T14:59:48.137330+0000","last_active":"2024-01-16T14:59:49.169542+0000","last_peered":"2024-01-16T14:59:49.169542+0000","last_clean":"2024-01-16T14:59:49.169542+0000","last_became_active":"2024-01-16T14:59:48.136959+0000","last_became_peered":"2024-01-16T14:59:48.136959+0000","last_unstale":"2024-01-16T14:59:49.169542+0000","last_undegraded":"2024-01-16T14:59:49.169542+0000","last_fullsized":"2024-01-16T14:59:49.169542+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:39.353399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.130678+0000","last_change":"2024-01-16T14:59:48.130678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.130678+0000","last_undegraded":"2024-01-16T14:59:48.130678+0000","last_fullsized":"2024-01-16T14:59:48.130678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160262+0000","last_change":"2024-01-16T14:59:48.137753+0000","last_active":"2024-01-16T14:59:49.160262+0000","last_peered":"2024-01-16T14:59:49.160262+0000","last_clean":"2024-01-16T14:59:49.160262+0000","last_became_active":"2024-01-16T14:59:48.137414+0000","last_became_peered":"2024-01-16T14:59:48.137414+0000","last_unstale":"2024-01-16T14:59:49.160262+0000","last_undegraded":"2024-01-16T14:59:49.160262+0000","last_fullsized":"2024-01-16T14:59:49.160262+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123243+0000","last_change":"2024-01-16T14:59:48.123243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123243+0000","last_undegraded":"2024-01-16T14:59:48.123243+0000","last_fullsized":"2024-01-16T14:59:48.123243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159457+0000","last_change":"2024-01-16T14:59:48.134911+0000","last_active":"2024-01-16T14:59:49.159457+0000","last_peered":"2024-01-16T14:59:49.159457+0000","last_clean":"2024-01-16T14:59:49.159457+0000","last_became_active":"2024-01-16T14:59:48.134259+0000","last_became_peered":"2024-01-16T14:59:48.134259+0000","last_unstale":"2024-01-16T14:59:49.159457+0000","last_undegraded":"2024-01-16T14:59:49.159457+0000","last_fullsized":"2024-01-16T14:59:49.159457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:28.124634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167122+0000","last_change":"2024-01-16T14:59:49.165223+0000","last_active":"2024-01-16T14:59:49.167122+0000","last_peered":"2024-01-16T14:59:49.167122+0000","last_clean":"2024-01-16T14:59:49.167122+0000","last_became_active":"2024-01-16T14:59:49.164935+0000","last_became_peered":"2024-01-16T14:59:49.164935+0000","last_unstale":"2024-01-16T14:59:49.167122+0000","last_undegraded":"2024-01-16T14:59:49.167122+0000","last_fullsized":"2024-01-16T14:59:49.167122+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.134046+0000","last_change":"2024-01-16T14:59:48.131434+0000","last_active":"2024-01-16T14:59:48.134046+0000","last_peered":"2024-01-16T14:59:48.134046+0000","last_clean":"2024-01-16T14:59:48.134046+0000","last_became_active":"2024-01-16T14:59:48.131165+0000","last_became_peered":"2024-01-16T14:59:48.131165+0000","last_unstale":"2024-01-16T14:59:48.134046+0000","last_undegraded":"2024-01-16T14:59:48.134046+0000","last_fullsized":"2024-01-16T14:59:48.134046+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159531+0000","last_change":"2024-01-16T14:59:49.159531+0000","last_active":"2024-01-16T14:59:49.159531+0000","last_peered":"2024-01-16T14:59:49.159531+0000","last_clean":"2024-01-16T14:59:49.159531+0000","last_became_active":"2024-01-16T14:59:49.159274+0000","last_became_peered":"2024-01-16T14:59:49.159274+0000","last_unstale":"2024-01-16T14:59:49.159531+0000","last_undegraded":"2024-01-16T14:59:49.159531+0000","last_fullsized":"2024-01-16T14:59:49.159531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168278+0000","last_change":"2024-01-16T14:59:48.128798+0000","last_active":"2024-01-16T14:59:49.168278+0000","last_peered":"2024-01-16T14:59:49.168278+0000","last_clean":"2024-01-16T14:59:49.168278+0000","last_became_active":"2024-01-16T14:59:48.128328+0000","last_became_peered":"2024-01-16T14:59:48.128328+0000","last_unstale":"2024-01-16T14:59:49.168278+0000","last_undegraded":"2024-01-16T14:59:49.168278+0000","last_fullsized":"2024-01-16T14:59:49.168278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:54.681593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113310+0000","last_change":"2024-01-16T14:59:48.113310+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113310+0000","last_undegraded":"2024-01-16T14:59:48.113310+0000","last_fullsized":"2024-01-16T14:59:48.113310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.161777+0000","last_change":"2024-01-16T14:59:48.131571+0000","last_active":"2024-01-16T14:59:49.161777+0000","last_peered":"2024-01-16T14:59:49.161777+0000","last_clean":"2024-01-16T14:59:49.161777+0000","last_became_active":"2024-01-16T14:59:48.131245+0000","last_became_peered":"2024-01-16T14:59:48.131245+0000","last_unstale":"2024-01-16T14:59:49.161777+0000","last_undegraded":"2024-01-16T14:59:49.161777+0000","last_fullsized":"2024-01-16T14:59:49.161777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113674+0000","last_change":"2024-01-16T14:59:48.113674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113674+0000","last_undegraded":"2024-01-16T14:59:48.113674+0000","last_fullsized":"2024-01-16T14:59:48.113674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.133471+0000","last_change":"2024-01-16T14:59:48.130388+0000","last_active":"2024-01-16T14:59:48.133471+0000","last_peered":"2024-01-16T14:59:48.133471+0000","last_clean":"2024-01-16T14:59:48.133471+0000","last_became_active":"2024-01-16T14:59:48.129990+0000","last_became_peered":"2024-01-16T14:59:48.129990+0000","last_unstale":"2024-01-16T14:59:48.133471+0000","last_undegraded":"2024-01-16T14:59:48.133471+0000","last_fullsized":"2024-01-16T14:59:48.133471+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:33.461419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.132738+0000","last_change":"2024-01-16T14:59:48.132738+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.132738+0000","last_undegraded":"2024-01-16T14:59:48.132738+0000","last_fullsized":"2024-01-16T14:59:48.132738+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.128590+0000","last_change":"2024-01-16T14:59:48.122577+0000","last_active":"2024-01-16T14:59:48.128590+0000","last_peered":"2024-01-16T14:59:48.128590+0000","last_clean":"2024-01-16T14:59:48.128590+0000","last_became_active":"2024-01-16T14:59:48.122280+0000","last_became_peered":"2024-01-16T14:59:48.122280+0000","last_unstale":"2024-01-16T14:59:48.128590+0000","last_undegraded":"2024-01-16T14:59:48.128590+0000","last_fullsized":"2024-01-16T14:59:48.128590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:53.539387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159679+0000","last_change":"2024-01-16T14:59:49.146563+0000","last_active":"2024-01-16T14:59:49.159679+0000","last_peered":"2024-01-16T14:59:49.159679+0000","last_clean":"2024-01-16T14:59:49.159679+0000","last_became_active":"2024-01-16T14:59:49.145334+0000","last_became_peered":"2024-01-16T14:59:49.145334+0000","last_unstale":"2024-01-16T14:59:49.159679+0000","last_undegraded":"2024-01-16T14:59:49.159679+0000","last_fullsized":"2024-01-16T14:59:49.159679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:13.595065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.140096+0000","last_change":"2024-01-16T14:59:48.127440+0000","last_active":"2024-01-16T14:59:49.140096+0000","last_peered":"2024-01-16T14:59:49.140096+0000","last_clean":"2024-01-16T14:59:49.140096+0000","last_became_active":"2024-01-16T14:59:48.126114+0000","last_became_peered":"2024-01-16T14:59:48.126114+0000","last_unstale":"2024-01-16T14:59:49.140096+0000","last_undegraded":"2024-01-16T14:59:49.140096+0000","last_fullsized":"2024-01-16T14:59:49.140096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114330+0000","last_change":"2024-01-16T14:59:48.114330+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114330+0000","last_undegraded":"2024-01-16T14:59:48.114330+0000","last_fullsized":"2024-01-16T14:59:48.114330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136234+0000","last_change":"2024-01-16T14:59:48.128532+0000","last_active":"2024-01-16T14:59:48.136234+0000","last_peered":"2024-01-16T14:59:48.136234+0000","last_clean":"2024-01-16T14:59:48.136234+0000","last_became_active":"2024-01-16T14:59:48.127045+0000","last_became_peered":"2024-01-16T14:59:48.127045+0000","last_unstale":"2024-01-16T14:59:48.136234+0000","last_undegraded":"2024-01-16T14:59:48.136234+0000","last_fullsized":"2024-01-16T14:59:48.136234+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:12.546318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165687+0000","last_change":"2024-01-16T14:59:49.157462+0000","last_active":"2024-01-16T14:59:49.165687+0000","last_peered":"2024-01-16T14:59:49.165687+0000","last_clean":"2024-01-16T14:59:49.165687+0000","last_became_active":"2024-01-16T14:59:49.157150+0000","last_became_peered":"2024-01-16T14:59:49.157150+0000","last_unstale":"2024-01-16T14:59:49.165687+0000","last_undegraded":"2024-01-16T14:59:49.165687+0000","last_fullsized":"2024-01-16T14:59:49.165687+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136759+0000","last_change":"2024-01-16T14:59:48.135944+0000","last_active":"2024-01-16T14:59:48.136759+0000","last_peered":"2024-01-16T14:59:48.136759+0000","last_clean":"2024-01-16T14:59:48.136759+0000","last_became_active":"2024-01-16T14:59:48.135643+0000","last_became_peered":"2024-01-16T14:59:48.135643+0000","last_unstale":"2024-01-16T14:59:48.136759+0000","last_undegraded":"2024-01-16T14:59:48.136759+0000","last_fullsized":"2024-01-16T14:59:48.136759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:39.871898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115155+0000","last_change":"2024-01-16T14:59:48.115155+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115155+0000","last_undegraded":"2024-01-16T14:59:48.115155+0000","last_fullsized":"2024-01-16T14:59:48.115155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132942+0000","last_change":"2024-01-16T14:59:48.129597+0000","last_active":"2024-01-16T14:59:48.132942+0000","last_peered":"2024-01-16T14:59:48.132942+0000","last_clean":"2024-01-16T14:59:48.132942+0000","last_became_active":"2024-01-16T14:59:48.128905+0000","last_became_peered":"2024-01-16T14:59:48.128905+0000","last_unstale":"2024-01-16T14:59:48.132942+0000","last_undegraded":"2024-01-16T14:59:48.132942+0000","last_fullsized":"2024-01-16T14:59:48.132942+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:15.588352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.125716+0000","last_change":"2024-01-16T14:59:48.125716+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.125716+0000","last_undegraded":"2024-01-16T14:59:48.125716+0000","last_fullsized":"2024-01-16T14:59:48.125716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169439+0000","last_change":"2024-01-16T14:59:48.128640+0000","last_active":"2024-01-16T14:59:49.169439+0000","last_peered":"2024-01-16T14:59:49.169439+0000","last_clean":"2024-01-16T14:59:49.169439+0000","last_became_active":"2024-01-16T14:59:48.128032+0000","last_became_peered":"2024-01-16T14:59:48.128032+0000","last_unstale":"2024-01-16T14:59:49.169439+0000","last_undegraded":"2024-01-16T14:59:49.169439+0000","last_fullsized":"2024-01-16T14:59:49.169439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:41.977214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112452+0000","last_change":"2024-01-16T14:59:48.112452+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112452+0000","last_undegraded":"2024-01-16T14:59:48.112452+0000","last_fullsized":"2024-01-16T14:59:48.112452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.130910+0000","last_change":"2024-01-16T14:59:48.126828+0000","last_active":"2024-01-16T14:59:48.130910+0000","last_peered":"2024-01-16T14:59:48.130910+0000","last_clean":"2024-01-16T14:59:48.130910+0000","last_became_active":"2024-01-16T14:59:48.126513+0000","last_became_peered":"2024-01-16T14:59:48.126513+0000","last_unstale":"2024-01-16T14:59:48.130910+0000","last_undegraded":"2024-01-16T14:59:48.130910+0000","last_fullsized":"2024-01-16T14:59:48.130910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153727+0000","last_change":"2024-01-16T14:59:49.150175+0000","last_active":"2024-01-16T14:59:49.153727+0000","last_peered":"2024-01-16T14:59:49.153727+0000","last_clean":"2024-01-16T14:59:49.153727+0000","last_became_active":"2024-01-16T14:59:49.149869+0000","last_became_peered":"2024-01-16T14:59:49.149869+0000","last_unstale":"2024-01-16T14:59:49.153727+0000","last_undegraded":"2024-01-16T14:59:49.153727+0000","last_fullsized":"2024-01-16T14:59:49.153727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138602+0000","last_change":"2024-01-16T14:59:48.129480+0000","last_active":"2024-01-16T14:59:49.138602+0000","last_peered":"2024-01-16T14:59:49.138602+0000","last_clean":"2024-01-16T14:59:49.138602+0000","last_became_active":"2024-01-16T14:59:48.129231+0000","last_became_peered":"2024-01-16T14:59:48.129231+0000","last_unstale":"2024-01-16T14:59:49.138602+0000","last_undegraded":"2024-01-16T14:59:49.138602+0000","last_fullsized":"2024-01-16T14:59:49.138602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114361+0000","last_change":"2024-01-16T14:59:48.114361+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114361+0000","last_undegraded":"2024-01-16T14:59:48.114361+0000","last_fullsized":"2024-01-16T14:59:48.114361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.137414+0000","last_change":"2024-01-16T14:59:48.133029+0000","last_active":"2024-01-16T14:59:48.137414+0000","last_peered":"2024-01-16T14:59:48.137414+0000","last_clean":"2024-01-16T14:59:48.137414+0000","last_became_active":"2024-01-16T14:59:48.132732+0000","last_became_peered":"2024-01-16T14:59:48.132732+0000","last_unstale":"2024-01-16T14:59:48.137414+0000","last_undegraded":"2024-01-16T14:59:48.137414+0000","last_fullsized":"2024-01-16T14:59:48.137414+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:36.691574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112029+0000","last_change":"2024-01-16T14:59:48.112029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112029+0000","last_undegraded":"2024-01-16T14:59:48.112029+0000","last_fullsized":"2024-01-16T14:59:48.112029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132892+0000","last_change":"2024-01-16T14:59:48.129568+0000","last_active":"2024-01-16T14:59:48.132892+0000","last_peered":"2024-01-16T14:59:48.132892+0000","last_clean":"2024-01-16T14:59:48.132892+0000","last_became_active":"2024-01-16T14:59:48.129222+0000","last_became_peered":"2024-01-16T14:59:48.129222+0000","last_unstale":"2024-01-16T14:59:48.132892+0000","last_undegraded":"2024-01-16T14:59:48.132892+0000","last_fullsized":"2024-01-16T14:59:48.132892+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:51.864931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114427+0000","last_change":"2024-01-16T14:59:48.114427+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114427+0000","last_undegraded":"2024-01-16T14:59:48.114427+0000","last_fullsized":"2024-01-16T14:59:48.114427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.166154+0000","last_change":"2024-01-16T14:59:48.128976+0000","last_active":"2024-01-16T14:59:49.166154+0000","last_peered":"2024-01-16T14:59:49.166154+0000","last_clean":"2024-01-16T14:59:49.166154+0000","last_became_active":"2024-01-16T14:59:48.128687+0000","last_became_peered":"2024-01-16T14:59:48.128687+0000","last_unstale":"2024-01-16T14:59:49.166154+0000","last_undegraded":"2024-01-16T14:59:49.166154+0000","last_fullsized":"2024-01-16T14:59:49.166154+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.129867+0000","last_change":"2024-01-16T14:59:48.129867+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.129867+0000","last_undegraded":"2024-01-16T14:59:48.129867+0000","last_fullsized":"2024-01-16T14:59:48.129867+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.138620+0000","last_change":"2024-01-16T14:59:48.134109+0000","last_active":"2024-01-16T14:59:48.138620+0000","last_peered":"2024-01-16T14:59:48.138620+0000","last_clean":"2024-01-16T14:59:48.138620+0000","last_became_active":"2024-01-16T14:59:48.133228+0000","last_became_peered":"2024-01-16T14:59:48.133228+0000","last_unstale":"2024-01-16T14:59:48.138620+0000","last_undegraded":"2024-01-16T14:59:48.138620+0000","last_fullsized":"2024-01-16T14:59:48.138620+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127674+0000","last_change":"2024-01-16T14:59:48.127674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127674+0000","last_undegraded":"2024-01-16T14:59:48.127674+0000","last_fullsized":"2024-01-16T14:59:48.127674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142152+0000","last_change":"2024-01-16T14:59:48.137888+0000","last_active":"2024-01-16T14:59:49.142152+0000","last_peered":"2024-01-16T14:59:49.142152+0000","last_clean":"2024-01-16T14:59:49.142152+0000","last_became_active":"2024-01-16T14:59:48.137587+0000","last_became_peered":"2024-01-16T14:59:48.137587+0000","last_unstale":"2024-01-16T14:59:49.142152+0000","last_undegraded":"2024-01-16T14:59:49.142152+0000","last_fullsized":"2024-01-16T14:59:49.142152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:53.615671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152916+0000","last_change":"2024-01-16T14:59:49.149041+0000","last_active":"2024-01-16T14:59:49.152916+0000","last_peered":"2024-01-16T14:59:49.152916+0000","last_clean":"2024-01-16T14:59:49.152916+0000","last_became_active":"2024-01-16T14:59:49.148799+0000","last_became_peered":"2024-01-16T14:59:49.148799+0000","last_unstale":"2024-01-16T14:59:49.152916+0000","last_undegraded":"2024-01-16T14:59:49.152916+0000","last_fullsized":"2024-01-16T14:59:49.152916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168417+0000","last_change":"2024-01-16T14:59:48.127494+0000","last_active":"2024-01-16T14:59:49.168417+0000","last_peered":"2024-01-16T14:59:49.168417+0000","last_clean":"2024-01-16T14:59:49.168417+0000","last_became_active":"2024-01-16T14:59:48.127232+0000","last_became_peered":"2024-01-16T14:59:48.127232+0000","last_unstale":"2024-01-16T14:59:49.168417+0000","last_undegraded":"2024-01-16T14:59:49.168417+0000","last_fullsized":"2024-01-16T14:59:49.168417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165904+0000","last_change":"2024-01-16T14:59:49.162106+0000","last_active":"2024-01-16T14:59:49.165904+0000","last_peered":"2024-01-16T14:59:49.165904+0000","last_clean":"2024-01-16T14:59:49.165904+0000","last_became_active":"2024-01-16T14:59:49.161715+0000","last_became_peered":"2024-01-16T14:59:49.161715+0000","last_unstale":"2024-01-16T14:59:49.165904+0000","last_undegraded":"2024-01-16T14:59:49.165904+0000","last_fullsized":"2024-01-16T14:59:49.165904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.140520+0000","last_change":"2024-01-16T14:59:48.139613+0000","last_active":"2024-01-16T14:59:48.140520+0000","last_peered":"2024-01-16T14:59:48.140520+0000","last_clean":"2024-01-16T14:59:48.140520+0000","last_became_active":"2024-01-16T14:59:48.139289+0000","last_became_peered":"2024-01-16T14:59:48.139289+0000","last_unstale":"2024-01-16T14:59:48.140520+0000","last_undegraded":"2024-01-16T14:59:48.140520+0000","last_fullsized":"2024-01-16T14:59:48.140520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112106+0000","last_change":"2024-01-16T14:59:48.112106+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112106+0000","last_undegraded":"2024-01-16T14:59:48.112106+0000","last_fullsized":"2024-01-16T14:59:48.112106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132124+0000","last_change":"2024-01-16T14:59:48.127338+0000","last_active":"2024-01-16T14:59:48.132124+0000","last_peered":"2024-01-16T14:59:48.132124+0000","last_clean":"2024-01-16T14:59:48.132124+0000","last_became_active":"2024-01-16T14:59:48.127065+0000","last_became_peered":"2024-01-16T14:59:48.127065+0000","last_unstale":"2024-01-16T14:59:48.132124+0000","last_undegraded":"2024-01-16T14:59:48.132124+0000","last_fullsized":"2024-01-16T14:59:48.132124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:20.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112846+0000","last_change":"2024-01-16T14:59:48.112846+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112846+0000","last_undegraded":"2024-01-16T14:59:48.112846+0000","last_fullsized":"2024-01-16T14:59:48.112846+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132404+0000","last_change":"2024-01-16T14:59:48.126842+0000","last_active":"2024-01-16T14:59:48.132404+0000","last_peered":"2024-01-16T14:59:48.132404+0000","last_clean":"2024-01-16T14:59:48.132404+0000","last_became_active":"2024-01-16T14:59:48.126505+0000","last_became_peered":"2024-01-16T14:59:48.126505+0000","last_unstale":"2024-01-16T14:59:48.132404+0000","last_undegraded":"2024-01-16T14:59:48.132404+0000","last_fullsized":"2024-01-16T14:59:48.132404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:30.105655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127029+0000","last_change":"2024-01-16T14:59:48.127029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127029+0000","last_undegraded":"2024-01-16T14:59:48.127029+0000","last_fullsized":"2024-01-16T14:59:48.127029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.141674+0000","last_change":"2024-01-16T14:59:48.130422+0000","last_active":"2024-01-16T14:59:49.141674+0000","last_peered":"2024-01-16T14:59:49.141674+0000","last_clean":"2024-01-16T14:59:49.141674+0000","last_became_active":"2024-01-16T14:59:48.130133+0000","last_became_peered":"2024-01-16T14:59:48.130133+0000","last_unstale":"2024-01-16T14:59:49.141674+0000","last_undegraded":"2024-01-16T14:59:49.141674+0000","last_fullsized":"2024-01-16T14:59:49.141674+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:55.956689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.117678+0000","last_change":"2024-01-16T14:59:48.117678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.117678+0000","last_undegraded":"2024-01-16T14:59:48.117678+0000","last_fullsized":"2024-01-16T14:59:48.117678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.139316+0000","last_change":"2024-01-16T14:59:48.138337+0000","last_active":"2024-01-16T14:59:48.139316+0000","last_peered":"2024-01-16T14:59:48.139316+0000","last_clean":"2024-01-16T14:59:48.139316+0000","last_became_active":"2024-01-16T14:59:48.137540+0000","last_became_peered":"2024-01-16T14:59:48.137540+0000","last_unstale":"2024-01-16T14:59:48.139316+0000","last_undegraded":"2024-01-16T14:59:48.139316+0000","last_fullsized":"2024-01-16T14:59:48.139316+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114300+0000","last_change":"2024-01-16T14:59:48.114300+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114300+0000","last_undegraded":"2024-01-16T14:59:48.114300+0000","last_fullsized":"2024-01-16T14:59:48.114300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.136941+0000","last_change":"2024-01-16T14:59:48.134570+0000","last_active":"2024-01-16T14:59:49.136941+0000","last_peered":"2024-01-16T14:59:49.136941+0000","last_clean":"2024-01-16T14:59:49.136941+0000","last_became_active":"2024-01-16T14:59:48.133797+0000","last_became_peered":"2024-01-16T14:59:48.133797+0000","last_unstale":"2024-01-16T14:59:49.136941+0000","last_undegraded":"2024-01-16T14:59:49.136941+0000","last_fullsized":"2024-01-16T14:59:49.136941+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:12.580412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152657+0000","last_change":"2024-01-16T14:59:49.147430+0000","last_active":"2024-01-16T14:59:49.152657+0000","last_peered":"2024-01-16T14:59:49.152657+0000","last_clean":"2024-01-16T14:59:49.152657+0000","last_became_active":"2024-01-16T14:59:49.147118+0000","last_became_peered":"2024-01-16T14:59:49.147118+0000","last_unstale":"2024-01-16T14:59:49.152657+0000","last_undegraded":"2024-01-16T14:59:49.152657+0000","last_fullsized":"2024-01-16T14:59:49.152657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136626+0000","last_change":"2024-01-16T14:59:48.134611+0000","last_active":"2024-01-16T14:59:48.136626+0000","last_peered":"2024-01-16T14:59:48.136626+0000","last_clean":"2024-01-16T14:59:48.136626+0000","last_became_active":"2024-01-16T14:59:48.134321+0000","last_became_peered":"2024-01-16T14:59:48.134321+0000","last_unstale":"2024-01-16T14:59:48.136626+0000","last_undegraded":"2024-01-16T14:59:48.136626+0000","last_fullsized":"2024-01-16T14:59:48.136626+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153730+0000","last_change":"2024-01-16T14:59:49.149860+0000","last_active":"2024-01-16T14:59:49.153730+0000","last_peered":"2024-01-16T14:59:49.153730+0000","last_clean":"2024-01-16T14:59:49.153730+0000","last_became_active":"2024-01-16T14:59:49.149557+0000","last_became_peered":"2024-01-16T14:59:49.149557+0000","last_unstale":"2024-01-16T14:59:49.153730+0000","last_undegraded":"2024-01-16T14:59:49.153730+0000","last_fullsized":"2024-01-16T14:59:49.153730+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.139001+0000","last_change":"2024-01-16T14:59:48.131348+0000","last_active":"2024-01-16T14:59:49.139001+0000","last_peered":"2024-01-16T14:59:49.139001+0000","last_clean":"2024-01-16T14:59:49.139001+0000","last_became_active":"2024-01-16T14:59:48.131045+0000","last_became_peered":"2024-01-16T14:59:48.131045+0000","last_unstale":"2024-01-16T14:59:49.139001+0000","last_undegraded":"2024-01-16T14:59:49.139001+0000","last_fullsized":"2024-01-16T14:59:49.139001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.119715+0000","last_change":"2024-01-16T14:59:48.119715+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.119715+0000","last_undegraded":"2024-01-16T14:59:48.119715+0000","last_fullsized":"2024-01-16T14:59:48.119715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138671+0000","last_change":"2024-01-16T14:59:48.136397+0000","last_active":"2024-01-16T14:59:49.138671+0000","last_peered":"2024-01-16T14:59:49.138671+0000","last_clean":"2024-01-16T14:59:49.138671+0000","last_became_active":"2024-01-16T14:59:48.136052+0000","last_became_peered":"2024-01-16T14:59:48.136052+0000","last_unstale":"2024-01-16T14:59:49.138671+0000","last_undegraded":"2024-01-16T14:59:49.138671+0000","last_fullsized":"2024-01-16T14:59:49.138671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:47.000083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157313+0000","last_change":"2024-01-16T14:59:49.157313+0000","last_active":"2024-01-16T14:59:49.157313+0000","last_peered":"2024-01-16T14:59:49.157313+0000","last_clean":"2024-01-16T14:59:49.157313+0000","last_became_active":"2024-01-16T14:59:49.156979+0000","last_became_peered":"2024-01-16T14:59:49.156979+0000","last_unstale":"2024-01-16T14:59:49.157313+0000","last_undegraded":"2024-01-16T14:59:49.157313+0000","last_fullsized":"2024-01-16T14:59:49.157313+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136547+0000","last_change":"2024-01-16T14:59:48.131399+0000","last_active":"2024-01-16T14:59:48.136547+0000","last_peered":"2024-01-16T14:59:48.136547+0000","last_clean":"2024-01-16T14:59:48.136547+0000","last_became_active":"2024-01-16T14:59:48.131079+0000","last_became_peered":"2024-01-16T14:59:48.131079+0000","last_unstale":"2024-01-16T14:59:48.136547+0000","last_undegraded":"2024-01-16T14:59:48.136547+0000","last_fullsized":"2024-01-16T14:59:48.136547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:33.060395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116250+0000","last_change":"2024-01-16T14:59:48.116250+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116250+0000","last_undegraded":"2024-01-16T14:59:48.116250+0000","last_fullsized":"2024-01-16T14:59:48.116250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136657+0000","last_change":"2024-01-16T14:59:48.132319+0000","last_active":"2024-01-16T14:59:48.136657+0000","last_peered":"2024-01-16T14:59:48.136657+0000","last_clean":"2024-01-16T14:59:48.136657+0000","last_became_active":"2024-01-16T14:59:48.132030+0000","last_became_peered":"2024-01-16T14:59:48.132030+0000","last_unstale":"2024-01-16T14:59:48.136657+0000","last_undegraded":"2024-01-16T14:59:48.136657+0000","last_fullsized":"2024-01-16T14:59:48.136657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-01-16T14:59:47.105463+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:59:47.105463+0000","last_peered":"2024-01-16T14:59:47.105463+0000","last_clean":"2024-01-16T14:59:47.105463+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:59:47.105463+0000","last_undegraded":"2024-01-16T14:59:47.105463+0000","last_fullsized":"2024-01-16T14:59:47.105463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0660000000000001,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.335,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.22,"15min":1.1319999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":1.044}]},{"osd":1,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1359999999999999,"15min":1.105},"min":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5629999999999999,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.196},{"interface":"front","average":{"1min":1.218,"5min":1.222,"15min":1.216},"min":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.746,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.4399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2050000000000001,"15min":1.1919999999999999},"min":{"1min":0.74299999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.278,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":1.413},{"interface":"front","average":{"1min":1.236,"5min":1.254,"15min":1.22},"min":{"1min":0.877,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4369999999999998,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2190000000000001,"15min":1.2230000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.419,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":5.7969999999999997,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.3169999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.425,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":3.234,"5min":1.7110000000000001,"15min":1.4870000000000001},"min":{"1min":0.82799999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":36.374000000000002,"5min":36.374000000000002,"15min":36.374000000000002},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.407,"15min":1.361},"min":{"1min":0.61299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.9889999999999999,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.431,"15min":1.4099999999999999},"min":{"1min":0.69699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.5139999999999998,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.5029999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"min":{"1min":0.879,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.4849999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.246},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.601},"min":{"1min":0.628,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.5230000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.087},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.254,"15min":1.24},"min":{"1min":0.75800000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.1220000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.081,"15min":1.0549999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.143,"15min":1.1339999999999999},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.643,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2230000000000001,"15min":1.2110000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":1.26},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.337}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.367,"15min":1.377},"min":{"1min":0.97399999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.526},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4159999999999999,"15min":1.4379999999999999},"min":{"1min":0.89300000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8640000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.4419999999999999,"15min":1.468},"min":{"1min":0.91500000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.3740000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0169999999999999,"5min":2.202,"15min":2.202},"last":1.171},{"interface":"front","average":{"1min":1.702,"5min":1.6000000000000001,"15min":1.579},"min":{"1min":1.1839999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.419}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.597,"15min":1.591},"min":{"1min":1.008,"5min":1,"15min":1},"max":{"1min":4.3550000000000004,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.468},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.234}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":1.0369999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.216,"15min":1.147},"min":{"1min":0.76100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.9780000000000002,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.3740000000000001,"15min":1.2490000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.6500000000000004,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1830000000000001,"15min":1.167},"min":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.272,"15min":1.3089999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.319,"15min":1.2669999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3480000000000001,"15min":1.3660000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3660000000000001,"15min":1.3520000000000001},"min":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4620000000000002,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.4810000000000001,"15min":1.496},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.339}]},{"osd":6,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.492,"15min":1.516},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.431,"15min":1.3859999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":10.015000000000001,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.018}]},{"osd":7,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.524,"15min":1.5660000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.583,"15min":1.5740000000000001},"min":{"1min":0.94699999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.357}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2330000000000001,"15min":1.1870000000000001},"min":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9550000000000001,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.383,"5min":1.27,"15min":1.252},"min":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.1870000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.141,"15min":1.089},"min":{"1min":0.76900000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.145,"15min":1.133},"min":{"1min":1.01,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.2789999999999999,"15min":1.2010000000000001},"min":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":8.0579999999999998,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.298,"15min":1.3},"min":{"1min":0.96699999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.0089999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2669999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2280000000000002,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.874},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":6.327,"5min":7.0650000000000004,"15min":7.0650000000000004},"last":0.96699999999999997}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.929,"5min":1.5720000000000001,"15min":1.5600000000000001},"min":{"1min":0.92900000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.289999999999999,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.089}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.173,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.395},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5660000000000001,"15min":1.6020000000000001},"min":{"1min":0.72299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.6259999999999999},"min":{"1min":1.02,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3580000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.349},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.7210000000000001,"15min":1.6950000000000001},"min":{"1min":0.92100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":3.8300000000000001,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.2310000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.91500000000000004,"15min":0.91900000000000004},"min":{"1min":0.57199999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.625,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.97799999999999998,"15min":0.93200000000000005},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.335,"5min":6.6669999999999998,"15min":6.6669999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.56100000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0700000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7370000000000001,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.7230000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.9930000000000001,"5min":6.734,"15min":6.734},"last":1.222},{"interface":"front","average":{"1min":1.115,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.67600000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.80800000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.3540000000000001},"min":{"1min":0.78100000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.6339999999999999,"5min":6.9640000000000004,"15min":6.9640000000000004},"last":1.113},{"interface":"front","average":{"1min":1.262,"5min":1.3200000000000001,"15min":1.3440000000000001},"min":{"1min":0.82499999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6319999999999999,"5min":2.762,"15min":2.762},"last":1.0720000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4179999999999999,"15min":1.466},"min":{"1min":0.88,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0659999999999998,"5min":3.169,"15min":3.169},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4359999999999999,"15min":1.4730000000000001},"min":{"1min":0.93700000000000006,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.956,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.829}]},{"osd":6,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4350000000000001,"15min":1.4099999999999999},"min":{"1min":0.95599999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.716,"5min":6.8460000000000001,"15min":6.8460000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.4379999999999999,"15min":1.4690000000000001},"min":{"1min":0.92800000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.835,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":2.6030000000000002}]},{"osd":7,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.482,"15min":1.448},"min":{"1min":0.92000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.865,"5min":6.9130000000000003,"15min":6.9130000000000003},"last":1.179},{"interface":"front","average":{"1min":1.615,"5min":1.5049999999999999,"15min":1.51},"min":{"1min":0.88400000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":2.3220000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.72099999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.3149999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.754},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.069,"15min":1.105},"min":{"1min":0.56499999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.5760000000000001,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.23},"min":{"1min":0.68200000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.718,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.27,"15min":1.3480000000000001},"min":{"1min":0.54600000000000004,"5min":0.373,"15min":0.373},"max":{"1min":2.6309999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.2509999999999999,"5min":6.141,"15min":6.141},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.331,"15min":1.397},"min":{"1min":0.88700000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.889,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0940000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9619999999999997,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4039999999999999,"15min":1.4399999999999999},"min":{"1min":0.71999999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.3679999999999999,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.4199999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.5009999999999999,"15min":1.5509999999999999},"min":{"1min":1.024,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.2069999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.1479999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.8290000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.278}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4550000000000001,"15min":1.4870000000000001},"min":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1150000000000002,"5min":5.024,"15min":5.024},"last":1.325},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6000000000000001,"15min":1.655},"min":{"1min":1.1140000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.3340000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.373}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6000000000000001,"15min":1.6559999999999999},"min":{"1min":1.196,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.7970000000000002,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5680000000000001,"15min":1.617},"min":{"1min":0.81399999999999995,"5min":0.754,"15min":0.754},"max":{"1min":2.7240000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.5009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.54500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.145,"5min":0.96899999999999997,"15min":0.88500000000000001},"min":{"1min":0.51300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.54,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.90700000000000003,"15min":0.91200000000000003},"min":{"1min":0.64400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.27,"5min":1.107,"15min":1.089},"min":{"1min":0.71599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.876}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2370000000000001,"15min":1.1870000000000001},"min":{"1min":0.753,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.1479999999999999,"15min":1.044},"min":{"1min":0.67500000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.3530000000000002,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.1559999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.4790000000000001,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.833,"5min":1.4239999999999999,"15min":1.3580000000000001},"min":{"1min":0.85499999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.9480000000000004,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.454,"15min":1.4039999999999999},"min":{"1min":1.1160000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":3.3250000000000002,"5min":4.8550000000000004,"15min":4.8550000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.411,"15min":1.421},"min":{"1min":1.0900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.53,"15min":1.492},"min":{"1min":1.0980000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":5.798,"5min":5.798,"15min":5.798},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.456,"15min":1.4039999999999999},"min":{"1min":0.95199999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.0250000000000004,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.534}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.534,"15min":1.504},"min":{"1min":0.90000000000000002,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.458,"15min":1.462},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":791535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.93500000000000005,"15min":0.93700000000000006},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0189999999999999,"15min":1.054},"min":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4690000000000001,"5min":1.587,"15min":1.587},"last":1.046},{"interface":"front","average":{"1min":1.075,"5min":1,"15min":1},"min":{"1min":0.53400000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.339}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.998,"15min":0.96799999999999997},"min":{"1min":0.45100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.145,"5min":1.198,"15min":1.228},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.397}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.607},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.24},"min":{"1min":0.72099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.177}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.4950000000000001},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4399999999999999,"15min":1.4350000000000001},"min":{"1min":0.89200000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.9429999999999996},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.488,"15min":1.5129999999999999},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.423,"15min":1.4610000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.496},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.468,"15min":1.466},"min":{"1min":0.98799999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.516}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:54.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-01-16T14:59:54.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.116216+0000', 'last_change': '2024-01-16T14:59:48.116216+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.116216+0000', 'last_undegraded': '2024-01-16T14:59:48.116216+0000', 'last_fullsized': '2024-01-16T14:59:48.116216+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-01-16T14:59:54.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.122243+0000', 'last_change': '2024-01-16T14:59:48.122243+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.122243+0000', 'last_undegraded': '2024-01-16T14:59:48.122243+0000', 'last_fullsized': '2024-01-16T14:59:48.122243+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.118153+0000', 'last_change': '2024-01-16T14:59:48.118153+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.118153+0000', 'last_undegraded': '2024-01-16T14:59:48.118153+0000', 'last_fullsized': '2024-01-16T14:59:48.118153+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.121232+0000', 'last_change': '2024-01-16T14:59:48.121232+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.121232+0000', 'last_undegraded': '2024-01-16T14:59:48.121232+0000', 'last_fullsized': '2024-01-16T14:59:48.121232+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.116927+0000', 'last_change': '2024-01-16T14:59:48.116927+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.116927+0000', 'last_undegraded': '2024-01-16T14:59:48.116927+0000', 'last_fullsized': '2024-01-16T14:59:48.116927+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.116447+0000', 'last_change': '2024-01-16T14:59:48.116447+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.116447+0000', 'last_undegraded': '2024-01-16T14:59:48.116447+0000', 'last_fullsized': '2024-01-16T14:59:48.116447+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114791+0000', 'last_change': '2024-01-16T14:59:48.114791+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114791+0000', 'last_undegraded': '2024-01-16T14:59:48.114791+0000', 'last_fullsized': '2024-01-16T14:59:48.114791+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-01-16T14:59:54.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.116913+0000', 'last_change': '2024-01-16T14:59:48.116913+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.116913+0000', 'last_undegraded': '2024-01-16T14:59:48.116913+0000', 'last_fullsized': '2024-01-16T14:59:48.116913+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.115973+0000', 'last_change': '2024-01-16T14:59:48.115973+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.115973+0000', 'last_undegraded': '2024-01-16T14:59:48.115973+0000', 'last_fullsized': '2024-01-16T14:59:48.115973+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.124245+0000', 'last_change': '2024-01-16T14:59:48.124245+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.124245+0000', 'last_undegraded': '2024-01-16T14:59:48.124245+0000', 'last_fullsized': '2024-01-16T14:59:48.124245+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114997+0000', 'last_change': '2024-01-16T14:59:48.114997+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114997+0000', 'last_undegraded': '2024-01-16T14:59:48.114997+0000', 'last_fullsized': '2024-01-16T14:59:48.114997+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114309+0000', 'last_change': '2024-01-16T14:59:48.114309+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114309+0000', 'last_undegraded': '2024-01-16T14:59:48.114309+0000', 'last_fullsized': '2024-01-16T14:59:48.114309+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.124125+0000', 'last_change': '2024-01-16T14:59:48.124125+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.124125+0000', 'last_undegraded': '2024-01-16T14:59:48.124125+0000', 'last_fullsized': '2024-01-16T14:59:48.124125+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.126633+0000', 'last_change': '2024-01-16T14:59:48.126633+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.126633+0000', 'last_undegraded': '2024-01-16T14:59:48.126633+0000', 'last_fullsized': '2024-01-16T14:59:48.126633+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.115055+0000', 'last_change': '2024-01-16T14:59:48.115055+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.115055+0000', 'last_undegraded': '2024-01-16T14:59:48.115055+0000', 'last_fullsized': '2024-01-16T14:59:48.115055+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112923+0000', 'last_change': '2024-01-16T14:59:48.112923+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112923+0000', 'last_undegraded': '2024-01-16T14:59:48.112923+0000', 'last_fullsized': '2024-01-16T14:59:48.112923+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.131707+0000', 'last_change': '2024-01-16T14:59:48.131707+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.131707+0000', 'last_undegraded': '2024-01-16T14:59:48.131707+0000', 'last_fullsized': '2024-01-16T14:59:48.131707+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.123440+0000', 'last_change': '2024-01-16T14:59:48.123440+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.123440+0000', 'last_undegraded': '2024-01-16T14:59:48.123440+0000', 'last_fullsized': '2024-01-16T14:59:48.123440+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-01-16T14:59:54.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.131918+0000', 'last_change': '2024-01-16T14:59:48.131918+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.131918+0000', 'last_undegraded': '2024-01-16T14:59:48.131918+0000', 'last_fullsized': '2024-01-16T14:59:48.131918+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.113430+0000', 'last_change': '2024-01-16T14:59:48.113430+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.113430+0000', 'last_undegraded': '2024-01-16T14:59:48.113430+0000', 'last_fullsized': '2024-01-16T14:59:48.113430+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.123754+0000', 'last_change': '2024-01-16T14:59:48.123754+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.123754+0000', 'last_undegraded': '2024-01-16T14:59:48.123754+0000', 'last_fullsized': '2024-01-16T14:59:48.123754+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.126693+0000', 'last_change': '2024-01-16T14:59:48.126693+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.126693+0000', 'last_undegraded': '2024-01-16T14:59:48.126693+0000', 'last_fullsized': '2024-01-16T14:59:48.126693+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112726+0000', 'last_change': '2024-01-16T14:59:48.112726+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112726+0000', 'last_undegraded': '2024-01-16T14:59:48.112726+0000', 'last_fullsized': '2024-01-16T14:59:48.112726+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.130678+0000', 'last_change': '2024-01-16T14:59:48.130678+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.130678+0000', 'last_undegraded': '2024-01-16T14:59:48.130678+0000', 'last_fullsized': '2024-01-16T14:59:48.130678+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.123243+0000', 'last_change': '2024-01-16T14:59:48.123243+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.123243+0000', 'last_undegraded': '2024-01-16T14:59:48.123243+0000', 'last_fullsized': '2024-01-16T14:59:48.123243+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.113310+0000', 'last_change': '2024-01-16T14:59:48.113310+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.113310+0000', 'last_undegraded': '2024-01-16T14:59:48.113310+0000', 'last_fullsized': '2024-01-16T14:59:48.113310+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.113674+0000', 'last_change': '2024-01-16T14:59:48.113674+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.113674+0000', 'last_undegraded': '2024-01-16T14:59:48.113674+0000', 'last_fullsized': '2024-01-16T14:59:48.113674+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.132738+0000', 'last_change': '2024-01-16T14:59:48.132738+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.132738+0000', 'last_undegraded': '2024-01-16T14:59:48.132738+0000', 'last_fullsized': '2024-01-16T14:59:48.132738+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114330+0000', 'last_change': '2024-01-16T14:59:48.114330+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114330+0000', 'last_undegraded': '2024-01-16T14:59:48.114330+0000', 'last_fullsized': '2024-01-16T14:59:48.114330+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-01-16T14:59:54.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.115155+0000', 'last_change': '2024-01-16T14:59:48.115155+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.115155+0000', 'last_undegraded': '2024-01-16T14:59:48.115155+0000', 'last_fullsized': '2024-01-16T14:59:48.115155+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.125716+0000', 'last_change': '2024-01-16T14:59:48.125716+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.125716+0000', 'last_undegraded': '2024-01-16T14:59:48.125716+0000', 'last_fullsized': '2024-01-16T14:59:48.125716+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112452+0000', 'last_change': '2024-01-16T14:59:48.112452+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112452+0000', 'last_undegraded': '2024-01-16T14:59:48.112452+0000', 'last_fullsized': '2024-01-16T14:59:48.112452+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114361+0000', 'last_change': '2024-01-16T14:59:48.114361+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114361+0000', 'last_undegraded': '2024-01-16T14:59:48.114361+0000', 'last_fullsized': '2024-01-16T14:59:48.114361+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112029+0000', 'last_change': '2024-01-16T14:59:48.112029+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112029+0000', 'last_undegraded': '2024-01-16T14:59:48.112029+0000', 'last_fullsized': '2024-01-16T14:59:48.112029+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114427+0000', 'last_change': '2024-01-16T14:59:48.114427+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114427+0000', 'last_undegraded': '2024-01-16T14:59:48.114427+0000', 'last_fullsized': '2024-01-16T14:59:48.114427+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.129867+0000', 'last_change': '2024-01-16T14:59:48.129867+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.129867+0000', 'last_undegraded': '2024-01-16T14:59:48.129867+0000', 'last_fullsized': '2024-01-16T14:59:48.129867+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.127674+0000', 'last_change': '2024-01-16T14:59:48.127674+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.127674+0000', 'last_undegraded': '2024-01-16T14:59:48.127674+0000', 'last_fullsized': '2024-01-16T14:59:48.127674+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112106+0000', 'last_change': '2024-01-16T14:59:48.112106+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112106+0000', 'last_undegraded': '2024-01-16T14:59:48.112106+0000', 'last_fullsized': '2024-01-16T14:59:48.112106+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.112846+0000', 'last_change': '2024-01-16T14:59:48.112846+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.112846+0000', 'last_undegraded': '2024-01-16T14:59:48.112846+0000', 'last_fullsized': '2024-01-16T14:59:48.112846+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.127029+0000', 'last_change': '2024-01-16T14:59:48.127029+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.127029+0000', 'last_undegraded': '2024-01-16T14:59:48.127029+0000', 'last_fullsized': '2024-01-16T14:59:48.127029+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-01-16T14:59:54.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.117678+0000', 'last_change': '2024-01-16T14:59:48.117678+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.117678+0000', 'last_undegraded': '2024-01-16T14:59:48.117678+0000', 'last_fullsized': '2024-01-16T14:59:48.117678+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.114300+0000', 'last_change': '2024-01-16T14:59:48.114300+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.114300+0000', 'last_undegraded': '2024-01-16T14:59:48.114300+0000', 'last_fullsized': '2024-01-16T14:59:48.114300+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.119715+0000', 'last_change': '2024-01-16T14:59:48.119715+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.119715+0000', 'last_undegraded': '2024-01-16T14:59:48.119715+0000', 'last_fullsized': '2024-01-16T14:59:48.119715+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-01-16T14:59:54.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-16T14:59:48.116250+0000', 'last_change': '2024-01-16T14:59:48.116250+0000', 'last_active': '2024-01-16T14:59:48.095777+0000', 'last_peered': '2024-01-16T14:59:48.095777+0000', 'last_clean': '2024-01-16T14:59:48.095777+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T14:59:48.116250+0000', 'last_undegraded': '2024-01-16T14:59:48.116250+0000', 'last_fullsized': '2024-01-16T14:59:48.116250+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'last_clean_scrub_stamp': '2024-01-16T14:59:48.095777+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T14:59:54.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T14:59:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.807+0000 7f538fe00640 1 Processor -- start 2024-01-16T14:59:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.808+0000 7f538fe00640 1 -- start start 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.809+0000 7f538fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f53900d0100 0x7f53900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.809+0000 7f538fe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53900d0a10 con 0x7f5390160920 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.809+0000 7f538fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53900cf080 con 0x7f53900d0f40 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.809+0000 7f538fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53900c6be0 con 0x7f53900d0100 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5390160920 0x7f53900cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52856/0 (socket says 172.21.15.125:52856) 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538ea00640 1 -- 172.21.15.125:0/2324545091 learned_addr learned my addr 172.21.15.125:0/2324545091 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949709465 0 0) 0x7f53900d0a10 con 0x7f5390160920 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370003520 con 0x7f5390160920 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538e000640 1 --2- 172.21.15.125:0/2324545091 >> v2:172.21.15.137:3300/0 conn(0x7f53900d0100 0x7f53900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896490603 0 0) 0x7f5370003520 con 0x7f5390160920 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 >> v2:172.21.15.137:3300/0 conn(0x7f53900d0100 msgr2=0x7f53900b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 --2- 172.21.15.125:0/2324545091 >> v2:172.21.15.137:3300/0 conn(0x7f53900d0100 0x7f53900b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 >> v1:172.21.15.137:6789/0 conn(0x7f53900d0f40 legacy=0x7f53900d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900ce7e0 con 0x7f5390160920 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5380001670 con 0x7f5390160920 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.810+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 >> v1:172.21.15.125:6789/0 conn(0x7f5390160920 legacy=0x7f53900cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5370003970 0x7f5370003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53700047c0 0x7f5370004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f537000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900c6be0 con 0x7f5370005180 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c0b0 con 0x7f5370003970 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c270 con 0x7f53700047c0 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f537000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538ea00640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5370003970 0x7f5370003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538e000640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53700047c0 0x7f5370004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 -- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53700047c0 msgr2=0x7f5370004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53700047c0 0x7f5370004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 -- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5370003970 msgr2=0x7f5370003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5370003970 0x7f5370003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.811+0000 7f538f400640 1 -- 172.21.15.125:0/2324545091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370003520 con 0x7f5370005180 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.812+0000 7f538f400640 1 -- 172.21.15.125:0/2324545091 reap_dead start 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.812+0000 7f538f400640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f537000b980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5388004480 tx=0x7f53700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86187642c7988bd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388006b50 con 0x7f5370005180 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388006cd0 con 0x7f5370005180 2024-01-16T14:59:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538d600640 1 -- 172.21.15.125:0/2324545091 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388007260 con 0x7f5370005180 2024-01-16T14:59:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 -- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 msgr2=0x7f537000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f537000b980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5388004480 tx=0x7f53700057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 -- 172.21.15.125:0/2324545091 shutdown_connections 2024-01-16T14:59:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 --2- 172.21.15.125:0/2324545091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f537000b980 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 -- 172.21.15.125:0/2324545091 >> 172.21.15.125:0/2324545091 conn(0x7f53900d3090 msgr2=0x7f53900d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 -- 172.21.15.125:0/2324545091 shutdown_connections 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.813+0000 7f538fe00640 1 -- 172.21.15.125:0/2324545091 wait complete. 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.815+0000 7f538fe00640 1 Processor -- start 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.815+0000 7f538fe00640 1 -- start start 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f539015baf0 0x7f53900afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53900b3470 0x7f53900ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900b3050 con 0x7f5370005180 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900b23d0 con 0x7f539015baf0 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.816+0000 7f538fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900b26b0 con 0x7f53900b3470 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33392/0 (socket says 172.21.15.125:33392) 2024-01-16T14:59:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 -- 172.21.15.125:0/1098759783 learned_addr learned my addr 172.21.15.125:0/1098759783 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538e000640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f539015baf0 0x7f53900afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 -- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53900b3470 msgr2=0x7f53900ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53900b3470 0x7f53900ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 -- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f539015baf0 msgr2=0x7f53900afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f539015baf0 0x7f53900afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 -- 172.21.15.125:0/1098759783 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900b2990 con 0x7f5370005180 2024-01-16T14:59:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.817+0000 7f538ea00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f538000e780 tx=0x7f538000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.818+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000ee90 con 0x7f5370005180 2024-01-16T14:59:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.818+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538000c870 con 0x7f5370005180 2024-01-16T14:59:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.818+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000cc00 con 0x7f5370005180 2024-01-16T14:59:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.818+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53900b2c70 con 0x7f5370005180 2024-01-16T14:59:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.818+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53900caeb0 con 0x7f5370005180 2024-01-16T14:59:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.819+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390067450 con 0x7f5370005180 2024-01-16T14:59:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.820+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f538001d460 con 0x7f5370005180 2024-01-16T14:59:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.821+0000 7f5386a00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 0x7f536806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.821+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5380089d40 con 0x7f5370005180 2024-01-16T14:59:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.821+0000 7f538e000640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 0x7f536806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.822+0000 7f538e000640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 0x7f536806d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f53900c9020 tx=0x7f5378008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:54.825+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53800549b0 con 0x7f5370005180 2024-01-16T14:59:55.059 INFO:tasks.ceph.mds.c.smithi125.stderr:2024-01-16T14:59:55.078+0000 7f9eb9a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:59:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.084+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5390067450 con 0x7f5370005180 2024-01-16T14:59:55.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.085+0000 7f5386a00640 1 -- 172.21.15.125:0/1098759783 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1097 (secure 0 0 0) 0x7f5380057fd0 con 0x7f5370005180 2024-01-16T14:59:55.066 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:55.066 INFO:teuthology.orchestra.run.smithi125.stdout:{"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":214,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1705416993,"num_in_osds":8,"osd_in_since":1705416984,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":53},{"state_name":"creating+peering","count":44}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":590984,"bytes_used":260673536,"bytes_avail":772833439744,"bytes_total":773094113280,"inactive_pgs_ratio":0.45360824465751648,"write_bytes_sec":1364,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":28,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5742}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-01-16T14:59:19.968596+0000","services":{}},"progress_events":{}} 2024-01-16T14:59:55.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 msgr2=0x7f536806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 0x7f536806d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f53900c9020 tx=0x7f5378008040 comp rx=0 tx=0).stop 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 msgr2=0x7f539014f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f538000e780 tx=0x7f538000ec40 comp rx=0 tx=0).stop 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 shutdown_connections 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53900b3470 0x7f53900ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f539015baf0 0x7f53900afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f536806b500 0x7f536806d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 --2- 172.21.15.125:0/1098759783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5370005180 0x7f539014f5f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 >> 172.21.15.125:0/1098759783 conn(0x7f53900d3090 msgr2=0x7f539006c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 shutdown_connections 2024-01-16T14:59:55.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.088+0000 7f538fe00640 1 -- 172.21.15.125:0/1098759783 wait complete. 2024-01-16T14:59:55.087 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T14:59:55.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:55.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.317+0000 7fd98f400640 1 Processor -- start 2024-01-16T14:59:55.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.317+0000 7fd98f400640 1 -- start start 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd9880cf9a0 0x7fd988070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98f400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd988071690 con 0x7fd988070ea0 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98f400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd988071970 con 0x7fd98814b2c0 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98f400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98814b6a0 con 0x7fd9880cf9a0 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98d400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd988070ea0 0x7fd9880cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52862/0 (socket says 172.21.15.125:52862) 2024-01-16T14:59:55.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd98d400640 1 -- 172.21.15.125:0/2356914049 learned_addr learned my addr 172.21.15.125:0/2356914049 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:55.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd97fe00640 1 --2- 172.21.15.125:0/2356914049 >> v2:172.21.15.137:3300/0 conn(0x7fd9880cf9a0 0x7fd988070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.319+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457450996 0 0) 0x7fd988071690 con 0x7fd988070ea0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968003520 con 0x7fd988070ea0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137223660 0 0) 0x7fd968003520 con 0x7fd988070ea0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 >> v2:172.21.15.137:3300/0 conn(0x7fd9880cf9a0 msgr2=0x7fd988070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 --2- 172.21.15.125:0/2356914049 >> v2:172.21.15.137:3300/0 conn(0x7fd9880cf9a0 0x7fd988070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 >> v1:172.21.15.137:6789/0 conn(0x7fd98814b2c0 legacy=0x7fd9880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988071c50 con 0x7fd988070ea0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd9800019f0 con 0x7fd988070ea0 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 >> v1:172.21.15.125:6789/0 conn(0x7fd988070ea0 legacy=0x7fd9880cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd968003970 0x7fd968003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 0x7fd968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd968005180 0x7fd96800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98814b6a0 con 0x7fd9680047c0 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96800c0d0 con 0x7fd968005180 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96800c290 con 0x7fd968003970 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.320+0000 7fd97fe00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 0x7fd968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd98ca00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd968003970 0x7fd968003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd98d400640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd968005180 0x7fd96800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd97fe00640 1 -- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd968003970 msgr2=0x7fd968003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd97fe00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd968003970 0x7fd968003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd97fe00640 1 -- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd968005180 msgr2=0x7fd96800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd97fe00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd968005180 0x7fd96800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd97fe00640 1 -- 172.21.15.125:0/2356914049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd968003520 con 0x7fd9680047c0 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd98d400640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd968005180 0x7fd96800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:55.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.321+0000 7fd98d400640 1 -- 172.21.15.125:0/2356914049 reap_dead start 2024-01-16T14:59:55.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.322+0000 7fd97fe00640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 0x7fd968004c40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd97000ba30 tx=0x7fd9680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03f86fed476c4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.322+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970032640 con 0x7fd9680047c0 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.322+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700327a0 con 0x7fd9680047c0 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.322+0000 7fd987e00640 1 -- 172.21.15.125:0/2356914049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970032b30 con 0x7fd9680047c0 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 -- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 msgr2=0x7fd968004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 0x7fd968004c40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd97000ba30 tx=0x7fd9680057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 -- 172.21.15.125:0/2356914049 shutdown_connections 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 --2- 172.21.15.125:0/2356914049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9680047c0 0x7fd968004c40 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 -- 172.21.15.125:0/2356914049 >> 172.21.15.125:0/2356914049 conn(0x7fd98806a800 msgr2=0x7fd98806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 -- 172.21.15.125:0/2356914049 shutdown_connections 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.323+0000 7fd98f400640 1 -- 172.21.15.125:0/2356914049 wait complete. 2024-01-16T14:59:55.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.325+0000 7fd98f400640 1 Processor -- start 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.325+0000 7fd98f400640 1 -- start start 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9680047c0 0x7fd9880ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9880cd740 0x7fd9880cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880ce130 con 0x7fd988155470 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880aba20 con 0x7fd9880cd740 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881fafc0 con 0x7fd9680047c0 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33406/0 (socket says 172.21.15.125:33406) 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 -- 172.21.15.125:0/3236379450 learned_addr learned my addr 172.21.15.125:0/3236379450 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd98ca00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9680047c0 0x7fd9880ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 -- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9680047c0 msgr2=0x7fd9880ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9680047c0 0x7fd9880ab4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 -- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9880cd740 msgr2=0x7fd9880cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9880cd740 0x7fd9880cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.326+0000 7fd97fe00640 1 -- 172.21.15.125:0/3236379450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9881fb260 con 0x7fd988155470 2024-01-16T14:59:55.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd97fe00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd97002bec0 tx=0x7fd970004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97003baf0 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97003bc50 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970030410 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9881fb540 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.327+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9881fbb00 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.328+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9881fbc90 con 0x7fd988155470 2024-01-16T14:59:55.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.329+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd970030570 con 0x7fd988155470 2024-01-16T14:59:55.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.330+0000 7fd985600640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 0x7fd96006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.330+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9700ac680 con 0x7fd988155470 2024-01-16T14:59:55.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.330+0000 7fd98ca00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 0x7fd96006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.331+0000 7fd98ca00640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 0x7fd96006d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd978007fd0 tx=0x7fd978009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.335+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd970076fe0 con 0x7fd988155470 2024-01-16T14:59:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.521+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9880cbc40 con 0x7fd96006b500 2024-01-16T14:59:55.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.543+0000 7fd985600640 1 -- 172.21.15.125:0/3236379450 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283891 (secure 0 0 0) 0x7fd97800d050 con 0x7fd96006b500 2024-01-16T14:59:55.527 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:55.529 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:59:55.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.547+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 msgr2=0x7fd96006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.547+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 0x7fd96006d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd978007fd0 tx=0x7fd978009040 comp rx=0 tx=0).stop 2024-01-16T14:59:55.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.547+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 msgr2=0x7fd9880a8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.547+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd97002bec0 tx=0x7fd970004730 comp rx=0 tx=0).stop 2024-01-16T14:59:55.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 shutdown_connections 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9880cd740 0x7fd9880cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd96006b500 0x7fd96006d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd988155470 0x7fd9880a8500 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 --2- 172.21.15.125:0/3236379450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9680047c0 0x7fd9880ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 >> 172.21.15.125:0/3236379450 conn(0x7fd98806a800 msgr2=0x7fd9880b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 shutdown_connections 2024-01-16T14:59:55.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:55.548+0000 7fd98f400640 1 -- 172.21.15.125:0/3236379450 wait complete. 2024-01-16T14:59:55.546 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-16T14:59:53.979363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590984,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":173,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254564,"kb_used_data":5092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720156,"statfs":{"total":773094113280,"available":772833439744,"internally_reserved":0,"allocated":5214208,"data_stored":2797205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116216+0000","last_change":"2024-01-16T14:59:48.116216+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116216+0000","last_undegraded":"2024-01-16T14:59:48.116216+0000","last_fullsized":"2024-01-16T14:59:48.116216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.122243+0000","last_change":"2024-01-16T14:59:48.122243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.122243+0000","last_undegraded":"2024-01-16T14:59:48.122243+0000","last_fullsized":"2024-01-16T14:59:48.122243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157265+0000","last_change":"2024-01-16T14:59:49.157265+0000","last_active":"2024-01-16T14:59:49.157265+0000","last_peered":"2024-01-16T14:59:49.157265+0000","last_clean":"2024-01-16T14:59:49.157265+0000","last_became_active":"2024-01-16T14:59:49.156985+0000","last_became_peered":"2024-01-16T14:59:49.156985+0000","last_unstale":"2024-01-16T14:59:49.157265+0000","last_undegraded":"2024-01-16T14:59:49.157265+0000","last_fullsized":"2024-01-16T14:59:49.157265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.118153+0000","last_change":"2024-01-16T14:59:48.118153+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.118153+0000","last_undegraded":"2024-01-16T14:59:48.118153+0000","last_fullsized":"2024-01-16T14:59:48.118153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.121232+0000","last_change":"2024-01-16T14:59:48.121232+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.121232+0000","last_undegraded":"2024-01-16T14:59:48.121232+0000","last_fullsized":"2024-01-16T14:59:48.121232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164334+0000","last_change":"2024-01-16T14:59:49.160920+0000","last_active":"2024-01-16T14:59:49.164334+0000","last_peered":"2024-01-16T14:59:49.164334+0000","last_clean":"2024-01-16T14:59:49.164334+0000","last_became_active":"2024-01-16T14:59:49.158310+0000","last_became_peered":"2024-01-16T14:59:49.158310+0000","last_unstale":"2024-01-16T14:59:49.164334+0000","last_undegraded":"2024-01-16T14:59:49.164334+0000","last_fullsized":"2024-01-16T14:59:49.164334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.670502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116927+0000","last_change":"2024-01-16T14:59:48.116927+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116927+0000","last_undegraded":"2024-01-16T14:59:48.116927+0000","last_fullsized":"2024-01-16T14:59:48.116927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116447+0000","last_change":"2024-01-16T14:59:48.116447+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116447+0000","last_undegraded":"2024-01-16T14:59:48.116447+0000","last_fullsized":"2024-01-16T14:59:48.116447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114791+0000","last_change":"2024-01-16T14:59:48.114791+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114791+0000","last_undegraded":"2024-01-16T14:59:48.114791+0000","last_fullsized":"2024-01-16T14:59:48.114791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116913+0000","last_change":"2024-01-16T14:59:48.116913+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116913+0000","last_undegraded":"2024-01-16T14:59:48.116913+0000","last_fullsized":"2024-01-16T14:59:48.116913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115973+0000","last_change":"2024-01-16T14:59:48.115973+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115973+0000","last_undegraded":"2024-01-16T14:59:48.115973+0000","last_fullsized":"2024-01-16T14:59:48.115973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124245+0000","last_change":"2024-01-16T14:59:48.124245+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124245+0000","last_undegraded":"2024-01-16T14:59:48.124245+0000","last_fullsized":"2024-01-16T14:59:48.124245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114997+0000","last_change":"2024-01-16T14:59:48.114997+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114997+0000","last_undegraded":"2024-01-16T14:59:48.114997+0000","last_fullsized":"2024-01-16T14:59:48.114997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114309+0000","last_change":"2024-01-16T14:59:48.114309+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114309+0000","last_undegraded":"2024-01-16T14:59:48.114309+0000","last_fullsized":"2024-01-16T14:59:48.114309+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163054+0000","last_change":"2024-01-16T14:59:49.158516+0000","last_active":"2024-01-16T14:59:49.163054+0000","last_peered":"2024-01-16T14:59:49.163054+0000","last_clean":"2024-01-16T14:59:49.163054+0000","last_became_active":"2024-01-16T14:59:49.158217+0000","last_became_peered":"2024-01-16T14:59:49.158217+0000","last_unstale":"2024-01-16T14:59:49.163054+0000","last_undegraded":"2024-01-16T14:59:49.163054+0000","last_fullsized":"2024-01-16T14:59:49.163054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.124125+0000","last_change":"2024-01-16T14:59:48.124125+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.124125+0000","last_undegraded":"2024-01-16T14:59:48.124125+0000","last_fullsized":"2024-01-16T14:59:48.124125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.162728+0000","last_change":"2024-01-16T14:59:49.162728+0000","last_active":"2024-01-16T14:59:49.162728+0000","last_peered":"2024-01-16T14:59:49.162728+0000","last_clean":"2024-01-16T14:59:49.162728+0000","last_became_active":"2024-01-16T14:59:49.162473+0000","last_became_peered":"2024-01-16T14:59:49.162473+0000","last_unstale":"2024-01-16T14:59:49.162728+0000","last_undegraded":"2024-01-16T14:59:49.162728+0000","last_fullsized":"2024-01-16T14:59:49.162728+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:29.139287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159712+0000","last_change":"2024-01-16T14:59:49.146953+0000","last_active":"2024-01-16T14:59:49.159712+0000","last_peered":"2024-01-16T14:59:49.159712+0000","last_clean":"2024-01-16T14:59:49.159712+0000","last_became_active":"2024-01-16T14:59:49.146074+0000","last_became_peered":"2024-01-16T14:59:49.146074+0000","last_unstale":"2024-01-16T14:59:49.159712+0000","last_undegraded":"2024-01-16T14:59:49.159712+0000","last_fullsized":"2024-01-16T14:59:49.159712+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:06.294703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126633+0000","last_change":"2024-01-16T14:59:48.126633+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126633+0000","last_undegraded":"2024-01-16T14:59:48.126633+0000","last_fullsized":"2024-01-16T14:59:48.126633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115055+0000","last_change":"2024-01-16T14:59:48.115055+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115055+0000","last_undegraded":"2024-01-16T14:59:48.115055+0000","last_fullsized":"2024-01-16T14:59:48.115055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.163749+0000","last_change":"2024-01-16T14:59:49.158956+0000","last_active":"2024-01-16T14:59:49.163749+0000","last_peered":"2024-01-16T14:59:49.163749+0000","last_clean":"2024-01-16T14:59:49.163749+0000","last_became_active":"2024-01-16T14:59:49.158665+0000","last_became_peered":"2024-01-16T14:59:49.158665+0000","last_unstale":"2024-01-16T14:59:49.163749+0000","last_undegraded":"2024-01-16T14:59:49.163749+0000","last_fullsized":"2024-01-16T14:59:49.163749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112923+0000","last_change":"2024-01-16T14:59:48.112923+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112923+0000","last_undegraded":"2024-01-16T14:59:48.112923+0000","last_fullsized":"2024-01-16T14:59:48.112923+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131707+0000","last_change":"2024-01-16T14:59:48.131707+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131707+0000","last_undegraded":"2024-01-16T14:59:48.131707+0000","last_fullsized":"2024-01-16T14:59:48.131707+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123440+0000","last_change":"2024-01-16T14:59:48.123440+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123440+0000","last_undegraded":"2024-01-16T14:59:48.123440+0000","last_fullsized":"2024-01-16T14:59:48.123440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.131918+0000","last_change":"2024-01-16T14:59:48.131918+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.131918+0000","last_undegraded":"2024-01-16T14:59:48.131918+0000","last_fullsized":"2024-01-16T14:59:48.131918+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113430+0000","last_change":"2024-01-16T14:59:48.113430+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113430+0000","last_undegraded":"2024-01-16T14:59:48.113430+0000","last_fullsized":"2024-01-16T14:59:48.113430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164250+0000","last_change":"2024-01-16T14:59:49.164250+0000","last_active":"2024-01-16T14:59:49.164250+0000","last_peered":"2024-01-16T14:59:49.164250+0000","last_clean":"2024-01-16T14:59:49.164250+0000","last_became_active":"2024-01-16T14:59:49.163931+0000","last_became_peered":"2024-01-16T14:59:49.163931+0000","last_unstale":"2024-01-16T14:59:49.164250+0000","last_undegraded":"2024-01-16T14:59:49.164250+0000","last_fullsized":"2024-01-16T14:59:49.164250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:48.014735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123754+0000","last_change":"2024-01-16T14:59:48.123754+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123754+0000","last_undegraded":"2024-01-16T14:59:48.123754+0000","last_fullsized":"2024-01-16T14:59:48.123754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.126693+0000","last_change":"2024-01-16T14:59:48.126693+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.126693+0000","last_undegraded":"2024-01-16T14:59:48.126693+0000","last_fullsized":"2024-01-16T14:59:48.126693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160966+0000","last_change":"2024-01-16T14:59:49.160966+0000","last_active":"2024-01-16T14:59:49.160966+0000","last_peered":"2024-01-16T14:59:49.160966+0000","last_clean":"2024-01-16T14:59:49.160966+0000","last_became_active":"2024-01-16T14:59:49.160612+0000","last_became_peered":"2024-01-16T14:59:49.160612+0000","last_unstale":"2024-01-16T14:59:49.160966+0000","last_undegraded":"2024-01-16T14:59:49.160966+0000","last_fullsized":"2024-01-16T14:59:49.160966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164208+0000","last_change":"2024-01-16T14:59:49.159653+0000","last_active":"2024-01-16T14:59:49.164208+0000","last_peered":"2024-01-16T14:59:49.164208+0000","last_clean":"2024-01-16T14:59:49.164208+0000","last_became_active":"2024-01-16T14:59:49.159375+0000","last_became_peered":"2024-01-16T14:59:49.159375+0000","last_unstale":"2024-01-16T14:59:49.164208+0000","last_undegraded":"2024-01-16T14:59:49.164208+0000","last_fullsized":"2024-01-16T14:59:49.164208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112726+0000","last_change":"2024-01-16T14:59:48.112726+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112726+0000","last_undegraded":"2024-01-16T14:59:48.112726+0000","last_fullsized":"2024-01-16T14:59:48.112726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165746+0000","last_change":"2024-01-16T14:59:49.156116+0000","last_active":"2024-01-16T14:59:49.165746+0000","last_peered":"2024-01-16T14:59:49.165746+0000","last_clean":"2024-01-16T14:59:49.165746+0000","last_became_active":"2024-01-16T14:59:49.155198+0000","last_became_peered":"2024-01-16T14:59:49.155198+0000","last_unstale":"2024-01-16T14:59:49.165746+0000","last_undegraded":"2024-01-16T14:59:49.165746+0000","last_fullsized":"2024-01-16T14:59:49.165746+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:12:21.166819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169542+0000","last_change":"2024-01-16T14:59:48.137330+0000","last_active":"2024-01-16T14:59:49.169542+0000","last_peered":"2024-01-16T14:59:49.169542+0000","last_clean":"2024-01-16T14:59:49.169542+0000","last_became_active":"2024-01-16T14:59:48.136959+0000","last_became_peered":"2024-01-16T14:59:48.136959+0000","last_unstale":"2024-01-16T14:59:49.169542+0000","last_undegraded":"2024-01-16T14:59:49.169542+0000","last_fullsized":"2024-01-16T14:59:49.169542+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:39.353399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.130678+0000","last_change":"2024-01-16T14:59:48.130678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.130678+0000","last_undegraded":"2024-01-16T14:59:48.130678+0000","last_fullsized":"2024-01-16T14:59:48.130678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.160262+0000","last_change":"2024-01-16T14:59:48.137753+0000","last_active":"2024-01-16T14:59:49.160262+0000","last_peered":"2024-01-16T14:59:49.160262+0000","last_clean":"2024-01-16T14:59:49.160262+0000","last_became_active":"2024-01-16T14:59:48.137414+0000","last_became_peered":"2024-01-16T14:59:48.137414+0000","last_unstale":"2024-01-16T14:59:49.160262+0000","last_undegraded":"2024-01-16T14:59:49.160262+0000","last_fullsized":"2024-01-16T14:59:49.160262+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.123243+0000","last_change":"2024-01-16T14:59:48.123243+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.123243+0000","last_undegraded":"2024-01-16T14:59:48.123243+0000","last_fullsized":"2024-01-16T14:59:48.123243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159457+0000","last_change":"2024-01-16T14:59:48.134911+0000","last_active":"2024-01-16T14:59:49.159457+0000","last_peered":"2024-01-16T14:59:49.159457+0000","last_clean":"2024-01-16T14:59:49.159457+0000","last_became_active":"2024-01-16T14:59:48.134259+0000","last_became_peered":"2024-01-16T14:59:48.134259+0000","last_unstale":"2024-01-16T14:59:49.159457+0000","last_undegraded":"2024-01-16T14:59:49.159457+0000","last_fullsized":"2024-01-16T14:59:49.159457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:28.124634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167122+0000","last_change":"2024-01-16T14:59:49.165223+0000","last_active":"2024-01-16T14:59:49.167122+0000","last_peered":"2024-01-16T14:59:49.167122+0000","last_clean":"2024-01-16T14:59:49.167122+0000","last_became_active":"2024-01-16T14:59:49.164935+0000","last_became_peered":"2024-01-16T14:59:49.164935+0000","last_unstale":"2024-01-16T14:59:49.167122+0000","last_undegraded":"2024-01-16T14:59:49.167122+0000","last_fullsized":"2024-01-16T14:59:49.167122+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.134046+0000","last_change":"2024-01-16T14:59:48.131434+0000","last_active":"2024-01-16T14:59:48.134046+0000","last_peered":"2024-01-16T14:59:48.134046+0000","last_clean":"2024-01-16T14:59:48.134046+0000","last_became_active":"2024-01-16T14:59:48.131165+0000","last_became_peered":"2024-01-16T14:59:48.131165+0000","last_unstale":"2024-01-16T14:59:48.134046+0000","last_undegraded":"2024-01-16T14:59:48.134046+0000","last_fullsized":"2024-01-16T14:59:48.134046+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159531+0000","last_change":"2024-01-16T14:59:49.159531+0000","last_active":"2024-01-16T14:59:49.159531+0000","last_peered":"2024-01-16T14:59:49.159531+0000","last_clean":"2024-01-16T14:59:49.159531+0000","last_became_active":"2024-01-16T14:59:49.159274+0000","last_became_peered":"2024-01-16T14:59:49.159274+0000","last_unstale":"2024-01-16T14:59:49.159531+0000","last_undegraded":"2024-01-16T14:59:49.159531+0000","last_fullsized":"2024-01-16T14:59:49.159531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168278+0000","last_change":"2024-01-16T14:59:48.128798+0000","last_active":"2024-01-16T14:59:49.168278+0000","last_peered":"2024-01-16T14:59:49.168278+0000","last_clean":"2024-01-16T14:59:49.168278+0000","last_became_active":"2024-01-16T14:59:48.128328+0000","last_became_peered":"2024-01-16T14:59:48.128328+0000","last_unstale":"2024-01-16T14:59:49.168278+0000","last_undegraded":"2024-01-16T14:59:49.168278+0000","last_fullsized":"2024-01-16T14:59:49.168278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:54.681593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113310+0000","last_change":"2024-01-16T14:59:48.113310+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113310+0000","last_undegraded":"2024-01-16T14:59:48.113310+0000","last_fullsized":"2024-01-16T14:59:48.113310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.161777+0000","last_change":"2024-01-16T14:59:48.131571+0000","last_active":"2024-01-16T14:59:49.161777+0000","last_peered":"2024-01-16T14:59:49.161777+0000","last_clean":"2024-01-16T14:59:49.161777+0000","last_became_active":"2024-01-16T14:59:48.131245+0000","last_became_peered":"2024-01-16T14:59:48.131245+0000","last_unstale":"2024-01-16T14:59:49.161777+0000","last_undegraded":"2024-01-16T14:59:49.161777+0000","last_fullsized":"2024-01-16T14:59:49.161777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.113674+0000","last_change":"2024-01-16T14:59:48.113674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.113674+0000","last_undegraded":"2024-01-16T14:59:48.113674+0000","last_fullsized":"2024-01-16T14:59:48.113674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.133471+0000","last_change":"2024-01-16T14:59:48.130388+0000","last_active":"2024-01-16T14:59:48.133471+0000","last_peered":"2024-01-16T14:59:48.133471+0000","last_clean":"2024-01-16T14:59:48.133471+0000","last_became_active":"2024-01-16T14:59:48.129990+0000","last_became_peered":"2024-01-16T14:59:48.129990+0000","last_unstale":"2024-01-16T14:59:48.133471+0000","last_undegraded":"2024-01-16T14:59:48.133471+0000","last_fullsized":"2024-01-16T14:59:48.133471+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:33.461419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.132738+0000","last_change":"2024-01-16T14:59:48.132738+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.132738+0000","last_undegraded":"2024-01-16T14:59:48.132738+0000","last_fullsized":"2024-01-16T14:59:48.132738+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.128590+0000","last_change":"2024-01-16T14:59:48.122577+0000","last_active":"2024-01-16T14:59:48.128590+0000","last_peered":"2024-01-16T14:59:48.128590+0000","last_clean":"2024-01-16T14:59:48.128590+0000","last_became_active":"2024-01-16T14:59:48.122280+0000","last_became_peered":"2024-01-16T14:59:48.122280+0000","last_unstale":"2024-01-16T14:59:48.128590+0000","last_undegraded":"2024-01-16T14:59:48.128590+0000","last_fullsized":"2024-01-16T14:59:48.128590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:53.539387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.159679+0000","last_change":"2024-01-16T14:59:49.146563+0000","last_active":"2024-01-16T14:59:49.159679+0000","last_peered":"2024-01-16T14:59:49.159679+0000","last_clean":"2024-01-16T14:59:49.159679+0000","last_became_active":"2024-01-16T14:59:49.145334+0000","last_became_peered":"2024-01-16T14:59:49.145334+0000","last_unstale":"2024-01-16T14:59:49.159679+0000","last_undegraded":"2024-01-16T14:59:49.159679+0000","last_fullsized":"2024-01-16T14:59:49.159679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:13.595065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.140096+0000","last_change":"2024-01-16T14:59:48.127440+0000","last_active":"2024-01-16T14:59:49.140096+0000","last_peered":"2024-01-16T14:59:49.140096+0000","last_clean":"2024-01-16T14:59:49.140096+0000","last_became_active":"2024-01-16T14:59:48.126114+0000","last_became_peered":"2024-01-16T14:59:48.126114+0000","last_unstale":"2024-01-16T14:59:49.140096+0000","last_undegraded":"2024-01-16T14:59:49.140096+0000","last_fullsized":"2024-01-16T14:59:49.140096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114330+0000","last_change":"2024-01-16T14:59:48.114330+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114330+0000","last_undegraded":"2024-01-16T14:59:48.114330+0000","last_fullsized":"2024-01-16T14:59:48.114330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136234+0000","last_change":"2024-01-16T14:59:48.128532+0000","last_active":"2024-01-16T14:59:48.136234+0000","last_peered":"2024-01-16T14:59:48.136234+0000","last_clean":"2024-01-16T14:59:48.136234+0000","last_became_active":"2024-01-16T14:59:48.127045+0000","last_became_peered":"2024-01-16T14:59:48.127045+0000","last_unstale":"2024-01-16T14:59:48.136234+0000","last_undegraded":"2024-01-16T14:59:48.136234+0000","last_fullsized":"2024-01-16T14:59:48.136234+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:12.546318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165687+0000","last_change":"2024-01-16T14:59:49.157462+0000","last_active":"2024-01-16T14:59:49.165687+0000","last_peered":"2024-01-16T14:59:49.165687+0000","last_clean":"2024-01-16T14:59:49.165687+0000","last_became_active":"2024-01-16T14:59:49.157150+0000","last_became_peered":"2024-01-16T14:59:49.157150+0000","last_unstale":"2024-01-16T14:59:49.165687+0000","last_undegraded":"2024-01-16T14:59:49.165687+0000","last_fullsized":"2024-01-16T14:59:49.165687+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136759+0000","last_change":"2024-01-16T14:59:48.135944+0000","last_active":"2024-01-16T14:59:48.136759+0000","last_peered":"2024-01-16T14:59:48.136759+0000","last_clean":"2024-01-16T14:59:48.136759+0000","last_became_active":"2024-01-16T14:59:48.135643+0000","last_became_peered":"2024-01-16T14:59:48.135643+0000","last_unstale":"2024-01-16T14:59:48.136759+0000","last_undegraded":"2024-01-16T14:59:48.136759+0000","last_fullsized":"2024-01-16T14:59:48.136759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:39.871898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.115155+0000","last_change":"2024-01-16T14:59:48.115155+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.115155+0000","last_undegraded":"2024-01-16T14:59:48.115155+0000","last_fullsized":"2024-01-16T14:59:48.115155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132942+0000","last_change":"2024-01-16T14:59:48.129597+0000","last_active":"2024-01-16T14:59:48.132942+0000","last_peered":"2024-01-16T14:59:48.132942+0000","last_clean":"2024-01-16T14:59:48.132942+0000","last_became_active":"2024-01-16T14:59:48.128905+0000","last_became_peered":"2024-01-16T14:59:48.128905+0000","last_unstale":"2024-01-16T14:59:48.132942+0000","last_undegraded":"2024-01-16T14:59:48.132942+0000","last_fullsized":"2024-01-16T14:59:48.132942+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:15.588352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.125716+0000","last_change":"2024-01-16T14:59:48.125716+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.125716+0000","last_undegraded":"2024-01-16T14:59:48.125716+0000","last_fullsized":"2024-01-16T14:59:48.125716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.169439+0000","last_change":"2024-01-16T14:59:48.128640+0000","last_active":"2024-01-16T14:59:49.169439+0000","last_peered":"2024-01-16T14:59:49.169439+0000","last_clean":"2024-01-16T14:59:49.169439+0000","last_became_active":"2024-01-16T14:59:48.128032+0000","last_became_peered":"2024-01-16T14:59:48.128032+0000","last_unstale":"2024-01-16T14:59:49.169439+0000","last_undegraded":"2024-01-16T14:59:49.169439+0000","last_fullsized":"2024-01-16T14:59:49.169439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:41.977214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112452+0000","last_change":"2024-01-16T14:59:48.112452+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112452+0000","last_undegraded":"2024-01-16T14:59:48.112452+0000","last_fullsized":"2024-01-16T14:59:48.112452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.130910+0000","last_change":"2024-01-16T14:59:48.126828+0000","last_active":"2024-01-16T14:59:48.130910+0000","last_peered":"2024-01-16T14:59:48.130910+0000","last_clean":"2024-01-16T14:59:48.130910+0000","last_became_active":"2024-01-16T14:59:48.126513+0000","last_became_peered":"2024-01-16T14:59:48.126513+0000","last_unstale":"2024-01-16T14:59:48.130910+0000","last_undegraded":"2024-01-16T14:59:48.130910+0000","last_fullsized":"2024-01-16T14:59:48.130910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153727+0000","last_change":"2024-01-16T14:59:49.150175+0000","last_active":"2024-01-16T14:59:49.153727+0000","last_peered":"2024-01-16T14:59:49.153727+0000","last_clean":"2024-01-16T14:59:49.153727+0000","last_became_active":"2024-01-16T14:59:49.149869+0000","last_became_peered":"2024-01-16T14:59:49.149869+0000","last_unstale":"2024-01-16T14:59:49.153727+0000","last_undegraded":"2024-01-16T14:59:49.153727+0000","last_fullsized":"2024-01-16T14:59:49.153727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138602+0000","last_change":"2024-01-16T14:59:48.129480+0000","last_active":"2024-01-16T14:59:49.138602+0000","last_peered":"2024-01-16T14:59:49.138602+0000","last_clean":"2024-01-16T14:59:49.138602+0000","last_became_active":"2024-01-16T14:59:48.129231+0000","last_became_peered":"2024-01-16T14:59:48.129231+0000","last_unstale":"2024-01-16T14:59:49.138602+0000","last_undegraded":"2024-01-16T14:59:49.138602+0000","last_fullsized":"2024-01-16T14:59:49.138602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114361+0000","last_change":"2024-01-16T14:59:48.114361+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114361+0000","last_undegraded":"2024-01-16T14:59:48.114361+0000","last_fullsized":"2024-01-16T14:59:48.114361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.137414+0000","last_change":"2024-01-16T14:59:48.133029+0000","last_active":"2024-01-16T14:59:48.137414+0000","last_peered":"2024-01-16T14:59:48.137414+0000","last_clean":"2024-01-16T14:59:48.137414+0000","last_became_active":"2024-01-16T14:59:48.132732+0000","last_became_peered":"2024-01-16T14:59:48.132732+0000","last_unstale":"2024-01-16T14:59:48.137414+0000","last_undegraded":"2024-01-16T14:59:48.137414+0000","last_fullsized":"2024-01-16T14:59:48.137414+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:36.691574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112029+0000","last_change":"2024-01-16T14:59:48.112029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112029+0000","last_undegraded":"2024-01-16T14:59:48.112029+0000","last_fullsized":"2024-01-16T14:59:48.112029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132892+0000","last_change":"2024-01-16T14:59:48.129568+0000","last_active":"2024-01-16T14:59:48.132892+0000","last_peered":"2024-01-16T14:59:48.132892+0000","last_clean":"2024-01-16T14:59:48.132892+0000","last_became_active":"2024-01-16T14:59:48.129222+0000","last_became_peered":"2024-01-16T14:59:48.129222+0000","last_unstale":"2024-01-16T14:59:48.132892+0000","last_undegraded":"2024-01-16T14:59:48.132892+0000","last_fullsized":"2024-01-16T14:59:48.132892+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:51.864931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114427+0000","last_change":"2024-01-16T14:59:48.114427+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114427+0000","last_undegraded":"2024-01-16T14:59:48.114427+0000","last_fullsized":"2024-01-16T14:59:48.114427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.166154+0000","last_change":"2024-01-16T14:59:48.128976+0000","last_active":"2024-01-16T14:59:49.166154+0000","last_peered":"2024-01-16T14:59:49.166154+0000","last_clean":"2024-01-16T14:59:49.166154+0000","last_became_active":"2024-01-16T14:59:48.128687+0000","last_became_peered":"2024-01-16T14:59:48.128687+0000","last_unstale":"2024-01-16T14:59:49.166154+0000","last_undegraded":"2024-01-16T14:59:49.166154+0000","last_fullsized":"2024-01-16T14:59:49.166154+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.129867+0000","last_change":"2024-01-16T14:59:48.129867+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.129867+0000","last_undegraded":"2024-01-16T14:59:48.129867+0000","last_fullsized":"2024-01-16T14:59:48.129867+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.138620+0000","last_change":"2024-01-16T14:59:48.134109+0000","last_active":"2024-01-16T14:59:48.138620+0000","last_peered":"2024-01-16T14:59:48.138620+0000","last_clean":"2024-01-16T14:59:48.138620+0000","last_became_active":"2024-01-16T14:59:48.133228+0000","last_became_peered":"2024-01-16T14:59:48.133228+0000","last_unstale":"2024-01-16T14:59:48.138620+0000","last_undegraded":"2024-01-16T14:59:48.138620+0000","last_fullsized":"2024-01-16T14:59:48.138620+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127674+0000","last_change":"2024-01-16T14:59:48.127674+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127674+0000","last_undegraded":"2024-01-16T14:59:48.127674+0000","last_fullsized":"2024-01-16T14:59:48.127674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142152+0000","last_change":"2024-01-16T14:59:48.137888+0000","last_active":"2024-01-16T14:59:49.142152+0000","last_peered":"2024-01-16T14:59:49.142152+0000","last_clean":"2024-01-16T14:59:49.142152+0000","last_became_active":"2024-01-16T14:59:48.137587+0000","last_became_peered":"2024-01-16T14:59:48.137587+0000","last_unstale":"2024-01-16T14:59:49.142152+0000","last_undegraded":"2024-01-16T14:59:49.142152+0000","last_fullsized":"2024-01-16T14:59:49.142152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:53.615671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152916+0000","last_change":"2024-01-16T14:59:49.149041+0000","last_active":"2024-01-16T14:59:49.152916+0000","last_peered":"2024-01-16T14:59:49.152916+0000","last_clean":"2024-01-16T14:59:49.152916+0000","last_became_active":"2024-01-16T14:59:49.148799+0000","last_became_peered":"2024-01-16T14:59:49.148799+0000","last_unstale":"2024-01-16T14:59:49.152916+0000","last_undegraded":"2024-01-16T14:59:49.152916+0000","last_fullsized":"2024-01-16T14:59:49.152916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.168417+0000","last_change":"2024-01-16T14:59:48.127494+0000","last_active":"2024-01-16T14:59:49.168417+0000","last_peered":"2024-01-16T14:59:49.168417+0000","last_clean":"2024-01-16T14:59:49.168417+0000","last_became_active":"2024-01-16T14:59:48.127232+0000","last_became_peered":"2024-01-16T14:59:48.127232+0000","last_unstale":"2024-01-16T14:59:49.168417+0000","last_undegraded":"2024-01-16T14:59:49.168417+0000","last_fullsized":"2024-01-16T14:59:49.168417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165904+0000","last_change":"2024-01-16T14:59:49.162106+0000","last_active":"2024-01-16T14:59:49.165904+0000","last_peered":"2024-01-16T14:59:49.165904+0000","last_clean":"2024-01-16T14:59:49.165904+0000","last_became_active":"2024-01-16T14:59:49.161715+0000","last_became_peered":"2024-01-16T14:59:49.161715+0000","last_unstale":"2024-01-16T14:59:49.165904+0000","last_undegraded":"2024-01-16T14:59:49.165904+0000","last_fullsized":"2024-01-16T14:59:49.165904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.140520+0000","last_change":"2024-01-16T14:59:48.139613+0000","last_active":"2024-01-16T14:59:48.140520+0000","last_peered":"2024-01-16T14:59:48.140520+0000","last_clean":"2024-01-16T14:59:48.140520+0000","last_became_active":"2024-01-16T14:59:48.139289+0000","last_became_peered":"2024-01-16T14:59:48.139289+0000","last_unstale":"2024-01-16T14:59:48.140520+0000","last_undegraded":"2024-01-16T14:59:48.140520+0000","last_fullsized":"2024-01-16T14:59:48.140520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112106+0000","last_change":"2024-01-16T14:59:48.112106+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112106+0000","last_undegraded":"2024-01-16T14:59:48.112106+0000","last_fullsized":"2024-01-16T14:59:48.112106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132124+0000","last_change":"2024-01-16T14:59:48.127338+0000","last_active":"2024-01-16T14:59:48.132124+0000","last_peered":"2024-01-16T14:59:48.132124+0000","last_clean":"2024-01-16T14:59:48.132124+0000","last_became_active":"2024-01-16T14:59:48.127065+0000","last_became_peered":"2024-01-16T14:59:48.127065+0000","last_unstale":"2024-01-16T14:59:48.132124+0000","last_undegraded":"2024-01-16T14:59:48.132124+0000","last_fullsized":"2024-01-16T14:59:48.132124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:20.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.112846+0000","last_change":"2024-01-16T14:59:48.112846+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.112846+0000","last_undegraded":"2024-01-16T14:59:48.112846+0000","last_fullsized":"2024-01-16T14:59:48.112846+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.132404+0000","last_change":"2024-01-16T14:59:48.126842+0000","last_active":"2024-01-16T14:59:48.132404+0000","last_peered":"2024-01-16T14:59:48.132404+0000","last_clean":"2024-01-16T14:59:48.132404+0000","last_became_active":"2024-01-16T14:59:48.126505+0000","last_became_peered":"2024-01-16T14:59:48.126505+0000","last_unstale":"2024-01-16T14:59:48.132404+0000","last_undegraded":"2024-01-16T14:59:48.132404+0000","last_fullsized":"2024-01-16T14:59:48.132404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:30.105655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.127029+0000","last_change":"2024-01-16T14:59:48.127029+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.127029+0000","last_undegraded":"2024-01-16T14:59:48.127029+0000","last_fullsized":"2024-01-16T14:59:48.127029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.141674+0000","last_change":"2024-01-16T14:59:48.130422+0000","last_active":"2024-01-16T14:59:49.141674+0000","last_peered":"2024-01-16T14:59:49.141674+0000","last_clean":"2024-01-16T14:59:49.141674+0000","last_became_active":"2024-01-16T14:59:48.130133+0000","last_became_peered":"2024-01-16T14:59:48.130133+0000","last_unstale":"2024-01-16T14:59:49.141674+0000","last_undegraded":"2024-01-16T14:59:49.141674+0000","last_fullsized":"2024-01-16T14:59:49.141674+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:55.956689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.117678+0000","last_change":"2024-01-16T14:59:48.117678+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.117678+0000","last_undegraded":"2024-01-16T14:59:48.117678+0000","last_fullsized":"2024-01-16T14:59:48.117678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.139316+0000","last_change":"2024-01-16T14:59:48.138337+0000","last_active":"2024-01-16T14:59:48.139316+0000","last_peered":"2024-01-16T14:59:48.139316+0000","last_clean":"2024-01-16T14:59:48.139316+0000","last_became_active":"2024-01-16T14:59:48.137540+0000","last_became_peered":"2024-01-16T14:59:48.137540+0000","last_unstale":"2024-01-16T14:59:48.139316+0000","last_undegraded":"2024-01-16T14:59:48.139316+0000","last_fullsized":"2024-01-16T14:59:48.139316+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.114300+0000","last_change":"2024-01-16T14:59:48.114300+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.114300+0000","last_undegraded":"2024-01-16T14:59:48.114300+0000","last_fullsized":"2024-01-16T14:59:48.114300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.136941+0000","last_change":"2024-01-16T14:59:48.134570+0000","last_active":"2024-01-16T14:59:49.136941+0000","last_peered":"2024-01-16T14:59:49.136941+0000","last_clean":"2024-01-16T14:59:49.136941+0000","last_became_active":"2024-01-16T14:59:48.133797+0000","last_became_peered":"2024-01-16T14:59:48.133797+0000","last_unstale":"2024-01-16T14:59:49.136941+0000","last_undegraded":"2024-01-16T14:59:49.136941+0000","last_fullsized":"2024-01-16T14:59:49.136941+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:12.580412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.152657+0000","last_change":"2024-01-16T14:59:49.147430+0000","last_active":"2024-01-16T14:59:49.152657+0000","last_peered":"2024-01-16T14:59:49.152657+0000","last_clean":"2024-01-16T14:59:49.152657+0000","last_became_active":"2024-01-16T14:59:49.147118+0000","last_became_peered":"2024-01-16T14:59:49.147118+0000","last_unstale":"2024-01-16T14:59:49.152657+0000","last_undegraded":"2024-01-16T14:59:49.152657+0000","last_fullsized":"2024-01-16T14:59:49.152657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136626+0000","last_change":"2024-01-16T14:59:48.134611+0000","last_active":"2024-01-16T14:59:48.136626+0000","last_peered":"2024-01-16T14:59:48.136626+0000","last_clean":"2024-01-16T14:59:48.136626+0000","last_became_active":"2024-01-16T14:59:48.134321+0000","last_became_peered":"2024-01-16T14:59:48.134321+0000","last_unstale":"2024-01-16T14:59:48.136626+0000","last_undegraded":"2024-01-16T14:59:48.136626+0000","last_fullsized":"2024-01-16T14:59:48.136626+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153730+0000","last_change":"2024-01-16T14:59:49.149860+0000","last_active":"2024-01-16T14:59:49.153730+0000","last_peered":"2024-01-16T14:59:49.153730+0000","last_clean":"2024-01-16T14:59:49.153730+0000","last_became_active":"2024-01-16T14:59:49.149557+0000","last_became_peered":"2024-01-16T14:59:49.149557+0000","last_unstale":"2024-01-16T14:59:49.153730+0000","last_undegraded":"2024-01-16T14:59:49.153730+0000","last_fullsized":"2024-01-16T14:59:49.153730+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.139001+0000","last_change":"2024-01-16T14:59:48.131348+0000","last_active":"2024-01-16T14:59:49.139001+0000","last_peered":"2024-01-16T14:59:49.139001+0000","last_clean":"2024-01-16T14:59:49.139001+0000","last_became_active":"2024-01-16T14:59:48.131045+0000","last_became_peered":"2024-01-16T14:59:48.131045+0000","last_unstale":"2024-01-16T14:59:49.139001+0000","last_undegraded":"2024-01-16T14:59:49.139001+0000","last_fullsized":"2024-01-16T14:59:49.139001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.119715+0000","last_change":"2024-01-16T14:59:48.119715+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.119715+0000","last_undegraded":"2024-01-16T14:59:48.119715+0000","last_fullsized":"2024-01-16T14:59:48.119715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.138671+0000","last_change":"2024-01-16T14:59:48.136397+0000","last_active":"2024-01-16T14:59:49.138671+0000","last_peered":"2024-01-16T14:59:49.138671+0000","last_clean":"2024-01-16T14:59:49.138671+0000","last_became_active":"2024-01-16T14:59:48.136052+0000","last_became_peered":"2024-01-16T14:59:48.136052+0000","last_unstale":"2024-01-16T14:59:49.138671+0000","last_undegraded":"2024-01-16T14:59:49.138671+0000","last_fullsized":"2024-01-16T14:59:49.138671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:47.000083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.157313+0000","last_change":"2024-01-16T14:59:49.157313+0000","last_active":"2024-01-16T14:59:49.157313+0000","last_peered":"2024-01-16T14:59:49.157313+0000","last_clean":"2024-01-16T14:59:49.157313+0000","last_became_active":"2024-01-16T14:59:49.156979+0000","last_became_peered":"2024-01-16T14:59:49.156979+0000","last_unstale":"2024-01-16T14:59:49.157313+0000","last_undegraded":"2024-01-16T14:59:49.157313+0000","last_fullsized":"2024-01-16T14:59:49.157313+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136547+0000","last_change":"2024-01-16T14:59:48.131399+0000","last_active":"2024-01-16T14:59:48.136547+0000","last_peered":"2024-01-16T14:59:48.136547+0000","last_clean":"2024-01-16T14:59:48.136547+0000","last_became_active":"2024-01-16T14:59:48.131079+0000","last_became_peered":"2024-01-16T14:59:48.131079+0000","last_unstale":"2024-01-16T14:59:48.136547+0000","last_undegraded":"2024-01-16T14:59:48.136547+0000","last_fullsized":"2024-01-16T14:59:48.136547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:33.060395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-16T14:59:48.116250+0000","last_change":"2024-01-16T14:59:48.116250+0000","last_active":"2024-01-16T14:59:48.095777+0000","last_peered":"2024-01-16T14:59:48.095777+0000","last_clean":"2024-01-16T14:59:48.095777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:59:48.116250+0000","last_undegraded":"2024-01-16T14:59:48.116250+0000","last_fullsized":"2024-01-16T14:59:48.116250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T14:59:48.136657+0000","last_change":"2024-01-16T14:59:48.132319+0000","last_active":"2024-01-16T14:59:48.136657+0000","last_peered":"2024-01-16T14:59:48.136657+0000","last_clean":"2024-01-16T14:59:48.136657+0000","last_became_active":"2024-01-16T14:59:48.132030+0000","last_became_peered":"2024-01-16T14:59:48.132030+0000","last_unstale":"2024-01-16T14:59:48.136657+0000","last_undegraded":"2024-01-16T14:59:48.136657+0000","last_fullsized":"2024-01-16T14:59:48.136657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-01-16T14:59:47.105463+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:59:47.105463+0000","last_peered":"2024-01-16T14:59:47.105463+0000","last_clean":"2024-01-16T14:59:47.105463+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:59:47.105463+0000","last_undegraded":"2024-01-16T14:59:47.105463+0000","last_fullsized":"2024-01-16T14:59:47.105463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":616,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0660000000000001,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.335,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.22,"15min":1.1319999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":1.044}]},{"osd":1,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1359999999999999,"15min":1.105},"min":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5629999999999999,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.196},{"interface":"front","average":{"1min":1.218,"5min":1.222,"15min":1.216},"min":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.746,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.4399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2050000000000001,"15min":1.1919999999999999},"min":{"1min":0.74299999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.278,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":1.413},{"interface":"front","average":{"1min":1.236,"5min":1.254,"15min":1.22},"min":{"1min":0.877,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4369999999999998,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2190000000000001,"15min":1.2230000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.419,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":5.7969999999999997,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.3169999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.425,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":3.234,"5min":1.7110000000000001,"15min":1.4870000000000001},"min":{"1min":0.82799999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":36.374000000000002,"5min":36.374000000000002,"15min":36.374000000000002},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.407,"15min":1.361},"min":{"1min":0.61299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.9889999999999999,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.431,"15min":1.4099999999999999},"min":{"1min":0.69699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.5139999999999998,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.5029999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"min":{"1min":0.879,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.4849999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.246},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.601},"min":{"1min":0.628,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.5230000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.087},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.254,"15min":1.24},"min":{"1min":0.75800000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.1220000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.081,"15min":1.0549999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.143,"15min":1.1339999999999999},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.643,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2230000000000001,"15min":1.2110000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":1.26},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.337}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.367,"15min":1.377},"min":{"1min":0.97399999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.526},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4159999999999999,"15min":1.4379999999999999},"min":{"1min":0.89300000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8640000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.4419999999999999,"15min":1.468},"min":{"1min":0.91500000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.3740000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0169999999999999,"5min":2.202,"15min":2.202},"last":1.171},{"interface":"front","average":{"1min":1.702,"5min":1.6000000000000001,"15min":1.579},"min":{"1min":1.1839999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.419}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.597,"15min":1.591},"min":{"1min":1.008,"5min":1,"15min":1},"max":{"1min":4.3550000000000004,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.468},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.234}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":1.0369999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.216,"15min":1.147},"min":{"1min":0.76100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.9780000000000002,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.3740000000000001,"15min":1.2490000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.6500000000000004,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1830000000000001,"15min":1.167},"min":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.272,"15min":1.3089999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.319,"15min":1.2669999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.0940000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3480000000000001,"15min":1.3660000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3660000000000001,"15min":1.3520000000000001},"min":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4620000000000002,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.4810000000000001,"15min":1.496},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.339}]},{"osd":6,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.492,"15min":1.516},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.431,"15min":1.3859999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":10.015000000000001,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.018}]},{"osd":7,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.524,"15min":1.5660000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.583,"15min":1.5740000000000001},"min":{"1min":0.94699999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.357}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2330000000000001,"15min":1.1870000000000001},"min":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9550000000000001,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.383,"5min":1.27,"15min":1.252},"min":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.1870000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.141,"15min":1.089},"min":{"1min":0.76900000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.145,"15min":1.133},"min":{"1min":1.01,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.2789999999999999,"15min":1.2010000000000001},"min":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":8.0579999999999998,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.298,"15min":1.3},"min":{"1min":0.96699999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.0089999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2669999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2280000000000002,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.874},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":6.327,"5min":7.0650000000000004,"15min":7.0650000000000004},"last":0.96699999999999997}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.929,"5min":1.5720000000000001,"15min":1.5600000000000001},"min":{"1min":0.92900000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.289999999999999,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.089}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.173,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.395},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5660000000000001,"15min":1.6020000000000001},"min":{"1min":0.72299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.6259999999999999},"min":{"1min":1.02,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3580000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.349},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.7210000000000001,"15min":1.6950000000000001},"min":{"1min":0.92100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":3.8300000000000001,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.2310000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.91500000000000004,"15min":0.91900000000000004},"min":{"1min":0.57199999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.625,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.97799999999999998,"15min":0.93200000000000005},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.335,"5min":6.6669999999999998,"15min":6.6669999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.56100000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0700000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7370000000000001,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.7230000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.9930000000000001,"5min":6.734,"15min":6.734},"last":1.222},{"interface":"front","average":{"1min":1.115,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.67600000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.80800000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.3540000000000001},"min":{"1min":0.78100000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.6339999999999999,"5min":6.9640000000000004,"15min":6.9640000000000004},"last":1.113},{"interface":"front","average":{"1min":1.262,"5min":1.3200000000000001,"15min":1.3440000000000001},"min":{"1min":0.82499999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6319999999999999,"5min":2.762,"15min":2.762},"last":1.0720000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4179999999999999,"15min":1.466},"min":{"1min":0.88,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0659999999999998,"5min":3.169,"15min":3.169},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4359999999999999,"15min":1.4730000000000001},"min":{"1min":0.93700000000000006,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.956,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.829}]},{"osd":6,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4350000000000001,"15min":1.4099999999999999},"min":{"1min":0.95599999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.716,"5min":6.8460000000000001,"15min":6.8460000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.4379999999999999,"15min":1.4690000000000001},"min":{"1min":0.92800000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.835,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":2.6030000000000002}]},{"osd":7,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.482,"15min":1.448},"min":{"1min":0.92000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.865,"5min":6.9130000000000003,"15min":6.9130000000000003},"last":1.179},{"interface":"front","average":{"1min":1.615,"5min":1.5049999999999999,"15min":1.51},"min":{"1min":0.88400000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":2.3220000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.72099999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.3149999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.754},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.069,"15min":1.105},"min":{"1min":0.56499999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.5760000000000001,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.23},"min":{"1min":0.68200000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.718,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.27,"15min":1.3480000000000001},"min":{"1min":0.54600000000000004,"5min":0.373,"15min":0.373},"max":{"1min":2.6309999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.2509999999999999,"5min":6.141,"15min":6.141},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.331,"15min":1.397},"min":{"1min":0.88700000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.889,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0940000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9619999999999997,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4039999999999999,"15min":1.4399999999999999},"min":{"1min":0.71999999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.3679999999999999,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.4199999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.5009999999999999,"15min":1.5509999999999999},"min":{"1min":1.024,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.2069999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.1479999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.8290000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.278}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4550000000000001,"15min":1.4870000000000001},"min":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1150000000000002,"5min":5.024,"15min":5.024},"last":1.325},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6000000000000001,"15min":1.655},"min":{"1min":1.1140000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.3340000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.373}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6000000000000001,"15min":1.6559999999999999},"min":{"1min":1.196,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.7970000000000002,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5680000000000001,"15min":1.617},"min":{"1min":0.81399999999999995,"5min":0.754,"15min":0.754},"max":{"1min":2.7240000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.5009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.54500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.145,"5min":0.96899999999999997,"15min":0.88500000000000001},"min":{"1min":0.51300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.54,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.90700000000000003,"15min":0.91200000000000003},"min":{"1min":0.64400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.27,"5min":1.107,"15min":1.089},"min":{"1min":0.71599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.876}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2370000000000001,"15min":1.1870000000000001},"min":{"1min":0.753,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.1479999999999999,"15min":1.044},"min":{"1min":0.67500000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.3530000000000002,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.1559999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.4790000000000001,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.833,"5min":1.4239999999999999,"15min":1.3580000000000001},"min":{"1min":0.85499999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.9480000000000004,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.454,"15min":1.4039999999999999},"min":{"1min":1.1160000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":3.3250000000000002,"5min":4.8550000000000004,"15min":4.8550000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.411,"15min":1.421},"min":{"1min":1.0900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.53,"15min":1.492},"min":{"1min":1.0980000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":5.798,"5min":5.798,"15min":5.798},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.456,"15min":1.4039999999999999},"min":{"1min":0.95199999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.0250000000000004,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.534}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.534,"15min":1.504},"min":{"1min":0.90000000000000002,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.458,"15min":1.462},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":791535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.93500000000000005,"15min":0.93700000000000006},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0189999999999999,"15min":1.054},"min":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4690000000000001,"5min":1.587,"15min":1.587},"last":1.046},{"interface":"front","average":{"1min":1.075,"5min":1,"15min":1},"min":{"1min":0.53400000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.339}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.998,"15min":0.96799999999999997},"min":{"1min":0.45100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.145,"5min":1.198,"15min":1.228},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.397}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.607},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.24},"min":{"1min":0.72099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.177}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.4950000000000001},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4399999999999999,"15min":1.4350000000000001},"min":{"1min":0.89200000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.9429999999999996},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.488,"15min":1.5129999999999999},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.423,"15min":1.4610000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.496},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.468,"15min":1.466},"min":{"1min":0.98799999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.516}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:58.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.788+0000 7f2a24400640 1 Processor -- start 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.788+0000 7f2a24400640 1 -- start start 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a24400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2a1c0cf9a0 0x7f2a1c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a24400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2a1c15e1e0 0x7f2a1c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a24400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0b1ee0 con 0x7f2a1c143ae0 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a24400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c0b2780 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a24400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c15e990 con 0x7f2a1c15e1e0 2024-01-16T14:59:58.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a21a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2a1c143ae0 0x7f2a1c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51782/0 (socket says 172.21.15.125:51782) 2024-01-16T14:59:58.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a21a00640 1 -- 172.21.15.125:0/580824425 learned_addr learned my addr 172.21.15.125:0/580824425 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:58.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a21000640 1 --2- 172.21.15.125:0/580824425 >> v2:172.21.15.125:3300/0 conn(0x7f2a1c0cf9a0 0x7f2a1c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a22400640 1 --2- 172.21.15.125:0/580824425 >> v2:172.21.15.137:3300/0 conn(0x7f2a1c15e1e0 0x7f2a1c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.790+0000 7f2a1be00640 1 -- 172.21.15.125:0/580824425 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711255990 0 0) 0x7f2a1c0b1ee0 con 0x7f2a1c143ae0 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.791+0000 7f2a1be00640 1 -- 172.21.15.125:0/580824425 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29fc003520 con 0x7f2a1c143ae0 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.791+0000 7f2a21000640 1 -- 172.21.15.125:0/580824425 >> v2:172.21.15.137:3300/0 conn(0x7f2a1c15e1e0 msgr2=0x7f2a1c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.791+0000 7f2a21000640 1 --2- 172.21.15.125:0/580824425 >> v2:172.21.15.137:3300/0 conn(0x7f2a1c15e1e0 0x7f2a1c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.791+0000 7f2a21000640 1 -- 172.21.15.125:0/580824425 >> v1:172.21.15.125:6789/0 conn(0x7f2a1c143ae0 legacy=0x7f2a1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.791+0000 7f2a21000640 1 -- 172.21.15.125:0/580824425 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0b2130 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a21000640 1 --2- 172.21.15.125:0/580824425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c15dca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0ae9d0 tx=0x7f2a0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a462730765548ad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a1be00640 1 -- 172.21.15.125:0/580824425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04031830 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a1be00640 1 -- 172.21.15.125:0/580824425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a04031990 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a1be00640 1 -- 172.21.15.125:0/580824425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04031d00 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a24400640 1 -- 172.21.15.125:0/580824425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 msgr2=0x7f2a1c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.792+0000 7f2a24400640 1 --2- 172.21.15.125:0/580824425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c15dca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0ae9d0 tx=0x7f2a0402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 -- 172.21.15.125:0/580824425 shutdown_connections 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 --2- 172.21.15.125:0/580824425 >> v2:172.21.15.137:3300/0 conn(0x7f2a1c15e1e0 0x7f2a1c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 --2- 172.21.15.125:0/580824425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c15dca0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 -- 172.21.15.125:0/580824425 >> 172.21.15.125:0/580824425 conn(0x7f2a1c06a800 msgr2=0x7f2a1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 -- 172.21.15.125:0/580824425 shutdown_connections 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.793+0000 7f2a24400640 1 -- 172.21.15.125:0/580824425 wait complete. 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.794+0000 7f2a24400640 1 Processor -- start 2024-01-16T14:59:58.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 -- start start 2024-01-16T14:59:58.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a1c143ae0 0x7f2a1c13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a1c15e1e0 0x7f2a1c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0a93e0 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0a98d0 con 0x7f2a1c15e1e0 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.795+0000 7f2a24400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0aa090 con 0x7f2a1c143ae0 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53740/0 (socket says 172.21.15.125:53740) 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 -- 172.21.15.125:0/552414355 learned_addr learned my addr 172.21.15.125:0/552414355 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21000640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a1c143ae0 0x7f2a1c13ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 -- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a1c143ae0 msgr2=0x7f2a1c13ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a1c143ae0 0x7f2a1c13ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 -- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a1c15e1e0 msgr2=0x7f2a1c141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a1c15e1e0 0x7f2a1c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 -- 172.21.15.125:0/552414355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04009400 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.796+0000 7f2a21a00640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c430 tx=0x7f2a0c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.797+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c005ce0 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.797+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c005e40 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.797+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0072f0 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.797+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c0aa370 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.797+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c0aa930 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.798+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c0675d0 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.799+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2a0c007450 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.800+0000 7f2a19600640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 0x7f29f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.800+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a0c0895f0 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.800+0000 7f2a21000640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 0x7f29f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.801+0000 7f2a21000640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 0x7f29f406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2a04036000 tx=0x7f2a0403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.804+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a0c054190 con 0x7f2a1c0cf9a0 2024-01-16T14:59:58.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:58.990+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a1c15e670 con 0x7f29f406b500 2024-01-16T14:59:58.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.012+0000 7f2a19600640 1 -- 172.21.15.125:0/552414355 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287346 (secure 0 0 0) 0x7f2a04031630 con 0x7f29f406b500 2024-01-16T14:59:58.997 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T14:59:58.998 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 msgr2=0x7f29f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 0x7f29f406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2a04036000 tx=0x7f2a0403c040 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 msgr2=0x7f2a1c0a8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c430 tx=0x7f2a0c00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 shutdown_connections 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a1c15e1e0 0x7f2a1c141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a1c143ae0 0x7f2a1c13ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29f406b500 0x7f29f406d9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 --2- 172.21.15.125:0/552414355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0cf9a0 0x7f2a1c0a8ea0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 >> 172.21.15.125:0/552414355 conn(0x7f2a1c06a800 msgr2=0x7f2a1c142af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 shutdown_connections 2024-01-16T14:59:59.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.018+0000 7f2a24400640 1 -- 172.21.15.125:0/552414355 wait complete. 2024-01-16T14:59:59.022 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-01-16T14:59:57.981300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254788,"kb_used_data":5348,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719932,"statfs":{"total":773094113280,"available":772833210368,"internally_reserved":0,"allocated":5476352,"data_stored":2906625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.855513"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.160759+0000","last_change":"2024-01-16T14:59:49.163276+0000","last_active":"2024-01-16T14:59:50.160759+0000","last_peered":"2024-01-16T14:59:50.160759+0000","last_clean":"2024-01-16T14:59:50.160759+0000","last_became_active":"2024-01-16T14:59:49.162997+0000","last_became_peered":"2024-01-16T14:59:49.162997+0000","last_unstale":"2024-01-16T14:59:50.160759+0000","last_undegraded":"2024-01-16T14:59:50.160759+0000","last_fullsized":"2024-01-16T14:59:50.160759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:54.630323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.164095+0000","last_change":"2024-01-16T14:59:49.170977+0000","last_active":"2024-01-16T14:59:50.164095+0000","last_peered":"2024-01-16T14:59:50.164095+0000","last_clean":"2024-01-16T14:59:50.164095+0000","last_became_active":"2024-01-16T14:59:49.170625+0000","last_became_peered":"2024-01-16T14:59:49.170625+0000","last_unstale":"2024-01-16T14:59:50.164095+0000","last_undegraded":"2024-01-16T14:59:50.164095+0000","last_fullsized":"2024-01-16T14:59:50.164095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:53.153378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761783+0000","last_change":"2024-01-16T14:59:49.157265+0000","last_active":"2024-01-16T14:59:50.761783+0000","last_peered":"2024-01-16T14:59:50.761783+0000","last_clean":"2024-01-16T14:59:50.761783+0000","last_became_active":"2024-01-16T14:59:49.156985+0000","last_became_peered":"2024-01-16T14:59:49.156985+0000","last_unstale":"2024-01-16T14:59:50.761783+0000","last_undegraded":"2024-01-16T14:59:50.761783+0000","last_fullsized":"2024-01-16T14:59:50.761783+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441440+0000","last_change":"2024-01-16T14:59:49.161893+0000","last_active":"2024-01-16T14:59:50.441440+0000","last_peered":"2024-01-16T14:59:50.441440+0000","last_clean":"2024-01-16T14:59:50.441440+0000","last_became_active":"2024-01-16T14:59:49.160443+0000","last_became_peered":"2024-01-16T14:59:49.160443+0000","last_unstale":"2024-01-16T14:59:50.441440+0000","last_undegraded":"2024-01-16T14:59:50.441440+0000","last_fullsized":"2024-01-16T14:59:50.441440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.164004+0000","last_change":"2024-01-16T14:59:49.171122+0000","last_active":"2024-01-16T14:59:50.164004+0000","last_peered":"2024-01-16T14:59:50.164004+0000","last_clean":"2024-01-16T14:59:50.164004+0000","last_became_active":"2024-01-16T14:59:49.170700+0000","last_became_peered":"2024-01-16T14:59:49.170700+0000","last_unstale":"2024-01-16T14:59:50.164004+0000","last_undegraded":"2024-01-16T14:59:50.164004+0000","last_fullsized":"2024-01-16T14:59:50.164004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.760989+0000","last_change":"2024-01-16T14:59:49.160920+0000","last_active":"2024-01-16T14:59:50.760989+0000","last_peered":"2024-01-16T14:59:50.760989+0000","last_clean":"2024-01-16T14:59:50.760989+0000","last_became_active":"2024-01-16T14:59:49.158310+0000","last_became_peered":"2024-01-16T14:59:49.158310+0000","last_unstale":"2024-01-16T14:59:50.760989+0000","last_undegraded":"2024-01-16T14:59:50.760989+0000","last_fullsized":"2024-01-16T14:59:50.760989+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:02:05.786276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.161227+0000","last_change":"2024-01-16T14:59:49.151664+0000","last_active":"2024-01-16T14:59:50.161227+0000","last_peered":"2024-01-16T14:59:50.161227+0000","last_clean":"2024-01-16T14:59:50.161227+0000","last_became_active":"2024-01-16T14:59:49.147394+0000","last_became_peered":"2024-01-16T14:59:49.147394+0000","last_unstale":"2024-01-16T14:59:50.161227+0000","last_undegraded":"2024-01-16T14:59:50.161227+0000","last_fullsized":"2024-01-16T14:59:50.161227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:05.867339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.440815+0000","last_change":"2024-01-16T14:59:49.168585+0000","last_active":"2024-01-16T14:59:50.440815+0000","last_peered":"2024-01-16T14:59:50.440815+0000","last_clean":"2024-01-16T14:59:50.440815+0000","last_became_active":"2024-01-16T14:59:49.166514+0000","last_became_peered":"2024-01-16T14:59:49.166514+0000","last_unstale":"2024-01-16T14:59:50.440815+0000","last_undegraded":"2024-01-16T14:59:50.440815+0000","last_fullsized":"2024-01-16T14:59:50.440815+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:31.120308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441800+0000","last_change":"2024-01-16T14:59:49.148994+0000","last_active":"2024-01-16T14:59:50.441800+0000","last_peered":"2024-01-16T14:59:50.441800+0000","last_clean":"2024-01-16T14:59:50.441800+0000","last_became_active":"2024-01-16T14:59:49.148683+0000","last_became_peered":"2024-01-16T14:59:49.148683+0000","last_unstale":"2024-01-16T14:59:50.441800+0000","last_undegraded":"2024-01-16T14:59:50.441800+0000","last_fullsized":"2024-01-16T14:59:50.441800+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:00.684681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441951+0000","last_change":"2024-01-16T14:59:49.160577+0000","last_active":"2024-01-16T14:59:50.441951+0000","last_peered":"2024-01-16T14:59:50.441951+0000","last_clean":"2024-01-16T14:59:50.441951+0000","last_became_active":"2024-01-16T14:59:49.160358+0000","last_became_peered":"2024-01-16T14:59:49.160358+0000","last_unstale":"2024-01-16T14:59:50.441951+0000","last_undegraded":"2024-01-16T14:59:50.441951+0000","last_fullsized":"2024-01-16T14:59:50.441951+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:53.153378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441430+0000","last_change":"2024-01-16T14:59:49.151889+0000","last_active":"2024-01-16T14:59:50.441430+0000","last_peered":"2024-01-16T14:59:50.441430+0000","last_clean":"2024-01-16T14:59:50.441430+0000","last_became_active":"2024-01-16T14:59:49.151589+0000","last_became_peered":"2024-01-16T14:59:49.151589+0000","last_unstale":"2024-01-16T14:59:50.441430+0000","last_undegraded":"2024-01-16T14:59:50.441430+0000","last_fullsized":"2024-01-16T14:59:50.441430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:49.237963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.163670+0000","last_change":"2024-01-16T14:59:49.167197+0000","last_active":"2024-01-16T14:59:50.163670+0000","last_peered":"2024-01-16T14:59:50.163670+0000","last_clean":"2024-01-16T14:59:50.163670+0000","last_became_active":"2024-01-16T14:59:49.166900+0000","last_became_peered":"2024-01-16T14:59:49.166900+0000","last_unstale":"2024-01-16T14:59:50.163670+0000","last_undegraded":"2024-01-16T14:59:50.163670+0000","last_fullsized":"2024-01-16T14:59:50.163670+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:55.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.155962+0000","last_change":"2024-01-16T14:59:49.145182+0000","last_active":"2024-01-16T14:59:50.155962+0000","last_peered":"2024-01-16T14:59:50.155962+0000","last_clean":"2024-01-16T14:59:50.155962+0000","last_became_active":"2024-01-16T14:59:49.144893+0000","last_became_peered":"2024-01-16T14:59:49.144893+0000","last_unstale":"2024-01-16T14:59:50.155962+0000","last_undegraded":"2024-01-16T14:59:50.155962+0000","last_fullsized":"2024-01-16T14:59:50.155962+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:26.663192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441150+0000","last_change":"2024-01-16T14:59:49.155328+0000","last_active":"2024-01-16T14:59:50.441150+0000","last_peered":"2024-01-16T14:59:50.441150+0000","last_clean":"2024-01-16T14:59:50.441150+0000","last_became_active":"2024-01-16T14:59:49.154994+0000","last_became_peered":"2024-01-16T14:59:49.154994+0000","last_unstale":"2024-01-16T14:59:50.441150+0000","last_undegraded":"2024-01-16T14:59:50.441150+0000","last_fullsized":"2024-01-16T14:59:50.441150+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.760858+0000","last_change":"2024-01-16T14:59:49.158516+0000","last_active":"2024-01-16T14:59:50.760858+0000","last_peered":"2024-01-16T14:59:50.760858+0000","last_clean":"2024-01-16T14:59:50.760858+0000","last_became_active":"2024-01-16T14:59:49.158217+0000","last_became_peered":"2024-01-16T14:59:49.158217+0000","last_unstale":"2024-01-16T14:59:50.760858+0000","last_undegraded":"2024-01-16T14:59:50.760858+0000","last_fullsized":"2024-01-16T14:59:50.760858+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:22.714011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762245+0000","last_change":"2024-01-16T14:59:49.172526+0000","last_active":"2024-01-16T14:59:50.762245+0000","last_peered":"2024-01-16T14:59:50.762245+0000","last_clean":"2024-01-16T14:59:50.762245+0000","last_became_active":"2024-01-16T14:59:49.172275+0000","last_became_peered":"2024-01-16T14:59:49.172275+0000","last_unstale":"2024-01-16T14:59:50.762245+0000","last_undegraded":"2024-01-16T14:59:50.762245+0000","last_fullsized":"2024-01-16T14:59:50.762245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:22.678485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762308+0000","last_change":"2024-01-16T14:59:49.162728+0000","last_active":"2024-01-16T14:59:50.762308+0000","last_peered":"2024-01-16T14:59:50.762308+0000","last_clean":"2024-01-16T14:59:50.762308+0000","last_became_active":"2024-01-16T14:59:49.162473+0000","last_became_peered":"2024-01-16T14:59:49.162473+0000","last_unstale":"2024-01-16T14:59:50.762308+0000","last_undegraded":"2024-01-16T14:59:50.762308+0000","last_fullsized":"2024-01-16T14:59:50.762308+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:44.168064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.763105+0000","last_change":"2024-01-16T14:59:49.146953+0000","last_active":"2024-01-16T14:59:50.763105+0000","last_peered":"2024-01-16T14:59:50.763105+0000","last_clean":"2024-01-16T14:59:50.763105+0000","last_became_active":"2024-01-16T14:59:49.146074+0000","last_became_peered":"2024-01-16T14:59:49.146074+0000","last_unstale":"2024-01-16T14:59:50.763105+0000","last_undegraded":"2024-01-16T14:59:50.763105+0000","last_fullsized":"2024-01-16T14:59:50.763105+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:17.526674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.764255+0000","last_change":"2024-01-16T14:59:49.166711+0000","last_active":"2024-01-16T14:59:50.764255+0000","last_peered":"2024-01-16T14:59:50.764255+0000","last_clean":"2024-01-16T14:59:50.764255+0000","last_became_active":"2024-01-16T14:59:49.165305+0000","last_became_peered":"2024-01-16T14:59:49.165305+0000","last_unstale":"2024-01-16T14:59:50.764255+0000","last_undegraded":"2024-01-16T14:59:50.764255+0000","last_fullsized":"2024-01-16T14:59:50.764255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:40.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.156573+0000","last_change":"2024-01-16T14:59:49.148890+0000","last_active":"2024-01-16T14:59:50.156573+0000","last_peered":"2024-01-16T14:59:50.156573+0000","last_clean":"2024-01-16T14:59:50.156573+0000","last_became_active":"2024-01-16T14:59:49.148606+0000","last_became_peered":"2024-01-16T14:59:49.148606+0000","last_unstale":"2024-01-16T14:59:50.156573+0000","last_undegraded":"2024-01-16T14:59:50.156573+0000","last_fullsized":"2024-01-16T14:59:50.156573+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:13.595065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761126+0000","last_change":"2024-01-16T14:59:49.158956+0000","last_active":"2024-01-16T14:59:50.761126+0000","last_peered":"2024-01-16T14:59:50.761126+0000","last_clean":"2024-01-16T14:59:50.761126+0000","last_became_active":"2024-01-16T14:59:49.158665+0000","last_became_peered":"2024-01-16T14:59:49.158665+0000","last_unstale":"2024-01-16T14:59:50.761126+0000","last_undegraded":"2024-01-16T14:59:50.761126+0000","last_fullsized":"2024-01-16T14:59:50.761126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:59.062594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441096+0000","last_change":"2024-01-16T14:59:49.161951+0000","last_active":"2024-01-16T14:59:50.441096+0000","last_peered":"2024-01-16T14:59:50.441096+0000","last_clean":"2024-01-16T14:59:50.441096+0000","last_became_active":"2024-01-16T14:59:49.161422+0000","last_became_peered":"2024-01-16T14:59:49.161422+0000","last_unstale":"2024-01-16T14:59:50.441096+0000","last_undegraded":"2024-01-16T14:59:50.441096+0000","last_fullsized":"2024-01-16T14:59:50.441096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762644+0000","last_change":"2024-01-16T14:59:49.161508+0000","last_active":"2024-01-16T14:59:50.762644+0000","last_peered":"2024-01-16T14:59:50.762644+0000","last_clean":"2024-01-16T14:59:50.762644+0000","last_became_active":"2024-01-16T14:59:49.161246+0000","last_became_peered":"2024-01-16T14:59:49.161246+0000","last_unstale":"2024-01-16T14:59:50.762644+0000","last_undegraded":"2024-01-16T14:59:50.762644+0000","last_fullsized":"2024-01-16T14:59:50.762644+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:48.014735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762868+0000","last_change":"2024-01-16T14:59:49.167480+0000","last_active":"2024-01-16T14:59:50.762868+0000","last_peered":"2024-01-16T14:59:50.762868+0000","last_clean":"2024-01-16T14:59:50.762868+0000","last_became_active":"2024-01-16T14:59:49.164808+0000","last_became_peered":"2024-01-16T14:59:49.164808+0000","last_unstale":"2024-01-16T14:59:50.762868+0000","last_undegraded":"2024-01-16T14:59:50.762868+0000","last_fullsized":"2024-01-16T14:59:50.762868+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.764147+0000","last_change":"2024-01-16T14:59:49.167175+0000","last_active":"2024-01-16T14:59:50.764147+0000","last_peered":"2024-01-16T14:59:50.764147+0000","last_clean":"2024-01-16T14:59:50.764147+0000","last_became_active":"2024-01-16T14:59:49.166896+0000","last_became_peered":"2024-01-16T14:59:49.166896+0000","last_unstale":"2024-01-16T14:59:50.764147+0000","last_undegraded":"2024-01-16T14:59:50.764147+0000","last_fullsized":"2024-01-16T14:59:50.764147+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:27.049078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.156239+0000","last_change":"2024-01-16T14:59:49.144234+0000","last_active":"2024-01-16T14:59:50.156239+0000","last_peered":"2024-01-16T14:59:50.156239+0000","last_clean":"2024-01-16T14:59:50.156239+0000","last_became_active":"2024-01-16T14:59:49.143798+0000","last_became_peered":"2024-01-16T14:59:49.143798+0000","last_unstale":"2024-01-16T14:59:50.156239+0000","last_undegraded":"2024-01-16T14:59:50.156239+0000","last_fullsized":"2024-01-16T14:59:50.156239+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:12:21.166819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762148+0000","last_change":"2024-01-16T14:59:49.164250+0000","last_active":"2024-01-16T14:59:50.762148+0000","last_peered":"2024-01-16T14:59:50.762148+0000","last_clean":"2024-01-16T14:59:50.762148+0000","last_became_active":"2024-01-16T14:59:49.163931+0000","last_became_peered":"2024-01-16T14:59:49.163931+0000","last_unstale":"2024-01-16T14:59:50.762148+0000","last_undegraded":"2024-01-16T14:59:50.762148+0000","last_fullsized":"2024-01-16T14:59:50.762148+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:59.627534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.163722+0000","last_change":"2024-01-16T14:59:49.163361+0000","last_active":"2024-01-16T14:59:50.163722+0000","last_peered":"2024-01-16T14:59:50.163722+0000","last_clean":"2024-01-16T14:59:50.163722+0000","last_became_active":"2024-01-16T14:59:49.163036+0000","last_became_peered":"2024-01-16T14:59:49.163036+0000","last_unstale":"2024-01-16T14:59:50.163722+0000","last_undegraded":"2024-01-16T14:59:50.163722+0000","last_fullsized":"2024-01-16T14:59:50.163722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:42.991867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762997+0000","last_change":"2024-01-16T14:59:49.163292+0000","last_active":"2024-01-16T14:59:50.762997+0000","last_peered":"2024-01-16T14:59:50.762997+0000","last_clean":"2024-01-16T14:59:50.762997+0000","last_became_active":"2024-01-16T14:59:49.162987+0000","last_became_peered":"2024-01-16T14:59:49.162987+0000","last_unstale":"2024-01-16T14:59:50.762997+0000","last_undegraded":"2024-01-16T14:59:50.762997+0000","last_fullsized":"2024-01-16T14:59:50.762997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761754+0000","last_change":"2024-01-16T14:59:49.160966+0000","last_active":"2024-01-16T14:59:50.761754+0000","last_peered":"2024-01-16T14:59:50.761754+0000","last_clean":"2024-01-16T14:59:50.761754+0000","last_became_active":"2024-01-16T14:59:49.160612+0000","last_became_peered":"2024-01-16T14:59:49.160612+0000","last_unstale":"2024-01-16T14:59:50.761754+0000","last_undegraded":"2024-01-16T14:59:50.761754+0000","last_fullsized":"2024-01-16T14:59:50.761754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:01.886358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.760982+0000","last_change":"2024-01-16T14:59:49.159653+0000","last_active":"2024-01-16T14:59:50.760982+0000","last_peered":"2024-01-16T14:59:50.760982+0000","last_clean":"2024-01-16T14:59:50.760982+0000","last_became_active":"2024-01-16T14:59:49.159375+0000","last_became_peered":"2024-01-16T14:59:49.159375+0000","last_unstale":"2024-01-16T14:59:50.760982+0000","last_undegraded":"2024-01-16T14:59:50.760982+0000","last_fullsized":"2024-01-16T14:59:50.760982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:47.411917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.155942+0000","last_change":"2024-01-16T14:59:49.145998+0000","last_active":"2024-01-16T14:59:50.155942+0000","last_peered":"2024-01-16T14:59:50.155942+0000","last_clean":"2024-01-16T14:59:50.155942+0000","last_became_active":"2024-01-16T14:59:49.145704+0000","last_became_peered":"2024-01-16T14:59:49.145704+0000","last_unstale":"2024-01-16T14:59:50.155942+0000","last_undegraded":"2024-01-16T14:59:50.155942+0000","last_fullsized":"2024-01-16T14:59:50.155942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:44.382458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.763248+0000","last_change":"2024-01-16T14:59:49.156116+0000","last_active":"2024-01-16T14:59:50.763248+0000","last_peered":"2024-01-16T14:59:50.763248+0000","last_clean":"2024-01-16T14:59:50.763248+0000","last_became_active":"2024-01-16T14:59:49.155198+0000","last_became_peered":"2024-01-16T14:59:49.155198+0000","last_unstale":"2024-01-16T14:59:50.763248+0000","last_undegraded":"2024-01-16T14:59:50.763248+0000","last_fullsized":"2024-01-16T14:59:50.763248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:50.375876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225224+0000","last_change":"2024-01-16T14:59:48.137330+0000","last_active":"2024-01-16T14:59:49.225224+0000","last_peered":"2024-01-16T14:59:49.225224+0000","last_clean":"2024-01-16T14:59:49.225224+0000","last_became_active":"2024-01-16T14:59:48.136959+0000","last_became_peered":"2024-01-16T14:59:48.136959+0000","last_unstale":"2024-01-16T14:59:49.225224+0000","last_undegraded":"2024-01-16T14:59:49.225224+0000","last_fullsized":"2024-01-16T14:59:49.225224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:39.353399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.764243+0000","last_change":"2024-01-16T14:59:49.168398+0000","last_active":"2024-01-16T14:59:50.764243+0000","last_peered":"2024-01-16T14:59:50.764243+0000","last_clean":"2024-01-16T14:59:50.764243+0000","last_became_active":"2024-01-16T14:59:49.165002+0000","last_became_peered":"2024-01-16T14:59:49.165002+0000","last_unstale":"2024-01-16T14:59:50.764243+0000","last_undegraded":"2024-01-16T14:59:50.764243+0000","last_fullsized":"2024-01-16T14:59:50.764243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:50.278294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167656+0000","last_change":"2024-01-16T14:59:48.137753+0000","last_active":"2024-01-16T14:59:49.167656+0000","last_peered":"2024-01-16T14:59:49.167656+0000","last_clean":"2024-01-16T14:59:49.167656+0000","last_became_active":"2024-01-16T14:59:48.137414+0000","last_became_peered":"2024-01-16T14:59:48.137414+0000","last_unstale":"2024-01-16T14:59:49.167656+0000","last_undegraded":"2024-01-16T14:59:49.167656+0000","last_fullsized":"2024-01-16T14:59:49.167656+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.164145+0000","last_change":"2024-01-16T14:59:49.160498+0000","last_active":"2024-01-16T14:59:50.164145+0000","last_peered":"2024-01-16T14:59:50.164145+0000","last_clean":"2024-01-16T14:59:50.164145+0000","last_became_active":"2024-01-16T14:59:49.160159+0000","last_became_peered":"2024-01-16T14:59:49.160159+0000","last_unstale":"2024-01-16T14:59:50.164145+0000","last_undegraded":"2024-01-16T14:59:50.164145+0000","last_fullsized":"2024-01-16T14:59:50.164145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:21.559282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.215629+0000","last_change":"2024-01-16T14:59:48.134911+0000","last_active":"2024-01-16T14:59:49.215629+0000","last_peered":"2024-01-16T14:59:49.215629+0000","last_clean":"2024-01-16T14:59:49.215629+0000","last_became_active":"2024-01-16T14:59:48.134259+0000","last_became_peered":"2024-01-16T14:59:48.134259+0000","last_unstale":"2024-01-16T14:59:49.215629+0000","last_undegraded":"2024-01-16T14:59:49.215629+0000","last_fullsized":"2024-01-16T14:59:49.215629+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:28.124634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762053+0000","last_change":"2024-01-16T14:59:49.165223+0000","last_active":"2024-01-16T14:59:50.762053+0000","last_peered":"2024-01-16T14:59:50.762053+0000","last_clean":"2024-01-16T14:59:50.762053+0000","last_became_active":"2024-01-16T14:59:49.164935+0000","last_became_peered":"2024-01-16T14:59:49.164935+0000","last_unstale":"2024-01-16T14:59:50.762053+0000","last_undegraded":"2024-01-16T14:59:50.762053+0000","last_fullsized":"2024-01-16T14:59:50.762053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:24.444381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.162588+0000","last_change":"2024-01-16T14:59:48.131434+0000","last_active":"2024-01-16T14:59:49.162588+0000","last_peered":"2024-01-16T14:59:49.162588+0000","last_clean":"2024-01-16T14:59:49.162588+0000","last_became_active":"2024-01-16T14:59:48.131165+0000","last_became_peered":"2024-01-16T14:59:48.131165+0000","last_unstale":"2024-01-16T14:59:49.162588+0000","last_undegraded":"2024-01-16T14:59:49.162588+0000","last_fullsized":"2024-01-16T14:59:49.162588+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:16.512021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762155+0000","last_change":"2024-01-16T14:59:49.159531+0000","last_active":"2024-01-16T14:59:50.762155+0000","last_peered":"2024-01-16T14:59:50.762155+0000","last_clean":"2024-01-16T14:59:50.762155+0000","last_became_active":"2024-01-16T14:59:49.159274+0000","last_became_peered":"2024-01-16T14:59:49.159274+0000","last_unstale":"2024-01-16T14:59:50.762155+0000","last_undegraded":"2024-01-16T14:59:50.762155+0000","last_fullsized":"2024-01-16T14:59:50.762155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225141+0000","last_change":"2024-01-16T14:59:48.128798+0000","last_active":"2024-01-16T14:59:49.225141+0000","last_peered":"2024-01-16T14:59:49.225141+0000","last_clean":"2024-01-16T14:59:49.225141+0000","last_became_active":"2024-01-16T14:59:48.128328+0000","last_became_peered":"2024-01-16T14:59:48.128328+0000","last_unstale":"2024-01-16T14:59:49.225141+0000","last_undegraded":"2024-01-16T14:59:49.225141+0000","last_fullsized":"2024-01-16T14:59:49.225141+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:54.681593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441320+0000","last_change":"2024-01-16T14:59:49.148854+0000","last_active":"2024-01-16T14:59:50.441320+0000","last_peered":"2024-01-16T14:59:50.441320+0000","last_clean":"2024-01-16T14:59:50.441320+0000","last_became_active":"2024-01-16T14:59:49.148339+0000","last_became_peered":"2024-01-16T14:59:49.148339+0000","last_unstale":"2024-01-16T14:59:50.441320+0000","last_undegraded":"2024-01-16T14:59:50.441320+0000","last_fullsized":"2024-01-16T14:59:50.441320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167540+0000","last_change":"2024-01-16T14:59:48.131571+0000","last_active":"2024-01-16T14:59:49.167540+0000","last_peered":"2024-01-16T14:59:49.167540+0000","last_clean":"2024-01-16T14:59:49.167540+0000","last_became_active":"2024-01-16T14:59:48.131245+0000","last_became_peered":"2024-01-16T14:59:48.131245+0000","last_unstale":"2024-01-16T14:59:49.167540+0000","last_undegraded":"2024-01-16T14:59:49.167540+0000","last_fullsized":"2024-01-16T14:59:49.167540+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.440967+0000","last_change":"2024-01-16T14:59:49.158773+0000","last_active":"2024-01-16T14:59:50.440967+0000","last_peered":"2024-01-16T14:59:50.440967+0000","last_clean":"2024-01-16T14:59:50.440967+0000","last_became_active":"2024-01-16T14:59:49.158471+0000","last_became_peered":"2024-01-16T14:59:49.158471+0000","last_unstale":"2024-01-16T14:59:50.440967+0000","last_undegraded":"2024-01-16T14:59:50.440967+0000","last_fullsized":"2024-01-16T14:59:50.440967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:01.886358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.153642+0000","last_change":"2024-01-16T14:59:48.130388+0000","last_active":"2024-01-16T14:59:49.153642+0000","last_peered":"2024-01-16T14:59:49.153642+0000","last_clean":"2024-01-16T14:59:49.153642+0000","last_became_active":"2024-01-16T14:59:48.129990+0000","last_became_peered":"2024-01-16T14:59:48.129990+0000","last_unstale":"2024-01-16T14:59:49.153642+0000","last_undegraded":"2024-01-16T14:59:49.153642+0000","last_fullsized":"2024-01-16T14:59:49.153642+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:44.655685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762435+0000","last_change":"2024-01-16T14:59:49.165793+0000","last_active":"2024-01-16T14:59:50.762435+0000","last_peered":"2024-01-16T14:59:50.762435+0000","last_clean":"2024-01-16T14:59:50.762435+0000","last_became_active":"2024-01-16T14:59:49.165321+0000","last_became_peered":"2024-01-16T14:59:49.165321+0000","last_unstale":"2024-01-16T14:59:50.762435+0000","last_undegraded":"2024-01-16T14:59:50.762435+0000","last_fullsized":"2024-01-16T14:59:50.762435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:29.139287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142085+0000","last_change":"2024-01-16T14:59:48.122577+0000","last_active":"2024-01-16T14:59:49.142085+0000","last_peered":"2024-01-16T14:59:49.142085+0000","last_clean":"2024-01-16T14:59:49.142085+0000","last_became_active":"2024-01-16T14:59:48.122280+0000","last_became_peered":"2024-01-16T14:59:48.122280+0000","last_unstale":"2024-01-16T14:59:49.142085+0000","last_undegraded":"2024-01-16T14:59:49.142085+0000","last_fullsized":"2024-01-16T14:59:49.142085+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762371+0000","last_change":"2024-01-16T14:59:49.146563+0000","last_active":"2024-01-16T14:59:50.762371+0000","last_peered":"2024-01-16T14:59:50.762371+0000","last_clean":"2024-01-16T14:59:50.762371+0000","last_became_active":"2024-01-16T14:59:49.145334+0000","last_became_peered":"2024-01-16T14:59:49.145334+0000","last_unstale":"2024-01-16T14:59:50.762371+0000","last_undegraded":"2024-01-16T14:59:50.762371+0000","last_fullsized":"2024-01-16T14:59:50.762371+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:06.763683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167705+0000","last_change":"2024-01-16T14:59:48.127440+0000","last_active":"2024-01-16T14:59:49.167705+0000","last_peered":"2024-01-16T14:59:49.167705+0000","last_clean":"2024-01-16T14:59:49.167705+0000","last_became_active":"2024-01-16T14:59:48.126114+0000","last_became_peered":"2024-01-16T14:59:48.126114+0000","last_unstale":"2024-01-16T14:59:49.167705+0000","last_undegraded":"2024-01-16T14:59:49.167705+0000","last_fullsized":"2024-01-16T14:59:49.167705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.160828+0000","last_change":"2024-01-16T14:59:49.160956+0000","last_active":"2024-01-16T14:59:50.160828+0000","last_peered":"2024-01-16T14:59:50.160828+0000","last_clean":"2024-01-16T14:59:50.160828+0000","last_became_active":"2024-01-16T14:59:49.160601+0000","last_became_peered":"2024-01-16T14:59:49.160601+0000","last_unstale":"2024-01-16T14:59:50.160828+0000","last_undegraded":"2024-01-16T14:59:50.160828+0000","last_fullsized":"2024-01-16T14:59:50.160828+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.670502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.155367+0000","last_change":"2024-01-16T14:59:48.128532+0000","last_active":"2024-01-16T14:59:49.155367+0000","last_peered":"2024-01-16T14:59:49.155367+0000","last_clean":"2024-01-16T14:59:49.155367+0000","last_became_active":"2024-01-16T14:59:48.127045+0000","last_became_peered":"2024-01-16T14:59:48.127045+0000","last_unstale":"2024-01-16T14:59:49.155367+0000","last_undegraded":"2024-01-16T14:59:49.155367+0000","last_fullsized":"2024-01-16T14:59:49.155367+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:49.263642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762032+0000","last_change":"2024-01-16T14:59:49.157462+0000","last_active":"2024-01-16T14:59:50.762032+0000","last_peered":"2024-01-16T14:59:50.762032+0000","last_clean":"2024-01-16T14:59:50.762032+0000","last_became_active":"2024-01-16T14:59:49.157150+0000","last_became_peered":"2024-01-16T14:59:49.157150+0000","last_unstale":"2024-01-16T14:59:50.762032+0000","last_undegraded":"2024-01-16T14:59:50.762032+0000","last_fullsized":"2024-01-16T14:59:50.762032+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:31.119871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167579+0000","last_change":"2024-01-16T14:59:48.135944+0000","last_active":"2024-01-16T14:59:49.167579+0000","last_peered":"2024-01-16T14:59:49.167579+0000","last_clean":"2024-01-16T14:59:49.167579+0000","last_became_active":"2024-01-16T14:59:48.135643+0000","last_became_peered":"2024-01-16T14:59:48.135643+0000","last_unstale":"2024-01-16T14:59:49.167579+0000","last_undegraded":"2024-01-16T14:59:49.167579+0000","last_fullsized":"2024-01-16T14:59:49.167579+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.160574+0000","last_change":"2024-01-16T14:59:49.155345+0000","last_active":"2024-01-16T14:59:50.160574+0000","last_peered":"2024-01-16T14:59:50.160574+0000","last_clean":"2024-01-16T14:59:50.160574+0000","last_became_active":"2024-01-16T14:59:49.154578+0000","last_became_peered":"2024-01-16T14:59:49.154578+0000","last_unstale":"2024-01-16T14:59:50.160574+0000","last_undegraded":"2024-01-16T14:59:50.160574+0000","last_fullsized":"2024-01-16T14:59:50.160574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:56.971342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.175264+0000","last_change":"2024-01-16T14:59:48.129597+0000","last_active":"2024-01-16T14:59:49.175264+0000","last_peered":"2024-01-16T14:59:49.175264+0000","last_clean":"2024-01-16T14:59:49.175264+0000","last_became_active":"2024-01-16T14:59:48.128905+0000","last_became_peered":"2024-01-16T14:59:48.128905+0000","last_unstale":"2024-01-16T14:59:49.175264+0000","last_undegraded":"2024-01-16T14:59:49.175264+0000","last_fullsized":"2024-01-16T14:59:49.175264+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:33.060395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762725+0000","last_change":"2024-01-16T14:59:49.165421+0000","last_active":"2024-01-16T14:59:50.762725+0000","last_peered":"2024-01-16T14:59:50.762725+0000","last_clean":"2024-01-16T14:59:50.762725+0000","last_became_active":"2024-01-16T14:59:49.165085+0000","last_became_peered":"2024-01-16T14:59:49.165085+0000","last_unstale":"2024-01-16T14:59:50.762725+0000","last_undegraded":"2024-01-16T14:59:50.762725+0000","last_fullsized":"2024-01-16T14:59:50.762725+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:06.294703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225151+0000","last_change":"2024-01-16T14:59:48.128640+0000","last_active":"2024-01-16T14:59:49.225151+0000","last_peered":"2024-01-16T14:59:49.225151+0000","last_clean":"2024-01-16T14:59:49.225151+0000","last_became_active":"2024-01-16T14:59:48.128032+0000","last_became_peered":"2024-01-16T14:59:48.128032+0000","last_unstale":"2024-01-16T14:59:49.225151+0000","last_undegraded":"2024-01-16T14:59:49.225151+0000","last_fullsized":"2024-01-16T14:59:49.225151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:41.977214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441502+0000","last_change":"2024-01-16T14:59:49.162381+0000","last_active":"2024-01-16T14:59:50.441502+0000","last_peered":"2024-01-16T14:59:50.441502+0000","last_clean":"2024-01-16T14:59:50.441502+0000","last_became_active":"2024-01-16T14:59:49.162078+0000","last_became_peered":"2024-01-16T14:59:49.162078+0000","last_unstale":"2024-01-16T14:59:50.441502+0000","last_undegraded":"2024-01-16T14:59:50.441502+0000","last_fullsized":"2024-01-16T14:59:50.441502+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.130634+0000","last_change":"2024-01-16T14:59:48.126828+0000","last_active":"2024-01-16T14:59:49.130634+0000","last_peered":"2024-01-16T14:59:49.130634+0000","last_clean":"2024-01-16T14:59:49.130634+0000","last_became_active":"2024-01-16T14:59:48.126513+0000","last_became_peered":"2024-01-16T14:59:48.126513+0000","last_unstale":"2024-01-16T14:59:49.130634+0000","last_undegraded":"2024-01-16T14:59:49.130634+0000","last_fullsized":"2024-01-16T14:59:49.130634+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:58.612882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.760776+0000","last_change":"2024-01-16T14:59:49.150175+0000","last_active":"2024-01-16T14:59:50.760776+0000","last_peered":"2024-01-16T14:59:50.760776+0000","last_clean":"2024-01-16T14:59:50.760776+0000","last_became_active":"2024-01-16T14:59:49.149869+0000","last_became_peered":"2024-01-16T14:59:49.149869+0000","last_unstale":"2024-01-16T14:59:50.760776+0000","last_undegraded":"2024-01-16T14:59:50.760776+0000","last_fullsized":"2024-01-16T14:59:50.760776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:29.213011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225086+0000","last_change":"2024-01-16T14:59:48.129480+0000","last_active":"2024-01-16T14:59:49.225086+0000","last_peered":"2024-01-16T14:59:49.225086+0000","last_clean":"2024-01-16T14:59:49.225086+0000","last_became_active":"2024-01-16T14:59:48.129231+0000","last_became_peered":"2024-01-16T14:59:48.129231+0000","last_unstale":"2024-01-16T14:59:49.225086+0000","last_undegraded":"2024-01-16T14:59:49.225086+0000","last_fullsized":"2024-01-16T14:59:49.225086+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.161415+0000","last_change":"2024-01-16T14:59:49.159817+0000","last_active":"2024-01-16T14:59:50.161415+0000","last_peered":"2024-01-16T14:59:50.161415+0000","last_clean":"2024-01-16T14:59:50.161415+0000","last_became_active":"2024-01-16T14:59:49.159285+0000","last_became_peered":"2024-01-16T14:59:49.159285+0000","last_unstale":"2024-01-16T14:59:50.161415+0000","last_undegraded":"2024-01-16T14:59:50.161415+0000","last_fullsized":"2024-01-16T14:59:50.161415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:29.213011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.172152+0000","last_change":"2024-01-16T14:59:48.133029+0000","last_active":"2024-01-16T14:59:49.172152+0000","last_peered":"2024-01-16T14:59:49.172152+0000","last_clean":"2024-01-16T14:59:49.172152+0000","last_became_active":"2024-01-16T14:59:48.132732+0000","last_became_peered":"2024-01-16T14:59:48.132732+0000","last_unstale":"2024-01-16T14:59:49.172152+0000","last_undegraded":"2024-01-16T14:59:49.172152+0000","last_fullsized":"2024-01-16T14:59:49.172152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.156357+0000","last_change":"2024-01-16T14:59:49.160596+0000","last_active":"2024-01-16T14:59:50.156357+0000","last_peered":"2024-01-16T14:59:50.156357+0000","last_clean":"2024-01-16T14:59:50.156357+0000","last_became_active":"2024-01-16T14:59:49.160315+0000","last_became_peered":"2024-01-16T14:59:49.160315+0000","last_unstale":"2024-01-16T14:59:50.156357+0000","last_undegraded":"2024-01-16T14:59:50.156357+0000","last_fullsized":"2024-01-16T14:59:50.156357+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:32.794937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.164183+0000","last_change":"2024-01-16T14:59:48.129568+0000","last_active":"2024-01-16T14:59:49.164183+0000","last_peered":"2024-01-16T14:59:49.164183+0000","last_clean":"2024-01-16T14:59:49.164183+0000","last_became_active":"2024-01-16T14:59:48.129222+0000","last_became_peered":"2024-01-16T14:59:48.129222+0000","last_unstale":"2024-01-16T14:59:49.164183+0000","last_undegraded":"2024-01-16T14:59:49.164183+0000","last_fullsized":"2024-01-16T14:59:49.164183+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:40:44.538277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441723+0000","last_change":"2024-01-16T14:59:49.146836+0000","last_active":"2024-01-16T14:59:50.441723+0000","last_peered":"2024-01-16T14:59:50.441723+0000","last_clean":"2024-01-16T14:59:50.441723+0000","last_became_active":"2024-01-16T14:59:49.146538+0000","last_became_peered":"2024-01-16T14:59:49.146538+0000","last_unstale":"2024-01-16T14:59:50.441723+0000","last_undegraded":"2024-01-16T14:59:50.441723+0000","last_fullsized":"2024-01-16T14:59:50.441723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:55.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167674+0000","last_change":"2024-01-16T14:59:48.128976+0000","last_active":"2024-01-16T14:59:49.167674+0000","last_peered":"2024-01-16T14:59:49.167674+0000","last_clean":"2024-01-16T14:59:49.167674+0000","last_became_active":"2024-01-16T14:59:48.128687+0000","last_became_peered":"2024-01-16T14:59:48.128687+0000","last_unstale":"2024-01-16T14:59:49.167674+0000","last_undegraded":"2024-01-16T14:59:49.167674+0000","last_fullsized":"2024-01-16T14:59:49.167674+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:26.034426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.763835+0000","last_change":"2024-01-16T14:59:49.157272+0000","last_active":"2024-01-16T14:59:50.763835+0000","last_peered":"2024-01-16T14:59:50.763835+0000","last_clean":"2024-01-16T14:59:50.763835+0000","last_became_active":"2024-01-16T14:59:49.156988+0000","last_became_peered":"2024-01-16T14:59:49.156988+0000","last_unstale":"2024-01-16T14:59:50.763835+0000","last_undegraded":"2024-01-16T14:59:50.763835+0000","last_fullsized":"2024-01-16T14:59:50.763835+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:50.375876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.165622+0000","last_change":"2024-01-16T14:59:48.134109+0000","last_active":"2024-01-16T14:59:49.165622+0000","last_peered":"2024-01-16T14:59:49.165622+0000","last_clean":"2024-01-16T14:59:49.165622+0000","last_became_active":"2024-01-16T14:59:48.133228+0000","last_became_peered":"2024-01-16T14:59:48.133228+0000","last_unstale":"2024-01-16T14:59:49.165622+0000","last_undegraded":"2024-01-16T14:59:49.165622+0000","last_fullsized":"2024-01-16T14:59:49.165622+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:05.749031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.763950+0000","last_change":"2024-01-16T14:59:49.168099+0000","last_active":"2024-01-16T14:59:50.763950+0000","last_peered":"2024-01-16T14:59:50.763950+0000","last_clean":"2024-01-16T14:59:50.763950+0000","last_became_active":"2024-01-16T14:59:49.167461+0000","last_became_peered":"2024-01-16T14:59:49.167461+0000","last_unstale":"2024-01-16T14:59:50.763950+0000","last_undegraded":"2024-01-16T14:59:50.763950+0000","last_fullsized":"2024-01-16T14:59:50.763950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:16.518645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225194+0000","last_change":"2024-01-16T14:59:48.137888+0000","last_active":"2024-01-16T14:59:49.225194+0000","last_peered":"2024-01-16T14:59:49.225194+0000","last_clean":"2024-01-16T14:59:49.225194+0000","last_became_active":"2024-01-16T14:59:48.137587+0000","last_became_peered":"2024-01-16T14:59:48.137587+0000","last_unstale":"2024-01-16T14:59:49.225194+0000","last_undegraded":"2024-01-16T14:59:49.225194+0000","last_fullsized":"2024-01-16T14:59:49.225194+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:53.615671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761336+0000","last_change":"2024-01-16T14:59:49.149041+0000","last_active":"2024-01-16T14:59:50.761336+0000","last_peered":"2024-01-16T14:59:50.761336+0000","last_clean":"2024-01-16T14:59:50.761336+0000","last_became_active":"2024-01-16T14:59:49.148799+0000","last_became_peered":"2024-01-16T14:59:49.148799+0000","last_unstale":"2024-01-16T14:59:50.761336+0000","last_undegraded":"2024-01-16T14:59:50.761336+0000","last_fullsized":"2024-01-16T14:59:50.761336+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:52.166570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.225161+0000","last_change":"2024-01-16T14:59:48.127494+0000","last_active":"2024-01-16T14:59:49.225161+0000","last_peered":"2024-01-16T14:59:49.225161+0000","last_clean":"2024-01-16T14:59:49.225161+0000","last_became_active":"2024-01-16T14:59:48.127232+0000","last_became_peered":"2024-01-16T14:59:48.127232+0000","last_unstale":"2024-01-16T14:59:49.225161+0000","last_undegraded":"2024-01-16T14:59:49.225161+0000","last_fullsized":"2024-01-16T14:59:49.225161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:20.544629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.762932+0000","last_change":"2024-01-16T14:59:49.162106+0000","last_active":"2024-01-16T14:59:50.762932+0000","last_peered":"2024-01-16T14:59:50.762932+0000","last_clean":"2024-01-16T14:59:50.762932+0000","last_became_active":"2024-01-16T14:59:49.161715+0000","last_became_peered":"2024-01-16T14:59:49.161715+0000","last_unstale":"2024-01-16T14:59:50.762932+0000","last_undegraded":"2024-01-16T14:59:50.762932+0000","last_fullsized":"2024-01-16T14:59:50.762932+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:34.476071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.154088+0000","last_change":"2024-01-16T14:59:48.139613+0000","last_active":"2024-01-16T14:59:49.154088+0000","last_peered":"2024-01-16T14:59:49.154088+0000","last_clean":"2024-01-16T14:59:49.154088+0000","last_became_active":"2024-01-16T14:59:48.139289+0000","last_became_peered":"2024-01-16T14:59:48.139289+0000","last_unstale":"2024-01-16T14:59:49.154088+0000","last_undegraded":"2024-01-16T14:59:49.154088+0000","last_fullsized":"2024-01-16T14:59:49.154088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:43.153411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.156175+0000","last_change":"2024-01-16T14:59:49.158499+0000","last_active":"2024-01-16T14:59:50.156175+0000","last_peered":"2024-01-16T14:59:50.156175+0000","last_clean":"2024-01-16T14:59:50.156175+0000","last_became_active":"2024-01-16T14:59:49.158237+0000","last_became_peered":"2024-01-16T14:59:49.158237+0000","last_unstale":"2024-01-16T14:59:50.156175+0000","last_undegraded":"2024-01-16T14:59:50.156175+0000","last_fullsized":"2024-01-16T14:59:50.156175+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:35.310451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142059+0000","last_change":"2024-01-16T14:59:48.127338+0000","last_active":"2024-01-16T14:59:49.142059+0000","last_peered":"2024-01-16T14:59:49.142059+0000","last_clean":"2024-01-16T14:59:49.142059+0000","last_became_active":"2024-01-16T14:59:48.127065+0000","last_became_peered":"2024-01-16T14:59:48.127065+0000","last_unstale":"2024-01-16T14:59:49.142059+0000","last_undegraded":"2024-01-16T14:59:49.142059+0000","last_fullsized":"2024-01-16T14:59:49.142059+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:49.361224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441018+0000","last_change":"2024-01-16T14:59:49.150886+0000","last_active":"2024-01-16T14:59:50.441018+0000","last_peered":"2024-01-16T14:59:50.441018+0000","last_clean":"2024-01-16T14:59:50.441018+0000","last_became_active":"2024-01-16T14:59:49.150599+0000","last_became_peered":"2024-01-16T14:59:49.150599+0000","last_unstale":"2024-01-16T14:59:50.441018+0000","last_undegraded":"2024-01-16T14:59:50.441018+0000","last_fullsized":"2024-01-16T14:59:50.441018+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:45.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.130446+0000","last_change":"2024-01-16T14:59:48.126842+0000","last_active":"2024-01-16T14:59:49.130446+0000","last_peered":"2024-01-16T14:59:49.130446+0000","last_clean":"2024-01-16T14:59:49.130446+0000","last_became_active":"2024-01-16T14:59:48.126505+0000","last_became_peered":"2024-01-16T14:59:48.126505+0000","last_unstale":"2024-01-16T14:59:49.130446+0000","last_undegraded":"2024-01-16T14:59:49.130446+0000","last_fullsized":"2024-01-16T14:59:49.130446+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:44.655685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.764463+0000","last_change":"2024-01-16T14:59:49.166624+0000","last_active":"2024-01-16T14:59:50.764463+0000","last_peered":"2024-01-16T14:59:50.764463+0000","last_clean":"2024-01-16T14:59:50.764463+0000","last_became_active":"2024-01-16T14:59:49.166217+0000","last_became_peered":"2024-01-16T14:59:49.166217+0000","last_unstale":"2024-01-16T14:59:50.764463+0000","last_undegraded":"2024-01-16T14:59:50.764463+0000","last_fullsized":"2024-01-16T14:59:50.764463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:40.886551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761639+0000","last_change":"2024-01-16T14:59:48.130422+0000","last_active":"2024-01-16T14:59:50.761639+0000","last_peered":"2024-01-16T14:59:50.761639+0000","last_clean":"2024-01-16T14:59:50.761639+0000","last_became_active":"2024-01-16T14:59:48.130133+0000","last_became_peered":"2024-01-16T14:59:48.130133+0000","last_unstale":"2024-01-16T14:59:50.761639+0000","last_undegraded":"2024-01-16T14:59:50.761639+0000","last_fullsized":"2024-01-16T14:59:50.761639+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:55.956689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.441776+0000","last_change":"2024-01-16T14:59:49.166387+0000","last_active":"2024-01-16T14:59:50.441776+0000","last_peered":"2024-01-16T14:59:50.441776+0000","last_clean":"2024-01-16T14:59:50.441776+0000","last_became_active":"2024-01-16T14:59:49.166065+0000","last_became_peered":"2024-01-16T14:59:49.166065+0000","last_unstale":"2024-01-16T14:59:50.441776+0000","last_undegraded":"2024-01-16T14:59:50.441776+0000","last_fullsized":"2024-01-16T14:59:50.441776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:42.991867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.134423+0000","last_change":"2024-01-16T14:59:48.138337+0000","last_active":"2024-01-16T14:59:49.134423+0000","last_peered":"2024-01-16T14:59:49.134423+0000","last_clean":"2024-01-16T14:59:49.134423+0000","last_became_active":"2024-01-16T14:59:48.137540+0000","last_became_peered":"2024-01-16T14:59:48.137540+0000","last_unstale":"2024-01-16T14:59:49.134423+0000","last_undegraded":"2024-01-16T14:59:49.134423+0000","last_fullsized":"2024-01-16T14:59:49.134423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:52.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.160698+0000","last_change":"2024-01-16T14:59:49.144692+0000","last_active":"2024-01-16T14:59:50.160698+0000","last_peered":"2024-01-16T14:59:50.160698+0000","last_clean":"2024-01-16T14:59:50.160698+0000","last_became_active":"2024-01-16T14:59:49.144193+0000","last_became_peered":"2024-01-16T14:59:49.144193+0000","last_unstale":"2024-01-16T14:59:50.160698+0000","last_undegraded":"2024-01-16T14:59:50.160698+0000","last_fullsized":"2024-01-16T14:59:50.160698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:40.368051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.215602+0000","last_change":"2024-01-16T14:59:48.134570+0000","last_active":"2024-01-16T14:59:49.215602+0000","last_peered":"2024-01-16T14:59:49.215602+0000","last_clean":"2024-01-16T14:59:49.215602+0000","last_became_active":"2024-01-16T14:59:48.133797+0000","last_became_peered":"2024-01-16T14:59:48.133797+0000","last_unstale":"2024-01-16T14:59:49.215602+0000","last_undegraded":"2024-01-16T14:59:49.215602+0000","last_fullsized":"2024-01-16T14:59:49.215602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:14:12.580412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761677+0000","last_change":"2024-01-16T14:59:49.147430+0000","last_active":"2024-01-16T14:59:50.761677+0000","last_peered":"2024-01-16T14:59:50.761677+0000","last_clean":"2024-01-16T14:59:50.761677+0000","last_became_active":"2024-01-16T14:59:49.147118+0000","last_became_peered":"2024-01-16T14:59:49.147118+0000","last_unstale":"2024-01-16T14:59:50.761677+0000","last_undegraded":"2024-01-16T14:59:50.761677+0000","last_fullsized":"2024-01-16T14:59:50.761677+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:27.485442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.131876+0000","last_change":"2024-01-16T14:59:48.134611+0000","last_active":"2024-01-16T14:59:49.131876+0000","last_peered":"2024-01-16T14:59:49.131876+0000","last_clean":"2024-01-16T14:59:49.131876+0000","last_became_active":"2024-01-16T14:59:48.134321+0000","last_became_peered":"2024-01-16T14:59:48.134321+0000","last_unstale":"2024-01-16T14:59:49.131876+0000","last_undegraded":"2024-01-16T14:59:49.131876+0000","last_fullsized":"2024-01-16T14:59:49.131876+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:05.749031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761456+0000","last_change":"2024-01-16T14:59:49.149860+0000","last_active":"2024-01-16T14:59:50.761456+0000","last_peered":"2024-01-16T14:59:50.761456+0000","last_clean":"2024-01-16T14:59:50.761456+0000","last_became_active":"2024-01-16T14:59:49.149557+0000","last_became_peered":"2024-01-16T14:59:49.149557+0000","last_unstale":"2024-01-16T14:59:50.761456+0000","last_undegraded":"2024-01-16T14:59:50.761456+0000","last_fullsized":"2024-01-16T14:59:50.761456+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:49.692016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167559+0000","last_change":"2024-01-16T14:59:48.131348+0000","last_active":"2024-01-16T14:59:49.167559+0000","last_peered":"2024-01-16T14:59:49.167559+0000","last_clean":"2024-01-16T14:59:49.167559+0000","last_became_active":"2024-01-16T14:59:48.131045+0000","last_became_peered":"2024-01-16T14:59:48.131045+0000","last_unstale":"2024-01-16T14:59:49.167559+0000","last_undegraded":"2024-01-16T14:59:49.167559+0000","last_fullsized":"2024-01-16T14:59:49.167559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:15.503992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.163462+0000","last_change":"2024-01-16T14:59:49.165904+0000","last_active":"2024-01-16T14:59:50.163462+0000","last_peered":"2024-01-16T14:59:50.163462+0000","last_clean":"2024-01-16T14:59:50.163462+0000","last_became_active":"2024-01-16T14:59:49.165635+0000","last_became_peered":"2024-01-16T14:59:49.165635+0000","last_unstale":"2024-01-16T14:59:50.163462+0000","last_undegraded":"2024-01-16T14:59:50.163462+0000","last_fullsized":"2024-01-16T14:59:50.163462+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:52.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.215578+0000","last_change":"2024-01-16T14:59:48.136397+0000","last_active":"2024-01-16T14:59:49.215578+0000","last_peered":"2024-01-16T14:59:49.215578+0000","last_clean":"2024-01-16T14:59:49.215578+0000","last_became_active":"2024-01-16T14:59:48.136052+0000","last_became_peered":"2024-01-16T14:59:48.136052+0000","last_unstale":"2024-01-16T14:59:49.215578+0000","last_undegraded":"2024-01-16T14:59:49.215578+0000","last_fullsized":"2024-01-16T14:59:49.215578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:47.000083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.761640+0000","last_change":"2024-01-16T14:59:49.157313+0000","last_active":"2024-01-16T14:59:50.761640+0000","last_peered":"2024-01-16T14:59:50.761640+0000","last_clean":"2024-01-16T14:59:50.761640+0000","last_became_active":"2024-01-16T14:59:49.156979+0000","last_became_peered":"2024-01-16T14:59:49.156979+0000","last_unstale":"2024-01-16T14:59:50.761640+0000","last_undegraded":"2024-01-16T14:59:50.761640+0000","last_fullsized":"2024-01-16T14:59:50.761640+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:31.120308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.142273+0000","last_change":"2024-01-16T14:59:48.131399+0000","last_active":"2024-01-16T14:59:49.142273+0000","last_peered":"2024-01-16T14:59:49.142273+0000","last_clean":"2024-01-16T14:59:49.142273+0000","last_became_active":"2024-01-16T14:59:48.131079+0000","last_became_peered":"2024-01-16T14:59:48.131079+0000","last_unstale":"2024-01-16T14:59:49.142273+0000","last_undegraded":"2024-01-16T14:59:49.142273+0000","last_fullsized":"2024-01-16T14:59:49.142273+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:38.988772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T14:59:50.442001+0000","last_change":"2024-01-16T14:59:49.162852+0000","last_active":"2024-01-16T14:59:50.442001+0000","last_peered":"2024-01-16T14:59:50.442001+0000","last_clean":"2024-01-16T14:59:50.442001+0000","last_became_active":"2024-01-16T14:59:49.162358+0000","last_became_peered":"2024-01-16T14:59:49.162358+0000","last_unstale":"2024-01-16T14:59:50.442001+0000","last_undegraded":"2024-01-16T14:59:50.442001+0000","last_fullsized":"2024-01-16T14:59:50.442001+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:48.095777+0000","last_clean_scrub_stamp":"2024-01-16T14:59:48.095777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:21.559282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.167776+0000","last_change":"2024-01-16T14:59:48.132319+0000","last_active":"2024-01-16T14:59:49.167776+0000","last_peered":"2024-01-16T14:59:49.167776+0000","last_clean":"2024-01-16T14:59:49.167776+0000","last_became_active":"2024-01-16T14:59:48.132030+0000","last_became_peered":"2024-01-16T14:59:48.132030+0000","last_unstale":"2024-01-16T14:59:49.167776+0000","last_undegraded":"2024-01-16T14:59:49.167776+0000","last_fullsized":"2024-01-16T14:59:49.167776+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:47.081125+0000","last_clean_scrub_stamp":"2024-01-16T14:59:47.081125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:48.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":191,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T14:59:49.130371+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T14:59:49.130371+0000","last_peered":"2024-01-16T14:59:49.130371+0000","last_clean":"2024-01-16T14:59:49.130371+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T14:59:49.130371+0000","last_undegraded":"2024-01-16T14:59:49.130371+0000","last_fullsized":"2024-01-16T14:59:49.130371+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":805719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0660000000000001,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.335,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.22,"15min":1.1319999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":0.92400000000000004}]},{"osd":1,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1359999999999999,"15min":1.105},"min":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5629999999999999,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.222,"15min":1.216},"min":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.746,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.319}]},{"osd":2,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2050000000000001,"15min":1.1919999999999999},"min":{"1min":0.74299999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.278,"5min":2.8140000000000001,"15min":2.8140000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.254,"15min":1.22},"min":{"1min":0.877,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4369999999999998,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":1.2789999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2190000000000001,"15min":1.2230000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.419,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":5.7969999999999997,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.751}]},{"osd":4,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.425,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.1859999999999999},{"interface":"front","average":{"1min":3.234,"5min":1.7110000000000001,"15min":1.4870000000000001},"min":{"1min":0.82799999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":36.374000000000002,"5min":36.374000000000002,"15min":36.374000000000002},"last":1.385}]},{"osd":5,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.407,"15min":1.361},"min":{"1min":0.61299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.9889999999999999,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":1.139},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.431,"15min":1.4099999999999999},"min":{"1min":0.69699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.5139999999999998,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.427}]},{"osd":6,"last update":"Tue Jan 16 14:59:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"min":{"1min":0.879,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.4849999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.601},"min":{"1min":0.628,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.5230000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.46}]}]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":215933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.087},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.254,"15min":1.24},"min":{"1min":0.75800000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.23}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.081,"15min":1.0549999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.143,"15min":1.1339999999999999},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.643,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.70399999999999996}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2230000000000001,"15min":1.2110000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.4770000000000001}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.367,"15min":1.377},"min":{"1min":0.97399999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.282}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4159999999999999,"15min":1.4379999999999999},"min":{"1min":0.89300000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8640000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.4419999999999999,"15min":1.468},"min":{"1min":0.91500000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.1279999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0169999999999999,"5min":2.202,"15min":2.202},"last":1.373},{"interface":"front","average":{"1min":1.702,"5min":1.6000000000000001,"15min":1.579},"min":{"1min":1.1839999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.597,"15min":1.591},"min":{"1min":1.008,"5min":1,"15min":1},"max":{"1min":4.3550000000000004,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.5600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":215331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":1.0369999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.216,"15min":1.147},"min":{"1min":0.76100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.9780000000000002,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.3740000000000001,"15min":1.2490000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.6500000000000004,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.708},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1830000000000001,"15min":1.167},"min":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.083}]},{"osd":2,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.272,"15min":1.3089999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.319,"15min":1.2669999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.272}]},{"osd":3,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3480000000000001,"15min":1.3660000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":1.212},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3660000000000001,"15min":1.3520000000000001},"min":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.4770000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4620000000000002,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.4810000000000001,"15min":1.496},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.77}]},{"osd":6,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.492,"15min":1.516},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":1.605},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.431,"15min":1.3859999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":10.015000000000001,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.524,"15min":1.5660000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.583,"15min":1.5740000000000001},"min":{"1min":0.94699999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.671}]}]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":216873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2330000000000001,"15min":1.1870000000000001},"min":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9550000000000001,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":1.53},{"interface":"front","average":{"1min":1.383,"5min":1.27,"15min":1.252},"min":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.125}]},{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.141,"15min":1.089},"min":{"1min":0.76900000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.145,"15min":1.133},"min":{"1min":1.01,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.056}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.2789999999999999,"15min":1.2010000000000001},"min":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":8.0579999999999998,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":1.343},{"interface":"front","average":{"1min":1.349,"5min":1.298,"15min":1.3},"min":{"1min":0.96699999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.228}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2669999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2280000000000002,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":6.327,"5min":7.0650000000000004,"15min":7.0650000000000004},"last":1.5900000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.304},{"interface":"front","average":{"1min":1.929,"5min":1.5720000000000001,"15min":1.5600000000000001},"min":{"1min":0.92900000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.289999999999999,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.1719999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.173,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5660000000000001,"15min":1.6020000000000001},"min":{"1min":0.72299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.2770000000000001,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.377}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.6259999999999999},"min":{"1min":1.02,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3580000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.268},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.7210000000000001,"15min":1.6950000000000001},"min":{"1min":0.92100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":3.8300000000000001,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.722}]}]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":215899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.91500000000000004,"15min":0.91900000000000004},"min":{"1min":0.57199999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.625,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.97799999999999998,"15min":0.93200000000000005},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.335,"5min":6.6669999999999998,"15min":6.6669999999999998},"last":0.62}]},{"osd":1,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.56100000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.753},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0700000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7370000000000001,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.181}]},{"osd":2,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.9930000000000001,"5min":6.734,"15min":6.734},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.67600000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.0529999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.3540000000000001},"min":{"1min":0.78100000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.6339999999999999,"5min":6.9640000000000004,"15min":6.9640000000000004},"last":1.333},{"interface":"front","average":{"1min":1.262,"5min":1.3200000000000001,"15min":1.3440000000000001},"min":{"1min":0.82499999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6319999999999999,"5min":2.762,"15min":2.762},"last":1.284}]},{"osd":5,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4179999999999999,"15min":1.466},"min":{"1min":0.88,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0659999999999998,"5min":3.169,"15min":3.169},"last":1.425},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4359999999999999,"15min":1.4730000000000001},"min":{"1min":0.93700000000000006,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.956,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.595}]},{"osd":6,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4350000000000001,"15min":1.4099999999999999},"min":{"1min":0.95599999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.716,"5min":6.8460000000000001,"15min":6.8460000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.4379999999999999,"15min":1.4690000000000001},"min":{"1min":0.92800000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.835,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.3779999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:59:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.482,"15min":1.448},"min":{"1min":0.92000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.865,"5min":6.9130000000000003,"15min":6.9130000000000003},"last":1.232},{"interface":"front","average":{"1min":1.615,"5min":1.5049999999999999,"15min":1.51},"min":{"1min":0.88400000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":1.4850000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":216895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.72099999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.3149999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.069,"15min":1.105},"min":{"1min":0.56499999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.5760000000000001,"5min":5.9889999999999999,"15min":5.9889999999999999},"last":1.153}]},{"osd":1,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.23},"min":{"1min":0.68200000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.718,"5min":6.1130000000000004,"15min":6.1130000000000004},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.27,"15min":1.3480000000000001},"min":{"1min":0.54600000000000004,"5min":0.373,"15min":0.373},"max":{"1min":2.6309999999999998,"5min":6.0259999999999998,"15min":6.0259999999999998},"last":0.96199999999999997}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.2509999999999999,"5min":6.141,"15min":6.141},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.331,"15min":1.397},"min":{"1min":0.88700000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.889,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.089}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9619999999999997,"5min":5.2009999999999996,"15min":5.2009999999999996},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4039999999999999,"15min":1.4399999999999999},"min":{"1min":0.71999999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.3679999999999999,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":1.044}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.5009999999999999,"15min":1.5509999999999999},"min":{"1min":1.024,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.2069999999999999,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.198},{"interface":"front","average":{"1min":1.524,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.1479999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.8290000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.375}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4550000000000001,"15min":1.4870000000000001},"min":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1150000000000002,"5min":5.024,"15min":5.024},"last":1.26},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6000000000000001,"15min":1.655},"min":{"1min":1.1140000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.3340000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.429}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6000000000000001,"15min":1.6559999999999999},"min":{"1min":1.196,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.7970000000000002,"5min":5.1660000000000004,"15min":5.1660000000000004},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5680000000000001,"15min":1.617},"min":{"1min":0.81399999999999995,"5min":0.754,"15min":0.754},"max":{"1min":2.7240000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":1.393}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":215331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.54500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.145,"5min":0.96899999999999997,"15min":0.88500000000000001},"min":{"1min":0.51300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.54,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.76900000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.90700000000000003,"15min":0.91200000000000003},"min":{"1min":0.64400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.022},{"interface":"front","average":{"1min":1.27,"5min":1.107,"15min":1.089},"min":{"1min":0.71599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.169}]},{"osd":3,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2370000000000001,"15min":1.1870000000000001},"min":{"1min":0.753,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.248},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.1479999999999999,"15min":1.044},"min":{"1min":0.67500000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.3530000000000002,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":0.65300000000000002}]},{"osd":4,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.4790000000000001,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.323},{"interface":"front","average":{"1min":1.833,"5min":1.4239999999999999,"15min":1.3580000000000001},"min":{"1min":0.85499999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.9480000000000004,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.3}]},{"osd":5,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.454,"15min":1.4039999999999999},"min":{"1min":1.1160000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":3.3250000000000002,"5min":4.8550000000000004,"15min":4.8550000000000004},"last":1.421},{"interface":"front","average":{"1min":1.492,"5min":1.411,"15min":1.421},"min":{"1min":1.0900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.2110000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.53,"15min":1.492},"min":{"1min":1.0980000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":5.798,"5min":5.798,"15min":5.798},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.456,"15min":1.4039999999999999},"min":{"1min":0.95199999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.0250000000000004,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.1000000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:59:41 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.534,"15min":1.504},"min":{"1min":0.90000000000000002,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":1.268},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.458,"15min":1.462},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.45}]}]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1122304,"data_stored":804644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":1.421},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.93500000000000005,"15min":0.93700000000000006},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":0.72599999999999998}]},{"osd":2,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0189999999999999,"15min":1.054},"min":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4690000000000001,"5min":1.587,"15min":1.587},"last":1.337},{"interface":"front","average":{"1min":1.075,"5min":1,"15min":1},"min":{"1min":0.53400000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":0.60099999999999998}]},{"osd":3,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.998,"15min":0.96799999999999997},"min":{"1min":0.45100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.198,"15min":1.228},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.1659999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.002},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.24},"min":{"1min":0.72099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.2150000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.4950000000000001},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4399999999999999,"15min":1.4350000000000001},"min":{"1min":0.89200000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.9429999999999996},"last":1.3080000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":1.127},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.488,"15min":1.5129999999999999},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.264}]},{"osd":7,"last update":"Tue Jan 16 14:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.423,"15min":1.4610000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.468,"15min":1.466},"min":{"1min":0.98799999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.387}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:59.026 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:59:59.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T14:59:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.306+0000 7f05aee00640 1 Processor -- start 2024-01-16T14:59:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.306+0000 7f05aee00640 1 -- start start 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.307+0000 7f05aee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f05a8157dc0 0x7f05a8070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05aee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a815ba40 con 0x7f05a8071230 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05aee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a815eb10 con 0x7f05a815e450 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05aee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a81581a0 con 0x7f05a8157dc0 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f059f200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f05a8071230 0x7f05a815dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51788/0 (socket says 172.21.15.125:51788) 2024-01-16T14:59:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f059f200640 1 -- 172.21.15.125:0/3645638562 learned_addr learned my addr 172.21.15.125:0/3645638562 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300327676 0 0) 0x7f05a815ba40 con 0x7f05a8071230 2024-01-16T14:59:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588003520 con 0x7f05a8071230 2024-01-16T14:59:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.308+0000 7f05a7e00640 1 --2- 172.21.15.125:0/3645638562 >> v2:172.21.15.137:3300/0 conn(0x7f05a8157dc0 0x7f05a8070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3888097705 0 0) 0x7f0588003520 con 0x7f05a8071230 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 >> v2:172.21.15.137:3300/0 conn(0x7f05a8157dc0 msgr2=0x7f05a8070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 --2- 172.21.15.125:0/3645638562 >> v2:172.21.15.137:3300/0 conn(0x7f05a8157dc0 0x7f05a8070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 >> v1:172.21.15.137:6789/0 conn(0x7f05a815e450 legacy=0x7f05a815bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8071860 con 0x7f05a8071230 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0590001670 con 0x7f05a8071230 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 >> v1:172.21.15.125:6789/0 conn(0x7f05a8071230 legacy=0x7f05a815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05880039c0 0x7f0588003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 0x7f0588004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880051d0 0x7f058800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81581a0 con 0x7f05880047c0 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058800c120 con 0x7f05880051d0 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058800e4f0 con 0x7f05880039c0 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f059f200640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 0x7f0588004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.309+0000 7f05a7e00640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05880039c0 0x7f0588003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f05ace00640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880051d0 0x7f058800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f059f200640 1 -- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05880039c0 msgr2=0x7f0588003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f059f200640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05880039c0 0x7f0588003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f059f200640 1 -- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880051d0 msgr2=0x7f058800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f059f200640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880051d0 0x7f058800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f05ace00640 1 -- 172.21.15.125:0/3645638562 reap_dead start 2024-01-16T14:59:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.310+0000 7f059f200640 1 -- 172.21.15.125:0/3645638562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588003520 con 0x7f05880047c0 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f059f200640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 0x7f0588004c40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f059000c0e0 tx=0x7f05880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f725e6f9d62fe49 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590005ad0 con 0x7f05880047c0 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0590005c30 con 0x7f05880047c0 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f05a7400640 1 -- 172.21.15.125:0/3645638562 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059002ca50 con 0x7f05880047c0 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f05aee00640 1 -- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 msgr2=0x7f0588004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.311+0000 7f05aee00640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 0x7f0588004c40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f059000c0e0 tx=0x7f05880057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.312+0000 7f05aee00640 1 -- 172.21.15.125:0/3645638562 shutdown_connections 2024-01-16T14:59:59.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.312+0000 7f05aee00640 1 --2- 172.21.15.125:0/3645638562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880047c0 0x7f0588004c40 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.312+0000 7f05aee00640 1 -- 172.21.15.125:0/3645638562 >> 172.21.15.125:0/3645638562 conn(0x7f05a80b0be0 msgr2=0x7f05a80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.312+0000 7f05aee00640 1 -- 172.21.15.125:0/3645638562 shutdown_connections 2024-01-16T14:59:59.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.312+0000 7f05aee00640 1 -- 172.21.15.125:0/3645638562 wait complete. 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.313+0000 7f05aee00640 1 Processor -- start 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 -- start start 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880047c0 0x7f05a8156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05a80ce1b0 0x7f05a81447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8147b90 con 0x7f05a81471a0 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8156da0 con 0x7f05880047c0 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.314+0000 7f05aee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8145290 con 0x7f05a80ce1b0 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53754/0 (socket says 172.21.15.125:53754) 2024-01-16T14:59:59.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 -- 172.21.15.125:0/2484557980 learned_addr learned my addr 172.21.15.125:0/2484557980 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f059f200640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05a80ce1b0 0x7f05a81447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05a7e00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880047c0 0x7f05a8156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 -- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05a80ce1b0 msgr2=0x7f05a81447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05a80ce1b0 0x7f05a81447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 -- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880047c0 msgr2=0x7f05a8156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880047c0 0x7f05a8156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 -- 172.21.15.125:0/2484557980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80ce920 con 0x7f05a81471a0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.315+0000 7f05ace00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f05a80726c0 tx=0x7f05a000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.316+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0005280 con 0x7f05a81471a0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.316+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a00053e0 con 0x7f05a81471a0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.316+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0007270 con 0x7f05a81471a0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.316+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80cec00 con 0x7f05a81471a0 2024-01-16T14:59:59.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.316+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80ce630 con 0x7f05a81471a0 2024-01-16T14:59:59.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.317+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a81454b0 con 0x7f05a81471a0 2024-01-16T14:59:59.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.318+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f05a00073d0 con 0x7f05a81471a0 2024-01-16T14:59:59.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.319+0000 7f05a4c00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 0x7f058006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.319+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a0089230 con 0x7f05a81471a0 2024-01-16T14:59:59.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.319+0000 7f05a7e00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 0x7f058006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.320+0000 7f05a7e00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 0x7f058006d9f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f05a8067b00 tx=0x7f059800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.323+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05a81454b0 con 0x7f05a81471a0 2024-01-16T14:59:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.576+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f05a80cf120 con 0x7f05a81471a0 2024-01-16T14:59:59.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.580+0000 7f05a4c00640 1 -- 172.21.15.125:0/2484557980 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f05a0053ca0 con 0x7f05a81471a0 2024-01-16T14:59:59.560 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T14:59:59.560 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T14:59:59.560 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T14:59:59.560 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T14:59:59.561 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T14:59:59.561 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 msgr2=0x7f058006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 0x7f058006d9f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f05a8067b00 tx=0x7f059800b040 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 msgr2=0x7f05a8147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f05a80726c0 tx=0x7f05a000e8a0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 shutdown_connections 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f058006b530 0x7f058006d9f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05a81471a0 0x7f05a8147650 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05a80ce1b0 0x7f05a81447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 --2- 172.21.15.125:0/2484557980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05880047c0 0x7f05a8156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.582+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 >> 172.21.15.125:0/2484557980 conn(0x7f05a80b0be0 msgr2=0x7f05a815bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.583+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 shutdown_connections 2024-01-16T14:59:59.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.583+0000 7f05aee00640 1 -- 172.21.15.125:0/2484557980 wait complete. 2024-01-16T14:59:59.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.789+0000 7f407d600640 1 Processor -- start 2024-01-16T14:59:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.789+0000 7f407d600640 1 -- start start 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.790+0000 7f407d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f407813df80 0x7f407815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.790+0000 7f407d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f407813e360 0x7f40780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f407d600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407806c470 con 0x7f407815f9c0 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f407d600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407806c750 con 0x7f407813e360 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f407d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407806ca30 con 0x7f407813df80 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4077400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f407815f9c0 0x7f40781408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51792/0 (socket says 172.21.15.125:51792) 2024-01-16T14:59:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4077400640 1 -- 172.21.15.125:0/2124501416 learned_addr learned my addr 172.21.15.125:0/2124501416 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T14:59:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4076000640 1 --2- 172.21.15.125:0/2124501416 >> v2:172.21.15.125:3300/0 conn(0x7f407813e360 0x7f40780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4076a00640 1 --2- 172.21.15.125:0/2124501416 >> v2:172.21.15.137:3300/0 conn(0x7f407813df80 0x7f407815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4075600640 1 -- 172.21.15.125:0/2124501416 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579631675 0 0) 0x7f407806c470 con 0x7f407815f9c0 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4075600640 1 -- 172.21.15.125:0/2124501416 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4058003520 con 0x7f407815f9c0 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4076000640 1 -- 172.21.15.125:0/2124501416 >> v2:172.21.15.137:3300/0 conn(0x7f407813df80 msgr2=0x7f407815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4076000640 1 --2- 172.21.15.125:0/2124501416 >> v2:172.21.15.137:3300/0 conn(0x7f407813df80 0x7f407815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.791+0000 7f4076000640 1 -- 172.21.15.125:0/2124501416 >> v1:172.21.15.125:6789/0 conn(0x7f407815f9c0 legacy=0x7f40781408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.792+0000 7f4076000640 1 -- 172.21.15.125:0/2124501416 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407806cd10 con 0x7f407813e360 2024-01-16T14:59:59.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f4076000640 1 --2- 172.21.15.125:0/2124501416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407813e360 0x7f40780cf290 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4078072770 tx=0x7f406002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1226e37125a0e5ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f4075600640 1 -- 172.21.15.125:0/2124501416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031830 con 0x7f407813e360 2024-01-16T14:59:59.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f4075600640 1 -- 172.21.15.125:0/2124501416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060031990 con 0x7f407813e360 2024-01-16T14:59:59.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f4075600640 1 -- 172.21.15.125:0/2124501416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031d20 con 0x7f407813e360 2024-01-16T14:59:59.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f407d600640 1 -- 172.21.15.125:0/2124501416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407813e360 msgr2=0x7f40780cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.793+0000 7f407d600640 1 --2- 172.21.15.125:0/2124501416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407813e360 0x7f40780cf290 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4078072770 tx=0x7f406002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 -- 172.21.15.125:0/2124501416 shutdown_connections 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 --2- 172.21.15.125:0/2124501416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407813e360 0x7f40780cf290 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 --2- 172.21.15.125:0/2124501416 >> v2:172.21.15.137:3300/0 conn(0x7f407813df80 0x7f407815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 -- 172.21.15.125:0/2124501416 >> 172.21.15.125:0/2124501416 conn(0x7f407806a800 msgr2=0x7f407806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 -- 172.21.15.125:0/2124501416 shutdown_connections 2024-01-16T14:59:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.794+0000 7f407d600640 1 -- 172.21.15.125:0/2124501416 wait complete. 2024-01-16T14:59:59.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.795+0000 7f407d600640 1 Processor -- start 2024-01-16T14:59:59.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.795+0000 7f407d600640 1 -- start start 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f407813df80 0x7f407814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f407813e360 0x7f407814b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407814f1c0 con 0x7f407815f9c0 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407814f4a0 con 0x7f407813e360 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.796+0000 7f407d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407814f780 con 0x7f407813df80 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53772/0 (socket says 172.21.15.125:53772) 2024-01-16T14:59:59.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 -- 172.21.15.125:0/2327782650 learned_addr learned my addr 172.21.15.125:0/2327782650 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4076a00640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f407813df80 0x7f407814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4076000640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f407813e360 0x7f407814b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 -- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f407813df80 msgr2=0x7f407814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f407813df80 0x7f407814ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 -- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f407813e360 msgr2=0x7f407814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f407813e360 0x7f407814b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 -- 172.21.15.125:0/2327782650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060009400 con 0x7f407815f9c0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.797+0000 7f4077400640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f407000cbb0 tx=0x7f407000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.798+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407000ad40 con 0x7f407815f9c0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.798+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4070012400 con 0x7f407815f9c0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.798+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407000aea0 con 0x7f407815f9c0 2024-01-16T14:59:59.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.798+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407814fa60 con 0x7f407815f9c0 2024-01-16T14:59:59.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.798+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40781f0890 con 0x7f407815f9c0 2024-01-16T14:59:59.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.799+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40780675d0 con 0x7f407815f9c0 2024-01-16T14:59:59.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.800+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4070007070 con 0x7f407815f9c0 2024-01-16T14:59:59.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.801+0000 7f406ea00640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.801+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4070089ee0 con 0x7f407815f9c0 2024-01-16T14:59:59.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.801+0000 7f4076a00640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.802+0000 7f4076a00640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 0x7f405006d9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4068006fd0 tx=0x7f4068008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T14:59:59.805+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4070054a90 con 0x7f407815f9c0 2024-01-16T15:00:00.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.034+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f407806df00 con 0x7f407815f9c0 2024-01-16T15:00:00.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.035+0000 7f406ea00640 1 -- 172.21.15.125:0/2327782650 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f4070017020 con 0x7f407815f9c0 2024-01-16T15:00:00.015 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:00:00.015 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/3050283375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3050283375},{"type":"v1","addr":"172.21.15.137:6833","nonce":3050283375}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4258955541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4258955541},{"type":"v1","addr":"172.21.15.137:6835","nonce":4258955541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5739,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1745104106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1745104106},{"type":"v1","addr":"172.21.15.125:6836","nonce":1745104106}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:48.571459+0000","modified":"2024-01-16T14:59:51.649467+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5742":{"gid":5742,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/351760567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":351760567},{"type":"v1","addr":"172.21.15.125:6837","nonce":351760567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:00:00.015 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 28 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 msgr2=0x7f405006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 0x7f405006d9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4068006fd0 tx=0x7f4068008040 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 msgr2=0x7f40781f0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f407000cbb0 tx=0x7f407000a430 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 shutdown_connections 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407815f9c0 0x7f40781f0350 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f407813e360 0x7f407814b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 --2- 172.21.15.125:0/2327782650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f407813df80 0x7f407814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.037+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 >> 172.21.15.125:0/2327782650 conn(0x7f407806a800 msgr2=0x7f4078153e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.038+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 shutdown_connections 2024-01-16T15:00:00.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.038+0000 7f407d600640 1 -- 172.21.15.125:0/2327782650 wait complete. 2024-01-16T15:00:00.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:59:48.571459+0000', 'modified': '2024-01-16T14:59:51.649467+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5742}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5742': {'gid': 5742, 'name': 'c', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/351760567', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6835', 'nonce': 351760567}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 351760567}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:00:00.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:00:00.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:00:00.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.680+0000 7f55c3c00640 1 Processor -- start 2024-01-16T15:00:00.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.681+0000 7f55c3c00640 1 -- start start 2024-01-16T15:00:00.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.682+0000 7f55c3c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55bc13df80 0x7f55bc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.682+0000 7f55c3c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 0x7f55bc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.682+0000 7f55c3c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc06c470 con 0x7f55bc13e360 2024-01-16T15:00:00.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.682+0000 7f55c3c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55bc06c750 con 0x7f55bc13df80 2024-01-16T15:00:00.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.682+0000 7f55c3c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc06ca30 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55b3c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f55bc13e360 0x7f55bc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51802/0 (socket says 172.21.15.125:51802) 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55b3c00640 1 -- 172.21.15.125:0/757102365 learned_addr learned my addr 172.21.15.125:0/757102365 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1200640 1 --2- 172.21.15.125:0/757102365 >> v2:172.21.15.125:3300/0 conn(0x7f55bc13df80 0x7f55bc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1c00640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 0x7f55bc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55bbe00640 1 -- 172.21.15.125:0/757102365 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068588276 0 0) 0x7f55bc06c470 con 0x7f55bc13e360 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55bbe00640 1 -- 172.21.15.125:0/757102365 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c003520 con 0x7f55bc13e360 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1200640 1 -- 172.21.15.125:0/757102365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 msgr2=0x7f55bc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1200640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 0x7f55bc0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1200640 1 -- 172.21.15.125:0/757102365 >> v1:172.21.15.125:6789/0 conn(0x7f55bc13e360 legacy=0x7f55bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1200640 1 -- 172.21.15.125:0/757102365 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc06cd10 con 0x7f55bc13df80 2024-01-16T15:00:00.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.683+0000 7f55c1c00640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 0x7f55bc0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:00.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.685+0000 7f55c1200640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc13df80 0x7f55bc15bee0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0097a0 tx=0x7f55ac02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422428f59f6ac1e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.685+0000 7f55bbe00640 1 -- 172.21.15.125:0/757102365 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac02d040 con 0x7f55bc13df80 2024-01-16T15:00:00.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.685+0000 7f55bbe00640 1 -- 172.21.15.125:0/757102365 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac031810 con 0x7f55bc13df80 2024-01-16T15:00:00.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.685+0000 7f55bbe00640 1 -- 172.21.15.125:0/757102365 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac031ba0 con 0x7f55bc13df80 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 -- 172.21.15.125:0/757102365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc13df80 msgr2=0x7f55bc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc13df80 0x7f55bc15bee0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0097a0 tx=0x7f55ac02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 -- 172.21.15.125:0/757102365 shutdown_connections 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc0cf9a0 0x7f55bc0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 --2- 172.21.15.125:0/757102365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc13df80 0x7f55bc15bee0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 -- 172.21.15.125:0/757102365 >> 172.21.15.125:0/757102365 conn(0x7f55bc06a800 msgr2=0x7f55bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 -- 172.21.15.125:0/757102365 shutdown_connections 2024-01-16T15:00:00.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.686+0000 7f55c3c00640 1 -- 172.21.15.125:0/757102365 wait complete. 2024-01-16T15:00:00.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.688+0000 7f55c3c00640 1 Processor -- start 2024-01-16T15:00:00.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.688+0000 7f55c3c00640 1 -- start start 2024-01-16T15:00:00.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 0x7f55bc12f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55bc13e360 0x7f55bc12fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc131130 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc130270 con 0x7f55bc13e360 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c3c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc130520 con 0x7f55bc13df80 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53794/0 (socket says 172.21.15.125:53794) 2024-01-16T15:00:00.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c1200640 1 -- 172.21.15.125:0/4094853424 learned_addr learned my addr 172.21.15.125:0/4094853424 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55b3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 0x7f55bc12f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c1200640 1 -- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 msgr2=0x7f55bc12f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.689+0000 7f55c1200640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 0x7f55bc12f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c1200640 1 -- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55bc13e360 msgr2=0x7f55bc12fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c1c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55bc13e360 0x7f55bc12fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c1200640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55bc13e360 0x7f55bc12fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c1200640 1 -- 172.21.15.125:0/4094853424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac009400 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55b3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 0x7f55bc12f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c1200640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0098c0 tx=0x7f55ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55b9600640 1 -- 172.21.15.125:0/4094853424 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac006e10 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55b9600640 1 -- 172.21.15.125:0/4094853424 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac03ad60 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc130800 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55b9600640 1 -- 172.21.15.125:0/4094853424 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac044620 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.690+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc130dc0 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.693+0000 7f55b9600640 1 -- 172.21.15.125:0/4094853424 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f55ac04d440 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.693+0000 7f55b9600640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 0x7f559406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.694+0000 7f55b9600640 1 -- 172.21.15.125:0/4094853424 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55ac0ac540 con 0x7f55bc0cf9a0 2024-01-16T15:00:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.694+0000 7f55b3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 0x7f559406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.694+0000 7f55c69b0740 1 Processor -- start 2024-01-16T15:00:00.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.694+0000 7f55b3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 0x7f559406d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f55bc072040 tx=0x7f55a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 -- start start 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e49280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091da7880 con 0x560091e485e0 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091db6ae0 con 0x560091e48ea0 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.695+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x560091db41d0 con 0x560091e43ba0 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53802/0 (socket says 172.21.15.125:53802) 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 -- 172.21.15.125:0/313739050 learned_addr learned my addr 172.21.15.125:0/313739050 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55b3c00640 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 -- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 msgr2=0x560091e480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e480a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 -- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 msgr2=0x560091e49280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e49280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 -- 172.21.15.125:0/313739050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560091d77860 con 0x560091e485e0 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.696+0000 7f55c1c00640 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009f60 tx=0x7f55b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad66cc233a2faa2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55b1400640 1 -- 172.21.15.125:0/313739050 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b400e050 con 0x560091e485e0 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55b1400640 1 -- 172.21.15.125:0/313739050 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b400f040 con 0x560091e485e0 2024-01-16T15:00:00.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55b1400640 1 -- 172.21.15.125:0/313739050 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40139f0 con 0x560091e485e0 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55c69b0740 1 -- 172.21.15.125:0/313739050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 msgr2=0x560091e8c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55c69b0740 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009f60 tx=0x7f55b4009f90 comp rx=0 tx=0).stop 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.697+0000 7f55c69b0740 1 -- 172.21.15.125:0/313739050 shutdown_connections 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e8c560 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 --2- 172.21.15.125:0/313739050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e49280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 -- 172.21.15.125:0/313739050 >> 172.21.15.125:0/313739050 conn(0x560091e0ac70 msgr2=0x560091e8e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 -- 172.21.15.125:0/313739050 shutdown_connections 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 -- 172.21.15.125:0/313739050 wait complete. 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.698+0000 7f55c69b0740 1 -- start start 2024-01-16T15:00:00.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.699+0000 7f55c69b0740 1 Processor -- start 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.699+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.699+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55c69b0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091dafac0 con 0x560091e485e0 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091d4c2f0 con 0x560091e48ea0 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x560091d69c70 con 0x560091e43ba0 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55c69b0740 10 client.0 _ll_register_callbacks cb 0x560091d6e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53806/0 (socket says 172.21.15.125:53806) 2024-01-16T15:00:00.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 -- 172.21.15.125:0/2452240134 learned_addr learned my addr 172.21.15.125:0/2452240134 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 -- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 msgr2=0x560091e726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 -- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 msgr2=0x560091e712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 -- 172.21.15.125:0/2452240134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b4009b60 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55b3c00640 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f55a4007330 tx=0x7f55a4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eceef77b45bb3fb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.700+0000 7f55aa000640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4021630 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55a4020a80 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560091da8aa0 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a4020c40 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55c69b0740 10 client.5910 fetch_fsmap learned FSMap version 28 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x560091dae140 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560091da8aa0 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f55a4020420 con 0x560091e485e0 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55c69b0740 10 client.5910 fetch_fsmap finished waiting for FSMap version 28 2024-01-16T15:00:00.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.701+0000 7f55c69b0740 10 client.5910 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c69b0740 20 client.5910 populate_metadata read hostname 'smithi125' 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e71cb0 0x560091ea23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c69b0740 4 client.5910 mds_command: new command op to 5742 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 --> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560091da87c0 con 0x560091e71cb0 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c1200640 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e71cb0 0x560091ea23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55c1200640 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e71cb0 0x560091ea23e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe648f0dfe495b20 server_cookie=3c2cb4d4c5760598 in_seq=0 out_seq=0 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.702+0000 7f55aa000640 10 client.5910 ms_handle_connect on v2:172.21.15.125:6835/351760567 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55aa000640 1 -- 172.21.15.125:0/2452240134 <== mds.0 v2:172.21.15.125:6835/351760567 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560091da87c0 con 0x560091e71cb0 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55aa000640 10 client.5910 handle_command_reply: tid=0 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 client.5910 shutdown 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 msgr2=0x560091e6df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f55a4007330 tx=0x7f55a4000ed0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 shutdown_connections 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e71cb0 0x560091ea23e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e485e0 0x560091e6df70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 --2- 172.21.15.125:0/2452240134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e43ba0 0x560091e726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 >> 172.21.15.125:0/2452240134 conn(0x560091e0ac70 msgr2=0x560091e42b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.703+0000 7f55aa000640 0 client.5910 ms_handle_reset on v2:172.21.15.125:6835/351760567 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.704+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 shutdown_connections 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.704+0000 7f55c69b0740 1 -- 172.21.15.125:0/2452240134 wait complete. 2024-01-16T15:00:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.704+0000 7f55c69b0740 20 client.5910 trim_cache size 0 max 16384 2024-01-16T15:00:00.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.720+0000 7f55c69b0740 1 Processor -- start 2024-01-16T15:00:00.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 -- start start 2024-01-16T15:00:00.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e73850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091b12f20 con 0x560091e43ba0 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091e70d40 con 0x560091e48ea0 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.721+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x560091e6d2c0 con 0x560091e485e0 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53816/0 (socket says 172.21.15.125:53816) 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 -- 172.21.15.125:0/513254844 learned_addr learned my addr 172.21.15.125:0/513254844 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 -- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 msgr2=0x560091e73850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e73850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 -- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 msgr2=0x560091e6bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 -- 172.21.15.125:0/513254844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560091d4bb80 con 0x560091e43ba0 2024-01-16T15:00:00.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.722+0000 7f55c1200640 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f55ac006dc0 tx=0x7f55ac0ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d40cd435dc6a8bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55b1e00640 1 -- 172.21.15.125:0/513254844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac082e60 con 0x560091e43ba0 2024-01-16T15:00:00.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55b1e00640 1 -- 172.21.15.125:0/513254844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac07be90 con 0x560091e43ba0 2024-01-16T15:00:00.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55b1e00640 1 -- 172.21.15.125:0/513254844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac077270 con 0x560091e43ba0 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 -- 172.21.15.125:0/513254844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 msgr2=0x560091e597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f55ac006dc0 tx=0x7f55ac0ac040 comp rx=0 tx=0).stop 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 -- 172.21.15.125:0/513254844 shutdown_connections 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e73850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 --2- 172.21.15.125:0/513254844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091e597e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 -- 172.21.15.125:0/513254844 >> 172.21.15.125:0/513254844 conn(0x560091e0ac70 msgr2=0x560091e3f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 -- 172.21.15.125:0/513254844 shutdown_connections 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.723+0000 7f55c69b0740 1 -- 172.21.15.125:0/513254844 wait complete. 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.724+0000 7f55c69b0740 1 -- start start 2024-01-16T15:00:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.724+0000 7f55c69b0740 1 Processor -- start 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e8d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091e44600 con 0x560091e43ba0 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091e8d030 con 0x560091e48ea0 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x560091e70850 con 0x560091e485e0 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c69b0740 10 client.0 _ll_register_callbacks cb 0x560091e97160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53818/0 (socket says 172.21.15.125:53818) 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.725+0000 7f55c1200640 1 -- 172.21.15.125:0/1704085492 learned_addr learned my addr 172.21.15.125:0/1704085492 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 -- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 msgr2=0x560091e8d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55b3c00640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e8d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e8d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 -- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 msgr2=0x560091e6fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 -- 172.21.15.125:0/1704085492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc130dc0 con 0x560091e43ba0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c1200640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f55ac077060 tx=0x7f55a4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7edcb4279d8f27 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55aa000640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac097070 con 0x560091e43ba0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55ac02f2c0 con 0x560091e43ba0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.726+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55b400f040 con 0x560091e43ba0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55ac02f4a0 con 0x560091e43ba0 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 10 client.5916 fetch_fsmap learned FSMap version 28 2024-01-16T15:00:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x560091d71170 con 0x560091e43ba0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b400f040 con 0x560091e43ba0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f55ac04b060 con 0x560091e43ba0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 10 client.5916 fetch_fsmap finished waiting for FSMap version 28 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 10 client.5916 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 20 client.5916 populate_metadata read hostname 'smithi125' 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e95e30 0x560091e96210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 4 client.5916 mds_command: new command op to 5742 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 --> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560091ab06c0 con 0x560091e95e30 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.727+0000 7f55b3c00640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e95e30 0x560091e96210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55b3c00640 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e95e30 0x560091e96210 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41a0562404969d8 server_cookie=bdd6f22dde125b37 in_seq=0 out_seq=0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55aa000640 10 client.5916 ms_handle_connect on v2:172.21.15.125:6835/351760567 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55aa000640 1 -- 172.21.15.125:0/1704085492 <== mds.0 v2:172.21.15.125:6835/351760567 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x560091ab06c0 con 0x560091e95e30 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55aa000640 10 client.5916 handle_command_reply: tid=0 2024-01-16T15:00:00.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 client.5916 shutdown 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 msgr2=0x560091ef8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f55ac077060 tx=0x7f55a4011000 comp rx=0 tx=0).stop 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 shutdown_connections 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x560091e48ea0 0x560091e6fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x560091e485e0 0x560091e8d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:6835/351760567,v1:172.21.15.125:6837/351760567] conn(0x560091e95e30 0x560091e96210 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 --2- 172.21.15.125:0/1704085492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x560091e43ba0 0x560091ef8930 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 >> 172.21.15.125:0/1704085492 conn(0x560091e0ac70 msgr2=0x560091e6da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55aa000640 0 client.5916 ms_handle_reset on v2:172.21.15.125:6835/351760567 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 shutdown_connections 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.728+0000 7f55c69b0740 1 -- 172.21.15.125:0/1704085492 wait complete. 2024-01-16T15:00:00.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c69b0740 20 client.5916 trim_cache size 0 max 16384 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5742, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 11.605029891999999, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 28, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 45, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 45, 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 17.067119641000001 2024-01-16T15:00:00.714 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:00:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 msgr2=0x7f559406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 0x7f559406d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f55bc072040 tx=0x7f55a4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 msgr2=0x7f55bc0b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0098c0 tx=0x7f55ac0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 shutdown_connections 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55bc13e360 0x7f55bc12fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f55bc13df80 0x7f55bc12f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f559406b500 0x7f559406d9c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 --2- 172.21.15.125:0/4094853424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55bc0cf9a0 0x7f55bc0b2f60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 >> 172.21.15.125:0/4094853424 conn(0x7f55bc06a800 msgr2=0x7f55bc0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 shutdown_connections 2024-01-16T15:00:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.729+0000 7f55c3c00640 1 -- 172.21.15.125:0/4094853424 wait complete. 2024-01-16T15:00:00.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:00.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.918+0000 7f1a00a00640 1 Processor -- start 2024-01-16T15:00:00.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.918+0000 7f1a00a00640 1 -- start start 2024-01-16T15:00:00.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.919+0000 7f1a00a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19f815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.919+0000 7f1a00a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f19f80cf9a0 0x7f19f80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.919+0000 7f1a00a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f806c470 con 0x7f19f813e360 2024-01-16T15:00:00.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.919+0000 7f1a00a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f806c750 con 0x7f19f80cf9a0 2024-01-16T15:00:00.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.919+0000 7f1a00a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f806ca30 con 0x7f19f813df80 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fd600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f19f813e360 0x7f19f80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51812/0 (socket says 172.21.15.125:51812) 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fd600640 1 -- 172.21.15.125:0/3273620232 learned_addr learned my addr 172.21.15.125:0/3273620232 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fe000640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19f815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fea00640 1 --2- 172.21.15.125:0/3273620232 >> v2:172.21.15.137:3300/0 conn(0x7f19f80cf9a0 0x7f19f80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965916342 0 0) 0x7f19f806c470 con 0x7f19f813e360 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.920+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8003520 con 0x7f19f813e360 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 238639080 0 0) 0x7f19d8003520 con 0x7f19f813e360 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 msgr2=0x7f19f815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19f815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 >> v2:172.21.15.137:3300/0 conn(0x7f19f80cf9a0 msgr2=0x7f19f80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 --2- 172.21.15.125:0/3273620232 >> v2:172.21.15.137:3300/0 conn(0x7f19f80cf9a0 0x7f19f80cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f806cd10 con 0x7f19f813e360 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f19e0001670 con 0x7f19f813e360 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 >> v1:172.21.15.125:6789/0 conn(0x7f19f813e360 legacy=0x7f19f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19d8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d8003ef0 0x7f19d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8004d00 0x7f19d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f806c750 con 0x7f19d8004d00 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f806ca30 con 0x7f19d8003ef0 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800b980 con 0x7f19f813df80 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fea00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8004d00 0x7f19d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.921+0000 7f19fe000640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19d8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fd600640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d8003ef0 0x7f19d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fea00640 1 -- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 msgr2=0x7f19d8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fea00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f813df80 0x7f19d8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fea00640 1 -- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d8003ef0 msgr2=0x7f19d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fea00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d8003ef0 0x7f19d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.922+0000 7f19fea00640 1 -- 172.21.15.125:0/3273620232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8003520 con 0x7f19d8004d00 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.923+0000 7f19fea00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8004d00 0x7f19d8005180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009fc0 tx=0x7f19d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b755d5f5ecb375 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.923+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f002cc40 con 0x7f19d8004d00 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.923+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f000f040 con 0x7f19d8004d00 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.923+0000 7f19fcc00640 1 -- 172.21.15.125:0/3273620232 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0034440 con 0x7f19d8004d00 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 -- 172.21.15.125:0/3273620232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8004d00 msgr2=0x7f19d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 --2- 172.21.15.125:0/3273620232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8004d00 0x7f19d8005180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009fc0 tx=0x7f19d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f19fea00640 1 -- 172.21.15.125:0/3273620232 reap_dead start 2024-01-16T15:00:00.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 -- 172.21.15.125:0/3273620232 shutdown_connections 2024-01-16T15:00:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 -- 172.21.15.125:0/3273620232 >> 172.21.15.125:0/3273620232 conn(0x7f19f806a800 msgr2=0x7f19f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 -- 172.21.15.125:0/3273620232 shutdown_connections 2024-01-16T15:00:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.924+0000 7f1a00a00640 1 -- 172.21.15.125:0/3273620232 wait complete. 2024-01-16T15:00:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.926+0000 7f1a00a00640 1 Processor -- start 2024-01-16T15:00:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.926+0000 7f1a00a00640 1 -- start start 2024-01-16T15:00:00.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 0x7f19f813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 0x7f19f8157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8150ee0 con 0x7f19f8150520 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f81f5160 con 0x7f19f8157860 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f1a00a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f81f4cd0 con 0x7f19d8004d00 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f19fd600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f19fd600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53836/0 (socket says 172.21.15.125:53836) 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.927+0000 7f19fd600640 1 -- 172.21.15.125:0/117798651 learned_addr learned my addr 172.21.15.125:0/117798651 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fe000640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 0x7f19f813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fea00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 0x7f19f8157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 -- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 msgr2=0x7f19f813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 0x7f19f813d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 -- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 msgr2=0x7f19f8157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 0x7f19f8157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 -- 172.21.15.125:0/117798651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f81f43a0 con 0x7f19f8150520 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fe000640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 0x7f19f813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fea00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 0x7f19f8157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19fd600640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f19e000e8d0 tx=0x7f19e000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e000c8d0 con 0x7f19f8150520 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e000ca50 con 0x7f19f8150520 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f81f4680 con 0x7f19f8150520 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e000ce00 con 0x7f19f8150520 2024-01-16T15:00:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.928+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f8158220 con 0x7f19f8150520 2024-01-16T15:00:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.930+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f80675d0 con 0x7f19f8150520 2024-01-16T15:00:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.931+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f19e001d460 con 0x7f19f8150520 2024-01-16T15:00:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.932+0000 7f19f6000640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 0x7f19d006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.932+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19e0089ff0 con 0x7f19f8150520 2024-01-16T15:00:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.932+0000 7f19fe000640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 0x7f19d006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.933+0000 7f19fe000640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 0x7f19d006d9f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009a00 tx=0x7f19e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:00.936+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19e0054b30 con 0x7f19f8150520 2024-01-16T15:00:01.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.157+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f19f81f4e10 con 0x7f19f8150520 2024-01-16T15:00:01.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.158+0000 7f19f6000640 1 -- 172.21.15.125:0/117798651 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7f19e0058150 con 0x7f19f8150520 2024-01-16T15:00:01.139 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:01.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 msgr2=0x7f19d006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 0x7f19d006d9f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009a00 tx=0x7f19e8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:01.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 msgr2=0x7f19f81509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f19e000e8d0 tx=0x7f19e000ed90 comp rx=0 tx=0).stop 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 shutdown_connections 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f8157860 0x7f19f8157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006b530 0x7f19d006d9f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f8150520 0x7f19f81509a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 --2- 172.21.15.125:0/117798651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8004d00 0x7f19f813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 >> 172.21.15.125:0/117798651 conn(0x7f19f806a800 msgr2=0x7f19f80b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 shutdown_connections 2024-01-16T15:00:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:01.160+0000 7f1a00a00640 1 -- 172.21.15.125:0/117798651 wait complete. 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:01.158 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:01.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.159 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:00:01.206 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:01.206 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:01.206 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:01.206 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.206 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:01.207 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:00:01.208 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:01.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.209 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:01.333 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:00:01.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.337 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:00:01.388 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:00:01.389 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:01.389 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:01.389 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:01.389 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:01.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:01.390 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:00:01.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.568 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:00:01.617 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:00:01.617 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:00:01.618 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:01.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:00:01.709 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:00:01.709 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:01.709 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:01.762 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:01.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:01.762 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:01.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:00:01.908 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:00:01.962 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:01.963 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:01.964 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:01.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:01.965 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:00:02.013 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:02.013 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:00:02.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:00:02.095 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:02.135 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:00:02.136 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:02.137 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:02.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:02.139 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:00:02.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:00:02.225+0000 7fb3d4442480 -1 init, newargv = 0x5637ba9c5a20 newargc=15 2024-01-16T15:00:02.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[53863]: starting ceph client 2024-01-16T15:00:02.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[53863]: starting fuse 2024-01-16T15:00:02.223 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:00:02.223 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:00:02.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:02.223 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:00:02.223 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:00:02.223 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:00:02.224 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:00:02.225 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:00:02.411 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.53863.asok 2024-01-16T15:00:02.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:02.428 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53863.asok 2024-01-16T15:00:02.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:02.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53863.asok status 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "53863", 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:00:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5928, 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 5928 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2469370392 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:00:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2469370392 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.5928 192.168.0.1:0/2469370392", 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/2469370392", 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 28, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 45, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:00:02.666 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:00:02.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:02.679 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:00:02.679 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:02.729 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:00:02.730 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:02.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:02.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:02.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:03.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.058+0000 7fb92b000640 1 Processor -- start 2024-01-16T15:00:03.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.058+0000 7fb92b000640 1 -- start start 2024-01-16T15:00:03.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.059+0000 7fb92b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb92406ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.059+0000 7fb92b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb924070490 0x7fb924099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.059+0000 7fb92b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9240cfa70 con 0x7fb924099fa0 2024-01-16T15:00:03.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.059+0000 7fb92b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92409a410 con 0x7fb924070490 2024-01-16T15:00:03.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.059+0000 7fb92b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9240cff00 con 0x7fb92415e880 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.060+0000 7fb923e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb92406ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.060+0000 7fb923e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb92406ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:53812/0 (socket says 172.21.15.125:53812) 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.060+0000 7fb91b200640 1 --2- 172.21.15.125:0/2176624291 >> v2:172.21.15.137:3300/0 conn(0x7fb924070490 0x7fb924099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.060+0000 7fb923e00640 1 -- 172.21.15.125:0/2176624291 learned_addr learned my addr 172.21.15.125:0/2176624291 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.061+0000 7fb91b200640 1 -- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 msgr2=0x7fb92406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.061+0000 7fb91b200640 1 --2- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb92406ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.061+0000 7fb91b200640 1 -- 172.21.15.125:0/2176624291 >> v1:172.21.15.137:6789/0 conn(0x7fb924099fa0 legacy=0x7fb9240cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.061+0000 7fb91b200640 1 -- 172.21.15.125:0/2176624291 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92406fc70 con 0x7fb924070490 2024-01-16T15:00:03.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.062+0000 7fb91b200640 1 --2- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb924099a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0097a0 tx=0x7fb90c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8edcafbe39bb6501 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.063+0000 7fb923400640 1 -- 172.21.15.125:0/2176624291 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c030800 con 0x7fb924070490 2024-01-16T15:00:03.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.063+0000 7fb923400640 1 -- 172.21.15.125:0/2176624291 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c030960 con 0x7fb924070490 2024-01-16T15:00:03.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.063+0000 7fb923400640 1 -- 172.21.15.125:0/2176624291 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c030cf0 con 0x7fb924070490 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 -- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 msgr2=0x7fb924099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 --2- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb924099a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0097a0 tx=0x7fb90c02b940 comp rx=0 tx=0).stop 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 -- 172.21.15.125:0/2176624291 shutdown_connections 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 --2- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb924099a60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 --2- 172.21.15.125:0/2176624291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb92406ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 -- 172.21.15.125:0/2176624291 >> 172.21.15.125:0/2176624291 conn(0x7fb9240b2d00 msgr2=0x7fb9240b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 -- 172.21.15.125:0/2176624291 shutdown_connections 2024-01-16T15:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.064+0000 7fb92b000640 1 -- 172.21.15.125:0/2176624291 wait complete. 2024-01-16T15:00:03.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.065+0000 7fb92b000640 1 Processor -- start 2024-01-16T15:00:03.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.066+0000 7fb92b000640 1 -- start start 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb92406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb9240c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92406ccb0 con 0x7fb924099fa0 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9240c5e00 con 0x7fb924070490 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb92b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92406bb60 con 0x7fb92415e880 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb91b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb91b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53846/0 (socket says 172.21.15.125:53846) 2024-01-16T15:00:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb91b200640 1 -- 172.21.15.125:0/257947695 learned_addr learned my addr 172.21.15.125:0/257947695 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb929000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb9240c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.067+0000 7fb923e00640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb92406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 -- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 msgr2=0x7fb9240c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb9240c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 -- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 msgr2=0x7fb92406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb92406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 -- 172.21.15.125:0/257947695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c009400 con 0x7fb924099fa0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb91b200640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0098c0 tx=0x7fb90c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c03e070 con 0x7fb924099fa0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c039a10 con 0x7fb924099fa0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c039da0 con 0x7fb924099fa0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92406be40 con 0x7fb924099fa0 2024-01-16T15:00:03.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.068+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92406c400 con 0x7fb924099fa0 2024-01-16T15:00:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.070+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9240675d0 con 0x7fb924099fa0 2024-01-16T15:00:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.070+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb90c04b400 con 0x7fb924099fa0 2024-01-16T15:00:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.071+0000 7fb920c00640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 0x7fb8fc06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.071+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb90c0ac5f0 con 0x7fb924099fa0 2024-01-16T15:00:03.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.072+0000 7fb923e00640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 0x7fb8fc06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.072+0000 7fb923e00640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 0x7fb8fc06d9f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb914009d80 tx=0x7fb91400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.076+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb90c076fe0 con 0x7fb924099fa0 2024-01-16T15:00:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.307+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb9240cedb0 con 0x7fb924099fa0 2024-01-16T15:00:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.307+0000 7fb920c00640 1 -- 172.21.15.125:0/257947695 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb90c07a600 con 0x7fb924099fa0 2024-01-16T15:00:03.289 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 msgr2=0x7fb8fc06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 0x7fb8fc06d9f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb914009d80 tx=0x7fb91400b040 comp rx=0 tx=0).stop 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 msgr2=0x7fb9240ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0098c0 tx=0x7fb90c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 shutdown_connections 2024-01-16T15:00:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb92415e880 0x7fb9240c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8fc06b530 0x7fb8fc06d9f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb924099fa0 0x7fb9240ca6c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 --2- 172.21.15.125:0/257947695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb924070490 0x7fb92406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 >> 172.21.15.125:0/257947695 conn(0x7fb9240b2d00 msgr2=0x7fb9240cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 shutdown_connections 2024-01-16T15:00:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:03.310+0000 7fb92b000640 1 -- 172.21.15.125:0/257947695 wait complete. 2024-01-16T15:00:03.310 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:03.310 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:03.310 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:00:03.310 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:00:03.310 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:03.311 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:03.311 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:03.311 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:03.311 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:03.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.311 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:00:03.369 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:03.369 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:03.369 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:03.370 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42434sec preferred_lft 42434sec 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:00:03.372 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:03.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.373 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:03.492 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:00:03.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.496 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:00:03.546 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:00:03.546 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:03.547 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:03.547 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:03.547 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:03.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:03.547 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:00:03.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.700 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:00:03.740 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:03.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:03.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:00:03.848 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:00:03.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:03.848 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:03.884 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:03.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:03.884 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:03.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:00:04.100 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:04.152 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:00:04.152 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:00:04.153 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:04.154 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:04.155 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:04.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:04.155 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:00:04.202 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:04.202 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:00:04.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:00:04.329 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:00:04.373 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:04.374 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:04.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:04.376 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:00:04.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:00:04.460+0000 7fe704c3a480 -1 init, newargv = 0x557cd21ee940 newargc=15 2024-01-16T15:00:04.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43620]: starting ceph client 2024-01-16T15:00:04.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43620]: starting fuse 2024-01-16T15:00:04.463 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:00:04.463 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:00:04.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:04.463 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:00:04.463 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:00:04.465 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:00:04.662 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.43620.asok 2024-01-16T15:00:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:00:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:04.679 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43620.asok 2024-01-16T15:00:04.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:04.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43620.asok status 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "43620", 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:00:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5167, 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5167 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 42029925 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 42029925 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5167 192.168.0.1:0/42029925", 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/42029925", 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 28, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 45, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:00:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:00:04.920 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:00:04.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:04.932 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:00:04.933 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:04.985 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:00:04.985 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:04.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:04.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:05.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:00:05.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.317+0000 7f8947e00640 1 Processor -- start 2024-01-16T15:00:05.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.317+0000 7f8947e00640 1 -- start start 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8948143c40 0x7f8948071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f89480ccc10 0x7f894806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8948070e50 con 0x7f89480717d0 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89480723c0 con 0x7f8948143c40 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8948144020 con 0x7f89480ccc10 2024-01-16T15:00:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8948143c40 0x7f8948071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8948143c40 0x7f8948071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53852/0 (socket says 172.21.15.125:53852) 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 -- 172.21.15.125:0/605705073 learned_addr learned my addr 172.21.15.125:0/605705073 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f893fe00640 1 --1- 172.21.15.125:0/605705073 >> v1:172.21.15.125:6789/0 conn(0x7f89480717d0 0x7f89480cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51820/0 (socket says 172.21.15.125:51820) 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8947400640 1 --2- 172.21.15.125:0/605705073 >> v2:172.21.15.137:3300/0 conn(0x7f89480ccc10 0x7f894806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 -- 172.21.15.125:0/605705073 >> v2:172.21.15.137:3300/0 conn(0x7f89480ccc10 msgr2=0x7f894806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 --2- 172.21.15.125:0/605705073 >> v2:172.21.15.137:3300/0 conn(0x7f89480ccc10 0x7f894806d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.319+0000 7f8946a00640 1 -- 172.21.15.125:0/605705073 >> v1:172.21.15.125:6789/0 conn(0x7f89480717d0 legacy=0x7f89480cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.320+0000 7f8946a00640 1 -- 172.21.15.125:0/605705073 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948071e00 con 0x7f8948143c40 2024-01-16T15:00:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.320+0000 7f8946000640 1 -- 172.21.15.125:0/605705073 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984437565 0 0) 0x7f8948070e50 con 0x7f89480717d0 2024-01-16T15:00:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.321+0000 7f8946a00640 1 --2- 172.21.15.125:0/605705073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f8948071290 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8948070690 tx=0x7f893802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f2a4eb22899dbf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.321+0000 7f8946000640 1 -- 172.21.15.125:0/605705073 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938031870 con 0x7f8948143c40 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.321+0000 7f8946000640 1 -- 172.21.15.125:0/605705073 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89380319d0 con 0x7f8948143c40 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.321+0000 7f8946000640 1 -- 172.21.15.125:0/605705073 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938031d60 con 0x7f8948143c40 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 -- 172.21.15.125:0/605705073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 msgr2=0x7f8948071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 --2- 172.21.15.125:0/605705073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f8948071290 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8948070690 tx=0x7f893802b910 comp rx=0 tx=0).stop 2024-01-16T15:00:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 -- 172.21.15.125:0/605705073 shutdown_connections 2024-01-16T15:00:05.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 --2- 172.21.15.125:0/605705073 >> v2:172.21.15.137:3300/0 conn(0x7f89480ccc10 0x7f894806d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 --2- 172.21.15.125:0/605705073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f8948071290 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 -- 172.21.15.125:0/605705073 >> 172.21.15.125:0/605705073 conn(0x7f894806a800 msgr2=0x7f894806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 -- 172.21.15.125:0/605705073 shutdown_connections 2024-01-16T15:00:05.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.322+0000 7f8947e00640 1 -- 172.21.15.125:0/605705073 wait complete. 2024-01-16T15:00:05.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.324+0000 7f8947e00640 1 Processor -- start 2024-01-16T15:00:05.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.324+0000 7f8947e00640 1 -- start start 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89480717d0 0x7f89480aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89480ccc10 0x7f894815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480abb40 con 0x7f8948143c40 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480b3170 con 0x7f89480ccc10 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89480ab860 con 0x7f89480717d0 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53856/0 (socket says 172.21.15.125:53856) 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 -- 172.21.15.125:0/2927475092 learned_addr learned my addr 172.21.15.125:0/2927475092 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 -- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89480717d0 msgr2=0x7f89480aa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89480717d0 0x7f89480aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 -- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89480ccc10 msgr2=0x7f894815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89480ccc10 0x7f894815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.325+0000 7f8947400640 1 -- 172.21.15.125:0/2927475092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380094d0 con 0x7f8948143c40 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f8947400640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f894000cd70 tx=0x7f894000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940003b90 con 0x7f8948143c40 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8940003cf0 con 0x7f8948143c40 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940004240 con 0x7f8948143c40 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89480b2ab0 con 0x7f8948143c40 2024-01-16T15:00:05.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.326+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89480aaf30 con 0x7f8948143c40 2024-01-16T15:00:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.327+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89480b2910 con 0x7f8948143c40 2024-01-16T15:00:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.328+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f89400043a0 con 0x7f8948143c40 2024-01-16T15:00:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.329+0000 7f893ea00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 0x7f892006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.329+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8940012070 con 0x7f8948143c40 2024-01-16T15:00:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.329+0000 7f8946a00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 0x7f892006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.330+0000 7f8946a00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 0x7f892006d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8938036000 tx=0x7f8938030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.333+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89480b2910 con 0x7f8948143c40 2024-01-16T15:00:05.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:05.564+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f89480b2910 con 0x7f8948143c40 2024-01-16T15:00:06.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.063+0000 7f893ea00640 1 -- 172.21.15.125:0/2927475092 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 136+0+0 (secure 0 0 0) 0x7f89480b2910 con 0x7f8948143c40 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 msgr2=0x7f892006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 0x7f892006d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8938036000 tx=0x7f8938030000 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 msgr2=0x7f894812fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f894000cd70 tx=0x7f894000a660 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 shutdown_connections 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f892006b500 0x7f892006d9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8948143c40 0x7f894812fcd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89480ccc10 0x7f894815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.067+0000 7f8947e00640 1 --2- 172.21.15.125:0/2927475092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89480717d0 0x7f89480aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.068+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 >> 172.21.15.125:0/2927475092 conn(0x7f894806a800 msgr2=0x7f89480b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.068+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 shutdown_connections 2024-01-16T15:00:06.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.068+0000 7f8947e00640 1 -- 172.21.15.125:0/2927475092 wait complete. 2024-01-16T15:00:06.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:00:06.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.300+0000 7f7ab1000640 1 Processor -- start 2024-01-16T15:00:06.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.300+0000 7f7ab1000640 1 -- start start 2024-01-16T15:00:06.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.301+0000 7f7ab1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.301+0000 7f7ab1000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7aac13fae0 0x7f7aac0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.301+0000 7f7ab1000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac13ff00 con 0x7f7aac13cfa0 2024-01-16T15:00:06.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.301+0000 7f7ab1000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0cdc40 con 0x7f7aac13fae0 2024-01-16T15:00:06.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.301+0000 7f7ab1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0cdec0 con 0x7f7aac15afb0 2024-01-16T15:00:06.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.302+0000 7f7aaaa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.302+0000 7f7aabe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7aac13fae0 0x7f7aac0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.302+0000 7f7aab400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7aac13cfa0 0x7f7aac0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:49500/0 (socket says 172.21.15.125:49500) 2024-01-16T15:00:06.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.302+0000 7f7aab400640 1 -- 172.21.15.125:0/2685000568 learned_addr learned my addr 172.21.15.125:0/2685000568 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:06.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaa000640 1 -- 172.21.15.125:0/2685000568 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174244104 0 0) 0x7f7aac13ff00 con 0x7f7aac13cfa0 2024-01-16T15:00:06.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaa000640 1 -- 172.21.15.125:0/2685000568 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c003520 con 0x7f7aac13cfa0 2024-01-16T15:00:06.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaaa00640 1 -- 172.21.15.125:0/2685000568 >> v2:172.21.15.137:3300/0 conn(0x7f7aac13fae0 msgr2=0x7f7aac0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaaa00640 1 --2- 172.21.15.125:0/2685000568 >> v2:172.21.15.137:3300/0 conn(0x7f7aac13fae0 0x7f7aac0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaaa00640 1 -- 172.21.15.125:0/2685000568 >> v1:172.21.15.137:6789/0 conn(0x7f7aac13cfa0 legacy=0x7f7aac0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.303+0000 7f7aaaa00640 1 -- 172.21.15.125:0/2685000568 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac0ce1a0 con 0x7f7aac15afb0 2024-01-16T15:00:06.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.305+0000 7f7aaaa00640 1 --2- 172.21.15.125:0/2685000568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac13f5a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009d90 tx=0x7f7a9402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9729f18e28be254b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.305+0000 7f7aaa000640 1 -- 172.21.15.125:0/2685000568 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402cbc0 con 0x7f7aac15afb0 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.305+0000 7f7aaa000640 1 -- 172.21.15.125:0/2685000568 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9402cd20 con 0x7f7aac15afb0 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.305+0000 7f7aaa000640 1 -- 172.21.15.125:0/2685000568 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94034660 con 0x7f7aac15afb0 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 -- 172.21.15.125:0/2685000568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 msgr2=0x7f7aac13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2685000568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac13f5a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009d90 tx=0x7f7a9402be10 comp rx=0 tx=0).stop 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 -- 172.21.15.125:0/2685000568 shutdown_connections 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2685000568 >> v2:172.21.15.137:3300/0 conn(0x7f7aac13fae0 0x7f7aac0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2685000568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac13f5a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 -- 172.21.15.125:0/2685000568 >> 172.21.15.125:0/2685000568 conn(0x7f7aac06a800 msgr2=0x7f7aac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 -- 172.21.15.125:0/2685000568 shutdown_connections 2024-01-16T15:00:06.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.306+0000 7f7ab1000640 1 -- 172.21.15.125:0/2685000568 wait complete. 2024-01-16T15:00:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.307+0000 7f7ab1000640 1 Processor -- start 2024-01-16T15:00:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.308+0000 7f7ab1000640 1 -- start start 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.308+0000 7f7ab1000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.308+0000 7f7ab1000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7aac13fae0 0x7f7aac150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7ab1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac06d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7ab1000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac1541c0 con 0x7f7aac13cfa0 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7ab1000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac1f0bc0 con 0x7f7aac13fae0 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7ab1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac155a10 con 0x7f7aac15afb0 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53872/0 (socket says 172.21.15.125:53872) 2024-01-16T15:00:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 -- 172.21.15.125:0/2647084220 learned_addr learned my addr 172.21.15.125:0/2647084220 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:06.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aaaa00640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7aac13fae0 0x7f7aac150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 -- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 msgr2=0x7f7aac06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aabe00640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac06d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 -- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7aac13fae0 msgr2=0x7f7aac150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7aac13fae0 0x7f7aac150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.309+0000 7f7aab400640 1 -- 172.21.15.125:0/2647084220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94009a70 con 0x7f7aac13cfa0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7aab400640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00c500 tx=0x7f7a9c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c005ce0 con 0x7f7aac13cfa0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c005e40 con 0x7f7aac13cfa0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac139700 con 0x7f7aac13cfa0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c007330 con 0x7f7aac13cfa0 2024-01-16T15:00:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.310+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac139cc0 con 0x7f7aac13cfa0 2024-01-16T15:00:06.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.311+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac067450 con 0x7f7aac13cfa0 2024-01-16T15:00:06.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.312+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7a9c007550 con 0x7f7aac13cfa0 2024-01-16T15:00:06.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.313+0000 7f7aa3400640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 0x7f7a8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.313+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a9c0896a0 con 0x7f7aac13cfa0 2024-01-16T15:00:06.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.313+0000 7f7aaaa00640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 0x7f7a8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.314+0000 7f7aaaa00640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 0x7f7a8406d9c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7a94036000 tx=0x7f7a94030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.317+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a9c054260 con 0x7f7aac13cfa0 2024-01-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.531+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7aac0cea40 con 0x7f7a8406b500 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.533+0000 7f7aa3400640 1 -- 172.21.15.125:0/2647084220 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7a94044080 con 0x7f7a8406b500 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 msgr2=0x7f7a8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 0x7f7a8406d9c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7a94036000 tx=0x7f7a94030000 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 msgr2=0x7f7aac155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00c500 tx=0x7f7a9c00c9c0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 shutdown_connections 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7aac15afb0 0x7f7aac06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7aac13fae0 0x7f7aac150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7a8406b500 0x7f7a8406d9c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 --2- 172.21.15.125:0/2647084220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aac13cfa0 0x7f7aac155310 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 >> 172.21.15.125:0/2647084220 conn(0x7f7aac06a800 msgr2=0x7f7aac06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 shutdown_connections 2024-01-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.536+0000 7f7ab1000640 1 -- 172.21.15.125:0/2647084220 wait complete. 2024-01-16T15:00:06.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:00:06.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.769+0000 7fc43e800640 1 Processor -- start 2024-01-16T15:00:06.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.769+0000 7fc43e800640 1 -- start start 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.770+0000 7fc43e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc43813df80 0x7fc43815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.770+0000 7fc43e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 0x7fc4380cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc43e800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43806c470 con 0x7fc43813e360 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc43e800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43806c750 con 0x7fc43813df80 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc43e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43806ca30 con 0x7fc4380cf9a0 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc436a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc43813e360 0x7fc4380cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44230/0 (socket says 172.21.15.125:44230) 2024-01-16T15:00:06.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc436a00640 1 -- 172.21.15.125:0/1418174284 learned_addr learned my addr 172.21.15.125:0/1418174284 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:06.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc437e00640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 0x7fc4380cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc437400640 1 --2- 172.21.15.125:0/1418174284 >> v2:172.21.15.137:3300/0 conn(0x7fc43813df80 0x7fc43815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.771+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352654629 0 0) 0x7fc43806c470 con 0x7fc43813e360 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc418003520 con 0x7fc43813e360 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256583545 0 0) 0x7fc418003520 con 0x7fc43813e360 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 msgr2=0x7fc4380cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 0x7fc4380cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 >> v2:172.21.15.137:3300/0 conn(0x7fc43813df80 msgr2=0x7fc43815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 --2- 172.21.15.125:0/1418174284 >> v2:172.21.15.137:3300/0 conn(0x7fc43813df80 0x7fc43815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43806cd10 con 0x7fc43813e360 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc420001670 con 0x7fc43813e360 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 >> v1:172.21.15.125:6789/0 conn(0x7fc43813e360 legacy=0x7fc4380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc418003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 0x7fc4180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc418004d00 0x7fc418005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43806c750 con 0x7fc418003970 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43806ca30 con 0x7fc418004d00 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41800b980 con 0x7fc4380cf9a0 2024-01-16T15:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.772+0000 7fc437400640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc418003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.773+0000 7fc437400640 1 -- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 msgr2=0x7fc4180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.773+0000 7fc437400640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380cf9a0 0x7fc4180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.773+0000 7fc437400640 1 -- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc418004d00 msgr2=0x7fc418005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.773+0000 7fc437400640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc418004d00 0x7fc418005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.773+0000 7fc437400640 1 -- 172.21.15.125:0/1418174284 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418003520 con 0x7fc418003970 2024-01-16T15:00:06.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.774+0000 7fc437400640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc418003d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc428009af0 tx=0x7fc4180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e043d87d8b7c859 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.774+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42802c710 con 0x7fc418003970 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.774+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42802c870 con 0x7fc418003970 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.774+0000 7fc436000640 1 -- 172.21.15.125:0/1418174284 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42802cc00 con 0x7fc418003970 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 -- 172.21.15.125:0/1418174284 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 msgr2=0x7fc418003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 --2- 172.21.15.125:0/1418174284 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc418003d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc428009af0 tx=0x7fc4180057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc437e00640 1 -- 172.21.15.125:0/1418174284 reap_dead start 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 -- 172.21.15.125:0/1418174284 shutdown_connections 2024-01-16T15:00:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 -- 172.21.15.125:0/1418174284 >> 172.21.15.125:0/1418174284 conn(0x7fc43806a800 msgr2=0x7fc43806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 -- 172.21.15.125:0/1418174284 shutdown_connections 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.775+0000 7fc43e800640 1 -- 172.21.15.125:0/1418174284 wait complete. 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.777+0000 7fc43e800640 1 Processor -- start 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.777+0000 7fc43e800640 1 -- start start 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4381f0ec0 0x7fc43815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4381f1340 0x7fc43815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438157be0 con 0x7fc418003970 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438157ec0 con 0x7fc4381f0ec0 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc43e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381581a0 con 0x7fc4381f1340 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49462/0 (socket says 172.21.15.125:49462) 2024-01-16T15:00:06.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 -- 172.21.15.125:0/3272152165 learned_addr learned my addr 172.21.15.125:0/3272152165 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 -- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4381f1340 msgr2=0x7fc43815e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437e00640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4381f1340 0x7fc43815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc436a00640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4381f0ec0 0x7fc43815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4381f1340 0x7fc43815e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 -- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4381f0ec0 msgr2=0x7fc43815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4381f0ec0 0x7fc43815e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.778+0000 7fc437400640 1 -- 172.21.15.125:0/3272152165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438158480 con 0x7fc418003970 2024-01-16T15:00:06.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc437400640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc428037040 tx=0x7fc428004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428003810 con 0x7fc418003970 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42802de90 con 0x7fc418003970 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438158760 con 0x7fc418003970 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428003b70 con 0x7fc418003970 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.779+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438158c80 con 0x7fc418003970 2024-01-16T15:00:06.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.780+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4380cc390 con 0x7fc418003970 2024-01-16T15:00:06.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.781+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc428003cd0 con 0x7fc418003970 2024-01-16T15:00:06.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.782+0000 7fc42f400640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.782+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4280abe90 con 0x7fc418003970 2024-01-16T15:00:06.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.782+0000 7fc436a00640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.783+0000 7fc436a00640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 0x7fc41006d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc420000f60 tx=0x7fc420009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.787+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc428076a20 con 0x7fc418003970 2024-01-16T15:00:06.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.977+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc4380675d0 con 0x7fc418003970 2024-01-16T15:00:06.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.978+0000 7fc42f400640 1 -- 172.21.15.125:0/3272152165 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc42807a040 con 0x7fc418003970 2024-01-16T15:00:06.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.980+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 msgr2=0x7fc41006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 0x7fc41006d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc420000f60 tx=0x7fc420009040 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 msgr2=0x7fc43813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc428037040 tx=0x7fc428004730 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 shutdown_connections 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4381f1340 0x7fc43815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4381f0ec0 0x7fc43815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 --2- 172.21.15.125:0/3272152165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc418003970 0x7fc43813d730 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 >> 172.21.15.125:0/3272152165 conn(0x7fc43806a800 msgr2=0x7fc43815d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 shutdown_connections 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:06.981+0000 7fc43e800640 1 -- 172.21.15.125:0/3272152165 wait complete. 2024-01-16T15:00:06.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:00:07.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:07.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:07.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.201+0000 7fe932c00640 1 Processor -- start 2024-01-16T15:00:07.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.202+0000 7fe932c00640 1 -- start start 2024-01-16T15:00:07.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe932c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe92c0cd240 0x7fe92c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe932c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c06c6b0 0x7fe92c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe932c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c06cea0 con 0x7fe92c099fa0 2024-01-16T15:00:07.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe932c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c09a430 con 0x7fe92c0cd240 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe932c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c06cbc0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe92be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe92c0cd240 0x7fe92c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe930c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe92c099fa0 0x7fe92c06b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44238/0 (socket says 172.21.15.125:44238) 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.203+0000 7fe930c00640 1 -- 172.21.15.125:0/3600918653 learned_addr learned my addr 172.21.15.125:0/3600918653 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.204+0000 7fe92be00640 1 -- 172.21.15.125:0/3600918653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c06c6b0 msgr2=0x7fe92c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.204+0000 7fe92be00640 1 --2- 172.21.15.125:0/3600918653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c06c6b0 0x7fe92c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.204+0000 7fe92be00640 1 -- 172.21.15.125:0/3600918653 >> v1:172.21.15.125:6789/0 conn(0x7fe92c099fa0 legacy=0x7fe92c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.204+0000 7fe92be00640 1 -- 172.21.15.125:0/3600918653 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c06f350 con 0x7fe92c0cd240 2024-01-16T15:00:07.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.204+0000 7fe92aa00640 1 -- 172.21.15.125:0/3600918653 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120403211 0 0) 0x7fe92c06cea0 con 0x7fe92c099fa0 2024-01-16T15:00:07.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.205+0000 7fe92be00640 1 --2- 172.21.15.125:0/3600918653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0cd240 0x7fe92c0b3820 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0097a0 tx=0x7fe91c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc76394e39ad7ed server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.205+0000 7fe92aa00640 1 -- 172.21.15.125:0/3600918653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c031830 con 0x7fe92c0cd240 2024-01-16T15:00:07.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.205+0000 7fe92aa00640 1 -- 172.21.15.125:0/3600918653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c031990 con 0x7fe92c0cd240 2024-01-16T15:00:07.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.205+0000 7fe92aa00640 1 -- 172.21.15.125:0/3600918653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c031d20 con 0x7fe92c0cd240 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 -- 172.21.15.125:0/3600918653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0cd240 msgr2=0x7fe92c0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 --2- 172.21.15.125:0/3600918653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0cd240 0x7fe92c0b3820 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0097a0 tx=0x7fe91c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 -- 172.21.15.125:0/3600918653 shutdown_connections 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 --2- 172.21.15.125:0/3600918653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c06c6b0 0x7fe92c099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 --2- 172.21.15.125:0/3600918653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0cd240 0x7fe92c0b3820 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 -- 172.21.15.125:0/3600918653 >> 172.21.15.125:0/3600918653 conn(0x7fe92c0b2d00 msgr2=0x7fe92c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 -- 172.21.15.125:0/3600918653 shutdown_connections 2024-01-16T15:00:07.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.206+0000 7fe932c00640 1 -- 172.21.15.125:0/3600918653 wait complete. 2024-01-16T15:00:07.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.207+0000 7fe932c00640 1 Processor -- start 2024-01-16T15:00:07.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 -- start start 2024-01-16T15:00:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe92c099fa0 0x7fe92c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c0cd240 0x7fe92c0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c151b80 con 0x7fe92c06c6b0 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0b4a70 con 0x7fe92c099fa0 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.208+0000 7fe932c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0b55f0 con 0x7fe92c0cd240 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49468/0 (socket says 172.21.15.125:49468) 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 -- 172.21.15.125:0/1570570953 learned_addr learned my addr 172.21.15.125:0/1570570953 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 -- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c0cd240 msgr2=0x7fe92c0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c0cd240 0x7fe92c0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 -- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe92c099fa0 msgr2=0x7fe92c0ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe92c099fa0 0x7fe92c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 -- 172.21.15.125:0/1570570953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c009400 con 0x7fe92c06c6b0 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.209+0000 7fe92be00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0098c0 tx=0x7fe91c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.210+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c03ba70 con 0x7fe92c06c6b0 2024-01-16T15:00:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.210+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c03bbf0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.210+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0453f0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.210+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c0b4cc0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.210+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c0b5280 con 0x7fe92c06c6b0 2024-01-16T15:00:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.211+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c06d0d0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.212+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe91c04e450 con 0x7fe92c06c6b0 2024-01-16T15:00:07.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.212+0000 7fe923e00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 0x7fe90406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.213+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe91c0ac440 con 0x7fe92c06c6b0 2024-01-16T15:00:07.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.213+0000 7fe92b400640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 0x7fe90406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.213+0000 7fe92b400640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 0x7fe90406d9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe92c09f2d0 tx=0x7fe914009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.217+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe91c076da0 con 0x7fe92c06c6b0 2024-01-16T15:00:07.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.416+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fe92c0cd620 con 0x7fe90406b500 2024-01-16T15:00:07.406 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:07.425+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:07.406 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:07.425+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:07.406 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:07.425+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:07.406 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:07.425+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:07.406 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:07.425+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.470+0000 7fe923e00640 1 -- 172.21.15.125:0/1570570953 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe91400a7a0 con 0x7fe90406b500 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.472+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 msgr2=0x7fe90406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.472+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 0x7fe90406d9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe92c09f2d0 tx=0x7fe914009250 comp rx=0 tx=0).stop 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 msgr2=0x7fe92c151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0098c0 tx=0x7fe91c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 shutdown_connections 2024-01-16T15:00:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92c0cd240 0x7fe92c0ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe92c099fa0 0x7fe92c0ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90406b500 0x7fe90406d9c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 --2- 172.21.15.125:0/1570570953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06c6b0 0x7fe92c151640 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 >> 172.21.15.125:0/1570570953 conn(0x7fe92c0b2d00 msgr2=0x7fe92c06e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 shutdown_connections 2024-01-16T15:00:07.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.473+0000 7fe932c00640 1 -- 172.21.15.125:0/1570570953 wait complete. 2024-01-16T15:00:07.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:00:07.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.678+0000 7f1568a00640 1 Processor -- start 2024-01-16T15:00:07.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.678+0000 7f1568a00640 1 -- start start 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1568a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1560143ae0 0x7f15600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1568a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600cfc30 con 0x7f15600cf7d0 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1568a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600b1ee0 con 0x7f156015e3b0 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1568a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15600b2780 con 0x7f1560143ae0 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1565600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f15600cf7d0 0x7f156015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44254/0 (socket says 172.21.15.125:44254) 2024-01-16T15:00:07.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1565600640 1 -- 172.21.15.125:0/924131154 learned_addr learned my addr 172.21.15.125:0/924131154 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:07.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1566000640 1 --2- 172.21.15.125:0/924131154 >> v2:172.21.15.137:3300/0 conn(0x7f1560143ae0 0x7f15600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.680+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707073675 0 0) 0x7f15600cfc30 con 0x7f15600cf7d0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540003520 con 0x7f15600cf7d0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888248506 0 0) 0x7f1540003520 con 0x7f15600cf7d0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 >> v2:172.21.15.137:3300/0 conn(0x7f1560143ae0 msgr2=0x7f15600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 --2- 172.21.15.125:0/924131154 >> v2:172.21.15.137:3300/0 conn(0x7f1560143ae0 0x7f15600cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 >> v1:172.21.15.137:6789/0 conn(0x7f156015e3b0 legacy=0x7f15600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156015e990 con 0x7f15600cf7d0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1548001670 con 0x7f15600cf7d0 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1566000640 1 --2- 172.21.15.125:0/924131154 >> v2:172.21.15.137:3300/0 conn(0x7f1560143ae0 0x7f15600cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 >> v1:172.21.15.125:6789/0 conn(0x7f15600cf7d0 legacy=0x7f156015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1540003970 0x7f1540003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f1540004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1540005180 0x7f154000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15600b2780 con 0x7f15400047c0 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154000e2e0 con 0x7f1540005180 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154000e4a0 con 0x7f1540003970 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.681+0000 7f1565600640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f1540004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1566000640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1540003970 0x7f1540003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1566a00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1540005180 0x7f154000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1565600640 1 -- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1540003970 msgr2=0x7f1540003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1565600640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1540003970 0x7f1540003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1565600640 1 -- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1540005180 msgr2=0x7f154000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1565600640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1540005180 0x7f154000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1565600640 1 -- 172.21.15.125:0/924131154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540003520 con 0x7f15400047c0 2024-01-16T15:00:07.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1566a00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1540005180 0x7f154000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:07.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.682+0000 7f1566a00640 1 -- 172.21.15.125:0/924131154 reap_dead start 2024-01-16T15:00:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.683+0000 7f1565600640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f1540004c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f154800c190 tx=0x7f15400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23465e35b48d4372 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.683+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548005ad0 con 0x7f15400047c0 2024-01-16T15:00:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.683+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548005c30 con 0x7f15400047c0 2024-01-16T15:00:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.683+0000 7f1564c00640 1 -- 172.21.15.125:0/924131154 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154802cae0 con 0x7f15400047c0 2024-01-16T15:00:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 -- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 msgr2=0x7f1540004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f1540004c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f154800c190 tx=0x7f15400057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 -- 172.21.15.125:0/924131154 shutdown_connections 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 --2- 172.21.15.125:0/924131154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f1540004c40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 -- 172.21.15.125:0/924131154 >> 172.21.15.125:0/924131154 conn(0x7f156006a800 msgr2=0x7f156006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 -- 172.21.15.125:0/924131154 shutdown_connections 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.684+0000 7f1568a00640 1 -- 172.21.15.125:0/924131154 wait complete. 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.686+0000 7f1568a00640 1 Processor -- start 2024-01-16T15:00:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.686+0000 7f1568a00640 1 -- start start 2024-01-16T15:00:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15600c63a0 0x7f15600afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15600c6820 0x7f15600b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f2cf0 con 0x7f15400047c0 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f2fd0 con 0x7f15600c6820 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1568a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f32b0 con 0x7f15600c63a0 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1566000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1566000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49496/0 (socket says 172.21.15.125:49496) 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1566000640 1 -- 172.21.15.125:0/218660873 learned_addr learned my addr 172.21.15.125:0/218660873 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.687+0000 7f1566a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15600c6820 0x7f15600b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1565600640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15600c63a0 0x7f15600afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 -- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15600c63a0 msgr2=0x7f15600afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15600c63a0 0x7f15600afba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 -- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15600c6820 msgr2=0x7f15600b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15600c6820 0x7f15600b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 -- 172.21.15.125:0/218660873 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15601f3590 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1566000640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f15500107b0 tx=0x7f1550010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000eb00 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155000c050 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000eea0 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15601f3ab0 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.688+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15600c6de0 con 0x7f15400047c0 2024-01-16T15:00:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.689+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15601f3780 con 0x7f15400047c0 2024-01-16T15:00:07.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.690+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1550015470 con 0x7f15400047c0 2024-01-16T15:00:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.691+0000 7f155e000640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 0x7f153806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.691+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f155008aad0 con 0x7f15400047c0 2024-01-16T15:00:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.692+0000 7f1565600640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 0x7f153806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.692+0000 7f1565600640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 0x7f153806d9f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1548034000 tx=0x7f1548009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.696+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15500554c0 con 0x7f15400047c0 2024-01-16T15:00:07.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.912+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f15600b2410 con 0x7f153806b530 2024-01-16T15:00:07.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.921+0000 7f155e000640 1 -- 172.21.15.125:0/218660873 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f154803d040 con 0x7f153806b530 2024-01-16T15:00:07.903 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02 2024-01-16T15:00:07.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 msgr2=0x7f153806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 0x7f153806d9f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1548034000 tx=0x7f1548009030 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 msgr2=0x7f15600cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f15500107b0 tx=0x7f1550010c70 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 shutdown_connections 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15600c6820 0x7f15600b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15600c63a0 0x7f15600afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153806b530 0x7f153806d9f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 --2- 172.21.15.125:0/218660873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15400047c0 0x7f15600cae20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 >> 172.21.15.125:0/218660873 conn(0x7f156006a800 msgr2=0x7f156015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 shutdown_connections 2024-01-16T15:00:07.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:07.924+0000 7f1568a00640 1 -- 172.21.15.125:0/218660873 wait complete. 2024-01-16T15:00:07.924 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02 2024-01-16T15:00:07.924 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:00:07.924 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:07.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:07.972 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:22.898 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:00:22.899 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:00:22.899 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.8022 s, 72.5 MB/s 2024-01-16T15:00:22.900 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:00:22.900 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:22.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:22.951 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:34.122 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:00:34.122 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:00:34.122 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.067 s, 97.0 MB/s 2024-01-16T15:00:34.124 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:00:34.124 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:34.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:34.174 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:37.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:37.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:00:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:00:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.0466 s, 97.2 MB/s 2024-01-16T15:00:45.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:00:45.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.550+0000 7f8086a00640 1 Processor -- start 2024-01-16T15:00:45.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.550+0000 7f8086a00640 1 -- start start 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f8086a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f808013e360 0x7f80800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f8086a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808015c5f0 con 0x7f808013df80 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f8086a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808006c470 con 0x7f808015f9c0 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f8086a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808006c750 con 0x7f808013e360 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f807fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f808013df80 0x7f808015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39628/0 (socket says 172.21.15.125:39628) 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f807fe00640 1 -- 172.21.15.125:0/447450254 learned_addr learned my addr 172.21.15.125:0/447450254 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:45.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.552+0000 7f807f400640 1 --2- 172.21.15.125:0/447450254 >> v2:172.21.15.125:3300/0 conn(0x7f808013e360 0x7f80800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.553+0000 7f807ea00640 1 -- 172.21.15.125:0/447450254 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244274958 0 0) 0x7f808015c5f0 con 0x7f808013df80 2024-01-16T15:00:45.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.553+0000 7f807ea00640 1 -- 172.21.15.125:0/447450254 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8060003520 con 0x7f808013df80 2024-01-16T15:00:45.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.553+0000 7f807f400640 1 -- 172.21.15.125:0/447450254 >> v1:172.21.15.137:6789/0 conn(0x7f808015f9c0 legacy=0x7f80800cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.553+0000 7f807f400640 1 -- 172.21.15.125:0/447450254 >> v1:172.21.15.125:6789/0 conn(0x7f808013df80 legacy=0x7f808015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.553+0000 7f807f400640 1 -- 172.21.15.125:0/447450254 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808006ca30 con 0x7f808013e360 2024-01-16T15:00:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.554+0000 7f807f400640 1 --2- 172.21.15.125:0/447450254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013e360 0x7f80800cf290 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f80680097a0 tx=0x7f806802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99059f121c3b1e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.554+0000 7f807ea00640 1 -- 172.21.15.125:0/447450254 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068031830 con 0x7f808013e360 2024-01-16T15:00:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f807ea00640 1 -- 172.21.15.125:0/447450254 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8068031990 con 0x7f808013e360 2024-01-16T15:00:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f807ea00640 1 -- 172.21.15.125:0/447450254 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068031d20 con 0x7f808013e360 2024-01-16T15:00:45.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 -- 172.21.15.125:0/447450254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013e360 msgr2=0x7f80800cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 --2- 172.21.15.125:0/447450254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013e360 0x7f80800cf290 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f80680097a0 tx=0x7f806802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 -- 172.21.15.125:0/447450254 shutdown_connections 2024-01-16T15:00:45.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 --2- 172.21.15.125:0/447450254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013e360 0x7f80800cf290 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 -- 172.21.15.125:0/447450254 >> 172.21.15.125:0/447450254 conn(0x7f808006a800 msgr2=0x7f808006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 -- 172.21.15.125:0/447450254 shutdown_connections 2024-01-16T15:00:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.555+0000 7f8086a00640 1 -- 172.21.15.125:0/447450254 wait complete. 2024-01-16T15:00:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.557+0000 7f8086a00640 1 Processor -- start 2024-01-16T15:00:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.557+0000 7f8086a00640 1 -- start start 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.558+0000 7f8086a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.558+0000 7f8086a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f808013e360 0x7f80801f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.558+0000 7f8086a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f808015f9c0 0x7f80800b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.558+0000 7f8086a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808014d110 con 0x7f808013df80 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.558+0000 7f8086a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808013de10 con 0x7f808015f9c0 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f8086a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80801f3790 con 0x7f808013e360 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57886/0 (socket says 172.21.15.125:57886) 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 -- 172.21.15.125:0/1254805048 learned_addr learned my addr 172.21.15.125:0/1254805048 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 -- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f808013e360 msgr2=0x7f80801f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f808013e360 0x7f80801f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 -- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f808015f9c0 msgr2=0x7f80800b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f808015f9c0 0x7f80800b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 -- 172.21.15.125:0/1254805048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8068009400 con 0x7f808013df80 2024-01-16T15:00:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.559+0000 7f807fe00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f807000c400 tx=0x7f807000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.560+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070005ce0 con 0x7f808013df80 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.560+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8070005e40 con 0x7f808013df80 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.560+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80700072d0 con 0x7f808013df80 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.560+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80801f3a70 con 0x7f808013df80 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.560+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80800b2d90 con 0x7f808013df80 2024-01-16T15:00:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.561+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80800675d0 con 0x7f808013df80 2024-01-16T15:00:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.562+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f80700074b0 con 0x7f808013df80 2024-01-16T15:00:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.563+0000 7f8077e00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 0x7f805806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.563+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8070089630 con 0x7f808013df80 2024-01-16T15:00:45.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.563+0000 7f807f400640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 0x7f805806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.564+0000 7f807f400640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 0x7f805806da50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8068036000 tx=0x7f8068030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.568+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80700540e0 con 0x7f808013df80 2024-01-16T15:00:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.759+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f808006df00 con 0x7f808013df80 2024-01-16T15:00:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.765+0000 7f8077e00640 1 -- 172.21.15.125:0/1254805048 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7f8070057700 con 0x7f808013df80 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.767+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 msgr2=0x7f805806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.767+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 0x7f805806da50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8068036000 tx=0x7f8068030000 comp rx=0 tx=0).stop 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 msgr2=0x7f808013d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f807000c400 tx=0x7f807000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 shutdown_connections 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f808015f9c0 0x7f80800b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f808013e360 0x7f80801f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f805806b590 0x7f805806da50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 --2- 172.21.15.125:0/1254805048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f808013df80 0x7f808013d8d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 >> 172.21.15.125:0/1254805048 conn(0x7f808006a800 msgr2=0x7f808015dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 shutdown_connections 2024-01-16T15:00:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.768+0000 7f8086a00640 1 -- 172.21.15.125:0/1254805048 wait complete. 2024-01-16T15:00:45.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:00:45.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.996+0000 7f930a200640 1 Processor -- start 2024-01-16T15:00:45.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.996+0000 7f930a200640 1 -- start start 2024-01-16T15:00:45.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f930a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930413e360 0x7f93040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f930a200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f930415f9c0 0x7f93040cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f930a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930415c5f0 con 0x7f930413df80 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f930a200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930406c470 con 0x7f930415f9c0 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f930a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930406c750 con 0x7f930413e360 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.998+0000 7f9303e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f930415f9c0 0x7f93040cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f930415f9c0 0x7f93040cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57900/0 (socket says 172.21.15.125:57900) 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 -- 172.21.15.125:0/2816548824 learned_addr learned my addr 172.21.15.125:0/2816548824 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f92fa800640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930413e360 0x7f93040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 -- 172.21.15.125:0/2816548824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930413e360 msgr2=0x7f93040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930413e360 0x7f93040cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 -- 172.21.15.125:0/2816548824 >> v1:172.21.15.137:6789/0 conn(0x7f930413df80 legacy=0x7f930415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:45.999+0000 7f9303e00640 1 -- 172.21.15.125:0/2816548824 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930406ca30 con 0x7f930415f9c0 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.000+0000 7f9303e00640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930415f9c0 0x7f93040cf7d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e36fc1afee86a1a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.000+0000 7f9302a00640 1 -- 172.21.15.125:0/2816548824 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc030840 con 0x7f930415f9c0 2024-01-16T15:00:45.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f9302a00640 1 -- 172.21.15.125:0/2816548824 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc0309a0 con 0x7f930415f9c0 2024-01-16T15:00:45.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f9302a00640 1 -- 172.21.15.125:0/2816548824 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc030d30 con 0x7f930415f9c0 2024-01-16T15:00:45.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 -- 172.21.15.125:0/2816548824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930415f9c0 msgr2=0x7f93040cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930415f9c0 0x7f93040cf7d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:45.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 -- 172.21.15.125:0/2816548824 shutdown_connections 2024-01-16T15:00:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930415f9c0 0x7f93040cf7d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 --2- 172.21.15.125:0/2816548824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930413e360 0x7f93040cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 -- 172.21.15.125:0/2816548824 >> 172.21.15.125:0/2816548824 conn(0x7f930406a800 msgr2=0x7f930406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 -- 172.21.15.125:0/2816548824 shutdown_connections 2024-01-16T15:00:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.001+0000 7f930a200640 1 -- 172.21.15.125:0/2816548824 wait complete. 2024-01-16T15:00:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.003+0000 7f930a200640 1 Processor -- start 2024-01-16T15:00:45.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.003+0000 7f930a200640 1 -- start start 2024-01-16T15:00:45.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 0x7f93041f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 0x7f930415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040702c0 con 0x7f930413e360 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040704f0 con 0x7f930413df80 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f930a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93040707d0 con 0x7f930415f9c0 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.004+0000 7f92fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57904/0 (socket says 172.21.15.125:57904) 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 -- 172.21.15.125:0/269189848 learned_addr learned my addr 172.21.15.125:0/269189848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f9303400640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 0x7f93041f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f9303e00640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 0x7f930415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 -- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 msgr2=0x7f930415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 0x7f930415a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 -- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 msgr2=0x7f93041f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 0x7f93041f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 -- 172.21.15.125:0/269189848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc0094d0 con 0x7f930413e360 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f9303400640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 0x7f93041f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f9303e00640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 0x7f930415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.005+0000 7f92fa800640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00e9b0 tx=0x7f92ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.006+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec009670 con 0x7f930413e360 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.006+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec00c9a0 con 0x7f930413e360 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.006+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00cd50 con 0x7f930413e360 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.006+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9304070ab0 con 0x7f930413e360 2024-01-16T15:00:45.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.006+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9304071070 con 0x7f930413e360 2024-01-16T15:00:45.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.007+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93040675d0 con 0x7f930413e360 2024-01-16T15:00:45.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.008+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f92ec013470 con 0x7f930413e360 2024-01-16T15:00:45.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.008+0000 7f92fbe00640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 0x7f92dc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.009+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f92ec08a2b0 con 0x7f930413e360 2024-01-16T15:00:45.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.009+0000 7f9303400640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 0x7f92dc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.009+0000 7f9303400640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 0x7f92dc06da50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f92f40012e0 tx=0x7f92f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.013+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92ec054d50 con 0x7f930413e360 2024-01-16T15:00:46.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.205+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f93040b43e0 con 0x7f930413e360 2024-01-16T15:00:46.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.212+0000 7f92fbe00640 1 -- 172.21.15.125:0/269189848 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7f92ec058370 con 0x7f930413e360 2024-01-16T15:00:46.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 msgr2=0x7f92dc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 0x7f92dc06da50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f92f40012e0 tx=0x7f92f4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 msgr2=0x7f930413d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00e9b0 tx=0x7f92ec00ee70 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 shutdown_connections 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f930415f9c0 0x7f930415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92dc06b590 0x7f92dc06da50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f930413e360 0x7f930413d730 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 --2- 172.21.15.125:0/269189848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f930413df80 0x7f93041f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.215+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 >> 172.21.15.125:0/269189848 conn(0x7f930406a800 msgr2=0x7f930415ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.216+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 shutdown_connections 2024-01-16T15:00:46.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.216+0000 7f930a200640 1 -- 172.21.15.125:0/269189848 wait complete. 2024-01-16T15:00:46.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:00:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.442+0000 7f455d400640 1 Processor -- start 2024-01-16T15:00:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.442+0000 7f455d400640 1 -- start start 2024-01-16T15:00:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f455d400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f455815e880 0x7f455806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f455d400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580cfa70 con 0x7f4558070490 2024-01-16T15:00:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f455d400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580cff00 con 0x7f455809a170 2024-01-16T15:00:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f455d400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455806fc70 con 0x7f455815e880 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f4556800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f455815e880 0x7f455806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f4556800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f455815e880 0x7f455806ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57912/0 (socket says 172.21.15.125:57912) 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f4555e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4558070490 0x7f4558099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39630/0 (socket says 172.21.15.125:39630) 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.444+0000 7f4556800640 1 -- 172.21.15.125:0/2487406206 learned_addr learned my addr 172.21.15.125:0/2487406206 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.445+0000 7f4556800640 1 -- 172.21.15.125:0/2487406206 >> v1:172.21.15.137:6789/0 conn(0x7f455809a170 legacy=0x7f455806e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.445+0000 7f4556800640 1 -- 172.21.15.125:0/2487406206 >> v1:172.21.15.125:6789/0 conn(0x7f4558070490 legacy=0x7f4558099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.445+0000 7f4556800640 1 -- 172.21.15.125:0/2487406206 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580cf420 con 0x7f455815e880 2024-01-16T15:00:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.446+0000 7f4556800640 1 --2- 172.21.15.125:0/2487406206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f455806ff50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4548009e50 tx=0x7f454802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7c3a8da1e05a71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.446+0000 7f4555400640 1 -- 172.21.15.125:0/2487406206 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454802cb90 con 0x7f455815e880 2024-01-16T15:00:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.446+0000 7f4555400640 1 -- 172.21.15.125:0/2487406206 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454802ccf0 con 0x7f455815e880 2024-01-16T15:00:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.446+0000 7f4555400640 1 -- 172.21.15.125:0/2487406206 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548034610 con 0x7f455815e880 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 -- 172.21.15.125:0/2487406206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 msgr2=0x7f455806ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 --2- 172.21.15.125:0/2487406206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f455806ff50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4548009e50 tx=0x7f454802bff0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 -- 172.21.15.125:0/2487406206 shutdown_connections 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 --2- 172.21.15.125:0/2487406206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f455806ff50 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 -- 172.21.15.125:0/2487406206 >> 172.21.15.125:0/2487406206 conn(0x7f45580b2d00 msgr2=0x7f45580b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 -- 172.21.15.125:0/2487406206 shutdown_connections 2024-01-16T15:00:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.447+0000 7f455d400640 1 -- 172.21.15.125:0/2487406206 wait complete. 2024-01-16T15:00:46.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.449+0000 7f455d400640 1 Processor -- start 2024-01-16T15:00:46.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.449+0000 7f455d400640 1 -- start start 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 0x7f455814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455809a170 0x7f45581f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580c6730 con 0x7f455815e880 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580b48b0 con 0x7f455809a170 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f455d400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45580c6f90 con 0x7f4558070490 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f4557200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f4557200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57920/0 (socket says 172.21.15.125:57920) 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.450+0000 7f4557200640 1 -- 172.21.15.125:0/292231038 learned_addr learned my addr 172.21.15.125:0/292231038 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4555e00640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455809a170 0x7f45581f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4556800640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 0x7f455814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 -- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 msgr2=0x7f455814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 0x7f455814ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 -- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455809a170 msgr2=0x7f45581f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455809a170 0x7f45581f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 -- 172.21.15.125:0/292231038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548009b30 con 0x7f455815e880 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4556800640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 0x7f455814ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f4557200640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f455000c410 tx=0x7f455000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550011790 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45500118f0 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550011ce0 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.451+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455814b6a0 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.452+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45580c6940 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.453+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45580675d0 con 0x7f455815e880 2024-01-16T15:00:46.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.454+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f455001c460 con 0x7f455815e880 2024-01-16T15:00:46.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.455+0000 7f454ea00640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 0x7f453006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.455+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4550089d70 con 0x7f455815e880 2024-01-16T15:00:46.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.455+0000 7f4556800640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 0x7f453006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.456+0000 7f4556800640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 0x7f453006da50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4548035000 tx=0x7f4548030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.460+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4550054810 con 0x7f455815e880 2024-01-16T15:00:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.681+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f45580ccb30 con 0x7f453006b590 2024-01-16T15:00:46.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.712+0000 7f454ea00640 1 -- 172.21.15.125:0/292231038 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4548040540 con 0x7f453006b590 2024-01-16T15:00:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 msgr2=0x7f453006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 0x7f453006da50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4548035000 tx=0x7f4548030000 comp rx=0 tx=0).stop 2024-01-16T15:00:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 msgr2=0x7f45581472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f455000c410 tx=0x7f455000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 shutdown_connections 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f453006b590 0x7f453006da50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f455815e880 0x7f45581472a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455809a170 0x7f45581f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 --2- 172.21.15.125:0/292231038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4558070490 0x7f455814ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 >> 172.21.15.125:0/292231038 conn(0x7f45580b2d00 msgr2=0x7f45580ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 shutdown_connections 2024-01-16T15:00:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.715+0000 7f455d400640 1 -- 172.21.15.125:0/292231038 wait complete. 2024-01-16T15:00:46.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:00:46.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.942+0000 7fcee3200640 1 Processor -- start 2024-01-16T15:00:46.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.942+0000 7fcee3200640 1 -- start start 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee3200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcee4143810 0x7fcee4071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee3200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40ae620 con 0x7fcee415d770 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee3200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee406c730 con 0x7fcee40ae170 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee3200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee4143bf0 con 0x7fcee4143810 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee2800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcee415d770 0x7fcee415dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44532/0 (socket says 172.21.15.125:44532) 2024-01-16T15:00:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee2800640 1 -- 172.21.15.125:0/4006359588 learned_addr learned my addr 172.21.15.125:0/4006359588 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:46.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee1e00640 1 --2- 172.21.15.125:0/4006359588 >> v2:172.21.15.137:3300/0 conn(0x7fcee4143810 0x7fcee4071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.944+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700713945 0 0) 0x7fcee40ae620 con 0x7fcee415d770 2024-01-16T15:00:46.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec4003520 con 0x7fcee415d770 2024-01-16T15:00:46.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854757973 0 0) 0x7fcec4003520 con 0x7fcee415d770 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 >> v2:172.21.15.137:3300/0 conn(0x7fcee4143810 msgr2=0x7fcee4071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 --2- 172.21.15.125:0/4006359588 >> v2:172.21.15.137:3300/0 conn(0x7fcee4143810 0x7fcee4071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 >> v1:172.21.15.137:6789/0 conn(0x7fcee40ae170 legacy=0x7fcee415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40724d0 con 0x7fcee415d770 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcedc001670 con 0x7fcee415d770 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 >> v1:172.21.15.125:6789/0 conn(0x7fcee415d770 legacy=0x7fcee415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 0x7fcec4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec40047c0 0x7fcec4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcec40052a0 0x7fcec400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4143bf0 con 0x7fcec4003970 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec400c170 con 0x7fcec40047c0 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec400c330 con 0x7fcec40052a0 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.945+0000 7fcee1e00640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 0x7fcec4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fced9200640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec40047c0 0x7fcec4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee1e00640 1 -- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcec40052a0 msgr2=0x7fcec400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee1e00640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcec40052a0 0x7fcec400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee1e00640 1 -- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec40047c0 msgr2=0x7fcec4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee1e00640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec40047c0 0x7fcec4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee1e00640 1 -- 172.21.15.125:0/4006359588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4003520 con 0x7fcec4003970 2024-01-16T15:00:46.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.946+0000 7fcee2800640 1 -- 172.21.15.125:0/4006359588 reap_dead start 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.947+0000 7fcee1e00640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 0x7fcec4003d90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fced4009f90 tx=0x7fcec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322ded7d81ce958c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.947+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced402cbd0 con 0x7fcec4003970 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.947+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced400f040 con 0x7fcec4003970 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.947+0000 7fcee1400640 1 -- 172.21.15.125:0/4006359588 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4034440 con 0x7fcec4003970 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 -- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 msgr2=0x7fcec4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 0x7fcec4003d90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fced4009f90 tx=0x7fcec40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 -- 172.21.15.125:0/4006359588 shutdown_connections 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 --2- 172.21.15.125:0/4006359588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcec4003970 0x7fcec4003d90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 -- 172.21.15.125:0/4006359588 >> 172.21.15.125:0/4006359588 conn(0x7fcee406a7d0 msgr2=0x7fcee406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 -- 172.21.15.125:0/4006359588 shutdown_connections 2024-01-16T15:00:46.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.948+0000 7fcee3200640 1 -- 172.21.15.125:0/4006359588 wait complete. 2024-01-16T15:00:46.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.950+0000 7fcee3200640 1 Processor -- start 2024-01-16T15:00:46.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.950+0000 7fcee3200640 1 -- start start 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec4003970 0x7fcee40b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b09a0 0x7fcee40b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee41442f0 con 0x7fcee40cd0d0 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40cbc90 con 0x7fcec4003970 2024-01-16T15:00:46.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40ce750 con 0x7fcee40b09a0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51542/0 (socket says 172.21.15.125:51542) 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee2800640 1 -- 172.21.15.125:0/2786315566 learned_addr learned my addr 172.21.15.125:0/2786315566 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fcee1e00640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec4003970 0x7fcee40b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.951+0000 7fced9200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b09a0 0x7fcee40b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 -- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b09a0 msgr2=0x7fcee40b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b09a0 0x7fcee40b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 -- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec4003970 msgr2=0x7fcee40b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec4003970 0x7fcee40b0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 -- 172.21.15.125:0/2786315566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40cb710 con 0x7fcee40cd0d0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee2800640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcee415db50 tx=0x7fcedc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0074e0 con 0x7fcee40cd0d0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc007640 con 0x7fcee40cd0d0 2024-01-16T15:00:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4144710 con 0x7fcee40cd0d0 2024-01-16T15:00:46.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc004a20 con 0x7fcee40cd0d0 2024-01-16T15:00:46.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.952+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee415cb90 con 0x7fcee40cd0d0 2024-01-16T15:00:46.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.953+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4144430 con 0x7fcee40cd0d0 2024-01-16T15:00:46.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.954+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fcedc004b80 con 0x7fcee40cd0d0 2024-01-16T15:00:46.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.955+0000 7fcedaa00640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 0x7fcebc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.955+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcedc089200 con 0x7fcee40cd0d0 2024-01-16T15:00:46.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.956+0000 7fcee1e00640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 0x7fcebc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.956+0000 7fcee1e00640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 0x7fcebc06da50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fced4036000 tx=0x7fced402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:46.960+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcee4144430 con 0x7fcee40cd0d0 2024-01-16T15:00:47.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.178+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee4144090 con 0x7fcebc06b590 2024-01-16T15:00:47.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.186+0000 7fcedaa00640 1 -- 172.21.15.125:0/2786315566 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fced402caf0 con 0x7fcebc06b590 2024-01-16T15:00:47.166 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.188+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 msgr2=0x7fcebc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.188+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 0x7fcebc06da50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fced4036000 tx=0x7fced402e000 comp rx=0 tx=0).stop 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 msgr2=0x7fcee40cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcee415db50 tx=0x7fcedc00c890 comp rx=0 tx=0).stop 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 shutdown_connections 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcebc06b590 0x7fcebc06da50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee40cd0d0 0x7fcee40cd550 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b09a0 0x7fcee40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 --2- 172.21.15.125:0/2786315566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcec4003970 0x7fcee40b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 >> 172.21.15.125:0/2786315566 conn(0x7fcee406a7d0 msgr2=0x7fcee4142020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 shutdown_connections 2024-01-16T15:00:47.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:47.189+0000 7fcee3200640 1 -- 172.21.15.125:0/2786315566 wait complete. 2024-01-16T15:00:47.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:47.188 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02) 2024-01-16T15:00:47.240 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02 2024-01-16T15:00:47.241 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:00:47.241 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:00:47.241 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:00:49.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:49.243 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02) 2024-01-16T15:00:49.306 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/820b732e-a605-4d90-be06-45928342fc02 2024-01-16T15:00:49.306 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:00:49.306 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:00:49.307 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:00:49.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone20151 2024-01-16T15:00:49.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.537+0000 7f1196a00640 1 Processor -- start 2024-01-16T15:00:49.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.537+0000 7f1196a00640 1 -- start start 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1190160920 0x7f11900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11900cf9f0 0x7f11900cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11900d0a10 con 0x7f1190161010 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11900d03b0 con 0x7f1190160920 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900d1f50 con 0x7f11900cf9f0 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1195600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1190160920 0x7f11900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1195600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1190160920 0x7f11900b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51558/0 (socket says 172.21.15.125:51558) 2024-01-16T15:00:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1195600640 1 -- 172.21.15.125:0/3754573458 learned_addr learned my addr 172.21.15.125:0/3754573458 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.539+0000 7f1196000640 1 --1- 172.21.15.125:0/3754573458 >> v1:172.21.15.125:6789/0 conn(0x7f1190161010 0x7f11901614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44540/0 (socket says 172.21.15.125:44540) 2024-01-16T15:00:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.540+0000 7f1194c00640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11900cf9f0 0x7f11900cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.540+0000 7f1195600640 1 -- 172.21.15.125:0/3754573458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11900cf9f0 msgr2=0x7f11900cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.540+0000 7f1195600640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11900cf9f0 0x7f11900cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.540+0000 7f1195600640 1 -- 172.21.15.125:0/3754573458 >> v1:172.21.15.125:6789/0 conn(0x7f1190161010 legacy=0x7f11901614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.540+0000 7f1195600640 1 -- 172.21.15.125:0/3754573458 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900cf080 con 0x7f1190160920 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.541+0000 7f1195600640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190160920 0x7f11900b3f30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1180009870 tx=0x7f118002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b4205da8a18a6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.541+0000 7f118fe00640 1 -- 172.21.15.125:0/3754573458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180030840 con 0x7f1190160920 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.541+0000 7f118fe00640 1 -- 172.21.15.125:0/3754573458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11800309a0 con 0x7f1190160920 2024-01-16T15:00:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.541+0000 7f118fe00640 1 -- 172.21.15.125:0/3754573458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180030d30 con 0x7f1190160920 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 -- 172.21.15.125:0/3754573458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190160920 msgr2=0x7f11900b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190160920 0x7f11900b3f30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1180009870 tx=0x7f118002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 -- 172.21.15.125:0/3754573458 shutdown_connections 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11900cf9f0 0x7f11900cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 --2- 172.21.15.125:0/3754573458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190160920 0x7f11900b3f30 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 -- 172.21.15.125:0/3754573458 >> 172.21.15.125:0/3754573458 conn(0x7f11900d3090 msgr2=0x7f11900d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 -- 172.21.15.125:0/3754573458 shutdown_connections 2024-01-16T15:00:49.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.542+0000 7f1196a00640 1 -- 172.21.15.125:0/3754573458 wait complete. 2024-01-16T15:00:49.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.544+0000 7f1196a00640 1 Processor -- start 2024-01-16T15:00:49.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.544+0000 7f1196a00640 1 -- start start 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11900cf9f0 0x7f119015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1190160920 0x7f119014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11900b6720 con 0x7f1190161010 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119015f700 con 0x7f11900cf9f0 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119015f9e0 con 0x7f1190160920 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51572/0 (socket says 172.21.15.125:51572) 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.545+0000 7f1196000640 1 -- 172.21.15.125:0/2512972681 learned_addr learned my addr 172.21.15.125:0/2512972681 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1195600640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11900cf9f0 0x7f119015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1194c00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1190160920 0x7f119014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 -- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1190160920 msgr2=0x7f119014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1190160920 0x7f119014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 -- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11900cf9f0 msgr2=0x7f119015c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11900cf9f0 0x7f119015c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 -- 172.21.15.125:0/2512972681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11800094d0 con 0x7f1190161010 2024-01-16T15:00:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196000640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1188001eb0 tx=0x7f118800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11880074e0 con 0x7f1190161010 2024-01-16T15:00:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188007640 con 0x7f1190161010 2024-01-16T15:00:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11880049e0 con 0x7f1190161010 2024-01-16T15:00:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.546+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119015fcc0 con 0x7f1190161010 2024-01-16T15:00:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.547+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119014c8d0 con 0x7f1190161010 2024-01-16T15:00:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.548+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11900b6890 con 0x7f1190161010 2024-01-16T15:00:49.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.549+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f1188004b40 con 0x7f1190161010 2024-01-16T15:00:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.550+0000 7f118d600640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 0x7f116806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.550+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1188089a20 con 0x7f1190161010 2024-01-16T15:00:49.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.551+0000 7f1195600640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 0x7f116806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.551+0000 7f1195600640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 0x7f116806da50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1180036000 tx=0x7f1180008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.555+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11900b6890 con 0x7f1190161010 2024-01-16T15:00:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.781+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone20151", "target": ["mon-mgr", ""]}) v1 -- 0x7f11900ceac0 con 0x7f116806b590 2024-01-16T15:00:49.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.941+0000 7f118d600640 1 -- 172.21.15.125:0/2512972681 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1180030b90 con 0x7f116806b590 2024-01-16T15:00:49.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 msgr2=0x7f116806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 0x7f116806da50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1180036000 tx=0x7f1180008030 comp rx=0 tx=0).stop 2024-01-16T15:00:49.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 msgr2=0x7f119015ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1188001eb0 tx=0x7f118800c820 comp rx=0 tx=0).stop 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 shutdown_connections 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f116806b590 0x7f116806da50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1190161010 0x7f119015ced0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1190160920 0x7f119014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 --2- 172.21.15.125:0/2512972681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11900cf9f0 0x7f119015c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 >> 172.21.15.125:0/2512972681 conn(0x7f11900d3090 msgr2=0x7f119006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 shutdown_connections 2024-01-16T15:00:49.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:49.945+0000 7f1196a00640 1 -- 172.21.15.125:0/2512972681 wait complete. 2024-01-16T15:00:49.929 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.949+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.929 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.949+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.929 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.949+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.929 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.949+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.929 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.949+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone20152 2024-01-16T15:00:49.951 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.952 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.952 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.952 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:49.952 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:00:49.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.159+0000 7f45bb800640 1 Processor -- start 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.159+0000 7f45bb800640 1 -- start start 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 0x7f45b40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b40d0b60 0x7f45b40aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45b406a7c0 0x7f45b406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40af320 con 0x7f45b406a7c0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b4071de0 con 0x7f45b406a3e0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45bb800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4072030 con 0x7f45b40d0b60 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45b9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45b406a7c0 0x7f45b406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45b9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45b406a7c0 0x7f45b406f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51604/0 (socket says 172.21.15.125:51604) 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.161+0000 7f45b9800640 1 -- 172.21.15.125:0/3010153453 learned_addr learned my addr 172.21.15.125:0/3010153453 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b3e00640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b40d0b60 0x7f45b40aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b8e00640 1 --2- 172.21.15.125:0/3010153453 >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 0x7f45b40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b9800640 1 -- 172.21.15.125:0/3010153453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b40d0b60 msgr2=0x7f45b40aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b9800640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b40d0b60 0x7f45b40aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b9800640 1 -- 172.21.15.125:0/3010153453 >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 msgr2=0x7f45b40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b9800640 1 --2- 172.21.15.125:0/3010153453 >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 0x7f45b40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b9800640 1 -- 172.21.15.125:0/3010153453 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b4072310 con 0x7f45b406a7c0 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.162+0000 7f45b8e00640 1 --2- 172.21.15.125:0/3010153453 >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 0x7f45b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.163+0000 7f45b9800640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b406f080 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f45b4073fe0 tx=0x7f45ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0a228aea9ddad7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.163+0000 7f45b3400640 1 -- 172.21.15.125:0/3010153453 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac031870 con 0x7f45b406a7c0 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.163+0000 7f45b3400640 1 -- 172.21.15.125:0/3010153453 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0319d0 con 0x7f45b406a7c0 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45b3400640 1 -- 172.21.15.125:0/3010153453 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac031d60 con 0x7f45b406a7c0 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 -- 172.21.15.125:0/3010153453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 msgr2=0x7f45b406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b406f080 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f45b4073fe0 tx=0x7f45ac02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 -- 172.21.15.125:0/3010153453 shutdown_connections 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b406f080 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 --2- 172.21.15.125:0/3010153453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b40d0b60 0x7f45b40aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 --2- 172.21.15.125:0/3010153453 >> v2:172.21.15.137:3300/0 conn(0x7f45b406a3e0 0x7f45b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 -- 172.21.15.125:0/3010153453 >> 172.21.15.125:0/3010153453 conn(0x7f45b40cfef0 msgr2=0x7f45b40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 -- 172.21.15.125:0/3010153453 shutdown_connections 2024-01-16T15:00:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.164+0000 7f45bb800640 1 -- 172.21.15.125:0/3010153453 wait complete. 2024-01-16T15:00:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.166+0000 7f45bb800640 1 Processor -- start 2024-01-16T15:00:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.166+0000 7f45bb800640 1 -- start start 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b406a3e0 0x7f45b4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45b40d0b60 0x7f45b4156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41f3a70 con 0x7f45b406a7c0 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41f34b0 con 0x7f45b40d0b60 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45bb800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41f3790 con 0x7f45b406a3e0 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51612/0 (socket says 172.21.15.125:51612) 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.167+0000 7f45b3e00640 1 -- 172.21.15.125:0/283644576 learned_addr learned my addr 172.21.15.125:0/283644576 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b8e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b406a3e0 0x7f45b4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b9800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45b40d0b60 0x7f45b4156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 -- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b406a3e0 msgr2=0x7f45b4152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b406a3e0 0x7f45b4152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 -- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45b40d0b60 msgr2=0x7f45b4156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45b40d0b60 0x7f45b4156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 -- 172.21.15.125:0/283644576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0094d0 con 0x7f45b406a7c0 2024-01-16T15:00:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b3e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f45a4007e30 tx=0x7f45a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4014620 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a400ee90 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.168+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a40149c0 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.169+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b40cdf70 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.169+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b415b510 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.170+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b415b730 con 0x7f45b406a7c0 2024-01-16T15:00:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.171+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f45a4014b20 con 0x7f45b406a7c0 2024-01-16T15:00:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.172+0000 7f45b0c00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 0x7f458c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.172+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f45a408ab50 con 0x7f45b406a7c0 2024-01-16T15:00:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.172+0000 7f45b8e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 0x7f458c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.173+0000 7f45b8e00640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 0x7f458c06db70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f459c00a990 tx=0x7f459c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.176+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45a4055430 con 0x7f45b406a7c0 2024-01-16T15:00:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.404+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone20152", "target": ["mon-mgr", ""]}) v1 -- 0x7f45b41f3900 con 0x7f458c06b6b0 2024-01-16T15:00:50.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.553+0000 7f45b0c00640 1 -- 172.21.15.125:0/283644576 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45b41f3900 con 0x7f458c06b6b0 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.556+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 msgr2=0x7f458c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.556+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 0x7f458c06db70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f459c00a990 tx=0x7f459c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.556+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 msgr2=0x7f45b41f2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.556+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f45a4007e30 tx=0x7f45a400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 shutdown_connections 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45b40d0b60 0x7f45b4156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f458c06b6b0 0x7f458c06db70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b406a7c0 0x7f45b41f2a80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 --2- 172.21.15.125:0/283644576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45b406a3e0 0x7f45b4152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 >> 172.21.15.125:0/283644576 conn(0x7f45b40cfef0 msgr2=0x7f45b40ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 shutdown_connections 2024-01-16T15:00:50.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.557+0000 7f45bb800640 1 -- 172.21.15.125:0/283644576 wait complete. 2024-01-16T15:00:50.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone20153 2024-01-16T15:00:50.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.772+0000 7fdc03000640 1 Processor -- start 2024-01-16T15:00:50.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.772+0000 7fdc03000640 1 -- start start 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc03000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbfc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc03000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdbfc0717d0 0x7fdbfc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc03000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc070e50 con 0x7fdbfc0cca40 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc03000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc0723c0 con 0x7fdbfc0717d0 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc03000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0cce70 con 0x7fdbfc143c40 2024-01-16T15:00:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc02600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdbfc0cca40 0x7fdbfc15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44544/0 (socket says 172.21.15.125:44544) 2024-01-16T15:00:50.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.774+0000 7fdc02600640 1 -- 172.21.15.125:0/3505287312 learned_addr learned my addr 172.21.15.125:0/3505287312 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdc01200640 1 --2- 172.21.15.125:0/3505287312 >> v2:172.21.15.137:3300/0 conn(0x7fdbfc0717d0 0x7fdbfc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbfc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441765594 0 0) 0x7fdbfc070e50 con 0x7fdbfc0cca40 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbdc003520 con 0x7fdbfc0cca40 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 852101 0 0) 0x7fdbdc003520 con 0x7fdbfc0cca40 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 msgr2=0x7fdbfc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbfc071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 >> v2:172.21.15.137:3300/0 conn(0x7fdbfc0717d0 msgr2=0x7fdbfc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 --2- 172.21.15.125:0/3505287312 >> v2:172.21.15.137:3300/0 conn(0x7fdbfc0717d0 0x7fdbfc0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc06e170 con 0x7fdbfc0cca40 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdbf40019f0 con 0x7fdbfc0cca40 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbfc071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 >> v1:172.21.15.125:6789/0 conn(0x7fdbfc0cca40 legacy=0x7fdbfc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbdc0047c0 0x7fdbdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0723c0 con 0x7fdbdc003970 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0cce70 con 0x7fdbdc0047c0 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc00bab0 con 0x7fdbfc143c40 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.775+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc02600640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01200640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbdc0047c0 0x7fdbdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01c00640 1 -- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 msgr2=0x7fdbdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc143c40 0x7fdbdc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01c00640 1 -- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbdc0047c0 msgr2=0x7fdbdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbdc0047c0 0x7fdbdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.776+0000 7fdc01c00640 1 -- 172.21.15.125:0/3505287312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc003520 con 0x7fdbdc003970 2024-01-16T15:00:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.777+0000 7fdc01c00640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbdc003d90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdbec006fd0 tx=0x7fdbdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0b08ad2ca3b435 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.777+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec004230 con 0x7fdbdc003970 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.777+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec004390 con 0x7fdbdc003970 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.777+0000 7fdbfbe00640 1 -- 172.21.15.125:0/3505287312 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec003830 con 0x7fdbdc003970 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 -- 172.21.15.125:0/3505287312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 msgr2=0x7fdbdc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 --2- 172.21.15.125:0/3505287312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbdc003d90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdbec006fd0 tx=0x7fdbdc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc02600640 1 -- 172.21.15.125:0/3505287312 reap_dead start 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 -- 172.21.15.125:0/3505287312 shutdown_connections 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 -- 172.21.15.125:0/3505287312 >> 172.21.15.125:0/3505287312 conn(0x7fdbfc06a800 msgr2=0x7fdbfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 -- 172.21.15.125:0/3505287312 shutdown_connections 2024-01-16T15:00:50.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.778+0000 7fdc03000640 1 -- 172.21.15.125:0/3505287312 wait complete. 2024-01-16T15:00:50.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.780+0000 7fdc03000640 1 Processor -- start 2024-01-16T15:00:50.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.780+0000 7fdc03000640 1 -- start start 2024-01-16T15:00:50.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc13cfc0 0x7fdbfc137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbfc13e540 0x7fdbfc0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1f07d0 con 0x7fdbdc003970 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc13d4a0 con 0x7fdbfc13e540 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc03000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc13d780 con 0x7fdbfc13cfc0 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51634/0 (socket says 172.21.15.125:51634) 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 -- 172.21.15.125:0/1032262550 learned_addr learned my addr 172.21.15.125:0/1032262550 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 -- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc13cfc0 msgr2=0x7fdbfc137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc13cfc0 0x7fdbfc137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 -- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbfc13e540 msgr2=0x7fdbfc0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbfc13e540 0x7fdbfc0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.781+0000 7fdc01c00640 1 -- 172.21.15.125:0/1032262550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc13da60 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdc01c00640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02be80 tx=0x7fdbec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec043600 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec043760 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc13dd40 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec043b10 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.782+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc137f80 con 0x7fdbdc003970 2024-01-16T15:00:50.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.783+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc067450 con 0x7fdbdc003970 2024-01-16T15:00:50.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.784+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fdbec0383f0 con 0x7fdbdc003970 2024-01-16T15:00:50.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.785+0000 7fdbf9600640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 0x7fdbd406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.785+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdbec0ac350 con 0x7fdbdc003970 2024-01-16T15:00:50.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.785+0000 7fdc01200640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 0x7fdbd406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.786+0000 7fdc01200640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 0x7fdbd406db70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4006d40 tx=0x7fdbe4006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:50.789+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbec076b60 con 0x7fdbdc003970 2024-01-16T15:00:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:51.002+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone20153", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbfc13d8f0 con 0x7fdbd406b6b0 2024-01-16T15:00:53.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.859+0000 7fdbf9600640 1 -- 172.21.15.125:0/1032262550 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdbfc13d8f0 con 0x7fdbd406b6b0 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 msgr2=0x7fdbd406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 0x7fdbd406db70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4006d40 tx=0x7fdbe4006bd0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 msgr2=0x7fdbfc1f0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02be80 tx=0x7fdbec004730 comp rx=0 tx=0).stop 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 shutdown_connections 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbfc13e540 0x7fdbfc0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbfc13cfc0 0x7fdbfc137a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbd406b6b0 0x7fdbd406db70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 --2- 172.21.15.125:0/1032262550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbdc003970 0x7fdbfc1f0290 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 >> 172.21.15.125:0/1032262550 conn(0x7fdbfc06a800 msgr2=0x7fdbfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 shutdown_connections 2024-01-16T15:00:53.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:53.863+0000 7fdc03000640 1 -- 172.21.15.125:0/1032262550 wait complete. 2024-01-16T15:00:53.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone20154 2024-01-16T15:00:54.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.097+0000 7f085f200640 1 Processor -- start 2024-01-16T15:00:54.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.098+0000 7f085f200640 1 -- start start 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f086013cfa0 0x7f086015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08600cd9b0 0x7f086013fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08600cdec0 con 0x7f086013cfa0 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08600ce1a0 con 0x7f08600cd9b0 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08600ce480 con 0x7f08600cd530 2024-01-16T15:00:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f086013cfa0 0x7f086015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f086013cfa0 0x7f086015afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51638/0 (socket says 172.21.15.125:51638) 2024-01-16T15:00:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.099+0000 7f085de00640 1 -- 172.21.15.125:0/2209536384 learned_addr learned my addr 172.21.15.125:0/2209536384 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085e800640 1 --2- 172.21.15.125:0/2209536384 >> v2:172.21.15.137:3300/0 conn(0x7f08600cd9b0 0x7f086013fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085de00640 1 -- 172.21.15.125:0/2209536384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 msgr2=0x7f086013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f0857e00640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085de00640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085de00640 1 -- 172.21.15.125:0/2209536384 >> v2:172.21.15.137:3300/0 conn(0x7f08600cd9b0 msgr2=0x7f086013fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085de00640 1 --2- 172.21.15.125:0/2209536384 >> v2:172.21.15.137:3300/0 conn(0x7f08600cd9b0 0x7f086013fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.100+0000 7f085de00640 1 -- 172.21.15.125:0/2209536384 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600ce760 con 0x7f086013cfa0 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.101+0000 7f085de00640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f086015afb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f086006ce70 tx=0x7f085002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a194d31a5d0e0672 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.101+0000 7f085d400640 1 -- 172.21.15.125:0/2209536384 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500317e0 con 0x7f086013cfa0 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.101+0000 7f085d400640 1 -- 172.21.15.125:0/2209536384 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0850031940 con 0x7f086013cfa0 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.101+0000 7f085d400640 1 -- 172.21.15.125:0/2209536384 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850031cd0 con 0x7f086013cfa0 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 -- 172.21.15.125:0/2209536384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 msgr2=0x7f086015afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f086015afb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f086006ce70 tx=0x7f085002b880 comp rx=0 tx=0).stop 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 -- 172.21.15.125:0/2209536384 shutdown_connections 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 --2- 172.21.15.125:0/2209536384 >> v2:172.21.15.137:3300/0 conn(0x7f08600cd9b0 0x7f086013fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 --2- 172.21.15.125:0/2209536384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f086015afb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 -- 172.21.15.125:0/2209536384 >> 172.21.15.125:0/2209536384 conn(0x7f086006a800 msgr2=0x7f086006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 -- 172.21.15.125:0/2209536384 shutdown_connections 2024-01-16T15:00:54.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.102+0000 7f085f200640 1 -- 172.21.15.125:0/2209536384 wait complete. 2024-01-16T15:00:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.104+0000 7f085f200640 1 Processor -- start 2024-01-16T15:00:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.104+0000 7f085f200640 1 -- start start 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 0x7f08601508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086013bdd0 con 0x7f086013cfa0 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08601f68b0 con 0x7f08600cd9b0 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.105+0000 7f085f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08601f6b50 con 0x7f08600cd530 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51646/0 (socket says 172.21.15.125:51646) 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 -- 172.21.15.125:0/3914343034 learned_addr learned my addr 172.21.15.125:0/3914343034 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085de00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f0857e00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 0x7f08601508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 -- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 msgr2=0x7f086013b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 -- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 msgr2=0x7f08601508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 0x7f08601508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 -- 172.21.15.125:0/3914343034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500094d0 con 0x7f086013cfa0 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f0857e00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 0x7f08601508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085de00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:54.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.106+0000 7f085e800640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f085800cd70 tx=0x7f085800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.107+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858003b90 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.107+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0858003cf0 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.107+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858004240 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.107+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08601f6e30 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.107+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601f7630 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.108+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08601f7350 con 0x7f086013cfa0 2024-01-16T15:00:54.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.109+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f08580043a0 con 0x7f086013cfa0 2024-01-16T15:00:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.110+0000 7f0856000640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 0x7f083806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.110+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0858012070 con 0x7f086013cfa0 2024-01-16T15:00:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.110+0000 7f085de00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 0x7f083806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.111+0000 7f085de00640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 0x7f083806dba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0850036000 tx=0x7f085003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.115+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08601f7350 con 0x7f086013cfa0 2024-01-16T15:00:54.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:54.341+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone20154", "target": ["mon-mgr", ""]}) v1 -- 0x7f08601f7350 con 0x7f083806b6e0 2024-01-16T15:00:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.860+0000 7f0856000640 1 -- 172.21.15.125:0/3914343034 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f08601f7350 con 0x7f083806b6e0 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 msgr2=0x7f083806dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 0x7f083806dba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0850036000 tx=0x7f085003d040 comp rx=0 tx=0).stop 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 msgr2=0x7f0860152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f085800cd70 tx=0x7f085800a660 comp rx=0 tx=0).stop 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 shutdown_connections 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f083806b6e0 0x7f083806dba0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f086013cfa0 0x7f0860152220 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08600cd9b0 0x7f08601508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 --2- 172.21.15.125:0/3914343034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08600cd530 0x7f086013b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.864+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 >> 172.21.15.125:0/3914343034 conn(0x7f086006a800 msgr2=0x7f08601582e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.865+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 shutdown_connections 2024-01-16T15:00:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:58.865+0000 7f085f200640 1 -- 172.21.15.125:0/3914343034 wait complete. 2024-01-16T15:00:58.868 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:00:58.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20151 2024-01-16T15:00:59.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.090+0000 7f2084800640 1 Processor -- start 2024-01-16T15:00:59.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.091+0000 7f2084800640 1 -- start start 2024-01-16T15:00:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.092+0000 7f2084800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f207c099a60 0x7f207c099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.092+0000 7f2084800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.092+0000 7f2084800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c0cfa70 con 0x7f207c15e880 2024-01-16T15:00:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.092+0000 7f2084800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c09a3e0 con 0x7f207c099a60 2024-01-16T15:00:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.092+0000 7f2084800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0cff00 con 0x7f207c06e7c0 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.093+0000 7f2082800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.093+0000 7f2081400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f207c099a60 0x7f207c099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.093+0000 7f2082800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:39248/0 (socket says 172.21.15.125:39248) 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.093+0000 7f2082800640 1 -- 172.21.15.125:0/529175969 learned_addr learned my addr 172.21.15.125:0/529175969 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.094+0000 7f2082800640 1 -- 172.21.15.125:0/529175969 >> v2:172.21.15.137:3300/0 conn(0x7f207c099a60 msgr2=0x7f207c099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.094+0000 7f2082800640 1 --2- 172.21.15.125:0/529175969 >> v2:172.21.15.137:3300/0 conn(0x7f207c099a60 0x7f207c099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.094+0000 7f2082800640 1 -- 172.21.15.125:0/529175969 >> v1:172.21.15.137:6789/0 conn(0x7f207c15e880 legacy=0x7f207c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.094+0000 7f2082800640 1 -- 172.21.15.125:0/529175969 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c06fc70 con 0x7f207c06e7c0 2024-01-16T15:00:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.095+0000 7f2082800640 1 --2- 172.21.15.125:0/529175969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f207402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77dd0fdcc52dac5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2080a00640 1 -- 172.21.15.125:0/529175969 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207402cb90 con 0x7f207c06e7c0 2024-01-16T15:00:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2080a00640 1 -- 172.21.15.125:0/529175969 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207402ccf0 con 0x7f207c06e7c0 2024-01-16T15:00:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2080a00640 1 -- 172.21.15.125:0/529175969 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074034610 con 0x7f207c06e7c0 2024-01-16T15:00:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 -- 172.21.15.125:0/529175969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 msgr2=0x7f207c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 --2- 172.21.15.125:0/529175969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f207402bff0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 -- 172.21.15.125:0/529175969 shutdown_connections 2024-01-16T15:00:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 --2- 172.21.15.125:0/529175969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c06e7c0 0x7f207c06ec10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 --2- 172.21.15.125:0/529175969 >> v2:172.21.15.137:3300/0 conn(0x7f207c099a60 0x7f207c099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 -- 172.21.15.125:0/529175969 >> 172.21.15.125:0/529175969 conn(0x7f207c0b2d00 msgr2=0x7f207c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.096+0000 7f2084800640 1 -- 172.21.15.125:0/529175969 shutdown_connections 2024-01-16T15:00:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.097+0000 7f2084800640 1 -- 172.21.15.125:0/529175969 wait complete. 2024-01-16T15:00:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.098+0000 7f2084800640 1 Processor -- start 2024-01-16T15:00:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.098+0000 7f2084800640 1 -- start start 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f207c06e7c0 0x7f207c0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c099a60 0x7f207c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0c6980 con 0x7f207c15e880 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0c8820 con 0x7f207c06e7c0 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.099+0000 7f2084800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0cb260 con 0x7f207c099a60 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43494/0 (socket says 172.21.15.125:43494) 2024-01-16T15:00:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 -- 172.21.15.125:0/92134073 learned_addr learned my addr 172.21.15.125:0/92134073 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2081e00640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f207c06e7c0 0x7f207c0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2081400640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c099a60 0x7f207c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 -- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c099a60 msgr2=0x7f207c14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c099a60 0x7f207c14f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 -- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f207c06e7c0 msgr2=0x7f207c0c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f207c06e7c0 0x7f207c0c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.100+0000 7f2082800640 1 -- 172.21.15.125:0/92134073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2074009b30 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f2082800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f2074004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074009510 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207403e450 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207c0cb540 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207403e800 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.101+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207c0c7350 con 0x7f207c15e880 2024-01-16T15:00:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.102+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207c0c6b30 con 0x7f207c15e880 2024-01-16T15:00:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.103+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f207403e960 con 0x7f207c15e880 2024-01-16T15:00:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.104+0000 7f207a000640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 0x7f205406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.104+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f20740ace10 con 0x7f207c15e880 2024-01-16T15:00:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.104+0000 7f2081e00640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 0x7f205406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.105+0000 7f2081e00640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 0x7f205406dba0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f206c000f60 tx=0x7f206c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.109+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f207c0c6b30 con 0x7f207c15e880 2024-01-16T15:00:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:00:59.343+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20151", "target": ["mon-mgr", ""]}) v1 -- 0x7f207c0c6770 con 0x7f205406b6e0 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.707+0000 7f207a000640 1 -- 172.21.15.125:0/92134073 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f206c0012c0 con 0x7f205406b6e0 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:01:03.689 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:01:03.690 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:01:03.690 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:01:03.690 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:03.690 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:03.690 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:03.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 msgr2=0x7f205406dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 0x7f205406dba0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f206c000f60 tx=0x7f206c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:03.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 msgr2=0x7f207c0c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f2074004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 shutdown_connections 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f205406b6e0 0x7f205406dba0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f207c15e880 0x7f207c0c6e10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f207c099a60 0x7f207c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 --2- 172.21.15.125:0/92134073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f207c06e7c0 0x7f207c0c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.711+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 >> 172.21.15.125:0/92134073 conn(0x7f207c0b2d00 msgr2=0x7f207c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.712+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 shutdown_connections 2024-01-16T15:01:03.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:03.712+0000 7f2084800640 1 -- 172.21.15.125:0/92134073 wait complete. 2024-01-16T15:01:03.715 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:01:04.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20151 2024-01-16T15:01:04.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.958+0000 7f697a200640 1 Processor -- start 2024-01-16T15:01:04.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.958+0000 7f697a200640 1 -- start start 2024-01-16T15:01:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.960+0000 7f697a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 0x7f697415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.960+0000 7f697a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f697415e1e0 0x7f69740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.960+0000 7f697a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69740b1ee0 con 0x7f6974143ae0 2024-01-16T15:01:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.960+0000 7f697a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69740b2780 con 0x7f697415e1e0 2024-01-16T15:01:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.960+0000 7f697a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697415e990 con 0x7f69740cf9a0 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.961+0000 7f696a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 0x7f697415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.961+0000 7f6973e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f697415e1e0 0x7f69740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.961+0000 7f6973e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f697415e1e0 0x7f69740702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:57546/0 (socket says 172.21.15.125:57546) 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.961+0000 7f6973e00640 1 -- 172.21.15.125:0/2036109486 learned_addr learned my addr 172.21.15.125:0/2036109486 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.961+0000 7f696a800640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 0x7f697415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:39276/0 (socket says 172.21.15.125:39276) 2024-01-16T15:01:04.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.962+0000 7f6973e00640 1 -- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 msgr2=0x7f697415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.962+0000 7f6973e00640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 0x7f697415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.962+0000 7f6973e00640 1 -- 172.21.15.125:0/2036109486 >> v1:172.21.15.137:6789/0 conn(0x7f6974143ae0 legacy=0x7f69740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.962+0000 7f6973e00640 1 -- 172.21.15.125:0/2036109486 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69740b2130 con 0x7f697415e1e0 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.963+0000 7f6973e00640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f697415e1e0 0x7f69740702c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f696c009e50 tx=0x7f696c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73b5301d2a3d450 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f6972a00640 1 -- 172.21.15.125:0/2036109486 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c02cb00 con 0x7f697415e1e0 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f6972a00640 1 -- 172.21.15.125:0/2036109486 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c02cc60 con 0x7f697415e1e0 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f6972a00640 1 -- 172.21.15.125:0/2036109486 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0343f0 con 0x7f697415e1e0 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 -- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f697415e1e0 msgr2=0x7f69740702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f697415e1e0 0x7f69740702c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f696c009e50 tx=0x7f696c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:01:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 -- 172.21.15.125:0/2036109486 shutdown_connections 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f697415e1e0 0x7f69740702c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 --2- 172.21.15.125:0/2036109486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69740cf9a0 0x7f697415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 -- 172.21.15.125:0/2036109486 >> 172.21.15.125:0/2036109486 conn(0x7f697406a800 msgr2=0x7f697406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.964+0000 7f697a200640 1 -- 172.21.15.125:0/2036109486 shutdown_connections 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.965+0000 7f697a200640 1 -- 172.21.15.125:0/2036109486 wait complete. 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.966+0000 7f697a200640 1 Processor -- start 2024-01-16T15:01:04.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.966+0000 7f697a200640 1 -- start start 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69740cf9a0 0x7f69740cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6974143ae0 0x7f69740b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697415a240 con 0x7f697415e1e0 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69740c4180 con 0x7f69740cf9a0 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.967+0000 7f697a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6974131ec0 con 0x7f6974143ae0 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43504/0 (socket says 172.21.15.125:43504) 2024-01-16T15:01:04.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 -- 172.21.15.125:0/1845541795 learned_addr learned my addr 172.21.15.125:0/1845541795 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973400640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69740cf9a0 0x7f69740cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f696a800640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6974143ae0 0x7f69740b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 -- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6974143ae0 msgr2=0x7f69740b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6974143ae0 0x7f69740b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 -- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69740cf9a0 msgr2=0x7f69740cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69740cf9a0 0x7f69740cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 -- 172.21.15.125:0/1845541795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c009b30 con 0x7f697415e1e0 2024-01-16T15:01:04.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.968+0000 7f6973e00640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f696c036040 tx=0x7f696c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.969+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c040670 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.969+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c0407f0 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.969+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c040ba0 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.969+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69741321a0 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.969+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6974132760 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.970+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69740675d0 con 0x7f697415e1e0 2024-01-16T15:01:04.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.971+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f696c040d00 con 0x7f697415e1e0 2024-01-16T15:01:04.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.972+0000 7f696be00640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 0x7f694c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.972+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f696c0ad390 con 0x7f697415e1e0 2024-01-16T15:01:04.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.972+0000 7f6973400640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 0x7f694c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.973+0000 7f6973400640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 0x7f694c06db70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6964001620 tx=0x7f6964009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:04.976+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f696c077c30 con 0x7f697415e1e0 2024-01-16T15:01:05.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.208+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20151", "target": ["mon-mgr", ""]}) v1 -- 0x7f697415e640 con 0x7f694c06b6b0 2024-01-16T15:01:05.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.225+0000 7f696be00640 1 -- 172.21.15.125:0/1845541795 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f69640012c0 con 0x7f694c06b6b0 2024-01-16T15:01:05.206 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:05.206 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 0.0, 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "0/3", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.207 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 msgr2=0x7f694c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 0x7f694c06db70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6964001620 tx=0x7f6964009040 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 msgr2=0x7f69740b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f696c036040 tx=0x7f696c004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 shutdown_connections 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f694c06b6b0 0x7f694c06db70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f697415e1e0 0x7f69740b1100 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6974143ae0 0x7f69740b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 --2- 172.21.15.125:0/1845541795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69740cf9a0 0x7f69740cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.228+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 >> 172.21.15.125:0/1845541795 conn(0x7f697406a800 msgr2=0x7f697415b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.229+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 shutdown_connections 2024-01-16T15:01:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.229+0000 7f697a200640 1 -- 172.21.15.125:0/1845541795 wait complete. 2024-01-16T15:01:05.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.458+0000 7f61bd000640 1 Processor -- start 2024-01-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.458+0000 7f61bd000640 1 -- start start 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61bd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61bd000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61b813e360 0x7f61b80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61bd000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b806c470 con 0x7f61b815f9c0 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61bd000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b806c750 con 0x7f61b813e360 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61bd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b806ca30 con 0x7f61b813df80 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61b6800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f61b815f9c0 0x7f61b81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40592/0 (socket says 172.21.15.125:40592) 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61b6800640 1 -- 172.21.15.125:0/1325570463 learned_addr learned my addr 172.21.15.125:0/1325570463 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.460+0000 7f61afe00640 1 --2- 172.21.15.125:0/1325570463 >> v2:172.21.15.125:3300/0 conn(0x7f61b813e360 0x7f61b80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61b5e00640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61b5400640 1 -- 172.21.15.125:0/1325570463 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014932186 0 0) 0x7f61b806c470 con 0x7f61b815f9c0 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61b5400640 1 -- 172.21.15.125:0/1325570463 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198003520 con 0x7f61b815f9c0 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61afe00640 1 -- 172.21.15.125:0/1325570463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 msgr2=0x7f61b815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61afe00640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61afe00640 1 -- 172.21.15.125:0/1325570463 >> v1:172.21.15.125:6789/0 conn(0x7f61b815f9c0 legacy=0x7f61b81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.461+0000 7f61afe00640 1 -- 172.21.15.125:0/1325570463 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b806cd10 con 0x7f61b813e360 2024-01-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.462+0000 7f61afe00640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b80cf290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f61a00097a0 tx=0x7f61a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f7c60ae1f93838 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.462+0000 7f61b5400640 1 -- 172.21.15.125:0/1325570463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a002d040 con 0x7f61b813e360 2024-01-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.462+0000 7f61b5400640 1 -- 172.21.15.125:0/1325570463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a0031810 con 0x7f61b813e360 2024-01-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.462+0000 7f61b5400640 1 -- 172.21.15.125:0/1325570463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0031ba0 con 0x7f61b813e360 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 -- 172.21.15.125:0/1325570463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 msgr2=0x7f61b80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b80cf290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f61a00097a0 tx=0x7f61a002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 -- 172.21.15.125:0/1325570463 shutdown_connections 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b80cf290 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 --2- 172.21.15.125:0/1325570463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 -- 172.21.15.125:0/1325570463 >> 172.21.15.125:0/1325570463 conn(0x7f61b806a800 msgr2=0x7f61b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 -- 172.21.15.125:0/1325570463 shutdown_connections 2024-01-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.463+0000 7f61bd000640 1 -- 172.21.15.125:0/1325570463 wait complete. 2024-01-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.465+0000 7f61bd000640 1 Processor -- start 2024-01-16T15:01:05.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.465+0000 7f61bd000640 1 -- start start 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b814c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61b815f9c0 0x7f61b8136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b814e380 con 0x7f61b813e360 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b814ccc0 con 0x7f61b815f9c0 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61bd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b814cfa0 con 0x7f61b813df80 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43516/0 (socket says 172.21.15.125:43516) 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.466+0000 7f61afe00640 1 -- 172.21.15.125:0/592971202 learned_addr learned my addr 172.21.15.125:0/592971202 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61b5e00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b814c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 -- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 msgr2=0x7f61b814c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b814c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 -- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61b815f9c0 msgr2=0x7f61b8136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61b815f9c0 0x7f61b8136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 -- 172.21.15.125:0/592971202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a0009400 con 0x7f61b813e360 2024-01-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61afe00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f61a00098c0 tx=0x7f61a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0006e10 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a003ad60 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b814d280 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.467+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0044620 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.468+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b814d840 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.469+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b80675d0 con 0x7f61b813e360 2024-01-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.470+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f61a004d440 con 0x7f61b813e360 2024-01-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.471+0000 7f61ae000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 0x7f619006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.471+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61a00ac700 con 0x7f61b813e360 2024-01-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.471+0000 7f61b5e00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 0x7f619006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.472+0000 7f61b5e00640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 0x7f619006db70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.475+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61a0076ef0 con 0x7f61b813e360 2024-01-16T15:01:05.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.737+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f61b806df00 con 0x7f61b813e360 2024-01-16T15:01:05.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.738+0000 7f61ae000640 1 -- 172.21.15.125:0/592971202 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2093 (secure 0 0 0) 0x7f61a0035840 con 0x7f61b813e360 2024-01-16T15:01:05.719 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:05.720 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 284, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:05.721 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 801, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222204981, 2024-01-16T15:01:05.722 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6704279552, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766389833728, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 511, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 18084, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 4 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:05.724 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:01:05.725 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:01:05.726 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:05.726 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.726 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:05.726 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 msgr2=0x7f619006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 0x7f619006db70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8008040 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 msgr2=0x7f61b8151090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f61a00098c0 tx=0x7f61a00046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 shutdown_connections 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61b815f9c0 0x7f61b8136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619006b6b0 0x7f619006db70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b813e360 0x7f61b8151090 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 --2- 172.21.15.125:0/592971202 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61b813df80 0x7f61b814c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 >> 172.21.15.125:0/592971202 conn(0x7f61b806a800 msgr2=0x7f61b80b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 shutdown_connections 2024-01-16T15:01:05.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:05.741+0000 7f61bd000640 1 -- 172.21.15.125:0/592971202 wait complete. 2024-01-16T15:01:07.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:07.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:07.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:08.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.116+0000 7f11ba400640 1 Processor -- start 2024-01-16T15:01:08.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.117+0000 7f11ba400640 1 -- start start 2024-01-16T15:01:08.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.119+0000 7f11ba400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f11b415d460 0x7f11b40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11ba400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40b30b0 0x7f11b406d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11ba400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b415d840 con 0x7f11b406e380 2024-01-16T15:01:08.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11ba400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b4071490 con 0x7f11b40b30b0 2024-01-16T15:01:08.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11ba400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b40b3790 con 0x7f11b415d460 2024-01-16T15:01:08.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11b3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40b30b0 0x7f11b406d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11b3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40b30b0 0x7f11b406d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59254/0 (socket says 172.21.15.125:59254) 2024-01-16T15:01:08.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.120+0000 7f11b3e00640 1 -- 172.21.15.125:0/4196568205 learned_addr learned my addr 172.21.15.125:0/4196568205 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:08.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.121+0000 7f11b3e00640 1 -- 172.21.15.125:0/4196568205 >> v2:172.21.15.137:3300/0 conn(0x7f11b415d460 msgr2=0x7f11b40ac8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:01:08.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.121+0000 7f11b3e00640 1 --2- 172.21.15.125:0/4196568205 >> v2:172.21.15.137:3300/0 conn(0x7f11b415d460 0x7f11b40ac8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.121+0000 7f11b3e00640 1 -- 172.21.15.125:0/4196568205 >> v1:172.21.15.137:6789/0 conn(0x7f11b406e380 legacy=0x7f11b40b29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:01:08.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.121+0000 7f11b3e00640 1 -- 172.21.15.125:0/4196568205 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b4072190 con 0x7f11b40b30b0 2024-01-16T15:01:08.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.123+0000 7f11b3e00640 1 --2- 172.21.15.125:0/4196568205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b406d280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f11b40b1760 tx=0x7f11ac02bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5a1e28ef7e70d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.124+0000 7f11b2000640 1 -- 172.21.15.125:0/4196568205 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac02caa0 con 0x7f11b40b30b0 2024-01-16T15:01:08.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.124+0000 7f11b2000640 1 -- 172.21.15.125:0/4196568205 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac02cc00 con 0x7f11b40b30b0 2024-01-16T15:01:08.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.124+0000 7f11b2000640 1 -- 172.21.15.125:0/4196568205 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0343f0 con 0x7f11b40b30b0 2024-01-16T15:01:08.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 -- 172.21.15.125:0/4196568205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 msgr2=0x7f11b406d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 --2- 172.21.15.125:0/4196568205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b406d280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f11b40b1760 tx=0x7f11ac02bf00 comp rx=0 tx=0).stop 2024-01-16T15:01:08.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 -- 172.21.15.125:0/4196568205 shutdown_connections 2024-01-16T15:01:08.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 --2- 172.21.15.125:0/4196568205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b406d280 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 --2- 172.21.15.125:0/4196568205 >> v2:172.21.15.137:3300/0 conn(0x7f11b415d460 0x7f11b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.125+0000 7f11ba400640 1 -- 172.21.15.125:0/4196568205 >> 172.21.15.125:0/4196568205 conn(0x7f11b40b0be0 msgr2=0x7f11b40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:08.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.126+0000 7f11ba400640 1 -- 172.21.15.125:0/4196568205 shutdown_connections 2024-01-16T15:01:08.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.127+0000 7f11ba400640 1 -- 172.21.15.125:0/4196568205 wait complete. 2024-01-16T15:01:08.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.130+0000 7f11ba400640 1 Processor -- start 2024-01-16T15:01:08.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.130+0000 7f11ba400640 1 -- start start 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b4158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b415d460 0x7f11b415da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b415be40 con 0x7f11b40b30b0 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4158780 con 0x7f11b406e380 2024-01-16T15:01:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.133+0000 7f11ba400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b414cb70 con 0x7f11b415d460 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b415d460 0x7f11b415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:46140/0 (socket says 172.21.15.125:46140) 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 -- 172.21.15.125:0/1256693223 learned_addr learned my addr 172.21.15.125:0/1256693223 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b2a00640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b4158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 -- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b415d460 msgr2=0x7f11b415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b415d460 0x7f11b415da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 -- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 msgr2=0x7f11b4158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b4158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.134+0000 7f11b3400640 1 -- 172.21.15.125:0/1256693223 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac009b30 con 0x7f11b406e380 2024-01-16T15:01:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.135+0000 7f11b3400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11a400b450 tx=0x7f11a400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.136+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4011890 con 0x7f11b406e380 2024-01-16T15:01:08.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.136+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b40ce680 con 0x7f11b406e380 2024-01-16T15:01:08.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.136+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a40119f0 con 0x7f11b406e380 2024-01-16T15:01:08.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.136+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4011da0 con 0x7f11b406e380 2024-01-16T15:01:08.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.138+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b415c330 con 0x7f11b406e380 2024-01-16T15:01:08.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.139+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b415c4f0 con 0x7f11b406e380 2024-01-16T15:01:08.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.140+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f11a401a400 con 0x7f11b406e380 2024-01-16T15:01:08.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.141+0000 7f11ab400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 0x7f118c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.141+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f11a408a630 con 0x7f11b406e380 2024-01-16T15:01:08.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.147+0000 7f11b2a00640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 0x7f118c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.149+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11a4054e40 con 0x7f11b406e380 2024-01-16T15:01:08.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.199+0000 7f11b2a00640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 0x7f118c06db70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f119c006fd0 tx=0x7f119c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.674+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f11b40710c0 con 0x7f11b406e380 2024-01-16T15:01:08.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.676+0000 7f11ab400640 1 -- 172.21.15.125:0/1256693223 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2093 (secure 0 0 0) 0x7f11a4058460 con 0x7f11b406e380 2024-01-16T15:01:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:08.661 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 287, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 802, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222246191, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6704390144, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766389723136, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 597, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 10064, 2024-01-16T15:01:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 3 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:08.667 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:08.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.688+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 msgr2=0x7f118c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.689+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 0x7f118c06db70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f119c006fd0 tx=0x7f119c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:08.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.689+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 msgr2=0x7f11b40ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.689+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11a400b450 tx=0x7f11a400b910 comp rx=0 tx=0).stop 2024-01-16T15:01:08.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 shutdown_connections 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b415d460 0x7f11b415da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06b6b0 0x7f118c06db70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40b30b0 0x7f11b4158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 --2- 172.21.15.125:0/1256693223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b406e380 0x7f11b40ce140 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.692+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 >> 172.21.15.125:0/1256693223 conn(0x7f11b40b0be0 msgr2=0x7f11b40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.693+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 shutdown_connections 2024-01-16T15:01:08.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:08.694+0000 7f11ba400640 1 -- 172.21.15.125:0/1256693223 wait complete. 2024-01-16T15:01:08.769 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:08.787+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:08.769 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:08.787+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:08.770 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:08.788+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:08.770 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:08.788+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:08.770 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:08.788+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:10.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:10.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.977+0000 7fa429600640 1 Processor -- start 2024-01-16T15:01:10.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.978+0000 7fa429600640 1 -- start start 2024-01-16T15:01:10.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240ae5a0 0x7fa42415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa4240b3ae0 0x7fa42406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa42406d120 0x7fa4240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa424071450 con 0x7fa42406d120 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa424070c40 con 0x7fa4240b3ae0 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.979+0000 7fa429600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42406c470 con 0x7fa4240ae5a0 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa42406d120 0x7fa4240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa42406d120 0x7fa4240cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59294/0 (socket says 172.21.15.125:59294) 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 -- 172.21.15.125:0/3900043928 learned_addr learned my addr 172.21.15.125:0/3900043928 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa422a00640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240ae5a0 0x7fa42415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 -- 172.21.15.125:0/3900043928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240ae5a0 msgr2=0x7fa42415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240ae5a0 0x7fa42415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 -- 172.21.15.125:0/3900043928 >> v2:172.21.15.137:3300/0 conn(0x7fa4240b3ae0 msgr2=0x7fa42406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 --2- 172.21.15.125:0/3900043928 >> v2:172.21.15.137:3300/0 conn(0x7fa4240b3ae0 0x7fa42406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.980+0000 7fa423400640 1 -- 172.21.15.125:0/3900043928 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42406c750 con 0x7fa42406d120 2024-01-16T15:01:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.981+0000 7fa423400640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4240cf290 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa4240cfed0 tx=0x7fa41c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6653f68f0b72238b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.981+0000 7fa422000640 1 -- 172.21.15.125:0/3900043928 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c02ca70 con 0x7fa42406d120 2024-01-16T15:01:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.981+0000 7fa422000640 1 -- 172.21.15.125:0/3900043928 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c02cbd0 con 0x7fa42406d120 2024-01-16T15:01:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.981+0000 7fa422000640 1 -- 172.21.15.125:0/3900043928 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c0343f0 con 0x7fa42406d120 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 -- 172.21.15.125:0/3900043928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 msgr2=0x7fa4240cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4240cf290 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa4240cfed0 tx=0x7fa41c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 -- 172.21.15.125:0/3900043928 shutdown_connections 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4240cf290 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 --2- 172.21.15.125:0/3900043928 >> v2:172.21.15.137:3300/0 conn(0x7fa4240b3ae0 0x7fa42406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 --2- 172.21.15.125:0/3900043928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240ae5a0 0x7fa42415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 -- 172.21.15.125:0/3900043928 >> 172.21.15.125:0/3900043928 conn(0x7fa42406a800 msgr2=0x7fa42406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 -- 172.21.15.125:0/3900043928 shutdown_connections 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.982+0000 7fa429600640 1 -- 172.21.15.125:0/3900043928 wait complete. 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.984+0000 7fa429600640 1 Processor -- start 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.984+0000 7fa429600640 1 -- start start 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4240ae5a0 0x7fa4240c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240b3ae0 0x7fa4241f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240c4c90 con 0x7fa42406d120 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4241f70d0 con 0x7fa4240ae5a0 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa429600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240c48d0 con 0x7fa4240b3ae0 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa422a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa422a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59298/0 (socket says 172.21.15.125:59298) 2024-01-16T15:01:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.985+0000 7fa422a00640 1 -- 172.21.15.125:0/3225489774 learned_addr learned my addr 172.21.15.125:0/3225489774 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa423400640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240b3ae0 0x7fa4241f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa419e00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4240ae5a0 0x7fa4240c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 -- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240b3ae0 msgr2=0x7fa4241f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240b3ae0 0x7fa4241f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 -- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4240ae5a0 msgr2=0x7fa4240c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4240ae5a0 0x7fa4240c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 -- 172.21.15.125:0/3225489774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c009b30 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa422a00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c7f0 tx=0x7fa41400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414004220 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4140043a0 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414007860 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4240cc590 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.986+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4241611d0 con 0x7fa42406d120 2024-01-16T15:01:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.988+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa424072160 con 0x7fa42406d120 2024-01-16T15:01:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.988+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7fa414007bd0 con 0x7fa42406d120 2024-01-16T15:01:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.989+0000 7fa41b400640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 0x7fa3fc06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.989+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa41408a580 con 0x7fa42406d120 2024-01-16T15:01:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.989+0000 7fa419e00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 0x7fa3fc06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.990+0000 7fa419e00640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 0x7fa3fc06dc00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa424160a70 tx=0x7fa40c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:10.994+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa414054ba0 con 0x7fa42406d120 2024-01-16T15:01:11.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.254+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa4240675d0 con 0x7fa42406d120 2024-01-16T15:01:11.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.255+0000 7fa41b400640 1 -- 172.21.15.125:0/3225489774 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2143 (secure 0 0 0) 0x7fa41401e870 con 0x7fa42406d120 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:11.236 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 290, 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:11.237 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:11.238 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1278, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5223001734, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 10118713344, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 762975399936, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 175736980, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 166669329, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 42, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 45 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:11.239 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:11.240 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:11.241 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 19s)", 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22220000624656677, 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:11.242 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 msgr2=0x7fa3fc06dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 0x7fa3fc06dc00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa424160a70 tx=0x7fa40c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 msgr2=0x7fa4241f26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c7f0 tx=0x7fa41400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 shutdown_connections 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4240b3ae0 0x7fa4241f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4240ae5a0 0x7fa4240c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3fc06b740 0x7fa3fc06dc00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 --2- 172.21.15.125:0/3225489774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa42406d120 0x7fa4241f26f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.258+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 >> 172.21.15.125:0/3225489774 conn(0x7fa42406a800 msgr2=0x7fa42406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:11.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.259+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 shutdown_connections 2024-01-16T15:01:11.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:11.259+0000 7fa429600640 1 -- 172.21.15.125:0/3225489774 wait complete. 2024-01-16T15:01:13.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:13.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.759+0000 7fade4c00640 1 Processor -- start 2024-01-16T15:01:13.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.759+0000 7fade4c00640 1 -- start start 2024-01-16T15:01:13.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.761+0000 7fade4c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 0x7fade00d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.761+0000 7fade4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fade00cf9a0 0x7fade00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.764+0000 7fadd7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fade00cf9a0 0x7fade00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.764+0000 7fadd7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fade00cf9a0 0x7fade00cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59308/0 (socket says 172.21.15.125:59308) 2024-01-16T15:01:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.765+0000 7faddfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fade00cddf0 0x7fade00ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40390/0 (socket says 172.21.15.125:40390) 2024-01-16T15:01:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.765+0000 7faddf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 0x7fade00d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.765+0000 7faddf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 0x7fade00d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:46174/0 (socket says 172.21.15.125:46174) 2024-01-16T15:01:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.766+0000 7fadd7e00640 1 -- 172.21.15.125:0/1548778396 learned_addr learned my addr 172.21.15.125:0/1548778396 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:13.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.766+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00d0320 con 0x7fade00cddf0 2024-01-16T15:01:13.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.767+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00b2730 con 0x7fade00cf9a0 2024-01-16T15:01:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.767+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00d1ec0 con 0x7fade015d4b0 2024-01-16T15:01:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.767+0000 7faddea00640 1 -- 172.21.15.125:0/1548778396 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789098758 0 0) 0x7fade00d0320 con 0x7fade00cddf0 2024-01-16T15:01:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.768+0000 7faddea00640 1 -- 172.21.15.125:0/1548778396 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003520 con 0x7fade00cddf0 2024-01-16T15:01:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.768+0000 7fadd7e00640 1 -- 172.21.15.125:0/1548778396 >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 msgr2=0x7fade00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.768+0000 7fadd7e00640 1 --2- 172.21.15.125:0/1548778396 >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 0x7fade00d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.768+0000 7fadd7e00640 1 -- 172.21.15.125:0/1548778396 >> v1:172.21.15.125:6789/0 conn(0x7fade00cddf0 legacy=0x7fade00ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.768+0000 7fadd7e00640 1 -- 172.21.15.125:0/1548778396 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00b29d0 con 0x7fade00cf9a0 2024-01-16T15:01:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.771+0000 7fadd7e00640 1 --2- 172.21.15.125:0/1548778396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade00cf9a0 0x7fade00cfde0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fade008b5b0 tx=0x7fadc802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49a1b6150923940 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.772+0000 7faddea00640 1 -- 172.21.15.125:0/1548778396 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8030870 con 0x7fade00cf9a0 2024-01-16T15:01:13.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.772+0000 7faddea00640 1 -- 172.21.15.125:0/1548778396 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc80309d0 con 0x7fade00cf9a0 2024-01-16T15:01:13.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.772+0000 7faddea00640 1 -- 172.21.15.125:0/1548778396 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8030d60 con 0x7fade00cf9a0 2024-01-16T15:01:13.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.773+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade00cf9a0 msgr2=0x7fade00cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.773+0000 7fade4c00640 1 --2- 172.21.15.125:0/1548778396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade00cf9a0 0x7fade00cfde0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fade008b5b0 tx=0x7fadc802b9b0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.775+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 shutdown_connections 2024-01-16T15:01:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.775+0000 7fade4c00640 1 --2- 172.21.15.125:0/1548778396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade00cf9a0 0x7fade00cfde0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.776+0000 7fade4c00640 1 --2- 172.21.15.125:0/1548778396 >> v2:172.21.15.137:3300/0 conn(0x7fade015d4b0 0x7fade00d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.776+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 >> 172.21.15.125:0/1548778396 conn(0x7fade00d3000 msgr2=0x7fade00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.776+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 shutdown_connections 2024-01-16T15:01:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.781+0000 7fade4c00640 1 -- 172.21.15.125:0/1548778396 wait complete. 2024-01-16T15:01:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.801+0000 7fade4c00640 1 Processor -- start 2024-01-16T15:01:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.802+0000 7fade4c00640 1 -- start start 2024-01-16T15:01:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fade00cf9a0 0x7fade0156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade015d4b0 0x7fade00ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade015b400 con 0x7fade015d4b0 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0157640 con 0x7fade00cf9a0 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.803+0000 7fade4c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00a02f0 con 0x7fade00cddf0 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.804+0000 7faddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.804+0000 7faddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:45810/0 (socket says 172.21.15.125:45810) 2024-01-16T15:01:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.804+0000 7faddf400640 1 -- 172.21.15.125:0/2877224908 learned_addr learned my addr 172.21.15.125:0/2877224908 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:13.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.807+0000 7faddf400640 1 -- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fade00cf9a0 msgr2=0x7fade0156730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:01:13.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.808+0000 7faddf400640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fade00cf9a0 0x7fade0156730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.808+0000 7faddf400640 1 -- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade015d4b0 msgr2=0x7fade00ce1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:01:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.808+0000 7faddf400640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade015d4b0 0x7fade00ce1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.808+0000 7faddf400640 1 -- 172.21.15.125:0/2877224908 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc80094d0 con 0x7fade00cddf0 2024-01-16T15:01:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.811+0000 7faddf400640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadd000e870 tx=0x7fadd000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.811+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00094f0 con 0x7fade00cddf0 2024-01-16T15:01:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.812+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd000c940 con 0x7fade00cddf0 2024-01-16T15:01:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.812+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd000ccf0 con 0x7fade00cddf0 2024-01-16T15:01:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.812+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade015da10 con 0x7fade00cddf0 2024-01-16T15:01:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.813+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade0157080 con 0x7fade00cddf0 2024-01-16T15:01:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.816+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0067450 con 0x7fade00cddf0 2024-01-16T15:01:13.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.825+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7fadd0005990 con 0x7fade00cddf0 2024-01-16T15:01:13.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.826+0000 7fadd7400640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 0x7fadb806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.835+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fadd008ab80 con 0x7fade00cddf0 2024-01-16T15:01:13.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.838+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadd00b6f30 con 0x7fade00cddf0 2024-01-16T15:01:13.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.843+0000 7fadd7e00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 0x7fadb806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:13.877+0000 7fadd7e00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 0x7fadb806dc50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fadc8036000 tx=0x7fadc803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.879 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:13.898+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:13.887 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:13.898+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:13.887 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:13.898+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:13.887 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:13.898+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:13.887 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:13.898+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:14.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.475+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fade00b4240 con 0x7fade00cddf0 2024-01-16T15:01:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.477+0000 7fadd7400640 1 -- 172.21.15.125:0/2877224908 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2143 (secure 0 0 0) 0x7fadd0021090 con 0x7fade00cddf0 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:14.458 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 293, 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:14.459 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:14.460 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1278, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5223001734, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 10118713344, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 762975399936, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 175739427, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 166666012, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 42, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 44 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:14.461 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.462 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:14.463 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 19s)", 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22220000624656677, 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:14.464 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:14.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.485+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 msgr2=0x7fadb806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.485+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 0x7fadb806dc50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fadc8036000 tx=0x7fadc803b040 comp rx=0 tx=0).stop 2024-01-16T15:01:14.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 msgr2=0x7fade0153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadd000e870 tx=0x7fadd000ed30 comp rx=0 tx=0).stop 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 shutdown_connections 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadb806b790 0x7fadb806dc50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fade015d4b0 0x7fade00ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fade00cf9a0 0x7fade0156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 --2- 172.21.15.125:0/2877224908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fade00cddf0 0x7fade0153e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 >> 172.21.15.125:0/2877224908 conn(0x7fade00d3000 msgr2=0x7fade00cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.486+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 shutdown_connections 2024-01-16T15:01:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:14.487+0000 7fade4c00640 1 -- 172.21.15.125:0/2877224908 wait complete. 2024-01-16T15:01:16.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:16.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.742+0000 7fc87aa00640 1 Processor -- start 2024-01-16T15:01:16.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.742+0000 7fc87aa00640 1 -- start start 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc87aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc8740cd600 0x7fc874070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc87aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 0x7fc87406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc87aa00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc874071450 con 0x7fc874157c30 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc87aa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87415ea30 con 0x7fc8740cd600 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc87aa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87406d860 con 0x7fc874071030 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc873e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc874157c30 0x7fc8741618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48350/0 (socket says 172.21.15.125:48350) 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc873400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc8740cd600 0x7fc874070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.744+0000 7fc873e00640 1 -- 172.21.15.125:0/1808342543 learned_addr learned my addr 172.21.15.125:0/1808342543 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc878a00640 1 --2- 172.21.15.125:0/1808342543 >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 0x7fc87406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc873400640 1 -- 172.21.15.125:0/1808342543 >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 msgr2=0x7fc87406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc873400640 1 --2- 172.21.15.125:0/1808342543 >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 0x7fc87406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc873400640 1 -- 172.21.15.125:0/1808342543 >> v1:172.21.15.125:6789/0 conn(0x7fc874157c30 legacy=0x7fc8741618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc873400640 1 -- 172.21.15.125:0/1808342543 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87415deb0 con 0x7fc8740cd600 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc872a00640 1 -- 172.21.15.125:0/1808342543 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044661060 0 0) 0x7fc874071450 con 0x7fc874157c30 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.745+0000 7fc878a00640 1 --2- 172.21.15.125:0/1808342543 >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 0x7fc87406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.746+0000 7fc873400640 1 --2- 172.21.15.125:0/1808342543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874070af0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc8740cbb60 tx=0x7fc85c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5450112e57f12006 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.746+0000 7fc872a00640 1 -- 172.21.15.125:0/1808342543 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c02cc70 con 0x7fc8740cd600 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.746+0000 7fc872a00640 1 -- 172.21.15.125:0/1808342543 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c02cdd0 con 0x7fc8740cd600 2024-01-16T15:01:16.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.746+0000 7fc872a00640 1 -- 172.21.15.125:0/1808342543 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c034660 con 0x7fc8740cd600 2024-01-16T15:01:16.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 -- 172.21.15.125:0/1808342543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 msgr2=0x7fc874070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 --2- 172.21.15.125:0/1808342543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874070af0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc8740cbb60 tx=0x7fc85c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 -- 172.21.15.125:0/1808342543 shutdown_connections 2024-01-16T15:01:16.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 --2- 172.21.15.125:0/1808342543 >> v2:172.21.15.137:3300/0 conn(0x7fc874071030 0x7fc87406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 --2- 172.21.15.125:0/1808342543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874070af0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 -- 172.21.15.125:0/1808342543 >> 172.21.15.125:0/1808342543 conn(0x7fc8740cf700 msgr2=0x7fc8740cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 -- 172.21.15.125:0/1808342543 shutdown_connections 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.747+0000 7fc87aa00640 1 -- 172.21.15.125:0/1808342543 wait complete. 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.749+0000 7fc87aa00640 1 Processor -- start 2024-01-16T15:01:16.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.749+0000 7fc87aa00640 1 -- start start 2024-01-16T15:01:16.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.749+0000 7fc87aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc874071030 0x7fc8740ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc87aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc87aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc874157c30 0x7fc8740af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc87aa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740adf70 con 0x7fc8740cd600 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc87aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741466d0 con 0x7fc874157c30 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc87aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740afb70 con 0x7fc874071030 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53814/0 (socket says 172.21.15.125:53814) 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 -- 172.21.15.125:0/3287458563 learned_addr learned my addr 172.21.15.125:0/3287458563 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 -- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc874071030 msgr2=0x7fc8740ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc874071030 0x7fc8740ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 -- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc874157c30 msgr2=0x7fc8740af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc874157c30 0x7fc8740af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.750+0000 7fc873400640 1 -- 172.21.15.125:0/3287458563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c009a70 con 0x7fc8740cd600 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc873400640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e80 tx=0x7fc85c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c003b90 con 0x7fc8740cd600 2024-01-16T15:01:16.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c003cf0 con 0x7fc8740cd600 2024-01-16T15:01:16.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c004260 con 0x7fc8740cd600 2024-01-16T15:01:16.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8740afe50 con 0x7fc8740cd600 2024-01-16T15:01:16.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.751+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8740b0410 con 0x7fc8740cd600 2024-01-16T15:01:16.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.752+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8740675d0 con 0x7fc8740cd600 2024-01-16T15:01:16.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.753+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7fc85c03b050 con 0x7fc8740cd600 2024-01-16T15:01:16.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.754+0000 7fc86be00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 0x7fc84c06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.754+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc85c0acba0 con 0x7fc8740cd600 2024-01-16T15:01:16.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.754+0000 7fc873e00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 0x7fc84c06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.754+0000 7fc873e00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 0x7fc84c06dc00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc864007fd0 tx=0x7fc864007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:16.759+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc85c0772d0 con 0x7fc8740cd600 2024-01-16T15:01:16.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.008+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc874160dc0 con 0x7fc8740cd600 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.009+0000 7fc86be00640 1 -- 172.21.15.125:0/3287458563 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7fc85c039090 con 0x7fc8740cd600 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:16.990 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 296, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:16.991 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:16.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1750, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 7207019798, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 14212591616, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 758881521664, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 344144376, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 331936656, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 82, 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 87 2024-01-16T15:01:16.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:16.994 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:16.995 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.335% (11s)\n [=========...................] (remaining: 23s)", 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.33335000276565552, 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:16.996 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 msgr2=0x7fc84c06dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 0x7fc84c06dc00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc864007fd0 tx=0x7fc864007920 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 msgr2=0x7fc874146190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e80 tx=0x7fc85c004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 shutdown_connections 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc874157c30 0x7fc8740af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc84c06b740 0x7fc84c06dc00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8740cd600 0x7fc874146190 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 --2- 172.21.15.125:0/3287458563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc874071030 0x7fc8740ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 >> 172.21.15.125:0/3287458563 conn(0x7fc8740cf700 msgr2=0x7fc8740cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 shutdown_connections 2024-01-16T15:01:16.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:17.012+0000 7fc87aa00640 1 -- 172.21.15.125:0/3287458563 wait complete. 2024-01-16T15:01:19.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:19.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.433+0000 7fd148e00640 1 Processor -- start 2024-01-16T15:01:19.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.434+0000 7fd148e00640 1 -- start start 2024-01-16T15:01:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.437+0000 7fd148e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 0x7fd1440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.437+0000 7fd148e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1440cfbf0 con 0x7fd144071070 2024-01-16T15:01:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.437+0000 7fd148e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd144071690 con 0x7fd1440cf7d0 2024-01-16T15:01:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.437+0000 7fd148e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14414b6a0 con 0x7fd14414b2c0 2024-01-16T15:01:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.438+0000 7fd142800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd144071070 0x7fd1440cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48360/0 (socket says 172.21.15.125:48360) 2024-01-16T15:01:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.438+0000 7fd142800640 1 -- 172.21.15.125:0/4017775960 learned_addr learned my addr 172.21.15.125:0/4017775960 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.438+0000 7fd141e00640 1 --2- 172.21.15.125:0/4017775960 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 0x7fd1440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.440+0000 7fd140a00640 1 -- 172.21.15.125:0/4017775960 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928195456 0 0) 0x7fd144071690 con 0x7fd1440cf7d0 2024-01-16T15:01:19.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.440+0000 7fd140a00640 1 -- 172.21.15.125:0/4017775960 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124003520 con 0x7fd1440cf7d0 2024-01-16T15:01:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.442+0000 7fd141e00640 1 -- 172.21.15.125:0/4017775960 >> v1:172.21.15.137:6789/0 conn(0x7fd1440cf7d0 legacy=0x7fd144070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.442+0000 7fd141e00640 1 -- 172.21.15.125:0/4017775960 >> v1:172.21.15.125:6789/0 conn(0x7fd144071070 legacy=0x7fd1440cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.442+0000 7fd141e00640 1 -- 172.21.15.125:0/4017775960 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144071970 con 0x7fd14414b2c0 2024-01-16T15:01:19.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.449+0000 7fd141e00640 1 --2- 172.21.15.125:0/4017775960 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 0x7fd1440cf290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009dc0 tx=0x7fd13c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5dbb8a42102a6d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.450+0000 7fd140a00640 1 -- 172.21.15.125:0/4017775960 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c02ccc0 con 0x7fd14414b2c0 2024-01-16T15:01:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.450+0000 7fd140a00640 1 -- 172.21.15.125:0/4017775960 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02ce20 con 0x7fd14414b2c0 2024-01-16T15:01:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.450+0000 7fd140a00640 1 -- 172.21.15.125:0/4017775960 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c034660 con 0x7fd14414b2c0 2024-01-16T15:01:19.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.452+0000 7fd148e00640 1 -- 172.21.15.125:0/4017775960 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 msgr2=0x7fd1440cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.453+0000 7fd148e00640 1 --2- 172.21.15.125:0/4017775960 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 0x7fd1440cf290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009dc0 tx=0x7fd13c02bf10 comp rx=0 tx=0).stop 2024-01-16T15:01:19.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.456+0000 7fd148e00640 1 -- 172.21.15.125:0/4017775960 shutdown_connections 2024-01-16T15:01:19.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.457+0000 7fd148e00640 1 --2- 172.21.15.125:0/4017775960 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd14414b2c0 0x7fd1440cf290 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.457+0000 7fd148e00640 1 -- 172.21.15.125:0/4017775960 >> 172.21.15.125:0/4017775960 conn(0x7fd14406a800 msgr2=0x7fd14406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.459+0000 7fd148e00640 1 -- 172.21.15.125:0/4017775960 shutdown_connections 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.469+0000 7fd148e00640 1 -- 172.21.15.125:0/4017775960 wait complete. 2024-01-16T15:01:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.472+0000 7fd148e00640 1 Processor -- start 2024-01-16T15:01:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.472+0000 7fd148e00640 1 -- start start 2024-01-16T15:01:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd144071070 0x7fd144160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd14414b2c0 0x7fd144145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144161260 con 0x7fd1440cf7d0 2024-01-16T15:01:19.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144163510 con 0x7fd14414b2c0 2024-01-16T15:01:19.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.477+0000 7fd148e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd144144840 con 0x7fd144071070 2024-01-16T15:01:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.483+0000 7fd141400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.483+0000 7fd142800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd14414b2c0 0x7fd144145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.483+0000 7fd141400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53818/0 (socket says 172.21.15.125:53818) 2024-01-16T15:01:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.483+0000 7fd141400640 1 -- 172.21.15.125:0/1590936738 learned_addr learned my addr 172.21.15.125:0/1590936738 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.483+0000 7fd141e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd144071070 0x7fd144160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.484+0000 7fd141400640 1 -- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd144071070 msgr2=0x7fd144160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.484+0000 7fd141400640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd144071070 0x7fd144160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.484+0000 7fd141400640 1 -- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd14414b2c0 msgr2=0x7fd144145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.484+0000 7fd141400640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd14414b2c0 0x7fd144145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.484+0000 7fd141400640 1 -- 172.21.15.125:0/1590936738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c009a70 con 0x7fd1440cf7d0 2024-01-16T15:01:19.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.486+0000 7fd141400640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd13400c490 tx=0x7fd13400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.487+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1340052d0 con 0x7fd1440cf7d0 2024-01-16T15:01:19.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.487+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd134005430 con 0x7fd1440cf7d0 2024-01-16T15:01:19.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.487+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134012930 con 0x7fd1440cf7d0 2024-01-16T15:01:19.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.487+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd144144b20 con 0x7fd1440cf7d0 2024-01-16T15:01:19.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.489+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1441631a0 con 0x7fd1440cf7d0 2024-01-16T15:01:19.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.500+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1441636c0 con 0x7fd1440cf7d0 2024-01-16T15:01:19.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.505+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd134012a90 con 0x7fd1440cf7d0 2024-01-16T15:01:19.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.506+0000 7fd13a000640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 0x7fd11c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.507+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1340182b0 con 0x7fd1440cf7d0 2024-01-16T15:01:19.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.515+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1441636c0 con 0x7fd1440cf7d0 2024-01-16T15:01:19.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.533+0000 7fd141e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 0x7fd11c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:19.535+0000 7fd141e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 0x7fd11c06dce0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c036000 tx=0x7fd13c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.158+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd144163680 con 0x7fd1440cf7d0 2024-01-16T15:01:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.163+0000 7fd13a000640 1 -- 172.21.15.125:0/1590936738 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7fd13408e060 con 0x7fd1440cf7d0 2024-01-16T15:01:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 299, 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1750, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 7207019798, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 14212591616, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 758881521664, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 344144921, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 331933328, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 82, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 85 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 17s)", 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.41667500138282776, 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:20.166 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:20.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.194+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 msgr2=0x7fd11c06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.194+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 0x7fd11c06dce0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c036000 tx=0x7fd13c030000 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.194+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 msgr2=0x7fd144143e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.194+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd13400c490 tx=0x7fd13400c950 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 shutdown_connections 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd14414b2c0 0x7fd144145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd11c06b820 0x7fd11c06dce0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1440cf7d0 0x7fd144143e60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 --2- 172.21.15.125:0/1590936738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd144071070 0x7fd144160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 >> 172.21.15.125:0/1590936738 conn(0x7fd14406a800 msgr2=0x7fd144148720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.195+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 shutdown_connections 2024-01-16T15:01:20.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:20.196+0000 7fd148e00640 1 -- 172.21.15.125:0/1590936738 wait complete. 2024-01-16T15:01:22.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.688+0000 7f3798e00640 1 Processor -- start 2024-01-16T15:01:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.688+0000 7f3798e00640 1 -- start start 2024-01-16T15:01:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f378c03c0a0 0x7f378c058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f378c053cb0 0x7f378c054120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f378c058590 con 0x7f378c03c0a0 2024-01-16T15:01:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f378c054660 con 0x7f378c053cb0 2024-01-16T15:01:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.691+0000 7f3798e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c058810 con 0x7f378c0e3160 2024-01-16T15:01:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.693+0000 7f3791400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f378c03c0a0 0x7f378c058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.693+0000 7f3791e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.693+0000 7f3791400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f378c03c0a0 0x7f378c058050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53828/0 (socket says 172.21.15.125:53828) 2024-01-16T15:01:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.693+0000 7f3791400640 1 -- 172.21.15.125:0/3462568403 learned_addr learned my addr 172.21.15.125:0/3462568403 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.694+0000 7f3791400640 1 -- 172.21.15.125:0/3462568403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 msgr2=0x7f378c0e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.694+0000 7f3791400640 1 --2- 172.21.15.125:0/3462568403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.694+0000 7f3791400640 1 -- 172.21.15.125:0/3462568403 >> v2:172.21.15.137:3300/0 conn(0x7f378c053cb0 msgr2=0x7f378c054120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.694+0000 7f3791400640 1 --2- 172.21.15.125:0/3462568403 >> v2:172.21.15.137:3300/0 conn(0x7f378c053cb0 0x7f378c054120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.694+0000 7f3791400640 1 -- 172.21.15.125:0/3462568403 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c058af0 con 0x7f378c03c0a0 2024-01-16T15:01:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.696+0000 7f3791400640 1 --2- 172.21.15.125:0/3462568403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c03c0a0 0x7f378c058050 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f378c054cd0 tx=0x7f377c02bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93d30d61bb150d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.701+0000 7f3790a00640 1 -- 172.21.15.125:0/3462568403 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c02c750 con 0x7f378c03c0a0 2024-01-16T15:01:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.701+0000 7f3790a00640 1 -- 172.21.15.125:0/3462568403 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c02c8b0 con 0x7f378c03c0a0 2024-01-16T15:01:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.701+0000 7f3790a00640 1 -- 172.21.15.125:0/3462568403 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c02cc80 con 0x7f378c03c0a0 2024-01-16T15:01:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.702+0000 7f3798e00640 1 -- 172.21.15.125:0/3462568403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c03c0a0 msgr2=0x7f378c058050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.702+0000 7f3798e00640 1 --2- 172.21.15.125:0/3462568403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c03c0a0 0x7f378c058050 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f378c054cd0 tx=0x7f377c02bbb0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.704+0000 7f3798e00640 1 -- 172.21.15.125:0/3462568403 shutdown_connections 2024-01-16T15:01:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.704+0000 7f3798e00640 1 --2- 172.21.15.125:0/3462568403 >> v2:172.21.15.137:3300/0 conn(0x7f378c053cb0 0x7f378c054120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.704+0000 7f3798e00640 1 --2- 172.21.15.125:0/3462568403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c03c0a0 0x7f378c058050 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.704+0000 7f3798e00640 1 --2- 172.21.15.125:0/3462568403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.704+0000 7f3798e00640 1 -- 172.21.15.125:0/3462568403 >> 172.21.15.125:0/3462568403 conn(0x7f378c058ee0 msgr2=0x7f378c0592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.709+0000 7f3798e00640 1 -- 172.21.15.125:0/3462568403 shutdown_connections 2024-01-16T15:01:22.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.709+0000 7f3798e00640 1 -- 172.21.15.125:0/3462568403 wait complete. 2024-01-16T15:01:22.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.717+0000 7f3798e00640 1 Processor -- start 2024-01-16T15:01:22.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.719+0000 7f3798e00640 1 -- start start 2024-01-16T15:01:22.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.721+0000 7f3798e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f378c03c0a0 0x7f378c03da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3798e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3798e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3798e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c181810 con 0x7f378c053cb0 2024-01-16T15:01:22.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3798e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c052970 con 0x7f378c03c0a0 2024-01-16T15:01:22.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3798e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c0e37f0 con 0x7f378c0e3160 2024-01-16T15:01:22.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.722+0000 7f3791400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.723+0000 7f3791400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53832/0 (socket says 172.21.15.125:53832) 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.723+0000 7f3791400640 1 -- 172.21.15.125:0/3342637797 learned_addr learned my addr 172.21.15.125:0/3342637797 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.724+0000 7f3791400640 1 -- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 msgr2=0x7f378c0522c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.724+0000 7f3791400640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0522c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.724+0000 7f3791400640 1 -- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f378c03c0a0 msgr2=0x7f378c03da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.724+0000 7f3791400640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f378c03c0a0 0x7f378c03da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.725+0000 7f3791400640 1 -- 172.21.15.125:0/3342637797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0094d0 con 0x7f378c053cb0 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.725+0000 7f3791400640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f377c0090b0 tx=0x7f377c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.727+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c034c80 con 0x7f378c053cb0 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.727+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c034e00 con 0x7f378c053cb0 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.727+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c03f660 con 0x7f378c053cb0 2024-01-16T15:01:22.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.727+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c0e3ad0 con 0x7f378c053cb0 2024-01-16T15:01:22.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.730+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c0e4090 con 0x7f378c053cb0 2024-01-16T15:01:22.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.738+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c0e4280 con 0x7f378c053cb0 2024-01-16T15:01:22.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.747+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f377c03b040 con 0x7f378c053cb0 2024-01-16T15:01:22.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.750+0000 7f378a000640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 0x7f376c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.751+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f377c0ac840 con 0x7f378c053cb0 2024-01-16T15:01:22.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.754+0000 7f3791e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 0x7f376c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.756+0000 7f3791e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 0x7f376c06dce0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f378400a810 tx=0x7f3784008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:22.757+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f377c0772d0 con 0x7f378c053cb0 2024-01-16T15:01:23.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.347+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f378c005390 con 0x7f378c053cb0 2024-01-16T15:01:23.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.348+0000 7f378a000640 1 -- 172.21.15.125:0/3342637797 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2128 (secure 0 0 0) 0x7f377c07a8f0 con 0x7f378c053cb0 2024-01-16T15:01:23.328 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:23.328 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:23.329 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 302, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:23.330 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2396, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 9916670036, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 19430600704, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 753663512576, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:23.331 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 397983549, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 391017012, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 95, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 100 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:23.332 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:23.333 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (17s)\n [==============..............] (remaining: 17s)", 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.5, 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:23.334 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.352+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 msgr2=0x7f376c06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.352+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 0x7f376c06dce0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f378400a810 tx=0x7f3784008040 comp rx=0 tx=0).stop 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.352+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 msgr2=0x7f378c051d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.352+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f377c0090b0 tx=0x7f377c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 shutdown_connections 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f378c0e3160 0x7f378c0522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f376c06b820 0x7f376c06dce0 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f378400a810 tx=0x7f3784008040 comp rx=0 tx=0).stop 2024-01-16T15:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f378c053cb0 0x7f378c051d80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 --2- 172.21.15.125:0/3342637797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f378c03c0a0 0x7f378c03da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.353+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 >> 172.21.15.125:0/3342637797 conn(0x7f378c058ee0 msgr2=0x7f378c057d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.354+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 shutdown_connections 2024-01-16T15:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:23.354+0000 7f3798e00640 1 -- 172.21.15.125:0/3342637797 wait complete. 2024-01-16T15:01:25.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:25.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.562+0000 7f3f41000640 1 Processor -- start 2024-01-16T15:01:25.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.562+0000 7f3f41000640 1 -- start start 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.563+0000 7f3f41000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f3c15d3c0 0x7f3f3c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.563+0000 7f3f41000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.563+0000 7f3f41000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b2c70 con 0x7f3f3c0cde90 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.563+0000 7f3f41000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0d03f0 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.563+0000 7f3f41000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0cf6d0 con 0x7f3f3c0cfa40 2024-01-16T15:01:25.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f3c15d3c0 0x7f3f3c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3f3c0cde90 0x7f3f3c0ce320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48364/0 (socket says 172.21.15.125:48364) 2024-01-16T15:01:25.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3be00640 1 -- 172.21.15.125:0/3267211884 learned_addr learned my addr 172.21.15.125:0/3267211884 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:25.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3a000640 1 -- 172.21.15.125:0/3267211884 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612305410 0 0) 0x7f3f3c0b2c70 con 0x7f3f3c0cde90 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3a000640 1 -- 172.21.15.125:0/3267211884 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c003520 con 0x7f3f3c0cde90 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3aa00640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3b400640 1 -- 172.21.15.125:0/3267211884 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 msgr2=0x7f3f3c0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3b400640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3b400640 1 -- 172.21.15.125:0/3267211884 >> v1:172.21.15.125:6789/0 conn(0x7f3f3c0cde90 legacy=0x7f3f3c0ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3b400640 1 -- 172.21.15.125:0/3267211884 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0cf3f0 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.564+0000 7f3f3aa00640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:25.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f3b400640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0d1f20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7444e3d55c81dbb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f3a000640 1 -- 172.21.15.125:0/3267211884 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3402cc50 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f3a000640 1 -- 172.21.15.125:0/3267211884 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3402cdb0 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f3a000640 1 -- 172.21.15.125:0/3267211884 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34034660 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f41000640 1 -- 172.21.15.125:0/3267211884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 msgr2=0x7f3f3c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.566+0000 7f3f41000640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0d1f20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3402bea0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 -- 172.21.15.125:0/3267211884 shutdown_connections 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 --2- 172.21.15.125:0/3267211884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0d1f20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 -- 172.21.15.125:0/3267211884 >> 172.21.15.125:0/3267211884 conn(0x7f3f3c0d3050 msgr2=0x7f3f3c0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 -- 172.21.15.125:0/3267211884 shutdown_connections 2024-01-16T15:01:25.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.567+0000 7f3f41000640 1 -- 172.21.15.125:0/3267211884 wait complete. 2024-01-16T15:01:25.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.568+0000 7f3f41000640 1 Processor -- start 2024-01-16T15:01:25.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.568+0000 7f3f41000640 1 -- start start 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f3c0cde90 0x7f3f3c0c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c7a70 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c15b0f0 con 0x7f3f3c0cde90 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.569+0000 7f3f41000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c6b80 con 0x7f3f3c0cfa40 2024-01-16T15:01:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53854/0 (socket says 172.21.15.125:53854) 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 -- 172.21.15.125:0/2672369344 learned_addr learned my addr 172.21.15.125:0/2672369344 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3aa00640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3b400640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f3c0cde90 0x7f3f3c0c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 -- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 msgr2=0x7f3f3c15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c15abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 -- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f3c0cde90 msgr2=0x7f3f3c0c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f3c0cde90 0x7f3f3c0c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 -- 172.21.15.125:0/2672369344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34009a70 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f3be00640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c430 tx=0x7f3f2c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.570+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c005ce0 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.571+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c005e40 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.571+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c0c6e60 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.571+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c007310 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.571+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c0c7420 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.572+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c0c7c20 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.573+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f3f2c007470 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.573+0000 7f3f33400640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 0x7f3f1406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.573+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3f2c089ae0 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.574+0000 7f3f3b400640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 0x7f3f1406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.574+0000 7f3f3b400640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 0x7f3f1406dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3f34036000 tx=0x7f3f34030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.578+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f2c057d10 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.862+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3f3c08ba80 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.863+0000 7f3f33400640 1 -- 172.21.15.125:0/2672369344 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2111 (secure 0 0 0) 0x7f3f2c017850 con 0x7f3f3c15d3c0 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 304, 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.844 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:25.845 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2396, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 9916670036, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 19430600704, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 753663512576, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 398003687, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 391036798, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 95, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 100 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:25.846 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "33cab34c-3d29-488f-b6c4-6604158aa880": { 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (0s)\n [==============..............] ", 2024-01-16T15:01:25.848 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.5, 2024-01-16T15:01:25.849 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:01:25.849 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:25.849 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:25.849 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:25.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.870+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 msgr2=0x7f3f1406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.870+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 0x7f3f1406dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3f34036000 tx=0x7f3f34030000 comp rx=0 tx=0).stop 2024-01-16T15:01:25.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.870+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 msgr2=0x7f3f3c0c7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.870+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c430 tx=0x7f3f2c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 shutdown_connections 2024-01-16T15:01:25.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f1406b7d0 0x7f3f1406dc90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f3c15d3c0 0x7f3f3c0c7f00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f3c0cfa40 0x7f3f3c15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 --2- 172.21.15.125:0/2672369344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f3c0cde90 0x7f3f3c0c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.871+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 >> 172.21.15.125:0/2672369344 conn(0x7f3f3c0d3050 msgr2=0x7f3f3c0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.874+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 shutdown_connections 2024-01-16T15:01:25.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:25.874+0000 7f3f41000640 1 -- 172.21.15.125:0/2672369344 wait complete. 2024-01-16T15:01:27.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:28.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.224+0000 7f818be00640 1 Processor -- start 2024-01-16T15:01:28.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.224+0000 7f818be00640 1 -- start start 2024-01-16T15:01:28.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.227+0000 7f818be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818413e140 0x7f818415bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.228+0000 7f818be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818415c2f0 0x7f81840cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.228+0000 7f818be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f81840cf850 0x7f818415ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.228+0000 7f818be00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81840cfce0 con 0x7f818413e140 2024-01-16T15:01:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.228+0000 7f818be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f818406d040 con 0x7f81840cf850 2024-01-16T15:01:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.228+0000 7f818be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818406c470 con 0x7f818415c2f0 2024-01-16T15:01:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.229+0000 7f8189400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818413e140 0x7f818415bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.229+0000 7f8189400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f818413e140 0x7f818415bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35532/0 (socket says 172.21.15.125:35532) 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.229+0000 7f8189400640 1 -- 172.21.15.125:0/73837430 learned_addr learned my addr 172.21.15.125:0/73837430 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.230+0000 7f8189400640 1 -- 172.21.15.125:0/73837430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818415c2f0 msgr2=0x7f81840cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.230+0000 7f8189400640 1 --2- 172.21.15.125:0/73837430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818415c2f0 0x7f81840cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.230+0000 7f8189400640 1 -- 172.21.15.125:0/73837430 >> v2:172.21.15.137:3300/0 conn(0x7f81840cf850 msgr2=0x7f818415ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.230+0000 7f8189400640 1 --2- 172.21.15.125:0/73837430 >> v2:172.21.15.137:3300/0 conn(0x7f81840cf850 0x7f818415ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.230+0000 7f8189400640 1 -- 172.21.15.125:0/73837430 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818413e520 con 0x7f818413e140 2024-01-16T15:01:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.233+0000 7f8189400640 1 --2- 172.21.15.125:0/73837430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818413e140 0x7f818415bdb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f81840ae730 tx=0x7f817c02fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73890d5e8776f792 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.236+0000 7f8183e00640 1 -- 172.21.15.125:0/73837430 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c030ae0 con 0x7f818413e140 2024-01-16T15:01:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.241+0000 7f8183e00640 1 -- 172.21.15.125:0/73837430 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c030c40 con 0x7f818413e140 2024-01-16T15:01:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.241+0000 7f8183e00640 1 -- 172.21.15.125:0/73837430 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c038440 con 0x7f818413e140 2024-01-16T15:01:28.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.245+0000 7f818be00640 1 -- 172.21.15.125:0/73837430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818413e140 msgr2=0x7f818415bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.245+0000 7f818be00640 1 --2- 172.21.15.125:0/73837430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818413e140 0x7f818415bdb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f81840ae730 tx=0x7f817c02fd30 comp rx=0 tx=0).stop 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.249+0000 7f818be00640 1 -- 172.21.15.125:0/73837430 shutdown_connections 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.249+0000 7f818be00640 1 --2- 172.21.15.125:0/73837430 >> v2:172.21.15.137:3300/0 conn(0x7f81840cf850 0x7f818415ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.249+0000 7f818be00640 1 --2- 172.21.15.125:0/73837430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818415c2f0 0x7f81840cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.249+0000 7f818be00640 1 --2- 172.21.15.125:0/73837430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f818413e140 0x7f818415bdb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.250+0000 7f818be00640 1 -- 172.21.15.125:0/73837430 >> 172.21.15.125:0/73837430 conn(0x7f818406a800 msgr2=0x7f818406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:28.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.251+0000 7f818be00640 1 -- 172.21.15.125:0/73837430 shutdown_connections 2024-01-16T15:01:28.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.251+0000 7f818be00640 1 -- 172.21.15.125:0/73837430 wait complete. 2024-01-16T15:01:28.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.254+0000 7f818be00640 1 Processor -- start 2024-01-16T15:01:28.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.254+0000 7f818be00640 1 -- start start 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.256+0000 7f818be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.256+0000 7f818be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 0x7f81840b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.256+0000 7f818be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818415c2f0 0x7f81840b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f818be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841f83a0 con 0x7f81840cf850 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f818be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841f8680 con 0x7f818415c2f0 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f818be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81841f8960 con 0x7f818413e140 2024-01-16T15:01:28.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f8189400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f8189400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35536/0 (socket says 172.21.15.125:35536) 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f8189400640 1 -- 172.21.15.125:0/1184490312 learned_addr learned my addr 172.21.15.125:0/1184490312 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f8189e00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818415c2f0 0x7f81840b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.257+0000 7f8188a00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 0x7f81840b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8189400640 1 -- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 msgr2=0x7f81840b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8189400640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 0x7f81840b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8189400640 1 -- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818415c2f0 msgr2=0x7f81840b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8189400640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818415c2f0 0x7f81840b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8189400640 1 -- 172.21.15.125:0/1184490312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0094d0 con 0x7f81840cf850 2024-01-16T15:01:28.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.258+0000 7f8188a00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 0x7f81840b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:28.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.260+0000 7f8189400640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f817c000c00 tx=0x7f817c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.263+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c043440 con 0x7f81840cf850 2024-01-16T15:01:28.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.263+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c0435c0 con 0x7f81840cf850 2024-01-16T15:01:28.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.263+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c043970 con 0x7f81840cf850 2024-01-16T15:01:28.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.268+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81841f8c40 con 0x7f81840cf850 2024-01-16T15:01:28.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.268+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818413bee0 con 0x7f81840cf850 2024-01-16T15:01:28.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.271+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81840675d0 con 0x7f81840cf850 2024-01-16T15:01:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.274+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f817c043ad0 con 0x7f81840cf850 2024-01-16T15:01:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.274+0000 7f8181600640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 0x7f815c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.275+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f817c0ad020 con 0x7f81840cf850 2024-01-16T15:01:28.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.275+0000 7f8188a00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 0x7f815c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.279+0000 7f8188a00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 0x7f815c06dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81840b3dc0 tx=0x7f816c009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.289+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f817c077760 con 0x7f81840cf850 2024-01-16T15:01:28.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.763+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8184155360 con 0x7f81840cf850 2024-01-16T15:01:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.764+0000 7f8181600640 1 -- 172.21.15.125:0/1184490312 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1870 (secure 0 0 0) 0x7f817c03d020 con 0x7f81840cf850 2024-01-16T15:01:28.745 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:28.745 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:28.745 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:01:28.745 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:01:28.746 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 307, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 47, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:01:28.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:01:28.748 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3215, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 13326840741, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 26486272000, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 746607841280, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 499636744, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 509798996, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 119, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 130 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 29, 2024-01-16T15:01:28.749 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 3, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 3, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 5742 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:28.750 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:01:28.751 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 10, 2024-01-16T15:01:28.752 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:00:56.002869+0000", 2024-01-16T15:01:28.752 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:01:28.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:28.752 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": {} 2024-01-16T15:01:28.752 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 msgr2=0x7f815c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 0x7f815c06dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81840b3dc0 tx=0x7f816c009110 comp rx=0 tx=0).stop 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 msgr2=0x7f818415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f817c000c00 tx=0x7f817c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 shutdown_connections 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f818415c2f0 0x7f81840b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f818413e140 0x7f81840b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f815c06b7d0 0x7f815c06dc90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 --2- 172.21.15.125:0/1184490312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81840cf850 0x7f818415a420 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 >> 172.21.15.125:0/1184490312 conn(0x7f818406a800 msgr2=0x7f8184070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:28.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 shutdown_connections 2024-01-16T15:01:28.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:28.769+0000 7f818be00640 1 -- 172.21.15.125:0/1184490312 wait complete. 2024-01-16T15:01:28.783 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:28.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20151 2024-01-16T15:01:29.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.098+0000 7f6dee200640 1 Processor -- start 2024-01-16T15:01:29.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.099+0000 7f6dee200640 1 -- start start 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6dee200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de8150760 0x7f6de8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6dee200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de80b32e0 0x7f6de815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6dee200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80b3750 con 0x7f6de815d300 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6dee200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de815dc80 con 0x7f6de8150760 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6dee200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8071190 con 0x7f6de80b32e0 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6de7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de8150760 0x7f6de8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6de7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de8150760 0x7f6de8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35552/0 (socket says 172.21.15.125:35552) 2024-01-16T15:01:29.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.101+0000 7f6de7400640 1 -- 172.21.15.125:0/1217741152 learned_addr learned my addr 172.21.15.125:0/1217741152 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:29.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.102+0000 7f6de7e00640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de80b32e0 0x7f6de815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.103+0000 7f6de7400640 1 -- 172.21.15.125:0/1217741152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de80b32e0 msgr2=0x7f6de815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:29.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.103+0000 7f6de7400640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de80b32e0 0x7f6de815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:29.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.103+0000 7f6de7400640 1 -- 172.21.15.125:0/1217741152 >> v1:172.21.15.125:6789/0 conn(0x7f6de815d300 legacy=0x7f6de80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:29.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.104+0000 7f6de7400640 1 -- 172.21.15.125:0/1217741152 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de8071470 con 0x7f6de8150760 2024-01-16T15:01:29.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.106+0000 7f6de7400640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de8150760 0x7f6de8072230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009e50 tx=0x7f6dd802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40af81c2822352ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:29.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.106+0000 7f6de6000640 1 -- 172.21.15.125:0/1217741152 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd802cb00 con 0x7f6de8150760 2024-01-16T15:01:29.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.106+0000 7f6de6000640 1 -- 172.21.15.125:0/1217741152 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd802cc60 con 0x7f6de8150760 2024-01-16T15:01:29.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.106+0000 7f6de6000640 1 -- 172.21.15.125:0/1217741152 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80343f0 con 0x7f6de8150760 2024-01-16T15:01:29.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.107+0000 7f6dee200640 1 -- 172.21.15.125:0/1217741152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de8150760 msgr2=0x7f6de8072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:29.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.107+0000 7f6dee200640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de8150760 0x7f6de8072230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009e50 tx=0x7f6dd802bf60 comp rx=0 tx=0).stop 2024-01-16T15:01:29.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.108+0000 7f6dee200640 1 -- 172.21.15.125:0/1217741152 shutdown_connections 2024-01-16T15:01:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.108+0000 7f6dee200640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de80b32e0 0x7f6de815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.108+0000 7f6dee200640 1 --2- 172.21.15.125:0/1217741152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de8150760 0x7f6de8072230 secure :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009e50 tx=0x7f6dd802bf60 comp rx=0 tx=0).stop 2024-01-16T15:01:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.108+0000 7f6dee200640 1 -- 172.21.15.125:0/1217741152 >> 172.21.15.125:0/1217741152 conn(0x7f6de806a800 msgr2=0x7f6de806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.109+0000 7f6dee200640 1 -- 172.21.15.125:0/1217741152 shutdown_connections 2024-01-16T15:01:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.109+0000 7f6dee200640 1 -- 172.21.15.125:0/1217741152 wait complete. 2024-01-16T15:01:29.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.111+0000 7f6dee200640 1 Processor -- start 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.111+0000 7f6dee200640 1 -- start start 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de815d300 0x7f6de8141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de814fd60 0x7f6de8140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de816d720 con 0x7f6de80b32e0 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80b0930 con 0x7f6de815d300 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6dee200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de81578a0 con 0x7f6de814fd60 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.112+0000 7f6de7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35560/0 (socket says 172.21.15.125:35560) 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 -- 172.21.15.125:0/2567214461 learned_addr learned my addr 172.21.15.125:0/2567214461 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de6a00640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de815d300 0x7f6de8141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7e00640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de814fd60 0x7f6de8140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 -- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de814fd60 msgr2=0x7f6de8140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de814fd60 0x7f6de8140580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 -- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de815d300 msgr2=0x7f6de8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de815d300 0x7f6de8141d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:29.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.113+0000 7f6de7400640 1 -- 172.21.15.125:0/2567214461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8009b30 con 0x7f6de80b32e0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.114+0000 7f6de7400640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8036040 tx=0x7f6dd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.114+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd803e450 con 0x7f6de80b32e0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.114+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd803e5b0 con 0x7f6de80b32e0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.114+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd803e960 con 0x7f6de80b32e0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.114+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de81411e0 con 0x7f6de80b32e0 2024-01-16T15:01:29.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.115+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de8157b60 con 0x7f6de80b32e0 2024-01-16T15:01:29.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.117+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de80675d0 con 0x7f6de80b32e0 2024-01-16T15:01:29.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.122+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f6dd803eac0 con 0x7f6de80b32e0 2024-01-16T15:01:29.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.125+0000 7f6ddf400640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 0x7f6dc006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:29.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.125+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6dd80ad010 con 0x7f6de80b32e0 2024-01-16T15:01:29.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.131+0000 7f6de6a00640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 0x7f6dc006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:29.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.132+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6dd80778a0 con 0x7f6de80b32e0 2024-01-16T15:01:29.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.132+0000 7f6de6a00640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 0x7f6dc006dc90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6de80cede0 tx=0x7f6dd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:29.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:29.483+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20151", "target": ["mon-mgr", ""]}) v1 -- 0x7f6de8071d10 con 0x7f6dc006b7d0 2024-01-16T15:01:30.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.548+0000 7f6ddf400640 1 -- 172.21.15.125:0/2567214461 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f6dd0001380 con 0x7f6dc006b7d0 2024-01-16T15:01:30.528 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:30.528 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:30.529 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:01:30.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:30.529 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:30.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.551+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 msgr2=0x7f6dc006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 0x7f6dc006dc90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6de80cede0 tx=0x7f6dd0009040 comp rx=0 tx=0).stop 2024-01-16T15:01:30.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 msgr2=0x7f6de80ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8036040 tx=0x7f6dd8004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:30.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 shutdown_connections 2024-01-16T15:01:30.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6de814fd60 0x7f6de8140580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de815d300 0x7f6de8141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6dc006b7d0 0x7f6dc006dc90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 --2- 172.21.15.125:0/2567214461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de80b32e0 0x7f6de80ce770 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 >> 172.21.15.125:0/2567214461 conn(0x7f6de806a800 msgr2=0x7f6de815b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:30.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 shutdown_connections 2024-01-16T15:01:30.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.552+0000 7f6dee200640 1 -- 172.21.15.125:0/2567214461 wait complete. 2024-01-16T15:01:30.551 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:30.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20152 2024-01-16T15:01:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.787+0000 7f6b44a00640 1 Processor -- start 2024-01-16T15:01:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.787+0000 7f6b44a00640 1 -- start start 2024-01-16T15:01:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b44a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c143ae0 0x7f6b3c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b44a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b3c15e3b0 0x7f6b3c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b44a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0cfc30 con 0x7f6b3c0cf7d0 2024-01-16T15:01:30.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b44a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3c0b1ee0 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b44a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0b2780 con 0x7f6b3c143ae0 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b41600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6b3c0cf7d0 0x7f6b3c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59402/0 (socket says 172.21.15.125:59402) 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.789+0000 7f6b41600640 1 -- 172.21.15.125:0/1710246379 learned_addr learned my addr 172.21.15.125:0/1710246379 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42a00640 1 --2- 172.21.15.125:0/1710246379 >> v2:172.21.15.125:3300/0 conn(0x7f6b3c15e3b0 0x7f6b3c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42000640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c143ae0 0x7f6b3c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b40c00640 1 -- 172.21.15.125:0/1710246379 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997718319 0 0) 0x7f6b3c0cfc30 con 0x7f6b3c0cf7d0 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b40c00640 1 -- 172.21.15.125:0/1710246379 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1c005730 con 0x7f6b3c0cf7d0 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42a00640 1 -- 172.21.15.125:0/1710246379 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c143ae0 msgr2=0x7f6b3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42a00640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c143ae0 0x7f6b3c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42a00640 1 -- 172.21.15.125:0/1710246379 >> v1:172.21.15.125:6789/0 conn(0x7f6b3c0cf7d0 legacy=0x7f6b3c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.790+0000 7f6b42a00640 1 -- 172.21.15.125:0/1710246379 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c15e990 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.791+0000 7f6b42a00640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c0702c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009d90 tx=0x7f6b2c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71bbfcd012e10dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.791+0000 7f6b40c00640 1 -- 172.21.15.125:0/1710246379 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c02cc60 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.791+0000 7f6b40c00640 1 -- 172.21.15.125:0/1710246379 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c02cdc0 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.791+0000 7f6b40c00640 1 -- 172.21.15.125:0/1710246379 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c034660 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 -- 172.21.15.125:0/1710246379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c15e3b0 msgr2=0x7f6b3c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c0702c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009d90 tx=0x7f6b2c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 -- 172.21.15.125:0/1710246379 shutdown_connections 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c0702c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 --2- 172.21.15.125:0/1710246379 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c143ae0 0x7f6b3c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 -- 172.21.15.125:0/1710246379 >> 172.21.15.125:0/1710246379 conn(0x7f6b3c06a800 msgr2=0x7f6b3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 -- 172.21.15.125:0/1710246379 shutdown_connections 2024-01-16T15:01:30.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.792+0000 7f6b44a00640 1 -- 172.21.15.125:0/1710246379 wait complete. 2024-01-16T15:01:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.794+0000 7f6b44a00640 1 Processor -- start 2024-01-16T15:01:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.794+0000 7f6b44a00640 1 -- start start 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c0cf7d0 0x7f6b3c1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0cc610 con 0x7f6b3c143ae0 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0cc860 con 0x7f6b3c15e3b0 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b44a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0ccb40 con 0x7f6b3c0cf7d0 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b41600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b41600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35580/0 (socket says 172.21.15.125:35580) 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.795+0000 7f6b41600640 1 -- 172.21.15.125:0/3610200661 learned_addr learned my addr 172.21.15.125:0/3610200661 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 -- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c0cf7d0 msgr2=0x7f6b3c1f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b42000640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c0cf7d0 0x7f6b3c1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c0cf7d0 0x7f6b3c1f2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 -- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b3c15e3b0 msgr2=0x7f6b3c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 -- 172.21.15.125:0/3610200661 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c009a70 con 0x7f6b3c143ae0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b41600640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c500 tx=0x7f6b2400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24005ce0 con 0x7f6b3c143ae0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b24005e40 con 0x7f6b3c143ae0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c0cce20 con 0x7f6b3c143ae0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24007330 con 0x7f6b3c143ae0 2024-01-16T15:01:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.796+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c1410c0 con 0x7f6b3c143ae0 2024-01-16T15:01:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.797+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c0cd340 con 0x7f6b3c143ae0 2024-01-16T15:01:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.798+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f6b24007550 con 0x7f6b3c143ae0 2024-01-16T15:01:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.799+0000 7f6b3a000640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 0x7f6b1406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.799+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b24089a30 con 0x7f6b3c143ae0 2024-01-16T15:01:30.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.799+0000 7f6b42000640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 0x7f6b1406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.800+0000 7f6b42000640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 0x7f6b1406dc90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009a10 tx=0x7f6b34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:30.804+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b3c0cd340 con 0x7f6b3c143ae0 2024-01-16T15:01:31.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.041+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20152", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b3c0b2410 con 0x7f6b1406b7d0 2024-01-16T15:01:31.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.058+0000 7f6b3a000640 1 -- 172.21.15.125:0/3610200661 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f6b34010870 con 0x7f6b1406b7d0 2024-01-16T15:01:31.039 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:31.039 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:31.039 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:01:31.039 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:31.039 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 msgr2=0x7f6b1406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 0x7f6b1406dc90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009a10 tx=0x7f6b34008040 comp rx=0 tx=0).stop 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 msgr2=0x7f6b3c0acb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c500 tx=0x7f6b2400c9c0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 shutdown_connections 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b3c15e3b0 0x7f6b3c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6b1406b7d0 0x7f6b1406dc90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b3c143ae0 0x7f6b3c0acb40 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 --2- 172.21.15.125:0/3610200661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b3c0cf7d0 0x7f6b3c1f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.061+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 >> 172.21.15.125:0/3610200661 conn(0x7f6b3c06a800 msgr2=0x7f6b3c142db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:31.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.062+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 shutdown_connections 2024-01-16T15:01:31.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.062+0000 7f6b44a00640 1 -- 172.21.15.125:0/3610200661 wait complete. 2024-01-16T15:01:31.059 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:31.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20153 2024-01-16T15:01:31.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.283+0000 7fd63fe00640 1 Processor -- start 2024-01-16T15:01:31.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.283+0000 7fd63fe00640 1 -- start start 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.284+0000 7fd63fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6400cf9a0 0x7fd6400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.284+0000 7fd63fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400b31d0 0x7fd6400b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.284+0000 7fd63fe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400d0320 con 0x7fd64015d4b0 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.284+0000 7fd63fe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400d1c60 con 0x7fd6400cf9a0 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.284+0000 7fd63fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400b2730 con 0x7fd6400b31d0 2024-01-16T15:01:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd63ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd64015d4b0 0x7fd6400d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59410/0 (socket says 172.21.15.125:59410) 2024-01-16T15:01:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd63ea00640 1 -- 172.21.15.125:0/3850183349 learned_addr learned my addr 172.21.15.125:0/3850183349 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd635e00640 1 --2- 172.21.15.125:0/3850183349 >> v2:172.21.15.125:3300/0 conn(0x7fd6400cf9a0 0x7fd6400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd63f400640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400b31d0 0x7fd6400b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd63e000640 1 -- 172.21.15.125:0/3850183349 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504926624 0 0) 0x7fd6400d0320 con 0x7fd64015d4b0 2024-01-16T15:01:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd63e000640 1 -- 172.21.15.125:0/3850183349 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd620003520 con 0x7fd64015d4b0 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd635e00640 1 -- 172.21.15.125:0/3850183349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400b31d0 msgr2=0x7fd6400b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd635e00640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400b31d0 0x7fd6400b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd635e00640 1 -- 172.21.15.125:0/3850183349 >> v1:172.21.15.125:6789/0 conn(0x7fd64015d4b0 legacy=0x7fd6400d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.285+0000 7fd635e00640 1 -- 172.21.15.125:0/3850183349 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400d1ec0 con 0x7fd6400cf9a0 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.286+0000 7fd635e00640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400cf9a0 0x7fd6400cfde0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd6280097a0 tx=0x7fd62802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc2c1401731a971 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63e000640 1 -- 172.21.15.125:0/3850183349 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62802d040 con 0x7fd6400cf9a0 2024-01-16T15:01:31.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63e000640 1 -- 172.21.15.125:0/3850183349 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628031810 con 0x7fd6400cf9a0 2024-01-16T15:01:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63e000640 1 -- 172.21.15.125:0/3850183349 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628031ba0 con 0x7fd6400cf9a0 2024-01-16T15:01:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 -- 172.21.15.125:0/3850183349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400cf9a0 msgr2=0x7fd6400cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400cf9a0 0x7fd6400cfde0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd6280097a0 tx=0x7fd62802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 -- 172.21.15.125:0/3850183349 shutdown_connections 2024-01-16T15:01:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400b31d0 0x7fd6400b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 --2- 172.21.15.125:0/3850183349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400cf9a0 0x7fd6400cfde0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.287+0000 7fd63fe00640 1 -- 172.21.15.125:0/3850183349 >> 172.21.15.125:0/3850183349 conn(0x7fd6400d3000 msgr2=0x7fd6400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.288+0000 7fd63fe00640 1 -- 172.21.15.125:0/3850183349 shutdown_connections 2024-01-16T15:01:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.288+0000 7fd63fe00640 1 -- 172.21.15.125:0/3850183349 wait complete. 2024-01-16T15:01:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.289+0000 7fd63fe00640 1 Processor -- start 2024-01-16T15:01:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.290+0000 7fd63fe00640 1 -- start start 2024-01-16T15:01:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.290+0000 7fd63fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.290+0000 7fd63fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400cf9a0 0x7fd640154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd64015d4b0 0x7fd64009daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63fe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400cb790 con 0x7fd6400b31d0 2024-01-16T15:01:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400a0650 con 0x7fd64015d4b0 2024-01-16T15:01:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400a0930 con 0x7fd6400cf9a0 2024-01-16T15:01:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35598/0 (socket says 172.21.15.125:35598) 2024-01-16T15:01:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 -- 172.21.15.125:0/1282084742 learned_addr learned my addr 172.21.15.125:0/1282084742 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63f400640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd64015d4b0 0x7fd64009daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd635e00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400cf9a0 0x7fd640154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 -- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400cf9a0 msgr2=0x7fd640154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400cf9a0 0x7fd640154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 -- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd64015d4b0 msgr2=0x7fd64009daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd64015d4b0 0x7fd64009daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.291+0000 7fd63ea00640 1 -- 172.21.15.125:0/1282084742 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628009400 con 0x7fd6400b31d0 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd63ea00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd63000cbc0 tx=0x7fd63000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000ad40 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630012400 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000aea0 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6400a0c10 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.292+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6401548c0 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.294+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6400a0da0 con 0x7fd6400b31d0 2024-01-16T15:01:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.294+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd630007070 con 0x7fd6400b31d0 2024-01-16T15:01:31.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.295+0000 7fd637400640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 0x7fd61806dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:31.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.295+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd63008a3b0 con 0x7fd6400b31d0 2024-01-16T15:01:31.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.296+0000 7fd635e00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 0x7fd61806dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:31.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.297+0000 7fd635e00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 0x7fd61806dcc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd628036000 tx=0x7fd62803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:31.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.301+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd630054c40 con 0x7fd6400b31d0 2024-01-16T15:01:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:31.530+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20153", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6400d08e0 con 0x7fd61806b800 2024-01-16T15:01:33.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.696+0000 7fd637400640 1 -- 172.21.15.125:0/1282084742 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd6400d08e0 con 0x7fd61806b800 2024-01-16T15:01:33.677 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:33.677 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:33.677 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:01:33.677 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:33.677 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:33.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 msgr2=0x7fd61806dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 0x7fd61806dcc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd628036000 tx=0x7fd62803c040 comp rx=0 tx=0).stop 2024-01-16T15:01:33.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 msgr2=0x7fd6400a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd63000cbc0 tx=0x7fd63000a430 comp rx=0 tx=0).stop 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 shutdown_connections 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd64015d4b0 0x7fd64009daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6400cf9a0 0x7fd640154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd61806b800 0x7fd61806dcc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 --2- 172.21.15.125:0/1282084742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400b31d0 0x7fd6400a0020 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.700+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 >> 172.21.15.125:0/1282084742 conn(0x7fd6400d3000 msgr2=0x7fd6400cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.701+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 shutdown_connections 2024-01-16T15:01:33.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.701+0000 7fd63fe00640 1 -- 172.21.15.125:0/1282084742 wait complete. 2024-01-16T15:01:33.700 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:33.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone20154 2024-01-16T15:01:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.922+0000 7fede6a00640 1 Processor -- start 2024-01-16T15:01:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.922+0000 7fede6a00640 1 -- start start 2024-01-16T15:01:33.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fede0143810 0x7fede0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fede015d770 0x7fede015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00ae620 con 0x7fede00ae170 2024-01-16T15:01:33.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede0143bf0 con 0x7fede015d770 2024-01-16T15:01:33.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede006c730 con 0x7fede0143810 2024-01-16T15:01:33.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fede015d770 0x7fede015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fede015d770 0x7fede015dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35600/0 (socket says 172.21.15.125:35600) 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.924+0000 7fede6000640 1 -- 172.21.15.125:0/1831665191 learned_addr learned my addr 172.21.15.125:0/1831665191 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.925+0000 7fede5600640 1 --2- 172.21.15.125:0/1831665191 >> v2:172.21.15.137:3300/0 conn(0x7fede0143810 0x7fede0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.925+0000 7fede6000640 1 -- 172.21.15.125:0/1831665191 >> v2:172.21.15.137:3300/0 conn(0x7fede0143810 msgr2=0x7fede0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.925+0000 7fede6000640 1 --2- 172.21.15.125:0/1831665191 >> v2:172.21.15.137:3300/0 conn(0x7fede0143810 0x7fede0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.925+0000 7fede6000640 1 -- 172.21.15.125:0/1831665191 >> v1:172.21.15.137:6789/0 conn(0x7fede00ae170 legacy=0x7fede015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.925+0000 7fede6000640 1 -- 172.21.15.125:0/1831665191 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00724d0 con 0x7fede015d770 2024-01-16T15:01:33.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.926+0000 7fede6000640 1 --2- 172.21.15.125:0/1831665191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede015dbc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fede00cec40 tx=0x7fedd802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0f7f04d47aa5e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.926+0000 7feddfe00640 1 -- 172.21.15.125:0/1831665191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8031750 con 0x7fede015d770 2024-01-16T15:01:33.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.926+0000 7feddfe00640 1 -- 172.21.15.125:0/1831665191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80318b0 con 0x7fede015d770 2024-01-16T15:01:33.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.926+0000 7feddfe00640 1 -- 172.21.15.125:0/1831665191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8031c40 con 0x7fede015d770 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 -- 172.21.15.125:0/1831665191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 msgr2=0x7fede015dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 --2- 172.21.15.125:0/1831665191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede015dbc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fede00cec40 tx=0x7fedd802b7f0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 -- 172.21.15.125:0/1831665191 shutdown_connections 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 --2- 172.21.15.125:0/1831665191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede015dbc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 --2- 172.21.15.125:0/1831665191 >> v2:172.21.15.137:3300/0 conn(0x7fede0143810 0x7fede0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 -- 172.21.15.125:0/1831665191 >> 172.21.15.125:0/1831665191 conn(0x7fede006a7d0 msgr2=0x7fede006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 -- 172.21.15.125:0/1831665191 shutdown_connections 2024-01-16T15:01:33.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.927+0000 7fede6a00640 1 -- 172.21.15.125:0/1831665191 wait complete. 2024-01-16T15:01:33.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.929+0000 7fede6a00640 1 Processor -- start 2024-01-16T15:01:33.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.929+0000 7fede6a00640 1 -- start start 2024-01-16T15:01:33.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 0x7fede013b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fede0143810 0x7fede006d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0144240 con 0x7fede015d770 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0143000 con 0x7fede00ae170 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede013bed0 con 0x7fede0143810 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35606/0 (socket says 172.21.15.125:35606) 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.930+0000 7fede6000640 1 -- 172.21.15.125:0/2621020050 learned_addr learned my addr 172.21.15.125:0/2621020050 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede5600640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 0x7fede013b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede4c00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fede0143810 0x7fede006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 -- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fede0143810 msgr2=0x7fede006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fede0143810 0x7fede006d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 -- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 msgr2=0x7fede013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 0x7fede013b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 -- 172.21.15.125:0/2621020050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80094d0 con 0x7fede015d770 2024-01-16T15:01:33.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede5600640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 0x7fede013b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:33.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7fede6000640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fedd802b830 tx=0x7fedd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8033c80 con 0x7fede015d770 2024-01-16T15:01:33.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8033de0 con 0x7fede015d770 2024-01-16T15:01:33.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.931+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd803c640 con 0x7fede015d770 2024-01-16T15:01:33.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.932+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede0142a40 con 0x7fede015d770 2024-01-16T15:01:33.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.932+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede0143e60 con 0x7fede015d770 2024-01-16T15:01:33.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.933+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede0144380 con 0x7fede015d770 2024-01-16T15:01:33.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.934+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fedd803a040 con 0x7fede015d770 2024-01-16T15:01:33.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.935+0000 7feddd600640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 0x7fedb806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.935+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fedd80abf30 con 0x7fede015d770 2024-01-16T15:01:33.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.935+0000 7fede5600640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 0x7fedb806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.936+0000 7fede5600640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 0x7fedb806dc90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007330 tx=0x7fedd000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:33.939+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fede0144380 con 0x7fede015d770 2024-01-16T15:01:34.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:34.168+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone20154", "target": ["mon-mgr", ""]}) v1 -- 0x7fede006ccd0 con 0x7fedb806b7d0 2024-01-16T15:01:37.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:37.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:38.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.695+0000 7feddd600640 1 -- 172.21.15.125:0/2621020050 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fedd0009c00 con 0x7fedb806b7d0 2024-01-16T15:01:38.675 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:01:38.675 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:01:38.675 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:01:38.675 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:38.675 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 msgr2=0x7fedb806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 0x7fedb806dc90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007330 tx=0x7fedd000b040 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 msgr2=0x7fede006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fedd802b830 tx=0x7fedd80046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 shutdown_connections 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedb806b7d0 0x7fedb806dc90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fede015d770 0x7fede006d860 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fede0143810 0x7fede006d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 --2- 172.21.15.125:0/2621020050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fede00ae170 0x7fede013b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 >> 172.21.15.125:0/2621020050 conn(0x7fede006a7d0 msgr2=0x7fede0140620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 shutdown_connections 2024-01-16T15:01:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.698+0000 7fede6a00640 1 -- 172.21.15.125:0/2621020050 wait complete. 2024-01-16T15:01:38.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:01:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.927+0000 7f9a7aa00640 1 Processor -- start 2024-01-16T15:01:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.927+0000 7f9a7aa00640 1 -- start start 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a7aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a74070b20 0x7f9a7416c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a7aa00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7406d9d0 con 0x7f9a740b3e80 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a7aa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7406dcb0 con 0x7f9a741619c0 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a7aa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406df90 con 0x7f9a74070b20 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a6b200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9a740b3e80 0x7f9a74070410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41590/0 (socket says 172.21.15.125:41590) 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a6b200640 1 -- 172.21.15.125:0/3464869371 learned_addr learned my addr 172.21.15.125:0/3464869371 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.929+0000 7f9a78a00640 1 --2- 172.21.15.125:0/3464869371 >> v2:172.21.15.125:3300/0 conn(0x7f9a74070b20 0x7f9a7416c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.930+0000 7f9a78a00640 1 -- 172.21.15.125:0/3464869371 >> v1:172.21.15.137:6789/0 conn(0x7f9a741619c0 legacy=0x7f9a7406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:38.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.930+0000 7f9a73400640 1 -- 172.21.15.125:0/3464869371 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179156099 0 0) 0x7f9a7406d9d0 con 0x7f9a740b3e80 2024-01-16T15:01:38.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.930+0000 7f9a78a00640 1 -- 172.21.15.125:0/3464869371 >> v1:172.21.15.125:6789/0 conn(0x7f9a740b3e80 legacy=0x7f9a74070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.930+0000 7f9a78a00640 1 -- 172.21.15.125:0/3464869371 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b2670 con 0x7f9a74070b20 2024-01-16T15:01:38.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.931+0000 7f9a78a00640 1 --2- 172.21.15.125:0/3464869371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a74070b20 0x7f9a7416c3d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009d90 tx=0x7f9a6c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b7566df83b9a80 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.931+0000 7f9a73400640 1 -- 172.21.15.125:0/3464869371 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c02cd00 con 0x7f9a74070b20 2024-01-16T15:01:38.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.931+0000 7f9a73400640 1 -- 172.21.15.125:0/3464869371 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c02ce60 con 0x7f9a74070b20 2024-01-16T15:01:38.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.931+0000 7f9a73400640 1 -- 172.21.15.125:0/3464869371 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c034660 con 0x7f9a74070b20 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3464869371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a74070b20 msgr2=0x7f9a7416c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3464869371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a74070b20 0x7f9a7416c3d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009d90 tx=0x7f9a6c02bf50 comp rx=0 tx=0).stop 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3464869371 shutdown_connections 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3464869371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a74070b20 0x7f9a7416c3d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3464869371 >> 172.21.15.125:0/3464869371 conn(0x7f9a740b0c00 msgr2=0x7f9a740b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3464869371 shutdown_connections 2024-01-16T15:01:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.932+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3464869371 wait complete. 2024-01-16T15:01:38.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.934+0000 7f9a7aa00640 1 Processor -- start 2024-01-16T15:01:38.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.934+0000 7f9a7aa00640 1 -- start start 2024-01-16T15:01:38.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a74070b20 0x7f9a741480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a740b3e80 0x7f9a7415db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7415e0d0 con 0x7f9a741619c0 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740cbdc0 con 0x7f9a74070b20 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a7aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740ccc00 con 0x7f9a740b3e80 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59252/0 (socket says 172.21.15.125:59252) 2024-01-16T15:01:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a78a00640 1 -- 172.21.15.125:0/3122522974 learned_addr learned my addr 172.21.15.125:0/3122522974 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:38.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.935+0000 7f9a73e00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a74070b20 0x7f9a741480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a6b200640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a740b3e80 0x7f9a7415db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 -- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a740b3e80 msgr2=0x7f9a7415db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a740b3e80 0x7f9a7415db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 -- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a74070b20 msgr2=0x7f9a741480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a74070b20 0x7f9a741480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 -- 172.21.15.125:0/3122522974 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c009a70 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a78a00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009eb0 tx=0x7f9a6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c003b90 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c003cf0 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c004230 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.936+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740cbff0 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.937+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740cc5b0 con 0x7f9a741619c0 2024-01-16T15:01:38.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.938+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74067640 con 0x7f9a741619c0 2024-01-16T15:01:38.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.939+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f9a6c03b050 con 0x7f9a741619c0 2024-01-16T15:01:38.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.940+0000 7f9a70c00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 0x7f9a4c06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.940+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9a6c0accd0 con 0x7f9a741619c0 2024-01-16T15:01:38.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.940+0000 7f9a73e00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 0x7f9a4c06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.941+0000 7f9a73e00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 0x7f9a4c06dcc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9a64000f60 tx=0x7f9a64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:38.944+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a6c077540 con 0x7f9a741619c0 2024-01-16T15:01:39.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.163+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a740b2190 con 0x7f9a4c06b800 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.165+0000 7f9a70c00640 1 -- 172.21.15.125:0/3122522974 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9a64001c70 con 0x7f9a4c06b800 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:39.145 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:01:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.167+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 msgr2=0x7f9a4c06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.167+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 0x7f9a4c06dcc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9a64000f60 tx=0x7f9a64009040 comp rx=0 tx=0).stop 2024-01-16T15:01:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.167+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 msgr2=0x7f9a740cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.167+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009eb0 tx=0x7f9a6c004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 shutdown_connections 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a4c06b800 0x7f9a4c06dcc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a741619c0 0x7f9a740cb880 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a740b3e80 0x7f9a7415db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 --2- 172.21.15.125:0/3122522974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a74070b20 0x7f9a741480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 >> 172.21.15.125:0/3122522974 conn(0x7f9a740b0c00 msgr2=0x7f9a740adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 shutdown_connections 2024-01-16T15:01:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.168+0000 7f9a7aa00640 1 -- 172.21.15.125:0/3122522974 wait complete. 2024-01-16T15:01:39.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:01:39.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.400+0000 7f7782400640 1 Processor -- start 2024-01-16T15:01:39.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.400+0000 7f7782400640 1 -- start start 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f7782400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0ae5a0 0x7f777c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f7782400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f777c0cf290 0x7f777c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f7782400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c070c40 con 0x7f777c0b3ae0 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f7782400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777c071450 con 0x7f777c0cf290 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f7782400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0cfc60 con 0x7f777c0ae5a0 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f777aa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f777c0b3ae0 0x7f777c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41606/0 (socket says 172.21.15.125:41606) 2024-01-16T15:01:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.402+0000 7f777aa00640 1 -- 172.21.15.125:0/1330739145 learned_addr learned my addr 172.21.15.125:0/1330739145 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777be00640 1 --2- 172.21.15.125:0/1330739145 >> v2:172.21.15.125:3300/0 conn(0x7f777c0cf290 0x7f777c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777b400640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0ae5a0 0x7f777c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777a000640 1 -- 172.21.15.125:0/1330739145 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465926332 0 0) 0x7f777c070c40 con 0x7f777c0b3ae0 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777a000640 1 -- 172.21.15.125:0/1330739145 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c003520 con 0x7f777c0b3ae0 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777be00640 1 -- 172.21.15.125:0/1330739145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0ae5a0 msgr2=0x7f777c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777be00640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0ae5a0 0x7f777c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777be00640 1 -- 172.21.15.125:0/1330739145 >> v1:172.21.15.125:6789/0 conn(0x7f777c0b3ae0 legacy=0x7f777c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.403+0000 7f777be00640 1 -- 172.21.15.125:0/1330739145 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c06c470 con 0x7f777c0cf290 2024-01-16T15:01:39.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.404+0000 7f777be00640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0cf290 0x7f777c0cf720 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f77740097a0 tx=0x7f777402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863651d9179808f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.404+0000 7f777a000640 1 -- 172.21.15.125:0/1330739145 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774030800 con 0x7f777c0cf290 2024-01-16T15:01:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f777a000640 1 -- 172.21.15.125:0/1330739145 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774030960 con 0x7f777c0cf290 2024-01-16T15:01:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f777a000640 1 -- 172.21.15.125:0/1330739145 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774030cd0 con 0x7f777c0cf290 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 -- 172.21.15.125:0/1330739145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0cf290 msgr2=0x7f777c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0cf290 0x7f777c0cf720 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f77740097a0 tx=0x7f777402b940 comp rx=0 tx=0).stop 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 -- 172.21.15.125:0/1330739145 shutdown_connections 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0cf290 0x7f777c0cf720 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 --2- 172.21.15.125:0/1330739145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0ae5a0 0x7f777c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 -- 172.21.15.125:0/1330739145 >> 172.21.15.125:0/1330739145 conn(0x7f777c06a800 msgr2=0x7f777c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 -- 172.21.15.125:0/1330739145 shutdown_connections 2024-01-16T15:01:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.405+0000 7f7782400640 1 -- 172.21.15.125:0/1330739145 wait complete. 2024-01-16T15:01:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.407+0000 7f7782400640 1 Processor -- start 2024-01-16T15:01:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.407+0000 7f7782400640 1 -- start start 2024-01-16T15:01:39.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c0b3ae0 0x7f777c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0cf290 0x7f777c1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c154750 con 0x7f777c0ae5a0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c157420 con 0x7f777c0b3ae0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f7782400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c157700 con 0x7f777c0cf290 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f777b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f777b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59270/0 (socket says 172.21.15.125:59270) 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.408+0000 7f777b400640 1 -- 172.21.15.125:0/2321833997 learned_addr learned my addr 172.21.15.125:0/2321833997 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777aa00640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c0b3ae0 0x7f777c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777be00640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0cf290 0x7f777c1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 -- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0cf290 msgr2=0x7f777c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0cf290 0x7f777c1581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 -- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c0b3ae0 msgr2=0x7f777c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c0b3ae0 0x7f777c15eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 -- 172.21.15.125:0/2321833997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774009400 con 0x7f777c0ae5a0 2024-01-16T15:01:39.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f777b400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f776c007ef0 tx=0x7f776c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c014620 con 0x7f777c0ae5a0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c014780 con 0x7f777c0ae5a0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c014b30 con 0x7f777c0ae5a0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.409+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c1579e0 con 0x7f777c0ae5a0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.410+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c152cd0 con 0x7f777c0ae5a0 2024-01-16T15:01:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.411+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c0675d0 con 0x7f777c0ae5a0 2024-01-16T15:01:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.411+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f776c014c90 con 0x7f777c0ae5a0 2024-01-16T15:01:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.413+0000 7f7773400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 0x7f775406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.413+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f776c08ae60 con 0x7f777c0ae5a0 2024-01-16T15:01:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.413+0000 7f777aa00640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 0x7f775406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.414+0000 7f777aa00640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 0x7f775406dc90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f777c0ae130 tx=0x7f7764009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.417+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f776c055600 con 0x7f777c0ae5a0 2024-01-16T15:01:39.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.633+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f777c071170 con 0x7f775406b7d0 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.649+0000 7f7773400640 1 -- 172.21.15.125:0/2321833997 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7f77640058d0 con 0x7f775406b7d0 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone20151" 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone20153" 2024-01-16T15:01:39.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone20154" 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone20152" 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:39.630 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 msgr2=0x7f775406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 0x7f775406dc90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f777c0ae130 tx=0x7f7764009040 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 msgr2=0x7f777c155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f776c007ef0 tx=0x7f776c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 shutdown_connections 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0cf290 0x7f777c1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c0b3ae0 0x7f777c15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406b7d0 0x7f775406dc90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 --2- 172.21.15.125:0/2321833997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f777c0ae5a0 0x7f777c155d30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 >> 172.21.15.125:0/2321833997 conn(0x7f777c06a800 msgr2=0x7f777c15a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:39.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 shutdown_connections 2024-01-16T15:01:39.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.652+0000 7f7782400640 1 -- 172.21.15.125:0/2321833997 wait complete. 2024-01-16T15:01:39.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone20151 2024-01-16T15:01:39.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.880+0000 7f3515400640 1 Processor -- start 2024-01-16T15:01:39.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.880+0000 7f3515400640 1 -- start start 2024-01-16T15:01:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f3515400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 0x7f3510072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f3515400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f351015d300 0x7f351015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f3515400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351015dc80 con 0x7f35100b2bd0 2024-01-16T15:01:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f3515400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35100b3770 con 0x7f351015d300 2024-01-16T15:01:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f3515400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3510071190 con 0x7f3510150760 2024-01-16T15:01:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f350e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 0x7f3510072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f350de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f351015d300 0x7f351015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f350e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 0x7f3510072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:38100/0 (socket says 172.21.15.125:38100) 2024-01-16T15:01:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.882+0000 7f350e800640 1 -- 172.21.15.125:0/3675771647 learned_addr learned my addr 172.21.15.125:0/3675771647 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.883+0000 7f350de00640 1 -- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 msgr2=0x7f3510072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.883+0000 7f350de00640 1 --2- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 0x7f3510072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.883+0000 7f350de00640 1 -- 172.21.15.125:0/3675771647 >> v1:172.21.15.137:6789/0 conn(0x7f35100b2bd0 legacy=0x7f35100b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.883+0000 7f350de00640 1 -- 172.21.15.125:0/3675771647 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510071470 con 0x7f351015d300 2024-01-16T15:01:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.884+0000 7f350de00640 1 --2- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f351015d300 0x7f351015d740 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f34f800ba40 tx=0x7f34f802de60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d4ce65b3245435 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.885+0000 7f350d400640 1 -- 172.21.15.125:0/3675771647 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f802ec10 con 0x7f351015d300 2024-01-16T15:01:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.885+0000 7f350d400640 1 -- 172.21.15.125:0/3675771647 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8032070 con 0x7f351015d300 2024-01-16T15:01:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.885+0000 7f350d400640 1 -- 172.21.15.125:0/3675771647 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8038400 con 0x7f351015d300 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.885+0000 7f3515400640 1 -- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f351015d300 msgr2=0x7f351015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.885+0000 7f3515400640 1 --2- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f351015d300 0x7f351015d740 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f34f800ba40 tx=0x7f34f802de60 comp rx=0 tx=0).stop 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 -- 172.21.15.125:0/3675771647 shutdown_connections 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 --2- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f351015d300 0x7f351015d740 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 --2- 172.21.15.125:0/3675771647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3510150760 0x7f3510072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 -- 172.21.15.125:0/3675771647 >> 172.21.15.125:0/3675771647 conn(0x7f351006a800 msgr2=0x7f351006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 -- 172.21.15.125:0/3675771647 shutdown_connections 2024-01-16T15:01:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.886+0000 7f3515400640 1 -- 172.21.15.125:0/3675771647 wait complete. 2024-01-16T15:01:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.887+0000 7f3515400640 1 Processor -- start 2024-01-16T15:01:39.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.887+0000 7f3515400640 1 -- start start 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35100b2bd0 0x7f3510149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f351015d300 0x7f35101f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35101f4120 con 0x7f3510150760 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100cdd70 con 0x7f35100b2bd0 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.888+0000 7f3515400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35100cd6f0 con 0x7f351015d300 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59272/0 (socket says 172.21.15.125:59272) 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 -- 172.21.15.125:0/4015213886 learned_addr learned my addr 172.21.15.125:0/4015213886 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350e800640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35100b2bd0 0x7f3510149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 -- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f351015d300 msgr2=0x7f35101f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f351015d300 0x7f35101f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 -- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35100b2bd0 msgr2=0x7f3510149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35100b2bd0 0x7f3510149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 -- 172.21.15.125:0/4015213886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f800b6a0 con 0x7f3510150760 2024-01-16T15:01:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.889+0000 7f350de00640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f34f800bb60 tx=0x7f34f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.890+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8006d10 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.890+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8042430 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.890+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80427c0 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.890+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101f3b60 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.890+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35100cd990 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.891+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35100cdeb0 con 0x7f3510150760 2024-01-16T15:01:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.892+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f34f8042920 con 0x7f3510150760 2024-01-16T15:01:39.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.893+0000 7f3506a00640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 0x7f34e806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:39.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.893+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f34f80acf20 con 0x7f3510150760 2024-01-16T15:01:39.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.893+0000 7f350e800640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 0x7f34e806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:39.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.894+0000 7f350e800640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 0x7f34e806dc90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3500009d80 tx=0x7f350000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:39.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:39.897+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35100cdeb0 con 0x7f3510150760 2024-01-16T15:01:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.115+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone20151", "target": ["mon-mgr", ""]}) v1 -- 0x7f351015de30 con 0x7f34e806b7d0 2024-01-16T15:01:40.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.147+0000 7f3506a00640 1 -- 172.21.15.125:0/4015213886 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f351015de30 con 0x7f34e806b7d0 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 msgr2=0x7f34e806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 0x7f34e806dc90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3500009d80 tx=0x7f350000b040 comp rx=0 tx=0).stop 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 msgr2=0x7f35100ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f34f800bb60 tx=0x7f34f80046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 shutdown_connections 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f351015d300 0x7f35101f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f34e806b7d0 0x7f34e806dc90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3510150760 0x7f35100ac8d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 --2- 172.21.15.125:0/4015213886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35100b2bd0 0x7f3510149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.149+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 >> 172.21.15.125:0/4015213886 conn(0x7f351006a800 msgr2=0x7f35100cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:40.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.150+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 shutdown_connections 2024-01-16T15:01:40.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.150+0000 7f3515400640 1 -- 172.21.15.125:0/4015213886 wait complete. 2024-01-16T15:01:40.132 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.153+0000 7f00fe200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.132 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.153+0000 7f00fe200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.132 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.153+0000 7f00fe200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.132 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.153+0000 7f00fe200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.132 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.153+0000 7f00fe200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone20153 2024-01-16T15:01:40.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.173+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.173+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.173+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.173+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:01:40.173+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:40.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.346+0000 7f4def400640 1 Processor -- start 2024-01-16T15:01:40.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.346+0000 7f4def400640 1 -- start start 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4def400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4de80cc500 0x7f4de80cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4def400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 0x7f4de806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4def400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de8070e50 con 0x7f4de8143c40 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4def400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de80723c0 con 0x7f4de80cc500 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4def400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80cced0 con 0x7f4de806d160 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4dee000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4de8143c40 0x7f4de8071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41614/0 (socket says 172.21.15.125:41614) 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.347+0000 7f4dee000640 1 -- 172.21.15.125:0/428800776 learned_addr learned my addr 172.21.15.125:0/428800776 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:40.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4ded600640 1 --2- 172.21.15.125:0/428800776 >> v2:172.21.15.125:3300/0 conn(0x7f4de80cc500 0x7f4de80cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4decc00640 1 -- 172.21.15.125:0/428800776 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442797272 0 0) 0x7f4de8070e50 con 0x7f4de8143c40 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4deea00640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 0x7f4de806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4decc00640 1 -- 172.21.15.125:0/428800776 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc8003520 con 0x7f4de8143c40 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4ded600640 1 -- 172.21.15.125:0/428800776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 msgr2=0x7f4de806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4ded600640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 0x7f4de806d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4ded600640 1 -- 172.21.15.125:0/428800776 >> v1:172.21.15.125:6789/0 conn(0x7f4de8143c40 legacy=0x7f4de8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4ded600640 1 -- 172.21.15.125:0/428800776 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de8144020 con 0x7f4de80cc500 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.348+0000 7f4deea00640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 0x7f4de806d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.349+0000 7f4ded600640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de80cc500 0x7f4de80cc990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00097a0 tx=0x7f4dd002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8f6c147bc77f30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.349+0000 7f4decc00640 1 -- 172.21.15.125:0/428800776 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0031780 con 0x7f4de80cc500 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.349+0000 7f4decc00640 1 -- 172.21.15.125:0/428800776 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd00318e0 con 0x7f4de80cc500 2024-01-16T15:01:40.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.349+0000 7f4decc00640 1 -- 172.21.15.125:0/428800776 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0031c70 con 0x7f4de80cc500 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 -- 172.21.15.125:0/428800776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de80cc500 msgr2=0x7f4de80cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de80cc500 0x7f4de80cc990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00097a0 tx=0x7f4dd002b820 comp rx=0 tx=0).stop 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 -- 172.21.15.125:0/428800776 shutdown_connections 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de806d160 0x7f4de806d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 --2- 172.21.15.125:0/428800776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de80cc500 0x7f4de80cc990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 -- 172.21.15.125:0/428800776 >> 172.21.15.125:0/428800776 conn(0x7f4de806a800 msgr2=0x7f4de806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:40.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 -- 172.21.15.125:0/428800776 shutdown_connections 2024-01-16T15:01:40.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.350+0000 7f4def400640 1 -- 172.21.15.125:0/428800776 wait complete. 2024-01-16T15:01:40.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.352+0000 7f4def400640 1 Processor -- start 2024-01-16T15:01:40.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.352+0000 7f4def400640 1 -- start start 2024-01-16T15:01:40.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de806d160 0x7f4de813c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de80cc500 0x7f4de80aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80ac020 con 0x7f4de8143c40 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80ab410 con 0x7f4de806d160 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4def400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80ab6f0 con 0x7f4de80cc500 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59314/0 (socket says 172.21.15.125:59314) 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 -- 172.21.15.125:0/1641063882 learned_addr learned my addr 172.21.15.125:0/1641063882 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 -- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de80cc500 msgr2=0x7f4de80aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de80cc500 0x7f4de80aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 -- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de806d160 msgr2=0x7f4de813c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de806d160 0x7f4de813c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.353+0000 7f4deea00640 1 -- 172.21.15.125:0/1641063882 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0009400 con 0x7f4de8143c40 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4deea00640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4de000cef0 tx=0x7f4de000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000d840 con 0x7f4de8143c40 2024-01-16T15:01:40.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de000d9c0 con 0x7f4de8143c40 2024-01-16T15:01:40.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de80ab9d0 con 0x7f4de8143c40 2024-01-16T15:01:40.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000dd70 con 0x7f4de8143c40 2024-01-16T15:01:40.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.354+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de813cac0 con 0x7f4de8143c40 2024-01-16T15:01:40.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.355+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de8067450 con 0x7f4de8143c40 2024-01-16T15:01:40.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.356+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f4de001e3b0 con 0x7f4de8143c40 2024-01-16T15:01:40.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.357+0000 7f4de6000640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 0x7f4dc006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.357+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4de008aeb0 con 0x7f4de8143c40 2024-01-16T15:01:40.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.357+0000 7f4dee000640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 0x7f4dc006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.358+0000 7f4dee000640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 0x7f4dc006dc90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4de813c270 tx=0x7f4dd8007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.361+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4de00557a0 con 0x7f4de8143c40 2024-01-16T15:01:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.561+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone20153", "target": ["mon-mgr", ""]}) v1 -- 0x7f4de806c960 con 0x7f4dc006b7d0 2024-01-16T15:01:40.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.591+0000 7f4de6000640 1 -- 172.21.15.125:0/1641063882 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4dd80058b0 con 0x7f4dc006b7d0 2024-01-16T15:01:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.593+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 msgr2=0x7f4dc006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.593+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 0x7f4dc006dc90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4de813c270 tx=0x7f4dd8007930 comp rx=0 tx=0).stop 2024-01-16T15:01:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 msgr2=0x7f4de815aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4de000cef0 tx=0x7f4de000a710 comp rx=0 tx=0).stop 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 shutdown_connections 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dc006b7d0 0x7f4dc006dc90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4de8143c40 0x7f4de815aa80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de80cc500 0x7f4de80aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 --2- 172.21.15.125:0/1641063882 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de806d160 0x7f4de813c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 >> 172.21.15.125:0/1641063882 conn(0x7f4de806a800 msgr2=0x7f4de80afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 shutdown_connections 2024-01-16T15:01:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.594+0000 7f4def400640 1 -- 172.21.15.125:0/1641063882 wait complete. 2024-01-16T15:01:40.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:01:40.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.794+0000 7fed9fc00640 1 Processor -- start 2024-01-16T15:01:40.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.794+0000 7fed9fc00640 1 -- start start 2024-01-16T15:01:40.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.795+0000 7fed9fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 0x7fed980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.795+0000 7fed9fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed980cf7d0 0x7fed9815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.795+0000 7fed9fc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed980cfc30 con 0x7fed9815e1e0 2024-01-16T15:01:40.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.795+0000 7fed9fc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed980b1ee0 con 0x7fed980cf7d0 2024-01-16T15:01:40.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.795+0000 7fed9fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed980b2780 con 0x7fed98143ae0 2024-01-16T15:01:40.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.796+0000 7fed8fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed980cf7d0 0x7fed9815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.796+0000 7fed8fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed980cf7d0 0x7fed9815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45322/0 (socket says 172.21.15.125:45322) 2024-01-16T15:01:40.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.796+0000 7fed8fc00640 1 -- 172.21.15.125:0/947943453 learned_addr learned my addr 172.21.15.125:0/947943453 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:40.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.796+0000 7fed9d200640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 0x7fed980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.797+0000 7fed8fc00640 1 -- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 msgr2=0x7fed980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.797+0000 7fed8fc00640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 0x7fed980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.797+0000 7fed8fc00640 1 -- 172.21.15.125:0/947943453 >> v1:172.21.15.137:6789/0 conn(0x7fed9815e1e0 legacy=0x7fed9815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.797+0000 7fed8fc00640 1 -- 172.21.15.125:0/947943453 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9815e990 con 0x7fed980cf7d0 2024-01-16T15:01:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.797+0000 7fed9d200640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 0x7fed980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.798+0000 7fed8fc00640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed9815dca0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fed80009e50 tx=0x7fed8002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27ab7d25b89d0954 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed97e00640 1 -- 172.21.15.125:0/947943453 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8002cb00 con 0x7fed980cf7d0 2024-01-16T15:01:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed97e00640 1 -- 172.21.15.125:0/947943453 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8002cc60 con 0x7fed980cf7d0 2024-01-16T15:01:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed97e00640 1 -- 172.21.15.125:0/947943453 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800343f0 con 0x7fed980cf7d0 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 -- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 msgr2=0x7fed9815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed9815dca0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fed80009e50 tx=0x7fed8002bf60 comp rx=0 tx=0).stop 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 -- 172.21.15.125:0/947943453 shutdown_connections 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed9815dca0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 --2- 172.21.15.125:0/947943453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed98143ae0 0x7fed980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 -- 172.21.15.125:0/947943453 >> 172.21.15.125:0/947943453 conn(0x7fed9806a800 msgr2=0x7fed9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 -- 172.21.15.125:0/947943453 shutdown_connections 2024-01-16T15:01:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.799+0000 7fed9fc00640 1 -- 172.21.15.125:0/947943453 wait complete. 2024-01-16T15:01:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.801+0000 7fed9fc00640 1 Processor -- start 2024-01-16T15:01:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.801+0000 7fed9fc00640 1 -- start start 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed98159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 0x7fed981419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed98159eb0 con 0x7fed98143ae0 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed980cc610 con 0x7fed980cf7d0 2024-01-16T15:01:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed9fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed980cc860 con 0x7fed9815e1e0 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed8fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed8fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59318/0 (socket says 172.21.15.125:59318) 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.802+0000 7fed8fc00640 1 -- 172.21.15.125:0/1488432530 learned_addr learned my addr 172.21.15.125:0/1488432530 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed9d200640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed98159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed9dc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 0x7fed981419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 -- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 msgr2=0x7fed981419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 0x7fed981419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 -- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 msgr2=0x7fed98159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed98159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 -- 172.21.15.125:0/1488432530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88009400 con 0x7fed98143ae0 2024-01-16T15:01:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed9dc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 0x7fed981419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed8fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed80036040 tx=0x7fed80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80009510 con 0x7fed98143ae0 2024-01-16T15:01:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8003e450 con 0x7fed98143ae0 2024-01-16T15:01:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.803+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8003e820 con 0x7fed98143ae0 2024-01-16T15:01:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.804+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80009b30 con 0x7fed98143ae0 2024-01-16T15:01:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.804+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed980cce20 con 0x7fed98143ae0 2024-01-16T15:01:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.805+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed980cf5f0 con 0x7fed98143ae0 2024-01-16T15:01:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.806+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fed8003e980 con 0x7fed98143ae0 2024-01-16T15:01:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.807+0000 7fed95600640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 0x7fed7006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.807+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed800ad0f0 con 0x7fed98143ae0 2024-01-16T15:01:40.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.807+0000 7fed9d200640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 0x7fed7006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:40.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.808+0000 7fed9d200640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 0x7fed7006dc90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed88009c60 tx=0x7fed88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:40.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:40.811+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed80077960 con 0x7fed98143ae0 2024-01-16T15:01:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:41.015+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed9815e640 con 0x7fed7006b7d0 2024-01-16T15:01:42.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:42.131+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fed80077d30 con 0x7fed98143ae0 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.704+0000 7fed95600640 1 -- 172.21.15.125:0/1488432530 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fed880098b0 con 0x7fed7006b7d0 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:01:43.685 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 msgr2=0x7fed7006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 0x7fed7006dc90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed88009c60 tx=0x7fed88008040 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 msgr2=0x7fed9813a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed80036040 tx=0x7fed80004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 shutdown_connections 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed9815e1e0 0x7fed981419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed7006b7d0 0x7fed7006dc90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed98143ae0 0x7fed9813a2c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 --2- 172.21.15.125:0/1488432530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed980cf7d0 0x7fed98159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 >> 172.21.15.125:0/1488432530 conn(0x7fed9806a800 msgr2=0x7fed9815ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 shutdown_connections 2024-01-16T15:01:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.708+0000 7fed9fc00640 1 -- 172.21.15.125:0/1488432530 wait complete. 2024-01-16T15:01:43.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:01:43.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.948+0000 7ff6d6400640 1 Processor -- start 2024-01-16T15:01:43.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.948+0000 7ff6d6400640 1 -- start start 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6d6400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6d013e360 0x7ff6d00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6d6400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d015c5f0 con 0x7ff6d015f9c0 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6d6400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d006c470 con 0x7ff6d013df80 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6d6400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d006c750 con 0x7ff6d013e360 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6cfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff6d015f9c0 0x7ff6d00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41616/0 (socket says 172.21.15.125:41616) 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.950+0000 7ff6cfe00640 1 -- 172.21.15.125:0/319652057 learned_addr learned my addr 172.21.15.125:0/319652057 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:43.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6c7e00640 1 --2- 172.21.15.125:0/319652057 >> v2:172.21.15.125:3300/0 conn(0x7ff6d013e360 0x7ff6d00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6cea00640 1 -- 172.21.15.125:0/319652057 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436876162 0 0) 0x7ff6d015c5f0 con 0x7ff6d015f9c0 2024-01-16T15:01:43.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6cea00640 1 -- 172.21.15.125:0/319652057 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b0003520 con 0x7ff6d015f9c0 2024-01-16T15:01:43.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6c7e00640 1 -- 172.21.15.125:0/319652057 >> v1:172.21.15.137:6789/0 conn(0x7ff6d013df80 legacy=0x7ff6d015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:43.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6c7e00640 1 -- 172.21.15.125:0/319652057 >> v1:172.21.15.125:6789/0 conn(0x7ff6d015f9c0 legacy=0x7ff6d00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.951+0000 7ff6c7e00640 1 -- 172.21.15.125:0/319652057 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d006ca30 con 0x7ff6d013e360 2024-01-16T15:01:43.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6c7e00640 1 --2- 172.21.15.125:0/319652057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d00cf290 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80097a0 tx=0x7ff6b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a9ca4b199b3161 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:43.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6cea00640 1 -- 172.21.15.125:0/319652057 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8030610 con 0x7ff6d013e360 2024-01-16T15:01:43.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6cea00640 1 -- 172.21.15.125:0/319652057 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b8030770 con 0x7ff6d013e360 2024-01-16T15:01:43.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6cea00640 1 -- 172.21.15.125:0/319652057 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8030b00 con 0x7ff6d013e360 2024-01-16T15:01:43.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6d6400640 1 -- 172.21.15.125:0/319652057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 msgr2=0x7ff6d00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.953+0000 7ff6d6400640 1 --2- 172.21.15.125:0/319652057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d00cf290 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80097a0 tx=0x7ff6b802b960 comp rx=0 tx=0).stop 2024-01-16T15:01:43.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.954+0000 7ff6d6400640 1 -- 172.21.15.125:0/319652057 shutdown_connections 2024-01-16T15:01:43.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.954+0000 7ff6d6400640 1 --2- 172.21.15.125:0/319652057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d00cf290 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.954+0000 7ff6d6400640 1 -- 172.21.15.125:0/319652057 >> 172.21.15.125:0/319652057 conn(0x7ff6d006a800 msgr2=0x7ff6d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:43.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.954+0000 7ff6d6400640 1 -- 172.21.15.125:0/319652057 shutdown_connections 2024-01-16T15:01:43.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.954+0000 7ff6d6400640 1 -- 172.21.15.125:0/319652057 wait complete. 2024-01-16T15:01:43.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.955+0000 7ff6d6400640 1 Processor -- start 2024-01-16T15:01:43.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.956+0000 7ff6d6400640 1 -- start start 2024-01-16T15:01:43.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.956+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6d013df80 0x7ff6d0153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.956+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.956+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6d015f9c0 0x7ff6d01e7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.956+0000 7ff6d6400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d01e83b0 con 0x7ff6d013e360 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6d6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0071bc0 con 0x7ff6d013df80 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6d6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d012a4c0 con 0x7ff6d015f9c0 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59334/0 (socket says 172.21.15.125:59334) 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 -- 172.21.15.125:0/1544775675 learned_addr learned my addr 172.21.15.125:0/1544775675 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 -- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6d015f9c0 msgr2=0x7ff6d01e7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6cf400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6d013df80 0x7ff6d0153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6cfe00640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6d015f9c0 0x7ff6d01e7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6d015f9c0 0x7ff6d01e7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 -- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6d013df80 msgr2=0x7ff6d0153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6d013df80 0x7ff6d0153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.957+0000 7ff6c7e00640 1 -- 172.21.15.125:0/1544775675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b8009400 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6c7e00640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff6b802bf70 tx=0x7ff6b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b803e040 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b8039cb0 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8044630 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d012a7a0 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.958+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d0153580 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.959+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d00675d0 con 0x7ff6d013e360 2024-01-16T15:01:43.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.960+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7ff6b8043390 con 0x7ff6d013e360 2024-01-16T15:01:43.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.961+0000 7ff6c7400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 0x7ff6a806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.961+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6b80aca00 con 0x7ff6d013e360 2024-01-16T15:01:43.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.961+0000 7ff6cf400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 0x7ff6a806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.962+0000 7ff6cf400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 0x7ff6a806ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000f60 tx=0x7ff6c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:43.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:43.965+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6b8076de0 con 0x7ff6d013e360 2024-01-16T15:01:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.174+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6d0155720 con 0x7ff6a806b8f0 2024-01-16T15:01:44.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.226+0000 7ff6c7400640 1 -- 172.21.15.125:0/1544775675 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6d0155720 con 0x7ff6a806b8f0 2024-01-16T15:01:44.206 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:44.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 msgr2=0x7ff6a806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 0x7ff6a806ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000f60 tx=0x7ff6c0009040 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 msgr2=0x7ff6d01e7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff6b802bf70 tx=0x7ff6b8004730 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 shutdown_connections 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6d015f9c0 0x7ff6d01e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff6a806b8f0 0x7ff6a806ddb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d013e360 0x7ff6d01e7930 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 --2- 172.21.15.125:0/1544775675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6d013df80 0x7ff6d0153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 >> 172.21.15.125:0/1544775675 conn(0x7ff6d006a800 msgr2=0x7ff6d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 shutdown_connections 2024-01-16T15:01:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.229+0000 7ff6d6400640 1 -- 172.21.15.125:0/1544775675 wait complete. 2024-01-16T15:01:44.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:01:44.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.444+0000 7f0594200640 1 Processor -- start 2024-01-16T15:01:44.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.444+0000 7f0594200640 1 -- start start 2024-01-16T15:01:44.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0594200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0594200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f058c0cf290 0x7f058c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0594200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c070c40 con 0x7f058c0b3ae0 2024-01-16T15:01:44.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0594200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c071450 con 0x7f058c0cf290 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0594200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0cfc60 con 0x7f058c0ae5a0 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0592200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f058c0cf290 0x7f058c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0590e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f058c0b3ae0 0x7f058c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41620/0 (socket says 172.21.15.125:41620) 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.446+0000 7f0590e00640 1 -- 172.21.15.125:0/1898328580 learned_addr learned my addr 172.21.15.125:0/1898328580 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f0591800640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f058be00640 1 -- 172.21.15.125:0/1898328580 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868110140 0 0) 0x7f058c070c40 con 0x7f058c0b3ae0 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f0592200640 1 -- 172.21.15.125:0/1898328580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 msgr2=0x7f058c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f0592200640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f0592200640 1 -- 172.21.15.125:0/1898328580 >> v1:172.21.15.125:6789/0 conn(0x7f058c0b3ae0 legacy=0x7f058c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.447+0000 7f0592200640 1 -- 172.21.15.125:0/1898328580 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c06c470 con 0x7f058c0cf290 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.448+0000 7f0592200640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c0cf720 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0584009d90 tx=0x7f058402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39e988a33ea0f09 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.448+0000 7f058be00640 1 -- 172.21.15.125:0/1898328580 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058402cce0 con 0x7f058c0cf290 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.448+0000 7f058be00640 1 -- 172.21.15.125:0/1898328580 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058402ce40 con 0x7f058c0cf290 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.448+0000 7f058be00640 1 -- 172.21.15.125:0/1898328580 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584034660 con 0x7f058c0cf290 2024-01-16T15:01:44.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 -- 172.21.15.125:0/1898328580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 msgr2=0x7f058c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c0cf720 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0584009d90 tx=0x7f058402bf30 comp rx=0 tx=0).stop 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 -- 172.21.15.125:0/1898328580 shutdown_connections 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c0cf720 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 --2- 172.21.15.125:0/1898328580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 -- 172.21.15.125:0/1898328580 >> 172.21.15.125:0/1898328580 conn(0x7f058c06a800 msgr2=0x7f058c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 -- 172.21.15.125:0/1898328580 shutdown_connections 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.449+0000 7f0594200640 1 -- 172.21.15.125:0/1898328580 wait complete. 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.450+0000 7f0594200640 1 Processor -- start 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 -- start start 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0b3ae0 0x7f058c160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c155eb0 con 0x7f058c0cf290 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c160a50 con 0x7f058c0b3ae0 2024-01-16T15:01:44.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.451+0000 7f0594200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0cda60 con 0x7f058c0ae5a0 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59346/0 (socket says 172.21.15.125:59346) 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 -- 172.21.15.125:0/3890658980 learned_addr learned my addr 172.21.15.125:0/3890658980 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 -- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 msgr2=0x7f058c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 -- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0b3ae0 msgr2=0x7f058c160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0590e00640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0b3ae0 0x7f058c160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0b3ae0 0x7f058c160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.452+0000 7f0592200640 1 -- 172.21.15.125:0/3890658980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0584009a70 con 0x7f058c0cf290 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0592200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0584009d90 tx=0x7f0584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058403e670 con 0x7f058c0cf290 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058403e7f0 con 0x7f058c0cf290 2024-01-16T15:01:44.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058403eba0 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c0cdd40 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.453+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c156150 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.454+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c156340 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.455+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f058403b040 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.456+0000 7f0589600640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 0x7f056406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.456+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05840ad230 con 0x7f058c0cf290 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.456+0000 7f0591800640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 0x7f056406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.457+0000 7f0591800640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 0x7f056406ddb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f057c0098f0 tx=0x7f057c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.460+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0584077720 con 0x7f058c0cf290 2024-01-16T15:01:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.659+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f058c071170 con 0x7f056406b8f0 2024-01-16T15:01:44.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.687+0000 7f0589600640 1 -- 172.21.15.125:0/3890658980 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f057c009e70 con 0x7f056406b8f0 2024-01-16T15:01:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 msgr2=0x7f056406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 0x7f056406ddb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f057c0098f0 tx=0x7f057c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 msgr2=0x7f058c15a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0584009d90 tx=0x7f0584004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 shutdown_connections 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f056406b8f0 0x7f056406ddb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f058c0cf290 0x7f058c15a820 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0b3ae0 0x7f058c160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 --2- 172.21.15.125:0/3890658980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0ae5a0 0x7f058c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 >> 172.21.15.125:0/3890658980 conn(0x7f058c06a800 msgr2=0x7f058c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 shutdown_connections 2024-01-16T15:01:44.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.690+0000 7f0594200640 1 -- 172.21.15.125:0/3890658980 wait complete. 2024-01-16T15:01:44.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone20154 2024-01-16T15:01:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.902+0000 7f1517c00640 1 Processor -- start 2024-01-16T15:01:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.903+0000 7f1517c00640 1 -- start start 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1517c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f15100b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1517c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151006cea0 con 0x7f151006c6b0 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1517c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151006cbc0 con 0x7f151009a170 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1517c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006f350 con 0x7f15100cd240 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1507e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f151006c6b0 0x7f1510099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41626/0 (socket says 172.21.15.125:41626) 2024-01-16T15:01:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.904+0000 7f1507e00640 1 -- 172.21.15.125:0/1466179181 learned_addr learned my addr 172.21.15.125:0/1466179181 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f1515200640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f15100b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678641761 0 0) 0x7f151006cea0 con 0x7f151006c6b0 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f0003520 con 0x7f151006c6b0 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396566841 0 0) 0x7f14f0003520 con 0x7f151006c6b0 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 msgr2=0x7f15100b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f15100b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 >> v1:172.21.15.137:6789/0 conn(0x7f151009a170 legacy=0x7f15100cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151006f630 con 0x7f151006c6b0 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f14f8001670 con 0x7f151006c6b0 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 >> v1:172.21.15.125:6789/0 conn(0x7f151006c6b0 legacy=0x7f1510099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.905+0000 7f150fe00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14f00039a0 0x7f14f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f150fe00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f14f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f150fe00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14f0004d00 0x7f14f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151006f350 con 0x7f14f0004d00 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f000ba30 con 0x7f14f00039a0 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f000bbf0 con 0x7f15100cd240 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14f0004d00 0x7f14f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1507e00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f14f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 -- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 msgr2=0x7f14f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15100cd240 0x7f14f00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 -- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14f00039a0 msgr2=0x7f14f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515200640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14f00039a0 0x7f14f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14f00039a0 0x7f14f0003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.906+0000 7f1515c00640 1 -- 172.21.15.125:0/1466179181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f0003520 con 0x7f14f0004d00 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.907+0000 7f1515c00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14f0004d00 0x7f14f0005180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1508001e20 tx=0x7f14f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442ceea9b31f0606 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508003b70 con 0x7f14f0004d00 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1508003cd0 con 0x7f14f0004d00 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f150fe00640 1 -- 172.21.15.125:0/1466179181 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508005310 con 0x7f14f0004d00 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 -- 172.21.15.125:0/1466179181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14f0004d00 msgr2=0x7f14f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 --2- 172.21.15.125:0/1466179181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14f0004d00 0x7f14f0005180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1508001e20 tx=0x7f14f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1515c00640 1 -- 172.21.15.125:0/1466179181 reap_dead start 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 -- 172.21.15.125:0/1466179181 shutdown_connections 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 -- 172.21.15.125:0/1466179181 >> 172.21.15.125:0/1466179181 conn(0x7f15100b2d00 msgr2=0x7f15100b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 -- 172.21.15.125:0/1466179181 shutdown_connections 2024-01-16T15:01:44.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.908+0000 7f1517c00640 1 -- 172.21.15.125:0/1466179181 wait complete. 2024-01-16T15:01:44.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.910+0000 7f1517c00640 1 Processor -- start 2024-01-16T15:01:44.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.910+0000 7f1517c00640 1 -- start start 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14f0004d00 0x7f151014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15100c82e0 0x7f15101519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15100c7ea0 con 0x7f15100c50f0 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151014fea0 con 0x7f15100c82e0 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1517c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15100c5660 con 0x7f14f0004d00 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1507e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1507e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59358/0 (socket says 172.21.15.125:59358) 2024-01-16T15:01:44.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.911+0000 7f1507e00640 1 -- 172.21.15.125:0/2504278069 learned_addr learned my addr 172.21.15.125:0/2504278069 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:44.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1515200640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14f0004d00 0x7f151014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1515c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15100c82e0 0x7f15101519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 -- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14f0004d00 msgr2=0x7f151014f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14f0004d00 0x7f151014f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 -- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15100c82e0 msgr2=0x7f15101519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15100c82e0 0x7f15101519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 -- 172.21.15.125:0/2504278069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100cb130 con 0x7f15100c50f0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1507e00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f14f800d840 tx=0x7f14f800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80074e0 con 0x7f15100c50f0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8007640 con 0x7f15100c50f0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8004a20 con 0x7f15100c50f0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.912+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15100cb410 con 0x7f15100c50f0 2024-01-16T15:01:44.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.913+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510151f30 con 0x7f15100c50f0 2024-01-16T15:01:44.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.914+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15100cb5a0 con 0x7f15100c50f0 2024-01-16T15:01:44.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.914+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f14f8004b80 con 0x7f15100c50f0 2024-01-16T15:01:44.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.915+0000 7f150d600640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 0x7f14e806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.915+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f14f808a290 con 0x7f15100c50f0 2024-01-16T15:01:44.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.916+0000 7f1515200640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 0x7f14e806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.916+0000 7f1515200640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 0x7f14e806dde0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1500009910 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:44.920+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14f80545a0 con 0x7f15100c50f0 2024-01-16T15:01:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.140+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone20154", "target": ["mon-mgr", ""]}) v1 -- 0x7f15100c8940 con 0x7f14e806b920 2024-01-16T15:01:45.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.164+0000 7f150d600640 1 -- 172.21.15.125:0/2504278069 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f15000118e0 con 0x7f14e806b920 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 msgr2=0x7f14e806dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 0x7f14e806dde0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1500009910 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 msgr2=0x7f151014f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f14f800d840 tx=0x7f14f800dd00 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 shutdown_connections 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f15100c82e0 0x7f15101519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f14e806b920 0x7f14e806dde0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15100c50f0 0x7f151014f960 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 --2- 172.21.15.125:0/2504278069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14f0004d00 0x7f151014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 >> 172.21.15.125:0/2504278069 conn(0x7f15100b2d00 msgr2=0x7f15100cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:45.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 shutdown_connections 2024-01-16T15:01:45.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.167+0000 7f1517c00640 1 -- 172.21.15.125:0/2504278069 wait complete. 2024-01-16T15:01:45.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone20152 2024-01-16T15:01:45.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.371+0000 7fa9ccc00640 1 Processor -- start 2024-01-16T15:01:45.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.371+0000 7fa9ccc00640 1 -- start start 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.372+0000 7fa9ccc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9c815e880 0x7fa9c806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.372+0000 7fa9ccc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa9c8070490 0x7fa9c8099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9ccc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8099fa0 0x7fa9c80cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9ccc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c809a3f0 con 0x7fa9c815e880 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9ccc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c80cfa70 con 0x7fa9c8070490 2024-01-16T15:01:45.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9ccc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c80cff00 con 0x7fa9c8099fa0 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9c815e880 0x7fa9c806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9c815e880 0x7fa9c806ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59374/0 (socket says 172.21.15.125:59374) 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 -- 172.21.15.125:0/501016170 learned_addr learned my addr 172.21.15.125:0/501016170 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 -- 172.21.15.125:0/501016170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8099fa0 msgr2=0x7fa9c80cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c6800640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8099fa0 0x7fa9c80cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9bfe00640 1 --2- 172.21.15.125:0/501016170 >> v2:172.21.15.137:3300/0 conn(0x7fa9c8070490 0x7fa9c8099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8099fa0 0x7fa9c80cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 -- 172.21.15.125:0/501016170 >> v2:172.21.15.137:3300/0 conn(0x7fa9c8070490 msgr2=0x7fa9c8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/501016170 >> v2:172.21.15.137:3300/0 conn(0x7fa9c8070490 0x7fa9c8099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.373+0000 7fa9c5e00640 1 -- 172.21.15.125:0/501016170 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c806fc70 con 0x7fa9c815e880 2024-01-16T15:01:45.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.375+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c806ff50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8071800 tx=0x7fa9b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b695ccf3ef30e302 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.375+0000 7fa9c5400640 1 -- 172.21.15.125:0/501016170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8030860 con 0x7fa9c815e880 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.375+0000 7fa9c5400640 1 -- 172.21.15.125:0/501016170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b80309c0 con 0x7fa9c815e880 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.375+0000 7fa9c5400640 1 -- 172.21.15.125:0/501016170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8030d50 con 0x7fa9c815e880 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 -- 172.21.15.125:0/501016170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 msgr2=0x7fa9c806ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c806ff50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8071800 tx=0x7fa9b802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 -- 172.21.15.125:0/501016170 shutdown_connections 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8099fa0 0x7fa9c80cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/501016170 >> v2:172.21.15.137:3300/0 conn(0x7fa9c8070490 0x7fa9c8099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/501016170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c806ff50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 -- 172.21.15.125:0/501016170 >> 172.21.15.125:0/501016170 conn(0x7fa9c80b2d00 msgr2=0x7fa9c80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 -- 172.21.15.125:0/501016170 shutdown_connections 2024-01-16T15:01:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.376+0000 7fa9ccc00640 1 -- 172.21.15.125:0/501016170 wait complete. 2024-01-16T15:01:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.378+0000 7fa9ccc00640 1 Processor -- start 2024-01-16T15:01:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.378+0000 7fa9ccc00640 1 -- start start 2024-01-16T15:01:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8070490 0x7fa9c8154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9c8099fa0 0x7fa9c80b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c81527e0 con 0x7fa9c815e880 2024-01-16T15:01:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8152220 con 0x7fa9c8099fa0 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.379+0000 7fa9ccc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8152500 con 0x7fa9c8070490 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59386/0 (socket says 172.21.15.125:59386) 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 -- 172.21.15.125:0/3139735697 learned_addr learned my addr 172.21.15.125:0/3139735697 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8070490 0x7fa9c8154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9bfe00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9c8099fa0 0x7fa9c80b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 -- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8070490 msgr2=0x7fa9c8154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8070490 0x7fa9c8154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 -- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9c8099fa0 msgr2=0x7fa9c80b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9c8099fa0 0x7fa9c80b5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 -- 172.21.15.125:0/3139735697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80094d0 con 0x7fa9c815e880 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.380+0000 7fa9c6800640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000cd70 tx=0x7fa9c000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.381+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0003b90 con 0x7fa9c815e880 2024-01-16T15:01:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.381+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c0003cf0 con 0x7fa9c815e880 2024-01-16T15:01:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.381+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c80c8410 con 0x7fa9c815e880 2024-01-16T15:01:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.381+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0004240 con 0x7fa9c815e880 2024-01-16T15:01:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.381+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c8151eb0 con 0x7fa9c815e880 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.382+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c80675d0 con 0x7fa9c815e880 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.383+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fa9c00043a0 con 0x7fa9c815e880 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.383+0000 7fa9be000640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 0x7fa9a006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.383+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9c0012070 con 0x7fa9c815e880 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.384+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 0x7fa9a006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.384+0000 7fa9c5e00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 0x7fa9a006ddb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8036000 tx=0x7fa9b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.387+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9c0055660 con 0x7fa9c815e880 2024-01-16T15:01:45.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.601+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone20152", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9c80cf700 con 0x7fa9a006b8f0 2024-01-16T15:01:45.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.631+0000 7fa9be000640 1 -- 172.21.15.125:0/3139735697 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa9b8044470 con 0x7fa9a006b8f0 2024-01-16T15:01:45.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 msgr2=0x7fa9a006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 0x7fa9a006ddb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8036000 tx=0x7fa9b8008030 comp rx=0 tx=0).stop 2024-01-16T15:01:45.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 msgr2=0x7fa9c80b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000cd70 tx=0x7fa9c000a660 comp rx=0 tx=0).stop 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 shutdown_connections 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa9a006b8f0 0x7fa9a006ddb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c815e880 0x7fa9c80b6430 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9c8099fa0 0x7fa9c80b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 --2- 172.21.15.125:0/3139735697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa9c8070490 0x7fa9c8154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.634+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 >> 172.21.15.125:0/3139735697 conn(0x7fa9c80b2d00 msgr2=0x7fa9c80cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.635+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 shutdown_connections 2024-01-16T15:01:45.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.635+0000 7fa9ccc00640 1 -- 172.21.15.125:0/3139735697 wait complete. 2024-01-16T15:01:45.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:45.634 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:01:45.683 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T15:01:45.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:01:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.899+0000 7f69a0e00640 1 Processor -- start 2024-01-16T15:01:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.900+0000 7f69a0e00640 1 -- start start 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c0cf9a0 0x7f699c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f69a0e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f699c070ea0 0x7f699c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f69a0e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c071690 con 0x7f699c14b2c0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f69a0e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c14b6a0 con 0x7f699c070ea0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f69a0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c071970 con 0x7f699c0cf9a0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f6999e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f699c14b2c0 0x7f699c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41642/0 (socket says 172.21.15.125:41642) 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f6999e00640 1 -- 172.21.15.125:0/4177795534 learned_addr learned my addr 172.21.15.125:0/4177795534 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.901+0000 7f699a800640 1 --2- 172.21.15.125:0/4177795534 >> v2:172.21.15.125:3300/0 conn(0x7f699c070ea0 0x7f699c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f6991200640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c0cf9a0 0x7f699c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f6999400640 1 -- 172.21.15.125:0/4177795534 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706513452 0 0) 0x7f699c071690 con 0x7f699c14b2c0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f6999400640 1 -- 172.21.15.125:0/4177795534 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f697c003520 con 0x7f699c14b2c0 2024-01-16T15:01:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f699a800640 1 -- 172.21.15.125:0/4177795534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c0cf9a0 msgr2=0x7f699c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f699a800640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c0cf9a0 0x7f699c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f699a800640 1 -- 172.21.15.125:0/4177795534 >> v1:172.21.15.125:6789/0 conn(0x7f699c14b2c0 legacy=0x7f699c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.902+0000 7f699a800640 1 -- 172.21.15.125:0/4177795534 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c071c50 con 0x7f699c070ea0 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.903+0000 7f699a800640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c070ea0 0x7f699c0cc160 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5222d2a39066a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f6999400640 1 -- 172.21.15.125:0/4177795534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699402cc50 con 0x7f699c070ea0 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f6999400640 1 -- 172.21.15.125:0/4177795534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699402cdb0 con 0x7f699c070ea0 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f6999400640 1 -- 172.21.15.125:0/4177795534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994034660 con 0x7f699c070ea0 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 -- 172.21.15.125:0/4177795534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c070ea0 msgr2=0x7f699c0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c070ea0 0x7f699c0cc160 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699402bea0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 -- 172.21.15.125:0/4177795534 shutdown_connections 2024-01-16T15:01:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c070ea0 0x7f699c0cc160 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 --2- 172.21.15.125:0/4177795534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c0cf9a0 0x7f699c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 -- 172.21.15.125:0/4177795534 >> 172.21.15.125:0/4177795534 conn(0x7f699c06a800 msgr2=0x7f699c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 -- 172.21.15.125:0/4177795534 shutdown_connections 2024-01-16T15:01:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.904+0000 7f69a0e00640 1 -- 172.21.15.125:0/4177795534 wait complete. 2024-01-16T15:01:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.906+0000 7f69a0e00640 1 Processor -- start 2024-01-16T15:01:45.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.906+0000 7f69a0e00640 1 -- start start 2024-01-16T15:01:45.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 0x7f699c1fb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 0x7f699c142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1623f0 con 0x7f699c0cf9a0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1520f0 con 0x7f699c070ea0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f69a0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c152390 con 0x7f699c14b2c0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f6991200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f6991200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59390/0 (socket says 172.21.15.125:59390) 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.907+0000 7f6991200640 1 -- 172.21.15.125:0/523564329 learned_addr learned my addr 172.21.15.125:0/523564329 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6999e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 0x7f699c1fb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f699a800640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 0x7f699c142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 -- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 msgr2=0x7f699c142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 0x7f699c142580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 -- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 msgr2=0x7f699c1fb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 0x7f699c1fb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 -- 172.21.15.125:0/523564329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009a70 con 0x7f699c0cf9a0 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f699a800640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 0x7f699c142580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:45.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6999e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 0x7f699c1fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6991200640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f698400d800 tx=0x7f698400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400b6c0 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698400b820 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c152670 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.908+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400bbb0 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.909+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c162620 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.910+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c0675d0 con 0x7f699c0cf9a0 2024-01-16T15:01:45.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.911+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f698400bd10 con 0x7f699c0cf9a0 2024-01-16T15:01:45.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.912+0000 7f6992a00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 0x7f697406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:45.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.912+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f698408ad40 con 0x7f699c0cf9a0 2024-01-16T15:01:45.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.912+0000 7f6999e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 0x7f697406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:45.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.913+0000 7f6999e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 0x7f697406ddb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f698c0011e0 tx=0x7f698c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:45.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:45.917+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6984055230 con 0x7f699c0cf9a0 2024-01-16T15:01:46.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:46.137+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f699c162840 con 0x7f697406b8f0 2024-01-16T15:01:46.479 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T15:01:46.499+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:01:46.592 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:01:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:48.540+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6984055600 con 0x7f699c0cf9a0 2024-01-16T15:01:49.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.536+0000 7f6992a00640 1 -- 172.21.15.125:0/523564329 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f699c162840 con 0x7f697406b8f0 2024-01-16T15:01:49.517 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.541+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 msgr2=0x7f697406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.541+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 0x7f697406ddb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f698c0011e0 tx=0x7f698c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.541+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 msgr2=0x7f699c161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.541+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f698400d800 tx=0x7f698400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 shutdown_connections 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f699c14b2c0 0x7f699c142580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f697406b8f0 0x7f697406ddb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f699c0cf9a0 0x7f699c161eb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 --2- 172.21.15.125:0/523564329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f699c070ea0 0x7f699c1fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.542+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 >> 172.21.15.125:0/523564329 conn(0x7f699c06a800 msgr2=0x7f699c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.543+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 shutdown_connections 2024-01-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:49.543+0000 7f69a0e00640 1 -- 172.21.15.125:0/523564329 wait complete. 2024-01-16T15:01:49.555 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:01:49.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:01:49.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:01:49.828 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:01:49.828 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:01:49.828 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:49.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:50.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.188+0000 7fedcf000640 1 Processor -- start 2024-01-16T15:01:50.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.189+0000 7fedcf000640 1 -- start start 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcf000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fedc813df80 0x7fedc815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcf000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc806c470 con 0x7fedc80cf9a0 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcf000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc806c750 con 0x7fedc813e360 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcf000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc806ca30 con 0x7fedc813df80 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcd000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fedc80cf9a0 0x7fedc80cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38602/0 (socket says 172.21.15.125:38602) 2024-01-16T15:01:50.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.190+0000 7fedcd000640 1 -- 172.21.15.125:0/4078719857 learned_addr learned my addr 172.21.15.125:0/4078719857 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:50.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7e00640 1 --2- 172.21.15.125:0/4078719857 >> v2:172.21.15.137:3300/0 conn(0x7fedc813df80 0x7fedc815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834440916 0 0) 0x7fedc806c470 con 0x7fedc80cf9a0 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda8003520 con 0x7fedc80cf9a0 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401458597 0 0) 0x7feda8003520 con 0x7fedc80cf9a0 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7e00640 1 -- 172.21.15.125:0/4078719857 >> v1:172.21.15.137:6789/0 conn(0x7fedc813e360 legacy=0x7fedc80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7e00640 1 -- 172.21.15.125:0/4078719857 >> v1:172.21.15.125:6789/0 conn(0x7fedc80cf9a0 legacy=0x7fedc80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.191+0000 7fedc7e00640 1 -- 172.21.15.125:0/4078719857 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc806cd10 con 0x7fedc813df80 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.193+0000 7fedc7e00640 1 --2- 172.21.15.125:0/4078719857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc813df80 0x7fedc815bee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fedb80097a0 tx=0x7fedb802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66ec8b644bb628b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.193+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb802f050 con 0x7fedc813df80 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.193+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb8031500 con 0x7fedc813df80 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.193+0000 7fedc7400640 1 -- 172.21.15.125:0/4078719857 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc00019f0 con 0x7fedc813df80 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 -- 172.21.15.125:0/4078719857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc813df80 msgr2=0x7fedc815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 --2- 172.21.15.125:0/4078719857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc813df80 0x7fedc815bee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fedb80097a0 tx=0x7fedb802b820 comp rx=0 tx=0).stop 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 -- 172.21.15.125:0/4078719857 shutdown_connections 2024-01-16T15:01:50.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 --2- 172.21.15.125:0/4078719857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc813df80 0x7fedc815bee0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 -- 172.21.15.125:0/4078719857 >> 172.21.15.125:0/4078719857 conn(0x7fedc806a800 msgr2=0x7fedc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 -- 172.21.15.125:0/4078719857 shutdown_connections 2024-01-16T15:01:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.194+0000 7fedcf000640 1 -- 172.21.15.125:0/4078719857 wait complete. 2024-01-16T15:01:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.195+0000 7fedcf000640 1 Processor -- start 2024-01-16T15:01:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 -- start start 2024-01-16T15:01:50.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc80cf9a0 0x7fedc81f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedc813e360 0x7fedc80b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8150c80 con 0x7fedc813df80 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8152da0 con 0x7fedc80cf9a0 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.196+0000 7fedcf000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8153080 con 0x7fedc813e360 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45592/0 (socket says 172.21.15.125:45592) 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 -- 172.21.15.125:0/3060054441 learned_addr learned my addr 172.21.15.125:0/3060054441 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedc7e00640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc80cf9a0 0x7fedc81f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 -- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedc813e360 msgr2=0x7fedc80b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedc813e360 0x7fedc80b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 -- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc80cf9a0 msgr2=0x7fedc81f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc80cf9a0 0x7fedc81f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 -- 172.21.15.125:0/3060054441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8009400 con 0x7fedc813df80 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.197+0000 7fedbf200640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c400 tx=0x7fedb000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.198+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb00052d0 con 0x7fedc813df80 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.198+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0005430 con 0x7fedc813df80 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.198+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0003830 con 0x7fedc813df80 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.198+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc8153360 con 0x7fedc813df80 2024-01-16T15:01:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.198+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc813da60 con 0x7fedc813df80 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.200+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc80ac8d0 con 0x7fedc813df80 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.200+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fedb0003990 con 0x7fedc813df80 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.201+0000 7fedc4c00640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 0x7feda006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.201+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fedb00898e0 con 0x7fedc813df80 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.201+0000 7fedc7e00640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 0x7feda006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.202+0000 7fedc7e00640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 0x7feda006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fedb8035000 tx=0x7fedb8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.205+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fedc80ac8d0 con 0x7fedc813df80 2024-01-16T15:01:50.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.385+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedc80ac8d0 con 0x7fedc813df80 2024-01-16T15:01:50.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.387+0000 7fedc4c00640 1 -- 172.21.15.125:0/3060054441 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13034 (secure 0 0 0) 0x7fedc80ac8d0 con 0x7fedc813df80 2024-01-16T15:01:50.367 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:50.367 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":50,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:01:49.523477+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/351760567":"2024-01-17T15:01:46.499505+0000","172.21.15.125:6835/351760567":"2024-01-17T15:01:46.499505+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.389+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 msgr2=0x7feda006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.389+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 0x7feda006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fedb8035000 tx=0x7fedb8008030 comp rx=0 tx=0).stop 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.389+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 msgr2=0x7fedc8150740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.389+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c400 tx=0x7fedb000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 shutdown_connections 2024-01-16T15:01:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedc813e360 0x7fedc80b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feda006b860 0x7feda006dd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc813df80 0x7fedc8150740 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 --2- 172.21.15.125:0/3060054441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedc80cf9a0 0x7fedc81f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 >> 172.21.15.125:0/3060054441 conn(0x7fedc806a800 msgr2=0x7fedc80cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 shutdown_connections 2024-01-16T15:01:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.390+0000 7fedcf000640 1 -- 172.21.15.125:0/3060054441 wait complete. 2024-01-16T15:01:50.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:01:50.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:01:50.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.606+0000 7f867dc00640 1 Processor -- start 2024-01-16T15:01:50.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.606+0000 7f867dc00640 1 -- start start 2024-01-16T15:01:50.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.607+0000 7f867dc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f867806a3e0 0x7f86780ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f867dc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8678071de0 con 0x7f86780d0b60 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f867dc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86780af270 con 0x7f86780aede0 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f867dc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8678072030 con 0x7f867806a3e0 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f8676a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f867806a3e0 0x7f86780ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f866de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f86780d0b60 0x7f867806a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38616/0 (socket says 172.21.15.125:38616) 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f866de00640 1 -- 172.21.15.125:0/4235064897 learned_addr learned my addr 172.21.15.125:0/4235064897 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f8676a00640 1 -- 172.21.15.125:0/4235064897 >> v1:172.21.15.137:6789/0 conn(0x7f86780aede0 legacy=0x7f867806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f8676a00640 1 -- 172.21.15.125:0/4235064897 >> v1:172.21.15.125:6789/0 conn(0x7f86780d0b60 legacy=0x7f867806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f8676a00640 1 -- 172.21.15.125:0/4235064897 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8678072310 con 0x7f867806a3e0 2024-01-16T15:01:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.608+0000 7f8676000640 1 -- 172.21.15.125:0/4235064897 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470908619 0 0) 0x7f8678071de0 con 0x7f86780d0b60 2024-01-16T15:01:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.609+0000 7f8676a00640 1 --2- 172.21.15.125:0/4235064897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867806a3e0 0x7f86780ac8d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8668009d90 tx=0x7f866802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708990f46cf2bfdb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f8676000640 1 -- 172.21.15.125:0/4235064897 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866802cc60 con 0x7f867806a3e0 2024-01-16T15:01:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f8676000640 1 -- 172.21.15.125:0/4235064897 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866802cdc0 con 0x7f867806a3e0 2024-01-16T15:01:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f8676000640 1 -- 172.21.15.125:0/4235064897 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668034660 con 0x7f867806a3e0 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 -- 172.21.15.125:0/4235064897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867806a3e0 msgr2=0x7f86780ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 --2- 172.21.15.125:0/4235064897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867806a3e0 0x7f86780ac8d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8668009d90 tx=0x7f866802beb0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 -- 172.21.15.125:0/4235064897 shutdown_connections 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 --2- 172.21.15.125:0/4235064897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867806a3e0 0x7f86780ac8d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 -- 172.21.15.125:0/4235064897 >> 172.21.15.125:0/4235064897 conn(0x7f86780cfef0 msgr2=0x7f86780d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 -- 172.21.15.125:0/4235064897 shutdown_connections 2024-01-16T15:01:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.610+0000 7f867dc00640 1 -- 172.21.15.125:0/4235064897 wait complete. 2024-01-16T15:01:50.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.612+0000 7f867dc00640 1 Processor -- start 2024-01-16T15:01:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.612+0000 7f867dc00640 1 -- start start 2024-01-16T15:01:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f867806a3e0 0x7f86780cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86780aede0 0x7f86781535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8678153ae0 con 0x7f86780d0b60 2024-01-16T15:01:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86781e2640 con 0x7f867806a3e0 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f867dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86780cd9f0 con 0x7f86780aede0 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f8677400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f8677400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45604/0 (socket says 172.21.15.125:45604) 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.613+0000 7f8677400640 1 -- 172.21.15.125:0/3101899319 learned_addr learned my addr 172.21.15.125:0/3101899319 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f866de00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86780aede0 0x7f86781535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8676a00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f867806a3e0 0x7f86780cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 -- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86780aede0 msgr2=0x7f86781535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86780aede0 0x7f86781535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 -- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f867806a3e0 msgr2=0x7f86780cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f867806a3e0 0x7f86780cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 -- 172.21.15.125:0/3101899319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8668009a70 con 0x7f86780d0b60 2024-01-16T15:01:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f8677400640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f867000c410 tx=0x7f867000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670011790 con 0x7f86780d0b60 2024-01-16T15:01:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86700118f0 con 0x7f86780d0b60 2024-01-16T15:01:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670011ce0 con 0x7f86780d0b60 2024-01-16T15:01:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.614+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86780cd750 con 0x7f86780d0b60 2024-01-16T15:01:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.615+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86781545b0 con 0x7f86780d0b60 2024-01-16T15:01:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.616+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86780675d0 con 0x7f86780d0b60 2024-01-16T15:01:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.616+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f867001c460 con 0x7f86780d0b60 2024-01-16T15:01:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.617+0000 7f866f400640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 0x7f865006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.617+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f867008a050 con 0x7f86780d0b60 2024-01-16T15:01:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.617+0000 7f8676a00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 0x7f865006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.618+0000 7f8676a00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 0x7f865006dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8668036000 tx=0x7f8668030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.622+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8670054910 con 0x7f86780d0b60 2024-01-16T15:01:50.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.796+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f867806e090 con 0x7f86780d0b60 2024-01-16T15:01:50.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.797+0000 7f866f400640 1 -- 172.21.15.125:0/3101899319 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8670057f30 con 0x7f86780d0b60 2024-01-16T15:01:50.776 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 msgr2=0x7f865006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 0x7f865006dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8668036000 tx=0x7f8668030000 comp rx=0 tx=0).stop 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 msgr2=0x7f86780cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f867000c410 tx=0x7f867000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 shutdown_connections 2024-01-16T15:01:50.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f865006b860 0x7f865006dd20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86780d0b60 0x7f86780cd210 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86780aede0 0x7f86781535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 --2- 172.21.15.125:0/3101899319 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f867806a3e0 0x7f86780cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.799+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 >> 172.21.15.125:0/3101899319 conn(0x7f86780cfef0 msgr2=0x7f86780ad9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.800+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 shutdown_connections 2024-01-16T15:01:50.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:50.800+0000 7f867dc00640 1 -- 172.21.15.125:0/3101899319 wait complete. 2024-01-16T15:01:50.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:01:51.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.023+0000 7f04e0a00640 1 Processor -- start 2024-01-16T15:01:51.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.023+0000 7f04e0a00640 1 -- start start 2024-01-16T15:01:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04e0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04dc143c40 0x7f04dc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04e0a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f04dc0717d0 0x7f04dc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04e0a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc070e50 con 0x7f04dc0cca40 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04e0a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0723c0 con 0x7f04dc143c40 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04e0a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0cce40 con 0x7f04dc0717d0 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04db200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f04dc0cca40 0x7f04dc15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38630/0 (socket says 172.21.15.125:38630) 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.025+0000 7f04db200640 1 -- 172.21.15.125:0/2466338532 learned_addr learned my addr 172.21.15.125:0/2466338532 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04da800640 1 --2- 172.21.15.125:0/2466338532 >> v2:172.21.15.125:3300/0 conn(0x7f04dc143c40 0x7f04dc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04d1c00640 1 --2- 172.21.15.125:0/2466338532 >> v2:172.21.15.137:3300/0 conn(0x7f04dc0717d0 0x7f04dc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04d9e00640 1 -- 172.21.15.125:0/2466338532 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367513470 0 0) 0x7f04dc070e50 con 0x7f04dc0cca40 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04da800640 1 -- 172.21.15.125:0/2466338532 >> v2:172.21.15.137:3300/0 conn(0x7f04dc0717d0 msgr2=0x7f04dc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04da800640 1 --2- 172.21.15.125:0/2466338532 >> v2:172.21.15.137:3300/0 conn(0x7f04dc0717d0 0x7f04dc0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04da800640 1 -- 172.21.15.125:0/2466338532 >> v1:172.21.15.125:6789/0 conn(0x7f04dc0cca40 legacy=0x7f04dc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.026+0000 7f04da800640 1 -- 172.21.15.125:0/2466338532 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc06e170 con 0x7f04dc143c40 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.027+0000 7f04da800640 1 --2- 172.21.15.125:0/2466338532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc143c40 0x7f04dc071290 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0b44a0 tx=0x7f04cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a28bbf24807f9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.027+0000 7f04d9e00640 1 -- 172.21.15.125:0/2466338532 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc02e070 con 0x7f04dc143c40 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.027+0000 7f04d9e00640 1 -- 172.21.15.125:0/2466338532 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc0315a0 con 0x7f04dc143c40 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.027+0000 7f04d9e00640 1 -- 172.21.15.125:0/2466338532 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc031930 con 0x7f04dc143c40 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 -- 172.21.15.125:0/2466338532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc143c40 msgr2=0x7f04dc071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 --2- 172.21.15.125:0/2466338532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc143c40 0x7f04dc071290 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0b44a0 tx=0x7f04cc02b840 comp rx=0 tx=0).stop 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 -- 172.21.15.125:0/2466338532 shutdown_connections 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 --2- 172.21.15.125:0/2466338532 >> v2:172.21.15.137:3300/0 conn(0x7f04dc0717d0 0x7f04dc0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 --2- 172.21.15.125:0/2466338532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc143c40 0x7f04dc071290 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 -- 172.21.15.125:0/2466338532 >> 172.21.15.125:0/2466338532 conn(0x7f04dc06a800 msgr2=0x7f04dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 -- 172.21.15.125:0/2466338532 shutdown_connections 2024-01-16T15:01:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.028+0000 7f04e0a00640 1 -- 172.21.15.125:0/2466338532 wait complete. 2024-01-16T15:01:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.030+0000 7f04e0a00640 1 Processor -- start 2024-01-16T15:01:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.030+0000 7f04e0a00640 1 -- start start 2024-01-16T15:01:51.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04dc0cca40 0x7f04dc0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04dc143c40 0x7f04dc0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc13b9d0 con 0x7f04dc0717d0 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc13bcb0 con 0x7f04dc0cca40 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04e0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc13bf90 con 0x7f04dc143c40 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45618/0 (socket says 172.21.15.125:45618) 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 -- 172.21.15.125:0/1294233911 learned_addr learned my addr 172.21.15.125:0/1294233911 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 -- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04dc143c40 msgr2=0x7f04dc0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04dc143c40 0x7f04dc0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 -- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04dc0cca40 msgr2=0x7f04dc0b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04dc0cca40 0x7f04dc0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.031+0000 7f04da800640 1 -- 172.21.15.125:0/1294233911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc009400 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04da800640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009890 tx=0x7f04cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc032840 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc0329c0 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc032d70 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc13c270 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.032+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc0b3ce0 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.033+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc0b3ed0 con 0x7f04dc0717d0 2024-01-16T15:01:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.034+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f04cc03b3f0 con 0x7f04dc0717d0 2024-01-16T15:01:51.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.035+0000 7f04d3400640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 0x7f04b406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.035+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f04cc0ac3a0 con 0x7f04dc0717d0 2024-01-16T15:01:51.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.035+0000 7f04d1c00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 0x7f04b406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.036+0000 7f04d1c00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 0x7f04b406dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0cd630 tx=0x7f04c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.039+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04dc0b3ed0 con 0x7f04dc0717d0 2024-01-16T15:01:51.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.245+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f04dc13c790 con 0x7f04dc0717d0 2024-01-16T15:01:51.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.245+0000 7f04d3400640 1 -- 172.21.15.125:0/1294233911 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v50) v1 ==== 80+0+122 (secure 0 0 0) 0x7f04dc13c790 con 0x7f04dc0717d0 2024-01-16T15:01:51.225 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/351760567 2024-01-17T15:01:46.499505+0000 2024-01-16T15:01:51.225 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/351760567 2024-01-17T15:01:46.499505+0000 2024-01-16T15:01:51.226 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:01:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 msgr2=0x7f04b406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 0x7f04b406dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0cd630 tx=0x7f04c4008040 comp rx=0 tx=0).stop 2024-01-16T15:01:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 msgr2=0x7f04dc138450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009890 tx=0x7f04cc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 shutdown_connections 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04dc143c40 0x7f04dc0b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04dc0cca40 0x7f04dc0b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f04b406b860 0x7f04b406dd20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 --2- 172.21.15.125:0/1294233911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04dc0717d0 0x7f04dc138450 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 >> 172.21.15.125:0/1294233911 conn(0x7f04dc06a800 msgr2=0x7f04dc0aeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 shutdown_connections 2024-01-16T15:01:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.248+0000 7f04e0a00640 1 -- 172.21.15.125:0/1294233911 wait complete. 2024-01-16T15:01:51.246 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:01:51.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:51.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:51.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[53863]: fuse finished with error 0 and tester_r 0 2024-01-16T15:01:51.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:01:51.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:51.338 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:01:51.580 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:01:51.580 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:51.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:51.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.868+0000 7f7db3400640 1 Processor -- start 2024-01-16T15:01:51.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.869+0000 7f7db3400640 1 -- start start 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.870+0000 7f7db3400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7dac0cf9a0 0x7f7dac15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.870+0000 7f7db3400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac070c40 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.870+0000 7f7db3400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac06c470 con 0x7f7dac0b3ae0 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.870+0000 7f7db3400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dac071450 con 0x7f7dac0cf9a0 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.870+0000 7f7db0a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7dac0ae5a0 0x7f7dac06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38644/0 (socket says 172.21.15.125:38644) 2024-01-16T15:01:51.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7db0a00640 1 -- 172.21.15.125:0/2299619528 learned_addr learned my addr 172.21.15.125:0/2299619528 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7db1400640 1 --2- 172.21.15.125:0/2299619528 >> v2:172.21.15.137:3300/0 conn(0x7f7dac0cf9a0 0x7f7dac15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123998269 0 0) 0x7f7dac070c40 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c003520 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441301274 0 0) 0x7f7d8c003520 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 >> v2:172.21.15.137:3300/0 conn(0x7f7dac0cf9a0 msgr2=0x7f7dac15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 --2- 172.21.15.125:0/2299619528 >> v2:172.21.15.137:3300/0 conn(0x7f7dac0cf9a0 0x7f7dac15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 >> v1:172.21.15.137:6789/0 conn(0x7f7dac0b3ae0 legacy=0x7f7dac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac06c750 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7d9c001670 con 0x7f7dac0ae5a0 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.871+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 >> v1:172.21.15.125:6789/0 conn(0x7f7dac0ae5a0 legacy=0x7f7dac06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7d8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d8c0047c0 0x7f7d8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d8c005180 0x7f7d8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac071450 con 0x7f7d8c003970 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c00c030 con 0x7f7d8c0047c0 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c00c1f0 con 0x7f7d8c005180 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7d8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7da3c00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d8c0047c0 0x7f7d8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 -- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d8c005180 msgr2=0x7f7d8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d8c005180 0x7f7d8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 -- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d8c0047c0 msgr2=0x7f7d8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d8c0047c0 0x7f7d8c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db0a00640 1 -- 172.21.15.125:0/2299619528 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c003520 con 0x7f7d8c003970 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.872+0000 7f7db1400640 1 -- 172.21.15.125:0/2299619528 reap_dead start 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.873+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7d8c003d90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00c190 tx=0x7f7d8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c5cf51d046f4c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c005ad0 con 0x7f7d8c003970 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c005c30 con 0x7f7d8c003970 2024-01-16T15:01:51.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7dabe00640 1 -- 172.21.15.125:0/2299619528 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c02cab0 con 0x7f7d8c003970 2024-01-16T15:01:51.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 -- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 msgr2=0x7f7d8c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7d8c003d90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00c190 tx=0x7f7d8c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 -- 172.21.15.125:0/2299619528 shutdown_connections 2024-01-16T15:01:51.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 --2- 172.21.15.125:0/2299619528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7d8c003d90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 -- 172.21.15.125:0/2299619528 >> 172.21.15.125:0/2299619528 conn(0x7f7dac06a800 msgr2=0x7f7dac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:51.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 -- 172.21.15.125:0/2299619528 shutdown_connections 2024-01-16T15:01:51.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.874+0000 7f7db3400640 1 -- 172.21.15.125:0/2299619528 wait complete. 2024-01-16T15:01:51.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.876+0000 7f7db3400640 1 Processor -- start 2024-01-16T15:01:51.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.876+0000 7f7db3400640 1 -- start start 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac15eea0 0x7f7dac1592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac15f320 0x7f7dac159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac15f8f0 con 0x7f7d8c003970 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac15fbd0 con 0x7f7dac15eea0 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.877+0000 7f7db3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac15feb0 con 0x7f7dac15f320 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45640/0 (socket says 172.21.15.125:45640) 2024-01-16T15:01:51.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 -- 172.21.15.125:0/2088729462 learned_addr learned my addr 172.21.15.125:0/2088729462 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 -- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac15f320 msgr2=0x7f7dac159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7da3c00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac15eea0 0x7f7dac1592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db1400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac15f320 0x7f7dac159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac15f320 0x7f7dac159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 -- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac15eea0 msgr2=0x7f7dac1592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac15eea0 0x7f7dac1592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 -- 172.21.15.125:0/2088729462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac160190 con 0x7f7d8c003970 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.878+0000 7f7db0a00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c02c410 tx=0x7f7d9c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.879+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c03c040 con 0x7f7d8c003970 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.879+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c02cd00 con 0x7f7d8c003970 2024-01-16T15:01:51.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.879+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac160470 con 0x7f7d8c003970 2024-01-16T15:01:51.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.879+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c007290 con 0x7f7d8c003970 2024-01-16T15:01:51.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.879+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac14fe10 con 0x7f7d8c003970 2024-01-16T15:01:51.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.880+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac0675d0 con 0x7f7d8c003970 2024-01-16T15:01:51.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.881+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7d9c032070 con 0x7f7d8c003970 2024-01-16T15:01:51.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.882+0000 7f7da9600640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 0x7f7d8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.882+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7d9c0ab540 con 0x7f7d8c003970 2024-01-16T15:01:51.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.882+0000 7f7da3c00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 0x7f7d8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.883+0000 7f7da3c00640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 0x7f7d8406dd20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7d94000c00 tx=0x7f7d94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:51.887+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d9c075d70 con 0x7f7d8c003970 2024-01-16T15:01:52.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.079+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dac15e400 con 0x7f7d8c003970 2024-01-16T15:01:52.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.080+0000 7f7da9600640 1 -- 172.21.15.125:0/2088729462 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13034 (secure 0 0 0) 0x7f7d9c045090 con 0x7f7d8c003970 2024-01-16T15:01:52.060 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:52.060 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":50,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:01:49.523477+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/351760567":"2024-01-17T15:01:46.499505+0000","172.21.15.125:6835/351760567":"2024-01-17T15:01:46.499505+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 msgr2=0x7f7d8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 0x7f7d8406dd20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7d94000c00 tx=0x7f7d94009040 comp rx=0 tx=0).stop 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 msgr2=0x7f7dac14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c02c410 tx=0x7f7d9c02e070 comp rx=0 tx=0).stop 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 shutdown_connections 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac15f320 0x7f7dac159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac15eea0 0x7f7dac1592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d8406b860 0x7f7d8406dd20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 --2- 172.21.15.125:0/2088729462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d8c003970 0x7f7dac14f8d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 >> 172.21.15.125:0/2088729462 conn(0x7f7dac06a800 msgr2=0x7f7dac15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:52.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.083+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 shutdown_connections 2024-01-16T15:01:52.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.084+0000 7f7db3400640 1 -- 172.21.15.125:0/2088729462 wait complete. 2024-01-16T15:01:52.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:01:52.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:01:52.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.316+0000 7f6b1bc00640 1 Processor -- start 2024-01-16T15:01:52.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.316+0000 7f6b1bc00640 1 -- start start 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b1bc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6b1413df80 0x7f6b1415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b1bc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b140cf9a0 0x7f6b140cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b1bc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1406c470 con 0x7f6b1413e360 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b1bc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1406c750 con 0x7f6b140cf9a0 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b1bc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1406ca30 con 0x7f6b1413df80 2024-01-16T15:01:52.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b19c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b140cf9a0 0x7f6b140cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b13e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6b1413e360 0x7f6b140cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38652/0 (socket says 172.21.15.125:38652) 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.318+0000 7f6b13e00640 1 -- 172.21.15.125:0/142687824 learned_addr learned my addr 172.21.15.125:0/142687824 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.319+0000 7f6b19200640 1 --2- 172.21.15.125:0/142687824 >> v2:172.21.15.137:3300/0 conn(0x7f6b1413df80 0x7f6b1415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.319+0000 7f6b19c00640 1 -- 172.21.15.125:0/142687824 >> v2:172.21.15.137:3300/0 conn(0x7f6b1413df80 msgr2=0x7f6b1415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.319+0000 7f6b19c00640 1 --2- 172.21.15.125:0/142687824 >> v2:172.21.15.137:3300/0 conn(0x7f6b1413df80 0x7f6b1415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.319+0000 7f6b19c00640 1 -- 172.21.15.125:0/142687824 >> v1:172.21.15.125:6789/0 conn(0x7f6b1413e360 legacy=0x7f6b140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.319+0000 7f6b19c00640 1 -- 172.21.15.125:0/142687824 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1406cd10 con 0x7f6b140cf9a0 2024-01-16T15:01:52.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.320+0000 7f6b19c00640 1 --2- 172.21.15.125:0/142687824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b140cd840 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6b14072770 tx=0x7f6b0c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7c524d700ec6ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.320+0000 7f6b13400640 1 -- 172.21.15.125:0/142687824 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c02cc70 con 0x7f6b140cf9a0 2024-01-16T15:01:52.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.320+0000 7f6b13400640 1 -- 172.21.15.125:0/142687824 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c02cdd0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.320+0000 7f6b13400640 1 -- 172.21.15.125:0/142687824 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c034660 con 0x7f6b140cf9a0 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 -- 172.21.15.125:0/142687824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 msgr2=0x7f6b140cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/142687824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b140cd840 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6b14072770 tx=0x7f6b0c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 -- 172.21.15.125:0/142687824 shutdown_connections 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/142687824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b140cd840 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/142687824 >> v2:172.21.15.137:3300/0 conn(0x7f6b1413df80 0x7f6b1415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 -- 172.21.15.125:0/142687824 >> 172.21.15.125:0/142687824 conn(0x7f6b1406a800 msgr2=0x7f6b1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 -- 172.21.15.125:0/142687824 shutdown_connections 2024-01-16T15:01:52.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.321+0000 7f6b1bc00640 1 -- 172.21.15.125:0/142687824 wait complete. 2024-01-16T15:01:52.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.323+0000 7f6b1bc00640 1 Processor -- start 2024-01-16T15:01:52.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.323+0000 7f6b1bc00640 1 -- start start 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b1413df80 0x7f6b141501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 0x7f6b140711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1413dda0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1414f160 con 0x7f6b1413e360 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b1bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1414f400 con 0x7f6b1413df80 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b19200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b19200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45658/0 (socket says 172.21.15.125:45658) 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b19200640 1 -- 172.21.15.125:0/3636111298 learned_addr learned my addr 172.21.15.125:0/3636111298 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b19c00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 0x7f6b140711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b19200640 1 -- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b1413df80 msgr2=0x7f6b141501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.324+0000 7f6b13e00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b1413df80 0x7f6b141501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19200640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b1413df80 0x7f6b141501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19200640 1 -- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 msgr2=0x7f6b140711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19200640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 0x7f6b140711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19200640 1 -- 172.21.15.125:0/3636111298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c009a70 con 0x7f6b140cf9a0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19c00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 0x7f6b140711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b19200640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400d800 tx=0x7f6b0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400b6c0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0400b820 con 0x7f6b140cf9a0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400bbb0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1414f6e0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.325+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1414fca0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.327+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b140675d0 con 0x7f6b140cf9a0 2024-01-16T15:01:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.327+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6b0400bd10 con 0x7f6b140cf9a0 2024-01-16T15:01:52.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.328+0000 7f6b10c00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 0x7f6aec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.328+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6b0408a970 con 0x7f6b140cf9a0 2024-01-16T15:01:52.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.328+0000 7f6b13e00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 0x7f6aec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.329+0000 7f6b13e00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 0x7f6aec06dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6b1415f1c0 tx=0x7f6afc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.333+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b04055230 con 0x7f6b140cf9a0 2024-01-16T15:01:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.523+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b1406df00 con 0x7f6b140cf9a0 2024-01-16T15:01:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.524+0000 7f6b10c00640 1 -- 172.21.15.125:0/3636111298 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b04058850 con 0x7f6b140cf9a0 2024-01-16T15:01:52.504 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 msgr2=0x7f6aec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 0x7f6aec06dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6b1415f1c0 tx=0x7f6afc009040 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 msgr2=0x7f6b1413d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400d800 tx=0x7f6b0400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 shutdown_connections 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b1413e360 0x7f6b140711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6b1413df80 0x7f6b141501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6aec06b860 0x7f6aec06dd20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 --2- 172.21.15.125:0/3636111298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b140cf9a0 0x7f6b1413d860 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 >> 172.21.15.125:0/3636111298 conn(0x7f6b1406a800 msgr2=0x7f6b1415d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 shutdown_connections 2024-01-16T15:01:52.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.527+0000 7f6b1bc00640 1 -- 172.21.15.125:0/3636111298 wait complete. 2024-01-16T15:01:52.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:01:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.758+0000 7f5ca5c00640 1 Processor -- start 2024-01-16T15:01:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.759+0000 7f5ca5c00640 1 -- start start 2024-01-16T15:01:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ca013df80 0x7f5ca015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5ca013e360 0x7f5ca00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca006c470 con 0x7f5ca013df80 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca006c750 con 0x7f5ca013e360 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.760+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca006ca30 con 0x7f5ca015f9c0 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ca013df80 0x7f5ca015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ca013df80 0x7f5ca015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45670/0 (socket says 172.21.15.125:45670) 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 -- 172.21.15.125:0/26911563 learned_addr learned my addr 172.21.15.125:0/26911563 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c97e00640 1 --2- 172.21.15.125:0/26911563 >> v2:172.21.15.137:3300/0 conn(0x7f5ca013e360 0x7f5ca00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 -- 172.21.15.125:0/26911563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 msgr2=0x7f5ca00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9f400640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 -- 172.21.15.125:0/26911563 >> v2:172.21.15.137:3300/0 conn(0x7f5ca013e360 msgr2=0x7f5ca00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:52.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/26911563 >> v2:172.21.15.137:3300/0 conn(0x7f5ca013e360 0x7f5ca00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.761+0000 7f5c9ea00640 1 -- 172.21.15.125:0/26911563 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca006cd10 con 0x7f5ca013df80 2024-01-16T15:01:52.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.762+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca015bee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5ca006ded0 tx=0x7f5c9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb7be63538d6627 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5c9e000640 1 -- 172.21.15.125:0/26911563 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90031870 con 0x7f5ca013df80 2024-01-16T15:01:52.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5c9e000640 1 -- 172.21.15.125:0/26911563 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c900319d0 con 0x7f5ca013df80 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5c9e000640 1 -- 172.21.15.125:0/26911563 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90031d60 con 0x7f5ca013df80 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 -- 172.21.15.125:0/26911563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 msgr2=0x7f5ca015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca015bee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5ca006ded0 tx=0x7f5c9002b910 comp rx=0 tx=0).stop 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 -- 172.21.15.125:0/26911563 shutdown_connections 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/26911563 >> v2:172.21.15.137:3300/0 conn(0x7f5ca013e360 0x7f5ca00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/26911563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca015bee0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 -- 172.21.15.125:0/26911563 >> 172.21.15.125:0/26911563 conn(0x7f5ca006a800 msgr2=0x7f5ca006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.763+0000 7f5ca5c00640 1 -- 172.21.15.125:0/26911563 shutdown_connections 2024-01-16T15:01:52.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.764+0000 7f5ca5c00640 1 -- 172.21.15.125:0/26911563 wait complete. 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.766+0000 7f5ca5c00640 1 Processor -- start 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.766+0000 7f5ca5c00640 1 -- start start 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ca013e360 0x7f5ca0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca013b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca00a8ae0 con 0x7f5ca013df80 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca013bda0 con 0x7f5ca013e360 2024-01-16T15:01:52.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca013bfd0 con 0x7f5ca015f9c0 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45684/0 (socket says 172.21.15.125:45684) 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.767+0000 7f5c9ea00640 1 -- 172.21.15.125:0/1546566809 learned_addr learned my addr 172.21.15.125:0/1546566809 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c97e00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ca013e360 0x7f5ca0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9f400640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca013b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 -- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 msgr2=0x7f5ca013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca013b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 -- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ca013e360 msgr2=0x7f5ca0154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ca013e360 0x7f5ca0154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 -- 172.21.15.125:0/1546566809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c900094d0 con 0x7f5ca013df80 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c9ea00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002b950 tx=0x7f5c900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90031860 con 0x7f5ca013df80 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c900443f0 con 0x7f5ca013df80 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c900447a0 con 0x7f5ca013df80 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.768+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca013c2b0 con 0x7f5ca013df80 2024-01-16T15:01:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.769+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca013c870 con 0x7f5ca013df80 2024-01-16T15:01:52.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.770+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca01f2fa0 con 0x7f5ca013df80 2024-01-16T15:01:52.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.771+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f5c9004d450 con 0x7f5ca013df80 2024-01-16T15:01:52.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.771+0000 7f5c96a00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 0x7f5c7806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:52.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.772+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c9002d040 con 0x7f5ca013df80 2024-01-16T15:01:52.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.772+0000 7f5c97e00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 0x7f5c7806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:52.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.772+0000 7f5c97e00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 0x7f5c7806dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006fd0 tx=0x7f5c88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:52.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.776+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ca01f2fa0 con 0x7f5ca013df80 2024-01-16T15:01:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.981+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5ca01f2f70 con 0x7f5ca013df80 2024-01-16T15:01:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.982+0000 7f5c96a00640 1 -- 172.21.15.125:0/1546566809 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v50) v1 ==== 80+0+122 (secure 0 0 0) 0x7f5c90035020 con 0x7f5ca013df80 2024-01-16T15:01:52.963 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/351760567 2024-01-17T15:01:46.499505+0000 2024-01-16T15:01:52.963 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/351760567 2024-01-17T15:01:46.499505+0000 2024-01-16T15:01:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 msgr2=0x7f5c7806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 0x7f5c7806dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006fd0 tx=0x7f5c88008040 comp rx=0 tx=0).stop 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 msgr2=0x7f5ca00b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002b950 tx=0x7f5c900046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 shutdown_connections 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ca015f9c0 0x7f5ca013b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ca013e360 0x7f5ca0154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c7806b860 0x7f5c7806dd20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 --2- 172.21.15.125:0/1546566809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca013df80 0x7f5ca00b28e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 >> 172.21.15.125:0/1546566809 conn(0x7f5ca006a800 msgr2=0x7f5ca006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.985+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 shutdown_connections 2024-01-16T15:01:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:52.986+0000 7f5ca5c00640 1 -- 172.21.15.125:0/1546566809 wait complete. 2024-01-16T15:01:52.986 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:01:52.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:52.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:53.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43620]: fuse finished with error 0 and tester_r 0 2024-01-16T15:01:53.081 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:01:53.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:53.082 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:01:53.254 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:01:53.254 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:53.310 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:01:53.310 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:01:53.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:53.311 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:01:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:01:53.363 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:01:53.363 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:53.421 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:01:53.421 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:01:53.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:53.421 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:01:53.469 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:01:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:01:53.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.710+0000 7fe38ca00640 1 Processor -- start 2024-01-16T15:01:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.710+0000 7fe38ca00640 1 -- start start 2024-01-16T15:01:53.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe38ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388162810 0x7fe388071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe38ca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe388071780 0x7fe38806c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe38ca00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880702e0 con 0x7fe388162430 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe38ca00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3880705c0 con 0x7fe388071780 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe38ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880708a0 con 0x7fe388162810 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe386800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe388162430 0x7fe3880b3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38662/0 (socket says 172.21.15.125:38662) 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe386800640 1 -- 172.21.15.125:0/1476196772 learned_addr learned my addr 172.21.15.125:0/1476196772 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.712+0000 7fe387200640 1 --2- 172.21.15.125:0/1476196772 >> v2:172.21.15.125:3300/0 conn(0x7fe388071780 0x7fe38806c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe385e00640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388162810 0x7fe388071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe385400640 1 -- 172.21.15.125:0/1476196772 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821056672 0 0) 0x7fe3880702e0 con 0x7fe388162430 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe385400640 1 -- 172.21.15.125:0/1476196772 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe368003520 con 0x7fe388162430 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe387200640 1 -- 172.21.15.125:0/1476196772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388162810 msgr2=0x7fe388071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe387200640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388162810 0x7fe388071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe387200640 1 -- 172.21.15.125:0/1476196772 >> v1:172.21.15.125:6789/0 conn(0x7fe388162430 legacy=0x7fe3880b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.713+0000 7fe387200640 1 -- 172.21.15.125:0/1476196772 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388070b80 con 0x7fe388071780 2024-01-16T15:01:53.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.714+0000 7fe387200640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388071780 0x7fe38806c8c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14e02296bab43ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.714+0000 7fe385400640 1 -- 172.21.15.125:0/1476196772 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38002d040 con 0x7fe388071780 2024-01-16T15:01:53.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.714+0000 7fe385400640 1 -- 172.21.15.125:0/1476196772 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380031810 con 0x7fe388071780 2024-01-16T15:01:53.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.714+0000 7fe385400640 1 -- 172.21.15.125:0/1476196772 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380031ba0 con 0x7fe388071780 2024-01-16T15:01:53.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 -- 172.21.15.125:0/1476196772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388071780 msgr2=0x7fe38806c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388071780 0x7fe38806c8c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 -- 172.21.15.125:0/1476196772 shutdown_connections 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388071780 0x7fe38806c8c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 --2- 172.21.15.125:0/1476196772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388162810 0x7fe388071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 -- 172.21.15.125:0/1476196772 >> 172.21.15.125:0/1476196772 conn(0x7fe38806a800 msgr2=0x7fe38806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 -- 172.21.15.125:0/1476196772 shutdown_connections 2024-01-16T15:01:53.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.715+0000 7fe38ca00640 1 -- 172.21.15.125:0/1476196772 wait complete. 2024-01-16T15:01:53.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.718+0000 7fe38ca00640 1 Processor -- start 2024-01-16T15:01:53.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.718+0000 7fe38ca00640 1 -- start start 2024-01-16T15:01:53.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388071780 0x7fe38815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388165380 0x7fe3881419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe388150730 con 0x7fe388162430 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe388164660 con 0x7fe388165380 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe38ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3881572b0 con 0x7fe388071780 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe385e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe385e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45708/0 (socket says 172.21.15.125:45708) 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.719+0000 7fe385e00640 1 -- 172.21.15.125:0/4239722817 learned_addr learned my addr 172.21.15.125:0/4239722817 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe386800640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388071780 0x7fe38815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe387200640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388165380 0x7fe3881419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 -- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388071780 msgr2=0x7fe38815a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388071780 0x7fe38815a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 -- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388165380 msgr2=0x7fe3881419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388165380 0x7fe3881419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 -- 172.21.15.125:0/4239722817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380009400 con 0x7fe388162430 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.720+0000 7fe385e00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe37000d5f0 tx=0x7fe37000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.721+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3700125f0 con 0x7fe388162430 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.721+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370012750 con 0x7fe388162430 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.721+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370012b00 con 0x7fe388162430 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.721+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe388157590 con 0x7fe388162430 2024-01-16T15:01:53.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.721+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe388164920 con 0x7fe388162430 2024-01-16T15:01:53.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.722+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe388164ab0 con 0x7fe388162430 2024-01-16T15:01:53.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.723+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe370012c60 con 0x7fe388162430 2024-01-16T15:01:53.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.724+0000 7fe37ea00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 0x7fe36006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.724+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe37008a8c0 con 0x7fe388162430 2024-01-16T15:01:53.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.724+0000 7fe386800640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 0x7fe36006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.724+0000 7fe386800640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 0x7fe36006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe378001560 tx=0x7fe378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.728+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe370055130 con 0x7fe388162430 2024-01-16T15:01:53.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.961+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe388067450 con 0x7fe388162430 2024-01-16T15:01:53.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.962+0000 7fe37ea00640 1 -- 172.21.15.125:0/4239722817 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+3711 (secure 0 0 0) 0x7fe370058750 con 0x7fe388162430 2024-01-16T15:01:53.943 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:53.943 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":31,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/3050283375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3050283375},{"type":"v1","addr":"172.21.15.137:6833","nonce":3050283375}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4258955541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4258955541},{"type":"v1","addr":"172.21.15.137:6835","nonce":4258955541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5739,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1745104106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1745104106},{"type":"v1","addr":"172.21.15.125:6836","nonce":1745104106}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6237,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/4055907802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":4055907802},{"type":"v1","addr":"172.21.15.125:6837","nonce":4055907802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[]} 2024-01-16T15:01:53.943 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 31 2024-01-16T15:01:53.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.964+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 msgr2=0x7fe36006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.964+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 0x7fe36006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe378001560 tx=0x7fe378009040 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.964+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 msgr2=0x7fe388164e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.964+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe37000d5f0 tx=0x7fe37000dab0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 shutdown_connections 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388165380 0x7fe3881419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b860 0x7fe36006dd20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388162430 0x7fe388164e40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 --2- 172.21.15.125:0/4239722817 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe388071780 0x7fe38815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 >> 172.21.15.125:0/4239722817 conn(0x7fe38806a800 msgr2=0x7fe38806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 shutdown_connections 2024-01-16T15:01:53.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:53.965+0000 7fe38ca00640 1 -- 172.21.15.125:0/4239722817 wait complete. 2024-01-16T15:01:53.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:01:54.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.188+0000 7f922d200640 1 Processor -- start 2024-01-16T15:01:54.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.188+0000 7f922d200640 1 -- start start 2024-01-16T15:01:54.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f922d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f922813fcb0 0x7f92280cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f922d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92280b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f922d200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280ce440 con 0x7f922813cf10 2024-01-16T15:01:54.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f922d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280ce720 con 0x7f922813fcb0 2024-01-16T15:01:54.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f922d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92280cea00 con 0x7f92280cdc20 2024-01-16T15:01:54.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f9227400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f922813cf10 0x7f922813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38668/0 (socket says 172.21.15.125:38668) 2024-01-16T15:01:54.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f9227400640 1 -- 172.21.15.125:0/2803369497 learned_addr learned my addr 172.21.15.125:0/2803369497 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962338441 0 0) 0x7f92280ce440 con 0x7f922813cf10 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.190+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208005730 con 0x7f922813cf10 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9227e00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92280b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f921e800640 1 --2- 172.21.15.125:0/2803369497 >> v2:172.21.15.137:3300/0 conn(0x7f922813fcb0 0x7f92280cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163275208 0 0) 0x7f9208005730 con 0x7f922813cf10 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 msgr2=0x7f92280b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92280b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 >> v2:172.21.15.137:3300/0 conn(0x7f922813fcb0 msgr2=0x7f92280cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 --2- 172.21.15.125:0/2803369497 >> v2:172.21.15.137:3300/0 conn(0x7f922813fcb0 0x7f92280cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280b1d90 con 0x7f922813cf10 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f9218001670 con 0x7f922813cf10 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 >> v1:172.21.15.125:6789/0 conn(0x7f922813cf10 legacy=0x7f922813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208005b80 0x7f9208005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92080069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9208006f10 0x7f9208007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92280ce720 con 0x7f9208006f10 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92280cea00 con 0x7f9208005b80 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920800dd20 con 0x7f92280cdc20 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f921e800640 1 --2- 172.21.15.125:0/2803369497 >> v2:172.21.15.137:3300/0 conn(0x7f922813fcb0 0x7f92280cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.191+0000 7f9227e00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9208006f10 0x7f9208007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227400640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208005b80 0x7f9208005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f921e800640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92080069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227e00640 1 -- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 msgr2=0x7f92080069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227e00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92280cdc20 0x7f92080069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227e00640 1 -- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208005b80 msgr2=0x7f9208005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227e00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208005b80 0x7f9208005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.192+0000 7f9227e00640 1 -- 172.21.15.125:0/2803369497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9208005730 con 0x7f9208006f10 2024-01-16T15:01:54.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.193+0000 7f9227e00640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9208006f10 0x7f9208007390 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9220007360 tx=0x7f9208007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8284954912b9d386 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.193+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922002ccd0 con 0x7f9208006f10 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.193+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922002ce30 con 0x7f9208006f10 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.193+0000 7f9226a00640 1 -- 172.21.15.125:0/2803369497 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220034660 con 0x7f9208006f10 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 -- 172.21.15.125:0/2803369497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9208006f10 msgr2=0x7f9208007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 --2- 172.21.15.125:0/2803369497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9208006f10 0x7f9208007390 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9220007360 tx=0x7f9208007a00 comp rx=0 tx=0).stop 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f9227e00640 1 -- 172.21.15.125:0/2803369497 reap_dead start 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 -- 172.21.15.125:0/2803369497 shutdown_connections 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 -- 172.21.15.125:0/2803369497 >> 172.21.15.125:0/2803369497 conn(0x7f922806a7d0 msgr2=0x7f922806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 -- 172.21.15.125:0/2803369497 shutdown_connections 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.194+0000 7f922d200640 1 -- 172.21.15.125:0/2803369497 wait complete. 2024-01-16T15:01:54.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.196+0000 7f922d200640 1 Processor -- start 2024-01-16T15:01:54.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.196+0000 7f922d200640 1 -- start start 2024-01-16T15:01:54.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208006f10 0x7f922815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9228137cf0 0x7f92281525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922814e970 con 0x7f9228137870 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922815bcf0 con 0x7f9208006f10 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f922d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922814ddf0 con 0x7f9228137cf0 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f921e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f921e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45722/0 (socket says 172.21.15.125:45722) 2024-01-16T15:01:54.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f921e800640 1 -- 172.21.15.125:0/4273475172 learned_addr learned my addr 172.21.15.125:0/4273475172 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f921e800640 1 -- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9228137cf0 msgr2=0x7f92281525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.197+0000 7f921e800640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9228137cf0 0x7f92281525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921e800640 1 -- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208006f10 msgr2=0x7f922815b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921e800640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208006f10 0x7f922815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921e800640 1 -- 172.21.15.125:0/4273475172 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922814e0d0 con 0x7f9228137870 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921e800640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f921000e7f0 tx=0x7f921000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921000eea0 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921000a920 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921000acd0 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922814e3b0 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.198+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922814da80 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.199+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9228159f60 con 0x7f9228137870 2024-01-16T15:01:54.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.200+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f9210015480 con 0x7f9228137870 2024-01-16T15:01:54.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.201+0000 7f921fe00640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 0x7f920006dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.201+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f921008aa70 con 0x7f9228137870 2024-01-16T15:01:54.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.201+0000 7f9227400640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 0x7f920006dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.202+0000 7f9227400640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 0x7f920006dd70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9218000f60 tx=0x7f9218000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.205+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92100552a0 con 0x7f9228137870 2024-01-16T15:01:54.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.400+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f9228067450 con 0x7f9228137870 2024-01-16T15:01:54.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.407+0000 7f921fe00640 1 -- 172.21.15.125:0/4273475172 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v17) v1 ==== 115+0+0 (secure 0 0 0) 0x7f92100588c0 con 0x7f9228137870 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 msgr2=0x7f920006dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 0x7f920006dd70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9218000f60 tx=0x7f9218000ed0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 msgr2=0x7f9228138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f921000e7f0 tx=0x7f921000ecb0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 shutdown_connections 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9228137cf0 0x7f92281525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f920006b8b0 0x7f920006dd70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9228137870 0x7f9228138100 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 --2- 172.21.15.125:0/4273475172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208006f10 0x7f922815b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 >> 172.21.15.125:0/4273475172 conn(0x7f922806a7d0 msgr2=0x7f9228070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 shutdown_connections 2024-01-16T15:01:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.410+0000 7f922d200640 1 -- 172.21.15.125:0/4273475172 wait complete. 2024-01-16T15:01:54.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:01:54.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.633+0000 7f35fdc00640 1 Processor -- start 2024-01-16T15:01:54.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.633+0000 7f35fdc00640 1 -- start start 2024-01-16T15:01:54.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35fdc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f35f813df80 0x7f35f815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35fdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35f80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35fdc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f806c470 con 0x7f35f813e360 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35fdc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f806c750 con 0x7f35f813df80 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35fdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f806ca30 con 0x7f35f80cf9a0 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35ede00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f35f813e360 0x7f35f80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38680/0 (socket says 172.21.15.125:38680) 2024-01-16T15:01:54.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.635+0000 7f35ede00640 1 -- 172.21.15.125:0/1598650017 learned_addr learned my addr 172.21.15.125:0/1598650017 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008465881 0 0) 0x7f35f806c470 con 0x7f35f813e360 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6a00640 1 --2- 172.21.15.125:0/1598650017 >> v2:172.21.15.137:3300/0 conn(0x7f35f813df80 0x7f35f815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8003520 con 0x7f35f813e360 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f7400640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35f80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2925848007 0 0) 0x7f35d8003520 con 0x7f35f813e360 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 msgr2=0x7f35f80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35f80cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 >> v2:172.21.15.137:3300/0 conn(0x7f35f813df80 msgr2=0x7f35f815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 --2- 172.21.15.125:0/1598650017 >> v2:172.21.15.137:3300/0 conn(0x7f35f813df80 0x7f35f815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f806cd10 con 0x7f35f813e360 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f35e0001670 con 0x7f35f813e360 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6a00640 1 --2- 172.21.15.125:0/1598650017 >> v2:172.21.15.137:3300/0 conn(0x7f35f813df80 0x7f35f815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:54.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 >> v1:172.21.15.125:6789/0 conn(0x7f35f813e360 legacy=0x7f35f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d8003970 0x7f35d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d8004d00 0x7f35d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f806c750 con 0x7f35d8004d00 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f806ca30 con 0x7f35d8003970 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.636+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d800bae0 con 0x7f35f80cf9a0 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d8004d00 0x7f35d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35ede00640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 -- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 msgr2=0x7f35d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80cf9a0 0x7f35d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 -- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d8003970 msgr2=0x7f35d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d8003970 0x7f35d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.637+0000 7f35f7400640 1 -- 172.21.15.125:0/1598650017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8003520 con 0x7f35d8004d00 2024-01-16T15:01:54.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.638+0000 7f35f7400640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d8004d00 0x7f35d8005180 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f35f0007330 tx=0x7f35d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50b261d89bce3ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f002cca0 con 0x7f35d8004d00 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002ce00 con 0x7f35d8004d00 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35f6000640 1 -- 172.21.15.125:0/1598650017 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0034660 con 0x7f35d8004d00 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 -- 172.21.15.125:0/1598650017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d8004d00 msgr2=0x7f35d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 --2- 172.21.15.125:0/1598650017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d8004d00 0x7f35d8005180 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f35f0007330 tx=0x7f35d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35f7400640 1 -- 172.21.15.125:0/1598650017 reap_dead start 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 -- 172.21.15.125:0/1598650017 shutdown_connections 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 -- 172.21.15.125:0/1598650017 >> 172.21.15.125:0/1598650017 conn(0x7f35f806a800 msgr2=0x7f35f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:54.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 -- 172.21.15.125:0/1598650017 shutdown_connections 2024-01-16T15:01:54.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.639+0000 7f35fdc00640 1 -- 172.21.15.125:0/1598650017 wait complete. 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.641+0000 7f35fdc00640 1 Processor -- start 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.641+0000 7f35fdc00640 1 -- start start 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f813d3f0 0x7f35f80a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 0x7f35f80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80b04e0 con 0x7f35f815a180 2024-01-16T15:01:54.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81f3b40 con 0x7f35f813d3f0 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35fdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f813d7d0 con 0x7f35f80afb20 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.642+0000 7f35f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45742/0 (socket says 172.21.15.125:45742) 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 -- 172.21.15.125:0/3471036081 learned_addr learned my addr 172.21.15.125:0/3471036081 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f6a00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f813d3f0 0x7f35f80a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35ede00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 0x7f35f80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 -- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 msgr2=0x7f35f80affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 0x7f35f80affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 -- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f813d3f0 msgr2=0x7f35f80a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f813d3f0 0x7f35f80a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 -- 172.21.15.125:0/3471036081 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f81f2cd0 con 0x7f35f815a180 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35ede00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 0x7f35f80affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.643+0000 7f35f7400640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f35f0007330 tx=0x7f35f0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.644+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f003e600 con 0x7f35f815a180 2024-01-16T15:01:54.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.644+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f003e780 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.644+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f003eb30 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.644+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f81f2f30 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.644+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f81f34f0 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.645+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f81f3cb0 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.646+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f35f003ec90 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.647+0000 7f35ef400640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 0x7f35d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.647+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f35f00ac4f0 con 0x7f35f815a180 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.647+0000 7f35f6a00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 0x7f35d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.648+0000 7f35f6a00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 0x7f35d006dd20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35f80b42b0 tx=0x7f35e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:54.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.651+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35f0076e40 con 0x7f35f815a180 2024-01-16T15:01:54.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.842+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f35f80675d0 con 0x7f35f815a180 2024-01-16T15:01:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.848+0000 7f35ef400640 1 -- 172.21.15.125:0/3471036081 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v18) v1 ==== 120+0+0 (secure 0 0 0) 0x7f35f007a460 con 0x7f35f815a180 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 msgr2=0x7f35d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 0x7f35d006dd20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35f80b42b0 tx=0x7f35e8008040 comp rx=0 tx=0).stop 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 msgr2=0x7f35f815a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f35f0007330 tx=0x7f35f0004ab0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 shutdown_connections 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35d006b860 0x7f35d006dd20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35f815a180 0x7f35f815a600 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f80afb20 0x7f35f80affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 --2- 172.21.15.125:0/3471036081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f813d3f0 0x7f35f80a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.851+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 >> 172.21.15.125:0/3471036081 conn(0x7f35f806a800 msgr2=0x7f35f80ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.852+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 shutdown_connections 2024-01-16T15:01:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:54.852+0000 7f35fdc00640 1 -- 172.21.15.125:0/3471036081 wait complete. 2024-01-16T15:01:54.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:01:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.068+0000 7fb727400640 1 Processor -- start 2024-01-16T15:01:55.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.068+0000 7fb727400640 1 -- start start 2024-01-16T15:01:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb727400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb7200b3ae0 0x7fb72015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb727400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb720070c40 con 0x7fb7200ae5a0 2024-01-16T15:01:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb727400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72006c470 con 0x7fb7200cf290 2024-01-16T15:01:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb727400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb720071450 con 0x7fb7200b3ae0 2024-01-16T15:01:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb724a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb7200ae5a0 0x7fb72006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38692/0 (socket says 172.21.15.125:38692) 2024-01-16T15:01:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.070+0000 7fb724a00640 1 -- 172.21.15.125:0/1602790232 learned_addr learned my addr 172.21.15.125:0/1602790232 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71fe00640 1 --2- 172.21.15.125:0/1602790232 >> v2:172.21.15.137:3300/0 conn(0x7fb7200b3ae0 0x7fb72015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308591361 0 0) 0x7fb720070c40 con 0x7fb7200ae5a0 2024-01-16T15:01:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb700003520 con 0x7fb7200ae5a0 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085354404 0 0) 0x7fb700003520 con 0x7fb7200ae5a0 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 >> v2:172.21.15.137:3300/0 conn(0x7fb7200b3ae0 msgr2=0x7fb72015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 --2- 172.21.15.125:0/1602790232 >> v2:172.21.15.137:3300/0 conn(0x7fb7200b3ae0 0x7fb72015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 >> v1:172.21.15.137:6789/0 conn(0x7fb7200cf290 legacy=0x7fb7200b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72006c750 con 0x7fb7200ae5a0 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb710001670 con 0x7fb7200ae5a0 2024-01-16T15:01:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 >> v1:172.21.15.125:6789/0 conn(0x7fb7200ae5a0 legacy=0x7fb72006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 0x7fb700003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7000047c0 0x7fb700004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb700005180 0x7fb70000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.071+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720071450 con 0x7fb700003970 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70000c030 con 0x7fb7000047c0 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70000c1f0 con 0x7fb700005180 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 0x7fb700003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb71fe00640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7000047c0 0x7fb700004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 -- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb700005180 msgr2=0x7fb70000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb700005180 0x7fb70000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 -- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7000047c0 msgr2=0x7fb700004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7000047c0 0x7fb700004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb724a00640 1 -- 172.21.15.125:0/1602790232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700003520 con 0x7fb700003970 2024-01-16T15:01:55.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.072+0000 7fb725400640 1 -- 172.21.15.125:0/1602790232 reap_dead start 2024-01-16T15:01:55.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.073+0000 7fb724a00640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 0x7fb700003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb71000c210 tx=0x7fb7000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85089b3ddc16239 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710005ad0 con 0x7fb700003970 2024-01-16T15:01:55.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710005c30 con 0x7fb700003970 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb71f400640 1 -- 172.21.15.125:0/1602790232 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71002cad0 con 0x7fb700003970 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 -- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 msgr2=0x7fb700003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 0x7fb700003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb71000c210 tx=0x7fb7000057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 -- 172.21.15.125:0/1602790232 shutdown_connections 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 --2- 172.21.15.125:0/1602790232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb700003970 0x7fb700003d90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 -- 172.21.15.125:0/1602790232 >> 172.21.15.125:0/1602790232 conn(0x7fb72006a800 msgr2=0x7fb72006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 -- 172.21.15.125:0/1602790232 shutdown_connections 2024-01-16T15:01:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.074+0000 7fb727400640 1 -- 172.21.15.125:0/1602790232 wait complete. 2024-01-16T15:01:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.076+0000 7fb727400640 1 Processor -- start 2024-01-16T15:01:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.076+0000 7fb727400640 1 -- start start 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb700003970 0x7fb7200cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7200acfb0 0x7fb7200ad430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200cc010 con 0x7fb7200ce920 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200ad970 con 0x7fb700003970 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb727400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7201f2bd0 con 0x7fb7200acfb0 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.077+0000 7fb71fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45764/0 (socket says 172.21.15.125:45764) 2024-01-16T15:01:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 -- 172.21.15.125:0/226256377 learned_addr learned my addr 172.21.15.125:0/226256377 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 -- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7200acfb0 msgr2=0x7fb7200ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7200acfb0 0x7fb7200ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 -- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb700003970 msgr2=0x7fb7200cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb700003970 0x7fb7200cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 -- 172.21.15.125:0/226256377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200cbde0 con 0x7fb7200ce920 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71fe00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb708010910 tx=0x7fb708010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708009690 con 0x7fb7200ce920 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70800ec20 con 0x7fb7200ce920 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.078+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70801c400 con 0x7fb7200ce920 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.079+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7201566f0 con 0x7fb7200ce920 2024-01-16T15:01:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.079+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb720156cb0 con 0x7fb7200ce920 2024-01-16T15:01:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.080+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7200675d0 con 0x7fb7200ce920 2024-01-16T15:01:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.081+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb70801c560 con 0x7fb7200ce920 2024-01-16T15:01:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.082+0000 7fb71cc00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 0x7fb6f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.082+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb70808a800 con 0x7fb7200ce920 2024-01-16T15:01:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.082+0000 7fb724a00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 0x7fb6f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.083+0000 7fb724a00640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 0x7fb6f806dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb710034000 tx=0x7fb710039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.087+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb708055240 con 0x7fb7200ce920 2024-01-16T15:01:55.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.278+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb72006db00 con 0x7fb7200ce920 2024-01-16T15:01:55.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.285+0000 7fb71cc00640 1 -- 172.21.15.125:0/226256377 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb708058860 con 0x7fb7200ce920 2024-01-16T15:01:55.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 msgr2=0x7fb6f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 0x7fb6f806dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb710034000 tx=0x7fb710039040 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 msgr2=0x7fb720155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb708010910 tx=0x7fb708010dd0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 shutdown_connections 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7200acfb0 0x7fb7200ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb6f806b860 0x7fb6f806dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7200ce920 0x7fb720155d00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 --2- 172.21.15.125:0/226256377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb700003970 0x7fb7200cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.288+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 >> 172.21.15.125:0/226256377 conn(0x7fb72006a800 msgr2=0x7fb7200b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.289+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 shutdown_connections 2024-01-16T15:01:55.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.289+0000 7fb727400640 1 -- 172.21.15.125:0/226256377 wait complete. 2024-01-16T15:01:55.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:01:55.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.512+0000 7fee07400640 1 Processor -- start 2024-01-16T15:01:55.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.512+0000 7fee07400640 1 -- start start 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee07400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 0x7fee000b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee07400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fee000cf9f0 0x7fee000cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee07400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee000d0a10 con 0x7fee00161010 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee07400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee000d03e0 con 0x7fee000cf9f0 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee07400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee000d1f50 con 0x7fee00160920 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee06a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fee00161010 0x7fee00161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38702/0 (socket says 172.21.15.125:38702) 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee06a00640 1 -- 172.21.15.125:0/849256575 learned_addr learned my addr 172.21.15.125:0/849256575 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.514+0000 7fee05600640 1 --2- 172.21.15.125:0/849256575 >> v2:172.21.15.125:3300/0 conn(0x7fee000cf9f0 0x7fee000cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee06000640 1 --2- 172.21.15.125:0/849256575 >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 0x7fee000b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee04c00640 1 -- 172.21.15.125:0/849256575 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915285609 0 0) 0x7fee000d0a10 con 0x7fee00161010 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee04c00640 1 -- 172.21.15.125:0/849256575 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede0003520 con 0x7fee00161010 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee05600640 1 -- 172.21.15.125:0/849256575 >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 msgr2=0x7fee000b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee05600640 1 --2- 172.21.15.125:0/849256575 >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 0x7fee000b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee05600640 1 -- 172.21.15.125:0/849256575 >> v1:172.21.15.125:6789/0 conn(0x7fee00161010 legacy=0x7fee00161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee05600640 1 -- 172.21.15.125:0/849256575 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000cf080 con 0x7fee000cf9f0 2024-01-16T15:01:55.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.515+0000 7fee06000640 1 --2- 172.21.15.125:0/849256575 >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 0x7fee000b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:55.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee05600640 1 --2- 172.21.15.125:0/849256575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee000cf9f0 0x7fee000cfea0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fee0008c060 tx=0x7fedf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be05331516a794d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee04c00640 1 -- 172.21.15.125:0/849256575 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf002cc80 con 0x7fee000cf9f0 2024-01-16T15:01:55.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee04c00640 1 -- 172.21.15.125:0/849256575 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf002cde0 con 0x7fee000cf9f0 2024-01-16T15:01:55.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee04c00640 1 -- 172.21.15.125:0/849256575 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0034660 con 0x7fee000cf9f0 2024-01-16T15:01:55.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee07400640 1 -- 172.21.15.125:0/849256575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee000cf9f0 msgr2=0x7fee000cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.517+0000 7fee07400640 1 --2- 172.21.15.125:0/849256575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee000cf9f0 0x7fee000cfea0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fee0008c060 tx=0x7fedf002bed0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 -- 172.21.15.125:0/849256575 shutdown_connections 2024-01-16T15:01:55.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 --2- 172.21.15.125:0/849256575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee000cf9f0 0x7fee000cfea0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 --2- 172.21.15.125:0/849256575 >> v2:172.21.15.137:3300/0 conn(0x7fee00160920 0x7fee000b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 -- 172.21.15.125:0/849256575 >> 172.21.15.125:0/849256575 conn(0x7fee000d3090 msgr2=0x7fee000d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 -- 172.21.15.125:0/849256575 shutdown_connections 2024-01-16T15:01:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.518+0000 7fee07400640 1 -- 172.21.15.125:0/849256575 wait complete. 2024-01-16T15:01:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.519+0000 7fee07400640 1 Processor -- start 2024-01-16T15:01:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.519+0000 7fee07400640 1 -- start start 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee000cf9f0 0x7fee001f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee00161010 0x7fee0006c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000cd5d0 con 0x7fee00160920 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0014cd90 con 0x7fee00161010 2024-01-16T15:01:55.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.520+0000 7fee07400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000cd010 con 0x7fee000cf9f0 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45788/0 (socket says 172.21.15.125:45788) 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 -- 172.21.15.125:0/335557670 learned_addr learned my addr 172.21.15.125:0/335557670 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee06000640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee000cf9f0 0x7fee001f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 -- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee000cf9f0 msgr2=0x7fee001f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee000cf9f0 0x7fee001f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 -- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee00161010 msgr2=0x7fee0006c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee00161010 0x7fee0006c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 -- 172.21.15.125:0/335557670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0009a70 con 0x7fee00160920 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.521+0000 7fee05600640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fedf0037040 tx=0x7fedf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.522+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0040650 con 0x7fee00160920 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.522+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf00407b0 con 0x7fee00160920 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.522+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0040b60 con 0x7fee00160920 2024-01-16T15:01:55.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.522+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee000cd2f0 con 0x7fee00160920 2024-01-16T15:01:55.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.522+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0014d6e0 con 0x7fee00160920 2024-01-16T15:01:55.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.523+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0014d260 con 0x7fee00160920 2024-01-16T15:01:55.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.524+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fedf0040cc0 con 0x7fee00160920 2024-01-16T15:01:55.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.525+0000 7fedfe000640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 0x7fedd806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.525+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fedf00acfe0 con 0x7fee00160920 2024-01-16T15:01:55.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.525+0000 7fee06000640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 0x7fedd806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.526+0000 7fee06000640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 0x7fedd806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fede800d8b0 tx=0x7fede8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.529+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee0014d260 con 0x7fee00160920 2024-01-16T15:01:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:55.790+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7fee0014d260 con 0x7fee00160920 2024-01-16T15:01:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.576+0000 7fedfe000640 1 -- 172.21.15.125:0/335557670 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v264) v1 ==== 153+0+0 (secure 0 0 0) 0x7fee0014d260 con 0x7fee00160920 2024-01-16T15:01:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 msgr2=0x7fedd806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 0x7fedd806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fede800d8b0 tx=0x7fede8008040 comp rx=0 tx=0).stop 2024-01-16T15:01:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 msgr2=0x7fee0006c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fedf0037040 tx=0x7fedf0004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 shutdown_connections 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.580+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee00161010 0x7fee0006c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fedd806b860 0x7fedd806dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee00160920 0x7fee0006c360 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 --2- 172.21.15.125:0/335557670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee000cf9f0 0x7fee001f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 >> 172.21.15.125:0/335557670 conn(0x7fee000d3090 msgr2=0x7fee0006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 shutdown_connections 2024-01-16T15:01:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.581+0000 7fee07400640 1 -- 172.21.15.125:0/335557670 wait complete. 2024-01-16T15:01:56.583 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:01:56.583 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:01:56.583 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:01:56.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:56.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.812+0000 7f8acb600640 1 Processor -- start 2024-01-16T15:01:56.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.812+0000 7f8acb600640 1 -- start start 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.813+0000 7f8acb600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ac415e3b0 0x7f8ac4164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.813+0000 7f8acb600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40b1ee0 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.813+0000 7f8acb600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40b2780 con 0x7f8ac4143ae0 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.813+0000 7f8acb600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac415e990 con 0x7f8ac415e3b0 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8ac40cf9a0 0x7f8ac415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42090/0 (socket says 172.21.15.125:42090) 2024-01-16T15:01:56.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac3e00640 1 -- 172.21.15.125:0/1782346240 learned_addr learned my addr 172.21.15.125:0/1782346240 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:56.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac9600640 1 --2- 172.21.15.125:0/1782346240 >> v2:172.21.15.125:3300/0 conn(0x7f8ac415e3b0 0x7f8ac4164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac9600640 1 -- 172.21.15.125:0/1782346240 >> v1:172.21.15.137:6789/0 conn(0x7f8ac4143ae0 legacy=0x7f8ac40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:56.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac9600640 1 -- 172.21.15.125:0/1782346240 >> v1:172.21.15.125:6789/0 conn(0x7f8ac40cf9a0 legacy=0x7f8ac415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac9600640 1 -- 172.21.15.125:0/1782346240 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac40b2130 con 0x7f8ac415e3b0 2024-01-16T15:01:56.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.814+0000 7f8ac3400640 1 -- 172.21.15.125:0/1782346240 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611020500 0 0) 0x7f8ac40b1ee0 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.815+0000 7f8ac9600640 1 --2- 172.21.15.125:0/1782346240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4164de0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009d90 tx=0x7f8abc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610e9ce00881a48d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8ac3400640 1 -- 172.21.15.125:0/1782346240 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02cc60 con 0x7f8ac415e3b0 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8ac3400640 1 -- 172.21.15.125:0/1782346240 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc02cdc0 con 0x7f8ac415e3b0 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8ac3400640 1 -- 172.21.15.125:0/1782346240 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc034660 con 0x7f8ac415e3b0 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 -- 172.21.15.125:0/1782346240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac415e3b0 msgr2=0x7f8ac4164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 --2- 172.21.15.125:0/1782346240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4164de0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009d90 tx=0x7f8abc02beb0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 -- 172.21.15.125:0/1782346240 shutdown_connections 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 --2- 172.21.15.125:0/1782346240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4164de0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 -- 172.21.15.125:0/1782346240 >> 172.21.15.125:0/1782346240 conn(0x7f8ac406a800 msgr2=0x7f8ac406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 -- 172.21.15.125:0/1782346240 shutdown_connections 2024-01-16T15:01:56.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.816+0000 7f8acb600640 1 -- 172.21.15.125:0/1782346240 wait complete. 2024-01-16T15:01:56.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.818+0000 7f8acb600640 1 Processor -- start 2024-01-16T15:01:56.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.818+0000 7f8acb600640 1 -- start start 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ac4143ae0 0x7f8ac413e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4070530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4159f60 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4142380 con 0x7f8ac415e3b0 2024-01-16T15:01:56.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8acb600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40cdd20 con 0x7f8ac4143ae0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58552/0 (socket says 172.21.15.125:58552) 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.819+0000 7f8ac8c00640 1 -- 172.21.15.125:0/2920215942 learned_addr learned my addr 172.21.15.125:0/2920215942 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac9600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac3e00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ac4143ae0 0x7f8ac413e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 -- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ac4143ae0 msgr2=0x7f8ac413e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ac4143ae0 0x7f8ac413e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 -- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 msgr2=0x7f8ac4070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4070530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 -- 172.21.15.125:0/2920215942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc009a70 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac9600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4070530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac8c00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400c430 tx=0x7f8ab400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4011790 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab40118f0 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac40ce580 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.820+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4011ce0 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.821+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac40ceb40 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.822+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac40cbf50 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.822+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8ab401c460 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.823+0000 7f8ac0c00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 0x7f8a9c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.823+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8ab408a230 con 0x7f8ac40cf9a0 2024-01-16T15:01:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.824+0000 7f8ac3e00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 0x7f8a9c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.824+0000 7f8ac3e00640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 0x7f8a9c06dd50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8aac007fd0 tx=0x7f8aac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:56.828+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ab4054b80 con 0x7f8ac40cf9a0 2024-01-16T15:01:57.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.022+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ac406c3f0 con 0x7f8ac40cf9a0 2024-01-16T15:01:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.024+0000 7f8ac0c00640 1 -- 172.21.15.125:0/2920215942 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13034 (secure 0 0 0) 0x7f8ab40581a0 con 0x7f8ac40cf9a0 2024-01-16T15:01:57.005 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:57.005 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":50,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:01:49.523477+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/351760567":"2024-01-17T15:01:46.499505+0000","172.21.15.125:6835/351760567":"2024-01-17T15:01:46.499505+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 msgr2=0x7f8a9c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 0x7f8a9c06dd50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8aac007fd0 tx=0x7f8aac009040 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 msgr2=0x7f8ac4159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400c430 tx=0x7f8ab400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 shutdown_connections 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ac415e3b0 0x7f8ac4070530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ac4143ae0 0x7f8ac413e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8a9c06b890 0x7f8a9c06dd50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 --2- 172.21.15.125:0/2920215942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ac40cf9a0 0x7f8ac4159220 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 >> 172.21.15.125:0/2920215942 conn(0x7f8ac406a800 msgr2=0x7f8ac413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 shutdown_connections 2024-01-16T15:01:57.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.027+0000 7f8acb600640 1 -- 172.21.15.125:0/2920215942 wait complete. 2024-01-16T15:01:57.025 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:01:57.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:01:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.249+0000 7f327ee00640 1 Processor -- start 2024-01-16T15:01:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.249+0000 7f327ee00640 1 -- start start 2024-01-16T15:01:57.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ee00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f327815e3b0 0x7f3278164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b1ee0 con 0x7f3278143ae0 2024-01-16T15:01:57.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b2780 con 0x7f32780cf9a0 2024-01-16T15:01:57.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ee00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327815e990 con 0x7f327815e3b0 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f327815e3b0 0x7f3278164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f3277e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3278143ae0 0x7f32780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42096/0 (socket says 172.21.15.125:42096) 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f3277e00640 1 -- 172.21.15.125:0/3590760099 learned_addr learned my addr 172.21.15.125:0/3590760099 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ce00640 1 -- 172.21.15.125:0/3590760099 >> v1:172.21.15.137:6789/0 conn(0x7f32780cf9a0 legacy=0x7f327815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.251+0000 7f327ce00640 1 -- 172.21.15.125:0/3590760099 >> v1:172.21.15.125:6789/0 conn(0x7f3278143ae0 legacy=0x7f32780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.252+0000 7f327ce00640 1 -- 172.21.15.125:0/3590760099 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780b2130 con 0x7f327815e3b0 2024-01-16T15:01:57.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f327ce00640 1 --2- 172.21.15.125:0/3590760099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f327815e3b0 0x7f3278164de0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f32680097a0 tx=0x7f326802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c75c0ce3beb095 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f3276a00640 1 -- 172.21.15.125:0/3590760099 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268031830 con 0x7f327815e3b0 2024-01-16T15:01:57.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f3276a00640 1 -- 172.21.15.125:0/3590760099 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268031990 con 0x7f327815e3b0 2024-01-16T15:01:57.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f3276a00640 1 -- 172.21.15.125:0/3590760099 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268031d20 con 0x7f327815e3b0 2024-01-16T15:01:57.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f327ee00640 1 -- 172.21.15.125:0/3590760099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f327815e3b0 msgr2=0x7f3278164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.253+0000 7f327ee00640 1 --2- 172.21.15.125:0/3590760099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f327815e3b0 0x7f3278164de0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f32680097a0 tx=0x7f326802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.254+0000 7f327ee00640 1 -- 172.21.15.125:0/3590760099 shutdown_connections 2024-01-16T15:01:57.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.254+0000 7f327ee00640 1 --2- 172.21.15.125:0/3590760099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f327815e3b0 0x7f3278164de0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.254+0000 7f327ee00640 1 -- 172.21.15.125:0/3590760099 >> 172.21.15.125:0/3590760099 conn(0x7f327806a800 msgr2=0x7f327806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:57.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.254+0000 7f327ee00640 1 -- 172.21.15.125:0/3590760099 shutdown_connections 2024-01-16T15:01:57.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.254+0000 7f327ee00640 1 -- 172.21.15.125:0/3590760099 wait complete. 2024-01-16T15:01:57.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.255+0000 7f327ee00640 1 Processor -- start 2024-01-16T15:01:57.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.255+0000 7f327ee00640 1 -- start start 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 0x7f3278159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f327815e3b0 0x7f32780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780acf30 con 0x7f3278143ae0 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327815a0d0 con 0x7f32780cf9a0 2024-01-16T15:01:57.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.256+0000 7f327ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327815eec0 con 0x7f327815e3b0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58570/0 (socket says 172.21.15.125:58570) 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 -- 172.21.15.125:0/4190792646 learned_addr learned my addr 172.21.15.125:0/4190792646 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f327ce00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f327815e3b0 0x7f32780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277e00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 0x7f3278159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 -- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f327815e3b0 msgr2=0x7f32780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f327815e3b0 0x7f32780702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 -- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 msgr2=0x7f3278159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 0x7f3278159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 -- 172.21.15.125:0/4190792646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3268009400 con 0x7f3278143ae0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277e00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 0x7f3278159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.257+0000 7f3277400640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f326000c460 tx=0x7f326000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.258+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260012830 con 0x7f3278143ae0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.258+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32600129d0 con 0x7f3278143ae0 2024-01-16T15:01:57.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.258+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32780acb90 con 0x7f3278143ae0 2024-01-16T15:01:57.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.258+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260012d80 con 0x7f3278143ae0 2024-01-16T15:01:57.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.258+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32781f7060 con 0x7f3278143ae0 2024-01-16T15:01:57.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.259+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32780675d0 con 0x7f3278143ae0 2024-01-16T15:01:57.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.260+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f326001b400 con 0x7f3278143ae0 2024-01-16T15:01:57.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.261+0000 7f326fe00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 0x7f325006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.261+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f326008a340 con 0x7f3278143ae0 2024-01-16T15:01:57.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.261+0000 7f3277e00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 0x7f325006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.262+0000 7f3277e00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 0x7f325006dd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3270005c00 tx=0x7f3270009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.265+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3260054c00 con 0x7f3278143ae0 2024-01-16T15:01:57.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.453+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32781416f0 con 0x7f3278143ae0 2024-01-16T15:01:57.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.454+0000 7f326fe00640 1 -- 172.21.15.125:0/4190792646 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3260058220 con 0x7f3278143ae0 2024-01-16T15:01:57.434 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 msgr2=0x7f325006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 0x7f325006dd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3270005c00 tx=0x7f3270009040 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 msgr2=0x7f32781f6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f326000c460 tx=0x7f326000c920 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 shutdown_connections 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f327815e3b0 0x7f32780702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f325006b860 0x7f325006dd20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3278143ae0 0x7f32781f6530 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 --2- 172.21.15.125:0/4190792646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32780cf9a0 0x7f3278159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 >> 172.21.15.125:0/4190792646 conn(0x7f327806a800 msgr2=0x7f3278142190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:57.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 shutdown_connections 2024-01-16T15:01:57.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.457+0000 7f327ee00640 1 -- 172.21.15.125:0/4190792646 wait complete. 2024-01-16T15:01:57.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:01:57.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.676+0000 7fb626600640 1 Processor -- start 2024-01-16T15:01:57.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.676+0000 7fb626600640 1 -- start start 2024-01-16T15:01:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb626600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb6200cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb626600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb6201618f0 0x7fb620070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb626600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb620071460 con 0x7fb620071030 2024-01-16T15:01:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb626600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62015ea30 con 0x7fb6201618f0 2024-01-16T15:01:57.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb626600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62006d860 con 0x7fb620157c30 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb61fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb620071030 0x7fb62006dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42100/0 (socket says 172.21.15.125:42100) 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.678+0000 7fb61fe00640 1 -- 172.21.15.125:0/523113324 learned_addr learned my addr 172.21.15.125:0/523113324 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61f400640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb6200cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61ea00640 1 --2- 172.21.15.125:0/523113324 >> v2:172.21.15.137:3300/0 conn(0x7fb6201618f0 0x7fb620070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59685528 0 0) 0x7fb620071460 con 0x7fb620071030 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb600003520 con 0x7fb620071030 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813133325 0 0) 0x7fb600003520 con 0x7fb620071030 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 msgr2=0x7fb6200cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb6200cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 >> v2:172.21.15.137:3300/0 conn(0x7fb6201618f0 msgr2=0x7fb620070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 --2- 172.21.15.125:0/523113324 >> v2:172.21.15.137:3300/0 conn(0x7fb6201618f0 0x7fb620070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62015deb0 con 0x7fb620071030 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb618001690 con 0x7fb620071030 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 >> v1:172.21.15.125:6789/0 conn(0x7fb620071030 legacy=0x7fb62006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb600003970 0x7fb600003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6000047c0 0x7fb600004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb600005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62015ea30 con 0x7fb600003970 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62006d860 con 0x7fb6000047c0 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.679+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60000b980 con 0x7fb620157c30 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb600003970 0x7fb600003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61fe00640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb600005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 -- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 msgr2=0x7fb600005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61ea00640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6000047c0 0x7fb600004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb620157c30 0x7fb600005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 -- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6000047c0 msgr2=0x7fb600004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6000047c0 0x7fb600004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.680+0000 7fb61f400640 1 -- 172.21.15.125:0/523113324 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb600003520 con 0x7fb600003970 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.681+0000 7fb61f400640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb600003970 0x7fb600003d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb610009fc0 tx=0x7fb6000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c601260752cdee9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61002cc20 con 0x7fb600003970 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61000f040 con 0x7fb600003970 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb61e000640 1 -- 172.21.15.125:0/523113324 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610034440 con 0x7fb600003970 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 -- 172.21.15.125:0/523113324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb600003970 msgr2=0x7fb600003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 --2- 172.21.15.125:0/523113324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb600003970 0x7fb600003d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb610009fc0 tx=0x7fb6000057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb61fe00640 1 -- 172.21.15.125:0/523113324 reap_dead start 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 -- 172.21.15.125:0/523113324 shutdown_connections 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 -- 172.21.15.125:0/523113324 >> 172.21.15.125:0/523113324 conn(0x7fb6200cf700 msgr2=0x7fb6200cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 -- 172.21.15.125:0/523113324 shutdown_connections 2024-01-16T15:01:57.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.682+0000 7fb626600640 1 -- 172.21.15.125:0/523113324 wait complete. 2024-01-16T15:01:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.684+0000 7fb626600640 1 Processor -- start 2024-01-16T15:01:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.684+0000 7fb626600640 1 -- start start 2024-01-16T15:01:57.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 0x7fb620145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 0x7fb6200b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200b43d0 con 0x7fb6200cd600 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6201583d0 con 0x7fb6200b3400 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.685+0000 7fb626600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200cdfa0 con 0x7fb600003970 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58598/0 (socket says 172.21.15.125:58598) 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 -- 172.21.15.125:0/363958939 learned_addr learned my addr 172.21.15.125:0/363958939 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61f400640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 0x7fb620145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61fe00640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 0x7fb6200b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 -- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 msgr2=0x7fb620145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 0x7fb620145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 -- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 msgr2=0x7fb6200b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 0x7fb6200b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 -- 172.21.15.125:0/363958939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb620146f10 con 0x7fb6200cd600 2024-01-16T15:01:57.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61f400640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 0x7fb620145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:57.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61fe00640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 0x7fb6200b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:57.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.686+0000 7fb61ea00640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb608010940 tx=0x7fb608010e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.687+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608009690 con 0x7fb6200cd600 2024-01-16T15:01:57.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.687+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60800ec20 con 0x7fb6200cd600 2024-01-16T15:01:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.687+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60801c400 con 0x7fb6200cd600 2024-01-16T15:01:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.687+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb620146410 con 0x7fb6200cd600 2024-01-16T15:01:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.687+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6201468c0 con 0x7fb6200cd600 2024-01-16T15:01:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.688+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb620146720 con 0x7fb6200cd600 2024-01-16T15:01:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.689+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb60801c560 con 0x7fb6200cd600 2024-01-16T15:01:57.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.690+0000 7fb617400640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 0x7fb5f806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.690+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb60808a960 con 0x7fb6200cd600 2024-01-16T15:01:57.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.690+0000 7fb61f400640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 0x7fb5f806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.691+0000 7fb61f400640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 0x7fb5f806dd50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb610036000 tx=0x7fb61002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.694+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb620146720 con 0x7fb6200cd600 2024-01-16T15:01:57.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:57.952+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7fb620146720 con 0x7fb6200cd600 2024-01-16T15:01:58.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.588+0000 7fb617400640 1 -- 172.21.15.125:0/363958939 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v266) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb620146720 con 0x7fb6200cd600 2024-01-16T15:01:58.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 msgr2=0x7fb5f806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 0x7fb5f806dd50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb610036000 tx=0x7fb61002e000 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 msgr2=0x7fb6200cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb608010940 tx=0x7fb608010e00 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 shutdown_connections 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6200b3400 0x7fb6200b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f806b890 0x7fb5f806dd50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6200cd600 0x7fb6200cda60 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 --2- 172.21.15.125:0/363958939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb600003970 0x7fb620145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 >> 172.21.15.125:0/363958939 conn(0x7fb6200cf700 msgr2=0x7fb6200b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 shutdown_connections 2024-01-16T15:01:58.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.593+0000 7fb626600640 1 -- 172.21.15.125:0/363958939 wait complete. 2024-01-16T15:01:58.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:01:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.829+0000 7fa712400640 1 Processor -- start 2024-01-16T15:01:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.829+0000 7fa712400640 1 -- start start 2024-01-16T15:01:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.830+0000 7fa712400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa70c13df80 0x7fa70c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa712400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa712400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c06c470 con 0x7fa70c13e360 2024-01-16T15:01:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa712400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70c06c750 con 0x7fa70c13df80 2024-01-16T15:01:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa712400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c06ca30 con 0x7fa70c0cf9a0 2024-01-16T15:01:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa70be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa702800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa70c13e360 0x7fa70c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:44664/0 (socket says 172.21.15.125:44664) 2024-01-16T15:01:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa70b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa70c13df80 0x7fa70c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.831+0000 7fa702800640 1 -- 172.21.15.125:0/617722522 learned_addr learned my addr 172.21.15.125:0/617722522 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.832+0000 7fa70b400640 1 -- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 msgr2=0x7fa70c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.832+0000 7fa70b400640 1 --2- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.832+0000 7fa70b400640 1 -- 172.21.15.125:0/617722522 >> v1:172.21.15.137:6789/0 conn(0x7fa70c13e360 legacy=0x7fa70c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.832+0000 7fa70b400640 1 -- 172.21.15.125:0/617722522 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c06cd10 con 0x7fa70c13df80 2024-01-16T15:01:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.834+0000 7fa70b400640 1 --2- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c15bee0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0097a0 tx=0x7fa6fc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce1f95743e9ef373 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.834+0000 7fa70aa00640 1 -- 172.21.15.125:0/617722522 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc02d040 con 0x7fa70c13df80 2024-01-16T15:01:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.834+0000 7fa70aa00640 1 -- 172.21.15.125:0/617722522 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc031810 con 0x7fa70c13df80 2024-01-16T15:01:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.834+0000 7fa70aa00640 1 -- 172.21.15.125:0/617722522 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc031ba0 con 0x7fa70c13df80 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 -- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 msgr2=0x7fa70c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 --2- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c15bee0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0097a0 tx=0x7fa6fc02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 -- 172.21.15.125:0/617722522 shutdown_connections 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 --2- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 --2- 172.21.15.125:0/617722522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c15bee0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 -- 172.21.15.125:0/617722522 >> 172.21.15.125:0/617722522 conn(0x7fa70c06a800 msgr2=0x7fa70c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 -- 172.21.15.125:0/617722522 shutdown_connections 2024-01-16T15:01:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.835+0000 7fa712400640 1 -- 172.21.15.125:0/617722522 wait complete. 2024-01-16T15:01:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.837+0000 7fa712400640 1 Processor -- start 2024-01-16T15:01:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.837+0000 7fa712400640 1 -- start start 2024-01-16T15:01:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c0b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c13cbe0 con 0x7fa70c13e360 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c13d3b0 con 0x7fa70c13df80 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa712400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c13a680 con 0x7fa70c0cf9a0 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.838+0000 7fa70be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58600/0 (socket says 172.21.15.125:58600) 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 -- 172.21.15.125:0/993191009 learned_addr learned my addr 172.21.15.125:0/993191009 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa702800640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c0b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70b400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 -- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 msgr2=0x7fa70c0b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 -- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 msgr2=0x7fa70c0b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c0b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 -- 172.21.15.125:0/993191009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc009400 con 0x7fa70c13e360 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.839+0000 7fa70be00640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa70400efc0 tx=0x7fa70400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.840+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704009b20 con 0x7fa70c13e360 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.840+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70400f040 con 0x7fa70c13e360 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.840+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040146c0 con 0x7fa70c13e360 2024-01-16T15:01:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.840+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c13cdf0 con 0x7fa70c13e360 2024-01-16T15:01:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.840+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c153150 con 0x7fa70c13e360 2024-01-16T15:01:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.841+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c144f10 con 0x7fa70c13e360 2024-01-16T15:01:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.842+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa704014820 con 0x7fa70c13e360 2024-01-16T15:01:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.843+0000 7fa703e00640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 0x7fa6e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.843+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa70408b1e0 con 0x7fa70c13e360 2024-01-16T15:01:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.843+0000 7fa70b400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 0x7fa6e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.844+0000 7fa70b400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 0x7fa6e406dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc036000 tx=0x7fa6fc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:58.847+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa70c144f10 con 0x7fa70c13e360 2024-01-16T15:01:59.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.039+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa70c144f10 con 0x7fa70c13e360 2024-01-16T15:01:59.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.046+0000 7fa703e00640 1 -- 172.21.15.125:0/993191009 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa70c144f10 con 0x7fa70c13e360 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 msgr2=0x7fa6e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 0x7fa6e406dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc036000 tx=0x7fa6fc03c040 comp rx=0 tx=0).stop 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 msgr2=0x7fa70c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa70400efc0 tx=0x7fa70400c490 comp rx=0 tx=0).stop 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 shutdown_connections 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e406b860 0x7fa6e406dd20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70c13e360 0x7fa70c0b21e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70c13df80 0x7fa70c0b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 --2- 172.21.15.125:0/993191009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70c0cf9a0 0x7fa70c0b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 >> 172.21.15.125:0/993191009 conn(0x7fa70c06a800 msgr2=0x7fa70c15a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:59.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.049+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 shutdown_connections 2024-01-16T15:01:59.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.050+0000 7fa712400640 1 -- 172.21.15.125:0/993191009 wait complete. 2024-01-16T15:01:59.049 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:01:59.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:01:59.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:01:59.269 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:01:59.269 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:01:59.269 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:59.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:01:59.326 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:01:59.326 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:59.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:01:59.382 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:01:59.382 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:59.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:01:59.439 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:01:59.439 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:59.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:01:59.494 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:01:59.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:59.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.738+0000 7fe538e00640 1 Processor -- start 2024-01-16T15:01:59.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.738+0000 7fe538e00640 1 -- start start 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.739+0000 7fe538e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5340cf9a0 0x7fe534070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.739+0000 7fe538e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe534070ea0 0x7fe5340cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.739+0000 7fe538e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534071690 con 0x7fe53414b2c0 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.739+0000 7fe538e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53414b6a0 con 0x7fe534070ea0 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.739+0000 7fe538e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe534071970 con 0x7fe5340cf9a0 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe531e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe53414b2c0 0x7fe5340cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42106/0 (socket says 172.21.15.125:42106) 2024-01-16T15:01:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe531e00640 1 -- 172.21.15.125:0/1524899857 learned_addr learned my addr 172.21.15.125:0/1524899857 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe532800640 1 --2- 172.21.15.125:0/1524899857 >> v2:172.21.15.125:3300/0 conn(0x7fe534070ea0 0x7fe5340cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe531400640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5340cf9a0 0x7fe534070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe530a00640 1 -- 172.21.15.125:0/1524899857 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210634040 0 0) 0x7fe534071690 con 0x7fe53414b2c0 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe530a00640 1 -- 172.21.15.125:0/1524899857 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514003520 con 0x7fe53414b2c0 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe532800640 1 -- 172.21.15.125:0/1524899857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5340cf9a0 msgr2=0x7fe534070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe532800640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5340cf9a0 0x7fe534070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe532800640 1 -- 172.21.15.125:0/1524899857 >> v1:172.21.15.125:6789/0 conn(0x7fe53414b2c0 legacy=0x7fe5340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.740+0000 7fe532800640 1 -- 172.21.15.125:0/1524899857 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534071c50 con 0x7fe534070ea0 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe532800640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534070ea0 0x7fe5340cc160 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009d90 tx=0x7fe52c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b975d104e60642a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe530a00640 1 -- 172.21.15.125:0/1524899857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c02cc60 con 0x7fe534070ea0 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe530a00640 1 -- 172.21.15.125:0/1524899857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c02cdc0 con 0x7fe534070ea0 2024-01-16T15:01:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe530a00640 1 -- 172.21.15.125:0/1524899857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c034660 con 0x7fe534070ea0 2024-01-16T15:01:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe538e00640 1 -- 172.21.15.125:0/1524899857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534070ea0 msgr2=0x7fe5340cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.742+0000 7fe538e00640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534070ea0 0x7fe5340cc160 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009d90 tx=0x7fe52c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 -- 172.21.15.125:0/1524899857 shutdown_connections 2024-01-16T15:01:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534070ea0 0x7fe5340cc160 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 --2- 172.21.15.125:0/1524899857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5340cf9a0 0x7fe534070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 -- 172.21.15.125:0/1524899857 >> 172.21.15.125:0/1524899857 conn(0x7fe53406a800 msgr2=0x7fe53406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 -- 172.21.15.125:0/1524899857 shutdown_connections 2024-01-16T15:01:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.743+0000 7fe538e00640 1 -- 172.21.15.125:0/1524899857 wait complete. 2024-01-16T15:01:59.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.744+0000 7fe538e00640 1 Processor -- start 2024-01-16T15:01:59.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.744+0000 7fe538e00640 1 -- start start 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe534070ea0 0x7fe5341fb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5340cf9a0 0x7fe534161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5341623f0 con 0x7fe53414b2c0 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5341520f0 con 0x7fe5340cf9a0 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.745+0000 7fe538e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe534152390 con 0x7fe534070ea0 2024-01-16T15:01:59.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58614/0 (socket says 172.21.15.125:58614) 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 -- 172.21.15.125:0/1054511458 learned_addr learned my addr 172.21.15.125:0/1054511458 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe531400640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5340cf9a0 0x7fe534161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe531e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe534070ea0 0x7fe5341fb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 -- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe534070ea0 msgr2=0x7fe5341fb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe534070ea0 0x7fe5341fb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 -- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5340cf9a0 msgr2=0x7fe534161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5340cf9a0 0x7fe534161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 -- 172.21.15.125:0/1054511458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c009a70 con 0x7fe53414b2c0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.746+0000 7fe532800640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009d90 tx=0x7fe52c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.747+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c040650 con 0x7fe53414b2c0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.747+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c0407b0 con 0x7fe53414b2c0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.747+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c040b60 con 0x7fe53414b2c0 2024-01-16T15:01:59.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.747+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe534152670 con 0x7fe53414b2c0 2024-01-16T15:01:59.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.747+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe534162620 con 0x7fe53414b2c0 2024-01-16T15:01:59.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.748+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534152b90 con 0x7fe53414b2c0 2024-01-16T15:01:59.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.749+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe52c040cc0 con 0x7fe53414b2c0 2024-01-16T15:01:59.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.750+0000 7fe52a000640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 0x7fe50c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.750+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe52c0ad0a0 con 0x7fe53414b2c0 2024-01-16T15:01:59.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.750+0000 7fe531e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 0x7fe50c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.751+0000 7fe531e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 0x7fe50c06dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5240011e0 tx=0x7fe524009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.754+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe52c0779a0 con 0x7fe53414b2c0 2024-01-16T15:01:59.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.981+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5340675d0 con 0x7fe53414b2c0 2024-01-16T15:01:59.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.982+0000 7fe52a000640 1 -- 172.21.15.125:0/1054511458 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+3711 (secure 0 0 0) 0x7fe52c044e90 con 0x7fe53414b2c0 2024-01-16T15:01:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:01:59.963 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":31,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/3050283375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3050283375},{"type":"v1","addr":"172.21.15.137:6833","nonce":3050283375}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4258955541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4258955541},{"type":"v1","addr":"172.21.15.137:6835","nonce":4258955541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5739,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1745104106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1745104106},{"type":"v1","addr":"172.21.15.125:6836","nonce":1745104106}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6237,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/4055907802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":4055907802},{"type":"v1","addr":"172.21.15.125:6837","nonce":4055907802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[]} 2024-01-16T15:01:59.963 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 31 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 msgr2=0x7fe50c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 0x7fe50c06dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5240011e0 tx=0x7fe524009040 comp rx=0 tx=0).stop 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 msgr2=0x7fe534142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009d90 tx=0x7fe52c004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 shutdown_connections 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe50c06b860 0x7fe50c06dd20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe53414b2c0 0x7fe534142580 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5340cf9a0 0x7fe534161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 --2- 172.21.15.125:0/1054511458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe534070ea0 0x7fe5341fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.984+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 >> 172.21.15.125:0/1054511458 conn(0x7fe53406a800 msgr2=0x7fe53406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:59.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.985+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 shutdown_connections 2024-01-16T15:01:59.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:01:59.985+0000 7fe538e00640 1 -- 172.21.15.125:0/1054511458 wait complete. 2024-01-16T15:01:59.985 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:01:59.985 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:01:59.985 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:01:59.985 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:59.985 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:01:59.985 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:01:59.985 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:01:59.985 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:59.986 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:01:59.986 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:01:59.986 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:01:59.986 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:59.986 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:01:59.986 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:01:59.986 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:01:59.986 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:00.007 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:02:00.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:02:00.008 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:02:00.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:02:00.010 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:02:00.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:02:00.013 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:02:00.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:02:00.057 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:02:00.057 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:00.061 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:02:00.061 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:00.064 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:02:00.064 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:00.067 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:02:00.067 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:00.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:02:00.211 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:02:00.219 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T15:02:00.227 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:02:00.230 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T15:02:00.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.275+0000 7f737ce00640 1 Processor -- start 2024-01-16T15:02:00.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.275+0000 7f737ce00640 1 -- start start 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.276+0000 7f737ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f73780cf9a0 0x7f7378166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.276+0000 7f737ce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737815c5f0 con 0x7f737813df80 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.276+0000 7f737ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737806c470 con 0x7f737813e360 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.276+0000 7f737ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737806c750 con 0x7f73780cf9a0 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7375e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f737813df80 0x7f737815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42112/0 (socket says 172.21.15.125:42112) 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7375e00640 1 -- 172.21.15.125:0/2713912688 learned_addr learned my addr 172.21.15.125:0/2713912688 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:00.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7376800640 1 --2- 172.21.15.125:0/2713912688 >> v2:172.21.15.137:3300/0 conn(0x7f73780cf9a0 0x7f7378166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434141030 0 0) 0x7f737815c5f0 con 0x7f737813df80 2024-01-16T15:02:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7358003520 con 0x7f737813df80 2024-01-16T15:02:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968826100 0 0) 0x7f737806c470 con 0x7f737813e360 2024-01-16T15:02:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7376800640 1 -- 172.21.15.125:0/2713912688 >> v1:172.21.15.137:6789/0 conn(0x7f737813e360 legacy=0x7f73780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7376800640 1 -- 172.21.15.125:0/2713912688 >> v1:172.21.15.125:6789/0 conn(0x7f737813df80 legacy=0x7f737815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.277+0000 7f7376800640 1 -- 172.21.15.125:0/2713912688 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737806ca30 con 0x7f73780cf9a0 2024-01-16T15:02:00.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f7376800640 1 --2- 172.21.15.125:0/2713912688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73780cf9a0 0x7f7378166030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7370009d90 tx=0x7f737002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=929afb33af5a594f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737002cd00 con 0x7f73780cf9a0 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737002ce60 con 0x7f73780cf9a0 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f7374a00640 1 -- 172.21.15.125:0/2713912688 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370034660 con 0x7f73780cf9a0 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f737ce00640 1 -- 172.21.15.125:0/2713912688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73780cf9a0 msgr2=0x7f7378166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.279+0000 7f737ce00640 1 --2- 172.21.15.125:0/2713912688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73780cf9a0 0x7f7378166030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7370009d90 tx=0x7f737002bf50 comp rx=0 tx=0).stop 2024-01-16T15:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.280+0000 7f737ce00640 1 -- 172.21.15.125:0/2713912688 shutdown_connections 2024-01-16T15:02:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.280+0000 7f737ce00640 1 --2- 172.21.15.125:0/2713912688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73780cf9a0 0x7f7378166030 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.280+0000 7f737ce00640 1 -- 172.21.15.125:0/2713912688 >> 172.21.15.125:0/2713912688 conn(0x7f737806a800 msgr2=0x7f737806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.280+0000 7f737ce00640 1 -- 172.21.15.125:0/2713912688 shutdown_connections 2024-01-16T15:02:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.280+0000 7f737ce00640 1 -- 172.21.15.125:0/2713912688 wait complete. 2024-01-16T15:02:00.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.281+0000 7f737ce00640 1 Processor -- start 2024-01-16T15:02:00.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.282+0000 7f737ce00640 1 -- start start 2024-01-16T15:02:00.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.282+0000 7f737ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.282+0000 7f737ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f737813df80 0x7f73780b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f737ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f737813e360 0x7f7378135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f737ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780b2b40 con 0x7f73780cf9a0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f737ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780b2e20 con 0x7f737813df80 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f737ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73780b3100 con 0x7f737813e360 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58652/0 (socket says 172.21.15.125:58652) 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 -- 172.21.15.125:0/1007976147 learned_addr learned my addr 172.21.15.125:0/1007976147 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7376800640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f737813e360 0x7f7378135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375400640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f737813df80 0x7f73780b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 -- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f737813e360 msgr2=0x7f7378135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f737813e360 0x7f7378135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 -- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f737813df80 msgr2=0x7f73780b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f737813df80 0x7f73780b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 -- 172.21.15.125:0/1007976147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009a70 con 0x7f73780cf9a0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.283+0000 7f7375e00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f736800c410 tx=0x7f736800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.284+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368005ce0 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.284+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7368005e40 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.284+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73680072f0 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.284+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73780b33e0 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.284+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737814d910 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.285+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73780675d0 con 0x7f73780cf9a0 2024-01-16T15:02:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.286+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7368007510 con 0x7f73780cf9a0 2024-01-16T15:02:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.287+0000 7f736e000640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 0x7f735006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.287+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7368089970 con 0x7f73780cf9a0 2024-01-16T15:02:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.287+0000 7f7375400640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 0x7f735006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.288+0000 7f7375400640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 0x7f735006dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f737814d060 tx=0x7f7360009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.291+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7368054380 con 0x7f73780cf9a0 2024-01-16T15:02:00.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.489+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f737806df00 con 0x7f73780cf9a0 2024-01-16T15:02:00.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.616+0000 7f736e000640 1 -- 172.21.15.125:0/1007976147 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7f73680579a0 con 0x7f73780cf9a0 2024-01-16T15:02:00.597 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T15:02:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.619+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 msgr2=0x7f735006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.619+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 0x7f735006dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f737814d060 tx=0x7f7360009040 comp rx=0 tx=0).stop 2024-01-16T15:02:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.619+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 msgr2=0x7f737814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.619+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f736800c410 tx=0x7f736800c8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 shutdown_connections 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f737813e360 0x7f7378135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f737813df80 0x7f73780b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f735006b860 0x7f735006dd20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 --2- 172.21.15.125:0/1007976147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73780cf9a0 0x7f737814bc60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 >> 172.21.15.125:0/1007976147 conn(0x7f737806a800 msgr2=0x7f737815d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 shutdown_connections 2024-01-16T15:02:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.620+0000 7f737ce00640 1 -- 172.21.15.125:0/1007976147 wait complete. 2024-01-16T15:02:00.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:02:00.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.835+0000 7f3b4ea00640 1 Processor -- start 2024-01-16T15:02:00.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.835+0000 7f3b4ea00640 1 -- start start 2024-01-16T15:02:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b4ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b480ac8d0 0x7f3b480aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b4ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b48071de0 con 0x7f3b4806a3e0 2024-01-16T15:02:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b4ea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b48072030 con 0x7f3b480d0b60 2024-01-16T15:02:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b4ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b480acd60 con 0x7f3b480ac8d0 2024-01-16T15:02:00.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b47e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b4806a3e0 0x7f3b4806a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42124/0 (socket says 172.21.15.125:42124) 2024-01-16T15:02:00.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b47e00640 1 -- 172.21.15.125:0/566993245 learned_addr learned my addr 172.21.15.125:0/566993245 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5730721 0 0) 0x7f3b48071de0 con 0x7f3b4806a3e0 2024-01-16T15:02:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.837+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b28003520 con 0x7f3b4806a3e0 2024-01-16T15:02:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b4ca00640 1 --2- 172.21.15.125:0/566993245 >> v2:172.21.15.137:3300/0 conn(0x7f3b480ac8d0 0x7f3b480aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196625943 0 0) 0x7f3b28003520 con 0x7f3b4806a3e0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 >> v2:172.21.15.137:3300/0 conn(0x7f3b480ac8d0 msgr2=0x7f3b480aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 --2- 172.21.15.125:0/566993245 >> v2:172.21.15.137:3300/0 conn(0x7f3b480ac8d0 0x7f3b480aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 >> v1:172.21.15.137:6789/0 conn(0x7f3b480d0b60 legacy=0x7f3b4806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b48072310 con 0x7f3b4806a3e0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3b38001670 con 0x7f3b4806a3e0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 >> v1:172.21.15.125:6789/0 conn(0x7f3b4806a3e0 legacy=0x7f3b4806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 0x7f3b28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280047c0 0x7f3b28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b280051d0 0x7f3b2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480acd60 con 0x7f3b280039c0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2800c100 con 0x7f3b280051d0 2024-01-16T15:02:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2800c2c0 con 0x7f3b280047c0 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.838+0000 7f3b47e00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 0x7f3b28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 -- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280047c0 msgr2=0x7f3b28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280047c0 0x7f3b28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 -- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b280051d0 msgr2=0x7f3b2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b280051d0 0x7f3b2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b4ca00640 1 -- 172.21.15.125:0/566993245 reap_dead start 2024-01-16T15:02:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 -- 172.21.15.125:0/566993245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28003520 con 0x7f3b280039c0 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.839+0000 7f3b47e00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 0x7f3b28003dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c160 tx=0x7f3b280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7333a082b8112d48 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38005ad0 con 0x7f3b280039c0 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38005c30 con 0x7f3b280039c0 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b46a00640 1 -- 172.21.15.125:0/566993245 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3802c7e0 con 0x7f3b280039c0 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 -- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 msgr2=0x7f3b28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 0x7f3b28003dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c160 tx=0x7f3b280057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 -- 172.21.15.125:0/566993245 shutdown_connections 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/566993245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b280039c0 0x7f3b28003dc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 -- 172.21.15.125:0/566993245 >> 172.21.15.125:0/566993245 conn(0x7f3b480cfef0 msgr2=0x7f3b480d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 -- 172.21.15.125:0/566993245 shutdown_connections 2024-01-16T15:02:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.840+0000 7f3b4ea00640 1 -- 172.21.15.125:0/566993245 wait complete. 2024-01-16T15:02:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.842+0000 7f3b4ea00640 1 Processor -- start 2024-01-16T15:02:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.842+0000 7f3b4ea00640 1 -- start start 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280039c0 0x7f3b48151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b48151ce0 0x7f3b480ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4815ff30 con 0x7f3b4815f570 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480cf810 con 0x7f3b48151ce0 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480ce770 con 0x7f3b280039c0 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58666/0 (socket says 172.21.15.125:58666) 2024-01-16T15:02:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.843+0000 7f3b4ca00640 1 -- 172.21.15.125:0/2375492824 learned_addr learned my addr 172.21.15.125:0/2375492824 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b47e00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280039c0 0x7f3b48151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b47400640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b48151ce0 0x7f3b480ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 -- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280039c0 msgr2=0x7f3b48151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280039c0 0x7f3b48151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 -- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b48151ce0 msgr2=0x7f3b480ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b48151ce0 0x7f3b480ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 -- 172.21.15.125:0/2375492824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480cea50 con 0x7f3b4815f570 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ca00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3b4806cfa0 tx=0x7f3b4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4000ed90 con 0x7f3b4815f570 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b40014420 con 0x7f3b4815f570 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400147d0 con 0x7f3b4815f570 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.844+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b480ced30 con 0x7f3b4815f570 2024-01-16T15:02:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.845+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b480cf2f0 con 0x7f3b4815f570 2024-01-16T15:02:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.846+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4815b2e0 con 0x7f3b4815f570 2024-01-16T15:02:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.846+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f3b40014930 con 0x7f3b4815f570 2024-01-16T15:02:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.847+0000 7f3b3fe00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 0x7f3b2006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.847+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b4008a930 con 0x7f3b4815f570 2024-01-16T15:02:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.848+0000 7f3b47e00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 0x7f3b2006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.848+0000 7f3b47e00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 0x7f3b2006dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b38034000 tx=0x7f3b38009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:00.852+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b40055260 con 0x7f3b4815f570 2024-01-16T15:02:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.109+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3b480675d0 con 0x7f3b4815f570 2024-01-16T15:02:01.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.110+0000 7f3b3fe00640 1 -- 172.21.15.125:0/2375492824 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3b40058880 con 0x7f3b4815f570 2024-01-16T15:02:01.090 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:01.090 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:02:01.090 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.091 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.093 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T15:02:01.094 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.095 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.097 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T15:02:01.098 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:02:01.099 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.100 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:01.101 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T15:02:01.102 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 msgr2=0x7f3b2006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 0x7f3b2006dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b38034000 tx=0x7f3b38009030 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 msgr2=0x7f3b4815f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3b4806cfa0 tx=0x7f3b4000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 shutdown_connections 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b2006b860 0x7f3b2006dd20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b4815f570 0x7f3b4815f9f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b48151ce0 0x7f3b480ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 --2- 172.21.15.125:0/2375492824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b280039c0 0x7f3b48151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 >> 172.21.15.125:0/2375492824 conn(0x7f3b480cfef0 msgr2=0x7f3b4806d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.112+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 shutdown_connections 2024-01-16T15:02:01.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.113+0000 7f3b4ea00640 1 -- 172.21.15.125:0/2375492824 wait complete. 2024-01-16T15:02:01.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:01.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.337+0000 7f7ef8200640 1 Processor -- start 2024-01-16T15:02:01.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.337+0000 7f7ef8200640 1 -- start start 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef8200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ef013e360 0x7f7ef00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef8200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef015c5f0 con 0x7f7ef013df80 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef8200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef006c470 con 0x7f7ef015f9c0 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef8200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef006c750 con 0x7f7ef013e360 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef4e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ef013e360 0x7f7ef00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef4e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ef013e360 0x7f7ef00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58676/0 (socket says 172.21.15.125:58676) 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef4e00640 1 -- 172.21.15.125:0/2874175803 learned_addr learned my addr 172.21.15.125:0/2874175803 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:01.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.339+0000 7f7ef5800640 1 --1- 172.21.15.125:0/2874175803 >> v1:172.21.15.125:6789/0 conn(0x7f7ef013df80 0x7f7ef015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42136/0 (socket says 172.21.15.125:42136) 2024-01-16T15:02:01.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.340+0000 7f7ef4e00640 1 -- 172.21.15.125:0/2874175803 >> v1:172.21.15.137:6789/0 conn(0x7f7ef015f9c0 legacy=0x7f7ef00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:01.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.340+0000 7f7ef4e00640 1 -- 172.21.15.125:0/2874175803 >> v1:172.21.15.125:6789/0 conn(0x7f7ef013df80 legacy=0x7f7ef015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.340+0000 7f7ef4e00640 1 -- 172.21.15.125:0/2874175803 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef006ca30 con 0x7f7ef013e360 2024-01-16T15:02:01.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.341+0000 7f7ef4e00640 1 --2- 172.21.15.125:0/2874175803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef013e360 0x7f7ef00cf290 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009870 tx=0x7f7ed802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741c33b3fe9dcd60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.341+0000 7f7eefe00640 1 -- 172.21.15.125:0/2874175803 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8030860 con 0x7f7ef013e360 2024-01-16T15:02:01.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.341+0000 7f7eefe00640 1 -- 172.21.15.125:0/2874175803 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed80309c0 con 0x7f7ef013e360 2024-01-16T15:02:01.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.341+0000 7f7eefe00640 1 -- 172.21.15.125:0/2874175803 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8030d50 con 0x7f7ef013e360 2024-01-16T15:02:01.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 -- 172.21.15.125:0/2874175803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef013e360 msgr2=0x7f7ef00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 --2- 172.21.15.125:0/2874175803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef013e360 0x7f7ef00cf290 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009870 tx=0x7f7ed802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 -- 172.21.15.125:0/2874175803 shutdown_connections 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 --2- 172.21.15.125:0/2874175803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef013e360 0x7f7ef00cf290 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 -- 172.21.15.125:0/2874175803 >> 172.21.15.125:0/2874175803 conn(0x7f7ef006a800 msgr2=0x7f7ef006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 -- 172.21.15.125:0/2874175803 shutdown_connections 2024-01-16T15:02:01.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.342+0000 7f7ef8200640 1 -- 172.21.15.125:0/2874175803 wait complete. 2024-01-16T15:02:01.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.344+0000 7f7ef8200640 1 Processor -- start 2024-01-16T15:02:01.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.344+0000 7f7ef8200640 1 -- start start 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ef013df80 0x7f7ef0150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ef013e360 0x7f7ef014dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef01f7440 con 0x7f7ef015f9c0 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0135160 con 0x7f7ef013df80 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef8200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef01f6aa0 con 0x7f7ef013e360 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58692/0 (socket says 172.21.15.125:58692) 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 -- 172.21.15.125:0/246810577 learned_addr learned my addr 172.21.15.125:0/246810577 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef5800640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ef013df80 0x7f7ef0150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 -- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ef013e360 msgr2=0x7f7ef014dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ef013e360 0x7f7ef014dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 -- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ef013df80 msgr2=0x7f7ef0150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ef013df80 0x7f7ef0150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.346+0000 7f7ef6200640 1 -- 172.21.15.125:0/246810577 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80094d0 con 0x7f7ef015f9c0 2024-01-16T15:02:01.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7ef6200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c360 tx=0x7f7ee800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80035f0 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8003750 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef01f6d80 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8003b40 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.347+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef01f7860 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.348+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef0067fd0 con 0x7f7ef015f9c0 2024-01-16T15:02:01.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.349+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7ee8003ca0 con 0x7f7ef015f9c0 2024-01-16T15:02:01.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.350+0000 7f7eed600640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 0x7f7ec806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.351+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7ee80898f0 con 0x7f7ef015f9c0 2024-01-16T15:02:01.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.351+0000 7f7ef5800640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 0x7f7ec806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.351+0000 7f7ef5800640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 0x7f7ec806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00012e0 tx=0x7f7ee0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.355+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ee8054290 con 0x7f7ef015f9c0 2024-01-16T15:02:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.547+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ef01f7580 con 0x7f7ef015f9c0 2024-01-16T15:02:01.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.549+0000 7f7eed600640 1 -- 172.21.15.125:0/246810577 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f7ef01f7580 con 0x7f7ef015f9c0 2024-01-16T15:02:01.530 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:01.530 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":54,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:02:00.634607+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:01.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.551+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 msgr2=0x7f7ec806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.551+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 0x7f7ec806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00012e0 tx=0x7f7ee0009040 comp rx=0 tx=0).stop 2024-01-16T15:02:01.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.551+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 msgr2=0x7f7ef014eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.551+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c360 tx=0x7f7ee800c820 comp rx=0 tx=0).stop 2024-01-16T15:02:01.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 shutdown_connections 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7ec806b860 0x7f7ec806dd20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ef015f9c0 0x7f7ef014eda0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ef013e360 0x7f7ef014dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 --2- 172.21.15.125:0/246810577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ef013df80 0x7f7ef0150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 >> 172.21.15.125:0/246810577 conn(0x7f7ef006a800 msgr2=0x7f7ef0070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 shutdown_connections 2024-01-16T15:02:01.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.552+0000 7f7ef8200640 1 -- 172.21.15.125:0/246810577 wait complete. 2024-01-16T15:02:01.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:01.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:01.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.787+0000 7f3356800640 1 Processor -- start 2024-01-16T15:02:01.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.788+0000 7f3356800640 1 -- start start 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3356800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 0x7f33500cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3356800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33500cd7e0 0x7f33500cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3356800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33500cb660 con 0x7f33500b2480 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3356800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33500ce1b0 con 0x7f33500cd7e0 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3356800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33500cb450 con 0x7f33500b3c50 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.789+0000 7f3355e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33500cd7e0 0x7f33500cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33500cd7e0 0x7f33500cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58696/0 (socket says 172.21.15.125:58696) 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 -- 172.21.15.125:0/4204024080 learned_addr learned my addr 172.21.15.125:0/4204024080 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3354a00640 1 --2- 172.21.15.125:0/4204024080 >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 0x7f33500cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 -- 172.21.15.125:0/4204024080 >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 msgr2=0x7f33500cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 --2- 172.21.15.125:0/4204024080 >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 0x7f33500cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 -- 172.21.15.125:0/4204024080 >> v1:172.21.15.137:6789/0 conn(0x7f33500b2480 legacy=0x7f33500d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:01.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3355e00640 1 -- 172.21.15.125:0/4204024080 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33500cb940 con 0x7f33500cd7e0 2024-01-16T15:02:01.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.790+0000 7f3354a00640 1 --2- 172.21.15.125:0/4204024080 >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 0x7f33500cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.791+0000 7f3355e00640 1 --2- 172.21.15.125:0/4204024080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500cdc70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f33500714d0 tx=0x7f334802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4578293911d56355 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f334fe00640 1 -- 172.21.15.125:0/4204024080 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33480317e0 con 0x7f33500cd7e0 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f334fe00640 1 -- 172.21.15.125:0/4204024080 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348031940 con 0x7f33500cd7e0 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f334fe00640 1 -- 172.21.15.125:0/4204024080 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348031cd0 con 0x7f33500cd7e0 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 -- 172.21.15.125:0/4204024080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 msgr2=0x7f33500cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 --2- 172.21.15.125:0/4204024080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500cdc70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f33500714d0 tx=0x7f334802b880 comp rx=0 tx=0).stop 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 -- 172.21.15.125:0/4204024080 shutdown_connections 2024-01-16T15:02:01.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 --2- 172.21.15.125:0/4204024080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500cdc70 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 --2- 172.21.15.125:0/4204024080 >> v2:172.21.15.137:3300/0 conn(0x7f33500b3c50 0x7f33500cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 -- 172.21.15.125:0/4204024080 >> 172.21.15.125:0/4204024080 conn(0x7f33500d0e00 msgr2=0x7f33500d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 -- 172.21.15.125:0/4204024080 shutdown_connections 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.792+0000 7f3356800640 1 -- 172.21.15.125:0/4204024080 wait complete. 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.794+0000 7f3356800640 1 Processor -- start 2024-01-16T15:02:01.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.794+0000 7f3356800640 1 -- start start 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 0x7f33501f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33500b3c50 0x7f335014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33501e1b40 con 0x7f33500cd7e0 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335014e580 con 0x7f33500b2480 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3356800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33501e1d90 con 0x7f33500b3c50 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3355e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3355e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58700/0 (socket says 172.21.15.125:58700) 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.795+0000 7f3355e00640 1 -- 172.21.15.125:0/3090734848 learned_addr learned my addr 172.21.15.125:0/3090734848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3354a00640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33500b3c50 0x7f335014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355400640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 0x7f33501f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 -- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33500b3c50 msgr2=0x7f335014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33500b3c50 0x7f335014e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 -- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 msgr2=0x7f33501f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 0x7f33501f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 -- 172.21.15.125:0/3090734848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480094d0 con 0x7f33500cd7e0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355400640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 0x7f33501f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f3355e00640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f33480090b0 tx=0x7f33480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.796+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348032880 con 0x7f33500cd7e0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.797+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348032a00 con 0x7f33500cd7e0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.797+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348032db0 con 0x7f33500cd7e0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.797+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33501f6360 con 0x7f33500cd7e0 2024-01-16T15:02:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.797+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33501f6920 con 0x7f33500cd7e0 2024-01-16T15:02:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.798+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33501f6b40 con 0x7f33500cd7e0 2024-01-16T15:02:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.799+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f334804e450 con 0x7f33500cd7e0 2024-01-16T15:02:01.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.799+0000 7f334d600640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 0x7f332806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.800+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33480ac670 con 0x7f33500cd7e0 2024-01-16T15:02:01.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.800+0000 7f3355400640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 0x7f332806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.801+0000 7f3355400640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 0x7f332806dd50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3338003eb0 tx=0x7f3338009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.804+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3348076e80 con 0x7f33500cd7e0 2024-01-16T15:02:01.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.995+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3350067b60 con 0x7f33500cd7e0 2024-01-16T15:02:01.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.996+0000 7f334d600640 1 -- 172.21.15.125:0/3090734848 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f334807a4a0 con 0x7f33500cd7e0 2024-01-16T15:02:01.977 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:02:01.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.998+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 msgr2=0x7f332806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.998+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 0x7f332806dd50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3338003eb0 tx=0x7f3338009040 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 msgr2=0x7f33500d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f33480090b0 tx=0x7f33480046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 shutdown_connections 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f332806b890 0x7f332806dd50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33500cd7e0 0x7f33500d11e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33500b3c50 0x7f335014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 --2- 172.21.15.125:0/3090734848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33500b2480 0x7f33501f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 >> 172.21.15.125:0/3090734848 conn(0x7f33500d0e00 msgr2=0x7f335006e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 shutdown_connections 2024-01-16T15:02:01.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:01.999+0000 7f3356800640 1 -- 172.21.15.125:0/3090734848 wait complete. 2024-01-16T15:02:01.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:02:02.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.226+0000 7f247a000640 1 Processor -- start 2024-01-16T15:02:02.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.227+0000 7f247a000640 1 -- start start 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.228+0000 7f247a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2474143ae0 0x7f24740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.228+0000 7f247a000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247415e3b0 0x7f24740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.228+0000 7f247a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740cfc30 con 0x7f24740cf7d0 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.228+0000 7f247a000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740b1ee0 con 0x7f247415e3b0 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.228+0000 7f247a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740b2780 con 0x7f2474143ae0 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247415e3b0 0x7f24740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f247415e3b0 0x7f24740702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58708/0 (socket says 172.21.15.125:58708) 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 -- 172.21.15.125:0/658680487 learned_addr learned my addr 172.21.15.125:0/658680487 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473400640 1 --2- 172.21.15.125:0/658680487 >> v2:172.21.15.137:3300/0 conn(0x7f2474143ae0 0x7f24740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 -- 172.21.15.125:0/658680487 >> v2:172.21.15.137:3300/0 conn(0x7f2474143ae0 msgr2=0x7f24740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 --2- 172.21.15.125:0/658680487 >> v2:172.21.15.137:3300/0 conn(0x7f2474143ae0 0x7f24740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 -- 172.21.15.125:0/658680487 >> v1:172.21.15.137:6789/0 conn(0x7f24740cf7d0 legacy=0x7f247415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.229+0000 7f2473e00640 1 -- 172.21.15.125:0/658680487 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247415e990 con 0x7f247415e3b0 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.230+0000 7f2473e00640 1 --2- 172.21.15.125:0/658680487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247415e3b0 0x7f24740702c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f24740cfea0 tx=0x7f246c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c35736ec19134b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f2472a00640 1 -- 172.21.15.125:0/658680487 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c02ea90 con 0x7f247415e3b0 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f2472a00640 1 -- 172.21.15.125:0/658680487 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c02ebf0 con 0x7f247415e3b0 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f2472a00640 1 -- 172.21.15.125:0/658680487 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c009600 con 0x7f247415e3b0 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 -- 172.21.15.125:0/658680487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247415e3b0 msgr2=0x7f24740702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 --2- 172.21.15.125:0/658680487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247415e3b0 0x7f24740702c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f24740cfea0 tx=0x7f246c02e0f0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 -- 172.21.15.125:0/658680487 shutdown_connections 2024-01-16T15:02:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 --2- 172.21.15.125:0/658680487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f247415e3b0 0x7f24740702c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 --2- 172.21.15.125:0/658680487 >> v2:172.21.15.137:3300/0 conn(0x7f2474143ae0 0x7f24740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 -- 172.21.15.125:0/658680487 >> 172.21.15.125:0/658680487 conn(0x7f247406a800 msgr2=0x7f247406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.231+0000 7f247a000640 1 -- 172.21.15.125:0/658680487 shutdown_connections 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.232+0000 7f247a000640 1 -- 172.21.15.125:0/658680487 wait complete. 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.233+0000 7f247a000640 1 Processor -- start 2024-01-16T15:02:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.233+0000 7f247a000640 1 -- start start 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 0x7f2474140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 0x7f2474141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247413f140 con 0x7f24740cf7d0 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247413ffa0 con 0x7f247415e3b0 2024-01-16T15:02:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f247a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247413f390 con 0x7f2474143ae0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.234+0000 7f2473400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58718/0 (socket says 172.21.15.125:58718) 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 -- 172.21.15.125:0/725915868 learned_addr learned my addr 172.21.15.125:0/725915868 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f246be00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 0x7f2474140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 -- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 msgr2=0x7f2474140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473e00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 0x7f2474141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 0x7f2474140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 -- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 msgr2=0x7f2474141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 0x7f2474141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 -- 172.21.15.125:0/725915868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c00bd40 con 0x7f24740cf7d0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f246be00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 0x7f2474140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473e00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 0x7f2474141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.235+0000 7f2473400640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f246400d800 tx=0x7f246400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.236+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246400b6c0 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.236+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246400b820 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.236+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246400bbb0 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.236+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247413f670 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.236+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247413fc30 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.237+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247413fdc0 con 0x7f24740cf7d0 2024-01-16T15:02:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.238+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f246400bd10 con 0x7f24740cf7d0 2024-01-16T15:02:02.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.239+0000 7f246b400640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.239+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f246408a890 con 0x7f24740cf7d0 2024-01-16T15:02:02.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.239+0000 7f246be00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.239+0000 7f246be00640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 0x7f244c06dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2474140350 tx=0x7f245c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.243+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24640551e0 con 0x7f24740cf7d0 2024-01-16T15:02:02.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.435+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f24740675d0 con 0x7f24740cf7d0 2024-01-16T15:02:02.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.436+0000 7f246b400640 1 -- 172.21.15.125:0/725915868 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2464058800 con 0x7f24740cf7d0 2024-01-16T15:02:02.416 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:02.417 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:02:02.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.438+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 msgr2=0x7f244c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.438+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 0x7f244c06dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2474140350 tx=0x7f245c009040 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 msgr2=0x7f24740adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f246400d800 tx=0x7f246400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 shutdown_connections 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f247415e3b0 0x7f2474141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2474143ae0 0x7f2474140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 --2- 172.21.15.125:0/725915868 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24740cf7d0 0x7f24740adb90 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 >> 172.21.15.125:0/725915868 conn(0x7f247406a800 msgr2=0x7f247415b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 shutdown_connections 2024-01-16T15:02:02.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.439+0000 7f247a000640 1 -- 172.21.15.125:0/725915868 wait complete. 2024-01-16T15:02:02.436 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:02:02.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:02:02.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.664+0000 7f44ffe00640 1 Processor -- start 2024-01-16T15:02:02.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.664+0000 7f44ffe00640 1 -- start start 2024-01-16T15:02:02.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44ffe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44f80cf9a0 0x7f44f8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44ffe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f8071690 con 0x7f44f8070ea0 2024-01-16T15:02:02.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44ffe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f8071970 con 0x7f44f814b2c0 2024-01-16T15:02:02.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44ffe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f814b6a0 con 0x7f44f80cf9a0 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44fde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f44f8070ea0 0x7f44f80cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42148/0 (socket says 172.21.15.125:42148) 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.666+0000 7f44fde00640 1 -- 172.21.15.125:0/3946024704 learned_addr learned my addr 172.21.15.125:0/3946024704 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44fca00640 1 --2- 172.21.15.125:0/3946024704 >> v2:172.21.15.137:3300/0 conn(0x7f44f80cf9a0 0x7f44f8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202913427 0 0) 0x7f44f8071690 con 0x7f44f8070ea0 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d8003520 con 0x7f44f8070ea0 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643398574 0 0) 0x7f44d8003520 con 0x7f44f8070ea0 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 >> v2:172.21.15.137:3300/0 conn(0x7f44f80cf9a0 msgr2=0x7f44f8070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 --2- 172.21.15.125:0/3946024704 >> v2:172.21.15.137:3300/0 conn(0x7f44f80cf9a0 0x7f44f8070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 >> v1:172.21.15.137:6789/0 conn(0x7f44f814b2c0 legacy=0x7f44f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8071c50 con 0x7f44f8070ea0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f44f00019f0 con 0x7f44f8070ea0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 >> v1:172.21.15.125:6789/0 conn(0x7f44f8070ea0 legacy=0x7f44f80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44d80047c0 0x7f44d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d8005180 0x7f44d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f814b6a0 con 0x7f44d80039c0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d800c080 con 0x7f44d8005180 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.667+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d800c240 con 0x7f44d80047c0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fde00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d8005180 0x7f44d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fca00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44d80047c0 0x7f44d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 -- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44d80047c0 msgr2=0x7f44d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44d80047c0 0x7f44d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 -- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d8005180 msgr2=0x7f44d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d8005180 0x7f44d800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fd400640 1 -- 172.21.15.125:0/3946024704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d8003520 con 0x7f44d80039c0 2024-01-16T15:02:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fde00640 1 -- 172.21.15.125:0/3946024704 reap_dead start 2024-01-16T15:02:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.668+0000 7f44fca00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44d80047c0 0x7f44d8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.669+0000 7f44fd400640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44d8003de0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001750 tx=0x7f44d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c227313c306c56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.669+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e80052d0 con 0x7f44d80039c0 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e802e050 con 0x7f44d80039c0 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44f7e00640 1 -- 172.21.15.125:0/3946024704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8003830 con 0x7f44d80039c0 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 -- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 msgr2=0x7f44d8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44d8003de0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001750 tx=0x7f44d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 -- 172.21.15.125:0/3946024704 shutdown_connections 2024-01-16T15:02:02.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 --2- 172.21.15.125:0/3946024704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44d8003de0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 -- 172.21.15.125:0/3946024704 >> 172.21.15.125:0/3946024704 conn(0x7f44f806a800 msgr2=0x7f44f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:02.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 -- 172.21.15.125:0/3946024704 shutdown_connections 2024-01-16T15:02:02.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.670+0000 7f44ffe00640 1 -- 172.21.15.125:0/3946024704 wait complete. 2024-01-16T15:02:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.672+0000 7f44ffe00640 1 Processor -- start 2024-01-16T15:02:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.672+0000 7f44ffe00640 1 -- start start 2024-01-16T15:02:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 0x7f44f8160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44f8163d50 0x7f44f8161250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80a8a70 con 0x7f44d80039c0 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8164320 con 0x7f44f81638d0 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44ffe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8164600 con 0x7f44f8163d50 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44fd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44fd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58736/0 (socket says 172.21.15.125:58736) 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.673+0000 7f44fd400640 1 -- 172.21.15.125:0/1215998377 learned_addr learned my addr 172.21.15.125:0/1215998377 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fca00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 0x7f44f8160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fde00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44f8163d50 0x7f44f8161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 -- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44f8163d50 msgr2=0x7f44f8161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44f8163d50 0x7f44f8161250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 -- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 msgr2=0x7f44f8160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 0x7f44f8160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 -- 172.21.15.125:0/1215998377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f81648e0 con 0x7f44d80039c0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fca00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 0x7f44f8160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44fd400640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e80076c0 con 0x7f44d80039c0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8007a30 con 0x7f44d80039c0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8006be0 con 0x7f44d80039c0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.674+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f80a8cc0 con 0x7f44d80039c0 2024-01-16T15:02:02.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.675+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f80abc40 con 0x7f44d80039c0 2024-01-16T15:02:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.676+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f80675d0 con 0x7f44d80039c0 2024-01-16T15:02:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.676+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f44e8006d40 con 0x7f44d80039c0 2024-01-16T15:02:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.677+0000 7f44f5600640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 0x7f44d006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.677+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44e80abf40 con 0x7f44d80039c0 2024-01-16T15:02:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.678+0000 7f44fca00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 0x7f44d006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.678+0000 7f44fca00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 0x7f44d006dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f44f80ab460 tx=0x7f44e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.682+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44e8076780 con 0x7f44d80039c0 2024-01-16T15:02:02.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:02.875+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f44f8164740 con 0x7f44d80039c0 2024-01-16T15:02:03.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.641+0000 7f44f5600640 1 -- 172.21.15.125:0/1215998377 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44f8164740 con 0x7f44d80039c0 2024-01-16T15:02:03.622 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 msgr2=0x7f44d006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 0x7f44d006dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f44f80ab460 tx=0x7f44e0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 msgr2=0x7f44f80ab700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8001120 comp rx=0 tx=0).stop 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 shutdown_connections 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44f8163d50 0x7f44f8161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44f81638d0 0x7f44f8160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44d006b890 0x7f44d006dd50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 --2- 172.21.15.125:0/1215998377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44d80039c0 0x7f44f80ab700 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 >> 172.21.15.125:0/1215998377 conn(0x7f44f806a800 msgr2=0x7f44f8148010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 shutdown_connections 2024-01-16T15:02:03.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.645+0000 7f44ffe00640 1 -- 172.21.15.125:0/1215998377 wait complete. 2024-01-16T15:02:03.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:02:03.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.870+0000 7f695d800640 1 Processor -- start 2024-01-16T15:02:03.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.870+0000 7f695d800640 1 -- start start 2024-01-16T15:02:03.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.872+0000 7f695d800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6958159730 0x7f69580cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.872+0000 7f695d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 0x7f695815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.872+0000 7f695d800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69580ccf10 con 0x7f695815e060 2024-01-16T15:02:03.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.872+0000 7f695d800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695815de20 con 0x7f6958159730 2024-01-16T15:02:03.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.872+0000 7f695d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69580b2580 con 0x7f695815d450 2024-01-16T15:02:03.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.873+0000 7f6957400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 0x7f695815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.873+0000 7f694de00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f695815e060 0x7f69580b1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:44734/0 (socket says 172.21.15.125:44734) 2024-01-16T15:02:03.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.873+0000 7f694de00640 1 -- 172.21.15.125:0/2935475352 learned_addr learned my addr 172.21.15.125:0/2935475352 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:03.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.873+0000 7f6956a00640 1 --2- 172.21.15.125:0/2935475352 >> v2:172.21.15.137:3300/0 conn(0x7f6958159730 0x7f69580cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.873+0000 7f6956000640 1 -- 172.21.15.125:0/2935475352 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385668683 0 0) 0x7f69580ccf10 con 0x7f695815e060 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6956000640 1 -- 172.21.15.125:0/2935475352 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6938003520 con 0x7f695815e060 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6956a00640 1 -- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 msgr2=0x7f695815d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6956a00640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 0x7f695815d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6956a00640 1 -- 172.21.15.125:0/2935475352 >> v1:172.21.15.137:6789/0 conn(0x7f695815e060 legacy=0x7f69580b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6956a00640 1 -- 172.21.15.125:0/2935475352 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580cd140 con 0x7f6958159730 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.874+0000 7f6957400640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 0x7f695815d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:03.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.875+0000 7f6956a00640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f69580cb380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69480097a0 tx=0x7f694802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e0bfcfbf1f4ba8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.876+0000 7f6956000640 1 -- 172.21.15.125:0/2935475352 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948031780 con 0x7f6958159730 2024-01-16T15:02:03.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.876+0000 7f6956000640 1 -- 172.21.15.125:0/2935475352 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69480318e0 con 0x7f6958159730 2024-01-16T15:02:03.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.876+0000 7f6956000640 1 -- 172.21.15.125:0/2935475352 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948031c70 con 0x7f6958159730 2024-01-16T15:02:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.876+0000 7f695d800640 1 -- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 msgr2=0x7f69580cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.876+0000 7f695d800640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f69580cb380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69480097a0 tx=0x7f694802b820 comp rx=0 tx=0).stop 2024-01-16T15:02:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 -- 172.21.15.125:0/2935475352 shutdown_connections 2024-01-16T15:02:03.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815d450 0x7f695815d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 --2- 172.21.15.125:0/2935475352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f69580cb380 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 -- 172.21.15.125:0/2935475352 >> 172.21.15.125:0/2935475352 conn(0x7f69580b0b70 msgr2=0x7f69580b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 -- 172.21.15.125:0/2935475352 shutdown_connections 2024-01-16T15:02:03.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.877+0000 7f695d800640 1 -- 172.21.15.125:0/2935475352 wait complete. 2024-01-16T15:02:03.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.878+0000 7f695d800640 1 Processor -- start 2024-01-16T15:02:03.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.879+0000 7f695d800640 1 -- start start 2024-01-16T15:02:03.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.879+0000 7f695d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f695813f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.879+0000 7f695d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.879+0000 7f695d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815e060 0x7f69580706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f695d800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695813fc60 con 0x7f695815d450 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f695d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69581f37c0 con 0x7f6958159730 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f695d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6958150dd0 con 0x7f695815e060 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58748/0 (socket says 172.21.15.125:58748) 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 -- 172.21.15.125:0/3885783074 learned_addr learned my addr 172.21.15.125:0/3885783074 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f6956a00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f695813f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f6957400640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815e060 0x7f69580706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 -- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815e060 msgr2=0x7f69580706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815e060 0x7f69580706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 -- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 msgr2=0x7f695813f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f695813f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.880+0000 7f694de00640 1 -- 172.21.15.125:0/3885783074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950009a70 con 0x7f695815d450 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f694de00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f694000c390 tx=0x7f694000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940005ce0 con 0x7f695815d450 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940005e40 con 0x7f695815d450 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6948009400 con 0x7f695815d450 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400072f0 con 0x7f695815d450 2024-01-16T15:02:03.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.881+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6958156e90 con 0x7f695815d450 2024-01-16T15:02:03.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.882+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69580cdc80 con 0x7f695815d450 2024-01-16T15:02:03.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.884+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6940007450 con 0x7f695815d450 2024-01-16T15:02:03.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.885+0000 7f694f400640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 0x7f693006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.885+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6940089eb0 con 0x7f695815d450 2024-01-16T15:02:03.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.885+0000 7f6956a00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 0x7f693006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.886+0000 7f6956a00640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 0x7f693006dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6948036000 tx=0x7f694803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:03.889+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69400546d0 con 0x7f695815d450 2024-01-16T15:02:04.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.081+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f69580675d0 con 0x7f695815d450 2024-01-16T15:02:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.647+0000 7f694f400640 1 -- 172.21.15.125:0/3885783074 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6940017020 con 0x7f695815d450 2024-01-16T15:02:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T15:02:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 msgr2=0x7f693006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 0x7f693006dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6948036000 tx=0x7f694803d040 comp rx=0 tx=0).stop 2024-01-16T15:02:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 msgr2=0x7f69581f3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f694000c390 tx=0x7f694000c850 comp rx=0 tx=0).stop 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 shutdown_connections 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f695815e060 0x7f69580706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f693006b860 0x7f693006dd20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f695815d450 0x7f69581f3280 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 --2- 172.21.15.125:0/3885783074 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6958159730 0x7f695813f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 >> 172.21.15.125:0/3885783074 conn(0x7f69580b0b70 msgr2=0x7f695814f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.652+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 shutdown_connections 2024-01-16T15:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.653+0000 7f695d800640 1 -- 172.21.15.125:0/3885783074 wait complete. 2024-01-16T15:02:04.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:02:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.871+0000 7efd13200640 1 Processor -- start 2024-01-16T15:02:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.871+0000 7efd13200640 1 -- start start 2024-01-16T15:02:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd13200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd1406c730 0x7efd140b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd13200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd1406cb10 0x7efd1406cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd13200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140708d0 con 0x7efd14070160 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd13200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd14070bb0 con 0x7efd1406cb10 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd13200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd14070e90 con 0x7efd1406c730 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd12800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd1406cb10 0x7efd1406cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd12800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd1406cb10 0x7efd1406cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58756/0 (socket says 172.21.15.125:58756) 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd12800640 1 -- 172.21.15.125:0/2938997331 learned_addr learned my addr 172.21.15.125:0/2938997331 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:04.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.873+0000 7efd11e00640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd1406c730 0x7efd140b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.874+0000 7efd12800640 1 -- 172.21.15.125:0/2938997331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd1406c730 msgr2=0x7efd140b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.874+0000 7efd12800640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd1406c730 0x7efd140b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.874+0000 7efd12800640 1 -- 172.21.15.125:0/2938997331 >> v1:172.21.15.137:6789/0 conn(0x7efd14070160 legacy=0x7efd14071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.874+0000 7efd12800640 1 -- 172.21.15.125:0/2938997331 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd14071170 con 0x7efd1406cb10 2024-01-16T15:02:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.875+0000 7efd12800640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd1406cf80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009e50 tx=0x7efd0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b2687c1f0c07cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.875+0000 7efd10a00640 1 -- 172.21.15.125:0/2938997331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c02cb00 con 0x7efd1406cb10 2024-01-16T15:02:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.875+0000 7efd10a00640 1 -- 172.21.15.125:0/2938997331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0c02cc60 con 0x7efd1406cb10 2024-01-16T15:02:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.875+0000 7efd10a00640 1 -- 172.21.15.125:0/2938997331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0343f0 con 0x7efd1406cb10 2024-01-16T15:02:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 -- 172.21.15.125:0/2938997331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 msgr2=0x7efd1406cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd1406cf80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009e50 tx=0x7efd0c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 -- 172.21.15.125:0/2938997331 shutdown_connections 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd1406cf80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 --2- 172.21.15.125:0/2938997331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd1406c730 0x7efd140b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 -- 172.21.15.125:0/2938997331 >> 172.21.15.125:0/2938997331 conn(0x7efd1406a7d0 msgr2=0x7efd1406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 -- 172.21.15.125:0/2938997331 shutdown_connections 2024-01-16T15:02:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.876+0000 7efd13200640 1 -- 172.21.15.125:0/2938997331 wait complete. 2024-01-16T15:02:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.878+0000 7efd13200640 1 Processor -- start 2024-01-16T15:02:04.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.878+0000 7efd13200640 1 -- start start 2024-01-16T15:02:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.879+0000 7efd13200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 0x7efd1415f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.879+0000 7efd13200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd13200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd14070160 0x7efd1415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd13200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd140a98b0 con 0x7efd1406cb10 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd13200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd140a8a00 con 0x7efd1406c730 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd13200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd140a8ca0 con 0x7efd14070160 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58772/0 (socket says 172.21.15.125:58772) 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 -- 172.21.15.125:0/3670880786 learned_addr learned my addr 172.21.15.125:0/3670880786 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11e00640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 0x7efd1415f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd12800640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd14070160 0x7efd1415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 -- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd14070160 msgr2=0x7efd1415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd14070160 0x7efd1415ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 -- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 msgr2=0x7efd1415f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 0x7efd1415f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11400640 1 -- 172.21.15.125:0/3670880786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c009b30 con 0x7efd1406cb10 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.880+0000 7efd11e00640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 0x7efd1415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd11400640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c400 tx=0x7efd0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd040052d0 con 0x7efd1406cb10 2024-01-16T15:02:04.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04005430 con 0x7efd1406cb10 2024-01-16T15:02:04.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd140a8f80 con 0x7efd1406cb10 2024-01-16T15:02:04.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04003830 con 0x7efd1406cb10 2024-01-16T15:02:04.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.881+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd140a9540 con 0x7efd1406cb10 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.882+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd14067450 con 0x7efd1406cb10 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.883+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7efd04003990 con 0x7efd1406cb10 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.884+0000 7efd0a000640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 0x7efcec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.884+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5609+0+0 (secure 0 0 0) 0x7efd04089ba0 con 0x7efd1406cb10 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.884+0000 7efd11e00640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 0x7efcec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.885+0000 7efd11e00640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 0x7efcec06dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:04.889+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd040541c0 con 0x7efd1406cb10 2024-01-16T15:02:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.113+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7efd140cd660 con 0x7efd1406cb10 2024-01-16T15:02:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.670+0000 7efd0a000640 1 -- 172.21.15.125:0/3670880786 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v34) v1 ==== 172+0+0 (secure 0 0 0) 0x7efd040577e0 con 0x7efd1406cb10 2024-01-16T15:02:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 8 and data pool 9 2024-01-16T15:02:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.673+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 msgr2=0x7efcec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.673+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 0x7efcec06dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc008040 comp rx=0 tx=0).stop 2024-01-16T15:02:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.673+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 msgr2=0x7efd140a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.673+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c400 tx=0x7efd0400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 shutdown_connections 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd14070160 0x7efd1415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efcec06b860 0x7efcec06dd20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd1406cb10 0x7efd140a84c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 --2- 172.21.15.125:0/3670880786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd1406c730 0x7efd1415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 >> 172.21.15.125:0/3670880786 conn(0x7efd1406a7d0 msgr2=0x7efd140cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 shutdown_connections 2024-01-16T15:02:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.674+0000 7efd13200640 1 -- 172.21.15.125:0/3670880786 wait complete. 2024-01-16T15:02:05.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:05.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.896+0000 7f167a400640 1 Processor -- start 2024-01-16T15:02:05.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.896+0000 7f167a400640 1 -- start start 2024-01-16T15:02:05.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f167a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 0x7f1674071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f167a400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16740717d0 0x7f16740cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f167a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1674070e50 con 0x7f16740cca40 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f167a400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16740723c0 con 0x7f16740717d0 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f167a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16740cce40 con 0x7f1674143c40 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16740717d0 0x7f16740cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16740717d0 0x7f16740cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58786/0 (socket says 172.21.15.125:58786) 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 -- 172.21.15.125:0/2960520402 learned_addr learned my addr 172.21.15.125:0/2960520402 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:05.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1679a00640 1 --1- 172.21.15.125:0/2960520402 >> v1:172.21.15.125:6789/0 conn(0x7f16740cca40 0x7f167415eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42164/0 (socket says 172.21.15.125:42164) 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1679000640 1 --2- 172.21.15.125:0/2960520402 >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 0x7f1674071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 -- 172.21.15.125:0/2960520402 >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 msgr2=0x7f1674071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 --2- 172.21.15.125:0/2960520402 >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 0x7f1674071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 -- 172.21.15.125:0/2960520402 >> v1:172.21.15.125:6789/0 conn(0x7f16740cca40 legacy=0x7f167415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673e00640 1 -- 172.21.15.125:0/2960520402 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167406e170 con 0x7f16740717d0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.898+0000 7f1673400640 1 -- 172.21.15.125:0/2960520402 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314656625 0 0) 0x7f1674070e50 con 0x7f16740cca40 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.899+0000 7f1679000640 1 --2- 172.21.15.125:0/2960520402 >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 0x7f1674071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f1673e00640 1 --2- 172.21.15.125:0/2960520402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f16740cc500 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f167406e410 tx=0x7f165c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa35ffd9558abb82 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f1673400640 1 -- 172.21.15.125:0/2960520402 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c031870 con 0x7f16740717d0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f1673400640 1 -- 172.21.15.125:0/2960520402 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c0319d0 con 0x7f16740717d0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f1673400640 1 -- 172.21.15.125:0/2960520402 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c031d60 con 0x7f16740717d0 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f167a400640 1 -- 172.21.15.125:0/2960520402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 msgr2=0x7f16740cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.900+0000 7f167a400640 1 --2- 172.21.15.125:0/2960520402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f16740cc500 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f167406e410 tx=0x7f165c02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 -- 172.21.15.125:0/2960520402 shutdown_connections 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 --2- 172.21.15.125:0/2960520402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f16740cc500 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 --2- 172.21.15.125:0/2960520402 >> v2:172.21.15.137:3300/0 conn(0x7f1674143c40 0x7f1674071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 -- 172.21.15.125:0/2960520402 >> 172.21.15.125:0/2960520402 conn(0x7f167406a800 msgr2=0x7f167406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 -- 172.21.15.125:0/2960520402 shutdown_connections 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.901+0000 7f167a400640 1 -- 172.21.15.125:0/2960520402 wait complete. 2024-01-16T15:02:05.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.902+0000 7f167a400640 1 Processor -- start 2024-01-16T15:02:05.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.903+0000 7f167a400640 1 -- start start 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.903+0000 7f167a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.903+0000 7f167a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16740cca40 0x7f16740c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.903+0000 7f167a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1674143c40 0x7f1674159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f167a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167413db70 con 0x7f16740717d0 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f167a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c5590 con 0x7f1674143c40 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f167a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c4980 con 0x7f16740cca40 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58802/0 (socket says 172.21.15.125:58802) 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 -- 172.21.15.125:0/3665687267 learned_addr learned my addr 172.21.15.125:0/3665687267 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 -- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16740cca40 msgr2=0x7f16740c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16740cca40 0x7f16740c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 -- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1674143c40 msgr2=0x7f1674159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1674143c40 0x7f1674159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 -- 172.21.15.125:0/3665687267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0094d0 con 0x7f16740717d0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.904+0000 7f1679000640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f166400c7f0 tx=0x7f166400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.905+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664007820 con 0x7f16740717d0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.905+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1664007980 con 0x7f16740717d0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.905+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664007cf0 con 0x7f16740717d0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.905+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16740c4c60 con 0x7f16740717d0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.905+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16740c5220 con 0x7f16740717d0 2024-01-16T15:02:05.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.906+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674067450 con 0x7f16740717d0 2024-01-16T15:02:05.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.907+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f16640113f0 con 0x7f16740717d0 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.908+0000 7f1670c00640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 0x7f164c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.908+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f166408ab00 con 0x7f16740717d0 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.908+0000 7f1673e00640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 0x7f164c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.909+0000 7f1673e00640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 0x7f164c06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f165c036000 tx=0x7f165c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:05.912+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1664054fb0 con 0x7f16740717d0 2024-01-16T15:02:06.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.098+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16740715b0 con 0x7f16740717d0 2024-01-16T15:02:06.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.101+0000 7f1670c00640 1 -- 172.21.15.125:0/3665687267 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f166401f090 con 0x7f16740717d0 2024-01-16T15:02:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:06.082 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":57,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:02:05.115302+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:02.926348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:04.133789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:06.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.103+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 msgr2=0x7f164c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.103+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 0x7f164c06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f165c036000 tx=0x7f165c030000 comp rx=0 tx=0).stop 2024-01-16T15:02:06.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.103+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 msgr2=0x7f1674140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.103+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f166400c7f0 tx=0x7f166400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 shutdown_connections 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1674143c40 0x7f1674159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16740cca40 0x7f16740c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f164c06b860 0x7f164c06dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 --2- 172.21.15.125:0/3665687267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16740717d0 0x7f1674140060 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 >> 172.21.15.125:0/3665687267 conn(0x7f167406a800 msgr2=0x7f16740ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 shutdown_connections 2024-01-16T15:02:06.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.104+0000 7f167a400640 1 -- 172.21.15.125:0/3665687267 wait complete. 2024-01-16T15:02:06.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:06.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.306+0000 7fd810400640 1 Processor -- start 2024-01-16T15:02:06.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.306+0000 7fd810400640 1 -- start start 2024-01-16T15:02:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd810400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8080cd170 0x7fd808071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd810400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd808071690 con 0x7fd80814b2c0 2024-01-16T15:02:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd810400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd808071970 con 0x7fd8080cf9a0 2024-01-16T15:02:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd810400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80814b6a0 con 0x7fd8080cd170 2024-01-16T15:02:06.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd80e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8080cd170 0x7fd808071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd80e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8080cd170 0x7fd808071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58818/0 (socket says 172.21.15.125:58818) 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd80e400640 1 -- 172.21.15.125:0/142545740 learned_addr learned my addr 172.21.15.125:0/142545740 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.308+0000 7fd80da00640 1 --1- 172.21.15.125:0/142545740 >> v1:172.21.15.125:6789/0 conn(0x7fd80814b2c0 0x7fd8080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42166/0 (socket says 172.21.15.125:42166) 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.309+0000 7fd80e400640 1 -- 172.21.15.125:0/142545740 >> v1:172.21.15.137:6789/0 conn(0x7fd8080cf9a0 legacy=0x7fd808070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.309+0000 7fd80e400640 1 -- 172.21.15.125:0/142545740 >> v1:172.21.15.125:6789/0 conn(0x7fd80814b2c0 legacy=0x7fd8080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.309+0000 7fd80e400640 1 -- 172.21.15.125:0/142545740 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808071c50 con 0x7fd8080cd170 2024-01-16T15:02:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.309+0000 7fd807e00640 1 -- 172.21.15.125:0/142545740 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590609681 0 0) 0x7fd808071690 con 0x7fd80814b2c0 2024-01-16T15:02:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd80e400640 1 --2- 172.21.15.125:0/142545740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080cd170 0x7fd808071070 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd800009e50 tx=0x7fd80002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b38861bcb35c64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd807e00640 1 -- 172.21.15.125:0/142545740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80002cb20 con 0x7fd8080cd170 2024-01-16T15:02:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd807e00640 1 -- 172.21.15.125:0/142545740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80002cc80 con 0x7fd8080cd170 2024-01-16T15:02:06.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd807e00640 1 -- 172.21.15.125:0/142545740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8000343f0 con 0x7fd8080cd170 2024-01-16T15:02:06.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd810400640 1 -- 172.21.15.125:0/142545740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080cd170 msgr2=0x7fd808071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.310+0000 7fd810400640 1 --2- 172.21.15.125:0/142545740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080cd170 0x7fd808071070 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd800009e50 tx=0x7fd80002bf80 comp rx=0 tx=0).stop 2024-01-16T15:02:06.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.311+0000 7fd810400640 1 -- 172.21.15.125:0/142545740 shutdown_connections 2024-01-16T15:02:06.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.311+0000 7fd810400640 1 --2- 172.21.15.125:0/142545740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080cd170 0x7fd808071070 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.311+0000 7fd810400640 1 -- 172.21.15.125:0/142545740 >> 172.21.15.125:0/142545740 conn(0x7fd80806a800 msgr2=0x7fd80806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.311+0000 7fd810400640 1 -- 172.21.15.125:0/142545740 shutdown_connections 2024-01-16T15:02:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.311+0000 7fd810400640 1 -- 172.21.15.125:0/142545740 wait complete. 2024-01-16T15:02:06.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.312+0000 7fd810400640 1 Processor -- start 2024-01-16T15:02:06.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.312+0000 7fd810400640 1 -- start start 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8080cd170 0x7fd808159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080cf9a0 0x7fd80813e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80806dc00 con 0x7fd80814b2c0 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80815a890 con 0x7fd8080cd170 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd810400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8081618f0 con 0x7fd8080cf9a0 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.313+0000 7fd80e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58820/0 (socket says 172.21.15.125:58820) 2024-01-16T15:02:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 -- 172.21.15.125:0/383753754 learned_addr learned my addr 172.21.15.125:0/383753754 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 -- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080cf9a0 msgr2=0x7fd80813e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80d000640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080cf9a0 0x7fd80813e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80da00640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8080cd170 0x7fd808159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080cf9a0 0x7fd80813e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 -- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8080cd170 msgr2=0x7fd808159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8080cd170 0x7fd808159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 -- 172.21.15.125:0/383753754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009b30 con 0x7fd80814b2c0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd80e400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd800036040 tx=0x7fd800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.314+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800009510 con 0x7fd80814b2c0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.315+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd800007e10 con 0x7fd80814b2c0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.315+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80003e670 con 0x7fd80814b2c0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.315+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80806cf80 con 0x7fd80814b2c0 2024-01-16T15:02:06.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.315+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80815a5b0 con 0x7fd80814b2c0 2024-01-16T15:02:06.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.316+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80806dd40 con 0x7fd80814b2c0 2024-01-16T15:02:06.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.317+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fd80003e7d0 con 0x7fd80814b2c0 2024-01-16T15:02:06.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.317+0000 7fd805600640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 0x7fd7e006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.318+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8000ad250 con 0x7fd80814b2c0 2024-01-16T15:02:06.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.318+0000 7fd80da00640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 0x7fd7e006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.318+0000 7fd80da00640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 0x7fd7e006dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8000f60 tx=0x7fd7f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.322+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd80806dd40 con 0x7fd80814b2c0 2024-01-16T15:02:06.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.497+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd80806dd40 con 0x7fd80814b2c0 2024-01-16T15:02:06.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.500+0000 7fd805600640 1 -- 172.21.15.125:0/383753754 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16626 (secure 0 0 0) 0x7fd80806dd40 con 0x7fd80814b2c0 2024-01-16T15:02:06.480 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:06.481 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":57,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:02:05.115302+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:02.926348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:04.133789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 msgr2=0x7fd7e006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 0x7fd7e006dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8000f60 tx=0x7fd7f8009040 comp rx=0 tx=0).stop 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 msgr2=0x7fd80813ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd800036040 tx=0x7fd800004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 shutdown_connections 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006b860 0x7fd7e006dd20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd80814b2c0 0x7fd80813ee60 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080cf9a0 0x7fd80813e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 --2- 172.21.15.125:0/383753754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8080cd170 0x7fd808159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.502+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 >> 172.21.15.125:0/383753754 conn(0x7fd80806a800 msgr2=0x7fd8080708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:06.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.503+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 shutdown_connections 2024-01-16T15:02:06.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.503+0000 7fd810400640 1 -- 172.21.15.125:0/383753754 wait complete. 2024-01-16T15:02:06.499 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:02:06.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.721+0000 7f99f9e00640 1 Processor -- start 2024-01-16T15:02:06.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.721+0000 7f99f9e00640 1 -- start start 2024-01-16T15:02:06.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.723+0000 7f99f9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.723+0000 7f99f9e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f99f413e360 0x7f99f40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.723+0000 7f99f9e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f406c470 con 0x7f99f415f9c0 2024-01-16T15:02:06.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.723+0000 7f99f9e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f406c750 con 0x7f99f413e360 2024-01-16T15:02:06.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.723+0000 7f99f9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f406ca30 con 0x7f99f413df80 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44254/0 (socket says 172.21.15.125:44254) 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 -- 172.21.15.125:0/3400237912 learned_addr learned my addr 172.21.15.125:0/3400237912 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f3400640 1 --1- 172.21.15.125:0/3400237912 >> v1:172.21.15.137:6789/0 conn(0x7f99f415f9c0 0x7f99f41408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:38752/0 (socket says 172.21.15.125:38752) 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2000640 1 --2- 172.21.15.125:0/3400237912 >> v2:172.21.15.137:3300/0 conn(0x7f99f413e360 0x7f99f40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 -- 172.21.15.125:0/3400237912 >> v2:172.21.15.137:3300/0 conn(0x7f99f413e360 msgr2=0x7f99f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:06.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 --2- 172.21.15.125:0/3400237912 >> v2:172.21.15.137:3300/0 conn(0x7f99f413e360 0x7f99f40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 -- 172.21.15.125:0/3400237912 >> v1:172.21.15.137:6789/0 conn(0x7f99f415f9c0 legacy=0x7f99f41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.724+0000 7f99f2a00640 1 -- 172.21.15.125:0/3400237912 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f406cd10 con 0x7f99f413df80 2024-01-16T15:02:06.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.726+0000 7f99f2a00640 1 --2- 172.21.15.125:0/3400237912 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009e50 tx=0x7f99e402bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300c99b4564c8043 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.726+0000 7f99f1600640 1 -- 172.21.15.125:0/3400237912 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e402caf0 con 0x7f99f413df80 2024-01-16T15:02:06.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.726+0000 7f99f1600640 1 -- 172.21.15.125:0/3400237912 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e402cc50 con 0x7f99f413df80 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.726+0000 7f99f1600640 1 -- 172.21.15.125:0/3400237912 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e40343f0 con 0x7f99f413df80 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 -- 172.21.15.125:0/3400237912 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 msgr2=0x7f99f415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3400237912 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009e50 tx=0x7f99e402bf50 comp rx=0 tx=0).stop 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 -- 172.21.15.125:0/3400237912 shutdown_connections 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3400237912 >> v2:172.21.15.137:3300/0 conn(0x7f99f413e360 0x7f99f40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3400237912 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f413df80 0x7f99f415bee0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 -- 172.21.15.125:0/3400237912 >> 172.21.15.125:0/3400237912 conn(0x7f99f406a800 msgr2=0x7f99f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 -- 172.21.15.125:0/3400237912 shutdown_connections 2024-01-16T15:02:06.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.727+0000 7f99f9e00640 1 -- 172.21.15.125:0/3400237912 wait complete. 2024-01-16T15:02:06.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.729+0000 7f99f9e00640 1 Processor -- start 2024-01-16T15:02:06.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.729+0000 7f99f9e00640 1 -- start start 2024-01-16T15:02:06.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99f413df80 0x7f99f415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f415f9c0 0x7f99f414b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f415ea00 con 0x7f99f413e360 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f41f3030 con 0x7f99f413df80 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f414c000 con 0x7f99f415f9c0 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50756/0 (socket says 172.21.15.125:50756) 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 -- 172.21.15.125:0/3655977534 learned_addr learned my addr 172.21.15.125:0/3655977534 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f3400640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f415f9c0 0x7f99f414b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2a00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99f413df80 0x7f99f415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 -- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f415f9c0 msgr2=0x7f99f414b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f415f9c0 0x7f99f414b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 -- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99f413df80 msgr2=0x7f99f415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.730+0000 7f99f2000640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99f413df80 0x7f99f415a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99f2000640 1 -- 172.21.15.125:0/3655977534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e4009b30 con 0x7f99f413e360 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99f2000640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e9b0 tx=0x7f99dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc009670 con 0x7f99f413e360 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc00c9a0 con 0x7f99f413e360 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f414c2e0 con 0x7f99f413e360 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc00cd50 con 0x7f99f413e360 2024-01-16T15:02:06.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.731+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f412f6d0 con 0x7f99f413e360 2024-01-16T15:02:06.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.732+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f40675d0 con 0x7f99f413e360 2024-01-16T15:02:06.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.733+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f99dc013470 con 0x7f99f413e360 2024-01-16T15:02:06.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.734+0000 7f99eb400640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 0x7f99cc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:06.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.734+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99dc08a850 con 0x7f99f413e360 2024-01-16T15:02:06.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.734+0000 7f99f2a00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 0x7f99cc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:06.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.735+0000 7f99f2a00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 0x7f99cc06dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f99e4035000 tx=0x7f99e4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:06.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.738+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99dc054ee0 con 0x7f99f413e360 2024-01-16T15:02:06.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:06.970+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99f406df00 con 0x7f99f413e360 2024-01-16T15:02:07.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.700+0000 7f99eb400640 1 -- 172.21.15.125:0/3655977534 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7f99dc058500 con 0x7f99f413e360 2024-01-16T15:02:07.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.704+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 msgr2=0x7f99cc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.704+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 0x7f99cc06dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f99e4035000 tx=0x7f99e4030000 comp rx=0 tx=0).stop 2024-01-16T15:02:07.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.704+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 msgr2=0x7f99f415a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.704+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e9b0 tx=0x7f99dc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 shutdown_connections 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99f415f9c0 0x7f99f414b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99cc06b860 0x7f99cc06dd20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99f413e360 0x7f99f415a680 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 --2- 172.21.15.125:0/3655977534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99f413df80 0x7f99f415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 >> 172.21.15.125:0/3655977534 conn(0x7f99f406a800 msgr2=0x7f99f40b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 shutdown_connections 2024-01-16T15:02:07.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.705+0000 7f99f9e00640 1 -- 172.21.15.125:0/3655977534 wait complete. 2024-01-16T15:02:07.706 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:02:07.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:02:07.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:07.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.939+0000 7f106d600640 1 Processor -- start 2024-01-16T15:02:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.939+0000 7f106d600640 1 -- start start 2024-01-16T15:02:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.941+0000 7f106d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f10680b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.941+0000 7f106d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f106806c6b0 0x7f10680cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.941+0000 7f106d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106806cea0 con 0x7f10680cd240 2024-01-16T15:02:07.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.941+0000 7f106d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106806cbc0 con 0x7f106806c6b0 2024-01-16T15:02:07.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.941+0000 7f106d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106806f350 con 0x7f106809a170 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1066000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f10680b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1066a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f10680cd240 0x7f1068099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:38754/0 (socket says 172.21.15.125:38754) 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1066a00640 1 -- 172.21.15.125:0/3988653743 learned_addr learned my addr 172.21.15.125:0/3988653743 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1067400640 1 --2- 172.21.15.125:0/3988653743 >> v2:172.21.15.137:3300/0 conn(0x7f106806c6b0 0x7f10680cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807653498 0 0) 0x7f106806cea0 con 0x7f10680cd240 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.942+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048003520 con 0x7f10680cd240 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1067400640 1 -- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 msgr2=0x7f10680b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3213384174 0 0) 0x7f1048003520 con 0x7f10680cd240 2024-01-16T15:02:07.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1067400640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f10680b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1067400640 1 -- 172.21.15.125:0/3988653743 >> v1:172.21.15.137:6789/0 conn(0x7f10680cd240 legacy=0x7f1068099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1067400640 1 -- 172.21.15.125:0/3988653743 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106806f630 con 0x7f106806c6b0 2024-01-16T15:02:07.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.943+0000 7f1066000640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f10680b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:07.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.944+0000 7f1067400640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106806c6b0 0x7f10680cf640 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a948b5d6a4140e56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.944+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058001670 con 0x7f106806c6b0 2024-01-16T15:02:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.944+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106002c8a0 con 0x7f106806c6b0 2024-01-16T15:02:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.944+0000 7f1065600640 1 -- 172.21.15.125:0/3988653743 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106002cc30 con 0x7f106806c6b0 2024-01-16T15:02:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 -- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106806c6b0 msgr2=0x7f10680cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106806c6b0 0x7f10680cf640 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106002bd70 comp rx=0 tx=0).stop 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 -- 172.21.15.125:0/3988653743 shutdown_connections 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f106806c6b0 0x7f10680cf640 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 --2- 172.21.15.125:0/3988653743 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f10680b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 -- 172.21.15.125:0/3988653743 >> 172.21.15.125:0/3988653743 conn(0x7f10680b2d00 msgr2=0x7f10680b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 -- 172.21.15.125:0/3988653743 shutdown_connections 2024-01-16T15:02:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.945+0000 7f106d600640 1 -- 172.21.15.125:0/3988653743 wait complete. 2024-01-16T15:02:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.947+0000 7f106d600640 1 Processor -- start 2024-01-16T15:02:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.948+0000 7f106d600640 1 -- start start 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f1068157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 0x7f10680b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680c5c70 con 0x7f106806c6b0 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068158370 con 0x7f10680cd240 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f106d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680b4750 con 0x7f106809a170 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f1066a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f1066a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50766/0 (socket says 172.21.15.125:50766) 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.949+0000 7f1066a00640 1 -- 172.21.15.125:0/2221395313 learned_addr learned my addr 172.21.15.125:0/2221395313 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1067400640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 0x7f10680b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066000640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f1068157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 -- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 msgr2=0x7f1068157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f1068157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 -- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 msgr2=0x7f10680b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 0x7f10680b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 -- 172.21.15.125:0/2221395313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050009400 con 0x7f106806c6b0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066000640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f1068157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1067400640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 0x7f10680b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f1066a00640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f105800c760 tx=0x7f105800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058005ad0 con 0x7f106806c6b0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1058005c30 con 0x7f106806c6b0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1060009a70 con 0x7f106806c6b0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.950+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058007070 con 0x7f106806c6b0 2024-01-16T15:02:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.951+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10680c6ae0 con 0x7f106806c6b0 2024-01-16T15:02:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.952+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1068073d90 con 0x7f106806c6b0 2024-01-16T15:02:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.953+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f10580071d0 con 0x7f106806c6b0 2024-01-16T15:02:07.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.953+0000 7f105ea00640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 0x7f104006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.953+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f105808a320 con 0x7f106806c6b0 2024-01-16T15:02:07.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.954+0000 7f1066000640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 0x7f104006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.954+0000 7f1066000640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 0x7f104006dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f10681f5ef0 tx=0x7f1050008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:07.958+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1058054920 con 0x7f106806c6b0 2024-01-16T15:02:08.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.189+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f10680675d0 con 0x7f106806c6b0 2024-01-16T15:02:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.711+0000 7f105ea00640 1 -- 172.21.15.125:0/2221395313 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1058057f40 con 0x7f106806c6b0 2024-01-16T15:02:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 msgr2=0x7f104006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 0x7f104006dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f10681f5ef0 tx=0x7f1050008040 comp rx=0 tx=0).stop 2024-01-16T15:02:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 msgr2=0x7f10680c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f105800c760 tx=0x7f105800cc20 comp rx=0 tx=0).stop 2024-01-16T15:02:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 shutdown_connections 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10680cd240 0x7f10680b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f106809a170 0x7f1068157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f104006b890 0x7f104006dd50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 --2- 172.21.15.125:0/2221395313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f106806c6b0 0x7f10680c5730 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 >> 172.21.15.125:0/2221395313 conn(0x7f10680b2d00 msgr2=0x7f10680cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 shutdown_connections 2024-01-16T15:02:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.716+0000 7f106d600640 1 -- 172.21.15.125:0/2221395313 wait complete. 2024-01-16T15:02:08.717 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:02:08.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:08.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.992+0000 7fef29400640 1 Processor -- start 2024-01-16T15:02:08.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.992+0000 7fef29400640 1 -- start start 2024-01-16T15:02:08.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef29400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 0x7fef2415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef29400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2406c470 con 0x7fef2413e360 2024-01-16T15:02:08.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef29400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2406c750 con 0x7fef240cf9a0 2024-01-16T15:02:08.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef29400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2406ca30 con 0x7fef2413df80 2024-01-16T15:02:08.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef21e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fef2413e360 0x7fef240cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60734/0 (socket says 172.21.15.125:60734) 2024-01-16T15:02:08.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef21e00640 1 -- 172.21.15.125:0/870228162 learned_addr learned my addr 172.21.15.125:0/870228162 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:08.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.994+0000 7fef22800640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 0x7fef2415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125278331 0 0) 0x7fef2406c470 con 0x7fef2413e360 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef04003520 con 0x7fef2413e360 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109158096 0 0) 0x7fef04003520 con 0x7fef2413e360 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 msgr2=0x7fef2415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 0x7fef2415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 >> v1:172.21.15.137:6789/0 conn(0x7fef240cf9a0 legacy=0x7fef240cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2406cd10 con 0x7fef2413e360 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fef0c001670 con 0x7fef2413e360 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 >> v1:172.21.15.125:6789/0 conn(0x7fef2413e360 legacy=0x7fef240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 0x7fef040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef04004d00 0x7fef04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2406ca30 con 0x7fef040039a0 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0400ba60 con 0x7fef04004d00 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0400bc20 con 0x7fef2413df80 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.995+0000 7fef22800640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef22800640 1 -- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 msgr2=0x7fef040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef22800640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef2413df80 0x7fef040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef22800640 1 -- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef04004d00 msgr2=0x7fef04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef23200640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef04004d00 0x7fef04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef22800640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef04004d00 0x7fef04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef22800640 1 -- 172.21.15.125:0/870228162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef04003520 con 0x7fef040039a0 2024-01-16T15:02:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.996+0000 7fef23200640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef04004d00 0x7fef04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:08.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.997+0000 7fef22800640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef04003dc0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fef14009a90 tx=0x7fef040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d166c3a41df5307 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.997+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1402c6b0 con 0x7fef040039a0 2024-01-16T15:02:08.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.997+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1402c810 con 0x7fef040039a0 2024-01-16T15:02:08.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.997+0000 7fef21400640 1 -- 172.21.15.125:0/870228162 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1402cba0 con 0x7fef040039a0 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 -- 172.21.15.125:0/870228162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 msgr2=0x7fef04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 --2- 172.21.15.125:0/870228162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef04003dc0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fef14009a90 tx=0x7fef040057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef23200640 1 -- 172.21.15.125:0/870228162 reap_dead start 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 -- 172.21.15.125:0/870228162 shutdown_connections 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 -- 172.21.15.125:0/870228162 >> 172.21.15.125:0/870228162 conn(0x7fef2406a800 msgr2=0x7fef2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 -- 172.21.15.125:0/870228162 shutdown_connections 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:08.998+0000 7fef29400640 1 -- 172.21.15.125:0/870228162 wait complete. 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.000+0000 7fef29400640 1 Processor -- start 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.000+0000 7fef29400640 1 -- start start 2024-01-16T15:02:08.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef24151470 0x7fef24157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 0x7fef2415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef240b3640 con 0x7fef040039a0 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef240b2060 con 0x7fef24157840 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef29400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef240b2340 con 0x7fef24151470 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50778/0 (socket says 172.21.15.125:50778) 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 -- 172.21.15.125:0/472426927 learned_addr learned my addr 172.21.15.125:0/472426927 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef23200640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 0x7fef2415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 -- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef24151470 msgr2=0x7fef24157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef24151470 0x7fef24157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 -- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 msgr2=0x7fef2415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 0x7fef2415a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.001+0000 7fef22800640 1 -- 172.21.15.125:0/472426927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef240b2620 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef23200640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 0x7fef2415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef22800640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fef1402c250 tx=0x7fef14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14003810 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef14003970 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14003d00 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef240b2900 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.002+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef240b1ca0 con 0x7fef040039a0 2024-01-16T15:02:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.003+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef240675d0 con 0x7fef040039a0 2024-01-16T15:02:08.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.004+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fef14047450 con 0x7fef040039a0 2024-01-16T15:02:08.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.006+0000 7fef1aa00640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 0x7feefc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.006+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef140ac6f0 con 0x7fef040039a0 2024-01-16T15:02:08.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.006+0000 7fef21e00640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 0x7feefc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.006+0000 7fef21e00640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 0x7feefc06dd50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef241528a0 tx=0x7fef0c001540 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.010+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef14076ff0 con 0x7fef040039a0 2024-01-16T15:02:09.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.235+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2406df00 con 0x7fef040039a0 2024-01-16T15:02:09.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.237+0000 7fef1aa00640 1 -- 172.21.15.125:0/472426927 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fef140044e0 con 0x7fef040039a0 2024-01-16T15:02:09.217 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:09.217 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2646664931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2646664931},{"type":"v1","addr":"172.21.15.125:6837","nonce":2646664931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5054,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1721910358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1721910358},{"type":"v1","addr":"172.21.15.137:6835","nonce":1721910358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5362,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/28677566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":28677566},{"type":"v1","addr":"172.21.15.137:6833","nonce":28677566}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:05.115789+0000","modified":"2024-01-16T15:02:08.191947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6835/2216643169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2216643169},{"type":"v1","addr":"172.21.15.125:6835","nonce":2216643169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:02:09.218 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 37 2024-01-16T15:02:09.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 msgr2=0x7feefc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 0x7feefc06dd50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef241528a0 tx=0x7fef0c001540 comp rx=0 tx=0).stop 2024-01-16T15:02:09.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 msgr2=0x7fef240b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fef1402c250 tx=0x7fef14004730 comp rx=0 tx=0).stop 2024-01-16T15:02:09.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 shutdown_connections 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef24157840 0x7fef2415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef24151470 0x7fef24157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feefc06b890 0x7feefc06dd50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 --2- 172.21.15.125:0/472426927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef040039a0 0x7fef240b3100 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 >> 172.21.15.125:0/472426927 conn(0x7fef2406a800 msgr2=0x7fef240b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 shutdown_connections 2024-01-16T15:02:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.239+0000 7fef29400640 1 -- 172.21.15.125:0/472426927 wait complete. 2024-01-16T15:02:09.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:09.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.453+0000 7f3e0b600640 1 Processor -- start 2024-01-16T15:02:09.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.454+0000 7f3e0b600640 1 -- start start 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e0b600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e04143ae0 0x7f3e040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e0b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e040cf7d0 0x7f3e0415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e0b600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e040cfc30 con 0x7f3e0415e1e0 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e0b600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e040b1ee0 con 0x7f3e04143ae0 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e0b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040b2780 con 0x7f3e040cf7d0 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.455+0000 7f3e08c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e04143ae0 0x7f3e040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e04143ae0 0x7f3e040cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50792/0 (socket says 172.21.15.125:50792) 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 -- 172.21.15.125:0/309756598 learned_addr learned my addr 172.21.15.125:0/309756598 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3dfbc00640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e040cf7d0 0x7f3e0415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 -- 172.21.15.125:0/309756598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e040cf7d0 msgr2=0x7f3e0415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e040cf7d0 0x7f3e0415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 -- 172.21.15.125:0/309756598 >> v1:172.21.15.125:6789/0 conn(0x7f3e0415e1e0 legacy=0x7f3e0415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.456+0000 7f3e08c00640 1 -- 172.21.15.125:0/309756598 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0415e990 con 0x7f3e04143ae0 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.457+0000 7f3e08c00640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e040cf290 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009e50 tx=0x7f3df402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a7300eedee5bef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.457+0000 7f3e03e00640 1 -- 172.21.15.125:0/309756598 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df402cb00 con 0x7f3e04143ae0 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e03e00640 1 -- 172.21.15.125:0/309756598 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df402cc60 con 0x7f3e04143ae0 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e03e00640 1 -- 172.21.15.125:0/309756598 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40343f0 con 0x7f3e04143ae0 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 -- 172.21.15.125:0/309756598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 msgr2=0x7f3e040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e040cf290 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009e50 tx=0x7f3df402bf60 comp rx=0 tx=0).stop 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 -- 172.21.15.125:0/309756598 shutdown_connections 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e040cf7d0 0x7f3e0415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 --2- 172.21.15.125:0/309756598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e040cf290 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 -- 172.21.15.125:0/309756598 >> 172.21.15.125:0/309756598 conn(0x7f3e0406a800 msgr2=0x7f3e0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 -- 172.21.15.125:0/309756598 shutdown_connections 2024-01-16T15:02:09.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.458+0000 7f3e0b600640 1 -- 172.21.15.125:0/309756598 wait complete. 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.460+0000 7f3e0b600640 1 Processor -- start 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.460+0000 7f3e0b600640 1 -- start start 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e040cf7d0 0x7f3e040ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e0415e1e0 0x7f3e040acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040cd7f0 con 0x7f3e04143ae0 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040cd360 con 0x7f3e040cf7d0 2024-01-16T15:02:09.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3e0b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040cc6b0 con 0x7f3e0415e1e0 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3dfbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3dfbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50808/0 (socket says 172.21.15.125:50808) 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.461+0000 7f3dfbc00640 1 -- 172.21.15.125:0/3221645210 learned_addr learned my addr 172.21.15.125:0/3221645210 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3e08c00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e040cf7d0 0x7f3e040ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 -- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e0415e1e0 msgr2=0x7f3e040acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3e09600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e0415e1e0 0x7f3e040acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e0415e1e0 0x7f3e040acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 -- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e040cf7d0 msgr2=0x7f3e040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e040cf7d0 0x7f3e040ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 -- 172.21.15.125:0/3221645210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4009b30 con 0x7f3e04143ae0 2024-01-16T15:02:09.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.462+0000 7f3dfbc00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00d860 tx=0x7f3dec00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.463+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00de50 con 0x7f3e04143ae0 2024-01-16T15:02:09.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.463+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec00b8d0 con 0x7f3e04143ae0 2024-01-16T15:02:09.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.463+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00bc60 con 0x7f3e04143ae0 2024-01-16T15:02:09.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.463+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e04159b10 con 0x7f3e04143ae0 2024-01-16T15:02:09.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.463+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e040cc920 con 0x7f3e04143ae0 2024-01-16T15:02:09.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.464+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e040b2bf0 con 0x7f3e04143ae0 2024-01-16T15:02:09.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.465+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f3dec01e460 con 0x7f3e04143ae0 2024-01-16T15:02:09.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.465+0000 7f3e01600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 0x7f3ddc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.466+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dec08ac80 con 0x7f3e04143ae0 2024-01-16T15:02:09.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.466+0000 7f3e08c00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 0x7f3ddc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.466+0000 7f3e08c00640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 0x7f3ddc06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3df4035000 tx=0x7f3df4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.470+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e040b2bf0 con 0x7f3e04143ae0 2024-01-16T15:02:09.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.664+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e040b2bf0 con 0x7f3e04143ae0 2024-01-16T15:02:09.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.666+0000 7f3e01600640 1 -- 172.21.15.125:0/3221645210 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16613 (secure 0 0 0) 0x7f3e040b2bf0 con 0x7f3e04143ae0 2024-01-16T15:02:09.647 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:09.647 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:02:06.686697+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:02.926348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:04.133789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.668+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 msgr2=0x7f3ddc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.668+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 0x7f3ddc06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3df4035000 tx=0x7f3df4030000 comp rx=0 tx=0).stop 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.668+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 msgr2=0x7f3e04159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.668+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00d860 tx=0x7f3dec00dd20 comp rx=0 tx=0).stop 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 shutdown_connections 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e0415e1e0 0x7f3e040acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ddc06b860 0x7f3ddc06dd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e04143ae0 0x7f3e04159010 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 --2- 172.21.15.125:0/3221645210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e040cf7d0 0x7f3e040ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 >> 172.21.15.125:0/3221645210 conn(0x7f3e0406a800 msgr2=0x7f3e0406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 shutdown_connections 2024-01-16T15:02:09.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.669+0000 7f3e0b600640 1 -- 172.21.15.125:0/3221645210 wait complete. 2024-01-16T15:02:09.669 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:02:09.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:09.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.888+0000 7fb37ae00640 1 Processor -- start 2024-01-16T15:02:09.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.888+0000 7fb37ae00640 1 -- start start 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.889+0000 7fb37ae00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb374143ae0 0x7fb3740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb37ae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3740cfc30 con 0x7fb3740cf7d0 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb37ae00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3740b1ee0 con 0x7fb37415e3b0 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb37ae00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3740b2780 con 0x7fb374143ae0 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb374143ae0 0x7fb3740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb374143ae0 0x7fb3740cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50812/0 (socket says 172.21.15.125:50812) 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 -- 172.21.15.125:0/1884375226 learned_addr learned my addr 172.21.15.125:0/1884375226 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:09.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb36b200640 1 --1- 172.21.15.125:0/1884375226 >> v1:172.21.15.125:6789/0 conn(0x7fb3740cf7d0 0x7fb37415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60744/0 (socket says 172.21.15.125:60744) 2024-01-16T15:02:09.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 -- 172.21.15.125:0/1884375226 >> v1:172.21.15.137:6789/0 conn(0x7fb37415e3b0 legacy=0x7fb3740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 -- 172.21.15.125:0/1884375226 >> v1:172.21.15.125:6789/0 conn(0x7fb3740cf7d0 legacy=0x7fb37415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373e00640 1 -- 172.21.15.125:0/1884375226 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37415e990 con 0x7fb374143ae0 2024-01-16T15:02:09.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.890+0000 7fb373400640 1 -- 172.21.15.125:0/1884375226 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851869892 0 0) 0x7fb3740cfc30 con 0x7fb3740cf7d0 2024-01-16T15:02:09.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.891+0000 7fb373e00640 1 --2- 172.21.15.125:0/1884375226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb374143ae0 0x7fb3740cf290 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb364009870 tx=0x7fb36402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa4e4b506b4a76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb373400640 1 -- 172.21.15.125:0/1884375226 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364030860 con 0x7fb374143ae0 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb373400640 1 -- 172.21.15.125:0/1884375226 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640309c0 con 0x7fb374143ae0 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb373400640 1 -- 172.21.15.125:0/1884375226 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364030d50 con 0x7fb374143ae0 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 -- 172.21.15.125:0/1884375226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb374143ae0 msgr2=0x7fb3740cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 --2- 172.21.15.125:0/1884375226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb374143ae0 0x7fb3740cf290 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb364009870 tx=0x7fb36402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 -- 172.21.15.125:0/1884375226 shutdown_connections 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 --2- 172.21.15.125:0/1884375226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb374143ae0 0x7fb3740cf290 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 -- 172.21.15.125:0/1884375226 >> 172.21.15.125:0/1884375226 conn(0x7fb37406a800 msgr2=0x7fb37406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.892+0000 7fb37ae00640 1 -- 172.21.15.125:0/1884375226 shutdown_connections 2024-01-16T15:02:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.893+0000 7fb37ae00640 1 -- 172.21.15.125:0/1884375226 wait complete. 2024-01-16T15:02:09.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.894+0000 7fb37ae00640 1 Processor -- start 2024-01-16T15:02:09.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.894+0000 7fb37ae00640 1 -- start start 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb374143ae0 0x7fb374142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 0x7fb37415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413ce40 con 0x7fb3740cf7d0 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413d6e0 con 0x7fb37415e3b0 2024-01-16T15:02:09.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb37ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413e0b0 con 0x7fb374143ae0 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.895+0000 7fb373e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50828/0 (socket says 172.21.15.125:50828) 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 -- 172.21.15.125:0/953458519 learned_addr learned my addr 172.21.15.125:0/953458519 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb36b200640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb374143ae0 0x7fb374142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb378e00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 0x7fb37415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 -- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb374143ae0 msgr2=0x7fb374142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb374143ae0 0x7fb374142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 -- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 msgr2=0x7fb37415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 0x7fb37415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 -- 172.21.15.125:0/953458519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640094d0 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb378e00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 0x7fb37415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.896+0000 7fb373e00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.897+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364039d60 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.897+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364043400 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.897+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3740af9d0 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.897+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640437b0 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.897+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37413d370 con 0x7fb3740cf7d0 2024-01-16T15:02:09.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.898+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740675d0 con 0x7fb3740cf7d0 2024-01-16T15:02:09.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.899+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb364043910 con 0x7fb3740cf7d0 2024-01-16T15:02:09.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.900+0000 7fb370c00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 0x7fb34c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.900+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb36403e070 con 0x7fb3740cf7d0 2024-01-16T15:02:09.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.900+0000 7fb36b200640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 0x7fb34c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.901+0000 7fb36b200640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 0x7fb34c06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb35c007fd0 tx=0x7fb35c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:09.904+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb364077fd0 con 0x7fb3740cf7d0 2024-01-16T15:02:10.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.090+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3740b2410 con 0x7fb34c06b860 2024-01-16T15:02:10.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.116+0000 7fb370c00640 1 -- 172.21.15.125:0/953458519 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287335 (secure 0 0 0) 0x7fb35c001c70 con 0x7fb34c06b860 2024-01-16T15:02:10.101 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:10.103 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.121+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 msgr2=0x7fb34c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.121+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 0x7fb34c06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb35c007fd0 tx=0x7fb35c009040 comp rx=0 tx=0).stop 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.121+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 msgr2=0x7fb37413db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.121+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364004730 comp rx=0 tx=0).stop 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 shutdown_connections 2024-01-16T15:02:10.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb37415e3b0 0x7fb37415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb374143ae0 0x7fb374142750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb34c06b860 0x7fb34c06dd20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 --2- 172.21.15.125:0/953458519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740cf7d0 0x7fb37413db70 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 >> 172.21.15.125:0/953458519 conn(0x7fb37406a800 msgr2=0x7fb37406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 shutdown_connections 2024-01-16T15:02:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.122+0000 7fb37ae00640 1 -- 172.21.15.125:0/953458519 wait complete. 2024-01-16T15:02:10.121 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-16T15:02:10.028024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289408,"kb_used_data":6660,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685312,"statfs":{"total":773094113280,"available":772797759488,"internally_reserved":0,"allocated":6819840,"data_stored":3509425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001825"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.704001+0000","last_change":"2024-01-16T15:02:05.714070+0000","last_active":"2024-01-16T15:02:06.704001+0000","last_peered":"2024-01-16T15:02:06.704001+0000","last_clean":"2024-01-16T15:02:06.704001+0000","last_became_active":"2024-01-16T15:02:05.713522+0000","last_became_peered":"2024-01-16T15:02:05.713522+0000","last_unstale":"2024-01-16T15:02:06.704001+0000","last_undegraded":"2024-01-16T15:02:06.704001+0000","last_fullsized":"2024-01-16T15:02:06.704001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:01.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763124+0000","last_change":"2024-01-16T15:02:05.723395+0000","last_active":"2024-01-16T15:02:06.763124+0000","last_peered":"2024-01-16T15:02:06.763124+0000","last_clean":"2024-01-16T15:02:06.763124+0000","last_became_active":"2024-01-16T15:02:05.723125+0000","last_became_peered":"2024-01-16T15:02:05.723125+0000","last_unstale":"2024-01-16T15:02:06.763124+0000","last_undegraded":"2024-01-16T15:02:06.763124+0000","last_fullsized":"2024-01-16T15:02:06.763124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:45.802093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707914+0000","last_change":"2024-01-16T15:02:05.709039+0000","last_active":"2024-01-16T15:02:06.707914+0000","last_peered":"2024-01-16T15:02:06.707914+0000","last_clean":"2024-01-16T15:02:06.707914+0000","last_became_active":"2024-01-16T15:02:05.707242+0000","last_became_peered":"2024-01-16T15:02:05.707242+0000","last_unstale":"2024-01-16T15:02:06.707914+0000","last_undegraded":"2024-01-16T15:02:06.707914+0000","last_fullsized":"2024-01-16T15:02:06.707914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764834+0000","last_change":"2024-01-16T15:02:05.714976+0000","last_active":"2024-01-16T15:02:06.764834+0000","last_peered":"2024-01-16T15:02:06.764834+0000","last_clean":"2024-01-16T15:02:06.764834+0000","last_became_active":"2024-01-16T15:02:05.714437+0000","last_became_peered":"2024-01-16T15:02:05.714437+0000","last_unstale":"2024-01-16T15:02:06.764834+0000","last_undegraded":"2024-01-16T15:02:06.764834+0000","last_fullsized":"2024-01-16T15:02:06.764834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703496+0000","last_change":"2024-01-16T15:02:05.717167+0000","last_active":"2024-01-16T15:02:06.703496+0000","last_peered":"2024-01-16T15:02:06.703496+0000","last_clean":"2024-01-16T15:02:06.703496+0000","last_became_active":"2024-01-16T15:02:05.716887+0000","last_became_peered":"2024-01-16T15:02:05.716887+0000","last_unstale":"2024-01-16T15:02:06.703496+0000","last_undegraded":"2024-01-16T15:02:06.703496+0000","last_fullsized":"2024-01-16T15:02:06.703496+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:56.700044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763422+0000","last_change":"2024-01-16T15:02:05.695959+0000","last_active":"2024-01-16T15:02:06.763422+0000","last_peered":"2024-01-16T15:02:06.763422+0000","last_clean":"2024-01-16T15:02:06.763422+0000","last_became_active":"2024-01-16T15:02:05.694589+0000","last_became_peered":"2024-01-16T15:02:05.694589+0000","last_unstale":"2024-01-16T15:02:06.763422+0000","last_undegraded":"2024-01-16T15:02:06.763422+0000","last_fullsized":"2024-01-16T15:02:06.763422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762937+0000","last_change":"2024-01-16T15:02:05.710640+0000","last_active":"2024-01-16T15:02:06.762937+0000","last_peered":"2024-01-16T15:02:06.762937+0000","last_clean":"2024-01-16T15:02:06.762937+0000","last_became_active":"2024-01-16T15:02:05.710314+0000","last_became_peered":"2024-01-16T15:02:05.710314+0000","last_unstale":"2024-01-16T15:02:06.762937+0000","last_undegraded":"2024-01-16T15:02:06.762937+0000","last_fullsized":"2024-01-16T15:02:06.762937+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:09.731041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761071+0000","last_change":"2024-01-16T15:02:05.713833+0000","last_active":"2024-01-16T15:02:06.761071+0000","last_peered":"2024-01-16T15:02:06.761071+0000","last_clean":"2024-01-16T15:02:06.761071+0000","last_became_active":"2024-01-16T15:02:05.713568+0000","last_became_peered":"2024-01-16T15:02:05.713568+0000","last_unstale":"2024-01-16T15:02:06.761071+0000","last_undegraded":"2024-01-16T15:02:06.761071+0000","last_fullsized":"2024-01-16T15:02:06.761071+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:50.726560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764373+0000","last_change":"2024-01-16T15:02:05.709577+0000","last_active":"2024-01-16T15:02:06.764373+0000","last_peered":"2024-01-16T15:02:06.764373+0000","last_clean":"2024-01-16T15:02:06.764373+0000","last_became_active":"2024-01-16T15:02:05.709256+0000","last_became_peered":"2024-01-16T15:02:05.709256+0000","last_unstale":"2024-01-16T15:02:06.764373+0000","last_undegraded":"2024-01-16T15:02:06.764373+0000","last_fullsized":"2024-01-16T15:02:06.764373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:19.298378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761074+0000","last_change":"2024-01-16T15:02:05.711654+0000","last_active":"2024-01-16T15:02:06.761074+0000","last_peered":"2024-01-16T15:02:06.761074+0000","last_clean":"2024-01-16T15:02:06.761074+0000","last_became_active":"2024-01-16T15:02:05.711368+0000","last_became_peered":"2024-01-16T15:02:05.711368+0000","last_unstale":"2024-01-16T15:02:06.761074+0000","last_undegraded":"2024-01-16T15:02:06.761074+0000","last_fullsized":"2024-01-16T15:02:06.761074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.546955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761517+0000","last_change":"2024-01-16T15:02:05.706192+0000","last_active":"2024-01-16T15:02:06.761517+0000","last_peered":"2024-01-16T15:02:06.761517+0000","last_clean":"2024-01-16T15:02:06.761517+0000","last_became_active":"2024-01-16T15:02:05.705880+0000","last_became_peered":"2024-01-16T15:02:05.705880+0000","last_unstale":"2024-01-16T15:02:06.761517+0000","last_undegraded":"2024-01-16T15:02:06.761517+0000","last_fullsized":"2024-01-16T15:02:06.761517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:34.237454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764893+0000","last_change":"2024-01-16T15:02:05.714792+0000","last_active":"2024-01-16T15:02:06.764893+0000","last_peered":"2024-01-16T15:02:06.764893+0000","last_clean":"2024-01-16T15:02:06.764893+0000","last_became_active":"2024-01-16T15:02:05.714103+0000","last_became_peered":"2024-01-16T15:02:05.714103+0000","last_unstale":"2024-01-16T15:02:06.764893+0000","last_undegraded":"2024-01-16T15:02:06.764893+0000","last_fullsized":"2024-01-16T15:02:06.764893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708928+0000","last_change":"2024-01-16T15:02:05.707657+0000","last_active":"2024-01-16T15:02:06.708928+0000","last_peered":"2024-01-16T15:02:06.708928+0000","last_clean":"2024-01-16T15:02:06.708928+0000","last_became_active":"2024-01-16T15:02:05.707260+0000","last_became_peered":"2024-01-16T15:02:05.707260+0000","last_unstale":"2024-01-16T15:02:06.708928+0000","last_undegraded":"2024-01-16T15:02:06.708928+0000","last_fullsized":"2024-01-16T15:02:06.708928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:40.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763558+0000","last_change":"2024-01-16T15:02:05.704314+0000","last_active":"2024-01-16T15:02:06.763558+0000","last_peered":"2024-01-16T15:02:06.763558+0000","last_clean":"2024-01-16T15:02:06.763558+0000","last_became_active":"2024-01-16T15:02:05.703400+0000","last_became_peered":"2024-01-16T15:02:05.703400+0000","last_unstale":"2024-01-16T15:02:06.763558+0000","last_undegraded":"2024-01-16T15:02:06.763558+0000","last_fullsized":"2024-01-16T15:02:06.763558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:32:49.727167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763211+0000","last_change":"2024-01-16T15:02:05.703743+0000","last_active":"2024-01-16T15:02:06.763211+0000","last_peered":"2024-01-16T15:02:06.763211+0000","last_clean":"2024-01-16T15:02:06.763211+0000","last_became_active":"2024-01-16T15:02:05.703430+0000","last_became_peered":"2024-01-16T15:02:05.703430+0000","last_unstale":"2024-01-16T15:02:06.763211+0000","last_undegraded":"2024-01-16T15:02:06.763211+0000","last_fullsized":"2024-01-16T15:02:06.763211+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:31:22.593889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762237+0000","last_change":"2024-01-16T15:02:05.707433+0000","last_active":"2024-01-16T15:02:06.762237+0000","last_peered":"2024-01-16T15:02:06.762237+0000","last_clean":"2024-01-16T15:02:06.762237+0000","last_became_active":"2024-01-16T15:02:05.706513+0000","last_became_peered":"2024-01-16T15:02:05.706513+0000","last_unstale":"2024-01-16T15:02:06.762237+0000","last_undegraded":"2024-01-16T15:02:06.762237+0000","last_fullsized":"2024-01-16T15:02:06.762237+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:31.344585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763670+0000","last_change":"2024-01-16T15:02:05.715908+0000","last_active":"2024-01-16T15:02:06.763670+0000","last_peered":"2024-01-16T15:02:06.763670+0000","last_clean":"2024-01-16T15:02:06.763670+0000","last_became_active":"2024-01-16T15:02:05.715613+0000","last_became_peered":"2024-01-16T15:02:05.715613+0000","last_unstale":"2024-01-16T15:02:06.763670+0000","last_undegraded":"2024-01-16T15:02:06.763670+0000","last_fullsized":"2024-01-16T15:02:06.763670+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:01.259511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708872+0000","last_change":"2024-01-16T15:02:05.702398+0000","last_active":"2024-01-16T15:02:06.708872+0000","last_peered":"2024-01-16T15:02:06.708872+0000","last_clean":"2024-01-16T15:02:06.708872+0000","last_became_active":"2024-01-16T15:02:05.702092+0000","last_became_peered":"2024-01-16T15:02:05.702092+0000","last_unstale":"2024-01-16T15:02:06.708872+0000","last_undegraded":"2024-01-16T15:02:06.708872+0000","last_fullsized":"2024-01-16T15:02:06.708872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703535+0000","last_change":"2024-01-16T15:02:05.694202+0000","last_active":"2024-01-16T15:02:06.703535+0000","last_peered":"2024-01-16T15:02:06.703535+0000","last_clean":"2024-01-16T15:02:06.703535+0000","last_became_active":"2024-01-16T15:02:05.693873+0000","last_became_peered":"2024-01-16T15:02:05.693873+0000","last_unstale":"2024-01-16T15:02:06.703535+0000","last_undegraded":"2024-01-16T15:02:06.703535+0000","last_fullsized":"2024-01-16T15:02:06.703535+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:53.845071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762787+0000","last_change":"2024-01-16T15:02:05.705564+0000","last_active":"2024-01-16T15:02:06.762787+0000","last_peered":"2024-01-16T15:02:06.762787+0000","last_clean":"2024-01-16T15:02:06.762787+0000","last_became_active":"2024-01-16T15:02:05.705315+0000","last_became_peered":"2024-01-16T15:02:05.705315+0000","last_unstale":"2024-01-16T15:02:06.762787+0000","last_undegraded":"2024-01-16T15:02:06.762787+0000","last_fullsized":"2024-01-16T15:02:06.762787+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:11.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707914+0000","last_change":"2024-01-16T15:02:05.709428+0000","last_active":"2024-01-16T15:02:06.707914+0000","last_peered":"2024-01-16T15:02:06.707914+0000","last_clean":"2024-01-16T15:02:06.707914+0000","last_became_active":"2024-01-16T15:02:05.709149+0000","last_became_peered":"2024-01-16T15:02:05.709149+0000","last_unstale":"2024-01-16T15:02:06.707914+0000","last_undegraded":"2024-01-16T15:02:06.707914+0000","last_fullsized":"2024-01-16T15:02:06.707914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.760967+0000","last_change":"2024-01-16T15:02:05.713281+0000","last_active":"2024-01-16T15:02:06.760967+0000","last_peered":"2024-01-16T15:02:06.760967+0000","last_clean":"2024-01-16T15:02:06.760967+0000","last_became_active":"2024-01-16T15:02:05.713026+0000","last_became_peered":"2024-01-16T15:02:05.713026+0000","last_unstale":"2024-01-16T15:02:06.760967+0000","last_undegraded":"2024-01-16T15:02:06.760967+0000","last_fullsized":"2024-01-16T15:02:06.760967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:55.945285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709164+0000","last_change":"2024-01-16T15:02:05.696922+0000","last_active":"2024-01-16T15:02:06.709164+0000","last_peered":"2024-01-16T15:02:06.709164+0000","last_clean":"2024-01-16T15:02:06.709164+0000","last_became_active":"2024-01-16T15:02:05.696343+0000","last_became_peered":"2024-01-16T15:02:05.696343+0000","last_unstale":"2024-01-16T15:02:06.709164+0000","last_undegraded":"2024-01-16T15:02:06.709164+0000","last_fullsized":"2024-01-16T15:02:06.709164+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:18.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707357+0000","last_change":"2024-01-16T15:02:05.705546+0000","last_active":"2024-01-16T15:02:06.707357+0000","last_peered":"2024-01-16T15:02:06.707357+0000","last_clean":"2024-01-16T15:02:06.707357+0000","last_became_active":"2024-01-16T15:02:05.705188+0000","last_became_peered":"2024-01-16T15:02:05.705188+0000","last_unstale":"2024-01-16T15:02:06.707357+0000","last_undegraded":"2024-01-16T15:02:06.707357+0000","last_fullsized":"2024-01-16T15:02:06.707357+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:51.004607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703915+0000","last_change":"2024-01-16T15:02:05.689999+0000","last_active":"2024-01-16T15:02:06.703915+0000","last_peered":"2024-01-16T15:02:06.703915+0000","last_clean":"2024-01-16T15:02:06.703915+0000","last_became_active":"2024-01-16T15:02:05.689755+0000","last_became_peered":"2024-01-16T15:02:05.689755+0000","last_unstale":"2024-01-16T15:02:06.703915+0000","last_undegraded":"2024-01-16T15:02:06.703915+0000","last_fullsized":"2024-01-16T15:02:06.703915+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:25.508676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765184+0000","last_change":"2024-01-16T15:02:05.713769+0000","last_active":"2024-01-16T15:02:06.765184+0000","last_peered":"2024-01-16T15:02:06.765184+0000","last_clean":"2024-01-16T15:02:06.765184+0000","last_became_active":"2024-01-16T15:02:05.713228+0000","last_became_peered":"2024-01-16T15:02:05.713228+0000","last_unstale":"2024-01-16T15:02:06.765184+0000","last_undegraded":"2024-01-16T15:02:06.765184+0000","last_fullsized":"2024-01-16T15:02:06.765184+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762676+0000","last_change":"2024-01-16T15:02:05.709062+0000","last_active":"2024-01-16T15:02:06.762676+0000","last_peered":"2024-01-16T15:02:06.762676+0000","last_clean":"2024-01-16T15:02:06.762676+0000","last_became_active":"2024-01-16T15:02:05.706597+0000","last_became_peered":"2024-01-16T15:02:05.706597+0000","last_unstale":"2024-01-16T15:02:06.762676+0000","last_undegraded":"2024-01-16T15:02:06.762676+0000","last_fullsized":"2024-01-16T15:02:06.762676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703369+0000","last_change":"2024-01-16T15:02:05.718074+0000","last_active":"2024-01-16T15:02:06.703369+0000","last_peered":"2024-01-16T15:02:06.703369+0000","last_clean":"2024-01-16T15:02:06.703369+0000","last_became_active":"2024-01-16T15:02:05.717777+0000","last_became_peered":"2024-01-16T15:02:05.717777+0000","last_unstale":"2024-01-16T15:02:06.703369+0000","last_undegraded":"2024-01-16T15:02:06.703369+0000","last_fullsized":"2024-01-16T15:02:06.703369+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:56.077780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762335+0000","last_change":"2024-01-16T15:02:05.721079+0000","last_active":"2024-01-16T15:02:06.762335+0000","last_peered":"2024-01-16T15:02:06.762335+0000","last_clean":"2024-01-16T15:02:06.762335+0000","last_became_active":"2024-01-16T15:02:05.720714+0000","last_became_peered":"2024-01-16T15:02:05.720714+0000","last_unstale":"2024-01-16T15:02:06.762335+0000","last_undegraded":"2024-01-16T15:02:06.762335+0000","last_fullsized":"2024-01-16T15:02:06.762335+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:10:07.921797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708137+0000","last_change":"2024-01-16T15:02:05.708735+0000","last_active":"2024-01-16T15:02:06.708137+0000","last_peered":"2024-01-16T15:02:06.708137+0000","last_clean":"2024-01-16T15:02:06.708137+0000","last_became_active":"2024-01-16T15:02:05.708446+0000","last_became_peered":"2024-01-16T15:02:05.708446+0000","last_unstale":"2024-01-16T15:02:06.708137+0000","last_undegraded":"2024-01-16T15:02:06.708137+0000","last_fullsized":"2024-01-16T15:02:06.708137+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.702823+0000","last_change":"2024-01-16T15:02:05.690002+0000","last_active":"2024-01-16T15:02:06.702823+0000","last_peered":"2024-01-16T15:02:06.702823+0000","last_clean":"2024-01-16T15:02:06.702823+0000","last_became_active":"2024-01-16T15:02:05.689516+0000","last_became_peered":"2024-01-16T15:02:05.689516+0000","last_unstale":"2024-01-16T15:02:06.702823+0000","last_undegraded":"2024-01-16T15:02:06.702823+0000","last_fullsized":"2024-01-16T15:02:06.702823+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:52.702885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762242+0000","last_change":"2024-01-16T15:02:05.718995+0000","last_active":"2024-01-16T15:02:06.762242+0000","last_peered":"2024-01-16T15:02:06.762242+0000","last_clean":"2024-01-16T15:02:06.762242+0000","last_became_active":"2024-01-16T15:02:05.718727+0000","last_became_peered":"2024-01-16T15:02:05.718727+0000","last_unstale":"2024-01-16T15:02:06.762242+0000","last_undegraded":"2024-01-16T15:02:06.762242+0000","last_fullsized":"2024-01-16T15:02:06.762242+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:58.504291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763703+0000","last_change":"2024-01-16T15:02:05.711882+0000","last_active":"2024-01-16T15:02:06.763703+0000","last_peered":"2024-01-16T15:02:06.763703+0000","last_clean":"2024-01-16T15:02:06.763703+0000","last_became_active":"2024-01-16T15:02:05.711545+0000","last_became_peered":"2024-01-16T15:02:05.711545+0000","last_unstale":"2024-01-16T15:02:06.763703+0000","last_undegraded":"2024-01-16T15:02:06.763703+0000","last_fullsized":"2024-01-16T15:02:06.763703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.719724+0000","last_change":"2024-01-16T15:02:04.677845+0000","last_active":"2024-01-16T15:02:05.719724+0000","last_peered":"2024-01-16T15:02:05.719724+0000","last_clean":"2024-01-16T15:02:05.719724+0000","last_became_active":"2024-01-16T15:02:04.674448+0000","last_became_peered":"2024-01-16T15:02:04.674448+0000","last_unstale":"2024-01-16T15:02:05.719724+0000","last_undegraded":"2024-01-16T15:02:05.719724+0000","last_fullsized":"2024-01-16T15:02:05.719724+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.508381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762560+0000","last_change":"2024-01-16T15:02:05.701672+0000","last_active":"2024-01-16T15:02:06.762560+0000","last_peered":"2024-01-16T15:02:06.762560+0000","last_clean":"2024-01-16T15:02:06.762560+0000","last_became_active":"2024-01-16T15:02:05.701181+0000","last_became_peered":"2024-01-16T15:02:05.701181+0000","last_unstale":"2024-01-16T15:02:06.762560+0000","last_undegraded":"2024-01-16T15:02:06.762560+0000","last_fullsized":"2024-01-16T15:02:06.762560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:40.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.717340+0000","last_change":"2024-01-16T15:02:04.678078+0000","last_active":"2024-01-16T15:02:05.717340+0000","last_peered":"2024-01-16T15:02:05.717340+0000","last_clean":"2024-01-16T15:02:05.717340+0000","last_became_active":"2024-01-16T15:02:04.677785+0000","last_became_peered":"2024-01-16T15:02:04.677785+0000","last_unstale":"2024-01-16T15:02:05.717340+0000","last_undegraded":"2024-01-16T15:02:05.717340+0000","last_fullsized":"2024-01-16T15:02:05.717340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:56.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763117+0000","last_change":"2024-01-16T15:02:05.695668+0000","last_active":"2024-01-16T15:02:06.763117+0000","last_peered":"2024-01-16T15:02:06.763117+0000","last_clean":"2024-01-16T15:02:06.763117+0000","last_became_active":"2024-01-16T15:02:05.695320+0000","last_became_peered":"2024-01-16T15:02:05.695320+0000","last_unstale":"2024-01-16T15:02:06.763117+0000","last_undegraded":"2024-01-16T15:02:06.763117+0000","last_fullsized":"2024-01-16T15:02:06.763117+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.702762+0000","last_change":"2024-01-16T15:02:04.677451+0000","last_active":"2024-01-16T15:02:05.702762+0000","last_peered":"2024-01-16T15:02:05.702762+0000","last_clean":"2024-01-16T15:02:05.702762+0000","last_became_active":"2024-01-16T15:02:04.676639+0000","last_became_peered":"2024-01-16T15:02:04.676639+0000","last_unstale":"2024-01-16T15:02:05.702762+0000","last_undegraded":"2024-01-16T15:02:05.702762+0000","last_fullsized":"2024-01-16T15:02:05.702762+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:45.344394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762756+0000","last_change":"2024-01-16T15:02:05.719489+0000","last_active":"2024-01-16T15:02:06.762756+0000","last_peered":"2024-01-16T15:02:06.762756+0000","last_clean":"2024-01-16T15:02:06.762756+0000","last_became_active":"2024-01-16T15:02:05.719272+0000","last_became_peered":"2024-01-16T15:02:05.719272+0000","last_unstale":"2024-01-16T15:02:06.762756+0000","last_undegraded":"2024-01-16T15:02:06.762756+0000","last_fullsized":"2024-01-16T15:02:06.762756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.717131+0000","last_change":"2024-01-16T15:02:04.680997+0000","last_active":"2024-01-16T15:02:05.717131+0000","last_peered":"2024-01-16T15:02:05.717131+0000","last_clean":"2024-01-16T15:02:05.717131+0000","last_became_active":"2024-01-16T15:02:04.680677+0000","last_became_peered":"2024-01-16T15:02:04.680677+0000","last_unstale":"2024-01-16T15:02:05.717131+0000","last_undegraded":"2024-01-16T15:02:05.717131+0000","last_fullsized":"2024-01-16T15:02:05.717131+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:30.329959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761987+0000","last_change":"2024-01-16T15:02:05.712735+0000","last_active":"2024-01-16T15:02:06.761987+0000","last_peered":"2024-01-16T15:02:06.761987+0000","last_clean":"2024-01-16T15:02:06.761987+0000","last_became_active":"2024-01-16T15:02:05.712459+0000","last_became_peered":"2024-01-16T15:02:05.712459+0000","last_unstale":"2024-01-16T15:02:06.761987+0000","last_undegraded":"2024-01-16T15:02:06.761987+0000","last_fullsized":"2024-01-16T15:02:06.761987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:24.174184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.713467+0000","last_change":"2024-01-16T15:02:04.688609+0000","last_active":"2024-01-16T15:02:05.713467+0000","last_peered":"2024-01-16T15:02:05.713467+0000","last_clean":"2024-01-16T15:02:05.713467+0000","last_became_active":"2024-01-16T15:02:04.688253+0000","last_became_peered":"2024-01-16T15:02:04.688253+0000","last_unstale":"2024-01-16T15:02:05.713467+0000","last_undegraded":"2024-01-16T15:02:05.713467+0000","last_fullsized":"2024-01-16T15:02:05.713467+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:21.918624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762078+0000","last_change":"2024-01-16T15:02:05.700757+0000","last_active":"2024-01-16T15:02:06.762078+0000","last_peered":"2024-01-16T15:02:06.762078+0000","last_clean":"2024-01-16T15:02:06.762078+0000","last_became_active":"2024-01-16T15:02:05.700491+0000","last_became_peered":"2024-01-16T15:02:05.700491+0000","last_unstale":"2024-01-16T15:02:06.762078+0000","last_undegraded":"2024-01-16T15:02:06.762078+0000","last_fullsized":"2024-01-16T15:02:06.762078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:06:29.331207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.715892+0000","last_change":"2024-01-16T15:02:04.673177+0000","last_active":"2024-01-16T15:02:05.715892+0000","last_peered":"2024-01-16T15:02:05.715892+0000","last_clean":"2024-01-16T15:02:05.715892+0000","last_became_active":"2024-01-16T15:02:04.672346+0000","last_became_peered":"2024-01-16T15:02:04.672346+0000","last_unstale":"2024-01-16T15:02:05.715892+0000","last_undegraded":"2024-01-16T15:02:05.715892+0000","last_fullsized":"2024-01-16T15:02:05.715892+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:45.746043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707796+0000","last_change":"2024-01-16T15:02:05.717824+0000","last_active":"2024-01-16T15:02:06.707796+0000","last_peered":"2024-01-16T15:02:06.707796+0000","last_clean":"2024-01-16T15:02:06.707796+0000","last_became_active":"2024-01-16T15:02:05.717046+0000","last_became_peered":"2024-01-16T15:02:05.717046+0000","last_unstale":"2024-01-16T15:02:06.707796+0000","last_undegraded":"2024-01-16T15:02:06.707796+0000","last_fullsized":"2024-01-16T15:02:06.707796+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:34.237454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694691+0000","last_change":"2024-01-16T15:02:04.682008+0000","last_active":"2024-01-16T15:02:05.694691+0000","last_peered":"2024-01-16T15:02:05.694691+0000","last_clean":"2024-01-16T15:02:05.694691+0000","last_became_active":"2024-01-16T15:02:04.681608+0000","last_became_peered":"2024-01-16T15:02:04.681608+0000","last_unstale":"2024-01-16T15:02:05.694691+0000","last_undegraded":"2024-01-16T15:02:05.694691+0000","last_fullsized":"2024-01-16T15:02:05.694691+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:32:48.712541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761301+0000","last_change":"2024-01-16T15:02:05.710209+0000","last_active":"2024-01-16T15:02:06.761301+0000","last_peered":"2024-01-16T15:02:06.761301+0000","last_clean":"2024-01-16T15:02:06.761301+0000","last_became_active":"2024-01-16T15:02:05.709954+0000","last_became_peered":"2024-01-16T15:02:05.709954+0000","last_unstale":"2024-01-16T15:02:06.761301+0000","last_undegraded":"2024-01-16T15:02:06.761301+0000","last_fullsized":"2024-01-16T15:02:06.761301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:21.899539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.702674+0000","last_change":"2024-01-16T15:02:04.676299+0000","last_active":"2024-01-16T15:02:05.702674+0000","last_peered":"2024-01-16T15:02:05.702674+0000","last_clean":"2024-01-16T15:02:05.702674+0000","last_became_active":"2024-01-16T15:02:04.675978+0000","last_became_peered":"2024-01-16T15:02:04.675978+0000","last_unstale":"2024-01-16T15:02:05.702674+0000","last_undegraded":"2024-01-16T15:02:05.702674+0000","last_fullsized":"2024-01-16T15:02:05.702674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:56.132562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763833+0000","last_change":"2024-01-16T15:02:05.708487+0000","last_active":"2024-01-16T15:02:06.763833+0000","last_peered":"2024-01-16T15:02:06.763833+0000","last_clean":"2024-01-16T15:02:06.763833+0000","last_became_active":"2024-01-16T15:02:05.708156+0000","last_became_peered":"2024-01-16T15:02:05.708156+0000","last_unstale":"2024-01-16T15:02:06.763833+0000","last_undegraded":"2024-01-16T15:02:06.763833+0000","last_fullsized":"2024-01-16T15:02:06.763833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:30.597006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.685434+0000","last_change":"2024-01-16T15:02:04.684694+0000","last_active":"2024-01-16T15:02:05.685434+0000","last_peered":"2024-01-16T15:02:05.685434+0000","last_clean":"2024-01-16T15:02:05.685434+0000","last_became_active":"2024-01-16T15:02:04.684270+0000","last_became_peered":"2024-01-16T15:02:04.684270+0000","last_unstale":"2024-01-16T15:02:05.685434+0000","last_undegraded":"2024-01-16T15:02:05.685434+0000","last_fullsized":"2024-01-16T15:02:05.685434+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:34.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709753+0000","last_change":"2024-01-16T15:02:05.709225+0000","last_active":"2024-01-16T15:02:06.709753+0000","last_peered":"2024-01-16T15:02:06.709753+0000","last_clean":"2024-01-16T15:02:06.709753+0000","last_became_active":"2024-01-16T15:02:05.706336+0000","last_became_peered":"2024-01-16T15:02:05.706336+0000","last_unstale":"2024-01-16T15:02:06.709753+0000","last_undegraded":"2024-01-16T15:02:06.709753+0000","last_fullsized":"2024-01-16T15:02:06.709753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:06.881538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.686092+0000","last_change":"2024-01-16T15:02:04.690135+0000","last_active":"2024-01-16T15:02:05.686092+0000","last_peered":"2024-01-16T15:02:05.686092+0000","last_clean":"2024-01-16T15:02:05.686092+0000","last_became_active":"2024-01-16T15:02:04.689704+0000","last_became_peered":"2024-01-16T15:02:04.689704+0000","last_unstale":"2024-01-16T15:02:05.686092+0000","last_undegraded":"2024-01-16T15:02:05.686092+0000","last_fullsized":"2024-01-16T15:02:05.686092+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:41.180213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761413+0000","last_change":"2024-01-16T15:02:05.721151+0000","last_active":"2024-01-16T15:02:06.761413+0000","last_peered":"2024-01-16T15:02:06.761413+0000","last_clean":"2024-01-16T15:02:06.761413+0000","last_became_active":"2024-01-16T15:02:05.720706+0000","last_became_peered":"2024-01-16T15:02:05.720706+0000","last_unstale":"2024-01-16T15:02:06.761413+0000","last_undegraded":"2024-01-16T15:02:06.761413+0000","last_fullsized":"2024-01-16T15:02:06.761413+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:41.520010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.718037+0000","last_change":"2024-01-16T15:02:04.683511+0000","last_active":"2024-01-16T15:02:05.718037+0000","last_peered":"2024-01-16T15:02:05.718037+0000","last_clean":"2024-01-16T15:02:05.718037+0000","last_became_active":"2024-01-16T15:02:04.683083+0000","last_became_peered":"2024-01-16T15:02:04.683083+0000","last_unstale":"2024-01-16T15:02:05.718037+0000","last_undegraded":"2024-01-16T15:02:05.718037+0000","last_fullsized":"2024-01-16T15:02:05.718037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:45.344394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708016+0000","last_change":"2024-01-16T15:02:05.694323+0000","last_active":"2024-01-16T15:02:06.708016+0000","last_peered":"2024-01-16T15:02:06.708016+0000","last_clean":"2024-01-16T15:02:06.708016+0000","last_became_active":"2024-01-16T15:02:05.694043+0000","last_became_peered":"2024-01-16T15:02:05.694043+0000","last_unstale":"2024-01-16T15:02:06.708016+0000","last_undegraded":"2024-01-16T15:02:06.708016+0000","last_fullsized":"2024-01-16T15:02:06.708016+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.726616+0000","last_change":"2024-01-16T15:02:04.676811+0000","last_active":"2024-01-16T15:02:05.726616+0000","last_peered":"2024-01-16T15:02:05.726616+0000","last_clean":"2024-01-16T15:02:05.726616+0000","last_became_active":"2024-01-16T15:02:04.676452+0000","last_became_peered":"2024-01-16T15:02:04.676452+0000","last_unstale":"2024-01-16T15:02:05.726616+0000","last_undegraded":"2024-01-16T15:02:05.726616+0000","last_fullsized":"2024-01-16T15:02:05.726616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:34.921650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764153+0000","last_change":"2024-01-16T15:02:05.722270+0000","last_active":"2024-01-16T15:02:06.764153+0000","last_peered":"2024-01-16T15:02:06.764153+0000","last_clean":"2024-01-16T15:02:06.764153+0000","last_became_active":"2024-01-16T15:02:05.722018+0000","last_became_peered":"2024-01-16T15:02:05.722018+0000","last_unstale":"2024-01-16T15:02:06.764153+0000","last_undegraded":"2024-01-16T15:02:06.764153+0000","last_fullsized":"2024-01-16T15:02:06.764153+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:58.504291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721472+0000","last_change":"2024-01-16T15:02:04.689950+0000","last_active":"2024-01-16T15:02:05.721472+0000","last_peered":"2024-01-16T15:02:05.721472+0000","last_clean":"2024-01-16T15:02:05.721472+0000","last_became_active":"2024-01-16T15:02:04.689392+0000","last_became_peered":"2024-01-16T15:02:04.689392+0000","last_unstale":"2024-01-16T15:02:05.721472+0000","last_undegraded":"2024-01-16T15:02:05.721472+0000","last_fullsized":"2024-01-16T15:02:05.721472+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:21.190356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708298+0000","last_change":"2024-01-16T15:02:05.709542+0000","last_active":"2024-01-16T15:02:06.708298+0000","last_peered":"2024-01-16T15:02:06.708298+0000","last_clean":"2024-01-16T15:02:06.708298+0000","last_became_active":"2024-01-16T15:02:05.709256+0000","last_became_peered":"2024-01-16T15:02:05.709256+0000","last_unstale":"2024-01-16T15:02:06.708298+0000","last_undegraded":"2024-01-16T15:02:06.708298+0000","last_fullsized":"2024-01-16T15:02:06.708298+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698374+0000","last_change":"2024-01-16T15:02:04.670268+0000","last_active":"2024-01-16T15:02:05.698374+0000","last_peered":"2024-01-16T15:02:05.698374+0000","last_clean":"2024-01-16T15:02:05.698374+0000","last_became_active":"2024-01-16T15:02:04.669950+0000","last_became_peered":"2024-01-16T15:02:04.669950+0000","last_unstale":"2024-01-16T15:02:05.698374+0000","last_undegraded":"2024-01-16T15:02:05.698374+0000","last_fullsized":"2024-01-16T15:02:05.698374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:15.532329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762898+0000","last_change":"2024-01-16T15:02:05.719151+0000","last_active":"2024-01-16T15:02:06.762898+0000","last_peered":"2024-01-16T15:02:06.762898+0000","last_clean":"2024-01-16T15:02:06.762898+0000","last_became_active":"2024-01-16T15:02:05.716626+0000","last_became_peered":"2024-01-16T15:02:05.716626+0000","last_unstale":"2024-01-16T15:02:06.762898+0000","last_undegraded":"2024-01-16T15:02:06.762898+0000","last_fullsized":"2024-01-16T15:02:06.762898+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:24:00.524827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694838+0000","last_change":"2024-01-16T15:02:04.678906+0000","last_active":"2024-01-16T15:02:05.694838+0000","last_peered":"2024-01-16T15:02:05.694838+0000","last_clean":"2024-01-16T15:02:05.694838+0000","last_became_active":"2024-01-16T15:02:04.678448+0000","last_became_peered":"2024-01-16T15:02:04.678448+0000","last_unstale":"2024-01-16T15:02:05.694838+0000","last_undegraded":"2024-01-16T15:02:05.694838+0000","last_fullsized":"2024-01-16T15:02:05.694838+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.582380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709295+0000","last_change":"2024-01-16T15:02:05.697150+0000","last_active":"2024-01-16T15:02:06.709295+0000","last_peered":"2024-01-16T15:02:06.709295+0000","last_clean":"2024-01-16T15:02:06.709295+0000","last_became_active":"2024-01-16T15:02:05.696650+0000","last_became_peered":"2024-01-16T15:02:05.696650+0000","last_unstale":"2024-01-16T15:02:06.709295+0000","last_undegraded":"2024-01-16T15:02:06.709295+0000","last_fullsized":"2024-01-16T15:02:06.709295+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:53.498507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.720102+0000","last_change":"2024-01-16T15:02:04.682082+0000","last_active":"2024-01-16T15:02:05.720102+0000","last_peered":"2024-01-16T15:02:05.720102+0000","last_clean":"2024-01-16T15:02:05.720102+0000","last_became_active":"2024-01-16T15:02:04.681666+0000","last_became_peered":"2024-01-16T15:02:04.681666+0000","last_unstale":"2024-01-16T15:02:05.720102+0000","last_undegraded":"2024-01-16T15:02:05.720102+0000","last_fullsized":"2024-01-16T15:02:05.720102+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:41:45.515329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709241+0000","last_change":"2024-01-16T15:02:05.715186+0000","last_active":"2024-01-16T15:02:06.709241+0000","last_peered":"2024-01-16T15:02:06.709241+0000","last_clean":"2024-01-16T15:02:06.709241+0000","last_became_active":"2024-01-16T15:02:05.714827+0000","last_became_peered":"2024-01-16T15:02:05.714827+0000","last_unstale":"2024-01-16T15:02:06.709241+0000","last_undegraded":"2024-01-16T15:02:06.709241+0000","last_fullsized":"2024-01-16T15:02:06.709241+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:14.377080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721478+0000","last_change":"2024-01-16T15:02:04.683297+0000","last_active":"2024-01-16T15:02:05.721478+0000","last_peered":"2024-01-16T15:02:05.721478+0000","last_clean":"2024-01-16T15:02:05.721478+0000","last_became_active":"2024-01-16T15:02:04.682784+0000","last_became_peered":"2024-01-16T15:02:04.682784+0000","last_unstale":"2024-01-16T15:02:05.721478+0000","last_undegraded":"2024-01-16T15:02:05.721478+0000","last_fullsized":"2024-01-16T15:02:05.721478+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:36.189068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765550+0000","last_change":"2024-01-16T15:02:05.707115+0000","last_active":"2024-01-16T15:02:06.765550+0000","last_peered":"2024-01-16T15:02:06.765550+0000","last_clean":"2024-01-16T15:02:06.765550+0000","last_became_active":"2024-01-16T15:02:05.706616+0000","last_became_peered":"2024-01-16T15:02:05.706616+0000","last_unstale":"2024-01-16T15:02:06.765550+0000","last_undegraded":"2024-01-16T15:02:06.765550+0000","last_fullsized":"2024-01-16T15:02:06.765550+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.716177+0000","last_change":"2024-01-16T15:02:04.687138+0000","last_active":"2024-01-16T15:02:05.716177+0000","last_peered":"2024-01-16T15:02:05.716177+0000","last_clean":"2024-01-16T15:02:05.716177+0000","last_became_active":"2024-01-16T15:02:04.686853+0000","last_became_peered":"2024-01-16T15:02:04.686853+0000","last_unstale":"2024-01-16T15:02:05.716177+0000","last_undegraded":"2024-01-16T15:02:05.716177+0000","last_fullsized":"2024-01-16T15:02:05.716177+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:47.375738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762595+0000","last_change":"2024-01-16T15:02:05.716760+0000","last_active":"2024-01-16T15:02:06.762595+0000","last_peered":"2024-01-16T15:02:06.762595+0000","last_clean":"2024-01-16T15:02:06.762595+0000","last_became_active":"2024-01-16T15:02:05.716500+0000","last_became_peered":"2024-01-16T15:02:05.716500+0000","last_unstale":"2024-01-16T15:02:06.762595+0000","last_undegraded":"2024-01-16T15:02:06.762595+0000","last_fullsized":"2024-01-16T15:02:06.762595+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:34.557637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.715974+0000","last_change":"2024-01-16T15:02:04.682056+0000","last_active":"2024-01-16T15:02:05.715974+0000","last_peered":"2024-01-16T15:02:05.715974+0000","last_clean":"2024-01-16T15:02:05.715974+0000","last_became_active":"2024-01-16T15:02:04.681766+0000","last_became_peered":"2024-01-16T15:02:04.681766+0000","last_unstale":"2024-01-16T15:02:05.715974+0000","last_undegraded":"2024-01-16T15:02:05.715974+0000","last_fullsized":"2024-01-16T15:02:05.715974+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:59.510201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709504+0000","last_change":"2024-01-16T15:02:05.703598+0000","last_active":"2024-01-16T15:02:06.709504+0000","last_peered":"2024-01-16T15:02:06.709504+0000","last_clean":"2024-01-16T15:02:06.709504+0000","last_became_active":"2024-01-16T15:02:05.702981+0000","last_became_peered":"2024-01-16T15:02:05.702981+0000","last_unstale":"2024-01-16T15:02:06.709504+0000","last_undegraded":"2024-01-16T15:02:06.709504+0000","last_fullsized":"2024-01-16T15:02:06.709504+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:37.203695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.693262+0000","last_change":"2024-01-16T15:02:04.681295+0000","last_active":"2024-01-16T15:02:05.693262+0000","last_peered":"2024-01-16T15:02:05.693262+0000","last_clean":"2024-01-16T15:02:05.693262+0000","last_became_active":"2024-01-16T15:02:04.680918+0000","last_became_peered":"2024-01-16T15:02:04.680918+0000","last_unstale":"2024-01-16T15:02:05.693262+0000","last_undegraded":"2024-01-16T15:02:05.693262+0000","last_fullsized":"2024-01-16T15:02:05.693262+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:44:27.577105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764456+0000","last_change":"2024-01-16T15:02:05.722557+0000","last_active":"2024-01-16T15:02:06.764456+0000","last_peered":"2024-01-16T15:02:06.764456+0000","last_clean":"2024-01-16T15:02:06.764456+0000","last_became_active":"2024-01-16T15:02:05.722276+0000","last_became_peered":"2024-01-16T15:02:05.722276+0000","last_unstale":"2024-01-16T15:02:06.764456+0000","last_undegraded":"2024-01-16T15:02:06.764456+0000","last_fullsized":"2024-01-16T15:02:06.764456+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:54.427231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721289+0000","last_change":"2024-01-16T15:02:04.680136+0000","last_active":"2024-01-16T15:02:05.721289+0000","last_peered":"2024-01-16T15:02:05.721289+0000","last_clean":"2024-01-16T15:02:05.721289+0000","last_became_active":"2024-01-16T15:02:04.679781+0000","last_became_peered":"2024-01-16T15:02:04.679781+0000","last_unstale":"2024-01-16T15:02:05.721289+0000","last_undegraded":"2024-01-16T15:02:05.721289+0000","last_fullsized":"2024-01-16T15:02:05.721289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:17.898530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762833+0000","last_change":"2024-01-16T15:02:05.708900+0000","last_active":"2024-01-16T15:02:06.762833+0000","last_peered":"2024-01-16T15:02:06.762833+0000","last_clean":"2024-01-16T15:02:06.762833+0000","last_became_active":"2024-01-16T15:02:05.708600+0000","last_became_peered":"2024-01-16T15:02:05.708600+0000","last_unstale":"2024-01-16T15:02:06.762833+0000","last_undegraded":"2024-01-16T15:02:06.762833+0000","last_fullsized":"2024-01-16T15:02:06.762833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.679302+0000","last_change":"2024-01-16T15:02:04.688154+0000","last_active":"2024-01-16T15:02:05.679302+0000","last_peered":"2024-01-16T15:02:05.679302+0000","last_clean":"2024-01-16T15:02:05.679302+0000","last_became_active":"2024-01-16T15:02:04.683089+0000","last_became_peered":"2024-01-16T15:02:04.683089+0000","last_unstale":"2024-01-16T15:02:05.679302+0000","last_undegraded":"2024-01-16T15:02:05.679302+0000","last_fullsized":"2024-01-16T15:02:05.679302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:21.797023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762594+0000","last_change":"2024-01-16T15:02:05.707619+0000","last_active":"2024-01-16T15:02:06.762594+0000","last_peered":"2024-01-16T15:02:06.762594+0000","last_clean":"2024-01-16T15:02:06.762594+0000","last_became_active":"2024-01-16T15:02:05.706744+0000","last_became_peered":"2024-01-16T15:02:05.706744+0000","last_unstale":"2024-01-16T15:02:06.762594+0000","last_undegraded":"2024-01-16T15:02:06.762594+0000","last_fullsized":"2024-01-16T15:02:06.762594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:54.427231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.697413+0000","last_change":"2024-01-16T15:02:04.679627+0000","last_active":"2024-01-16T15:02:05.697413+0000","last_peered":"2024-01-16T15:02:05.697413+0000","last_clean":"2024-01-16T15:02:05.697413+0000","last_became_active":"2024-01-16T15:02:04.679163+0000","last_became_peered":"2024-01-16T15:02:04.679163+0000","last_unstale":"2024-01-16T15:02:05.697413+0000","last_undegraded":"2024-01-16T15:02:05.697413+0000","last_fullsized":"2024-01-16T15:02:05.697413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:47.375738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765261+0000","last_change":"2024-01-16T15:02:05.713955+0000","last_active":"2024-01-16T15:02:06.765261+0000","last_peered":"2024-01-16T15:02:06.765261+0000","last_clean":"2024-01-16T15:02:06.765261+0000","last_became_active":"2024-01-16T15:02:05.713526+0000","last_became_peered":"2024-01-16T15:02:05.713526+0000","last_unstale":"2024-01-16T15:02:06.765261+0000","last_undegraded":"2024-01-16T15:02:06.765261+0000","last_fullsized":"2024-01-16T15:02:06.765261+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.719125+0000","last_change":"2024-01-16T15:02:04.685777+0000","last_active":"2024-01-16T15:02:05.719125+0000","last_peered":"2024-01-16T15:02:05.719125+0000","last_clean":"2024-01-16T15:02:05.719125+0000","last_became_active":"2024-01-16T15:02:04.685481+0000","last_became_peered":"2024-01-16T15:02:04.685481+0000","last_unstale":"2024-01-16T15:02:05.719125+0000","last_undegraded":"2024-01-16T15:02:05.719125+0000","last_fullsized":"2024-01-16T15:02:05.719125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:12.921478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765659+0000","last_change":"2024-01-16T15:02:05.708736+0000","last_active":"2024-01-16T15:02:06.765659+0000","last_peered":"2024-01-16T15:02:06.765659+0000","last_clean":"2024-01-16T15:02:06.765659+0000","last_became_active":"2024-01-16T15:02:05.708470+0000","last_became_peered":"2024-01-16T15:02:05.708470+0000","last_unstale":"2024-01-16T15:02:06.765659+0000","last_undegraded":"2024-01-16T15:02:06.765659+0000","last_fullsized":"2024-01-16T15:02:06.765659+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694340+0000","last_change":"2024-01-16T15:02:04.671324+0000","last_active":"2024-01-16T15:02:05.694340+0000","last_peered":"2024-01-16T15:02:05.694340+0000","last_clean":"2024-01-16T15:02:05.694340+0000","last_became_active":"2024-01-16T15:02:04.671056+0000","last_became_peered":"2024-01-16T15:02:04.671056+0000","last_unstale":"2024-01-16T15:02:05.694340+0000","last_undegraded":"2024-01-16T15:02:05.694340+0000","last_fullsized":"2024-01-16T15:02:05.694340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:34.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761500+0000","last_change":"2024-01-16T15:02:05.720952+0000","last_active":"2024-01-16T15:02:06.761500+0000","last_peered":"2024-01-16T15:02:06.761500+0000","last_clean":"2024-01-16T15:02:06.761500+0000","last_became_active":"2024-01-16T15:02:05.720361+0000","last_became_peered":"2024-01-16T15:02:05.720361+0000","last_unstale":"2024-01-16T15:02:06.761500+0000","last_undegraded":"2024-01-16T15:02:06.761500+0000","last_fullsized":"2024-01-16T15:02:06.761500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:51.004607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698305+0000","last_change":"2024-01-16T15:02:04.677076+0000","last_active":"2024-01-16T15:02:05.698305+0000","last_peered":"2024-01-16T15:02:05.698305+0000","last_clean":"2024-01-16T15:02:05.698305+0000","last_became_active":"2024-01-16T15:02:04.674921+0000","last_became_peered":"2024-01-16T15:02:04.674921+0000","last_unstale":"2024-01-16T15:02:05.698305+0000","last_undegraded":"2024-01-16T15:02:05.698305+0000","last_fullsized":"2024-01-16T15:02:05.698305+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:15.371747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763397+0000","last_change":"2024-01-16T15:02:05.717087+0000","last_active":"2024-01-16T15:02:06.763397+0000","last_peered":"2024-01-16T15:02:06.763397+0000","last_clean":"2024-01-16T15:02:06.763397+0000","last_became_active":"2024-01-16T15:02:05.716773+0000","last_became_peered":"2024-01-16T15:02:05.716773+0000","last_unstale":"2024-01-16T15:02:06.763397+0000","last_undegraded":"2024-01-16T15:02:06.763397+0000","last_fullsized":"2024-01-16T15:02:06.763397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.701829+0000","last_change":"2024-01-16T15:02:04.683141+0000","last_active":"2024-01-16T15:02:05.701829+0000","last_peered":"2024-01-16T15:02:05.701829+0000","last_clean":"2024-01-16T15:02:05.701829+0000","last_became_active":"2024-01-16T15:02:04.682515+0000","last_became_peered":"2024-01-16T15:02:04.682515+0000","last_unstale":"2024-01-16T15:02:05.701829+0000","last_undegraded":"2024-01-16T15:02:05.701829+0000","last_fullsized":"2024-01-16T15:02:05.701829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.508381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762011+0000","last_change":"2024-01-16T15:02:05.723101+0000","last_active":"2024-01-16T15:02:06.762011+0000","last_peered":"2024-01-16T15:02:06.762011+0000","last_clean":"2024-01-16T15:02:06.762011+0000","last_became_active":"2024-01-16T15:02:05.722293+0000","last_became_peered":"2024-01-16T15:02:05.722293+0000","last_unstale":"2024-01-16T15:02:06.762011+0000","last_undegraded":"2024-01-16T15:02:06.762011+0000","last_fullsized":"2024-01-16T15:02:06.762011+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:03.615829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.677315+0000","last_change":"2024-01-16T15:02:04.681828+0000","last_active":"2024-01-16T15:02:05.677315+0000","last_peered":"2024-01-16T15:02:05.677315+0000","last_clean":"2024-01-16T15:02:05.677315+0000","last_became_active":"2024-01-16T15:02:04.681571+0000","last_became_peered":"2024-01-16T15:02:04.681571+0000","last_unstale":"2024-01-16T15:02:05.677315+0000","last_undegraded":"2024-01-16T15:02:05.677315+0000","last_fullsized":"2024-01-16T15:02:05.677315+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:24.494050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708624+0000","last_change":"2024-01-16T15:02:05.697525+0000","last_active":"2024-01-16T15:02:06.708624+0000","last_peered":"2024-01-16T15:02:06.708624+0000","last_clean":"2024-01-16T15:02:06.708624+0000","last_became_active":"2024-01-16T15:02:05.697166+0000","last_became_peered":"2024-01-16T15:02:05.697166+0000","last_unstale":"2024-01-16T15:02:06.708624+0000","last_undegraded":"2024-01-16T15:02:06.708624+0000","last_fullsized":"2024-01-16T15:02:06.708624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.704166+0000","last_change":"2024-01-16T15:02:04.684667+0000","last_active":"2024-01-16T15:02:05.704166+0000","last_peered":"2024-01-16T15:02:05.704166+0000","last_clean":"2024-01-16T15:02:05.704166+0000","last_became_active":"2024-01-16T15:02:04.684301+0000","last_became_peered":"2024-01-16T15:02:04.684301+0000","last_unstale":"2024-01-16T15:02:05.704166+0000","last_undegraded":"2024-01-16T15:02:05.704166+0000","last_fullsized":"2024-01-16T15:02:05.704166+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:11.450047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.760698+0000","last_change":"2024-01-16T15:02:05.705898+0000","last_active":"2024-01-16T15:02:06.760698+0000","last_peered":"2024-01-16T15:02:06.760698+0000","last_clean":"2024-01-16T15:02:06.760698+0000","last_became_active":"2024-01-16T15:02:05.705594+0000","last_became_peered":"2024-01-16T15:02:05.705594+0000","last_unstale":"2024-01-16T15:02:06.760698+0000","last_undegraded":"2024-01-16T15:02:06.760698+0000","last_fullsized":"2024-01-16T15:02:06.760698+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:16.386373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.678340+0000","last_change":"2024-01-16T15:02:04.685022+0000","last_active":"2024-01-16T15:02:05.678340+0000","last_peered":"2024-01-16T15:02:05.678340+0000","last_clean":"2024-01-16T15:02:05.678340+0000","last_became_active":"2024-01-16T15:02:04.684477+0000","last_became_peered":"2024-01-16T15:02:04.684477+0000","last_unstale":"2024-01-16T15:02:05.678340+0000","last_undegraded":"2024-01-16T15:02:05.678340+0000","last_fullsized":"2024-01-16T15:02:05.678340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:52.853624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761590+0000","last_change":"2024-01-16T15:02:05.710367+0000","last_active":"2024-01-16T15:02:06.761590+0000","last_peered":"2024-01-16T15:02:06.761590+0000","last_clean":"2024-01-16T15:02:06.761590+0000","last_became_active":"2024-01-16T15:02:05.709916+0000","last_became_peered":"2024-01-16T15:02:05.709916+0000","last_unstale":"2024-01-16T15:02:06.761590+0000","last_undegraded":"2024-01-16T15:02:06.761590+0000","last_fullsized":"2024-01-16T15:02:06.761590+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:05.382187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694881+0000","last_change":"2024-01-16T15:02:04.676924+0000","last_active":"2024-01-16T15:02:05.694881+0000","last_peered":"2024-01-16T15:02:05.694881+0000","last_clean":"2024-01-16T15:02:05.694881+0000","last_became_active":"2024-01-16T15:02:04.676651+0000","last_became_peered":"2024-01-16T15:02:04.676651+0000","last_unstale":"2024-01-16T15:02:05.694881+0000","last_undegraded":"2024-01-16T15:02:05.694881+0000","last_fullsized":"2024-01-16T15:02:05.694881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:00.305823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761218+0000","last_change":"2024-01-16T15:02:05.712715+0000","last_active":"2024-01-16T15:02:06.761218+0000","last_peered":"2024-01-16T15:02:06.761218+0000","last_clean":"2024-01-16T15:02:06.761218+0000","last_became_active":"2024-01-16T15:02:05.712413+0000","last_became_peered":"2024-01-16T15:02:05.712413+0000","last_unstale":"2024-01-16T15:02:06.761218+0000","last_undegraded":"2024-01-16T15:02:06.761218+0000","last_fullsized":"2024-01-16T15:02:06.761218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:42.286960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698184+0000","last_change":"2024-01-16T15:02:04.679821+0000","last_active":"2024-01-16T15:02:05.698184+0000","last_peered":"2024-01-16T15:02:05.698184+0000","last_clean":"2024-01-16T15:02:05.698184+0000","last_became_active":"2024-01-16T15:02:04.679528+0000","last_became_peered":"2024-01-16T15:02:04.679528+0000","last_unstale":"2024-01-16T15:02:05.698184+0000","last_undegraded":"2024-01-16T15:02:05.698184+0000","last_fullsized":"2024-01-16T15:02:05.698184+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:59.388150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.682313+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:02:05.682313+0000","last_peered":"2024-01-16T15:02:05.682313+0000","last_clean":"2024-01-16T15:02:05.682313+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:02:05.682313+0000","last_undegraded":"2024-01-16T15:02:05.682313+0000","last_fullsized":"2024-01-16T15:02:05.682313+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1298432,"data_stored":881139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":1.5249999999999999,"15min":1.1579999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":45.296999999999997,"5min":45.296999999999997,"15min":45.296999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.724,"5min":1.3560000000000001,"15min":1.1779999999999999},"min":{"1min":0.76000000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.2690000000000001,"5min":9.2690000000000001,"15min":9.2690000000000001},"last":0.86199999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.2809999999999999,"15min":1.153},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":10.085000000000001,"5min":10.085000000000001,"15min":10.085000000000001},"last":1.022},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.3779999999999999,"15min":1.268},"min":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":11.356999999999999,"5min":11.356999999999999,"15min":11.356999999999999},"last":1.484}]},{"osd":2,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.256,"15min":1.2090000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.3900000000000001,"5min":1.536,"15min":1.3140000000000001},"min":{"1min":0.71999999999999997,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":23.594000000000001,"5min":23.594000000000001,"15min":23.594000000000001},"last":1.2330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.296,"15min":1.248},"min":{"1min":0.85599999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.4510000000000001,"15min":1.369},"min":{"1min":0.81699999999999995,"5min":0.751,"15min":0.751},"max":{"1min":4.8239999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.347}]},{"osd":4,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.4059999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":3.5939999999999999,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.702},{"interface":"front","average":{"1min":1.252,"5min":1.7,"15min":1.484},"min":{"1min":0.626,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.274,"5min":36.374000000000002,"15min":36.374000000000002},"last":1.4119999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.409,"15min":1.3620000000000001},"min":{"1min":0.48599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.073,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.476,"15min":1.425},"min":{"1min":0.56699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.327,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.835}]},{"osd":6,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4790000000000001,"15min":1.5249999999999999},"min":{"1min":0.627,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":4.1289999999999996,"5min":4.1289999999999996,"15min":4.1289999999999996},"last":2.052},{"interface":"front","average":{"1min":1.591,"5min":1.613,"15min":1.6000000000000001},"min":{"1min":0.64900000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":4.0940000000000003,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.778}]}]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":696320,"data_stored":290611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":5.7859999999999996,"5min":2.0579999999999998,"15min":1.4019999999999999},"min":{"1min":0.69599999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":71.870999999999995,"5min":71.870999999999995,"15min":71.870999999999995},"last":1.157},{"interface":"front","average":{"1min":5.7779999999999996,"5min":2.169,"15min":1.5449999999999999},"min":{"1min":0.98899999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":71.846999999999994,"5min":71.846999999999994,"15min":71.846999999999994},"last":1.306}]},{"osd":1,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.1399999999999999,"15min":1.075},"min":{"1min":0.73199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":3.0430000000000001,"5min":3.0430000000000001,"15min":3.0430000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":2.5009999999999999,"5min":1.405,"15min":1.2210000000000001},"min":{"1min":0.84099999999999997,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":13.839,"5min":13.839,"15min":13.839},"last":0.91500000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.282,"15min":1.23},"min":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":4.9080000000000004,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":1.571},{"interface":"front","average":{"1min":1.53,"5min":1.353,"15min":1.3049999999999999},"min":{"1min":0.86599999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":1.46}]},{"osd":3,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.345,"15min":1.369},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.214,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.206},{"interface":"front","average":{"1min":1.835,"5min":1.4430000000000001,"15min":1.423},"min":{"1min":1.0349999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":1.3400000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.4099999999999999,"15min":1.4359999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":7.4320000000000004,"5min":7.4320000000000004,"15min":7.4320000000000004},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4179999999999999,"15min":1.46},"min":{"1min":0.71299999999999997,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.391}]},{"osd":5,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.413,"15min":1.4490000000000001},"min":{"1min":0.68799999999999994,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.829,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.6120000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":1.6899999999999999,"15min":1.609},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":7.508,"5min":7.508,"15min":7.508},"last":1.427}]},{"osd":7,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5620000000000001,"15min":1.5800000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.9510000000000001,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.546,"15min":1.5760000000000001},"min":{"1min":0.68999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.5329999999999999,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":1.7170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":291263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.417,"15min":1.165},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":25.129999999999999,"5min":25.129999999999999,"15min":25.129999999999999},"last":1.595},{"interface":"front","average":{"1min":1.377,"5min":1.2829999999999999,"15min":1.169},"min":{"1min":0.503,"5min":0.499,"15min":0.499},"max":{"1min":4.7759999999999998,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":1.002}]},{"osd":1,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.359,"15min":1.244},"min":{"1min":0.69599999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.871,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.157},{"interface":"front","average":{"1min":1.367,"5min":1.242,"15min":1.1870000000000001},"min":{"1min":0.67300000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":4.8259999999999996,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":1.381}]},{"osd":2,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.3029999999999999,"15min":1.319},"min":{"1min":0.71099999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":10.321999999999999,"5min":10.321999999999999,"15min":10.321999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.3480000000000001,"15min":1.2769999999999999},"min":{"1min":0.83099999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":4.7160000000000002,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.4810000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1899999999999999,"5min":1.55,"15min":1.4330000000000001},"min":{"1min":0.755,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":1.4339999999999999},{"interface":"front","average":{"1min":2.319,"5min":1.5700000000000001,"15min":1.4199999999999999},"min":{"1min":0.98899999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":11.811,"5min":11.811,"15min":11.811},"last":1.111}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":1.375,"15min":1.2789999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.5209999999999999,"15min":1.5089999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":7.1769999999999996,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.647}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4279999999999999,"15min":1.4950000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.8769999999999998,"5min":4.8769999999999998,"15min":4.8769999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.4690000000000001,"15min":1.399},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":8.7070000000000007,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.2390000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.4630000000000001,"15min":1.546},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.901,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.742,"5min":1.597,"15min":1.579},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":1.284}]}]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":708608,"data_stored":292223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.325,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.9500000000000002,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.3,"15min":1.262},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":3.9769999999999999},"last":1.258}]},{"osd":1,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.2509999999999999,"15min":1.1259999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":7.5720000000000001,"5min":7.5720000000000001,"15min":7.5720000000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.1719999999999999,"15min":1.1419999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.552}]},{"osd":2,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.377,"15min":1.2330000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.4900000000000002,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.3700000000000001,"15min":1.3240000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":8.4239999999999995,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.70199999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2370000000000001,"15min":1.2649999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":1.4870000000000001},{"interface":"front","average":{"1min":2.762,"5min":1.8069999999999999,"15min":1.55},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":27.626999999999999,"5min":27.626999999999999,"15min":27.626999999999999},"last":1.325}]},{"osd":5,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.3859999999999999,"15min":1.3999999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.2599999999999998},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.4410000000000001,"15min":1.5169999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.788,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.3660000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":5.9210000000000003,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4990000000000001,"15min":1.579},"min":{"1min":0.70199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.4820000000000002,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.4750000000000001,"15min":1.5880000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1779999999999999,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.603},{"interface":"front","average":{"1min":2.4020000000000001,"5min":1.8959999999999999,"15min":1.7529999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":19.001000000000001,"5min":19.001000000000001,"15min":19.001000000000001},"last":1.524}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":290667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":0.94099999999999995,"15min":0.92700000000000005},"min":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":4.4820000000000002,"5min":4.4820000000000002,"15min":4.4820000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.1629999999999998,"5min":1.2350000000000001,"15min":1.018},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":20.012,"5min":20.012,"15min":20.012},"last":0.78900000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3519999999999999,"5min":1.238,"15min":1.0189999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":19.722000000000001,"5min":19.722000000000001,"15min":19.722000000000001},"last":0.876},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.0449999999999999,"15min":1.0720000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"last":1.081}]},{"osd":2,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":1.327,"15min":1.1160000000000001},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":20.074000000000002,"5min":20.074000000000002,"15min":20.074000000000002},"last":1.385},{"interface":"front","average":{"1min":2.5289999999999999,"5min":1.419,"15min":1.204},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":19.777999999999999,"5min":19.777999999999999,"15min":19.777999999999999},"last":1.2669999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":1.5469999999999999,"15min":1.413},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":21.736000000000001,"5min":21.736000000000001,"15min":21.736000000000001},"last":1.155},{"interface":"front","average":{"1min":3.1909999999999998,"5min":1.7030000000000001,"15min":1.472},"min":{"1min":0.96799999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":20.43,"5min":20.43,"15min":20.43},"last":1.409}]},{"osd":5,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.3919999999999999,"15min":1.4570000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":4.6909999999999998,"5min":4.6909999999999998,"15min":4.6909999999999998},"last":1.206},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4379999999999999,"15min":1.474},"min":{"1min":0.80400000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"last":1.502}]},{"osd":6,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3969999999999998,"5min":1.6359999999999999,"15min":1.4770000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":20.459,"5min":20.459,"15min":20.459},"last":1.4410000000000001},{"interface":"front","average":{"1min":2.8010000000000002,"5min":1.752,"15min":1.5740000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":19.928999999999998,"5min":19.928999999999998,"15min":19.928999999999998},"last":1.3480000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4140000000000001,"5min":1.74,"15min":1.534},"min":{"1min":1.0389999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":20.109999999999999,"5min":20.109999999999999,"15min":20.109999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":1.7949999999999999,"15min":1.607},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":19.975000000000001,"5min":19.975000000000001,"15min":19.975000000000001},"last":1.4670000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":290667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.419,"5min":1.2649999999999999,"15min":1.149},"min":{"1min":0.42299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":25.335999999999999,"5min":25.335999999999999,"15min":25.335999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":2.5800000000000001,"5min":1.3320000000000001,"15min":1.1919999999999999},"min":{"1min":0.47499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":24.798999999999999,"5min":24.798999999999999,"15min":24.798999999999999},"last":0.85699999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.2509999999999999,"15min":1.2569999999999999},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.8109999999999999,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":1.329},{"interface":"front","average":{"1min":2.3399999999999999,"5min":1.4330000000000001,"15min":1.403},"min":{"1min":0.376,"5min":0.373,"15min":0.373},"max":{"1min":21.439,"5min":21.439,"15min":21.439},"last":1.163}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.1479999999999999,"15min":1.218},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":8.1980000000000004,"5min":8.1980000000000004,"15min":8.1980000000000004},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.4950000000000001,"15min":1.4510000000000001},"min":{"1min":0.53200000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":28.045999999999999,"5min":28.045999999999999,"15min":28.045999999999999},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":3.2189999999999999,"5min":1.7709999999999999,"15min":1.589},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":21.532,"5min":21.532,"15min":21.532},"last":1.421},{"interface":"front","average":{"1min":2.5539999999999998,"5min":1.6020000000000001,"15min":1.506},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":14.597,"5min":14.597,"15min":14.597},"last":1.512}]},{"osd":5,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.454,"15min":1.5349999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.762,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":2.726,"5min":1.75,"15min":1.637},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":21.608000000000001,"5min":21.608000000000001,"15min":21.608000000000001},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.605,"15min":1.5369999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":1.577},{"interface":"front","average":{"1min":1.522,"5min":1.5349999999999999,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.871,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.6950000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":3.0110000000000001,"5min":1.8560000000000001,"15min":1.7410000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":21.571999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.5369999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":12.456,"5min":12.456,"15min":12.456},"last":1.385}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":700416,"data_stored":290701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.1299999999999999,"15min":1.01},"min":{"1min":0.65500000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.147,"5min":10.147,"15min":10.147},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.863,"5min":1.1930000000000001,"15min":0.95899999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":10.563000000000001,"5min":10.563000000000001,"15min":10.563000000000001},"last":0.879}]},{"osd":2,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":1.329,"15min":1.0529999999999999},"min":{"1min":0.48199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":24.709,"5min":24.709,"15min":24.709},"last":1.268},{"interface":"front","average":{"1min":2.7759999999999998,"5min":1.468,"15min":1.21},"min":{"1min":0.73499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":30.128,"5min":30.128,"15min":30.128},"last":1.071}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.3109999999999999,"15min":1.212},"min":{"1min":0.61099999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":5.1890000000000001,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.331},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.1699999999999999,"15min":1.0509999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7709999999999999,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.01}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.901,"5min":1.4339999999999999,"15min":1.302},"min":{"1min":0.877,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":14.242000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.891,"5min":1.5509999999999999,"15min":1.3999999999999999},"min":{"1min":0.81999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.734,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.516}]},{"osd":5,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.5189999999999999,"15min":1.425},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":7.3490000000000002,"5min":7.3490000000000002,"15min":7.3490000000000002},"last":1.4630000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.6060000000000001,"15min":1.486},"min":{"1min":1.008,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":14.153,"5min":14.153,"15min":14.153},"last":1.6200000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.5880000000000001,"15min":1.512},"min":{"1min":0.78200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.458,"15min":1.405},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.7589999999999999,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.55,"15min":1.51},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"last":1.113},{"interface":"front","average":{"1min":2.2799999999999998,"5min":1.6279999999999999,"15min":1.518},"min":{"1min":0.81000000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":14.331,"5min":14.331,"15min":14.331},"last":1.3799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":882154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":1.1100000000000001,"15min":0.95799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.948,"5min":18.948,"15min":18.948},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.1200000000000001,"5min":1.161,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.750999999999999,"5min":15.750999999999999,"15min":15.750999999999999},"last":0.65900000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":1.2609999999999999,"15min":1.135},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":13.44,"5min":13.44,"15min":13.44},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.2150000000000001,"15min":1.071},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":16.190999999999999,"5min":16.190999999999999,"15min":16.190999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.2310000000000001,"15min":1.0449999999999999},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.102,"5min":15.102,"15min":15.102},"last":0.626},{"interface":"front","average":{"1min":2.6600000000000001,"5min":1.476,"15min":1.3200000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":15.365,"5min":15.365,"15min":15.365},"last":0.69299999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":1.6319999999999999,"15min":1.4259999999999999},"min":{"1min":0.747,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":15.832000000000001,"5min":15.832000000000001,"15min":15.832000000000001},"last":1.1990000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.401,"15min":1.2889999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":11.404999999999999,"5min":11.404999999999999,"15min":11.404999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3149999999999999,"5min":1.617,"15min":1.54},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":13.794,"5min":13.794,"15min":13.794},"last":1.054},{"interface":"front","average":{"1min":2.2839999999999998,"5min":1.597,"15min":1.4870000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":15.906000000000001,"5min":15.906000000000001,"15min":15.906000000000001},"last":1.2689999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":1.6499999999999999,"15min":1.4430000000000001},"min":{"1min":0.77200000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":11.367000000000001,"5min":16.960999999999999,"15min":16.960999999999999},"last":1.3},{"interface":"front","average":{"1min":2.2930000000000001,"5min":1.609,"15min":1.5529999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":13.499000000000001,"5min":13.499000000000001,"15min":13.499000000000001},"last":0.95099999999999996}]},{"osd":7,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":1.5629999999999999,"15min":1.5069999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":16.039999999999999,"5min":16.039999999999999,"15min":16.039999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":2.54,"5min":1.8109999999999999,"15min":1.5800000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":11.433,"5min":17.038,"15min":17.038},"last":1.071}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:10.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:10.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.396+0000 7fc407200640 1 Processor -- start 2024-01-16T15:02:10.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.397+0000 7fc407200640 1 -- start start 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.398+0000 7fc407200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40815afb0 0x7fc40813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.398+0000 7fc407200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc40813fae0 0x7fc4080b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.398+0000 7fc407200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40813ff00 con 0x7fc40813cfa0 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.398+0000 7fc407200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4080cdc40 con 0x7fc40813fae0 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.398+0000 7fc407200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4080cdec0 con 0x7fc40815afb0 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc405e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc40813cfa0 0x7fc4080cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60750/0 (socket says 172.21.15.125:60750) 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc405e00640 1 -- 172.21.15.125:0/3771745185 learned_addr learned my addr 172.21.15.125:0/3771745185 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc406800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc40813fae0 0x7fc4080b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc3ffe00640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40815afb0 0x7fc40813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc405400640 1 -- 172.21.15.125:0/3771745185 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019412185 0 0) 0x7fc40813ff00 con 0x7fc40813cfa0 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc405400640 1 -- 172.21.15.125:0/3771745185 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8003520 con 0x7fc40813cfa0 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc406800640 1 -- 172.21.15.125:0/3771745185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40815afb0 msgr2=0x7fc40813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc406800640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40815afb0 0x7fc40813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc406800640 1 -- 172.21.15.125:0/3771745185 >> v1:172.21.15.125:6789/0 conn(0x7fc40813cfa0 legacy=0x7fc4080cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.399+0000 7fc406800640 1 -- 172.21.15.125:0/3771745185 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4080ce1a0 con 0x7fc40813fae0 2024-01-16T15:02:10.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.400+0000 7fc406800640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813fae0 0x7fc4080b2040 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc400009d90 tx=0x7fc40002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ac7b4048f11b25 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.400+0000 7fc405400640 1 -- 172.21.15.125:0/3771745185 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40002cbc0 con 0x7fc40813fae0 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc405400640 1 -- 172.21.15.125:0/3771745185 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40002cd20 con 0x7fc40813fae0 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc405400640 1 -- 172.21.15.125:0/3771745185 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400034660 con 0x7fc40813fae0 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 -- 172.21.15.125:0/3771745185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813fae0 msgr2=0x7fc4080b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813fae0 0x7fc4080b2040 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc400009d90 tx=0x7fc40002be10 comp rx=0 tx=0).stop 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 -- 172.21.15.125:0/3771745185 shutdown_connections 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813fae0 0x7fc4080b2040 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 --2- 172.21.15.125:0/3771745185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40815afb0 0x7fc40813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 -- 172.21.15.125:0/3771745185 >> 172.21.15.125:0/3771745185 conn(0x7fc40806a800 msgr2=0x7fc40806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 -- 172.21.15.125:0/3771745185 shutdown_connections 2024-01-16T15:02:10.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.401+0000 7fc407200640 1 -- 172.21.15.125:0/3771745185 wait complete. 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.403+0000 7fc407200640 1 Processor -- start 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.403+0000 7fc407200640 1 -- start start 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40813fae0 0x7fc408159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40815afb0 0x7fc408159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc408166140 con 0x7fc40813cfa0 2024-01-16T15:02:10.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc408134f70 con 0x7fc40815afb0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc407200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc408135250 con 0x7fc40813fae0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc405e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc405e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50838/0 (socket says 172.21.15.125:50838) 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.404+0000 7fc405e00640 1 -- 172.21.15.125:0/3135869346 learned_addr learned my addr 172.21.15.125:0/3135869346 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc3ffe00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40813fae0 0x7fc408159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc406800640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40815afb0 0x7fc408159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 -- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40813fae0 msgr2=0x7fc408159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40813fae0 0x7fc408159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 -- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40815afb0 msgr2=0x7fc408159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40815afb0 0x7fc408159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 -- 172.21.15.125:0/3135869346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400009a70 con 0x7fc40813cfa0 2024-01-16T15:02:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc405e00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800c500 tx=0x7fc3f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8005ce0 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8005e40 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc408135530 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.405+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8007330 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.406+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc408134260 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.407+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc408067450 con 0x7fc40813cfa0 2024-01-16T15:02:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.408+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc3f8007550 con 0x7fc40813cfa0 2024-01-16T15:02:10.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.409+0000 7fc3fe000640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 0x7fc3e006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.409+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3f8089c90 con 0x7fc40813cfa0 2024-01-16T15:02:10.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.409+0000 7fc3ffe00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 0x7fc3e006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.410+0000 7fc3ffe00640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 0x7fc3e006dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0006fd0 tx=0x7fc3f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.413+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3f80543a0 con 0x7fc40813cfa0 2024-01-16T15:02:10.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.599+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4080cea40 con 0x7fc3e006b860 2024-01-16T15:02:10.588 INFO:tasks.ceph.mds.c.smithi125.stderr:2024-01-16T15:02:10.608+0000 7f34b4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:02:10.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.621+0000 7fc3fe000640 1 -- 172.21.15.125:0/3135869346 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287335 (secure 0 0 0) 0x7fc3f000c050 con 0x7fc3e006b860 2024-01-16T15:02:10.605 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:10.607 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.625+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 msgr2=0x7fc3e006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.625+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 0x7fc3e006dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0006fd0 tx=0x7fc3f0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.625+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 msgr2=0x7fc40814fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.625+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800c500 tx=0x7fc3f800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 shutdown_connections 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40815afb0 0x7fc408159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc40813fae0 0x7fc408159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3e006b860 0x7fc3e006dd20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 --2- 172.21.15.125:0/3135869346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc40813cfa0 0x7fc40814fb40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 >> 172.21.15.125:0/3135869346 conn(0x7fc40806a800 msgr2=0x7fc408136d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 shutdown_connections 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.626+0000 7fc407200640 1 -- 172.21.15.125:0/3135869346 wait complete. 2024-01-16T15:02:10.625 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-16T15:02:10.028024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289408,"kb_used_data":6660,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685312,"statfs":{"total":773094113280,"available":772797759488,"internally_reserved":0,"allocated":6819840,"data_stored":3509425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001825"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.704001+0000","last_change":"2024-01-16T15:02:05.714070+0000","last_active":"2024-01-16T15:02:06.704001+0000","last_peered":"2024-01-16T15:02:06.704001+0000","last_clean":"2024-01-16T15:02:06.704001+0000","last_became_active":"2024-01-16T15:02:05.713522+0000","last_became_peered":"2024-01-16T15:02:05.713522+0000","last_unstale":"2024-01-16T15:02:06.704001+0000","last_undegraded":"2024-01-16T15:02:06.704001+0000","last_fullsized":"2024-01-16T15:02:06.704001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:01.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763124+0000","last_change":"2024-01-16T15:02:05.723395+0000","last_active":"2024-01-16T15:02:06.763124+0000","last_peered":"2024-01-16T15:02:06.763124+0000","last_clean":"2024-01-16T15:02:06.763124+0000","last_became_active":"2024-01-16T15:02:05.723125+0000","last_became_peered":"2024-01-16T15:02:05.723125+0000","last_unstale":"2024-01-16T15:02:06.763124+0000","last_undegraded":"2024-01-16T15:02:06.763124+0000","last_fullsized":"2024-01-16T15:02:06.763124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:45.802093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707914+0000","last_change":"2024-01-16T15:02:05.709039+0000","last_active":"2024-01-16T15:02:06.707914+0000","last_peered":"2024-01-16T15:02:06.707914+0000","last_clean":"2024-01-16T15:02:06.707914+0000","last_became_active":"2024-01-16T15:02:05.707242+0000","last_became_peered":"2024-01-16T15:02:05.707242+0000","last_unstale":"2024-01-16T15:02:06.707914+0000","last_undegraded":"2024-01-16T15:02:06.707914+0000","last_fullsized":"2024-01-16T15:02:06.707914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764834+0000","last_change":"2024-01-16T15:02:05.714976+0000","last_active":"2024-01-16T15:02:06.764834+0000","last_peered":"2024-01-16T15:02:06.764834+0000","last_clean":"2024-01-16T15:02:06.764834+0000","last_became_active":"2024-01-16T15:02:05.714437+0000","last_became_peered":"2024-01-16T15:02:05.714437+0000","last_unstale":"2024-01-16T15:02:06.764834+0000","last_undegraded":"2024-01-16T15:02:06.764834+0000","last_fullsized":"2024-01-16T15:02:06.764834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703496+0000","last_change":"2024-01-16T15:02:05.717167+0000","last_active":"2024-01-16T15:02:06.703496+0000","last_peered":"2024-01-16T15:02:06.703496+0000","last_clean":"2024-01-16T15:02:06.703496+0000","last_became_active":"2024-01-16T15:02:05.716887+0000","last_became_peered":"2024-01-16T15:02:05.716887+0000","last_unstale":"2024-01-16T15:02:06.703496+0000","last_undegraded":"2024-01-16T15:02:06.703496+0000","last_fullsized":"2024-01-16T15:02:06.703496+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:56.700044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763422+0000","last_change":"2024-01-16T15:02:05.695959+0000","last_active":"2024-01-16T15:02:06.763422+0000","last_peered":"2024-01-16T15:02:06.763422+0000","last_clean":"2024-01-16T15:02:06.763422+0000","last_became_active":"2024-01-16T15:02:05.694589+0000","last_became_peered":"2024-01-16T15:02:05.694589+0000","last_unstale":"2024-01-16T15:02:06.763422+0000","last_undegraded":"2024-01-16T15:02:06.763422+0000","last_fullsized":"2024-01-16T15:02:06.763422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762937+0000","last_change":"2024-01-16T15:02:05.710640+0000","last_active":"2024-01-16T15:02:06.762937+0000","last_peered":"2024-01-16T15:02:06.762937+0000","last_clean":"2024-01-16T15:02:06.762937+0000","last_became_active":"2024-01-16T15:02:05.710314+0000","last_became_peered":"2024-01-16T15:02:05.710314+0000","last_unstale":"2024-01-16T15:02:06.762937+0000","last_undegraded":"2024-01-16T15:02:06.762937+0000","last_fullsized":"2024-01-16T15:02:06.762937+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:09.731041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761071+0000","last_change":"2024-01-16T15:02:05.713833+0000","last_active":"2024-01-16T15:02:06.761071+0000","last_peered":"2024-01-16T15:02:06.761071+0000","last_clean":"2024-01-16T15:02:06.761071+0000","last_became_active":"2024-01-16T15:02:05.713568+0000","last_became_peered":"2024-01-16T15:02:05.713568+0000","last_unstale":"2024-01-16T15:02:06.761071+0000","last_undegraded":"2024-01-16T15:02:06.761071+0000","last_fullsized":"2024-01-16T15:02:06.761071+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:50.726560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764373+0000","last_change":"2024-01-16T15:02:05.709577+0000","last_active":"2024-01-16T15:02:06.764373+0000","last_peered":"2024-01-16T15:02:06.764373+0000","last_clean":"2024-01-16T15:02:06.764373+0000","last_became_active":"2024-01-16T15:02:05.709256+0000","last_became_peered":"2024-01-16T15:02:05.709256+0000","last_unstale":"2024-01-16T15:02:06.764373+0000","last_undegraded":"2024-01-16T15:02:06.764373+0000","last_fullsized":"2024-01-16T15:02:06.764373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:19.298378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761074+0000","last_change":"2024-01-16T15:02:05.711654+0000","last_active":"2024-01-16T15:02:06.761074+0000","last_peered":"2024-01-16T15:02:06.761074+0000","last_clean":"2024-01-16T15:02:06.761074+0000","last_became_active":"2024-01-16T15:02:05.711368+0000","last_became_peered":"2024-01-16T15:02:05.711368+0000","last_unstale":"2024-01-16T15:02:06.761074+0000","last_undegraded":"2024-01-16T15:02:06.761074+0000","last_fullsized":"2024-01-16T15:02:06.761074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.546955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761517+0000","last_change":"2024-01-16T15:02:05.706192+0000","last_active":"2024-01-16T15:02:06.761517+0000","last_peered":"2024-01-16T15:02:06.761517+0000","last_clean":"2024-01-16T15:02:06.761517+0000","last_became_active":"2024-01-16T15:02:05.705880+0000","last_became_peered":"2024-01-16T15:02:05.705880+0000","last_unstale":"2024-01-16T15:02:06.761517+0000","last_undegraded":"2024-01-16T15:02:06.761517+0000","last_fullsized":"2024-01-16T15:02:06.761517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:34.237454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764893+0000","last_change":"2024-01-16T15:02:05.714792+0000","last_active":"2024-01-16T15:02:06.764893+0000","last_peered":"2024-01-16T15:02:06.764893+0000","last_clean":"2024-01-16T15:02:06.764893+0000","last_became_active":"2024-01-16T15:02:05.714103+0000","last_became_peered":"2024-01-16T15:02:05.714103+0000","last_unstale":"2024-01-16T15:02:06.764893+0000","last_undegraded":"2024-01-16T15:02:06.764893+0000","last_fullsized":"2024-01-16T15:02:06.764893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708928+0000","last_change":"2024-01-16T15:02:05.707657+0000","last_active":"2024-01-16T15:02:06.708928+0000","last_peered":"2024-01-16T15:02:06.708928+0000","last_clean":"2024-01-16T15:02:06.708928+0000","last_became_active":"2024-01-16T15:02:05.707260+0000","last_became_peered":"2024-01-16T15:02:05.707260+0000","last_unstale":"2024-01-16T15:02:06.708928+0000","last_undegraded":"2024-01-16T15:02:06.708928+0000","last_fullsized":"2024-01-16T15:02:06.708928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:40.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763558+0000","last_change":"2024-01-16T15:02:05.704314+0000","last_active":"2024-01-16T15:02:06.763558+0000","last_peered":"2024-01-16T15:02:06.763558+0000","last_clean":"2024-01-16T15:02:06.763558+0000","last_became_active":"2024-01-16T15:02:05.703400+0000","last_became_peered":"2024-01-16T15:02:05.703400+0000","last_unstale":"2024-01-16T15:02:06.763558+0000","last_undegraded":"2024-01-16T15:02:06.763558+0000","last_fullsized":"2024-01-16T15:02:06.763558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:32:49.727167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763211+0000","last_change":"2024-01-16T15:02:05.703743+0000","last_active":"2024-01-16T15:02:06.763211+0000","last_peered":"2024-01-16T15:02:06.763211+0000","last_clean":"2024-01-16T15:02:06.763211+0000","last_became_active":"2024-01-16T15:02:05.703430+0000","last_became_peered":"2024-01-16T15:02:05.703430+0000","last_unstale":"2024-01-16T15:02:06.763211+0000","last_undegraded":"2024-01-16T15:02:06.763211+0000","last_fullsized":"2024-01-16T15:02:06.763211+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:31:22.593889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762237+0000","last_change":"2024-01-16T15:02:05.707433+0000","last_active":"2024-01-16T15:02:06.762237+0000","last_peered":"2024-01-16T15:02:06.762237+0000","last_clean":"2024-01-16T15:02:06.762237+0000","last_became_active":"2024-01-16T15:02:05.706513+0000","last_became_peered":"2024-01-16T15:02:05.706513+0000","last_unstale":"2024-01-16T15:02:06.762237+0000","last_undegraded":"2024-01-16T15:02:06.762237+0000","last_fullsized":"2024-01-16T15:02:06.762237+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:31.344585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763670+0000","last_change":"2024-01-16T15:02:05.715908+0000","last_active":"2024-01-16T15:02:06.763670+0000","last_peered":"2024-01-16T15:02:06.763670+0000","last_clean":"2024-01-16T15:02:06.763670+0000","last_became_active":"2024-01-16T15:02:05.715613+0000","last_became_peered":"2024-01-16T15:02:05.715613+0000","last_unstale":"2024-01-16T15:02:06.763670+0000","last_undegraded":"2024-01-16T15:02:06.763670+0000","last_fullsized":"2024-01-16T15:02:06.763670+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:01.259511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708872+0000","last_change":"2024-01-16T15:02:05.702398+0000","last_active":"2024-01-16T15:02:06.708872+0000","last_peered":"2024-01-16T15:02:06.708872+0000","last_clean":"2024-01-16T15:02:06.708872+0000","last_became_active":"2024-01-16T15:02:05.702092+0000","last_became_peered":"2024-01-16T15:02:05.702092+0000","last_unstale":"2024-01-16T15:02:06.708872+0000","last_undegraded":"2024-01-16T15:02:06.708872+0000","last_fullsized":"2024-01-16T15:02:06.708872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703535+0000","last_change":"2024-01-16T15:02:05.694202+0000","last_active":"2024-01-16T15:02:06.703535+0000","last_peered":"2024-01-16T15:02:06.703535+0000","last_clean":"2024-01-16T15:02:06.703535+0000","last_became_active":"2024-01-16T15:02:05.693873+0000","last_became_peered":"2024-01-16T15:02:05.693873+0000","last_unstale":"2024-01-16T15:02:06.703535+0000","last_undegraded":"2024-01-16T15:02:06.703535+0000","last_fullsized":"2024-01-16T15:02:06.703535+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:53.845071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762787+0000","last_change":"2024-01-16T15:02:05.705564+0000","last_active":"2024-01-16T15:02:06.762787+0000","last_peered":"2024-01-16T15:02:06.762787+0000","last_clean":"2024-01-16T15:02:06.762787+0000","last_became_active":"2024-01-16T15:02:05.705315+0000","last_became_peered":"2024-01-16T15:02:05.705315+0000","last_unstale":"2024-01-16T15:02:06.762787+0000","last_undegraded":"2024-01-16T15:02:06.762787+0000","last_fullsized":"2024-01-16T15:02:06.762787+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:11.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707914+0000","last_change":"2024-01-16T15:02:05.709428+0000","last_active":"2024-01-16T15:02:06.707914+0000","last_peered":"2024-01-16T15:02:06.707914+0000","last_clean":"2024-01-16T15:02:06.707914+0000","last_became_active":"2024-01-16T15:02:05.709149+0000","last_became_peered":"2024-01-16T15:02:05.709149+0000","last_unstale":"2024-01-16T15:02:06.707914+0000","last_undegraded":"2024-01-16T15:02:06.707914+0000","last_fullsized":"2024-01-16T15:02:06.707914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.760967+0000","last_change":"2024-01-16T15:02:05.713281+0000","last_active":"2024-01-16T15:02:06.760967+0000","last_peered":"2024-01-16T15:02:06.760967+0000","last_clean":"2024-01-16T15:02:06.760967+0000","last_became_active":"2024-01-16T15:02:05.713026+0000","last_became_peered":"2024-01-16T15:02:05.713026+0000","last_unstale":"2024-01-16T15:02:06.760967+0000","last_undegraded":"2024-01-16T15:02:06.760967+0000","last_fullsized":"2024-01-16T15:02:06.760967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:55.945285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709164+0000","last_change":"2024-01-16T15:02:05.696922+0000","last_active":"2024-01-16T15:02:06.709164+0000","last_peered":"2024-01-16T15:02:06.709164+0000","last_clean":"2024-01-16T15:02:06.709164+0000","last_became_active":"2024-01-16T15:02:05.696343+0000","last_became_peered":"2024-01-16T15:02:05.696343+0000","last_unstale":"2024-01-16T15:02:06.709164+0000","last_undegraded":"2024-01-16T15:02:06.709164+0000","last_fullsized":"2024-01-16T15:02:06.709164+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:18.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707357+0000","last_change":"2024-01-16T15:02:05.705546+0000","last_active":"2024-01-16T15:02:06.707357+0000","last_peered":"2024-01-16T15:02:06.707357+0000","last_clean":"2024-01-16T15:02:06.707357+0000","last_became_active":"2024-01-16T15:02:05.705188+0000","last_became_peered":"2024-01-16T15:02:05.705188+0000","last_unstale":"2024-01-16T15:02:06.707357+0000","last_undegraded":"2024-01-16T15:02:06.707357+0000","last_fullsized":"2024-01-16T15:02:06.707357+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:51.004607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703915+0000","last_change":"2024-01-16T15:02:05.689999+0000","last_active":"2024-01-16T15:02:06.703915+0000","last_peered":"2024-01-16T15:02:06.703915+0000","last_clean":"2024-01-16T15:02:06.703915+0000","last_became_active":"2024-01-16T15:02:05.689755+0000","last_became_peered":"2024-01-16T15:02:05.689755+0000","last_unstale":"2024-01-16T15:02:06.703915+0000","last_undegraded":"2024-01-16T15:02:06.703915+0000","last_fullsized":"2024-01-16T15:02:06.703915+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:25.508676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765184+0000","last_change":"2024-01-16T15:02:05.713769+0000","last_active":"2024-01-16T15:02:06.765184+0000","last_peered":"2024-01-16T15:02:06.765184+0000","last_clean":"2024-01-16T15:02:06.765184+0000","last_became_active":"2024-01-16T15:02:05.713228+0000","last_became_peered":"2024-01-16T15:02:05.713228+0000","last_unstale":"2024-01-16T15:02:06.765184+0000","last_undegraded":"2024-01-16T15:02:06.765184+0000","last_fullsized":"2024-01-16T15:02:06.765184+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762676+0000","last_change":"2024-01-16T15:02:05.709062+0000","last_active":"2024-01-16T15:02:06.762676+0000","last_peered":"2024-01-16T15:02:06.762676+0000","last_clean":"2024-01-16T15:02:06.762676+0000","last_became_active":"2024-01-16T15:02:05.706597+0000","last_became_peered":"2024-01-16T15:02:05.706597+0000","last_unstale":"2024-01-16T15:02:06.762676+0000","last_undegraded":"2024-01-16T15:02:06.762676+0000","last_fullsized":"2024-01-16T15:02:06.762676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:08.221884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.703369+0000","last_change":"2024-01-16T15:02:05.718074+0000","last_active":"2024-01-16T15:02:06.703369+0000","last_peered":"2024-01-16T15:02:06.703369+0000","last_clean":"2024-01-16T15:02:06.703369+0000","last_became_active":"2024-01-16T15:02:05.717777+0000","last_became_peered":"2024-01-16T15:02:05.717777+0000","last_unstale":"2024-01-16T15:02:06.703369+0000","last_undegraded":"2024-01-16T15:02:06.703369+0000","last_fullsized":"2024-01-16T15:02:06.703369+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:56.077780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762335+0000","last_change":"2024-01-16T15:02:05.721079+0000","last_active":"2024-01-16T15:02:06.762335+0000","last_peered":"2024-01-16T15:02:06.762335+0000","last_clean":"2024-01-16T15:02:06.762335+0000","last_became_active":"2024-01-16T15:02:05.720714+0000","last_became_peered":"2024-01-16T15:02:05.720714+0000","last_unstale":"2024-01-16T15:02:06.762335+0000","last_undegraded":"2024-01-16T15:02:06.762335+0000","last_fullsized":"2024-01-16T15:02:06.762335+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:10:07.921797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708137+0000","last_change":"2024-01-16T15:02:05.708735+0000","last_active":"2024-01-16T15:02:06.708137+0000","last_peered":"2024-01-16T15:02:06.708137+0000","last_clean":"2024-01-16T15:02:06.708137+0000","last_became_active":"2024-01-16T15:02:05.708446+0000","last_became_peered":"2024-01-16T15:02:05.708446+0000","last_unstale":"2024-01-16T15:02:06.708137+0000","last_undegraded":"2024-01-16T15:02:06.708137+0000","last_fullsized":"2024-01-16T15:02:06.708137+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.702823+0000","last_change":"2024-01-16T15:02:05.690002+0000","last_active":"2024-01-16T15:02:06.702823+0000","last_peered":"2024-01-16T15:02:06.702823+0000","last_clean":"2024-01-16T15:02:06.702823+0000","last_became_active":"2024-01-16T15:02:05.689516+0000","last_became_peered":"2024-01-16T15:02:05.689516+0000","last_unstale":"2024-01-16T15:02:06.702823+0000","last_undegraded":"2024-01-16T15:02:06.702823+0000","last_fullsized":"2024-01-16T15:02:06.702823+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:52.702885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762242+0000","last_change":"2024-01-16T15:02:05.718995+0000","last_active":"2024-01-16T15:02:06.762242+0000","last_peered":"2024-01-16T15:02:06.762242+0000","last_clean":"2024-01-16T15:02:06.762242+0000","last_became_active":"2024-01-16T15:02:05.718727+0000","last_became_peered":"2024-01-16T15:02:05.718727+0000","last_unstale":"2024-01-16T15:02:06.762242+0000","last_undegraded":"2024-01-16T15:02:06.762242+0000","last_fullsized":"2024-01-16T15:02:06.762242+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:58.504291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763703+0000","last_change":"2024-01-16T15:02:05.711882+0000","last_active":"2024-01-16T15:02:06.763703+0000","last_peered":"2024-01-16T15:02:06.763703+0000","last_clean":"2024-01-16T15:02:06.763703+0000","last_became_active":"2024-01-16T15:02:05.711545+0000","last_became_peered":"2024-01-16T15:02:05.711545+0000","last_unstale":"2024-01-16T15:02:06.763703+0000","last_undegraded":"2024-01-16T15:02:06.763703+0000","last_fullsized":"2024-01-16T15:02:06.763703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.719724+0000","last_change":"2024-01-16T15:02:04.677845+0000","last_active":"2024-01-16T15:02:05.719724+0000","last_peered":"2024-01-16T15:02:05.719724+0000","last_clean":"2024-01-16T15:02:05.719724+0000","last_became_active":"2024-01-16T15:02:04.674448+0000","last_became_peered":"2024-01-16T15:02:04.674448+0000","last_unstale":"2024-01-16T15:02:05.719724+0000","last_undegraded":"2024-01-16T15:02:05.719724+0000","last_fullsized":"2024-01-16T15:02:05.719724+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.508381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762560+0000","last_change":"2024-01-16T15:02:05.701672+0000","last_active":"2024-01-16T15:02:06.762560+0000","last_peered":"2024-01-16T15:02:06.762560+0000","last_clean":"2024-01-16T15:02:06.762560+0000","last_became_active":"2024-01-16T15:02:05.701181+0000","last_became_peered":"2024-01-16T15:02:05.701181+0000","last_unstale":"2024-01-16T15:02:06.762560+0000","last_undegraded":"2024-01-16T15:02:06.762560+0000","last_fullsized":"2024-01-16T15:02:06.762560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:40.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.717340+0000","last_change":"2024-01-16T15:02:04.678078+0000","last_active":"2024-01-16T15:02:05.717340+0000","last_peered":"2024-01-16T15:02:05.717340+0000","last_clean":"2024-01-16T15:02:05.717340+0000","last_became_active":"2024-01-16T15:02:04.677785+0000","last_became_peered":"2024-01-16T15:02:04.677785+0000","last_unstale":"2024-01-16T15:02:05.717340+0000","last_undegraded":"2024-01-16T15:02:05.717340+0000","last_fullsized":"2024-01-16T15:02:05.717340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:56.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763117+0000","last_change":"2024-01-16T15:02:05.695668+0000","last_active":"2024-01-16T15:02:06.763117+0000","last_peered":"2024-01-16T15:02:06.763117+0000","last_clean":"2024-01-16T15:02:06.763117+0000","last_became_active":"2024-01-16T15:02:05.695320+0000","last_became_peered":"2024-01-16T15:02:05.695320+0000","last_unstale":"2024-01-16T15:02:06.763117+0000","last_undegraded":"2024-01-16T15:02:06.763117+0000","last_fullsized":"2024-01-16T15:02:06.763117+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.702762+0000","last_change":"2024-01-16T15:02:04.677451+0000","last_active":"2024-01-16T15:02:05.702762+0000","last_peered":"2024-01-16T15:02:05.702762+0000","last_clean":"2024-01-16T15:02:05.702762+0000","last_became_active":"2024-01-16T15:02:04.676639+0000","last_became_peered":"2024-01-16T15:02:04.676639+0000","last_unstale":"2024-01-16T15:02:05.702762+0000","last_undegraded":"2024-01-16T15:02:05.702762+0000","last_fullsized":"2024-01-16T15:02:05.702762+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:45.344394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762756+0000","last_change":"2024-01-16T15:02:05.719489+0000","last_active":"2024-01-16T15:02:06.762756+0000","last_peered":"2024-01-16T15:02:06.762756+0000","last_clean":"2024-01-16T15:02:06.762756+0000","last_became_active":"2024-01-16T15:02:05.719272+0000","last_became_peered":"2024-01-16T15:02:05.719272+0000","last_unstale":"2024-01-16T15:02:06.762756+0000","last_undegraded":"2024-01-16T15:02:06.762756+0000","last_fullsized":"2024-01-16T15:02:06.762756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:44.866166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.717131+0000","last_change":"2024-01-16T15:02:04.680997+0000","last_active":"2024-01-16T15:02:05.717131+0000","last_peered":"2024-01-16T15:02:05.717131+0000","last_clean":"2024-01-16T15:02:05.717131+0000","last_became_active":"2024-01-16T15:02:04.680677+0000","last_became_peered":"2024-01-16T15:02:04.680677+0000","last_unstale":"2024-01-16T15:02:05.717131+0000","last_undegraded":"2024-01-16T15:02:05.717131+0000","last_fullsized":"2024-01-16T15:02:05.717131+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:30.329959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761987+0000","last_change":"2024-01-16T15:02:05.712735+0000","last_active":"2024-01-16T15:02:06.761987+0000","last_peered":"2024-01-16T15:02:06.761987+0000","last_clean":"2024-01-16T15:02:06.761987+0000","last_became_active":"2024-01-16T15:02:05.712459+0000","last_became_peered":"2024-01-16T15:02:05.712459+0000","last_unstale":"2024-01-16T15:02:06.761987+0000","last_undegraded":"2024-01-16T15:02:06.761987+0000","last_fullsized":"2024-01-16T15:02:06.761987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:24.174184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.713467+0000","last_change":"2024-01-16T15:02:04.688609+0000","last_active":"2024-01-16T15:02:05.713467+0000","last_peered":"2024-01-16T15:02:05.713467+0000","last_clean":"2024-01-16T15:02:05.713467+0000","last_became_active":"2024-01-16T15:02:04.688253+0000","last_became_peered":"2024-01-16T15:02:04.688253+0000","last_unstale":"2024-01-16T15:02:05.713467+0000","last_undegraded":"2024-01-16T15:02:05.713467+0000","last_fullsized":"2024-01-16T15:02:05.713467+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:21.918624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762078+0000","last_change":"2024-01-16T15:02:05.700757+0000","last_active":"2024-01-16T15:02:06.762078+0000","last_peered":"2024-01-16T15:02:06.762078+0000","last_clean":"2024-01-16T15:02:06.762078+0000","last_became_active":"2024-01-16T15:02:05.700491+0000","last_became_peered":"2024-01-16T15:02:05.700491+0000","last_unstale":"2024-01-16T15:02:06.762078+0000","last_undegraded":"2024-01-16T15:02:06.762078+0000","last_fullsized":"2024-01-16T15:02:06.762078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:06:29.331207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.715892+0000","last_change":"2024-01-16T15:02:04.673177+0000","last_active":"2024-01-16T15:02:05.715892+0000","last_peered":"2024-01-16T15:02:05.715892+0000","last_clean":"2024-01-16T15:02:05.715892+0000","last_became_active":"2024-01-16T15:02:04.672346+0000","last_became_peered":"2024-01-16T15:02:04.672346+0000","last_unstale":"2024-01-16T15:02:05.715892+0000","last_undegraded":"2024-01-16T15:02:05.715892+0000","last_fullsized":"2024-01-16T15:02:05.715892+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:45.746043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.707796+0000","last_change":"2024-01-16T15:02:05.717824+0000","last_active":"2024-01-16T15:02:06.707796+0000","last_peered":"2024-01-16T15:02:06.707796+0000","last_clean":"2024-01-16T15:02:06.707796+0000","last_became_active":"2024-01-16T15:02:05.717046+0000","last_became_peered":"2024-01-16T15:02:05.717046+0000","last_unstale":"2024-01-16T15:02:06.707796+0000","last_undegraded":"2024-01-16T15:02:06.707796+0000","last_fullsized":"2024-01-16T15:02:06.707796+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:34.237454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694691+0000","last_change":"2024-01-16T15:02:04.682008+0000","last_active":"2024-01-16T15:02:05.694691+0000","last_peered":"2024-01-16T15:02:05.694691+0000","last_clean":"2024-01-16T15:02:05.694691+0000","last_became_active":"2024-01-16T15:02:04.681608+0000","last_became_peered":"2024-01-16T15:02:04.681608+0000","last_unstale":"2024-01-16T15:02:05.694691+0000","last_undegraded":"2024-01-16T15:02:05.694691+0000","last_fullsized":"2024-01-16T15:02:05.694691+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:32:48.712541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761301+0000","last_change":"2024-01-16T15:02:05.710209+0000","last_active":"2024-01-16T15:02:06.761301+0000","last_peered":"2024-01-16T15:02:06.761301+0000","last_clean":"2024-01-16T15:02:06.761301+0000","last_became_active":"2024-01-16T15:02:05.709954+0000","last_became_peered":"2024-01-16T15:02:05.709954+0000","last_unstale":"2024-01-16T15:02:06.761301+0000","last_undegraded":"2024-01-16T15:02:06.761301+0000","last_fullsized":"2024-01-16T15:02:06.761301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:21.899539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.702674+0000","last_change":"2024-01-16T15:02:04.676299+0000","last_active":"2024-01-16T15:02:05.702674+0000","last_peered":"2024-01-16T15:02:05.702674+0000","last_clean":"2024-01-16T15:02:05.702674+0000","last_became_active":"2024-01-16T15:02:04.675978+0000","last_became_peered":"2024-01-16T15:02:04.675978+0000","last_unstale":"2024-01-16T15:02:05.702674+0000","last_undegraded":"2024-01-16T15:02:05.702674+0000","last_fullsized":"2024-01-16T15:02:05.702674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:56.132562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763833+0000","last_change":"2024-01-16T15:02:05.708487+0000","last_active":"2024-01-16T15:02:06.763833+0000","last_peered":"2024-01-16T15:02:06.763833+0000","last_clean":"2024-01-16T15:02:06.763833+0000","last_became_active":"2024-01-16T15:02:05.708156+0000","last_became_peered":"2024-01-16T15:02:05.708156+0000","last_unstale":"2024-01-16T15:02:06.763833+0000","last_undegraded":"2024-01-16T15:02:06.763833+0000","last_fullsized":"2024-01-16T15:02:06.763833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:30.597006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.685434+0000","last_change":"2024-01-16T15:02:04.684694+0000","last_active":"2024-01-16T15:02:05.685434+0000","last_peered":"2024-01-16T15:02:05.685434+0000","last_clean":"2024-01-16T15:02:05.685434+0000","last_became_active":"2024-01-16T15:02:04.684270+0000","last_became_peered":"2024-01-16T15:02:04.684270+0000","last_unstale":"2024-01-16T15:02:05.685434+0000","last_undegraded":"2024-01-16T15:02:05.685434+0000","last_fullsized":"2024-01-16T15:02:05.685434+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:34.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709753+0000","last_change":"2024-01-16T15:02:05.709225+0000","last_active":"2024-01-16T15:02:06.709753+0000","last_peered":"2024-01-16T15:02:06.709753+0000","last_clean":"2024-01-16T15:02:06.709753+0000","last_became_active":"2024-01-16T15:02:05.706336+0000","last_became_peered":"2024-01-16T15:02:05.706336+0000","last_unstale":"2024-01-16T15:02:06.709753+0000","last_undegraded":"2024-01-16T15:02:06.709753+0000","last_fullsized":"2024-01-16T15:02:06.709753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:06.881538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.686092+0000","last_change":"2024-01-16T15:02:04.690135+0000","last_active":"2024-01-16T15:02:05.686092+0000","last_peered":"2024-01-16T15:02:05.686092+0000","last_clean":"2024-01-16T15:02:05.686092+0000","last_became_active":"2024-01-16T15:02:04.689704+0000","last_became_peered":"2024-01-16T15:02:04.689704+0000","last_unstale":"2024-01-16T15:02:05.686092+0000","last_undegraded":"2024-01-16T15:02:05.686092+0000","last_fullsized":"2024-01-16T15:02:05.686092+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:41.180213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761413+0000","last_change":"2024-01-16T15:02:05.721151+0000","last_active":"2024-01-16T15:02:06.761413+0000","last_peered":"2024-01-16T15:02:06.761413+0000","last_clean":"2024-01-16T15:02:06.761413+0000","last_became_active":"2024-01-16T15:02:05.720706+0000","last_became_peered":"2024-01-16T15:02:05.720706+0000","last_unstale":"2024-01-16T15:02:06.761413+0000","last_undegraded":"2024-01-16T15:02:06.761413+0000","last_fullsized":"2024-01-16T15:02:06.761413+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:41.520010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.718037+0000","last_change":"2024-01-16T15:02:04.683511+0000","last_active":"2024-01-16T15:02:05.718037+0000","last_peered":"2024-01-16T15:02:05.718037+0000","last_clean":"2024-01-16T15:02:05.718037+0000","last_became_active":"2024-01-16T15:02:04.683083+0000","last_became_peered":"2024-01-16T15:02:04.683083+0000","last_unstale":"2024-01-16T15:02:05.718037+0000","last_undegraded":"2024-01-16T15:02:05.718037+0000","last_fullsized":"2024-01-16T15:02:05.718037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:45.344394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708016+0000","last_change":"2024-01-16T15:02:05.694323+0000","last_active":"2024-01-16T15:02:06.708016+0000","last_peered":"2024-01-16T15:02:06.708016+0000","last_clean":"2024-01-16T15:02:06.708016+0000","last_became_active":"2024-01-16T15:02:05.694043+0000","last_became_peered":"2024-01-16T15:02:05.694043+0000","last_unstale":"2024-01-16T15:02:06.708016+0000","last_undegraded":"2024-01-16T15:02:06.708016+0000","last_fullsized":"2024-01-16T15:02:06.708016+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.726616+0000","last_change":"2024-01-16T15:02:04.676811+0000","last_active":"2024-01-16T15:02:05.726616+0000","last_peered":"2024-01-16T15:02:05.726616+0000","last_clean":"2024-01-16T15:02:05.726616+0000","last_became_active":"2024-01-16T15:02:04.676452+0000","last_became_peered":"2024-01-16T15:02:04.676452+0000","last_unstale":"2024-01-16T15:02:05.726616+0000","last_undegraded":"2024-01-16T15:02:05.726616+0000","last_fullsized":"2024-01-16T15:02:05.726616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:34.921650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764153+0000","last_change":"2024-01-16T15:02:05.722270+0000","last_active":"2024-01-16T15:02:06.764153+0000","last_peered":"2024-01-16T15:02:06.764153+0000","last_clean":"2024-01-16T15:02:06.764153+0000","last_became_active":"2024-01-16T15:02:05.722018+0000","last_became_peered":"2024-01-16T15:02:05.722018+0000","last_unstale":"2024-01-16T15:02:06.764153+0000","last_undegraded":"2024-01-16T15:02:06.764153+0000","last_fullsized":"2024-01-16T15:02:06.764153+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:58.504291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721472+0000","last_change":"2024-01-16T15:02:04.689950+0000","last_active":"2024-01-16T15:02:05.721472+0000","last_peered":"2024-01-16T15:02:05.721472+0000","last_clean":"2024-01-16T15:02:05.721472+0000","last_became_active":"2024-01-16T15:02:04.689392+0000","last_became_peered":"2024-01-16T15:02:04.689392+0000","last_unstale":"2024-01-16T15:02:05.721472+0000","last_undegraded":"2024-01-16T15:02:05.721472+0000","last_fullsized":"2024-01-16T15:02:05.721472+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:21.190356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708298+0000","last_change":"2024-01-16T15:02:05.709542+0000","last_active":"2024-01-16T15:02:06.708298+0000","last_peered":"2024-01-16T15:02:06.708298+0000","last_clean":"2024-01-16T15:02:06.708298+0000","last_became_active":"2024-01-16T15:02:05.709256+0000","last_became_peered":"2024-01-16T15:02:05.709256+0000","last_unstale":"2024-01-16T15:02:06.708298+0000","last_undegraded":"2024-01-16T15:02:06.708298+0000","last_fullsized":"2024-01-16T15:02:06.708298+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698374+0000","last_change":"2024-01-16T15:02:04.670268+0000","last_active":"2024-01-16T15:02:05.698374+0000","last_peered":"2024-01-16T15:02:05.698374+0000","last_clean":"2024-01-16T15:02:05.698374+0000","last_became_active":"2024-01-16T15:02:04.669950+0000","last_became_peered":"2024-01-16T15:02:04.669950+0000","last_unstale":"2024-01-16T15:02:05.698374+0000","last_undegraded":"2024-01-16T15:02:05.698374+0000","last_fullsized":"2024-01-16T15:02:05.698374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:15.532329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762898+0000","last_change":"2024-01-16T15:02:05.719151+0000","last_active":"2024-01-16T15:02:06.762898+0000","last_peered":"2024-01-16T15:02:06.762898+0000","last_clean":"2024-01-16T15:02:06.762898+0000","last_became_active":"2024-01-16T15:02:05.716626+0000","last_became_peered":"2024-01-16T15:02:05.716626+0000","last_unstale":"2024-01-16T15:02:06.762898+0000","last_undegraded":"2024-01-16T15:02:06.762898+0000","last_fullsized":"2024-01-16T15:02:06.762898+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:24:00.524827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694838+0000","last_change":"2024-01-16T15:02:04.678906+0000","last_active":"2024-01-16T15:02:05.694838+0000","last_peered":"2024-01-16T15:02:05.694838+0000","last_clean":"2024-01-16T15:02:05.694838+0000","last_became_active":"2024-01-16T15:02:04.678448+0000","last_became_peered":"2024-01-16T15:02:04.678448+0000","last_unstale":"2024-01-16T15:02:05.694838+0000","last_undegraded":"2024-01-16T15:02:05.694838+0000","last_fullsized":"2024-01-16T15:02:05.694838+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.582380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709295+0000","last_change":"2024-01-16T15:02:05.697150+0000","last_active":"2024-01-16T15:02:06.709295+0000","last_peered":"2024-01-16T15:02:06.709295+0000","last_clean":"2024-01-16T15:02:06.709295+0000","last_became_active":"2024-01-16T15:02:05.696650+0000","last_became_peered":"2024-01-16T15:02:05.696650+0000","last_unstale":"2024-01-16T15:02:06.709295+0000","last_undegraded":"2024-01-16T15:02:06.709295+0000","last_fullsized":"2024-01-16T15:02:06.709295+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:53.498507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.720102+0000","last_change":"2024-01-16T15:02:04.682082+0000","last_active":"2024-01-16T15:02:05.720102+0000","last_peered":"2024-01-16T15:02:05.720102+0000","last_clean":"2024-01-16T15:02:05.720102+0000","last_became_active":"2024-01-16T15:02:04.681666+0000","last_became_peered":"2024-01-16T15:02:04.681666+0000","last_unstale":"2024-01-16T15:02:05.720102+0000","last_undegraded":"2024-01-16T15:02:05.720102+0000","last_fullsized":"2024-01-16T15:02:05.720102+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:41:45.515329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709241+0000","last_change":"2024-01-16T15:02:05.715186+0000","last_active":"2024-01-16T15:02:06.709241+0000","last_peered":"2024-01-16T15:02:06.709241+0000","last_clean":"2024-01-16T15:02:06.709241+0000","last_became_active":"2024-01-16T15:02:05.714827+0000","last_became_peered":"2024-01-16T15:02:05.714827+0000","last_unstale":"2024-01-16T15:02:06.709241+0000","last_undegraded":"2024-01-16T15:02:06.709241+0000","last_fullsized":"2024-01-16T15:02:06.709241+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:14.377080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721478+0000","last_change":"2024-01-16T15:02:04.683297+0000","last_active":"2024-01-16T15:02:05.721478+0000","last_peered":"2024-01-16T15:02:05.721478+0000","last_clean":"2024-01-16T15:02:05.721478+0000","last_became_active":"2024-01-16T15:02:04.682784+0000","last_became_peered":"2024-01-16T15:02:04.682784+0000","last_unstale":"2024-01-16T15:02:05.721478+0000","last_undegraded":"2024-01-16T15:02:05.721478+0000","last_fullsized":"2024-01-16T15:02:05.721478+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:36.189068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765550+0000","last_change":"2024-01-16T15:02:05.707115+0000","last_active":"2024-01-16T15:02:06.765550+0000","last_peered":"2024-01-16T15:02:06.765550+0000","last_clean":"2024-01-16T15:02:06.765550+0000","last_became_active":"2024-01-16T15:02:05.706616+0000","last_became_peered":"2024-01-16T15:02:05.706616+0000","last_unstale":"2024-01-16T15:02:06.765550+0000","last_undegraded":"2024-01-16T15:02:06.765550+0000","last_fullsized":"2024-01-16T15:02:06.765550+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.716177+0000","last_change":"2024-01-16T15:02:04.687138+0000","last_active":"2024-01-16T15:02:05.716177+0000","last_peered":"2024-01-16T15:02:05.716177+0000","last_clean":"2024-01-16T15:02:05.716177+0000","last_became_active":"2024-01-16T15:02:04.686853+0000","last_became_peered":"2024-01-16T15:02:04.686853+0000","last_unstale":"2024-01-16T15:02:05.716177+0000","last_undegraded":"2024-01-16T15:02:05.716177+0000","last_fullsized":"2024-01-16T15:02:05.716177+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:47.375738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762595+0000","last_change":"2024-01-16T15:02:05.716760+0000","last_active":"2024-01-16T15:02:06.762595+0000","last_peered":"2024-01-16T15:02:06.762595+0000","last_clean":"2024-01-16T15:02:06.762595+0000","last_became_active":"2024-01-16T15:02:05.716500+0000","last_became_peered":"2024-01-16T15:02:05.716500+0000","last_unstale":"2024-01-16T15:02:06.762595+0000","last_undegraded":"2024-01-16T15:02:06.762595+0000","last_fullsized":"2024-01-16T15:02:06.762595+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:34.557637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.715974+0000","last_change":"2024-01-16T15:02:04.682056+0000","last_active":"2024-01-16T15:02:05.715974+0000","last_peered":"2024-01-16T15:02:05.715974+0000","last_clean":"2024-01-16T15:02:05.715974+0000","last_became_active":"2024-01-16T15:02:04.681766+0000","last_became_peered":"2024-01-16T15:02:04.681766+0000","last_unstale":"2024-01-16T15:02:05.715974+0000","last_undegraded":"2024-01-16T15:02:05.715974+0000","last_fullsized":"2024-01-16T15:02:05.715974+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:59.510201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.709504+0000","last_change":"2024-01-16T15:02:05.703598+0000","last_active":"2024-01-16T15:02:06.709504+0000","last_peered":"2024-01-16T15:02:06.709504+0000","last_clean":"2024-01-16T15:02:06.709504+0000","last_became_active":"2024-01-16T15:02:05.702981+0000","last_became_peered":"2024-01-16T15:02:05.702981+0000","last_unstale":"2024-01-16T15:02:06.709504+0000","last_undegraded":"2024-01-16T15:02:06.709504+0000","last_fullsized":"2024-01-16T15:02:06.709504+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:37.203695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.693262+0000","last_change":"2024-01-16T15:02:04.681295+0000","last_active":"2024-01-16T15:02:05.693262+0000","last_peered":"2024-01-16T15:02:05.693262+0000","last_clean":"2024-01-16T15:02:05.693262+0000","last_became_active":"2024-01-16T15:02:04.680918+0000","last_became_peered":"2024-01-16T15:02:04.680918+0000","last_unstale":"2024-01-16T15:02:05.693262+0000","last_undegraded":"2024-01-16T15:02:05.693262+0000","last_fullsized":"2024-01-16T15:02:05.693262+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:44:27.577105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.764456+0000","last_change":"2024-01-16T15:02:05.722557+0000","last_active":"2024-01-16T15:02:06.764456+0000","last_peered":"2024-01-16T15:02:06.764456+0000","last_clean":"2024-01-16T15:02:06.764456+0000","last_became_active":"2024-01-16T15:02:05.722276+0000","last_became_peered":"2024-01-16T15:02:05.722276+0000","last_unstale":"2024-01-16T15:02:06.764456+0000","last_undegraded":"2024-01-16T15:02:06.764456+0000","last_fullsized":"2024-01-16T15:02:06.764456+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:54.427231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.721289+0000","last_change":"2024-01-16T15:02:04.680136+0000","last_active":"2024-01-16T15:02:05.721289+0000","last_peered":"2024-01-16T15:02:05.721289+0000","last_clean":"2024-01-16T15:02:05.721289+0000","last_became_active":"2024-01-16T15:02:04.679781+0000","last_became_peered":"2024-01-16T15:02:04.679781+0000","last_unstale":"2024-01-16T15:02:05.721289+0000","last_undegraded":"2024-01-16T15:02:05.721289+0000","last_fullsized":"2024-01-16T15:02:05.721289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:17.898530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762833+0000","last_change":"2024-01-16T15:02:05.708900+0000","last_active":"2024-01-16T15:02:06.762833+0000","last_peered":"2024-01-16T15:02:06.762833+0000","last_clean":"2024-01-16T15:02:06.762833+0000","last_became_active":"2024-01-16T15:02:05.708600+0000","last_became_peered":"2024-01-16T15:02:05.708600+0000","last_unstale":"2024-01-16T15:02:06.762833+0000","last_undegraded":"2024-01-16T15:02:06.762833+0000","last_fullsized":"2024-01-16T15:02:06.762833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:24.021974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.679302+0000","last_change":"2024-01-16T15:02:04.688154+0000","last_active":"2024-01-16T15:02:05.679302+0000","last_peered":"2024-01-16T15:02:05.679302+0000","last_clean":"2024-01-16T15:02:05.679302+0000","last_became_active":"2024-01-16T15:02:04.683089+0000","last_became_peered":"2024-01-16T15:02:04.683089+0000","last_unstale":"2024-01-16T15:02:05.679302+0000","last_undegraded":"2024-01-16T15:02:05.679302+0000","last_fullsized":"2024-01-16T15:02:05.679302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:21.797023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762594+0000","last_change":"2024-01-16T15:02:05.707619+0000","last_active":"2024-01-16T15:02:06.762594+0000","last_peered":"2024-01-16T15:02:06.762594+0000","last_clean":"2024-01-16T15:02:06.762594+0000","last_became_active":"2024-01-16T15:02:05.706744+0000","last_became_peered":"2024-01-16T15:02:05.706744+0000","last_unstale":"2024-01-16T15:02:06.762594+0000","last_undegraded":"2024-01-16T15:02:06.762594+0000","last_fullsized":"2024-01-16T15:02:06.762594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:54.427231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.697413+0000","last_change":"2024-01-16T15:02:04.679627+0000","last_active":"2024-01-16T15:02:05.697413+0000","last_peered":"2024-01-16T15:02:05.697413+0000","last_clean":"2024-01-16T15:02:05.697413+0000","last_became_active":"2024-01-16T15:02:04.679163+0000","last_became_peered":"2024-01-16T15:02:04.679163+0000","last_unstale":"2024-01-16T15:02:05.697413+0000","last_undegraded":"2024-01-16T15:02:05.697413+0000","last_fullsized":"2024-01-16T15:02:05.697413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:47.375738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765261+0000","last_change":"2024-01-16T15:02:05.713955+0000","last_active":"2024-01-16T15:02:06.765261+0000","last_peered":"2024-01-16T15:02:06.765261+0000","last_clean":"2024-01-16T15:02:06.765261+0000","last_became_active":"2024-01-16T15:02:05.713526+0000","last_became_peered":"2024-01-16T15:02:05.713526+0000","last_unstale":"2024-01-16T15:02:06.765261+0000","last_undegraded":"2024-01-16T15:02:06.765261+0000","last_fullsized":"2024-01-16T15:02:06.765261+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:44.415304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.719125+0000","last_change":"2024-01-16T15:02:04.685777+0000","last_active":"2024-01-16T15:02:05.719125+0000","last_peered":"2024-01-16T15:02:05.719125+0000","last_clean":"2024-01-16T15:02:05.719125+0000","last_became_active":"2024-01-16T15:02:04.685481+0000","last_became_peered":"2024-01-16T15:02:04.685481+0000","last_unstale":"2024-01-16T15:02:05.719125+0000","last_undegraded":"2024-01-16T15:02:05.719125+0000","last_fullsized":"2024-01-16T15:02:05.719125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:12.921478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.765659+0000","last_change":"2024-01-16T15:02:05.708736+0000","last_active":"2024-01-16T15:02:06.765659+0000","last_peered":"2024-01-16T15:02:06.765659+0000","last_clean":"2024-01-16T15:02:06.765659+0000","last_became_active":"2024-01-16T15:02:05.708470+0000","last_became_peered":"2024-01-16T15:02:05.708470+0000","last_unstale":"2024-01-16T15:02:06.765659+0000","last_undegraded":"2024-01-16T15:02:06.765659+0000","last_fullsized":"2024-01-16T15:02:06.765659+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694340+0000","last_change":"2024-01-16T15:02:04.671324+0000","last_active":"2024-01-16T15:02:05.694340+0000","last_peered":"2024-01-16T15:02:05.694340+0000","last_clean":"2024-01-16T15:02:05.694340+0000","last_became_active":"2024-01-16T15:02:04.671056+0000","last_became_peered":"2024-01-16T15:02:04.671056+0000","last_unstale":"2024-01-16T15:02:05.694340+0000","last_undegraded":"2024-01-16T15:02:05.694340+0000","last_fullsized":"2024-01-16T15:02:05.694340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:34.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761500+0000","last_change":"2024-01-16T15:02:05.720952+0000","last_active":"2024-01-16T15:02:06.761500+0000","last_peered":"2024-01-16T15:02:06.761500+0000","last_clean":"2024-01-16T15:02:06.761500+0000","last_became_active":"2024-01-16T15:02:05.720361+0000","last_became_peered":"2024-01-16T15:02:05.720361+0000","last_unstale":"2024-01-16T15:02:06.761500+0000","last_undegraded":"2024-01-16T15:02:06.761500+0000","last_fullsized":"2024-01-16T15:02:06.761500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:51.004607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698305+0000","last_change":"2024-01-16T15:02:04.677076+0000","last_active":"2024-01-16T15:02:05.698305+0000","last_peered":"2024-01-16T15:02:05.698305+0000","last_clean":"2024-01-16T15:02:05.698305+0000","last_became_active":"2024-01-16T15:02:04.674921+0000","last_became_peered":"2024-01-16T15:02:04.674921+0000","last_unstale":"2024-01-16T15:02:05.698305+0000","last_undegraded":"2024-01-16T15:02:05.698305+0000","last_fullsized":"2024-01-16T15:02:05.698305+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:15.371747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.763397+0000","last_change":"2024-01-16T15:02:05.717087+0000","last_active":"2024-01-16T15:02:06.763397+0000","last_peered":"2024-01-16T15:02:06.763397+0000","last_clean":"2024-01-16T15:02:06.763397+0000","last_became_active":"2024-01-16T15:02:05.716773+0000","last_became_peered":"2024-01-16T15:02:05.716773+0000","last_unstale":"2024-01-16T15:02:06.763397+0000","last_undegraded":"2024-01-16T15:02:06.763397+0000","last_fullsized":"2024-01-16T15:02:06.763397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:42.194839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.701829+0000","last_change":"2024-01-16T15:02:04.683141+0000","last_active":"2024-01-16T15:02:05.701829+0000","last_peered":"2024-01-16T15:02:05.701829+0000","last_clean":"2024-01-16T15:02:05.701829+0000","last_became_active":"2024-01-16T15:02:04.682515+0000","last_became_peered":"2024-01-16T15:02:04.682515+0000","last_unstale":"2024-01-16T15:02:05.701829+0000","last_undegraded":"2024-01-16T15:02:05.701829+0000","last_fullsized":"2024-01-16T15:02:05.701829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:41.508381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.762011+0000","last_change":"2024-01-16T15:02:05.723101+0000","last_active":"2024-01-16T15:02:06.762011+0000","last_peered":"2024-01-16T15:02:06.762011+0000","last_clean":"2024-01-16T15:02:06.762011+0000","last_became_active":"2024-01-16T15:02:05.722293+0000","last_became_peered":"2024-01-16T15:02:05.722293+0000","last_unstale":"2024-01-16T15:02:06.762011+0000","last_undegraded":"2024-01-16T15:02:06.762011+0000","last_fullsized":"2024-01-16T15:02:06.762011+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:03.615829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.677315+0000","last_change":"2024-01-16T15:02:04.681828+0000","last_active":"2024-01-16T15:02:05.677315+0000","last_peered":"2024-01-16T15:02:05.677315+0000","last_clean":"2024-01-16T15:02:05.677315+0000","last_became_active":"2024-01-16T15:02:04.681571+0000","last_became_peered":"2024-01-16T15:02:04.681571+0000","last_unstale":"2024-01-16T15:02:05.677315+0000","last_undegraded":"2024-01-16T15:02:05.677315+0000","last_fullsized":"2024-01-16T15:02:05.677315+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:24.494050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.708624+0000","last_change":"2024-01-16T15:02:05.697525+0000","last_active":"2024-01-16T15:02:06.708624+0000","last_peered":"2024-01-16T15:02:06.708624+0000","last_clean":"2024-01-16T15:02:06.708624+0000","last_became_active":"2024-01-16T15:02:05.697166+0000","last_became_peered":"2024-01-16T15:02:05.697166+0000","last_unstale":"2024-01-16T15:02:06.708624+0000","last_undegraded":"2024-01-16T15:02:06.708624+0000","last_fullsized":"2024-01-16T15:02:06.708624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:57.687490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.704166+0000","last_change":"2024-01-16T15:02:04.684667+0000","last_active":"2024-01-16T15:02:05.704166+0000","last_peered":"2024-01-16T15:02:05.704166+0000","last_clean":"2024-01-16T15:02:05.704166+0000","last_became_active":"2024-01-16T15:02:04.684301+0000","last_became_peered":"2024-01-16T15:02:04.684301+0000","last_unstale":"2024-01-16T15:02:05.704166+0000","last_undegraded":"2024-01-16T15:02:05.704166+0000","last_fullsized":"2024-01-16T15:02:05.704166+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:11.450047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.760698+0000","last_change":"2024-01-16T15:02:05.705898+0000","last_active":"2024-01-16T15:02:06.760698+0000","last_peered":"2024-01-16T15:02:06.760698+0000","last_clean":"2024-01-16T15:02:06.760698+0000","last_became_active":"2024-01-16T15:02:05.705594+0000","last_became_peered":"2024-01-16T15:02:05.705594+0000","last_unstale":"2024-01-16T15:02:06.760698+0000","last_undegraded":"2024-01-16T15:02:06.760698+0000","last_fullsized":"2024-01-16T15:02:06.760698+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:16.386373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.678340+0000","last_change":"2024-01-16T15:02:04.685022+0000","last_active":"2024-01-16T15:02:05.678340+0000","last_peered":"2024-01-16T15:02:05.678340+0000","last_clean":"2024-01-16T15:02:05.678340+0000","last_became_active":"2024-01-16T15:02:04.684477+0000","last_became_peered":"2024-01-16T15:02:04.684477+0000","last_unstale":"2024-01-16T15:02:05.678340+0000","last_undegraded":"2024-01-16T15:02:05.678340+0000","last_fullsized":"2024-01-16T15:02:05.678340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:52.853624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761590+0000","last_change":"2024-01-16T15:02:05.710367+0000","last_active":"2024-01-16T15:02:06.761590+0000","last_peered":"2024-01-16T15:02:06.761590+0000","last_clean":"2024-01-16T15:02:06.761590+0000","last_became_active":"2024-01-16T15:02:05.709916+0000","last_became_peered":"2024-01-16T15:02:05.709916+0000","last_unstale":"2024-01-16T15:02:06.761590+0000","last_undegraded":"2024-01-16T15:02:06.761590+0000","last_fullsized":"2024-01-16T15:02:06.761590+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:05.382187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.694881+0000","last_change":"2024-01-16T15:02:04.676924+0000","last_active":"2024-01-16T15:02:05.694881+0000","last_peered":"2024-01-16T15:02:05.694881+0000","last_clean":"2024-01-16T15:02:05.694881+0000","last_became_active":"2024-01-16T15:02:04.676651+0000","last_became_peered":"2024-01-16T15:02:04.676651+0000","last_unstale":"2024-01-16T15:02:05.694881+0000","last_undegraded":"2024-01-16T15:02:05.694881+0000","last_fullsized":"2024-01-16T15:02:05.694881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:00.305823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:06.761218+0000","last_change":"2024-01-16T15:02:05.712715+0000","last_active":"2024-01-16T15:02:06.761218+0000","last_peered":"2024-01-16T15:02:06.761218+0000","last_clean":"2024-01-16T15:02:06.761218+0000","last_became_active":"2024-01-16T15:02:05.712413+0000","last_became_peered":"2024-01-16T15:02:05.712413+0000","last_unstale":"2024-01-16T15:02:06.761218+0000","last_undegraded":"2024-01-16T15:02:06.761218+0000","last_fullsized":"2024-01-16T15:02:06.761218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:04.642731+0000","last_clean_scrub_stamp":"2024-01-16T15:02:04.642731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:42.286960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.698184+0000","last_change":"2024-01-16T15:02:04.679821+0000","last_active":"2024-01-16T15:02:05.698184+0000","last_peered":"2024-01-16T15:02:05.698184+0000","last_clean":"2024-01-16T15:02:05.698184+0000","last_became_active":"2024-01-16T15:02:04.679528+0000","last_became_peered":"2024-01-16T15:02:04.679528+0000","last_unstale":"2024-01-16T15:02:05.698184+0000","last_undegraded":"2024-01-16T15:02:05.698184+0000","last_fullsized":"2024-01-16T15:02:05.698184+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:03.628105+0000","last_clean_scrub_stamp":"2024-01-16T15:02:03.628105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:59.388150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:05.682313+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:02:05.682313+0000","last_peered":"2024-01-16T15:02:05.682313+0000","last_clean":"2024-01-16T15:02:05.682313+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:02:05.682313+0000","last_undegraded":"2024-01-16T15:02:05.682313+0000","last_fullsized":"2024-01-16T15:02:05.682313+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1298432,"data_stored":881139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":1.5249999999999999,"15min":1.1579999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":45.296999999999997,"5min":45.296999999999997,"15min":45.296999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.724,"5min":1.3560000000000001,"15min":1.1779999999999999},"min":{"1min":0.76000000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":9.2690000000000001,"5min":9.2690000000000001,"15min":9.2690000000000001},"last":0.86199999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.2809999999999999,"15min":1.153},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":10.085000000000001,"5min":10.085000000000001,"15min":10.085000000000001},"last":1.022},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.3779999999999999,"15min":1.268},"min":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":11.356999999999999,"5min":11.356999999999999,"15min":11.356999999999999},"last":1.484}]},{"osd":2,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.256,"15min":1.2090000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.3900000000000001,"5min":1.536,"15min":1.3140000000000001},"min":{"1min":0.71999999999999997,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":23.594000000000001,"5min":23.594000000000001,"15min":23.594000000000001},"last":1.2330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.296,"15min":1.248},"min":{"1min":0.85599999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.4510000000000001,"15min":1.369},"min":{"1min":0.81699999999999995,"5min":0.751,"15min":0.751},"max":{"1min":4.8239999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":1.347}]},{"osd":4,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.4059999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":3.5939999999999999,"5min":5.1870000000000003,"15min":5.1870000000000003},"last":1.702},{"interface":"front","average":{"1min":1.252,"5min":1.7,"15min":1.484},"min":{"1min":0.626,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.274,"5min":36.374000000000002,"15min":36.374000000000002},"last":1.4119999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.409,"15min":1.3620000000000001},"min":{"1min":0.48599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.073,"5min":6.7190000000000003,"15min":6.7190000000000003},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.476,"15min":1.425},"min":{"1min":0.56699999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.327,"5min":3.5139999999999998,"15min":3.5139999999999998},"last":1.835}]},{"osd":6,"last update":"Tue Jan 16 15:01:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4790000000000001,"15min":1.5249999999999999},"min":{"1min":0.627,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":4.1289999999999996,"5min":4.1289999999999996,"15min":4.1289999999999996},"last":2.052},{"interface":"front","average":{"1min":1.591,"5min":1.613,"15min":1.6000000000000001},"min":{"1min":0.64900000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":4.0940000000000003,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.778}]}]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":696320,"data_stored":290611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":5.7859999999999996,"5min":2.0579999999999998,"15min":1.4019999999999999},"min":{"1min":0.69599999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":71.870999999999995,"5min":71.870999999999995,"15min":71.870999999999995},"last":1.157},{"interface":"front","average":{"1min":5.7779999999999996,"5min":2.169,"15min":1.5449999999999999},"min":{"1min":0.98899999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":71.846999999999994,"5min":71.846999999999994,"15min":71.846999999999994},"last":1.306}]},{"osd":1,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.1399999999999999,"15min":1.075},"min":{"1min":0.73199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":3.0430000000000001,"5min":3.0430000000000001,"15min":3.0430000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":2.5009999999999999,"5min":1.405,"15min":1.2210000000000001},"min":{"1min":0.84099999999999997,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":13.839,"5min":13.839,"15min":13.839},"last":0.91500000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.282,"15min":1.23},"min":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":4.9080000000000004,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":1.571},{"interface":"front","average":{"1min":1.53,"5min":1.353,"15min":1.3049999999999999},"min":{"1min":0.86599999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":1.46}]},{"osd":3,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.345,"15min":1.369},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.214,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.206},{"interface":"front","average":{"1min":1.835,"5min":1.4430000000000001,"15min":1.423},"min":{"1min":1.0349999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":1.3400000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.4099999999999999,"15min":1.4359999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":7.4320000000000004,"5min":7.4320000000000004,"15min":7.4320000000000004},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4179999999999999,"15min":1.46},"min":{"1min":0.71299999999999997,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.391}]},{"osd":5,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.413,"15min":1.4490000000000001},"min":{"1min":0.68799999999999994,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.829,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.6120000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":1.6899999999999999,"15min":1.609},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":7.508,"5min":7.508,"15min":7.508},"last":1.427}]},{"osd":7,"last update":"Tue Jan 16 15:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5620000000000001,"15min":1.5800000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.9510000000000001,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.546,"15min":1.5760000000000001},"min":{"1min":0.68999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.5329999999999999,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":1.7170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":291263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.417,"15min":1.165},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":25.129999999999999,"5min":25.129999999999999,"15min":25.129999999999999},"last":1.595},{"interface":"front","average":{"1min":1.377,"5min":1.2829999999999999,"15min":1.169},"min":{"1min":0.503,"5min":0.499,"15min":0.499},"max":{"1min":4.7759999999999998,"5min":6.0960000000000001,"15min":6.0960000000000001},"last":1.002}]},{"osd":1,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.359,"15min":1.244},"min":{"1min":0.69599999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.871,"5min":11.298999999999999,"15min":11.298999999999999},"last":1.157},{"interface":"front","average":{"1min":1.367,"5min":1.242,"15min":1.1870000000000001},"min":{"1min":0.67300000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":4.8259999999999996,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":1.381}]},{"osd":2,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.3029999999999999,"15min":1.319},"min":{"1min":0.71099999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":10.321999999999999,"5min":10.321999999999999,"15min":10.321999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.3480000000000001,"15min":1.2769999999999999},"min":{"1min":0.83099999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":4.7160000000000002,"5min":6.2809999999999997,"15min":6.2809999999999997},"last":1.4810000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1899999999999999,"5min":1.55,"15min":1.4330000000000001},"min":{"1min":0.755,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":1.4339999999999999},{"interface":"front","average":{"1min":2.319,"5min":1.5700000000000001,"15min":1.4199999999999999},"min":{"1min":0.98899999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":11.811,"5min":11.811,"15min":11.811},"last":1.111}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":1.375,"15min":1.2789999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.5209999999999999,"15min":1.5089999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":7.1769999999999996,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.647}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4279999999999999,"15min":1.4950000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.8769999999999998,"5min":4.8769999999999998,"15min":4.8769999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.4690000000000001,"15min":1.399},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":8.7070000000000007,"5min":10.015000000000001,"15min":10.015000000000001},"last":1.2390000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.4630000000000001,"15min":1.546},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.901,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.742,"5min":1.597,"15min":1.579},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":1.284}]}]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":708608,"data_stored":292223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.325,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.9500000000000002,"5min":6.5579999999999998,"15min":6.5579999999999998},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.3,"15min":1.262},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":3.9769999999999999},"last":1.258}]},{"osd":1,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.2509999999999999,"15min":1.1259999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":7.5720000000000001,"5min":7.5720000000000001,"15min":7.5720000000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.1719999999999999,"15min":1.1419999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.552}]},{"osd":2,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.377,"15min":1.2330000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.4900000000000002,"5min":8.0579999999999998,"15min":8.0579999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.3700000000000001,"15min":1.3240000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":8.4239999999999995,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.70199999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2370000000000001,"15min":1.2649999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":1.4870000000000001},{"interface":"front","average":{"1min":2.762,"5min":1.8069999999999999,"15min":1.55},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":27.626999999999999,"5min":27.626999999999999,"15min":27.626999999999999},"last":1.325}]},{"osd":5,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.3859999999999999,"15min":1.3999999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.2599999999999998},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.4410000000000001,"15min":1.5169999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.788,"5min":11.289999999999999,"15min":11.289999999999999},"last":1.3660000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.5189999999999999,"15min":1.5449999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":5.9210000000000003,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4990000000000001,"15min":1.579},"min":{"1min":0.70199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.4820000000000002,"5min":3.2770000000000001,"15min":3.2770000000000001},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.4750000000000001,"15min":1.5880000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1779999999999999,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.603},{"interface":"front","average":{"1min":2.4020000000000001,"5min":1.8959999999999999,"15min":1.7529999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":19.001000000000001,"5min":19.001000000000001,"15min":19.001000000000001},"last":1.524}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":290667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":0.94099999999999995,"15min":0.92700000000000005},"min":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":4.4820000000000002,"5min":4.4820000000000002,"15min":4.4820000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.1629999999999998,"5min":1.2350000000000001,"15min":1.018},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":20.012,"5min":20.012,"15min":20.012},"last":0.78900000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3519999999999999,"5min":1.238,"15min":1.0189999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":19.722000000000001,"5min":19.722000000000001,"15min":19.722000000000001},"last":0.876},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.0449999999999999,"15min":1.0720000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"last":1.081}]},{"osd":2,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":1.327,"15min":1.1160000000000001},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":20.074000000000002,"5min":20.074000000000002,"15min":20.074000000000002},"last":1.385},{"interface":"front","average":{"1min":2.5289999999999999,"5min":1.419,"15min":1.204},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":19.777999999999999,"5min":19.777999999999999,"15min":19.777999999999999},"last":1.2669999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":1.5469999999999999,"15min":1.413},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":21.736000000000001,"5min":21.736000000000001,"15min":21.736000000000001},"last":1.155},{"interface":"front","average":{"1min":3.1909999999999998,"5min":1.7030000000000001,"15min":1.472},"min":{"1min":0.96799999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":20.43,"5min":20.43,"15min":20.43},"last":1.409}]},{"osd":5,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.3919999999999999,"15min":1.4570000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":4.6909999999999998,"5min":4.6909999999999998,"15min":4.6909999999999998},"last":1.206},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4379999999999999,"15min":1.474},"min":{"1min":0.80400000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"last":1.502}]},{"osd":6,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.3969999999999998,"5min":1.6359999999999999,"15min":1.4770000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":20.459,"5min":20.459,"15min":20.459},"last":1.4410000000000001},{"interface":"front","average":{"1min":2.8010000000000002,"5min":1.752,"15min":1.5740000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":19.928999999999998,"5min":19.928999999999998,"15min":19.928999999999998},"last":1.3480000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4140000000000001,"5min":1.74,"15min":1.534},"min":{"1min":1.0389999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":20.109999999999999,"5min":20.109999999999999,"15min":20.109999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":1.7949999999999999,"15min":1.607},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":19.975000000000001,"5min":19.975000000000001,"15min":19.975000000000001},"last":1.4670000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":704512,"data_stored":290667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.419,"5min":1.2649999999999999,"15min":1.149},"min":{"1min":0.42299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":25.335999999999999,"5min":25.335999999999999,"15min":25.335999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":2.5800000000000001,"5min":1.3320000000000001,"15min":1.1919999999999999},"min":{"1min":0.47499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":24.798999999999999,"5min":24.798999999999999,"15min":24.798999999999999},"last":0.85699999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.2509999999999999,"15min":1.2569999999999999},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.8109999999999999,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":1.329},{"interface":"front","average":{"1min":2.3399999999999999,"5min":1.4330000000000001,"15min":1.403},"min":{"1min":0.376,"5min":0.373,"15min":0.373},"max":{"1min":21.439,"5min":21.439,"15min":21.439},"last":1.163}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.1479999999999999,"15min":1.218},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":8.1980000000000004,"5min":8.1980000000000004,"15min":8.1980000000000004},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.4950000000000001,"15min":1.4510000000000001},"min":{"1min":0.53200000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":28.045999999999999,"5min":28.045999999999999,"15min":28.045999999999999},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":3.2189999999999999,"5min":1.7709999999999999,"15min":1.589},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":21.532,"5min":21.532,"15min":21.532},"last":1.421},{"interface":"front","average":{"1min":2.5539999999999998,"5min":1.6020000000000001,"15min":1.506},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":14.597,"5min":14.597,"15min":14.597},"last":1.512}]},{"osd":5,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.454,"15min":1.5349999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.762,"5min":4.9199999999999999,"15min":4.9199999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":2.726,"5min":1.75,"15min":1.637},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":21.608000000000001,"5min":21.608000000000001,"15min":21.608000000000001},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.605,"15min":1.5369999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":1.577},{"interface":"front","average":{"1min":1.522,"5min":1.5349999999999999,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.871,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.6950000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":3.0110000000000001,"5min":1.8560000000000001,"15min":1.7410000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":21.571999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.5369999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":12.456,"5min":12.456,"15min":12.456},"last":1.385}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":700416,"data_stored":290701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.1299999999999999,"15min":1.01},"min":{"1min":0.65500000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.147,"5min":10.147,"15min":10.147},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.863,"5min":1.1930000000000001,"15min":0.95899999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":10.563000000000001,"5min":10.563000000000001,"15min":10.563000000000001},"last":0.879}]},{"osd":2,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":1.329,"15min":1.0529999999999999},"min":{"1min":0.48199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":24.709,"5min":24.709,"15min":24.709},"last":1.268},{"interface":"front","average":{"1min":2.7759999999999998,"5min":1.468,"15min":1.21},"min":{"1min":0.73499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":30.128,"5min":30.128,"15min":30.128},"last":1.071}]},{"osd":3,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.3109999999999999,"15min":1.212},"min":{"1min":0.61099999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":5.1890000000000001,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.331},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.1699999999999999,"15min":1.0509999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7709999999999999,"5min":6.5890000000000004,"15min":6.5890000000000004},"last":1.01}]},{"osd":4,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.901,"5min":1.4339999999999999,"15min":1.302},"min":{"1min":0.877,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":14.242000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.891,"5min":1.5509999999999999,"15min":1.3999999999999999},"min":{"1min":0.81999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":6.734,"5min":6.9480000000000004,"15min":6.9480000000000004},"last":1.516}]},{"osd":5,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.5189999999999999,"15min":1.425},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":7.3490000000000002,"5min":7.3490000000000002,"15min":7.3490000000000002},"last":1.4630000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.6060000000000001,"15min":1.486},"min":{"1min":1.008,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":14.153,"5min":14.153,"15min":14.153},"last":1.6200000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.5880000000000001,"15min":1.512},"min":{"1min":0.78200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.458,"15min":1.405},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.7589999999999999,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.55,"15min":1.51},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"last":1.113},{"interface":"front","average":{"1min":2.2799999999999998,"5min":1.6279999999999999,"15min":1.518},"min":{"1min":0.81000000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":14.331,"5min":14.331,"15min":14.331},"last":1.3799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":882154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":1.1100000000000001,"15min":0.95799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.948,"5min":18.948,"15min":18.948},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.1200000000000001,"5min":1.161,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.750999999999999,"5min":15.750999999999999,"15min":15.750999999999999},"last":0.65900000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":1.2609999999999999,"15min":1.135},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":13.44,"5min":13.44,"15min":13.44},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.2150000000000001,"15min":1.071},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":16.190999999999999,"5min":16.190999999999999,"15min":16.190999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.2310000000000001,"15min":1.0449999999999999},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.102,"5min":15.102,"15min":15.102},"last":0.626},{"interface":"front","average":{"1min":2.6600000000000001,"5min":1.476,"15min":1.3200000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":15.365,"5min":15.365,"15min":15.365},"last":0.69299999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":1.6319999999999999,"15min":1.4259999999999999},"min":{"1min":0.747,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":15.832000000000001,"5min":15.832000000000001,"15min":15.832000000000001},"last":1.1990000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.401,"15min":1.2889999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":11.404999999999999,"5min":11.404999999999999,"15min":11.404999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3149999999999999,"5min":1.617,"15min":1.54},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":13.794,"5min":13.794,"15min":13.794},"last":1.054},{"interface":"front","average":{"1min":2.2839999999999998,"5min":1.597,"15min":1.4870000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":15.906000000000001,"5min":15.906000000000001,"15min":15.906000000000001},"last":1.2689999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":1.6499999999999999,"15min":1.4430000000000001},"min":{"1min":0.77200000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":11.367000000000001,"5min":16.960999999999999,"15min":16.960999999999999},"last":1.3},{"interface":"front","average":{"1min":2.2930000000000001,"5min":1.609,"15min":1.5529999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":13.499000000000001,"5min":13.499000000000001,"15min":13.499000000000001},"last":0.95099999999999996}]},{"osd":7,"last update":"Tue Jan 16 15:01:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":1.5629999999999999,"15min":1.5069999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":16.039999999999999,"5min":16.039999999999999,"15min":16.039999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":2.54,"5min":1.8109999999999999,"15min":1.5800000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":11.433,"5min":17.038,"15min":17.038},"last":1.071}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:10.629 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:02:10.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:02:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.898+0000 7f1a55a00640 1 Processor -- start 2024-01-16T15:02:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.898+0000 7f1a55a00640 1 -- start start 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a55a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a500d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a55a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a500b22a0 con 0x7f1a500d0140 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a55a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a500afec0 con 0x7f1a5015da20 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a55a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500b01a0 con 0x7f1a5015d640 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a4f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1a500d0140 0x7f1a500b0f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60756/0 (socket says 172.21.15.125:60756) 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.900+0000 7f1a4f400640 1 -- 172.21.15.125:0/4034003398 learned_addr learned my addr 172.21.15.125:0/4034003398 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:10.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4ea00640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a500d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802135276 0 0) 0x7f1a500b22a0 con 0x7f1a500d0140 2024-01-16T15:02:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a30003520 con 0x7f1a500d0140 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299844888 0 0) 0x7f1a30003520 con 0x7f1a500d0140 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 msgr2=0x7f1a500d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a500d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 >> v1:172.21.15.137:6789/0 conn(0x7f1a5015da20 legacy=0x7f1a500cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500b0480 con 0x7f1a500d0140 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1a40001670 con 0x7f1a500d0140 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 >> v1:172.21.15.125:6789/0 conn(0x7f1a500d0140 legacy=0x7f1a500b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a300039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a30003f20 0x7f1a300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.901+0000 7f1a4d600640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500b01a0 con 0x7f1a30004d00 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3000ba60 con 0x7f1a30003f20 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3000bc20 con 0x7f1a5015d640 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4ea00640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a300039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4e000640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a30003f20 0x7f1a300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 -- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 msgr2=0x7f1a300039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a5015d640 0x7f1a300039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 -- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a30003f20 msgr2=0x7f1a300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a30003f20 0x7f1a300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.902+0000 7f1a4f400640 1 -- 172.21.15.125:0/4034003398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a30003520 con 0x7f1a30004d00 2024-01-16T15:02:10.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.903+0000 7f1a4f400640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a30005180 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c240 tx=0x7f1a300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700ea19a60ee8159 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40005ad0 con 0x7f1a30004d00 2024-01-16T15:02:10.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a40005c30 con 0x7f1a30004d00 2024-01-16T15:02:10.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a4d600640 1 -- 172.21.15.125:0/4034003398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4002caf0 con 0x7f1a30004d00 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a55a00640 1 -- 172.21.15.125:0/4034003398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 msgr2=0x7f1a30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4034003398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a30005180 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c240 tx=0x7f1a300057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.904+0000 7f1a4f400640 1 -- 172.21.15.125:0/4034003398 reap_dead start 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.905+0000 7f1a55a00640 1 -- 172.21.15.125:0/4034003398 shutdown_connections 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.905+0000 7f1a55a00640 1 -- 172.21.15.125:0/4034003398 >> 172.21.15.125:0/4034003398 conn(0x7f1a5009c560 msgr2=0x7f1a500d3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.905+0000 7f1a55a00640 1 -- 172.21.15.125:0/4034003398 shutdown_connections 2024-01-16T15:02:10.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.905+0000 7f1a55a00640 1 -- 172.21.15.125:0/4034003398 wait complete. 2024-01-16T15:02:10.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.906+0000 7f1a55a00640 1 Processor -- start 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.907+0000 7f1a55a00640 1 -- start start 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.907+0000 7f1a55a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a55a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a500ccd70 0x7f1a5014e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a55a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a500cb4f0 0x7f1a500b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a55a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5014ec50 con 0x7f1a30004d00 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a55a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500cd1f0 con 0x7f1a500cb4f0 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a55a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500cbcd0 con 0x7f1a500ccd70 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50854/0 (socket says 172.21.15.125:50854) 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 -- 172.21.15.125:0/4278671337 learned_addr learned my addr 172.21.15.125:0/4278671337 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 -- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a500ccd70 msgr2=0x7f1a5014e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a500ccd70 0x7f1a5014e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 -- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a500cb4f0 msgr2=0x7f1a500b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a500cb4f0 0x7f1a500b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.908+0000 7f1a4ea00640 1 -- 172.21.15.125:0/4278671337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500cbfb0 con 0x7f1a30004d00 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a4ea00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1a48011600 tx=0x7f1a48011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48015870 con 0x7f1a30004d00 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a480159d0 con 0x7f1a30004d00 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a500cc290 con 0x7f1a30004d00 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48015d40 con 0x7f1a30004d00 2024-01-16T15:02:10.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.909+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a500cc850 con 0x7f1a30004d00 2024-01-16T15:02:10.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.910+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a50067670 con 0x7f1a30004d00 2024-01-16T15:02:10.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.911+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1a48013430 con 0x7f1a30004d00 2024-01-16T15:02:10.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.912+0000 7f1a46a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 0x7f1a2806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.912+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a4808aa70 con 0x7f1a30004d00 2024-01-16T15:02:10.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.913+0000 7f1a4e000640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 0x7f1a2806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.913+0000 7f1a4e000640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 0x7f1a2806dd50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1a38007fd0 tx=0x7f1a38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:10.917+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a48055030 con 0x7f1a30004d00 2024-01-16T15:02:11.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.174+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1a5015df70 con 0x7f1a30004d00 2024-01-16T15:02:11.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.179+0000 7f1a46a00640 1 -- 172.21.15.125:0/4278671337 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f1a480220b0 con 0x7f1a30004d00 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:11.160 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T15:02:11.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.181+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 msgr2=0x7f1a2806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 0x7f1a2806dd50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1a38007fd0 tx=0x7f1a38009040 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 msgr2=0x7f1a500b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1a48011600 tx=0x7f1a48011ac0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 shutdown_connections 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a500cb4f0 0x7f1a500b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a500ccd70 0x7f1a5014e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a2806b890 0x7f1a2806dd50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 --2- 172.21.15.125:0/4278671337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a30004d00 0x7f1a500b36e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 >> 172.21.15.125:0/4278671337 conn(0x7f1a5009c560 msgr2=0x7f1a5006c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 shutdown_connections 2024-01-16T15:02:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.182+0000 7f1a55a00640 1 -- 172.21.15.125:0/4278671337 wait complete. 2024-01-16T15:02:11.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:11.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.401+0000 7fa3a7a00640 1 Processor -- start 2024-01-16T15:02:11.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.401+0000 7fa3a7a00640 1 -- start start 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.402+0000 7fa3a7a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 0x7fa3a015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa3a7a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3a013e360 0x7fa3a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa3a7a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a006c470 con 0x7fa3a015f9c0 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa3a7a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a006c750 con 0x7fa3a013e360 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa3a7a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a006ca30 con 0x7fa3a013df80 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3a013e360 0x7fa3a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3a013e360 0x7fa3a00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50860/0 (socket says 172.21.15.125:50860) 2024-01-16T15:02:11.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 -- 172.21.15.125:0/2055345492 learned_addr learned my addr 172.21.15.125:0/2055345492 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa3a5000640 1 --2- 172.21.15.125:0/2055345492 >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 0x7fa3a015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 -- 172.21.15.125:0/2055345492 >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 msgr2=0x7fa3a015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 --2- 172.21.15.125:0/2055345492 >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 0x7fa3a015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.403+0000 7fa39fe00640 1 -- 172.21.15.125:0/2055345492 >> v1:172.21.15.125:6789/0 conn(0x7fa3a015f9c0 legacy=0x7fa3a01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.404+0000 7fa39fe00640 1 -- 172.21.15.125:0/2055345492 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a006cd10 con 0x7fa3a013e360 2024-01-16T15:02:11.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.404+0000 7fa3a5000640 1 --2- 172.21.15.125:0/2055345492 >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 0x7fa3a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa39fe00640 1 --2- 172.21.15.125:0/2055345492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a013e360 0x7fa3a00cf290 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0072770 tx=0x7fa38802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a094d9ac166ca1a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa39f400640 1 -- 172.21.15.125:0/2055345492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388030860 con 0x7fa3a013e360 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa39f400640 1 -- 172.21.15.125:0/2055345492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3880309c0 con 0x7fa3a013e360 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa39f400640 1 -- 172.21.15.125:0/2055345492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388030d50 con 0x7fa3a013e360 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa3a7a00640 1 -- 172.21.15.125:0/2055345492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a013e360 msgr2=0x7fa3a00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.405+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/2055345492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a013e360 0x7fa3a00cf290 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0072770 tx=0x7fa38802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 -- 172.21.15.125:0/2055345492 shutdown_connections 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/2055345492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a013e360 0x7fa3a00cf290 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/2055345492 >> v2:172.21.15.137:3300/0 conn(0x7fa3a013df80 0x7fa3a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 -- 172.21.15.125:0/2055345492 >> 172.21.15.125:0/2055345492 conn(0x7fa3a006a800 msgr2=0x7fa3a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 -- 172.21.15.125:0/2055345492 shutdown_connections 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.406+0000 7fa3a7a00640 1 -- 172.21.15.125:0/2055345492 wait complete. 2024-01-16T15:02:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.407+0000 7fa3a7a00640 1 Processor -- start 2024-01-16T15:02:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.407+0000 7fa3a7a00640 1 -- start start 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a013df80 0x7fa3a0156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a013e360 0x7fa3a015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a015ab90 con 0x7fa3a015f9c0 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0157450 con 0x7fa3a013e360 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.408+0000 7fa3a7a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0157730 con 0x7fa3a013df80 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50872/0 (socket says 172.21.15.125:50872) 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 -- 172.21.15.125:0/844198479 learned_addr learned my addr 172.21.15.125:0/844198479 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa39fe00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a013e360 0x7fa3a015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5000640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a013df80 0x7fa3a0156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 -- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a013df80 msgr2=0x7fa3a0156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a013df80 0x7fa3a0156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 -- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a013e360 msgr2=0x7fa3a015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a013e360 0x7fa3a015a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 -- 172.21.15.125:0/844198479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880094d0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.409+0000 7fa3a5a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa39800c730 tx=0x7fa39800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.410+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398003b70 con 0x7fa3a015f9c0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.410+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398003cd0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.410+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0157a10 con 0x7fa3a015f9c0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.410+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39800aa90 con 0x7fa3a015f9c0 2024-01-16T15:02:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.410+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a0157fd0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.411+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a00675d0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.412+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa39800abf0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.413+0000 7fa39cc00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 0x7fa37806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.413+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa39808a4a0 con 0x7fa3a015f9c0 2024-01-16T15:02:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.413+0000 7fa3a5000640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 0x7fa37806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.414+0000 7fa3a5000640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 0x7fa37806dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa39000b840 tx=0x7fa390008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.417+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa398054a60 con 0x7fa3a015f9c0 2024-01-16T15:02:11.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.643+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3a006df00 con 0x7fa3a015f9c0 2024-01-16T15:02:11.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.644+0000 7fa39cc00640 1 -- 172.21.15.125:0/844198479 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fa398058080 con 0x7fa3a015f9c0 2024-01-16T15:02:11.625 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:11.625 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2646664931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2646664931},{"type":"v1","addr":"172.21.15.125:6837","nonce":2646664931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5054,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1721910358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1721910358},{"type":"v1","addr":"172.21.15.137:6835","nonce":1721910358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5362,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/28677566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":28677566},{"type":"v1","addr":"172.21.15.137:6833","nonce":28677566}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:05.115789+0000","modified":"2024-01-16T15:02:08.191947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6835/2216643169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2216643169},{"type":"v1","addr":"172.21.15.125:6835","nonce":2216643169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:02:11.625 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 37 2024-01-16T15:02:11.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 msgr2=0x7fa37806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 0x7fa37806dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa39000b840 tx=0x7fa390008040 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 msgr2=0x7fa3a01f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa39800c730 tx=0x7fa39800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 shutdown_connections 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa37806b860 0x7fa37806dd20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3a015f9c0 0x7fa3a01f0200 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3a013e360 0x7fa3a015a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 --2- 172.21.15.125:0/844198479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3a013df80 0x7fa3a0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 >> 172.21.15.125:0/844198479 conn(0x7fa3a006a800 msgr2=0x7fa3a00b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 shutdown_connections 2024-01-16T15:02:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:11.647+0000 7fa3a7a00640 1 -- 172.21.15.125:0/844198479 wait complete. 2024-01-16T15:02:12.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:02:05.115789+0000', 'modified': '2024-01-16T15:02:08.191947+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6366}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6366': {'gid': 6366, 'name': 'c', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6835/2216643169', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6834', 'nonce': 2216643169}, {'type': 'v1', 'addr': '172.21.15.125:6835', 'nonce': 2216643169}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:02:12.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:02:12.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:02:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.290+0000 7f3779200640 1 Processor -- start 2024-01-16T15:02:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.290+0000 7f3779200640 1 -- start start 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3779200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37740ae5a0 0x7f377415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3779200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f37740b3ae0 0x7f377406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3779200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3774070c40 con 0x7f377406d120 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3779200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3774071450 con 0x7f37740ae5a0 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3779200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377406c470 con 0x7f37740b3ae0 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3771e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37740ae5a0 0x7f377415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.292+0000 7f3771e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37740ae5a0 0x7f377415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50882/0 (socket says 172.21.15.125:50882) 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.293+0000 7f3771e00640 1 -- 172.21.15.125:0/1108330059 learned_addr learned my addr 172.21.15.125:0/1108330059 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.293+0000 7f3771e00640 1 -- 172.21.15.125:0/1108330059 >> v2:172.21.15.137:3300/0 conn(0x7f37740b3ae0 msgr2=0x7f377406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.293+0000 7f3771e00640 1 --2- 172.21.15.125:0/1108330059 >> v2:172.21.15.137:3300/0 conn(0x7f37740b3ae0 0x7f377406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.293+0000 7f3771e00640 1 -- 172.21.15.125:0/1108330059 >> v1:172.21.15.137:6789/0 conn(0x7f377406d120 legacy=0x7f37740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.293+0000 7f3771e00640 1 -- 172.21.15.125:0/1108330059 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377406c750 con 0x7f37740ae5a0 2024-01-16T15:02:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.294+0000 7f3771e00640 1 --2- 172.21.15.125:0/1108330059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740ae5a0 0x7f377415e430 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f37740cfea0 tx=0x7f376402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32163e505c51eb72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3770a00640 1 -- 172.21.15.125:0/1108330059 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764031870 con 0x7f37740ae5a0 2024-01-16T15:02:12.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3770a00640 1 -- 172.21.15.125:0/1108330059 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37640319d0 con 0x7f37740ae5a0 2024-01-16T15:02:12.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3770a00640 1 -- 172.21.15.125:0/1108330059 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764031d60 con 0x7f37740ae5a0 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 -- 172.21.15.125:0/1108330059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740ae5a0 msgr2=0x7f377415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 --2- 172.21.15.125:0/1108330059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740ae5a0 0x7f377415e430 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f37740cfea0 tx=0x7f376402b910 comp rx=0 tx=0).stop 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 -- 172.21.15.125:0/1108330059 shutdown_connections 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 --2- 172.21.15.125:0/1108330059 >> v2:172.21.15.137:3300/0 conn(0x7f37740b3ae0 0x7f377406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 --2- 172.21.15.125:0/1108330059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740ae5a0 0x7f377415e430 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.295+0000 7f3779200640 1 -- 172.21.15.125:0/1108330059 >> 172.21.15.125:0/1108330059 conn(0x7f377406a800 msgr2=0x7f377406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.296+0000 7f3779200640 1 -- 172.21.15.125:0/1108330059 shutdown_connections 2024-01-16T15:02:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.296+0000 7f3779200640 1 -- 172.21.15.125:0/1108330059 wait complete. 2024-01-16T15:02:12.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.297+0000 7f3779200640 1 Processor -- start 2024-01-16T15:02:12.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.297+0000 7f3779200640 1 -- start start 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 0x7f37740cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 0x7f377414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37741f70b0 con 0x7f37740b3ae0 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37740cb940 con 0x7f377406d120 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.298+0000 7f3779200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37740cbc20 con 0x7f37740ae5a0 2024-01-16T15:02:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50888/0 (socket says 172.21.15.125:50888) 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 -- 172.21.15.125:0/2233163804 learned_addr learned my addr 172.21.15.125:0/2233163804 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3771e00640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 0x7f37740cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3771400640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 0x7f377414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 -- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 msgr2=0x7f377414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 0x7f377414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 -- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 msgr2=0x7f37740cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 0x7f37740cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 -- 172.21.15.125:0/2233163804 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640094d0 con 0x7f37740b3ae0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3771e00640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 0x7f37740cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3771400640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 0x7f377414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.299+0000 7f3772800640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c390 tx=0x7f376c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.300+0000 7f376a000640 1 -- 172.21.15.125:0/2233163804 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c0035f0 con 0x7f37740b3ae0 2024-01-16T15:02:12.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.300+0000 7f376a000640 1 -- 172.21.15.125:0/2233163804 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c003750 con 0x7f37740b3ae0 2024-01-16T15:02:12.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.300+0000 7f376a000640 1 -- 172.21.15.125:0/2233163804 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c003b00 con 0x7f37740b3ae0 2024-01-16T15:02:12.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.300+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37740cbf00 con 0x7f37740b3ae0 2024-01-16T15:02:12.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.300+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37740ccc40 con 0x7f37740b3ae0 2024-01-16T15:02:12.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.302+0000 7f376a000640 1 -- 172.21.15.125:0/2233163804 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f376c003c60 con 0x7f37740b3ae0 2024-01-16T15:02:12.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.303+0000 7f376a000640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 0x7f374c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.303+0000 7f376a000640 1 -- 172.21.15.125:0/2233163804 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f376c0898f0 con 0x7f37740b3ae0 2024-01-16T15:02:12.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.303+0000 7f3771e00640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 0x7f374c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.303+0000 7f377c009740 1 Processor -- start 2024-01-16T15:02:12.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 -- start start 2024-01-16T15:02:12.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f3771e00640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 0x7f374c06dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3764036000 tx=0x7f376403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695ce6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d29560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695c44880 con 0x55f695ce0ba0 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695c53ae0 con 0x55f695ce55e0 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.304+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f695c511d0 con 0x55f695ce5ea0 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50890/0 (socket says 172.21.15.125:50890) 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 -- 172.21.15.125:0/3741156680 learned_addr learned my addr 172.21.15.125:0/3741156680 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 -- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 msgr2=0x55f695ce6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695ce6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 -- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 msgr2=0x55f695d29560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d29560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 -- 172.21.15.125:0/3741156680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f695c14860 con 0x55f695ce0ba0 2024-01-16T15:02:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.305+0000 7f3771400640 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f375c007fb0 tx=0x7f375c00c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfe3c4f105c36fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f3762a00640 1 -- 172.21.15.125:0/3741156680 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c00f850 con 0x55f695ce0ba0 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f3762a00640 1 -- 172.21.15.125:0/3741156680 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c00f9b0 con 0x55f695ce0ba0 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f3762a00640 1 -- 172.21.15.125:0/3741156680 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c00fd60 con 0x55f695ce0ba0 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 -- 172.21.15.125:0/3741156680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 msgr2=0x55f695ce50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f375c007fb0 tx=0x7f375c00c750 comp rx=0 tx=0).stop 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 -- 172.21.15.125:0/3741156680 shutdown_connections 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d29560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce0ba0 0x55f695ce50a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 --2- 172.21.15.125:0/3741156680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695ce6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 -- 172.21.15.125:0/3741156680 >> 172.21.15.125:0/3741156680 conn(0x55f695ca7c70 msgr2=0x55f695d2b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 -- 172.21.15.125:0/3741156680 shutdown_connections 2024-01-16T15:02:12.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.306+0000 7f377c009740 1 -- 172.21.15.125:0/3741156680 wait complete. 2024-01-16T15:02:12.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.307+0000 7f377c009740 1 -- start start 2024-01-16T15:02:12.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.307+0000 7f377c009740 1 Processor -- start 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d32e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695d0eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695c4cac0 con 0x55f695ce55e0 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695be92f0 con 0x55f695ce0ba0 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f695c06c70 con 0x55f695ce5ea0 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f377c009740 10 client.0 _ll_register_callbacks cb 0x55f695c0b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.308+0000 7f3771400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50902/0 (socket says 172.21.15.125:50902) 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 -- 172.21.15.125:0/3282768594 learned_addr learned my addr 172.21.15.125:0/3282768594 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771e00640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d32e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 -- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 msgr2=0x55f695d0eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3772800640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695d0eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695d0eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 -- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 msgr2=0x55f695d32e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d32e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 -- 172.21.15.125:0/3282768594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c007b80 con 0x55f695ce55e0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771e00640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d32e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f3771400640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f375c007f80 tx=0x7f375c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c937dd55c0857d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.309+0000 7f375b400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c019860 con 0x55f695ce55e0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f695c45aa0 con 0x55f695ce55e0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f375c0199c0 con 0x55f695ce55e0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f375c00fe50 con 0x55f695ce55e0 2024-01-16T15:02:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 10 client.6510 fetch_fsmap learned FSMap version 37 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55f695c4b140 con 0x55f695ce55e0 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f695c45aa0 con 0x55f695ce55e0 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f375c022070 con 0x55f695ce55e0 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 10 client.6510 fetch_fsmap finished waiting for FSMap version 37 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 10 client.6510 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 20 client.6510 populate_metadata read hostname 'smithi125' 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695cf6930 0x55f695d0bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 4 client.6510 mds_command: new command op to 6366 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.310+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 --> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f695c457c0 con 0x55f695cf6930 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.311+0000 7f3771e00640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695cf6930 0x55f695d0bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.311+0000 7f3771e00640 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695cf6930 0x55f695d0bb30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d4a33b9d0262a9 server_cookie=6614741179503283 in_seq=0 out_seq=0 2024-01-16T15:02:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.311+0000 7f375b400640 10 client.6510 ms_handle_connect on v2:172.21.15.125:6834/2216643169 2024-01-16T15:02:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f375b400640 1 -- 172.21.15.125:0/3282768594 <== mds.0 v2:172.21.15.125:6834/2216643169 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f695c457c0 con 0x55f695cf6930 2024-01-16T15:02:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f375b400640 10 client.6510 handle_command_reply: tid=0 2024-01-16T15:02:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 client.6510 shutdown 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 msgr2=0x55f695d0f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f375c007f80 tx=0x7f375c004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 shutdown_connections 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695cf6930 0x55f695d0bb30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce5ea0 0x55f695d0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce55e0 0x55f695d0f870 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f375b400640 0 client.6510 ms_handle_reset on v2:172.21.15.125:6834/2216643169 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 --2- 172.21.15.125:0/3282768594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d32e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.312+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 >> 172.21.15.125:0/3282768594 conn(0x55f695ca7c70 msgr2=0x55f695d28b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.313+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 shutdown_connections 2024-01-16T15:02:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.313+0000 7f377c009740 1 -- 172.21.15.125:0/3282768594 wait complete. 2024-01-16T15:02:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.313+0000 7f377c009740 20 client.6510 trim_cache size 0 max 16384 2024-01-16T15:02:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.329+0000 7f377c009740 1 Processor -- start 2024-01-16T15:02:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.329+0000 7f377c009740 1 -- start start 2024-01-16T15:02:12.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce0ba0 0x55f695d32e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d13e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695ce1910 con 0x55f695ce5ea0 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6959aff20 con 0x55f695ce55e0 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f695d2c2e0 con 0x55f695ce0ba0 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50906/0 (socket says 172.21.15.125:50906) 2024-01-16T15:02:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.330+0000 7f3772800640 1 -- 172.21.15.125:0/3923509294 learned_addr learned my addr 172.21.15.125:0/3923509294 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 -- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce0ba0 msgr2=0x55f695d32e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce0ba0 0x55f695d32e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 -- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 msgr2=0x55f695d13e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d13e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 -- 172.21.15.125:0/3923509294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f695be8b80 con 0x55f695ce5ea0 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3772800640 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f376c054070 tx=0x7f376c0574d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2684ed3627cfde6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3763400640 1 -- 172.21.15.125:0/3923509294 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c058ea0 con 0x55f695ce5ea0 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3763400640 1 -- 172.21.15.125:0/3923509294 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c057b70 con 0x55f695ce5ea0 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.331+0000 7f3763400640 1 -- 172.21.15.125:0/3923509294 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c087440 con 0x55f695ce5ea0 2024-01-16T15:02:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- 172.21.15.125:0/3923509294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 msgr2=0x55f695d4f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f376c054070 tx=0x7f376c0574d0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- 172.21.15.125:0/3923509294 shutdown_connections 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695d4f420 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce55e0 0x55f695d13e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 --2- 172.21.15.125:0/3923509294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce0ba0 0x55f695d32e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- 172.21.15.125:0/3923509294 >> 172.21.15.125:0/3923509294 conn(0x55f695ca7c70 msgr2=0x55f695cdc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- 172.21.15.125:0/3923509294 shutdown_connections 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- 172.21.15.125:0/3923509294 wait complete. 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.332+0000 7f377c009740 1 -- start start 2024-01-16T15:02:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.333+0000 7f377c009740 1 Processor -- start 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d95a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce55e0 0x55f695cdfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695d4ba80 con 0x55f695ce5ea0 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f695d4e1e0 con 0x55f695ce0ba0 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f695d0f2a0 con 0x55f695ce55e0 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f377c009740 10 client.0 _ll_register_callbacks cb 0x55f695ce5530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50922/0 (socket says 172.21.15.125:50922) 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 -- 172.21.15.125:0/2064293107 learned_addr learned my addr 172.21.15.125:0/2064293107 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 -- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce55e0 msgr2=0x55f695cdfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce55e0 0x55f695cdfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 -- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 msgr2=0x55f695d95a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d95a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 -- 172.21.15.125:0/2064293107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37740ccc40 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f3772800640 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c360 tx=0x7f376c014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5934ce6bcf1ebc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.334+0000 7f375b400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c05bcc0 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f376c05fe70 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f695d0b1e0 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f376c06f170 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 10 client.6516 fetch_fsmap learned FSMap version 37 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55f695c0e170 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f695d0b1e0 con 0x55f695ce5ea0 2024-01-16T15:02:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f376c06f2d0 con 0x55f695ce5ea0 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 10 client.6516 fetch_fsmap finished waiting for FSMap version 37 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 10 client.6516 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 20 client.6516 populate_metadata read hostname 'smithi125' 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695d4d270 0x55f695d4d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 4 client.6516 mds_command: new command op to 6366 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.335+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 --> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f375c00f9b0 con 0x55f695d4d270 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f3771e00640 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695d4d270 0x55f695d4d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f3771e00640 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695d4d270 0x55f695d4d650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d52a2f383467716 server_cookie=4cc1a717c87ca89f in_seq=0 out_seq=0 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f375b400640 10 client.6516 ms_handle_connect on v2:172.21.15.125:6834/2216643169 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f375b400640 1 -- 172.21.15.125:0/2064293107 <== mds.0 v2:172.21.15.125:6834/2216643169 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f375c00f9b0 con 0x55f695d4d270 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f375b400640 10 client.6516 handle_command_reply: tid=0 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 client.6516 shutdown 2024-01-16T15:02:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 msgr2=0x55f695ce00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c360 tx=0x7f376c014070 comp rx=0 tx=0).stop 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 shutdown_connections 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:6834/2216643169,v1:172.21.15.125:6835/2216643169] conn(0x55f695d4d270 0x55f695d4d650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55f695ce5ea0 0x55f695ce00c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55f695ce55e0 0x55f695cdfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 --2- 172.21.15.125:0/2064293107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55f695ce0ba0 0x55f695d95a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f375b400640 0 client.6516 ms_handle_reset on v2:172.21.15.125:6834/2216643169 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.336+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 >> 172.21.15.125:0/2064293107 conn(0x55f695ca7c70 msgr2=0x55f695ce6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 shutdown_connections 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f377c009740 1 -- 172.21.15.125:0/2064293107 wait complete. 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f377c009740 20 client.6516 trim_cache size 0 max 16384 2024-01-16T15:02:12.321 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 6366, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.6648188599999996, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 37, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 58, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 58, 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 12.101433352000001 2024-01-16T15:02:12.322 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 msgr2=0x7f374c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 0x7f374c06dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3764036000 tx=0x7f376403c040 comp rx=0 tx=0).stop 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 msgr2=0x7f37740ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c390 tx=0x7f376c00c850 comp rx=0 tx=0).stop 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 shutdown_connections 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f374c06b860 0x7f374c06dd20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37740b3ae0 0x7f37740ce120 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37740ae5a0 0x7f377414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 --2- 172.21.15.125:0/2233163804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f377406d120 0x7f37740cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.337+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 >> 172.21.15.125:0/2233163804 conn(0x7f377406a800 msgr2=0x7f37740cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.338+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 shutdown_connections 2024-01-16T15:02:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.338+0000 7f3779200640 1 -- 172.21.15.125:0/2233163804 wait complete. 2024-01-16T15:02:12.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:12.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.561+0000 7f7e61800640 1 Processor -- start 2024-01-16T15:02:12.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.561+0000 7f7e61800640 1 -- start start 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.562+0000 7f7e61800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e5c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.562+0000 7f7e61800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c15dba0 con 0x7f7e5c146650 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.562+0000 7f7e61800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c0cd130 con 0x7f7e5c0726f0 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.562+0000 7f7e61800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c06f800 con 0x7f7e5c15e740 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7e5c146650 0x7f7e5c0cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60784/0 (socket says 172.21.15.125:60784) 2024-01-16T15:02:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5be00640 1 -- 172.21.15.125:0/1780097864 learned_addr learned my addr 172.21.15.125:0/1780097864 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:12.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e60e00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e5c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380323991 0 0) 0x7f7e5c15dba0 con 0x7f7e5c146650 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c003520 con 0x7f7e5c146650 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812769142 0 0) 0x7f7e3c003520 con 0x7f7e5c146650 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 msgr2=0x7f7e5c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e5c0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 >> v1:172.21.15.137:6789/0 conn(0x7f7e5c0726f0 legacy=0x7f7e5c15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c06f240 con 0x7f7e5c146650 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.563+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7e4c001670 con 0x7f7e5c146650 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 >> v1:172.21.15.125:6789/0 conn(0x7f7e5c146650 legacy=0x7f7e5c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e3c0039a0 0x7f7e3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c06f800 con 0x7f7e3c0047c0 2024-01-16T15:02:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c00ba60 con 0x7f7e3c0039a0 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c00bc20 con 0x7f7e5c15e740 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5be00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e3c0039a0 0x7f7e3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 -- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 msgr2=0x7f7e3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e60e00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15e740 0x7f7e3c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 -- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e3c0039a0 msgr2=0x7f7e3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e3c0039a0 0x7f7e3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.564+0000 7f7e5b400640 1 -- 172.21.15.125:0/1780097864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c003520 con 0x7f7e3c0047c0 2024-01-16T15:02:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.565+0000 7f7e5be00640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e3c0039a0 0x7f7e3c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.566+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c072af0 tx=0x7f7e3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627d864d3403e687 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.566+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440052d0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.566+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4402e050 con 0x7f7e3c0047c0 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.566+0000 7f7e5aa00640 1 -- 172.21.15.125:0/1780097864 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440035f0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.566+0000 7f7e61800640 1 -- 172.21.15.125:0/1780097864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 msgr2=0x7f7e3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e61800640 1 --2- 172.21.15.125:0/1780097864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c072af0 tx=0x7f7e3c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e60e00640 1 -- 172.21.15.125:0/1780097864 reap_dead start 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e61800640 1 -- 172.21.15.125:0/1780097864 shutdown_connections 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e61800640 1 -- 172.21.15.125:0/1780097864 >> 172.21.15.125:0/1780097864 conn(0x7f7e5c0b1d40 msgr2=0x7f7e5c0b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e61800640 1 -- 172.21.15.125:0/1780097864 shutdown_connections 2024-01-16T15:02:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.567+0000 7f7e61800640 1 -- 172.21.15.125:0/1780097864 wait complete. 2024-01-16T15:02:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.568+0000 7f7e61800640 1 Processor -- start 2024-01-16T15:02:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.569+0000 7f7e61800640 1 -- start start 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15cc20 0x7f7e5c13c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5c13ca60 0x7f7e5c13cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0ae960 con 0x7f7e3c0047c0 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0aec40 con 0x7f7e5c13ca60 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e61800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0aef20 con 0x7f7e5c15cc20 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50928/0 (socket says 172.21.15.125:50928) 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 -- 172.21.15.125:0/1095983780 learned_addr learned my addr 172.21.15.125:0/1095983780 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 -- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15cc20 msgr2=0x7f7e5c13c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15cc20 0x7f7e5c13c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 -- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5c13ca60 msgr2=0x7f7e5c13cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5c13ca60 0x7f7e5c13cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.570+0000 7f7e5be00640 1 -- 172.21.15.125:0/1095983780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0af200 con 0x7f7e3c0047c0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e5be00640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00d810 tx=0x7f7e4c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0074e0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c007640 con 0x7f7e3c0047c0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0049e0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c0af4e0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.571+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c15d0a0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.572+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c067b60 con 0x7f7e3c0047c0 2024-01-16T15:02:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.573+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7e4c004b40 con 0x7f7e3c0047c0 2024-01-16T15:02:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.574+0000 7f7e53e00640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 0x7f7e3406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.574+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4c08a040 con 0x7f7e3c0047c0 2024-01-16T15:02:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.574+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 0x7f7e3406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.575+0000 7f7e5b400640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 0x7f7e3406dd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e44033000 tx=0x7f7e4402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.578+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e4c054530 con 0x7f7e3c0047c0 2024-01-16T15:02:12.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.803+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7e5c06b0f0 con 0x7f7e3c0047c0 2024-01-16T15:02:12.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.804+0000 7f7e53e00640 1 -- 172.21.15.125:0/1095983780 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7e4c019020 con 0x7f7e3c0047c0 2024-01-16T15:02:12.784 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:12.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 msgr2=0x7f7e3406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 0x7f7e3406dd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e44033000 tx=0x7f7e4402f000 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 msgr2=0x7f7e5c0ae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00d810 tx=0x7f7e4c00dcd0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 shutdown_connections 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5c13ca60 0x7f7e5c13cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e5c15cc20 0x7f7e5c13c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e3406b860 0x7f7e3406dd20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 --2- 172.21.15.125:0/1095983780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3c0047c0 0x7f7e5c0ae2f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 >> 172.21.15.125:0/1095983780 conn(0x7f7e5c0b1d40 msgr2=0x7f7e5c0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 shutdown_connections 2024-01-16T15:02:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:12.807+0000 7f7e61800640 1 -- 172.21.15.125:0/1095983780 wait complete. 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:12.807 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:12.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:12.808 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42316sec preferred_lft 42316sec 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:02:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:02:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:12.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:12.864 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:12.944 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:02:12.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:12.948 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:02:13.000 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:02:13.001 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:13.001 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:02:13.001 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:02:13.001 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:13.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:13.001 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:02:13.001 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:02:13.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:02:13.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:02:13.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:02:13.002 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:02:13.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:13.170 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:02:13.222 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:02:13.222 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:02:13.223 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:13.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:13.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:02:13.317 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:02:13.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:13.317 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:13.369 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:13.369 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:13.369 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:13.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:02:13.518 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:02:13.572 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:13.573 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:13.575 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:13.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:13.575 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:02:13.624 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:13.625 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:02:13.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:02:13.707 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:02:13.751 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:02:13.752 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:13.753 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:13.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:13.754 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:02:13.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:02:13.834+0000 7fa00be52480 -1 init, newargv = 0x560a723f2970 newargc=15 2024-01-16T15:02:13.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[58036]: starting ceph client 2024-01-16T15:02:13.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[58036]: starting fuse 2024-01-16T15:02:13.838 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:02:13.839 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:02:13.839 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:02:13.840 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:02:14.042 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.58036.asok 2024-01-16T15:02:14.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:14.060 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58036.asok 2024-01-16T15:02:14.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:14.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58036.asok status 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "58036", 2024-01-16T15:02:14.295 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 6525, 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 6525 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1997000057 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:14.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1997000057 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.6525 192.168.0.1:0/1997000057", 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/1997000057", 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 37, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 58, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:02:14.297 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:02:14.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:14.310 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:02:14.310 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:14.364 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:02:14.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:14.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:14.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:14.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:14.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.688+0000 7f3125200640 1 Processor -- start 2024-01-16T15:02:14.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.688+0000 7f3125200640 1 -- start start 2024-01-16T15:02:14.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.689+0000 7f3125200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f312013df80 0x7f312015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.689+0000 7f3125200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013e360 0x7f31200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.689+0000 7f3125200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 0x7f31200cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f3125200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312006c470 con 0x7f312013df80 2024-01-16T15:02:14.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f3125200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312006c750 con 0x7f312015f9c0 2024-01-16T15:02:14.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f3125200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312006ca30 con 0x7f312013e360 2024-01-16T15:02:14.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f311de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f312013df80 0x7f312015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f311de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f312013df80 0x7f312015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50932/0 (socket says 172.21.15.125:50932) 2024-01-16T15:02:14.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f311de00640 1 -- 172.21.15.125:0/4016709433 learned_addr learned my addr 172.21.15.125:0/4016709433 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f311e800640 1 --2- 172.21.15.125:0/4016709433 >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 0x7f31200cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.690+0000 7f3117e00640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013e360 0x7f31200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311de00640 1 -- 172.21.15.125:0/4016709433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013e360 msgr2=0x7f31200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311de00640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013e360 0x7f31200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311de00640 1 -- 172.21.15.125:0/4016709433 >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 msgr2=0x7f31200cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311de00640 1 --2- 172.21.15.125:0/4016709433 >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 0x7f31200cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311de00640 1 -- 172.21.15.125:0/4016709433 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312006cd10 con 0x7f312013df80 2024-01-16T15:02:14.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.691+0000 7f311e800640 1 --2- 172.21.15.125:0/4016709433 >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 0x7f31200cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.692+0000 7f311de00640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013df80 0x7f312015bee0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f312006e6d0 tx=0x7f311002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b73549ff3cf876 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.692+0000 7f311d400640 1 -- 172.21.15.125:0/4016709433 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110031860 con 0x7f312013df80 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.692+0000 7f311d400640 1 -- 172.21.15.125:0/4016709433 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31100319c0 con 0x7f312013df80 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f311d400640 1 -- 172.21.15.125:0/4016709433 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110031d50 con 0x7f312013df80 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 -- 172.21.15.125:0/4016709433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013df80 msgr2=0x7f312015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013df80 0x7f312015bee0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f312006e6d0 tx=0x7f311002b900 comp rx=0 tx=0).stop 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 -- 172.21.15.125:0/4016709433 shutdown_connections 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 --2- 172.21.15.125:0/4016709433 >> v2:172.21.15.137:3300/0 conn(0x7f312015f9c0 0x7f31200cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013e360 0x7f31200cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 --2- 172.21.15.125:0/4016709433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013df80 0x7f312015bee0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 -- 172.21.15.125:0/4016709433 >> 172.21.15.125:0/4016709433 conn(0x7f312006a800 msgr2=0x7f312006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 -- 172.21.15.125:0/4016709433 shutdown_connections 2024-01-16T15:02:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.693+0000 7f3125200640 1 -- 172.21.15.125:0/4016709433 wait complete. 2024-01-16T15:02:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.695+0000 7f3125200640 1 Processor -- start 2024-01-16T15:02:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.695+0000 7f3125200640 1 -- start start 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013df80 0x7f312013c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 0x7f31200ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200aa830 con 0x7f312013e360 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200a9bd0 con 0x7f312015f9c0 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3125200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200a9e90 con 0x7f312013df80 2024-01-16T15:02:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3117e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3117e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50940/0 (socket says 172.21.15.125:50940) 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.696+0000 7f3117e00640 1 -- 172.21.15.125:0/2043791118 learned_addr learned my addr 172.21.15.125:0/2043791118 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f311e800640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 0x7f31200ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f311de00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013df80 0x7f312013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 -- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013df80 msgr2=0x7f312013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013df80 0x7f312013c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 -- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 msgr2=0x7f31200ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 0x7f31200ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 -- 172.21.15.125:0/2043791118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100094d0 con 0x7f312013e360 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f311e800640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 0x7f31200ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3117e00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f310800d800 tx=0x7f310800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800b6c0 con 0x7f312013e360 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310800b820 con 0x7f312013e360 2024-01-16T15:02:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200aa170 con 0x7f312013e360 2024-01-16T15:02:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.697+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800bbb0 con 0x7f312013e360 2024-01-16T15:02:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.698+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312013cc60 con 0x7f312013e360 2024-01-16T15:02:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.699+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312015b130 con 0x7f312013e360 2024-01-16T15:02:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.700+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f310800bd10 con 0x7f312013e360 2024-01-16T15:02:14.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.700+0000 7f3116000640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 0x7f30f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.701+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f310808ab10 con 0x7f312013e360 2024-01-16T15:02:14.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.701+0000 7f311de00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 0x7f30f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.701+0000 7f311de00640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 0x7f30f806dd20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3110036000 tx=0x7f311003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.705+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31080551a0 con 0x7f312013e360 2024-01-16T15:02:14.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.932+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f31200675d0 con 0x7f312013e360 2024-01-16T15:02:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.933+0000 7f3116000640 1 -- 172.21.15.125:0/2043791118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7f31080587c0 con 0x7f312013e360 2024-01-16T15:02:14.912 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.935+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 msgr2=0x7f30f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.935+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 0x7f30f806dd20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3110036000 tx=0x7f311003c040 comp rx=0 tx=0).stop 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 msgr2=0x7f31200cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f310800d800 tx=0x7f310800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 shutdown_connections 2024-01-16T15:02:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f312015f9c0 0x7f31200ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f30f806b860 0x7f30f806dd20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f312013e360 0x7f31200cdfc0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 --2- 172.21.15.125:0/2043791118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f312013df80 0x7f312013c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 >> 172.21.15.125:0/2043791118 conn(0x7f312006a800 msgr2=0x7f31200aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 shutdown_connections 2024-01-16T15:02:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:14.936+0000 7f3125200640 1 -- 172.21.15.125:0/2043791118 wait complete. 2024-01-16T15:02:14.932 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:14.933 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:14.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:14.933 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:14.990 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42303sec preferred_lft 42303sec 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:14.991 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:14.992 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.992 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:02:14.992 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:14.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:14.994 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:02:15.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.090 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:15.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.141 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:02:15.142 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:02:15.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:02:15.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:02:15.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:02:15.142 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:02:15.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.315 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:02:15.360 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:02:15.360 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:02:15.361 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:15.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:02:15.497 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:02:15.497 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:15.497 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:15.550 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:15.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:15.551 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:15.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:02:15.693 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:15.751 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:15.751 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:15.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.752 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:02:15.799 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:15.800 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:02:15.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:02:15.882 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:02:15.927 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:15.928 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:15.929 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:15.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.930 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:02:15.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:02:16.017+0000 7fea0ba3c480 -1 init, newargv = 0x55c4a6498a40 newargc=15 2024-01-16T15:02:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44351]: starting ceph client 2024-01-16T15:02:16.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44351]: starting fuse 2024-01-16T15:02:16.014 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:02:16.014 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:02:16.014 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:02:16.015 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:02:16.208 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.44351.asok 2024-01-16T15:02:16.212 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:16.225 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44351.asok 2024-01-16T15:02:16.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:16.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44351.asok status 2024-01-16T15:02:16.460 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "44351", 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5105, 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:02:16.461 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5105 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 976509315 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 976509315 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:02:16.462 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5105 192.168.0.1:0/976509315", 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/976509315", 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 37, 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 58, 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:02:16.463 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:02:16.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:16.476 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:02:16.476 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:16.530 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:02:16.531 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:16.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:16.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:16.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:02:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.867+0000 7fe38cc00640 1 Processor -- start 2024-01-16T15:02:16.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.868+0000 7fe38cc00640 1 -- start start 2024-01-16T15:02:16.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.869+0000 7fe38cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe388160920 0x7fe3880b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.869+0000 7fe38cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.869+0000 7fe38cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880d0a10 con 0x7fe3880cf9f0 2024-01-16T15:02:16.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.869+0000 7fe38cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3880d0330 con 0x7fe388160920 2024-01-16T15:02:16.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.869+0000 7fe38cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880d1f50 con 0x7fe3880cfea0 2024-01-16T15:02:16.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.870+0000 7fe387e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.870+0000 7fe387e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:42310/0 (socket says 172.21.15.125:42310) 2024-01-16T15:02:16.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.870+0000 7fe387e00640 1 -- 172.21.15.125:0/3422835551 learned_addr learned my addr 172.21.15.125:0/3422835551 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:16.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.870+0000 7fe387400640 1 --2- 172.21.15.125:0/3422835551 >> v2:172.21.15.137:3300/0 conn(0x7fe388160920 0x7fe3880b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.871+0000 7fe387400640 1 -- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 msgr2=0x7fe388161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.871+0000 7fe387400640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.871+0000 7fe387400640 1 -- 172.21.15.125:0/3422835551 >> v1:172.21.15.137:6789/0 conn(0x7fe3880cf9f0 legacy=0x7fe388161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.871+0000 7fe387400640 1 -- 172.21.15.125:0/3422835551 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880cf080 con 0x7fe388160920 2024-01-16T15:02:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.871+0000 7fe387e00640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:16.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.872+0000 7fe387400640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388160920 0x7fe3880b3f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe3700097a0 tx=0x7fe37002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74bd44ce98f0e65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.873+0000 7fe386000640 1 -- 172.21.15.125:0/3422835551 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37002d040 con 0x7fe388160920 2024-01-16T15:02:16.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.873+0000 7fe386000640 1 -- 172.21.15.125:0/3422835551 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370031810 con 0x7fe388160920 2024-01-16T15:02:16.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.873+0000 7fe386000640 1 -- 172.21.15.125:0/3422835551 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370031ba0 con 0x7fe388160920 2024-01-16T15:02:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.873+0000 7fe38cc00640 1 -- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388160920 msgr2=0x7fe3880b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.873+0000 7fe38cc00640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388160920 0x7fe3880b3f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe3700097a0 tx=0x7fe37002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 -- 172.21.15.125:0/3422835551 shutdown_connections 2024-01-16T15:02:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe388161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 --2- 172.21.15.125:0/3422835551 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe388160920 0x7fe3880b3f30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 -- 172.21.15.125:0/3422835551 >> 172.21.15.125:0/3422835551 conn(0x7fe3880d3090 msgr2=0x7fe3880d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 -- 172.21.15.125:0/3422835551 shutdown_connections 2024-01-16T15:02:16.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.874+0000 7fe38cc00640 1 -- 172.21.15.125:0/3422835551 wait complete. 2024-01-16T15:02:16.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.876+0000 7fe38cc00640 1 Processor -- start 2024-01-16T15:02:16.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.876+0000 7fe38cc00640 1 -- start start 2024-01-16T15:02:16.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3880cf9f0 0x7fe3880b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe3880ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880ca6a0 con 0x7fe388160920 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880ca8d0 con 0x7fe3880cf9f0 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.877+0000 7fe38cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880cabb0 con 0x7fe3880cfea0 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47960/0 (socket says 172.21.15.125:47960) 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 -- 172.21.15.125:0/351518861 learned_addr learned my addr 172.21.15.125:0/351518861 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387400640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3880cf9f0 0x7fe3880b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe386a00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe3880ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 -- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 msgr2=0x7fe3880ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe3880ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 -- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3880cf9f0 msgr2=0x7fe3880b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3880cf9f0 0x7fe3880b67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe387e00640 1 -- 172.21.15.125:0/351518861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3800094d0 con 0x7fe388160920 2024-01-16T15:02:16.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.878+0000 7fe386a00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe3880ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe387e00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe38000e930 tx=0x7fe38000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380009720 con 0x7fe388160920 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000ca00 con 0x7fe388160920 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe370009400 con 0x7fe388160920 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000cd90 con 0x7fe388160920 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.879+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3880cb170 con 0x7fe388160920 2024-01-16T15:02:16.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.880+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe388067f30 con 0x7fe388160920 2024-01-16T15:02:16.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.881+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe380013470 con 0x7fe388160920 2024-01-16T15:02:16.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.882+0000 7fe37f400640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 0x7fe36006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.882+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe38008a220 con 0x7fe388160920 2024-01-16T15:02:16.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.882+0000 7fe387400640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 0x7fe36006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.883+0000 7fe387400640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 0x7fe36006dd50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe37003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:16.887+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3800547e0 con 0x7fe388160920 2024-01-16T15:02:17.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.121+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe388067450 con 0x7fe388160920 2024-01-16T15:02:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.197+0000 7fe37f400640 1 -- 172.21.15.125:0/351518861 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe380057e00 con 0x7fe388160920 2024-01-16T15:02:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 msgr2=0x7fe36006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 0x7fe36006dd50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe37003c040 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 msgr2=0x7fe3881513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe38000e930 tx=0x7fe38000edf0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 shutdown_connections 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe36006b890 0x7fe36006dd50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe388160920 0x7fe3881513c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3880cfea0 0x7fe3880ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 --2- 172.21.15.125:0/351518861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3880cf9f0 0x7fe3880b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 >> 172.21.15.125:0/351518861 conn(0x7fe3880d3090 msgr2=0x7fe388160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 shutdown_connections 2024-01-16T15:02:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.201+0000 7fe38cc00640 1 -- 172.21.15.125:0/351518861 wait complete. 2024-01-16T15:02:17.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:17.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.430+0000 7fe476400640 1 Processor -- start 2024-01-16T15:02:17.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.430+0000 7fe476400640 1 -- start start 2024-01-16T15:02:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.431+0000 7fe476400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.431+0000 7fe476400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe47013e360 0x7fe4700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe476400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47006c470 con 0x7fe47015f9c0 2024-01-16T15:02:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe476400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47006c750 con 0x7fe47013e360 2024-01-16T15:02:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe476400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47006ca30 con 0x7fe47013df80 2024-01-16T15:02:17.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe46f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe46f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:42332/0 (socket says 172.21.15.125:42332) 2024-01-16T15:02:17.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe46f400640 1 -- 172.21.15.125:0/775829866 learned_addr learned my addr 172.21.15.125:0/775829866 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:17.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.432+0000 7fe466800640 1 --2- 172.21.15.125:0/775829866 >> v2:172.21.15.137:3300/0 conn(0x7fe47013e360 0x7fe4700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.433+0000 7fe466800640 1 -- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 msgr2=0x7fe47015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.433+0000 7fe466800640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.433+0000 7fe466800640 1 -- 172.21.15.125:0/775829866 >> v1:172.21.15.137:6789/0 conn(0x7fe47015f9c0 legacy=0x7fe4701408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.433+0000 7fe466800640 1 -- 172.21.15.125:0/775829866 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47006cd10 con 0x7fe47013e360 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.433+0000 7fe46f400640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:17.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.434+0000 7fe466800640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013e360 0x7fe4700cf290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe4580097a0 tx=0x7fe45802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b7c8ef46f2a3716 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.435+0000 7fe46ea00640 1 -- 172.21.15.125:0/775829866 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45802d040 con 0x7fe47013e360 2024-01-16T15:02:17.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.435+0000 7fe46ea00640 1 -- 172.21.15.125:0/775829866 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458031810 con 0x7fe47013e360 2024-01-16T15:02:17.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.435+0000 7fe46ea00640 1 -- 172.21.15.125:0/775829866 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458031ba0 con 0x7fe47013e360 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 -- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013e360 msgr2=0x7fe4700cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013e360 0x7fe4700cf290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe4580097a0 tx=0x7fe45802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 -- 172.21.15.125:0/775829866 shutdown_connections 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013e360 0x7fe4700cf290 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 --2- 172.21.15.125:0/775829866 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013df80 0x7fe47015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 -- 172.21.15.125:0/775829866 >> 172.21.15.125:0/775829866 conn(0x7fe47006a800 msgr2=0x7fe47006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 -- 172.21.15.125:0/775829866 shutdown_connections 2024-01-16T15:02:17.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.436+0000 7fe476400640 1 -- 172.21.15.125:0/775829866 wait complete. 2024-01-16T15:02:17.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.438+0000 7fe476400640 1 Processor -- start 2024-01-16T15:02:17.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.438+0000 7fe476400640 1 -- start start 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013df80 0x7fe47013b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013e360 0x7fe470149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470070e70 con 0x7fe47015f9c0 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470071150 con 0x7fe47013df80 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe476400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe470071430 con 0x7fe47013e360 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.439+0000 7fe46fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47972/0 (socket says 172.21.15.125:47972) 2024-01-16T15:02:17.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 -- 172.21.15.125:0/716889149 learned_addr learned my addr 172.21.15.125:0/716889149 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:17.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46f400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013df80 0x7fe47013b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe466800640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013e360 0x7fe470149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 -- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013e360 msgr2=0x7fe470149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013e360 0x7fe470149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 -- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013df80 msgr2=0x7fe47013b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013df80 0x7fe47013b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 -- 172.21.15.125:0/716889149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600094d0 con 0x7fe47015f9c0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.440+0000 7fe46fe00640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c7a0 tx=0x7fe46800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.441+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468003b70 con 0x7fe47015f9c0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.441+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468003cf0 con 0x7fe47015f9c0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.441+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe458009400 con 0x7fe47015f9c0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.441+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468003ea0 con 0x7fe47015f9c0 2024-01-16T15:02:17.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.441+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4700719f0 con 0x7fe47015f9c0 2024-01-16T15:02:17.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.442+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4700675d0 con 0x7fe47015f9c0 2024-01-16T15:02:17.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.443+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe46800aa90 con 0x7fe47015f9c0 2024-01-16T15:02:17.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.444+0000 7fe467e00640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 0x7fe44806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.444+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe46808a400 con 0x7fe47015f9c0 2024-01-16T15:02:17.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.444+0000 7fe46f400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 0x7fe44806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.445+0000 7fe46f400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 0x7fe44806dd20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe460006ca0 tx=0x7fe460009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.448+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4680549c0 con 0x7fe47015f9c0 2024-01-16T15:02:17.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.666+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4700ce850 con 0x7fe44806b860 2024-01-16T15:02:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.668+0000 7fe467e00640 1 -- 172.21.15.125:0/716889149 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe460009940 con 0x7fe44806b860 2024-01-16T15:02:17.647 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:02:17.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:02:17.647 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:02:17.648 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:02:17.648 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:02:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 msgr2=0x7fe44806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 0x7fe44806dd20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe460006ca0 tx=0x7fe460009240 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 msgr2=0x7fe47014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c7a0 tx=0x7fe46800cc60 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 shutdown_connections 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe44806b860 0x7fe44806dd20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47015f9c0 0x7fe47014a360 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe47013e360 0x7fe470149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 --2- 172.21.15.125:0/716889149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe47013df80 0x7fe47013b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.670+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 >> 172.21.15.125:0/716889149 conn(0x7fe47006a800 msgr2=0x7fe47014ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.671+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 shutdown_connections 2024-01-16T15:02:17.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.671+0000 7fe476400640 1 -- 172.21.15.125:0/716889149 wait complete. 2024-01-16T15:02:17.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:17.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.900+0000 7f85c3000640 1 Processor -- start 2024-01-16T15:02:17.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.901+0000 7f85c3000640 1 -- start start 2024-01-16T15:02:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc06a3e0 0x7f85bc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c3000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85bc0d0b60 0x7f85bc0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c3000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc071de0 con 0x7f85bc06a7c0 2024-01-16T15:02:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c3000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0af320 con 0x7f85bc0d0b60 2024-01-16T15:02:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc072030 con 0x7f85bc06a3e0 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85b3200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85bc0d0b60 0x7f85bc0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c1000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f85bc06a7c0 0x7f85bc06f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55108/0 (socket says 172.21.15.125:55108) 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85b3200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85bc0d0b60 0x7f85bc0aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47984/0 (socket says 172.21.15.125:47984) 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.902+0000 7f85c1000640 1 -- 172.21.15.125:0/2683763312 learned_addr learned my addr 172.21.15.125:0/2683763312 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85bbe00640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc06a3e0 0x7f85bc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85b3200640 1 -- 172.21.15.125:0/2683763312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc06a3e0 msgr2=0x7f85bc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85b3200640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc06a3e0 0x7f85bc0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85b3200640 1 -- 172.21.15.125:0/2683763312 >> v1:172.21.15.125:6789/0 conn(0x7f85bc06a7c0 legacy=0x7f85bc06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85b3200640 1 -- 172.21.15.125:0/2683763312 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc072310 con 0x7f85bc0d0b60 2024-01-16T15:02:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.903+0000 7f85bb400640 1 -- 172.21.15.125:0/2683763312 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912382050 0 0) 0x7f85bc071de0 con 0x7f85bc06a7c0 2024-01-16T15:02:17.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.904+0000 7f85b3200640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc0d0b60 0x7f85bc0aede0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f85a40097a0 tx=0x7f85a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafbaf07b75531d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.904+0000 7f85bb400640 1 -- 172.21.15.125:0/2683763312 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4031830 con 0x7f85bc0d0b60 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.904+0000 7f85bb400640 1 -- 172.21.15.125:0/2683763312 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a4031990 con 0x7f85bc0d0b60 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.904+0000 7f85bb400640 1 -- 172.21.15.125:0/2683763312 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4031d20 con 0x7f85bc0d0b60 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 -- 172.21.15.125:0/2683763312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc0d0b60 msgr2=0x7f85bc0aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc0d0b60 0x7f85bc0aede0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f85a40097a0 tx=0x7f85a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 -- 172.21.15.125:0/2683763312 shutdown_connections 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc0d0b60 0x7f85bc0aede0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 --2- 172.21.15.125:0/2683763312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc06a3e0 0x7f85bc0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 -- 172.21.15.125:0/2683763312 >> 172.21.15.125:0/2683763312 conn(0x7f85bc0cfef0 msgr2=0x7f85bc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 -- 172.21.15.125:0/2683763312 shutdown_connections 2024-01-16T15:02:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.905+0000 7f85c3000640 1 -- 172.21.15.125:0/2683763312 wait complete. 2024-01-16T15:02:17.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.907+0000 7f85c3000640 1 Processor -- start 2024-01-16T15:02:17.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.907+0000 7f85c3000640 1 -- start start 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 0x7f85bc156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc0d0b60 0x7f85bc0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc156b70 con 0x7f85bc06a3e0 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc155ef0 con 0x7f85bc06a7c0 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1561d0 con 0x7f85bc0d0b60 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85bbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85bbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47996/0 (socket says 172.21.15.125:47996) 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85bbe00640 1 -- 172.21.15.125:0/3323786809 learned_addr learned my addr 172.21.15.125:0/3323786809 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.908+0000 7f85c1000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc0d0b60 0x7f85bc0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 -- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc0d0b60 msgr2=0x7f85bc0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc0d0b60 0x7f85bc0cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85b3200640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 0x7f85bc156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 -- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 msgr2=0x7f85bc156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 0x7f85bc156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 -- 172.21.15.125:0/3323786809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a4009400 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85b3200640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 0x7f85bc156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85bbe00640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c7f0 tx=0x7f85ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac004220 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0043a0 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc1564b0 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac007860 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.909+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc0ccb10 con 0x7f85bc06a3e0 2024-01-16T15:02:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.911+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc0ae3a0 con 0x7f85bc06a3e0 2024-01-16T15:02:17.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.912+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f85ac007bd0 con 0x7f85bc06a3e0 2024-01-16T15:02:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.913+0000 7f85b8c00640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 0x7f859406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.913+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85ac08a640 con 0x7f85bc06a3e0 2024-01-16T15:02:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.913+0000 7f85b3200640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 0x7f859406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.913+0000 7f85b3200640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 0x7f859406dd20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f85a4036000 tx=0x7f85a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:17.917+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85ac054bc0 con 0x7f85bc06a3e0 2024-01-16T15:02:18.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.110+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f85bc0675d0 con 0x7f85bc06a3e0 2024-01-16T15:02:18.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.111+0000 7f85b8c00640 1 -- 172.21.15.125:0/3323786809 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85ac0581e0 con 0x7f85bc06a3e0 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 msgr2=0x7f859406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 0x7f859406dd20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f85a4036000 tx=0x7f85a403c040 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 msgr2=0x7f85bc142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c7f0 tx=0x7f85ac00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 shutdown_connections 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85bc0d0b60 0x7f85bc0cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85bc06a7c0 0x7f85bc156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f859406b860 0x7f859406dd20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 --2- 172.21.15.125:0/3323786809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85bc06a3e0 0x7f85bc142ab0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.114+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 >> 172.21.15.125:0/3323786809 conn(0x7f85bc0cfef0 msgr2=0x7f85bc0ad510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.115+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 shutdown_connections 2024-01-16T15:02:18.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.115+0000 7f85c3000640 1 -- 172.21.15.125:0/3323786809 wait complete. 2024-01-16T15:02:18.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:02:18.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.350+0000 7f5821800640 1 Processor -- start 2024-01-16T15:02:18.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.351+0000 7f5821800640 1 -- start start 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.352+0000 7f5821800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 0x7f581c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.352+0000 7f5821800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f581c13e360 0x7f581c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.352+0000 7f5821800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c06c470 con 0x7f581c15f9c0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.352+0000 7f5821800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581c06c750 con 0x7f581c13e360 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.352+0000 7f5821800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c06ca30 con 0x7f581c13df80 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f5813e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f581c13e360 0x7f581c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f581b400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f581c15f9c0 0x7f581c1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55112/0 (socket says 172.21.15.125:55112) 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f581b400640 1 -- 172.21.15.125:0/3698062107 learned_addr learned my addr 172.21.15.125:0/3698062107 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f581aa00640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 0x7f581c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f5813e00640 1 -- 172.21.15.125:0/3698062107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 msgr2=0x7f581c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f5813e00640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 0x7f581c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f5813e00640 1 -- 172.21.15.125:0/3698062107 >> v1:172.21.15.125:6789/0 conn(0x7f581c15f9c0 legacy=0x7f581c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f5813e00640 1 -- 172.21.15.125:0/3698062107 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c06cd10 con 0x7f581c13e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f581a000640 1 -- 172.21.15.125:0/3698062107 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824583722 0 0) 0x7f581c06c470 con 0x7f581c15f9c0 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.353+0000 7f581aa00640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 0x7f581c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.354+0000 7f5813e00640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c13e360 0x7f581c0cf290 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f58040097a0 tx=0x7f580402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f23dadff95e9867 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f581a000640 1 -- 172.21.15.125:0/3698062107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580402d040 con 0x7f581c13e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f581a000640 1 -- 172.21.15.125:0/3698062107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5804031810 con 0x7f581c13e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f581a000640 1 -- 172.21.15.125:0/3698062107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5804031ba0 con 0x7f581c13e360 2024-01-16T15:02:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 -- 172.21.15.125:0/3698062107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c13e360 msgr2=0x7f581c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c13e360 0x7f581c0cf290 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f58040097a0 tx=0x7f580402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 -- 172.21.15.125:0/3698062107 shutdown_connections 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c13e360 0x7f581c0cf290 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 --2- 172.21.15.125:0/3698062107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13df80 0x7f581c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 -- 172.21.15.125:0/3698062107 >> 172.21.15.125:0/3698062107 conn(0x7f581c06a800 msgr2=0x7f581c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 -- 172.21.15.125:0/3698062107 shutdown_connections 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.355+0000 7f5821800640 1 -- 172.21.15.125:0/3698062107 wait complete. 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.357+0000 7f5821800640 1 Processor -- start 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.357+0000 7f5821800640 1 -- start start 2024-01-16T15:02:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f581c13df80 0x7f581c0709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13e360 0x7f581c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c070f30 con 0x7f581c15f9c0 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c070810 con 0x7f581c13df80 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f5821800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c13dd80 con 0x7f581c13e360 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f581b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.358+0000 7f581b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48020/0 (socket says 172.21.15.125:48020) 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 -- 172.21.15.125:0/1695899145 learned_addr learned my addr 172.21.15.125:0/1695899145 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f5813e00640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13e360 0x7f581c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581aa00640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f581c13df80 0x7f581c0709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 -- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13e360 msgr2=0x7f581c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13e360 0x7f581c13d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 -- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f581c13df80 msgr2=0x7f581c0709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f581c13df80 0x7f581c0709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 -- 172.21.15.125:0/1695899145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804009400 con 0x7f581c15f9c0 2024-01-16T15:02:18.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f581b400640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f581400c7e0 tx=0x7f581400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.359+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581400a640 con 0x7f581c15f9c0 2024-01-16T15:02:18.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.360+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581400a7c0 con 0x7f581c15f9c0 2024-01-16T15:02:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.360+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c071140 con 0x7f581c15f9c0 2024-01-16T15:02:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.360+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581400ab70 con 0x7f581c15f9c0 2024-01-16T15:02:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.360+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c071700 con 0x7f581c15f9c0 2024-01-16T15:02:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.361+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c0675d0 con 0x7f581c15f9c0 2024-01-16T15:02:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.362+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f581400acd0 con 0x7f581c15f9c0 2024-01-16T15:02:18.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.362+0000 7f5812a00640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 0x7f57f406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.363+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5814089eb0 con 0x7f581c15f9c0 2024-01-16T15:02:18.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.363+0000 7f581aa00640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 0x7f57f406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.363+0000 7f581aa00640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 0x7f57f406dd20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f581c14d900 tx=0x7f580c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.367+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5814054500 con 0x7f581c15f9c0 2024-01-16T15:02:18.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.582+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f581c0ce850 con 0x7f57f406b860 2024-01-16T15:02:18.567 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:02:18.588+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:18.567 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:02:18.588+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:18.567 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:02:18.588+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:18.567 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:02:18.588+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:18.568 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:02:18.588+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:18.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.634+0000 7f5812a00640 1 -- 172.21.15.125:0/1695899145 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f580c00c050 con 0x7f57f406b860 2024-01-16T15:02:18.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 msgr2=0x7f57f406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 0x7f57f406dd20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f581c14d900 tx=0x7f580c008040 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 msgr2=0x7f581c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f581400c7e0 tx=0x7f581400cca0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 shutdown_connections 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57f406b860 0x7f57f406dd20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f581c15f9c0 0x7f581c159f80 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f581c13e360 0x7f581c13d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 --2- 172.21.15.125:0/1695899145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f581c13df80 0x7f581c0709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 >> 172.21.15.125:0/1695899145 conn(0x7f581c06a800 msgr2=0x7f581c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 shutdown_connections 2024-01-16T15:02:18.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.637+0000 7f5821800640 1 -- 172.21.15.125:0/1695899145 wait complete. 2024-01-16T15:02:18.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.849+0000 7facfcc00640 1 Processor -- start 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.849+0000 7facfcc00640 1 -- start start 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facfcc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7facf815de80 0x7facf80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facfcc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf806c730 con 0x7facf8143810 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facfcc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80724d0 con 0x7facf8071e40 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facfcc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf8143bf0 con 0x7facf815de80 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facf7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7facf8143810 0x7facf815e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55124/0 (socket says 172.21.15.125:55124) 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facf7400640 1 -- 172.21.15.125:0/241419928 learned_addr learned my addr 172.21.15.125:0/241419928 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.851+0000 7facf7e00640 1 --2- 172.21.15.125:0/241419928 >> v2:172.21.15.125:3300/0 conn(0x7facf815de80 0x7facf80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.852+0000 7facf6a00640 1 -- 172.21.15.125:0/241419928 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108874541 0 0) 0x7facf806c730 con 0x7facf8143810 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.852+0000 7facf6a00640 1 -- 172.21.15.125:0/241419928 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd8003520 con 0x7facf8143810 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.852+0000 7facf7e00640 1 -- 172.21.15.125:0/241419928 >> v1:172.21.15.137:6789/0 conn(0x7facf8071e40 legacy=0x7facf815d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.852+0000 7facf7e00640 1 -- 172.21.15.125:0/241419928 >> v1:172.21.15.125:6789/0 conn(0x7facf8143810 legacy=0x7facf815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.852+0000 7facf7e00640 1 -- 172.21.15.125:0/241419928 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf806c9f0 con 0x7facf815de80 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.853+0000 7facf7e00640 1 --2- 172.21.15.125:0/241419928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf815de80 0x7facf80ae170 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7facf0009d90 tx=0x7facf002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba5f3b429b05bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.853+0000 7facf6a00640 1 -- 172.21.15.125:0/241419928 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf002cbd0 con 0x7facf815de80 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.853+0000 7facf6a00640 1 -- 172.21.15.125:0/241419928 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf002cd30 con 0x7facf815de80 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.853+0000 7facf6a00640 1 -- 172.21.15.125:0/241419928 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf002cea0 con 0x7facf815de80 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 -- 172.21.15.125:0/241419928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf815de80 msgr2=0x7facf80ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 --2- 172.21.15.125:0/241419928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf815de80 0x7facf80ae170 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7facf0009d90 tx=0x7facf002be20 comp rx=0 tx=0).stop 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 -- 172.21.15.125:0/241419928 shutdown_connections 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 --2- 172.21.15.125:0/241419928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf815de80 0x7facf80ae170 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 -- 172.21.15.125:0/241419928 >> 172.21.15.125:0/241419928 conn(0x7facf806a7d0 msgr2=0x7facf806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 -- 172.21.15.125:0/241419928 shutdown_connections 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.854+0000 7facfcc00640 1 -- 172.21.15.125:0/241419928 wait complete. 2024-01-16T15:02:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.856+0000 7facfcc00640 1 Processor -- start 2024-01-16T15:02:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.856+0000 7facfcc00640 1 -- start start 2024-01-16T15:02:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf8071e40 0x7facf815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf815de80 0x7facf8142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80722f0 con 0x7facf8143810 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80b00b0 con 0x7facf8071e40 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facfcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf80b1850 con 0x7facf815de80 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48056/0 (socket says 172.21.15.125:48056) 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 -- 172.21.15.125:0/1460919869 learned_addr learned my addr 172.21.15.125:0/1460919869 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 -- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf815de80 msgr2=0x7facf8142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf815de80 0x7facf8142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 -- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf8071e40 msgr2=0x7facf815cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf8071e40 0x7facf815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.857+0000 7facefe00640 1 -- 172.21.15.125:0/1460919869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf0009a70 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facefe00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7face000c480 tx=0x7face000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face00052d0 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face0005430 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0012870 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf80b0a10 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.858+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf80af9f0 con 0x7facf8143810 2024-01-16T15:02:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.859+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf8067450 con 0x7facf8143810 2024-01-16T15:02:18.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.860+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7face00129d0 con 0x7facf8143810 2024-01-16T15:02:18.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.861+0000 7facef400640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 0x7facd006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.861+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7face008a190 con 0x7facf8143810 2024-01-16T15:02:18.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.862+0000 7facf7400640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 0x7facd006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.862+0000 7facf7400640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 0x7facd006dd20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7face8001610 tx=0x7face8007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:18.866+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7face00547e0 con 0x7facf8143810 2024-01-16T15:02:19.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.082+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7facf806cfb0 con 0x7facd006b860 2024-01-16T15:02:19.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.090+0000 7facef400640 1 -- 172.21.15.125:0/1460919869 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7face80058b0 con 0x7facd006b860 2024-01-16T15:02:19.070 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42 2024-01-16T15:02:19.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 msgr2=0x7facd006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 0x7facd006dd20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7face8001610 tx=0x7face8007930 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 msgr2=0x7facf80b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7face000c480 tx=0x7face000c940 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 shutdown_connections 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf815de80 0x7facf8142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facd006b860 0x7facd006dd20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7facf8143810 0x7facf80b04d0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 --2- 172.21.15.125:0/1460919869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf8071e40 0x7facf815cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 >> 172.21.15.125:0/1460919869 conn(0x7facf806a7d0 msgr2=0x7facf8140690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 shutdown_connections 2024-01-16T15:02:19.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:19.093+0000 7facfcc00640 1 -- 172.21.15.125:0/1460919869 wait complete. 2024-01-16T15:02:19.089 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42 2024-01-16T15:02:19.089 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:02:19.090 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:19.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:19.142 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:34.012 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:02:34.013 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:02:34.013 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.7964 s, 72.6 MB/s 2024-01-16T15:02:34.014 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:02:34.014 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:34.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:34.065 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:37.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:37.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:44.992 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:02:44.992 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:02:44.992 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8232 s, 99.2 MB/s 2024-01-16T15:02:44.994 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:02:44.994 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:45.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:45.049 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:02:56.345 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:02:56.345 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2362 s, 95.6 MB/s 2024-01-16T15:02:56.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:02:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.580+0000 7fddfa800640 1 Processor -- start 2024-01-16T15:02:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.580+0000 7fddfa800640 1 -- start start 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.582+0000 7fddfa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fddf413cf10 0x7fddf413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.582+0000 7fddfa800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf40cddf0 0x7fddf40b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.582+0000 7fddfa800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40ce440 con 0x7fddf413fae0 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.582+0000 7fddfa800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf413ff90 con 0x7fddf413cf10 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.582+0000 7fddfa800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40ce260 con 0x7fddf40cddf0 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fddf413cf10 0x7fddf413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fddf413cf10 0x7fddf413f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50032/0 (socket says 172.21.15.125:50032) 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 -- 172.21.15.125:0/3632833865 learned_addr learned my addr 172.21.15.125:0/3632833865 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9e00640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf40cddf0 0x7fddf40b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 -- 172.21.15.125:0/3632833865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf40cddf0 msgr2=0x7fddf40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf40cddf0 0x7fddf40b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 -- 172.21.15.125:0/3632833865 >> v1:172.21.15.137:6789/0 conn(0x7fddf413fae0 legacy=0x7fddf40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:56.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.583+0000 7fddf9400640 1 -- 172.21.15.125:0/3632833865 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40ce720 con 0x7fddf413cf10 2024-01-16T15:02:56.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.584+0000 7fddf9400640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf413cf10 0x7fddf413f5a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009870 tx=0x7fdde402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff23cc7b122596c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.584+0000 7fddf3e00640 1 -- 172.21.15.125:0/3632833865 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4031640 con 0x7fddf413cf10 2024-01-16T15:02:56.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.584+0000 7fddf3e00640 1 -- 172.21.15.125:0/3632833865 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde40317a0 con 0x7fddf413cf10 2024-01-16T15:02:56.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddf3e00640 1 -- 172.21.15.125:0/3632833865 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4031b30 con 0x7fddf413cf10 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 -- 172.21.15.125:0/3632833865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf413cf10 msgr2=0x7fddf413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf413cf10 0x7fddf413f5a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009870 tx=0x7fdde402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 -- 172.21.15.125:0/3632833865 shutdown_connections 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf40cddf0 0x7fddf40b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 --2- 172.21.15.125:0/3632833865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf413cf10 0x7fddf413f5a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 -- 172.21.15.125:0/3632833865 >> 172.21.15.125:0/3632833865 conn(0x7fddf406a7d0 msgr2=0x7fddf406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 -- 172.21.15.125:0/3632833865 shutdown_connections 2024-01-16T15:02:56.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.585+0000 7fddfa800640 1 -- 172.21.15.125:0/3632833865 wait complete. 2024-01-16T15:02:56.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.587+0000 7fddfa800640 1 Processor -- start 2024-01-16T15:02:56.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.587+0000 7fddfa800640 1 -- start start 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 0x7fddf40a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 0x7fddf415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf415be30 con 0x7fddf40cddf0 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf413c3d0 con 0x7fddf413cf10 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddfa800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40a9d80 con 0x7fddf413fae0 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddf9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddf9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50044/0 (socket says 172.21.15.125:50044) 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.588+0000 7fddf9400640 1 -- 172.21.15.125:0/2050133249 learned_addr learned my addr 172.21.15.125:0/2050133249 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9e00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 0x7fddf415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf8a00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 0x7fddf40a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 -- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 msgr2=0x7fddf415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 0x7fddf415ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 -- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 msgr2=0x7fddf40a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 0x7fddf40a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 -- 172.21.15.125:0/2050133249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40094d0 con 0x7fddf40cddf0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9e00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 0x7fddf415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf8a00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 0x7fddf40a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf9400640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009870 tx=0x7fdde40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.589+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde403ac60 con 0x7fddf40cddf0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.590+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde403adc0 con 0x7fddf40cddf0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.590+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4044630 con 0x7fddf40cddf0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.590+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40a84c0 con 0x7fddf40cddf0 2024-01-16T15:02:56.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.590+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40a8a80 con 0x7fddf40cddf0 2024-01-16T15:02:56.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.591+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4072240 con 0x7fddf40cddf0 2024-01-16T15:02:56.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.592+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fdde402f050 con 0x7fddf40cddf0 2024-01-16T15:02:56.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.592+0000 7fddf1600640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 0x7fddcc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.593+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdde40ac670 con 0x7fddf40cddf0 2024-01-16T15:02:56.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.593+0000 7fddf8a00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 0x7fddcc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.593+0000 7fddf8a00640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 0x7fddcc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fddf415b390 tx=0x7fdddc0073a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.597+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdde4076ba0 con 0x7fddf40cddf0 2024-01-16T15:02:56.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.790+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fddf4067450 con 0x7fddf40cddf0 2024-01-16T15:02:56.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.797+0000 7fddf1600640 1 -- 172.21.15.125:0/2050133249 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7fdde407a1c0 con 0x7fddf40cddf0 2024-01-16T15:02:56.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 msgr2=0x7fddcc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 0x7fddcc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fddf415b390 tx=0x7fdddc0073a0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 msgr2=0x7fddf413be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009870 tx=0x7fdde40046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 shutdown_connections 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddf413fae0 0x7fddf415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddf413cf10 0x7fddf40a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fddcc06b8f0 0x7fddcc06ddb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 --2- 172.21.15.125:0/2050133249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fddf40cddf0 0x7fddf413be90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 >> 172.21.15.125:0/2050133249 conn(0x7fddf406a7d0 msgr2=0x7fddf413f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 shutdown_connections 2024-01-16T15:02:56.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:56.800+0000 7fddfa800640 1 -- 172.21.15.125:0/2050133249 wait complete. 2024-01-16T15:02:56.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:02:57.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.026+0000 7f130ae00640 1 Processor -- start 2024-01-16T15:02:57.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.026+0000 7f130ae00640 1 -- start start 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.027+0000 7f130ae00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1304143ae0 0x7f13040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.027+0000 7f130ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f130415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.027+0000 7f130ae00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 0x7f130415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.027+0000 7f130ae00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040cfc30 con 0x7f1304143ae0 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f130ae00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040b1ee0 con 0x7f130415e1e0 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f130ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040b2780 con 0x7f13040cf7d0 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1304143ae0 0x7f13040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1304143ae0 0x7f13040cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50056/0 (socket says 172.21.15.125:50056) 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 -- 172.21.15.125:0/3094702789 learned_addr learned my addr 172.21.15.125:0/3094702789 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 -- 172.21.15.125:0/3094702789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 msgr2=0x7f130415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 --2- 172.21.15.125:0/3094702789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f130415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1308e00640 1 --2- 172.21.15.125:0/3094702789 >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 0x7f130415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 -- 172.21.15.125:0/3094702789 >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 msgr2=0x7f130415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 --2- 172.21.15.125:0/3094702789 >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 0x7f130415a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1303e00640 1 -- 172.21.15.125:0/3094702789 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130415e990 con 0x7f1304143ae0 2024-01-16T15:02:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.028+0000 7f1308e00640 1 --2- 172.21.15.125:0/3094702789 >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 0x7f130415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.030+0000 7f1303e00640 1 --2- 172.21.15.125:0/3094702789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040cf290 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1304072160 tx=0x7f12f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5b1b8b47712632 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.030+0000 7f1302a00640 1 -- 172.21.15.125:0/3094702789 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4031870 con 0x7f1304143ae0 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.030+0000 7f1302a00640 1 -- 172.21.15.125:0/3094702789 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f40319d0 con 0x7f1304143ae0 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.030+0000 7f1302a00640 1 -- 172.21.15.125:0/3094702789 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4031d60 con 0x7f1304143ae0 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.030+0000 7f130ae00640 1 -- 172.21.15.125:0/3094702789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 msgr2=0x7f13040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 --2- 172.21.15.125:0/3094702789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040cf290 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1304072160 tx=0x7f12f402b910 comp rx=0 tx=0).stop 2024-01-16T15:02:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 -- 172.21.15.125:0/3094702789 shutdown_connections 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 --2- 172.21.15.125:0/3094702789 >> v2:172.21.15.137:3300/0 conn(0x7f130415e1e0 0x7f130415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 --2- 172.21.15.125:0/3094702789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f130415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 --2- 172.21.15.125:0/3094702789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040cf290 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 -- 172.21.15.125:0/3094702789 >> 172.21.15.125:0/3094702789 conn(0x7f130406a800 msgr2=0x7f130406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 -- 172.21.15.125:0/3094702789 shutdown_connections 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.031+0000 7f130ae00640 1 -- 172.21.15.125:0/3094702789 wait complete. 2024-01-16T15:02:57.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.033+0000 7f130ae00640 1 Processor -- start 2024-01-16T15:02:57.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.033+0000 7f130ae00640 1 -- start start 2024-01-16T15:02:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f1304159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f130415e1e0 0x7f13041f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304134f50 con 0x7f1304143ae0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130415a0f0 con 0x7f130415e1e0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f130ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c8c10 con 0x7f13040cf7d0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f1303400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f1303400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50062/0 (socket says 172.21.15.125:50062) 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f1303400640 1 -- 172.21.15.125:0/3058359690 learned_addr learned my addr 172.21.15.125:0/3058359690 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.034+0000 7f1308e00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f130415e1e0 0x7f13041f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303e00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f1304159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 -- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 msgr2=0x7f1304159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f1304159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 -- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f130415e1e0 msgr2=0x7f13041f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f130415e1e0 0x7f13041f6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 -- 172.21.15.125:0/3058359690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40094d0 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f1303400640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00b4d0 tx=0x7f12ec00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec010870 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec0109d0 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec010d80 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304134340 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.035+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1304134900 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.037+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13040b2bf0 con 0x7f1304143ae0 2024-01-16T15:02:57.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.037+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f12ec00f3c0 con 0x7f1304143ae0 2024-01-16T15:02:57.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.038+0000 7f12fbe00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 0x7f12dc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.038+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12ec08a6a0 con 0x7f1304143ae0 2024-01-16T15:02:57.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.039+0000 7f1303e00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 0x7f12dc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.039+0000 7f1303e00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 0x7f12dc06ddb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12f4036000 tx=0x7f12f403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.043+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13040b2bf0 con 0x7f1304143ae0 2024-01-16T15:02:57.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.234+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f13040b2bf0 con 0x7f1304143ae0 2024-01-16T15:02:57.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.241+0000 7f12fbe00640 1 -- 172.21.15.125:0/3058359690 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f13040b2bf0 con 0x7f1304143ae0 2024-01-16T15:02:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 msgr2=0x7f12dc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 0x7f12dc06ddb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12f4036000 tx=0x7f12f403c040 comp rx=0 tx=0).stop 2024-01-16T15:02:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 msgr2=0x7f13040c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00b4d0 tx=0x7f12ec00b990 comp rx=0 tx=0).stop 2024-01-16T15:02:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 shutdown_connections 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f130415e1e0 0x7f13041f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12dc06b8f0 0x7f12dc06ddb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1304143ae0 0x7f13040c86d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 --2- 172.21.15.125:0/3058359690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cf7d0 0x7f1304159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 >> 172.21.15.125:0/3058359690 conn(0x7f130406a800 msgr2=0x7f13040702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.244+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 shutdown_connections 2024-01-16T15:02:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.245+0000 7f130ae00640 1 -- 172.21.15.125:0/3058359690 wait complete. 2024-01-16T15:02:57.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:02:57.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.473+0000 7fd234a00640 1 Processor -- start 2024-01-16T15:02:57.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.473+0000 7fd234a00640 1 -- start start 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd234a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2300b3c20 0x7fd230070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd234a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd230071790 con 0x7fd23006d050 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd234a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd230071100 con 0x7fd230159590 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd234a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd230070860 con 0x7fd2300b3c20 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd22ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2300b3c20 0x7fd230070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd22ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2300b3c20 0x7fd230070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50064/0 (socket says 172.21.15.125:50064) 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.475+0000 7fd22ea00640 1 -- 172.21.15.125:0/3345967152 learned_addr learned my addr 172.21.15.125:0/3345967152 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.476+0000 7fd22ea00640 1 -- 172.21.15.125:0/3345967152 >> v1:172.21.15.137:6789/0 conn(0x7fd230159590 legacy=0x7fd2300ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.476+0000 7fd22ea00640 1 -- 172.21.15.125:0/3345967152 >> v1:172.21.15.125:6789/0 conn(0x7fd23006d050 legacy=0x7fd23006d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.476+0000 7fd22ea00640 1 -- 172.21.15.125:0/3345967152 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230070b40 con 0x7fd2300b3c20 2024-01-16T15:02:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.477+0000 7fd22ea00640 1 --2- 172.21.15.125:0/3345967152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300b3c20 0x7fd230070160 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd218009870 tx=0x7fd21802ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a814b9f5b1291b10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.477+0000 7fd22e000640 1 -- 172.21.15.125:0/3345967152 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218030840 con 0x7fd2300b3c20 2024-01-16T15:02:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.477+0000 7fd22e000640 1 -- 172.21.15.125:0/3345967152 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2180309a0 con 0x7fd2300b3c20 2024-01-16T15:02:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.477+0000 7fd22e000640 1 -- 172.21.15.125:0/3345967152 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218030d30 con 0x7fd2300b3c20 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 -- 172.21.15.125:0/3345967152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300b3c20 msgr2=0x7fd230070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 --2- 172.21.15.125:0/3345967152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300b3c20 0x7fd230070160 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd218009870 tx=0x7fd21802ca20 comp rx=0 tx=0).stop 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 -- 172.21.15.125:0/3345967152 shutdown_connections 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 --2- 172.21.15.125:0/3345967152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300b3c20 0x7fd230070160 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 -- 172.21.15.125:0/3345967152 >> 172.21.15.125:0/3345967152 conn(0x7fd23006a800 msgr2=0x7fd23006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 -- 172.21.15.125:0/3345967152 shutdown_connections 2024-01-16T15:02:57.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.478+0000 7fd234a00640 1 -- 172.21.15.125:0/3345967152 wait complete. 2024-01-16T15:02:57.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.480+0000 7fd234a00640 1 Processor -- start 2024-01-16T15:02:57.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.480+0000 7fd234a00640 1 -- start start 2024-01-16T15:02:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2300b3c20 0x7fd230139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd230159590 0x7fd23015a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23015c900 con 0x7fd23006d050 2024-01-16T15:02:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd230139fe0 con 0x7fd230159590 2024-01-16T15:02:57.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd234a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23015ab60 con 0x7fd2300b3c20 2024-01-16T15:02:57.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd22f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd22f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50072/0 (socket says 172.21.15.125:50072) 2024-01-16T15:02:57.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.481+0000 7fd22f400640 1 -- 172.21.15.125:0/3498893329 learned_addr learned my addr 172.21.15.125:0/3498893329 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22ea00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2300b3c20 0x7fd230139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 -- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2300b3c20 msgr2=0x7fd230139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22fe00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd230159590 0x7fd23015a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2300b3c20 0x7fd230139aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 -- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd230159590 msgr2=0x7fd23015a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd230159590 0x7fd23015a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 -- 172.21.15.125:0/3498893329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180094d0 con 0x7fd23006d050 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd22f400640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b3f0 tx=0x7fd22000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011830 con 0x7fd23006d050 2024-01-16T15:02:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd220011990 con 0x7fd23006d050 2024-01-16T15:02:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011d40 con 0x7fd23006d050 2024-01-16T15:02:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23015bcf0 con 0x7fd23006d050 2024-01-16T15:02:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.482+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23015c2b0 con 0x7fd23006d050 2024-01-16T15:02:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.483+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd230072210 con 0x7fd23006d050 2024-01-16T15:02:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.484+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fd22001a400 con 0x7fd23006d050 2024-01-16T15:02:57.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.485+0000 7fd227400640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 0x7fd20806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.485+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd22008a680 con 0x7fd23006d050 2024-01-16T15:02:57.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.485+0000 7fd22ea00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 0x7fd20806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.486+0000 7fd22ea00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 0x7fd20806ddb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd218037000 tx=0x7fd21803a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.490+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd220054b20 con 0x7fd23006d050 2024-01-16T15:02:57.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.712+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd230159970 con 0x7fd20806b8f0 2024-01-16T15:02:57.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.741+0000 7fd227400640 1 -- 172.21.15.125:0/3498893329 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd218030c00 con 0x7fd20806b8f0 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 msgr2=0x7fd20806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 0x7fd20806ddb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd218037000 tx=0x7fd21803a040 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 msgr2=0x7fd23015b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b3f0 tx=0x7fd22000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 shutdown_connections 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd230159590 0x7fd23015a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2300b3c20 0x7fd230139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd20806b8f0 0x7fd20806ddb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 --2- 172.21.15.125:0/3498893329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd23006d050 0x7fd23015b670 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 >> 172.21.15.125:0/3498893329 conn(0x7fd23006a800 msgr2=0x7fd2300ccce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 shutdown_connections 2024-01-16T15:02:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.744+0000 7fd234a00640 1 -- 172.21.15.125:0/3498893329 wait complete. 2024-01-16T15:02:57.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:02:57.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.972+0000 7f00a2800640 1 Processor -- start 2024-01-16T15:02:57.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.972+0000 7f00a2800640 1 -- start start 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.973+0000 7f00a2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009c15e3b0 0x7f009c164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.973+0000 7f00a2800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0b1ee0 con 0x7f009c143ae0 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.973+0000 7f00a2800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0b2780 con 0x7f009c0cf9a0 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.973+0000 7f00a2800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c15e990 con 0x7f009c15e3b0 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009c15e3b0 0x7f009c164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009c15e3b0 0x7f009c164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50082/0 (socket says 172.21.15.125:50082) 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 -- 172.21.15.125:0/2829330886 learned_addr learned my addr 172.21.15.125:0/2829330886 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 -- 172.21.15.125:0/2829330886 >> v1:172.21.15.137:6789/0 conn(0x7f009c0cf9a0 legacy=0x7f009c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 -- 172.21.15.125:0/2829330886 >> v1:172.21.15.125:6789/0 conn(0x7f009c143ae0 legacy=0x7f009c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.974+0000 7f009be00640 1 -- 172.21.15.125:0/2829330886 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0b2130 con 0x7f009c15e3b0 2024-01-16T15:02:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.975+0000 7f009be00640 1 --2- 172.21.15.125:0/2829330886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c15e3b0 0x7f009c164de0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0094009e50 tx=0x7f009402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d7f8a95caffd07 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.975+0000 7f009a000640 1 -- 172.21.15.125:0/2829330886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402cb20 con 0x7f009c15e3b0 2024-01-16T15:02:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.975+0000 7f009a000640 1 -- 172.21.15.125:0/2829330886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009402cc80 con 0x7f009c15e3b0 2024-01-16T15:02:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.975+0000 7f009a000640 1 -- 172.21.15.125:0/2829330886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940343f0 con 0x7f009c15e3b0 2024-01-16T15:02:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 -- 172.21.15.125:0/2829330886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c15e3b0 msgr2=0x7f009c164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 --2- 172.21.15.125:0/2829330886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c15e3b0 0x7f009c164de0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0094009e50 tx=0x7f009402bf80 comp rx=0 tx=0).stop 2024-01-16T15:02:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 -- 172.21.15.125:0/2829330886 shutdown_connections 2024-01-16T15:02:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 --2- 172.21.15.125:0/2829330886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c15e3b0 0x7f009c164de0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 -- 172.21.15.125:0/2829330886 >> 172.21.15.125:0/2829330886 conn(0x7f009c06a800 msgr2=0x7f009c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 -- 172.21.15.125:0/2829330886 shutdown_connections 2024-01-16T15:02:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.976+0000 7f00a2800640 1 -- 172.21.15.125:0/2829330886 wait complete. 2024-01-16T15:02:57.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.978+0000 7f00a2800640 1 Processor -- start 2024-01-16T15:02:57.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.978+0000 7f00a2800640 1 -- start start 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 0x7f009c13d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c15e3b0 0x7f009c140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0a8cd0 con 0x7f009c0cf9a0 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0a93c0 con 0x7f009c143ae0 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f00a2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0a84c0 con 0x7f009c15e3b0 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f009b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f009b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50084/0 (socket says 172.21.15.125:50084) 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.979+0000 7f009b400640 1 -- 172.21.15.125:0/335728166 learned_addr learned my addr 172.21.15.125:0/335728166 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:02:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009aa00640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 0x7f009c13d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009be00640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c15e3b0 0x7f009c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 -- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c15e3b0 msgr2=0x7f009c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c15e3b0 0x7f009c140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 -- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 msgr2=0x7f009c13d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 0x7f009c13d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 -- 172.21.15.125:0/335728166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094009b30 con 0x7f009c0cf9a0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009aa00640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 0x7f009c13d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f009b400640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c480 tx=0x7f008c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c0052d0 con 0x7f009c0cf9a0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c005430 con 0x7f009c0cf9a0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c0cd530 con 0x7f009c0cf9a0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.980+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c012870 con 0x7f009c0cf9a0 2024-01-16T15:02:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.981+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c0a8710 con 0x7f009c0cf9a0 2024-01-16T15:02:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.982+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c06c3f0 con 0x7f009c0cf9a0 2024-01-16T15:02:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.983+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f008c0129d0 con 0x7f009c0cf9a0 2024-01-16T15:02:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.983+0000 7f0093400640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 0x7f007406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.984+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f008c08a250 con 0x7f009c0cf9a0 2024-01-16T15:02:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.984+0000 7f009aa00640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 0x7f007406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.985+0000 7f009aa00640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 0x7f007406dde0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f009c1f22c0 tx=0x7f0084009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:57.988+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f008c054810 con 0x7f009c0cf9a0 2024-01-16T15:02:58.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.207+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f009c0cbd00 con 0x7f007406b920 2024-01-16T15:02:58.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.216+0000 7f0093400640 1 -- 172.21.15.125:0/335728166 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f009c0cbd00 con 0x7f007406b920 2024-01-16T15:02:58.197 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 msgr2=0x7f007406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 0x7f007406dde0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f009c1f22c0 tx=0x7f0084009040 comp rx=0 tx=0).stop 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 msgr2=0x7f009c0a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c480 tx=0x7f008c00c940 comp rx=0 tx=0).stop 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 shutdown_connections 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c15e3b0 0x7f009c140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c143ae0 0x7f009c13d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f007406b920 0x7f007406dde0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 --2- 172.21.15.125:0/335728166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009c0cf9a0 0x7f009c0a8e80 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.219+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 >> 172.21.15.125:0/335728166 conn(0x7f009c06a800 msgr2=0x7f009c0b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:58.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.220+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 shutdown_connections 2024-01-16T15:02:58.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:02:58.220+0000 7f00a2800640 1 -- 172.21.15.125:0/335728166 wait complete. 2024-01-16T15:02:58.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:58.218 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42) 2024-01-16T15:02:58.270 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42 2024-01-16T15:02:58.270 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:02:58.270 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:02:58.272 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:03:00.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:00.272 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42) 2024-01-16T15:03:00.334 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/de0f7b03-a46a-4357-b305-a8c65ac7ac42 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:00.336 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:03:00.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27071 2024-01-16T15:03:00.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.573+0000 7fc921c00640 1 Processor -- start 2024-01-16T15:03:00.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.573+0000 7fc921c00640 1 -- start start 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.574+0000 7fc921c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 0x7fc91c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc921c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc91c15e1e0 0x7fc91c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc921c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91c0b1ee0 con 0x7fc91c143ae0 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc921c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91c0b2780 con 0x7fc91c15e1e0 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc921c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c15e990 con 0x7fc91c0cf9a0 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc91aa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc91c143ae0 0x7fc91c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56096/0 (socket says 172.21.15.125:56096) 2024-01-16T15:03:00.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc91aa00640 1 -- 172.21.15.125:0/696457272 learned_addr learned my addr 172.21.15.125:0/696457272 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc91b400640 1 --2- 172.21.15.125:0/696457272 >> v2:172.21.15.137:3300/0 conn(0x7fc91c15e1e0 0x7fc91c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc911e00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 0x7fc91c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853006571 0 0) 0x7fc91c0b1ee0 con 0x7fc91c143ae0 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.575+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8fc003520 con 0x7fc91c143ae0 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233892589 0 0) 0x7fc8fc003520 con 0x7fc91c143ae0 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 msgr2=0x7fc91c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 0x7fc91c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 >> v2:172.21.15.137:3300/0 conn(0x7fc91c15e1e0 msgr2=0x7fc91c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 --2- 172.21.15.125:0/696457272 >> v2:172.21.15.137:3300/0 conn(0x7fc91c15e1e0 0x7fc91c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c0b2130 con 0x7fc91c143ae0 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc90c0019f0 con 0x7fc91c143ae0 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 >> v1:172.21.15.125:6789/0 conn(0x7fc91c143ae0 legacy=0x7fc91c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8fc003970 0x7fc8fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc0047c0 0x7fc8fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 0x7fc8fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c0b2780 con 0x7fc8fc003970 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c15e990 con 0x7fc8fc0047c0 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc00b980 con 0x7fc91c0cf9a0 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.576+0000 7fc91aa00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8fc003970 0x7fc8fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.577+0000 7fc91aa00640 1 -- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 msgr2=0x7fc8fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.577+0000 7fc91aa00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c0cf9a0 0x7fc8fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.577+0000 7fc91aa00640 1 -- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc0047c0 msgr2=0x7fc8fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.577+0000 7fc91aa00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc0047c0 0x7fc8fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.577+0000 7fc91aa00640 1 -- 172.21.15.125:0/696457272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc003520 con 0x7fc8fc003970 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.578+0000 7fc91aa00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8fc003970 0x7fc8fc003d90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00a830 tx=0x7fc8fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5decb6af497531 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.578+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c034630 con 0x7fc8fc003970 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.578+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90c02f070 con 0x7fc8fc003970 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.578+0000 7fc91a000640 1 -- 172.21.15.125:0/696457272 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c0349f0 con 0x7fc8fc003970 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 -- 172.21.15.125:0/696457272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8fc003970 msgr2=0x7fc8fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 --2- 172.21.15.125:0/696457272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8fc003970 0x7fc8fc003d90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00a830 tx=0x7fc8fc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc91b400640 1 -- 172.21.15.125:0/696457272 reap_dead start 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 -- 172.21.15.125:0/696457272 shutdown_connections 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 -- 172.21.15.125:0/696457272 >> 172.21.15.125:0/696457272 conn(0x7fc91c06a800 msgr2=0x7fc91c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 -- 172.21.15.125:0/696457272 shutdown_connections 2024-01-16T15:03:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.579+0000 7fc921c00640 1 -- 172.21.15.125:0/696457272 wait complete. 2024-01-16T15:03:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.581+0000 7fc921c00640 1 Processor -- start 2024-01-16T15:03:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.581+0000 7fc921c00640 1 -- start start 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 0x7fc91c1669c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c167470 0x7fc91c142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c15aed0 con 0x7fc91c15a4e0 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c157180 con 0x7fc8fc003970 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc921c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c141640 con 0x7fc91c167470 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50090/0 (socket says 172.21.15.125:50090) 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 -- 172.21.15.125:0/3969796764 learned_addr learned my addr 172.21.15.125:0/3969796764 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91aa00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 0x7fc91c1669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 -- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c167470 msgr2=0x7fc91c142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc911e00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c167470 0x7fc91c142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c167470 0x7fc91c142c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 -- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 msgr2=0x7fc91c1669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 0x7fc91c1669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91b400640 1 -- 172.21.15.125:0/3969796764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c140a30 con 0x7fc91c15a4e0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.582+0000 7fc91aa00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 0x7fc91c1669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc91b400640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0cc3c0 tx=0x7fc91400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400c680 con 0x7fc91c15a4e0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91400c7e0 con 0x7fc91c15a4e0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400cb90 con 0x7fc91c15a4e0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91c140d10 con 0x7fc91c15a4e0 2024-01-16T15:03:00.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.583+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c1412d0 con 0x7fc91c15a4e0 2024-01-16T15:03:00.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.584+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91c0675d0 con 0x7fc91c15a4e0 2024-01-16T15:03:00.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.585+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fc91400ccf0 con 0x7fc91c15a4e0 2024-01-16T15:03:00.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.586+0000 7fc913400640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 0x7fc8f406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.586+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc91408a090 con 0x7fc91c15a4e0 2024-01-16T15:03:00.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.587+0000 7fc91aa00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 0x7fc8f406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.587+0000 7fc91aa00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 0x7fc8f406ddb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc90c034000 tx=0x7fc90c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.591+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc914054610 con 0x7fc91c15a4e0 2024-01-16T15:03:00.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.820+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27071", "target": ["mon-mgr", ""]}) v1 -- 0x7fc91c15e640 con 0x7fc8f406b8f0 2024-01-16T15:03:00.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.951+0000 7fc913400640 1 -- 172.21.15.125:0/3969796764 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc90c004b60 con 0x7fc8f406b8f0 2024-01-16T15:03:00.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 msgr2=0x7fc8f406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 0x7fc8f406ddb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc90c034000 tx=0x7fc90c009030 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 msgr2=0x7fc91c15a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0cc3c0 tx=0x7fc91400ec00 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 shutdown_connections 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc8f406b8f0 0x7fc8f406ddb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc91c15a4e0 0x7fc91c15a990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc91c167470 0x7fc91c142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 --2- 172.21.15.125:0/3969796764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8fc003970 0x7fc91c1669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 >> 172.21.15.125:0/3969796764 conn(0x7fc91c06a800 msgr2=0x7fc91c13f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 shutdown_connections 2024-01-16T15:03:00.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:00.955+0000 7fc921c00640 1 -- 172.21.15.125:0/3969796764 wait complete. 2024-01-16T15:03:00.939 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.959+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.939 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.959+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.939 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.959+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.939 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.959+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.939 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.959+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27072 2024-01-16T15:03:00.962 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.982+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.962 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.982+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.962 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.982+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.962 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.982+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:00.962 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:00.982+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:01.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.165+0000 7f87b7400640 1 Processor -- start 2024-01-16T15:03:01.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.165+0000 7f87b7400640 1 -- start start 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87b013e360 0x7f87b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b7400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b006c470 con 0x7f87b015f9c0 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b7400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b006c750 con 0x7f87b013e360 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b006ca30 con 0x7f87b013df80 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87afe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87b013e360 0x7f87b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87afe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87b013e360 0x7f87b00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50126/0 (socket says 172.21.15.125:50126) 2024-01-16T15:03:01.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87afe00640 1 -- 172.21.15.125:0/4204573329 learned_addr learned my addr 172.21.15.125:0/4204573329 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:01.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87afe00640 1 -- 172.21.15.125:0/4204573329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 msgr2=0x7f87b015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.167+0000 7f87b4a00640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.168+0000 7f87afe00640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.168+0000 7f87afe00640 1 -- 172.21.15.125:0/4204573329 >> v1:172.21.15.137:6789/0 conn(0x7f87b015f9c0 legacy=0x7f87b01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.168+0000 7f87afe00640 1 -- 172.21.15.125:0/4204573329 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b006cd10 con 0x7f87b013e360 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.168+0000 7f87b4a00640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:01.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.169+0000 7f87afe00640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00cf290 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8798009870 tx=0x7f879802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09806d557750324 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.169+0000 7f87af400640 1 -- 172.21.15.125:0/4204573329 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798031640 con 0x7f87b013e360 2024-01-16T15:03:01.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.169+0000 7f87af400640 1 -- 172.21.15.125:0/4204573329 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980317a0 con 0x7f87b013e360 2024-01-16T15:03:01.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.169+0000 7f87af400640 1 -- 172.21.15.125:0/4204573329 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798031b30 con 0x7f87b013e360 2024-01-16T15:03:01.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 -- 172.21.15.125:0/4204573329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 msgr2=0x7f87b00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00cf290 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8798009870 tx=0x7f879802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 -- 172.21.15.125:0/4204573329 shutdown_connections 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00cf290 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 --2- 172.21.15.125:0/4204573329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 -- 172.21.15.125:0/4204573329 >> 172.21.15.125:0/4204573329 conn(0x7f87b006a800 msgr2=0x7f87b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 -- 172.21.15.125:0/4204573329 shutdown_connections 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.170+0000 7f87b7400640 1 -- 172.21.15.125:0/4204573329 wait complete. 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.172+0000 7f87b7400640 1 Processor -- start 2024-01-16T15:03:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.172+0000 7f87b7400640 1 -- start start 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b01541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87b015f9c0 0x7f87b0153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b013da70 con 0x7f87b013e360 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01546e0 con 0x7f87b015f9c0 2024-01-16T15:03:01.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00a95b0 con 0x7f87b013df80 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50138/0 (socket says 172.21.15.125:50138) 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 -- 172.21.15.125:0/3473788050 learned_addr learned my addr 172.21.15.125:0/3473788050 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87b4a00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b01541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 -- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 msgr2=0x7f87b01541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b01541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 -- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87b015f9c0 msgr2=0x7f87b0153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87b015f9c0 0x7f87b0153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.173+0000 7f87afe00640 1 -- 172.21.15.125:0/3473788050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980094d0 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87afe00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8798009870 tx=0x7f87980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879803ac60 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879803adc0 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798044630 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b00a8a30 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.174+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b00a8ff0 con 0x7f87b013e360 2024-01-16T15:03:01.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.175+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b00a9810 con 0x7f87b013e360 2024-01-16T15:03:01.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.176+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f879802f050 con 0x7f87b013e360 2024-01-16T15:03:01.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.177+0000 7f87acc00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 0x7f878806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.177+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87980ac7e0 con 0x7f87b013e360 2024-01-16T15:03:01.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.177+0000 7f87b4a00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 0x7f878806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.178+0000 7f87b4a00640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 0x7f878806dde0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f87b015ad70 tx=0x7f87a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.182+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87b00a9810 con 0x7f87b013e360 2024-01-16T15:03:01.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.408+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27072", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b00ce850 con 0x7f878806b920 2024-01-16T15:03:01.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.543+0000 7f87acc00640 1 -- 172.21.15.125:0/3473788050 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f87a000c040 con 0x7f878806b920 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 msgr2=0x7f878806dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 0x7f878806dde0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f87b015ad70 tx=0x7f87a0008040 comp rx=0 tx=0).stop 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 msgr2=0x7f87b00b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8798009870 tx=0x7f87980046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 shutdown_connections 2024-01-16T15:03:01.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87b015f9c0 0x7f87b0153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878806b920 0x7f878806dde0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87b013e360 0x7f87b00b3a10 secure :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8798009870 tx=0x7f87980046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 --2- 172.21.15.125:0/3473788050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87b013df80 0x7f87b01541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 >> 172.21.15.125:0/3473788050 conn(0x7f87b006a800 msgr2=0x7f87b006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 shutdown_connections 2024-01-16T15:03:01.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.547+0000 7f87b7400640 1 -- 172.21.15.125:0/3473788050 wait complete. 2024-01-16T15:03:01.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27073 2024-01-16T15:03:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.834+0000 7fde32c00640 1 Processor -- start 2024-01-16T15:03:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.835+0000 7fde32c00640 1 -- start start 2024-01-16T15:03:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.836+0000 7fde32c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.836+0000 7fde32c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fde2c071070 0x7fde2c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.836+0000 7fde32c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0cfbf0 con 0x7fde2c0cf7d0 2024-01-16T15:03:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.836+0000 7fde32c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c14b6a0 con 0x7fde2c071070 2024-01-16T15:03:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.836+0000 7fde32c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c071690 con 0x7fde2c14b2c0 2024-01-16T15:03:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.837+0000 7fde2be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.837+0000 7fde2be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56854/0 (socket says 172.21.15.125:56854) 2024-01-16T15:03:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.837+0000 7fde2be00640 1 -- 172.21.15.125:0/3406513529 learned_addr learned my addr 172.21.15.125:0/3406513529 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.837+0000 7fde30c00640 1 --2- 172.21.15.125:0/3406513529 >> v2:172.21.15.137:3300/0 conn(0x7fde2c071070 0x7fde2c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2aa00640 1 -- 172.21.15.125:0/3406513529 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077134568 0 0) 0x7fde2c0cfbf0 con 0x7fde2c0cf7d0 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2aa00640 1 -- 172.21.15.125:0/3406513529 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0c003520 con 0x7fde2c0cf7d0 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2be00640 1 -- 172.21.15.125:0/3406513529 >> v2:172.21.15.137:3300/0 conn(0x7fde2c071070 msgr2=0x7fde2c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2be00640 1 --2- 172.21.15.125:0/3406513529 >> v2:172.21.15.137:3300/0 conn(0x7fde2c071070 0x7fde2c0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2be00640 1 -- 172.21.15.125:0/3406513529 >> v1:172.21.15.137:6789/0 conn(0x7fde2c0cf7d0 legacy=0x7fde2c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.838+0000 7fde2be00640 1 -- 172.21.15.125:0/3406513529 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c071970 con 0x7fde2c14b2c0 2024-01-16T15:03:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.839+0000 7fde2be00640 1 --2- 172.21.15.125:0/3406513529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009e50 tx=0x7fde1c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e67f936fe979ced server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.840+0000 7fde2aa00640 1 -- 172.21.15.125:0/3406513529 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c02caf0 con 0x7fde2c14b2c0 2024-01-16T15:03:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.840+0000 7fde2aa00640 1 -- 172.21.15.125:0/3406513529 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c02cc50 con 0x7fde2c14b2c0 2024-01-16T15:03:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.840+0000 7fde2aa00640 1 -- 172.21.15.125:0/3406513529 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0343f0 con 0x7fde2c14b2c0 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 -- 172.21.15.125:0/3406513529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 msgr2=0x7fde2c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 --2- 172.21.15.125:0/3406513529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009e50 tx=0x7fde1c02bf50 comp rx=0 tx=0).stop 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 -- 172.21.15.125:0/3406513529 shutdown_connections 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 --2- 172.21.15.125:0/3406513529 >> v2:172.21.15.137:3300/0 conn(0x7fde2c071070 0x7fde2c0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 --2- 172.21.15.125:0/3406513529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c0cf290 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 -- 172.21.15.125:0/3406513529 >> 172.21.15.125:0/3406513529 conn(0x7fde2c06a800 msgr2=0x7fde2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 -- 172.21.15.125:0/3406513529 shutdown_connections 2024-01-16T15:03:01.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.841+0000 7fde32c00640 1 -- 172.21.15.125:0/3406513529 wait complete. 2024-01-16T15:03:01.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.843+0000 7fde32c00640 1 Processor -- start 2024-01-16T15:03:01.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.843+0000 7fde32c00640 1 -- start start 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde2c0cf7d0 0x7fde2c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c163c60 con 0x7fde2c071070 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c1f8160 con 0x7fde2c0cf7d0 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde32c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c162db0 con 0x7fde2c14b2c0 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50146/0 (socket says 172.21.15.125:50146) 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 -- 172.21.15.125:0/975403586 learned_addr learned my addr 172.21.15.125:0/975403586 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2b400640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde2c0cf7d0 0x7fde2c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde30c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 -- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 msgr2=0x7fde2c160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 -- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde2c0cf7d0 msgr2=0x7fde2c1640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde2c0cf7d0 0x7fde2c1640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.845+0000 7fde2be00640 1 -- 172.21.15.125:0/975403586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c009b30 con 0x7fde2c071070 2024-01-16T15:03:01.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.846+0000 7fde2be00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fde1c036040 tx=0x7fde1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.846+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c03e450 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.846+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c03e5b0 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.846+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c163050 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.846+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c03e960 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.847+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c163610 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.848+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c0675d0 con 0x7fde2c071070 2024-01-16T15:03:01.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.849+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fde1c03eac0 con 0x7fde2c071070 2024-01-16T15:03:01.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.850+0000 7fde23e00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 0x7fde0406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.850+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde1c0ad310 con 0x7fde2c071070 2024-01-16T15:03:01.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.850+0000 7fde2b400640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 0x7fde0406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.851+0000 7fde2b400640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 0x7fde0406dde0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06b2c0 tx=0x7fde14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:01.855+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde1c077840 con 0x7fde2c071070 2024-01-16T15:03:02.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:02.079+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27073", "target": ["mon-mgr", ""]}) v1 -- 0x7fde2c0cbc40 con 0x7fde0406b920 2024-01-16T15:03:03.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:03.132+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fde1c03ed70 con 0x7fde2c071070 2024-01-16T15:03:05.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.452+0000 7fde23e00640 1 -- 172.21.15.125:0/975403586 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde14001330 con 0x7fde0406b920 2024-01-16T15:03:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 msgr2=0x7fde0406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 0x7fde0406dde0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06b2c0 tx=0x7fde14009040 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 msgr2=0x7fde2c1f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fde1c036040 tx=0x7fde1c004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 shutdown_connections 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde2c14b2c0 0x7fde2c160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde2c0cf7d0 0x7fde2c1640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fde0406b920 0x7fde0406dde0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 --2- 172.21.15.125:0/975403586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c071070 0x7fde2c1f7c20 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.455+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 >> 172.21.15.125:0/975403586 conn(0x7fde2c06a800 msgr2=0x7fde2c0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.456+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 shutdown_connections 2024-01-16T15:03:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.456+0000 7fde32c00640 1 -- 172.21.15.125:0/975403586 wait complete. 2024-01-16T15:03:05.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27074 2024-01-16T15:03:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.685+0000 7f16aac00640 1 Processor -- start 2024-01-16T15:03:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.686+0000 7f16aac00640 1 -- start start 2024-01-16T15:03:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.687+0000 7f16aac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16a40cf9a0 0x7f16a4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.687+0000 7f16aac00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a4071690 con 0x7f16a414b2c0 2024-01-16T15:03:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.687+0000 7f16aac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a4071970 con 0x7f16a4070ea0 2024-01-16T15:03:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.687+0000 7f16aac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a414b6a0 con 0x7f16a40cf9a0 2024-01-16T15:03:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f16a3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f16a414b2c0 0x7f16a40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56102/0 (socket says 172.21.15.125:56102) 2024-01-16T15:03:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f16a3e00640 1 -- 172.21.15.125:0/1267868717 learned_addr learned my addr 172.21.15.125:0/1267868717 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f169b200640 1 --2- 172.21.15.125:0/1267868717 >> v2:172.21.15.125:3300/0 conn(0x7f16a40cf9a0 0x7f16a4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f16a3400640 1 -- 172.21.15.125:0/1267868717 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279387124 0 0) 0x7f16a4071690 con 0x7f16a414b2c0 2024-01-16T15:03:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f16a3400640 1 -- 172.21.15.125:0/1267868717 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684003520 con 0x7f16a414b2c0 2024-01-16T15:03:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f169b200640 1 -- 172.21.15.125:0/1267868717 >> v1:172.21.15.137:6789/0 conn(0x7f16a4070ea0 legacy=0x7f16a40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f169b200640 1 -- 172.21.15.125:0/1267868717 >> v1:172.21.15.125:6789/0 conn(0x7f16a414b2c0 legacy=0x7f16a40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.688+0000 7f169b200640 1 -- 172.21.15.125:0/1267868717 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4071c50 con 0x7f16a40cf9a0 2024-01-16T15:03:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.689+0000 7f169b200640 1 --2- 172.21.15.125:0/1267868717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a40cf9a0 0x7f16a4070960 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f168c0097a0 tx=0x7f168c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775a46e93263beab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16a3400640 1 -- 172.21.15.125:0/1267868717 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c031830 con 0x7f16a40cf9a0 2024-01-16T15:03:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16a3400640 1 -- 172.21.15.125:0/1267868717 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c031990 con 0x7f16a40cf9a0 2024-01-16T15:03:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16a3400640 1 -- 172.21.15.125:0/1267868717 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c031d20 con 0x7f16a40cf9a0 2024-01-16T15:03:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16aac00640 1 -- 172.21.15.125:0/1267868717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a40cf9a0 msgr2=0x7f16a4070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16aac00640 1 --2- 172.21.15.125:0/1267868717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a40cf9a0 0x7f16a4070960 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f168c0097a0 tx=0x7f168c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.690+0000 7f16aac00640 1 -- 172.21.15.125:0/1267868717 shutdown_connections 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.691+0000 7f16aac00640 1 --2- 172.21.15.125:0/1267868717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a40cf9a0 0x7f16a4070960 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.691+0000 7f16aac00640 1 -- 172.21.15.125:0/1267868717 >> 172.21.15.125:0/1267868717 conn(0x7f16a406a800 msgr2=0x7f16a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.691+0000 7f16aac00640 1 -- 172.21.15.125:0/1267868717 shutdown_connections 2024-01-16T15:03:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.691+0000 7f16aac00640 1 -- 172.21.15.125:0/1267868717 wait complete. 2024-01-16T15:03:05.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.692+0000 7f16aac00640 1 Processor -- start 2024-01-16T15:03:05.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.692+0000 7f16aac00640 1 -- start start 2024-01-16T15:03:05.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16a4070ea0 0x7f16a415f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16a40cf9a0 0x7f16a4146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4162fb0 con 0x7f16a414b2c0 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4149430 con 0x7f16a40cf9a0 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.693+0000 7f16aac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4148b90 con 0x7f16a4070ea0 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50152/0 (socket says 172.21.15.125:50152) 2024-01-16T15:03:05.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 -- 172.21.15.125:0/307030968 learned_addr learned my addr 172.21.15.125:0/307030968 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 -- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16a4070ea0 msgr2=0x7f16a415f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16a4070ea0 0x7f16a415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 -- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16a40cf9a0 msgr2=0x7f16a4146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16a40cf9a0 0x7f16a4146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 -- 172.21.15.125:0/307030968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c009400 con 0x7f16a414b2c0 2024-01-16T15:03:05.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.694+0000 7f16a8c00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f169c00c410 tx=0x7f169c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.695+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c011790 con 0x7f16a414b2c0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.695+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c0118f0 con 0x7f16a414b2c0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.695+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c011ce0 con 0x7f16a414b2c0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.695+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a4148e70 con 0x7f16a414b2c0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.695+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a41634a0 con 0x7f16a414b2c0 2024-01-16T15:03:05.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.696+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a40af5a0 con 0x7f16a414b2c0 2024-01-16T15:03:05.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.697+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f169c01c460 con 0x7f16a414b2c0 2024-01-16T15:03:05.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.698+0000 7f16a0c00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 0x7f167c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.698+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f169c08a9b0 con 0x7f16a414b2c0 2024-01-16T15:03:05.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.698+0000 7f16a3e00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 0x7f167c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.699+0000 7f16a3e00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 0x7f167c06ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1694002000 tx=0x7f16940015f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.702+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f169c087f00 con 0x7f16a414b2c0 2024-01-16T15:03:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:05.927+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27074", "target": ["mon-mgr", ""]}) v1 -- 0x7f16a40cbc40 con 0x7f167c06ba10 2024-01-16T15:03:08.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:08.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:10.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.435+0000 7f16a0c00640 1 -- 172.21.15.125:0/307030968 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f16940020f0 con 0x7f167c06ba10 2024-01-16T15:03:10.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.438+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 msgr2=0x7f167c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.438+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 0x7f167c06ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1694002000 tx=0x7f16940015f0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.438+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 msgr2=0x7f16a4142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.438+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f169c00c410 tx=0x7f169c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 shutdown_connections 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f167c06ba10 0x7f167c06ded0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16a414b2c0 0x7f16a4142180 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16a40cf9a0 0x7f16a4146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 --2- 172.21.15.125:0/307030968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16a4070ea0 0x7f16a415f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 >> 172.21.15.125:0/307030968 conn(0x7f16a406a800 msgr2=0x7f16a40ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 shutdown_connections 2024-01-16T15:03:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.440+0000 7f16aac00640 1 -- 172.21.15.125:0/307030968 wait complete. 2024-01-16T15:03:10.441 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27075 2024-01-16T15:03:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.670+0000 7f2811200640 1 Processor -- start 2024-01-16T15:03:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.670+0000 7f2811200640 1 -- start start 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2811200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c143ae0 0x7f280c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2811200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f280c0cf7d0 0x7f280c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2811200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c0cfc30 con 0x7f280c15e1e0 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2811200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280c0b1ee0 con 0x7f280c0cf7d0 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2811200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280c0b2780 con 0x7f280c143ae0 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f280c0cf7d0 0x7f280c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f280c0cf7d0 0x7f280c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43278/0 (socket says 172.21.15.125:43278) 2024-01-16T15:03:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 -- 172.21.15.125:0/238359190 learned_addr learned my addr 172.21.15.125:0/238359190 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2809e00640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c143ae0 0x7f280c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 -- 172.21.15.125:0/238359190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c143ae0 msgr2=0x7f280c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c143ae0 0x7f280c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 -- 172.21.15.125:0/238359190 >> v1:172.21.15.137:6789/0 conn(0x7f280c15e1e0 legacy=0x7f280c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.672+0000 7f2803e00640 1 -- 172.21.15.125:0/238359190 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c15e990 con 0x7f280c0cf7d0 2024-01-16T15:03:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.673+0000 7f2803e00640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c0cf7d0 0x7f280c15dca0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009870 tx=0x7f27f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50be4741b57876da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.673+0000 7f2809400640 1 -- 172.21.15.125:0/238359190 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4031640 con 0x7f280c0cf7d0 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.673+0000 7f2809400640 1 -- 172.21.15.125:0/238359190 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f40317a0 con 0x7f280c0cf7d0 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.673+0000 7f2809400640 1 -- 172.21.15.125:0/238359190 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4031b30 con 0x7f280c0cf7d0 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 -- 172.21.15.125:0/238359190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c0cf7d0 msgr2=0x7f280c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c0cf7d0 0x7f280c15dca0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009870 tx=0x7f27f402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 -- 172.21.15.125:0/238359190 shutdown_connections 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c0cf7d0 0x7f280c15dca0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 --2- 172.21.15.125:0/238359190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c143ae0 0x7f280c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 -- 172.21.15.125:0/238359190 >> 172.21.15.125:0/238359190 conn(0x7f280c06a800 msgr2=0x7f280c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 -- 172.21.15.125:0/238359190 shutdown_connections 2024-01-16T15:03:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.674+0000 7f2811200640 1 -- 172.21.15.125:0/238359190 wait complete. 2024-01-16T15:03:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.676+0000 7f2811200640 1 Processor -- start 2024-01-16T15:03:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.676+0000 7f2811200640 1 -- start start 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 0x7f280c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 0x7f280c158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c13c310 con 0x7f280c15e1e0 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c159160 con 0x7f280c143ae0 2024-01-16T15:03:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f2811200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280c159440 con 0x7f280c0cf7d0 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f280a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f280a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43294/0 (socket says 172.21.15.125:43294) 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.677+0000 7f280a800640 1 -- 172.21.15.125:0/2103723595 learned_addr learned my addr 172.21.15.125:0/2103723595 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2809e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 0x7f280c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2803e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 0x7f280c158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 -- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 msgr2=0x7f280c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 0x7f280c1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 -- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 msgr2=0x7f280c158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 0x7f280c158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 -- 172.21.15.125:0/2103723595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40094d0 con 0x7f280c15e1e0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2803e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 0x7f280c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2809e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 0x7f280c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f280a800640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f280400b970 tx=0x7f280400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400cb40 con 0x7f280c15e1e0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280400cca0 con 0x7f280c15e1e0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804014680 con 0x7f280c15e1e0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c159720 con 0x7f280c15e1e0 2024-01-16T15:03:10.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.678+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c154ef0 con 0x7f280c15e1e0 2024-01-16T15:03:10.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.680+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c13c480 con 0x7f280c15e1e0 2024-01-16T15:03:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.681+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f280401d3b0 con 0x7f280c15e1e0 2024-01-16T15:03:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.681+0000 7f2802000640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 0x7f27e406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.682+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2804088380 con 0x7f280c15e1e0 2024-01-16T15:03:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.682+0000 7f2809e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 0x7f27e406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.682+0000 7f2809e00640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 0x7f27e406ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006fd0 tx=0x7f27fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.686+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2804088090 con 0x7f280c15e1e0 2024-01-16T15:03:10.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:10.911+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27075", "target": ["mon-mgr", ""]}) v1 -- 0x7f280c15bef0 con 0x7f27e406ba10 2024-01-16T15:03:15.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.418+0000 7f2802000640 1 -- 172.21.15.125:0/2103723595 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f280c15bef0 con 0x7f27e406ba10 2024-01-16T15:03:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 msgr2=0x7f27e406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 0x7f27e406ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006fd0 tx=0x7f27fc008040 comp rx=0 tx=0).stop 2024-01-16T15:03:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 msgr2=0x7f280c0afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f280400b970 tx=0x7f280400be30 comp rx=0 tx=0).stop 2024-01-16T15:03:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 shutdown_connections 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f27e406ba10 0x7f27e406ded0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f280c15e1e0 0x7f280c0afba0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f280c143ae0 0x7f280c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 --2- 172.21.15.125:0/2103723595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f280c0cf7d0 0x7f280c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.422+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 >> 172.21.15.125:0/2103723595 conn(0x7f280c06a800 msgr2=0x7f280c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.423+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 shutdown_connections 2024-01-16T15:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.423+0000 7f2811200640 1 -- 172.21.15.125:0/2103723595 wait complete. 2024-01-16T15:03:15.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27076 2024-01-16T15:03:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.661+0000 7fc933200640 1 Processor -- start 2024-01-16T15:03:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.662+0000 7fc933200640 1 -- start start 2024-01-16T15:03:15.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc92c1547a0 0x7fc92c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc92c0715a0 0x7fc92c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c06b930 0x7fc92c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92c15e7b0 con 0x7fc92c0715a0 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92c06bda0 con 0x7fc92c1547a0 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.663+0000 7fc933200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c071d40 con 0x7fc92c06b930 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc92c0715a0 0x7fc92c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc92c0715a0 0x7fc92c06b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43300/0 (socket says 172.21.15.125:43300) 2024-01-16T15:03:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 -- 172.21.15.125:0/1922691271 learned_addr learned my addr 172.21.15.125:0/1922691271 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 -- 172.21.15.125:0/1922691271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c06b930 msgr2=0x7fc92c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc931200640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c06b930 0x7fc92c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92be00640 1 --2- 172.21.15.125:0/1922691271 >> v2:172.21.15.137:3300/0 conn(0x7fc92c1547a0 0x7fc92c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c06b930 0x7fc92c06cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 -- 172.21.15.125:0/1922691271 >> v2:172.21.15.137:3300/0 conn(0x7fc92c1547a0 msgr2=0x7fc92c15b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 --2- 172.21.15.125:0/1922691271 >> v2:172.21.15.137:3300/0 conn(0x7fc92c1547a0 0x7fc92c15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.664+0000 7fc92b400640 1 -- 172.21.15.125:0/1922691271 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c154b80 con 0x7fc92c0715a0 2024-01-16T15:03:15.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.665+0000 7fc92b400640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c0715a0 0x7fc92c06b3f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc92c074430 tx=0x7fc91402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce14d1bbb95af693 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc92aa00640 1 -- 172.21.15.125:0/1922691271 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9140317e0 con 0x7fc92c0715a0 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc92aa00640 1 -- 172.21.15.125:0/1922691271 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc914031940 con 0x7fc92c0715a0 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc92aa00640 1 -- 172.21.15.125:0/1922691271 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914031cd0 con 0x7fc92c0715a0 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 -- 172.21.15.125:0/1922691271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c0715a0 msgr2=0x7fc92c06b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c0715a0 0x7fc92c06b3f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc92c074430 tx=0x7fc91402b880 comp rx=0 tx=0).stop 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 -- 172.21.15.125:0/1922691271 shutdown_connections 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c06b930 0x7fc92c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 --2- 172.21.15.125:0/1922691271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c0715a0 0x7fc92c06b3f0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 --2- 172.21.15.125:0/1922691271 >> v2:172.21.15.137:3300/0 conn(0x7fc92c1547a0 0x7fc92c15b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 -- 172.21.15.125:0/1922691271 >> 172.21.15.125:0/1922691271 conn(0x7fc92c070a80 msgr2=0x7fc92c070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 -- 172.21.15.125:0/1922691271 shutdown_connections 2024-01-16T15:03:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.666+0000 7fc933200640 1 -- 172.21.15.125:0/1922691271 wait complete. 2024-01-16T15:03:15.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.668+0000 7fc933200640 1 Processor -- start 2024-01-16T15:03:15.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.668+0000 7fc933200640 1 -- start start 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 0x7fc92c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c1547a0 0x7fc92c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c157c60 con 0x7fc92c06b930 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c1594b0 con 0x7fc92c0715a0 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc933200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c159750 con 0x7fc92c1547a0 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.669+0000 7fc92be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43306/0 (socket says 172.21.15.125:43306) 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 -- 172.21.15.125:0/1132811242 learned_addr learned my addr 172.21.15.125:0/1132811242 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc931200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c1547a0 0x7fc92c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92b400640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 0x7fc92c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 -- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c1547a0 msgr2=0x7fc92c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c1547a0 0x7fc92c15a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 -- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 msgr2=0x7fc92c157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 0x7fc92c157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 -- 172.21.15.125:0/1132811242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140094d0 con 0x7fc92c06b930 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92b400640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 0x7fc92c157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.670+0000 7fc92be00640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c7f0 tx=0x7fc91c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.671+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c004220 con 0x7fc92c06b930 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.671+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c0043a0 con 0x7fc92c06b930 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.671+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92c159a30 con 0x7fc92c06b930 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.671+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c007860 con 0x7fc92c06b930 2024-01-16T15:03:15.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.671+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92c146690 con 0x7fc92c06b930 2024-01-16T15:03:15.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.672+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92c0675d0 con 0x7fc92c06b930 2024-01-16T15:03:15.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.673+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fc91c007bd0 con 0x7fc92c06b930 2024-01-16T15:03:15.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.674+0000 7fc923e00640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 0x7fc90406df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.674+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc91c08ae00 con 0x7fc92c06b930 2024-01-16T15:03:15.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.674+0000 7fc92b400640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 0x7fc90406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.675+0000 7fc92b400640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 0x7fc90406df00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc914036000 tx=0x7fc91403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.678+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc91c0881b0 con 0x7fc92c06b930 2024-01-16T15:03:15.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:15.903+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27076", "target": ["mon-mgr", ""]}) v1 -- 0x7fc92c15e0f0 con 0x7fc90406ba40 2024-01-16T15:03:16.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.057+0000 7fc923e00640 1 -- 172.21.15.125:0/1132811242 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc914031c30 con 0x7fc90406ba40 2024-01-16T15:03:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 msgr2=0x7fc90406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 0x7fc90406df00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc914036000 tx=0x7fc91403d040 comp rx=0 tx=0).stop 2024-01-16T15:03:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 msgr2=0x7fc92c146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c7f0 tx=0x7fc91c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 shutdown_connections 2024-01-16T15:03:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc92c1547a0 0x7fc92c15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc92c0715a0 0x7fc92c157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc90406ba40 0x7fc90406df00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 --2- 172.21.15.125:0/1132811242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92c06b930 0x7fc92c146150 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 >> 172.21.15.125:0/1132811242 conn(0x7fc92c070a80 msgr2=0x7fc92c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 shutdown_connections 2024-01-16T15:03:16.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.060+0000 7fc933200640 1 -- 172.21.15.125:0/1132811242 wait complete. 2024-01-16T15:03:16.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27077 2024-01-16T15:03:16.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.295+0000 7f4dd9000640 1 Processor -- start 2024-01-16T15:03:16.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.295+0000 7f4dd9000640 1 -- start start 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd9000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4dd40cf9a0 0x7f4dd415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd9000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 0x7f4dd40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40b1ee0 con 0x7f4dd4143ae0 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd9000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40b2780 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd9000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd415e990 con 0x7f4dd415e1e0 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd1400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4dd40cf9a0 0x7f4dd415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd1400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4dd40cf9a0 0x7f4dd415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43322/0 (socket says 172.21.15.125:43322) 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.297+0000 7f4dd1400640 1 -- 172.21.15.125:0/3049851461 learned_addr learned my addr 172.21.15.125:0/3049851461 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd2800640 1 --2- 172.21.15.125:0/3049851461 >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 0x7f4dd40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd1400640 1 -- 172.21.15.125:0/3049851461 >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 msgr2=0x7f4dd40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd1400640 1 --2- 172.21.15.125:0/3049851461 >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 0x7f4dd40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd1400640 1 -- 172.21.15.125:0/3049851461 >> v1:172.21.15.137:6789/0 conn(0x7f4dd4143ae0 legacy=0x7f4dd40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd1400640 1 -- 172.21.15.125:0/3049851461 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40b2130 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.298+0000 7f4dd2800640 1 --2- 172.21.15.125:0/3049851461 >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 0x7f4dd40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.299+0000 7f4dd1400640 1 --2- 172.21.15.125:0/3049851461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415dca0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40ae9d0 tx=0x7f4dbc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33b5ac08e9b71c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.299+0000 7f4dd0a00640 1 -- 172.21.15.125:0/3049851461 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc02ca80 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.299+0000 7f4dd0a00640 1 -- 172.21.15.125:0/3049851461 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc02cbe0 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.299+0000 7f4dd0a00640 1 -- 172.21.15.125:0/3049851461 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0343f0 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 -- 172.21.15.125:0/3049851461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd40cf9a0 msgr2=0x7f4dd415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 --2- 172.21.15.125:0/3049851461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415dca0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40ae9d0 tx=0x7f4dbc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 -- 172.21.15.125:0/3049851461 shutdown_connections 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 --2- 172.21.15.125:0/3049851461 >> v2:172.21.15.137:3300/0 conn(0x7f4dd415e1e0 0x7f4dd40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 --2- 172.21.15.125:0/3049851461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415dca0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 -- 172.21.15.125:0/3049851461 >> 172.21.15.125:0/3049851461 conn(0x7f4dd406a800 msgr2=0x7f4dd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 -- 172.21.15.125:0/3049851461 shutdown_connections 2024-01-16T15:03:16.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.300+0000 7f4dd9000640 1 -- 172.21.15.125:0/3049851461 wait complete. 2024-01-16T15:03:16.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.302+0000 7f4dd9000640 1 Processor -- start 2024-01-16T15:03:16.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.302+0000 7f4dd9000640 1 -- start start 2024-01-16T15:03:16.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dd415e1e0 0x7f4dd4159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd413e170 con 0x7f4dd4143ae0 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd415a060 con 0x7f4dd40cf9a0 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40cd830 con 0x7f4dd415e1e0 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43330/0 (socket says 172.21.15.125:43330) 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 -- 172.21.15.125:0/1222093977 learned_addr learned my addr 172.21.15.125:0/1222093977 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1e00640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 -- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dd415e1e0 msgr2=0x7f4dd4159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dd415e1e0 0x7f4dd4159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 -- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 msgr2=0x7f4dd415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.303+0000 7f4dd1400640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dd1400640 1 -- 172.21.15.125:0/1222093977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009b30 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dd1e00640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dd1400640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0093e0 tx=0x7f4dbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc03e450 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc03e5b0 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc03e960 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd413e3a0 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.304+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd413e960 con 0x7f4dd4143ae0 2024-01-16T15:03:16.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.306+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd413eb50 con 0x7f4dd4143ae0 2024-01-16T15:03:16.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.306+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f4dbc02cd50 con 0x7f4dd4143ae0 2024-01-16T15:03:16.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.307+0000 7f4dca000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 0x7f4dac06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.307+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dbc0ada70 con 0x7f4dd4143ae0 2024-01-16T15:03:16.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.308+0000 7f4dd1e00640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 0x7f4dac06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.308+0000 7f4dd1e00640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 0x7f4dac06ded0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4139230 tx=0x7f4dc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.312+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4dbc0aaf30 con 0x7f4dd4143ae0 2024-01-16T15:03:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.535+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27077", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dd415e670 con 0x7f4dac06ba10 2024-01-16T15:03:16.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.682+0000 7f4dca000640 1 -- 172.21.15.125:0/1222093977 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4dc4003a80 con 0x7f4dac06ba10 2024-01-16T15:03:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.685+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 msgr2=0x7f4dac06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 0x7f4dac06ded0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4139230 tx=0x7f4dc4009040 comp rx=0 tx=0).stop 2024-01-16T15:03:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 msgr2=0x7f4dd415cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0093e0 tx=0x7f4dbc004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 shutdown_connections 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dd415e1e0 0x7f4dd4159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4dac06ba10 0x7f4dac06ded0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dd4143ae0 0x7f4dd415cc10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 --2- 172.21.15.125:0/1222093977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd40cf9a0 0x7f4dd415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 >> 172.21.15.125:0/1222093977 conn(0x7f4dd406a800 msgr2=0x7f4dd40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 shutdown_connections 2024-01-16T15:03:16.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.686+0000 7f4dd9000640 1 -- 172.21.15.125:0/1222093977 wait complete. 2024-01-16T15:03:16.685 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:03:16.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27071 2024-01-16T15:03:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.910+0000 7fe331c00640 1 Processor -- start 2024-01-16T15:03:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.910+0000 7fe331c00640 1 -- start start 2024-01-16T15:03:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe331c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32c0cf9a0 0x7fe32c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe331c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe32c15e1e0 0x7fe32c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe331c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0b1ee0 con 0x7fe32c143ae0 2024-01-16T15:03:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe331c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c0b2780 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe331c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c15e990 con 0x7fe32c15e1e0 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe32aa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe32c143ae0 0x7fe32c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60676/0 (socket says 172.21.15.125:60676) 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe321e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32c0cf9a0 0x7fe32c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.912+0000 7fe32aa00640 1 -- 172.21.15.125:0/3726719279 learned_addr learned my addr 172.21.15.125:0/3726719279 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.913+0000 7fe32b400640 1 --2- 172.21.15.125:0/3726719279 >> v2:172.21.15.137:3300/0 conn(0x7fe32c15e1e0 0x7fe32c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.913+0000 7fe321e00640 1 -- 172.21.15.125:0/3726719279 >> v2:172.21.15.137:3300/0 conn(0x7fe32c15e1e0 msgr2=0x7fe32c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.913+0000 7fe321e00640 1 --2- 172.21.15.125:0/3726719279 >> v2:172.21.15.137:3300/0 conn(0x7fe32c15e1e0 0x7fe32c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.913+0000 7fe321e00640 1 -- 172.21.15.125:0/3726719279 >> v1:172.21.15.125:6789/0 conn(0x7fe32c143ae0 legacy=0x7fe32c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.913+0000 7fe321e00640 1 -- 172.21.15.125:0/3726719279 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c0b2130 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.914+0000 7fe321e00640 1 --2- 172.21.15.125:0/3726719279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cf9a0 0x7fe32c15dca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0ae9d0 tx=0x7fe31402dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5437276111196205 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.914+0000 7fe32a000640 1 -- 172.21.15.125:0/3726719279 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31402f8a0 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.914+0000 7fe32a000640 1 -- 172.21.15.125:0/3726719279 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31402fa00 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.914+0000 7fe32a000640 1 -- 172.21.15.125:0/3726719279 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31402fd90 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 -- 172.21.15.125:0/3726719279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cf9a0 msgr2=0x7fe32c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 --2- 172.21.15.125:0/3726719279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cf9a0 0x7fe32c15dca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0ae9d0 tx=0x7fe31402dae0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 -- 172.21.15.125:0/3726719279 shutdown_connections 2024-01-16T15:03:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 --2- 172.21.15.125:0/3726719279 >> v2:172.21.15.137:3300/0 conn(0x7fe32c15e1e0 0x7fe32c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 --2- 172.21.15.125:0/3726719279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cf9a0 0x7fe32c15dca0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 -- 172.21.15.125:0/3726719279 >> 172.21.15.125:0/3726719279 conn(0x7fe32c06a800 msgr2=0x7fe32c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 -- 172.21.15.125:0/3726719279 shutdown_connections 2024-01-16T15:03:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.915+0000 7fe331c00640 1 -- 172.21.15.125:0/3726719279 wait complete. 2024-01-16T15:03:16.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.918+0000 7fe331c00640 1 Processor -- start 2024-01-16T15:03:16.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.918+0000 7fe331c00640 1 -- start start 2024-01-16T15:03:16.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 0x7fe32c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 0x7fe32c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0ae540 con 0x7fe32c15e1e0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0ad930 con 0x7fe32c143ae0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe331c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0adc10 con 0x7fe32c0cf9a0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe32b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe32b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45216/0 (socket says 172.21.15.125:45216) 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.919+0000 7fe32b400640 1 -- 172.21.15.125:0/2732503206 learned_addr learned my addr 172.21.15.125:0/2732503206 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe321e00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 0x7fe32c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32aa00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 0x7fe32c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 -- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 msgr2=0x7fe32c1f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 0x7fe32c1f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 -- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 msgr2=0x7fe32c0acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 0x7fe32c0acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 -- 172.21.15.125:0/2732503206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314009400 con 0x7fe32c15e1e0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe321e00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 0x7fe32c0acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32aa00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 0x7fe32c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.920+0000 7fe32b400640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe324007e40 tx=0x7fe32400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.921+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324014620 con 0x7fe32c15e1e0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.921+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32400ee90 con 0x7fe32c15e1e0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.921+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240149e0 con 0x7fe32c15e1e0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.921+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c0adef0 con 0x7fe32c15e1e0 2024-01-16T15:03:16.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.921+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c0cc8c0 con 0x7fe32c15e1e0 2024-01-16T15:03:16.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.922+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c0ae6b0 con 0x7fe32c15e1e0 2024-01-16T15:03:16.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.923+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fe324014b40 con 0x7fe32c15e1e0 2024-01-16T15:03:16.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.924+0000 7fe323400640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 0x7fe30406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.924+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe32408b680 con 0x7fe32c15e1e0 2024-01-16T15:03:16.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.924+0000 7fe32aa00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 0x7fe30406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.925+0000 7fe32aa00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 0x7fe30406ded0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005880 tx=0x7fe31c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:16.928+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe324088c80 con 0x7fe32c15e1e0 2024-01-16T15:03:17.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.154+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27071", "target": ["mon-mgr", ""]}) v1 -- 0x7fe32c071aa0 con 0x7fe30406ba10 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.174+0000 7fe323400640 1 -- 172.21.15.125:0/2732503206 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fe32c071aa0 con 0x7fe30406ba10 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:03:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 0.0, 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "0/3", 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:17.154 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:17.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 msgr2=0x7fe30406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 0x7fe30406ded0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005880 tx=0x7fe31c009040 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 msgr2=0x7fe32c0ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe324007e40 tx=0x7fe32400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 shutdown_connections 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe30406ba10 0x7fe30406ded0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c15e1e0 0x7fe32c0ad300 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32c143ae0 0x7fe32c0acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 --2- 172.21.15.125:0/2732503206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32c0cf9a0 0x7fe32c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 >> 172.21.15.125:0/2732503206 conn(0x7fe32c06a800 msgr2=0x7fe32c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 shutdown_connections 2024-01-16T15:03:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.177+0000 7fe331c00640 1 -- 172.21.15.125:0/2732503206 wait complete. 2024-01-16T15:03:17.176 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:03:17.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27077 2024-01-16T15:03:17.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.409+0000 7f3165c00640 1 Processor -- start 2024-01-16T15:03:17.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.409+0000 7f3165c00640 1 -- start start 2024-01-16T15:03:17.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3165c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f316015e3b0 0x7f3160164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3165c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600b1ee0 con 0x7f31600cf9a0 2024-01-16T15:03:17.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3165c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600b2780 con 0x7f3160143ae0 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3165c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316015e990 con 0x7f316015e3b0 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3155e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f31600cf9a0 0x7f316015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60690/0 (socket says 172.21.15.125:60690) 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.411+0000 7f3155e00640 1 -- 172.21.15.125:0/3833612324 learned_addr learned my addr 172.21.15.125:0/3833612324 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315f400640 1 --2- 172.21.15.125:0/3833612324 >> v2:172.21.15.125:3300/0 conn(0x7f316015e3b0 0x7f3160164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315e000640 1 -- 172.21.15.125:0/3833612324 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728251566 0 0) 0x7f31600b1ee0 con 0x7f31600cf9a0 2024-01-16T15:03:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315e000640 1 -- 172.21.15.125:0/3833612324 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3140003520 con 0x7f31600cf9a0 2024-01-16T15:03:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315f400640 1 -- 172.21.15.125:0/3833612324 >> v1:172.21.15.137:6789/0 conn(0x7f3160143ae0 legacy=0x7f31600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315f400640 1 -- 172.21.15.125:0/3833612324 >> v1:172.21.15.125:6789/0 conn(0x7f31600cf9a0 legacy=0x7f316015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.412+0000 7f315f400640 1 -- 172.21.15.125:0/3833612324 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600b2130 con 0x7f316015e3b0 2024-01-16T15:03:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.413+0000 7f315f400640 1 --2- 172.21.15.125:0/3833612324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316015e3b0 0x7f3160164de0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3158009d90 tx=0x7f315802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49782cbc3ba2744d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.413+0000 7f315e000640 1 -- 172.21.15.125:0/3833612324 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315802ccf0 con 0x7f316015e3b0 2024-01-16T15:03:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.413+0000 7f315e000640 1 -- 172.21.15.125:0/3833612324 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315802ce50 con 0x7f316015e3b0 2024-01-16T15:03:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.413+0000 7f315e000640 1 -- 172.21.15.125:0/3833612324 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158034660 con 0x7f316015e3b0 2024-01-16T15:03:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 -- 172.21.15.125:0/3833612324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316015e3b0 msgr2=0x7f3160164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 --2- 172.21.15.125:0/3833612324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316015e3b0 0x7f3160164de0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3158009d90 tx=0x7f315802bf40 comp rx=0 tx=0).stop 2024-01-16T15:03:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 -- 172.21.15.125:0/3833612324 shutdown_connections 2024-01-16T15:03:17.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 --2- 172.21.15.125:0/3833612324 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316015e3b0 0x7f3160164de0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 -- 172.21.15.125:0/3833612324 >> 172.21.15.125:0/3833612324 conn(0x7f316006a800 msgr2=0x7f316006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 -- 172.21.15.125:0/3833612324 shutdown_connections 2024-01-16T15:03:17.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.414+0000 7f3165c00640 1 -- 172.21.15.125:0/3833612324 wait complete. 2024-01-16T15:03:17.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.416+0000 7f3165c00640 1 Processor -- start 2024-01-16T15:03:17.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.416+0000 7f3165c00640 1 -- start start 2024-01-16T15:03:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 0x7f3160163800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 0x7f316015a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3160164ce0 con 0x7f3160143ae0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31600ce180 con 0x7f31600cf9a0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3165c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31600ce610 con 0x7f316015e3b0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3155e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3155e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45244/0 (socket says 172.21.15.125:45244) 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.417+0000 7f3155e00640 1 -- 172.21.15.125:0/2629483833 learned_addr learned my addr 172.21.15.125:0/2629483833 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f315ea00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 0x7f3160163800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f315f400640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 0x7f316015a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 -- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 msgr2=0x7f316015a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 0x7f316015a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 -- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 msgr2=0x7f3160163800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 0x7f3160163800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 -- 172.21.15.125:0/2629483833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009a70 con 0x7f3160143ae0 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f315f400640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 0x7f316015a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f315ea00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 0x7f3160163800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3155e00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f314800c500 tx=0x7f314800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.418+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148005ce0 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.419+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148005e40 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.419+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148007330 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.419+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3160163ec0 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.419+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31600cde10 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.420+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31600cbf50 con 0x7f3160143ae0 2024-01-16T15:03:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.421+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f3148007490 con 0x7f3160143ae0 2024-01-16T15:03:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.421+0000 7f3157400640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 0x7f313806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.422+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f314808a340 con 0x7f3160143ae0 2024-01-16T15:03:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.422+0000 7f315ea00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 0x7f313806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.422+0000 7f315ea00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 0x7f313806df00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3150000f60 tx=0x7f3150009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.426+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3148087890 con 0x7f3160143ae0 2024-01-16T15:03:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.671+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27077", "target": ["mon-mgr", ""]}) v1 -- 0x7f3160165320 con 0x7f313806ba40 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.682+0000 7f3157400640 1 -- 172.21.15.125:0/2629483833 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f3150001d30 con 0x7f313806ba40 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:03:17.661 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:17.662 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:17.662 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 msgr2=0x7f313806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 0x7f313806df00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3150000f60 tx=0x7f3150009040 comp rx=0 tx=0).stop 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 msgr2=0x7f31601647a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f314800c500 tx=0x7f314800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 shutdown_connections 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316015e3b0 0x7f316015a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f313806ba40 0x7f313806df00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3160143ae0 0x7f31601647a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 --2- 172.21.15.125:0/2629483833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31600cf9a0 0x7f3160163800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.685+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 >> 172.21.15.125:0/2629483833 conn(0x7f316006a800 msgr2=0x7f31600cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.686+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 shutdown_connections 2024-01-16T15:03:17.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:17.686+0000 7f3165c00640 1 -- 172.21.15.125:0/2629483833 wait complete. 2024-01-16T15:03:17.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:18.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.042+0000 7fe71cc00640 1 Processor -- start 2024-01-16T15:03:18.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.042+0000 7fe71cc00640 1 -- start start 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.045+0000 7fe71cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181610e0 0x7fe7180d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.045+0000 7fe71cc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7180b2920 con 0x7fe71815cc90 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.045+0000 7fe71cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7180cb670 con 0x7fe7180cf4c0 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.045+0000 7fe71cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180cb950 con 0x7fe7181610e0 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.046+0000 7fe715400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe7180cf4c0 0x7fe7181614c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:33064/0 (socket says 172.21.15.125:33064) 2024-01-16T15:03:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.046+0000 7fe715400640 1 -- 172.21.15.125:0/585886397 learned_addr learned my addr 172.21.15.125:0/585886397 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.047+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169598063 0 0) 0x7fe7180cb670 con 0x7fe7180cf4c0 2024-01-16T15:03:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.047+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f8003520 con 0x7fe7180cf4c0 2024-01-16T15:03:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560764775 0 0) 0x7fe7180b2920 con 0x7fe71815cc90 2024-01-16T15:03:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7180cb670 con 0x7fe71815cc90 2024-01-16T15:03:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2195031024 0 0) 0x7fe6f8003520 con 0x7fe7180cf4c0 2024-01-16T15:03:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181610e0 msgr2=0x7fe7180d0340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181610e0 0x7fe7180d0340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.049+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 >> v1:172.21.15.125:6789/0 conn(0x7fe71815cc90 legacy=0x7fe71815d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180cbc30 con 0x7fe7180cf4c0 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe7000015f0 con 0x7fe7180cf4c0 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 >> v1:172.21.15.137:6789/0 conn(0x7fe7180cf4c0 legacy=0x7fe7181614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6f8003b00 0x7fe6f8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80047c0 0x7fe6f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.050+0000 7fe714a00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f80051f0 0x7fe6f8014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.051+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180cb950 con 0x7fe6f80051f0 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.051+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8005950 con 0x7fe6f80047c0 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.051+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8005b10 con 0x7fe6f8003b00 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.051+0000 7fe716800640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f80051f0 0x7fe6f8014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe715e00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6f8003b00 0x7fe6f8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe715400640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80047c0 0x7fe6f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe716800640 1 -- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6f8003b00 msgr2=0x7fe6f8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe716800640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6f8003b00 0x7fe6f8003f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe716800640 1 -- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80047c0 msgr2=0x7fe6f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe716800640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80047c0 0x7fe6f8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.052+0000 7fe716800640 1 -- 172.21.15.125:0/585886397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180cbc30 con 0x7fe6f80051f0 2024-01-16T15:03:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.054+0000 7fe716800640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f80051f0 0x7fe6f8014190 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe710002190 tx=0x7fe6f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e54175a2c5c04e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:18.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.055+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71002c880 con 0x7fe6f80051f0 2024-01-16T15:03:18.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.055+0000 7fe714a00640 1 -- 172.21.15.125:0/585886397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71002c9e0 con 0x7fe6f80051f0 2024-01-16T15:03:18.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.056+0000 7fe71cc00640 1 -- 172.21.15.125:0/585886397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f80051f0 msgr2=0x7fe6f8014190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.056+0000 7fe71cc00640 1 --2- 172.21.15.125:0/585886397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f80051f0 0x7fe6f8014190 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe710002190 tx=0x7fe6f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.057+0000 7fe716800640 1 -- 172.21.15.125:0/585886397 reap_dead start 2024-01-16T15:03:18.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.058+0000 7fe71cc00640 1 -- 172.21.15.125:0/585886397 shutdown_connections 2024-01-16T15:03:18.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.058+0000 7fe71cc00640 1 -- 172.21.15.125:0/585886397 >> 172.21.15.125:0/585886397 conn(0x7fe7180d1020 msgr2=0x7fe7180d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:18.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.064+0000 7fe71cc00640 1 -- 172.21.15.125:0/585886397 shutdown_connections 2024-01-16T15:03:18.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.064+0000 7fe71cc00640 1 -- 172.21.15.125:0/585886397 wait complete. 2024-01-16T15:03:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.071+0000 7fe71cc00640 1 Processor -- start 2024-01-16T15:03:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.071+0000 7fe71cc00640 1 -- start start 2024-01-16T15:03:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.074+0000 7fe71cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80051f0 0x7fe7181519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.075+0000 7fe71cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181520c0 0x7fe7180b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.075+0000 7fe71cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe716800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe716800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45262/0 (socket says 172.21.15.125:45262) 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe71cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180cb280 con 0x7fe718152610 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe71cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe718151f30 con 0x7fe6f80051f0 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe71cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b3b70 con 0x7fe7181520c0 2024-01-16T15:03:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.076+0000 7fe716800640 1 -- 172.21.15.125:0/2092560143 learned_addr learned my addr 172.21.15.125:0/2092560143 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:18.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.077+0000 7fe715e00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80051f0 0x7fe7181519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.078+0000 7fe716800640 1 -- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181520c0 msgr2=0x7fe7180b5d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:03:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.078+0000 7fe716800640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181520c0 0x7fe7180b5d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.078+0000 7fe716800640 1 -- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80051f0 msgr2=0x7fe7181519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.078+0000 7fe716800640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80051f0 0x7fe7181519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.078+0000 7fe716800640 1 -- 172.21.15.125:0/2092560143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180b6810 con 0x7fe718152610 2024-01-16T15:03:18.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.079+0000 7fe716800640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe7100021f0 tx=0x7fe710001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.080+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710045070 con 0x7fe718152610 2024-01-16T15:03:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.080+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710034b50 con 0x7fe718152610 2024-01-16T15:03:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.080+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710006b50 con 0x7fe718152610 2024-01-16T15:03:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.082+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7180b3dc0 con 0x7fe718152610 2024-01-16T15:03:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.083+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7180b4380 con 0x7fe718152610 2024-01-16T15:03:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.084+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7180675d0 con 0x7fe718152610 2024-01-16T15:03:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.087+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fe71003b070 con 0x7fe718152610 2024-01-16T15:03:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.088+0000 7fe70e000640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 0x7fe6f006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.089+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7100a9cb0 con 0x7fe718152610 2024-01-16T15:03:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.090+0000 7fe715e00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 0x7fe6f006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.098+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7100a99a0 con 0x7fe718152610 2024-01-16T15:03:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.100+0000 7fe715e00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 0x7fe6f006df20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.695+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe7180675d0 con 0x7fe718152610 2024-01-16T15:03:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.698+0000 7fe70e000640 1 -- 172.21.15.125:0/2092560143 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fe710039070 con 0x7fe718152610 2024-01-16T15:03:18.678 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:18.678 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:18.678 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:18.678 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:18.678 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:18.679 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 417, 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:18.680 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:18.681 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 800, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222213199, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6756626432, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766337486848, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 341, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 10491, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 3 2024-01-16T15:03:18.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:18.683 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:18.684 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:18.685 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:18.686 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.706+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 msgr2=0x7fe6f006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.706+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 0x7fe6f006df20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708008040 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.706+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 msgr2=0x7fe7180b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.706+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe7100021f0 tx=0x7fe710001630 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 shutdown_connections 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe6f006ba60 0x7fe6f006df20 secure :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708008040 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe718152610 0x7fe7180b62d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7181520c0 0x7fe7180b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 --2- 172.21.15.125:0/2092560143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6f80051f0 0x7fe7181519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.707+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 >> 172.21.15.125:0/2092560143 conn(0x7fe7180d1020 msgr2=0x7fe71806cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:18.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.709+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 shutdown_connections 2024-01-16T15:03:18.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:18.710+0000 7fe71cc00640 1 -- 172.21.15.125:0/2092560143 wait complete. 2024-01-16T15:03:19.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:20.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.148+0000 7f7995000640 1 Processor -- start 2024-01-16T15:03:20.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.149+0000 7f7995000640 1 -- start start 2024-01-16T15:03:20.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.152+0000 7f7995000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:20.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.152+0000 7f7995000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7988052670 con 0x7f7988050710 2024-01-16T15:03:20.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.152+0000 7f7995000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7988053820 con 0x7f79880541d0 2024-01-16T15:03:20.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.152+0000 7f7995000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7988050b90 con 0x7f79880df740 2024-01-16T15:03:20.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.155+0000 7f798e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.155+0000 7f798e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:51748/0 (socket says 172.21.15.125:51748) 2024-01-16T15:03:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.155+0000 7f798e800640 1 -- 172.21.15.125:0/3991672275 learned_addr learned my addr 172.21.15.125:0/3991672275 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.156+0000 7f798e800640 1 -- 172.21.15.125:0/3991672275 >> v1:172.21.15.137:6789/0 conn(0x7f79880541d0 legacy=0x7f7988053110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:03:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.156+0000 7f798e800640 1 -- 172.21.15.125:0/3991672275 >> v1:172.21.15.125:6789/0 conn(0x7f7988050710 legacy=0x7f79880384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.156+0000 7f798e800640 1 -- 172.21.15.125:0/3991672275 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7988053c50 con 0x7f79880df740 2024-01-16T15:03:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.159+0000 7f798e800640 1 --2- 172.21.15.125:0/3991672275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7990072da0 tx=0x7f79900ae6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e6c846d9349cc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.162+0000 7f798ca00640 1 -- 172.21.15.125:0/3991672275 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79900b37d0 con 0x7f79880df740 2024-01-16T15:03:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.162+0000 7f798ca00640 1 -- 172.21.15.125:0/3991672275 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79900b1050 con 0x7f79880df740 2024-01-16T15:03:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.162+0000 7f798ca00640 1 -- 172.21.15.125:0/3991672275 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79900b3bb0 con 0x7f79880df740 2024-01-16T15:03:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.165+0000 7f7995000640 1 -- 172.21.15.125:0/3991672275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 msgr2=0x7f7988038c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.165+0000 7f7995000640 1 --2- 172.21.15.125:0/3991672275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7990072da0 tx=0x7f79900ae6b0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.166+0000 7f7995000640 1 -- 172.21.15.125:0/3991672275 shutdown_connections 2024-01-16T15:03:20.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.166+0000 7f7995000640 1 --2- 172.21.15.125:0/3991672275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880df740 0x7f7988038c00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.166+0000 7f7995000640 1 -- 172.21.15.125:0/3991672275 >> 172.21.15.125:0/3991672275 conn(0x7f7988009f80 msgr2=0x7f798800a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:20.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.167+0000 7f7995000640 1 -- 172.21.15.125:0/3991672275 shutdown_connections 2024-01-16T15:03:20.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.167+0000 7f7995000640 1 -- 172.21.15.125:0/3991672275 wait complete. 2024-01-16T15:03:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.170+0000 7f7995000640 1 Processor -- start 2024-01-16T15:03:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.171+0000 7f7995000640 1 -- start start 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880541d0 0x7f79880d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79880df740 0x7f798804e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79880e2760 con 0x7f7988050710 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79880dff00 con 0x7f79880df740 2024-01-16T15:03:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.172+0000 7f7995000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798804e970 con 0x7f79880541d0 2024-01-16T15:03:20.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.173+0000 7f798de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:20.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.174+0000 7f798de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45272/0 (socket says 172.21.15.125:45272) 2024-01-16T15:03:20.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.174+0000 7f798de00640 1 -- 172.21.15.125:0/2965873370 learned_addr learned my addr 172.21.15.125:0/2965873370 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798d400640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880541d0 0x7f79880d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798e800640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79880df740 0x7f798804e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 -- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880541d0 msgr2=0x7f79880d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880541d0 0x7f79880d1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 -- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79880df740 msgr2=0x7f798804e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79880df740 0x7f798804e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 -- 172.21.15.125:0/2965873370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799004f090 con 0x7f7988050710 2024-01-16T15:03:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.175+0000 7f798de00640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f797800b450 tx=0x7f797800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.177+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978011890 con 0x7f7988050710 2024-01-16T15:03:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.177+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978011a10 con 0x7f7988050710 2024-01-16T15:03:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.177+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978011dc0 con 0x7f7988050710 2024-01-16T15:03:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.177+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79880e1ae0 con 0x7f7988050710 2024-01-16T15:03:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.177+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79880e20a0 con 0x7f7988050710 2024-01-16T15:03:20.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.181+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7988005360 con 0x7f7988050710 2024-01-16T15:03:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.186+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f797801a400 con 0x7f7988050710 2024-01-16T15:03:20.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.188+0000 7f7986000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 0x7f796806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:20.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.188+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7978088f00 con 0x7f7988050710 2024-01-16T15:03:20.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.191+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7978088bf0 con 0x7f7988050710 2024-01-16T15:03:20.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.228+0000 7f798d400640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 0x7f796806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:20.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.247+0000 7f798d400640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 0x7f796806ded0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f798804ba30 tx=0x7f7980008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:20.292 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:20.313+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:20.293 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:20.313+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:20.293 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:20.313+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:20.293 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:20.313+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:20.293 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:20.313+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:20.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.764+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f79880074c0 con 0x7f7988050710 2024-01-16T15:03:20.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.765+0000 7f7986000640 1 -- 172.21.15.125:0/2965873370 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2348 (secure 0 0 0) 0x7f7978015020 con 0x7f7988050710 2024-01-16T15:03:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 419, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1058, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 4287768251, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 8365178880, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 764728934400, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 97132475, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 88776036, 2024-01-16T15:03:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 24, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 29 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:20.754 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:20.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 msgr2=0x7f796806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:20.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 0x7f796806ded0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f798804ba30 tx=0x7f7980008040 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 msgr2=0x7f79880e1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f797800b450 tx=0x7f797800b910 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 shutdown_connections 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79880df740 0x7f798804e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79880541d0 0x7f79880d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f796806ba10 0x7f796806ded0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 --2- 172.21.15.125:0/2965873370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7988050710 0x7f79880e1500 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 >> 172.21.15.125:0/2965873370 conn(0x7f7988009f80 msgr2=0x7f79880d8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.767+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 shutdown_connections 2024-01-16T15:03:20.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:20.768+0000 7f7995000640 1 -- 172.21.15.125:0/2965873370 wait complete. 2024-01-16T15:03:22.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:23.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.176+0000 7f1cc4a00640 1 Processor -- start 2024-01-16T15:03:23.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.178+0000 7f1cc4a00640 1 -- start start 2024-01-16T15:03:23.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.183+0000 7f1cc4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cbc14b2c0 0x7f1cbc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.184+0000 7f1cc4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.184+0000 7f1cc4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cbc0cfbf0 con 0x7f1cbc070ea0 2024-01-16T15:03:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.184+0000 7f1cc4a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cbc14b6a0 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.185+0000 7f1cc4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc071690 con 0x7f1cbc0cf7d0 2024-01-16T15:03:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.185+0000 7f1cc2000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cbc14b2c0 0x7f1cbc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.185+0000 7f1cc2a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f1cbc070ea0 0x7f1cbc0b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:33080/0 (socket says 172.21.15.125:33080) 2024-01-16T15:03:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.185+0000 7f1cc2a00640 1 -- 172.21.15.125:0/1698542861 learned_addr learned my addr 172.21.15.125:0/1698542861 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:23.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.187+0000 7f1cc2000640 1 -- 172.21.15.125:0/1698542861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 msgr2=0x7f1cbc070960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:03:23.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.191+0000 7f1cc1600640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.192+0000 7f1cc2000640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.192+0000 7f1cc2000640 1 -- 172.21.15.125:0/1698542861 >> v1:172.21.15.137:6789/0 conn(0x7f1cbc070ea0 legacy=0x7f1cbc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:23.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.192+0000 7f1cc2000640 1 -- 172.21.15.125:0/1698542861 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc071970 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.197+0000 7f1cc1600640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:23.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.203+0000 7f1cc2000640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc0cf290 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009d90 tx=0x7f1cb402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b498a25251a880 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:23.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.204+0000 7f1cc0c00640 1 -- 172.21.15.125:0/1698542861 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb402cbf0 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.206+0000 7f1cc0c00640 1 -- 172.21.15.125:0/1698542861 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb402cd50 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.206+0000 7f1cc0c00640 1 -- 172.21.15.125:0/1698542861 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4034660 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.213+0000 7f1cc4a00640 1 -- 172.21.15.125:0/1698542861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 msgr2=0x7f1cbc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:23.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.213+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc0cf290 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009d90 tx=0x7f1cb402be40 comp rx=0 tx=0).stop 2024-01-16T15:03:23.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.216+0000 7f1cc4a00640 1 -- 172.21.15.125:0/1698542861 shutdown_connections 2024-01-16T15:03:23.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.216+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.216+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/1698542861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc0cf290 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.216+0000 7f1cc4a00640 1 -- 172.21.15.125:0/1698542861 >> 172.21.15.125:0/1698542861 conn(0x7f1cbc06a800 msgr2=0x7f1cbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:23.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.223+0000 7f1cc4a00640 1 -- 172.21.15.125:0/1698542861 shutdown_connections 2024-01-16T15:03:23.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.224+0000 7f1cc4a00640 1 -- 172.21.15.125:0/1698542861 wait complete. 2024-01-16T15:03:23.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.228+0000 7f1cc4a00640 1 Processor -- start 2024-01-16T15:03:23.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.228+0000 7f1cc4a00640 1 -- start start 2024-01-16T15:03:23.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.231+0000 7f1cc4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.231+0000 7f1cc4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.232+0000 7f1cc4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.232+0000 7f1cc2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.232+0000 7f1cc2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:49384/0 (socket says 172.21.15.125:49384) 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.232+0000 7f1cc1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.233+0000 7f1cc1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:51794/0 (socket says 172.21.15.125:51794) 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.233+0000 7f1cc1600640 1 -- 172.21.15.125:0/508455185 learned_addr learned my addr 172.21.15.125:0/508455185 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.233+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc1522d0 con 0x7f1cbc14b2c0 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.234+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc152590 con 0x7f1cbc070ea0 2024-01-16T15:03:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.234+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc152870 con 0x7f1cbc0cf7d0 2024-01-16T15:03:23.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.234+0000 7f1cc2a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc2000640 1 -- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 msgr2=0x7f1cbc0adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:23.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc2000640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc2000640 1 -- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 msgr2=0x7f1cbc1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:23.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc2000640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc2000640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb4009a70 con 0x7f1cbc070ea0 2024-01-16T15:03:23.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.235+0000 7f1cc1600640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:23.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.236+0000 7f1cc2a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:23.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.236+0000 7f1cc2000640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4033000 tx=0x7f1cb4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:23.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.237+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb403e820 con 0x7f1cbc070ea0 2024-01-16T15:03:23.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.238+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb403e980 con 0x7f1cbc070ea0 2024-01-16T15:03:23.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.238+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb403ecf0 con 0x7f1cbc070ea0 2024-01-16T15:03:23.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.238+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc152b50 con 0x7f1cbc070ea0 2024-01-16T15:03:23.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.238+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc0aaf20 con 0x7f1cbc070ea0 2024-01-16T15:03:23.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.244+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c005250 con 0x7f1cbc070ea0 2024-01-16T15:03:23.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.246+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 18) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f1cb403b050 con 0x7f1cbc070ea0 2024-01-16T15:03:23.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.248+0000 7f1cba000640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 0x7f1c9406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:23.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.249+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cb40aacb0 con 0x7f1cbc070ea0 2024-01-16T15:03:23.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.252+0000 7f1cc1600640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 0x7f1c9406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:23.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.254+0000 7f1cc1600640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 0x7f1c9406e030 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1552a0 tx=0x7f1ca4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:23.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.277+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cb4077720 con 0x7f1cbc070ea0 2024-01-16T15:03:23.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.879+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1c8c0057d0 con 0x7f1cbc070ea0 2024-01-16T15:03:23.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.883+0000 7f1cba000640 1 -- 172.21.15.125:0/508455185 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2421 (secure 0 0 0) 0x7f1cb4039990 con 0x7f1cbc070ea0 2024-01-16T15:03:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:23.864 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 422, 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1058, 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 4287768251, 2024-01-16T15:03:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 8365178880, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 764728934400, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 97124660, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 88764699, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 24, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 28 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:23.874 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.890+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 msgr2=0x7f1c9406e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.890+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 0x7f1c9406e030 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1552a0 tx=0x7f1ca4008040 comp rx=0 tx=0).stop 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.890+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 msgr2=0x7f1cbc0ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.890+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4033000 tx=0x7f1cb4004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 shutdown_connections 2024-01-16T15:03:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c9406bb70 0x7f1c9406e030 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cbc14b2c0 0x7f1cbc1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cbc0cf7d0 0x7f1cbc0adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 --2- 172.21.15.125:0/508455185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cbc070ea0 0x7f1cbc0ad730 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.893+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 >> 172.21.15.125:0/508455185 conn(0x7f1cbc06a800 msgr2=0x7f1cbc0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.894+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 shutdown_connections 2024-01-16T15:03:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:23.894+0000 7f1cc4a00640 1 -- 172.21.15.125:0/508455185 wait complete. 2024-01-16T15:03:25.358 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:25.378+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:25.358 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:25.378+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:25.358 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:25.378+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:25.358 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:25.378+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:25.358 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:03:25.378+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:25.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:26.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.399+0000 7fcb0da00640 1 Processor -- start 2024-01-16T15:03:26.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.399+0000 7fcb0da00640 1 -- start start 2024-01-16T15:03:26.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.402+0000 7fcb0da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb0814b2c0 0x7fcb080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.402+0000 7fcb0da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcb080cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.402+0000 7fcb0da00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080cfbf0 con 0x7fcb080cf7d0 2024-01-16T15:03:26.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.402+0000 7fcb0da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0814b6a0 con 0x7fcb0814b2c0 2024-01-16T15:03:26.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.402+0000 7fcb0da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08071690 con 0x7fcb08071070 2024-01-16T15:03:26.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.403+0000 7fcb06000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb080cf7d0 0x7fcb08070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60724/0 (socket says 172.21.15.125:60724) 2024-01-16T15:03:26.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.403+0000 7fcb06000640 1 -- 172.21.15.125:0/335542289 learned_addr learned my addr 172.21.15.125:0/335542289 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:26.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.403+0000 7fcb06a00640 1 --2- 172.21.15.125:0/335542289 >> v2:172.21.15.125:3300/0 conn(0x7fcb0814b2c0 0x7fcb080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.404+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672401160 0 0) 0x7fcb080cfbf0 con 0x7fcb080cf7d0 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.404+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae8003520 con 0x7fcb080cf7d0 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410282727 0 0) 0x7fcae8003520 con 0x7fcb080cf7d0 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 msgr2=0x7fcb080cc160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcb080cc160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 >> v2:172.21.15.125:3300/0 conn(0x7fcb0814b2c0 msgr2=0x7fcb080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 --2- 172.21.15.125:0/335542289 >> v2:172.21.15.125:3300/0 conn(0x7fcb0814b2c0 0x7fcb080cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.405+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08071970 con 0x7fcb080cf7d0 2024-01-16T15:03:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcaf0001670 con 0x7fcb080cf7d0 2024-01-16T15:03:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 >> v1:172.21.15.125:6789/0 conn(0x7fcb080cf7d0 legacy=0x7fcb08070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcae8003970 0x7fcae8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcae80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0814b6a0 con 0x7fcae8004d00 2024-01-16T15:03:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08071690 con 0x7fcae8003970 2024-01-16T15:03:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.406+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae800bb10 con 0x7fcb08071070 2024-01-16T15:03:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.407+0000 7fcb07400640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.409+0000 7fcb06a00640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcae8003970 0x7fcae8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb07400640 1 -- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 msgr2=0x7fcae80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.410+0000 7fcb06000640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcae80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb07400640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcae80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb07400640 1 -- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcae8003970 msgr2=0x7fcae8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb07400640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcae8003970 0x7fcae8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb07400640 1 -- 172.21.15.125:0/335542289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae8003520 con 0x7fcae8004d00 2024-01-16T15:03:26.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.411+0000 7fcb06000640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08071070 0x7fcae80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.415+0000 7fcb07400640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009e60 tx=0x7fcae80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb17d817e89e7cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.418+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0002cd90 con 0x7fcae8004d00 2024-01-16T15:03:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.418+0000 7fcb05600640 1 -- 172.21.15.125:0/335542289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00034440 con 0x7fcae8004d00 2024-01-16T15:03:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.419+0000 7fcb0da00640 1 -- 172.21.15.125:0/335542289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 msgr2=0x7fcae8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.419+0000 7fcb0da00640 1 --2- 172.21.15.125:0/335542289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009e60 tx=0x7fcae80057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.420+0000 7fcb07400640 1 -- 172.21.15.125:0/335542289 reap_dead start 2024-01-16T15:03:26.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.422+0000 7fcb0da00640 1 -- 172.21.15.125:0/335542289 shutdown_connections 2024-01-16T15:03:26.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.422+0000 7fcb0da00640 1 -- 172.21.15.125:0/335542289 >> 172.21.15.125:0/335542289 conn(0x7fcb0806a800 msgr2=0x7fcb0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.423+0000 7fcb0da00640 1 -- 172.21.15.125:0/335542289 shutdown_connections 2024-01-16T15:03:26.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.423+0000 7fcb0da00640 1 -- 172.21.15.125:0/335542289 wait complete. 2024-01-16T15:03:26.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.428+0000 7fcb0da00640 1 Processor -- start 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.429+0000 7fcb0da00640 1 -- start start 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.431+0000 7fcb0da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcb0813f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb0da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb08162b60 0x7fcb08145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb0da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb0da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08146050 con 0x7fcae8004d00 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb0da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0813f640 con 0x7fcb08162b60 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb0da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb081630d0 con 0x7fcb08145bd0 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb07400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb07400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:51808/0 (socket says 172.21.15.125:51808) 2024-01-16T15:03:26.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.432+0000 7fcb07400640 1 -- 172.21.15.125:0/1313996590 learned_addr learned my addr 172.21.15.125:0/1313996590 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:26.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb06a00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcb0813f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb07400640 1 -- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb08162b60 msgr2=0x7fcb08145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb07400640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb08162b60 0x7fcb08145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb07400640 1 -- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 msgr2=0x7fcb0813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb07400640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcb0813f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb07400640 1 -- 172.21.15.125:0/1313996590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb081633b0 con 0x7fcb08145bd0 2024-01-16T15:03:26.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.434+0000 7fcb06a00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcb0813f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:26.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.438+0000 7fcb07400640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009f80 tx=0x7fcb00004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.442+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0003f630 con 0x7fcb08145bd0 2024-01-16T15:03:26.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.442+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0003f7b0 con 0x7fcb08145bd0 2024-01-16T15:03:26.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.442+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0003fb60 con 0x7fcb08145bd0 2024-01-16T15:03:26.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.442+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb08163690 con 0x7fcb08145bd0 2024-01-16T15:03:26.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.443+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb081436e0 con 0x7fcb08145bd0 2024-01-16T15:03:26.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.445+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb080675d0 con 0x7fcb08145bd0 2024-01-16T15:03:26.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.451+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fcb00002040 con 0x7fcb08145bd0 2024-01-16T15:03:26.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.454+0000 7fcafea00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 0x7fcae006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.454+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb0007b730 con 0x7fcb08145bd0 2024-01-16T15:03:26.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.464+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb00078200 con 0x7fcb08145bd0 2024-01-16T15:03:26.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.466+0000 7fcb06a00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 0x7fcae006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.470+0000 7fcb06a00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 0x7fcae006e020 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb08161910 tx=0x7fcaf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.985+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcb0806dff0 con 0x7fcb08145bd0 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.986+0000 7fcafea00640 1 -- 172.21.15.125:0/1313996590 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2424 (secure 0 0 0) 0x7fcb00078200 con 0x7fcb08145bd0 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 426, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1536, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 6296964876, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 12502827008, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 760591286272, 2024-01-16T15:03:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 272849821, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 256116805, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 66, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 72 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-16T15:03:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:26.972 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 msgr2=0x7fcae006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 0x7fcae006e020 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb08161910 tx=0x7fcaf8008040 comp rx=0 tx=0).stop 2024-01-16T15:03:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 msgr2=0x7fcb081431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009f80 tx=0x7fcb00004a80 comp rx=0 tx=0).stop 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 shutdown_connections 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb08145bd0 0x7fcb081431a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb08162b60 0x7fcb08145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcae006bb60 0x7fcae006e020 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 --2- 172.21.15.125:0/1313996590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae8004d00 0x7fcb0813f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 >> 172.21.15.125:0/1313996590 conn(0x7fcb0806a800 msgr2=0x7fcb08148200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 shutdown_connections 2024-01-16T15:03:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:26.989+0000 7fcb0da00640 1 -- 172.21.15.125:0/1313996590 wait complete. 2024-01-16T15:03:28.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.442+0000 7fb276e00640 1 Processor -- start 2024-01-16T15:03:29.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.443+0000 7fb276e00640 1 -- start start 2024-01-16T15:03:29.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.446+0000 7fb276e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb27014b1a0 0x7fb2700b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb276e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27006aaf0 0x7fb27006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb276e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27006e870 con 0x7fb2701652a0 2024-01-16T15:03:29.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb276e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2700d0d40 con 0x7fb27014b1a0 2024-01-16T15:03:29.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb276e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27014b580 con 0x7fb27006aaf0 2024-01-16T15:03:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb26fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb27014b1a0 0x7fb2700b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb26fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb27014b1a0 0x7fb2700b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54492/0 (socket says 172.21.15.125:54492) 2024-01-16T15:03:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.447+0000 7fb26fe00640 1 -- 172.21.15.125:0/2333019136 learned_addr learned my addr 172.21.15.125:0/2333019136 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.448+0000 7fb274e00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27006aaf0 0x7fb27006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.448+0000 7fb26fe00640 1 -- 172.21.15.125:0/2333019136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27006aaf0 msgr2=0x7fb27006ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.448+0000 7fb26fe00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27006aaf0 0x7fb27006ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.448+0000 7fb26fe00640 1 -- 172.21.15.125:0/2333019136 >> v1:172.21.15.125:6789/0 conn(0x7fb2701652a0 legacy=0x7fb27006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.448+0000 7fb26fe00640 1 -- 172.21.15.125:0/2333019136 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2700b3eb0 con 0x7fb27014b1a0 2024-01-16T15:03:29.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.450+0000 7fb26fe00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27014b1a0 0x7fb2700b3970 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb268009e50 tx=0x7fb26802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9ce8f7a69458cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.451+0000 7fb26ea00640 1 -- 172.21.15.125:0/2333019136 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26802cb00 con 0x7fb27014b1a0 2024-01-16T15:03:29.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.451+0000 7fb26ea00640 1 -- 172.21.15.125:0/2333019136 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26802cc60 con 0x7fb27014b1a0 2024-01-16T15:03:29.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.451+0000 7fb26ea00640 1 -- 172.21.15.125:0/2333019136 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680343f0 con 0x7fb27014b1a0 2024-01-16T15:03:29.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.453+0000 7fb276e00640 1 -- 172.21.15.125:0/2333019136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27014b1a0 msgr2=0x7fb2700b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.454+0000 7fb276e00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27014b1a0 0x7fb2700b3970 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb268009e50 tx=0x7fb26802bf60 comp rx=0 tx=0).stop 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.454+0000 7fb276e00640 1 -- 172.21.15.125:0/2333019136 shutdown_connections 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.454+0000 7fb276e00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27006aaf0 0x7fb27006ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.455+0000 7fb276e00640 1 --2- 172.21.15.125:0/2333019136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27014b1a0 0x7fb2700b3970 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.455+0000 7fb276e00640 1 -- 172.21.15.125:0/2333019136 >> 172.21.15.125:0/2333019136 conn(0x7fb2700cfef0 msgr2=0x7fb2700d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.455+0000 7fb276e00640 1 -- 172.21.15.125:0/2333019136 shutdown_connections 2024-01-16T15:03:29.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.455+0000 7fb276e00640 1 -- 172.21.15.125:0/2333019136 wait complete. 2024-01-16T15:03:29.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.464+0000 7fb276e00640 1 Processor -- start 2024-01-16T15:03:29.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.468+0000 7fb276e00640 1 -- start start 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27014b1a0 0x7fb2700ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2701652a0 0x7fb2700ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb270161b00 con 0x7fb27006aaf0 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb270163320 con 0x7fb2701652a0 2024-01-16T15:03:29.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb276e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb270163600 con 0x7fb27014b1a0 2024-01-16T15:03:29.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.470+0000 7fb26fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54494/0 (socket says 172.21.15.125:54494) 2024-01-16T15:03:29.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 -- 172.21.15.125:0/2921818642 learned_addr learned my addr 172.21.15.125:0/2921818642 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 -- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27014b1a0 msgr2=0x7fb2700ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27014b1a0 0x7fb2700ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 -- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2701652a0 msgr2=0x7fb2700ce960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2701652a0 0x7fb2700ce960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.471+0000 7fb26fe00640 1 -- 172.21.15.125:0/2921818642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb268009b30 con 0x7fb27006aaf0 2024-01-16T15:03:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.472+0000 7fb26fe00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb268000f80 tx=0x7fb268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.473+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26803e450 con 0x7fb27006aaf0 2024-01-16T15:03:29.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.473+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26803e5b0 con 0x7fb27006aaf0 2024-01-16T15:03:29.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.473+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26803e960 con 0x7fb27006aaf0 2024-01-16T15:03:29.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.473+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb270161d50 con 0x7fb27006aaf0 2024-01-16T15:03:29.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.473+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2701463c0 con 0x7fb27006aaf0 2024-01-16T15:03:29.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.477+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb270162270 con 0x7fb27006aaf0 2024-01-16T15:03:29.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.480+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb26803eac0 con 0x7fb27006aaf0 2024-01-16T15:03:29.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.482+0000 7fb267e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 0x7fb24806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.483+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2680aba30 con 0x7fb27006aaf0 2024-01-16T15:03:29.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.483+0000 7fb26f400640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 0x7fb24806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.484+0000 7fb26f400640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 0x7fb24806e0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb270164a10 tx=0x7fb2580079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:29.487+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2680781b0 con 0x7fb27006aaf0 2024-01-16T15:03:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.066+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb2700675d0 con 0x7fb27006aaf0 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.067+0000 7fb267e00640 1 -- 172.21.15.125:0/2921818642 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fb268043020 con 0x7fb27006aaf0 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:30.046 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:30.047 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 429, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:30.048 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1536, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 6296964876, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 12502827008, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 760591286272, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:30.049 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 272844558, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 256105883, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 66, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 70 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:30.050 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.051 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 37s)", 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.23810000717639923, 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.41667500138282776, 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:30.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:30.053 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:30.053 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 msgr2=0x7fb24806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 0x7fb24806e0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb270164a10 tx=0x7fb2580079d0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 msgr2=0x7fb2701453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb268000f80 tx=0x7fb268004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 shutdown_connections 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2701652a0 0x7fb2700ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb27014b1a0 0x7fb2700ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb24806bc00 0x7fb24806e0c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 --2- 172.21.15.125:0/2921818642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb27006aaf0 0x7fb2701453f0 secure :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb268000f80 tx=0x7fb268004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.071+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 >> 172.21.15.125:0/2921818642 conn(0x7fb2700cfef0 msgr2=0x7fb270160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.072+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 shutdown_connections 2024-01-16T15:03:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:30.072+0000 7fb276e00640 1 -- 172.21.15.125:0/2921818642 wait complete. 2024-01-16T15:03:32.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:32.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.582+0000 7f6fdbe00640 1 Processor -- start 2024-01-16T15:03:32.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.583+0000 7f6fdbe00640 1 -- start start 2024-01-16T15:03:32.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.586+0000 7f6fdbe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fd414b2c0 0x7f6fd40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.586+0000 7f6fdbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fd4071070 0x7f6fd40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.586+0000 7f6fdbe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40cfbf0 con 0x7f6fd40cf7d0 2024-01-16T15:03:32.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.586+0000 7f6fdbe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd414b6a0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.586+0000 7f6fdbe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd4071690 con 0x7f6fd4071070 2024-01-16T15:03:32.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.587+0000 7f6fd8a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6fd40cf7d0 0x7f6fd4070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49286/0 (socket says 172.21.15.125:49286) 2024-01-16T15:03:32.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.588+0000 7f6fd8a00640 1 -- 172.21.15.125:0/332159669 learned_addr learned my addr 172.21.15.125:0/332159669 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:32.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.589+0000 7f6fd9400640 1 --2- 172.21.15.125:0/332159669 >> v2:172.21.15.125:3300/0 conn(0x7f6fd414b2c0 0x7f6fd40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.589+0000 7f6fd3e00640 1 -- 172.21.15.125:0/332159669 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807232583 0 0) 0x7f6fd40cfbf0 con 0x7f6fd40cf7d0 2024-01-16T15:03:32.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.591+0000 7f6fd3e00640 1 -- 172.21.15.125:0/332159669 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003520 con 0x7f6fd40cf7d0 2024-01-16T15:03:32.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.592+0000 7f6fd9400640 1 -- 172.21.15.125:0/332159669 >> v2:172.21.15.137:3300/0 conn(0x7f6fd4071070 msgr2=0x7f6fd40cc160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:03:32.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.593+0000 7f6fd9400640 1 --2- 172.21.15.125:0/332159669 >> v2:172.21.15.137:3300/0 conn(0x7f6fd4071070 0x7f6fd40cc160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.593+0000 7f6fd9400640 1 -- 172.21.15.125:0/332159669 >> v1:172.21.15.125:6789/0 conn(0x7f6fd40cf7d0 legacy=0x7f6fd4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.594+0000 7f6fd9400640 1 -- 172.21.15.125:0/332159669 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd4071970 con 0x7f6fd414b2c0 2024-01-16T15:03:32.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.596+0000 7f6fd9400640 1 --2- 172.21.15.125:0/332159669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd40cf290 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4165330 tx=0x7f6fc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6c797577849b20 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:32.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.596+0000 7f6fd3e00640 1 -- 172.21.15.125:0/332159669 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4031640 con 0x7f6fd414b2c0 2024-01-16T15:03:32.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.596+0000 7f6fd3e00640 1 -- 172.21.15.125:0/332159669 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc40317a0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.596+0000 7f6fd3e00640 1 -- 172.21.15.125:0/332159669 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4031b30 con 0x7f6fd414b2c0 2024-01-16T15:03:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.598+0000 7f6fdbe00640 1 -- 172.21.15.125:0/332159669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 msgr2=0x7f6fd40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.598+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/332159669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd40cf290 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4165330 tx=0x7f6fc402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.601+0000 7f6fdbe00640 1 -- 172.21.15.125:0/332159669 shutdown_connections 2024-01-16T15:03:32.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.601+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/332159669 >> v2:172.21.15.137:3300/0 conn(0x7f6fd4071070 0x7f6fd40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.601+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/332159669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd40cf290 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.601+0000 7f6fdbe00640 1 -- 172.21.15.125:0/332159669 >> 172.21.15.125:0/332159669 conn(0x7f6fd406a800 msgr2=0x7f6fd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:32.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.601+0000 7f6fdbe00640 1 -- 172.21.15.125:0/332159669 shutdown_connections 2024-01-16T15:03:32.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.602+0000 7f6fdbe00640 1 -- 172.21.15.125:0/332159669 wait complete. 2024-01-16T15:03:32.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.605+0000 7f6fdbe00640 1 Processor -- start 2024-01-16T15:03:32.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.606+0000 7f6fdbe00640 1 -- start start 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fd4071070 0x7f6fd415f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fd40cf7d0 0x7f6fd4147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4145bb0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd415fb60 con 0x7f6fd4071070 2024-01-16T15:03:32.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.609+0000 7f6fdbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4147950 con 0x7f6fd40cf7d0 2024-01-16T15:03:32.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.611+0000 7f6fd9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.611+0000 7f6fd9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54512/0 (socket says 172.21.15.125:54512) 2024-01-16T15:03:32.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.611+0000 7f6fd9e00640 1 -- 172.21.15.125:0/389357378 learned_addr learned my addr 172.21.15.125:0/389357378 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:32.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.611+0000 7f6fd8a00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fd40cf7d0 0x7f6fd4147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.612+0000 7f6fd9e00640 1 -- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fd40cf7d0 msgr2=0x7f6fd4147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.612+0000 7f6fd9e00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fd40cf7d0 0x7f6fd4147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.612+0000 7f6fd9e00640 1 -- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fd4071070 msgr2=0x7f6fd415f620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:03:32.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.612+0000 7f6fd9e00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fd4071070 0x7f6fd415f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.612+0000 7f6fd9e00640 1 -- 172.21.15.125:0/389357378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4009400 con 0x7f6fd414b2c0 2024-01-16T15:03:32.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.613+0000 7f6fd9e00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00b9c0 tx=0x7f6fcc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:32.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.614+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00cba0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.615+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc00cd00 con 0x7f6fd414b2c0 2024-01-16T15:03:32.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.615+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc014680 con 0x7f6fd414b2c0 2024-01-16T15:03:32.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.616+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd4148320 con 0x7f6fd414b2c0 2024-01-16T15:03:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.621+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd4147300 con 0x7f6fd414b2c0 2024-01-16T15:03:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.622+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6fcc01d3b0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.623+0000 7f6fd1600640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 0x7f6fac06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.624+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fcc08cbc0 con 0x7f6fd414b2c0 2024-01-16T15:03:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.624+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd4071f30 con 0x7f6fd414b2c0 2024-01-16T15:03:32.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.636+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fcc090060 con 0x7f6fd414b2c0 2024-01-16T15:03:32.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.642+0000 7f6fd9400640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 0x7f6fac06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:32.652+0000 7f6fd9400640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 0x7f6fac06e040 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4036000 tx=0x7f6fc4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.308+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6fd40675d0 con 0x7f6fd414b2c0 2024-01-16T15:03:33.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.314+0000 7f6fd1600640 1 -- 172.21.15.125:0/389357378 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f6fcc019030 con 0x7f6fd414b2c0 2024-01-16T15:03:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:33.294 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 432, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2153, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 8884951217, 2024-01-16T15:03:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 17808080896, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 755286032384, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 385529197, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 382756058, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 92, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 98 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 37s)", 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.23810000717639923, 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.41667500138282776, 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:33.301 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:33.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.322+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 msgr2=0x7f6fac06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.322+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 0x7f6fac06e040 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4036000 tx=0x7f6fc4008030 comp rx=0 tx=0).stop 2024-01-16T15:03:33.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.323+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 msgr2=0x7f6fd4144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.324+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00b9c0 tx=0x7f6fcc00be80 comp rx=0 tx=0).stop 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 shutdown_connections 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6fac06bb80 0x7f6fac06e040 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd414b2c0 0x7f6fd4144e10 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fd40cf7d0 0x7f6fd4147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 --2- 172.21.15.125:0/389357378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fd4071070 0x7f6fd415f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.326+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 >> 172.21.15.125:0/389357378 conn(0x7f6fd406a800 msgr2=0x7f6fd406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:33.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.330+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 shutdown_connections 2024-01-16T15:03:33.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:33.331+0000 7f6fdbe00640 1 -- 172.21.15.125:0/389357378 wait complete. 2024-01-16T15:03:35.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:35.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.603+0000 7f7303800640 1 Processor -- start 2024-01-16T15:03:35.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.604+0000 7f7303800640 1 -- start start 2024-01-16T15:03:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7303800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f72fc0cd960 0x7f72fc15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7303800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc06eb00 con 0x7f72fc06f550 2024-01-16T15:03:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7303800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc06f0c0 con 0x7f72fc06a3e0 2024-01-16T15:03:35.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7303800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc06ab80 con 0x7f72fc0cd960 2024-01-16T15:03:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7301800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f72fc06f550 0x7f72fc15c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49298/0 (socket says 172.21.15.125:49298) 2024-01-16T15:03:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.605+0000 7f7301800640 1 -- 172.21.15.125:0/2104980274 learned_addr learned my addr 172.21.15.125:0/2104980274 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f7300e00640 1 --2- 172.21.15.125:0/2104980274 >> v2:172.21.15.137:3300/0 conn(0x7f72fc0cd960 0x7f72fc15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869488717 0 0) 0x7f72fc06eb00 con 0x7f72fc06f550 2024-01-16T15:03:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72dc003520 con 0x7f72fc06f550 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388739453 0 0) 0x7f72dc003520 con 0x7f72fc06f550 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 >> v2:172.21.15.137:3300/0 conn(0x7f72fc0cd960 msgr2=0x7f72fc15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 --2- 172.21.15.125:0/2104980274 >> v2:172.21.15.137:3300/0 conn(0x7f72fc0cd960 0x7f72fc15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 >> v1:172.21.15.137:6789/0 conn(0x7f72fc06a3e0 legacy=0x7f72fc0cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc06ae60 con 0x7f72fc06f550 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f72f40019f0 con 0x7f72fc06f550 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 >> v1:172.21.15.125:6789/0 conn(0x7f72fc06f550 legacy=0x7f72fc15c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 0x7f72dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc0047c0 0x7f72dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72dc005180 0x7f72dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc06ab80 con 0x7f72dc003970 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc00c120 con 0x7f72dc0047c0 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.606+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc00c2e0 con 0x7f72dc005180 2024-01-16T15:03:35.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 0x7f72dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f72fbe00640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc0047c0 0x7f72dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7301800640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72dc005180 0x7f72dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 -- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72dc005180 msgr2=0x7f72dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72dc005180 0x7f72dc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 -- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc0047c0 msgr2=0x7f72dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc0047c0 0x7f72dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7300e00640 1 -- 172.21.15.125:0/2104980274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc003520 con 0x7f72dc003970 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7301800640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72dc005180 0x7f72dc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.607+0000 7f7301800640 1 -- 172.21.15.125:0/2104980274 reap_dead start 2024-01-16T15:03:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.609+0000 7f7300e00640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 0x7f72dc003d90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009ac0 tx=0x7f72dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62085a641cacf9b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.609+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec02c6b0 con 0x7f72dc003970 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.609+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec02c810 con 0x7f72dc003970 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.609+0000 7f72fb400640 1 -- 172.21.15.125:0/2104980274 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec02cba0 con 0x7f72dc003970 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 -- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 msgr2=0x7f72dc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 0x7f72dc003d90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009ac0 tx=0x7f72dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 -- 172.21.15.125:0/2104980274 shutdown_connections 2024-01-16T15:03:35.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 --2- 172.21.15.125:0/2104980274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72dc003970 0x7f72dc003d90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 -- 172.21.15.125:0/2104980274 >> 172.21.15.125:0/2104980274 conn(0x7f72fc0b1dd0 msgr2=0x7f72fc0b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:35.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 -- 172.21.15.125:0/2104980274 shutdown_connections 2024-01-16T15:03:35.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.610+0000 7f7303800640 1 -- 172.21.15.125:0/2104980274 wait complete. 2024-01-16T15:03:35.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.612+0000 7f7303800640 1 Processor -- start 2024-01-16T15:03:35.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.612+0000 7f7303800640 1 -- start start 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc003970 0x7f72fc1401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72fc15def0 0x7f72fc15e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc1f5040 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc15e8b0 con 0x7f72dc003970 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7303800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0ccdb0 con 0x7f72fc15def0 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54526/0 (socket says 172.21.15.125:54526) 2024-01-16T15:03:35.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 -- 172.21.15.125:0/1275403651 learned_addr learned my addr 172.21.15.125:0/1275403651 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7300e00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc003970 0x7f72fc1401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f7301800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72fc15def0 0x7f72fc15e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 -- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72fc15def0 msgr2=0x7f72fc15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72fc15def0 0x7f72fc15e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 -- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc003970 msgr2=0x7f72fc1401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc003970 0x7f72fc1401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.613+0000 7f72fbe00640 1 -- 172.21.15.125:0/1275403651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc1f43e0 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f72fbe00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e550 tx=0x7f72e400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40052e0 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4005440 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4003810 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc0aaa70 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.614+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1f4980 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.615+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0675d0 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.616+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f72e4003970 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.617+0000 7f72f8c00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 0x7f72d406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:35.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.617+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e4088340 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.618+0000 7f7300e00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 0x7f72d406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:35.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.618+0000 7f7300e00640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 0x7f72d406dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f72ec036000 tx=0x7f72ec030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:35.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.622+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72e4054a30 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.859+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f72fc0cd350 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.860+0000 7f72f8c00640 1 -- 172.21.15.125:0/1275403651 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f72e4058050 con 0x7f72fc0aa5f0 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.839 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 434, 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:35.840 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:35.841 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2153, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 8884951217, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 17808080896, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 755286032384, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 385553132, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 382779821, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 92, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 98 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:35.842 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:35.843 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:35.844 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 37s)", 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.23810000717639923, 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.41667500138282776, 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:35.845 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.862+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 msgr2=0x7f72d406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.862+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 0x7f72d406dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f72ec036000 tx=0x7f72ec030000 comp rx=0 tx=0).stop 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.862+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 msgr2=0x7f72fc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.862+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e550 tx=0x7f72e400ea10 comp rx=0 tx=0).stop 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 shutdown_connections 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72fc15def0 0x7f72fc15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f72d406bb30 0x7f72d406dff0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72fc0aa5f0 0x7f72fc0cc870 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 --2- 172.21.15.125:0/1275403651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72dc003970 0x7f72fc1401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:35.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 >> 172.21.15.125:0/1275403651 conn(0x7f72fc0b1dd0 msgr2=0x7f72fc159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:35.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 shutdown_connections 2024-01-16T15:03:35.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:35.863+0000 7f7303800640 1 -- 172.21.15.125:0/1275403651 wait complete. 2024-01-16T15:03:37.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.275+0000 7f017be00640 1 Processor -- start 2024-01-16T15:03:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.276+0000 7f017be00640 1 -- start start 2024-01-16T15:03:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.278+0000 7f017be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f017c074520 0x7f017c153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.278+0000 7f017be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c159be0 0x7f017c159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.278+0000 7f017be00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017c06d140 con 0x7f017c06c430 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.278+0000 7f017be00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017c153cb0 con 0x7f017c074520 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.278+0000 7f017be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c06d420 con 0x7f017c159be0 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.279+0000 7f017aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f017c074520 0x7f017c153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.279+0000 7f017aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f017c074520 0x7f017c153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57606/0 (socket says 172.21.15.125:57606) 2024-01-16T15:03:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.279+0000 7f017aa00640 1 -- 172.21.15.125:0/313071627 learned_addr learned my addr 172.21.15.125:0/313071627 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:38.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.280+0000 7f017aa00640 1 -- 172.21.15.125:0/313071627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c159be0 msgr2=0x7f017c159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:38.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.281+0000 7f0179600640 1 -- 172.21.15.125:0/313071627 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944032164 0 0) 0x7f017c06d140 con 0x7f017c06c430 2024-01-16T15:03:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.283+0000 7f017b400640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c159be0 0x7f017c159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.283+0000 7f017aa00640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c159be0 0x7f017c159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.283+0000 7f017aa00640 1 -- 172.21.15.125:0/313071627 >> v1:172.21.15.125:6789/0 conn(0x7f017c06c430 legacy=0x7f017c06c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.283+0000 7f017aa00640 1 -- 172.21.15.125:0/313071627 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c06d700 con 0x7f017c074520 2024-01-16T15:03:38.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.286+0000 7f017aa00640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c074520 0x7f017c153770 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f016c009870 tx=0x7f016c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db19564e4c808f6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:38.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.286+0000 7f0179600640 1 -- 172.21.15.125:0/313071627 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c030860 con 0x7f017c074520 2024-01-16T15:03:38.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.287+0000 7f0179600640 1 -- 172.21.15.125:0/313071627 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0309c0 con 0x7f017c074520 2024-01-16T15:03:38.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.287+0000 7f0179600640 1 -- 172.21.15.125:0/313071627 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c030d90 con 0x7f017c074520 2024-01-16T15:03:38.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 -- 172.21.15.125:0/313071627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c074520 msgr2=0x7f017c153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c074520 0x7f017c153770 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f016c009870 tx=0x7f016c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 -- 172.21.15.125:0/313071627 shutdown_connections 2024-01-16T15:03:38.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c159be0 0x7f017c159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 --2- 172.21.15.125:0/313071627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c074520 0x7f017c153770 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 -- 172.21.15.125:0/313071627 >> 172.21.15.125:0/313071627 conn(0x7f017c0b0c70 msgr2=0x7f017c0b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:38.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 -- 172.21.15.125:0/313071627 shutdown_connections 2024-01-16T15:03:38.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.288+0000 7f017be00640 1 -- 172.21.15.125:0/313071627 wait complete. 2024-01-16T15:03:38.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.292+0000 7f017be00640 1 Processor -- start 2024-01-16T15:03:38.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.292+0000 7f017be00640 1 -- start start 2024-01-16T15:03:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c074520 0x7f017c144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c1562d0 con 0x7f017c06c430 2024-01-16T15:03:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c14b620 con 0x7f017c159be0 2024-01-16T15:03:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.296+0000 7f017be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c1559a0 con 0x7f017c074520 2024-01-16T15:03:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c074520 0x7f017c144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57618/0 (socket says 172.21.15.125:57618) 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:36110/0 (socket says 172.21.15.125:36110) 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.297+0000 7f017aa00640 1 -- 172.21.15.125:0/168035489 learned_addr learned my addr 172.21.15.125:0/168035489 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 -- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c074520 msgr2=0x7f017c144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c074520 0x7f017c144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 -- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 msgr2=0x7f017c156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 -- 172.21.15.125:0/168035489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0094d0 con 0x7f017c06c430 2024-01-16T15:03:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017b400640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:03:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.298+0000 7f017aa00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f016c000c00 tx=0x7f016c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.299+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c039d60 con 0x7f017c06c430 2024-01-16T15:03:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.299+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c043400 con 0x7f017c06c430 2024-01-16T15:03:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.299+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0437b0 con 0x7f017c06c430 2024-01-16T15:03:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.299+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c155c80 con 0x7f017c06c430 2024-01-16T15:03:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.299+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c1451a0 con 0x7f017c06c430 2024-01-16T15:03:38.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:38.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.303+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c067450 con 0x7f017c06c430 2024-01-16T15:03:38.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.304+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f016c043910 con 0x7f017c06c430 2024-01-16T15:03:38.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.308+0000 7f0173400640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 0x7f015406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:38.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.311+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f016c03e070 con 0x7f017c06c430 2024-01-16T15:03:38.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.315+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f016c077e80 con 0x7f017c06c430 2024-01-16T15:03:38.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.316+0000 7f017a000640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 0x7f015406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.329+0000 7f017a000640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 0x7f015406dff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f017c154eb0 tx=0x7f0164009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:38.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.929+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f017c0b11f0 con 0x7f017c06c430 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.935+0000 7f0173400640 1 -- 172.21.15.125:0/168035489 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f016c034070 con 0x7f017c06c430 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:38.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 438, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:38.915 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:38.916 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3051, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 12643151716, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 25445466112, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 747648647168, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 517226772, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 528420910, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 123, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 132 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:38.917 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:38.918 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:38.919 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (19s)\n [============................] (remaining: 25s)", 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.4285714328289032, 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (19s)\n [==============..............] (remaining: 19s)", 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.5, 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:38.920 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 msgr2=0x7f015406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 0x7f015406dff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f017c154eb0 tx=0x7f0164009040 comp rx=0 tx=0).stop 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 msgr2=0x7f017c14afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f016c000c00 tx=0x7f016c004730 comp rx=0 tx=0).stop 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 shutdown_connections 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017c159be0 0x7f017c156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f017c074520 0x7f017c144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f015406bb30 0x7f015406dff0 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f017c154eb0 tx=0x7f0164009040 comp rx=0 tx=0).stop 2024-01-16T15:03:38.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 --2- 172.21.15.125:0/168035489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f017c06c430 0x7f017c14afb0 secure :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f016c000c00 tx=0x7f016c004730 comp rx=0 tx=0).stop 2024-01-16T15:03:38.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.943+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 >> 172.21.15.125:0/168035489 conn(0x7f017c0b0c70 msgr2=0x7f017c0cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.946+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 shutdown_connections 2024-01-16T15:03:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:38.946+0000 7f017be00640 1 -- 172.21.15.125:0/168035489 wait complete. 2024-01-16T15:03:40.987 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.224+0000 7f624a400640 1 Processor -- start 2024-01-16T15:03:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.225+0000 7f624a400640 1 -- start start 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.226+0000 7f624a400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f624415e670 0x7f62440b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.226+0000 7f624a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 0x7f62440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.226+0000 7f624a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62440cca90 con 0x7f624415ea50 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.226+0000 7f624a400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62440cf170 con 0x7f624415e670 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.226+0000 7f624a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62440ccd70 con 0x7f62440ce6a0 2024-01-16T15:03:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f624415e670 0x7f62440b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f624415ea50 0x7f624415eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41320/0 (socket says 172.21.15.125:41320) 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249a00640 1 -- 172.21.15.125:0/3177455069 learned_addr learned my addr 172.21.15.125:0/3177455069 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249000640 1 -- 172.21.15.125:0/3177455069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 msgr2=0x7f62440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6243e00640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 0x7f62440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6243400640 1 -- 172.21.15.125:0/3177455069 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318277504 0 0) 0x7f62440cca90 con 0x7f624415ea50 2024-01-16T15:03:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249000640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 0x7f62440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249000640 1 -- 172.21.15.125:0/3177455069 >> v1:172.21.15.125:6789/0 conn(0x7f624415ea50 legacy=0x7f624415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6249000640 1 -- 172.21.15.125:0/3177455069 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62440ceab0 con 0x7f624415e670 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.227+0000 7f6243e00640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 0x7f62440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.228+0000 7f6249000640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f624415e670 0x7f62440b3d30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6234009d90 tx=0x7f623402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6552874c0518b025 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.228+0000 7f6243400640 1 -- 172.21.15.125:0/3177455069 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623402cc50 con 0x7f624415e670 2024-01-16T15:03:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f6243400640 1 -- 172.21.15.125:0/3177455069 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623402cdb0 con 0x7f624415e670 2024-01-16T15:03:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f6243400640 1 -- 172.21.15.125:0/3177455069 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234034660 con 0x7f624415e670 2024-01-16T15:03:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 -- 172.21.15.125:0/3177455069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f624415e670 msgr2=0x7f62440b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f624415e670 0x7f62440b3d30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6234009d90 tx=0x7f623402bea0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 -- 172.21.15.125:0/3177455069 shutdown_connections 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62440ce6a0 0x7f62440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 --2- 172.21.15.125:0/3177455069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f624415e670 0x7f62440b3d30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 -- 172.21.15.125:0/3177455069 >> 172.21.15.125:0/3177455069 conn(0x7f62440b2a30 msgr2=0x7f62440b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 -- 172.21.15.125:0/3177455069 shutdown_connections 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.229+0000 7f624a400640 1 -- 172.21.15.125:0/3177455069 wait complete. 2024-01-16T15:03:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.231+0000 7f624a400640 1 Processor -- start 2024-01-16T15:03:41.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.231+0000 7f624a400640 1 -- start start 2024-01-16T15:03:41.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 0x7f62441f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f624415ea50 0x7f624414e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62441f38a0 con 0x7f62440c8220 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62441f6da0 con 0x7f624415e670 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f624a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62441f32e0 con 0x7f624415ea50 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f6249a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f6249a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57640/0 (socket says 172.21.15.125:57640) 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.232+0000 7f6249a00640 1 -- 172.21.15.125:0/224890218 learned_addr learned my addr 172.21.15.125:0/224890218 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249000640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 0x7f62441f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6243e00640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f624415ea50 0x7f624414e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 -- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f624415ea50 msgr2=0x7f624414e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f624415ea50 0x7f624414e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 -- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 msgr2=0x7f62441f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 0x7f62441f6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 -- 172.21.15.125:0/224890218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6234009a70 con 0x7f62440c8220 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249000640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 0x7f62441f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6249a00640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f623c00c410 tx=0x7f623c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c0074e0 con 0x7f62440c8220 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c007640 con 0x7f62440c8220 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c004a20 con 0x7f62440c8220 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62441f35c0 con 0x7f62440c8220 2024-01-16T15:03:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.233+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62441f6340 con 0x7f62440c8220 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.234+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62441f6560 con 0x7f62440c8220 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.235+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f623c004b80 con 0x7f62440c8220 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.236+0000 7f6240c00640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 0x7f621c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.236+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f623c0883d0 con 0x7f62440c8220 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.236+0000 7f6249000640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 0x7f621c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.237+0000 7f6249000640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 0x7f621c06dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6234036000 tx=0x7f6234030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.241+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f623c054b80 con 0x7f62440c8220 2024-01-16T15:03:41.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.480+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6244067b60 con 0x7f62440c8220 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.481+0000 7f6240c00640 1 -- 172.21.15.125:0/224890218 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2438 (secure 0 0 0) 0x7f623c0581a0 con 0x7f62440c8220 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:41.460 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 440, 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:41.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:41.462 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3543, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 14715315997, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 29306253312, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 743787859968, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 688023227, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 700979923, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 164, 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 178 2024-01-16T15:03:41.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:41.464 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 52.381% (24s)\n [==============..............] (remaining: 22s)", 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.52381426095962524, 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "Subvolume \"subvol_clone27077\" has been 66.67% cloned (24s)\n [==================..........] (remaining: 12s)", 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.66670000553131104, 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 msgr2=0x7f621c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 0x7f621c06dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6234036000 tx=0x7f6234030000 comp rx=0 tx=0).stop 2024-01-16T15:03:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 msgr2=0x7f62441f2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f623c00c410 tx=0x7f623c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 shutdown_connections 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f621c06bb30 0x7f621c06dff0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62440c8220 0x7f62441f2900 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f624415ea50 0x7f624414e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 --2- 172.21.15.125:0/224890218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f624415e670 0x7f62441f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.483+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 >> 172.21.15.125:0/224890218 conn(0x7f62440b2a30 msgr2=0x7f62440ccfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.484+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 shutdown_connections 2024-01-16T15:03:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:41.484+0000 7f624a400640 1 -- 172.21.15.125:0/224890218 wait complete. 2024-01-16T15:03:43.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:43.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.790+0000 7fe99d000640 1 Processor -- start 2024-01-16T15:03:43.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.791+0000 7fe99d000640 1 -- start start 2024-01-16T15:03:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.793+0000 7fe99d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980cc650 0x7fe998168300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.793+0000 7fe99d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9980d0530 con 0x7fe99815d6e0 2024-01-16T15:03:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.793+0000 7fe99d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9980b0070 con 0x7fe9980b02b0 2024-01-16T15:03:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.793+0000 7fe99d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b3a80 con 0x7fe9980cc650 2024-01-16T15:03:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.794+0000 7fe995400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe9980b02b0 0x7fe99806c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55138/0 (socket says 172.21.15.125:55138) 2024-01-16T15:03:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.794+0000 7fe995400640 1 -- 172.21.15.125:0/2804729810 learned_addr learned my addr 172.21.15.125:0/2804729810 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.794+0000 7fe996800640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980cc650 0x7fe998168300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526504051 0 0) 0x7fe9980b0070 con 0x7fe9980b02b0 2024-01-16T15:03:43.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe978003520 con 0x7fe9980b02b0 2024-01-16T15:03:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2926072464 0 0) 0x7fe978003520 con 0x7fe9980b02b0 2024-01-16T15:03:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980cc650 msgr2=0x7fe998168300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980cc650 0x7fe998168300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 >> v1:172.21.15.125:6789/0 conn(0x7fe99815d6e0 legacy=0x7fe99806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99806c740 con 0x7fe9980b02b0 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.795+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe980001670 con 0x7fe9980b02b0 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 >> v1:172.21.15.137:6789/0 conn(0x7fe9980b02b0 legacy=0x7fe99806c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe978003970 0x7fe978003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9780047c0 0x7fe978004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978005180 0x7fe97800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b3a80 con 0x7fe978005180 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97800c020 con 0x7fe9780047c0 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97800c1e0 con 0x7fe978003970 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.796+0000 7fe995e00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe978003970 0x7fe978003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995400640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9780047c0 0x7fe978004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995e00640 1 -- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9780047c0 msgr2=0x7fe978004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995e00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9780047c0 0x7fe978004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe996800640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978005180 0x7fe97800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995e00640 1 -- 172.21.15.125:0/2804729810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978005180 msgr2=0x7fe97800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995e00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978005180 0x7fe97800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe995e00640 1 -- 172.21.15.125:0/2804729810 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99806c740 con 0x7fe978003970 2024-01-16T15:03:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.797+0000 7fe996800640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978005180 0x7fe97800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:43.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.800+0000 7fe995e00640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe978003970 0x7fe978003d90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe988001750 tx=0x7fe9780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7fa51a6e6276ce4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:43.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.801+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9880052d0 con 0x7fe978003970 2024-01-16T15:03:43.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.801+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98802e050 con 0x7fe978003970 2024-01-16T15:03:43.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.801+0000 7fe994a00640 1 -- 172.21.15.125:0/2804729810 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988003830 con 0x7fe978003970 2024-01-16T15:03:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.802+0000 7fe99d000640 1 -- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe978003970 msgr2=0x7fe978003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.802+0000 7fe99d000640 1 --2- 172.21.15.125:0/2804729810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe978003970 0x7fe978003d90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe988001750 tx=0x7fe9780057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.802+0000 7fe996800640 1 -- 172.21.15.125:0/2804729810 reap_dead start 2024-01-16T15:03:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.802+0000 7fe99d000640 1 -- 172.21.15.125:0/2804729810 shutdown_connections 2024-01-16T15:03:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.802+0000 7fe99d000640 1 -- 172.21.15.125:0/2804729810 >> 172.21.15.125:0/2804729810 conn(0x7fe9980b2ae0 msgr2=0x7fe9980b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.803+0000 7fe99d000640 1 -- 172.21.15.125:0/2804729810 shutdown_connections 2024-01-16T15:03:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.803+0000 7fe99d000640 1 -- 172.21.15.125:0/2804729810 wait complete. 2024-01-16T15:03:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.807+0000 7fe99d000640 1 Processor -- start 2024-01-16T15:03:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.807+0000 7fe99d000640 1 -- start start 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe998159480 0x7fe998159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 0x7fe9980af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980c61e0 con 0x7fe978003970 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980afa30 con 0x7fe998159480 2024-01-16T15:03:43.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.809+0000 7fe99d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe998158e70 con 0x7fe9980af070 2024-01-16T15:03:43.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57656/0 (socket says 172.21.15.125:57656) 2024-01-16T15:03:43.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 -- 172.21.15.125:0/427373098 learned_addr learned my addr 172.21.15.125:0/427373098 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:43.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995400640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe998159480 0x7fe998159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe996800640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 0x7fe9980af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 -- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 msgr2=0x7fe9980af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 0x7fe9980af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 -- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe998159480 msgr2=0x7fe998159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe998159480 0x7fe998159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.810+0000 7fe995e00640 1 -- 172.21.15.125:0/427373098 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980c7000 con 0x7fe978003970 2024-01-16T15:03:43.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.811+0000 7fe995e00640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe988005040 tx=0x7fe988001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:43.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.811+0000 7fe996800640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 0x7fe9980af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:43.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.812+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988000ed0 con 0x7fe978003970 2024-01-16T15:03:43.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.812+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe988007640 con 0x7fe978003970 2024-01-16T15:03:43.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.812+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988006c20 con 0x7fe978003970 2024-01-16T15:03:43.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.813+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9980c63f0 con 0x7fe978003970 2024-01-16T15:03:43.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.814+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9980c69b0 con 0x7fe978003970 2024-01-16T15:03:43.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.816+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9980675d0 con 0x7fe978003970 2024-01-16T15:03:43.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.818+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe9880035f0 con 0x7fe978003970 2024-01-16T15:03:43.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.819+0000 7fe98e000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 0x7fe97006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:43.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.819+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9880aa1f0 con 0x7fe978003970 2024-01-16T15:03:43.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.820+0000 7fe995400640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 0x7fe97006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:43.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.824+0000 7fe995400640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 0x7fe97006e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe980000fb0 tx=0x7fe980009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:43.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:43.828+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9880768e0 con 0x7fe978003970 2024-01-16T15:03:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.102+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe9980d0be0 con 0x7fe978003970 2024-01-16T15:03:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.103+0000 7fe98e000640 1 -- 172.21.15.125:0/427373098 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2438 (secure 0 0 0) 0x7fe98804b020 con 0x7fe978003970 2024-01-16T15:03:44.082 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:44.082 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:44.082 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:44.082 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:44.083 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 443, 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.084 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:44.085 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3543, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 14715315997, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 29306253312, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 743787859968, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 478250920, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 485612365, 2024-01-16T15:03:44.086 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 114, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:44.087 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:44.088 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:44.089 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 52.381% (24s)\n [==============..............] (remaining: 22s)", 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.52381426095962524, 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "Subvolume \"subvol_clone27077\" has been 66.67% cloned (24s)\n [==================..........] (remaining: 12s)", 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.66670000553131104, 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:44.090 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:44.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 msgr2=0x7fe97006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 0x7fe97006e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe980000fb0 tx=0x7fe980009040 comp rx=0 tx=0).stop 2024-01-16T15:03:44.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 msgr2=0x7fe998150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe988005040 tx=0x7fe988001120 comp rx=0 tx=0).stop 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 shutdown_connections 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9980af070 0x7fe9980af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe998159480 0x7fe998159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe97006bb60 0x7fe97006e020 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 --2- 172.21.15.125:0/427373098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe978003970 0x7fe998150ba0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.105+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 >> 172.21.15.125:0/427373098 conn(0x7fe9980b2ae0 msgr2=0x7fe99806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.106+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 shutdown_connections 2024-01-16T15:03:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:44.106+0000 7fe99d000640 1 -- 172.21.15.125:0/427373098 wait complete. 2024-01-16T15:03:46.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:46.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.333+0000 7f44e0a00640 1 Processor -- start 2024-01-16T15:03:46.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.334+0000 7f44e0a00640 1 -- start start 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.335+0000 7f44e0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44dc160920 0x7f44dc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.335+0000 7f44e0a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44dc0cf9f0 0x7f44dc0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.335+0000 7f44e0a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc0d0a10 con 0x7f44dc161010 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.335+0000 7f44e0a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44dc0d03e0 con 0x7f44dc160920 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.335+0000 7f44e0a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44dc0d1f50 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44dc160920 0x7f44dc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44dc160920 0x7f44dc0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57672/0 (socket says 172.21.15.125:57672) 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 -- 172.21.15.125:0/1246110580 learned_addr learned my addr 172.21.15.125:0/1246110580 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:46.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44d9e00640 1 --2- 172.21.15.125:0/1246110580 >> v2:172.21.15.137:3300/0 conn(0x7f44dc0cf9f0 0x7f44dc0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 -- 172.21.15.125:0/1246110580 >> v2:172.21.15.137:3300/0 conn(0x7f44dc0cf9f0 msgr2=0x7f44dc0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:46.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 --2- 172.21.15.125:0/1246110580 >> v2:172.21.15.137:3300/0 conn(0x7f44dc0cf9f0 0x7f44dc0cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 -- 172.21.15.125:0/1246110580 >> v1:172.21.15.137:6789/0 conn(0x7f44dc161010 legacy=0x7f44dc161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:46.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.336+0000 7f44da800640 1 -- 172.21.15.125:0/1246110580 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc0cf080 con 0x7f44dc160920 2024-01-16T15:03:46.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.337+0000 7f44da800640 1 --2- 172.21.15.125:0/1246110580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc160920 0x7f44dc0b3f30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f44dc08c1f0 tx=0x7f44cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181b2607dc4f70b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.338+0000 7f44d9400640 1 -- 172.21.15.125:0/1246110580 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc031870 con 0x7f44dc160920 2024-01-16T15:03:46.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.338+0000 7f44d9400640 1 -- 172.21.15.125:0/1246110580 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc0319d0 con 0x7f44dc160920 2024-01-16T15:03:46.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.338+0000 7f44d9400640 1 -- 172.21.15.125:0/1246110580 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc031d60 con 0x7f44dc160920 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 -- 172.21.15.125:0/1246110580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc160920 msgr2=0x7f44dc0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 --2- 172.21.15.125:0/1246110580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc160920 0x7f44dc0b3f30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f44dc08c1f0 tx=0x7f44cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 -- 172.21.15.125:0/1246110580 shutdown_connections 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 --2- 172.21.15.125:0/1246110580 >> v2:172.21.15.137:3300/0 conn(0x7f44dc0cf9f0 0x7f44dc0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 --2- 172.21.15.125:0/1246110580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc160920 0x7f44dc0b3f30 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 -- 172.21.15.125:0/1246110580 >> 172.21.15.125:0/1246110580 conn(0x7f44dc0d3090 msgr2=0x7f44dc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 -- 172.21.15.125:0/1246110580 shutdown_connections 2024-01-16T15:03:46.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.339+0000 7f44e0a00640 1 -- 172.21.15.125:0/1246110580 wait complete. 2024-01-16T15:03:46.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.341+0000 7f44e0a00640 1 Processor -- start 2024-01-16T15:03:46.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.341+0000 7f44e0a00640 1 -- start start 2024-01-16T15:03:46.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 0x7f44dc1f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44dc161010 0x7f44dc0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc1f4520 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc1f3f60 con 0x7f44dc160920 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44e0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc1f4240 con 0x7f44dc161010 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57684/0 (socket says 172.21.15.125:57684) 2024-01-16T15:03:46.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 -- 172.21.15.125:0/4944536 learned_addr learned my addr 172.21.15.125:0/4944536 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:46.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44d9e00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 0x7f44dc1f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 -- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44dc161010 msgr2=0x7f44dc0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44db200640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44dc161010 0x7f44dc0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44dc161010 0x7f44dc0c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 -- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 msgr2=0x7f44dc1f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 0x7f44dc1f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44da800640 1 -- 172.21.15.125:0/4944536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc0094d0 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.342+0000 7f44d9e00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 0x7f44dc1f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44da800640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f44cc000c00 tx=0x7f44cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc03adf0 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc0443f0 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc0447a0 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44dc0cb500 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.343+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44dc14a970 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.344+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44dc0cb360 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.345+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f44cc04d450 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.347+0000 7f44d2a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 0x7f44b406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.347+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44cc02d040 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.347+0000 7f44d9e00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 0x7f44b406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.348+0000 7f44d9e00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 0x7f44b406dff0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f44dc0b63c0 tx=0x7f44c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.352+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44dc0cb360 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.609+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f44dc0cb360 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.610+0000 7f44d2a00640 1 -- 172.21.15.125:0/4944536 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2402 (secure 0 0 0) 0x7f44dc0cb360 con 0x7f44dc0cf9f0 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:46.589 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 445, 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:46.590 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:46.591 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3878, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 16107988611, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 32573976576, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 740520136704, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 590477200, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 601698160, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 141, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 155 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:46.592 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:46.593 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.594 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (30s)\n [================............] (remaining: 22s)", 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.57142859697341919, 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "Subvolume \"subvol_clone27077\" has been 0.0% cloned (30s)\n [............................] ", 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:46.595 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 msgr2=0x7f44b406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 0x7f44b406dff0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f44dc0b63c0 tx=0x7f44c4008040 comp rx=0 tx=0).stop 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 msgr2=0x7f44dc14a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f44cc000c00 tx=0x7f44cc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 shutdown_connections 2024-01-16T15:03:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44dc161010 0x7f44dc0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44dc160920 0x7f44dc1f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f44b406bb30 0x7f44b406dff0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 --2- 172.21.15.125:0/4944536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44dc0cf9f0 0x7f44dc14a430 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 >> 172.21.15.125:0/4944536 conn(0x7f44dc0d3090 msgr2=0x7f44dc0b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 shutdown_connections 2024-01-16T15:03:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:46.612+0000 7f44e0a00640 1 -- 172.21.15.125:0/4944536 wait complete. 2024-01-16T15:03:48.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.936+0000 7fceee400640 1 Processor -- start 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.936+0000 7fceee400640 1 -- start start 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fceee400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcee80cd240 0x7fcee80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fceee400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee8099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fceee400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee806cea0 con 0x7fcee8099fa0 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fceee400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee809a430 con 0x7fcee80cd240 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fceee400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee806cbc0 con 0x7fcee806c6b0 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fcee7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcee8099fa0 0x7fcee806b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59958/0 (socket says 172.21.15.125:59958) 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fcee7e00640 1 -- 172.21.15.125:0/3015109560 learned_addr learned my addr 172.21.15.125:0/3015109560 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.939+0000 7fcee6a00640 1 --2- 172.21.15.125:0/3015109560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee8099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.940+0000 7fcee7400640 1 --2- 172.21.15.125:0/3015109560 >> v2:172.21.15.137:3300/0 conn(0x7fcee80cd240 0x7fcee80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.940+0000 7fcee6000640 1 -- 172.21.15.125:0/3015109560 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531783468 0 0) 0x7fcee806cea0 con 0x7fcee8099fa0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.940+0000 7fcee6000640 1 -- 172.21.15.125:0/3015109560 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec8003520 con 0x7fcee8099fa0 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.941+0000 7fcee6a00640 1 -- 172.21.15.125:0/3015109560 >> v2:172.21.15.137:3300/0 conn(0x7fcee80cd240 msgr2=0x7fcee80b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.941+0000 7fcee6a00640 1 --2- 172.21.15.125:0/3015109560 >> v2:172.21.15.137:3300/0 conn(0x7fcee80cd240 0x7fcee80b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.941+0000 7fcee6a00640 1 -- 172.21.15.125:0/3015109560 >> v1:172.21.15.125:6789/0 conn(0x7fcee8099fa0 legacy=0x7fcee806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.941+0000 7fcee6a00640 1 -- 172.21.15.125:0/3015109560 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee806f350 con 0x7fcee806c6b0 2024-01-16T15:03:48.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.943+0000 7fcee6a00640 1 --2- 172.21.15.125:0/3015109560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee8099a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fced00097a0 tx=0x7fced002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77848b7d5fb34d9e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:48.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.943+0000 7fcee6000640 1 -- 172.21.15.125:0/3015109560 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031830 con 0x7fcee806c6b0 2024-01-16T15:03:48.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.943+0000 7fcee6000640 1 -- 172.21.15.125:0/3015109560 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0031990 con 0x7fcee806c6b0 2024-01-16T15:03:48.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.944+0000 7fcee6000640 1 -- 172.21.15.125:0/3015109560 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031d40 con 0x7fcee806c6b0 2024-01-16T15:03:48.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.944+0000 7fceee400640 1 -- 172.21.15.125:0/3015109560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 msgr2=0x7fcee8099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.944+0000 7fceee400640 1 --2- 172.21.15.125:0/3015109560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee8099a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fced00097a0 tx=0x7fced002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 -- 172.21.15.125:0/3015109560 shutdown_connections 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 --2- 172.21.15.125:0/3015109560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee8099a60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 --2- 172.21.15.125:0/3015109560 >> v2:172.21.15.137:3300/0 conn(0x7fcee80cd240 0x7fcee80b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 -- 172.21.15.125:0/3015109560 >> 172.21.15.125:0/3015109560 conn(0x7fcee80b2d00 msgr2=0x7fcee80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 -- 172.21.15.125:0/3015109560 shutdown_connections 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.945+0000 7fceee400640 1 -- 172.21.15.125:0/3015109560 wait complete. 2024-01-16T15:03:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.948+0000 7fceee400640 1 Processor -- start 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.950+0000 7fceee400640 1 -- start start 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee815fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee8099fa0 0x7fcee80b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee815ffd0 con 0x7fcee80cd240 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80b45a0 con 0x7fcee8099fa0 2024-01-16T15:03:48.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.953+0000 7fceee400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80c86c0 con 0x7fcee806c6b0 2024-01-16T15:03:48.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50160/0 (socket says 172.21.15.125:50160) 2024-01-16T15:03:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 -- 172.21.15.125:0/100610507 learned_addr learned my addr 172.21.15.125:0/100610507 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee6a00640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee8099fa0 0x7fcee80b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee815fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 -- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 msgr2=0x7fcee815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee815fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 -- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee8099fa0 msgr2=0x7fcee80b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee8099fa0 0x7fcee80b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.955+0000 7fcee7e00640 1 -- 172.21.15.125:0/100610507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0009400 con 0x7fcee80cd240 2024-01-16T15:03:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.956+0000 7fcee7e00640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fcee000caa0 tx=0x7fcee000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.957+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000aba0 con 0x7fcee80cd240 2024-01-16T15:03:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.957+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000ad00 con 0x7fcee80cd240 2024-01-16T15:03:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.957+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0012620 con 0x7fcee80cd240 2024-01-16T15:03:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.957+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee80c89a0 con 0x7fcee80cd240 2024-01-16T15:03:48.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.958+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee80c8f60 con 0x7fcee80cd240 2024-01-16T15:03:48.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.960+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee80675d0 con 0x7fcee80cd240 2024-01-16T15:03:48.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.960+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fcee0007070 con 0x7fcee80cd240 2024-01-16T15:03:48.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.961+0000 7fcedf400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 0x7fcec006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.962+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcee0088560 con 0x7fcee80cd240 2024-01-16T15:03:48.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.962+0000 7fcee7400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 0x7fcec006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.962+0000 7fcee7400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 0x7fcec006dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcee80d0b50 tx=0x7fced8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:48.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:48.967+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcee0054cb0 con 0x7fcee80cd240 2024-01-16T15:03:49.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.425+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcee80cfc40 con 0x7fcee80cd240 2024-01-16T15:03:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.429+0000 7fcedf400640 1 -- 172.21.15.125:0/100610507 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2391 (secure 0 0 0) 0x7fcee0017020 con 0x7fcee80cd240 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 448, 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3878, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 16107988611, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 32573976576, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 740520136704, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 282992350, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 288613437, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 68, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 77 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (32s)\n [================............] (remaining: 24s)", 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.57142859697341919, 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (32s)\n [............................] ", 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:49.417 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:49.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 msgr2=0x7fcec006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:49.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 0x7fcec006dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcee80d0b50 tx=0x7fced8008040 comp rx=0 tx=0).stop 2024-01-16T15:03:49.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 msgr2=0x7fcee8160290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:49.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fcee000caa0 tx=0x7fcee000cf60 comp rx=0 tx=0).stop 2024-01-16T15:03:49.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 shutdown_connections 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcec006bb30 0x7fcec006dff0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcee80cd240 0x7fcee8160290 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee8099fa0 0x7fcee80b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 --2- 172.21.15.125:0/100610507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee806c6b0 0x7fcee815fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 >> 172.21.15.125:0/100610507 conn(0x7fcee80b2d00 msgr2=0x7fcee80cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.435+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 shutdown_connections 2024-01-16T15:03:49.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:49.436+0000 7fceee400640 1 -- 172.21.15.125:0/100610507 wait complete. 2024-01-16T15:03:51.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.724+0000 7f2d85800640 1 Processor -- start 2024-01-16T15:03:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.724+0000 7f2d85800640 1 -- start start 2024-01-16T15:03:51.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.726+0000 7f2d85800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2d8014b2c0 0x7f2d800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.727+0000 7f2d85800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d800cfbf0 con 0x7f2d80071070 2024-01-16T15:03:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.727+0000 7f2d85800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d80071690 con 0x7f2d800cf7d0 2024-01-16T15:03:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.727+0000 7f2d85800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8014b6a0 con 0x7f2d8014b2c0 2024-01-16T15:03:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.727+0000 7f2d7f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2d80071070 0x7f2d800cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59970/0 (socket says 172.21.15.125:59970) 2024-01-16T15:03:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.727+0000 7f2d7f400640 1 -- 172.21.15.125:0/1727766837 learned_addr learned my addr 172.21.15.125:0/1727766837 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:03:51.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900805283 0 0) 0x7f2d800cfbf0 con 0x7f2d80071070 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60003520 con 0x7f2d80071070 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528343289 0 0) 0x7f2d60003520 con 0x7f2d80071070 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 >> v2:172.21.15.137:3300/0 conn(0x7f2d8014b2c0 msgr2=0x7f2d800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 --2- 172.21.15.125:0/1727766837 >> v2:172.21.15.137:3300/0 conn(0x7f2d8014b2c0 0x7f2d800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 >> v1:172.21.15.137:6789/0 conn(0x7f2d800cf7d0 legacy=0x7f2d80070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.728+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d80071970 con 0x7f2d80071070 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2d78001670 con 0x7f2d80071070 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 >> v1:172.21.15.125:6789/0 conn(0x7f2d80071070 legacy=0x7f2d800cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d60003970 0x7f2d60003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d600047c0 0x7f2d60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 0x7f2d6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8014b6a0 con 0x7f2d600051f0 2024-01-16T15:03:51.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c150 con 0x7f2d600047c0 2024-01-16T15:03:51.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c310 con 0x7f2d60003970 2024-01-16T15:03:51.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7f400640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 0x7f2d6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7f400640 1 -- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d60003970 msgr2=0x7f2d60003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7f400640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d60003970 0x7f2d60003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7f400640 1 -- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d600047c0 msgr2=0x7f2d60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.729+0000 7f2d7e000640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d600047c0 0x7f2d60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.730+0000 7f2d7f400640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d600047c0 0x7f2d60004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.730+0000 7f2d7f400640 1 -- 172.21.15.125:0/1727766837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d80071970 con 0x7f2d600051f0 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.731+0000 7f2d7f400640 1 -- 172.21.15.125:0/1727766837 reap_dead start 2024-01-16T15:03:51.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.731+0000 7f2d7f400640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 0x7f2d6000b980 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c370 tx=0x7f2d600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca345c613ec5fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.732+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7802cc20 con 0x7f2d600051f0 2024-01-16T15:03:51.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.732+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7802cdc0 con 0x7f2d600051f0 2024-01-16T15:03:51.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.732+0000 7f2d7d600640 1 -- 172.21.15.125:0/1727766837 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78005b10 con 0x7f2d600051f0 2024-01-16T15:03:51.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.733+0000 7f2d85800640 1 -- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 msgr2=0x7f2d6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.733+0000 7f2d85800640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 0x7f2d6000b980 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c370 tx=0x7f2d600057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.734+0000 7f2d85800640 1 -- 172.21.15.125:0/1727766837 shutdown_connections 2024-01-16T15:03:51.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.734+0000 7f2d85800640 1 --2- 172.21.15.125:0/1727766837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d600051f0 0x7f2d6000b980 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.734+0000 7f2d85800640 1 -- 172.21.15.125:0/1727766837 >> 172.21.15.125:0/1727766837 conn(0x7f2d8006a800 msgr2=0x7f2d8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:51.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.734+0000 7f2d85800640 1 -- 172.21.15.125:0/1727766837 shutdown_connections 2024-01-16T15:03:51.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.734+0000 7f2d85800640 1 -- 172.21.15.125:0/1727766837 wait complete. 2024-01-16T15:03:51.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.738+0000 7f2d85800640 1 Processor -- start 2024-01-16T15:03:51.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.738+0000 7f2d85800640 1 -- start start 2024-01-16T15:03:51.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.739+0000 7f2d85800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 0x7f2d80142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.739+0000 7f2d85800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d80162ec0 0x7f2d80149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.739+0000 7f2d85800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 0x7f2d8014a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.742+0000 7f2d85800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80150c80 con 0x7f2d80162ec0 2024-01-16T15:03:51.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.742+0000 7f2d85800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80142890 con 0x7f2d80163300 2024-01-16T15:03:51.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.742+0000 7f2d85800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80163840 con 0x7f2d600051f0 2024-01-16T15:03:51.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.743+0000 7f2d7ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 0x7f2d80142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.743+0000 7f2d7f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 0x7f2d8014a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.743+0000 7f2d7f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 0x7f2d8014a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:49236/0 (socket says 172.21.15.125:49236) 2024-01-16T15:03:51.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.743+0000 7f2d7f400640 1 -- 172.21.15.125:0/2657768536 learned_addr learned my addr 172.21.15.125:0/2657768536 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:51.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.744+0000 7f2d7e000640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d80162ec0 0x7f2d80149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.746+0000 7f2d7ea00640 1 -- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 msgr2=0x7f2d8014a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.746+0000 7f2d7ea00640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 0x7f2d8014a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.746+0000 7f2d7ea00640 1 -- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d80162ec0 msgr2=0x7f2d80149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.746+0000 7f2d7ea00640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d80162ec0 0x7f2d80149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.746+0000 7f2d7ea00640 1 -- 172.21.15.125:0/2657768536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d80163b20 con 0x7f2d600051f0 2024-01-16T15:03:51.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.748+0000 7f2d7ea00640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 0x7f2d80142350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006fd0 tx=0x7f2d6800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.750+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011830 con 0x7f2d600051f0 2024-01-16T15:03:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.750+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68011990 con 0x7f2d600051f0 2024-01-16T15:03:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.750+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011d60 con 0x7f2d600051f0 2024-01-16T15:03:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.750+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d80163e00 con 0x7f2d600051f0 2024-01-16T15:03:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.750+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d80164c40 con 0x7f2d600051f0 2024-01-16T15:03:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.753+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50005250 con 0x7f2d600051f0 2024-01-16T15:03:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.759+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2d6801b460 con 0x7f2d600051f0 2024-01-16T15:03:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.760+0000 7f2d76a00640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 0x7f2d5806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.760+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d68088d70 con 0x7f2d600051f0 2024-01-16T15:03:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.761+0000 7f2d7e000640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 0x7f2d5806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.762+0000 7f2d7e000640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 0x7f2d5806e020 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2d80156780 tx=0x7f2d70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:51.768+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d68055510 con 0x7f2d600051f0 2024-01-16T15:03:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.231+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2d500057d0 con 0x7f2d600051f0 2024-01-16T15:03:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.233+0000 7f2d76a00640 1 -- 172.21.15.125:0/2657768536 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f2d68058b30 con 0x7f2d600051f0 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:52.217 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 451, 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:52.218 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:52.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 4079, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 16963717261, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 34043887616, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 739050225664, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 189707951, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 187285916, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 46, 2024-01-16T15:03:52.220 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 52 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:52.221 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:52.222 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 66.666% (36s)\n [==================..........] (remaining: 18s)", 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.66665714979171753, 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (36s)\n [======......................] (remaining: 2m)", 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:52.223 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:52.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.243+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 msgr2=0x7f2d5806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.243+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 0x7f2d5806e020 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2d80156780 tx=0x7f2d70009040 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.243+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 msgr2=0x7f2d80142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.243+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 0x7f2d80142350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006fd0 tx=0x7f2d6800b8e0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 shutdown_connections 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d80163300 0x7f2d8014a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2d5806bb60 0x7f2d5806e020 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d80162ec0 0x7f2d80149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 --2- 172.21.15.125:0/2657768536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d600051f0 0x7f2d80142350 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 >> 172.21.15.125:0/2657768536 conn(0x7f2d8006a800 msgr2=0x7f2d800aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 shutdown_connections 2024-01-16T15:03:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:52.244+0000 7f2d85800640 1 -- 172.21.15.125:0/2657768536 wait complete. 2024-01-16T15:03:54.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:54.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.689+0000 7fa0c4a00640 1 Processor -- start 2024-01-16T15:03:54.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.690+0000 7fa0c4a00640 1 -- start start 2024-01-16T15:03:54.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.694+0000 7fa0c4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0c0158f90 0x7fa0c00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.695+0000 7fa0c4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c006cf50 0x7fa0c0070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.695+0000 7fa0c4a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c00b1ec0 con 0x7fa0c0159370 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.695+0000 7fa0c4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c00b21a0 con 0x7fa0c0158f90 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.695+0000 7fa0c4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00b2480 con 0x7fa0c006cf50 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.696+0000 7fa0bf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0c0158f90 0x7fa0c00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.696+0000 7fa0bf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0c0158f90 0x7fa0c00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:49242/0 (socket says 172.21.15.125:49242) 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.696+0000 7fa0bf400640 1 -- 172.21.15.125:0/2308718411 learned_addr learned my addr 172.21.15.125:0/2308718411 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.696+0000 7fa0bf400640 1 -- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c006cf50 msgr2=0x7fa0c0070160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:03:54.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.696+0000 7fa0bf400640 1 --2- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c006cf50 0x7fa0c0070160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.697+0000 7fa0bf400640 1 -- 172.21.15.125:0/2308718411 >> v1:172.21.15.125:6789/0 conn(0x7fa0c0159370 legacy=0x7fa0c006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.697+0000 7fa0bf400640 1 -- 172.21.15.125:0/2308718411 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00b2760 con 0x7fa0c0158f90 2024-01-16T15:03:54.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.703+0000 7fa0bf400640 1 --2- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0158f90 0x7fa0c00ce190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009e50 tx=0x7fa0b802bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa87cfdecab7f03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.705+0000 7fa0be000640 1 -- 172.21.15.125:0/2308718411 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b802caa0 con 0x7fa0c0158f90 2024-01-16T15:03:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.705+0000 7fa0be000640 1 -- 172.21.15.125:0/2308718411 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b802cc00 con 0x7fa0c0158f90 2024-01-16T15:03:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.705+0000 7fa0be000640 1 -- 172.21.15.125:0/2308718411 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80343f0 con 0x7fa0c0158f90 2024-01-16T15:03:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 -- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0158f90 msgr2=0x7fa0c00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0158f90 0x7fa0c00ce190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009e50 tx=0x7fa0b802bf00 comp rx=0 tx=0).stop 2024-01-16T15:03:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 -- 172.21.15.125:0/2308718411 shutdown_connections 2024-01-16T15:03:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c006cf50 0x7fa0c0070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/2308718411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0158f90 0x7fa0c00ce190 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.707+0000 7fa0c4a00640 1 -- 172.21.15.125:0/2308718411 >> 172.21.15.125:0/2308718411 conn(0x7fa0c006a7d0 msgr2=0x7fa0c006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.708+0000 7fa0c4a00640 1 -- 172.21.15.125:0/2308718411 shutdown_connections 2024-01-16T15:03:54.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.708+0000 7fa0c4a00640 1 -- 172.21.15.125:0/2308718411 wait complete. 2024-01-16T15:03:54.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.713+0000 7fa0c4a00640 1 Processor -- start 2024-01-16T15:03:54.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.714+0000 7fa0c4a00640 1 -- start start 2024-01-16T15:03:54.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 0x7fa0c015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 0x7fa0c0149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0146750 con 0x7fa0c006cf50 2024-01-16T15:03:54.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c013ff10 con 0x7fa0c0159370 2024-01-16T15:03:54.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.718+0000 7fa0c4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0140d70 con 0x7fa0c0158f90 2024-01-16T15:03:54.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.719+0000 7fa0bf400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.719+0000 7fa0bf400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50166/0 (socket says 172.21.15.125:50166) 2024-01-16T15:03:54.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.719+0000 7fa0bf400640 1 -- 172.21.15.125:0/1804099700 learned_addr learned my addr 172.21.15.125:0/1804099700 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:54.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.719+0000 7fa0bea00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 0x7fa0c015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.720+0000 7fa0bf400640 1 -- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 msgr2=0x7fa0c015b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.720+0000 7fa0bfe00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 0x7fa0c0149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.721+0000 7fa0bf400640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 0x7fa0c015b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.721+0000 7fa0bf400640 1 -- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 msgr2=0x7fa0c0149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.721+0000 7fa0bf400640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 0x7fa0c0149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.721+0000 7fa0bf400640 1 -- 172.21.15.125:0/1804099700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8009b30 con 0x7fa0c006cf50 2024-01-16T15:03:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.724+0000 7fa0bfe00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 0x7fa0c0149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.724+0000 7fa0bea00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 0x7fa0c015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.724+0000 7fa0bf400640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009e50 tx=0x7fa0b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.725+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b803e450 con 0x7fa0c006cf50 2024-01-16T15:03:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.725+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b803e5b0 con 0x7fa0c006cf50 2024-01-16T15:03:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.725+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b803e940 con 0x7fa0c006cf50 2024-01-16T15:03:54.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.727+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c0140160 con 0x7fa0c006cf50 2024-01-16T15:03:54.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.728+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c0140a00 con 0x7fa0c006cf50 2024-01-16T15:03:54.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.731+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa0b803eaa0 con 0x7fa0c006cf50 2024-01-16T15:03:54.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.733+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0067450 con 0x7fa0c006cf50 2024-01-16T15:03:54.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.734+0000 7fa0b7400640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 0x7fa09806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.735+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0b80ab470 con 0x7fa0c006cf50 2024-01-16T15:03:54.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.745+0000 7fa0bea00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 0x7fa09806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.750+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0b8077c20 con 0x7fa0c006cf50 2024-01-16T15:03:54.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:54.768+0000 7fa0bea00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 0x7fa09806e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa0c015dd00 tx=0x7fa0a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:55.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.291+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa0c0148f00 con 0x7fa0c006cf50 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.292+0000 7fa0b7400640 1 -- 172.21.15.125:0/1804099700 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fa0b807b240 con 0x7fa0c006cf50 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 454, 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 4079, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 16963717261, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 34043887616, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 739050225664, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 189715049, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 187292923, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 46, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 52 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 71.429% (38s)\n [====================........] (remaining: 15s)", 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.71428573131561279, 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 33.333% (38s)\n [=========...................] (remaining: 76s)", 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.3333333432674408, 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:55.278 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.295+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 msgr2=0x7fa09806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.295+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 0x7fa09806e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa0c015dd00 tx=0x7fa0a8009040 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.295+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 msgr2=0x7fa0c0146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.295+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009e50 tx=0x7fa0b8004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 shutdown_connections 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0c0159370 0x7fa0c0149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0c0158f90 0x7fa0c015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa09806bc00 0x7fa09806e0c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 --2- 172.21.15.125:0/1804099700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0c006cf50 0x7fa0c0146210 secure :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009e50 tx=0x7fa0b8004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 >> 172.21.15.125:0/1804099700 conn(0x7fa0c006a7d0 msgr2=0x7fa0c00cb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:55.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.296+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 shutdown_connections 2024-01-16T15:03:55.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:55.301+0000 7fa0c4a00640 1 -- 172.21.15.125:0/1804099700 wait complete. 2024-01-16T15:03:57.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:57.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.672+0000 7fb3be000640 1 Processor -- start 2024-01-16T15:03:57.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.672+0000 7fb3be000640 1 -- start start 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.673+0000 7fb3be000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3b80cf9a0 0x7fb3b8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3be000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb3b8070ea0 0x7fb3b80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3b6a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3b80cf9a0 0x7fb3b8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3b6a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3b80cf9a0 0x7fb3b8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59288/0 (socket says 172.21.15.125:59288) 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3b6a00640 1 -- 172.21.15.125:0/3625376505 learned_addr learned my addr 172.21.15.125:0/3625376505 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b8071690 con 0x7fb3b814b2c0 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b814b6a0 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.674+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b8071970 con 0x7fb3b8070ea0 2024-01-16T15:03:57.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b7e00640 1 --2- 172.21.15.125:0/3625376505 >> v2:172.21.15.137:3300/0 conn(0x7fb3b8070ea0 0x7fb3b80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b6000640 1 -- 172.21.15.125:0/3625376505 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429256290 0 0) 0x7fb3b8071690 con 0x7fb3b814b2c0 2024-01-16T15:03:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b6a00640 1 -- 172.21.15.125:0/3625376505 >> v2:172.21.15.137:3300/0 conn(0x7fb3b8070ea0 msgr2=0x7fb3b80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b6a00640 1 --2- 172.21.15.125:0/3625376505 >> v2:172.21.15.137:3300/0 conn(0x7fb3b8070ea0 0x7fb3b80cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b6a00640 1 -- 172.21.15.125:0/3625376505 >> v1:172.21.15.125:6789/0 conn(0x7fb3b814b2c0 legacy=0x7fb3b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.675+0000 7fb3b6a00640 1 -- 172.21.15.125:0/3625376505 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8071c50 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.679+0000 7fb3b6a00640 1 --2- 172.21.15.125:0/3625376505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b80cf9a0 0x7fb3b8070960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8165330 tx=0x7fb3a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1dc10e1a64faa8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:57.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.681+0000 7fb3b6000640 1 -- 172.21.15.125:0/3625376505 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0030860 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.681+0000 7fb3b6000640 1 -- 172.21.15.125:0/3625376505 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a00309c0 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.681+0000 7fb3b6000640 1 -- 172.21.15.125:0/3625376505 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0030d50 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.683+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b80cf9a0 msgr2=0x7fb3b8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.683+0000 7fb3be000640 1 --2- 172.21.15.125:0/3625376505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b80cf9a0 0x7fb3b8070960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8165330 tx=0x7fb3a002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.685+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 shutdown_connections 2024-01-16T15:03:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.685+0000 7fb3be000640 1 --2- 172.21.15.125:0/3625376505 >> v2:172.21.15.137:3300/0 conn(0x7fb3b8070ea0 0x7fb3b80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.685+0000 7fb3be000640 1 --2- 172.21.15.125:0/3625376505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b80cf9a0 0x7fb3b8070960 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.685+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 >> 172.21.15.125:0/3625376505 conn(0x7fb3b806a800 msgr2=0x7fb3b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:57.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.686+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 shutdown_connections 2024-01-16T15:03:57.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.686+0000 7fb3be000640 1 -- 172.21.15.125:0/3625376505 wait complete. 2024-01-16T15:03:57.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.689+0000 7fb3be000640 1 Processor -- start 2024-01-16T15:03:57.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.689+0000 7fb3be000640 1 -- start start 2024-01-16T15:03:57.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.690+0000 7fb3be000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.690+0000 7fb3be000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3b80cf9a0 0x7fb3b813f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3be000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3b814b2c0 0x7fb3b8142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3be000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b815a4f0 con 0x7fb3b8070ea0 2024-01-16T15:03:57.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3be000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b815a7d0 con 0x7fb3b814b2c0 2024-01-16T15:03:57.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3be000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b815aab0 con 0x7fb3b80cf9a0 2024-01-16T15:03:57.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59296/0 (socket says 172.21.15.125:59296) 2024-01-16T15:03:57.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 -- 172.21.15.125:0/2301167520 learned_addr learned my addr 172.21.15.125:0/2301167520 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 -- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3b80cf9a0 msgr2=0x7fb3b813f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3b80cf9a0 0x7fb3b813f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 -- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3b814b2c0 msgr2=0x7fb3b8142440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3b814b2c0 0x7fb3b8142440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.691+0000 7fb3b7400640 1 -- 172.21.15.125:0/2301167520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00094d0 con 0x7fb3b8070ea0 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3b7400640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800c410 tx=0x7fb3a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:57.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80035f0 con 0x7fb3b8070ea0 2024-01-16T15:03:57.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a8003770 con 0x7fb3b8070ea0 2024-01-16T15:03:57.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8003b20 con 0x7fb3b8070ea0 2024-01-16T15:03:57.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b815ad90 con 0x7fb3b8070ea0 2024-01-16T15:03:57.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.692+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b813f5d0 con 0x7fb3b8070ea0 2024-01-16T15:03:57.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.694+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b80675d0 con 0x7fb3b8070ea0 2024-01-16T15:03:57.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.697+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb3a8003c80 con 0x7fb3b8070ea0 2024-01-16T15:03:57.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.699+0000 7fb3af400640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 0x7fb39006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.700+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3a8087d10 con 0x7fb3b8070ea0 2024-01-16T15:03:57.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.704+0000 7fb3b6a00640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 0x7fb39006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.707+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3a8054380 con 0x7fb3b8070ea0 2024-01-16T15:03:57.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:57.724+0000 7fb3b6a00640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 0x7fb39006dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0036000 tx=0x7fb3a003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:58.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.177+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb3b80b2a20 con 0x7fb3b8070ea0 2024-01-16T15:03:58.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.180+0000 7fb3af400640 1 -- 172.21.15.125:0/2301167520 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fb3a8017990 con 0x7fb3b8070ea0 2024-01-16T15:03:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:03:58.163 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:03:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:03:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:03:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 457, 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 4771, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 19866306304, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 39760863232, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 733333250048, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 320694623, 2024-01-16T15:03:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 313029279, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 77, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 82 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:03:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:03:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:03:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 80.953% (41s)\n [======================......] (remaining: 9s)", 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.80952858924865723, 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "ad6e41df-8e22-4275-9bce-0f3e4871311d": { 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "3 ongoing cloning jobs; Avg progress = 55.557% (41s)\n [===============.............] (remaining: 33s)", 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.55556666851043701, 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:03:58.171 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:03:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.185+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 msgr2=0x7fb39006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.185+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 0x7fb39006dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0036000 tx=0x7fb3a003b040 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.185+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 msgr2=0x7fb3b815b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.185+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800c410 tx=0x7fb3a800c8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 shutdown_connections 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3b814b2c0 0x7fb3b8142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3b80cf9a0 0x7fb3b813f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb39006bb30 0x7fb39006dff0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 --2- 172.21.15.125:0/2301167520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3b8070ea0 0x7fb3b815b590 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 >> 172.21.15.125:0/2301167520 conn(0x7fb3b806a800 msgr2=0x7fb3b80cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 shutdown_connections 2024-01-16T15:03:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:03:58.186+0000 7fb3be000640 1 -- 172.21.15.125:0/2301167520 wait complete. 2024-01-16T15:04:00.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:00.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.605+0000 7fa2da400640 1 Processor -- start 2024-01-16T15:04:00.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.605+0000 7fa2da400640 1 -- start start 2024-01-16T15:04:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.607+0000 7fa2da400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 0x7fa2d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.607+0000 7fa2da400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2d4071070 0x7fa2d40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.608+0000 7fa2da400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d40cfbf0 con 0x7fa2d40cf7d0 2024-01-16T15:04:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.608+0000 7fa2da400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d414b6a0 con 0x7fa2d4071070 2024-01-16T15:04:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.608+0000 7fa2da400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d4071690 con 0x7fa2d414b2c0 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.608+0000 7fa2d3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2d4071070 0x7fa2d40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.609+0000 7fa2d3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2d4071070 0x7fa2d40cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59310/0 (socket says 172.21.15.125:59310) 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.609+0000 7fa2d3e00640 1 -- 172.21.15.125:0/2262784982 learned_addr learned my addr 172.21.15.125:0/2262784982 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.609+0000 7fa2d3e00640 1 -- 172.21.15.125:0/2262784982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 msgr2=0x7fa2d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.609+0000 7fa2d3400640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 0x7fa2d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.610+0000 7fa2d3e00640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 0x7fa2d40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.610+0000 7fa2d3e00640 1 -- 172.21.15.125:0/2262784982 >> v1:172.21.15.137:6789/0 conn(0x7fa2d40cf7d0 legacy=0x7fa2d4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:00.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.610+0000 7fa2d3e00640 1 -- 172.21.15.125:0/2262784982 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d4071970 con 0x7fa2d4071070 2024-01-16T15:04:00.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.610+0000 7fa2d3400640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 0x7fa2d40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:00.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.611+0000 7fa2d3e00640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4071070 0x7fa2d40cc160 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009e50 tx=0x7fa2cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b729e93966a8f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.612+0000 7fa2d2000640 1 -- 172.21.15.125:0/2262784982 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc02ca80 con 0x7fa2d4071070 2024-01-16T15:04:00.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.612+0000 7fa2d2000640 1 -- 172.21.15.125:0/2262784982 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc02cbe0 con 0x7fa2d4071070 2024-01-16T15:04:00.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.612+0000 7fa2d2000640 1 -- 172.21.15.125:0/2262784982 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0343f0 con 0x7fa2d4071070 2024-01-16T15:04:00.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.613+0000 7fa2da400640 1 -- 172.21.15.125:0/2262784982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4071070 msgr2=0x7fa2d40cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.613+0000 7fa2da400640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4071070 0x7fa2d40cc160 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009e50 tx=0x7fa2cc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 -- 172.21.15.125:0/2262784982 shutdown_connections 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4071070 0x7fa2d40cc160 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 --2- 172.21.15.125:0/2262784982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d414b2c0 0x7fa2d40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 -- 172.21.15.125:0/2262784982 >> 172.21.15.125:0/2262784982 conn(0x7fa2d406a800 msgr2=0x7fa2d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 -- 172.21.15.125:0/2262784982 shutdown_connections 2024-01-16T15:04:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.614+0000 7fa2da400640 1 -- 172.21.15.125:0/2262784982 wait complete. 2024-01-16T15:04:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.619+0000 7fa2da400640 1 Processor -- start 2024-01-16T15:04:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.620+0000 7fa2da400640 1 -- start start 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d4071070 0x7fa2d4161440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d40cf7d0 0x7fa2d41605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d40cdf00 con 0x7fa2d414b2c0 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d4160ae0 con 0x7fa2d40cf7d0 2024-01-16T15:04:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.624+0000 7fa2da400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d41632a0 con 0x7fa2d4071070 2024-01-16T15:04:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.626+0000 7fa2d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59312/0 (socket says 172.21.15.125:59312) 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 -- 172.21.15.125:0/849261656 learned_addr learned my addr 172.21.15.125:0/849261656 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 -- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d4071070 msgr2=0x7fa2d4161440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d4071070 0x7fa2d4161440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 -- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d40cf7d0 msgr2=0x7fa2d41605a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d40cf7d0 0x7fa2d41605a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.627+0000 7fa2d3e00640 1 -- 172.21.15.125:0/849261656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc009b30 con 0x7fa2d414b2c0 2024-01-16T15:04:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.628+0000 7fa2d3e00640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009e50 tx=0x7fa2cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.628+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc03e450 con 0x7fa2d414b2c0 2024-01-16T15:04:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.629+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc03e5d0 con 0x7fa2d414b2c0 2024-01-16T15:04:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.629+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc03e980 con 0x7fa2d414b2c0 2024-01-16T15:04:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.630+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d40cd940 con 0x7fa2d414b2c0 2024-01-16T15:04:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.630+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d4143f40 con 0x7fa2d414b2c0 2024-01-16T15:04:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.631+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d40aebc0 con 0x7fa2d414b2c0 2024-01-16T15:04:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.635+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa2cc02cd50 con 0x7fa2d414b2c0 2024-01-16T15:04:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.636+0000 7fa2cb400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 0x7fa2ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.636+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2cc0ab180 con 0x7fa2d414b2c0 2024-01-16T15:04:00.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.643+0000 7fa2d3400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 0x7fa2ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.644+0000 7fa2d3400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 0x7fa2ac06e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400a810 tx=0x7fa2c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:00.644+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2cc077870 con 0x7fa2d414b2c0 2024-01-16T15:04:01.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.199+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa2d4144130 con 0x7fa2d414b2c0 2024-01-16T15:04:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.207+0000 7fa2cb400640 1 -- 172.21.15.125:0/849261656 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7fa2cc043880 con 0x7fa2d414b2c0 2024-01-16T15:04:01.188 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:01.188 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:01.189 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 460, 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:01.190 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:01.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:01.192 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 565258289, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 559002470, 2024-01-16T15:04:01.193 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 135, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 144 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:01.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.195 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:01.196 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:01.198 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:01.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 msgr2=0x7fa2ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 0x7fa2ac06e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400a810 tx=0x7fa2c4008040 comp rx=0 tx=0).stop 2024-01-16T15:04:01.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 msgr2=0x7fa2d413eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009e50 tx=0x7fa2cc004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 shutdown_connections 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa2ac06bc00 0x7fa2ac06e0c0 secure :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400a810 tx=0x7fa2c4008040 comp rx=0 tx=0).stop 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d414b2c0 0x7fa2d413eb40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d40cf7d0 0x7fa2d41605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 --2- 172.21.15.125:0/849261656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d4071070 0x7fa2d4161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.215+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 >> 172.21.15.125:0/849261656 conn(0x7fa2d406a800 msgr2=0x7fa2d4149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.216+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 shutdown_connections 2024-01-16T15:04:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.216+0000 7fa2da400640 1 -- 172.21.15.125:0/849261656 wait complete. 2024-01-16T15:04:01.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:01.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.621+0000 7f8061400640 1 Processor -- start 2024-01-16T15:04:01.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.622+0000 7f8061400640 1 -- start start 2024-01-16T15:04:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.625+0000 7f8061400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.626+0000 7f8061400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 0x7f805c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.626+0000 7f8061400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c15d830 con 0x7f805c15ce90 2024-01-16T15:04:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.626+0000 7f8061400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c153d30 con 0x7f805c06d480 2024-01-16T15:04:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.626+0000 7f8061400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b2b70 con 0x7f805c153950 2024-01-16T15:04:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.627+0000 7f805b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 0x7f805c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.628+0000 7f805a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.628+0000 7f805a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44246/0 (socket says 172.21.15.125:44246) 2024-01-16T15:04:01.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.628+0000 7f805a800640 1 -- 172.21.15.125:0/1126494180 learned_addr learned my addr 172.21.15.125:0/1126494180 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:01.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.630+0000 7f805a800640 1 -- 172.21.15.125:0/1126494180 >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 msgr2=0x7f805c15d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:01.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.631+0000 7f805a800640 1 --2- 172.21.15.125:0/1126494180 >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 0x7f805c15d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.631+0000 7f805a800640 1 -- 172.21.15.125:0/1126494180 >> v1:172.21.15.137:6789/0 conn(0x7f805c15ce90 legacy=0x7f805c1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.631+0000 7f8059400640 1 -- 172.21.15.125:0/1126494180 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458526243 0 0) 0x7f805c15d830 con 0x7f805c15ce90 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.631+0000 7f805a800640 1 -- 172.21.15.125:0/1126494180 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0b2e50 con 0x7f805c153950 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.634+0000 7f805b200640 1 --2- 172.21.15.125:0/1126494180 >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 0x7f805c15d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.635+0000 7f805a800640 1 --2- 172.21.15.125:0/1126494180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8054009e50 tx=0x7f805402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42dad0214bf624c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.636+0000 7f8059400640 1 -- 172.21.15.125:0/1126494180 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805402cb40 con 0x7f805c153950 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.636+0000 7f8059400640 1 -- 172.21.15.125:0/1126494180 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805402cca0 con 0x7f805c153950 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.636+0000 7f8059400640 1 -- 172.21.15.125:0/1126494180 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054034610 con 0x7f805c153950 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.637+0000 7f8061400640 1 -- 172.21.15.125:0/1126494180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 msgr2=0x7f805c0cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.637+0000 7f8061400640 1 --2- 172.21.15.125:0/1126494180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8054009e50 tx=0x7f805402bfa0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.638+0000 7f8061400640 1 -- 172.21.15.125:0/1126494180 shutdown_connections 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.638+0000 7f8061400640 1 --2- 172.21.15.125:0/1126494180 >> v2:172.21.15.137:3300/0 conn(0x7f805c06d480 0x7f805c15d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.638+0000 7f8061400640 1 --2- 172.21.15.125:0/1126494180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c153950 0x7f805c0cb0d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.638+0000 7f8061400640 1 -- 172.21.15.125:0/1126494180 >> 172.21.15.125:0/1126494180 conn(0x7f805c0b0be0 msgr2=0x7f805c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.639+0000 7f8061400640 1 -- 172.21.15.125:0/1126494180 shutdown_connections 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.639+0000 7f8061400640 1 -- 172.21.15.125:0/1126494180 wait complete. 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.643+0000 7f8061400640 1 Processor -- start 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.644+0000 7f8061400640 1 -- start start 2024-01-16T15:04:01.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.644+0000 7f8061400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 0x7f805c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8061400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8061400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c15ce90 0x7f805c1409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8061400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c154770 con 0x7f805c153950 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8061400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c14e0e0 con 0x7f805c06d480 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8061400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0cc6e0 con 0x7f805c15ce90 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8059e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8059e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59318/0 (socket says 172.21.15.125:59318) 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f8059e00640 1 -- 172.21.15.125:0/635865237 learned_addr learned my addr 172.21.15.125:0/635865237 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f805b200640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c15ce90 0x7f805c1409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.645+0000 7f805a800640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 0x7f805c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f8059e00640 1 -- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c15ce90 msgr2=0x7f805c1409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f8059e00640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c15ce90 0x7f805c1409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f8059e00640 1 -- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 msgr2=0x7f805c157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f8059e00640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 0x7f805c157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f8059e00640 1 -- 172.21.15.125:0/635865237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c009400 con 0x7f805c153950 2024-01-16T15:04:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.646+0000 7f805a800640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 0x7f805c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.647+0000 7f8059e00640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f804400c410 tx=0x7f804400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.647+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80440035f0 con 0x7f805c153950 2024-01-16T15:04:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.647+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8044003750 con 0x7f805c153950 2024-01-16T15:04:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.647+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044003b00 con 0x7f805c153950 2024-01-16T15:04:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.647+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8054009b30 con 0x7f805c153950 2024-01-16T15:04:01.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.648+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c154c30 con 0x7f805c153950 2024-01-16T15:04:01.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.649+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c0675d0 con 0x7f805c153950 2024-01-16T15:04:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.652+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f8044003c60 con 0x7f805c153950 2024-01-16T15:04:01.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.653+0000 7f8052a00640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 0x7f803406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:01.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.654+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8044087ba0 con 0x7f805c153950 2024-01-16T15:04:01.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.656+0000 7f805a800640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 0x7f803406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.661+0000 7f805a800640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 0x7f803406dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8054035000 tx=0x7f8054030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:01.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:01.668+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8044054260 con 0x7f805c153950 2024-01-16T15:04:02.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.197+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f805c157250 con 0x7f805c153950 2024-01-16T15:04:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.199+0000 7f8052a00640 1 -- 172.21.15.125:0/635865237 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f8044017090 con 0x7f805c153950 2024-01-16T15:04:02.179 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:02.180 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:02.181 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 461, 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:02.182 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:02.183 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:02.184 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487686888, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487711025, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:02.185 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.186 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.187 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:02.188 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:02.189 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:02.189 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.189 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.189 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:02.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 msgr2=0x7f803406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 0x7f803406dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8054035000 tx=0x7f8054030000 comp rx=0 tx=0).stop 2024-01-16T15:04:02.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 msgr2=0x7f805c140480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f804400c410 tx=0x7f804400c810 comp rx=0 tx=0).stop 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 shutdown_connections 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f805c15ce90 0x7f805c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f803406bb30 0x7f803406dff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c153950 0x7f805c140480 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 --2- 172.21.15.125:0/635865237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f805c06d480 0x7f805c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 >> 172.21.15.125:0/635865237 conn(0x7f805c0b0be0 msgr2=0x7f805c15a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 shutdown_connections 2024-01-16T15:04:02.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.202+0000 7f8061400640 1 -- 172.21.15.125:0/635865237 wait complete. 2024-01-16T15:04:02.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:02.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.414+0000 7f037ca00640 1 Processor -- start 2024-01-16T15:04:02.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.414+0000 7f037ca00640 1 -- start start 2024-01-16T15:04:02.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.415+0000 7f037ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03740cf9a0 0x7f0374070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.415+0000 7f037ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0374070ea0 0x7f03740cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.415+0000 7f037ca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0374071690 con 0x7f037414b2c0 2024-01-16T15:04:02.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.415+0000 7f037ca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037414b6a0 con 0x7f03740cf9a0 2024-01-16T15:04:02.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.415+0000 7f037ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0374071970 con 0x7f0374070ea0 2024-01-16T15:04:02.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.416+0000 7f0379600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03740cf9a0 0x7f0374070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.416+0000 7f0379600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03740cf9a0 0x7f0374070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:55442/0 (socket says 172.21.15.125:55442) 2024-01-16T15:04:02.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.416+0000 7f0379600640 1 -- 172.21.15.125:0/1165859576 learned_addr learned my addr 172.21.15.125:0/1165859576 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:02.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.416+0000 7f037aa00640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0374070ea0 0x7f03740cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.417+0000 7f0379600640 1 -- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0374070ea0 msgr2=0x7f03740cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.417+0000 7f0379600640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0374070ea0 0x7f03740cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.417+0000 7f0379600640 1 -- 172.21.15.125:0/1165859576 >> v1:172.21.15.137:6789/0 conn(0x7f037414b2c0 legacy=0x7f03740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.417+0000 7f0379600640 1 -- 172.21.15.125:0/1165859576 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374071c50 con 0x7f03740cf9a0 2024-01-16T15:04:02.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.418+0000 7f0379600640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03740cf9a0 0x7f0374070960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f035c009870 tx=0x7f035c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=373cde281d476b85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f0378c00640 1 -- 172.21.15.125:0/1165859576 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c030840 con 0x7f03740cf9a0 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f0378c00640 1 -- 172.21.15.125:0/1165859576 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c0309a0 con 0x7f03740cf9a0 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f0378c00640 1 -- 172.21.15.125:0/1165859576 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c030d30 con 0x7f03740cf9a0 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 -- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03740cf9a0 msgr2=0x7f0374070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03740cf9a0 0x7f0374070960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f035c009870 tx=0x7f035c02b980 comp rx=0 tx=0).stop 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 -- 172.21.15.125:0/1165859576 shutdown_connections 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0374070ea0 0x7f03740cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 --2- 172.21.15.125:0/1165859576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03740cf9a0 0x7f0374070960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 -- 172.21.15.125:0/1165859576 >> 172.21.15.125:0/1165859576 conn(0x7f037406a800 msgr2=0x7f037406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 -- 172.21.15.125:0/1165859576 shutdown_connections 2024-01-16T15:04:02.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.419+0000 7f037ca00640 1 -- 172.21.15.125:0/1165859576 wait complete. 2024-01-16T15:04:02.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.421+0000 7f037ca00640 1 Processor -- start 2024-01-16T15:04:02.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.421+0000 7f037ca00640 1 -- start start 2024-01-16T15:04:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0374070ea0 0x7f0374145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 0x7f03741fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03741f7310 con 0x7f037414b2c0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374143050 con 0x7f0374070ea0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.422+0000 7f037ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03741faa80 con 0x7f03740cf9a0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59324/0 (socket says 172.21.15.125:59324) 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 -- 172.21.15.125:0/2247200302 learned_addr learned my addr 172.21.15.125:0/2247200302 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f0379600640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 0x7f03741fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037a000640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0374070ea0 0x7f0374145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 -- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 msgr2=0x7f03741fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 0x7f03741fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 -- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0374070ea0 msgr2=0x7f0374145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0374070ea0 0x7f0374145aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 -- 172.21.15.125:0/2247200302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0094d0 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f0379600640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 0x7f03741fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f037aa00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f036c00efc0 tx=0x7f036c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c014480 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c00f040 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.423+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c014820 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.424+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0374142bc0 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.424+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03741faf90 con 0x7f037414b2c0 2024-01-16T15:04:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.425+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03740675d0 con 0x7f037414b2c0 2024-01-16T15:04:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.426+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f036c014980 con 0x7f037414b2c0 2024-01-16T15:04:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.427+0000 7f0372000640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 0x7f034c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.427+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f036c08b6a0 con 0x7f037414b2c0 2024-01-16T15:04:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.427+0000 7f037a000640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 0x7f034c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.428+0000 7f037a000640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 0x7f034c06dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f03741479f0 tx=0x7f0364009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.431+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f036c0558b0 con 0x7f037414b2c0 2024-01-16T15:04:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.680+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f037406caa0 con 0x7f037414b2c0 2024-01-16T15:04:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.680+0000 7f0372000640 1 -- 172.21.15.125:0/2247200302 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f036c018020 con 0x7f037414b2c0 2024-01-16T15:04:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:02.660 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 461, 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:02.664 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487686888, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487711025, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:02.665 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.666 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.667 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:02.669 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 msgr2=0x7f034c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 0x7f034c06dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f03741479f0 tx=0x7f0364009040 comp rx=0 tx=0).stop 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 msgr2=0x7f0374142680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f036c00efc0 tx=0x7f036c00c490 comp rx=0 tx=0).stop 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 shutdown_connections 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f034c06bb30 0x7f034c06dff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f037414b2c0 0x7f0374142680 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03740cf9a0 0x7f03741fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.682+0000 7f037ca00640 1 --2- 172.21.15.125:0/2247200302 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0374070ea0 0x7f0374145aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.683+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 >> 172.21.15.125:0/2247200302 conn(0x7f037406a800 msgr2=0x7f03740cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.683+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 shutdown_connections 2024-01-16T15:04:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.683+0000 7f037ca00640 1 -- 172.21.15.125:0/2247200302 wait complete. 2024-01-16T15:04:02.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:02.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.896+0000 7f0fbca00640 1 Processor -- start 2024-01-16T15:04:02.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.896+0000 7f0fbca00640 1 -- start start 2024-01-16T15:04:02.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.898+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.898+0000 7f0fbca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0fb806d160 0x7f0fb806d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.898+0000 7f0fbca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb8070e50 con 0x7f0fb8143c40 2024-01-16T15:04:02.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.898+0000 7f0fbca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb80723c0 con 0x7f0fb806d160 2024-01-16T15:04:02.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.898+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8144020 con 0x7f0fb80cc500 2024-01-16T15:04:02.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0fb806d160 0x7f0fb806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fadc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0fb806d160 0x7f0fb806d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:55464/0 (socket says 172.21.15.125:55464) 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 -- 172.21.15.125:0/2850151715 learned_addr learned my addr 172.21.15.125:0/2850151715 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 -- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 msgr2=0x7f0fb80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 --2- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb80cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 -- 172.21.15.125:0/2850151715 >> v1:172.21.15.137:6789/0 conn(0x7f0fb8143c40 legacy=0x7f0fb8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.899+0000 7f0fb7200640 1 -- 172.21.15.125:0/2850151715 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb80ccea0 con 0x7f0fb806d160 2024-01-16T15:04:02.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.901+0000 7f0fb7200640 1 --2- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb806d160 0x7f0fb806d590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009e50 tx=0x7f0fb002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69aa1e9e98d9a9f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.901+0000 7f0fb5e00640 1 -- 172.21.15.125:0/2850151715 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb002ca80 con 0x7f0fb806d160 2024-01-16T15:04:02.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fb5e00640 1 -- 172.21.15.125:0/2850151715 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb002cbe0 con 0x7f0fb806d160 2024-01-16T15:04:02.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fb5e00640 1 -- 172.21.15.125:0/2850151715 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00343f0 con 0x7f0fb806d160 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 -- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb806d160 msgr2=0x7f0fb806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 --2- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb806d160 0x7f0fb806d590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009e50 tx=0x7f0fb002bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 -- 172.21.15.125:0/2850151715 shutdown_connections 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 --2- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb806d160 0x7f0fb806d590 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 --2- 172.21.15.125:0/2850151715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb80cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 -- 172.21.15.125:0/2850151715 >> 172.21.15.125:0/2850151715 conn(0x7f0fb806a800 msgr2=0x7f0fb806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 -- 172.21.15.125:0/2850151715 shutdown_connections 2024-01-16T15:04:02.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.902+0000 7f0fbca00640 1 -- 172.21.15.125:0/2850151715 wait complete. 2024-01-16T15:04:02.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.904+0000 7f0fbca00640 1 Processor -- start 2024-01-16T15:04:02.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.905+0000 7f0fbca00640 1 -- start start 2024-01-16T15:04:02.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb813c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb8143c40 0x7f0fb80ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb81e0710 con 0x7f0fb806d160 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80ace60 con 0x7f0fb8143c40 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80ad140 con 0x7f0fb80cc500 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59340/0 (socket says 172.21.15.125:59340) 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 -- 172.21.15.125:0/872073518 learned_addr learned my addr 172.21.15.125:0/872073518 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fadc00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb813c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 -- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 msgr2=0x7f0fb813c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb813c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 -- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb8143c40 msgr2=0x7f0fb80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb7200640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb8143c40 0x7f0fb80ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.906+0000 7f0fb6800640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb8143c40 0x7f0fb80ac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0fb6800640 1 -- 172.21.15.125:0/872073518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0009b30 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0fb6800640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c400 tx=0x7f0fa800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80052d0 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa8005430 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb80ad420 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8003830 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.907+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb80ad9e0 con 0x7f0fb806d160 2024-01-16T15:04:02.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.908+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb80ce860 con 0x7f0fb806d160 2024-01-16T15:04:02.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.909+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f0fa8003990 con 0x7f0fb806d160 2024-01-16T15:04:02.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.910+0000 7f0faf400640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 0x7f0f9006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.910+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fa8087b70 con 0x7f0fb806d160 2024-01-16T15:04:02.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.910+0000 7f0fadc00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 0x7f0f9006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.911+0000 7f0fadc00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 0x7f0f9006dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80cd660 tx=0x7f0fa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:02.914+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fa8054260 con 0x7f0fb806d160 2024-01-16T15:04:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.152+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0fb8067450 con 0x7f0fb806d160 2024-01-16T15:04:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.153+0000 7f0faf400640 1 -- 172.21.15.125:0/872073518 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f0fa8017880 con 0x7f0fb806d160 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:03.133 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:03.134 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 462, 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:03.135 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:03.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:03.137 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487686888, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487711025, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:03.138 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.139 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:03.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.141 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.142 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.155+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 msgr2=0x7f0f9006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.155+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 0x7f0f9006dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80cd660 tx=0x7f0fa0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.155+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 msgr2=0x7f0fb80adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.155+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c400 tx=0x7f0fa800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.155+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 shutdown_connections 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb8143c40 0x7f0fb80ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb80cc500 0x7f0fb813c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f9006bb30 0x7f0f9006dff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 --2- 172.21.15.125:0/872073518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fb806d160 0x7f0fb80adf00 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 >> 172.21.15.125:0/872073518 conn(0x7f0fb806a800 msgr2=0x7f0fb806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 shutdown_connections 2024-01-16T15:04:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.156+0000 7f0fbca00640 1 -- 172.21.15.125:0/872073518 wait complete. 2024-01-16T15:04:03.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:03.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.365+0000 7f6d47800640 1 Processor -- start 2024-01-16T15:04:03.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.365+0000 7f6d47800640 1 -- start start 2024-01-16T15:04:03.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.366+0000 7f6d47800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d47800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d400b32e0 0x7f6d4015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d47800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4015dc80 con 0x7f6d4015d300 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d47800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d400b3770 con 0x7f6d400b32e0 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d47800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d40071190 con 0x7f6d40150760 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d44e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d45800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d400b32e0 0x7f6d4015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d44e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44282/0 (socket says 172.21.15.125:44282) 2024-01-16T15:04:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.367+0000 7f6d44e00640 1 -- 172.21.15.125:0/2873903849 learned_addr learned my addr 172.21.15.125:0/2873903849 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:03.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.368+0000 7f6d44e00640 1 -- 172.21.15.125:0/2873903849 >> v2:172.21.15.137:3300/0 conn(0x7f6d400b32e0 msgr2=0x7f6d4015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.368+0000 7f6d44e00640 1 --2- 172.21.15.125:0/2873903849 >> v2:172.21.15.137:3300/0 conn(0x7f6d400b32e0 0x7f6d4015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.368+0000 7f6d44e00640 1 -- 172.21.15.125:0/2873903849 >> v1:172.21.15.137:6789/0 conn(0x7f6d4015d300 legacy=0x7f6d400b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.368+0000 7f6d44e00640 1 -- 172.21.15.125:0/2873903849 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d40071470 con 0x7f6d40150760 2024-01-16T15:04:03.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.370+0000 7f6d44e00640 1 --2- 172.21.15.125:0/2873903849 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d44d164ad0da30f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.370+0000 7f6d3fe00640 1 -- 172.21.15.125:0/2873903849 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30030840 con 0x7f6d40150760 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.370+0000 7f6d3fe00640 1 -- 172.21.15.125:0/2873903849 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d300309a0 con 0x7f6d40150760 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.370+0000 7f6d3fe00640 1 -- 172.21.15.125:0/2873903849 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30030d30 con 0x7f6d40150760 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 -- 172.21.15.125:0/2873903849 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 msgr2=0x7f6d40072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 --2- 172.21.15.125:0/2873903849 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b980 comp rx=0 tx=0).stop 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 -- 172.21.15.125:0/2873903849 shutdown_connections 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 --2- 172.21.15.125:0/2873903849 >> v2:172.21.15.137:3300/0 conn(0x7f6d400b32e0 0x7f6d4015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 --2- 172.21.15.125:0/2873903849 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d40150760 0x7f6d40072230 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 -- 172.21.15.125:0/2873903849 >> 172.21.15.125:0/2873903849 conn(0x7f6d4006a800 msgr2=0x7f6d4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:03.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 -- 172.21.15.125:0/2873903849 shutdown_connections 2024-01-16T15:04:03.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.371+0000 7f6d47800640 1 -- 172.21.15.125:0/2873903849 wait complete. 2024-01-16T15:04:03.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.373+0000 7f6d47800640 1 Processor -- start 2024-01-16T15:04:03.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.374+0000 7f6d47800640 1 -- start start 2024-01-16T15:04:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d400b32e0 0x7f6d40159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 0x7f6d400cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4014c6c0 con 0x7f6d40150760 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4014e820 con 0x7f6d4015d300 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d47800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4014def0 con 0x7f6d400b32e0 2024-01-16T15:04:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d37c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d37c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59356/0 (socket says 172.21.15.125:59356) 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d37c00640 1 -- 172.21.15.125:0/3252095888 learned_addr learned my addr 172.21.15.125:0/3252095888 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d44e00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d400b32e0 0x7f6d40159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.375+0000 7f6d45800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 0x7f6d400cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 -- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d400b32e0 msgr2=0x7f6d40159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d400b32e0 0x7f6d40159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 -- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 msgr2=0x7f6d400cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 0x7f6d400cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 -- 172.21.15.125:0/3252095888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d300094d0 con 0x7f6d40150760 2024-01-16T15:04:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d45800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 0x7f6d400cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d37c00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c450 tx=0x7f6d2800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d280052d0 con 0x7f6d40150760 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d28005430 con 0x7f6d40150760 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4014e1d0 con 0x7f6d40150760 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28012870 con 0x7f6d40150760 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.376+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4014dba0 con 0x7f6d40150760 2024-01-16T15:04:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.378+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4015d740 con 0x7f6d40150760 2024-01-16T15:04:03.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.378+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6d280129d0 con 0x7f6d40150760 2024-01-16T15:04:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.379+0000 7f6d3d600640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 0x7f6d1806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.379+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d28088170 con 0x7f6d40150760 2024-01-16T15:04:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.380+0000 7f6d44e00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 0x7f6d1806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.380+0000 7f6d44e00640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 0x7f6d1806dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d30036000 tx=0x7f6d30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.383+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d28054860 con 0x7f6d40150760 2024-01-16T15:04:03.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.623+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6d4014e990 con 0x7f6d40150760 2024-01-16T15:04:03.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.624+0000 7f6d3d600640 1 -- 172.21.15.125:0/3252095888 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f6d28057e80 con 0x7f6d40150760 2024-01-16T15:04:03.604 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:03.604 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:03.604 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:03.604 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:03.605 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 462, 2024-01-16T15:04:03.606 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:03.607 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487686888, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487711025, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:03.610 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:03.611 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:03.612 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:03.613 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:03.614 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:03.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 msgr2=0x7f6d1806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 0x7f6d1806dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d30036000 tx=0x7f6d30008030 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 msgr2=0x7f6d400cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c450 tx=0x7f6d2800c910 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 shutdown_connections 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d4015d300 0x7f6d400cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1806bb30 0x7f6d1806dff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d40150760 0x7f6d400cd010 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 --2- 172.21.15.125:0/3252095888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d400b32e0 0x7f6d40159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.626+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 >> 172.21.15.125:0/3252095888 conn(0x7f6d4006a800 msgr2=0x7f6d4006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:03.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.627+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 shutdown_connections 2024-01-16T15:04:03.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.627+0000 7f6d47800640 1 -- 172.21.15.125:0/3252095888 wait complete. 2024-01-16T15:04:03.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.844+0000 7f53fd600640 1 Processor -- start 2024-01-16T15:04:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.845+0000 7f53fd600640 1 -- start start 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.846+0000 7f53fd600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f53f8071410 0x7f53f8146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.846+0000 7f53fd600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53f80717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.846+0000 7f53fd600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f806cc20 con 0x7f53f8071d30 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.846+0000 7f53fd600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f806cec0 con 0x7f53f8071410 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.846+0000 7f53fd600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f806d1a0 con 0x7f53f806c7c0 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53fcc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f53f8071d30 0x7f53f80b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48330/0 (socket says 172.21.15.125:48330) 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53fcc00640 1 -- 172.21.15.125:0/672199344 learned_addr learned my addr 172.21.15.125:0/672199344 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7e00640 1 --2- 172.21.15.125:0/672199344 >> v2:172.21.15.137:3300/0 conn(0x7f53f8071410 0x7f53f8146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53ef200640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53f80717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96995396 0 0) 0x7f53f806cc20 con 0x7f53f8071d30 2024-01-16T15:04:03.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d8003520 con 0x7f53f8071d30 2024-01-16T15:04:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118448013 0 0) 0x7f53d8003520 con 0x7f53f8071d30 2024-01-16T15:04:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 msgr2=0x7f53f80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53f80717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 >> v2:172.21.15.137:3300/0 conn(0x7f53f8071410 msgr2=0x7f53f8146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.847+0000 7f53f7400640 1 --2- 172.21.15.125:0/672199344 >> v2:172.21.15.137:3300/0 conn(0x7f53f8071410 0x7f53f8146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f806d480 con 0x7f53f8071d30 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f53f00019f0 con 0x7f53f8071d30 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 >> v1:172.21.15.125:6789/0 conn(0x7f53f8071d30 legacy=0x7f53f80b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53d8003970 0x7f53d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53d8004d00 0x7f53d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f806cec0 con 0x7f53d8003970 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f806d1a0 con 0x7f53d8004d00 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d800b980 con 0x7f53f806c7c0 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53d8003970 0x7f53d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53ef200640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 -- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 msgr2=0x7f53d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53fcc00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53d8004d00 0x7f53d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53f806c7c0 0x7f53d80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 -- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53d8004d00 msgr2=0x7f53d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53d8004d00 0x7f53d8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53f7e00640 1 -- 172.21.15.125:0/672199344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d8003520 con 0x7f53d8003970 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.848+0000 7f53fcc00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53d8004d00 0x7f53d8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.849+0000 7f53f7e00640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53d8003970 0x7f53d8003d90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009af0 tx=0x7f53d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548ede5d0d173b37 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e802c710 con 0x7f53d8003970 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e802c870 con 0x7f53d8003970 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53f7400640 1 -- 172.21.15.125:0/672199344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e802cc00 con 0x7f53d8003970 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fd600640 1 -- 172.21.15.125:0/672199344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53d8003970 msgr2=0x7f53d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fd600640 1 --2- 172.21.15.125:0/672199344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53d8003970 0x7f53d8003d90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009af0 tx=0x7f53d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fcc00640 1 -- 172.21.15.125:0/672199344 reap_dead start 2024-01-16T15:04:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fd600640 1 -- 172.21.15.125:0/672199344 shutdown_connections 2024-01-16T15:04:03.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fd600640 1 -- 172.21.15.125:0/672199344 >> 172.21.15.125:0/672199344 conn(0x7f53f806a800 msgr2=0x7f53f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:03.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.850+0000 7f53fd600640 1 -- 172.21.15.125:0/672199344 shutdown_connections 2024-01-16T15:04:03.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.851+0000 7f53fd600640 1 -- 172.21.15.125:0/672199344 wait complete. 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.852+0000 7f53fd600640 1 Processor -- start 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.852+0000 7f53fd600640 1 -- start start 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53d8003970 0x7f53f80b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 0x7f53f813a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f813abd0 con 0x7f53f813a720 2024-01-16T15:04:03.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f813d060 con 0x7f53f813e100 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.853+0000 7f53fd600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f813d340 con 0x7f53d8003970 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59376/0 (socket says 172.21.15.125:59376) 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 -- 172.21.15.125:0/2939428464 learned_addr learned my addr 172.21.15.125:0/2939428464 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53f7e00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53d8003970 0x7f53f80b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53ef200640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 0x7f53f813a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 -- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53d8003970 msgr2=0x7f53f80b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53d8003970 0x7f53f80b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 -- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 msgr2=0x7f53f813a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 0x7f53f813a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53fcc00640 1 -- 172.21.15.125:0/2939428464 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f813d620 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.854+0000 7f53ef200640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 0x7f53f813a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53fcc00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f53f0000f80 tx=0x7f53f000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f000de90 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f000bb80 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f813d900 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0007070 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.855+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f80b2df0 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.856+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f813de20 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.857+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f53f00071d0 con 0x7f53f813a720 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.858+0000 7f53f4c00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 0x7f53d006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.858+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53f0088a40 con 0x7f53f813a720 2024-01-16T15:04:03.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.858+0000 7f53f7e00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 0x7f53d006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.859+0000 7f53f7e00640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 0x7f53d006e020 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53e8036000 tx=0x7f53e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:03.862+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53f813de20 con 0x7f53f813a720 2024-01-16T15:04:04.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.124+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f53f80b2fb0 con 0x7f53f813a720 2024-01-16T15:04:04.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.125+0000 7f53f4c00640 1 -- 172.21.15.125:0/2939428464 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f53f0055130 con 0x7f53f813a720 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.105 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:04.106 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 463, 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:04.107 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:04.108 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487686888, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487711025, 2024-01-16T15:04:04.109 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:04.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:04.111 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.112 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (42s)\n [========================....] (remaining: 7s)", 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.8571428656578064, 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.114 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.127+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 msgr2=0x7f53d006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.127+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 0x7f53d006e020 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53e8036000 tx=0x7f53e8030000 comp rx=0 tx=0).stop 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.127+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 msgr2=0x7f53f813c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.127+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f53f0000f80 tx=0x7f53f000b4a0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 shutdown_connections 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53d006bb60 0x7f53d006e020 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f813a720 0x7f53f813c330 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53f813e100 0x7f53f813a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 --2- 172.21.15.125:0/2939428464 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53d8003970 0x7f53f80b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 >> 172.21.15.125:0/2939428464 conn(0x7f53f806a800 msgr2=0x7f53f80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 shutdown_connections 2024-01-16T15:04:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.128+0000 7f53fd600640 1 -- 172.21.15.125:0/2939428464 wait complete. 2024-01-16T15:04:04.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:04.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.383+0000 7faaede00640 1 Processor -- start 2024-01-16T15:04:04.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.383+0000 7faaede00640 1 -- start start 2024-01-16T15:04:04.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faaede00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faae80cf9a0 0x7faae815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faaede00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faae815e1e0 0x7faae80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faaede00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae80b1ee0 con 0x7faae8143ae0 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faaede00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae80b2780 con 0x7faae815e1e0 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faaede00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae815e990 con 0x7faae80cf9a0 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.385+0000 7faae7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faae815e1e0 0x7faae80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faae815e1e0 0x7faae80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59380/0 (socket says 172.21.15.125:59380) 2024-01-16T15:04:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 -- 172.21.15.125:0/1651842239 learned_addr learned my addr 172.21.15.125:0/1651842239 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 -- 172.21.15.125:0/1651842239 >> v2:172.21.15.137:3300/0 conn(0x7faae80cf9a0 msgr2=0x7faae815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 --2- 172.21.15.125:0/1651842239 >> v2:172.21.15.137:3300/0 conn(0x7faae80cf9a0 0x7faae815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 -- 172.21.15.125:0/1651842239 >> v1:172.21.15.137:6789/0 conn(0x7faae8143ae0 legacy=0x7faae80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.386+0000 7faae7400640 1 -- 172.21.15.125:0/1651842239 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80b2130 con 0x7faae815e1e0 2024-01-16T15:04:04.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.388+0000 7faae7400640 1 --2- 172.21.15.125:0/1651842239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae815e1e0 0x7faae80702c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7faae80cb830 tx=0x7faae002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14c53c82ef3490c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.388+0000 7faae5600640 1 -- 172.21.15.125:0/1651842239 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae002ea90 con 0x7faae815e1e0 2024-01-16T15:04:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.388+0000 7faae5600640 1 -- 172.21.15.125:0/1651842239 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae002ebf0 con 0x7faae815e1e0 2024-01-16T15:04:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.388+0000 7faae5600640 1 -- 172.21.15.125:0/1651842239 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0009600 con 0x7faae815e1e0 2024-01-16T15:04:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 -- 172.21.15.125:0/1651842239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae815e1e0 msgr2=0x7faae80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 --2- 172.21.15.125:0/1651842239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae815e1e0 0x7faae80702c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7faae80cb830 tx=0x7faae002e0f0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 -- 172.21.15.125:0/1651842239 shutdown_connections 2024-01-16T15:04:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 --2- 172.21.15.125:0/1651842239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae815e1e0 0x7faae80702c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 --2- 172.21.15.125:0/1651842239 >> v2:172.21.15.137:3300/0 conn(0x7faae80cf9a0 0x7faae815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 -- 172.21.15.125:0/1651842239 >> 172.21.15.125:0/1651842239 conn(0x7faae806a800 msgr2=0x7faae806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 -- 172.21.15.125:0/1651842239 shutdown_connections 2024-01-16T15:04:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.390+0000 7faaede00640 1 -- 172.21.15.125:0/1651842239 wait complete. 2024-01-16T15:04:04.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.394+0000 7faaede00640 1 Processor -- start 2024-01-16T15:04:04.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.394+0000 7faaede00640 1 -- start start 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faae8143ae0 0x7faae80cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 0x7faae81f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80ce2c0 con 0x7faae80cf9a0 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae81f3090 con 0x7faae8143ae0 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.396+0000 7faaede00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae81f6bc0 con 0x7faae815e1e0 2024-01-16T15:04:04.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.397+0000 7faae6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.397+0000 7faae6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59382/0 (socket says 172.21.15.125:59382) 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.397+0000 7faae6000640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faae8143ae0 0x7faae80cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.398+0000 7faae7400640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 0x7faae81f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.397+0000 7faae6a00640 1 -- 172.21.15.125:0/1463141056 learned_addr learned my addr 172.21.15.125:0/1463141056 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae7400640 1 -- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faae8143ae0 msgr2=0x7faae80cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae7400640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faae8143ae0 0x7faae80cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae7400640 1 -- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 msgr2=0x7faae813d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae7400640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae7400640 1 -- 172.21.15.125:0/1463141056 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae000bd40 con 0x7faae815e1e0 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.399+0000 7faae6a00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.400+0000 7faae7400640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 0x7faae81f6680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faae0034000 tx=0x7faae0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.401+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0036800 con 0x7faae815e1e0 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.401+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0036980 con 0x7faae815e1e0 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.401+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0036d70 con 0x7faae815e1e0 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.401+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae81f6e90 con 0x7faae815e1e0 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.401+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae81f7450 con 0x7faae815e1e0 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.403+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae806e0d0 con 0x7faae815e1e0 2024-01-16T15:04:04.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.407+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7faae0040420 con 0x7faae815e1e0 2024-01-16T15:04:04.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.408+0000 7faadf400640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 0x7faac006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.408+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faae00ac420 con 0x7faae815e1e0 2024-01-16T15:04:04.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.412+0000 7faae6a00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 0x7faac006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.412+0000 7faae6a00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 0x7faac006e040 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7faad00043e0 tx=0x7faad0007ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.421+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faae0078c70 con 0x7faae815e1e0 2024-01-16T15:04:04.889 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:04.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.908+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faae806e0d0 con 0x7faae815e1e0 2024-01-16T15:04:04.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.909+0000 7faadf400640 1 -- 172.21.15.125:0/1463141056 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7faae000b720 con 0x7faae815e1e0 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 463, 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:04.891 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:04.892 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487714335, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487738473, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:04.893 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:04.894 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:04.895 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:04.896 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:04.896 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:04.896 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.896 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:04.896 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.916+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 msgr2=0x7faac006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.916+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 0x7faac006e040 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7faad00043e0 tx=0x7faad0007ef0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.916+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 msgr2=0x7faae81f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.916+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 0x7faae81f6680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faae0034000 tx=0x7faae0004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 shutdown_connections 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faae815e1e0 0x7faae81f6680 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faae8143ae0 0x7faae80cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faac006bb80 0x7faac006e040 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 --2- 172.21.15.125:0/1463141056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faae80cf9a0 0x7faae813d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 >> 172.21.15.125:0/1463141056 conn(0x7faae806a800 msgr2=0x7faae8070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 shutdown_connections 2024-01-16T15:04:04.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:04.917+0000 7faaede00640 1 -- 172.21.15.125:0/1463141056 wait complete. 2024-01-16T15:04:04.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:05.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.267+0000 7fc5fe000640 1 Processor -- start 2024-01-16T15:04:05.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.270+0000 7fc5fe000640 1 -- start start 2024-01-16T15:04:05.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5fe000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc5f8150760 0x7fc5f8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5fe000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80b3710 con 0x7fc5f815d300 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5fe000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f8071190 con 0x7fc5f80b32e0 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5fe000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f8071470 con 0x7fc5f8150760 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5f7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc5f8150760 0x7fc5f8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5f7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc5f8150760 0x7fc5f8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59388/0 (socket says 172.21.15.125:59388) 2024-01-16T15:04:05.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5efe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc5f815d300 0x7fc5f80b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48336/0 (socket says 172.21.15.125:48336) 2024-01-16T15:04:05.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.273+0000 7fc5f7400640 1 -- 172.21.15.125:0/1315210761 learned_addr learned my addr 172.21.15.125:0/1315210761 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:05.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.274+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137748318 0 0) 0x7fc5f80b3710 con 0x7fc5f815d300 2024-01-16T15:04:05.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.275+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8004630 con 0x7fc5f815d300 2024-01-16T15:04:05.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.275+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295732633 0 0) 0x7fc5f8071190 con 0x7fc5f80b32e0 2024-01-16T15:04:05.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.275+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f80b3710 con 0x7fc5f80b32e0 2024-01-16T15:04:05.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126027051 0 0) 0x7fc5d8004630 con 0x7fc5f815d300 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 >> v2:172.21.15.125:3300/0 conn(0x7fc5f8150760 msgr2=0x7fc5f8072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 --2- 172.21.15.125:0/1315210761 >> v2:172.21.15.125:3300/0 conn(0x7fc5f8150760 0x7fc5f8072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 >> v1:172.21.15.137:6789/0 conn(0x7fc5f80b32e0 legacy=0x7fc5f80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8071750 con 0x7fc5f815d300 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc5e0001670 con 0x7fc5f815d300 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 >> v1:172.21.15.125:6789/0 conn(0x7fc5f815d300 legacy=0x7fc5f80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 0x7fc5d80060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5d8004870 0x7fc5d8004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5d8005230 0x7fc5d8003440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8071470 con 0x7fc5d8005cc0 2024-01-16T15:04:05.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8003980 con 0x7fc5d8005230 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.276+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8003b40 con 0x7fc5d8004870 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5efe00640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5d8004870 0x7fc5d8004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7e00640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5d8005230 0x7fc5d8003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 0x7fc5d80060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 -- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5d8004870 msgr2=0x7fc5d8004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5d8004870 0x7fc5d8004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 -- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5d8005230 msgr2=0x7fc5d8003440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5d8005230 0x7fc5d8003440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.277+0000 7fc5f7400640 1 -- 172.21.15.125:0/1315210761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8004630 con 0x7fc5d8005cc0 2024-01-16T15:04:05.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.278+0000 7fc5f7e00640 1 -- 172.21.15.125:0/1315210761 reap_dead start 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.279+0000 7fc5f7400640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 0x7fc5d80060c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009c10 tx=0x7fc5d8006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1164a03e8516a10f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.279+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e802c7b0 con 0x7fc5d8005cc0 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.279+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e802c910 con 0x7fc5d8005cc0 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5f6a00640 1 -- 172.21.15.125:0/1315210761 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e802cca0 con 0x7fc5d8005cc0 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 -- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 msgr2=0x7fc5d80060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 0x7fc5d80060c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009c10 tx=0x7fc5d8006900 comp rx=0 tx=0).stop 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 -- 172.21.15.125:0/1315210761 shutdown_connections 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 --2- 172.21.15.125:0/1315210761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5d8005cc0 0x7fc5d80060c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 -- 172.21.15.125:0/1315210761 >> 172.21.15.125:0/1315210761 conn(0x7fc5f806a800 msgr2=0x7fc5f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 -- 172.21.15.125:0/1315210761 shutdown_connections 2024-01-16T15:04:05.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.280+0000 7fc5fe000640 1 -- 172.21.15.125:0/1315210761 wait complete. 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.284+0000 7fc5fe000640 1 Processor -- start 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.284+0000 7fc5fe000640 1 -- start start 2024-01-16T15:04:05.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5f81393d0 0x7fc5f81468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5f8144ca0 0x7fc5f8145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f814d1b0 con 0x7fc5f81393d0 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8145810 con 0x7fc5f814c7f0 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5fe000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8145af0 con 0x7fc5f8144ca0 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5f7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5f7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:55556/0 (socket says 172.21.15.125:55556) 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.286+0000 7fc5f7e00640 1 -- 172.21.15.125:0/2761981604 learned_addr learned my addr 172.21.15.125:0/2761981604 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7400640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5f81393d0 0x7fc5f81468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7e00640 1 -- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5f8144ca0 msgr2=0x7fc5f8145120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7e00640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5f8144ca0 0x7fc5f8145120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7e00640 1 -- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5f81393d0 msgr2=0x7fc5f81468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7e00640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5f81393d0 0x7fc5f81468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.287+0000 7fc5f7e00640 1 -- 172.21.15.125:0/2761981604 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8145dd0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.288+0000 7fc5f7e00640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5f806e190 tx=0x7fc5f000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.290+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0003970 con 0x7fc5f814c7f0 2024-01-16T15:04:05.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.290+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f0003ad0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.290+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0003e80 con 0x7fc5f814c7f0 2024-01-16T15:04:05.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.290+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f81460b0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.290+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f81397b0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.292+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f80675d0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.304+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc5f00050d0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.306+0000 7fc5ef400640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.306+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5f00888a0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.306+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5f008b3a0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.308+0000 7fc5f7400640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.309+0000 7fc5f7400640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 0x7fc5d006e0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8036000 tx=0x7fc5e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.791+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc5f80b26b0 con 0x7fc5f814c7f0 2024-01-16T15:04:05.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.796+0000 7fc5ef400640 1 -- 172.21.15.125:0/2761981604 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7fc5f0054f90 con 0x7fc5f814c7f0 2024-01-16T15:04:05.776 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 464, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487714335, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487738473, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:05.782 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:05.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.800+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 msgr2=0x7fc5d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.800+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 0x7fc5d006e0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8036000 tx=0x7fc5e8030000 comp rx=0 tx=0).stop 2024-01-16T15:04:05.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.800+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 msgr2=0x7fc5f814cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.800+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5f806e190 tx=0x7fc5f000eb90 comp rx=0 tx=0).stop 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 shutdown_connections 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5f814c7f0 0x7fc5f814cc70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5f8144ca0 0x7fc5f8145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 --2- 172.21.15.125:0/2761981604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5f81393d0 0x7fc5f81468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 >> 172.21.15.125:0/2761981604 conn(0x7fc5f806a800 msgr2=0x7fc5f815ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 shutdown_connections 2024-01-16T15:04:05.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:05.802+0000 7fc5fe000640 1 -- 172.21.15.125:0/2761981604 wait complete. 2024-01-16T15:04:05.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:06.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.141+0000 7f0caca00640 1 Processor -- start 2024-01-16T15:04:06.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.141+0000 7f0caca00640 1 -- start start 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.151+0000 7f0caca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ca40b3970 0x7f0ca415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.151+0000 7f0caca00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406d150 con 0x7f0ca40aee10 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.151+0000 7f0caca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406dcd0 con 0x7f0ca40b2780 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.151+0000 7f0caca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca406cbc0 con 0x7f0ca40b3970 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.152+0000 7f0caaa00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f0ca40b2780 0x7f0ca4156a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:38856/0 (socket says 172.21.15.125:38856) 2024-01-16T15:04:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.152+0000 7f0caaa00640 1 -- 172.21.15.125:0/3180065290 learned_addr learned my addr 172.21.15.125:0/3180065290 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.155+0000 7f0ca9600640 1 --2- 172.21.15.125:0/3180065290 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40b3970 0x7f0ca415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.155+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515418270 0 0) 0x7f0ca406d150 con 0x7f0ca40aee10 2024-01-16T15:04:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.155+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c84003520 con 0x7f0ca40aee10 2024-01-16T15:04:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.156+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668940651 0 0) 0x7f0c84003520 con 0x7f0ca40aee10 2024-01-16T15:04:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.156+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40b3970 msgr2=0x7f0ca415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.156+0000 7f0ca8c00640 1 --2- 172.21.15.125:0/3180065290 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40b3970 0x7f0ca415a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.156+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 >> v1:172.21.15.137:6789/0 conn(0x7f0ca40b2780 legacy=0x7f0ca4156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.156+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40af1f0 con 0x7f0ca40aee10 2024-01-16T15:04:06.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0c8c001670 con 0x7f0ca40aee10 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 >> v1:172.21.15.125:6789/0 conn(0x7f0ca40aee10 legacy=0x7f0ca40b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 0x7f0c84003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840047c0 0x7f0c84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c840052a0 0x7f0c8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca406cbc0 con 0x7f0c840047c0 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8400c1c0 con 0x7f0c840052a0 2024-01-16T15:04:06.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.157+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8400c380 con 0x7f0c840039c0 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.162+0000 7f0caa000640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 0x7f0c84003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.162+0000 7f0caaa00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c840052a0 0x7f0c8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0ca9600640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840047c0 0x7f0c84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0caa000640 1 -- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c840052a0 msgr2=0x7f0c8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0caa000640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c840052a0 0x7f0c8400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0caa000640 1 -- 172.21.15.125:0/3180065290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840047c0 msgr2=0x7f0c84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0caa000640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840047c0 0x7f0c84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.163+0000 7f0caa000640 1 -- 172.21.15.125:0/3180065290 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84003520 con 0x7f0c840039c0 2024-01-16T15:04:06.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.165+0000 7f0caaa00640 1 -- 172.21.15.125:0/3180065290 reap_dead start 2024-01-16T15:04:06.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.166+0000 7f0caa000640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 0x7f0c84003de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c007f10 tx=0x7f0c840057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4abf9afa27b20d89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.166+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c02c920 con 0x7f0c840039c0 2024-01-16T15:04:06.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.166+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c02ca80 con 0x7f0c840039c0 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.166+0000 7f0ca8c00640 1 -- 172.21.15.125:0/3180065290 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c034440 con 0x7f0c840039c0 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.168+0000 7f0caca00640 1 -- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 msgr2=0x7f0c84003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.168+0000 7f0caca00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 0x7f0c84003de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c007f10 tx=0x7f0c840057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.169+0000 7f0caca00640 1 -- 172.21.15.125:0/3180065290 shutdown_connections 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.169+0000 7f0caca00640 1 --2- 172.21.15.125:0/3180065290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c840039c0 0x7f0c84003de0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.169+0000 7f0caca00640 1 -- 172.21.15.125:0/3180065290 >> 172.21.15.125:0/3180065290 conn(0x7f0ca40cf770 msgr2=0x7f0ca40cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.169+0000 7f0caca00640 1 -- 172.21.15.125:0/3180065290 shutdown_connections 2024-01-16T15:04:06.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.169+0000 7f0caca00640 1 -- 172.21.15.125:0/3180065290 wait complete. 2024-01-16T15:04:06.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.173+0000 7f0caca00640 1 Processor -- start 2024-01-16T15:04:06.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.173+0000 7f0caca00640 1 -- start start 2024-01-16T15:04:06.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.175+0000 7f0caca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 0x7f0ca414f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.175+0000 7f0caca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca40a84c0 0x7f0ca414ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.175+0000 7f0caca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4154db0 con 0x7f0c840039c0 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca414fd90 con 0x7f0ca40a84c0 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41509c0 con 0x7f0ca4154870 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0ca9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca40a84c0 0x7f0ca414ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caaa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caaa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44322/0 (socket says 172.21.15.125:44322) 2024-01-16T15:04:06.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.176+0000 7f0caaa00640 1 -- 172.21.15.125:0/2114042852 learned_addr learned my addr 172.21.15.125:0/2114042852 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:06.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.177+0000 7f0caaa00640 1 -- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca40a84c0 msgr2=0x7f0ca414ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.177+0000 7f0caaa00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca40a84c0 0x7f0ca414ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.177+0000 7f0caaa00640 1 -- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 msgr2=0x7f0ca414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.178+0000 7f0caa000640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 0x7f0ca414f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.178+0000 7f0caaa00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 0x7f0ca414f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.178+0000 7f0caaa00640 1 -- 172.21.15.125:0/2114042852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4150ca0 con 0x7f0ca4154870 2024-01-16T15:04:06.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.178+0000 7f0caa000640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 0x7f0ca414f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:06.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.179+0000 7f0caaa00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40d06e0 tx=0x7f0c9c007420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.180+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c0122d0 con 0x7f0ca4154870 2024-01-16T15:04:06.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.180+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c018040 con 0x7f0ca4154870 2024-01-16T15:04:06.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.180+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c012670 con 0x7f0ca4154870 2024-01-16T15:04:06.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.182+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca4150f80 con 0x7f0ca4154870 2024-01-16T15:04:06.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.182+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41504d0 con 0x7f0ca4154870 2024-01-16T15:04:06.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.184+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca4155240 con 0x7f0ca4154870 2024-01-16T15:04:06.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.189+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f0c9c010850 con 0x7f0ca4154870 2024-01-16T15:04:06.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.191+0000 7f0ca2000640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 0x7f0c7c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.191+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c9c090490 con 0x7f0ca4154870 2024-01-16T15:04:06.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.199+0000 7f0caa000640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 0x7f0c7c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.199+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca4155240 con 0x7f0ca4154870 2024-01-16T15:04:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.204+0000 7f0caa000640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 0x7f0c7c06e0c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.499+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0ca4155240 con 0x7f0ca4154870 2024-01-16T15:04:06.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.501+0000 7f0ca2000640 1 -- 172.21.15.125:0/2114042852 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7f0ca4155240 con 0x7f0ca4154870 2024-01-16T15:04:06.480 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.481 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 465, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:06.482 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:06.483 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487714335, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487738473, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:06.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:06.485 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.486 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 msgr2=0x7f0c7c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 0x7f0c7c06e0c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c009030 comp rx=0 tx=0).stop 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 msgr2=0x7f0ca41526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40d06e0 tx=0x7f0c9c007420 comp rx=0 tx=0).stop 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 shutdown_connections 2024-01-16T15:04:06.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca4154870 0x7f0ca41526d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca40a84c0 0x7f0ca414ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c7c06bc00 0x7f0c7c06e0c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 --2- 172.21.15.125:0/2114042852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840039c0 0x7f0ca414f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.503+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 >> 172.21.15.125:0/2114042852 conn(0x7f0ca40cf770 msgr2=0x7f0ca4158540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.504+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 shutdown_connections 2024-01-16T15:04:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.504+0000 7f0caca00640 1 -- 172.21.15.125:0/2114042852 wait complete. 2024-01-16T15:04:06.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.704+0000 7f5b32e00640 1 Processor -- start 2024-01-16T15:04:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.705+0000 7f5b32e00640 1 -- start start 2024-01-16T15:04:06.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c160920 0x7f5b2c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5b2c161010 0x7f5b2c1614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2c0d03b0 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2c0d0a10 con 0x7f5b2c161010 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b32e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0cf080 con 0x7f5b2c160920 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38308/0 (socket says 172.21.15.125:38308) 2024-01-16T15:04:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 -- 172.21.15.125:0/1443444282 learned_addr learned my addr 172.21.15.125:0/1443444282 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 -- 172.21.15.125:0/1443444282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c160920 msgr2=0x7f5b2c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 --2- 172.21.15.125:0/1443444282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c160920 0x7f5b2c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 -- 172.21.15.125:0/1443444282 >> v2:172.21.15.137:3300/0 conn(0x7f5b2c161010 msgr2=0x7f5b2c1614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 --2- 172.21.15.125:0/1443444282 >> v2:172.21.15.137:3300/0 conn(0x7f5b2c161010 0x7f5b2c1614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.706+0000 7f5b31000640 1 -- 172.21.15.125:0/1443444282 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0c6be0 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.707+0000 7f5b31000640 1 --2- 172.21.15.125:0/1443444282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0c6930 tx=0x7f5b1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3e7cadb9e0339e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b2be00640 1 -- 172.21.15.125:0/1443444282 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14031870 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b2be00640 1 -- 172.21.15.125:0/1443444282 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b140319d0 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b2be00640 1 -- 172.21.15.125:0/1443444282 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14031d60 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 -- 172.21.15.125:0/1443444282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c0cf9f0 msgr2=0x7f5b2c0cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 --2- 172.21.15.125:0/1443444282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0c6930 tx=0x7f5b1402b910 comp rx=0 tx=0).stop 2024-01-16T15:04:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 -- 172.21.15.125:0/1443444282 shutdown_connections 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 --2- 172.21.15.125:0/1443444282 >> v2:172.21.15.137:3300/0 conn(0x7f5b2c161010 0x7f5b2c1614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 --2- 172.21.15.125:0/1443444282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c0cf9f0 0x7f5b2c0cfe70 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 --2- 172.21.15.125:0/1443444282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c160920 0x7f5b2c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.708+0000 7f5b32e00640 1 -- 172.21.15.125:0/1443444282 >> 172.21.15.125:0/1443444282 conn(0x7f5b2c0d3090 msgr2=0x7f5b2c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.709+0000 7f5b32e00640 1 -- 172.21.15.125:0/1443444282 shutdown_connections 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.709+0000 7f5b32e00640 1 -- 172.21.15.125:0/1443444282 wait complete. 2024-01-16T15:04:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.710+0000 7f5b32e00640 1 Processor -- start 2024-01-16T15:04:06.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.710+0000 7f5b32e00640 1 -- start start 2024-01-16T15:04:06.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c0cf9f0 0x7f5b2c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2c161010 0x7f5b2c0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0b6ba0 con 0x7f5b2c160920 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0cabe0 con 0x7f5b2c161010 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b32e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0ccf90 con 0x7f5b2c0cf9f0 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b31000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b31000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38310/0 (socket says 172.21.15.125:38310) 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.711+0000 7f5b31000640 1 -- 172.21.15.125:0/2192131584 learned_addr learned my addr 172.21.15.125:0/2192131584 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 -- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c0cf9f0 msgr2=0x7f5b2c15f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c0cf9f0 0x7f5b2c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 -- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2c161010 msgr2=0x7f5b2c0ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2c161010 0x7f5b2c0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 -- 172.21.15.125:0/2192131584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b140094d0 con 0x7f5b2c160920 2024-01-16T15:04:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b31000640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5b1402b950 tx=0x7f5b140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14031840 con 0x7f5b2c160920 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b140443f0 con 0x7f5b2c160920 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b140447a0 con 0x7f5b2c160920 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.712+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c0b6270 con 0x7f5b2c160920 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.713+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c0b6830 con 0x7f5b2c160920 2024-01-16T15:04:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.714+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c0b5a60 con 0x7f5b2c160920 2024-01-16T15:04:06.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.714+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5b14044900 con 0x7f5b2c160920 2024-01-16T15:04:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.715+0000 7f5b29600640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 0x7f5b0406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.715+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b1402d040 con 0x7f5b2c160920 2024-01-16T15:04:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.715+0000 7f5b31a00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 0x7f5b0406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.716+0000 7f5b31a00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 0x7f5b0406dff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400ab90 tx=0x7f5b24009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.720+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b14077660 con 0x7f5b2c160920 2024-01-16T15:04:06.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.960+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5b2c067450 con 0x7f5b2c160920 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.961+0000 7f5b29600640 1 -- 172.21.15.125:0/2192131584 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7f5b14035020 con 0x7f5b2c160920 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 466, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:06.942 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5478, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 22819235795, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 45754429440, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 727339683840, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:06.943 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 487714335, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 487738473, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 116, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 123 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:06.944 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:06.946 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 msgr2=0x7f5b0406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 0x7f5b0406dff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400ab90 tx=0x7f5b24009250 comp rx=0 tx=0).stop 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 msgr2=0x7f5b2c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5b1402b950 tx=0x7f5b140046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 shutdown_connections 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2c161010 0x7f5b2c0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b0406bb30 0x7f5b0406dff0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b2c160920 0x7f5b2c0ca160 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 --2- 172.21.15.125:0/2192131584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2c0cf9f0 0x7f5b2c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 >> 172.21.15.125:0/2192131584 conn(0x7f5b2c0d3090 msgr2=0x7f5b2c0b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 shutdown_connections 2024-01-16T15:04:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:06.963+0000 7f5b32e00640 1 -- 172.21.15.125:0/2192131584 wait complete. 2024-01-16T15:04:06.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:07.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.173+0000 7feef9600640 1 Processor -- start 2024-01-16T15:04:07.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.173+0000 7feef9600640 1 -- start start 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.175+0000 7feef9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.175+0000 7feef9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.175+0000 7feef9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef40b1ee0 con 0x7feef4143ae0 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.175+0000 7feef9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef40b2780 con 0x7feef40cf9a0 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.175+0000 7feef9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef415e990 con 0x7feef415e1e0 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.176+0000 7feef3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.176+0000 7feee9e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.176+0000 7feef3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:38846/0 (socket says 172.21.15.125:38846) 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.176+0000 7feee9e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:35358/0 (socket says 172.21.15.125:35358) 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.176+0000 7feef3400640 1 -- 172.21.15.125:0/891052206 learned_addr learned my addr 172.21.15.125:0/891052206 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.177+0000 7feef3400640 1 -- 172.21.15.125:0/891052206 >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 msgr2=0x7feef415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.177+0000 7feef3400640 1 --2- 172.21.15.125:0/891052206 >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.177+0000 7feef3400640 1 -- 172.21.15.125:0/891052206 >> v1:172.21.15.137:6789/0 conn(0x7feef4143ae0 legacy=0x7feef40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.177+0000 7feef3400640 1 -- 172.21.15.125:0/891052206 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40b2130 con 0x7feef415e1e0 2024-01-16T15:04:07.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.177+0000 7feee9e00640 1 --2- 172.21.15.125:0/891052206 >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:07.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.178+0000 7feef3400640 1 --2- 172.21.15.125:0/891052206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feeec009e50 tx=0x7feeec02e170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e191351802285a73 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.178+0000 7feef2000640 1 -- 172.21.15.125:0/891052206 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec02ed10 con 0x7feef415e1e0 2024-01-16T15:04:07.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.178+0000 7feef2000640 1 -- 172.21.15.125:0/891052206 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec02ee70 con 0x7feef415e1e0 2024-01-16T15:04:07.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.178+0000 7feef2000640 1 -- 172.21.15.125:0/891052206 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec0365f0 con 0x7feef415e1e0 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 -- 172.21.15.125:0/891052206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 msgr2=0x7feef40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 --2- 172.21.15.125:0/891052206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feeec009e50 tx=0x7feeec02e170 comp rx=0 tx=0).stop 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 -- 172.21.15.125:0/891052206 shutdown_connections 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 --2- 172.21.15.125:0/891052206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef415e1e0 0x7feef40702c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 --2- 172.21.15.125:0/891052206 >> v2:172.21.15.137:3300/0 conn(0x7feef40cf9a0 0x7feef415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 -- 172.21.15.125:0/891052206 >> 172.21.15.125:0/891052206 conn(0x7feef406a800 msgr2=0x7feef406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 -- 172.21.15.125:0/891052206 shutdown_connections 2024-01-16T15:04:07.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.179+0000 7feef9600640 1 -- 172.21.15.125:0/891052206 wait complete. 2024-01-16T15:04:07.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.181+0000 7feef9600640 1 Processor -- start 2024-01-16T15:04:07.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.181+0000 7feef9600640 1 -- start start 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feef40cf9a0 0x7feef41f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef4143ae0 0x7feef41f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41415c0 con 0x7feef415e1e0 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f6e60 con 0x7feef40cf9a0 2024-01-16T15:04:07.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f6b80 con 0x7feef4143ae0 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38322/0 (socket says 172.21.15.125:38322) 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef3400640 1 -- 172.21.15.125:0/1350231665 learned_addr learned my addr 172.21.15.125:0/1350231665 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feee9e00640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef4143ae0 0x7feef41f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.182+0000 7feef2a00640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feef40cf9a0 0x7feef41f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 -- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef4143ae0 msgr2=0x7feef41f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef4143ae0 0x7feef41f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 -- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feef40cf9a0 msgr2=0x7feef41f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feef40cf9a0 0x7feef41f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 -- 172.21.15.125:0/1350231665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc009400 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef3400640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7feeec038040 tx=0x7feeec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec009510 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec00e120 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec009b30 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec040660 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.183+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef4133710 con 0x7feef415e1e0 2024-01-16T15:04:07.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.185+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef40cbf50 con 0x7feef415e1e0 2024-01-16T15:04:07.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.185+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7feeec0407c0 con 0x7feef415e1e0 2024-01-16T15:04:07.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.186+0000 7feeeb400640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 0x7feecc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.186+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feeec0aacf0 con 0x7feef415e1e0 2024-01-16T15:04:07.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.186+0000 7feef2a00640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 0x7feecc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.187+0000 7feef2a00640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 0x7feecc06dff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feef40ad9a0 tx=0x7feee4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.190+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feeec0773e0 con 0x7feef415e1e0 2024-01-16T15:04:07.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.434+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feef40675d0 con 0x7feef415e1e0 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.435+0000 7feeeb400640 1 -- 172.21.15.125:0/1350231665 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7feeec07aa00 con 0x7feef415e1e0 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:07.415 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 466, 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:07.416 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:07.417 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5935, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 24736158037, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 49843859456, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 723250253824, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 643553084, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 647080593, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 154, 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 165 2024-01-16T15:04:07.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.419 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:07.420 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.421 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 msgr2=0x7feecc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 0x7feecc06dff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feef40ad9a0 tx=0x7feee4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 msgr2=0x7feef41f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7feeec038040 tx=0x7feeec004ab0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 shutdown_connections 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feecc06bb30 0x7feecc06dff0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feef415e1e0 0x7feef41f6500 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feef4143ae0 0x7feef41f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 --2- 172.21.15.125:0/1350231665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feef40cf9a0 0x7feef41f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 >> 172.21.15.125:0/1350231665 conn(0x7feef406a800 msgr2=0x7feef406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 shutdown_connections 2024-01-16T15:04:07.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.438+0000 7feef9600640 1 -- 172.21.15.125:0/1350231665 wait complete. 2024-01-16T15:04:07.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:07.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.661+0000 7f3fa3400640 1 Processor -- start 2024-01-16T15:04:07.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.661+0000 7f3fa3400640 1 -- start start 2024-01-16T15:04:07.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.662+0000 7f3fa3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f9c0cf9a0 0x7f3f9c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.662+0000 7f3fa3400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f9c15e1e0 0x7f3f9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.662+0000 7f3fa3400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c0b1ee0 con 0x7f3f9c143ae0 2024-01-16T15:04:07.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.662+0000 7f3fa3400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c0b2780 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.662+0000 7f3fa3400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c15e990 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3fa0a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3f9c143ae0 0x7f3f9c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47014/0 (socket says 172.21.15.125:47014) 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3fa0a00640 1 -- 172.21.15.125:0/3793730196 learned_addr learned my addr 172.21.15.125:0/3793730196 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f93c00640 1 --2- 172.21.15.125:0/3793730196 >> v2:172.21.15.125:3300/0 conn(0x7f3f9c0cf9a0 0x7f3f9c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3fa1400640 1 --2- 172.21.15.125:0/3793730196 >> v2:172.21.15.137:3300/0 conn(0x7f3f9c15e1e0 0x7f3f9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f9be00640 1 -- 172.21.15.125:0/3793730196 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556751416 0 0) 0x7f3f9c0b1ee0 con 0x7f3f9c143ae0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f93c00640 1 -- 172.21.15.125:0/3793730196 >> v2:172.21.15.137:3300/0 conn(0x7f3f9c15e1e0 msgr2=0x7f3f9c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f93c00640 1 --2- 172.21.15.125:0/3793730196 >> v2:172.21.15.137:3300/0 conn(0x7f3f9c15e1e0 0x7f3f9c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f93c00640 1 -- 172.21.15.125:0/3793730196 >> v1:172.21.15.125:6789/0 conn(0x7f3f9c143ae0 legacy=0x7f3f9c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.663+0000 7f3f93c00640 1 -- 172.21.15.125:0/3793730196 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c0b2130 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.664+0000 7f3f93c00640 1 --2- 172.21.15.125:0/3793730196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c15dca0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0ae9d0 tx=0x7f3f8402dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5554a09778a5506b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3f9be00640 1 -- 172.21.15.125:0/3793730196 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8402f8a0 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3f9be00640 1 -- 172.21.15.125:0/3793730196 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8402fa00 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3f9be00640 1 -- 172.21.15.125:0/3793730196 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8402fd90 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 -- 172.21.15.125:0/3793730196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0cf9a0 msgr2=0x7f3f9c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 --2- 172.21.15.125:0/3793730196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c15dca0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0ae9d0 tx=0x7f3f8402dae0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 -- 172.21.15.125:0/3793730196 shutdown_connections 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 --2- 172.21.15.125:0/3793730196 >> v2:172.21.15.137:3300/0 conn(0x7f3f9c15e1e0 0x7f3f9c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 --2- 172.21.15.125:0/3793730196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c15dca0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 -- 172.21.15.125:0/3793730196 >> 172.21.15.125:0/3793730196 conn(0x7f3f9c06a800 msgr2=0x7f3f9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 -- 172.21.15.125:0/3793730196 shutdown_connections 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.665+0000 7f3fa3400640 1 -- 172.21.15.125:0/3793730196 wait complete. 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.667+0000 7f3fa3400640 1 Processor -- start 2024-01-16T15:04:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.667+0000 7f3fa3400640 1 -- start start 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c143ae0 0x7f3f9c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0aa020 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0a9150 con 0x7f3f9c0cf9a0 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0a93a0 con 0x7f3f9c143ae0 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38344/0 (socket says 172.21.15.125:38344) 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 -- 172.21.15.125:0/2200663890 learned_addr learned my addr 172.21.15.125:0/2200663890 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 -- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c143ae0 msgr2=0x7f3f9c0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c143ae0 0x7f3f9c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 -- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0cf9a0 msgr2=0x7f3f9c159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.668+0000 7f3fa1400640 1 -- 172.21.15.125:0/2200663890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84009400 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3fa1400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3f94007e70 tx=0x7f3f9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94014620 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f940147a0 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94014b50 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c0a9680 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.669+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c0a9c40 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.670+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c0aa160 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.671+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3f94014cb0 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.672+0000 7f3f99600640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 0x7f3f7406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.672+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f94089000 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.672+0000 7f3fa0a00640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 0x7f3f7406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.673+0000 7f3fa0a00640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 0x7f3f7406dff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c007fd0 tx=0x7f3f8c007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.676+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f9c0aa160 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.931+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3f9c0aa160 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.932+0000 7f3f99600640 1 -- 172.21.15.125:0/2200663890 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2155 (secure 0 0 0) 0x7f3f9c0aa160 con 0x7f3f9c15e1e0 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:07.912 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 467, 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:07.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:07.914 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5935, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 24736158037, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 49843859456, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 723250253824, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 643553084, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 647080593, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 154, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 165 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.915 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:07.916 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:07.917 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "69458e4c-23b0-48f2-9afe-6acd3b837d50": { 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (46s)\n [==========================..] (remaining: 2s)", 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.95238572359085083, 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:07.918 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.934+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 msgr2=0x7f3f7406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.934+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 0x7f3f7406dff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c007fd0 tx=0x7f3f8c007920 comp rx=0 tx=0).stop 2024-01-16T15:04:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.934+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 msgr2=0x7f3f9c0ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.934+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3f94007e70 tx=0x7f3f9400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 shutdown_connections 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f7406bb30 0x7f3f7406dff0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c15e1e0 0x7f3f9c0ae3d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c143ae0 0x7f3f9c0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 --2- 172.21.15.125:0/2200663890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0cf9a0 0x7f3f9c159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 >> 172.21.15.125:0/2200663890 conn(0x7f3f9c06a800 msgr2=0x7f3f9c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 shutdown_connections 2024-01-16T15:04:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:07.935+0000 7f3fa3400640 1 -- 172.21.15.125:0/2200663890 wait complete. 2024-01-16T15:04:07.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.153+0000 7f9ddf400640 1 Processor -- start 2024-01-16T15:04:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.154+0000 7f9ddf400640 1 -- start start 2024-01-16T15:04:08.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddf400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9dd80d3910 0x7f9dd80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd8160e10 0x7f9dd80d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddf400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd80d4160 con 0x7f9dd80cfcf0 2024-01-16T15:04:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddf400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd80ceff0 con 0x7f9dd80d3910 2024-01-16T15:04:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80cf410 con 0x7f9dd8160e10 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddca00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9dd80cfcf0 0x7f9dd8160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47026/0 (socket says 172.21.15.125:47026) 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.155+0000 7f9ddca00640 1 -- 172.21.15.125:0/2764463476 learned_addr learned my addr 172.21.15.125:0/2764463476 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7e00640 1 --2- 172.21.15.125:0/2764463476 >> v2:172.21.15.125:3300/0 conn(0x7f9dd80d3910 0x7f9dd80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9ddd400640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd8160e10 0x7f9dd80d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7400640 1 -- 172.21.15.125:0/2764463476 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270859574 0 0) 0x7f9dd80d4160 con 0x7f9dd80cfcf0 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7400640 1 -- 172.21.15.125:0/2764463476 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db8003520 con 0x7f9dd80cfcf0 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7e00640 1 -- 172.21.15.125:0/2764463476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd8160e10 msgr2=0x7f9dd80d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7e00640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd8160e10 0x7f9dd80d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7e00640 1 -- 172.21.15.125:0/2764463476 >> v1:172.21.15.125:6789/0 conn(0x7f9dd80cfcf0 legacy=0x7f9dd8160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.156+0000 7f9dd7e00640 1 -- 172.21.15.125:0/2764463476 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd80b2420 con 0x7f9dd80d3910 2024-01-16T15:04:08.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.157+0000 7f9dd7e00640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd80d3910 0x7f9dd80d1100 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009dc0 tx=0x7f9dc802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c873a6c923607d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9dd7400640 1 -- 172.21.15.125:0/2764463476 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc802cbb0 con 0x7f9dd80d3910 2024-01-16T15:04:08.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9dd7400640 1 -- 172.21.15.125:0/2764463476 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc802cd10 con 0x7f9dd80d3910 2024-01-16T15:04:08.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9dd7400640 1 -- 172.21.15.125:0/2764463476 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8034660 con 0x7f9dd80d3910 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 -- 172.21.15.125:0/2764463476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd80d3910 msgr2=0x7f9dd80d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd80d3910 0x7f9dd80d1100 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009dc0 tx=0x7f9dc802be00 comp rx=0 tx=0).stop 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 -- 172.21.15.125:0/2764463476 shutdown_connections 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd8160e10 0x7f9dd80d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 --2- 172.21.15.125:0/2764463476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd80d3910 0x7f9dd80d1100 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 -- 172.21.15.125:0/2764463476 >> 172.21.15.125:0/2764463476 conn(0x7f9dd809c530 msgr2=0x7f9dd80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 -- 172.21.15.125:0/2764463476 shutdown_connections 2024-01-16T15:04:08.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.158+0000 7f9ddf400640 1 -- 172.21.15.125:0/2764463476 wait complete. 2024-01-16T15:04:08.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.160+0000 7f9ddf400640 1 Processor -- start 2024-01-16T15:04:08.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.160+0000 7f9ddf400640 1 -- start start 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd80cfcf0 0x7f9dd8154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd80d3910 0x7f9dd80b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81547a0 con 0x7f9dd8160e10 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80ca7d0 con 0x7f9dd80d3910 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80caab0 con 0x7f9dd80cfcf0 2024-01-16T15:04:08.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38366/0 (socket says 172.21.15.125:38366) 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.161+0000 7f9ddd400640 1 -- 172.21.15.125:0/4082166688 learned_addr learned my addr 172.21.15.125:0/4082166688 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddca00640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd80cfcf0 0x7f9dd8154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9dd7e00640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd80d3910 0x7f9dd80b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 -- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd80cfcf0 msgr2=0x7f9dd8154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd80cfcf0 0x7f9dd8154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 -- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd80d3910 msgr2=0x7f9dd80b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd80d3910 0x7f9dd80b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 -- 172.21.15.125:0/4082166688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8009a70 con 0x7f9dd8160e10 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddd400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c540 tx=0x7f9dd000ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0004220 con 0x7f9dd8160e10 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0004380 con 0x7f9dd8160e10 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0003850 con 0x7f9dd8160e10 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd80cad90 con 0x7f9dd8160e10 2024-01-16T15:04:08.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.162+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd80cb350 con 0x7f9dd8160e10 2024-01-16T15:04:08.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.163+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd80cb540 con 0x7f9dd8160e10 2024-01-16T15:04:08.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.164+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f9dd00039b0 con 0x7f9dd8160e10 2024-01-16T15:04:08.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.165+0000 7f9dd4c00640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 0x7f9db006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.165+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dd0088240 con 0x7f9dd8160e10 2024-01-16T15:04:08.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.165+0000 7f9ddca00640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 0x7f9db006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.166+0000 7f9ddca00640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 0x7f9db006e020 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00011e0 tx=0x7f9dc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.169+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dd008ce60 con 0x7f9dd8160e10 2024-01-16T15:04:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.414+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9dd80675d0 con 0x7f9dd8160e10 2024-01-16T15:04:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.415+0000 7f9dd4c00640 1 -- 172.21.15.125:0/4082166688 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1869 (secure 0 0 0) 0x7f9dd001e090 con 0x7f9dd8160e10 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:04:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 467, 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:04:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 58, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:04:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 5935, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 24736158037, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 49843859456, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 723250253824, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 400518297, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 405430875, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 95, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 103 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 38, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 4, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 6366 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 12, 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:03:12.053412+0000", 2024-01-16T15:04:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": {} 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 msgr2=0x7f9db006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 0x7f9db006e020 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00011e0 tx=0x7f9dc0009040 comp rx=0 tx=0).stop 2024-01-16T15:04:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 msgr2=0x7f9dd806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c540 tx=0x7f9dd000ca00 comp rx=0 tx=0).stop 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 shutdown_connections 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9db006bb60 0x7f9db006e020 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd8160e10 0x7f9dd806ba20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd80d3910 0x7f9dd80b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 --2- 172.21.15.125:0/4082166688 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9dd80cfcf0 0x7f9dd8154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 >> 172.21.15.125:0/4082166688 conn(0x7f9dd809c530 msgr2=0x7f9dd80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 shutdown_connections 2024-01-16T15:04:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.418+0000 7f9ddf400640 1 -- 172.21.15.125:0/4082166688 wait complete. 2024-01-16T15:04:08.417 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:08.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27071 2024-01-16T15:04:08.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:08.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:08.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.641+0000 7f541b400640 1 Processor -- start 2024-01-16T15:04:08.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.641+0000 7f541b400640 1 -- start start 2024-01-16T15:04:08.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541415d4b0 0x7f54140d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54140b41e0 0x7f54140cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541b400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140d0320 con 0x7f54140cf9a0 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541b400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140b2730 con 0x7f54140b41e0 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140d1ec0 con 0x7f541415d4b0 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54140b41e0 0x7f54140cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54140b41e0 0x7f54140cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38376/0 (socket says 172.21.15.125:38376) 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.643+0000 7f541aa00640 1 -- 172.21.15.125:0/3786172855 learned_addr learned my addr 172.21.15.125:0/3786172855 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.644+0000 7f541a000640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541415d4b0 0x7f54140d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.644+0000 7f541aa00640 1 -- 172.21.15.125:0/3786172855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541415d4b0 msgr2=0x7f54140d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.644+0000 7f541aa00640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541415d4b0 0x7f54140d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.644+0000 7f541aa00640 1 -- 172.21.15.125:0/3786172855 >> v1:172.21.15.137:6789/0 conn(0x7f54140cf9a0 legacy=0x7f54140cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.644+0000 7f541aa00640 1 -- 172.21.15.125:0/3786172855 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140b29d0 con 0x7f54140b41e0 2024-01-16T15:04:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.645+0000 7f541aa00640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140cfde0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f54040098a0 tx=0x7f540402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3767c5759e0af098 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f5418c00640 1 -- 172.21.15.125:0/3786172855 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404030840 con 0x7f54140b41e0 2024-01-16T15:04:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f5418c00640 1 -- 172.21.15.125:0/3786172855 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54040309a0 con 0x7f54140b41e0 2024-01-16T15:04:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f5418c00640 1 -- 172.21.15.125:0/3786172855 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404030d30 con 0x7f54140b41e0 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 -- 172.21.15.125:0/3786172855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 msgr2=0x7f54140cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140cfde0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f54040098a0 tx=0x7f540402b980 comp rx=0 tx=0).stop 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 -- 172.21.15.125:0/3786172855 shutdown_connections 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140cfde0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 --2- 172.21.15.125:0/3786172855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f541415d4b0 0x7f54140d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 -- 172.21.15.125:0/3786172855 >> 172.21.15.125:0/3786172855 conn(0x7f54140d3000 msgr2=0x7f54140d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 -- 172.21.15.125:0/3786172855 shutdown_connections 2024-01-16T15:04:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.646+0000 7f541b400640 1 -- 172.21.15.125:0/3786172855 wait complete. 2024-01-16T15:04:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.648+0000 7f541b400640 1 Processor -- start 2024-01-16T15:04:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.648+0000 7f541b400640 1 -- start start 2024-01-16T15:04:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54140cf9a0 0x7f541415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541415d4b0 0x7f54140a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140cb0c0 con 0x7f54140b41e0 2024-01-16T15:04:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140cb3a0 con 0x7f541415d4b0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140cb680 con 0x7f54140cf9a0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38392/0 (socket says 172.21.15.125:38392) 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.649+0000 7f541a000640 1 -- 172.21.15.125:0/2074345798 learned_addr learned my addr 172.21.15.125:0/2074345798 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f5419600640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54140cf9a0 0x7f541415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541aa00640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541415d4b0 0x7f54140a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 -- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54140cf9a0 msgr2=0x7f541415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54140cf9a0 0x7f541415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 -- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541415d4b0 msgr2=0x7f54140a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541415d4b0 0x7f54140a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 -- 172.21.15.125:0/2074345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54040094d0 con 0x7f54140b41e0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541a000640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f540c00d5d0 tx=0x7f540c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c0125f0 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c012770 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c012b20 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541415da10 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.650+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541415c140 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.651+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5414067450 con 0x7f54140b41e0 2024-01-16T15:04:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.653+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f540c012c80 con 0x7f54140b41e0 2024-01-16T15:04:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.654+0000 7f5412000640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 0x7f53ec06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.654+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f540c088b20 con 0x7f54140b41e0 2024-01-16T15:04:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.654+0000 7f5419600640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 0x7f53ec06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.654+0000 7f5419600640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 0x7f53ec06e020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5414160b70 tx=0x7f53fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.658+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f540c055210 con 0x7f54140b41e0 2024-01-16T15:04:08.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:08.893+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27071", "target": ["mon-mgr", ""]}) v1 -- 0x7f54140cb510 con 0x7f53ec06bb60 2024-01-16T15:04:10.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.275+0000 7f5412000640 1 -- 172.21.15.125:0/2074345798 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f54140cb510 con 0x7f53ec06bb60 2024-01-16T15:04:10.255 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:10.256 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:10.256 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:10.256 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:10.256 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:10.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 msgr2=0x7f53ec06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 0x7f53ec06e020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5414160b70 tx=0x7f53fc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 msgr2=0x7f54140caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f540c00d5d0 tx=0x7f540c00da90 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 shutdown_connections 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f541415d4b0 0x7f54140a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54140cf9a0 0x7f541415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f53ec06bb60 0x7f53ec06e020 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 --2- 172.21.15.125:0/2074345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54140b41e0 0x7f54140caae0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.279+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 >> 172.21.15.125:0/2074345798 conn(0x7f54140d3000 msgr2=0x7f541406ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.280+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 shutdown_connections 2024-01-16T15:04:10.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.280+0000 7f541b400640 1 -- 172.21.15.125:0/2074345798 wait complete. 2024-01-16T15:04:10.279 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:10.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27072 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.558+0000 7f7731c00640 1 Processor -- start 2024-01-16T15:04:10.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.558+0000 7f7731c00640 1 -- start start 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7731c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f772c0cf9a0 0x7f772c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7731c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c15c5f0 con 0x7f772c13e360 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7731c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c06c470 con 0x7f772c13df80 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7731c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772c06c750 con 0x7f772c0cf9a0 2024-01-16T15:04:10.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7723e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f772c13e360 0x7f772c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47030/0 (socket says 172.21.15.125:47030) 2024-01-16T15:04:10.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f772b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f772c0cf9a0 0x7f772c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f7723e00640 1 -- 172.21.15.125:0/2109335155 learned_addr learned my addr 172.21.15.125:0/2109335155 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:10.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.560+0000 7f772b400640 1 -- 172.21.15.125:0/2109335155 >> v1:172.21.15.137:6789/0 conn(0x7f772c13df80 legacy=0x7f772c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.561+0000 7f772b400640 1 -- 172.21.15.125:0/2109335155 >> v1:172.21.15.125:6789/0 conn(0x7f772c13e360 legacy=0x7f772c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.561+0000 7f772b400640 1 -- 172.21.15.125:0/2109335155 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c06ca30 con 0x7f772c0cf9a0 2024-01-16T15:04:10.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f772b400640 1 --2- 172.21.15.125:0/2109335155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c0cf9a0 0x7f772c166030 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7724009d90 tx=0x7f772402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0960c5b33d62b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f772a000640 1 -- 172.21.15.125:0/2109335155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772402cc60 con 0x7f772c0cf9a0 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f772a000640 1 -- 172.21.15.125:0/2109335155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772402cdc0 con 0x7f772c0cf9a0 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f772a000640 1 -- 172.21.15.125:0/2109335155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724034660 con 0x7f772c0cf9a0 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f7731c00640 1 -- 172.21.15.125:0/2109335155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c0cf9a0 msgr2=0x7f772c166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.562+0000 7f7731c00640 1 --2- 172.21.15.125:0/2109335155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c0cf9a0 0x7f772c166030 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7724009d90 tx=0x7f772402beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.563+0000 7f7731c00640 1 -- 172.21.15.125:0/2109335155 shutdown_connections 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.563+0000 7f7731c00640 1 --2- 172.21.15.125:0/2109335155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c0cf9a0 0x7f772c166030 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.563+0000 7f7731c00640 1 -- 172.21.15.125:0/2109335155 >> 172.21.15.125:0/2109335155 conn(0x7f772c06a800 msgr2=0x7f772c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.563+0000 7f7731c00640 1 -- 172.21.15.125:0/2109335155 shutdown_connections 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.563+0000 7f7731c00640 1 -- 172.21.15.125:0/2109335155 wait complete. 2024-01-16T15:04:10.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.564+0000 7f7731c00640 1 Processor -- start 2024-01-16T15:04:10.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 -- start start 2024-01-16T15:04:10.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772c0cf9a0 0x7f772c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772c13e360 0x7f772c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c158de0 con 0x7f772c13df80 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c135c90 con 0x7f772c0cf9a0 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.565+0000 7f7731c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c070800 con 0x7f772c13e360 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38410/0 (socket says 172.21.15.125:38410) 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 -- 172.21.15.125:0/1932466230 learned_addr learned my addr 172.21.15.125:0/1932466230 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 -- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772c13e360 msgr2=0x7f772c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772c13e360 0x7f772c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 -- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772c0cf9a0 msgr2=0x7f772c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772c0cf9a0 0x7f772c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 -- 172.21.15.125:0/1932466230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724009a70 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.566+0000 7f7723e00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f771400c500 tx=0x7f771400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.567+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714005ce0 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.567+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714005e40 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.567+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140072f0 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.567+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c070aa0 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.567+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c071060 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.568+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c0675d0 con 0x7f772c13df80 2024-01-16T15:04:10.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.569+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f7714007450 con 0x7f772c13df80 2024-01-16T15:04:10.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.570+0000 7f7722a00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 0x7f770406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.570+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7714087a50 con 0x7f772c13df80 2024-01-16T15:04:10.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.570+0000 7f772aa00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 0x7f770406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.571+0000 7f772aa00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 0x7f770406dff0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f771c006fd0 tx=0x7f771c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.574+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77140541f0 con 0x7f772c13df80 2024-01-16T15:04:10.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:10.805+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27072", "target": ["mon-mgr", ""]}) v1 -- 0x7f772c06cff0 con 0x7f770406bb30 2024-01-16T15:04:15.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.269+0000 7f7722a00640 1 -- 172.21.15.125:0/1932466230 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f771c009280 con 0x7f770406bb30 2024-01-16T15:04:15.249 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:15.249 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:15.249 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:15.249 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:15.249 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:15.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 msgr2=0x7f770406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 0x7f770406dff0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f771c006fd0 tx=0x7f771c006bc0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 msgr2=0x7f772c135750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f771400c500 tx=0x7f771400c9c0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 shutdown_connections 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772c13e360 0x7f772c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f770406bb30 0x7f770406dff0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f772c13df80 0x7f772c135750 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 --2- 172.21.15.125:0/1932466230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772c0cf9a0 0x7f772c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 >> 172.21.15.125:0/1932466230 conn(0x7f772c06a800 msgr2=0x7f772c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:15.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.273+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 shutdown_connections 2024-01-16T15:04:15.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.274+0000 7f7731c00640 1 -- 172.21.15.125:0/1932466230 wait complete. 2024-01-16T15:04:15.273 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:15.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27073 2024-01-16T15:04:15.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.504+0000 7f3f5ea00640 1 Processor -- start 2024-01-16T15:04:15.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.504+0000 7f3f5ea00640 1 -- start start 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f580cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f580b20d0 0x7f3f580b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580ce760 con 0x7f3f5815d960 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580d0a00 con 0x7f3f580b20d0 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580cf250 con 0x7f3f5815d580 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5cc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3f5815d960 0x7f3f580cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47042/0 (socket says 172.21.15.125:47042) 2024-01-16T15:04:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5cc00640 1 -- 172.21.15.125:0/2662256592 learned_addr learned my addr 172.21.15.125:0/2662256592 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5d600640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f580cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172657900 0 0) 0x7f3f580ce760 con 0x7f3f5815d960 2024-01-16T15:04:15.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.506+0000 7f3f5e000640 1 --2- 172.21.15.125:0/2662256592 >> v2:172.21.15.137:3300/0 conn(0x7f3f580b20d0 0x7f3f580b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f38003520 con 0x7f3f5815d960 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335684001 0 0) 0x7f3f38003520 con 0x7f3f5815d960 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 msgr2=0x7f3f580cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f580cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 >> v2:172.21.15.137:3300/0 conn(0x7f3f580b20d0 msgr2=0x7f3f580b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 --2- 172.21.15.125:0/2662256592 >> v2:172.21.15.137:3300/0 conn(0x7f3f580b20d0 0x7f3f580b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580d1f50 con 0x7f3f5815d960 2024-01-16T15:04:15.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3f400019f0 con 0x7f3f5815d960 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 >> v1:172.21.15.125:6789/0 conn(0x7f3f5815d960 legacy=0x7f3f580cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f38003970 0x7f3f38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f38004d00 0x7f3f38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580d0a00 con 0x7f3f38003970 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580cf250 con 0x7f3f38004d00 2024-01-16T15:04:15.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3800b980 con 0x7f3f5815d580 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.507+0000 7f3f5d600640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f38003970 0x7f3f38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5cc00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5d600640 1 -- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 msgr2=0x7f3f380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5d600640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5815d580 0x7f3f380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5d600640 1 -- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f38004d00 msgr2=0x7f3f38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5e000640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f38004d00 0x7f3f38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5d600640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f38004d00 0x7f3f38005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.508+0000 7f3f5d600640 1 -- 172.21.15.125:0/2662256592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38003520 con 0x7f3f38003970 2024-01-16T15:04:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.509+0000 7f3f5d600640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f38003970 0x7f3f38003d90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009ac0 tx=0x7f3f380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfddfa858d5fcebc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.509+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802c6b0 con 0x7f3f38003970 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.509+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4802c810 con 0x7f3f38003970 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.509+0000 7f3f57e00640 1 -- 172.21.15.125:0/2662256592 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802cba0 con 0x7f3f38003970 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 -- 172.21.15.125:0/2662256592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f38003970 msgr2=0x7f3f38003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/2662256592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f38003970 0x7f3f38003d90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009ac0 tx=0x7f3f380057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5e000640 1 -- 172.21.15.125:0/2662256592 reap_dead start 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 -- 172.21.15.125:0/2662256592 shutdown_connections 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 -- 172.21.15.125:0/2662256592 >> 172.21.15.125:0/2662256592 conn(0x7f3f580d3040 msgr2=0x7f3f580d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 -- 172.21.15.125:0/2662256592 shutdown_connections 2024-01-16T15:04:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.510+0000 7f3f5ea00640 1 -- 172.21.15.125:0/2662256592 wait complete. 2024-01-16T15:04:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.512+0000 7f3f5ea00640 1 Processor -- start 2024-01-16T15:04:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.512+0000 7f3f5ea00640 1 -- start start 2024-01-16T15:04:15.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f38003970 0x7f3f5815a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f5806b7c0 0x7f3f5806bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5806c180 con 0x7f3f5815e650 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5815f110 con 0x7f3f5806b7c0 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580cb510 con 0x7f3f38003970 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38432/0 (socket says 172.21.15.125:38432) 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5cc00640 1 -- 172.21.15.125:0/1449065847 learned_addr learned my addr 172.21.15.125:0/1449065847 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.513+0000 7f3f5d600640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f38003970 0x7f3f5815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5e000640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f5806b7c0 0x7f3f5806bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 -- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f38003970 msgr2=0x7f3f5815a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f38003970 0x7f3f5815a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 -- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f5806b7c0 msgr2=0x7f3f5806bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f5806b7c0 0x7f3f5806bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 -- 172.21.15.125:0/1449065847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580cb7f0 con 0x7f3f5815e650 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5cc00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007a10 tx=0x7f3f4000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000ee50 con 0x7f3f5815e650 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f40013420 con 0x7f3f5815e650 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f580cbad0 con 0x7f3f5815e650 2024-01-16T15:04:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.514+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f400137b0 con 0x7f3f5815e650 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.515+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5815ae90 con 0x7f3f5815e650 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.516+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f580cbf90 con 0x7f3f5815e650 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.516+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3f40013910 con 0x7f3f5815e650 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.517+0000 7f3f55600640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 0x7f3f3006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.517+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f40088d00 con 0x7f3f5815e650 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.518+0000 7f3f5d600640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 0x7f3f3006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.518+0000 7f3f5d600640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 0x7f3f3006e020 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3f48036000 tx=0x7f3f48030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.522+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f40055450 con 0x7f3f5815e650 2024-01-16T15:04:15.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:15.749+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27073", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f580d2c10 con 0x7f3f3006bb60 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.268+0000 7f3f55600640 1 -- 172.21.15.125:0/1449065847 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f3f580d2c10 con 0x7f3f3006bb60 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:20.247 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:20.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 msgr2=0x7f3f3006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 0x7f3f3006e020 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3f48036000 tx=0x7f3f48030000 comp rx=0 tx=0).stop 2024-01-16T15:04:20.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 msgr2=0x7f3f5815ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007a10 tx=0x7f3f4000e530 comp rx=0 tx=0).stop 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 shutdown_connections 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.271+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f5806b7c0 0x7f3f5806bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f3006bb60 0x7f3f3006e020 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5815e650 0x7f3f5815ead0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 --2- 172.21.15.125:0/1449065847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f38003970 0x7f3f5815a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 >> 172.21.15.125:0/1449065847 conn(0x7f3f580d3040 msgr2=0x7f3f5815b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 shutdown_connections 2024-01-16T15:04:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.272+0000 7f3f5ea00640 1 -- 172.21.15.125:0/1449065847 wait complete. 2024-01-16T15:04:20.273 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:20.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27074 2024-01-16T15:04:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.509+0000 7f1b29000640 1 Processor -- start 2024-01-16T15:04:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.509+0000 7f1b29000640 1 -- start start 2024-01-16T15:04:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b29000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b2415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b29000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1b240cf9a0 0x7f1b240cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b29000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2406c470 con 0x7f1b2413e360 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b29000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2406c750 con 0x7f1b240cf9a0 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b29000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2406ca30 con 0x7f1b2413df80 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b1be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1b2413e360 0x7f1b240cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58754/0 (socket says 172.21.15.125:58754) 2024-01-16T15:04:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b1be00640 1 -- 172.21.15.125:0/1142594957 learned_addr learned my addr 172.21.15.125:0/1142594957 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b22800640 1 --2- 172.21.15.125:0/1142594957 >> v2:172.21.15.137:3300/0 conn(0x7f1b240cf9a0 0x7f1b240cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.511+0000 7f1b21e00640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b2415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779403558 0 0) 0x7f1b2406c470 con 0x7f1b2413e360 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003520 con 0x7f1b2413e360 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297081949 0 0) 0x7f1b04003520 con 0x7f1b2413e360 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 msgr2=0x7f1b2415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b2415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 >> v2:172.21.15.137:3300/0 conn(0x7f1b240cf9a0 msgr2=0x7f1b240cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 --2- 172.21.15.125:0/1142594957 >> v2:172.21.15.137:3300/0 conn(0x7f1b240cf9a0 0x7f1b240cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2406cd10 con 0x7f1b2413e360 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1b0c001670 con 0x7f1b2413e360 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 >> v1:172.21.15.125:6789/0 conn(0x7f1b2413e360 legacy=0x7f1b240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b04003970 0x7f1b04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2406c750 con 0x7f1b04004d00 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2406ca30 con 0x7f1b04003970 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.512+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0400b980 con 0x7f1b2413df80 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b1be00640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b21e00640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b04003970 0x7f1b04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 -- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 msgr2=0x7f1b040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b2413df80 0x7f1b040047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 -- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b04003970 msgr2=0x7f1b04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b04003970 0x7f1b04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.513+0000 7f1b22800640 1 -- 172.21.15.125:0/1142594957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04003520 con 0x7f1b04004d00 2024-01-16T15:04:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.514+0000 7f1b22800640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009fc0 tx=0x7f1b040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc6e5df17214a5c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.514+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c02cc60 con 0x7f1b04004d00 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.514+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c00f040 con 0x7f1b04004d00 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.514+0000 7f1b21400640 1 -- 172.21.15.125:0/1142594957 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c034440 con 0x7f1b04004d00 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 -- 172.21.15.125:0/1142594957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04004d00 msgr2=0x7f1b04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 --2- 172.21.15.125:0/1142594957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009fc0 tx=0x7f1b040057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b22800640 1 -- 172.21.15.125:0/1142594957 reap_dead start 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 -- 172.21.15.125:0/1142594957 shutdown_connections 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 -- 172.21.15.125:0/1142594957 >> 172.21.15.125:0/1142594957 conn(0x7f1b2406a800 msgr2=0x7f1b2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 -- 172.21.15.125:0/1142594957 shutdown_connections 2024-01-16T15:04:20.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.515+0000 7f1b29000640 1 -- 172.21.15.125:0/1142594957 wait complete. 2024-01-16T15:04:20.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.517+0000 7f1b29000640 1 Processor -- start 2024-01-16T15:04:20.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.517+0000 7f1b29000640 1 -- start start 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b04004d00 0x7f1b240b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 0x7f1b241f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2414bdd0 con 0x7f1b2414b660 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2414c0b0 con 0x7f1b241f5560 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b29000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2414c390 con 0x7f1b04004d00 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b1be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b1be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53014/0 (socket says 172.21.15.125:53014) 2024-01-16T15:04:20.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.518+0000 7f1b1be00640 1 -- 172.21.15.125:0/2289973304 learned_addr learned my addr 172.21.15.125:0/2289973304 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:20.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b22800640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 0x7f1b241f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b21e00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b04004d00 0x7f1b240b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 -- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b04004d00 msgr2=0x7f1b240b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b04004d00 0x7f1b240b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 -- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 msgr2=0x7f1b241f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 0x7f1b241f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 -- 172.21.15.125:0/2289973304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2414c670 con 0x7f1b2414b660 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b22800640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 0x7f1b241f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1be00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d8e0 tx=0x7f1b0c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c0074e0 con 0x7f1b2414b660 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c007640 con 0x7f1b2414b660 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c004a20 con 0x7f1b2414b660 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.519+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2414c950 con 0x7f1b2414b660 2024-01-16T15:04:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.520+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b241f5f20 con 0x7f1b2414b660 2024-01-16T15:04:20.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.521+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b240675d0 con 0x7f1b2414b660 2024-01-16T15:04:20.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.522+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f1b0c004b80 con 0x7f1b2414b660 2024-01-16T15:04:20.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.522+0000 7f1b1a000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 0x7f1afc06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.523+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b0c088180 con 0x7f1b2414b660 2024-01-16T15:04:20.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.523+0000 7f1b21e00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 0x7f1afc06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.523+0000 7f1b21e00640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 0x7f1afc06e020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b2415ae30 tx=0x7f1b14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.527+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b0c054870 con 0x7f1b2414b660 2024-01-16T15:04:20.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:20.758+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27074", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b2414c220 con 0x7f1afc06bb60 2024-01-16T15:04:25.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.264+0000 7f1b1a000640 1 -- 172.21.15.125:0/2289973304 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f1b2414c220 con 0x7f1afc06bb60 2024-01-16T15:04:25.244 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:25.244 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:25.244 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:25.244 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:25.244 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 msgr2=0x7f1afc06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 0x7f1afc06e020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b2415ae30 tx=0x7f1b14008040 comp rx=0 tx=0).stop 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 msgr2=0x7f1b241e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d8e0 tx=0x7f1b0c00dda0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 shutdown_connections 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b241f5560 0x7f1b241f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1afc06bb60 0x7f1afc06e020 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b2414b660 0x7f1b241e8490 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 --2- 172.21.15.125:0/2289973304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b04004d00 0x7f1b240b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 >> 172.21.15.125:0/2289973304 conn(0x7f1b2406a800 msgr2=0x7f1b240afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.268+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 shutdown_connections 2024-01-16T15:04:25.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.269+0000 7f1b29000640 1 -- 172.21.15.125:0/2289973304 wait complete. 2024-01-16T15:04:25.267 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:25.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27075 2024-01-16T15:04:25.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.494+0000 7fee5ca00640 1 Processor -- start 2024-01-16T15:04:25.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.494+0000 7fee5ca00640 1 -- start start 2024-01-16T15:04:25.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 0x7fee5415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5ca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee540b1ee0 con 0x7fee54143ae0 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5ca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee540b2780 con 0x7fee5415e1e0 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5415e990 con 0x7fee540cf9a0 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee59600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 0x7fee5415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:58578/0 (socket says 172.21.15.125:58578) 2024-01-16T15:04:25.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.496+0000 7fee5aa00640 1 -- 172.21.15.125:0/3360257880 learned_addr learned my addr 172.21.15.125:0/3360257880 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:25.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.497+0000 7fee59600640 1 -- 172.21.15.125:0/3360257880 >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 msgr2=0x7fee540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:25.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.497+0000 7fee59600640 1 --2- 172.21.15.125:0/3360257880 >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.497+0000 7fee59600640 1 -- 172.21.15.125:0/3360257880 >> v1:172.21.15.137:6789/0 conn(0x7fee54143ae0 legacy=0x7fee540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.497+0000 7fee59600640 1 -- 172.21.15.125:0/3360257880 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee540b2130 con 0x7fee540cf9a0 2024-01-16T15:04:25.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.497+0000 7fee5aa00640 1 --2- 172.21.15.125:0/3360257880 >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:25.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.498+0000 7fee59600640 1 --2- 172.21.15.125:0/3360257880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 0x7fee5415dca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00ba40 tx=0x7fee3c02e080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9aa8c4e711c8c6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.499+0000 7fee58c00640 1 -- 172.21.15.125:0/3360257880 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c02ee30 con 0x7fee540cf9a0 2024-01-16T15:04:25.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.499+0000 7fee58c00640 1 -- 172.21.15.125:0/3360257880 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3c036440 con 0x7fee540cf9a0 2024-01-16T15:04:25.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.499+0000 7fee58c00640 1 -- 172.21.15.125:0/3360257880 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0367d0 con 0x7fee540cf9a0 2024-01-16T15:04:25.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.499+0000 7fee5ca00640 1 -- 172.21.15.125:0/3360257880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 msgr2=0x7fee5415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.499+0000 7fee5ca00640 1 --2- 172.21.15.125:0/3360257880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 0x7fee5415dca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00ba40 tx=0x7fee3c02e080 comp rx=0 tx=0).stop 2024-01-16T15:04:25.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 -- 172.21.15.125:0/3360257880 shutdown_connections 2024-01-16T15:04:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 --2- 172.21.15.125:0/3360257880 >> v2:172.21.15.137:3300/0 conn(0x7fee5415e1e0 0x7fee540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 --2- 172.21.15.125:0/3360257880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee540cf9a0 0x7fee5415dca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 -- 172.21.15.125:0/3360257880 >> 172.21.15.125:0/3360257880 conn(0x7fee5406a800 msgr2=0x7fee5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 -- 172.21.15.125:0/3360257880 shutdown_connections 2024-01-16T15:04:25.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.500+0000 7fee5ca00640 1 -- 172.21.15.125:0/3360257880 wait complete. 2024-01-16T15:04:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.501+0000 7fee5ca00640 1 Processor -- start 2024-01-16T15:04:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.502+0000 7fee5ca00640 1 -- start start 2024-01-16T15:04:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.502+0000 7fee5ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee54143ae0 0x7fee5415aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee5415e1e0 0x7fee5415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5ca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee540cec00 con 0x7fee540cf9a0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee541f3180 con 0x7fee5415e1e0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee540ce080 con 0x7fee54143ae0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53018/0 (socket says 172.21.15.125:53018) 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 -- 172.21.15.125:0/1124141146 learned_addr learned my addr 172.21.15.125:0/1124141146 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee59600640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee54143ae0 0x7fee5415aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5aa00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee5415e1e0 0x7fee5415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 -- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee54143ae0 msgr2=0x7fee5415aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee54143ae0 0x7fee5415aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 -- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee5415e1e0 msgr2=0x7fee5415afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee5415e1e0 0x7fee5415afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.503+0000 7fee5a000640 1 -- 172.21.15.125:0/1124141146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c009b30 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee5a000640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fee4400c450 tx=0x7fee4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440052d0 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44005430 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c00b6a0 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44012870 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.504+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee540ce640 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.505+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee541f32f0 con 0x7fee540cf9a0 2024-01-16T15:04:25.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.506+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fee440129d0 con 0x7fee540cf9a0 2024-01-16T15:04:25.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.507+0000 7fee52000640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 0x7fee2c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.507+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee44088190 con 0x7fee540cf9a0 2024-01-16T15:04:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.507+0000 7fee59600640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 0x7fee2c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.508+0000 7fee59600640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 0x7fee2c06dff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fee3c038000 tx=0x7fee3c032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.512+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee44054860 con 0x7fee540cf9a0 2024-01-16T15:04:25.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:25.737+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27075", "target": ["mon-mgr", ""]}) v1 -- 0x7fee5415e640 con 0x7fee2c06bb30 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:26.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.482+0000 7fee52000640 1 -- 172.21.15.125:0/1124141146 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fee3c0368b0 con 0x7fee2c06bb30 2024-01-16T15:04:26.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.487+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 msgr2=0x7fee2c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.487+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 0x7fee2c06dff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fee3c038000 tx=0x7fee3c032000 comp rx=0 tx=0).stop 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.487+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 msgr2=0x7fee5413f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.487+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fee4400c450 tx=0x7fee4400c910 comp rx=0 tx=0).stop 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 shutdown_connections 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee5415e1e0 0x7fee5415afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee54143ae0 0x7fee5415aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2c06bb30 0x7fee2c06dff0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 --2- 172.21.15.125:0/1124141146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee540cf9a0 0x7fee5413f7d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.488+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 >> 172.21.15.125:0/1124141146 conn(0x7fee5406a800 msgr2=0x7fee54070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.489+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 shutdown_connections 2024-01-16T15:04:26.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.489+0000 7fee5ca00640 1 -- 172.21.15.125:0/1124141146 wait complete. 2024-01-16T15:04:26.500 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:26.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27076 2024-01-16T15:04:26.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.700+0000 7fb002c00640 1 Processor -- start 2024-01-16T15:04:26.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.700+0000 7fb002c00640 1 -- start start 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.701+0000 7fb002c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faffc13e360 0x7faffc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.701+0000 7fb002c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faffc15f9c0 0x7faffc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.701+0000 7fb002c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc15c5f0 con 0x7faffc13df80 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.701+0000 7fb002c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc06c470 con 0x7faffc13e360 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.701+0000 7fb002c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc06c750 con 0x7faffc15f9c0 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffb400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faffc13e360 0x7faffc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faffc13df80 0x7faffc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45136/0 (socket says 172.21.15.125:45136) 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffbe00640 1 -- 172.21.15.125:0/3026081723 learned_addr learned my addr 172.21.15.125:0/3026081723 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7fb000c00640 1 --2- 172.21.15.125:0/3026081723 >> v2:172.21.15.137:3300/0 conn(0x7faffc15f9c0 0x7faffc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffb400640 1 -- 172.21.15.125:0/3026081723 >> v2:172.21.15.137:3300/0 conn(0x7faffc15f9c0 msgr2=0x7faffc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffb400640 1 --2- 172.21.15.125:0/3026081723 >> v2:172.21.15.137:3300/0 conn(0x7faffc15f9c0 0x7faffc0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffb400640 1 -- 172.21.15.125:0/3026081723 >> v1:172.21.15.125:6789/0 conn(0x7faffc13df80 legacy=0x7faffc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffb400640 1 -- 172.21.15.125:0/3026081723 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc06ca30 con 0x7faffc13e360 2024-01-16T15:04:26.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.702+0000 7faffaa00640 1 -- 172.21.15.125:0/3026081723 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332584922 0 0) 0x7faffc15c5f0 con 0x7faffc13df80 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.703+0000 7faffb400640 1 --2- 172.21.15.125:0/3026081723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc13e360 0x7faffc0cf290 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faffc0ccae0 tx=0x7fafe402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8115fd4890f9a069 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.703+0000 7faffaa00640 1 -- 172.21.15.125:0/3026081723 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4031830 con 0x7faffc13e360 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7faffaa00640 1 -- 172.21.15.125:0/3026081723 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4031990 con 0x7faffc13e360 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7faffaa00640 1 -- 172.21.15.125:0/3026081723 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4031d20 con 0x7faffc13e360 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 -- 172.21.15.125:0/3026081723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc13e360 msgr2=0x7faffc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 --2- 172.21.15.125:0/3026081723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc13e360 0x7faffc0cf290 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faffc0ccae0 tx=0x7fafe402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 -- 172.21.15.125:0/3026081723 shutdown_connections 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 --2- 172.21.15.125:0/3026081723 >> v2:172.21.15.137:3300/0 conn(0x7faffc15f9c0 0x7faffc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 --2- 172.21.15.125:0/3026081723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc13e360 0x7faffc0cf290 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 -- 172.21.15.125:0/3026081723 >> 172.21.15.125:0/3026081723 conn(0x7faffc06a800 msgr2=0x7faffc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 -- 172.21.15.125:0/3026081723 shutdown_connections 2024-01-16T15:04:26.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.704+0000 7fb002c00640 1 -- 172.21.15.125:0/3026081723 wait complete. 2024-01-16T15:04:26.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.706+0000 7fb002c00640 1 Processor -- start 2024-01-16T15:04:26.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.706+0000 7fb002c00640 1 -- start start 2024-01-16T15:04:26.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 0x7faffc15a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faffc13e360 0x7faffc0af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0715b0 con 0x7faffc15f9c0 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc071890 con 0x7faffc13df80 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb002c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc071b70 con 0x7faffc13e360 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb000c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb000c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34156/0 (socket says 172.21.15.125:34156) 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.707+0000 7fb000c00640 1 -- 172.21.15.125:0/617242160 learned_addr learned my addr 172.21.15.125:0/617242160 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7faffbe00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 0x7faffc15a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 -- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faffc13e360 msgr2=0x7faffc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faffc13e360 0x7faffc0af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 -- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 msgr2=0x7faffc15a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 0x7faffc15a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 -- 172.21.15.125:0/617242160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4009400 con 0x7faffc15f9c0 2024-01-16T15:04:26.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7faffbe00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 0x7faffc15a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:26.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb000c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faff400cd60 tx=0x7faff400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4003b90 con 0x7faffc15f9c0 2024-01-16T15:04:26.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4003d10 con 0x7faffc15f9c0 2024-01-16T15:04:26.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4004260 con 0x7faffc15f9c0 2024-01-16T15:04:26.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.708+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc071e50 con 0x7faffc15f9c0 2024-01-16T15:04:26.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.709+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc15a910 con 0x7faffc15f9c0 2024-01-16T15:04:26.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.710+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc0675d0 con 0x7faffc15f9c0 2024-01-16T15:04:26.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.711+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7faff40043c0 con 0x7faffc15f9c0 2024-01-16T15:04:26.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.711+0000 7faff3e00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 0x7fafd406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.711+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faff4012070 con 0x7faffc15f9c0 2024-01-16T15:04:26.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.712+0000 7faffbe00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 0x7fafd406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.712+0000 7faffbe00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 0x7fafd406dff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fafec000f60 tx=0x7fafec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.716+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faff4055570 con 0x7faffc15f9c0 2024-01-16T15:04:26.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:26.933+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27076", "target": ["mon-mgr", ""]}) v1 -- 0x7faffc06cff0 con 0x7fafd406bb30 2024-01-16T15:04:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:28.235+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7faff4055940 con 0x7faffc15f9c0 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.267+0000 7faff3e00640 1 -- 172.21.15.125:0/617242160 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fafec001cc0 con 0x7fafd406bb30 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:30.246 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:30.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.270+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 msgr2=0x7fafd406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.270+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 0x7fafd406dff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fafec000f60 tx=0x7fafec009040 comp rx=0 tx=0).stop 2024-01-16T15:04:30.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.270+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 msgr2=0x7faffc13d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.270+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faff400cd60 tx=0x7faff400a660 comp rx=0 tx=0).stop 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 shutdown_connections 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafd406bb30 0x7fafd406dff0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faffc15f9c0 0x7faffc13d430 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faffc13e360 0x7faffc0af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 --2- 172.21.15.125:0/617242160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faffc13df80 0x7faffc15a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 >> 172.21.15.125:0/617242160 conn(0x7faffc06a800 msgr2=0x7faffc0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 shutdown_connections 2024-01-16T15:04:30.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.271+0000 7fb002c00640 1 -- 172.21.15.125:0/617242160 wait complete. 2024-01-16T15:04:30.270 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:30.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27077 2024-01-16T15:04:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.503+0000 7f50cde00640 1 Processor -- start 2024-01-16T15:04:30.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.504+0000 7f50cde00640 1 -- start start 2024-01-16T15:04:30.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f50c806c730 0x7f50c80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cde00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c8071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cde00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c80708d0 con 0x7f50c806cb10 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cde00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c8070bb0 con 0x7f50c806c730 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cde00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c8070e90 con 0x7f50c8070160 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f50c806c730 0x7f50c80b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.505+0000 7f50cd400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f50c806cb10 0x7f50c806cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45152/0 (socket says 172.21.15.125:45152) 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50cd400640 1 -- 172.21.15.125:0/3713559179 learned_addr learned my addr 172.21.15.125:0/3713559179 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50bfc00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c8071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50cca00640 1 -- 172.21.15.125:0/3713559179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 msgr2=0x7f50c8071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50cca00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c8071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50cca00640 1 -- 172.21.15.125:0/3713559179 >> v1:172.21.15.125:6789/0 conn(0x7f50c806cb10 legacy=0x7f50c806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.506+0000 7f50cca00640 1 -- 172.21.15.125:0/3713559179 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c8071170 con 0x7f50c806c730 2024-01-16T15:04:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.507+0000 7f50cca00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806c730 0x7f50c80b2e00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f50b80097a0 tx=0x7f50b802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c4cdabde25e68f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50c7e00640 1 -- 172.21.15.125:0/3713559179 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8031780 con 0x7f50c806c730 2024-01-16T15:04:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50c7e00640 1 -- 172.21.15.125:0/3713559179 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b80318e0 con 0x7f50c806c730 2024-01-16T15:04:30.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50c7e00640 1 -- 172.21.15.125:0/3713559179 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8031c70 con 0x7f50c806c730 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 -- 172.21.15.125:0/3713559179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806c730 msgr2=0x7f50c80b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806c730 0x7f50c80b2e00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f50b80097a0 tx=0x7f50b802b820 comp rx=0 tx=0).stop 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 -- 172.21.15.125:0/3713559179 shutdown_connections 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c8071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 --2- 172.21.15.125:0/3713559179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806c730 0x7f50c80b2e00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.508+0000 7f50cde00640 1 -- 172.21.15.125:0/3713559179 >> 172.21.15.125:0/3713559179 conn(0x7f50c806a7d0 msgr2=0x7f50c806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.509+0000 7f50cde00640 1 -- 172.21.15.125:0/3713559179 shutdown_connections 2024-01-16T15:04:30.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.509+0000 7f50cde00640 1 -- 172.21.15.125:0/3713559179 wait complete. 2024-01-16T15:04:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.510+0000 7f50cde00640 1 Processor -- start 2024-01-16T15:04:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.510+0000 7f50cde00640 1 -- start start 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.511+0000 7f50cde00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50c806c730 0x7f50c8144d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.511+0000 7f50cde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cde00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c81438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cde00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c8145270 con 0x7f50c806cb10 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cde00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c80c4c50 con 0x7f50c806c730 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cde00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c80c4e80 con 0x7f50c8070160 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34180/0 (socket says 172.21.15.125:34180) 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 -- 172.21.15.125:0/352469728 learned_addr learned my addr 172.21.15.125:0/352469728 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cca00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50c806c730 0x7f50c8144d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50cd400640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c81438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 -- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 msgr2=0x7f50c81438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c81438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.512+0000 7f50bfc00640 1 -- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50c806c730 msgr2=0x7f50c8144d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50bfc00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50c806c730 0x7f50c8144d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50bfc00640 1 -- 172.21.15.125:0/352469728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8009400 con 0x7f50c806cb10 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50bfc00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f50b000d5f0 tx=0x7f50b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00125f0 con 0x7f50c806cb10 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0012750 con 0x7f50c806cb10 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0012b00 con 0x7f50c806cb10 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c80aa570 con 0x7f50c806cb10 2024-01-16T15:04:30.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.513+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c80aab30 con 0x7f50c806cb10 2024-01-16T15:04:30.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.514+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c80b26d0 con 0x7f50c806cb10 2024-01-16T15:04:30.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.515+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f50b0012c60 con 0x7f50c806cb10 2024-01-16T15:04:30.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.516+0000 7f50c5600640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 0x7f50a006df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.516+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50b0088760 con 0x7f50c806cb10 2024-01-16T15:04:30.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.517+0000 7f50cca00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 0x7f50a006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.517+0000 7f50cca00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 0x7f50a006df60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f50b8036000 tx=0x7f50b803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.521+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50b00bde90 con 0x7f50c806cb10 2024-01-16T15:04:30.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:30.749+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27077", "target": ["mon-mgr", ""]}) v1 -- 0x7f50c8071eb0 con 0x7f50a006baa0 2024-01-16T15:04:35.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.267+0000 7f50c5600640 1 -- 172.21.15.125:0/352469728 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f50b8031a50 con 0x7f50a006baa0 2024-01-16T15:04:35.246 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:04:35.247 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:04:35.247 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:04:35.247 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:35.247 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.270+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 msgr2=0x7f50a006df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.270+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 0x7f50a006df60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f50b8036000 tx=0x7f50b803d040 comp rx=0 tx=0).stop 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.270+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 msgr2=0x7f50c80c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.270+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f50b000d5f0 tx=0x7f50b000dab0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 shutdown_connections 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50c8070160 0x7f50c81438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a006baa0 0x7f50a006df60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c806cb10 0x7f50c80c4710 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 --2- 172.21.15.125:0/352469728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50c806c730 0x7f50c8144d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 >> 172.21.15.125:0/352469728 conn(0x7f50c806a7d0 msgr2=0x7f50c8146e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 shutdown_connections 2024-01-16T15:04:35.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.271+0000 7f50cde00640 1 -- 172.21.15.125:0/352469728 wait complete. 2024-01-16T15:04:35.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:04:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.507+0000 7f3443200640 1 Processor -- start 2024-01-16T15:04:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.508+0000 7f3443200640 1 -- start start 2024-01-16T15:04:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.509+0000 7f3443200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 0x7f343c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.509+0000 7f3443200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.509+0000 7f3443200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c070c40 con 0x7f343c06d120 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.509+0000 7f3443200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c071450 con 0x7f343c0ae5a0 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.509+0000 7f3443200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06c470 con 0x7f343c0b3ae0 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.510+0000 7f343b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.510+0000 7f343b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56468/0 (socket says 172.21.15.125:56468) 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.510+0000 7f343b400640 1 -- 172.21.15.125:0/2780520057 learned_addr learned my addr 172.21.15.125:0/2780520057 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.510+0000 7f343be00640 1 --2- 172.21.15.125:0/2780520057 >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 0x7f343c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.511+0000 7f343b400640 1 -- 172.21.15.125:0/2780520057 >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 msgr2=0x7f343c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.511+0000 7f343b400640 1 --2- 172.21.15.125:0/2780520057 >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 0x7f343c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.511+0000 7f343b400640 1 -- 172.21.15.125:0/2780520057 >> v1:172.21.15.137:6789/0 conn(0x7f343c06d120 legacy=0x7f343c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.511+0000 7f343b400640 1 -- 172.21.15.125:0/2780520057 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c06c750 con 0x7f343c0b3ae0 2024-01-16T15:04:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.511+0000 7f343be00640 1 --2- 172.21.15.125:0/2780520057 >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 0x7f343c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:35.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.513+0000 7f343b400640 1 --2- 172.21.15.125:0/2780520057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4538f222a65ef313 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.513+0000 7f343aa00640 1 -- 172.21.15.125:0/2780520057 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424030840 con 0x7f343c0b3ae0 2024-01-16T15:04:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.513+0000 7f343aa00640 1 -- 172.21.15.125:0/2780520057 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34240309a0 con 0x7f343c0b3ae0 2024-01-16T15:04:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.513+0000 7f343aa00640 1 -- 172.21.15.125:0/2780520057 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424030d30 con 0x7f343c0b3ae0 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 -- 172.21.15.125:0/2780520057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 msgr2=0x7f343c06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 --2- 172.21.15.125:0/2780520057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342402b980 comp rx=0 tx=0).stop 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 -- 172.21.15.125:0/2780520057 shutdown_connections 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 --2- 172.21.15.125:0/2780520057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0b3ae0 0x7f343c06cbe0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 --2- 172.21.15.125:0/2780520057 >> v2:172.21.15.137:3300/0 conn(0x7f343c0ae5a0 0x7f343c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 -- 172.21.15.125:0/2780520057 >> 172.21.15.125:0/2780520057 conn(0x7f343c06a800 msgr2=0x7f343c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 -- 172.21.15.125:0/2780520057 shutdown_connections 2024-01-16T15:04:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.514+0000 7f3443200640 1 -- 172.21.15.125:0/2780520057 wait complete. 2024-01-16T15:04:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.516+0000 7f3443200640 1 Processor -- start 2024-01-16T15:04:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.516+0000 7f3443200640 1 -- start start 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0ae5a0 0x7f343c1f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f343c0b3ae0 0x7f343c1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0cdfb0 con 0x7f343c06d120 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1f2ca0 con 0x7f343c0b3ae0 2024-01-16T15:04:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3443200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1f3260 con 0x7f343c0ae5a0 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f343be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f343be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34188/0 (socket says 172.21.15.125:34188) 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f343be00640 1 -- 172.21.15.125:0/2111371422 learned_addr learned my addr 172.21.15.125:0/2111371422 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f3441200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f343c0b3ae0 0x7f343c1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.517+0000 7f343b400640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0ae5a0 0x7f343c1f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 -- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0ae5a0 msgr2=0x7f343c1f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0ae5a0 0x7f343c1f36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 -- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f343c0b3ae0 msgr2=0x7f343c1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f343c0b3ae0 0x7f343c1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 -- 172.21.15.125:0/2111371422 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c009400 con 0x7f343c06d120 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f343be00640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f342c00ebe0 tx=0x7f342c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c009760 con 0x7f343c06d120 2024-01-16T15:04:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c00cc30 con 0x7f343c06d120 2024-01-16T15:04:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c00eee0 con 0x7f343c06d120 2024-01-16T15:04:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34240094d0 con 0x7f343c06d120 2024-01-16T15:04:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.518+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c1f3c30 con 0x7f343c06d120 2024-01-16T15:04:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.519+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c0675d0 con 0x7f343c06d120 2024-01-16T15:04:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.520+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f342c013470 con 0x7f343c06d120 2024-01-16T15:04:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.521+0000 7f3433e00640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 0x7f341406df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.521+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f342c088540 con 0x7f343c06d120 2024-01-16T15:04:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.522+0000 7f343b400640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 0x7f341406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.522+0000 7f343b400640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 0x7f341406df90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3424036000 tx=0x7f3424008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.526+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f342c054fb0 con 0x7f343c06d120 2024-01-16T15:04:35.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.740+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f343c070550 con 0x7f341406bad0 2024-01-16T15:04:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.742+0000 7f3433e00640 1 -- 172.21.15.125:0/2111371422 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3424030b90 con 0x7f341406bad0 2024-01-16T15:04:35.721 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:04:35.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:35.721 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:04:35.722 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:35.722 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:04:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 msgr2=0x7f341406df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 0x7f341406df90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3424036000 tx=0x7f3424008030 comp rx=0 tx=0).stop 2024-01-16T15:04:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 msgr2=0x7f343c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f342c00ebe0 tx=0x7f342c00c520 comp rx=0 tx=0).stop 2024-01-16T15:04:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 shutdown_connections 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f343c0b3ae0 0x7f343c1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f343c0ae5a0 0x7f343c1f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f341406bad0 0x7f341406df90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 --2- 172.21.15.125:0/2111371422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f343c06d120 0x7f343c0cda70 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 >> 172.21.15.125:0/2111371422 conn(0x7f343c06a800 msgr2=0x7f343c15a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 shutdown_connections 2024-01-16T15:04:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.745+0000 7f3443200640 1 -- 172.21.15.125:0/2111371422 wait complete. 2024-01-16T15:04:35.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:04:35.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.976+0000 7fc3d8000640 1 Processor -- start 2024-01-16T15:04:35.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.976+0000 7fc3d8000640 1 -- start start 2024-01-16T15:04:35.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d8000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3d00ae5a0 0x7fc3d015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 0x7fc3d00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d8000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d0070c40 con 0x7fc3d00b3ae0 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d8000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d0071450 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00cfc60 con 0x7fc3d00cf290 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d4c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc3d00b3ae0 0x7fc3d006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45160/0 (socket says 172.21.15.125:45160) 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d5600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3d00ae5a0 0x7fc3d015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.978+0000 7fc3d4c00640 1 -- 172.21.15.125:0/2206631567 learned_addr learned my addr 172.21.15.125:0/2206631567 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:35.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d6000640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 0x7fc3d00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d5600640 1 -- 172.21.15.125:0/2206631567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 msgr2=0x7fc3d00cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d5600640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 0x7fc3d00cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d5600640 1 -- 172.21.15.125:0/2206631567 >> v1:172.21.15.125:6789/0 conn(0x7fc3d00b3ae0 legacy=0x7fc3d006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d5600640 1 -- 172.21.15.125:0/2206631567 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d006c470 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3d6000640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 0x7fc3d00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.979+0000 7fc3cfe00640 1 -- 172.21.15.125:0/2206631567 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368124026 0 0) 0x7fc3d0070c40 con 0x7fc3d00b3ae0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.980+0000 7fc3d5600640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d015e430 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097a0 tx=0x7fc3c002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00102d3c4c0c38b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.980+0000 7fc3cfe00640 1 -- 172.21.15.125:0/2206631567 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0030610 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.980+0000 7fc3cfe00640 1 -- 172.21.15.125:0/2206631567 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0030770 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.980+0000 7fc3cfe00640 1 -- 172.21.15.125:0/2206631567 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0030b00 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 -- 172.21.15.125:0/2206631567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00ae5a0 msgr2=0x7fc3d015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d015e430 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097a0 tx=0x7fc3c002b960 comp rx=0 tx=0).stop 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 -- 172.21.15.125:0/2206631567 shutdown_connections 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00cf290 0x7fc3d00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 --2- 172.21.15.125:0/2206631567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d015e430 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 -- 172.21.15.125:0/2206631567 >> 172.21.15.125:0/2206631567 conn(0x7fc3d006a800 msgr2=0x7fc3d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 -- 172.21.15.125:0/2206631567 shutdown_connections 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.981+0000 7fc3d8000640 1 -- 172.21.15.125:0/2206631567 wait complete. 2024-01-16T15:04:35.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.983+0000 7fc3d8000640 1 Processor -- start 2024-01-16T15:04:35.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.983+0000 7fc3d8000640 1 -- start start 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d01f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00b3ae0 0x7fc3d015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01f2970 con 0x7fc3d00cf290 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d015f450 con 0x7fc3d00ae5a0 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d015f730 con 0x7fc3d00b3ae0 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34220/0 (socket says 172.21.15.125:34220) 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 -- 172.21.15.125:0/3504155881 learned_addr learned my addr 172.21.15.125:0/3504155881 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d4c00640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00b3ae0 0x7fc3d015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d5600640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d01f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 -- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00b3ae0 msgr2=0x7fc3d015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00b3ae0 0x7fc3d015eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 -- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3d00ae5a0 msgr2=0x7fc3d01f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d01f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.984+0000 7fc3d6000640 1 -- 172.21.15.125:0/3504155881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0009400 con 0x7fc3d00cf290 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3d6000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800cd50 tx=0x7fc3c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c800f680 con 0x7fc3d00cf290 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c800f800 con 0x7fc3d00cf290 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d015fa10 con 0x7fc3d00cf290 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c800fbb0 con 0x7fc3d00cf290 2024-01-16T15:04:35.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.985+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d0157110 con 0x7fc3d00cf290 2024-01-16T15:04:35.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.986+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d0157330 con 0x7fc3d00cf290 2024-01-16T15:04:35.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.987+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fc3c800b040 con 0x7fc3d00cf290 2024-01-16T15:04:35.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.988+0000 7fc3cd600640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 0x7fc3a806df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.988+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3c8088c90 con 0x7fc3d00cf290 2024-01-16T15:04:35.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.988+0000 7fc3d5600640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 0x7fc3a806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.989+0000 7fc3d5600640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 0x7fc3a806df60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0036000 tx=0x7fc3c003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:35.993+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3c8055700 con 0x7fc3d00cf290 2024-01-16T15:04:36.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.209+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3d0071170 con 0x7fc3a806baa0 2024-01-16T15:04:36.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.227+0000 7fc3cd600640 1 -- 172.21.15.125:0/3504155881 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7fc3c00308e0 con 0x7fc3a806baa0 2024-01-16T15:04:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:36.206 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:04:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27074" 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27072" 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27075" 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27077" 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27071" 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27073" 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone27076" 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:36.208 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:04:36.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 msgr2=0x7fc3a806df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 0x7fc3a806df60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0036000 tx=0x7fc3c003b040 comp rx=0 tx=0).stop 2024-01-16T15:04:36.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 msgr2=0x7fc3d015c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800cd50 tx=0x7fc3c800a650 comp rx=0 tx=0).stop 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 shutdown_connections 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc3a806baa0 0x7fc3a806df60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d00cf290 0x7fc3d015c5f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3d00b3ae0 0x7fc3d015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 --2- 172.21.15.125:0/3504155881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3d00ae5a0 0x7fc3d01f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 >> 172.21.15.125:0/3504155881 conn(0x7fc3d006a800 msgr2=0x7fc3d015d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 shutdown_connections 2024-01-16T15:04:36.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.230+0000 7fc3d8000640 1 -- 172.21.15.125:0/3504155881 wait complete. 2024-01-16T15:04:36.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27074 2024-01-16T15:04:36.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.447+0000 7fe8c0400640 1 Processor -- start 2024-01-16T15:04:36.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.447+0000 7fe8c0400640 1 -- start start 2024-01-16T15:04:36.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8c0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 0x7fe8b8071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8c0400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe8b8072210 0x7fe8b8070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8c0400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b8072660 con 0x7fe8b815b3f0 2024-01-16T15:04:36.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8c0400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b80ae8a0 con 0x7fe8b8072210 2024-01-16T15:04:36.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8c0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b815fd40 con 0x7fe8b80ae3f0 2024-01-16T15:04:36.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8bda00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe8b815b3f0 0x7fe8b80adce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45172/0 (socket says 172.21.15.125:45172) 2024-01-16T15:04:36.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8bda00640 1 -- 172.21.15.125:0/1576441963 learned_addr learned my addr 172.21.15.125:0/1576441963 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:36.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8be400640 1 --2- 172.21.15.125:0/1576441963 >> v2:172.21.15.137:3300/0 conn(0x7fe8b8072210 0x7fe8b8070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188497789 0 0) 0x7fe8a8001110 con 0x7fe8b815b3f0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.449+0000 7fe8bd000640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 0x7fe8b8071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898003520 con 0x7fe8b815b3f0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3922462114 0 0) 0x7fe898003520 con 0x7fe8b815b3f0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 msgr2=0x7fe8b8071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 0x7fe8b8071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 >> v2:172.21.15.137:3300/0 conn(0x7fe8b8072210 msgr2=0x7fe8b8070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 --2- 172.21.15.125:0/1576441963 >> v2:172.21.15.137:3300/0 conn(0x7fe8b8072210 0x7fe8b8070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b815ef80 con 0x7fe8b815b3f0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe8a8001800 con 0x7fe8b815b3f0 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 >> v1:172.21.15.125:6789/0 conn(0x7fe8b815b3f0 legacy=0x7fe8b80adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe898003970 0x7fe898003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8980047c0 0x7fe898004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 0x7fe898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b80ae8a0 con 0x7fe898003970 2024-01-16T15:04:36.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b815fd40 con 0x7fe8980047c0 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89800b980 con 0x7fe8b80ae3f0 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.450+0000 7fe8bda00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe898003970 0x7fe898003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bd000640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8980047c0 0x7fe898004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bda00640 1 -- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 msgr2=0x7fe898005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bda00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b80ae3f0 0x7fe898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bda00640 1 -- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8980047c0 msgr2=0x7fe898004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bda00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8980047c0 0x7fe898004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bda00640 1 -- 172.21.15.125:0/1576441963 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898003520 con 0x7fe898003970 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.451+0000 7fe8bd000640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8980047c0 0x7fe898004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8bda00640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe898003970 0x7fe898003d90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800c210 tx=0x7fe8980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225247910668896b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8005ad0 con 0x7fe898003970 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a8005c30 con 0x7fe898003970 2024-01-16T15:04:36.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8b7e00640 1 -- 172.21.15.125:0/1576441963 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a802c8d0 con 0x7fe898003970 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8c0400640 1 -- 172.21.15.125:0/1576441963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe898003970 msgr2=0x7fe898003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8c0400640 1 --2- 172.21.15.125:0/1576441963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe898003970 0x7fe898003d90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800c210 tx=0x7fe8980057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.452+0000 7fe8be400640 1 -- 172.21.15.125:0/1576441963 reap_dead start 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.453+0000 7fe8c0400640 1 -- 172.21.15.125:0/1576441963 shutdown_connections 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.453+0000 7fe8c0400640 1 -- 172.21.15.125:0/1576441963 >> 172.21.15.125:0/1576441963 conn(0x7fe8b80b0c00 msgr2=0x7fe8b80b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.453+0000 7fe8c0400640 1 -- 172.21.15.125:0/1576441963 shutdown_connections 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.453+0000 7fe8c0400640 1 -- 172.21.15.125:0/1576441963 wait complete. 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.454+0000 7fe8c0400640 1 Processor -- start 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.454+0000 7fe8c0400640 1 -- start start 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe898003970 0x7fe8b814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b814e9b0 0x7fe8b814ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b80ccee0 con 0x7fe8b80cc520 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b814e7b0 con 0x7fe898003970 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8c0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81f3020 con 0x7fe8b814e9b0 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.455+0000 7fe8bd000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34232/0 (socket says 172.21.15.125:34232) 2024-01-16T15:04:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 -- 172.21.15.125:0/4001137871 learned_addr learned my addr 172.21.15.125:0/4001137871 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 -- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b814e9b0 msgr2=0x7fe8b814ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b814e9b0 0x7fe8b814ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 -- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe898003970 msgr2=0x7fe8b814e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe898003970 0x7fe8b814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 -- 172.21.15.125:0/4001137871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b81f32c0 con 0x7fe8b80cc520 2024-01-16T15:04:36.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8bd000640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0010830 tx=0x7fe8a0010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a000eb60 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a000ecc0 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a001c400 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.456+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b81f35a0 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.457+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b81f3b60 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.458+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b8067640 con 0x7fe8b80cc520 2024-01-16T15:04:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.459+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fe8a001c560 con 0x7fe8b80cc520 2024-01-16T15:04:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.459+0000 7fe8b5600640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 0x7fe89006df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.460+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8a0088850 con 0x7fe8b80cc520 2024-01-16T15:04:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.460+0000 7fe8bda00640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 0x7fe89006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.460+0000 7fe8bda00640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 0x7fe89006df60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8034000 tx=0x7fe8a802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.464+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8a0055370 con 0x7fe8b80cc520 2024-01-16T15:04:36.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.664+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27074", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8b815f820 con 0x7fe89006baa0 2024-01-16T15:04:36.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.686+0000 7fe8b5600640 1 -- 172.21.15.125:0/4001137871 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe8a8001540 con 0x7fe89006baa0 2024-01-16T15:04:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 msgr2=0x7fe89006df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 0x7fe89006df60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8034000 tx=0x7fe8a802e000 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 msgr2=0x7fe8b80cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0010830 tx=0x7fe8a0010cf0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 shutdown_connections 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8b814e9b0 0x7fe8b814ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe89006baa0 0x7fe89006df60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8b80cc520 0x7fe8b80cc9a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 --2- 172.21.15.125:0/4001137871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe898003970 0x7fe8b814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 >> 172.21.15.125:0/4001137871 conn(0x7fe8b80b0c00 msgr2=0x7fe8b80cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 shutdown_connections 2024-01-16T15:04:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.689+0000 7fe8c0400640 1 -- 172.21.15.125:0/4001137871 wait complete. 2024-01-16T15:04:36.669 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.691+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.669 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.691+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.669 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.691+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.669 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.691+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.669 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.691+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:04:36.691 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.710+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.692 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.710+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.692 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.710+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.692 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.710+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.692 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:04:36.710+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:36.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.890+0000 7fb2e3800640 1 Processor -- start 2024-01-16T15:04:36.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.890+0000 7fb2e3800640 1 -- start start 2024-01-16T15:04:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.891+0000 7fb2e3800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc0cf9a0 0x7fb2dc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.891+0000 7fb2e3800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.891+0000 7fb2e3800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0b1ee0 con 0x7fb2dc143ae0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.891+0000 7fb2e3800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0b2780 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.891+0000 7fb2e3800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc15e990 con 0x7fb2dc0cf9a0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40830/0 (socket says 172.21.15.125:40830) 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 -- 172.21.15.125:0/4041875365 learned_addr learned my addr 172.21.15.125:0/4041875365 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2dbe00640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc0cf9a0 0x7fb2dc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 -- 172.21.15.125:0/4041875365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc0cf9a0 msgr2=0x7fb2dc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc0cf9a0 0x7fb2dc15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 -- 172.21.15.125:0/4041875365 >> v1:172.21.15.137:6789/0 conn(0x7fb2dc143ae0 legacy=0x7fb2dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.892+0000 7fb2e1800640 1 -- 172.21.15.125:0/4041875365 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0b2130 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.893+0000 7fb2e1800640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009870 tx=0x7fb2d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d171475f82614aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.893+0000 7fb2db400640 1 -- 172.21.15.125:0/4041875365 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4031640 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2db400640 1 -- 172.21.15.125:0/4041875365 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d40317a0 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2db400640 1 -- 172.21.15.125:0/4041875365 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4031b30 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 -- 172.21.15.125:0/4041875365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 msgr2=0x7fb2dc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009870 tx=0x7fb2d402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 -- 172.21.15.125:0/4041875365 shutdown_connections 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc0702c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 --2- 172.21.15.125:0/4041875365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc0cf9a0 0x7fb2dc15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 -- 172.21.15.125:0/4041875365 >> 172.21.15.125:0/4041875365 conn(0x7fb2dc06a800 msgr2=0x7fb2dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 -- 172.21.15.125:0/4041875365 shutdown_connections 2024-01-16T15:04:36.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.894+0000 7fb2e3800640 1 -- 172.21.15.125:0/4041875365 wait complete. 2024-01-16T15:04:36.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.896+0000 7fb2e3800640 1 Processor -- start 2024-01-16T15:04:36.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.896+0000 7fb2e3800640 1 -- start start 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc0cf9a0 0x7fb2dc142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc143ae0 0x7fb2dc15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc1600c0 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0cb760 con 0x7fb2dc0cf9a0 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e3800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc15f4b0 con 0x7fb2dc143ae0 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40842/0 (socket says 172.21.15.125:40842) 2024-01-16T15:04:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.897+0000 7fb2e1800640 1 -- 172.21.15.125:0/330186932 learned_addr learned my addr 172.21.15.125:0/330186932 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e0e00640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc0cf9a0 0x7fb2dc142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2dbe00640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc143ae0 0x7fb2dc15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 -- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc143ae0 msgr2=0x7fb2dc15edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc143ae0 0x7fb2dc15edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 -- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc0cf9a0 msgr2=0x7fb2dc142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc0cf9a0 0x7fb2dc142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 -- 172.21.15.125:0/330186932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40094d0 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2e1800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb2d402b930 tx=0x7fb2d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d403ac60 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d403ade0 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.899+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc15f790 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.898+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4044630 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.899+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc15fd50 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.900+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc160230 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.901+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 20) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fb2d402f050 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.901+0000 7fb2d8c00640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 0x7fb2b406df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.902+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d40aa0f0 con 0x7fb2dc15e1e0 2024-01-16T15:04:36.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.902+0000 7fb2e0e00640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 0x7fb2b406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.902+0000 7fb2e0e00640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 0x7fb2b406df90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc12a160 tx=0x7fb2cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:36.906+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2d40aee60 con 0x7fb2dc15e1e0 2024-01-16T15:04:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:37.126+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2dc15e640 con 0x7fb2b406bad0 2024-01-16T15:04:38.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:38.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:39.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:39.203+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb2d4076fe0 con 0x7fb2dc15e1e0 2024-01-16T15:04:40.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.274+0000 7fb2d8c00640 1 -- 172.21.15.125:0/330186932 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fb2cc005450 con 0x7fb2b406bad0 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:04:40.253 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:04:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.277+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 msgr2=0x7fb2b406df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.277+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 0x7fb2b406df90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc12a160 tx=0x7fb2cc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.277+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 msgr2=0x7fb2dc141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.277+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb2d402b930 tx=0x7fb2d40046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 shutdown_connections 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2b406bad0 0x7fb2b406df90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2dc15e1e0 0x7fb2dc141180 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc143ae0 0x7fb2dc15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 --2- 172.21.15.125:0/330186932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc0cf9a0 0x7fb2dc142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 >> 172.21.15.125:0/330186932 conn(0x7fb2dc06a800 msgr2=0x7fb2dc06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 shutdown_connections 2024-01-16T15:04:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.278+0000 7fb2e3800640 1 -- 172.21.15.125:0/330186932 wait complete. 2024-01-16T15:04:40.275 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:04:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.509+0000 7f7d00800640 1 Processor -- start 2024-01-16T15:04:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.510+0000 7f7d00800640 1 -- start start 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.511+0000 7f7d00800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.511+0000 7f7d00800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 0x7f7cf80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.511+0000 7f7d00800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8070c40 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.511+0000 7f7d00800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf8071450 con 0x7f7cf80cf290 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.511+0000 7f7d00800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf80cfc60 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.512+0000 7f7cfd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.512+0000 7f7cfd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56218/0 (socket says 172.21.15.125:56218) 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.512+0000 7f7cfd400640 1 -- 172.21.15.125:0/4058299592 learned_addr learned my addr 172.21.15.125:0/4058299592 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.512+0000 7f7cfe800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 0x7f7cf80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.512+0000 7f7cfde00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7cf80ae5a0 0x7f7cf806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:46480/0 (socket says 172.21.15.125:46480) 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.513+0000 7f7cfd400640 1 -- 172.21.15.125:0/4058299592 >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 msgr2=0x7f7cf80cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.513+0000 7f7cfd400640 1 --2- 172.21.15.125:0/4058299592 >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 0x7f7cf80cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.513+0000 7f7cfd400640 1 -- 172.21.15.125:0/4058299592 >> v1:172.21.15.137:6789/0 conn(0x7f7cf80ae5a0 legacy=0x7f7cf806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.513+0000 7f7cfd400640 1 -- 172.21.15.125:0/4058299592 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf806c470 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.513+0000 7f7cfe800640 1 --2- 172.21.15.125:0/4058299592 >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 0x7f7cf80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.514+0000 7f7cfd400640 1 --2- 172.21.15.125:0/4058299592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009870 tx=0x7f7ce002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3001819425bee552 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.514+0000 7f7cfca00640 1 -- 172.21.15.125:0/4058299592 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce002f830 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.514+0000 7f7cfca00640 1 -- 172.21.15.125:0/4058299592 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce002f990 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.514+0000 7f7cfca00640 1 -- 172.21.15.125:0/4058299592 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce002fd20 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 -- 172.21.15.125:0/4058299592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 msgr2=0x7f7cf815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 --2- 172.21.15.125:0/4058299592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009870 tx=0x7f7ce002ba10 comp rx=0 tx=0).stop 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 -- 172.21.15.125:0/4058299592 shutdown_connections 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 --2- 172.21.15.125:0/4058299592 >> v2:172.21.15.137:3300/0 conn(0x7f7cf80cf290 0x7f7cf80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 --2- 172.21.15.125:0/4058299592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80b3ae0 0x7f7cf815e430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 -- 172.21.15.125:0/4058299592 >> 172.21.15.125:0/4058299592 conn(0x7f7cf806a800 msgr2=0x7f7cf806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 -- 172.21.15.125:0/4058299592 shutdown_connections 2024-01-16T15:04:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.515+0000 7f7d00800640 1 -- 172.21.15.125:0/4058299592 wait complete. 2024-01-16T15:04:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.517+0000 7f7d00800640 1 Processor -- start 2024-01-16T15:04:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.517+0000 7f7d00800640 1 -- start start 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cf80b3ae0 0x7f7cf814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80cf290 0x7f7cf815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8159290 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8157c00 con 0x7f7cf80b3ae0 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7d00800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8156ff0 con 0x7f7cf80cf290 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40858/0 (socket says 172.21.15.125:40858) 2024-01-16T15:04:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfde00640 1 -- 172.21.15.125:0/2730643251 learned_addr learned my addr 172.21.15.125:0/2730643251 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfd400640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cf80b3ae0 0x7f7cf814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfe800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80cf290 0x7f7cf815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.518+0000 7f7cfde00640 1 -- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80cf290 msgr2=0x7f7cf815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cfde00640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80cf290 0x7f7cf815fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cfde00640 1 -- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cf80b3ae0 msgr2=0x7f7cf814e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cfde00640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cf80b3ae0 0x7f7cf814e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cfde00640 1 -- 172.21.15.125:0/2730643251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0009a70 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cfde00640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800c4d0 tx=0x7f7ce800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8007480 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce80075e0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80049e0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce00094d0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.519+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf81575b0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.520+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf80675d0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.521+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7ce8004b40 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.522+0000 7f7cf6000640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 0x7f7cd006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.522+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ce80874d0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.522+0000 7f7cfd400640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 0x7f7cd006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.523+0000 7f7cfd400640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 0x7f7cd006e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0036000 tx=0x7f7ce0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.527+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ce8053ea0 con 0x7f7cf80ae5a0 2024-01-16T15:04:40.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.747+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7cf8071170 con 0x7f7cd006bbc0 2024-01-16T15:04:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.803+0000 7f7cf6000640 1 -- 172.21.15.125:0/2730643251 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7ce002fc90 con 0x7f7cd006bbc0 2024-01-16T15:04:40.782 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 msgr2=0x7f7cd006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 0x7f7cd006e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0036000 tx=0x7f7ce0008030 comp rx=0 tx=0).stop 2024-01-16T15:04:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 msgr2=0x7f7cf81492a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800c4d0 tx=0x7f7ce800c990 comp rx=0 tx=0).stop 2024-01-16T15:04:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 shutdown_connections 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cf80cf290 0x7f7cf815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cf80b3ae0 0x7f7cf814e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7cd006bbc0 0x7f7cd006e080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 --2- 172.21.15.125:0/2730643251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf80ae5a0 0x7f7cf81492a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.806+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 >> 172.21.15.125:0/2730643251 conn(0x7f7cf806a800 msgr2=0x7f7cf815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.807+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 shutdown_connections 2024-01-16T15:04:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:40.807+0000 7f7d00800640 1 -- 172.21.15.125:0/2730643251 wait complete. 2024-01-16T15:04:40.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:04:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.034+0000 7faaf8c00640 1 Processor -- start 2024-01-16T15:04:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.035+0000 7faaf8c00640 1 -- start start 2024-01-16T15:04:41.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.036+0000 7faaf8c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faaf40ce8f0 0x7faaf40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.036+0000 7faaf8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf40b2a00 0x7faaf40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.036+0000 7faaf8c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf40d0de0 con 0x7faaf4160f80 2024-01-16T15:04:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.036+0000 7faaf8c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf40d10c0 con 0x7faaf40ce8f0 2024-01-16T15:04:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.036+0000 7faaf8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40d13a0 con 0x7faaf40b2a00 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faaf40ce8f0 0x7faaf40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faaf40ce8f0 0x7faaf40ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40862/0 (socket says 172.21.15.125:40862) 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 -- 172.21.15.125:0/2857693041 learned_addr learned my addr 172.21.15.125:0/2857693041 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faaf2800640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf40b2a00 0x7faaf40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 -- 172.21.15.125:0/2857693041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf40b2a00 msgr2=0x7faaf40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf40b2a00 0x7faaf40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 -- 172.21.15.125:0/2857693041 >> v1:172.21.15.137:6789/0 conn(0x7faaf4160f80 legacy=0x7faaf4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.037+0000 7faae9200640 1 -- 172.21.15.125:0/2857693041 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf40d1680 con 0x7faaf40ce8f0 2024-01-16T15:04:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.038+0000 7faae9200640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40ce8f0 0x7faaf40ced30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7faadc009870 tx=0x7faadc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109ab297761b78fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.038+0000 7faaf1400640 1 -- 172.21.15.125:0/2857693041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc030840 con 0x7faaf40ce8f0 2024-01-16T15:04:41.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf1400640 1 -- 172.21.15.125:0/2857693041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc0309a0 con 0x7faaf40ce8f0 2024-01-16T15:04:41.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf1400640 1 -- 172.21.15.125:0/2857693041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc030d30 con 0x7faaf40ce8f0 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 -- 172.21.15.125:0/2857693041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40ce8f0 msgr2=0x7faaf40ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40ce8f0 0x7faaf40ced30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7faadc009870 tx=0x7faadc02b980 comp rx=0 tx=0).stop 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 -- 172.21.15.125:0/2857693041 shutdown_connections 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf40b2a00 0x7faaf40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 --2- 172.21.15.125:0/2857693041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40ce8f0 0x7faaf40ced30 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 -- 172.21.15.125:0/2857693041 >> 172.21.15.125:0/2857693041 conn(0x7faaf409c550 msgr2=0x7faaf40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 -- 172.21.15.125:0/2857693041 shutdown_connections 2024-01-16T15:04:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.039+0000 7faaf8c00640 1 -- 172.21.15.125:0/2857693041 wait complete. 2024-01-16T15:04:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.041+0000 7faaf8c00640 1 Processor -- start 2024-01-16T15:04:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.041+0000 7faaf8c00640 1 -- start start 2024-01-16T15:04:41.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 0x7faaf41f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf4160f80 0x7faaf415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40cba90 con 0x7faaf40b2a00 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40b9120 con 0x7faaf40ce8f0 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf41f6620 con 0x7faaf4160f80 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.042+0000 7faaf1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40874/0 (socket says 172.21.15.125:40874) 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 -- 172.21.15.125:0/957084855 learned_addr learned my addr 172.21.15.125:0/957084855 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faae9200640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 0x7faaf41f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf2800640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf4160f80 0x7faaf415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 -- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf4160f80 msgr2=0x7faaf415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf4160f80 0x7faaf415e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 -- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 msgr2=0x7faaf41f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 0x7faaf41f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 -- 172.21.15.125:0/957084855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc0094d0 con 0x7faaf40b2a00 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faae9200640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 0x7faaf41f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.043+0000 7faaf1e00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7faae400c400 tx=0x7faae400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.044+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae40052d0 con 0x7faaf40b2a00 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.044+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4005430 con 0x7faaf40b2a00 2024-01-16T15:04:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.044+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf40b8e40 con 0x7faaf40b2a00 2024-01-16T15:04:41.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.044+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4003830 con 0x7faaf40b2a00 2024-01-16T15:04:41.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.044+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf40cb720 con 0x7faaf40b2a00 2024-01-16T15:04:41.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.045+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf40cb8b0 con 0x7faaf40b2a00 2024-01-16T15:04:41.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.046+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7faae4003990 con 0x7faaf40b2a00 2024-01-16T15:04:41.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.047+0000 7faaeaa00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 0x7faacc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.047+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faae4087970 con 0x7faaf40b2a00 2024-01-16T15:04:41.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.047+0000 7faae9200640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 0x7faacc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.048+0000 7faae9200640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 0x7faacc06e0b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faadc036000 tx=0x7faadc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.052+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faae4054290 con 0x7faaf40b2a00 2024-01-16T15:04:41.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.270+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf40d1c40 con 0x7faacc06bbf0 2024-01-16T15:04:41.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.306+0000 7faaeaa00640 1 -- 172.21.15.125:0/957084855 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faadc030b90 con 0x7faacc06bbf0 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 msgr2=0x7faacc06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 0x7faacc06e0b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faadc036000 tx=0x7faadc03b040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 msgr2=0x7faaf414a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7faae400c400 tx=0x7faae400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 shutdown_connections 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faaf4160f80 0x7faaf415e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaf40ce8f0 0x7faaf41f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faacc06bbf0 0x7faacc06e0b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 --2- 172.21.15.125:0/957084855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf40b2a00 0x7faaf414a410 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 >> 172.21.15.125:0/957084855 conn(0x7faaf409c550 msgr2=0x7faaf406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 shutdown_connections 2024-01-16T15:04:41.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.309+0000 7faaf8c00640 1 -- 172.21.15.125:0/957084855 wait complete. 2024-01-16T15:04:41.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27072 2024-01-16T15:04:41.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.521+0000 7f05f5200640 1 Processor -- start 2024-01-16T15:04:41.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.521+0000 7f05f5200640 1 -- start start 2024-01-16T15:04:41.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05f5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013df80 0x7f05f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05f5200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05f013e360 0x7f05f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05f5200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f006c470 con 0x7f05f015f9c0 2024-01-16T15:04:41.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05f5200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f006c750 con 0x7f05f013e360 2024-01-16T15:04:41.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05f5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f006ca30 con 0x7f05f013df80 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05ee800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f05f015f9c0 0x7f05f01408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48324/0 (socket says 172.21.15.125:48324) 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05e7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05f013e360 0x7f05f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.523+0000 7f05ee800640 1 -- 172.21.15.125:0/1901116791 learned_addr learned my addr 172.21.15.125:0/1901116791 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05ede00640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013df80 0x7f05f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05ed400640 1 -- 172.21.15.125:0/1901116791 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472704455 0 0) 0x7f05f006c470 con 0x7f05f015f9c0 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05ed400640 1 -- 172.21.15.125:0/1901116791 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d0003520 con 0x7f05f015f9c0 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05e7e00640 1 -- 172.21.15.125:0/1901116791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013df80 msgr2=0x7f05f015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05e7e00640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013df80 0x7f05f015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05e7e00640 1 -- 172.21.15.125:0/1901116791 >> v1:172.21.15.125:6789/0 conn(0x7f05f015f9c0 legacy=0x7f05f01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.524+0000 7f05e7e00640 1 -- 172.21.15.125:0/1901116791 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f006cd10 con 0x7f05f013e360 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.525+0000 7f05e7e00640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f013e360 0x7f05f00cf290 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f05d80097a0 tx=0x7f05d802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2c300ca15fab08 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.525+0000 7f05ed400640 1 -- 172.21.15.125:0/1901116791 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d802d040 con 0x7f05f013e360 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05ed400640 1 -- 172.21.15.125:0/1901116791 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d8031810 con 0x7f05f013e360 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05ed400640 1 -- 172.21.15.125:0/1901116791 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8031ba0 con 0x7f05f013e360 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 -- 172.21.15.125:0/1901116791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f013e360 msgr2=0x7f05f00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f013e360 0x7f05f00cf290 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f05d80097a0 tx=0x7f05d802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 -- 172.21.15.125:0/1901116791 shutdown_connections 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f013e360 0x7f05f00cf290 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 --2- 172.21.15.125:0/1901116791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013df80 0x7f05f015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 -- 172.21.15.125:0/1901116791 >> 172.21.15.125:0/1901116791 conn(0x7f05f006a800 msgr2=0x7f05f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 -- 172.21.15.125:0/1901116791 shutdown_connections 2024-01-16T15:04:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.526+0000 7f05f5200640 1 -- 172.21.15.125:0/1901116791 wait complete. 2024-01-16T15:04:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.528+0000 7f05f5200640 1 Processor -- start 2024-01-16T15:04:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.528+0000 7f05f5200640 1 -- start start 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05f013df80 0x7f05f014c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 0x7f05f0151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014e380 con 0x7f05f015f9c0 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014ccc0 con 0x7f05f013df80 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05f5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014cfa0 con 0x7f05f013e360 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05ee800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05ee800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40890/0 (socket says 172.21.15.125:40890) 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.530+0000 7f05ee800640 1 -- 172.21.15.125:0/4043136191 learned_addr learned my addr 172.21.15.125:0/4043136191 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05e7e00640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 0x7f05f0151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ede00640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05f013df80 0x7f05f014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 -- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 msgr2=0x7f05f0151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 0x7f05f0151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 -- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05f013df80 msgr2=0x7f05f014c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05f013df80 0x7f05f014c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 -- 172.21.15.125:0/4043136191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8009400 con 0x7f05f015f9c0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05e7e00640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 0x7f05f0151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05ee800640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f05e800cb40 tx=0x7f05e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800aca0 con 0x7f05f015f9c0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e800ae00 con 0x7f05f015f9c0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8012620 con 0x7f05f015f9c0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.531+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f014d280 con 0x7f05f015f9c0 2024-01-16T15:04:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.532+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f014d840 con 0x7f05f015f9c0 2024-01-16T15:04:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.533+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f00675d0 con 0x7f05f015f9c0 2024-01-16T15:04:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.534+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f05e8007070 con 0x7f05f015f9c0 2024-01-16T15:04:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.534+0000 7f05e6000640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 0x7f05c806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.535+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05e8088500 con 0x7f05f015f9c0 2024-01-16T15:04:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.535+0000 7f05ede00640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 0x7f05c806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.535+0000 7f05ede00640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 0x7f05c806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009970 tx=0x7f05e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.539+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05e8054dc0 con 0x7f05f015f9c0 2024-01-16T15:04:41.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.758+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27072", "target": ["mon-mgr", ""]}) v1 -- 0x7f05f00ce850 con 0x7f05c806bbc0 2024-01-16T15:04:41.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.786+0000 7f05e6000640 1 -- 172.21.15.125:0/4043136191 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f05e0010870 con 0x7f05c806bbc0 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.788+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 msgr2=0x7f05c806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.788+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 0x7f05c806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009970 tx=0x7f05e0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 msgr2=0x7f05f0136cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f05e800cb40 tx=0x7f05e800a060 comp rx=0 tx=0).stop 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 shutdown_connections 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f05c806bbc0 0x7f05c806e080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05f015f9c0 0x7f05f0136cb0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05f013e360 0x7f05f0151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 --2- 172.21.15.125:0/4043136191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05f013df80 0x7f05f014c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 >> 172.21.15.125:0/4043136191 conn(0x7f05f006a800 msgr2=0x7f05f00b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 shutdown_connections 2024-01-16T15:04:41.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.789+0000 7f05f5200640 1 -- 172.21.15.125:0/4043136191 wait complete. 2024-01-16T15:04:41.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27075 2024-01-16T15:04:41.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.992+0000 7fdd4b200640 1 Processor -- start 2024-01-16T15:04:41.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.992+0000 7fdd4b200640 1 -- start start 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd4b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdd44143ae0 0x7fdd440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd4b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 0x7fdd4415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd4b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd440cfc30 con 0x7fdd4415e1e0 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd4b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd440b1ee0 con 0x7fdd44143ae0 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd4b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440b2780 con 0x7fdd440cf7d0 2024-01-16T15:04:41.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd49200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdd4415e1e0 0x7fdd4415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48328/0 (socket says 172.21.15.125:48328) 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.993+0000 7fdd49200640 1 -- 172.21.15.125:0/2073864398 learned_addr learned my addr 172.21.15.125:0/2073864398 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd43400640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 0x7fdd4415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd43e00640 1 --2- 172.21.15.125:0/2073864398 >> v2:172.21.15.137:3300/0 conn(0x7fdd44143ae0 0x7fdd440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694937982 0 0) 0x7fdd440cfc30 con 0x7fdd4415e1e0 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd24003520 con 0x7fdd4415e1e0 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3344806601 0 0) 0x7fdd24003520 con 0x7fdd4415e1e0 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 msgr2=0x7fdd4415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 0x7fdd4415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 >> v2:172.21.15.137:3300/0 conn(0x7fdd44143ae0 msgr2=0x7fdd440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 --2- 172.21.15.125:0/2073864398 >> v2:172.21.15.137:3300/0 conn(0x7fdd44143ae0 0x7fdd440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4415e990 con 0x7fdd4415e1e0 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdd3c0019f0 con 0x7fdd4415e1e0 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 >> v1:172.21.15.125:6789/0 conn(0x7fdd4415e1e0 legacy=0x7fdd4415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 0x7fdd24003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd24003ef0 0x7fdd240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24004d00 0x7fdd24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440b1ee0 con 0x7fdd24003ef0 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440b2780 con 0x7fdd24004d00 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2400b9d0 con 0x7fdd440cf7d0 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.994+0000 7fdd43e00640 1 --2- 172.21.15.125:0/2073864398 >> v2:172.21.15.137:3300/0 conn(0x7fdd44143ae0 0x7fdd440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd24003ef0 0x7fdd240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 -- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 msgr2=0x7fdd24003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440cf7d0 0x7fdd24003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 -- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24004d00 msgr2=0x7fdd24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24004d00 0x7fdd24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.995+0000 7fdd43400640 1 -- 172.21.15.125:0/2073864398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24003520 con 0x7fdd24003ef0 2024-01-16T15:04:41.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.996+0000 7fdd43400640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd24003ef0 0x7fdd240047c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00bd60 tx=0x7fdd240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec15c6a6e2f6a593 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.996+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c02f710 con 0x7fdd24003ef0 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.996+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c02f870 con 0x7fdd24003ef0 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.996+0000 7fdd42a00640 1 -- 172.21.15.125:0/2073864398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c02fc00 con 0x7fdd24003ef0 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 -- 172.21.15.125:0/2073864398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd24003ef0 msgr2=0x7fdd240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2073864398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd24003ef0 0x7fdd240047c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00bd60 tx=0x7fdd240057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd49200640 1 -- 172.21.15.125:0/2073864398 reap_dead start 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 -- 172.21.15.125:0/2073864398 shutdown_connections 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 -- 172.21.15.125:0/2073864398 >> 172.21.15.125:0/2073864398 conn(0x7fdd4406a800 msgr2=0x7fdd4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 -- 172.21.15.125:0/2073864398 shutdown_connections 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.997+0000 7fdd4b200640 1 -- 172.21.15.125:0/2073864398 wait complete. 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.999+0000 7fdd4b200640 1 Processor -- start 2024-01-16T15:04:41.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:41.999+0000 7fdd4b200640 1 -- start start 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24003ef0 0x7fdd44158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440a8a00 0x7fdd4415c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4415bf30 con 0x7fdd441593f0 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440a8eb0 con 0x7fdd24003ef0 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd4b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440cea70 con 0x7fdd440a8a00 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40904/0 (socket says 172.21.15.125:40904) 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 -- 172.21.15.125:0/2640270568 learned_addr learned my addr 172.21.15.125:0/2640270568 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 -- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440a8a00 msgr2=0x7fdd4415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440a8a00 0x7fdd4415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 -- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24003ef0 msgr2=0x7fdd44158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24003ef0 0x7fdd44158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.000+0000 7fdd43400640 1 -- 172.21.15.125:0/2640270568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4415b590 con 0x7fdd441593f0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd43400640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c02f0b0 tx=0x7fdd2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c003810 con 0x7fdd441593f0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c030e90 con 0x7fdd441593f0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4415b870 con 0x7fdd441593f0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c003b90 con 0x7fdd441593f0 2024-01-16T15:04:41.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.001+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd44158fc0 con 0x7fdd441593f0 2024-01-16T15:04:41.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.002+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd440675d0 con 0x7fdd441593f0 2024-01-16T15:04:41.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.003+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fdd2c003cf0 con 0x7fdd441593f0 2024-01-16T15:04:41.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.005+0000 7fdd3be00640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 0x7fdd1c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.005+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd2c0adb30 con 0x7fdd441593f0 2024-01-16T15:04:41.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.005+0000 7fdd43e00640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 0x7fdd1c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.006+0000 7fdd43e00640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 0x7fdd1c06e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd34006fd0 tx=0x7fdd34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.009+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd2c07a3d0 con 0x7fdd441593f0 2024-01-16T15:04:42.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.226+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27075", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd4415e640 con 0x7fdd1c06bbc0 2024-01-16T15:04:42.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.257+0000 7fdd3be00640 1 -- 172.21.15.125:0/2640270568 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd34009da0 con 0x7fdd1c06bbc0 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 msgr2=0x7fdd1c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 0x7fdd1c06e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd34006fd0 tx=0x7fdd34008040 comp rx=0 tx=0).stop 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 msgr2=0x7fdd440a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c02f0b0 tx=0x7fdd2c004730 comp rx=0 tx=0).stop 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 shutdown_connections 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd440a8a00 0x7fdd4415c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd1c06bbc0 0x7fdd1c06e080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd441593f0 0x7fdd440a84c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 --2- 172.21.15.125:0/2640270568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd24003ef0 0x7fdd44158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 >> 172.21.15.125:0/2640270568 conn(0x7fdd4406a800 msgr2=0x7fdd4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 shutdown_connections 2024-01-16T15:04:42.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.260+0000 7fdd4b200640 1 -- 172.21.15.125:0/2640270568 wait complete. 2024-01-16T15:04:42.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27077 2024-01-16T15:04:42.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.476+0000 7f297f400640 1 Processor -- start 2024-01-16T15:04:42.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.476+0000 7f297f400640 1 -- start start 2024-01-16T15:04:42.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.477+0000 7f297f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f2978071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297f400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2978070e50 con 0x7f29780717d0 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297f400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2978144020 con 0x7f29780ccc10 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29780723c0 con 0x7f2978143c40 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297d600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f29780717d0 0x7f29780cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48344/0 (socket says 172.21.15.125:48344) 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297d600640 1 -- 172.21.15.125:0/609992148 learned_addr learned my addr 172.21.15.125:0/609992148 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297e000640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f2978071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004701578 0 0) 0x7f2978070e50 con 0x7f29780717d0 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.478+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2958003520 con 0x7f29780717d0 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4195030027 0 0) 0x7f2958003520 con 0x7f29780717d0 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 msgr2=0x7f2978071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f2978071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 >> v1:172.21.15.137:6789/0 conn(0x7f29780ccc10 legacy=0x7f297806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978071e00 con 0x7f29780717d0 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2960001670 con 0x7f29780717d0 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 >> v1:172.21.15.125:6789/0 conn(0x7f29780717d0 legacy=0x7f29780cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29580039a0 0x7f2958003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29580047c0 0x7f2958004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f29580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29780723c0 con 0x7f29580047c0 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295800bab0 con 0x7f29580039a0 2024-01-16T15:04:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295800bc70 con 0x7f2978143c40 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.479+0000 7f297d600640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29580047c0 0x7f2958004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297ea00640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f29580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297e000640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29580039a0 0x7f2958003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297d600640 1 -- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 msgr2=0x7f29580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297d600640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2978143c40 0x7f29580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297d600640 1 -- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29580039a0 msgr2=0x7f2958003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297d600640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29580039a0 0x7f2958003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.480+0000 7f297d600640 1 -- 172.21.15.125:0/609992148 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2958003520 con 0x7f29580047c0 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297d600640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29580047c0 0x7f2958004c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f296000c240 tx=0x7f29580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3823a7a2794ae4c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960005ad0 con 0x7f29580047c0 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2960005c50 con 0x7f29580047c0 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297cc00640 1 -- 172.21.15.125:0/609992148 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296002c8b0 con 0x7f29580047c0 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297f400640 1 -- 172.21.15.125:0/609992148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29580047c0 msgr2=0x7f2958004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.481+0000 7f297f400640 1 --2- 172.21.15.125:0/609992148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29580047c0 0x7f2958004c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f296000c240 tx=0x7f29580057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.482+0000 7f297ea00640 1 -- 172.21.15.125:0/609992148 reap_dead start 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.482+0000 7f297f400640 1 -- 172.21.15.125:0/609992148 shutdown_connections 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.482+0000 7f297f400640 1 -- 172.21.15.125:0/609992148 >> 172.21.15.125:0/609992148 conn(0x7f297806a800 msgr2=0x7f297806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.482+0000 7f297f400640 1 -- 172.21.15.125:0/609992148 shutdown_connections 2024-01-16T15:04:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.482+0000 7f297f400640 1 -- 172.21.15.125:0/609992148 wait complete. 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.483+0000 7f297f400640 1 Processor -- start 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.484+0000 7f297f400640 1 -- start start 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.484+0000 7f297f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29580047c0 0x7f297813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.484+0000 7f297f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f297813f520 0x7f297815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297f400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29780c62e0 con 0x7f29780aeb30 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29780b3970 con 0x7f297813f520 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29780c9b40 con 0x7f29580047c0 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40916/0 (socket says 172.21.15.125:40916) 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 -- 172.21.15.125:0/3657403477 learned_addr learned my addr 172.21.15.125:0/3657403477 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297e000640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29580047c0 0x7f297813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297d600640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f297813f520 0x7f297815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 -- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29580047c0 msgr2=0x7f297813fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29580047c0 0x7f297813fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 -- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f297813f520 msgr2=0x7f297815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f297813f520 0x7f297815aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.485+0000 7f297ea00640 1 -- 172.21.15.125:0/3657403477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780cabf0 con 0x7f29780aeb30 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f297ea00640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2978144d60 tx=0x7f297000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970003970 con 0x7f29780aeb30 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970003ad0 con 0x7f29780aeb30 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970003e80 con 0x7f29780aeb30 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29780ca630 con 0x7f29780aeb30 2024-01-16T15:04:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.486+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29780c5f00 con 0x7f29780aeb30 2024-01-16T15:04:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.487+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2978067450 con 0x7f29780aeb30 2024-01-16T15:04:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.489+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f29700050d0 con 0x7f29780aeb30 2024-01-16T15:04:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.490+0000 7f2976000640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 0x7f295006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.490+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2970088400 con 0x7f29780aeb30 2024-01-16T15:04:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.490+0000 7f297e000640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 0x7f295006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.490+0000 7f297e000640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 0x7f295006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29780cc930 tx=0x7f2968008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.494+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2970054b30 con 0x7f29780aeb30 2024-01-16T15:04:42.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.715+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27077", "target": ["mon-mgr", ""]}) v1 -- 0x7f29780af4f0 con 0x7f295006bbc0 2024-01-16T15:04:42.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.744+0000 7f2976000640 1 -- 172.21.15.125:0/3657403477 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f29780af4f0 con 0x7f295006bbc0 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.747+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 msgr2=0x7f295006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.747+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 0x7f295006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29780cc930 tx=0x7f2968008040 comp rx=0 tx=0).stop 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.747+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 msgr2=0x7f29780aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2978144d60 tx=0x7f297000ec10 comp rx=0 tx=0).stop 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 shutdown_connections 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f295006bbc0 0x7f295006e080 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29780aeb30 0x7f29780aefb0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f297813f520 0x7f297815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 --2- 172.21.15.125:0/3657403477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29580047c0 0x7f297813fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 >> 172.21.15.125:0/3657403477 conn(0x7f297806a800 msgr2=0x7f297806d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 shutdown_connections 2024-01-16T15:04:42.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.748+0000 7f297f400640 1 -- 172.21.15.125:0/3657403477 wait complete. 2024-01-16T15:04:42.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27071 2024-01-16T15:04:42.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.978+0000 7fa623600640 1 Processor -- start 2024-01-16T15:04:42.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.978+0000 7fa623600640 1 -- start start 2024-01-16T15:04:42.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa61c15d610 0x7fa61c0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c0ced60 0x7fa61c0d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa61c0d1d20 0x7fa61c0cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0d1390 con 0x7fa61c15d610 2024-01-16T15:04:42.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0d21b0 con 0x7fa61c0d1d20 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.980+0000 7fa623600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b3400 con 0x7fa61c0ced60 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.981+0000 7fa620c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa61c15d610 0x7fa61c0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.981+0000 7fa620c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa61c15d610 0x7fa61c0ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40922/0 (socket says 172.21.15.125:40922) 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.981+0000 7fa620c00640 1 -- 172.21.15.125:0/1724295125 learned_addr learned my addr 172.21.15.125:0/1724295125 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.981+0000 7fa621600640 1 --2- 172.21.15.125:0/1724295125 >> v2:172.21.15.137:3300/0 conn(0x7fa61c0d1d20 0x7fa61c0cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.981+0000 7fa61be00640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c0ced60 0x7fa61c0d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.982+0000 7fa620c00640 1 -- 172.21.15.125:0/1724295125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c0ced60 msgr2=0x7fa61c0d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.982+0000 7fa620c00640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c0ced60 0x7fa61c0d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.982+0000 7fa620c00640 1 -- 172.21.15.125:0/1724295125 >> v2:172.21.15.137:3300/0 conn(0x7fa61c0d1d20 msgr2=0x7fa61c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.982+0000 7fa620c00640 1 --2- 172.21.15.125:0/1724295125 >> v2:172.21.15.137:3300/0 conn(0x7fa61c0d1d20 0x7fa61c0cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.982+0000 7fa620c00640 1 -- 172.21.15.125:0/1724295125 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c0b27f0 con 0x7fa61c15d610 2024-01-16T15:04:42.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.983+0000 7fa620c00640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c15d610 0x7fa61c0ce820 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0d42d0 tx=0x7fa60c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373e3f25d24f6ef3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.984+0000 7fa61b400640 1 -- 172.21.15.125:0/1724295125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c031870 con 0x7fa61c15d610 2024-01-16T15:04:42.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.984+0000 7fa61b400640 1 -- 172.21.15.125:0/1724295125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c0319d0 con 0x7fa61c15d610 2024-01-16T15:04:42.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.984+0000 7fa61b400640 1 -- 172.21.15.125:0/1724295125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c031d60 con 0x7fa61c15d610 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 -- 172.21.15.125:0/1724295125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c15d610 msgr2=0x7fa61c0ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c15d610 0x7fa61c0ce820 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0d42d0 tx=0x7fa60c02b910 comp rx=0 tx=0).stop 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 -- 172.21.15.125:0/1724295125 shutdown_connections 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 --2- 172.21.15.125:0/1724295125 >> v2:172.21.15.137:3300/0 conn(0x7fa61c0d1d20 0x7fa61c0cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c0ced60 0x7fa61c0d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 --2- 172.21.15.125:0/1724295125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c15d610 0x7fa61c0ce820 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.985+0000 7fa623600640 1 -- 172.21.15.125:0/1724295125 >> 172.21.15.125:0/1724295125 conn(0x7fa61c09c540 msgr2=0x7fa61c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.986+0000 7fa623600640 1 -- 172.21.15.125:0/1724295125 shutdown_connections 2024-01-16T15:04:42.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.986+0000 7fa623600640 1 -- 172.21.15.125:0/1724295125 wait complete. 2024-01-16T15:04:42.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.989+0000 7fa623600640 1 Processor -- start 2024-01-16T15:04:42.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.989+0000 7fa623600640 1 -- start start 2024-01-16T15:04:42.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 0x7fa61c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 0x7fa61c0ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c15d610 0x7fa61c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0cd570 con 0x7fa61c0d1d20 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b8650 con 0x7fa61c0ced60 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa623600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c15fac0 con 0x7fa61c15d610 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa61be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 0x7fa61c0ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa61be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 0x7fa61c0ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40934/0 (socket says 172.21.15.125:40934) 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.990+0000 7fa61be00640 1 -- 172.21.15.125:0/1655236771 learned_addr learned my addr 172.21.15.125:0/1655236771 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 0x7fa61c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa621600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c15d610 0x7fa61c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 -- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c15d610 msgr2=0x7fa61c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c15d610 0x7fa61c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 -- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 msgr2=0x7fa61c0ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 0x7fa61c0ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.991+0000 7fa620c00640 1 -- 172.21.15.125:0/1655236771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0094d0 con 0x7fa61c0ced60 2024-01-16T15:04:42.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.992+0000 7fa620c00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 0x7fa61c0cd030 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa60c033000 tx=0x7fa60c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.993+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c03ac60 con 0x7fa61c0ced60 2024-01-16T15:04:42.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.993+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c03adc0 con 0x7fa61c0ced60 2024-01-16T15:04:42.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.993+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c044630 con 0x7fa61c0ced60 2024-01-16T15:04:42.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.993+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c15fda0 con 0x7fa61c0ced60 2024-01-16T15:04:42.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.994+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c0b6ba0 con 0x7fa61c0ced60 2024-01-16T15:04:42.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.995+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c15ff30 con 0x7fa61c0ced60 2024-01-16T15:04:42.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.997+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa60c04d450 con 0x7fa61c0ced60 2024-01-16T15:04:42.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.999+0000 7fa618c00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 0x7fa5f406e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:42.999+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa60c02d040 con 0x7fa61c0ced60 2024-01-16T15:04:42.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.000+0000 7fa61be00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 0x7fa5f406e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.002+0000 7fa61be00640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 0x7fa5f406e150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa61c160980 tx=0x7fa604009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.006+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa60c0774d0 con 0x7fa61c0ced60 2024-01-16T15:04:43.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.350+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27071", "target": ["mon-mgr", ""]}) v1 -- 0x7fa61c0b3090 con 0x7fa5f406bc90 2024-01-16T15:04:43.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.367+0000 7fa618c00640 1 -- 172.21.15.125:0/1655236771 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa604011a90 con 0x7fa5f406bc90 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 msgr2=0x7fa5f406e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 0x7fa5f406e150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa61c160980 tx=0x7fa604009190 comp rx=0 tx=0).stop 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 msgr2=0x7fa61c0cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 0x7fa61c0cd030 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa60c033000 tx=0x7fa60c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 shutdown_connections 2024-01-16T15:04:43.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa61c15d610 0x7fa61c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa5f406bc90 0x7fa5f406e150 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa61c0d1d20 0x7fa61c0ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 --2- 172.21.15.125:0/1655236771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa61c0ced60 0x7fa61c0cd030 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 >> 172.21.15.125:0/1655236771 conn(0x7fa61c09c540 msgr2=0x7fa61c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 shutdown_connections 2024-01-16T15:04:43.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.370+0000 7fa623600640 1 -- 172.21.15.125:0/1655236771 wait complete. 2024-01-16T15:04:43.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27073 2024-01-16T15:04:43.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.578+0000 7f8ce1c00640 1 Processor -- start 2024-01-16T15:04:43.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.578+0000 7f8ce1c00640 1 -- start start 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8ce1c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8cdc160980 0x7f8cdc06ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8ce1c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8cdc070af0 0x7f8cdc070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8ce1c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cdc071480 con 0x7f8cdc161a10 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8ce1c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cdc0cf1a0 con 0x7f8cdc160980 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8ce1c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cdc161000 con 0x7f8cdc070af0 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8cd1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8cdc161a10 0x7f8cdc0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48350/0 (socket says 172.21.15.125:48350) 2024-01-16T15:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8cd1e00640 1 -- 172.21.15.125:0/455350597 learned_addr learned my addr 172.21.15.125:0/455350597 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.580+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/455350597 >> v2:172.21.15.125:3300/0 conn(0x7f8cdc160980 0x7f8cdc06ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cdb400640 1 --2- 172.21.15.125:0/455350597 >> v2:172.21.15.137:3300/0 conn(0x7f8cdc070af0 0x7f8cdc070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cda000640 1 -- 172.21.15.125:0/455350597 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384869267 0 0) 0x7f8cc4001110 con 0x7f8cdc161a10 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cda000640 1 -- 172.21.15.125:0/455350597 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc003520 con 0x7f8cdc161a10 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cdaa00640 1 -- 172.21.15.125:0/455350597 >> v2:172.21.15.137:3300/0 conn(0x7f8cdc070af0 msgr2=0x7f8cdc070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/455350597 >> v2:172.21.15.137:3300/0 conn(0x7f8cdc070af0 0x7f8cdc070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cdaa00640 1 -- 172.21.15.125:0/455350597 >> v1:172.21.15.125:6789/0 conn(0x7f8cdc161a10 legacy=0x7f8cdc0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.581+0000 7f8cdaa00640 1 -- 172.21.15.125:0/455350597 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc161580 con 0x7f8cdc160980 2024-01-16T15:04:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.582+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/455350597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc160980 0x7f8cdc06ba30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc06eb60 tx=0x7f8ccc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c8448732f88f65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8cda000640 1 -- 172.21.15.125:0/455350597 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc031830 con 0x7f8cdc160980 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8cda000640 1 -- 172.21.15.125:0/455350597 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc031990 con 0x7f8cdc160980 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8cda000640 1 -- 172.21.15.125:0/455350597 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc031d20 con 0x7f8cdc160980 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 -- 172.21.15.125:0/455350597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc160980 msgr2=0x7f8cdc06ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/455350597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc160980 0x7f8cdc06ba30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc06eb60 tx=0x7f8ccc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 -- 172.21.15.125:0/455350597 shutdown_connections 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/455350597 >> v2:172.21.15.137:3300/0 conn(0x7f8cdc070af0 0x7f8cdc070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/455350597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc160980 0x7f8cdc06ba30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 -- 172.21.15.125:0/455350597 >> 172.21.15.125:0/455350597 conn(0x7f8cdc0cf700 msgr2=0x7f8cdc0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.583+0000 7f8ce1c00640 1 -- 172.21.15.125:0/455350597 shutdown_connections 2024-01-16T15:04:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.584+0000 7f8ce1c00640 1 -- 172.21.15.125:0/455350597 wait complete. 2024-01-16T15:04:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.585+0000 7f8ce1c00640 1 Processor -- start 2024-01-16T15:04:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.585+0000 7f8ce1c00640 1 -- start start 2024-01-16T15:04:43.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 0x7f8cdc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cdc161a10 0x7f8cdc140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc15de40 con 0x7f8cdc070af0 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc158620 con 0x7f8cdc160980 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.586+0000 7f8ce1c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc158900 con 0x7f8cdc161a10 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40960/0 (socket says 172.21.15.125:40960) 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 -- 172.21.15.125:0/100233998 learned_addr learned my addr 172.21.15.125:0/100233998 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cd1e00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 0x7f8cdc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 -- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cdc161a10 msgr2=0x7f8cdc140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdb400640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cdc161a10 0x7f8cdc140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cdc161a10 0x7f8cdc140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 -- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 msgr2=0x7f8cdc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 0x7f8cdc157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cdaa00640 1 -- 172.21.15.125:0/100233998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc009400 con 0x7f8cdc070af0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.587+0000 7f8cd1e00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 0x7f8cdc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8cdaa00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009890 tx=0x7f8ccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc03a6f0 con 0x7f8cdc070af0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc03a870 con 0x7f8cdc070af0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc03ac20 con 0x7f8cdc070af0 2024-01-16T15:04:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc15d780 con 0x7f8cdc070af0 2024-01-16T15:04:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.588+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc158330 con 0x7f8cdc070af0 2024-01-16T15:04:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.589+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc066390 con 0x7f8cdc070af0 2024-01-16T15:04:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.590+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8ccc043410 con 0x7f8cdc070af0 2024-01-16T15:04:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.591+0000 7f8cd3400640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 0x7f8cb406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.591+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ccc0a9fe0 con 0x7f8cdc070af0 2024-01-16T15:04:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.591+0000 7f8cd1e00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 0x7f8cb406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.592+0000 7f8cd1e00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 0x7f8cb406e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc1f6e50 tx=0x7f8cc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.595+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ccc0769c0 con 0x7f8cdc070af0 2024-01-16T15:04:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.805+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27073", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cdc161e70 con 0x7f8cb406bbc0 2024-01-16T15:04:43.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.833+0000 7f8cd3400640 1 -- 172.21.15.125:0/100233998 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8cc40013a0 con 0x7f8cb406bbc0 2024-01-16T15:04:43.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.835+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 msgr2=0x7f8cb406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.835+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 0x7f8cb406e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc1f6e50 tx=0x7f8cc4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.835+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 msgr2=0x7f8cdc155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.835+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009890 tx=0x7f8ccc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 shutdown_connections 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cdc161a10 0x7f8cdc140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cdc160980 0x7f8cdc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8cb406bbc0 0x7f8cb406e080 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 --2- 172.21.15.125:0/100233998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8cdc070af0 0x7f8cdc155a20 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 >> 172.21.15.125:0/100233998 conn(0x7f8cdc0cf700 msgr2=0x7f8cdc15b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 shutdown_connections 2024-01-16T15:04:43.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:43.836+0000 7f8ce1c00640 1 -- 172.21.15.125:0/100233998 wait complete. 2024-01-16T15:04:43.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27076 2024-01-16T15:04:44.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.098+0000 7f4bade00640 1 Processor -- start 2024-01-16T15:04:44.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.098+0000 7f4bade00640 1 -- start start 2024-01-16T15:04:44.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.100+0000 7f4bade00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4ba813df80 0x7f4ba815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.100+0000 7f4bade00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ba813e360 0x7f4ba80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.100+0000 7f4bade00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba806c470 con 0x7f4ba815f9c0 2024-01-16T15:04:44.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.100+0000 7f4bade00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba806c750 con 0x7f4ba813df80 2024-01-16T15:04:44.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.100+0000 7f4bade00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba806ca30 con 0x7f4ba813e360 2024-01-16T15:04:44.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba7400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4ba815f9c0 0x7f4ba81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:46522/0 (socket says 172.21.15.125:46522) 2024-01-16T15:04:44.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba7400640 1 -- 172.21.15.125:0/9730507 learned_addr learned my addr 172.21.15.125:0/9730507 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba6000640 1 --2- 172.21.15.125:0/9730507 >> v2:172.21.15.137:3300/0 conn(0x7f4ba813e360 0x7f4ba80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba6a00640 1 --2- 172.21.15.125:0/9730507 >> v2:172.21.15.125:3300/0 conn(0x7f4ba813df80 0x7f4ba815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825248757 0 0) 0x7f4ba806c470 con 0x7f4ba815f9c0 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.101+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88003520 con 0x7f4ba815f9c0 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.102+0000 7f4ba6000640 1 -- 172.21.15.125:0/9730507 >> v2:172.21.15.125:3300/0 conn(0x7f4ba813df80 msgr2=0x7f4ba815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.102+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28103489 0 0) 0x7f4b88003520 con 0x7f4ba815f9c0 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.102+0000 7f4ba6000640 1 --2- 172.21.15.125:0/9730507 >> v2:172.21.15.125:3300/0 conn(0x7f4ba813df80 0x7f4ba815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.102+0000 7f4ba6000640 1 -- 172.21.15.125:0/9730507 >> v1:172.21.15.137:6789/0 conn(0x7f4ba815f9c0 legacy=0x7f4ba81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.102+0000 7f4ba6000640 1 -- 172.21.15.125:0/9730507 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba806cd10 con 0x7f4ba813e360 2024-01-16T15:04:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.104+0000 7f4ba6000640 1 --2- 172.21.15.125:0/9730507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba813e360 0x7f4ba80cf290 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4ba806ded0 tx=0x7f4b9002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19de4cacbb7f0d32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.104+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00019d0 con 0x7f4ba813e360 2024-01-16T15:04:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.104+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90033440 con 0x7f4ba813e360 2024-01-16T15:04:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.104+0000 7f4ba5600640 1 -- 172.21.15.125:0/9730507 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900337d0 con 0x7f4ba813e360 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 -- 172.21.15.125:0/9730507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba813e360 msgr2=0x7f4ba80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 --2- 172.21.15.125:0/9730507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba813e360 0x7f4ba80cf290 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4ba806ded0 tx=0x7f4b9002b7a0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 -- 172.21.15.125:0/9730507 shutdown_connections 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 --2- 172.21.15.125:0/9730507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba813e360 0x7f4ba80cf290 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 --2- 172.21.15.125:0/9730507 >> v2:172.21.15.125:3300/0 conn(0x7f4ba813df80 0x7f4ba815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.106+0000 7f4bade00640 1 -- 172.21.15.125:0/9730507 >> 172.21.15.125:0/9730507 conn(0x7f4ba806a800 msgr2=0x7f4ba806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.107+0000 7f4bade00640 1 -- 172.21.15.125:0/9730507 shutdown_connections 2024-01-16T15:04:44.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.107+0000 7f4bade00640 1 -- 172.21.15.125:0/9730507 wait complete. 2024-01-16T15:04:44.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.108+0000 7f4bade00640 1 Processor -- start 2024-01-16T15:04:44.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.109+0000 7f4bade00640 1 -- start start 2024-01-16T15:04:44.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ba813e360 0x7f4ba80b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba815f9c0 0x7f4ba80b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba813dab0 con 0x7f4ba813df80 2024-01-16T15:04:44.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8151280 con 0x7f4ba815f9c0 2024-01-16T15:04:44.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.110+0000 7f4bade00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba814f510 con 0x7f4ba813e360 2024-01-16T15:04:44.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.111+0000 7f4ba6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.111+0000 7f4ba7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba815f9c0 0x7f4ba80b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.111+0000 7f4ba6000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ba813e360 0x7f4ba80b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.111+0000 7f4ba6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40988/0 (socket says 172.21.15.125:40988) 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.111+0000 7f4ba6a00640 1 -- 172.21.15.125:0/737068130 learned_addr learned my addr 172.21.15.125:0/737068130 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 -- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ba813e360 msgr2=0x7f4ba80b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ba813e360 0x7f4ba80b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 -- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba815f9c0 msgr2=0x7f4ba80b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba815f9c0 0x7f4ba80b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 -- 172.21.15.125:0/737068130 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90009400 con 0x7f4ba813df80 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.112+0000 7f4ba6a00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800d840 tx=0x7f4b9800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.113+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800b6c0 con 0x7f4ba813df80 2024-01-16T15:04:44.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.113+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9800b820 con 0x7f4ba813df80 2024-01-16T15:04:44.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.113+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800bbd0 con 0x7f4ba813df80 2024-01-16T15:04:44.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.113+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8151d00 con 0x7f4ba813df80 2024-01-16T15:04:44.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.113+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba81522c0 con 0x7f4ba813df80 2024-01-16T15:04:44.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.117+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba80675d0 con 0x7f4ba813df80 2024-01-16T15:04:44.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.120+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f4b9800bd30 con 0x7f4ba813df80 2024-01-16T15:04:44.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.121+0000 7f4b9f400640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 0x7f4b8006e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.121+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b98088b70 con 0x7f4ba813df80 2024-01-16T15:04:44.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.125+0000 7f4ba6000640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 0x7f4b8006e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.128+0000 7f4ba6000640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 0x7f4b8006e150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4b90035000 tx=0x7f4b9002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.132+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b98055240 con 0x7f4ba813df80 2024-01-16T15:04:44.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.544+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27076", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ba80ce850 con 0x7f4b8006bc90 2024-01-16T15:04:44.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.577+0000 7f4b9f400640 1 -- 172.21.15.125:0/737068130 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4b900338b0 con 0x7f4b8006bc90 2024-01-16T15:04:44.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.581+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 msgr2=0x7f4b8006e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.581+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 0x7f4b8006e150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4b90035000 tx=0x7f4b9002f000 comp rx=0 tx=0).stop 2024-01-16T15:04:44.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.581+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 msgr2=0x7f4ba8150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.581+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800d840 tx=0x7f4b9800dd00 comp rx=0 tx=0).stop 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 shutdown_connections 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ba815f9c0 0x7f4ba80b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ba813e360 0x7f4ba80b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b8006bc90 0x7f4b8006e150 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 --2- 172.21.15.125:0/737068130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba813df80 0x7f4ba8150d40 secure :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800d840 tx=0x7f4b9800dd00 comp rx=0 tx=0).stop 2024-01-16T15:04:44.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 >> 172.21.15.125:0/737068130 conn(0x7f4ba806a800 msgr2=0x7f4ba815d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 shutdown_connections 2024-01-16T15:04:44.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.582+0000 7f4bade00640 1 -- 172.21.15.125:0/737068130 wait complete. 2024-01-16T15:04:44.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:44.591 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:04:44.637 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T15:04:44.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:04:44.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.837+0000 7fc793200640 1 Processor -- start 2024-01-16T15:04:44.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.837+0000 7fc793200640 1 -- start start 2024-01-16T15:04:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc793200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc78c0cd240 0x7fc78c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc793200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc78c09a170 0x7fc78c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc793200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c06cea0 con 0x7fc78c06c6b0 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc793200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c06cbc0 con 0x7fc78c0cd240 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc793200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c06f350 con 0x7fc78c09a170 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc78b400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc78c06c6b0 0x7fc78c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48366/0 (socket says 172.21.15.125:48366) 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc78b400640 1 -- 172.21.15.125:0/830498611 learned_addr learned my addr 172.21.15.125:0/830498611 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.839+0000 7fc78be00640 1 --2- 172.21.15.125:0/830498611 >> v2:172.21.15.125:3300/0 conn(0x7fc78c0cd240 0x7fc78c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc791200640 1 --2- 172.21.15.125:0/830498611 >> v2:172.21.15.137:3300/0 conn(0x7fc78c09a170 0x7fc78c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc78be00640 1 -- 172.21.15.125:0/830498611 >> v2:172.21.15.137:3300/0 conn(0x7fc78c09a170 msgr2=0x7fc78c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc78be00640 1 --2- 172.21.15.125:0/830498611 >> v2:172.21.15.137:3300/0 conn(0x7fc78c09a170 0x7fc78c0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc78be00640 1 -- 172.21.15.125:0/830498611 >> v1:172.21.15.125:6789/0 conn(0x7fc78c06c6b0 legacy=0x7fc78c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc78be00640 1 -- 172.21.15.125:0/830498611 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c06f630 con 0x7fc78c0cd240 2024-01-16T15:04:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.840+0000 7fc78aa00640 1 -- 172.21.15.125:0/830498611 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274592304 0 0) 0x7fc78c06cea0 con 0x7fc78c06c6b0 2024-01-16T15:04:44.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.841+0000 7fc78be00640 1 --2- 172.21.15.125:0/830498611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c0b3820 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0cb7f0 tx=0x7fc77c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed90fc4cdd081b14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.841+0000 7fc78aa00640 1 -- 172.21.15.125:0/830498611 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c031830 con 0x7fc78c0cd240 2024-01-16T15:04:44.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.841+0000 7fc78aa00640 1 -- 172.21.15.125:0/830498611 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c031990 con 0x7fc78c0cd240 2024-01-16T15:04:44.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.841+0000 7fc78aa00640 1 -- 172.21.15.125:0/830498611 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c031d20 con 0x7fc78c0cd240 2024-01-16T15:04:44.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 -- 172.21.15.125:0/830498611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 msgr2=0x7fc78c0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 --2- 172.21.15.125:0/830498611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c0b3820 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0cb7f0 tx=0x7fc77c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 -- 172.21.15.125:0/830498611 shutdown_connections 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 --2- 172.21.15.125:0/830498611 >> v2:172.21.15.137:3300/0 conn(0x7fc78c09a170 0x7fc78c0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 --2- 172.21.15.125:0/830498611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c0b3820 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 -- 172.21.15.125:0/830498611 >> 172.21.15.125:0/830498611 conn(0x7fc78c0b2d00 msgr2=0x7fc78c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 -- 172.21.15.125:0/830498611 shutdown_connections 2024-01-16T15:04:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.842+0000 7fc793200640 1 -- 172.21.15.125:0/830498611 wait complete. 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.843+0000 7fc793200640 1 Processor -- start 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.844+0000 7fc793200640 1 -- start start 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc78c06c6b0 0x7fc78c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78c09a170 0x7fc78c0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c1f2ce0 con 0x7fc78c0cd240 2024-01-16T15:04:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0c57a0 con 0x7fc78c06c6b0 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc793200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0c5a80 con 0x7fc78c09a170 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41006/0 (socket says 172.21.15.125:41006) 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 -- 172.21.15.125:0/2701416987 learned_addr learned my addr 172.21.15.125:0/2701416987 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 -- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78c09a170 msgr2=0x7fc78c0c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc78b400640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78c09a170 0x7fc78c0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc78be00640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc78c06c6b0 0x7fc78c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78c09a170 0x7fc78c0c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 -- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc78c06c6b0 msgr2=0x7fc78c157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc78c06c6b0 0x7fc78c157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.845+0000 7fc791200640 1 -- 172.21.15.125:0/2701416987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c009400 con 0x7fc78c0cd240 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc791200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d5a0 tx=0x7fc78400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7840125f0 con 0x7fc78c0cd240 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784012750 con 0x7fc78c0cd240 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784012b00 con 0x7fc78c0cd240 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c0c5d60 con 0x7fc78c0cd240 2024-01-16T15:04:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.846+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c0c6320 con 0x7fc78c0cd240 2024-01-16T15:04:44.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.847+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c0c6540 con 0x7fc78c0cd240 2024-01-16T15:04:44.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.848+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc784012c60 con 0x7fc78c0cd240 2024-01-16T15:04:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.849+0000 7fc783e00640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 0x7fc76406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.849+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc784088bd0 con 0x7fc78c0cd240 2024-01-16T15:04:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.849+0000 7fc78be00640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 0x7fc76406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.850+0000 7fc78be00640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 0x7fc76406e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc77c036000 tx=0x7fc77c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:44.854+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc784055210 con 0x7fc78c0cd240 2024-01-16T15:04:45.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:45.164+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fc78c0cd620 con 0x7fc76406bbc0 2024-01-16T15:04:45.188 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T15:04:45.209+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:04:45.289 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:04:47.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:47.188+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc7840555e0 con 0x7fc78c0cd240 2024-01-16T15:04:48.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.234+0000 7fc783e00640 1 -- 172.21.15.125:0/2701416987 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fc77c031b80 con 0x7fc76406bbc0 2024-01-16T15:04:48.215 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:04:48.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.239+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 msgr2=0x7fc76406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.239+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 0x7fc76406e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc77c036000 tx=0x7fc77c03c040 comp rx=0 tx=0).stop 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.239+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 msgr2=0x7fc78c15e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.239+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d5a0 tx=0x7fc78400da60 comp rx=0 tx=0).stop 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 shutdown_connections 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc76406bbc0 0x7fc76406e080 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc78c0cd240 0x7fc78c15e420 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78c09a170 0x7fc78c0c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 --2- 172.21.15.125:0/2701416987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc78c06c6b0 0x7fc78c157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 >> 172.21.15.125:0/2701416987 conn(0x7fc78c0b2d00 msgr2=0x7fc78c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 shutdown_connections 2024-01-16T15:04:48.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.240+0000 7fc793200640 1 -- 172.21.15.125:0/2701416987 wait complete. 2024-01-16T15:04:48.252 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:04:48.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:48.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:48.538 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:04:48.538 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:04:48.538 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:48.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.816+0000 7f61c1800640 1 Processor -- start 2024-01-16T15:04:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.816+0000 7f61c1800640 1 -- start start 2024-01-16T15:04:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.817+0000 7f61c1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.817+0000 7f61c1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61bc15e3b0 0x7f61bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.817+0000 7f61c1800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61bc0cfc30 con 0x7f61bc0cf7d0 2024-01-16T15:04:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.817+0000 7f61c1800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc0b1ee0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.817+0000 7f61c1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0b2780 con 0x7f61bc143ae0 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61bc15e3b0 0x7f61bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61bc15e3b0 0x7f61bc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48814/0 (socket says 172.21.15.125:48814) 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 -- 172.21.15.125:0/832850502 learned_addr learned my addr 172.21.15.125:0/832850502 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 -- 172.21.15.125:0/832850502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 msgr2=0x7f61bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 --2- 172.21.15.125:0/832850502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 -- 172.21.15.125:0/832850502 >> v1:172.21.15.125:6789/0 conn(0x7f61bc0cf7d0 legacy=0x7f61bc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.818+0000 7f61bb400640 1 -- 172.21.15.125:0/832850502 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc15e990 con 0x7f61bc15e3b0 2024-01-16T15:04:48.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.819+0000 7f61bb400640 1 --2- 172.21.15.125:0/832850502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc0702c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009870 tx=0x7f61b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbd729bca6ccb81 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.819+0000 7f61ba000640 1 -- 172.21.15.125:0/832850502 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4030840 con 0x7f61bc15e3b0 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.819+0000 7f61ba000640 1 -- 172.21.15.125:0/832850502 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b40309a0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.819+0000 7f61ba000640 1 -- 172.21.15.125:0/832850502 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4030d30 con 0x7f61bc15e3b0 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 -- 172.21.15.125:0/832850502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 msgr2=0x7f61bc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 --2- 172.21.15.125:0/832850502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc0702c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009870 tx=0x7f61b402b980 comp rx=0 tx=0).stop 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 -- 172.21.15.125:0/832850502 shutdown_connections 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 --2- 172.21.15.125:0/832850502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc0702c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 --2- 172.21.15.125:0/832850502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 -- 172.21.15.125:0/832850502 >> 172.21.15.125:0/832850502 conn(0x7f61bc06a800 msgr2=0x7f61bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 -- 172.21.15.125:0/832850502 shutdown_connections 2024-01-16T15:04:48.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.820+0000 7f61c1800640 1 -- 172.21.15.125:0/832850502 wait complete. 2024-01-16T15:04:48.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.822+0000 7f61c1800640 1 Processor -- start 2024-01-16T15:04:48.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.822+0000 7f61c1800640 1 -- start start 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.822+0000 7f61c1800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0cf7d0 0x7f61bc159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.822+0000 7f61c1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc1f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.822+0000 7f61c1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61c1800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc1f5070 con 0x7f61bc15e3b0 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61c1800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc13d440 con 0x7f61bc0cf7d0 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61c1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc1f4ab0 con 0x7f61bc143ae0 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48816/0 (socket says 172.21.15.125:48816) 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 -- 172.21.15.125:0/747221118 learned_addr learned my addr 172.21.15.125:0/747221118 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 -- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 msgr2=0x7f61bc1f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc1f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 -- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0cf7d0 msgr2=0x7f61bc159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0cf7d0 0x7f61bc159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 -- 172.21.15.125:0/747221118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40094d0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.823+0000 7f61bb400640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009870 tx=0x7f61b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.824+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b403e070 con 0x7f61bc15e3b0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.824+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b40307d0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.824+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc1f4d90 con 0x7f61bc15e3b0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.824+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40423f0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.824+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc159eb0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.825+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc13cec0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.826+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f61b4042550 con 0x7f61bc15e3b0 2024-01-16T15:04:48.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.827+0000 7f61b3400640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 0x7f619406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.827+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f61b40aae20 con 0x7f61bc15e3b0 2024-01-16T15:04:48.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.827+0000 7f61baa00640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 0x7f619406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.828+0000 7f61baa00640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 0x7f619406dff0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009710 tx=0x7f61ac006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:48.832+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61bc13cec0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.008+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61bc13cec0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.010+0000 7f61b3400640 1 -- 172.21.15.125:0/747221118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f61bc13cec0 con 0x7f61bc15e3b0 2024-01-16T15:04:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:48.990 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:04:48.223107+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6835/2216643169":"2024-01-17T15:04:45.210297+0000","172.21.15.125:6834/2216643169":"2024-01-17T15:04:45.210297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:48.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.012+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 msgr2=0x7f619406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.012+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 0x7f619406dff0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009710 tx=0x7f61ac006c90 comp rx=0 tx=0).stop 2024-01-16T15:04:48.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.012+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 msgr2=0x7f61bc141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.012+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009870 tx=0x7f61b40046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 shutdown_connections 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f619406bb30 0x7f619406dff0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61bc15e3b0 0x7f61bc141a40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc143ae0 0x7f61bc1f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 --2- 172.21.15.125:0/747221118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0cf7d0 0x7f61bc159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 >> 172.21.15.125:0/747221118 conn(0x7f61bc06a800 msgr2=0x7f61bc15ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 shutdown_connections 2024-01-16T15:04:48.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.013+0000 7f61c1800640 1 -- 172.21.15.125:0/747221118 wait complete. 2024-01-16T15:04:49.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:49.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.234+0000 7f5e5ce00640 1 Processor -- start 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.234+0000 7f5e5ce00640 1 -- start start 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e5ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5e58143ae0 0x7f5e580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e5ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e5815e3b0 0x7f5e580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e5ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e580cfc30 con 0x7f5e580cf7d0 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e5ce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e580b1ee0 con 0x7f5e5815e3b0 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e5ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e580b2780 con 0x7f5e58143ae0 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e56800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e5815e3b0 0x7f5e580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.235+0000 7f5e56800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e5815e3b0 0x7f5e580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48818/0 (socket says 172.21.15.125:48818) 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.236+0000 7f5e56800640 1 -- 172.21.15.125:0/1653520082 learned_addr learned my addr 172.21.15.125:0/1653520082 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.236+0000 7f5e56800640 1 -- 172.21.15.125:0/1653520082 >> v2:172.21.15.137:3300/0 conn(0x7f5e58143ae0 msgr2=0x7f5e580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.236+0000 7f5e56800640 1 --2- 172.21.15.125:0/1653520082 >> v2:172.21.15.137:3300/0 conn(0x7f5e58143ae0 0x7f5e580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.236+0000 7f5e56800640 1 -- 172.21.15.125:0/1653520082 >> v1:172.21.15.137:6789/0 conn(0x7f5e580cf7d0 legacy=0x7f5e5815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.236+0000 7f5e56800640 1 -- 172.21.15.125:0/1653520082 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5815e990 con 0x7f5e5815e3b0 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.237+0000 7f5e56800640 1 --2- 172.21.15.125:0/1653520082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e5815e3b0 0x7f5e580702c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5e580cfea0 tx=0x7f5e4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b7d9b9498fe375 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.237+0000 7f5e54a00640 1 -- 172.21.15.125:0/1653520082 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48031870 con 0x7f5e5815e3b0 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.237+0000 7f5e54a00640 1 -- 172.21.15.125:0/1653520082 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e480319d0 con 0x7f5e5815e3b0 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.237+0000 7f5e54a00640 1 -- 172.21.15.125:0/1653520082 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48031d60 con 0x7f5e5815e3b0 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 -- 172.21.15.125:0/1653520082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e5815e3b0 msgr2=0x7f5e580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/1653520082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e5815e3b0 0x7f5e580702c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5e580cfea0 tx=0x7f5e4802b910 comp rx=0 tx=0).stop 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 -- 172.21.15.125:0/1653520082 shutdown_connections 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/1653520082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e5815e3b0 0x7f5e580702c0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/1653520082 >> v2:172.21.15.137:3300/0 conn(0x7f5e58143ae0 0x7f5e580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 -- 172.21.15.125:0/1653520082 >> 172.21.15.125:0/1653520082 conn(0x7f5e5806a800 msgr2=0x7f5e5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 -- 172.21.15.125:0/1653520082 shutdown_connections 2024-01-16T15:04:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.238+0000 7f5e5ce00640 1 -- 172.21.15.125:0/1653520082 wait complete. 2024-01-16T15:04:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.239+0000 7f5e5ce00640 1 Processor -- start 2024-01-16T15:04:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 -- start start 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 0x7f5e58159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5815e3b0 0x7f5e5813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5815a260 con 0x7f5e58143ae0 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5813cce0 con 0x7f5e5815e3b0 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.240+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5813cfc0 con 0x7f5e580cf7d0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48834/0 (socket says 172.21.15.125:48834) 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 -- 172.21.15.125:0/3874827411 learned_addr learned my addr 172.21.15.125:0/3874827411 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55e00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 0x7f5e58159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e56800640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5815e3b0 0x7f5e5813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 -- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 msgr2=0x7f5e58159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 0x7f5e58159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 -- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5815e3b0 msgr2=0x7f5e5813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5815e3b0 0x7f5e5813e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 -- 172.21.15.125:0/3874827411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480094d0 con 0x7f5e58143ae0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55e00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 0x7f5e58159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.241+0000 7f5e55400640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000c410 tx=0x7f5e4000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.242+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400035f0 con 0x7f5e58143ae0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.242+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40003750 con 0x7f5e58143ae0 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.242+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40003b40 con 0x7f5e58143ae0 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.242+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5813d2a0 con 0x7f5e58143ae0 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.242+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5813c9d0 con 0x7f5e58143ae0 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.243+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e580675d0 con 0x7f5e58143ae0 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.244+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5e40003ca0 con 0x7f5e58143ae0 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.245+0000 7f5e4e000640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 0x7f5e3006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.245+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5e400878a0 con 0x7f5e58143ae0 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.245+0000 7f5e55e00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 0x7f5e3006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.245+0000 7f5e55e00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 0x7f5e3006dff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009fa0 tx=0x7f5e50009570 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.249+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e40053ff0 con 0x7f5e58143ae0 2024-01-16T15:04:49.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.425+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e5806ad10 con 0x7f5e58143ae0 2024-01-16T15:04:49.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.426+0000 7f5e4e000640 1 -- 172.21.15.125:0/3874827411 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e400170a0 con 0x7f5e58143ae0 2024-01-16T15:04:49.406 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:04:49.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 msgr2=0x7f5e3006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 0x7f5e3006dff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009fa0 tx=0x7f5e50009570 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 msgr2=0x7f5e5813c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000c410 tx=0x7f5e4000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 shutdown_connections 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5815e3b0 0x7f5e5813e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e3006bb30 0x7f5e3006dff0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e58143ae0 0x7f5e5813c490 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 --2- 172.21.15.125:0/3874827411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e580cf7d0 0x7f5e58159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 >> 172.21.15.125:0/3874827411 conn(0x7f5e5806a800 msgr2=0x7f5e5806cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 shutdown_connections 2024-01-16T15:04:49.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.429+0000 7f5e5ce00640 1 -- 172.21.15.125:0/3874827411 wait complete. 2024-01-16T15:04:49.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:49.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.717+0000 7f5806400640 1 Processor -- start 2024-01-16T15:04:49.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.717+0000 7f5806400640 1 -- start start 2024-01-16T15:04:49.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.719+0000 7f5806400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5806400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5800159750 0x7f58000cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5806400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 0x7f58000b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5805000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5806400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5800070160 con 0x7f58000b15f0 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5806400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580015a010 con 0x7f5800159750 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5806400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580015a2f0 con 0x7f58001536f0 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5805000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:44628/0 (socket says 172.21.15.125:44628) 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5805000640 1 -- 172.21.15.125:0/3756019609 learned_addr learned my addr 172.21.15.125:0/3756019609 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f5805a00640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 0x7f58000b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.720+0000 7f57ffe00640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.137:3300/0 conn(0x7f5800159750 0x7f58000cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805000640 1 -- 172.21.15.125:0/3756019609 >> v2:172.21.15.137:3300/0 conn(0x7f5800159750 msgr2=0x7f58000cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805000640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.137:3300/0 conn(0x7f5800159750 0x7f58000cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805000640 1 -- 172.21.15.125:0/3756019609 >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 msgr2=0x7f58000b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805000640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 0x7f58000b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805000640 1 -- 172.21.15.125:0/3756019609 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580015a5d0 con 0x7f58001536f0 2024-01-16T15:04:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.721+0000 7f5805a00640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 0x7f58000b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:49.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.723+0000 7f5805000640 1 --2- 172.21.15.125:0/3756019609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f58000b1ba0 tx=0x7f57f80093b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd88265834fdd2cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.723+0000 7f57ff400640 1 -- 172.21.15.125:0/3756019609 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f802c880 con 0x7f58001536f0 2024-01-16T15:04:49.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.723+0000 7f57ff400640 1 -- 172.21.15.125:0/3756019609 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f802c9e0 con 0x7f58001536f0 2024-01-16T15:04:49.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.723+0000 7f57ff400640 1 -- 172.21.15.125:0/3756019609 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f802cd90 con 0x7f58001536f0 2024-01-16T15:04:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 -- 172.21.15.125:0/3756019609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 msgr2=0x7f5800072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 --2- 172.21.15.125:0/3756019609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f58000b1ba0 tx=0x7f57f80093b0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 -- 172.21.15.125:0/3756019609 shutdown_connections 2024-01-16T15:04:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.125:3300/0 conn(0x7f58000b15f0 0x7f58000b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 --2- 172.21.15.125:0/3756019609 >> v2:172.21.15.137:3300/0 conn(0x7f5800159750 0x7f58000cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 --2- 172.21.15.125:0/3756019609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f5800072290 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 -- 172.21.15.125:0/3756019609 >> 172.21.15.125:0/3756019609 conn(0x7f58000b0b00 msgr2=0x7f58000b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 -- 172.21.15.125:0/3756019609 shutdown_connections 2024-01-16T15:04:49.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.725+0000 7f5806400640 1 -- 172.21.15.125:0/3756019609 wait complete. 2024-01-16T15:04:49.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.727+0000 7f5806400640 1 Processor -- start 2024-01-16T15:04:49.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.728+0000 7f5806400640 1 -- start start 2024-01-16T15:04:49.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f58000ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5800159750 0x7f58001525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58001556b0 con 0x7f58000b15f0 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58001f2f30 con 0x7f5800159750 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5806400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58001403e0 con 0x7f58001536f0 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.729+0000 7f5805000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48844/0 (socket says 172.21.15.125:48844) 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 -- 172.21.15.125:0/3307276160 learned_addr learned my addr 172.21.15.125:0/3307276160 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805a00640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5800159750 0x7f58001525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f57ffe00640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f58000ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 -- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 msgr2=0x7f58000ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f58000ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 -- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5800159750 msgr2=0x7f58001525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5800159750 0x7f58001525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.730+0000 7f5805000640 1 -- 172.21.15.125:0/3307276160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f000d8a0 con 0x7f58000b15f0 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.732+0000 7f5805000640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f57f8036040 tx=0x7f57f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.732+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800b720 con 0x7f58000b15f0 2024-01-16T15:04:49.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.732+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f8004040 con 0x7f58000b15f0 2024-01-16T15:04:49.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.732+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f80043f0 con 0x7f58000b15f0 2024-01-16T15:04:49.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.732+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f800bb50 con 0x7f58000b15f0 2024-01-16T15:04:49.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.733+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5800154a30 con 0x7f58000b15f0 2024-01-16T15:04:49.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.735+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5800067450 con 0x7f58000b15f0 2024-01-16T15:04:49.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.736+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f57f8004550 con 0x7f58000b15f0 2024-01-16T15:04:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.737+0000 7f57fcc00640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 0x7f57d806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.737+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f57f80aaca0 con 0x7f58000b15f0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.740+0000 7f57ffe00640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 0x7f57d806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.741+0000 7f57ffe00640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 0x7f57d806dff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f58000cec90 tx=0x7f57e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.746+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57f80773f0 con 0x7f58000b15f0 2024-01-16T15:04:49.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.978+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5800159170 con 0x7f58000b15f0 2024-01-16T15:04:49.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.979+0000 7f57fcc00640 1 -- 172.21.15.125:0/3307276160 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v61) v1 ==== 80+0+124 (secure 0 0 0) 0x7f57f807aa10 con 0x7f58000b15f0 2024-01-16T15:04:49.959 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/2216643169 2024-01-17T15:04:45.210297+0000 2024-01-16T15:04:49.959 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2216643169 2024-01-17T15:04:45.210297+0000 2024-01-16T15:04:49.959 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 msgr2=0x7f57d806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 0x7f57d806dff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f58000cec90 tx=0x7f57e8008040 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 msgr2=0x7f58001f29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f57f8036040 tx=0x7f57f8004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 shutdown_connections 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5800159750 0x7f58001525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58001536f0 0x7f58000ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57d806bb30 0x7f57d806dff0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 --2- 172.21.15.125:0/3307276160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58000b15f0 0x7f58001f29f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 >> 172.21.15.125:0/3307276160 conn(0x7f58000b0b00 msgr2=0x7f580015c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 shutdown_connections 2024-01-16T15:04:49.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:49.982+0000 7f5806400640 1 -- 172.21.15.125:0/3307276160 wait complete. 2024-01-16T15:04:49.978 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:04:49.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:49.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:50.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[58036]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:50.087 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:04:50.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:50.087 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:04:50.335 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:04:50.335 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:50.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.579+0000 7f4d99200640 1 Processor -- start 2024-01-16T15:04:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.580+0000 7f4d99200640 1 -- start start 2024-01-16T15:04:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d99200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 0x7f4d940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d99200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d940cfc30 con 0x7f4d940cf7d0 2024-01-16T15:04:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d99200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d940b1ee0 con 0x7f4d9415e3b0 2024-01-16T15:04:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d99200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940b2780 con 0x7f4d94143ae0 2024-01-16T15:04:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d91400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4d940cf7d0 0x7f4d9415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48686/0 (socket says 172.21.15.125:48686) 2024-01-16T15:04:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.581+0000 7f4d91400640 1 -- 172.21.15.125:0/3134735183 learned_addr learned my addr 172.21.15.125:0/3134735183 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 0x7f4d940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155097145 0 0) 0x7f4d940cfc30 con 0x7f4d940cf7d0 2024-01-16T15:04:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74005730 con 0x7f4d940cf7d0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242449358 0 0) 0x7f4d74005730 con 0x7f4d940cf7d0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 msgr2=0x7f4d940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 0x7f4d940cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 >> v1:172.21.15.137:6789/0 conn(0x7f4d9415e3b0 legacy=0x7f4d940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9415e990 con 0x7f4d940cf7d0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f4d7c001670 con 0x7f4d940cf7d0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 >> v1:172.21.15.125:6789/0 conn(0x7f4d940cf7d0 legacy=0x7f4d9415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d74005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d740069d0 0x7f4d74006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 0x7f4d740073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940b2780 con 0x7f4d74005bb0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7400dcc0 con 0x7f4d740069d0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.582+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7400de80 con 0x7f4d94143ae0 2024-01-16T15:04:50.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d74005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 -- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 msgr2=0x7f4d740073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d94143ae0 0x7f4d740073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 -- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d740069d0 msgr2=0x7f4d74006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d740069d0 0x7f4d74006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.583+0000 7f4d91e00640 1 -- 172.21.15.125:0/3134735183 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d74005730 con 0x7f4d74005bb0 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.584+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d74005fd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400ba30 tx=0x7f4d74007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa17399f8d360244 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.584+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84032680 con 0x7f4d74005bb0 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.584+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d840327e0 con 0x7f4d74005bb0 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.584+0000 7f4d90a00640 1 -- 172.21.15.125:0/3134735183 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84032b70 con 0x7f4d74005bb0 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 -- 172.21.15.125:0/3134735183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 msgr2=0x7f4d74005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 --2- 172.21.15.125:0/3134735183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d74005fd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400ba30 tx=0x7f4d74007a00 comp rx=0 tx=0).stop 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d92800640 1 -- 172.21.15.125:0/3134735183 reap_dead start 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 -- 172.21.15.125:0/3134735183 shutdown_connections 2024-01-16T15:04:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 -- 172.21.15.125:0/3134735183 >> 172.21.15.125:0/3134735183 conn(0x7f4d9406a800 msgr2=0x7f4d9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 -- 172.21.15.125:0/3134735183 shutdown_connections 2024-01-16T15:04:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.585+0000 7f4d99200640 1 -- 172.21.15.125:0/3134735183 wait complete. 2024-01-16T15:04:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.586+0000 7f4d99200640 1 Processor -- start 2024-01-16T15:04:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 -- start start 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d940aff80 0x7f4d941ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d940ad920 0x7f4d940adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940ae2e0 con 0x7f4d74005bb0 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940b0750 con 0x7f4d940aff80 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.587+0000 7f4d99200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940b0a30 con 0x7f4d940ad920 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48862/0 (socket says 172.21.15.125:48862) 2024-01-16T15:04:50.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 -- 172.21.15.125:0/3559634957 learned_addr learned my addr 172.21.15.125:0/3559634957 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 -- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d940ad920 msgr2=0x7f4d940adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d940ad920 0x7f4d940adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 -- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d940aff80 msgr2=0x7f4d941ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d940aff80 0x7f4d941ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 -- 172.21.15.125:0/3559634957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d940b0d10 con 0x7f4d74005bb0 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.588+0000 7f4d91e00640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d8402bf00 tx=0x7f4d84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.589+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84004220 con 0x7f4d74005bb0 2024-01-16T15:04:50.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.589+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d84004380 con 0x7f4d74005bb0 2024-01-16T15:04:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.589+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84030ac0 con 0x7f4d74005bb0 2024-01-16T15:04:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.589+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d940b0ff0 con 0x7f4d74005bb0 2024-01-16T15:04:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.589+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d941ed750 con 0x7f4d74005bb0 2024-01-16T15:04:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.590+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d940675d0 con 0x7f4d74005bb0 2024-01-16T15:04:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.591+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4d84030c20 con 0x7f4d74005bb0 2024-01-16T15:04:50.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.592+0000 7f4d8a000640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 0x7f4d6c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.592+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4d840aaf80 con 0x7f4d74005bb0 2024-01-16T15:04:50.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.592+0000 7f4d91400640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 0x7f4d6c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.593+0000 7f4d91400640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 0x7f4d6c06dff0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d94142530 tx=0x7f4d7c001430 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.596+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d840776d0 con 0x7f4d74005bb0 2024-01-16T15:04:50.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.772+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d940675d0 con 0x7f4d74005bb0 2024-01-16T15:04:50.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.774+0000 7f4d8a000640 1 -- 172.21.15.125:0/3559634957 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f4d84006c30 con 0x7f4d74005bb0 2024-01-16T15:04:50.754 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:50.754 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:04:48.223107+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6835/2216643169":"2024-01-17T15:04:45.210297+0000","172.21.15.125:6834/2216643169":"2024-01-17T15:04:45.210297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 msgr2=0x7f4d6c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 0x7f4d6c06dff0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d94142530 tx=0x7f4d7c001430 comp rx=0 tx=0).stop 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 msgr2=0x7f4d941f3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d8402bf00 tx=0x7f4d84004730 comp rx=0 tx=0).stop 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 shutdown_connections 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d940ad920 0x7f4d940adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d940aff80 0x7f4d941ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d6c06bb30 0x7f4d6c06dff0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 --2- 172.21.15.125:0/3559634957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d74005bb0 0x7f4d941f3e90 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 >> 172.21.15.125:0/3559634957 conn(0x7f4d9406a800 msgr2=0x7f4d940aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 shutdown_connections 2024-01-16T15:04:50.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.777+0000 7f4d99200640 1 -- 172.21.15.125:0/3559634957 wait complete. 2024-01-16T15:04:50.775 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:50.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.992+0000 7f034f200640 1 Processor -- start 2024-01-16T15:04:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.992+0000 7f034f200640 1 -- start start 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f034f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f034813df80 0x7f034815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f034f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03480cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f034f200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034806c470 con 0x7f034813e360 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f034f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034806c750 con 0x7f034813df80 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f034f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034806ca30 con 0x7f03480cf9a0 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f033fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f034813e360 0x7f03480cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48692/0 (socket says 172.21.15.125:48692) 2024-01-16T15:04:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.993+0000 7f033fe00640 1 -- 172.21.15.125:0/3525593631 learned_addr learned my addr 172.21.15.125:0/3525593631 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338497271 0 0) 0x7f034806c470 con 0x7f034813e360 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0328003520 con 0x7f034813e360 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f034d200640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03480cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347e00640 1 --2- 172.21.15.125:0/3525593631 >> v2:172.21.15.137:3300/0 conn(0x7f034813df80 0x7f034815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4184963042 0 0) 0x7f0328003520 con 0x7f034813e360 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 msgr2=0x7f03480cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03480cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 >> v2:172.21.15.137:3300/0 conn(0x7f034813df80 msgr2=0x7f034815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 --2- 172.21.15.125:0/3525593631 >> v2:172.21.15.137:3300/0 conn(0x7f034813df80 0x7f034815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034806cd10 con 0x7f034813e360 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0330001670 con 0x7f034813e360 2024-01-16T15:04:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 >> v1:172.21.15.125:6789/0 conn(0x7f034813e360 legacy=0x7f03480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0328003970 0x7f0328003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328004d00 0x7f0328005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034806c750 con 0x7f0328003970 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034806ca30 con 0x7f0328004d00 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.994+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032800ba60 con 0x7f03480cf9a0 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0328003970 0x7f0328003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f033fe00640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 -- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 msgr2=0x7f03280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03480cf9a0 0x7f03280047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 -- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328004d00 msgr2=0x7f0328005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328004d00 0x7f0328005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.995+0000 7f0347e00640 1 -- 172.21.15.125:0/3525593631 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328003520 con 0x7f0328003970 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.996+0000 7f0347e00640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0328003970 0x7f0328003d90 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f033800ba30 tx=0x7f03280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af97752742bdd539 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.996+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338032660 con 0x7f0328003970 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.996+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03380327c0 con 0x7f0328003970 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.996+0000 7f0347400640 1 -- 172.21.15.125:0/3525593631 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338032b50 con 0x7f0328003970 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 -- 172.21.15.125:0/3525593631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0328003970 msgr2=0x7f0328003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 --2- 172.21.15.125:0/3525593631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0328003970 0x7f0328003d90 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f033800ba30 tx=0x7f03280057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034d200640 1 -- 172.21.15.125:0/3525593631 reap_dead start 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 -- 172.21.15.125:0/3525593631 shutdown_connections 2024-01-16T15:04:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 -- 172.21.15.125:0/3525593631 >> 172.21.15.125:0/3525593631 conn(0x7f034806a800 msgr2=0x7f034806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 -- 172.21.15.125:0/3525593631 shutdown_connections 2024-01-16T15:04:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.997+0000 7f034f200640 1 -- 172.21.15.125:0/3525593631 wait complete. 2024-01-16T15:04:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.998+0000 7f034f200640 1 Processor -- start 2024-01-16T15:04:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 -- start start 2024-01-16T15:04:50.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328003970 0x7f03480b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0348131f70 0x7f0348153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034812f570 con 0x7f034815a180 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034815ab40 con 0x7f0328003970 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:50.999+0000 7f034f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03480b32d0 con 0x7f0348131f70 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48870/0 (socket says 172.21.15.125:48870) 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 -- 172.21.15.125:0/355127470 learned_addr learned my addr 172.21.15.125:0/355127470 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 -- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0348131f70 msgr2=0x7f0348153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0348131f70 0x7f0348153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 -- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328003970 msgr2=0x7f03480b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328003970 0x7f03480b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 -- 172.21.15.125:0/355127470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03480b35b0 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.000+0000 7f034d200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f03480cfea0 tx=0x7f034000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.001+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340018070 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.001+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034000eb90 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.001+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340003990 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.001+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03480b3890 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.001+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0348153640 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.002+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034812f6b0 con 0x7f034815a180 2024-01-16T15:04:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.003+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f0340014420 con 0x7f034815a180 2024-01-16T15:04:50.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.004+0000 7f0344c00640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 0x7f032006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.004+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0340088470 con 0x7f034815a180 2024-01-16T15:04:50.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.004+0000 7f0347e00640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 0x7f032006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.005+0000 7f0347e00640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 0x7f032006dff0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f033800ba00 tx=0x7f0338032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.008+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f034812f6b0 con 0x7f034815a180 2024-01-16T15:04:51.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.186+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f034812f6b0 con 0x7f034815a180 2024-01-16T15:04:51.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.187+0000 7f0344c00640 1 -- 172.21.15.125:0/355127470 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f034812f6b0 con 0x7f034815a180 2024-01-16T15:04:51.166 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:04:51.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.189+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 msgr2=0x7f032006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.189+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 0x7f032006dff0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f033800ba00 tx=0x7f0338032000 comp rx=0 tx=0).stop 2024-01-16T15:04:51.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.189+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 msgr2=0x7f034815a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.189+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f03480cfea0 tx=0x7f034000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 shutdown_connections 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f032006bb30 0x7f032006dff0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f034815a180 0x7f034815a600 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0348131f70 0x7f0348153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 --2- 172.21.15.125:0/355127470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328003970 0x7f03480b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 >> 172.21.15.125:0/355127470 conn(0x7f034806a800 msgr2=0x7f03480b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 shutdown_connections 2024-01-16T15:04:51.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.190+0000 7f034f200640 1 -- 172.21.15.125:0/355127470 wait complete. 2024-01-16T15:04:51.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.409+0000 7f8875200640 1 Processor -- start 2024-01-16T15:04:51.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.409+0000 7f8875200640 1 -- start start 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f8875200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870071200 0x7f887006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f8875200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887015deb0 con 0x7f8870157c30 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f8875200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887015e190 con 0x7f88700cd600 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f8875200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887015ea30 con 0x7f8870071200 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f886de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8870157c30 0x7f88701618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48708/0 (socket says 172.21.15.125:48708) 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.411+0000 7f886de00640 1 -- 172.21.15.125:0/737015261 learned_addr learned my addr 172.21.15.125:0/737015261 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886e800640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870071200 0x7f887006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26579316 0 0) 0x7f887015deb0 con 0x7f8870157c30 2024-01-16T15:04:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8850003520 con 0x7f8870157c30 2024-01-16T15:04:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3459253403 0 0) 0x7f8850003520 con 0x7f8870157c30 2024-01-16T15:04:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870071200 msgr2=0x7f887006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870071200 0x7f887006dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 >> v1:172.21.15.137:6789/0 conn(0x7f88700cd600 legacy=0x7f8870070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887015e470 con 0x7f8870157c30 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8860001690 con 0x7f8870157c30 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 >> v1:172.21.15.125:6789/0 conn(0x7f8870157c30 legacy=0x7f88701618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88500039a0 0x7f8850003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500047c0 0x7f8850004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8850005180 0x7f885000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887015ea30 con 0x7f88500039a0 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885000c050 con 0x7f8850005180 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885000c210 con 0x7f88500047c0 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886e800640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870071200 0x7f887006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.412+0000 7f886de00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88500039a0 0x7f8850003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886d400640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500047c0 0x7f8850004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886de00640 1 -- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500047c0 msgr2=0x7f8850004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886de00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500047c0 0x7f8850004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886de00640 1 -- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8850005180 msgr2=0x7f885000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886de00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8850005180 0x7f885000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886de00640 1 -- 172.21.15.125:0/737015261 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8850003520 con 0x7f88500039a0 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.413+0000 7f886d400640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500047c0 0x7f8850004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.414+0000 7f886de00640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88500039a0 0x7f8850003dc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f886000c210 tx=0x7f88500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bb627aa390d367 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.414+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860005ad0 con 0x7f88500039a0 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.414+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8860005c30 con 0x7f88500039a0 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.414+0000 7f886ca00640 1 -- 172.21.15.125:0/737015261 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886002c870 con 0x7f88500039a0 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 -- 172.21.15.125:0/737015261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88500039a0 msgr2=0x7f8850003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 --2- 172.21.15.125:0/737015261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88500039a0 0x7f8850003dc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f886000c210 tx=0x7f88500057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f886e800640 1 -- 172.21.15.125:0/737015261 reap_dead start 2024-01-16T15:04:51.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 -- 172.21.15.125:0/737015261 shutdown_connections 2024-01-16T15:04:51.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 -- 172.21.15.125:0/737015261 >> 172.21.15.125:0/737015261 conn(0x7f88700cf700 msgr2=0x7f88700cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 -- 172.21.15.125:0/737015261 shutdown_connections 2024-01-16T15:04:51.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.415+0000 7f8875200640 1 -- 172.21.15.125:0/737015261 wait complete. 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.417+0000 7f8875200640 1 Processor -- start 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.417+0000 7f8875200640 1 -- start start 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500039a0 0x7f8870146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887015f460 0x7f887015f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700ace20 con 0x7f88701502d0 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887015fe20 con 0x7f887015f460 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f8875200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8870152550 con 0x7f88500039a0 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48890/0 (socket says 172.21.15.125:48890) 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 -- 172.21.15.125:0/2565187015 learned_addr learned my addr 172.21.15.125:0/2565187015 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 -- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500039a0 msgr2=0x7f8870146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500039a0 0x7f8870146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 -- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887015f460 msgr2=0x7f887015f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887015f460 0x7f887015f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.418+0000 7f886d400640 1 -- 172.21.15.125:0/2565187015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88701465a0 con 0x7f88701502d0 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f886d400640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f885800d7d0 tx=0x7f885800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858017070 con 0x7f88701502d0 2024-01-16T15:04:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858005070 con 0x7f88701502d0 2024-01-16T15:04:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858005440 con 0x7f88701502d0 2024-01-16T15:04:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700aff70 con 0x7f88701502d0 2024-01-16T15:04:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.419+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700b0530 con 0x7f88701502d0 2024-01-16T15:04:51.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.420+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88700675d0 con 0x7f88701502d0 2024-01-16T15:04:51.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.421+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f8858007030 con 0x7f88701502d0 2024-01-16T15:04:51.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.422+0000 7f8866000640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.422+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8858087df0 con 0x7f88701502d0 2024-01-16T15:04:51.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.422+0000 7f886de00640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.423+0000 7f886de00640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 0x7f884806dff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f886000c210 tx=0x7f8860007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.426+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88580544e0 con 0x7f88701502d0 2024-01-16T15:04:51.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.613+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8870071ae0 con 0x7f88701502d0 2024-01-16T15:04:51.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.614+0000 7f8866000640 1 -- 172.21.15.125:0/2565187015 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v61) v1 ==== 80+0+124 (secure 0 0 0) 0x7f8858057b00 con 0x7f88701502d0 2024-01-16T15:04:51.594 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/2216643169 2024-01-17T15:04:45.210297+0000 2024-01-16T15:04:51.594 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2216643169 2024-01-17T15:04:45.210297+0000 2024-01-16T15:04:51.594 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:04:51.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.616+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 msgr2=0x7f884806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.616+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 0x7f884806dff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f886000c210 tx=0x7f8860007000 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 msgr2=0x7f88700af700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f885800d7d0 tx=0x7f885800dc90 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 shutdown_connections 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887015f460 0x7f887015f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88701502d0 0x7f88700af700 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 --2- 172.21.15.125:0/2565187015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500039a0 0x7f8870146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 >> 172.21.15.125:0/2565187015 conn(0x7f88700cf700 msgr2=0x7f8870070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 shutdown_connections 2024-01-16T15:04:51.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:51.617+0000 7f8875200640 1 -- 172.21.15.125:0/2565187015 wait complete. 2024-01-16T15:04:51.615 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:04:51.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:51.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:51.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44351]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:51.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:04:51.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:51.713 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:04:51.890 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:04:51.890 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:51.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:51.945 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:51.946 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:04:51.993 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:04:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:51.995 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:04:51.995 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:52.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:52.051 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:04:52.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:52.052 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:04:52.100 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:04:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:52.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.340+0000 7f6d19200640 1 Processor -- start 2024-01-16T15:04:52.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.340+0000 7f6d19200640 1 -- start start 2024-01-16T15:04:52.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.341+0000 7f6d19200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d140b3ae0 0x7f6d1415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.341+0000 7f6d19200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 0x7f6d140cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d19200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d14070c40 con 0x7f6d140ae5a0 2024-01-16T15:04:52.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d19200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d14071450 con 0x7f6d140b3ae0 2024-01-16T15:04:52.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d19200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d140cfc60 con 0x7f6d140cf290 2024-01-16T15:04:52.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d09200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d140b3ae0 0x7f6d1415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d09200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d140b3ae0 0x7f6d1415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48904/0 (socket says 172.21.15.125:48904) 2024-01-16T15:04:52.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d09200640 1 -- 172.21.15.125:0/4247234482 learned_addr learned my addr 172.21.15.125:0/4247234482 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:52.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.342+0000 7f6d12800640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 0x7f6d140cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.343+0000 7f6d09200640 1 -- 172.21.15.125:0/4247234482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 msgr2=0x7f6d140cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.343+0000 7f6d09200640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 0x7f6d140cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.343+0000 7f6d09200640 1 -- 172.21.15.125:0/4247234482 >> v1:172.21.15.137:6789/0 conn(0x7f6d140ae5a0 legacy=0x7f6d1406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.343+0000 7f6d09200640 1 -- 172.21.15.125:0/4247234482 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1406c470 con 0x7f6d140b3ae0 2024-01-16T15:04:52.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.343+0000 7f6d12800640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 0x7f6d140cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:52.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.344+0000 7f6d09200640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140b3ae0 0x7f6d1415e430 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009870 tx=0x7f6cfc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4c91a2e8aa3766 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.344+0000 7f6d11400640 1 -- 172.21.15.125:0/4247234482 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc02f830 con 0x7f6d140b3ae0 2024-01-16T15:04:52.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.344+0000 7f6d11400640 1 -- 172.21.15.125:0/4247234482 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc02f990 con 0x7f6d140b3ae0 2024-01-16T15:04:52.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.344+0000 7f6d11400640 1 -- 172.21.15.125:0/4247234482 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc02fd20 con 0x7f6d140b3ae0 2024-01-16T15:04:52.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 -- 172.21.15.125:0/4247234482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140b3ae0 msgr2=0x7f6d1415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140b3ae0 0x7f6d1415e430 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009870 tx=0x7f6cfc02ba10 comp rx=0 tx=0).stop 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 -- 172.21.15.125:0/4247234482 shutdown_connections 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140cf290 0x7f6d140cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 --2- 172.21.15.125:0/4247234482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140b3ae0 0x7f6d1415e430 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 -- 172.21.15.125:0/4247234482 >> 172.21.15.125:0/4247234482 conn(0x7f6d1406a800 msgr2=0x7f6d1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 -- 172.21.15.125:0/4247234482 shutdown_connections 2024-01-16T15:04:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.345+0000 7f6d19200640 1 -- 172.21.15.125:0/4247234482 wait complete. 2024-01-16T15:04:52.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.346+0000 7f6d19200640 1 Processor -- start 2024-01-16T15:04:52.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.347+0000 7f6d19200640 1 -- start start 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.347+0000 7f6d19200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d19200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140b3ae0 0x7f6d1415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d19200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140cf290 0x7f6d14159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d19200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415f890 con 0x7f6d140ae5a0 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d19200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415fb70 con 0x7f6d140cf290 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d19200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415fe50 con 0x7f6d140b3ae0 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48906/0 (socket says 172.21.15.125:48906) 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 -- 172.21.15.125:0/401673423 learned_addr learned my addr 172.21.15.125:0/401673423 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 -- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140b3ae0 msgr2=0x7f6d1415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140b3ae0 0x7f6d1415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 -- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140cf290 msgr2=0x7f6d14159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140cf290 0x7f6d14159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.348+0000 7f6d11e00640 1 -- 172.21.15.125:0/401673423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc0094d0 con 0x7f6d140ae5a0 2024-01-16T15:04:52.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d11e00640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c870 tx=0x7f6d0400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003b70 con 0x7f6d140ae5a0 2024-01-16T15:04:52.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d04003cf0 con 0x7f6d140ae5a0 2024-01-16T15:04:52.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400aa90 con 0x7f6d140ae5a0 2024-01-16T15:04:52.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d14160130 con 0x7f6d140ae5a0 2024-01-16T15:04:52.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.349+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d14160930 con 0x7f6d140ae5a0 2024-01-16T15:04:52.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.350+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d14160650 con 0x7f6d140ae5a0 2024-01-16T15:04:52.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.351+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6d0400abf0 con 0x7f6d140ae5a0 2024-01-16T15:04:52.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.352+0000 7f6d0aa00640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 0x7f6cec06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.352+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6d04088510 con 0x7f6d140ae5a0 2024-01-16T15:04:52.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.352+0000 7f6d09200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 0x7f6cec06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.353+0000 7f6d09200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 0x7f6cec06dff0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc036000 tx=0x7f6cfc039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.356+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d04054c00 con 0x7f6d140ae5a0 2024-01-16T15:04:52.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.581+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d14160ac0 con 0x7f6d140ae5a0 2024-01-16T15:04:52.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.582+0000 7f6d0aa00640 1 -- 172.21.15.125:0/401673423 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6d04003e60 con 0x7f6d140ae5a0 2024-01-16T15:04:52.562 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:52.562 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2646664931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2646664931},{"type":"v1","addr":"172.21.15.125:6837","nonce":2646664931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5054,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1721910358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1721910358},{"type":"v1","addr":"172.21.15.137:6835","nonce":1721910358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5362,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/28677566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":28677566},{"type":"v1","addr":"172.21.15.137:6833","nonce":28677566}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":6996,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1998350661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1998350661},{"type":"v1","addr":"172.21.15.125:6835","nonce":1998350661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-16T15:04:52.562 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 40 2024-01-16T15:04:52.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.584+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 msgr2=0x7f6cec06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.584+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 0x7f6cec06dff0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc036000 tx=0x7f6cfc039040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 msgr2=0x7f6d14148e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c870 tx=0x7f6d0400cd30 comp rx=0 tx=0).stop 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 shutdown_connections 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140cf290 0x7f6d14159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d140b3ae0 0x7f6d1415ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cec06bb30 0x7f6cec06dff0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 --2- 172.21.15.125:0/401673423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d140ae5a0 0x7f6d14148e60 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 >> 172.21.15.125:0/401673423 conn(0x7f6d1406a800 msgr2=0x7f6d1415d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 shutdown_connections 2024-01-16T15:04:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.585+0000 7f6d19200640 1 -- 172.21.15.125:0/401673423 wait complete. 2024-01-16T15:04:52.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:04:52.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.804+0000 7f36dc600640 1 Processor -- start 2024-01-16T15:04:52.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.804+0000 7f36dc600640 1 -- start start 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.805+0000 7f36dc600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f36d414b2c0 0x7f36d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.805+0000 7f36dc600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f36d4071070 0x7f36d40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.805+0000 7f36dc600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40cfbf0 con 0x7f36d40cf7d0 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.805+0000 7f36dc600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d414b6a0 con 0x7f36d414b2c0 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.805+0000 7f36dc600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d4071690 con 0x7f36d4071070 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f36d40cf7d0 0x7f36d4070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48718/0 (socket says 172.21.15.125:48718) 2024-01-16T15:04:52.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f36d414b2c0 0x7f36d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9200640 1 -- 172.21.15.125:0/1087326359 learned_addr learned my addr 172.21.15.125:0/1087326359 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:52.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36da600640 1 --2- 172.21.15.125:0/1087326359 >> v2:172.21.15.137:3300/0 conn(0x7f36d4071070 0x7f36d40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9c00640 1 -- 172.21.15.125:0/1087326359 >> v2:172.21.15.137:3300/0 conn(0x7f36d4071070 msgr2=0x7f36d40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d3e00640 1 -- 172.21.15.125:0/1087326359 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528885167 0 0) 0x7f36d40cfbf0 con 0x7f36d40cf7d0 2024-01-16T15:04:52.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9c00640 1 --2- 172.21.15.125:0/1087326359 >> v2:172.21.15.137:3300/0 conn(0x7f36d4071070 0x7f36d40cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9c00640 1 -- 172.21.15.125:0/1087326359 >> v1:172.21.15.125:6789/0 conn(0x7f36d40cf7d0 legacy=0x7f36d4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.806+0000 7f36d9c00640 1 -- 172.21.15.125:0/1087326359 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4071970 con 0x7f36d414b2c0 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.807+0000 7f36d9c00640 1 --2- 172.21.15.125:0/1087326359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40cf290 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f36d4165330 tx=0x7f36c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ea0df1b0cbe998 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36d3e00640 1 -- 172.21.15.125:0/1087326359 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4031830 con 0x7f36d414b2c0 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36d3e00640 1 -- 172.21.15.125:0/1087326359 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c4031990 con 0x7f36d414b2c0 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36d3e00640 1 -- 172.21.15.125:0/1087326359 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4031d20 con 0x7f36d414b2c0 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 -- 172.21.15.125:0/1087326359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 msgr2=0x7f36d40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 --2- 172.21.15.125:0/1087326359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40cf290 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f36d4165330 tx=0x7f36c402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 -- 172.21.15.125:0/1087326359 shutdown_connections 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 --2- 172.21.15.125:0/1087326359 >> v2:172.21.15.137:3300/0 conn(0x7f36d4071070 0x7f36d40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 --2- 172.21.15.125:0/1087326359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40cf290 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.808+0000 7f36dc600640 1 -- 172.21.15.125:0/1087326359 >> 172.21.15.125:0/1087326359 conn(0x7f36d406a800 msgr2=0x7f36d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.809+0000 7f36dc600640 1 -- 172.21.15.125:0/1087326359 shutdown_connections 2024-01-16T15:04:52.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.809+0000 7f36dc600640 1 -- 172.21.15.125:0/1087326359 wait complete. 2024-01-16T15:04:52.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.810+0000 7f36dc600640 1 Processor -- start 2024-01-16T15:04:52.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.810+0000 7f36dc600640 1 -- start start 2024-01-16T15:04:52.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36d4071070 0x7f36d41fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36d40cf7d0 0x7f36d40c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4144660 con 0x7f36d414b2c0 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4147680 con 0x7f36d40cf7d0 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.811+0000 7f36dc600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4144030 con 0x7f36d4071070 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48930/0 (socket says 172.21.15.125:48930) 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 -- 172.21.15.125:0/2128416616 learned_addr learned my addr 172.21.15.125:0/2128416616 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36d9c00640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36d4071070 0x7f36d41fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 -- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36d4071070 msgr2=0x7f36d41fa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36d4071070 0x7f36d41fa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 -- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36d40cf7d0 msgr2=0x7f36d40c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36d40cf7d0 0x7f36d40c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 -- 172.21.15.125:0/2128416616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4009400 con 0x7f36d414b2c0 2024-01-16T15:04:52.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.812+0000 7f36da600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f36cc007e40 tx=0x7f36cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.813+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc014620 con 0x7f36d414b2c0 2024-01-16T15:04:52.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.813+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00ee90 con 0x7f36d414b2c0 2024-01-16T15:04:52.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.813+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0149e0 con 0x7f36d414b2c0 2024-01-16T15:04:52.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.813+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d41fb060 con 0x7f36d414b2c0 2024-01-16T15:04:52.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.813+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d4156530 con 0x7f36d414b2c0 2024-01-16T15:04:52.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.814+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d40675d0 con 0x7f36d414b2c0 2024-01-16T15:04:52.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.815+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f36cc014b40 con 0x7f36d414b2c0 2024-01-16T15:04:52.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.816+0000 7f36d1600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 0x7f36ac06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.816+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f36cc088fc0 con 0x7f36d414b2c0 2024-01-16T15:04:52.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.816+0000 7f36d9c00640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 0x7f36ac06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.817+0000 7f36d9c00640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 0x7f36ac06dff0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f36c4036000 tx=0x7f36c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:52.820+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36cc0556b0 con 0x7f36d414b2c0 2024-01-16T15:04:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.011+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f36d40b3a20 con 0x7f36d414b2c0 2024-01-16T15:04:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.018+0000 7f36d1600640 1 -- 172.21.15.125:0/2128416616 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f36cc058cd0 con 0x7f36d414b2c0 2024-01-16T15:04:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.020+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 msgr2=0x7f36ac06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.020+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 0x7f36ac06dff0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f36c4036000 tx=0x7f36c4008030 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.020+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 msgr2=0x7f36d40ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.020+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f36cc007e40 tx=0x7f36cc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 shutdown_connections 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ac06bb30 0x7f36ac06dff0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f36d414b2c0 0x7f36d40ca290 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36d40cf7d0 0x7f36d40c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 --2- 172.21.15.125:0/2128416616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f36d4071070 0x7f36d41fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 >> 172.21.15.125:0/2128416616 conn(0x7f36d406a800 msgr2=0x7f36d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 shutdown_connections 2024-01-16T15:04:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.021+0000 7f36dc600640 1 -- 172.21.15.125:0/2128416616 wait complete. 2024-01-16T15:04:53.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:04:53.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.250+0000 7f19e4000640 1 Processor -- start 2024-01-16T15:04:53.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.250+0000 7f19e4000640 1 -- start start 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f19dc0cf9a0 0x7f19dc166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e4000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc15c5f0 con 0x7f19dc13e360 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc06c470 con 0x7f19dc13df80 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc06c750 con 0x7f19dc0cf9a0 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e0c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f19dc13e360 0x7f19dc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48730/0 (socket says 172.21.15.125:48730) 2024-01-16T15:04:53.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e0c00640 1 -- 172.21.15.125:0/823192663 learned_addr learned my addr 172.21.15.125:0/823192663 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435450424 0 0) 0x7f19dc15c5f0 con 0x7f19dc13e360 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19e2000640 1 --2- 172.21.15.125:0/823192663 >> v2:172.21.15.137:3300/0 conn(0x7f19dc0cf9a0 0x7f19dc166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.252+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc003520 con 0x7f19dc13e360 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183950674 0 0) 0x7f19bc003520 con 0x7f19dc13e360 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 >> v2:172.21.15.137:3300/0 conn(0x7f19dc0cf9a0 msgr2=0x7f19dc166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 --2- 172.21.15.125:0/823192663 >> v2:172.21.15.137:3300/0 conn(0x7f19dc0cf9a0 0x7f19dc166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 >> v1:172.21.15.137:6789/0 conn(0x7f19dc13df80 legacy=0x7f19dc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc06ca30 con 0x7f19dc13e360 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f19c4001670 con 0x7f19dc13e360 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 >> v1:172.21.15.125:6789/0 conn(0x7f19dc13e360 legacy=0x7f19dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19bc003970 0x7f19bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc0047c0 0x7f19bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 0x7f19bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc06c750 con 0x7f19bc005180 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc00c020 con 0x7f19bc0047c0 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc00c1e0 con 0x7f19bc003970 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.253+0000 7f19e2000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 0x7f19bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e1600640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19bc003970 0x7f19bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e0c00640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc0047c0 0x7f19bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e2000640 1 -- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19bc003970 msgr2=0x7f19bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e2000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19bc003970 0x7f19bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e2000640 1 -- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc0047c0 msgr2=0x7f19bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e2000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc0047c0 0x7f19bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e2000640 1 -- 172.21.15.125:0/823192663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc003520 con 0x7f19bc005180 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.254+0000 7f19e0c00640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc0047c0 0x7f19bc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.255+0000 7f19e2000640 1 -- 172.21.15.125:0/823192663 reap_dead start 2024-01-16T15:04:53.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.255+0000 7f19e2000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 0x7f19bc00b980 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009f90 tx=0x7f19bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a641f9ef51fa43d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.255+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d402cc20 con 0x7f19bc005180 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.255+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d400f040 con 0x7f19bc005180 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.255+0000 7f19dbe00640 1 -- 172.21.15.125:0/823192663 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4003990 con 0x7f19bc005180 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 -- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 msgr2=0x7f19bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 0x7f19bc00b980 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009f90 tx=0x7f19bc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 -- 172.21.15.125:0/823192663 shutdown_connections 2024-01-16T15:04:53.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 --2- 172.21.15.125:0/823192663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19bc005180 0x7f19bc00b980 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 -- 172.21.15.125:0/823192663 >> 172.21.15.125:0/823192663 conn(0x7f19dc06a800 msgr2=0x7f19dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 -- 172.21.15.125:0/823192663 shutdown_connections 2024-01-16T15:04:53.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.256+0000 7f19e4000640 1 -- 172.21.15.125:0/823192663 wait complete. 2024-01-16T15:04:53.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.258+0000 7f19e4000640 1 Processor -- start 2024-01-16T15:04:53.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.258+0000 7f19e4000640 1 -- start start 2024-01-16T15:04:53.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 0x7f19dc15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19dc0a9580 0x7f19dc13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc070d20 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15d640 con 0x7f19bc005180 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0703f0 con 0x7f19dc0a9580 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48934/0 (socket says 172.21.15.125:48934) 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.259+0000 7f19e2000640 1 -- 172.21.15.125:0/1715086069 learned_addr learned my addr 172.21.15.125:0/1715086069 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e1600640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 0x7f19dc15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e0c00640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19dc0a9580 0x7f19dc13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 -- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19dc0a9580 msgr2=0x7f19dc13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19dc0a9580 0x7f19dc13d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 -- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 msgr2=0x7f19dc15cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 0x7f19dc15cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 -- 172.21.15.125:0/1715086069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0706d0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e1600640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 0x7f19dc15cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:53.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.260+0000 7f19e2000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f19d402c690 tx=0x7f19d4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.262+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d402cea0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.262+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d4004230 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.262+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0709b0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.262+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d40045e0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.262+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc13dcf0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.264+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc070e90 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.264+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f19d4004740 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.266+0000 7f19d9600640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 0x7f19b406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.266+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f19d40aad70 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.266+0000 7f19e1600640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 0x7f19b406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.266+0000 7f19e1600640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 0x7f19b406e040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000c00 tx=0x7f19cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.270+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19d40775c0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.463+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f19dc0675d0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.471+0000 7f19d9600640 1 -- 172.21.15.125:0/1715086069 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v24) v1 ==== 120+0+0 (secure 0 0 0) 0x7f19d407abe0 con 0x7f19dc0ac4e0 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 msgr2=0x7f19b406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 0x7f19b406e040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000c00 tx=0x7f19cc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 msgr2=0x7f19dc0aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f19d402c690 tx=0x7f19d4009750 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 shutdown_connections 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19b406bb80 0x7f19b406e040 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19dc0ac4e0 0x7f19dc0aa4d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19dc0a9580 0x7f19dc13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 --2- 172.21.15.125:0/1715086069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19bc005180 0x7f19dc15cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 >> 172.21.15.125:0/1715086069 conn(0x7f19dc06a800 msgr2=0x7f19dc15e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 shutdown_connections 2024-01-16T15:04:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.474+0000 7f19e4000640 1 -- 172.21.15.125:0/1715086069 wait complete. 2024-01-16T15:04:53.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:04:53.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.701+0000 7f5cfb200640 1 Processor -- start 2024-01-16T15:04:53.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.701+0000 7f5cfb200640 1 -- start start 2024-01-16T15:04:53.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5cfc071e40 0x7f5cfc0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfb200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cfc15d770 0x7f5cfc15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfb200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cfc06c730 con 0x7f5cfc143810 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfb200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cfc143bf0 con 0x7f5cfc15d770 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cfc0722f0 con 0x7f5cfc071e40 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cfa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cfc15d770 0x7f5cfc15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cf9e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5cfc143810 0x7f5cfc15e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48742/0 (socket says 172.21.15.125:48742) 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cf9e00640 1 -- 172.21.15.125:0/2785668558 learned_addr learned my addr 172.21.15.125:0/2785668558 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:53.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.703+0000 7f5cf3e00640 1 --2- 172.21.15.125:0/2785668558 >> v2:172.21.15.137:3300/0 conn(0x7f5cfc071e40 0x7f5cfc0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.704+0000 7f5cfa800640 1 -- 172.21.15.125:0/2785668558 >> v2:172.21.15.137:3300/0 conn(0x7f5cfc071e40 msgr2=0x7f5cfc0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.704+0000 7f5cfa800640 1 --2- 172.21.15.125:0/2785668558 >> v2:172.21.15.137:3300/0 conn(0x7f5cfc071e40 0x7f5cfc0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.704+0000 7f5cfa800640 1 -- 172.21.15.125:0/2785668558 >> v1:172.21.15.125:6789/0 conn(0x7f5cfc143810 legacy=0x7f5cfc15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.704+0000 7f5cfa800640 1 -- 172.21.15.125:0/2785668558 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc0724d0 con 0x7f5cfc15d770 2024-01-16T15:04:53.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.704+0000 7f5cf9400640 1 -- 172.21.15.125:0/2785668558 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696988217 0 0) 0x7f5cfc06c730 con 0x7f5cfc143810 2024-01-16T15:04:53.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.705+0000 7f5cfa800640 1 --2- 172.21.15.125:0/2785668558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc15d770 0x7f5cfc15dbc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0b0a40 tx=0x7f5cf402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd8f7c4a784e1ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.705+0000 7f5cf9400640 1 -- 172.21.15.125:0/2785668558 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf402cb60 con 0x7f5cfc15d770 2024-01-16T15:04:53.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.705+0000 7f5cf9400640 1 -- 172.21.15.125:0/2785668558 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf402ccc0 con 0x7f5cfc15d770 2024-01-16T15:04:53.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.705+0000 7f5cf9400640 1 -- 172.21.15.125:0/2785668558 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4034660 con 0x7f5cfc15d770 2024-01-16T15:04:53.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 -- 172.21.15.125:0/2785668558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc15d770 msgr2=0x7f5cfc15dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 --2- 172.21.15.125:0/2785668558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc15d770 0x7f5cfc15dbc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0b0a40 tx=0x7f5cf402bdb0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 -- 172.21.15.125:0/2785668558 shutdown_connections 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 --2- 172.21.15.125:0/2785668558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc15d770 0x7f5cfc15dbc0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 --2- 172.21.15.125:0/2785668558 >> v2:172.21.15.137:3300/0 conn(0x7f5cfc071e40 0x7f5cfc0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 -- 172.21.15.125:0/2785668558 >> 172.21.15.125:0/2785668558 conn(0x7f5cfc06a7d0 msgr2=0x7f5cfc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 -- 172.21.15.125:0/2785668558 shutdown_connections 2024-01-16T15:04:53.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.706+0000 7f5cfb200640 1 -- 172.21.15.125:0/2785668558 wait complete. 2024-01-16T15:04:53.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.708+0000 7f5cfb200640 1 Processor -- start 2024-01-16T15:04:53.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.708+0000 7f5cfb200640 1 -- start start 2024-01-16T15:04:53.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cfc143810 0x7f5cfc144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cfc15d770 0x7f5cfc144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc12fe90 con 0x7f5cfc071e40 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc130170 con 0x7f5cfc143810 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc130450 con 0x7f5cfc15d770 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48954/0 (socket says 172.21.15.125:48954) 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 -- 172.21.15.125:0/14874223 learned_addr learned my addr 172.21.15.125:0/14874223 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cfa800640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cfc15d770 0x7f5cfc144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 -- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cfc15d770 msgr2=0x7f5cfc144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cfc15d770 0x7f5cfc144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 -- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cfc143810 msgr2=0x7f5cfc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cfc143810 0x7f5cfc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.709+0000 7f5cf9e00640 1 -- 172.21.15.125:0/14874223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4009a70 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cf9e00640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00c4d0 tx=0x7f5cec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0074e0 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec007660 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec004a20 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc130730 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.710+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc146db0 con 0x7f5cfc071e40 2024-01-16T15:04:53.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.711+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc067450 con 0x7f5cfc071e40 2024-01-16T15:04:53.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.712+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5cec004b80 con 0x7f5cfc071e40 2024-01-16T15:04:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.713+0000 7f5cf2000640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 0x7f5cd406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.713+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5cec0877a0 con 0x7f5cfc071e40 2024-01-16T15:04:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.714+0000 7f5cf3e00640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 0x7f5cd406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.714+0000 7f5cf3e00640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 0x7f5cd406dff0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc140940 tx=0x7f5ce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.718+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cec053e90 con 0x7f5cfc071e40 2024-01-16T15:04:53.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.911+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5cfc0b2050 con 0x7f5cfc071e40 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.917+0000 7f5cf2000640 1 -- 172.21.15.125:0/14874223 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5cec0180a0 con 0x7f5cfc071e40 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 msgr2=0x7f5cd406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 0x7f5cd406dff0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc140940 tx=0x7f5ce4008040 comp rx=0 tx=0).stop 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 msgr2=0x7f5cfc13f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00c4d0 tx=0x7f5cec00c990 comp rx=0 tx=0).stop 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 shutdown_connections 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cfc15d770 0x7f5cfc144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cfc143810 0x7f5cfc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cd406bb30 0x7f5cd406dff0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 --2- 172.21.15.125:0/14874223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cfc071e40 0x7f5cfc13f7d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 >> 172.21.15.125:0/14874223 conn(0x7f5cfc06a7d0 msgr2=0x7f5cfc06abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 shutdown_connections 2024-01-16T15:04:53.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:53.920+0000 7f5cfb200640 1 -- 172.21.15.125:0/14874223 wait complete. 2024-01-16T15:04:53.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:04:54.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.150+0000 7f6130c00640 1 Processor -- start 2024-01-16T15:04:54.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.150+0000 7f6130c00640 1 -- start start 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f612c0ae5a0 0x7f612c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f612c0b3ae0 0x7f612c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c06d120 0x7f612c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c071450 con 0x7f612c0b3ae0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c070c40 con 0x7f612c0ae5a0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6130c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c06c470 con 0x7f612c06d120 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6123e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f612c0b3ae0 0x7f612c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6123e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f612c0b3ae0 0x7f612c06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48964/0 (socket says 172.21.15.125:48964) 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.152+0000 7f6123e00640 1 -- 172.21.15.125:0/1983036313 learned_addr learned my addr 172.21.15.125:0/1983036313 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6129e00640 1 --2- 172.21.15.125:0/1983036313 >> v2:172.21.15.137:3300/0 conn(0x7f612c0ae5a0 0x7f612c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f612a800640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c06d120 0x7f612c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6123e00640 1 -- 172.21.15.125:0/1983036313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c06d120 msgr2=0x7f612c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6123e00640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c06d120 0x7f612c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6123e00640 1 -- 172.21.15.125:0/1983036313 >> v2:172.21.15.137:3300/0 conn(0x7f612c0ae5a0 msgr2=0x7f612c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6123e00640 1 --2- 172.21.15.125:0/1983036313 >> v2:172.21.15.137:3300/0 conn(0x7f612c0ae5a0 0x7f612c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.153+0000 7f6123e00640 1 -- 172.21.15.125:0/1983036313 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c06c750 con 0x7f612c0b3ae0 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.154+0000 7f6123e00640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c0b3ae0 0x7f612c06cbe0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f612c0cfe20 tx=0x7f611402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa39625a11ed37d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.154+0000 7f6129400640 1 -- 172.21.15.125:0/1983036313 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114031870 con 0x7f612c0b3ae0 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6129400640 1 -- 172.21.15.125:0/1983036313 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61140319d0 con 0x7f612c0b3ae0 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6129400640 1 -- 172.21.15.125:0/1983036313 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114031d60 con 0x7f612c0b3ae0 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 -- 172.21.15.125:0/1983036313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c0b3ae0 msgr2=0x7f612c06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c0b3ae0 0x7f612c06cbe0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f612c0cfe20 tx=0x7f611402b910 comp rx=0 tx=0).stop 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 -- 172.21.15.125:0/1983036313 shutdown_connections 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c06d120 0x7f612c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 --2- 172.21.15.125:0/1983036313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c0b3ae0 0x7f612c06cbe0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 --2- 172.21.15.125:0/1983036313 >> v2:172.21.15.137:3300/0 conn(0x7f612c0ae5a0 0x7f612c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 -- 172.21.15.125:0/1983036313 >> 172.21.15.125:0/1983036313 conn(0x7f612c06a800 msgr2=0x7f612c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.155+0000 7f6130c00640 1 -- 172.21.15.125:0/1983036313 shutdown_connections 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.156+0000 7f6130c00640 1 -- 172.21.15.125:0/1983036313 wait complete. 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.157+0000 7f6130c00640 1 Processor -- start 2024-01-16T15:04:54.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.157+0000 7f6130c00640 1 -- start start 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 0x7f612c15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c0b3ae0 0x7f612c0cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c156370 con 0x7f612c06d120 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c15f4a0 con 0x7f612c0ae5a0 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6130c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c15f780 con 0x7f612c0b3ae0 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6129e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6129e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48970/0 (socket says 172.21.15.125:48970) 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.158+0000 7f6129e00640 1 -- 172.21.15.125:0/1353647523 learned_addr learned my addr 172.21.15.125:0/1353647523 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6123e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 0x7f612c15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 -- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c0b3ae0 msgr2=0x7f612c0cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c0b3ae0 0x7f612c0cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 -- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 msgr2=0x7f612c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 0x7f612c15eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 -- 172.21.15.125:0/1353647523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61140094d0 con 0x7f612c06d120 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6123e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 0x7f612c15eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6129e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d800 tx=0x7f611c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.159+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00b6c0 con 0x7f612c06d120 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.160+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c00b820 con 0x7f612c06d120 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.160+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c15fa60 con 0x7f612c06d120 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.160+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00bbb0 con 0x7f612c06d120 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.160+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c160020 con 0x7f612c06d120 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.161+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c1601b0 con 0x7f612c06d120 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.162+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f611c00bd10 con 0x7f612c06d120 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.163+0000 7f6122000640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 0x7f610406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.163+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f611c088b50 con 0x7f612c06d120 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.163+0000 7f6123e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 0x7f610406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.164+0000 7f6123e00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 0x7f610406dff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6114036000 tx=0x7f611403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.168+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f611c0552a0 con 0x7f612c06d120 2024-01-16T15:04:54.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:54.424+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f612c0675d0 con 0x7f612c06d120 2024-01-16T15:04:55.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.264+0000 7f6122000640 1 -- 172.21.15.125:0/1353647523 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v393) v1 ==== 154+0+0 (secure 0 0 0) 0x7f611c0588c0 con 0x7f612c06d120 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 msgr2=0x7f610406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 0x7f610406dff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6114036000 tx=0x7f611403c040 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 msgr2=0x7f612c155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d800 tx=0x7f611c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 shutdown_connections 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f612c0b3ae0 0x7f612c0cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f612c0ae5a0 0x7f612c15eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f610406bb30 0x7f610406dff0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 --2- 172.21.15.125:0/1353647523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f612c06d120 0x7f612c155e30 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 >> 172.21.15.125:0/1353647523 conn(0x7f612c06a800 msgr2=0x7f612c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 shutdown_connections 2024-01-16T15:04:55.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.268+0000 7f6130c00640 1 -- 172.21.15.125:0/1353647523 wait complete. 2024-01-16T15:04:55.267 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:04:55.268 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:04:55.268 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:04:55.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:55.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.483+0000 7f4654e00640 1 Processor -- start 2024-01-16T15:04:55.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.484+0000 7f4654e00640 1 -- start start 2024-01-16T15:04:55.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.485+0000 7f4654e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f465013e360 0x7f46500cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.485+0000 7f4654e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465015c5f0 con 0x7f465015f9c0 2024-01-16T15:04:55.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.485+0000 7f4654e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465006c470 con 0x7f465013df80 2024-01-16T15:04:55.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.485+0000 7f4654e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465006c750 con 0x7f465013e360 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f465015f9c0 0x7f46500cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48744/0 (socket says 172.21.15.125:48744) 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464e800640 1 -- 172.21.15.125:0/513018769 learned_addr learned my addr 172.21.15.125:0/513018769 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f4645200640 1 --2- 172.21.15.125:0/513018769 >> v2:172.21.15.137:3300/0 conn(0x7f465013e360 0x7f46500cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830971626 0 0) 0x7f465015c5f0 con 0x7f465015f9c0 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4630003520 con 0x7f465015f9c0 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759460806 0 0) 0x7f4630003520 con 0x7f465015f9c0 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 >> v2:172.21.15.137:3300/0 conn(0x7f465013e360 msgr2=0x7f46500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 --2- 172.21.15.125:0/513018769 >> v2:172.21.15.137:3300/0 conn(0x7f465013e360 0x7f46500cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 >> v1:172.21.15.137:6789/0 conn(0x7f465013df80 legacy=0x7f465015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465006ca30 con 0x7f465015f9c0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.486+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f4648001a10 con 0x7f465015f9c0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 >> v1:172.21.15.125:6789/0 conn(0x7f465015f9c0 legacy=0x7f46500cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 0x7f4630003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300047c0 0x7f4630004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4630005180 0x7f463000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465006c750 con 0x7f46300039c0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463000c080 con 0x7f4630005180 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463000c240 con 0x7f46300047c0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 0x7f4630003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464e800640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4630005180 0x7f463000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f4645200640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300047c0 0x7f4630004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 -- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300047c0 msgr2=0x7f4630004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300047c0 0x7f4630004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 -- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4630005180 msgr2=0x7f463000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4630005180 0x7f463000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464de00640 1 -- 172.21.15.125:0/513018769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630003520 con 0x7f46300039c0 2024-01-16T15:04:55.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464e800640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4630005180 0x7f463000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:55.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.487+0000 7f464e800640 1 -- 172.21.15.125:0/513018769 reap_dead start 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.488+0000 7f464de00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 0x7f4630003de0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4640001750 tx=0x7f46300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc630ea6e52523d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46400052d0 con 0x7f46300039c0 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464002e050 con 0x7f46300039c0 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f464d400640 1 -- 172.21.15.125:0/513018769 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640003830 con 0x7f46300039c0 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 -- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 msgr2=0x7f4630003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 0x7f4630003de0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4640001750 tx=0x7f46300057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 -- 172.21.15.125:0/513018769 shutdown_connections 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 --2- 172.21.15.125:0/513018769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46300039c0 0x7f4630003de0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 -- 172.21.15.125:0/513018769 >> 172.21.15.125:0/513018769 conn(0x7f465006a800 msgr2=0x7f465006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.489+0000 7f4654e00640 1 -- 172.21.15.125:0/513018769 shutdown_connections 2024-01-16T15:04:55.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.490+0000 7f4654e00640 1 -- 172.21.15.125:0/513018769 wait complete. 2024-01-16T15:04:55.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.491+0000 7f4654e00640 1 Processor -- start 2024-01-16T15:04:55.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.491+0000 7f4654e00640 1 -- start start 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300039c0 0x7f4650152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46501e5650 0x7f46500b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4650152d80 con 0x7f46501e51d0 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46501f4a90 con 0x7f46300039c0 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.492+0000 7f4654e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46501f4d70 con 0x7f46501e5650 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48998/0 (socket says 172.21.15.125:48998) 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 -- 172.21.15.125:0/3515185099 learned_addr learned my addr 172.21.15.125:0/3515185099 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f464de00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300039c0 0x7f4650152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 -- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46501e5650 msgr2=0x7f46500b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46501e5650 0x7f46500b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 -- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300039c0 msgr2=0x7f4650152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300039c0 0x7f4650152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 -- 172.21.15.125:0/3515185099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46501f5050 con 0x7f46501e51d0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.493+0000 7f4645200640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f46380108d0 tx=0x7f4638010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.494+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638009650 con 0x7f46501e51d0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.494+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463800eb00 con 0x7f46501e51d0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.494+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463800ee90 con 0x7f46501e51d0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.494+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46501f5330 con 0x7f46501e51d0 2024-01-16T15:04:55.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.494+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46501f5b30 con 0x7f46501e51d0 2024-01-16T15:04:55.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.495+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46500cd420 con 0x7f46501e51d0 2024-01-16T15:04:55.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.496+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f463801c400 con 0x7f46501e51d0 2024-01-16T15:04:55.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.497+0000 7f4646a00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 0x7f462806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.497+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4638088ac0 con 0x7f46501e51d0 2024-01-16T15:04:55.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.497+0000 7f464de00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 0x7f462806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.498+0000 7f464de00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 0x7f462806e020 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4640033000 tx=0x7f4640009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.501+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46380552c0 con 0x7f46501e51d0 2024-01-16T15:04:55.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.677+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46500675d0 con 0x7f46501e51d0 2024-01-16T15:04:55.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.679+0000 7f4646a00640 1 -- 172.21.15.125:0/3515185099 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f46380588e0 con 0x7f46501e51d0 2024-01-16T15:04:55.658 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:55.658 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:04:48.223107+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6835/2216643169":"2024-01-17T15:04:45.210297+0000","172.21.15.125:6834/2216643169":"2024-01-17T15:04:45.210297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:55.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.681+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 msgr2=0x7f462806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.681+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 0x7f462806e020 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4640033000 tx=0x7f4640009030 comp rx=0 tx=0).stop 2024-01-16T15:04:55.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.681+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 msgr2=0x7f46500afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.681+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f46380108d0 tx=0x7f4638010d90 comp rx=0 tx=0).stop 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 shutdown_connections 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46501e5650 0x7f46500b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f462806bb60 0x7f462806e020 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4640033000 tx=0x7f4640009030 comp rx=0 tx=0).stop 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46501e51d0 0x7f46500afd10 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 --2- 172.21.15.125:0/3515185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300039c0 0x7f4650152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 >> 172.21.15.125:0/3515185099 conn(0x7f465006a800 msgr2=0x7f46500ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 shutdown_connections 2024-01-16T15:04:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.682+0000 7f4654e00640 1 -- 172.21.15.125:0/3515185099 wait complete. 2024-01-16T15:04:55.679 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:55.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.905+0000 7f0924200640 1 Processor -- start 2024-01-16T15:04:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.905+0000 7f0924200640 1 -- start start 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.906+0000 7f0924200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f091c0cf9a0 0x7f091c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.906+0000 7f0924200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f091c15e1e0 0x7f091c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.906+0000 7f0924200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0b1ee0 con 0x7f091c143ae0 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.906+0000 7f0924200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0b2780 con 0x7f091c15e1e0 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.906+0000 7f0924200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c15e990 con 0x7f091c0cf9a0 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0921800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f091c143ae0 0x7f091c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48748/0 (socket says 172.21.15.125:48748) 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0921800640 1 -- 172.21.15.125:0/3280426122 learned_addr learned my addr 172.21.15.125:0/3280426122 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0922200640 1 --2- 172.21.15.125:0/3280426122 >> v2:172.21.15.125:3300/0 conn(0x7f091c15e1e0 0x7f091c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0920e00640 1 --2- 172.21.15.125:0/3280426122 >> v2:172.21.15.137:3300/0 conn(0x7f091c0cf9a0 0x7f091c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f091be00640 1 -- 172.21.15.125:0/3280426122 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181834386 0 0) 0x7f091c0b1ee0 con 0x7f091c143ae0 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0922200640 1 -- 172.21.15.125:0/3280426122 >> v2:172.21.15.137:3300/0 conn(0x7f091c0cf9a0 msgr2=0x7f091c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0922200640 1 --2- 172.21.15.125:0/3280426122 >> v2:172.21.15.137:3300/0 conn(0x7f091c0cf9a0 0x7f091c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0922200640 1 -- 172.21.15.125:0/3280426122 >> v1:172.21.15.125:6789/0 conn(0x7f091c143ae0 legacy=0x7f091c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.907+0000 7f0922200640 1 -- 172.21.15.125:0/3280426122 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0b2130 con 0x7f091c15e1e0 2024-01-16T15:04:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.908+0000 7f0922200640 1 --2- 172.21.15.125:0/3280426122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c0702c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f091c0cb830 tx=0x7f091402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead842d90cbc20c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f091be00640 1 -- 172.21.15.125:0/3280426122 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914031830 con 0x7f091c15e1e0 2024-01-16T15:04:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f091be00640 1 -- 172.21.15.125:0/3280426122 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914031990 con 0x7f091c15e1e0 2024-01-16T15:04:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f091be00640 1 -- 172.21.15.125:0/3280426122 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914031d20 con 0x7f091c15e1e0 2024-01-16T15:04:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f0924200640 1 -- 172.21.15.125:0/3280426122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 msgr2=0x7f091c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f0924200640 1 --2- 172.21.15.125:0/3280426122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c0702c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f091c0cb830 tx=0x7f091402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f0924200640 1 -- 172.21.15.125:0/3280426122 shutdown_connections 2024-01-16T15:04:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.909+0000 7f0924200640 1 --2- 172.21.15.125:0/3280426122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c0702c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.910+0000 7f0924200640 1 --2- 172.21.15.125:0/3280426122 >> v2:172.21.15.137:3300/0 conn(0x7f091c0cf9a0 0x7f091c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.910+0000 7f0924200640 1 -- 172.21.15.125:0/3280426122 >> 172.21.15.125:0/3280426122 conn(0x7f091c06a800 msgr2=0x7f091c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.910+0000 7f0924200640 1 -- 172.21.15.125:0/3280426122 shutdown_connections 2024-01-16T15:04:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.910+0000 7f0924200640 1 -- 172.21.15.125:0/3280426122 wait complete. 2024-01-16T15:04:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.911+0000 7f0924200640 1 Processor -- start 2024-01-16T15:04:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.912+0000 7f0924200640 1 -- start start 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.912+0000 7f0924200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f091c0cf9a0 0x7f091c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.912+0000 7f0924200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f091c143ae0 0x7f091c132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0924200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0924200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c12ffa0 con 0x7f091c15e1e0 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0924200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c132ad0 con 0x7f091c143ae0 2024-01-16T15:04:55.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0924200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c132db0 con 0x7f091c0cf9a0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49020/0 (socket says 172.21.15.125:49020) 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 -- 172.21.15.125:0/3926442125 learned_addr learned my addr 172.21.15.125:0/3926442125 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0920e00640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f091c143ae0 0x7f091c132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 -- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f091c0cf9a0 msgr2=0x7f091c159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f091c0cf9a0 0x7f091c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 -- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f091c143ae0 msgr2=0x7f091c132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f091c143ae0 0x7f091c132030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.913+0000 7f0922200640 1 -- 172.21.15.125:0/3926442125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914009400 con 0x7f091c15e1e0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0922200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f091402bee0 tx=0x7f09140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914004230 con 0x7f091c15e1e0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914004390 con 0x7f091c15e1e0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091403a920 con 0x7f091c15e1e0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c1301b0 con 0x7f091c15e1e0 2024-01-16T15:04:55.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.914+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c15d3f0 con 0x7f091c15e1e0 2024-01-16T15:04:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.915+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c1306d0 con 0x7f091c15e1e0 2024-01-16T15:04:55.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.916+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f091403aa80 con 0x7f091c15e1e0 2024-01-16T15:04:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.917+0000 7f0919600640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 0x7f08f406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.917+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09140aacc0 con 0x7f091c15e1e0 2024-01-16T15:04:55.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.917+0000 7f0921800640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 0x7f08f406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.918+0000 7f0921800640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 0x7f08f406dff0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f090c001560 tx=0x7f090c007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:55.921+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f091c1306d0 con 0x7f091c15e1e0 2024-01-16T15:04:56.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.112+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f091c1306d0 con 0x7f091c15e1e0 2024-01-16T15:04:56.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.113+0000 7f0919600640 1 -- 172.21.15.125:0/3926442125 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f091c1306d0 con 0x7f091c15e1e0 2024-01-16T15:04:56.092 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 msgr2=0x7f08f406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 0x7f08f406dff0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f090c001560 tx=0x7f090c007ca0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 msgr2=0x7f091c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f091402bee0 tx=0x7f09140046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 shutdown_connections 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08f406bb30 0x7f08f406dff0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c15e1e0 0x7f091c15a4e0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f091c143ae0 0x7f091c132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 --2- 172.21.15.125:0/3926442125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f091c0cf9a0 0x7f091c159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 >> 172.21.15.125:0/3926442125 conn(0x7f091c06a800 msgr2=0x7f091c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 shutdown_connections 2024-01-16T15:04:56.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.116+0000 7f0924200640 1 -- 172.21.15.125:0/3926442125 wait complete. 2024-01-16T15:04:56.114 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:04:56.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.341+0000 7f22c1e00640 1 Processor -- start 2024-01-16T15:04:56.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.341+0000 7f22c1e00640 1 -- start start 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22c1e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f22bc0ae5a0 0x7f22bc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22c1e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22bc0b3ae0 0x7f22bc06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22c1e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc070c40 con 0x7f22bc06d120 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22c1e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc071450 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22c1e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc06c470 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22bb400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f22bc06d120 0x7f22bc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48760/0 (socket says 172.21.15.125:48760) 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22bb400640 1 -- 172.21.15.125:0/2256085447 learned_addr learned my addr 172.21.15.125:0/2256085447 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.343+0000 7f22ba000640 1 --2- 172.21.15.125:0/2256085447 >> v2:172.21.15.125:3300/0 conn(0x7f22bc0b3ae0 0x7f22bc06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22baa00640 1 --2- 172.21.15.125:0/2256085447 >> v2:172.21.15.137:3300/0 conn(0x7f22bc0ae5a0 0x7f22bc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22b9600640 1 -- 172.21.15.125:0/2256085447 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364091047 0 0) 0x7f22bc070c40 con 0x7f22bc06d120 2024-01-16T15:04:56.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22ba000640 1 -- 172.21.15.125:0/2256085447 >> v2:172.21.15.137:3300/0 conn(0x7f22bc0ae5a0 msgr2=0x7f22bc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:56.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22ba000640 1 --2- 172.21.15.125:0/2256085447 >> v2:172.21.15.137:3300/0 conn(0x7f22bc0ae5a0 0x7f22bc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22ba000640 1 -- 172.21.15.125:0/2256085447 >> v1:172.21.15.125:6789/0 conn(0x7f22bc06d120 legacy=0x7f22bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.344+0000 7f22ba000640 1 -- 172.21.15.125:0/2256085447 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc06c750 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.345+0000 7f22ba000640 1 --2- 172.21.15.125:0/2256085447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0b3ae0 0x7f22bc06cbe0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0cfe20 tx=0x7f22a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c705756fd2af5d5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.345+0000 7f22b9600640 1 -- 172.21.15.125:0/2256085447 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031830 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.345+0000 7f22b9600640 1 -- 172.21.15.125:0/2256085447 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4031990 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.345+0000 7f22b9600640 1 -- 172.21.15.125:0/2256085447 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031d20 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 -- 172.21.15.125:0/2256085447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0b3ae0 msgr2=0x7f22bc06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 --2- 172.21.15.125:0/2256085447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0b3ae0 0x7f22bc06cbe0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0cfe20 tx=0x7f22a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 -- 172.21.15.125:0/2256085447 shutdown_connections 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 --2- 172.21.15.125:0/2256085447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0b3ae0 0x7f22bc06cbe0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 --2- 172.21.15.125:0/2256085447 >> v2:172.21.15.137:3300/0 conn(0x7f22bc0ae5a0 0x7f22bc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 -- 172.21.15.125:0/2256085447 >> 172.21.15.125:0/2256085447 conn(0x7f22bc06a800 msgr2=0x7f22bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 -- 172.21.15.125:0/2256085447 shutdown_connections 2024-01-16T15:04:56.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.346+0000 7f22c1e00640 1 -- 172.21.15.125:0/2256085447 wait complete. 2024-01-16T15:04:56.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.348+0000 7f22c1e00640 1 Processor -- start 2024-01-16T15:04:56.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.348+0000 7f22c1e00640 1 -- start start 2024-01-16T15:04:56.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22bc06d120 0x7f22bc0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22bc0b3ae0 0x7f22bc0cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0cce20 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0cd240 con 0x7f22bc06d120 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22c1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0cc1a0 con 0x7f22bc0b3ae0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49036/0 (socket says 172.21.15.125:49036) 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 -- 172.21.15.125:0/3797399673 learned_addr learned my addr 172.21.15.125:0/3797399673 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22baa00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22bc06d120 0x7f22bc0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22bb400640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22bc0b3ae0 0x7f22bc0cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 -- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22bc0b3ae0 msgr2=0x7f22bc0cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22bc0b3ae0 0x7f22bc0cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 -- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22bc06d120 msgr2=0x7f22bc0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.349+0000 7f22ba000640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22bc06d120 0x7f22bc0cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22ba000640 1 -- 172.21.15.125:0/3797399673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4009400 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22ba000640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f22a4000c00 tx=0x7f22a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a40317d0 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a403aa60 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a403ae10 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc0cc480 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.350+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc0cca40 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.351+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc0675d0 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.352+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f22a404d450 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.353+0000 7f22b3400640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 0x7f229406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.353+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f22a40aa750 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.354+0000 7f22baa00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 0x7f229406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.354+0000 7f22baa00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 0x7f229406dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009a60 tx=0x7f22ac003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.359+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22a4076e40 con 0x7f22bc0ae5a0 2024-01-16T15:04:56.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:56.618+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7f22bc0ccbd0 con 0x7f22bc0ae5a0 2024-01-16T15:04:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.280+0000 7f22b3400640 1 -- 172.21.15.125:0/3797399673 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v395) v1 ==== 169+0+0 (secure 0 0 0) 0x7f22a4035020 con 0x7f22bc0ae5a0 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 msgr2=0x7f229406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 0x7f229406dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009a60 tx=0x7f22ac003990 comp rx=0 tx=0).stop 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 msgr2=0x7f22bc15c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f22a4000c00 tx=0x7f22a40046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 shutdown_connections 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f22bc0b3ae0 0x7f22bc0cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f229406bb30 0x7f229406dff0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22bc0ae5a0 0x7f22bc15c5a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 --2- 172.21.15.125:0/3797399673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22bc06d120 0x7f22bc0cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.283+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 >> 172.21.15.125:0/3797399673 conn(0x7f22bc06a800 msgr2=0x7f22bc0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.284+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 shutdown_connections 2024-01-16T15:04:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.284+0000 7f22c1e00640 1 -- 172.21.15.125:0/3797399673 wait complete. 2024-01-16T15:04:57.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:04:57.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.518+0000 7f1d64a00640 1 Processor -- start 2024-01-16T15:04:57.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.518+0000 7f1d64a00640 1 -- start start 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.519+0000 7f1d64a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d5c14b2c0 0x7f1d5c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.519+0000 7f1d64a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0cfbf0 con 0x7f1d5c071070 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d64a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c071690 con 0x7f1d5c0cf7d0 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d64a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5c14b6a0 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d5c14b2c0 0x7f1d5c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1d5c071070 0x7f1d5c0cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45710/0 (socket says 172.21.15.125:45710) 2024-01-16T15:04:57.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62a00640 1 -- 172.21.15.125:0/1236567112 learned_addr learned my addr 172.21.15.125:0/1236567112 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:04:57.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62000640 1 -- 172.21.15.125:0/1236567112 >> v1:172.21.15.137:6789/0 conn(0x7f1d5c0cf7d0 legacy=0x7f1d5c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62000640 1 -- 172.21.15.125:0/1236567112 >> v1:172.21.15.125:6789/0 conn(0x7f1d5c071070 legacy=0x7f1d5c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.520+0000 7f1d62000640 1 -- 172.21.15.125:0/1236567112 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c071970 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d62000640 1 --2- 172.21.15.125:0/1236567112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c0cf290 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0097a0 tx=0x7f1d4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d718a927cbd627a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d60c00640 1 -- 172.21.15.125:0/1236567112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c031830 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d60c00640 1 -- 172.21.15.125:0/1236567112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c031990 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d60c00640 1 -- 172.21.15.125:0/1236567112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c031d20 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d64a00640 1 -- 172.21.15.125:0/1236567112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c14b2c0 msgr2=0x7f1d5c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.522+0000 7f1d64a00640 1 --2- 172.21.15.125:0/1236567112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c0cf290 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0097a0 tx=0x7f1d4c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.523+0000 7f1d64a00640 1 -- 172.21.15.125:0/1236567112 shutdown_connections 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.523+0000 7f1d64a00640 1 --2- 172.21.15.125:0/1236567112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c0cf290 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.523+0000 7f1d64a00640 1 -- 172.21.15.125:0/1236567112 >> 172.21.15.125:0/1236567112 conn(0x7f1d5c06a800 msgr2=0x7f1d5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.523+0000 7f1d64a00640 1 -- 172.21.15.125:0/1236567112 shutdown_connections 2024-01-16T15:04:57.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.523+0000 7f1d64a00640 1 -- 172.21.15.125:0/1236567112 wait complete. 2024-01-16T15:04:57.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.524+0000 7f1d64a00640 1 Processor -- start 2024-01-16T15:04:57.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.525+0000 7f1d64a00640 1 -- start start 2024-01-16T15:04:57.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.525+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.525+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d5c0cf7d0 0x7f1d5c146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.525+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c1f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d64a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c145d00 con 0x7f1d5c071070 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d64a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c148120 con 0x7f1d5c14b2c0 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d64a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c145f50 con 0x7f1d5c0cf7d0 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60012/0 (socket says 172.21.15.125:60012) 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 -- 172.21.15.125:0/3269519198 learned_addr learned my addr 172.21.15.125:0/3269519198 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 -- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d5c0cf7d0 msgr2=0x7f1d5c146d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d5c0cf7d0 0x7f1d5c146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 -- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d5c14b2c0 msgr2=0x7f1d5c1f6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c1f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 -- 172.21.15.125:0/3269519198 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c009400 con 0x7f1d5c071070 2024-01-16T15:04:57.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.526+0000 7f1d62000640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.527+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c03aa60 con 0x7f1d5c071070 2024-01-16T15:04:57.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.527+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c03abc0 con 0x7f1d5c071070 2024-01-16T15:04:57.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.527+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c0443f0 con 0x7f1d5c071070 2024-01-16T15:04:57.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.527+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c146230 con 0x7f1d5c071070 2024-01-16T15:04:57.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.527+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c1467f0 con 0x7f1d5c071070 2024-01-16T15:04:57.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.528+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c146a10 con 0x7f1d5c071070 2024-01-16T15:04:57.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.529+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f1d4c04d450 con 0x7f1d5c071070 2024-01-16T15:04:57.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.530+0000 7f1d5a000640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 0x7f1d3406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.530+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1d4c0aa790 con 0x7f1d5c071070 2024-01-16T15:04:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.530+0000 7f1d61600640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 0x7f1d3406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.531+0000 7f1d61600640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 0x7f1d3406dff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c157a00 tx=0x7f1d440079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.534+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d4c0b3080 con 0x7f1d5c071070 2024-01-16T15:04:57.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.727+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1d5c0675d0 con 0x7f1d5c071070 2024-01-16T15:04:57.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.732+0000 7f1d5a000640 1 -- 172.21.15.125:0/3269519198 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1d4c035070 con 0x7f1d5c071070 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 msgr2=0x7f1d3406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 0x7f1d3406dff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c157a00 tx=0x7f1d440079b0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 msgr2=0x7f1d5c147be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 shutdown_connections 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d5c14b2c0 0x7f1d5c1f6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d5c0cf7d0 0x7f1d5c146d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d3406bb30 0x7f1d3406dff0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 --2- 172.21.15.125:0/3269519198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d5c071070 0x7f1d5c147be0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 >> 172.21.15.125:0/3269519198 conn(0x7f1d5c06a800 msgr2=0x7f1d5c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 shutdown_connections 2024-01-16T15:04:57.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:57.735+0000 7f1d64a00640 1 -- 172.21.15.125:0/3269519198 wait complete. 2024-01-16T15:04:57.735 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:04:57.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:57.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:57.967 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:04:57.967 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:04:57.967 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:58.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:58.024 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:04:58.024 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:58.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:58.081 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:04:58.081 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:58.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:58.137 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:04:58.137 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:58.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:58.194 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:04:58.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:58.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.431+0000 7f09ac600640 1 Processor -- start 2024-01-16T15:04:58.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.431+0000 7f09ac600640 1 -- start start 2024-01-16T15:04:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.432+0000 7f09ac600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 0x7f09a40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09ac600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f09a415f9c0 0x7f09a40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09ac600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a415c5f0 con 0x7f09a413df80 2024-01-16T15:04:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09ac600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a406c470 con 0x7f09a415f9c0 2024-01-16T15:04:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09ac600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a406c750 con 0x7f09a413e360 2024-01-16T15:04:58.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09a9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 0x7f09a40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09aa600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f09a415f9c0 0x7f09a40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09a9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 0x7f09a40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:36630/0 (socket says 172.21.15.125:36630) 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09a9c00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f09a413df80 0x7f09a415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:49640/0 (socket says 172.21.15.125:49640) 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.433+0000 7f09a9200640 1 -- 172.21.15.125:0/2124690251 learned_addr learned my addr 172.21.15.125:0/2124690251 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.434+0000 7f09aa600640 1 -- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 msgr2=0x7f09a40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.434+0000 7f09aa600640 1 --2- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 0x7f09a40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.434+0000 7f09aa600640 1 -- 172.21.15.125:0/2124690251 >> v1:172.21.15.137:6789/0 conn(0x7f09a413df80 legacy=0x7f09a415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.434+0000 7f09aa600640 1 -- 172.21.15.125:0/2124690251 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a406ca30 con 0x7f09a415f9c0 2024-01-16T15:04:58.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.435+0000 7f09aa600640 1 --2- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a415f9c0 0x7f09a40cf7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d90 tx=0x7f099c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28eeec9b9824a4bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.436+0000 7f09a3e00640 1 -- 172.21.15.125:0/2124690251 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c02cc50 con 0x7f09a415f9c0 2024-01-16T15:04:58.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.436+0000 7f09a3e00640 1 -- 172.21.15.125:0/2124690251 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c02cdb0 con 0x7f09a415f9c0 2024-01-16T15:04:58.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.436+0000 7f09a3e00640 1 -- 172.21.15.125:0/2124690251 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c034660 con 0x7f09a415f9c0 2024-01-16T15:04:58.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 -- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a415f9c0 msgr2=0x7f09a40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 --2- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a415f9c0 0x7f09a40cf7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d90 tx=0x7f099c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 -- 172.21.15.125:0/2124690251 shutdown_connections 2024-01-16T15:04:58.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 --2- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a415f9c0 0x7f09a40cf7d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 --2- 172.21.15.125:0/2124690251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413e360 0x7f09a40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 -- 172.21.15.125:0/2124690251 >> 172.21.15.125:0/2124690251 conn(0x7f09a406a800 msgr2=0x7f09a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 -- 172.21.15.125:0/2124690251 shutdown_connections 2024-01-16T15:04:58.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.437+0000 7f09ac600640 1 -- 172.21.15.125:0/2124690251 wait complete. 2024-01-16T15:04:58.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.439+0000 7f09ac600640 1 Processor -- start 2024-01-16T15:04:58.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.439+0000 7f09ac600640 1 -- start start 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413df80 0x7f09a40adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a413e360 0x7f09a40cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41f6800 con 0x7f09a415f9c0 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41f6ae0 con 0x7f09a413e360 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09ac600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41f6dc0 con 0x7f09a413df80 2024-01-16T15:04:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60028/0 (socket says 172.21.15.125:60028) 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 -- 172.21.15.125:0/2826741182 learned_addr learned my addr 172.21.15.125:0/2826741182 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09a9c00640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413df80 0x7f09a40adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09a9200640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a413e360 0x7f09a40cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 -- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413df80 msgr2=0x7f09a40adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413df80 0x7f09a40adf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.440+0000 7f09aa600640 1 -- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a413e360 msgr2=0x7f09a40cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09aa600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a413e360 0x7f09a40cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09aa600640 1 -- 172.21.15.125:0/2826741182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c009a70 con 0x7f09a415f9c0 2024-01-16T15:04:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09aa600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d90 tx=0x7f099c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c040650 con 0x7f09a415f9c0 2024-01-16T15:04:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c0407b0 con 0x7f09a415f9c0 2024-01-16T15:04:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c040b60 con 0x7f09a415f9c0 2024-01-16T15:04:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a41f70a0 con 0x7f09a415f9c0 2024-01-16T15:04:58.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.441+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a40b4240 con 0x7f09a415f9c0 2024-01-16T15:04:58.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.443+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a412f490 con 0x7f09a415f9c0 2024-01-16T15:04:58.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.443+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f099c040cc0 con 0x7f09a415f9c0 2024-01-16T15:04:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.444+0000 7f09a1600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 0x7f097c06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.444+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f099c0ab410 con 0x7f09a415f9c0 2024-01-16T15:04:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.445+0000 7f09a9c00640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 0x7f097c06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.445+0000 7f09a9c00640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 0x7f097c06e020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0994000f60 tx=0x7f0994007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.449+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09a412f490 con 0x7f09a415f9c0 2024-01-16T15:04:58.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.668+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a412f490 con 0x7f09a415f9c0 2024-01-16T15:04:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.669+0000 7f09a1600640 1 -- 172.21.15.125:0/2826741182 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f099c044e90 con 0x7f09a415f9c0 2024-01-16T15:04:58.649 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:04:58.649 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2646664931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2646664931},{"type":"v1","addr":"172.21.15.125:6837","nonce":2646664931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5054,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1721910358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1721910358},{"type":"v1","addr":"172.21.15.137:6835","nonce":1721910358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":5362,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/28677566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":28677566},{"type":"v1","addr":"172.21.15.137:6833","nonce":28677566}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33},{"gid":6996,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1998350661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1998350661},{"type":"v1","addr":"172.21.15.125:6835","nonce":1998350661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-16T15:04:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 40 2024-01-16T15:04:58.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.671+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 msgr2=0x7f097c06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.671+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 0x7f097c06e020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0994000f60 tx=0x7f0994007c80 comp rx=0 tx=0).stop 2024-01-16T15:04:58.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.671+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 msgr2=0x7f09a40ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.671+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d90 tx=0x7f099c004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:58.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 shutdown_connections 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f097c06bb60 0x7f097c06e020 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09a415f9c0 0x7f09a40ce2e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09a413e360 0x7f09a40cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 --2- 172.21.15.125:0/2826741182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09a413df80 0x7f09a40adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 >> 172.21.15.125:0/2826741182 conn(0x7f09a406a800 msgr2=0x7f09a415beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 shutdown_connections 2024-01-16T15:04:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.672+0000 7f09ac600640 1 -- 172.21.15.125:0/2826741182 wait complete. 2024-01-16T15:04:58.670 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:04:58.670 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:04:58.671 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:04:58.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:58.671 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:04:58.671 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:04:58.671 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:04:58.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:58.671 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:04:58.672 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:04:58.672 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:04:58.672 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:58.672 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:04:58.672 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:04:58.672 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:04:58.672 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:58.688 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:04:58.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:04:58.693 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:04:58.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:04:58.694 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:04:58.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:04:58.696 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:04:58.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:04:58.741 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:04:58.741 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:58.743 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:04:58.743 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:58.747 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:04:58.747 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:58.749 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:04:58.749 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:58.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:04:58.892 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:04:58.893 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:04:58.900 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T15:04:58.901 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T15:04:58.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.949+0000 7f1f7ce00640 1 Processor -- start 2024-01-16T15:04:58.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.949+0000 7f1f7ce00640 1 -- start start 2024-01-16T15:04:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.950+0000 7f1f7ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780b3ae0 0x7f1f7815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.950+0000 7f1f7ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 0x7f1f780cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.950+0000 7f1f7ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f78070c40 con 0x7f1f780ae5a0 2024-01-16T15:04:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.950+0000 7f1f7ce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f78071450 con 0x7f1f780b3ae0 2024-01-16T15:04:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.950+0000 7f1f7ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780cfc60 con 0x7f1f780cf290 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780b3ae0 0x7f1f7815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780b3ae0 0x7f1f7815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60058/0 (socket says 172.21.15.125:60058) 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 -- 172.21.15.125:0/3100757412 learned_addr learned my addr 172.21.15.125:0/3100757412 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f76800640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 0x7f1f780cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 -- 172.21.15.125:0/3100757412 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 msgr2=0x7f1f780cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 0x7f1f780cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 -- 172.21.15.125:0/3100757412 >> v1:172.21.15.137:6789/0 conn(0x7f1f780ae5a0 legacy=0x7f1f7806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f75400640 1 -- 172.21.15.125:0/3100757412 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7806c470 con 0x7f1f780b3ae0 2024-01-16T15:04:58.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.951+0000 7f1f76800640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 0x7f1f780cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:58.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.952+0000 7f1f75400640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815e430 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009870 tx=0x7f1f6002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236da479d8dc5f42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.952+0000 7f1f74a00640 1 -- 172.21.15.125:0/3100757412 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6002f840 con 0x7f1f780b3ae0 2024-01-16T15:04:58.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.952+0000 7f1f74a00640 1 -- 172.21.15.125:0/3100757412 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6002f9a0 con 0x7f1f780b3ae0 2024-01-16T15:04:58.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.952+0000 7f1f74a00640 1 -- 172.21.15.125:0/3100757412 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6002fd30 con 0x7f1f780b3ae0 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 -- 172.21.15.125:0/3100757412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 msgr2=0x7f1f7815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815e430 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009870 tx=0x7f1f6002ba20 comp rx=0 tx=0).stop 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 -- 172.21.15.125:0/3100757412 shutdown_connections 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf290 0x7f1f780cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/3100757412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815e430 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 -- 172.21.15.125:0/3100757412 >> 172.21.15.125:0/3100757412 conn(0x7f1f7806a800 msgr2=0x7f1f7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 -- 172.21.15.125:0/3100757412 shutdown_connections 2024-01-16T15:04:58.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.953+0000 7f1f7ce00640 1 -- 172.21.15.125:0/3100757412 wait complete. 2024-01-16T15:04:58.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.955+0000 7f1f7ce00640 1 Processor -- start 2024-01-16T15:04:58.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.955+0000 7f1f7ce00640 1 -- start start 2024-01-16T15:04:58.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780ae5a0 0x7f1f78148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f780cf290 0x7f1f78159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815f890 con 0x7f1f780b3ae0 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815fb70 con 0x7f1f780cf290 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f7ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815fe50 con 0x7f1f780ae5a0 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f75400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f75400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60070/0 (socket says 172.21.15.125:60070) 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f75400640 1 -- 172.21.15.125:0/163383619 learned_addr learned my addr 172.21.15.125:0/163383619 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f76800640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f780cf290 0x7f1f78159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.956+0000 7f1f75e00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780ae5a0 0x7f1f78148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 -- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780ae5a0 msgr2=0x7f1f78148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780ae5a0 0x7f1f78148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 -- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f780cf290 msgr2=0x7f1f78159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f780cf290 0x7f1f78159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 -- 172.21.15.125:0/163383619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600094d0 con 0x7f1f780b3ae0 2024-01-16T15:04:58.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f75400640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1f60000c00 tx=0x7f1f60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60037c40 con 0x7f1f780b3ae0 2024-01-16T15:04:58.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60037da0 con 0x7f1f780b3ae0 2024-01-16T15:04:58.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60041640 con 0x7f1f780b3ae0 2024-01-16T15:04:58.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78160130 con 0x7f1f780b3ae0 2024-01-16T15:04:58.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.957+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f78160930 con 0x7f1f780b3ae0 2024-01-16T15:04:58.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.959+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f78160650 con 0x7f1f780b3ae0 2024-01-16T15:04:58.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.959+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f1f600417a0 con 0x7f1f780b3ae0 2024-01-16T15:04:58.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.960+0000 7f1f6e000640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 0x7f1f5006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.960+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1f6003c070 con 0x7f1f780b3ae0 2024-01-16T15:04:58.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.960+0000 7f1f75e00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 0x7f1f5006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.961+0000 7f1f75e00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 0x7f1f5006dff0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000f60 tx=0x7f1f68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:58.965+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f78160650 con 0x7f1f780b3ae0 2024-01-16T15:04:59.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:04:59.163+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1f78160650 con 0x7f1f780b3ae0 2024-01-16T15:05:00.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.326+0000 7f1f6e000640 1 -- 172.21.15.125:0/163383619 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v66) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1f78160650 con 0x7f1f780b3ae0 2024-01-16T15:05:00.306 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T15:05:00.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.329+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 msgr2=0x7f1f5006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.329+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 0x7f1f5006dff0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000f60 tx=0x7f1f68009040 comp rx=0 tx=0).stop 2024-01-16T15:05:00.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.329+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 msgr2=0x7f1f7815ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1f60000c00 tx=0x7f1f60004730 comp rx=0 tx=0).stop 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 shutdown_connections 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f780cf290 0x7f1f78159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006bb30 0x7f1f5006dff0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780b3ae0 0x7f1f7815ef20 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 --2- 172.21.15.125:0/163383619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780ae5a0 0x7f1f78148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 >> 172.21.15.125:0/163383619 conn(0x7f1f7806a800 msgr2=0x7f1f7815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 shutdown_connections 2024-01-16T15:05:00.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.330+0000 7f1f7ce00640 1 -- 172.21.15.125:0/163383619 wait complete. 2024-01-16T15:05:00.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:05:00.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.552+0000 7f9028800640 1 Processor -- start 2024-01-16T15:05:00.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.552+0000 7f9028800640 1 -- start start 2024-01-16T15:05:00.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9028800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90200ae5a0 0x7f902015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9028800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9028800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9020070c40 con 0x7f902006d120 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9028800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9020071450 con 0x7f90200ae5a0 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9028800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902006c470 con 0x7f90200b3ae0 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9025e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90200ae5a0 0x7f902015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9026800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f902006d120 0x7f90200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45722/0 (socket says 172.21.15.125:45722) 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9026800640 1 -- 172.21.15.125:0/1343833874 learned_addr learned my addr 172.21.15.125:0/1343833874 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:00.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.554+0000 7f9025400640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9024a00640 1 -- 172.21.15.125:0/1343833874 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394390816 0 0) 0x7f9020070c40 con 0x7f902006d120 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9024a00640 1 -- 172.21.15.125:0/1343833874 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9000003520 con 0x7f902006d120 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9025e00640 1 -- 172.21.15.125:0/1343833874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 msgr2=0x7f902006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9025e00640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9025e00640 1 -- 172.21.15.125:0/1343833874 >> v1:172.21.15.125:6789/0 conn(0x7f902006d120 legacy=0x7f90200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9025e00640 1 -- 172.21.15.125:0/1343833874 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902006c750 con 0x7f90200ae5a0 2024-01-16T15:05:00.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.555+0000 7f9025400640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:00.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.556+0000 7f9025e00640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015e430 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f90100097a0 tx=0x7f901002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce5758d057df0b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9024a00640 1 -- 172.21.15.125:0/1343833874 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010031830 con 0x7f90200ae5a0 2024-01-16T15:05:00.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9024a00640 1 -- 172.21.15.125:0/1343833874 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9010031990 con 0x7f90200ae5a0 2024-01-16T15:05:00.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9024a00640 1 -- 172.21.15.125:0/1343833874 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010031d20 con 0x7f90200ae5a0 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 -- 172.21.15.125:0/1343833874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 msgr2=0x7f902015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015e430 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f90100097a0 tx=0x7f901002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 -- 172.21.15.125:0/1343833874 shutdown_connections 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 --2- 172.21.15.125:0/1343833874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015e430 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.557+0000 7f9028800640 1 -- 172.21.15.125:0/1343833874 >> 172.21.15.125:0/1343833874 conn(0x7f902006a800 msgr2=0x7f902006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.558+0000 7f9028800640 1 -- 172.21.15.125:0/1343833874 shutdown_connections 2024-01-16T15:05:00.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.558+0000 7f9028800640 1 -- 172.21.15.125:0/1343833874 wait complete. 2024-01-16T15:05:00.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.559+0000 7f9028800640 1 Processor -- start 2024-01-16T15:05:00.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.559+0000 7f9028800640 1 -- start start 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f902006d120 0x7f90200cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90200cbf10 con 0x7f90200ae5a0 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902015d320 con 0x7f902006d120 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.560+0000 7f9028800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902015c860 con 0x7f90200b3ae0 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60084/0 (socket says 172.21.15.125:60084) 2024-01-16T15:05:00.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 -- 172.21.15.125:0/109806071 learned_addr learned my addr 172.21.15.125:0/109806071 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 -- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 msgr2=0x7f902015c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 -- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f902006d120 msgr2=0x7f90200cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f902006d120 0x7f90200cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 -- 172.21.15.125:0/109806071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010009400 con 0x7f90200ae5a0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.561+0000 7f9025400640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f900800e7e0 tx=0x7f900800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.562+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008004030 con 0x7f90200ae5a0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.562+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008004190 con 0x7f90200ae5a0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.562+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008004520 con 0x7f90200ae5a0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.562+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902015cb40 con 0x7f90200ae5a0 2024-01-16T15:05:00.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.562+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90200cbba0 con 0x7f90200ae5a0 2024-01-16T15:05:00.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.563+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90200add90 con 0x7f90200ae5a0 2024-01-16T15:05:00.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.564+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f900800c6d0 con 0x7f90200ae5a0 2024-01-16T15:05:00.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.565+0000 7f901e000640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 0x7f8ff806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.565+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f90080882d0 con 0x7f90200ae5a0 2024-01-16T15:05:00.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.565+0000 7f9025e00640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 0x7f8ff806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.566+0000 7f9025e00640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 0x7f8ff806e040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9010036000 tx=0x7f9010030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.569+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90080549c0 con 0x7f90200ae5a0 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.824+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f90200675d0 con 0x7f90200ae5a0 2024-01-16T15:05:00.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.825+0000 7f901e000640 1 -- 172.21.15.125:0/109806071 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9008057fe0 con 0x7f90200ae5a0 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.810 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:05:00.815 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T15:05:00.817 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:05:00.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 msgr2=0x7f8ff806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 0x7f8ff806e040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9010036000 tx=0x7f9010030000 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 msgr2=0x7f902015bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f900800e7e0 tx=0x7f900800eca0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 shutdown_connections 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90200b3ae0 0x7f902015c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8ff806bb80 0x7f8ff806e040 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90200ae5a0 0x7f902015bcf0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 --2- 172.21.15.125:0/109806071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f902006d120 0x7f90200cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 >> 172.21.15.125:0/109806071 conn(0x7f902006a800 msgr2=0x7f902006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 shutdown_connections 2024-01-16T15:05:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:00.828+0000 7f9028800640 1 -- 172.21.15.125:0/109806071 wait complete. 2024-01-16T15:05:00.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:01.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.041+0000 7f7143e00640 1 Processor -- start 2024-01-16T15:05:01.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.041+0000 7f7143e00640 1 -- start start 2024-01-16T15:05:01.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7143e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c13e360 0x7f713c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7143e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f713c15f9c0 0x7f713c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7143e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c15c5f0 con 0x7f713c13df80 2024-01-16T15:05:01.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7143e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713c06c470 con 0x7f713c15f9c0 2024-01-16T15:05:01.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7143e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c06c750 con 0x7f713c13e360 2024-01-16T15:05:01.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7141e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f713c15f9c0 0x7f713c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7141e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f713c15f9c0 0x7f713c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60096/0 (socket says 172.21.15.125:60096) 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7141e00640 1 -- 172.21.15.125:0/322225092 learned_addr learned my addr 172.21.15.125:0/322225092 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.043+0000 7f7141400640 1 --1- 172.21.15.125:0/322225092 >> v1:172.21.15.125:6789/0 conn(0x7f713c13df80 0x7f713c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45724/0 (socket says 172.21.15.125:45724) 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f7140a00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c13e360 0x7f713c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f7141e00640 1 -- 172.21.15.125:0/322225092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c13e360 msgr2=0x7f713c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f7141e00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c13e360 0x7f713c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f7141e00640 1 -- 172.21.15.125:0/322225092 >> v1:172.21.15.125:6789/0 conn(0x7f713c13df80 legacy=0x7f713c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f7141e00640 1 -- 172.21.15.125:0/322225092 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c06ca30 con 0x7f713c15f9c0 2024-01-16T15:05:01.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.044+0000 7f713be00640 1 -- 172.21.15.125:0/322225092 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67971729 0 0) 0x7f713c15c5f0 con 0x7f713c13df80 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.045+0000 7f7141e00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c15f9c0 0x7f713c0cf7d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e50 tx=0x7f713402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9d4f60c518faa3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.045+0000 7f713be00640 1 -- 172.21.15.125:0/322225092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713402cb00 con 0x7f713c15f9c0 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.045+0000 7f713be00640 1 -- 172.21.15.125:0/322225092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713402cc60 con 0x7f713c15f9c0 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.045+0000 7f713be00640 1 -- 172.21.15.125:0/322225092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71340343f0 con 0x7f713c15f9c0 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 -- 172.21.15.125:0/322225092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c15f9c0 msgr2=0x7f713c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c15f9c0 0x7f713c0cf7d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e50 tx=0x7f713402bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 -- 172.21.15.125:0/322225092 shutdown_connections 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c15f9c0 0x7f713c0cf7d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 --2- 172.21.15.125:0/322225092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c13e360 0x7f713c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 -- 172.21.15.125:0/322225092 >> 172.21.15.125:0/322225092 conn(0x7f713c06a800 msgr2=0x7f713c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 -- 172.21.15.125:0/322225092 shutdown_connections 2024-01-16T15:05:01.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.046+0000 7f7143e00640 1 -- 172.21.15.125:0/322225092 wait complete. 2024-01-16T15:05:01.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.048+0000 7f7143e00640 1 Processor -- start 2024-01-16T15:05:01.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.048+0000 7f7143e00640 1 -- start start 2024-01-16T15:05:01.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c13df80 0x7f713c0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15f9c0 0x7f713c0adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c12f630 con 0x7f713c13e360 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0ac700 con 0x7f713c13df80 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7143e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c12f880 con 0x7f713c15f9c0 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7140a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7140a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60106/0 (socket says 172.21.15.125:60106) 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.049+0000 7f7140a00640 1 -- 172.21.15.125:0/2728005842 learned_addr learned my addr 172.21.15.125:0/2728005842 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 -- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15f9c0 msgr2=0x7f713c0adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15f9c0 0x7f713c0adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 -- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c13df80 msgr2=0x7f713c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7141400640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c13df80 0x7f713c0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c13df80 0x7f713c0ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 -- 172.21.15.125:0/2728005842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7134009b30 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7140a00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f712400d800 tx=0x7f712400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712400b6c0 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712400b820 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c12fb60 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712400bbb0 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.050+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c130120 con 0x7f713c13e360 2024-01-16T15:05:01.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.051+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c0675d0 con 0x7f713c13e360 2024-01-16T15:05:01.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.053+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f712400bd10 con 0x7f713c13e360 2024-01-16T15:05:01.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.054+0000 7f7139600640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 0x7f711406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.054+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7124088b40 con 0x7f713c13e360 2024-01-16T15:05:01.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.054+0000 7f7141400640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 0x7f711406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.055+0000 7f7141400640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 0x7f711406dff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.058+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71240bde90 con 0x7f713c13e360 2024-01-16T15:05:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.250+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f713c15eb50 con 0x7f713c13e360 2024-01-16T15:05:01.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.252+0000 7f7139600640 1 -- 172.21.15.125:0/2728005842 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f7124055290 con 0x7f713c13e360 2024-01-16T15:05:01.231 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:01.232 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":66,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:05:00.324251+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.254+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 msgr2=0x7f711406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.254+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 0x7f711406dff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.254+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 msgr2=0x7f713c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.254+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f712400d800 tx=0x7f712400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 shutdown_connections 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15f9c0 0x7f713c0adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f711406bb30 0x7f711406dff0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f713c13e360 0x7f713c132b20 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 --2- 172.21.15.125:0/2728005842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c13df80 0x7f713c0ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 >> 172.21.15.125:0/2728005842 conn(0x7f713c06a800 msgr2=0x7f713c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 shutdown_connections 2024-01-16T15:05:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.255+0000 7f7143e00640 1 -- 172.21.15.125:0/2728005842 wait complete. 2024-01-16T15:05:01.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:01.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:01.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.486+0000 7fd665a00640 1 Processor -- start 2024-01-16T15:05:01.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.486+0000 7fd665a00640 1 -- start start 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.488+0000 7fd665a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd660164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.488+0000 7fd665a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600b1ee0 con 0x7fd6600cf9a0 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.488+0000 7fd665a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600b2780 con 0x7fd660143ae0 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.488+0000 7fd665a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66015e990 con 0x7fd66015e3b0 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd657e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd6600cf9a0 0x7fd66015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45736/0 (socket says 172.21.15.125:45736) 2024-01-16T15:05:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd657e00640 1 -- 172.21.15.125:0/4009682361 learned_addr learned my addr 172.21.15.125:0/4009682361 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65f400640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd660164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341705275 0 0) 0x7fd6600b1ee0 con 0x7fd6600cf9a0 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd640003520 con 0x7fd6600cf9a0 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583253908 0 0) 0x7fd640003520 con 0x7fd6600cf9a0 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 msgr2=0x7fd660164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd660164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 >> v1:172.21.15.137:6789/0 conn(0x7fd660143ae0 legacy=0x7fd6600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600b2130 con 0x7fd6600cf9a0 2024-01-16T15:05:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.489+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd648001670 con 0x7fd6600cf9a0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 >> v1:172.21.15.125:6789/0 conn(0x7fd6600cf9a0 legacy=0x7fd66015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd640003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd6400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd640004d00 0x7fd640005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66015e990 con 0x7fd6400039a0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64000bb10 con 0x7fd640004d00 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64000dee0 con 0x7fd66015e3b0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65ea00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd640003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd657e00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd6400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65f400640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd640004d00 0x7fd640005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65ea00640 1 -- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 msgr2=0x7fd6400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65ea00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd6400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65ea00640 1 -- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd640004d00 msgr2=0x7fd640005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.490+0000 7fd65ea00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd640004d00 0x7fd640005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.491+0000 7fd65ea00640 1 -- 172.21.15.125:0/4009682361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd640003520 con 0x7fd6400039a0 2024-01-16T15:05:01.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.491+0000 7fd657e00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd66015e3b0 0x7fd6400047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.492+0000 7fd65ea00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd640003dc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd650001750 tx=0x7fd6400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c5c509a1194e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.492+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500052d0 con 0x7fd6400039a0 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.492+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65002e050 con 0x7fd6400039a0 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.492+0000 7fd65e000640 1 -- 172.21.15.125:0/4009682361 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500035f0 con 0x7fd6400039a0 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 -- 172.21.15.125:0/4009682361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 msgr2=0x7fd640003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 --2- 172.21.15.125:0/4009682361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd640003dc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd650001750 tx=0x7fd6400057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd65f400640 1 -- 172.21.15.125:0/4009682361 reap_dead start 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 -- 172.21.15.125:0/4009682361 shutdown_connections 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 -- 172.21.15.125:0/4009682361 >> 172.21.15.125:0/4009682361 conn(0x7fd66006a800 msgr2=0x7fd66006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 -- 172.21.15.125:0/4009682361 shutdown_connections 2024-01-16T15:05:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.493+0000 7fd665a00640 1 -- 172.21.15.125:0/4009682361 wait complete. 2024-01-16T15:05:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.495+0000 7fd665a00640 1 Processor -- start 2024-01-16T15:05:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.495+0000 7fd665a00640 1 -- start start 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 0x7fd660137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 0x7fd66013d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601597c0 con 0x7fd6400039a0 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601f2cb0 con 0x7fd6601f3140 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd665a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd660138570 con 0x7fd660159f20 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd65ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd65ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60128/0 (socket says 172.21.15.125:60128) 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.496+0000 7fd65ea00640 1 -- 172.21.15.125:0/2707386861 learned_addr learned my addr 172.21.15.125:0/2707386861 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd657e00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 0x7fd660137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65f400640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 0x7fd66013d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 -- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 msgr2=0x7fd660137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 0x7fd660137c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 -- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 msgr2=0x7fd66013d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 0x7fd66013d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 -- 172.21.15.125:0/2707386861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6601f2660 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd657e00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 0x7fd660137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65f400640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 0x7fd66013d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd65ea00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd65002bc30 tx=0x7fd650001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500076c0 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd650007a30 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650006be0 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.497+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6601f2940 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.498+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6601381b0 con 0x7fd6400039a0 2024-01-16T15:05:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.499+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66006c3f0 con 0x7fd6400039a0 2024-01-16T15:05:01.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.500+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fd650006d40 con 0x7fd6400039a0 2024-01-16T15:05:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.501+0000 7fd656a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 0x7fd63806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.501+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6500aa250 con 0x7fd6400039a0 2024-01-16T15:05:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.501+0000 7fd657e00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 0x7fd63806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.502+0000 7fd657e00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 0x7fd63806e020 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd648001eb0 tx=0x7fd648001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.506+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6500769a0 con 0x7fd6400039a0 2024-01-16T15:05:01.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.693+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6600675d0 con 0x7fd6400039a0 2024-01-16T15:05:01.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.694+0000 7fd656a00640 1 -- 172.21.15.125:0/2707386861 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd650079fc0 con 0x7fd6400039a0 2024-01-16T15:05:01.674 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:05:01.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 msgr2=0x7fd63806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 0x7fd63806e020 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd648001eb0 tx=0x7fd648001150 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 msgr2=0x7fd660159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd65002bc30 tx=0x7fd650001120 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 shutdown_connections 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6601f3140 0x7fd66013d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd660159f20 0x7fd660137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd63806bb60 0x7fd63806e020 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 --2- 172.21.15.125:0/2707386861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6400039a0 0x7fd660159280 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 >> 172.21.15.125:0/2707386861 conn(0x7fd66006a800 msgr2=0x7fd66015bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 shutdown_connections 2024-01-16T15:05:01.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.697+0000 7fd665a00640 1 -- 172.21.15.125:0/2707386861 wait complete. 2024-01-16T15:05:01.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:05:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.925+0000 7fbc37000640 1 Processor -- start 2024-01-16T15:05:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.925+0000 7fbc37000640 1 -- start start 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.926+0000 7fbc37000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc30143ae0 0x7fbc300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.926+0000 7fbc37000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 0x7fbc300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.926+0000 7fbc37000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc300cfc30 con 0x7fbc300cf7d0 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.926+0000 7fbc37000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc300b1ee0 con 0x7fbc30143ae0 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.926+0000 7fbc37000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300b2780 con 0x7fbc3015e3b0 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc30143ae0 0x7fbc300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc30143ae0 0x7fbc300cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60136/0 (socket says 172.21.15.125:60136) 2024-01-16T15:05:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 -- 172.21.15.125:0/3630636069 learned_addr learned my addr 172.21.15.125:0/3630636069 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:01.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc35000640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 0x7fbc300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 -- 172.21.15.125:0/3630636069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 msgr2=0x7fbc300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 0x7fbc300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 -- 172.21.15.125:0/3630636069 >> v1:172.21.15.125:6789/0 conn(0x7fbc300cf7d0 legacy=0x7fbc3015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2fe00640 1 -- 172.21.15.125:0/3630636069 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3015e990 con 0x7fbc30143ae0 2024-01-16T15:05:01.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc35000640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 0x7fbc300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:01.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.927+0000 7fbc2ea00640 1 -- 172.21.15.125:0/3630636069 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484985833 0 0) 0x7fbc300cfc30 con 0x7fbc300cf7d0 2024-01-16T15:05:01.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.928+0000 7fbc2fe00640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc30143ae0 0x7fbc300cf290 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbc200098a0 tx=0x7fbc2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad5d3534cf54b6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc2ea00640 1 -- 172.21.15.125:0/3630636069 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20030840 con 0x7fbc30143ae0 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc2ea00640 1 -- 172.21.15.125:0/3630636069 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc200309a0 con 0x7fbc30143ae0 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc2ea00640 1 -- 172.21.15.125:0/3630636069 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20030d30 con 0x7fbc30143ae0 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 -- 172.21.15.125:0/3630636069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc30143ae0 msgr2=0x7fbc300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc30143ae0 0x7fbc300cf290 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbc200098a0 tx=0x7fbc2002b980 comp rx=0 tx=0).stop 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 -- 172.21.15.125:0/3630636069 shutdown_connections 2024-01-16T15:05:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc3015e3b0 0x7fbc300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 --2- 172.21.15.125:0/3630636069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc30143ae0 0x7fbc300cf290 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 -- 172.21.15.125:0/3630636069 >> 172.21.15.125:0/3630636069 conn(0x7fbc3006a800 msgr2=0x7fbc3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 -- 172.21.15.125:0/3630636069 shutdown_connections 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.929+0000 7fbc37000640 1 -- 172.21.15.125:0/3630636069 wait complete. 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.931+0000 7fbc37000640 1 Processor -- start 2024-01-16T15:05:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.931+0000 7fbc37000640 1 -- start start 2024-01-16T15:05:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc30143ae0 0x7fbc30165df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc3015e3b0 0x7fbc30159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30166670 con 0x7fbc300cf7d0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc301668c0 con 0x7fbc3015e3b0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc37000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30166ba0 con 0x7fbc30143ae0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60148/0 (socket says 172.21.15.125:60148) 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.932+0000 7fbc2fe00640 1 -- 172.21.15.125:0/2959222472 learned_addr learned my addr 172.21.15.125:0/2959222472 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc35000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc3015e3b0 0x7fbc30159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2f400640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc30143ae0 0x7fbc30165df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 -- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc30143ae0 msgr2=0x7fbc30165df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc30143ae0 0x7fbc30165df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 -- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc3015e3b0 msgr2=0x7fbc30159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc3015e3b0 0x7fbc30159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 -- 172.21.15.125:0/2959222472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200094d0 con 0x7fbc300cf7d0 2024-01-16T15:05:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc2fe00640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.933+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20030810 con 0x7fbc300cf7d0 2024-01-16T15:05:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.934+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20043400 con 0x7fbc300cf7d0 2024-01-16T15:05:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.934+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc200437d0 con 0x7fbc300cf7d0 2024-01-16T15:05:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.934+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc30166e80 con 0x7fbc300cf7d0 2024-01-16T15:05:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.934+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc30167440 con 0x7fbc300cf7d0 2024-01-16T15:05:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.935+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc300675d0 con 0x7fbc300cf7d0 2024-01-16T15:05:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.936+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fbc20043930 con 0x7fbc300cf7d0 2024-01-16T15:05:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.936+0000 7fbc27e00640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 0x7fbc0806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.937+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbc2003e070 con 0x7fbc300cf7d0 2024-01-16T15:05:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.937+0000 7fbc2f400640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 0x7fbc0806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.937+0000 7fbc2f400640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 0x7fbc0806e020 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbc3006cfc0 tx=0x7fbc18009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:01.941+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc20078180 con 0x7fbc300cf7d0 2024-01-16T15:05:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.134+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbc30167600 con 0x7fbc300cf7d0 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.135+0000 7fbc27e00640 1 -- 172.21.15.125:0/2959222472 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbc20034090 con 0x7fbc300cf7d0 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:02.115 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 msgr2=0x7fbc0806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 0x7fbc0806e020 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbc3006cfc0 tx=0x7fbc18009040 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 msgr2=0x7fbc30167960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc20004730 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 shutdown_connections 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc3015e3b0 0x7fbc30159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc30143ae0 0x7fbc30165df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbc0806bb60 0x7fbc0806e020 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 --2- 172.21.15.125:0/2959222472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc300cf7d0 0x7fbc30167960 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 >> 172.21.15.125:0/2959222472 conn(0x7fbc3006a800 msgr2=0x7fbc300cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:02.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 shutdown_connections 2024-01-16T15:05:02.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.138+0000 7fbc37000640 1 -- 172.21.15.125:0/2959222472 wait complete. 2024-01-16T15:05:02.135 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:05:02.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:05:02.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.369+0000 7f2935400640 1 Processor -- start 2024-01-16T15:05:02.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.369+0000 7f2935400640 1 -- start start 2024-01-16T15:05:02.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f2935400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 0x7f293015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f2935400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300b1ee0 con 0x7f2930143ae0 2024-01-16T15:05:02.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f2935400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300b2780 con 0x7f293015e1e0 2024-01-16T15:05:02.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f2935400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293015e990 con 0x7f29300cf9a0 2024-01-16T15:05:02.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f292e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2930143ae0 0x7f29300cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45752/0 (socket says 172.21.15.125:45752) 2024-01-16T15:05:02.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.371+0000 7f292e800640 1 -- 172.21.15.125:0/2852287012 learned_addr learned my addr 172.21.15.125:0/2852287012 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f2927e00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 0x7f293015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049032313 0 0) 0x7f29300b1ee0 con 0x7f2930143ae0 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910003520 con 0x7f2930143ae0 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348805458 0 0) 0x7f2910003520 con 0x7f2930143ae0 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 msgr2=0x7f293015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 0x7f293015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 >> v1:172.21.15.137:6789/0 conn(0x7f293015e1e0 legacy=0x7f29300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300b2130 con 0x7f2930143ae0 2024-01-16T15:05:02.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2920001670 con 0x7f2930143ae0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 >> v1:172.21.15.125:6789/0 conn(0x7f2930143ae0 legacy=0x7f29300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f2910003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.372+0000 7f292de00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29100047c0 0x7f2910004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292de00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 0x7f2910005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293015e990 con 0x7f29100039a0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291000ba60 con 0x7f29100047c0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291000de30 con 0x7f29300cf9a0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f2910003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f2927e00640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29100047c0 0x7f2910004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 -- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 msgr2=0x7f2910005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300cf9a0 0x7f2910005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 -- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29100047c0 msgr2=0x7f2910004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29100047c0 0x7f2910004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.373+0000 7f292e800640 1 -- 172.21.15.125:0/2852287012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910003520 con 0x7f29100039a0 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.374+0000 7f292e800640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f2910003dc0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2920007fb0 tx=0x7f29100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8046c5f6cbc4abe0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920005ad0 con 0x7f29100039a0 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920005c30 con 0x7f29100039a0 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f292de00640 1 -- 172.21.15.125:0/2852287012 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292002c790 con 0x7f29100039a0 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 -- 172.21.15.125:0/2852287012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 msgr2=0x7f2910003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 --2- 172.21.15.125:0/2852287012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f2910003dc0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2920007fb0 tx=0x7f29100057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f292f200640 1 -- 172.21.15.125:0/2852287012 reap_dead start 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 -- 172.21.15.125:0/2852287012 shutdown_connections 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 -- 172.21.15.125:0/2852287012 >> 172.21.15.125:0/2852287012 conn(0x7f293006a800 msgr2=0x7f293006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 -- 172.21.15.125:0/2852287012 shutdown_connections 2024-01-16T15:05:02.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.375+0000 7f2935400640 1 -- 172.21.15.125:0/2852287012 wait complete. 2024-01-16T15:05:02.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.377+0000 7f2935400640 1 Processor -- start 2024-01-16T15:05:02.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.377+0000 7f2935400640 1 -- start start 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29301596e0 0x7f29300cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300b1220 0x7f29300b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29300b0e00 con 0x7f29100039a0 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293013e7a0 con 0x7f29301596e0 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f2935400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29300b0180 con 0x7f29300b1220 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.378+0000 7f292e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60164/0 (socket says 172.21.15.125:60164) 2024-01-16T15:05:02.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 -- 172.21.15.125:0/823152089 learned_addr learned my addr 172.21.15.125:0/823152089 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292f200640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300b1220 0x7f29300b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f2927e00640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29301596e0 0x7f29300cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 -- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300b1220 msgr2=0x7f29300b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300b1220 0x7f29300b1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 -- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29301596e0 msgr2=0x7f29300cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29301596e0 0x7f29300cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 -- 172.21.15.125:0/823152089 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300b0460 con 0x7f29100039a0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.379+0000 7f292e800640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f292002c330 tx=0x7f29300b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.380+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292002cba0 con 0x7f29100039a0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.380+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292002cd00 con 0x7f29100039a0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.380+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920007290 con 0x7f29100039a0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.380+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29300b0740 con 0x7f29100039a0 2024-01-16T15:05:02.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.380+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29300cdef0 con 0x7f29100039a0 2024-01-16T15:05:02.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.381+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930067000 con 0x7f29100039a0 2024-01-16T15:05:02.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.382+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2920032070 con 0x7f29100039a0 2024-01-16T15:05:02.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.383+0000 7f2926a00640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 0x7f290806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.383+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f29200a9650 con 0x7f29100039a0 2024-01-16T15:05:02.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.383+0000 7f2927e00640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 0x7f290806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.384+0000 7f2927e00640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 0x7f290806e020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f29300aff70 tx=0x7f2918008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.387+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2920075d40 con 0x7f29100039a0 2024-01-16T15:05:02.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:02.582+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f29300675d0 con 0x7f29100039a0 2024-01-16T15:05:03.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.353+0000 7f2926a00640 1 -- 172.21.15.125:0/823152089 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2920045070 con 0x7f29100039a0 2024-01-16T15:05:03.333 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 msgr2=0x7f290806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 0x7f290806e020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f29300aff70 tx=0x7f2918008040 comp rx=0 tx=0).stop 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 msgr2=0x7f293013e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f292002c330 tx=0x7f29300b26f0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 shutdown_connections 2024-01-16T15:05:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29300b1220 0x7f29300b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29301596e0 0x7f29300cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f290806bb60 0x7f290806e020 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 --2- 172.21.15.125:0/823152089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29100039a0 0x7f293013e260 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.357+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 >> 172.21.15.125:0/823152089 conn(0x7f293006a800 msgr2=0x7f2930142a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.358+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 shutdown_connections 2024-01-16T15:05:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.358+0000 7f2935400640 1 -- 172.21.15.125:0/823152089 wait complete. 2024-01-16T15:05:03.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:05:03.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.579+0000 7f6000e00640 1 Processor -- start 2024-01-16T15:05:03.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.579+0000 7f6000e00640 1 -- start start 2024-01-16T15:05:03.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f6000e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5ffc0b19a0 0x7f5ffc0cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f6000e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ffc0cce20 0x7f5ffc15c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f6000e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc0cef50 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f6000e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ffc0cc8a0 con 0x7f5ffc0cce20 2024-01-16T15:05:03.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f6000e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ffc0cd2b0 con 0x7f5ffc0b19a0 2024-01-16T15:05:03.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f5ffa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ffc0cce20 0x7f5ffc15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f5ffa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ffc0cce20 0x7f5ffc15c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60180/0 (socket says 172.21.15.125:60180) 2024-01-16T15:05:03.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f5ffa800640 1 -- 172.21.15.125:0/3960737595 learned_addr learned my addr 172.21.15.125:0/3960737595 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:03.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.581+0000 7f5ff9e00640 1 --1- 172.21.15.125:0/3960737595 >> v1:172.21.15.125:6789/0 conn(0x7f5ffc0cbaf0 0x7f5ffc0ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45766/0 (socket says 172.21.15.125:45766) 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ff3e00640 1 --2- 172.21.15.125:0/3960737595 >> v2:172.21.15.137:3300/0 conn(0x7f5ffc0b19a0 0x7f5ffc0cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ffa800640 1 -- 172.21.15.125:0/3960737595 >> v2:172.21.15.137:3300/0 conn(0x7f5ffc0b19a0 msgr2=0x7f5ffc0cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ffa800640 1 --2- 172.21.15.125:0/3960737595 >> v2:172.21.15.137:3300/0 conn(0x7f5ffc0b19a0 0x7f5ffc0cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ffa800640 1 -- 172.21.15.125:0/3960737595 >> v1:172.21.15.125:6789/0 conn(0x7f5ffc0cbaf0 legacy=0x7f5ffc0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ffa800640 1 -- 172.21.15.125:0/3960737595 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0cc5c0 con 0x7f5ffc0cce20 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.582+0000 7f5ff9400640 1 -- 172.21.15.125:0/3960737595 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656741775 0 0) 0x7f5ffc0cef50 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.583+0000 7f5ffa800640 1 --2- 172.21.15.125:0/3960737595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cce20 0x7f5ffc15c100 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc06f430 tx=0x7f5ff402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad1b11e2e41427c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.583+0000 7f5ff9400640 1 -- 172.21.15.125:0/3960737595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff402cbb0 con 0x7f5ffc0cce20 2024-01-16T15:05:03.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.583+0000 7f5ff9400640 1 -- 172.21.15.125:0/3960737595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff402cd10 con 0x7f5ffc0cce20 2024-01-16T15:05:03.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.583+0000 7f5ff9400640 1 -- 172.21.15.125:0/3960737595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4034610 con 0x7f5ffc0cce20 2024-01-16T15:05:03.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 -- 172.21.15.125:0/3960737595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cce20 msgr2=0x7f5ffc15c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 --2- 172.21.15.125:0/3960737595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cce20 0x7f5ffc15c100 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc06f430 tx=0x7f5ff402c010 comp rx=0 tx=0).stop 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 -- 172.21.15.125:0/3960737595 shutdown_connections 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 --2- 172.21.15.125:0/3960737595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cce20 0x7f5ffc15c100 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 --2- 172.21.15.125:0/3960737595 >> v2:172.21.15.137:3300/0 conn(0x7f5ffc0b19a0 0x7f5ffc0cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 -- 172.21.15.125:0/3960737595 >> 172.21.15.125:0/3960737595 conn(0x7f5ffc0b0e80 msgr2=0x7f5ffc0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 -- 172.21.15.125:0/3960737595 shutdown_connections 2024-01-16T15:05:03.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.584+0000 7f6000e00640 1 -- 172.21.15.125:0/3960737595 wait complete. 2024-01-16T15:05:03.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.586+0000 7f6000e00640 1 Processor -- start 2024-01-16T15:05:03.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.586+0000 7f6000e00640 1 -- start start 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 0x7f5ffc14efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ffc0cce20 0x7f5ffc0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc1e3710 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc14f500 con 0x7f5ffc0b19a0 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f6000e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc1e2a90 con 0x7f5ffc0cce20 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f5ff3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f5ff3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60188/0 (socket says 172.21.15.125:60188) 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.587+0000 7f5ff3e00640 1 -- 172.21.15.125:0/2766597780 learned_addr learned my addr 172.21.15.125:0/2766597780 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff9e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 0x7f5ffc14efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ffa800640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ffc0cce20 0x7f5ffc0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 -- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ffc0cce20 msgr2=0x7f5ffc0c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ffc0cce20 0x7f5ffc0c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 -- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 msgr2=0x7f5ffc14efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 0x7f5ffc14efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 -- 172.21.15.125:0/2766597780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4009b30 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff9e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 0x7f5ffc14efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff3e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400d800 tx=0x7f5fe400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400b6c0 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400b820 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400bbb0 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.588+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ffc1e2d70 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.589+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ffc1e3330 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.590+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ffc06ec70 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.591+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5fe400bd10 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.592+0000 7f5ff2000640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 0x7f5fd406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.592+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5fe40896c0 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.592+0000 7f5ff9e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 0x7f5fd406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.593+0000 7f5ff9e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 0x7f5fd406dff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000f60 tx=0x7f5fec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.595+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fe4055db0 con 0x7f5ffc0cbaf0 2024-01-16T15:05:03.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:03.791+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5ffc067670 con 0x7f5ffc0cbaf0 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.371+0000 7f5ff2000640 1 -- 172.21.15.125:0/2766597780 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5fe40593d0 con 0x7f5ffc0cbaf0 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 msgr2=0x7f5fd406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 0x7f5fd406dff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000f60 tx=0x7f5fec009040 comp rx=0 tx=0).stop 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 msgr2=0x7f5ffc0c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400d800 tx=0x7f5fe400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 shutdown_connections 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ffc0cce20 0x7f5ffc0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5fd406bb30 0x7f5fd406dff0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ffc0cbaf0 0x7f5ffc0c5140 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 --2- 172.21.15.125:0/2766597780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ffc0b19a0 0x7f5ffc14efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.375+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 >> 172.21.15.125:0/2766597780 conn(0x7f5ffc0b0e80 msgr2=0x7f5ffc1595a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.376+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 shutdown_connections 2024-01-16T15:05:04.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.376+0000 7f6000e00640 1 -- 172.21.15.125:0/2766597780 wait complete. 2024-01-16T15:05:04.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:05:04.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.592+0000 7ff0ebe00640 1 Processor -- start 2024-01-16T15:05:04.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.592+0000 7ff0ebe00640 1 -- start start 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0ebe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff0e413e360 0x7ff0e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0ebe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e415c5f0 con 0x7ff0e413df80 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0ebe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e406c470 con 0x7ff0e415f9c0 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0ebe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e406c750 con 0x7ff0e413e360 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0e9400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff0e413df80 0x7ff0e415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45778/0 (socket says 172.21.15.125:45778) 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0e9400640 1 -- 172.21.15.125:0/1212127985 learned_addr learned my addr 172.21.15.125:0/1212127985 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797250752 0 0) 0x7ff0e415c5f0 con 0x7ff0e413df80 2024-01-16T15:05:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/1212127985 >> v2:172.21.15.137:3300/0 conn(0x7ff0e413e360 0x7ff0e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.594+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c4003520 con 0x7ff0e413df80 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200278217 0 0) 0x7ff0c4003520 con 0x7ff0e413df80 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 >> v2:172.21.15.137:3300/0 conn(0x7ff0e413e360 msgr2=0x7ff0e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 --2- 172.21.15.125:0/1212127985 >> v2:172.21.15.137:3300/0 conn(0x7ff0e413e360 0x7ff0e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 >> v1:172.21.15.137:6789/0 conn(0x7ff0e415f9c0 legacy=0x7ff0e40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e406ca30 con 0x7ff0e413df80 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff0d4001690 con 0x7ff0e413df80 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 >> v1:172.21.15.125:6789/0 conn(0x7ff0e413df80 legacy=0x7ff0e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0c4005180 0x7ff0c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e406c750 con 0x7ff0c40047c0 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c400c020 con 0x7ff0c4003970 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c400c1e0 con 0x7ff0c4005180 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.595+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e9400640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e8a00640 1 -- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0c4005180 msgr2=0x7ff0c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0c4005180 0x7ff0c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e8a00640 1 -- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c4003970 msgr2=0x7ff0c4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e8a00640 1 -- 172.21.15.125:0/1212127985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4003520 con 0x7ff0c40047c0 2024-01-16T15:05:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.596+0000 7ff0e9e00640 1 -- 172.21.15.125:0/1212127985 reap_dead start 2024-01-16T15:05:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.597+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009a90 tx=0x7ff0c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed00af27dbdf4ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.597+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc02c6b0 con 0x7ff0c40047c0 2024-01-16T15:05:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.597+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc02c810 con 0x7ff0c40047c0 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.597+0000 7ff0e3e00640 1 -- 172.21.15.125:0/1212127985 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc02cbc0 con 0x7ff0c40047c0 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 -- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 msgr2=0x7ff0c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009a90 tx=0x7ff0c40057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 -- 172.21.15.125:0/1212127985 shutdown_connections 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/1212127985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 -- 172.21.15.125:0/1212127985 >> 172.21.15.125:0/1212127985 conn(0x7ff0e406a800 msgr2=0x7ff0e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 -- 172.21.15.125:0/1212127985 shutdown_connections 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.598+0000 7ff0ebe00640 1 -- 172.21.15.125:0/1212127985 wait complete. 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.600+0000 7ff0ebe00640 1 Processor -- start 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.600+0000 7ff0ebe00640 1 -- start start 2024-01-16T15:05:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c40047c0 0x7ff0e413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e40cdeb0 0x7ff0e414d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e41f60a0 con 0x7ff0e414db10 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e413dd10 con 0x7ff0c40047c0 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0ebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e41f6310 con 0x7ff0e40cdeb0 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60212/0 (socket says 172.21.15.125:60212) 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 -- 172.21.15.125:0/817601651 learned_addr learned my addr 172.21.15.125:0/817601651 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 -- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e40cdeb0 msgr2=0x7ff0e414d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e8a00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e40cdeb0 0x7ff0e414d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9400640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c40047c0 0x7ff0e413d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e40cdeb0 0x7ff0e414d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 -- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c40047c0 msgr2=0x7ff0e413d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c40047c0 0x7ff0e413d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.601+0000 7ff0e9e00640 1 -- 172.21.15.125:0/817601651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e41f65f0 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0e9e00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff0e406dea0 tx=0x7ff0dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc003970 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc003ad0 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e41f68d0 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc004020 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.602+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e40b2500 con 0x7ff0e414db10 2024-01-16T15:05:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.603+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e41f6df0 con 0x7ff0e414db10 2024-01-16T15:05:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.604+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7ff0dc004180 con 0x7ff0e414db10 2024-01-16T15:05:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.605+0000 7ff0e1600640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 0x7ff0bc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.605+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff0dc087910 con 0x7ff0e414db10 2024-01-16T15:05:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.605+0000 7ff0e9400640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 0x7ff0bc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.606+0000 7ff0e9400640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 0x7ff0bc06dff0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001eb0 tx=0x7ff0d4001d60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.610+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0e41f6df0 con 0x7ff0e414db10 2024-01-16T15:05:04.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:04.839+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff0e41f6df0 con 0x7ff0e414db10 2024-01-16T15:05:05.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.385+0000 7ff0e1600640 1 -- 172.21.15.125:0/817601651 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff0e41f6df0 con 0x7ff0e414db10 2024-01-16T15:05:05.370 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 10 and data pool 11 2024-01-16T15:05:05.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.395+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 msgr2=0x7ff0bc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.395+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 0x7ff0bc06dff0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001eb0 tx=0x7ff0d4001d60 comp rx=0 tx=0).stop 2024-01-16T15:05:05.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.395+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 msgr2=0x7ff0e40afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.395+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff0e406dea0 tx=0x7ff0dc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 shutdown_connections 2024-01-16T15:05:05.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0bc06bb30 0x7ff0bc06dff0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e414db10 0x7ff0e40afc90 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e40cdeb0 0x7ff0e414d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 --2- 172.21.15.125:0/817601651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0c40047c0 0x7ff0e413d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 >> 172.21.15.125:0/817601651 conn(0x7ff0e406a800 msgr2=0x7ff0e40ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.397+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 shutdown_connections 2024-01-16T15:05:05.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.398+0000 7ff0ebe00640 1 -- 172.21.15.125:0/817601651 wait complete. 2024-01-16T15:05:05.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:05.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.599+0000 7fe8ad800640 1 Processor -- start 2024-01-16T15:05:05.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.599+0000 7fe8ad800640 1 -- start start 2024-01-16T15:05:05.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe8a813df80 0x7fe8a815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8a813e360 0x7fe8a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a815f9c0 0x7fe8a80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a806c470 con 0x7fe8a813e360 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a806c750 con 0x7fe8a813df80 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8ad800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806ca30 con 0x7fe8a815f9c0 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8a6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8a813e360 0x7fe8a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8a6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8a813e360 0x7fe8a80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60220/0 (socket says 172.21.15.125:60220) 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8a6000640 1 -- 172.21.15.125:0/1082767873 learned_addr learned my addr 172.21.15.125:0/1082767873 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8a6a00640 1 --2- 172.21.15.125:0/1082767873 >> v2:172.21.15.137:3300/0 conn(0x7fe8a813df80 0x7fe8a815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.601+0000 7fe8a7400640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a815f9c0 0x7fe8a80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.602+0000 7fe8a6000640 1 -- 172.21.15.125:0/1082767873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a815f9c0 msgr2=0x7fe8a80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.602+0000 7fe8a6000640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a815f9c0 0x7fe8a80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.602+0000 7fe8a6000640 1 -- 172.21.15.125:0/1082767873 >> v2:172.21.15.137:3300/0 conn(0x7fe8a813df80 msgr2=0x7fe8a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.602+0000 7fe8a6000640 1 --2- 172.21.15.125:0/1082767873 >> v2:172.21.15.137:3300/0 conn(0x7fe8a813df80 0x7fe8a815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.602+0000 7fe8a6000640 1 -- 172.21.15.125:0/1082767873 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a806cd10 con 0x7fe8a813e360 2024-01-16T15:05:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.603+0000 7fe8a6000640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813e360 0x7fe8a80cf290 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8072770 tx=0x7fe89002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f5ca2d29848284 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.603+0000 7fe8a5600640 1 -- 172.21.15.125:0/1082767873 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890031860 con 0x7fe8a813e360 2024-01-16T15:05:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.603+0000 7fe8a5600640 1 -- 172.21.15.125:0/1082767873 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8900319c0 con 0x7fe8a813e360 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.603+0000 7fe8a5600640 1 -- 172.21.15.125:0/1082767873 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890031d50 con 0x7fe8a813e360 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 -- 172.21.15.125:0/1082767873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813e360 msgr2=0x7fe8a80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813e360 0x7fe8a80cf290 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8072770 tx=0x7fe89002b900 comp rx=0 tx=0).stop 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 -- 172.21.15.125:0/1082767873 shutdown_connections 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a815f9c0 0x7fe8a80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 --2- 172.21.15.125:0/1082767873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813e360 0x7fe8a80cf290 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 --2- 172.21.15.125:0/1082767873 >> v2:172.21.15.137:3300/0 conn(0x7fe8a813df80 0x7fe8a815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 -- 172.21.15.125:0/1082767873 >> 172.21.15.125:0/1082767873 conn(0x7fe8a806a800 msgr2=0x7fe8a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 -- 172.21.15.125:0/1082767873 shutdown_connections 2024-01-16T15:05:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.604+0000 7fe8ad800640 1 -- 172.21.15.125:0/1082767873 wait complete. 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.606+0000 7fe8ad800640 1 Processor -- start 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.606+0000 7fe8ad800640 1 -- start start 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a813e360 0x7fe8a80aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8a815f9c0 0x7fe8a8152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80af7b0 con 0x7fe8a813df80 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80afa90 con 0x7fe8a815f9c0 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8ad800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80afd70 con 0x7fe8a813e360 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60230/0 (socket says 172.21.15.125:60230) 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 -- 172.21.15.125:0/3785088558 learned_addr learned my addr 172.21.15.125:0/3785088558 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 -- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a813e360 msgr2=0x7fe8a80aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a813e360 0x7fe8a80aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 -- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8a815f9c0 msgr2=0x7fe8a8152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8a815f9c0 0x7fe8a8152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.607+0000 7fe8a6a00640 1 -- 172.21.15.125:0/3785088558 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900094d0 con 0x7fe8a813df80 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe8a6a00640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe89800d5f0 tx=0x7fe89800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980125f0 con 0x7fe8a813df80 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898012750 con 0x7fe8a813df80 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898012b00 con 0x7fe8a813df80 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a80b0050 con 0x7fe8a813df80 2024-01-16T15:05:05.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.608+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a813bea0 con 0x7fe8a813df80 2024-01-16T15:05:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.609+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a80675d0 con 0x7fe8a813df80 2024-01-16T15:05:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.610+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe898012c60 con 0x7fe8a813df80 2024-01-16T15:05:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.611+0000 7fe89ea00640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 0x7fe88006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.611+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe898088a60 con 0x7fe8a813df80 2024-01-16T15:05:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.611+0000 7fe8a6000640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 0x7fe88006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.612+0000 7fe8a6000640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 0x7fe88006e020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe890030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.615+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8980551b0 con 0x7fe8a813df80 2024-01-16T15:05:05.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.794+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8a806df00 con 0x7fe8a813df80 2024-01-16T15:05:05.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.798+0000 7fe89ea00640 1 -- 172.21.15.125:0/3785088558 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16629 (secure 0 0 0) 0x7fe8980587d0 con 0x7fe8a813df80 2024-01-16T15:05:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:05.777 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:05:04.841883+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:02.632990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:03.846526+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:05.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 msgr2=0x7fe88006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 0x7fe88006e020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe890030000 comp rx=0 tx=0).stop 2024-01-16T15:05:05.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 msgr2=0x7fe8a813b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe89800d5f0 tx=0x7fe89800dab0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 shutdown_connections 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe8a815f9c0 0x7fe8a8152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe8a813e360 0x7fe8a80aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe88006bb60 0x7fe88006e020 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 --2- 172.21.15.125:0/3785088558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8a813df80 0x7fe8a813b680 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 >> 172.21.15.125:0/3785088558 conn(0x7fe8a806a800 msgr2=0x7fe8a813fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 shutdown_connections 2024-01-16T15:05:05.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:05.801+0000 7fe8ad800640 1 -- 172.21.15.125:0/3785088558 wait complete. 2024-01-16T15:05:05.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:06.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.026+0000 7f95dac00640 1 Processor -- start 2024-01-16T15:05:06.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.026+0000 7f95dac00640 1 -- start start 2024-01-16T15:05:06.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95dac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f95d413df80 0x7f95d415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95dac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95dac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d406c470 con 0x7f95d413e360 2024-01-16T15:05:06.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95dac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d406c750 con 0x7f95d413df80 2024-01-16T15:05:06.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95dac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d406ca30 con 0x7f95d40cf9a0 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95d8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95d3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f95d413df80 0x7f95d415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95d8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:36866/0 (socket says 172.21.15.125:36866) 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.027+0000 7f95d8c00640 1 -- 172.21.15.125:0/4202359118 learned_addr learned my addr 172.21.15.125:0/4202359118 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.028+0000 7f95d8c00640 1 -- 172.21.15.125:0/4202359118 >> v2:172.21.15.137:3300/0 conn(0x7f95d413df80 msgr2=0x7f95d415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.028+0000 7f95d8c00640 1 --2- 172.21.15.125:0/4202359118 >> v2:172.21.15.137:3300/0 conn(0x7f95d413df80 0x7f95d415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.028+0000 7f95d8c00640 1 -- 172.21.15.125:0/4202359118 >> v1:172.21.15.137:6789/0 conn(0x7f95d413e360 legacy=0x7f95d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.028+0000 7f95d8c00640 1 -- 172.21.15.125:0/4202359118 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d406cd10 con 0x7f95d40cf9a0 2024-01-16T15:05:06.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.029+0000 7f95d8c00640 1 --2- 172.21.15.125:0/4202359118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f7119db56ff943 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.030+0000 7f95d2a00640 1 -- 172.21.15.125:0/4202359118 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc02cb00 con 0x7f95d40cf9a0 2024-01-16T15:05:06.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.030+0000 7f95d2a00640 1 -- 172.21.15.125:0/4202359118 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc02cc60 con 0x7f95d40cf9a0 2024-01-16T15:05:06.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.030+0000 7f95d2a00640 1 -- 172.21.15.125:0/4202359118 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0343f0 con 0x7f95d40cf9a0 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 -- 172.21.15.125:0/4202359118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 msgr2=0x7f95d40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 --2- 172.21.15.125:0/4202359118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 -- 172.21.15.125:0/4202359118 shutdown_connections 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 --2- 172.21.15.125:0/4202359118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d40cf9a0 0x7f95d40cd840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 --2- 172.21.15.125:0/4202359118 >> v2:172.21.15.137:3300/0 conn(0x7f95d413df80 0x7f95d415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 -- 172.21.15.125:0/4202359118 >> 172.21.15.125:0/4202359118 conn(0x7f95d406a800 msgr2=0x7f95d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 -- 172.21.15.125:0/4202359118 shutdown_connections 2024-01-16T15:05:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.031+0000 7f95dac00640 1 -- 172.21.15.125:0/4202359118 wait complete. 2024-01-16T15:05:06.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.033+0000 7f95dac00640 1 Processor -- start 2024-01-16T15:05:06.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.033+0000 7f95dac00640 1 -- start start 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 0x7f95d414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d413df80 0x7f95d413d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d413de10 con 0x7f95d413e360 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4150580 con 0x7f95d40cf9a0 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95dac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4150860 con 0x7f95d413df80 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60240/0 (socket says 172.21.15.125:60240) 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 -- 172.21.15.125:0/2198952640 learned_addr learned my addr 172.21.15.125:0/2198952640 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d3e00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 0x7f95d414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d3400640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d413df80 0x7f95d413d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 -- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d413df80 msgr2=0x7f95d413d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d413df80 0x7f95d413d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 -- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 msgr2=0x7f95d414ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 0x7f95d414ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d8c00640 1 -- 172.21.15.125:0/2198952640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc009b30 con 0x7f95d413e360 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.035+0000 7f95d3e00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 0x7f95d414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95d8c00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc009510 con 0x7f95d413e360 2024-01-16T15:05:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc03e450 con 0x7f95d413e360 2024-01-16T15:05:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc03e800 con 0x7f95d413e360 2024-01-16T15:05:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d4150b40 con 0x7f95d413e360 2024-01-16T15:05:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.036+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d414c470 con 0x7f95d413e360 2024-01-16T15:05:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.037+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d40675d0 con 0x7f95d413e360 2024-01-16T15:05:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.038+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f95cc03e960 con 0x7f95d413e360 2024-01-16T15:05:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.039+0000 7f95cbe00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 0x7f95ac06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.039+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95cc0ab340 con 0x7f95d413e360 2024-01-16T15:05:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.039+0000 7f95d3e00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 0x7f95ac06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.040+0000 7f95d3e00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 0x7f95ac06dff0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009b90 tx=0x7f95c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.044+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95cc077a10 con 0x7f95d413e360 2024-01-16T15:05:06.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.239+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95d406df00 con 0x7f95d413e360 2024-01-16T15:05:06.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.241+0000 7f95cbe00640 1 -- 172.21.15.125:0/2198952640 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f95cc07b030 con 0x7f95d413e360 2024-01-16T15:05:06.221 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:06.221 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:05:04.841883+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:02.632990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:03.846526+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 msgr2=0x7f95ac06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 0x7f95ac06dff0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009b90 tx=0x7f95c4008040 comp rx=0 tx=0).stop 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 msgr2=0x7f95d414bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 shutdown_connections 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f95ac06bb30 0x7f95ac06dff0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95d413e360 0x7f95d414bf30 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95d413df80 0x7f95d413d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 --2- 172.21.15.125:0/2198952640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95d40cf9a0 0x7f95d414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 >> 172.21.15.125:0/2198952640 conn(0x7f95d406a800 msgr2=0x7f95d415e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 shutdown_connections 2024-01-16T15:05:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.244+0000 7f95dac00640 1 -- 172.21.15.125:0/2198952640 wait complete. 2024-01-16T15:05:06.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:05:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.479+0000 7f4db8e00640 1 Processor -- start 2024-01-16T15:05:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.479+0000 7f4db8e00640 1 -- start start 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db8e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4db40b3ae0 0x7f4db415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db8e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4db40cf290 0x7f4db40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db8e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db4070c40 con 0x7f4db40ae5a0 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db8e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db4071450 con 0x7f4db40cf290 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db8e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40cfc60 con 0x7f4db40b3ae0 2024-01-16T15:05:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4db40ae5a0 0x7f4db406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45788/0 (socket says 172.21.15.125:45788) 2024-01-16T15:05:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db1e00640 1 -- 172.21.15.125:0/4117061925 learned_addr learned my addr 172.21.15.125:0/4117061925 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db2800640 1 --2- 172.21.15.125:0/4117061925 >> v2:172.21.15.125:3300/0 conn(0x7f4db40cf290 0x7f4db40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.481+0000 7f4db1400640 1 --2- 172.21.15.125:0/4117061925 >> v2:172.21.15.137:3300/0 conn(0x7f4db40b3ae0 0x7f4db415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db0a00640 1 -- 172.21.15.125:0/4117061925 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927121262 0 0) 0x7f4db4070c40 con 0x7f4db40ae5a0 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db0a00640 1 -- 172.21.15.125:0/4117061925 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94003520 con 0x7f4db40ae5a0 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db2800640 1 -- 172.21.15.125:0/4117061925 >> v2:172.21.15.137:3300/0 conn(0x7f4db40b3ae0 msgr2=0x7f4db415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db2800640 1 --2- 172.21.15.125:0/4117061925 >> v2:172.21.15.137:3300/0 conn(0x7f4db40b3ae0 0x7f4db415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db2800640 1 -- 172.21.15.125:0/4117061925 >> v1:172.21.15.125:6789/0 conn(0x7f4db40ae5a0 legacy=0x7f4db406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.482+0000 7f4db2800640 1 -- 172.21.15.125:0/4117061925 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db406c470 con 0x7f4db40cf290 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.483+0000 7f4db2800640 1 --2- 172.21.15.125:0/4117061925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40cf290 0x7f4db40cf720 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f4db40cfea0 tx=0x7f4dac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d871ce90cd9d576c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.483+0000 7f4db0a00640 1 -- 172.21.15.125:0/4117061925 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac02cd00 con 0x7f4db40cf290 2024-01-16T15:05:06.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.483+0000 7f4db0a00640 1 -- 172.21.15.125:0/4117061925 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac02ce60 con 0x7f4db40cf290 2024-01-16T15:05:06.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.483+0000 7f4db0a00640 1 -- 172.21.15.125:0/4117061925 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac034640 con 0x7f4db40cf290 2024-01-16T15:05:06.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 -- 172.21.15.125:0/4117061925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40cf290 msgr2=0x7f4db40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 --2- 172.21.15.125:0/4117061925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40cf290 0x7f4db40cf720 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f4db40cfea0 tx=0x7f4dac02bf50 comp rx=0 tx=0).stop 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 -- 172.21.15.125:0/4117061925 shutdown_connections 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 --2- 172.21.15.125:0/4117061925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40cf290 0x7f4db40cf720 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 --2- 172.21.15.125:0/4117061925 >> v2:172.21.15.137:3300/0 conn(0x7f4db40b3ae0 0x7f4db415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 -- 172.21.15.125:0/4117061925 >> 172.21.15.125:0/4117061925 conn(0x7f4db406a800 msgr2=0x7f4db406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 -- 172.21.15.125:0/4117061925 shutdown_connections 2024-01-16T15:05:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.484+0000 7f4db8e00640 1 -- 172.21.15.125:0/4117061925 wait complete. 2024-01-16T15:05:06.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.486+0000 7f4db8e00640 1 Processor -- start 2024-01-16T15:05:06.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.486+0000 7f4db8e00640 1 -- start start 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4db40ae5a0 0x7f4db415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40cf290 0x7f4db40cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db415cf40 con 0x7f4db40b3ae0 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db415bea0 con 0x7f4db40cf290 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db415c180 con 0x7f4db40ae5a0 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60256/0 (socket says 172.21.15.125:60256) 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.487+0000 7f4db1400640 1 -- 172.21.15.125:0/674540593 learned_addr learned my addr 172.21.15.125:0/674540593 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4db40ae5a0 0x7f4db415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 -- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4db40ae5a0 msgr2=0x7f4db415b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db2800640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40cf290 0x7f4db40cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4db40ae5a0 0x7f4db415b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 -- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40cf290 msgr2=0x7f4db40cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40cf290 0x7f4db40cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 -- 172.21.15.125:0/674540593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac009a70 con 0x7f4db40b3ae0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db1400640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00d800 tx=0x7f4d9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00b6c0 con 0x7f4db40b3ae0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c00b820 con 0x7f4db40b3ae0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00bbb0 con 0x7f4db40b3ae0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db415c460 con 0x7f4db40b3ae0 2024-01-16T15:05:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.488+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db415ca20 con 0x7f4db40b3ae0 2024-01-16T15:05:06.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.490+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db415cbe0 con 0x7f4db40b3ae0 2024-01-16T15:05:06.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.491+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4d9c00bd10 con 0x7f4db40b3ae0 2024-01-16T15:05:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.492+0000 7f4daa000640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 0x7f4d8c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.492+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d9c088b70 con 0x7f4db40b3ae0 2024-01-16T15:05:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.492+0000 7f4db1e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 0x7f4d8c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.493+0000 7f4db1e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 0x7f4d8c06dff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4db40ced60 tx=0x7f4da4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.496+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d9c0552a0 con 0x7f4db40b3ae0 2024-01-16T15:05:06.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:06.726+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4db415e730 con 0x7f4db40b3ae0 2024-01-16T15:05:07.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.407+0000 7f4daa000640 1 -- 172.21.15.125:0/674540593 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4d9c0588c0 con 0x7f4db40b3ae0 2024-01-16T15:05:07.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 msgr2=0x7f4d8c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 0x7f4d8c06dff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4db40ced60 tx=0x7f4da4009040 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 msgr2=0x7f4db40b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00d800 tx=0x7f4d9c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 shutdown_connections 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40cf290 0x7f4db40cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4d8c06bb30 0x7f4d8c06dff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db40b3ae0 0x7f4db40b02d0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 --2- 172.21.15.125:0/674540593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4db40ae5a0 0x7f4db415b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 >> 172.21.15.125:0/674540593 conn(0x7f4db406a800 msgr2=0x7f4db4160350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 shutdown_connections 2024-01-16T15:05:07.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.411+0000 7f4db8e00640 1 -- 172.21.15.125:0/674540593 wait complete. 2024-01-16T15:05:07.411 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:05:07.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:05:07.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.650+0000 7f3d04c00640 1 Processor -- start 2024-01-16T15:05:07.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.651+0000 7f3d04c00640 1 -- start start 2024-01-16T15:05:07.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.652+0000 7f3d04c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d000c9870 0x7f3d000ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.652+0000 7f3d04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 0x7f3d00161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.652+0000 7f3d04c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000cf4c0 con 0x7f3d000d2260 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.652+0000 7f3d04c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d000d2ef0 con 0x7f3d000c9870 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.652+0000 7f3d04c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000c9c50 con 0x7f3d000d0560 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.653+0000 7f3cfd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 0x7f3d00161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.653+0000 7f3cfde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d000c9870 0x7f3d000ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.653+0000 7f3cfde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d000c9870 0x7f3d000ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:35918/0 (socket says 172.21.15.125:35918) 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.653+0000 7f3cfd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 0x7f3d00161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:57988/0 (socket says 172.21.15.125:57988) 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.653+0000 7f3cfde00640 1 -- 172.21.15.125:0/2732553047 learned_addr learned my addr 172.21.15.125:0/2732553047 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.654+0000 7f3cfde00640 1 -- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 msgr2=0x7f3d00161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.654+0000 7f3cfde00640 1 --2- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 0x7f3d00161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.654+0000 7f3cfde00640 1 -- 172.21.15.125:0/2732553047 >> v1:172.21.15.137:6789/0 conn(0x7f3d000d2260 legacy=0x7f3d000d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.654+0000 7f3cfde00640 1 -- 172.21.15.125:0/2732553047 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d000cf7a0 con 0x7f3d000c9870 2024-01-16T15:05:07.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.655+0000 7f3cfde00640 1 --2- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000c9870 0x7f3d000ceb50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009870 tx=0x7f3cf002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a18a7f7d11488406 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3cfca00640 1 -- 172.21.15.125:0/2732553047 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0030840 con 0x7f3d000c9870 2024-01-16T15:05:07.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3cfca00640 1 -- 172.21.15.125:0/2732553047 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf00309a0 con 0x7f3d000c9870 2024-01-16T15:05:07.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3cfca00640 1 -- 172.21.15.125:0/2732553047 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0030d30 con 0x7f3d000c9870 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 -- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000c9870 msgr2=0x7f3d000ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 --2- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000c9870 0x7f3d000ceb50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009870 tx=0x7f3cf002b980 comp rx=0 tx=0).stop 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 -- 172.21.15.125:0/2732553047 shutdown_connections 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 --2- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000d0560 0x7f3d00161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 --2- 172.21.15.125:0/2732553047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000c9870 0x7f3d000ceb50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 -- 172.21.15.125:0/2732553047 >> 172.21.15.125:0/2732553047 conn(0x7f3d0009c560 msgr2=0x7f3d000d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 -- 172.21.15.125:0/2732553047 shutdown_connections 2024-01-16T15:05:07.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.656+0000 7f3d04c00640 1 -- 172.21.15.125:0/2732553047 wait complete. 2024-01-16T15:05:07.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.658+0000 7f3d04c00640 1 Processor -- start 2024-01-16T15:05:07.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.658+0000 7f3d04c00640 1 -- start start 2024-01-16T15:05:07.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000c9870 0x7f3d000b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000d2260 0x7f3d000afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000b2d80 con 0x7f3d000d0560 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000b3060 con 0x7f3d000d2260 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.659+0000 7f3d04c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000b3340 con 0x7f3d000c9870 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51820/0 (socket says 172.21.15.125:51820) 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 -- 172.21.15.125:0/342255091 learned_addr learned my addr 172.21.15.125:0/342255091 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfde00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000c9870 0x7f3d000b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfe800640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000d2260 0x7f3d000afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 -- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000c9870 msgr2=0x7f3d000b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000c9870 0x7f3d000b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 -- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000d2260 msgr2=0x7f3d000afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000d2260 0x7f3d000afe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 -- 172.21.15.125:0/342255091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf00094d0 con 0x7f3d000d0560 2024-01-16T15:05:07.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.660+0000 7f3cfd400640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800e980 tx=0x7f3ce800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.661+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8009670 con 0x7f3d000d0560 2024-01-16T15:05:07.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.661+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce800ca00 con 0x7f3d000d0560 2024-01-16T15:05:07.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.661+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d000b3620 con 0x7f3d000d0560 2024-01-16T15:05:07.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.661+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce800cd90 con 0x7f3d000d0560 2024-01-16T15:05:07.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.661+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d000b0370 con 0x7f3d000d0560 2024-01-16T15:05:07.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.662+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00066640 con 0x7f3d000d0560 2024-01-16T15:05:07.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.663+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3ce8013470 con 0x7f3d000d0560 2024-01-16T15:05:07.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.664+0000 7f3cf6000640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 0x7f3cd806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.664+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ce8088840 con 0x7f3d000d0560 2024-01-16T15:05:07.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.664+0000 7f3cfde00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 0x7f3cd806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.665+0000 7f3cfde00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 0x7f3cd806dff0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0036000 tx=0x7f3cf003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.668+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ce8054f90 con 0x7f3d000d0560 2024-01-16T15:05:07.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:07.899+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3d000675d0 con 0x7f3d000d0560 2024-01-16T15:05:08.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.420+0000 7f3cf6000640 1 -- 172.21.15.125:0/342255091 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3ce80585b0 con 0x7f3d000d0560 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 msgr2=0x7f3cd806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 0x7f3cd806dff0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0036000 tx=0x7f3cf003b040 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 msgr2=0x7f3d001508e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800e980 tx=0x7f3ce800ee40 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 shutdown_connections 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d000d2260 0x7f3d000afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cd806bb30 0x7f3cd806dff0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000d0560 0x7f3d001508e0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 --2- 172.21.15.125:0/342255091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d000c9870 0x7f3d000b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 >> 172.21.15.125:0/342255091 conn(0x7f3d0009c560 msgr2=0x7f3d000d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 shutdown_connections 2024-01-16T15:05:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.424+0000 7f3d04c00640 1 -- 172.21.15.125:0/342255091 wait complete. 2024-01-16T15:05:08.424 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:05:08.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.645+0000 7fcc7b200640 1 Processor -- start 2024-01-16T15:05:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.646+0000 7fcc7b200640 1 -- start start 2024-01-16T15:05:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc7b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcc7413e360 0x7fcc740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc7b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcc7415f9c0 0x7fcc740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc7b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7415c5f0 con 0x7fcc7413df80 2024-01-16T15:05:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc7b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7406c470 con 0x7fcc7415f9c0 2024-01-16T15:05:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc7b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7406c750 con 0x7fcc7413e360 2024-01-16T15:05:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc73e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc7413df80 0x7fcc7415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47678/0 (socket says 172.21.15.125:47678) 2024-01-16T15:05:08.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc73e00640 1 -- 172.21.15.125:0/384891938 learned_addr learned my addr 172.21.15.125:0/384891938 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.647+0000 7fcc79200640 1 --2- 172.21.15.125:0/384891938 >> v2:172.21.15.125:3300/0 conn(0x7fcc7415f9c0 0x7fcc740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc6b200640 1 --2- 172.21.15.125:0/384891938 >> v2:172.21.15.137:3300/0 conn(0x7fcc7413e360 0x7fcc740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc73400640 1 -- 172.21.15.125:0/384891938 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067678024 0 0) 0x7fcc7415c5f0 con 0x7fcc7413df80 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc73400640 1 -- 172.21.15.125:0/384891938 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc54003520 con 0x7fcc7413df80 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc79200640 1 -- 172.21.15.125:0/384891938 >> v2:172.21.15.137:3300/0 conn(0x7fcc7413e360 msgr2=0x7fcc740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc79200640 1 --2- 172.21.15.125:0/384891938 >> v2:172.21.15.137:3300/0 conn(0x7fcc7413e360 0x7fcc740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc79200640 1 -- 172.21.15.125:0/384891938 >> v1:172.21.15.125:6789/0 conn(0x7fcc7413df80 legacy=0x7fcc7415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.648+0000 7fcc79200640 1 -- 172.21.15.125:0/384891938 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7406ca30 con 0x7fcc7415f9c0 2024-01-16T15:05:08.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.649+0000 7fcc79200640 1 --2- 172.21.15.125:0/384891938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc740cf7d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcc7406ded0 tx=0x7fcc6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb89f457aded2b88 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.649+0000 7fcc73400640 1 -- 172.21.15.125:0/384891938 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c02cc80 con 0x7fcc7415f9c0 2024-01-16T15:05:08.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.649+0000 7fcc73400640 1 -- 172.21.15.125:0/384891938 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c02cde0 con 0x7fcc7415f9c0 2024-01-16T15:05:08.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.649+0000 7fcc73400640 1 -- 172.21.15.125:0/384891938 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c034660 con 0x7fcc7415f9c0 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 -- 172.21.15.125:0/384891938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 msgr2=0x7fcc740cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 --2- 172.21.15.125:0/384891938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc740cf7d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcc7406ded0 tx=0x7fcc6c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 -- 172.21.15.125:0/384891938 shutdown_connections 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 --2- 172.21.15.125:0/384891938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc740cf7d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 --2- 172.21.15.125:0/384891938 >> v2:172.21.15.137:3300/0 conn(0x7fcc7413e360 0x7fcc740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 -- 172.21.15.125:0/384891938 >> 172.21.15.125:0/384891938 conn(0x7fcc7406a800 msgr2=0x7fcc7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 -- 172.21.15.125:0/384891938 shutdown_connections 2024-01-16T15:05:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.650+0000 7fcc7b200640 1 -- 172.21.15.125:0/384891938 wait complete. 2024-01-16T15:05:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.652+0000 7fcc7b200640 1 Processor -- start 2024-01-16T15:05:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.652+0000 7fcc7b200640 1 -- start start 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc7413df80 0x7fcc741f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc7413e360 0x7fcc741f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740a8d50 con 0x7fcc7415f9c0 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc741f6970 con 0x7fcc7413df80 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc7b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc741f6c50 con 0x7fcc7413e360 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51836/0 (socket says 172.21.15.125:51836) 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 -- 172.21.15.125:0/2258582844 learned_addr learned my addr 172.21.15.125:0/2258582844 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 -- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc7413e360 msgr2=0x7fcc741f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc7413e360 0x7fcc741f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 -- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc7413df80 msgr2=0x7fcc741f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc7413df80 0x7fcc741f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.653+0000 7fcc79200640 1 -- 172.21.15.125:0/2258582844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c009a70 con 0x7fcc7415f9c0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc79200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009e80 tx=0x7fcc6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c040650 con 0x7fcc7415f9c0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c0407b0 con 0x7fcc7415f9c0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc741f2b60 con 0x7fcc7415f9c0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c040b60 con 0x7fcc7415f9c0 2024-01-16T15:05:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.654+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc741f2810 con 0x7fcc7415f9c0 2024-01-16T15:05:08.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.656+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc741f3080 con 0x7fcc7415f9c0 2024-01-16T15:05:08.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.656+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fcc6c040cc0 con 0x7fcc7415f9c0 2024-01-16T15:05:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.657+0000 7fcc70c00640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 0x7fcc4c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.657+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc6c0ab2c0 con 0x7fcc7415f9c0 2024-01-16T15:05:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.657+0000 7fcc73e00640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 0x7fcc4c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.658+0000 7fcc73e00640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 0x7fcc4c06dff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc7415c780 tx=0x7fcc64007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.663+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc6c077a10 con 0x7fcc7415f9c0 2024-01-16T15:05:08.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:08.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:08.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.893+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc740675d0 con 0x7fcc7415f9c0 2024-01-16T15:05:08.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.895+0000 7fcc70c00640 1 -- 172.21.15.125:0/2258582844 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fcc6c009450 con 0x7fcc7415f9c0 2024-01-16T15:05:08.873 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2392328978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2392328978},{"type":"v1","addr":"172.21.15.137:6834","nonce":2392328978}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5779,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2251602125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2251602125},{"type":"v1","addr":"172.21.15.137:6835","nonce":2251602125}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2951255872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2951255872},{"type":"v1","addr":"172.21.15.125:6835","nonce":2951255872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:04.842732+0000","modified":"2024-01-16T15:05:07.901312+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7125":{"gid":7125,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/22640034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":22640034},{"type":"v1","addr":"172.21.15.125:6837","nonce":22640034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:05:08.873 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 45 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 msgr2=0x7fcc4c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 0x7fcc4c06dff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc7415c780 tx=0x7fcc64007930 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 msgr2=0x7fcc741f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009e80 tx=0x7fcc6c004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 shutdown_connections 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc4c06bb30 0x7fcc4c06dff0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc7415f9c0 0x7fcc741f6340 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc7413e360 0x7fcc741f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 --2- 172.21.15.125:0/2258582844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc7413df80 0x7fcc741f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 >> 172.21.15.125:0/2258582844 conn(0x7fcc7406a800 msgr2=0x7fcc740b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 shutdown_connections 2024-01-16T15:05:08.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:08.897+0000 7fcc7b200640 1 -- 172.21.15.125:0/2258582844 wait complete. 2024-01-16T15:05:08.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:09.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.166+0000 7f19fee00640 1 Processor -- start 2024-01-16T15:05:09.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.166+0000 7f19fee00640 1 -- start start 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f19f815e3b0 0x7f19f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f80cfc30 con 0x7f19f80cf7d0 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80b1ee0 con 0x7f19f815e3b0 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b2780 con 0x7f19f8143ae0 2024-01-16T15:05:09.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19f7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f19f80cf7d0 0x7f19f815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47688/0 (socket says 172.21.15.125:47688) 2024-01-16T15:05:09.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19f7400640 1 -- 172.21.15.125:0/2579891570 learned_addr learned my addr 172.21.15.125:0/2579891570 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:09.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19fce00640 1 --2- 172.21.15.125:0/2579891570 >> v2:172.21.15.137:3300/0 conn(0x7f19f815e3b0 0x7f19f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.168+0000 7f19f7e00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851505723 0 0) 0x7f19f80cfc30 con 0x7f19f80cf7d0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8003520 con 0x7f19f80cf7d0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951071108 0 0) 0x7f19d8003520 con 0x7f19f80cf7d0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 msgr2=0x7f19f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19f80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 >> v2:172.21.15.137:3300/0 conn(0x7f19f815e3b0 msgr2=0x7f19f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 --2- 172.21.15.125:0/2579891570 >> v2:172.21.15.137:3300/0 conn(0x7f19f815e3b0 0x7f19f80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f815e990 con 0x7f19f80cf7d0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f19e0001670 con 0x7f19f80cf7d0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 >> v1:172.21.15.125:6789/0 conn(0x7f19f80cf7d0 legacy=0x7f19f815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8003970 0x7f19d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d80047c0 0x7f19d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b1ee0 con 0x7f19d8003970 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b2780 con 0x7f19d80047c0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800b980 con 0x7f19f8143ae0 2024-01-16T15:05:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.169+0000 7f19f7e00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8003970 0x7f19d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19fce00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19f7e00640 1 -- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 msgr2=0x7f19d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19f7e00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19f8143ae0 0x7f19d8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19f7e00640 1 -- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d80047c0 msgr2=0x7f19d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19f7e00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19d80047c0 0x7f19d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.170+0000 7f19f7e00640 1 -- 172.21.15.125:0/2579891570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8003520 con 0x7f19d8003970 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.171+0000 7f19f7e00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8003970 0x7f19d8003d90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009ac0 tx=0x7f19d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d8f0fb6806b550 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.171+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e802c6b0 con 0x7f19d8003970 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.171+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e802c810 con 0x7f19d8003970 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.171+0000 7f19f6a00640 1 -- 172.21.15.125:0/2579891570 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e802cba0 con 0x7f19d8003970 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 -- 172.21.15.125:0/2579891570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8003970 msgr2=0x7f19d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 --2- 172.21.15.125:0/2579891570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19d8003970 0x7f19d8003d90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009ac0 tx=0x7f19d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fce00640 1 -- 172.21.15.125:0/2579891570 reap_dead start 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 -- 172.21.15.125:0/2579891570 shutdown_connections 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 -- 172.21.15.125:0/2579891570 >> 172.21.15.125:0/2579891570 conn(0x7f19f806a800 msgr2=0x7f19f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 -- 172.21.15.125:0/2579891570 shutdown_connections 2024-01-16T15:05:09.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.172+0000 7f19fee00640 1 -- 172.21.15.125:0/2579891570 wait complete. 2024-01-16T15:05:09.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.174+0000 7f19fee00640 1 Processor -- start 2024-01-16T15:05:09.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.174+0000 7f19fee00640 1 -- start start 2024-01-16T15:05:09.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8003970 0x7f19f80acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f80ce490 0x7f19f80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f815d120 con 0x7f19f80ce010 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f815d5b0 con 0x7f19f80ce490 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19fee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f815c270 con 0x7f19d8003970 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51844/0 (socket says 172.21.15.125:51844) 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19f7400640 1 -- 172.21.15.125:0/954488169 learned_addr learned my addr 172.21.15.125:0/954488169 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.175+0000 7f19f7e00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8003970 0x7f19f80acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19fce00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f80ce490 0x7f19f80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 -- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8003970 msgr2=0x7f19f80acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8003970 0x7f19f80acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 -- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f80ce490 msgr2=0x7f19f80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f80ce490 0x7f19f80ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 -- 172.21.15.125:0/954488169 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f815c510 con 0x7f19f80ce010 2024-01-16T15:05:09.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19f7400640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c700 tx=0x7f19e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00074e0 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0007660 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0004a20 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f815c7f0 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.176+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f815cdb0 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.178+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f815cf70 con 0x7f19f80ce010 2024-01-16T15:05:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.179+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f19e0004b80 con 0x7f19f80ce010 2024-01-16T15:05:09.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.179+0000 7f19efe00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 0x7f19d006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.179+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19e00879a0 con 0x7f19f80ce010 2024-01-16T15:05:09.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.180+0000 7f19f7e00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 0x7f19d006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.180+0000 7f19f7e00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 0x7f19d006dff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f19e8036000 tx=0x7f19e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.184+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19e0057af0 con 0x7f19f80ce010 2024-01-16T15:05:09.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.375+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19f80675d0 con 0x7f19f80ce010 2024-01-16T15:05:09.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.378+0000 7f19efe00640 1 -- 172.21.15.125:0/954488169 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16616 (secure 0 0 0) 0x7f19e0018090 con 0x7f19f80ce010 2024-01-16T15:05:09.357 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:09.357 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":70,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:05:06.386473+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:02.632990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:03.846526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 msgr2=0x7f19d006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 0x7f19d006dff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f19e8036000 tx=0x7f19e8030000 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 msgr2=0x7f19f80cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c700 tx=0x7f19e000cbc0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 shutdown_connections 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19f80ce490 0x7f19f80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19d006bb30 0x7f19d006dff0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19f80ce010 0x7f19f80cc610 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 --2- 172.21.15.125:0/954488169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19d8003970 0x7f19f80acb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.381+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 >> 172.21.15.125:0/954488169 conn(0x7f19f806a800 msgr2=0x7f19f80add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.382+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 shutdown_connections 2024-01-16T15:05:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.382+0000 7f19fee00640 1 -- 172.21.15.125:0/954488169 wait complete. 2024-01-16T15:05:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:05:09.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:09.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.599+0000 7fcb56400640 1 Processor -- start 2024-01-16T15:05:09.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.599+0000 7fcb56400640 1 -- start start 2024-01-16T15:05:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb56400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcb501437b0 0x7fcb5006f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb56400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb5006e990 0x7fcb5015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb56400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500cf3b0 con 0x7fcb5015da20 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb56400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5015e3f0 con 0x7fcb5006e990 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb56400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5006fc80 con 0x7fcb501437b0 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb55a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb5006e990 0x7fcb5015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb47c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb5015da20 0x7fcb500b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47694/0 (socket says 172.21.15.125:47694) 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb47c00640 1 -- 172.21.15.125:0/1577532926 learned_addr learned my addr 172.21.15.125:0/1577532926 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb55a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb5006e990 0x7fcb5015deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51846/0 (socket says 172.21.15.125:51846) 2024-01-16T15:05:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.601+0000 7fcb55000640 1 --2- 172.21.15.125:0/1577532926 >> v2:172.21.15.137:3300/0 conn(0x7fcb501437b0 0x7fcb5006f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.602+0000 7fcb55a00640 1 -- 172.21.15.125:0/1577532926 >> v2:172.21.15.137:3300/0 conn(0x7fcb501437b0 msgr2=0x7fcb5006f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.602+0000 7fcb55a00640 1 --2- 172.21.15.125:0/1577532926 >> v2:172.21.15.137:3300/0 conn(0x7fcb501437b0 0x7fcb5006f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.602+0000 7fcb55a00640 1 -- 172.21.15.125:0/1577532926 >> v1:172.21.15.125:6789/0 conn(0x7fcb5015da20 legacy=0x7fcb500b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.602+0000 7fcb55a00640 1 -- 172.21.15.125:0/1577532926 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5006ee00 con 0x7fcb5006e990 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.603+0000 7fcb55a00640 1 --2- 172.21.15.125:0/1577532926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5015deb0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fcb500726c0 tx=0x7fcb4802da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34aa8928fd7c5a2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.603+0000 7fcb4fe00640 1 -- 172.21.15.125:0/1577532926 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48031870 con 0x7fcb5006e990 2024-01-16T15:05:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.603+0000 7fcb4fe00640 1 -- 172.21.15.125:0/1577532926 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb480319d0 con 0x7fcb5006e990 2024-01-16T15:05:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.603+0000 7fcb4fe00640 1 -- 172.21.15.125:0/1577532926 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48031d60 con 0x7fcb5006e990 2024-01-16T15:05:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 -- 172.21.15.125:0/1577532926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 msgr2=0x7fcb5015deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 --2- 172.21.15.125:0/1577532926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5015deb0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fcb500726c0 tx=0x7fcb4802da50 comp rx=0 tx=0).stop 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 -- 172.21.15.125:0/1577532926 shutdown_connections 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 --2- 172.21.15.125:0/1577532926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5015deb0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 --2- 172.21.15.125:0/1577532926 >> v2:172.21.15.137:3300/0 conn(0x7fcb501437b0 0x7fcb5006f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 -- 172.21.15.125:0/1577532926 >> 172.21.15.125:0/1577532926 conn(0x7fcb500b1cd0 msgr2=0x7fcb500b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 -- 172.21.15.125:0/1577532926 shutdown_connections 2024-01-16T15:05:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.604+0000 7fcb56400640 1 -- 172.21.15.125:0/1577532926 wait complete. 2024-01-16T15:05:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.606+0000 7fcb56400640 1 Processor -- start 2024-01-16T15:05:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.607+0000 7fcb56400640 1 -- start start 2024-01-16T15:05:09.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb501437b0 0x7fcb5013d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 0x7fcb50159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5013eb10 con 0x7fcb5006e990 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50140650 con 0x7fcb5015da20 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb56400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5013de90 con 0x7fcb501437b0 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51856/0 (socket says 172.21.15.125:51856) 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 -- 172.21.15.125:0/2673679417 learned_addr learned my addr 172.21.15.125:0/2673679417 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55a00640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 0x7fcb50159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb47c00640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb501437b0 0x7fcb5013d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 -- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb501437b0 msgr2=0x7fcb5013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb501437b0 0x7fcb5013d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 -- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 msgr2=0x7fcb50159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 0x7fcb50159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55000640 1 -- 172.21.15.125:0/2673679417 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009400 con 0x7fcb5006e990 2024-01-16T15:05:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.608+0000 7fcb55a00640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 0x7fcb50159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb55000640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000efc0 tx=0x7fcb4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40014480 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4000f040 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40014800 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5013e170 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.609+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5013e730 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.610+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5013ed80 con 0x7fcb5006e990 2024-01-16T15:05:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.611+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fcb40014960 con 0x7fcb5006e990 2024-01-16T15:05:09.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.612+0000 7fcb4d600640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 0x7fcb2806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.612+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb40089430 con 0x7fcb5006e990 2024-01-16T15:05:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.613+0000 7fcb47c00640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 0x7fcb2806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.613+0000 7fcb47c00640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 0x7fcb2806e020 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcb500b4e00 tx=0x7fcb38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.617+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb40055b80 con 0x7fcb5006e990 2024-01-16T15:05:09.729 INFO:tasks.ceph.mds.c.smithi125.stderr:2024-01-16T15:05:09.751+0000 7f429b000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:05:09.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.804+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb500cfc10 con 0x7fcb2806bb60 2024-01-16T15:05:09.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.830+0000 7fcb4d600640 1 -- 172.21.15.125:0/2673679417 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277886 (secure 0 0 0) 0x7fcb380058b0 con 0x7fcb2806bb60 2024-01-16T15:05:09.812 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:05:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.836+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 msgr2=0x7fcb2806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.836+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 0x7fcb2806e020 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcb500b4e00 tx=0x7fcb38009040 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.836+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 msgr2=0x7fcb5013ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.836+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000efc0 tx=0x7fcb4000c490 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 shutdown_connections 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb5015da20 0x7fcb50159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb501437b0 0x7fcb5013d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb2806bb60 0x7fcb2806e020 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 --2- 172.21.15.125:0/2673679417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb5006e990 0x7fcb5013ef30 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 >> 172.21.15.125:0/2673679417 conn(0x7fcb500b1cd0 msgr2=0x7fcb500b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 shutdown_connections 2024-01-16T15:05:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:09.837+0000 7fcb56400640 1 -- 172.21.15.125:0/2673679417 wait complete. 2024-01-16T15:05:09.836 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-01-16T15:05:08.109011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":388892,"kb_used_data":7524,"kb_used_omap":12,"kb_used_meta":381235,"kb_avail":754585828,"statfs":{"total":773094113280,"available":772695887872,"internally_reserved":0,"allocated":7704576,"data_stored":3898383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":390385200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752651"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.384537+0000","last_change":"2024-01-16T15:05:04.384537+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.384537+0000","last_undegraded":"2024-01-16T15:05:04.384537+0000","last_fullsized":"2024-01-16T15:05:04.384537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.388365+0000","last_change":"2024-01-16T15:05:04.388365+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.388365+0000","last_undegraded":"2024-01-16T15:05:04.388365+0000","last_fullsized":"2024-01-16T15:05:04.388365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386925+0000","last_change":"2024-01-16T15:05:04.386925+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386925+0000","last_undegraded":"2024-01-16T15:05:04.386925+0000","last_fullsized":"2024-01-16T15:05:04.386925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.387278+0000","last_change":"2024-01-16T15:05:04.387278+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.387278+0000","last_undegraded":"2024-01-16T15:05:04.387278+0000","last_fullsized":"2024-01-16T15:05:04.387278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386218+0000","last_change":"2024-01-16T15:05:04.386218+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386218+0000","last_undegraded":"2024-01-16T15:05:04.386218+0000","last_fullsized":"2024-01-16T15:05:04.386218+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385374+0000","last_change":"2024-01-16T15:05:04.385374+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385374+0000","last_undegraded":"2024-01-16T15:05:04.385374+0000","last_fullsized":"2024-01-16T15:05:04.385374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385954+0000","last_change":"2024-01-16T15:05:04.385954+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385954+0000","last_undegraded":"2024-01-16T15:05:04.385954+0000","last_fullsized":"2024-01-16T15:05:04.385954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385210+0000","last_change":"2024-01-16T15:05:04.385210+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385210+0000","last_undegraded":"2024-01-16T15:05:04.385210+0000","last_fullsized":"2024-01-16T15:05:04.385210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386524+0000","last_change":"2024-01-16T15:05:04.386524+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386524+0000","last_undegraded":"2024-01-16T15:05:04.386524+0000","last_fullsized":"2024-01-16T15:05:04.386524+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385494+0000","last_change":"2024-01-16T15:05:04.385494+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385494+0000","last_undegraded":"2024-01-16T15:05:04.385494+0000","last_fullsized":"2024-01-16T15:05:04.385494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385123+0000","last_change":"2024-01-16T15:05:04.385123+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385123+0000","last_undegraded":"2024-01-16T15:05:04.385123+0000","last_fullsized":"2024-01-16T15:05:04.385123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.383954+0000","last_change":"2024-01-16T15:05:04.383954+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.383954+0000","last_undegraded":"2024-01-16T15:05:04.383954+0000","last_fullsized":"2024-01-16T15:05:04.383954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.384886+0000","last_change":"2024-01-16T15:05:04.384886+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.384886+0000","last_undegraded":"2024-01-16T15:05:04.384886+0000","last_fullsized":"2024-01-16T15:05:04.384886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.363803+0000","last_change":"2024-01-16T15:05:03.363803+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.363803+0000","last_undegraded":"2024-01-16T15:05:03.363803+0000","last_fullsized":"2024-01-16T15:05:03.363803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386020+0000","last_change":"2024-01-16T15:05:04.386020+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386020+0000","last_undegraded":"2024-01-16T15:05:04.386020+0000","last_fullsized":"2024-01-16T15:05:04.386020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.400487+0000","last_change":"2024-01-16T15:05:04.396236+0000","last_active":"2024-01-16T15:05:04.400487+0000","last_peered":"2024-01-16T15:05:04.400487+0000","last_clean":"2024-01-16T15:05:04.400487+0000","last_became_active":"2024-01-16T15:05:04.395960+0000","last_became_peered":"2024-01-16T15:05:04.395960+0000","last_unstale":"2024-01-16T15:05:04.400487+0000","last_undegraded":"2024-01-16T15:05:04.400487+0000","last_fullsized":"2024-01-16T15:05:04.400487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:41.845761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.360545+0000","last_change":"2024-01-16T15:05:03.360545+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.360545+0000","last_undegraded":"2024-01-16T15:05:03.360545+0000","last_fullsized":"2024-01-16T15:05:03.360545+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.365556+0000","last_change":"2024-01-16T15:05:03.365556+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.365556+0000","last_undegraded":"2024-01-16T15:05:03.365556+0000","last_fullsized":"2024-01-16T15:05:03.365556+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.394787+0000","last_change":"2024-01-16T15:05:04.394127+0000","last_active":"2024-01-16T15:05:04.394787+0000","last_peered":"2024-01-16T15:05:04.394787+0000","last_clean":"2024-01-16T15:05:04.394787+0000","last_became_active":"2024-01-16T15:05:04.393732+0000","last_became_peered":"2024-01-16T15:05:04.393732+0000","last_unstale":"2024-01-16T15:05:04.394787+0000","last_undegraded":"2024-01-16T15:05:04.394787+0000","last_fullsized":"2024-01-16T15:05:04.394787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:57.453053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.367645+0000","last_change":"2024-01-16T15:05:03.367645+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.367645+0000","last_undegraded":"2024-01-16T15:05:03.367645+0000","last_fullsized":"2024-01-16T15:05:03.367645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385579+0000","last_change":"2024-01-16T15:05:04.385579+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385579+0000","last_undegraded":"2024-01-16T15:05:04.385579+0000","last_fullsized":"2024-01-16T15:05:04.385579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.400335+0000","last_change":"2024-01-16T15:05:04.397445+0000","last_active":"2024-01-16T15:05:04.400335+0000","last_peered":"2024-01-16T15:05:04.400335+0000","last_clean":"2024-01-16T15:05:04.400335+0000","last_became_active":"2024-01-16T15:05:04.396381+0000","last_became_peered":"2024-01-16T15:05:04.396381+0000","last_unstale":"2024-01-16T15:05:04.400335+0000","last_undegraded":"2024-01-16T15:05:04.400335+0000","last_fullsized":"2024-01-16T15:05:04.400335+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:04.008696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.384576+0000","last_change":"2024-01-16T15:05:04.384576+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.384576+0000","last_undegraded":"2024-01-16T15:05:04.384576+0000","last_fullsized":"2024-01-16T15:05:04.384576+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":8,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.396016+0000","last_change":"2024-01-16T15:05:04.396016+0000","last_active":"2024-01-16T15:05:04.396016+0000","last_peered":"2024-01-16T15:05:04.396016+0000","last_clean":"2024-01-16T15:05:04.396016+0000","last_became_active":"2024-01-16T15:05:04.394556+0000","last_became_peered":"2024-01-16T15:05:04.394556+0000","last_unstale":"2024-01-16T15:05:04.396016+0000","last_undegraded":"2024-01-16T15:05:04.396016+0000","last_fullsized":"2024-01-16T15:05:04.396016+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:55.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.360551+0000","last_change":"2024-01-16T15:05:03.360551+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.360551+0000","last_undegraded":"2024-01-16T15:05:03.360551+0000","last_fullsized":"2024-01-16T15:05:03.360551+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.366103+0000","last_change":"2024-01-16T15:05:03.366103+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.366103+0000","last_undegraded":"2024-01-16T15:05:03.366103+0000","last_fullsized":"2024-01-16T15:05:03.366103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.406764+0000","last_change":"2024-01-16T15:05:04.405645+0000","last_active":"2024-01-16T15:05:04.406764+0000","last_peered":"2024-01-16T15:05:04.406764+0000","last_clean":"2024-01-16T15:05:04.406764+0000","last_became_active":"2024-01-16T15:05:04.405340+0000","last_became_peered":"2024-01-16T15:05:04.405340+0000","last_unstale":"2024-01-16T15:05:04.406764+0000","last_undegraded":"2024-01-16T15:05:04.406764+0000","last_fullsized":"2024-01-16T15:05:04.406764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:56.984663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.387748+0000","last_change":"2024-01-16T15:05:04.387748+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.387748+0000","last_undegraded":"2024-01-16T15:05:04.387748+0000","last_fullsized":"2024-01-16T15:05:04.387748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.395407+0000","last_change":"2024-01-16T15:05:04.394073+0000","last_active":"2024-01-16T15:05:04.395407+0000","last_peered":"2024-01-16T15:05:04.395407+0000","last_clean":"2024-01-16T15:05:04.395407+0000","last_became_active":"2024-01-16T15:05:04.393394+0000","last_became_peered":"2024-01-16T15:05:04.393394+0000","last_unstale":"2024-01-16T15:05:04.395407+0000","last_undegraded":"2024-01-16T15:05:04.395407+0000","last_fullsized":"2024-01-16T15:05:04.395407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:10.147136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.401576+0000","last_change":"2024-01-16T15:05:04.400225+0000","last_active":"2024-01-16T15:05:04.401576+0000","last_peered":"2024-01-16T15:05:04.401576+0000","last_clean":"2024-01-16T15:05:04.401576+0000","last_became_active":"2024-01-16T15:05:04.399790+0000","last_became_peered":"2024-01-16T15:05:04.399790+0000","last_unstale":"2024-01-16T15:05:04.401576+0000","last_undegraded":"2024-01-16T15:05:04.401576+0000","last_fullsized":"2024-01-16T15:05:04.401576+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:59.957523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.367038+0000","last_change":"2024-01-16T15:05:03.367038+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.367038+0000","last_undegraded":"2024-01-16T15:05:03.367038+0000","last_fullsized":"2024-01-16T15:05:03.367038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386058+0000","last_change":"2024-01-16T15:05:04.386058+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386058+0000","last_undegraded":"2024-01-16T15:05:04.386058+0000","last_fullsized":"2024-01-16T15:05:04.386058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.369241+0000","last_change":"2024-01-16T15:05:03.369241+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.369241+0000","last_undegraded":"2024-01-16T15:05:03.369241+0000","last_fullsized":"2024-01-16T15:05:03.369241+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.387271+0000","last_change":"2024-01-16T15:05:04.387271+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.387271+0000","last_undegraded":"2024-01-16T15:05:04.387271+0000","last_fullsized":"2024-01-16T15:05:04.387271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.360577+0000","last_change":"2024-01-16T15:05:03.360577+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.360577+0000","last_undegraded":"2024-01-16T15:05:03.360577+0000","last_fullsized":"2024-01-16T15:05:03.360577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.368190+0000","last_change":"2024-01-16T15:05:03.368190+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.368190+0000","last_undegraded":"2024-01-16T15:05:03.368190+0000","last_fullsized":"2024-01-16T15:05:03.368190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.361268+0000","last_change":"2024-01-16T15:05:03.361268+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.361268+0000","last_undegraded":"2024-01-16T15:05:03.361268+0000","last_fullsized":"2024-01-16T15:05:03.361268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.385718+0000","last_change":"2024-01-16T15:05:04.385718+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.385718+0000","last_undegraded":"2024-01-16T15:05:04.385718+0000","last_fullsized":"2024-01-16T15:05:04.385718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.365014+0000","last_change":"2024-01-16T15:05:03.365014+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.365014+0000","last_undegraded":"2024-01-16T15:05:03.365014+0000","last_fullsized":"2024-01-16T15:05:03.365014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.387258+0000","last_change":"2024-01-16T15:05:04.387258+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.387258+0000","last_undegraded":"2024-01-16T15:05:04.387258+0000","last_fullsized":"2024-01-16T15:05:04.387258+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.363216+0000","last_change":"2024-01-16T15:05:03.363216+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.363216+0000","last_undegraded":"2024-01-16T15:05:03.363216+0000","last_fullsized":"2024-01-16T15:05:03.363216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.383570+0000","last_change":"2024-01-16T15:05:04.383570+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.383570+0000","last_undegraded":"2024-01-16T15:05:04.383570+0000","last_fullsized":"2024-01-16T15:05:04.383570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.365738+0000","last_change":"2024-01-16T15:05:03.365738+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.365738+0000","last_undegraded":"2024-01-16T15:05:03.365738+0000","last_fullsized":"2024-01-16T15:05:03.365738+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.382926+0000","last_change":"2024-01-16T15:05:04.382926+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.382926+0000","last_undegraded":"2024-01-16T15:05:04.382926+0000","last_fullsized":"2024-01-16T15:05:04.382926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.368603+0000","last_change":"2024-01-16T15:05:03.368603+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.368603+0000","last_undegraded":"2024-01-16T15:05:03.368603+0000","last_fullsized":"2024-01-16T15:05:03.368603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.382932+0000","last_change":"2024-01-16T15:05:04.382932+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.382932+0000","last_undegraded":"2024-01-16T15:05:04.382932+0000","last_fullsized":"2024-01-16T15:05:04.382932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.395452+0000","last_change":"2024-01-16T15:05:04.393679+0000","last_active":"2024-01-16T15:05:04.395452+0000","last_peered":"2024-01-16T15:05:04.395452+0000","last_clean":"2024-01-16T15:05:04.395452+0000","last_became_active":"2024-01-16T15:05:04.392878+0000","last_became_peered":"2024-01-16T15:05:04.392878+0000","last_unstale":"2024-01-16T15:05:04.395452+0000","last_undegraded":"2024-01-16T15:05:04.395452+0000","last_fullsized":"2024-01-16T15:05:04.395452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:26.691809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.386891+0000","last_change":"2024-01-16T15:05:04.386891+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.386891+0000","last_undegraded":"2024-01-16T15:05:04.386891+0000","last_fullsized":"2024-01-16T15:05:04.386891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.367409+0000","last_change":"2024-01-16T15:05:03.367409+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.367409+0000","last_undegraded":"2024-01-16T15:05:03.367409+0000","last_fullsized":"2024-01-16T15:05:03.367409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.369124+0000","last_change":"2024-01-16T15:05:03.369124+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.369124+0000","last_undegraded":"2024-01-16T15:05:03.369124+0000","last_fullsized":"2024-01-16T15:05:03.369124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.395430+0000","last_change":"2024-01-16T15:05:04.393333+0000","last_active":"2024-01-16T15:05:04.395430+0000","last_peered":"2024-01-16T15:05:04.395430+0000","last_clean":"2024-01-16T15:05:04.395430+0000","last_became_active":"2024-01-16T15:05:04.393011+0000","last_became_peered":"2024-01-16T15:05:04.393011+0000","last_unstale":"2024-01-16T15:05:04.395430+0000","last_undegraded":"2024-01-16T15:05:04.395430+0000","last_fullsized":"2024-01-16T15:05:04.395430+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:54.643297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.401768+0000","last_change":"2024-01-16T15:05:04.399310+0000","last_active":"2024-01-16T15:05:04.401768+0000","last_peered":"2024-01-16T15:05:04.401768+0000","last_clean":"2024-01-16T15:05:04.401768+0000","last_became_active":"2024-01-16T15:05:04.399004+0000","last_became_peered":"2024-01-16T15:05:04.399004+0000","last_unstale":"2024-01-16T15:05:04.401768+0000","last_undegraded":"2024-01-16T15:05:04.401768+0000","last_fullsized":"2024-01-16T15:05:04.401768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:29.089830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.382968+0000","last_change":"2024-01-16T15:05:04.382968+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.382968+0000","last_undegraded":"2024-01-16T15:05:04.382968+0000","last_fullsized":"2024-01-16T15:05:04.382968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":3,"reported_epoch":68,"state":"creating+activating","last_fresh":"2024-01-16T15:05:04.377920+0000","last_change":"2024-01-16T15:05:04.377920+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.377920+0000","last_undegraded":"2024-01-16T15:05:04.377920+0000","last_fullsized":"2024-01-16T15:05:04.377920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:48:31.473394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-16T15:05:04.400396+0000","last_change":"2024-01-16T15:05:04.397840+0000","last_active":"2024-01-16T15:05:04.400396+0000","last_peered":"2024-01-16T15:05:04.400396+0000","last_clean":"2024-01-16T15:05:04.400396+0000","last_became_active":"2024-01-16T15:05:04.397170+0000","last_became_peered":"2024-01-16T15:05:04.397170+0000","last_unstale":"2024-01-16T15:05:04.400396+0000","last_undegraded":"2024-01-16T15:05:04.400396+0000","last_fullsized":"2024-01-16T15:05:04.400396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:44.500105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.388135+0000","last_change":"2024-01-16T15:05:04.388135+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.388135+0000","last_undegraded":"2024-01-16T15:05:04.388135+0000","last_fullsized":"2024-01-16T15:05:04.388135+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.360661+0000","last_change":"2024-01-16T15:05:03.360661+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.360661+0000","last_undegraded":"2024-01-16T15:05:03.360661+0000","last_fullsized":"2024-01-16T15:05:03.360661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:05:04.356166+0000","last_change":"2024-01-16T15:05:04.356166+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.356166+0000","last_undegraded":"2024-01-16T15:05:04.356166+0000","last_fullsized":"2024-01-16T15:05:04.356166+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.365261+0000","last_change":"2024-01-16T15:05:03.365261+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.365261+0000","last_undegraded":"2024-01-16T15:05:03.365261+0000","last_fullsized":"2024-01-16T15:05:03.365261+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-16T15:05:04.383361+0000","last_change":"2024-01-16T15:05:04.383361+0000","last_active":"2024-01-16T15:05:04.356166+0000","last_peered":"2024-01-16T15:05:04.356166+0000","last_clean":"2024-01-16T15:05:04.356166+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:04.383361+0000","last_undegraded":"2024-01-16T15:05:04.383361+0000","last_fullsized":"2024-01-16T15:05:04.383361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-16T15:05:03.366664+0000","last_change":"2024-01-16T15:05:03.366664+0000","last_active":"2024-01-16T15:05:03.340743+0000","last_peered":"2024-01-16T15:05:03.340743+0000","last_clean":"2024-01-16T15:05:03.340743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:05:03.366664+0000","last_undegraded":"2024-01-16T15:05:03.366664+0000","last_fullsized":"2024-01-16T15:05:03.366664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":216,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-01-16T15:05:00.333749+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:05:00.333749+0000","last_peered":"2024-01-16T15:05:00.333749+0000","last_clean":"2024-01-16T15:05:00.333749+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:05:00.333749+0000","last_undegraded":"2024-01-16T15:05:00.333749+0000","last_fullsized":"2024-01-16T15:05:00.333749+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":4},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50604,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321236,"statfs":{"total":96636764160,"available":96584945664,"internally_reserved":0,"allocated":1409024,"data_stored":930144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.9550000000000001,"15min":1.3320000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":8.5399999999999991,"5min":45.296999999999997,"15min":45.296999999999997},"last":1.288},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.5620000000000001,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":2.073,"5min":20.126999999999999,"15min":20.126999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.768,"15min":1.331},"min":{"1min":0.65700000000000003,"5min":0.499,"15min":0.499},"max":{"1min":6.6669999999999998,"5min":42.723999999999997,"15min":42.723999999999997},"last":1.196},{"interface":"front","average":{"1min":1.569,"5min":2.1600000000000001,"15min":1.5309999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.1820000000000004,"5min":43.572000000000003,"15min":43.572000000000003},"last":1.472}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.7609999999999999,"15min":1.3839999999999999},"min":{"1min":0.67100000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.8580000000000001,"5min":35.308,"15min":35.308},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.706,"5min":1.8480000000000001,"15min":1.4350000000000001},"min":{"1min":0.65800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":9.1899999999999995,"5min":23.594000000000001,"15min":23.594000000000001},"last":1.0760000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":2.0920000000000001,"15min":1.512},"min":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":7.1520000000000001,"5min":53.033999999999999,"15min":53.033999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.9650000000000001,"15min":1.5700000000000001},"min":{"1min":0.76200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":14.708,"5min":31.305,"15min":31.305},"last":1.6160000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2889999999999999,"15min":1.373},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.2410000000000001,"5min":3.5939999999999999,"15min":5.1870000000000003},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.355,"15min":1.4810000000000001},"min":{"1min":0.48199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.5070000000000001,"5min":7.8360000000000003,"15min":36.374000000000002},"last":1.137}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.5589999999999999,"15min":1.4350000000000001},"min":{"1min":0.55500000000000005,"5min":0.45700000000000002,"15min":0.41499999999999998},"max":{"1min":9.1370000000000005,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.54,"15min":1.4570000000000001},"min":{"1min":0.61599999999999999,"5min":0.53500000000000003,"15min":0.46899999999999997},"max":{"1min":2.1360000000000001,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":1.8939999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5489999999999999,"15min":1.532},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":4.6600000000000001,"5min":7.9349999999999996,"15min":7.9349999999999996},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.853,"5min":1.657,"15min":1.6220000000000001},"min":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.49099999999999999},"max":{"1min":8.4849999999999994,"5min":8.4849999999999994,"15min":8.4849999999999994},"last":1.9490000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45868,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325972,"statfs":{"total":96636764160,"available":96589795328,"internally_reserved":0,"allocated":815104,"data_stored":339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":2.4079999999999999,"15min":1.532},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60099999999999998},"max":{"1min":3.9260000000000002,"5min":71.870999999999995,"15min":71.870999999999995},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.6219999999999999,"15min":1.7030000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":5.7130000000000001,"5min":71.846999999999994,"15min":71.846999999999994},"last":1.4059999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.353,"15min":1.159},"min":{"1min":0.81899999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.5789999999999997,"5min":7.5949999999999998,"15min":7.5949999999999998},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.829,"5min":1.609,"15min":1.294},"min":{"1min":0.70899999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":12.436999999999999,"5min":13.839,"15min":13.839},"last":0.69699999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.448,"15min":1.292},"min":{"1min":0.71399999999999997,"5min":0.63800000000000001,"15min":0.63400000000000001},"max":{"1min":3.819,"5min":7.1740000000000004,"15min":7.1740000000000004},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.4490000000000001,"15min":1.337},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.8500000000000001,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":0.89600000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.9059999999999999,"5min":1.726,"15min":1.4910000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":9.2170000000000005,"5min":14.827,"15min":14.827},"last":1.528},{"interface":"front","average":{"1min":2.0710000000000002,"5min":1.718,"15min":1.506},"min":{"1min":0.89400000000000002,"5min":0.83899999999999997,"15min":0.57899999999999996},"max":{"1min":9.2460000000000004,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":1.167}]},{"osd":4,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4379999999999999,"15min":1.4339999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":3.6909999999999998,"5min":7.4320000000000004,"15min":7.4320000000000004},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.516,"15min":1.48},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.605,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":1.343}]},{"osd":5,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4079999999999999,"15min":1.431},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.38500000000000001},"max":{"1min":3.855,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.673,"15min":1.6140000000000001},"min":{"1min":0.77600000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":3.7269999999999999,"5min":7.508,"15min":7.508},"last":1.3700000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.7230000000000001,"15min":1.6359999999999999},"min":{"1min":0.61199999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.966,"5min":14.74,"15min":14.74},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.569,"15min":1.573},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.4490000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574954,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45868,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325972,"statfs":{"total":96636764160,"available":96589795328,"internally_reserved":0,"allocated":815104,"data_stored":339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.748,"15min":1.274},"min":{"1min":0.59099999999999997,"5min":0.46800000000000003,"15min":0.44400000000000001},"max":{"1min":5.3540000000000001,"5min":25.129999999999999,"15min":25.129999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.794,"15min":1.3740000000000001},"min":{"1min":0.66300000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":3.6659999999999999,"5min":34.959000000000003,"15min":34.959000000000003},"last":1.175}]},{"osd":1,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.3089999999999999,"15min":1.29},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.49099999999999999},"max":{"1min":1.9379999999999999,"5min":10.180999999999999,"15min":11.298999999999999},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.413,"15min":1.252},"min":{"1min":0.67500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.613,"5min":13.332000000000001,"15min":13.332000000000001},"last":1.5049999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.357,"15min":1.319},"min":{"1min":0.628,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.0670000000000002,"5min":10.321999999999999,"15min":10.321999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.6180000000000001,"15min":1.393},"min":{"1min":0.71099999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":6.7599999999999998,"5min":17.030000000000001,"15min":17.030000000000001},"last":1.7450000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.0030000000000001,"15min":1.575},"min":{"1min":0.625,"5min":0.54100000000000004,"15min":0.53300000000000003},"max":{"1min":7.3840000000000003,"5min":18.372,"15min":18.372},"last":1.1000000000000001},{"interface":"front","average":{"1min":2.2320000000000002,"5min":2.1019999999999999,"15min":1.605},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":17.713999999999999,"5min":22.963999999999999,"15min":22.963999999999999},"last":1.3500000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.341,"15min":1.27},"min":{"1min":0.45500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.7889999999999999,"5min":8.6050000000000004,"15min":8.6050000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.327,"5min":1.504,"15min":1.496},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.869,"5min":7.2229999999999999,"15min":7.2229999999999999},"last":1.304}]},{"osd":6,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4730000000000001,"15min":1.498},"min":{"1min":0.77200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.4220000000000002,"5min":13.404999999999999,"15min":13.404999999999999},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.357,"15min":1.3839999999999999},"min":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.0310000000000001,"5min":8.7070000000000007,"15min":10.015000000000001},"last":1.0680000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":1.679,"15min":1.597},"min":{"1min":0.47899999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":21.422000000000001,"5min":21.422000000000001,"15min":21.422000000000001},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.792,"15min":1.6479999999999999},"min":{"1min":0.54800000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.956,"5min":21.106000000000002,"15min":21.106000000000002},"last":1.6120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50024,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321816,"statfs":{"total":96636764160,"available":96585539584,"internally_reserved":0,"allocated":802816,"data_stored":333959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.2649999999999999},"min":{"1min":0.752,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":5.4560000000000004,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.4279999999999999,"15min":1.3129999999999999},"min":{"1min":0.88800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":5.524,"5min":7.2190000000000003,"15min":7.2190000000000003},"last":1.774}]},{"osd":1,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.853,"15min":1.353},"min":{"1min":0.68799999999999994,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":8.9179999999999993,"5min":39.823999999999998,"15min":39.823999999999998},"last":1.649},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.796,"15min":1.3560000000000001},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":6.9429999999999996,"5min":50.582000000000001,"15min":50.582000000000001},"last":0.93300000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.325,"15min":1.2549999999999999},"min":{"1min":0.629,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.7930000000000001,"5min":5.4619999999999997,"15min":8.0579999999999998},"last":1.125},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4419999999999999,"15min":1.347},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":3.0289999999999999,"5min":9.1509999999999998,"15min":9.1509999999999998},"last":1.704}]},{"osd":3,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.448,"15min":1.331},"min":{"1min":0.59699999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":6.2190000000000003,"5min":11.971,"15min":11.971},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.968,"15min":1.625},"min":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":9.4329999999999998,"5min":27.626999999999999,"15min":27.626999999999999},"last":1.4710000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.405,"15min":1.397},"min":{"1min":0.51800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.2799999999999998,"5min":8.2599999999999998,"15min":8.2599999999999998},"last":1.571},{"interface":"front","average":{"1min":1.385,"5min":1.3580000000000001,"15min":1.4950000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":4.2430000000000003,"5min":4.2430000000000003,"15min":11.289999999999999},"last":1.5189999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.458,"15min":1.512},"min":{"1min":0.55200000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":3.0089999999999999,"5min":5.9210000000000003,"15min":6.9630000000000001},"last":1.802},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.45,"15min":1.5449999999999999},"min":{"1min":0.58999999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.3700000000000001,"5min":2.9460000000000002,"15min":3.2770000000000001},"last":1.272}]},{"osd":7,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.444,"15min":1.5589999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":2.6400000000000001,"5min":3.048,"15min":3.048},"last":1.671},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.879,"15min":1.76},"min":{"1min":0.63400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.093,"5min":19.001000000000001,"15min":19.001000000000001},"last":1.964}]}]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50024,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321816,"statfs":{"total":96636764160,"available":96585539584,"internally_reserved":0,"allocated":815104,"data_stored":339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.5369999999999999,"15min":1.1240000000000001},"min":{"1min":0.54500000000000004,"5min":0.313,"15min":0.313},"max":{"1min":3.0139999999999998,"5min":38.423999999999999,"15min":38.423999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.133,"5min":1.405,"15min":1.097},"min":{"1min":0.48499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.4689999999999999,"5min":20.012,"15min":20.012},"last":1.3380000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":3.379,"5min":2.907,"15min":1.5800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.215,"5min":40.695999999999998,"15min":40.695999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.323,"15min":1.149},"min":{"1min":0.57399999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.879,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.94799999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.1110000000000002,"15min":1.3979999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.1720000000000002,"5min":59.009,"15min":59.009},"last":1.456},{"interface":"front","average":{"1min":3.5299999999999998,"5min":2.637,"15min":1.611},"min":{"1min":0.623,"5min":0.38,"15min":0.38},"max":{"1min":19.152000000000001,"5min":29.773,"15min":29.773},"last":1.071}]},{"osd":4,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.591,"15min":1.4350000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8740000000000001,"5min":21.736000000000001,"15min":21.736000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":3.7469999999999999,"5min":2.9079999999999999,"15min":1.8620000000000001},"min":{"1min":0.80000000000000004,"5min":0.752,"15min":0.752},"max":{"1min":19.292000000000002,"5min":29.524000000000001,"15min":29.524000000000001},"last":1.6080000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.617,"15min":1.508},"min":{"1min":0.65100000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.9239999999999999,"5min":9.8249999999999993,"15min":9.8249999999999993},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7210000000000001,"15min":1.5489999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":4.9119999999999999,"5min":9.9009999999999998,"15min":9.9009999999999998},"last":1.234}]},{"osd":6,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.7370000000000001,"15min":1.5229999999999999},"min":{"1min":0.81599999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.966,"5min":20.459,"15min":20.459},"last":1.7529999999999999},{"interface":"front","average":{"1min":3.8460000000000001,"5min":2.944,"15min":1.9550000000000001},"min":{"1min":0.84899999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":19.254999999999999,"5min":29.475000000000001,"15min":29.475000000000001},"last":1.504}]},{"osd":7,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.784,"15min":1.5660000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.9470000000000001,"5min":20.109999999999999,"15min":20.109999999999999},"last":1.655},{"interface":"front","average":{"1min":3.8519999999999999,"5min":2.887,"15min":1.968},"min":{"1min":0.98399999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":19.327999999999999,"5min":29.396999999999998,"15min":29.396999999999998},"last":1.7130000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45868,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325972,"statfs":{"total":96636764160,"available":96589795328,"internally_reserved":0,"allocated":815104,"data_stored":339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.1910000000000001,"15min":1.105},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28899999999999998},"max":{"1min":1.3400000000000001,"5min":25.335999999999999,"15min":25.335999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.552,"15min":1.248},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":25.199000000000002,"5min":25.199000000000002,"15min":25.199000000000002},"last":0.755}]},{"osd":1,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.27,"15min":1.234},"min":{"1min":0.442,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":3.1240000000000001,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":0.82299999999999995},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.6799999999999999,"15min":1.446},"min":{"1min":0.39100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":19.312000000000001,"5min":21.439,"15min":21.439},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.1970000000000001,"15min":1.198},"min":{"1min":0.41699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.7750000000000004,"5min":8.1980000000000004,"15min":8.1980000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.179,"5min":1.464,"15min":1.4079999999999999},"min":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.46500000000000002},"max":{"1min":3.3119999999999998,"5min":28.045999999999999,"15min":28.045999999999999},"last":1.389}]},{"osd":4,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3380000000000001,"5min":1.994,"15min":1.649},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.803000000000001,"5min":21.532,"15min":21.532},"last":1.498},{"interface":"front","average":{"1min":1.792,"5min":1.756,"15min":1.5389999999999999},"min":{"1min":0.57599999999999996,"5min":0.502,"15min":0.502},"max":{"1min":14.106999999999999,"5min":14.597,"15min":14.597},"last":1.3200000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.655,"15min":1.577},"min":{"1min":0.629,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.524,"5min":19.327000000000002,"15min":19.327000000000002},"last":1.234},{"interface":"front","average":{"1min":2.379,"5min":1.9750000000000001,"15min":1.6930000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":20.843,"5min":21.608000000000001,"15min":21.608000000000001},"last":1.2649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.6040000000000001,"15min":1.5209999999999999},"min":{"1min":0.68400000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7789999999999999,"5min":12.57,"15min":12.57},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.5269999999999999,"15min":1.603},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":3.1190000000000002,"5min":3.871,"15min":4.8849999999999998},"last":1.137}]},{"osd":7,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":2.464,"5min":2.0830000000000002,"15min":1.7889999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":20.742000000000001,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.361},{"interface":"front","average":{"1min":1.288,"5min":1.6539999999999999,"15min":1.621},"min":{"1min":0.72499999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6830000000000001,"5min":12.456,"15min":12.456},"last":1.4410000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50036,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321804,"statfs":{"total":96636764160,"available":96585527296,"internally_reserved":0,"allocated":827392,"data_stored":346051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.3740000000000001,"15min":1.0960000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.28799999999999998},"max":{"1min":17.210999999999999,"5min":17.210999999999999,"15min":17.210999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.921,"15min":1.244},"min":{"1min":0.34300000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":20.164000000000001,"5min":34.252000000000002,"15min":34.252000000000002},"last":1.1679999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.927,"15min":1.2490000000000001},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.36099999999999999},"max":{"1min":24.102,"5min":24.709,"15min":24.709},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.982,"5min":1.8240000000000001,"15min":1.337},"min":{"1min":0.60499999999999998,"5min":0.46600000000000003,"15min":0.377},"max":{"1min":16.988,"5min":30.128,"15min":30.128},"last":0.80800000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.573,"15min":1.3240000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":4.242,"5min":1.944,"15min":1.361},"min":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":41.972000000000001,"5min":41.972000000000001,"15min":41.972000000000001},"last":1.0720000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":1.7549999999999999,"15min":1.425},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":18.004999999999999,"5min":18.004999999999999,"15min":18.004999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":2.3700000000000001,"5min":1.746,"15min":1.498},"min":{"1min":0.83899999999999997,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":20.187999999999999,"5min":20.187999999999999,"15min":20.187999999999999},"last":1.302}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.5950000000000002,"5min":1.895,"15min":1.5760000000000001},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":20.905999999999999,"5min":20.905999999999999,"15min":20.905999999999999},"last":1.385},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.0499999999999998,"15min":1.629},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":17.957999999999998,"5min":17.957999999999998,"15min":17.957999999999998},"last":1.5760000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.829,"15min":1.611},"min":{"1min":1.087,"5min":0.78200000000000003,"15min":0.73099999999999998},"max":{"1min":20.276,"5min":20.276,"15min":20.276},"last":1.2589999999999999},{"interface":"front","average":{"1min":2.593,"5min":1.7629999999999999,"15min":1.5329999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":20.509,"5min":20.509,"15min":20.509},"last":1.518}]},{"osd":7,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.7949999999999999,"5min":1.9359999999999999,"15min":1.653},"min":{"1min":0.97399999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":20.669,"5min":20.669,"15min":20.669},"last":1.319},{"interface":"front","average":{"1min":2.363,"5min":1.95,"15min":1.6240000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":18.039999999999999,"5min":18.039999999999999,"15min":18.039999999999999},"last":1.349}]}]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50600,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321240,"statfs":{"total":96636764160,"available":96584949760,"internally_reserved":0,"allocated":1404928,"data_stored":929125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.5600000000000001,"15min":1.119},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":6.2889999999999997,"5min":26.751000000000001,"15min":26.751000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.4350000000000001,"15min":1.103},"min":{"1min":0.45000000000000001,"5min":0.42199999999999999,"15min":0.35899999999999999},"max":{"1min":12.648999999999999,"5min":15.750999999999999,"15min":15.750999999999999},"last":1.2849999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":2.0129999999999999,"15min":1.3680000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.9159999999999999,"5min":38.517000000000003,"15min":38.517000000000003},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.544,"15min":1.181},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.161,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.93200000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":3.25,"5min":2.1110000000000002,"15min":1.3540000000000001},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":29.808,"5min":29.808,"15min":29.808},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":2.4060000000000001,"15min":1.615},"min":{"1min":0.51700000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.1020000000000003,"5min":32.302,"15min":32.302},"last":1.587}]},{"osd":4,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.792,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.70799999999999996},"max":{"1min":9.4649999999999999,"5min":15.832000000000001,"15min":15.832000000000001},"last":1.72},{"interface":"front","average":{"1min":2.452,"5min":1.9159999999999999,"15min":1.4670000000000001},"min":{"1min":0.68200000000000005,"5min":0.432,"15min":0.432},"max":{"1min":14.685,"5min":19.062999999999999,"15min":19.062999999999999},"last":1.768}]},{"osd":5,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":2.024,"15min":1.669},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.8879999999999999,"5min":26.800000000000001,"15min":26.800000000000001},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.6930000000000001,"15min":1.522},"min":{"1min":0.80100000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":4.1289999999999996,"5min":15.906000000000001,"15min":15.906000000000001},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5470000000000002,"5min":2.181,"15min":1.6200000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":14.772,"5min":20.844000000000001,"15min":20.844000000000001},"last":1.494},{"interface":"front","average":{"1min":1.613,"5min":2.0499999999999998,"15min":1.6879999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.3200000000000003,"5min":26.832000000000001,"15min":26.832000000000001},"last":1.8520000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.8799999999999999,"15min":1.5940000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":14.789,"5min":16.039999999999999,"15min":16.039999999999999},"last":1.8129999999999999},{"interface":"front","average":{"1min":2.6000000000000001,"5min":2.5379999999999998,"15min":1.823},"min":{"1min":0.75,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":14.811,"5min":29.286999999999999,"15min":29.286999999999999},"last":1.6359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:09.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:10.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.111+0000 7f2106400640 1 Processor -- start 2024-01-16T15:05:10.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.112+0000 7f2106400640 1 -- start start 2024-01-16T15:05:10.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f210013df80 0x7f210015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013e360 0x7f21000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f210015f9c0 0x7f21000cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210006c470 con 0x7f210015f9c0 2024-01-16T15:05:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210006c750 con 0x7f210013df80 2024-01-16T15:05:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.113+0000 7f2106400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210006ca30 con 0x7f210013e360 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f210015f9c0 0x7f21000cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f210015f9c0 0x7f21000cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51860/0 (socket says 172.21.15.125:51860) 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20fea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013e360 0x7f21000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 -- 172.21.15.125:0/531763512 learned_addr learned my addr 172.21.15.125:0/531763512 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ff400640 1 --2- 172.21.15.125:0/531763512 >> v2:172.21.15.137:3300/0 conn(0x7f210013df80 0x7f210015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 -- 172.21.15.125:0/531763512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013e360 msgr2=0x7f21000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 --2- 172.21.15.125:0/531763512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013e360 0x7f21000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 -- 172.21.15.125:0/531763512 >> v2:172.21.15.137:3300/0 conn(0x7f210013df80 msgr2=0x7f210015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 --2- 172.21.15.125:0/531763512 >> v2:172.21.15.137:3300/0 conn(0x7f210013df80 0x7f210015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.114+0000 7f20ffe00640 1 -- 172.21.15.125:0/531763512 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210006cd10 con 0x7f210015f9c0 2024-01-16T15:05:10.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.115+0000 7f20ffe00640 1 --2- 172.21.15.125:0/531763512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210015f9c0 0x7f21000cf7d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2100072770 tx=0x7f20f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7f25cea31a8ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.116+0000 7f20fe000640 1 -- 172.21.15.125:0/531763512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8031870 con 0x7f210015f9c0 2024-01-16T15:05:10.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.116+0000 7f20fe000640 1 -- 172.21.15.125:0/531763512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f80319d0 con 0x7f210015f9c0 2024-01-16T15:05:10.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.116+0000 7f20fe000640 1 -- 172.21.15.125:0/531763512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8031d60 con 0x7f210015f9c0 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.116+0000 7f2106400640 1 -- 172.21.15.125:0/531763512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210015f9c0 msgr2=0x7f21000cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.116+0000 7f2106400640 1 --2- 172.21.15.125:0/531763512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210015f9c0 0x7f21000cf7d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2100072770 tx=0x7f20f802b910 comp rx=0 tx=0).stop 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 -- 172.21.15.125:0/531763512 shutdown_connections 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 --2- 172.21.15.125:0/531763512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210015f9c0 0x7f21000cf7d0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 --2- 172.21.15.125:0/531763512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013e360 0x7f21000cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 --2- 172.21.15.125:0/531763512 >> v2:172.21.15.137:3300/0 conn(0x7f210013df80 0x7f210015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 -- 172.21.15.125:0/531763512 >> 172.21.15.125:0/531763512 conn(0x7f210006a800 msgr2=0x7f210006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 -- 172.21.15.125:0/531763512 shutdown_connections 2024-01-16T15:05:10.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.117+0000 7f2106400640 1 -- 172.21.15.125:0/531763512 wait complete. 2024-01-16T15:05:10.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.118+0000 7f2106400640 1 Processor -- start 2024-01-16T15:05:10.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.119+0000 7f2106400640 1 -- start start 2024-01-16T15:05:10.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013df80 0x7f2100154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f210015f9c0 0x7f21001f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21001527e0 con 0x7f210013e360 2024-01-16T15:05:10.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b2310 con 0x7f210015f9c0 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f2106400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b25f0 con 0x7f210013df80 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51876/0 (socket says 172.21.15.125:51876) 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 -- 172.21.15.125:0/607073200 learned_addr learned my addr 172.21.15.125:0/607073200 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20ff400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013df80 0x7f2100154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20ffe00640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f210015f9c0 0x7f21001f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 -- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013df80 msgr2=0x7f2100154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013df80 0x7f2100154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 -- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f210015f9c0 msgr2=0x7f21001f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f210015f9c0 0x7f21001f04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.120+0000 7f20fea00640 1 -- 172.21.15.125:0/607073200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f80094d0 con 0x7f210013e360 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f20fea00640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f20f000efc0 tx=0x7f20f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0014480 con 0x7f210013e360 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f000f040 con 0x7f210013e360 2024-01-16T15:05:10.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0014800 con 0x7f210013e360 2024-01-16T15:05:10.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21000b28d0 con 0x7f210013e360 2024-01-16T15:05:10.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.121+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21000b2e90 con 0x7f210013e360 2024-01-16T15:05:10.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.122+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21000675d0 con 0x7f210013e360 2024-01-16T15:05:10.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.123+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f20f0014960 con 0x7f210013e360 2024-01-16T15:05:10.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.124+0000 7f20f7400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 0x7f20d806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.124+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20f0089450 con 0x7f210013e360 2024-01-16T15:05:10.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.124+0000 7f20ff400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 0x7f20d806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.125+0000 7f20ff400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 0x7f20d806dff0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009a30 tx=0x7f20e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.128+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20f0055ba0 con 0x7f210013e360 2024-01-16T15:05:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.317+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21001523c0 con 0x7f20d806bb30 2024-01-16T15:05:10.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.330+0000 7f20f7400640 1 -- 172.21.15.125:0/607073200 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286949 (secure 0 0 0) 0x7f21001523c0 con 0x7f20d806bb30 2024-01-16T15:05:10.312 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:10.314 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:05:10.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 msgr2=0x7f20d806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 0x7f20d806dff0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009a30 tx=0x7f20e8008040 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 msgr2=0x7f21000b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f20f000efc0 tx=0x7f20f000c490 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 shutdown_connections 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f210015f9c0 0x7f21001f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f20d806bb30 0x7f20d806dff0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210013e360 0x7f21000b1ca0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 --2- 172.21.15.125:0/607073200 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f210013df80 0x7f2100154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 >> 172.21.15.125:0/607073200 conn(0x7f210006a800 msgr2=0x7f210013e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 shutdown_connections 2024-01-16T15:05:10.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.336+0000 7f2106400640 1 -- 172.21.15.125:0/607073200 wait complete. 2024-01-16T15:05:10.334 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-01-16T15:05:10.109682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":389264,"kb_used_data":7876,"kb_used_omap":12,"kb_used_meta":381235,"kb_avail":754585456,"statfs":{"total":773094113280,"available":772695506944,"internally_reserved":0,"allocated":8065024,"data_stored":4057545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":390385200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001877"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.402244+0000","last_change":"2024-01-16T15:05:05.391815+0000","last_active":"2024-01-16T15:05:06.402244+0000","last_peered":"2024-01-16T15:05:06.402244+0000","last_clean":"2024-01-16T15:05:06.402244+0000","last_became_active":"2024-01-16T15:05:05.389066+0000","last_became_peered":"2024-01-16T15:05:05.389066+0000","last_unstale":"2024-01-16T15:05:06.402244+0000","last_undegraded":"2024-01-16T15:05:06.402244+0000","last_fullsized":"2024-01-16T15:05:06.402244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:51.979594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839438+0000","last_change":"2024-01-16T15:05:05.409332+0000","last_active":"2024-01-16T15:05:06.839438+0000","last_peered":"2024-01-16T15:05:06.839438+0000","last_clean":"2024-01-16T15:05:06.839438+0000","last_became_active":"2024-01-16T15:05:05.406751+0000","last_became_peered":"2024-01-16T15:05:05.406751+0000","last_unstale":"2024-01-16T15:05:06.839438+0000","last_undegraded":"2024-01-16T15:05:06.839438+0000","last_fullsized":"2024-01-16T15:05:06.839438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:30.105254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764573+0000","last_change":"2024-01-16T15:05:05.394876+0000","last_active":"2024-01-16T15:05:06.764573+0000","last_peered":"2024-01-16T15:05:06.764573+0000","last_clean":"2024-01-16T15:05:06.764573+0000","last_became_active":"2024-01-16T15:05:05.393799+0000","last_became_peered":"2024-01-16T15:05:05.393799+0000","last_unstale":"2024-01-16T15:05:06.764573+0000","last_undegraded":"2024-01-16T15:05:06.764573+0000","last_fullsized":"2024-01-16T15:05:06.764573+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:59:40.623632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.836978+0000","last_change":"2024-01-16T15:05:05.410712+0000","last_active":"2024-01-16T15:05:06.836978+0000","last_peered":"2024-01-16T15:05:06.836978+0000","last_clean":"2024-01-16T15:05:06.836978+0000","last_became_active":"2024-01-16T15:05:05.410065+0000","last_became_peered":"2024-01-16T15:05:05.410065+0000","last_unstale":"2024-01-16T15:05:06.836978+0000","last_undegraded":"2024-01-16T15:05:06.836978+0000","last_fullsized":"2024-01-16T15:05:06.836978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:09.630991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838709+0000","last_change":"2024-01-16T15:05:05.386201+0000","last_active":"2024-01-16T15:05:06.838709+0000","last_peered":"2024-01-16T15:05:06.838709+0000","last_clean":"2024-01-16T15:05:06.838709+0000","last_became_active":"2024-01-16T15:05:05.385848+0000","last_became_peered":"2024-01-16T15:05:05.385848+0000","last_unstale":"2024-01-16T15:05:06.838709+0000","last_undegraded":"2024-01-16T15:05:06.838709+0000","last_fullsized":"2024-01-16T15:05:06.838709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:45.247535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837817+0000","last_change":"2024-01-16T15:05:05.409535+0000","last_active":"2024-01-16T15:05:06.837817+0000","last_peered":"2024-01-16T15:05:06.837817+0000","last_clean":"2024-01-16T15:05:06.837817+0000","last_became_active":"2024-01-16T15:05:05.406669+0000","last_became_peered":"2024-01-16T15:05:05.406669+0000","last_unstale":"2024-01-16T15:05:06.837817+0000","last_undegraded":"2024-01-16T15:05:06.837817+0000","last_fullsized":"2024-01-16T15:05:06.837817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:29.909361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837240+0000","last_change":"2024-01-16T15:05:05.396470+0000","last_active":"2024-01-16T15:05:06.837240+0000","last_peered":"2024-01-16T15:05:06.837240+0000","last_clean":"2024-01-16T15:05:06.837240+0000","last_became_active":"2024-01-16T15:05:05.395190+0000","last_became_peered":"2024-01-16T15:05:05.395190+0000","last_unstale":"2024-01-16T15:05:06.837240+0000","last_undegraded":"2024-01-16T15:05:06.837240+0000","last_fullsized":"2024-01-16T15:05:06.837240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:55:15.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837491+0000","last_change":"2024-01-16T15:05:05.406015+0000","last_active":"2024-01-16T15:05:06.837491+0000","last_peered":"2024-01-16T15:05:06.837491+0000","last_clean":"2024-01-16T15:05:06.837491+0000","last_became_active":"2024-01-16T15:05:05.402774+0000","last_became_peered":"2024-01-16T15:05:05.402774+0000","last_unstale":"2024-01-16T15:05:06.837491+0000","last_undegraded":"2024-01-16T15:05:06.837491+0000","last_fullsized":"2024-01-16T15:05:06.837491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:33.292581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839613+0000","last_change":"2024-01-16T15:05:05.403141+0000","last_active":"2024-01-16T15:05:06.839613+0000","last_peered":"2024-01-16T15:05:06.839613+0000","last_clean":"2024-01-16T15:05:06.839613+0000","last_became_active":"2024-01-16T15:05:05.402703+0000","last_became_peered":"2024-01-16T15:05:05.402703+0000","last_unstale":"2024-01-16T15:05:06.839613+0000","last_undegraded":"2024-01-16T15:05:06.839613+0000","last_fullsized":"2024-01-16T15:05:06.839613+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:01.886903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.400529+0000","last_change":"2024-01-16T15:05:05.401094+0000","last_active":"2024-01-16T15:05:06.400529+0000","last_peered":"2024-01-16T15:05:06.400529+0000","last_clean":"2024-01-16T15:05:06.400529+0000","last_became_active":"2024-01-16T15:05:05.399888+0000","last_became_peered":"2024-01-16T15:05:05.399888+0000","last_unstale":"2024-01-16T15:05:06.400529+0000","last_undegraded":"2024-01-16T15:05:06.400529+0000","last_fullsized":"2024-01-16T15:05:06.400529+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:46.150689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.400563+0000","last_change":"2024-01-16T15:05:05.397065+0000","last_active":"2024-01-16T15:05:06.400563+0000","last_peered":"2024-01-16T15:05:06.400563+0000","last_clean":"2024-01-16T15:05:06.400563+0000","last_became_active":"2024-01-16T15:05:05.396782+0000","last_became_peered":"2024-01-16T15:05:05.396782+0000","last_unstale":"2024-01-16T15:05:06.400563+0000","last_undegraded":"2024-01-16T15:05:06.400563+0000","last_fullsized":"2024-01-16T15:05:06.400563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:57.490005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837196+0000","last_change":"2024-01-16T15:05:05.408436+0000","last_active":"2024-01-16T15:05:06.837196+0000","last_peered":"2024-01-16T15:05:06.837196+0000","last_clean":"2024-01-16T15:05:06.837196+0000","last_became_active":"2024-01-16T15:05:05.408167+0000","last_became_peered":"2024-01-16T15:05:05.408167+0000","last_unstale":"2024-01-16T15:05:06.837196+0000","last_undegraded":"2024-01-16T15:05:06.837196+0000","last_fullsized":"2024-01-16T15:05:06.837196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:59.697126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.393043+0000","last_change":"2024-01-16T15:05:05.403795+0000","last_active":"2024-01-16T15:05:06.393043+0000","last_peered":"2024-01-16T15:05:06.393043+0000","last_clean":"2024-01-16T15:05:06.393043+0000","last_became_active":"2024-01-16T15:05:05.403470+0000","last_became_peered":"2024-01-16T15:05:05.403470+0000","last_unstale":"2024-01-16T15:05:06.393043+0000","last_undegraded":"2024-01-16T15:05:06.393043+0000","last_fullsized":"2024-01-16T15:05:06.393043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:12.679118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838911+0000","last_change":"2024-01-16T15:05:05.400710+0000","last_active":"2024-01-16T15:05:06.838911+0000","last_peered":"2024-01-16T15:05:06.838911+0000","last_clean":"2024-01-16T15:05:06.838911+0000","last_became_active":"2024-01-16T15:05:05.397824+0000","last_became_peered":"2024-01-16T15:05:05.397824+0000","last_unstale":"2024-01-16T15:05:06.838911+0000","last_undegraded":"2024-01-16T15:05:06.838911+0000","last_fullsized":"2024-01-16T15:05:06.838911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:44.324652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.393331+0000","last_change":"2024-01-16T15:05:05.389689+0000","last_active":"2024-01-16T15:05:06.393331+0000","last_peered":"2024-01-16T15:05:06.393331+0000","last_clean":"2024-01-16T15:05:06.393331+0000","last_became_active":"2024-01-16T15:05:05.389275+0000","last_became_peered":"2024-01-16T15:05:05.389275+0000","last_unstale":"2024-01-16T15:05:06.393331+0000","last_undegraded":"2024-01-16T15:05:06.393331+0000","last_fullsized":"2024-01-16T15:05:06.393331+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:58.420066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.399115+0000","last_change":"2024-01-16T15:05:05.408653+0000","last_active":"2024-01-16T15:05:06.399115+0000","last_peered":"2024-01-16T15:05:06.399115+0000","last_clean":"2024-01-16T15:05:06.399115+0000","last_became_active":"2024-01-16T15:05:05.408300+0000","last_became_peered":"2024-01-16T15:05:05.408300+0000","last_unstale":"2024-01-16T15:05:06.399115+0000","last_undegraded":"2024-01-16T15:05:06.399115+0000","last_fullsized":"2024-01-16T15:05:06.399115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:42.861185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839666+0000","last_change":"2024-01-16T15:05:05.404216+0000","last_active":"2024-01-16T15:05:06.839666+0000","last_peered":"2024-01-16T15:05:06.839666+0000","last_clean":"2024-01-16T15:05:06.839666+0000","last_became_active":"2024-01-16T15:05:05.403942+0000","last_became_peered":"2024-01-16T15:05:05.403942+0000","last_unstale":"2024-01-16T15:05:06.839666+0000","last_undegraded":"2024-01-16T15:05:06.839666+0000","last_fullsized":"2024-01-16T15:05:06.839666+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:37.070048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837333+0000","last_change":"2024-01-16T15:05:05.409892+0000","last_active":"2024-01-16T15:05:06.837333+0000","last_peered":"2024-01-16T15:05:06.837333+0000","last_clean":"2024-01-16T15:05:06.837333+0000","last_became_active":"2024-01-16T15:05:05.409589+0000","last_became_peered":"2024-01-16T15:05:05.409589+0000","last_unstale":"2024-01-16T15:05:06.837333+0000","last_undegraded":"2024-01-16T15:05:06.837333+0000","last_fullsized":"2024-01-16T15:05:06.837333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:47.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838201+0000","last_change":"2024-01-16T15:05:05.404628+0000","last_active":"2024-01-16T15:05:06.838201+0000","last_peered":"2024-01-16T15:05:06.838201+0000","last_clean":"2024-01-16T15:05:06.838201+0000","last_became_active":"2024-01-16T15:05:05.401607+0000","last_became_peered":"2024-01-16T15:05:05.401607+0000","last_unstale":"2024-01-16T15:05:06.838201+0000","last_undegraded":"2024-01-16T15:05:06.838201+0000","last_fullsized":"2024-01-16T15:05:06.838201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:56:52.538120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.393078+0000","last_change":"2024-01-16T15:05:05.398701+0000","last_active":"2024-01-16T15:05:06.393078+0000","last_peered":"2024-01-16T15:05:06.393078+0000","last_clean":"2024-01-16T15:05:06.393078+0000","last_became_active":"2024-01-16T15:05:05.398267+0000","last_became_peered":"2024-01-16T15:05:05.398267+0000","last_unstale":"2024-01-16T15:05:06.393078+0000","last_undegraded":"2024-01-16T15:05:06.393078+0000","last_fullsized":"2024-01-16T15:05:06.393078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:38.145752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401379+0000","last_change":"2024-01-16T15:05:05.405639+0000","last_active":"2024-01-16T15:05:06.401379+0000","last_peered":"2024-01-16T15:05:06.401379+0000","last_clean":"2024-01-16T15:05:06.401379+0000","last_became_active":"2024-01-16T15:05:05.404121+0000","last_became_peered":"2024-01-16T15:05:05.404121+0000","last_unstale":"2024-01-16T15:05:06.401379+0000","last_undegraded":"2024-01-16T15:05:06.401379+0000","last_fullsized":"2024-01-16T15:05:06.401379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:53.990947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.399575+0000","last_change":"2024-01-16T15:05:05.386704+0000","last_active":"2024-01-16T15:05:06.399575+0000","last_peered":"2024-01-16T15:05:06.399575+0000","last_clean":"2024-01-16T15:05:06.399575+0000","last_became_active":"2024-01-16T15:05:05.386418+0000","last_became_peered":"2024-01-16T15:05:05.386418+0000","last_unstale":"2024-01-16T15:05:06.399575+0000","last_undegraded":"2024-01-16T15:05:06.399575+0000","last_fullsized":"2024-01-16T15:05:06.399575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:10.073739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401361+0000","last_change":"2024-01-16T15:05:05.399047+0000","last_active":"2024-01-16T15:05:06.401361+0000","last_peered":"2024-01-16T15:05:06.401361+0000","last_clean":"2024-01-16T15:05:06.401361+0000","last_became_active":"2024-01-16T15:05:05.398539+0000","last_became_peered":"2024-01-16T15:05:05.398539+0000","last_unstale":"2024-01-16T15:05:06.401361+0000","last_undegraded":"2024-01-16T15:05:06.401361+0000","last_fullsized":"2024-01-16T15:05:06.401361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:32.985457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.399187+0000","last_change":"2024-01-16T15:05:05.408802+0000","last_active":"2024-01-16T15:05:06.399187+0000","last_peered":"2024-01-16T15:05:06.399187+0000","last_clean":"2024-01-16T15:05:06.399187+0000","last_became_active":"2024-01-16T15:05:05.408523+0000","last_became_peered":"2024-01-16T15:05:05.408523+0000","last_unstale":"2024-01-16T15:05:06.399187+0000","last_undegraded":"2024-01-16T15:05:06.399187+0000","last_fullsized":"2024-01-16T15:05:06.399187+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:58.747034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.400870+0000","last_change":"2024-01-16T15:05:05.395423+0000","last_active":"2024-01-16T15:05:06.400870+0000","last_peered":"2024-01-16T15:05:06.400870+0000","last_clean":"2024-01-16T15:05:06.400870+0000","last_became_active":"2024-01-16T15:05:05.393506+0000","last_became_peered":"2024-01-16T15:05:05.393506+0000","last_unstale":"2024-01-16T15:05:06.400870+0000","last_undegraded":"2024-01-16T15:05:06.400870+0000","last_fullsized":"2024-01-16T15:05:06.400870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:18.864570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839928+0000","last_change":"2024-01-16T15:05:05.407602+0000","last_active":"2024-01-16T15:05:06.839928+0000","last_peered":"2024-01-16T15:05:06.839928+0000","last_clean":"2024-01-16T15:05:06.839928+0000","last_became_active":"2024-01-16T15:05:05.407293+0000","last_became_peered":"2024-01-16T15:05:05.407293+0000","last_unstale":"2024-01-16T15:05:06.839928+0000","last_undegraded":"2024-01-16T15:05:06.839928+0000","last_fullsized":"2024-01-16T15:05:06.839928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:58.468477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764676+0000","last_change":"2024-01-16T15:05:05.404556+0000","last_active":"2024-01-16T15:05:06.764676+0000","last_peered":"2024-01-16T15:05:06.764676+0000","last_clean":"2024-01-16T15:05:06.764676+0000","last_became_active":"2024-01-16T15:05:05.404288+0000","last_became_peered":"2024-01-16T15:05:05.404288+0000","last_unstale":"2024-01-16T15:05:06.764676+0000","last_undegraded":"2024-01-16T15:05:06.764676+0000","last_fullsized":"2024-01-16T15:05:06.764676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:26.694522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.400834+0000","last_change":"2024-01-16T15:05:05.400372+0000","last_active":"2024-01-16T15:05:06.400834+0000","last_peered":"2024-01-16T15:05:06.400834+0000","last_clean":"2024-01-16T15:05:06.400834+0000","last_became_active":"2024-01-16T15:05:05.399901+0000","last_became_peered":"2024-01-16T15:05:05.399901+0000","last_unstale":"2024-01-16T15:05:06.400834+0000","last_undegraded":"2024-01-16T15:05:06.400834+0000","last_fullsized":"2024-01-16T15:05:06.400834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:07.604455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764450+0000","last_change":"2024-01-16T15:05:05.402813+0000","last_active":"2024-01-16T15:05:06.764450+0000","last_peered":"2024-01-16T15:05:06.764450+0000","last_clean":"2024-01-16T15:05:06.764450+0000","last_became_active":"2024-01-16T15:05:05.401287+0000","last_became_peered":"2024-01-16T15:05:05.401287+0000","last_unstale":"2024-01-16T15:05:06.764450+0000","last_undegraded":"2024-01-16T15:05:06.764450+0000","last_fullsized":"2024-01-16T15:05:06.764450+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:02.382508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838175+0000","last_change":"2024-01-16T15:05:05.411447+0000","last_active":"2024-01-16T15:05:06.838175+0000","last_peered":"2024-01-16T15:05:06.838175+0000","last_clean":"2024-01-16T15:05:06.838175+0000","last_became_active":"2024-01-16T15:05:05.409802+0000","last_became_peered":"2024-01-16T15:05:05.409802+0000","last_unstale":"2024-01-16T15:05:06.838175+0000","last_undegraded":"2024-01-16T15:05:06.838175+0000","last_fullsized":"2024-01-16T15:05:06.838175+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:38.145752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838203+0000","last_change":"2024-01-16T15:05:05.397505+0000","last_active":"2024-01-16T15:05:06.838203+0000","last_peered":"2024-01-16T15:05:06.838203+0000","last_clean":"2024-01-16T15:05:06.838203+0000","last_became_active":"2024-01-16T15:05:05.397225+0000","last_became_peered":"2024-01-16T15:05:05.397225+0000","last_unstale":"2024-01-16T15:05:06.838203+0000","last_undegraded":"2024-01-16T15:05:06.838203+0000","last_fullsized":"2024-01-16T15:05:06.838203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:12.679118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837571+0000","last_change":"2024-01-16T15:05:05.398948+0000","last_active":"2024-01-16T15:05:06.837571+0000","last_peered":"2024-01-16T15:05:06.837571+0000","last_clean":"2024-01-16T15:05:06.837571+0000","last_became_active":"2024-01-16T15:05:05.398579+0000","last_became_peered":"2024-01-16T15:05:05.398579+0000","last_unstale":"2024-01-16T15:05:06.837571+0000","last_undegraded":"2024-01-16T15:05:06.837571+0000","last_fullsized":"2024-01-16T15:05:06.837571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.594259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.402304+0000","last_change":"2024-01-16T15:05:05.407065+0000","last_active":"2024-01-16T15:05:06.402304+0000","last_peered":"2024-01-16T15:05:06.402304+0000","last_clean":"2024-01-16T15:05:06.402304+0000","last_became_active":"2024-01-16T15:05:05.406575+0000","last_became_peered":"2024-01-16T15:05:05.406575+0000","last_unstale":"2024-01-16T15:05:06.402304+0000","last_undegraded":"2024-01-16T15:05:06.402304+0000","last_fullsized":"2024-01-16T15:05:06.402304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:57:57.236920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.420678+0000","last_change":"2024-01-16T15:05:04.406791+0000","last_active":"2024-01-16T15:05:05.420678+0000","last_peered":"2024-01-16T15:05:05.420678+0000","last_clean":"2024-01-16T15:05:05.420678+0000","last_became_active":"2024-01-16T15:05:04.404993+0000","last_became_peered":"2024-01-16T15:05:04.404993+0000","last_unstale":"2024-01-16T15:05:05.420678+0000","last_undegraded":"2024-01-16T15:05:05.420678+0000","last_fullsized":"2024-01-16T15:05:05.420678+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:56.984663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839847+0000","last_change":"2024-01-16T15:05:05.402959+0000","last_active":"2024-01-16T15:05:06.839847+0000","last_peered":"2024-01-16T15:05:06.839847+0000","last_clean":"2024-01-16T15:05:06.839847+0000","last_became_active":"2024-01-16T15:05:05.399808+0000","last_became_peered":"2024-01-16T15:05:05.399808+0000","last_unstale":"2024-01-16T15:05:06.839847+0000","last_undegraded":"2024-01-16T15:05:06.839847+0000","last_fullsized":"2024-01-16T15:05:06.839847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:57.458340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.413247+0000","last_change":"2024-01-16T15:05:04.396236+0000","last_active":"2024-01-16T15:05:05.413247+0000","last_peered":"2024-01-16T15:05:05.413247+0000","last_clean":"2024-01-16T15:05:05.413247+0000","last_became_active":"2024-01-16T15:05:04.395960+0000","last_became_peered":"2024-01-16T15:05:04.395960+0000","last_unstale":"2024-01-16T15:05:05.413247+0000","last_undegraded":"2024-01-16T15:05:05.413247+0000","last_fullsized":"2024-01-16T15:05:05.413247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:23.123971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839728+0000","last_change":"2024-01-16T15:05:05.405139+0000","last_active":"2024-01-16T15:05:06.839728+0000","last_peered":"2024-01-16T15:05:06.839728+0000","last_clean":"2024-01-16T15:05:06.839728+0000","last_became_active":"2024-01-16T15:05:05.404836+0000","last_became_peered":"2024-01-16T15:05:05.404836+0000","last_unstale":"2024-01-16T15:05:06.839728+0000","last_undegraded":"2024-01-16T15:05:06.839728+0000","last_fullsized":"2024-01-16T15:05:06.839728+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:42.861185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.423704+0000","last_change":"2024-01-16T15:05:04.406559+0000","last_active":"2024-01-16T15:05:05.423704+0000","last_peered":"2024-01-16T15:05:05.423704+0000","last_clean":"2024-01-16T15:05:05.423704+0000","last_became_active":"2024-01-16T15:05:04.406194+0000","last_became_peered":"2024-01-16T15:05:04.406194+0000","last_unstale":"2024-01-16T15:05:05.423704+0000","last_undegraded":"2024-01-16T15:05:05.423704+0000","last_fullsized":"2024-01-16T15:05:05.423704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:28.893937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401905+0000","last_change":"2024-01-16T15:05:05.400945+0000","last_active":"2024-01-16T15:05:06.401905+0000","last_peered":"2024-01-16T15:05:06.401905+0000","last_clean":"2024-01-16T15:05:06.401905+0000","last_became_active":"2024-01-16T15:05:05.394883+0000","last_became_peered":"2024-01-16T15:05:05.394883+0000","last_unstale":"2024-01-16T15:05:06.401905+0000","last_undegraded":"2024-01-16T15:05:06.401905+0000","last_fullsized":"2024-01-16T15:05:06.401905+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:16:14.777202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.418629+0000","last_change":"2024-01-16T15:05:04.404847+0000","last_active":"2024-01-16T15:05:05.418629+0000","last_peered":"2024-01-16T15:05:05.418629+0000","last_clean":"2024-01-16T15:05:05.418629+0000","last_became_active":"2024-01-16T15:05:04.404521+0000","last_became_peered":"2024-01-16T15:05:04.404521+0000","last_unstale":"2024-01-16T15:05:05.418629+0000","last_undegraded":"2024-01-16T15:05:05.418629+0000","last_fullsized":"2024-01-16T15:05:05.418629+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:12.898182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838579+0000","last_change":"2024-01-16T15:05:05.400056+0000","last_active":"2024-01-16T15:05:06.838579+0000","last_peered":"2024-01-16T15:05:06.838579+0000","last_clean":"2024-01-16T15:05:06.838579+0000","last_became_active":"2024-01-16T15:05:05.399275+0000","last_became_peered":"2024-01-16T15:05:05.399275+0000","last_unstale":"2024-01-16T15:05:06.838579+0000","last_undegraded":"2024-01-16T15:05:06.838579+0000","last_fullsized":"2024-01-16T15:05:06.838579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:58.420066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.421552+0000","last_change":"2024-01-16T15:05:04.394127+0000","last_active":"2024-01-16T15:05:05.421552+0000","last_peered":"2024-01-16T15:05:05.421552+0000","last_clean":"2024-01-16T15:05:05.421552+0000","last_became_active":"2024-01-16T15:05:04.393732+0000","last_became_peered":"2024-01-16T15:05:04.393732+0000","last_unstale":"2024-01-16T15:05:05.421552+0000","last_undegraded":"2024-01-16T15:05:05.421552+0000","last_fullsized":"2024-01-16T15:05:05.421552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:50.964170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401063+0000","last_change":"2024-01-16T15:05:05.397564+0000","last_active":"2024-01-16T15:05:06.401063+0000","last_peered":"2024-01-16T15:05:06.401063+0000","last_clean":"2024-01-16T15:05:06.401063+0000","last_became_active":"2024-01-16T15:05:05.394386+0000","last_became_peered":"2024-01-16T15:05:05.394386+0000","last_unstale":"2024-01-16T15:05:06.401063+0000","last_undegraded":"2024-01-16T15:05:06.401063+0000","last_fullsized":"2024-01-16T15:05:06.401063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:19.331162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.416346+0000","last_change":"2024-01-16T15:05:04.406546+0000","last_active":"2024-01-16T15:05:05.416346+0000","last_peered":"2024-01-16T15:05:05.416346+0000","last_clean":"2024-01-16T15:05:05.416346+0000","last_became_active":"2024-01-16T15:05:04.405987+0000","last_became_peered":"2024-01-16T15:05:04.405987+0000","last_unstale":"2024-01-16T15:05:05.416346+0000","last_undegraded":"2024-01-16T15:05:05.416346+0000","last_fullsized":"2024-01-16T15:05:05.416346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:16.608954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838039+0000","last_change":"2024-01-16T15:05:05.391926+0000","last_active":"2024-01-16T15:05:06.838039+0000","last_peered":"2024-01-16T15:05:06.838039+0000","last_clean":"2024-01-16T15:05:06.838039+0000","last_became_active":"2024-01-16T15:05:05.391558+0000","last_became_peered":"2024-01-16T15:05:05.391558+0000","last_unstale":"2024-01-16T15:05:06.838039+0000","last_undegraded":"2024-01-16T15:05:06.838039+0000","last_fullsized":"2024-01-16T15:05:06.838039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:52:35.300008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.396061+0000","last_change":"2024-01-16T15:05:04.397445+0000","last_active":"2024-01-16T15:05:05.396061+0000","last_peered":"2024-01-16T15:05:05.396061+0000","last_clean":"2024-01-16T15:05:05.396061+0000","last_became_active":"2024-01-16T15:05:04.396381+0000","last_became_peered":"2024-01-16T15:05:04.396381+0000","last_unstale":"2024-01-16T15:05:05.396061+0000","last_undegraded":"2024-01-16T15:05:05.396061+0000","last_fullsized":"2024-01-16T15:05:05.396061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:46.708567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.393408+0000","last_change":"2024-01-16T15:05:05.390783+0000","last_active":"2024-01-16T15:05:06.393408+0000","last_peered":"2024-01-16T15:05:06.393408+0000","last_clean":"2024-01-16T15:05:06.393408+0000","last_became_active":"2024-01-16T15:05:05.390438+0000","last_became_peered":"2024-01-16T15:05:05.390438+0000","last_unstale":"2024-01-16T15:05:06.393408+0000","last_undegraded":"2024-01-16T15:05:06.393408+0000","last_fullsized":"2024-01-16T15:05:06.393408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:47.703192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.368602+0000","last_change":"2024-01-16T15:05:04.396016+0000","last_active":"2024-01-16T15:05:05.368602+0000","last_peered":"2024-01-16T15:05:05.368602+0000","last_clean":"2024-01-16T15:05:05.368602+0000","last_became_active":"2024-01-16T15:05:04.394556+0000","last_became_peered":"2024-01-16T15:05:04.394556+0000","last_unstale":"2024-01-16T15:05:05.368602+0000","last_undegraded":"2024-01-16T15:05:05.368602+0000","last_fullsized":"2024-01-16T15:05:05.368602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:49.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839954+0000","last_change":"2024-01-16T15:05:05.404565+0000","last_active":"2024-01-16T15:05:06.839954+0000","last_peered":"2024-01-16T15:05:06.839954+0000","last_clean":"2024-01-16T15:05:06.839954+0000","last_became_active":"2024-01-16T15:05:05.404163+0000","last_became_peered":"2024-01-16T15:05:05.404163+0000","last_unstale":"2024-01-16T15:05:06.839954+0000","last_undegraded":"2024-01-16T15:05:06.839954+0000","last_fullsized":"2024-01-16T15:05:06.839954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:58.000087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.385426+0000","last_change":"2024-01-16T15:05:04.393705+0000","last_active":"2024-01-16T15:05:05.385426+0000","last_peered":"2024-01-16T15:05:05.385426+0000","last_clean":"2024-01-16T15:05:05.385426+0000","last_became_active":"2024-01-16T15:05:04.393361+0000","last_became_peered":"2024-01-16T15:05:04.393361+0000","last_unstale":"2024-01-16T15:05:05.385426+0000","last_undegraded":"2024-01-16T15:05:05.385426+0000","last_fullsized":"2024-01-16T15:05:05.385426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:20.578835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.402345+0000","last_change":"2024-01-16T15:05:05.403248+0000","last_active":"2024-01-16T15:05:06.402345+0000","last_peered":"2024-01-16T15:05:06.402345+0000","last_clean":"2024-01-16T15:05:06.402345+0000","last_became_active":"2024-01-16T15:05:05.402933+0000","last_became_peered":"2024-01-16T15:05:05.402933+0000","last_unstale":"2024-01-16T15:05:06.402345+0000","last_undegraded":"2024-01-16T15:05:06.402345+0000","last_fullsized":"2024-01-16T15:05:06.402345+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:40.245791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.402136+0000","last_change":"2024-01-16T15:05:04.410254+0000","last_active":"2024-01-16T15:05:05.402136+0000","last_peered":"2024-01-16T15:05:05.402136+0000","last_clean":"2024-01-16T15:05:05.402136+0000","last_became_active":"2024-01-16T15:05:04.409635+0000","last_became_peered":"2024-01-16T15:05:04.409635+0000","last_unstale":"2024-01-16T15:05:05.402136+0000","last_undegraded":"2024-01-16T15:05:05.402136+0000","last_fullsized":"2024-01-16T15:05:05.402136+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839651+0000","last_change":"2024-01-16T15:05:05.405487+0000","last_active":"2024-01-16T15:05:06.839651+0000","last_peered":"2024-01-16T15:05:06.839651+0000","last_clean":"2024-01-16T15:05:06.839651+0000","last_became_active":"2024-01-16T15:05:05.405250+0000","last_became_peered":"2024-01-16T15:05:05.405250+0000","last_unstale":"2024-01-16T15:05:06.839651+0000","last_undegraded":"2024-01-16T15:05:06.839651+0000","last_fullsized":"2024-01-16T15:05:06.839651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:27.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.421062+0000","last_change":"2024-01-16T15:05:04.405645+0000","last_active":"2024-01-16T15:05:05.421062+0000","last_peered":"2024-01-16T15:05:05.421062+0000","last_clean":"2024-01-16T15:05:05.421062+0000","last_became_active":"2024-01-16T15:05:04.405340+0000","last_became_peered":"2024-01-16T15:05:04.405340+0000","last_unstale":"2024-01-16T15:05:05.421062+0000","last_undegraded":"2024-01-16T15:05:05.421062+0000","last_fullsized":"2024-01-16T15:05:05.421062+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:57:56.221496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839099+0000","last_change":"2024-01-16T15:05:05.397290+0000","last_active":"2024-01-16T15:05:06.839099+0000","last_peered":"2024-01-16T15:05:06.839099+0000","last_clean":"2024-01-16T15:05:06.839099+0000","last_became_active":"2024-01-16T15:05:05.396984+0000","last_became_peered":"2024-01-16T15:05:05.396984+0000","last_unstale":"2024-01-16T15:05:06.839099+0000","last_undegraded":"2024-01-16T15:05:06.839099+0000","last_fullsized":"2024-01-16T15:05:06.839099+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:22.799623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.423151+0000","last_change":"2024-01-16T15:05:04.394073+0000","last_active":"2024-01-16T15:05:05.423151+0000","last_peered":"2024-01-16T15:05:05.423151+0000","last_clean":"2024-01-16T15:05:05.423151+0000","last_became_active":"2024-01-16T15:05:04.393394+0000","last_became_peered":"2024-01-16T15:05:04.393394+0000","last_unstale":"2024-01-16T15:05:05.423151+0000","last_undegraded":"2024-01-16T15:05:05.423151+0000","last_fullsized":"2024-01-16T15:05:05.423151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:36.054624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401420+0000","last_change":"2024-01-16T15:05:05.394096+0000","last_active":"2024-01-16T15:05:06.401420+0000","last_peered":"2024-01-16T15:05:06.401420+0000","last_clean":"2024-01-16T15:05:06.401420+0000","last_became_active":"2024-01-16T15:05:05.389244+0000","last_became_peered":"2024-01-16T15:05:05.389244+0000","last_unstale":"2024-01-16T15:05:06.401420+0000","last_undegraded":"2024-01-16T15:05:06.401420+0000","last_fullsized":"2024-01-16T15:05:06.401420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:03:34.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.423979+0000","last_change":"2024-01-16T15:05:04.400225+0000","last_active":"2024-01-16T15:05:05.423979+0000","last_peered":"2024-01-16T15:05:05.423979+0000","last_clean":"2024-01-16T15:05:05.423979+0000","last_became_active":"2024-01-16T15:05:04.399790+0000","last_became_peered":"2024-01-16T15:05:04.399790+0000","last_unstale":"2024-01-16T15:05:05.423979+0000","last_undegraded":"2024-01-16T15:05:05.423979+0000","last_fullsized":"2024-01-16T15:05:05.423979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:00.871479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839039+0000","last_change":"2024-01-16T15:05:05.405105+0000","last_active":"2024-01-16T15:05:06.839039+0000","last_peered":"2024-01-16T15:05:06.839039+0000","last_clean":"2024-01-16T15:05:06.839039+0000","last_became_active":"2024-01-16T15:05:05.404698+0000","last_became_peered":"2024-01-16T15:05:05.404698+0000","last_unstale":"2024-01-16T15:05:06.839039+0000","last_undegraded":"2024-01-16T15:05:06.839039+0000","last_fullsized":"2024-01-16T15:05:06.839039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:25.947350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.403091+0000","last_change":"2024-01-16T15:05:04.406429+0000","last_active":"2024-01-16T15:05:05.403091+0000","last_peered":"2024-01-16T15:05:05.403091+0000","last_clean":"2024-01-16T15:05:05.403091+0000","last_became_active":"2024-01-16T15:05:04.405623+0000","last_became_peered":"2024-01-16T15:05:04.405623+0000","last_unstale":"2024-01-16T15:05:05.403091+0000","last_undegraded":"2024-01-16T15:05:05.403091+0000","last_fullsized":"2024-01-16T15:05:05.403091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:13.637638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.392881+0000","last_change":"2024-01-16T15:05:05.409020+0000","last_active":"2024-01-16T15:05:06.392881+0000","last_peered":"2024-01-16T15:05:06.392881+0000","last_clean":"2024-01-16T15:05:06.392881+0000","last_became_active":"2024-01-16T15:05:05.408732+0000","last_became_peered":"2024-01-16T15:05:05.408732+0000","last_unstale":"2024-01-16T15:05:06.392881+0000","last_undegraded":"2024-01-16T15:05:06.392881+0000","last_fullsized":"2024-01-16T15:05:06.392881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:37.055136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.384653+0000","last_change":"2024-01-16T15:05:04.406376+0000","last_active":"2024-01-16T15:05:05.384653+0000","last_peered":"2024-01-16T15:05:05.384653+0000","last_clean":"2024-01-16T15:05:05.384653+0000","last_became_active":"2024-01-16T15:05:04.405029+0000","last_became_peered":"2024-01-16T15:05:04.405029+0000","last_unstale":"2024-01-16T15:05:05.384653+0000","last_undegraded":"2024-01-16T15:05:05.384653+0000","last_fullsized":"2024-01-16T15:05:05.384653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:52.975523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838431+0000","last_change":"2024-01-16T15:05:05.399515+0000","last_active":"2024-01-16T15:05:06.838431+0000","last_peered":"2024-01-16T15:05:06.838431+0000","last_clean":"2024-01-16T15:05:06.838431+0000","last_became_active":"2024-01-16T15:05:05.399172+0000","last_became_peered":"2024-01-16T15:05:05.399172+0000","last_unstale":"2024-01-16T15:05:06.838431+0000","last_undegraded":"2024-01-16T15:05:06.838431+0000","last_fullsized":"2024-01-16T15:05:06.838431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:04.552080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.424358+0000","last_change":"2024-01-16T15:05:04.394101+0000","last_active":"2024-01-16T15:05:05.424358+0000","last_peered":"2024-01-16T15:05:05.424358+0000","last_clean":"2024-01-16T15:05:05.424358+0000","last_became_active":"2024-01-16T15:05:04.392109+0000","last_became_peered":"2024-01-16T15:05:04.392109+0000","last_unstale":"2024-01-16T15:05:05.424358+0000","last_undegraded":"2024-01-16T15:05:05.424358+0000","last_fullsized":"2024-01-16T15:05:05.424358+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:29.089830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.400708+0000","last_change":"2024-01-16T15:05:05.396682+0000","last_active":"2024-01-16T15:05:06.400708+0000","last_peered":"2024-01-16T15:05:06.400708+0000","last_clean":"2024-01-16T15:05:06.400708+0000","last_became_active":"2024-01-16T15:05:05.396403+0000","last_became_peered":"2024-01-16T15:05:05.396403+0000","last_unstale":"2024-01-16T15:05:06.400708+0000","last_undegraded":"2024-01-16T15:05:06.400708+0000","last_fullsized":"2024-01-16T15:05:06.400708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:15:26.229009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.421164+0000","last_change":"2024-01-16T15:05:04.407760+0000","last_active":"2024-01-16T15:05:05.421164+0000","last_peered":"2024-01-16T15:05:05.421164+0000","last_clean":"2024-01-16T15:05:05.421164+0000","last_became_active":"2024-01-16T15:05:04.407043+0000","last_became_peered":"2024-01-16T15:05:04.407043+0000","last_unstale":"2024-01-16T15:05:05.421164+0000","last_undegraded":"2024-01-16T15:05:05.421164+0000","last_fullsized":"2024-01-16T15:05:05.421164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:36.140427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.402458+0000","last_change":"2024-01-16T15:05:05.393976+0000","last_active":"2024-01-16T15:05:06.402458+0000","last_peered":"2024-01-16T15:05:06.402458+0000","last_clean":"2024-01-16T15:05:06.402458+0000","last_became_active":"2024-01-16T15:05:05.392258+0000","last_became_peered":"2024-01-16T15:05:05.392258+0000","last_unstale":"2024-01-16T15:05:06.402458+0000","last_undegraded":"2024-01-16T15:05:06.402458+0000","last_fullsized":"2024-01-16T15:05:06.402458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:17.728661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.423664+0000","last_change":"2024-01-16T15:05:04.408404+0000","last_active":"2024-01-16T15:05:05.423664+0000","last_peered":"2024-01-16T15:05:05.423664+0000","last_clean":"2024-01-16T15:05:05.423664+0000","last_became_active":"2024-01-16T15:05:04.408105+0000","last_became_peered":"2024-01-16T15:05:04.408105+0000","last_unstale":"2024-01-16T15:05:05.423664+0000","last_undegraded":"2024-01-16T15:05:05.423664+0000","last_fullsized":"2024-01-16T15:05:05.423664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:45.132507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837343+0000","last_change":"2024-01-16T15:05:05.401777+0000","last_active":"2024-01-16T15:05:06.837343+0000","last_peered":"2024-01-16T15:05:06.837343+0000","last_clean":"2024-01-16T15:05:06.837343+0000","last_became_active":"2024-01-16T15:05:05.401456+0000","last_became_peered":"2024-01-16T15:05:05.401456+0000","last_unstale":"2024-01-16T15:05:06.837343+0000","last_undegraded":"2024-01-16T15:05:06.837343+0000","last_fullsized":"2024-01-16T15:05:06.837343+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:57:55.845235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.424063+0000","last_change":"2024-01-16T15:05:04.405985+0000","last_active":"2024-01-16T15:05:05.424063+0000","last_peered":"2024-01-16T15:05:05.424063+0000","last_clean":"2024-01-16T15:05:05.424063+0000","last_became_active":"2024-01-16T15:05:04.405663+0000","last_became_peered":"2024-01-16T15:05:04.405663+0000","last_unstale":"2024-01-16T15:05:05.424063+0000","last_undegraded":"2024-01-16T15:05:05.424063+0000","last_fullsized":"2024-01-16T15:05:05.424063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:03.389174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.839214+0000","last_change":"2024-01-16T15:05:05.401777+0000","last_active":"2024-01-16T15:05:06.839214+0000","last_peered":"2024-01-16T15:05:06.839214+0000","last_clean":"2024-01-16T15:05:06.839214+0000","last_became_active":"2024-01-16T15:05:05.400511+0000","last_became_peered":"2024-01-16T15:05:05.400511+0000","last_unstale":"2024-01-16T15:05:06.839214+0000","last_undegraded":"2024-01-16T15:05:06.839214+0000","last_fullsized":"2024-01-16T15:05:06.839214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:47.723990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.375886+0000","last_change":"2024-01-16T15:05:04.407769+0000","last_active":"2024-01-16T15:05:05.375886+0000","last_peered":"2024-01-16T15:05:05.375886+0000","last_clean":"2024-01-16T15:05:05.375886+0000","last_became_active":"2024-01-16T15:05:04.407439+0000","last_became_peered":"2024-01-16T15:05:04.407439+0000","last_unstale":"2024-01-16T15:05:05.375886+0000","last_undegraded":"2024-01-16T15:05:05.375886+0000","last_fullsized":"2024-01-16T15:05:05.375886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:11.663695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764316+0000","last_change":"2024-01-16T15:05:05.406725+0000","last_active":"2024-01-16T15:05:06.764316+0000","last_peered":"2024-01-16T15:05:06.764316+0000","last_clean":"2024-01-16T15:05:06.764316+0000","last_became_active":"2024-01-16T15:05:05.406408+0000","last_became_peered":"2024-01-16T15:05:05.406408+0000","last_unstale":"2024-01-16T15:05:06.764316+0000","last_undegraded":"2024-01-16T15:05:06.764316+0000","last_fullsized":"2024-01-16T15:05:06.764316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:18.750809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.406183+0000","last_change":"2024-01-16T15:05:04.406257+0000","last_active":"2024-01-16T15:05:05.406183+0000","last_peered":"2024-01-16T15:05:05.406183+0000","last_clean":"2024-01-16T15:05:05.406183+0000","last_became_active":"2024-01-16T15:05:04.405263+0000","last_became_peered":"2024-01-16T15:05:04.405263+0000","last_unstale":"2024-01-16T15:05:05.406183+0000","last_undegraded":"2024-01-16T15:05:05.406183+0000","last_fullsized":"2024-01-16T15:05:05.406183+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:38.326114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764159+0000","last_change":"2024-01-16T15:05:05.409141+0000","last_active":"2024-01-16T15:05:06.764159+0000","last_peered":"2024-01-16T15:05:06.764159+0000","last_clean":"2024-01-16T15:05:06.764159+0000","last_became_active":"2024-01-16T15:05:05.408826+0000","last_became_peered":"2024-01-16T15:05:05.408826+0000","last_unstale":"2024-01-16T15:05:06.764159+0000","last_undegraded":"2024-01-16T15:05:06.764159+0000","last_fullsized":"2024-01-16T15:05:06.764159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:19:20.587986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.384731+0000","last_change":"2024-01-16T15:05:04.399057+0000","last_active":"2024-01-16T15:05:05.384731+0000","last_peered":"2024-01-16T15:05:05.384731+0000","last_clean":"2024-01-16T15:05:05.384731+0000","last_became_active":"2024-01-16T15:05:04.398756+0000","last_became_peered":"2024-01-16T15:05:04.398756+0000","last_unstale":"2024-01-16T15:05:05.384731+0000","last_undegraded":"2024-01-16T15:05:05.384731+0000","last_fullsized":"2024-01-16T15:05:05.384731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:23.123971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764058+0000","last_change":"2024-01-16T15:05:05.398063+0000","last_active":"2024-01-16T15:05:06.764058+0000","last_peered":"2024-01-16T15:05:06.764058+0000","last_clean":"2024-01-16T15:05:06.764058+0000","last_became_active":"2024-01-16T15:05:05.397735+0000","last_became_peered":"2024-01-16T15:05:05.397735+0000","last_unstale":"2024-01-16T15:05:06.764058+0000","last_undegraded":"2024-01-16T15:05:06.764058+0000","last_fullsized":"2024-01-16T15:05:06.764058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:17.728661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.386967+0000","last_change":"2024-01-16T15:05:04.393679+0000","last_active":"2024-01-16T15:05:05.386967+0000","last_peered":"2024-01-16T15:05:05.386967+0000","last_clean":"2024-01-16T15:05:05.386967+0000","last_became_active":"2024-01-16T15:05:04.392878+0000","last_became_peered":"2024-01-16T15:05:04.392878+0000","last_unstale":"2024-01-16T15:05:05.386967+0000","last_undegraded":"2024-01-16T15:05:05.386967+0000","last_fullsized":"2024-01-16T15:05:05.386967+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:13.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.392966+0000","last_change":"2024-01-16T15:05:05.401166+0000","last_active":"2024-01-16T15:05:06.392966+0000","last_peered":"2024-01-16T15:05:06.392966+0000","last_clean":"2024-01-16T15:05:06.392966+0000","last_became_active":"2024-01-16T15:05:05.400494+0000","last_became_peered":"2024-01-16T15:05:05.400494+0000","last_unstale":"2024-01-16T15:05:06.392966+0000","last_undegraded":"2024-01-16T15:05:06.392966+0000","last_fullsized":"2024-01-16T15:05:06.392966+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:55:15.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.420931+0000","last_change":"2024-01-16T15:05:04.407801+0000","last_active":"2024-01-16T15:05:05.420931+0000","last_peered":"2024-01-16T15:05:05.420931+0000","last_clean":"2024-01-16T15:05:05.420931+0000","last_became_active":"2024-01-16T15:05:04.406787+0000","last_became_peered":"2024-01-16T15:05:04.406787+0000","last_unstale":"2024-01-16T15:05:05.420931+0000","last_undegraded":"2024-01-16T15:05:05.420931+0000","last_fullsized":"2024-01-16T15:05:05.420931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:02.362251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838840+0000","last_change":"2024-01-16T15:05:05.394859+0000","last_active":"2024-01-16T15:05:06.838840+0000","last_peered":"2024-01-16T15:05:06.838840+0000","last_clean":"2024-01-16T15:05:06.838840+0000","last_became_active":"2024-01-16T15:05:05.394605+0000","last_became_peered":"2024-01-16T15:05:05.394605+0000","last_unstale":"2024-01-16T15:05:06.838840+0000","last_undegraded":"2024-01-16T15:05:06.838840+0000","last_fullsized":"2024-01-16T15:05:06.838840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:47.703192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.840092+0000","last_change":"2024-01-16T15:05:04.405954+0000","last_active":"2024-01-16T15:05:06.840092+0000","last_peered":"2024-01-16T15:05:06.840092+0000","last_clean":"2024-01-16T15:05:06.840092+0000","last_became_active":"2024-01-16T15:05:04.405231+0000","last_became_peered":"2024-01-16T15:05:04.405231+0000","last_unstale":"2024-01-16T15:05:06.840092+0000","last_undegraded":"2024-01-16T15:05:06.840092+0000","last_fullsized":"2024-01-16T15:05:06.840092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:36.039712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.401843+0000","last_change":"2024-01-16T15:05:05.408627+0000","last_active":"2024-01-16T15:05:06.401843+0000","last_peered":"2024-01-16T15:05:06.401843+0000","last_clean":"2024-01-16T15:05:06.401843+0000","last_became_active":"2024-01-16T15:05:05.408279+0000","last_became_peered":"2024-01-16T15:05:05.408279+0000","last_unstale":"2024-01-16T15:05:06.401843+0000","last_undegraded":"2024-01-16T15:05:06.401843+0000","last_fullsized":"2024-01-16T15:05:06.401843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:26.426420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.369736+0000","last_change":"2024-01-16T15:05:04.393333+0000","last_active":"2024-01-16T15:05:05.369736+0000","last_peered":"2024-01-16T15:05:05.369736+0000","last_clean":"2024-01-16T15:05:05.369736+0000","last_became_active":"2024-01-16T15:05:04.393011+0000","last_became_peered":"2024-01-16T15:05:04.393011+0000","last_unstale":"2024-01-16T15:05:05.369736+0000","last_undegraded":"2024-01-16T15:05:05.369736+0000","last_fullsized":"2024-01-16T15:05:05.369736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:28.029219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.402147+0000","last_change":"2024-01-16T15:05:05.401591+0000","last_active":"2024-01-16T15:05:06.402147+0000","last_peered":"2024-01-16T15:05:06.402147+0000","last_clean":"2024-01-16T15:05:06.402147+0000","last_became_active":"2024-01-16T15:05:05.401330+0000","last_became_peered":"2024-01-16T15:05:05.401330+0000","last_unstale":"2024-01-16T15:05:06.402147+0000","last_undegraded":"2024-01-16T15:05:06.402147+0000","last_fullsized":"2024-01-16T15:05:06.402147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:52.063857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.385549+0000","last_change":"2024-01-16T15:05:04.399310+0000","last_active":"2024-01-16T15:05:05.385549+0000","last_peered":"2024-01-16T15:05:05.385549+0000","last_clean":"2024-01-16T15:05:05.385549+0000","last_became_active":"2024-01-16T15:05:04.399004+0000","last_became_peered":"2024-01-16T15:05:04.399004+0000","last_unstale":"2024-01-16T15:05:05.385549+0000","last_undegraded":"2024-01-16T15:05:05.385549+0000","last_fullsized":"2024-01-16T15:05:05.385549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:52.975523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838777+0000","last_change":"2024-01-16T15:05:05.405861+0000","last_active":"2024-01-16T15:05:06.838777+0000","last_peered":"2024-01-16T15:05:06.838777+0000","last_clean":"2024-01-16T15:05:06.838777+0000","last_became_active":"2024-01-16T15:05:05.399431+0000","last_became_peered":"2024-01-16T15:05:05.399431+0000","last_unstale":"2024-01-16T15:05:06.838777+0000","last_undegraded":"2024-01-16T15:05:06.838777+0000","last_fullsized":"2024-01-16T15:05:06.838777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:02:54.679251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.369920+0000","last_change":"2024-01-16T15:05:04.406564+0000","last_active":"2024-01-16T15:05:05.369920+0000","last_peered":"2024-01-16T15:05:05.369920+0000","last_clean":"2024-01-16T15:05:05.369920+0000","last_became_active":"2024-01-16T15:05:04.406309+0000","last_became_peered":"2024-01-16T15:05:04.406309+0000","last_unstale":"2024-01-16T15:05:05.369920+0000","last_undegraded":"2024-01-16T15:05:05.369920+0000","last_fullsized":"2024-01-16T15:05:05.369920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:33.255649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.838111+0000","last_change":"2024-01-16T15:05:05.408995+0000","last_active":"2024-01-16T15:05:06.838111+0000","last_peered":"2024-01-16T15:05:06.838111+0000","last_clean":"2024-01-16T15:05:06.838111+0000","last_became_active":"2024-01-16T15:05:05.408611+0000","last_became_peered":"2024-01-16T15:05:05.408611+0000","last_unstale":"2024-01-16T15:05:06.838111+0000","last_undegraded":"2024-01-16T15:05:06.838111+0000","last_fullsized":"2024-01-16T15:05:06.838111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:46.815864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.397324+0000","last_change":"2024-01-16T15:05:04.397840+0000","last_active":"2024-01-16T15:05:05.397324+0000","last_peered":"2024-01-16T15:05:05.397324+0000","last_clean":"2024-01-16T15:05:05.397324+0000","last_became_active":"2024-01-16T15:05:04.397170+0000","last_became_peered":"2024-01-16T15:05:04.397170+0000","last_unstale":"2024-01-16T15:05:05.397324+0000","last_undegraded":"2024-01-16T15:05:05.397324+0000","last_fullsized":"2024-01-16T15:05:05.397324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:56.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.836753+0000","last_change":"2024-01-16T15:05:05.401782+0000","last_active":"2024-01-16T15:05:06.836753+0000","last_peered":"2024-01-16T15:05:06.836753+0000","last_clean":"2024-01-16T15:05:06.836753+0000","last_became_active":"2024-01-16T15:05:05.401489+0000","last_became_peered":"2024-01-16T15:05:05.401489+0000","last_unstale":"2024-01-16T15:05:06.836753+0000","last_undegraded":"2024-01-16T15:05:06.836753+0000","last_fullsized":"2024-01-16T15:05:06.836753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:50.742658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.383664+0000","last_change":"2024-01-16T15:05:04.399514+0000","last_active":"2024-01-16T15:05:05.383664+0000","last_peered":"2024-01-16T15:05:05.383664+0000","last_clean":"2024-01-16T15:05:05.383664+0000","last_became_active":"2024-01-16T15:05:04.398092+0000","last_became_peered":"2024-01-16T15:05:04.398092+0000","last_unstale":"2024-01-16T15:05:05.383664+0000","last_undegraded":"2024-01-16T15:05:05.383664+0000","last_fullsized":"2024-01-16T15:05:05.383664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:39.419493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.837531+0000","last_change":"2024-01-16T15:05:05.403697+0000","last_active":"2024-01-16T15:05:06.837531+0000","last_peered":"2024-01-16T15:05:06.837531+0000","last_clean":"2024-01-16T15:05:06.837531+0000","last_became_active":"2024-01-16T15:05:05.403417+0000","last_became_peered":"2024-01-16T15:05:05.403417+0000","last_unstale":"2024-01-16T15:05:06.837531+0000","last_undegraded":"2024-01-16T15:05:06.837531+0000","last_fullsized":"2024-01-16T15:05:06.837531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:46.147931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.394624+0000","last_change":"2024-01-16T15:05:04.404689+0000","last_active":"2024-01-16T15:05:05.394624+0000","last_peered":"2024-01-16T15:05:05.394624+0000","last_clean":"2024-01-16T15:05:05.394624+0000","last_became_active":"2024-01-16T15:05:04.403370+0000","last_became_peered":"2024-01-16T15:05:04.403370+0000","last_unstale":"2024-01-16T15:05:05.394624+0000","last_undegraded":"2024-01-16T15:05:05.394624+0000","last_fullsized":"2024-01-16T15:05:05.394624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:21:26.814373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:05:06.764305+0000","last_change":"2024-01-16T15:05:05.411738+0000","last_active":"2024-01-16T15:05:06.764305+0000","last_peered":"2024-01-16T15:05:06.764305+0000","last_clean":"2024-01-16T15:05:06.764305+0000","last_became_active":"2024-01-16T15:05:05.411412+0000","last_became_peered":"2024-01-16T15:05:05.411412+0000","last_unstale":"2024-01-16T15:05:06.764305+0000","last_undegraded":"2024-01-16T15:05:06.764305+0000","last_fullsized":"2024-01-16T15:05:06.764305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:04.356166+0000","last_clean_scrub_stamp":"2024-01-16T15:05:04.356166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:09:23.446233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.418420+0000","last_change":"2024-01-16T15:05:04.405296+0000","last_active":"2024-01-16T15:05:05.418420+0000","last_peered":"2024-01-16T15:05:05.418420+0000","last_clean":"2024-01-16T15:05:05.418420+0000","last_became_active":"2024-01-16T15:05:04.403463+0000","last_became_peered":"2024-01-16T15:05:04.403463+0000","last_unstale":"2024-01-16T15:05:05.418420+0000","last_undegraded":"2024-01-16T15:05:05.418420+0000","last_fullsized":"2024-01-16T15:05:05.418420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:03.340743+0000","last_clean_scrub_stamp":"2024-01-16T15:05:03.340743+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:38.964725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":219,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-16T15:05:05.374626+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:05:05.374626+0000","last_peered":"2024-01-16T15:05:05.374626+0000","last_clean":"2024-01-16T15:05:05.374626+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:05:05.374626+0000","last_undegraded":"2024-01-16T15:05:05.374626+0000","last_fullsized":"2024-01-16T15:05:05.374626+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":950110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.9550000000000001,"15min":1.3320000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":8.5399999999999991,"5min":45.296999999999997,"15min":45.296999999999997},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.5620000000000001,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":2.073,"5min":20.126999999999999,"15min":20.126999999999999},"last":1.0529999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.768,"15min":1.331},"min":{"1min":0.65700000000000003,"5min":0.499,"15min":0.499},"max":{"1min":6.6669999999999998,"5min":42.723999999999997,"15min":42.723999999999997},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.569,"5min":2.1600000000000001,"15min":1.5309999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.1820000000000004,"5min":43.572000000000003,"15min":43.572000000000003},"last":1.284}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.7609999999999999,"15min":1.3839999999999999},"min":{"1min":0.67100000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.8580000000000001,"5min":35.308,"15min":35.308},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.706,"5min":1.8480000000000001,"15min":1.4350000000000001},"min":{"1min":0.65800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":9.1899999999999995,"5min":23.594000000000001,"15min":23.594000000000001},"last":1.2529999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":2.0920000000000001,"15min":1.512},"min":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":7.1520000000000001,"5min":53.033999999999999,"15min":53.033999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.9650000000000001,"15min":1.5700000000000001},"min":{"1min":0.76200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":14.708,"5min":31.305,"15min":31.305},"last":1.369}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2889999999999999,"15min":1.373},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.2410000000000001,"5min":3.5939999999999999,"15min":5.1870000000000003},"last":1.466},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.355,"15min":1.4810000000000001},"min":{"1min":0.48199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.5070000000000001,"5min":7.8360000000000003,"15min":36.374000000000002},"last":1.6379999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.5589999999999999,"15min":1.4350000000000001},"min":{"1min":0.55500000000000005,"5min":0.45700000000000002,"15min":0.41499999999999998},"max":{"1min":9.1370000000000005,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":1.534},{"interface":"front","average":{"1min":1.304,"5min":1.54,"15min":1.4570000000000001},"min":{"1min":0.61599999999999999,"5min":0.53500000000000003,"15min":0.46899999999999997},"max":{"1min":2.1360000000000001,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":1.599}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5489999999999999,"15min":1.532},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":4.6600000000000001,"5min":7.9349999999999996,"15min":7.9349999999999996},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.853,"5min":1.657,"15min":1.6220000000000001},"min":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.49099999999999999},"max":{"1min":8.4849999999999994,"5min":8.4849999999999994,"15min":8.4849999999999994},"last":1.6819999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45908,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325932,"statfs":{"total":96636764160,"available":96589754368,"internally_reserved":0,"allocated":860160,"data_stored":359218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":2.4079999999999999,"15min":1.532},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60099999999999998},"max":{"1min":3.9260000000000002,"5min":71.870999999999995,"15min":71.870999999999995},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.6219999999999999,"15min":1.7030000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":5.7130000000000001,"5min":71.846999999999994,"15min":71.846999999999994},"last":1.5109999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.353,"15min":1.159},"min":{"1min":0.81899999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.5789999999999997,"5min":7.5949999999999998,"15min":7.5949999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.829,"5min":1.609,"15min":1.294},"min":{"1min":0.70899999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":12.436999999999999,"5min":13.839,"15min":13.839},"last":0.94399999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.448,"15min":1.292},"min":{"1min":0.71399999999999997,"5min":0.63800000000000001,"15min":0.63400000000000001},"max":{"1min":3.819,"5min":7.1740000000000004,"15min":7.1740000000000004},"last":1.367},{"interface":"front","average":{"1min":1.431,"5min":1.4490000000000001,"15min":1.337},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.8500000000000001,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":1.28}]},{"osd":3,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.9059999999999999,"5min":1.726,"15min":1.4910000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":9.2170000000000005,"5min":14.827,"15min":14.827},"last":1.0620000000000001},{"interface":"front","average":{"1min":2.0710000000000002,"5min":1.718,"15min":1.506},"min":{"1min":0.89400000000000002,"5min":0.83899999999999997,"15min":0.57899999999999996},"max":{"1min":9.2460000000000004,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":1.319}]},{"osd":4,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4379999999999999,"15min":1.4339999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":3.6909999999999998,"5min":7.4320000000000004,"15min":7.4320000000000004},"last":1.222},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.516,"15min":1.48},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.605,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":1.6799999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4079999999999999,"15min":1.431},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.38500000000000001},"max":{"1min":3.855,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.673,"15min":1.6140000000000001},"min":{"1min":0.77600000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":3.7269999999999999,"5min":7.508,"15min":7.508},"last":1.5840000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.7230000000000001,"15min":1.6359999999999999},"min":{"1min":0.61199999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.966,"5min":14.74,"15min":14.74},"last":1.744},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.569,"15min":1.573},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.631}]}]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":856064,"data_stored":359216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.748,"15min":1.274},"min":{"1min":0.59099999999999997,"5min":0.46800000000000003,"15min":0.44400000000000001},"max":{"1min":5.3540000000000001,"5min":25.129999999999999,"15min":25.129999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.794,"15min":1.3740000000000001},"min":{"1min":0.66300000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":3.6659999999999999,"5min":34.959000000000003,"15min":34.959000000000003},"last":1.175}]},{"osd":1,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.3089999999999999,"15min":1.29},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.49099999999999999},"max":{"1min":1.9379999999999999,"5min":10.180999999999999,"15min":11.298999999999999},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.413,"15min":1.252},"min":{"1min":0.67500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.613,"5min":13.332000000000001,"15min":13.332000000000001},"last":1.5049999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.357,"15min":1.319},"min":{"1min":0.628,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.0670000000000002,"5min":10.321999999999999,"15min":10.321999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.6180000000000001,"15min":1.393},"min":{"1min":0.71099999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":6.7599999999999998,"5min":17.030000000000001,"15min":17.030000000000001},"last":1.7450000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.0030000000000001,"15min":1.575},"min":{"1min":0.625,"5min":0.54100000000000004,"15min":0.53300000000000003},"max":{"1min":7.3840000000000003,"5min":18.372,"15min":18.372},"last":1.1000000000000001},{"interface":"front","average":{"1min":2.2320000000000002,"5min":2.1019999999999999,"15min":1.605},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":17.713999999999999,"5min":22.963999999999999,"15min":22.963999999999999},"last":1.3500000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.341,"15min":1.27},"min":{"1min":0.45500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.7889999999999999,"5min":8.6050000000000004,"15min":8.6050000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.327,"5min":1.504,"15min":1.496},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.869,"5min":7.2229999999999999,"15min":7.2229999999999999},"last":1.304}]},{"osd":6,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4730000000000001,"15min":1.498},"min":{"1min":0.77200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.4220000000000002,"5min":13.404999999999999,"15min":13.404999999999999},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.357,"15min":1.3839999999999999},"min":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.0310000000000001,"5min":8.7070000000000007,"15min":10.015000000000001},"last":1.0680000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":1.679,"15min":1.597},"min":{"1min":0.47899999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":21.422000000000001,"5min":21.422000000000001,"15min":21.422000000000001},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.792,"15min":1.6479999999999999},"min":{"1min":0.54800000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.956,"5min":21.106000000000002,"15min":21.106000000000002},"last":1.6120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50072,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321768,"statfs":{"total":96636764160,"available":96585490432,"internally_reserved":0,"allocated":856064,"data_stored":359196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.2649999999999999},"min":{"1min":0.752,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":5.4560000000000004,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.232},{"interface":"front","average":{"1min":1.577,"5min":1.4279999999999999,"15min":1.3129999999999999},"min":{"1min":0.88800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":5.524,"5min":7.2190000000000003,"15min":7.2190000000000003},"last":1.6539999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.853,"15min":1.353},"min":{"1min":0.68799999999999994,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":8.9179999999999993,"5min":39.823999999999998,"15min":39.823999999999998},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.796,"15min":1.3560000000000001},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":6.9429999999999996,"5min":50.582000000000001,"15min":50.582000000000001},"last":1.147}]},{"osd":2,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.325,"15min":1.2549999999999999},"min":{"1min":0.629,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.7930000000000001,"5min":5.4619999999999997,"15min":8.0579999999999998},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4419999999999999,"15min":1.347},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":3.0289999999999999,"5min":9.1509999999999998,"15min":9.1509999999999998},"last":1.347}]},{"osd":3,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.448,"15min":1.331},"min":{"1min":0.59699999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":6.2190000000000003,"5min":11.971,"15min":11.971},"last":1.587},{"interface":"front","average":{"1min":1.754,"5min":1.968,"15min":1.625},"min":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":9.4329999999999998,"5min":27.626999999999999,"15min":27.626999999999999},"last":1.4119999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.405,"15min":1.397},"min":{"1min":0.51800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.2799999999999998,"5min":8.2599999999999998,"15min":8.2599999999999998},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3580000000000001,"15min":1.4950000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":4.2430000000000003,"5min":4.2430000000000003,"15min":11.289999999999999},"last":1.4650000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.458,"15min":1.512},"min":{"1min":0.55200000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":3.0089999999999999,"5min":5.9210000000000003,"15min":6.9630000000000001},"last":1.8779999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.45,"15min":1.5449999999999999},"min":{"1min":0.58999999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.3700000000000001,"5min":2.9460000000000002,"15min":3.2770000000000001},"last":1.5289999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:47 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.444,"15min":1.5589999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":2.6400000000000001,"5min":3.048,"15min":3.048},"last":1.7},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.879,"15min":1.76},"min":{"1min":0.63400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.093,"5min":19.001000000000001,"15min":19.001000000000001},"last":2.0030000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50076,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321764,"statfs":{"total":96636764160,"available":96585486336,"internally_reserved":0,"allocated":860160,"data_stored":359238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.5369999999999999,"15min":1.1240000000000001},"min":{"1min":0.54500000000000004,"5min":0.313,"15min":0.313},"max":{"1min":3.0139999999999998,"5min":38.423999999999999,"15min":38.423999999999999},"last":0.878},{"interface":"front","average":{"1min":1.133,"5min":1.405,"15min":1.097},"min":{"1min":0.48499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.4689999999999999,"5min":20.012,"15min":20.012},"last":0.99199999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":3.379,"5min":2.907,"15min":1.5800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.215,"5min":40.695999999999998,"15min":40.695999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.323,"15min":1.149},"min":{"1min":0.57399999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.879,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":1.246}]},{"osd":2,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.1110000000000002,"15min":1.3979999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.1720000000000002,"5min":59.009,"15min":59.009},"last":1.115},{"interface":"front","average":{"1min":3.5299999999999998,"5min":2.637,"15min":1.611},"min":{"1min":0.623,"5min":0.38,"15min":0.38},"max":{"1min":19.152000000000001,"5min":29.773,"15min":29.773},"last":1.1970000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.591,"15min":1.4350000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8740000000000001,"5min":21.736000000000001,"15min":21.736000000000001},"last":1.6659999999999999},{"interface":"front","average":{"1min":3.7469999999999999,"5min":2.9079999999999999,"15min":1.8620000000000001},"min":{"1min":0.80000000000000004,"5min":0.752,"15min":0.752},"max":{"1min":19.292000000000002,"5min":29.524000000000001,"15min":29.524000000000001},"last":1.304}]},{"osd":5,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.617,"15min":1.508},"min":{"1min":0.65100000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.9239999999999999,"5min":9.8249999999999993,"15min":9.8249999999999993},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7210000000000001,"15min":1.5489999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":4.9119999999999999,"5min":9.9009999999999998,"15min":9.9009999999999998},"last":1.448}]},{"osd":6,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.7370000000000001,"15min":1.5229999999999999},"min":{"1min":0.81599999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.966,"5min":20.459,"15min":20.459},"last":1.774},{"interface":"front","average":{"1min":3.8460000000000001,"5min":2.944,"15min":1.9550000000000001},"min":{"1min":0.84899999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":19.254999999999999,"5min":29.475000000000001,"15min":29.475000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.784,"15min":1.5660000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.9470000000000001,"5min":20.109999999999999,"15min":20.109999999999999},"last":1.833},{"interface":"front","average":{"1min":3.8519999999999999,"5min":2.887,"15min":1.968},"min":{"1min":0.98399999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":19.327999999999999,"5min":29.396999999999998,"15min":29.396999999999998},"last":1.534}]}]},{"osd":2,"up_from":13,"seq":55834574955,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":864256,"data_stored":360738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.1910000000000001,"15min":1.105},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28899999999999998},"max":{"1min":1.3400000000000001,"5min":25.335999999999999,"15min":25.335999999999999},"last":1.075},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.552,"15min":1.248},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":25.199000000000002,"5min":25.199000000000002,"15min":25.199000000000002},"last":0.72499999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.27,"15min":1.234},"min":{"1min":0.442,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":3.1240000000000001,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":1.242},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.6799999999999999,"15min":1.446},"min":{"1min":0.39100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":19.312000000000001,"5min":21.439,"15min":21.439},"last":1.272}]},{"osd":3,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.1970000000000001,"15min":1.198},"min":{"1min":0.41699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.7750000000000004,"5min":8.1980000000000004,"15min":8.1980000000000004},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.464,"15min":1.4079999999999999},"min":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.46500000000000002},"max":{"1min":3.3119999999999998,"5min":28.045999999999999,"15min":28.045999999999999},"last":1.2929999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3380000000000001,"5min":1.994,"15min":1.649},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.803000000000001,"5min":21.532,"15min":21.532},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.792,"5min":1.756,"15min":1.5389999999999999},"min":{"1min":0.57599999999999996,"5min":0.502,"15min":0.502},"max":{"1min":14.106999999999999,"5min":14.597,"15min":14.597},"last":1.034}]},{"osd":5,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.655,"15min":1.577},"min":{"1min":0.629,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.524,"5min":19.327000000000002,"15min":19.327000000000002},"last":1.389},{"interface":"front","average":{"1min":2.379,"5min":1.9750000000000001,"15min":1.6930000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":20.843,"5min":21.608000000000001,"15min":21.608000000000001},"last":1.349}]},{"osd":6,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.6040000000000001,"15min":1.5209999999999999},"min":{"1min":0.68400000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7789999999999999,"5min":12.57,"15min":12.57},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.5269999999999999,"15min":1.603},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":3.1190000000000002,"5min":3.871,"15min":4.8849999999999998},"last":1.456}]},{"osd":7,"last update":"Tue Jan 16 15:04:55 2024","interfaces":[{"interface":"back","average":{"1min":2.464,"5min":2.0830000000000002,"15min":1.7889999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":20.742000000000001,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.6539999999999999,"15min":1.621},"min":{"1min":0.72499999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6830000000000001,"5min":12.456,"15min":12.456},"last":1.1599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50088,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321752,"statfs":{"total":96636764160,"available":96585474048,"internally_reserved":0,"allocated":872448,"data_stored":361354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.3740000000000001,"15min":1.0960000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.28799999999999998},"max":{"1min":17.210999999999999,"5min":17.210999999999999,"15min":17.210999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.921,"15min":1.244},"min":{"1min":0.34300000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":20.164000000000001,"5min":34.252000000000002,"15min":34.252000000000002},"last":0.84199999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.927,"15min":1.2490000000000001},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.36099999999999999},"max":{"1min":24.102,"5min":24.709,"15min":24.709},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.982,"5min":1.8240000000000001,"15min":1.337},"min":{"1min":0.60499999999999998,"5min":0.46600000000000003,"15min":0.377},"max":{"1min":16.988,"5min":30.128,"15min":30.128},"last":1.127}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.573,"15min":1.3240000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":4.242,"5min":1.944,"15min":1.361},"min":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":41.972000000000001,"5min":41.972000000000001,"15min":41.972000000000001},"last":1.1579999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":1.7549999999999999,"15min":1.425},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":18.004999999999999,"5min":18.004999999999999,"15min":18.004999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":2.3700000000000001,"5min":1.746,"15min":1.498},"min":{"1min":0.83899999999999997,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":20.187999999999999,"5min":20.187999999999999,"15min":20.187999999999999},"last":1.288}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.5950000000000002,"5min":1.895,"15min":1.5760000000000001},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":20.905999999999999,"5min":20.905999999999999,"15min":20.905999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.0499999999999998,"15min":1.629},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":17.957999999999998,"5min":17.957999999999998,"15min":17.957999999999998},"last":1.2569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.829,"15min":1.611},"min":{"1min":1.087,"5min":0.78200000000000003,"15min":0.73099999999999998},"max":{"1min":20.276,"5min":20.276,"15min":20.276},"last":1.5449999999999999},{"interface":"front","average":{"1min":2.593,"5min":1.7629999999999999,"15min":1.5329999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":20.509,"5min":20.509,"15min":20.509},"last":1.107}]},{"osd":7,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.7949999999999999,"5min":1.9359999999999999,"15min":1.653},"min":{"1min":0.97399999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":20.669,"5min":20.669,"15min":20.669},"last":1.2050000000000001},{"interface":"front","average":{"1min":2.363,"5min":1.95,"15min":1.6240000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":18.039999999999999,"5min":18.039999999999999,"15min":18.039999999999999},"last":1.1759999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50640,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321200,"statfs":{"total":96636764160,"available":96584908800,"internally_reserved":0,"allocated":1441792,"data_stored":948475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.5600000000000001,"15min":1.119},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":6.2889999999999997,"5min":26.751000000000001,"15min":26.751000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.4350000000000001,"15min":1.103},"min":{"1min":0.45000000000000001,"5min":0.42199999999999999,"15min":0.35899999999999999},"max":{"1min":12.648999999999999,"5min":15.750999999999999,"15min":15.750999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":2.0129999999999999,"15min":1.3680000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.9159999999999999,"5min":38.517000000000003,"15min":38.517000000000003},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.232,"5min":1.544,"15min":1.181},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.161,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.77600000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":3.25,"5min":2.1110000000000002,"15min":1.3540000000000001},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":29.808,"5min":29.808,"15min":29.808},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":2.4060000000000001,"15min":1.615},"min":{"1min":0.51700000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.1020000000000003,"5min":32.302,"15min":32.302},"last":1.054}]},{"osd":4,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.792,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.70799999999999996},"max":{"1min":9.4649999999999999,"5min":15.832000000000001,"15min":15.832000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":2.452,"5min":1.9159999999999999,"15min":1.4670000000000001},"min":{"1min":0.68200000000000005,"5min":0.432,"15min":0.432},"max":{"1min":14.685,"5min":19.062999999999999,"15min":19.062999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":2.024,"15min":1.669},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.8879999999999999,"5min":26.800000000000001,"15min":26.800000000000001},"last":1.093},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.6930000000000001,"15min":1.522},"min":{"1min":0.80100000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":4.1289999999999996,"5min":15.906000000000001,"15min":15.906000000000001},"last":1.177}]},{"osd":6,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5470000000000002,"5min":2.181,"15min":1.6200000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":14.772,"5min":20.844000000000001,"15min":20.844000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.613,"5min":2.0499999999999998,"15min":1.6879999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.3200000000000003,"5min":26.832000000000001,"15min":26.832000000000001},"last":0.98199999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:04:48 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.8799999999999999,"15min":1.5940000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":14.789,"5min":16.039999999999999,"15min":16.039999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":2.6000000000000001,"5min":2.5379999999999998,"15min":1.823},"min":{"1min":0.75,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":14.811,"5min":29.286999999999999,"15min":29.286999999999999},"last":0.999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:10.339 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:05:10.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:05:10.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.613+0000 7f01d1600640 1 Processor -- start 2024-01-16T15:05:10.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.613+0000 7f01d1600640 1 -- start start 2024-01-16T15:05:10.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01d1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01cc15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01d1600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15e6e0 con 0x7f01cc157dc0 2024-01-16T15:05:10.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01d1600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15ba40 con 0x7f01cc15bed0 2024-01-16T15:05:10.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01d1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1581a0 con 0x7f01cc15dd40 2024-01-16T15:05:10.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01caa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01cc157dc0 0x7f01cc070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47700/0 (socket says 172.21.15.125:47700) 2024-01-16T15:05:10.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.615+0000 7f01caa00640 1 -- 172.21.15.125:0/1051594283 learned_addr learned my addr 172.21.15.125:0/1051594283 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01ca000640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01cc15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322288342 0 0) 0x7f01cc15e6e0 con 0x7f01cc157dc0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ac003520 con 0x7f01cc157dc0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113778634 0 0) 0x7f01ac003520 con 0x7f01cc157dc0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 msgr2=0x7f01cc15e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01cc15e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 >> v1:172.21.15.137:6789/0 conn(0x7f01cc15bed0 legacy=0x7f01cc15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc15eb10 con 0x7f01cc157dc0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f01bc0019f0 con 0x7f01cc157dc0 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 >> v1:172.21.15.125:6789/0 conn(0x7f01cc157dc0 legacy=0x7f01cc070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac004d00 0x7f01ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1581a0 con 0x7f01ac0039a0 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac00ba60 con 0x7f01ac004d00 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.616+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac00de30 con 0x7f01cc15dd40 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01cb400640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac004d00 0x7f01ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01ca000640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 -- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 msgr2=0x7f01ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 -- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac004d00 msgr2=0x7f01ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac004d00 0x7f01ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01caa00640 1 -- 172.21.15.125:0/1051594283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac003520 con 0x7f01ac0039a0 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01ca000640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc15dd40 0x7f01ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:10.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.617+0000 7f01cb400640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac004d00 0x7f01ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:10.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.618+0000 7f01caa00640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f01bc002230 tx=0x7f01ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb6b69482412fb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc034630 con 0x7f01ac0039a0 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc02f070 con 0x7f01ac0039a0 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01c9600640 1 -- 172.21.15.125:0/1051594283 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc034a10 con 0x7f01ac0039a0 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 -- 172.21.15.125:0/1051594283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac0039a0 msgr2=0x7f01ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 --2- 172.21.15.125:0/1051594283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f01bc002230 tx=0x7f01ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01cb400640 1 -- 172.21.15.125:0/1051594283 reap_dead start 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 -- 172.21.15.125:0/1051594283 shutdown_connections 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 -- 172.21.15.125:0/1051594283 >> 172.21.15.125:0/1051594283 conn(0x7f01cc0b0be0 msgr2=0x7f01cc0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 -- 172.21.15.125:0/1051594283 shutdown_connections 2024-01-16T15:05:10.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.619+0000 7f01d1600640 1 -- 172.21.15.125:0/1051594283 wait complete. 2024-01-16T15:05:10.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.621+0000 7f01d1600640 1 Processor -- start 2024-01-16T15:05:10.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.621+0000 7f01d1600640 1 -- start start 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 0x7f01cc0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc14ec70 0x7f01cc14f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc151ac0 con 0x7f01cc1509b0 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc151ee0 con 0x7f01ac0039a0 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01d1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc150e40 con 0x7f01cc14ec70 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.622+0000 7f01ca000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51894/0 (socket says 172.21.15.125:51894) 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 -- 172.21.15.125:0/2628996850 learned_addr learned my addr 172.21.15.125:0/2628996850 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01caa00640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 0x7f01cc0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01cb400640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc14ec70 0x7f01cc14f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 -- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc14ec70 msgr2=0x7f01cc14f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc14ec70 0x7f01cc14f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 -- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 msgr2=0x7f01cc0ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 0x7f01cc0ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 -- 172.21.15.125:0/2628996850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc151120 con 0x7f01cc1509b0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01caa00640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 0x7f01cc0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.623+0000 7f01ca000640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f01b40116d0 tx=0x7f01b4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.624+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400f9c0 con 0x7f01cc1509b0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.624+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b400fb20 con 0x7f01cc1509b0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.624+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400fed0 con 0x7f01cc1509b0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.624+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc151400 con 0x7f01cc1509b0 2024-01-16T15:05:10.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.624+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc14f620 con 0x7f01cc1509b0 2024-01-16T15:05:10.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.625+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc067670 con 0x7f01cc1509b0 2024-01-16T15:05:10.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.626+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f01b4015460 con 0x7f01cc1509b0 2024-01-16T15:05:10.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.627+0000 7f01c2a00640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 0x7f01a406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.627+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01b4088e30 con 0x7f01cc1509b0 2024-01-16T15:05:10.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.627+0000 7f01caa00640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 0x7f01a406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.628+0000 7f01caa00640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 0x7f01a406dff0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f01bc034000 tx=0x7f01bc02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.631+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01b4055520 con 0x7f01cc1509b0 2024-01-16T15:05:10.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.897+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f01cc06d3a0 con 0x7f01cc1509b0 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.901+0000 7f01c2a00640 1 -- 172.21.15.125:0/2628996850 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f01b4058b40 con 0x7f01cc1509b0 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T15:05:10.879 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:10.880 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T15:05:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 msgr2=0x7f01a406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 0x7f01a406dff0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f01bc034000 tx=0x7f01bc02f000 comp rx=0 tx=0).stop 2024-01-16T15:05:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 msgr2=0x7f01cc154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f01b40116d0 tx=0x7f01b4011b90 comp rx=0 tx=0).stop 2024-01-16T15:05:10.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 shutdown_connections 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01cc14ec70 0x7f01cc14f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a406bb30 0x7f01a406dff0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc1509b0 0x7f01cc154f50 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 --2- 172.21.15.125:0/2628996850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01ac0039a0 0x7f01cc0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 >> 172.21.15.125:0/2628996850 conn(0x7f01cc0b0be0 msgr2=0x7f01cc06cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 shutdown_connections 2024-01-16T15:05:10.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:10.904+0000 7f01d1600640 1 -- 172.21.15.125:0/2628996850 wait complete. 2024-01-16T15:05:10.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:11.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.119+0000 7f267da00640 1 Processor -- start 2024-01-16T15:05:11.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.120+0000 7f267da00640 1 -- start start 2024-01-16T15:05:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.121+0000 7f267da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f267813df80 0x7f267815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.121+0000 7f267da00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f267813e360 0x7f26780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f267da00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267806c470 con 0x7f267815f9c0 2024-01-16T15:05:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f267da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267806c750 con 0x7f267813df80 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f267da00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267806ca30 con 0x7f267813e360 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f267813df80 0x7f267815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f267813df80 0x7f267815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51898/0 (socket says 172.21.15.125:51898) 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 -- 172.21.15.125:0/2386139867 learned_addr learned my addr 172.21.15.125:0/2386139867 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f266de00640 1 --2- 172.21.15.125:0/2386139867 >> v2:172.21.15.137:3300/0 conn(0x7f267813e360 0x7f26780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 -- 172.21.15.125:0/2386139867 >> v2:172.21.15.137:3300/0 conn(0x7f267813e360 msgr2=0x7f26780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 --2- 172.21.15.125:0/2386139867 >> v2:172.21.15.137:3300/0 conn(0x7f267813e360 0x7f26780cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 -- 172.21.15.125:0/2386139867 >> v1:172.21.15.137:6789/0 conn(0x7f267815f9c0 legacy=0x7f26781408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.122+0000 7f2676a00640 1 -- 172.21.15.125:0/2386139867 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267806cd10 con 0x7f267813df80 2024-01-16T15:05:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.123+0000 7f2676a00640 1 --2- 172.21.15.125:0/2386139867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813df80 0x7f267815bee0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f267806ded0 tx=0x7f266802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc15ebeb9612c457 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.124+0000 7f2676000640 1 -- 172.21.15.125:0/2386139867 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668031870 con 0x7f267813df80 2024-01-16T15:05:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.124+0000 7f2676000640 1 -- 172.21.15.125:0/2386139867 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26680319d0 con 0x7f267813df80 2024-01-16T15:05:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.124+0000 7f2676000640 1 -- 172.21.15.125:0/2386139867 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668031d60 con 0x7f267813df80 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.124+0000 7f267da00640 1 -- 172.21.15.125:0/2386139867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813df80 msgr2=0x7f267815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.124+0000 7f267da00640 1 --2- 172.21.15.125:0/2386139867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813df80 0x7f267815bee0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f267806ded0 tx=0x7f266802b910 comp rx=0 tx=0).stop 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 -- 172.21.15.125:0/2386139867 shutdown_connections 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 --2- 172.21.15.125:0/2386139867 >> v2:172.21.15.137:3300/0 conn(0x7f267813e360 0x7f26780cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 --2- 172.21.15.125:0/2386139867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813df80 0x7f267815bee0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 -- 172.21.15.125:0/2386139867 >> 172.21.15.125:0/2386139867 conn(0x7f267806a800 msgr2=0x7f267806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 -- 172.21.15.125:0/2386139867 shutdown_connections 2024-01-16T15:05:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.125+0000 7f267da00640 1 -- 172.21.15.125:0/2386139867 wait complete. 2024-01-16T15:05:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.126+0000 7f267da00640 1 Processor -- start 2024-01-16T15:05:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.126+0000 7f267da00640 1 -- start start 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267813df80 0x7f267814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f267815f9c0 0x7f26780af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678070d90 con 0x7f267813e360 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267814c600 con 0x7f267815f9c0 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.127+0000 7f267da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267814b1b0 con 0x7f267813df80 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51900/0 (socket says 172.21.15.125:51900) 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 -- 172.21.15.125:0/1589423018 learned_addr learned my addr 172.21.15.125:0/1589423018 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 -- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267813df80 msgr2=0x7f267814aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f2676a00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267813df80 0x7f267814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f2677400640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f267815f9c0 0x7f26780af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267813df80 0x7f267814aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 -- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f267815f9c0 msgr2=0x7f26780af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f267815f9c0 0x7f26780af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.128+0000 7f266de00640 1 -- 172.21.15.125:0/1589423018 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680094d0 con 0x7f267813e360 2024-01-16T15:05:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f266de00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f266000c820 tx=0x7f266000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660004220 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2660004380 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660007a70 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26780703f0 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.129+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26780709b0 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.130+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26780675d0 con 0x7f267813e360 2024-01-16T15:05:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.131+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2660007bd0 con 0x7f267813e360 2024-01-16T15:05:11.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.132+0000 7f266f400640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 0x7f265006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.132+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2660088380 con 0x7f267813e360 2024-01-16T15:05:11.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.132+0000 7f2676a00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 0x7f265006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.133+0000 7f2676a00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 0x7f265006dff0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2668036000 tx=0x7f266803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.136+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2660054a70 con 0x7f267813e360 2024-01-16T15:05:11.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.367+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26780cc100 con 0x7f267813e360 2024-01-16T15:05:11.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.369+0000 7f266f400640 1 -- 172.21.15.125:0/1589423018 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f2660058090 con 0x7f267813e360 2024-01-16T15:05:11.347 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:11.347 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2392328978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2392328978},{"type":"v1","addr":"172.21.15.137:6834","nonce":2392328978}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5779,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2251602125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2251602125},{"type":"v1","addr":"172.21.15.137:6835","nonce":2251602125}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2951255872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2951255872},{"type":"v1","addr":"172.21.15.125:6835","nonce":2951255872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:04.842732+0000","modified":"2024-01-16T15:05:07.901312+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7125":{"gid":7125,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/22640034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":22640034},{"type":"v1","addr":"172.21.15.125:6837","nonce":22640034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:05:11.347 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 45 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 msgr2=0x7f265006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 0x7f265006dff0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2668036000 tx=0x7f266803c040 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 msgr2=0x7f26781355d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f266000c820 tx=0x7f266000cce0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 shutdown_connections 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f267815f9c0 0x7f26780af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f265006bb30 0x7f265006dff0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f267813e360 0x7f26781355d0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 --2- 172.21.15.125:0/1589423018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267813df80 0x7f267814aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.371+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 >> 172.21.15.125:0/1589423018 conn(0x7f267806a800 msgr2=0x7f267815ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.372+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 shutdown_connections 2024-01-16T15:05:11.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:11.372+0000 7f267da00640 1 -- 172.21.15.125:0/1589423018 wait complete. 2024-01-16T15:05:11.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:05:04.842732+0000', 'modified': '2024-01-16T15:05:07.901312+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7125}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7125': {'gid': 7125, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/22640034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6836', 'nonce': 22640034}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 22640034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:05:11.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:05:11.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:05:11.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.012+0000 7fa289000640 1 Processor -- start 2024-01-16T15:05:11.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.012+0000 7fa289000640 1 -- start start 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa289000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 0x7fa28415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa289000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa28413e360 0x7fa2840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa289000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28406c470 con 0x7fa28415f9c0 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa289000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28406c750 con 0x7fa28413e360 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa289000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28406ca30 con 0x7fa28413df80 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa279200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa28413e360 0x7fa2840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa282800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa28415f9c0 0x7fa2841408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47714/0 (socket says 172.21.15.125:47714) 2024-01-16T15:05:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa282800640 1 -- 172.21.15.125:0/941332198 learned_addr learned my addr 172.21.15.125:0/941332198 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa281e00640 1 --2- 172.21.15.125:0/941332198 >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 0x7fa28415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa279200640 1 -- 172.21.15.125:0/941332198 >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 msgr2=0x7fa28415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa279200640 1 --2- 172.21.15.125:0/941332198 >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 0x7fa28415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa279200640 1 -- 172.21.15.125:0/941332198 >> v1:172.21.15.125:6789/0 conn(0x7fa28415f9c0 legacy=0x7fa2841408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa279200640 1 -- 172.21.15.125:0/941332198 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28406cd10 con 0x7fa28413e360 2024-01-16T15:05:11.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.014+0000 7fa281400640 1 -- 172.21.15.125:0/941332198 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649010317 0 0) 0x7fa28406c470 con 0x7fa28415f9c0 2024-01-16T15:05:11.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.015+0000 7fa281e00640 1 --2- 172.21.15.125:0/941332198 >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 0x7fa28415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa279200640 1 --2- 172.21.15.125:0/941332198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28413e360 0x7fa2840cf290 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa284072770 tx=0x7fa26c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edc8e9948bc335e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa281400640 1 -- 172.21.15.125:0/941332198 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c031830 con 0x7fa28413e360 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa281400640 1 -- 172.21.15.125:0/941332198 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c031990 con 0x7fa28413e360 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa281400640 1 -- 172.21.15.125:0/941332198 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c031d20 con 0x7fa28413e360 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa289000640 1 -- 172.21.15.125:0/941332198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28413e360 msgr2=0x7fa2840cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa289000640 1 --2- 172.21.15.125:0/941332198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28413e360 0x7fa2840cf290 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa284072770 tx=0x7fa26c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa289000640 1 -- 172.21.15.125:0/941332198 shutdown_connections 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa289000640 1 --2- 172.21.15.125:0/941332198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28413e360 0x7fa2840cf290 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.016+0000 7fa289000640 1 --2- 172.21.15.125:0/941332198 >> v2:172.21.15.137:3300/0 conn(0x7fa28413df80 0x7fa28415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.017+0000 7fa289000640 1 -- 172.21.15.125:0/941332198 >> 172.21.15.125:0/941332198 conn(0x7fa28406a800 msgr2=0x7fa28406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.017+0000 7fa289000640 1 -- 172.21.15.125:0/941332198 shutdown_connections 2024-01-16T15:05:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.017+0000 7fa289000640 1 -- 172.21.15.125:0/941332198 wait complete. 2024-01-16T15:05:11.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.018+0000 7fa289000640 1 Processor -- start 2024-01-16T15:05:11.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.018+0000 7fa289000640 1 -- start start 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa28413df80 0x7fa28414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa28413e360 0x7fa28413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28414ddc0 con 0x7fa28415f9c0 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28413b020 con 0x7fa28413df80 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.019+0000 7fa289000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa284152fa0 con 0x7fa28413e360 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51916/0 (socket says 172.21.15.125:51916) 2024-01-16T15:05:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 -- 172.21.15.125:0/443016420 learned_addr learned my addr 172.21.15.125:0/443016420 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:11.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa281e00640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa28413df80 0x7fa28414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa279200640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa28413e360 0x7fa28413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 -- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa28413e360 msgr2=0x7fa28413aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa28413e360 0x7fa28413aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 -- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa28413df80 msgr2=0x7fa28414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa28413df80 0x7fa28414d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 -- 172.21.15.125:0/443016420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c009400 con 0x7fa28415f9c0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.020+0000 7fa282800640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa27c001560 tx=0x7fa27c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.021+0000 7fa27aa00640 1 -- 172.21.15.125:0/443016420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c00a640 con 0x7fa28415f9c0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.021+0000 7fa27aa00640 1 -- 172.21.15.125:0/443016420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c00a7a0 con 0x7fa28415f9c0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.021+0000 7fa27aa00640 1 -- 172.21.15.125:0/443016420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c00ab30 con 0x7fa28415f9c0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.021+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa284153280 con 0x7fa28415f9c0 2024-01-16T15:05:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.021+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa284153840 con 0x7fa28415f9c0 2024-01-16T15:05:12.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.023+0000 7fa27aa00640 1 -- 172.21.15.125:0/443016420 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa27c00ac90 con 0x7fa28415f9c0 2024-01-16T15:05:12.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.024+0000 7fa27aa00640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 0x7fa25c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.024+0000 7fa27aa00640 1 -- 172.21.15.125:0/443016420 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa27c087ad0 con 0x7fa28415f9c0 2024-01-16T15:05:12.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.024+0000 7fa281e00640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 0x7fa25c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.024+0000 7fa28bdb1740 1 Processor -- start 2024-01-16T15:05:12.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.025+0000 7fa28bdb1740 1 -- start start 2024-01-16T15:05:12.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.025+0000 7fa281e00640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 0x7fa25c06dff0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b840 tx=0x7fa274008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.025+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.025+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71e35aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71d93880 con 0x561b71e35fe0 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71da2ae0 con 0x561b71e2fba0 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b71da01d0 con 0x561b71e5fb80 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51926/0 (socket says 172.21.15.125:51926) 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 -- 172.21.15.125:0/4005887220 learned_addr learned my addr 172.21.15.125:0/4005887220 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 -- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 msgr2=0x561b71e5ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 -- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 msgr2=0x561b71e35aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71e35aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 -- 172.21.15.125:0/4005887220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b71d63860 con 0x561b71e35fe0 2024-01-16T15:05:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.026+0000 7fa282800640 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa27c05fc50 tx=0x7fa27c054240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea6666f2287296c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa272000640 1 -- 172.21.15.125:0/4005887220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c05e0d0 con 0x561b71e35fe0 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa272000640 1 -- 172.21.15.125:0/4005887220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c05bfd0 con 0x561b71e35fe0 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa272000640 1 -- 172.21.15.125:0/4005887220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c060180 con 0x561b71e35fe0 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 -- 172.21.15.125:0/4005887220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 msgr2=0x561b71e78560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa27c05fc50 tx=0x7fa27c054240 comp rx=0 tx=0).stop 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 -- 172.21.15.125:0/4005887220 shutdown_connections 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e35fe0 0x561b71e78560 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71e35aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/4005887220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 -- 172.21.15.125:0/4005887220 >> 172.21.15.125:0/4005887220 conn(0x561b71df6c70 msgr2=0x561b71e7a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 -- 172.21.15.125:0/4005887220 shutdown_connections 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.027+0000 7fa28bdb1740 1 -- 172.21.15.125:0/4005887220 wait complete. 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.028+0000 7fa28bdb1740 1 -- start start 2024-01-16T15:05:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.028+0000 7fa28bdb1740 1 Processor -- start 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e7e800 0x561b71e7bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71d9bac0 con 0x561b71e2fba0 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71d382f0 con 0x561b71e7e800 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.029+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b71d55c70 con 0x561b71e5fb80 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa28bdb1740 10 client.0 _ll_register_callbacks cb 0x561b71d5a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51930/0 (socket says 172.21.15.125:51930) 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 -- 172.21.15.125:0/2648722251 learned_addr learned my addr 172.21.15.125:0/2648722251 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 -- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 msgr2=0x561b71e5a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 -- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e7e800 msgr2=0x561b71e7bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e7e800 0x561b71e7bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 -- 172.21.15.125:0/2648722251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa284153840 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa281e00640 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fa274012750 tx=0x7fa274012840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1e2341f6c1a8a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.030+0000 7fa26aa00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400b050 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa274013800 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b71d94aa0 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa274012e70 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 10 client.7269 fetch_fsmap learned FSMap version 45 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x561b71d9a140 con 0x561b71e2fba0 2024-01-16T15:05:12.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b71d94aa0 con 0x561b71e2fba0 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 45) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa2740139c0 con 0x561b71e2fba0 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 10 client.7269 fetch_fsmap finished waiting for FSMap version 45 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 10 client.7269 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 20 client.7269 populate_metadata read hostname 'smithi125' 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.031+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e58a30 0x561b71e58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.032+0000 7fa28bdb1740 4 client.7269 mds_command: new command op to 7125 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.032+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 --> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b71d947c0 con 0x561b71e58a30 2024-01-16T15:05:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.032+0000 7fa279200640 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e58a30 0x561b71e58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.032+0000 7fa26aa00640 10 client.7269 ms_handle_connect on v2:172.21.15.125:6836/22640034 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.032+0000 7fa279200640 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e58a30 0x561b71e58e10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a30abfe0214dcb2e server_cookie=262e36f22f7a626a in_seq=0 out_seq=0 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.033+0000 7fa26aa00640 1 -- 172.21.15.125:0/2648722251 <== mds.0 v2:172.21.15.125:6836/22640034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b71d947c0 con 0x561b71e58a30 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.033+0000 7fa26aa00640 10 client.7269 handle_command_reply: tid=0 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.033+0000 7fa28bdb1740 1 client.7269 shutdown 2024-01-16T15:05:12.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.033+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 msgr2=0x561b71e5a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.033+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fa274012750 tx=0x7fa274012840 comp rx=0 tx=0).stop 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 shutdown_connections 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e58a30 0x561b71e58e10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e7e800 0x561b71e7bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e5fb80 0x561b71e5a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/2648722251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e2fba0 0x561b71e5a0c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa26aa00640 0 client.7269 ms_handle_reset on v2:172.21.15.125:6836/22640034 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 >> 172.21.15.125:0/2648722251 conn(0x561b71df6c70 msgr2=0x561b71e5d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 shutdown_connections 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 1 -- 172.21.15.125:0/2648722251 wait complete. 2024-01-16T15:05:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.034+0000 7fa28bdb1740 20 client.7269 trim_cache size 0 max 16384 2024-01-16T15:05:12.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.051+0000 7fa28bdb1740 1 Processor -- start 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.051+0000 7fa28bdb1740 1 -- start start 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 0x561b71e58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 0x561b71e59350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71afef20 con 0x561b71e7e800 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71e31b00 con 0x561b71e5fb80 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b71e5c920 con 0x561b71e2fba0 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.052+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51938/0 (socket says 172.21.15.125:51938) 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 -- 172.21.15.125:0/1728176310 learned_addr learned my addr 172.21.15.125:0/1728176310 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa279200640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 0x561b71e59350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 -- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 msgr2=0x561b71e58e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa281e00640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 0x561b71e58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 0x561b71e58e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 -- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 msgr2=0x561b71e59350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 0x561b71e59350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 -- 172.21.15.125:0/1728176310 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b71d37b80 con 0x561b71e7e800 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa281e00640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 0x561b71e58e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa279200640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 0x561b71e59350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa282800640 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00a430 tx=0x7fa27c08a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62db88c2eaa583d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa272a00640 1 -- 172.21.15.125:0/1728176310 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c060180 con 0x561b71e7e800 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.053+0000 7fa272a00640 1 -- 172.21.15.125:0/1728176310 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c056220 con 0x561b71e7e800 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa272a00640 1 -- 172.21.15.125:0/1728176310 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c054590 con 0x561b71e7e800 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 -- 172.21.15.125:0/1728176310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 msgr2=0x561b71e7ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00a430 tx=0x7fa27c08a050 comp rx=0 tx=0).stop 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 -- 172.21.15.125:0/1728176310 shutdown_connections 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e7e800 0x561b71e7ce10 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e5fb80 0x561b71e59350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/1728176310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e2fba0 0x561b71e58e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 -- 172.21.15.125:0/1728176310 >> 172.21.15.125:0/1728176310 conn(0x561b71df6c70 msgr2=0x561b71e2eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 -- 172.21.15.125:0/1728176310 shutdown_connections 2024-01-16T15:05:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.054+0000 7fa28bdb1740 1 -- 172.21.15.125:0/1728176310 wait complete. 2024-01-16T15:05:12.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.055+0000 7fa28bdb1740 1 -- start start 2024-01-16T15:05:12.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.055+0000 7fa28bdb1740 1 Processor -- start 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71eec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 0x561b71e8d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71e35240 con 0x561b71e5fb80 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b71e33b90 con 0x561b71e2fba0 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b71e590a0 con 0x561b71e7e800 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa28bdb1740 10 client.0 _ll_register_callbacks cb 0x7fa27400de30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa279200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa279200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51950/0 (socket says 172.21.15.125:51950) 2024-01-16T15:05:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.056+0000 7fa279200640 1 -- 172.21.15.125:0/559543312 learned_addr learned my addr 172.21.15.125:0/559543312 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa281e00640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71eec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa282800640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 0x561b71e8d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 -- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 msgr2=0x561b71e8d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 0x561b71e8d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 -- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 msgr2=0x561b71eec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71eec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 -- 172.21.15.125:0/559543312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b71d63860 con 0x561b71e5fb80 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa282800640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 0x561b71e8d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa279200640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009890 tx=0x7fa26c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad9274361df4e64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa26aa00640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c03f040 con 0x561b71e5fb80 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa26c046220 con 0x561b71e5fb80 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.057+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa27c05bfd0 con 0x561b71e5fb80 2024-01-16T15:05:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c0463e0 con 0x561b71e5fb80 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 10 client.7275 fetch_fsmap learned FSMap version 45 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x561b71d5d170 con 0x561b71e5fb80 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c05bfd0 con 0x561b71e5fb80 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 45) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa26c046540 con 0x561b71e5fb80 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 10 client.7275 fetch_fsmap finished waiting for FSMap version 45 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 10 client.7275 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 20 client.7275 populate_metadata read hostname 'smithi125' 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e33470 0x561b71e8d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 4 client.7275 mds_command: new command op to 7125 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 --> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561b71a9c6c0 con 0x561b71e33470 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.058+0000 7fa281e00640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e33470 0x561b71e8d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa281e00640 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e33470 0x561b71e8d9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185f55f9548de146 server_cookie=6ea64acf945b8839 in_seq=0 out_seq=0 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa26aa00640 10 client.7275 ms_handle_connect on v2:172.21.15.125:6836/22640034 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa26aa00640 1 -- 172.21.15.125:0/559543312 <== mds.0 v2:172.21.15.125:6836/22640034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561b71a9c6c0 con 0x561b71e33470 2024-01-16T15:05:12.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa26aa00640 10 client.7275 handle_command_reply: tid=0 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 client.7275 shutdown 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 msgr2=0x561b71e47cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009890 tx=0x7fa26c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 shutdown_connections 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:6836/22640034,v1:172.21.15.125:6837/22640034] conn(0x561b71e33470 0x561b71e8d9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x561b71e7e800 0x561b71e8d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x561b71e5fb80 0x561b71e47cb0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 --2- 172.21.15.125:0/559543312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x561b71e2fba0 0x561b71eec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa26aa00640 0 client.7275 ms_handle_reset on v2:172.21.15.125:6836/22640034 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.059+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 >> 172.21.15.125:0/559543312 conn(0x561b71df6c70 msgr2=0x561b71e7dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 shutdown_connections 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa28bdb1740 1 -- 172.21.15.125:0/559543312 wait complete. 2024-01-16T15:05:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa28bdb1740 20 client.7275 trim_cache size 0 max 16384 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 7125, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.6689600249999996, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 45, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 70, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 70, 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 13.137778399 2024-01-16T15:05:12.042 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 msgr2=0x7fa25c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 0x7fa25c06dff0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b840 tx=0x7fa274008040 comp rx=0 tx=0).stop 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 msgr2=0x7fa28414b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa27c001560 tx=0x7fa27c00cc70 comp rx=0 tx=0).stop 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 shutdown_connections 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa25c06bb30 0x7fa25c06dff0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa28415f9c0 0x7fa28414b830 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa28413e360 0x7fa28413aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 --2- 172.21.15.125:0/443016420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa28413df80 0x7fa28414d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.060+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 >> 172.21.15.125:0/443016420 conn(0x7fa28406a800 msgr2=0x7fa2840b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.061+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 shutdown_connections 2024-01-16T15:05:12.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.061+0000 7fa289000640 1 -- 172.21.15.125:0/443016420 wait complete. 2024-01-16T15:05:12.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:12.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.283+0000 7f871d800640 1 Processor -- start 2024-01-16T15:05:12.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.283+0000 7f871d800640 1 -- start start 2024-01-16T15:05:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f871d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8718143c40 0x7f8718071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f871d800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87180717d0 0x7f87180cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f871d800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8718070e50 con 0x7f87180cca40 2024-01-16T15:05:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f871d800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87180723c0 con 0x7f87180717d0 2024-01-16T15:05:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f871d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180cce70 con 0x7f8718143c40 2024-01-16T15:05:12.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f870f200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87180717d0 0x7f87180cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f870f200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87180717d0 0x7f87180cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51952/0 (socket says 172.21.15.125:51952) 2024-01-16T15:05:12.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.284+0000 7f870f200640 1 -- 172.21.15.125:0/2646076841 learned_addr learned my addr 172.21.15.125:0/2646076841 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.285+0000 7f8717e00640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8718143c40 0x7f8718071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.285+0000 7f870f200640 1 -- 172.21.15.125:0/2646076841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8718143c40 msgr2=0x7f8718071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.285+0000 7f870f200640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8718143c40 0x7f8718071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.285+0000 7f870f200640 1 -- 172.21.15.125:0/2646076841 >> v1:172.21.15.137:6789/0 conn(0x7f87180cca40 legacy=0x7f871815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.285+0000 7f870f200640 1 -- 172.21.15.125:0/2646076841 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871806e170 con 0x7f87180717d0 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.286+0000 7f870f200640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87180717d0 0x7f87180cc500 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8700009870 tx=0x7f870002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ac62500e23b50f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.286+0000 7f8717400640 1 -- 172.21.15.125:0/2646076841 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000317c0 con 0x7f87180717d0 2024-01-16T15:05:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.286+0000 7f8717400640 1 -- 172.21.15.125:0/2646076841 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700031920 con 0x7f87180717d0 2024-01-16T15:05:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.286+0000 7f8717400640 1 -- 172.21.15.125:0/2646076841 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700031cb0 con 0x7f87180717d0 2024-01-16T15:05:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 -- 172.21.15.125:0/2646076841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87180717d0 msgr2=0x7f87180cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87180717d0 0x7f87180cc500 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8700009870 tx=0x7f870002b860 comp rx=0 tx=0).stop 2024-01-16T15:05:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 -- 172.21.15.125:0/2646076841 shutdown_connections 2024-01-16T15:05:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87180717d0 0x7f87180cc500 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 --2- 172.21.15.125:0/2646076841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8718143c40 0x7f8718071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 -- 172.21.15.125:0/2646076841 >> 172.21.15.125:0/2646076841 conn(0x7f871806a800 msgr2=0x7f871806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 -- 172.21.15.125:0/2646076841 shutdown_connections 2024-01-16T15:05:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.287+0000 7f871d800640 1 -- 172.21.15.125:0/2646076841 wait complete. 2024-01-16T15:05:12.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.289+0000 7f871d800640 1 Processor -- start 2024-01-16T15:05:12.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.289+0000 7f871d800640 1 -- start start 2024-01-16T15:05:12.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87180717d0 0x7f87181f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 0x7f871813b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871813b940 con 0x7f8718143c40 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718159ad0 con 0x7f87180717d0 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871813bba0 con 0x7f87180cca40 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51958/0 (socket says 172.21.15.125:51958) 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f871ce00640 1 -- 172.21.15.125:0/1470945055 learned_addr learned my addr 172.21.15.125:0/1470945055 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f8717e00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87180717d0 0x7f87181f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.290+0000 7f870f200640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 0x7f871813b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 -- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 msgr2=0x7f871813b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 0x7f871813b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 -- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87180717d0 msgr2=0x7f87181f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87180717d0 0x7f87181f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 -- 172.21.15.125:0/1470945055 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87000094d0 con 0x7f8718143c40 2024-01-16T15:05:12.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f870f200640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 0x7f871813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871ce00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f871000c780 tx=0x7f871000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710004b60 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871000ce90 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871813be80 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871000a850 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.291+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871813a3a0 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.292+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8718067450 con 0x7f8718143c40 2024-01-16T15:05:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.293+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f871000a9b0 con 0x7f8718143c40 2024-01-16T15:05:12.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.294+0000 7f8714c00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 0x7f86f006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.294+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8710088730 con 0x7f8718143c40 2024-01-16T15:05:12.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.294+0000 7f8717e00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 0x7f86f006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.295+0000 7f8717e00640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 0x7f86f006dff0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8708006fd0 tx=0x7f8708008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.299+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8710054e80 con 0x7f8718143c40 2024-01-16T15:05:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.528+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f87180cd930 con 0x7f8718143c40 2024-01-16T15:05:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.528+0000 7f8714c00640 1 -- 172.21.15.125:0/1470945055 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7f87100584a0 con 0x7f8718143c40 2024-01-16T15:05:12.506 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.530+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 msgr2=0x7f86f006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.530+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 0x7f86f006dff0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8708006fd0 tx=0x7f8708008040 comp rx=0 tx=0).stop 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.530+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 msgr2=0x7f8718159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.530+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f871000c780 tx=0x7f871000cc40 comp rx=0 tx=0).stop 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 shutdown_connections 2024-01-16T15:05:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f86f006bb30 0x7f86f006dff0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8718143c40 0x7f8718159590 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87180cca40 0x7f871813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 --2- 172.21.15.125:0/1470945055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87180717d0 0x7f87181f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 >> 172.21.15.125:0/1470945055 conn(0x7f871806a800 msgr2=0x7f871806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 shutdown_connections 2024-01-16T15:05:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:12.531+0000 7f871d800640 1 -- 172.21.15.125:0/1470945055 wait complete. 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:12.526 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:12.527 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:12.527 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:12.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.527 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:05:12.578 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:12.579 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42136sec preferred_lft 42136sec 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.580 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:05:12.581 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:12.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.582 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:12.669 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:05:12.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.674 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:05:12.727 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:05:12.727 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:12.728 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:05:12.728 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:05:12.728 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:12.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:05:12.729 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:05:12.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.901 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:05:12.949 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:05:12.949 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:05:12.950 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:12.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:12.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:05:13.047 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:05:13.047 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:13.047 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:13.097 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:13.098 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:13.098 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:13.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:05:13.237 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:13.293 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:05:13.293 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:05:13.293 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:05:13.293 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:05:13.294 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:13.295 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:13.296 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:05:13.296 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:13.296 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:13.296 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:13.296 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:13.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:13.297 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:05:13.349 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:13.349 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:05:13.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:05:13.432 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:05:13.475 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:13.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:13.479 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:05:13.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:05:13.567+0000 7f6e35531480 -1 init, newargv = 0x5635366d7af0 newargc=15 2024-01-16T15:05:13.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[63461]: starting ceph client 2024-01-16T15:05:13.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[63461]: starting fuse 2024-01-16T15:05:13.566 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:05:13.566 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:05:13.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:13.566 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:05:13.566 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:05:13.566 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:05:13.566 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:13.567 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:05:13.568 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:05:13.760 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.63461.asok 2024-01-16T15:05:13.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:13.776 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63461.asok 2024-01-16T15:05:13.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:13.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63461.asok status 2024-01-16T15:05:14.009 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "63461", 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 7290, 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:05:14.010 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 7290 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 165712452 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 165712452 2024-01-16T15:05:14.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.7290 192.168.0.1:0/165712452", 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/165712452", 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 45, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 70, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:05:14.012 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:05:14.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.023 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:05:14.023 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:14.076 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:05:14.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:14.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:14.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.403+0000 7faff5c00640 1 Processor -- start 2024-01-16T15:05:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.403+0000 7faff5c00640 1 -- start start 2024-01-16T15:05:14.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.404+0000 7faff5c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faff013ff20 0x7faff015e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7faff5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7faff0141c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7faff5c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00b5440 con 0x7faff0140300 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7faff5c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff0161c80 con 0x7faff013ff20 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7faff5c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff015e6f0 con 0x7faff006aaf0 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7fafef400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faff0140300 0x7faff006a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47728/0 (socket says 172.21.15.125:47728) 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7fafef400640 1 -- 172.21.15.125:0/1391037367 learned_addr learned my addr 172.21.15.125:0/1391037367 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7fafefe00640 1 --2- 172.21.15.125:0/1391037367 >> v2:172.21.15.137:3300/0 conn(0x7faff013ff20 0x7faff015e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7faff5200640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7faff0141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.405+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194881365 0 0) 0x7faff00b5440 con 0x7faff0140300 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd0003520 con 0x7faff0140300 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279197603 0 0) 0x7fafd0003520 con 0x7faff0140300 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 msgr2=0x7faff0141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7faff0141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 >> v2:172.21.15.137:3300/0 conn(0x7faff013ff20 msgr2=0x7faff015e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 --2- 172.21.15.125:0/1391037367 >> v2:172.21.15.137:3300/0 conn(0x7faff013ff20 0x7faff015e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff00b4830 con 0x7faff0140300 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fafe0001670 con 0x7faff0140300 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 >> v1:172.21.15.125:6789/0 conn(0x7faff0140300 legacy=0x7faff006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7fafd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7fafd00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafd0004d00 0x7fafd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0161c80 con 0x7fafd0003970 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff015e6f0 con 0x7fafd0004d00 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd000b9d0 con 0x7faff006aaf0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.406+0000 7fafefe00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7fafd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafef400640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7fafd00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafefe00640 1 -- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 msgr2=0x7fafd00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7faff5200640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafd0004d00 0x7fafd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafefe00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff006aaf0 0x7fafd00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafefe00640 1 -- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafd0004d00 msgr2=0x7fafd0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafefe00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafd0004d00 0x7fafd0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7fafefe00640 1 -- 172.21.15.125:0/1391037367 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0003520 con 0x7fafd0003970 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.407+0000 7faff5200640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fafd0004d00 0x7fafd0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.408+0000 7fafefe00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7fafd0003d90 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009af0 tx=0x7fafd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe88d8c31ab5296 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.408+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd802c710 con 0x7fafd0003970 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.408+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd802c870 con 0x7fafd0003970 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.408+0000 7fafeea00640 1 -- 172.21.15.125:0/1391037367 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd802cc00 con 0x7fafd0003970 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 -- 172.21.15.125:0/1391037367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 msgr2=0x7fafd0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 --2- 172.21.15.125:0/1391037367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7fafd0003d90 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009af0 tx=0x7fafd00057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5200640 1 -- 172.21.15.125:0/1391037367 reap_dead start 2024-01-16T15:05:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 -- 172.21.15.125:0/1391037367 shutdown_connections 2024-01-16T15:05:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 -- 172.21.15.125:0/1391037367 >> 172.21.15.125:0/1391037367 conn(0x7faff00b1cd0 msgr2=0x7faff00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 -- 172.21.15.125:0/1391037367 shutdown_connections 2024-01-16T15:05:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.409+0000 7faff5c00640 1 -- 172.21.15.125:0/1391037367 wait complete. 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.411+0000 7faff5c00640 1 Processor -- start 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.411+0000 7faff5c00640 1 -- start start 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff0135190 0x7faff0135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faff01370d0 0x7faff0137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff013c110 con 0x7fafd0003970 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0135b50 con 0x7faff01370d0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7faff5c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff0134dc0 con 0x7faff0135190 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7fafefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7fafefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51988/0 (socket says 172.21.15.125:51988) 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.412+0000 7fafefe00640 1 -- 172.21.15.125:0/213329034 learned_addr learned my addr 172.21.15.125:0/213329034 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafef400640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff0135190 0x7faff0135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7faff5200640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faff01370d0 0x7faff0137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 -- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff0135190 msgr2=0x7faff0135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff0135190 0x7faff0135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 -- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faff01370d0 msgr2=0x7faff0137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faff01370d0 0x7faff0137580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 -- 172.21.15.125:0/213329034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0137e70 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafefe00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fafd8037040 tx=0x7fafd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8003810 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd802de90 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8003b90 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff0138150 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.413+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff013bd30 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.415+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff0067450 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.416+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fafd8003cf0 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.416+0000 7fafe7e00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 0x7fafc806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.417+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafd80aa800 con 0x7fafd0003970 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.417+0000 7fafef400640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 0x7fafc806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.417+0000 7fafef400640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 0x7fafc806dff0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fafe00018e0 tx=0x7fafe0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.421+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafd8077010 con 0x7fafd0003970 2024-01-16T15:05:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.650+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faff00cf280 con 0x7fafd0003970 2024-01-16T15:05:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.651+0000 7fafe7e00640 1 -- 172.21.15.125:0/213329034 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7fafd807a630 con 0x7fafd0003970 2024-01-16T15:05:14.630 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 msgr2=0x7fafc806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 0x7fafc806dff0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fafe00018e0 tx=0x7fafe0009040 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 msgr2=0x7faff0134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fafd8037040 tx=0x7fafd8004730 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 shutdown_connections 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faff01370d0 0x7faff0137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faff0135190 0x7faff0135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fafc806bb30 0x7fafc806dff0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 --2- 172.21.15.125:0/213329034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafd0003970 0x7faff0134880 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 >> 172.21.15.125:0/213329034 conn(0x7faff00b1cd0 msgr2=0x7faff013ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 shutdown_connections 2024-01-16T15:05:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:14.654+0000 7faff5c00640 1 -- 172.21.15.125:0/213329034 wait complete. 2024-01-16T15:05:14.651 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:14.652 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.653 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.710 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42123sec preferred_lft 42123sec 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:05:14.711 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:05:14.712 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.713 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:14.803 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:05:14.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.808 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.860 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:05:14.860 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:05:14.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:05:14.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:05:14.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:05:14.861 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:05:15.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.034 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:05:15.082 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:05:15.082 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:05:15.083 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:15.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:05:15.176 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:05:15.177 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:15.177 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:15.231 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:15.232 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:15.232 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:15.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:05:15.376 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:15.430 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:05:15.430 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:05:15.430 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:05:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:05:15.432 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:15.433 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:15.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.434 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:05:15.489 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:15.489 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:05:15.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:05:15.569 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:15.613 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:05:15.613 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:05:15.613 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:05:15.613 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:05:15.614 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:15.616 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:15.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.617 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:05:15.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:05:15.698+0000 7f109e9ff480 -1 init, newargv = 0x56170b7b8510 newargc=15 2024-01-16T15:05:15.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45115]: starting ceph client 2024-01-16T15:05:15.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45115]: starting fuse 2024-01-16T15:05:15.698 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:05:15.699 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:05:15.699 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:15.700 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:05:15.701 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:05:15.896 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.45115.asok 2024-01-16T15:05:15.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:05:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:15.913 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45115.asok 2024-01-16T15:05:15.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45115.asok status 2024-01-16T15:05:16.147 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "45115", 2024-01-16T15:05:16.148 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5384, 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5384 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:05:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 780182527 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 780182527 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:05:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5384 192.168.0.1:0/780182527", 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/780182527", 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 45, 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 70, 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:05:16.151 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:05:16.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.164 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:05:16.164 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:16.216 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:05:16.216 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:16.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:16.308 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:05:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.547+0000 7f40f8800640 1 Processor -- start 2024-01-16T15:05:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.547+0000 7f40f8800640 1 -- start start 2024-01-16T15:05:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.549+0000 7f40f8800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40f0143ae0 0x7f40f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.549+0000 7f40f8800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f40f015e3b0 0x7f40f00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.549+0000 7f40f8800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f00cfc30 con 0x7f40f00cf7d0 2024-01-16T15:05:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.549+0000 7f40f8800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f00b1ee0 con 0x7f40f0143ae0 2024-01-16T15:05:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.549+0000 7f40f8800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f00b2780 con 0x7f40f015e3b0 2024-01-16T15:05:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40f0143ae0 0x7f40f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40f0143ae0 0x7f40f00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51998/0 (socket says 172.21.15.125:51998) 2024-01-16T15:05:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 -- 172.21.15.125:0/2379688615 learned_addr learned my addr 172.21.15.125:0/2379688615 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f6800640 1 --2- 172.21.15.125:0/2379688615 >> v2:172.21.15.137:3300/0 conn(0x7f40f015e3b0 0x7f40f00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 -- 172.21.15.125:0/2379688615 >> v2:172.21.15.137:3300/0 conn(0x7f40f015e3b0 msgr2=0x7f40f00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 --2- 172.21.15.125:0/2379688615 >> v2:172.21.15.137:3300/0 conn(0x7f40f015e3b0 0x7f40f00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 -- 172.21.15.125:0/2379688615 >> v1:172.21.15.137:6789/0 conn(0x7f40f00cf7d0 legacy=0x7f40f015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.550+0000 7f40f5e00640 1 -- 172.21.15.125:0/2379688615 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f015e990 con 0x7f40f0143ae0 2024-01-16T15:05:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.551+0000 7f40f5e00640 1 --2- 172.21.15.125:0/2379688615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f0143ae0 0x7f40f00cf290 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f40f006e4f0 tx=0x7f40e002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606961f8cb9a2678 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.551+0000 7f40f4a00640 1 -- 172.21.15.125:0/2379688615 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e002f8e0 con 0x7f40f0143ae0 2024-01-16T15:05:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f4a00640 1 -- 172.21.15.125:0/2379688615 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e002fa40 con 0x7f40f0143ae0 2024-01-16T15:05:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f4a00640 1 -- 172.21.15.125:0/2379688615 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e002fdd0 con 0x7f40f0143ae0 2024-01-16T15:05:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 -- 172.21.15.125:0/2379688615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f0143ae0 msgr2=0x7f40f00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 --2- 172.21.15.125:0/2379688615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f0143ae0 0x7f40f00cf290 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f40f006e4f0 tx=0x7f40e002db20 comp rx=0 tx=0).stop 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 -- 172.21.15.125:0/2379688615 shutdown_connections 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 --2- 172.21.15.125:0/2379688615 >> v2:172.21.15.137:3300/0 conn(0x7f40f015e3b0 0x7f40f00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 --2- 172.21.15.125:0/2379688615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f0143ae0 0x7f40f00cf290 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 -- 172.21.15.125:0/2379688615 >> 172.21.15.125:0/2379688615 conn(0x7f40f006a800 msgr2=0x7f40f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 -- 172.21.15.125:0/2379688615 shutdown_connections 2024-01-16T15:05:16.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.552+0000 7f40f8800640 1 -- 172.21.15.125:0/2379688615 wait complete. 2024-01-16T15:05:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.554+0000 7f40f8800640 1 Processor -- start 2024-01-16T15:05:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.554+0000 7f40f8800640 1 -- start start 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 0x7f40f013fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40f015e3b0 0x7f40f00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f01f2a70 con 0x7f40f00cf7d0 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f015a060 con 0x7f40f0143ae0 2024-01-16T15:05:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f8800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0141210 con 0x7f40f015e3b0 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59022/0 (socket says 172.21.15.125:59022) 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.555+0000 7f40f5e00640 1 -- 172.21.15.125:0/798198475 learned_addr learned my addr 172.21.15.125:0/798198475 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5400640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 0x7f40f013fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 -- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40f015e3b0 msgr2=0x7f40f00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40f015e3b0 0x7f40f00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 -- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 msgr2=0x7f40f013fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 0x7f40f013fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 -- 172.21.15.125:0/798198475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00094d0 con 0x7f40f00cf7d0 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5400640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 0x7f40f013fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.556+0000 7f40f5e00640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f40e0000c00 tx=0x7f40e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.557+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0037d30 con 0x7f40f00cf7d0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.557+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0042470 con 0x7f40f00cf7d0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.557+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0042820 con 0x7f40f00cf7d0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.557+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f0140600 con 0x7f40f00cf7d0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.557+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f0140bc0 con 0x7f40f00cf7d0 2024-01-16T15:05:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.558+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f00675d0 con 0x7f40f00cf7d0 2024-01-16T15:05:16.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.559+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f40e0042980 con 0x7f40f00cf7d0 2024-01-16T15:05:16.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.560+0000 7f40ee000640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 0x7f40c806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.560+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40e0033070 con 0x7f40f00cf7d0 2024-01-16T15:05:16.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.560+0000 7f40f5400640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 0x7f40c806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.561+0000 7f40f5400640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 0x7f40c806dff0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f40f00ca570 tx=0x7f40d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.565+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40e0077e80 con 0x7f40f00cf7d0 2024-01-16T15:05:16.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:16.793+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f40f0143050 con 0x7f40f00cf7d0 2024-01-16T15:05:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.470+0000 7f40ee000640 1 -- 172.21.15.125:0/798198475 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 136+0+0 (secure 0 0 0) 0x7f40e0046020 con 0x7f40f00cf7d0 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 msgr2=0x7f40c806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 0x7f40c806dff0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f40f00ca570 tx=0x7f40d8009040 comp rx=0 tx=0).stop 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 msgr2=0x7f40f0159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f40e0000c00 tx=0x7f40e00046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 shutdown_connections 2024-01-16T15:05:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40f015e3b0 0x7f40f00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40f0143ae0 0x7f40f013fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f40c806bb30 0x7f40c806dff0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 --2- 172.21.15.125:0/798198475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f00cf7d0 0x7f40f0159b20 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 >> 172.21.15.125:0/798198475 conn(0x7f40f006a800 msgr2=0x7f40f006c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 shutdown_connections 2024-01-16T15:05:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.474+0000 7f40f8800640 1 -- 172.21.15.125:0/798198475 wait complete. 2024-01-16T15:05:17.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:05:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.708+0000 7fc2c2a00640 1 Processor -- start 2024-01-16T15:05:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.709+0000 7fc2c2a00640 1 -- start start 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.710+0000 7fc2c2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc2bc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.710+0000 7fc2c2a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc070c40 con 0x7fc2bc0cf290 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.710+0000 7fc2c2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc06c470 con 0x7fc2bc0b3ae0 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.710+0000 7fc2c2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc071450 con 0x7fc2bc0ae5a0 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2c0a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2bc0cf290 0x7fc2bc0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33042/0 (socket says 172.21.15.125:33042) 2024-01-16T15:05:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2c0a00640 1 -- 172.21.15.125:0/209616087 learned_addr learned my addr 172.21.15.125:0/209616087 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc2bc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528063714 0 0) 0x7fc2bc070c40 con 0x7fc2bc0cf290 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc29c003520 con 0x7fc2bc0cf290 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3121673837 0 0) 0x7fc29c003520 con 0x7fc2bc0cf290 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 msgr2=0x7fc2bc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc2bc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 >> v1:172.21.15.137:6789/0 conn(0x7fc2bc0b3ae0 legacy=0x7fc2bc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc06c750 con 0x7fc2bc0cf290 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.711+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc2b4001670 con 0x7fc2bc0cf290 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 >> v1:172.21.15.125:6789/0 conn(0x7fc2bc0cf290 legacy=0x7fc2bc0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc29c0039a0 0x7fc29c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0047c0 0x7fc29c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc29c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc071450 con 0x7fc29c0047c0 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c00bac0 con 0x7fc29c0039a0 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c00bc80 con 0x7fc2bc0ae5a0 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0047c0 0x7fc29c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc29c0039a0 0x7fc29c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2c0a00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc29c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 -- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 msgr2=0x7fc29c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc0ae5a0 0x7fc29c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 -- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc29c0039a0 msgr2=0x7fc29c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc29c0039a0 0x7fc29c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.712+0000 7fc2bb400640 1 -- 172.21.15.125:0/209616087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c003520 con 0x7fc29c0047c0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.713+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc29c0039a0 0x7fc29c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2bb400640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0047c0 0x7fc29c004c40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4001e20 tx=0x7fc29c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2defa6bce9c6bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4003b70 con 0x7fc29c0047c0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a4003cd0 con 0x7fc29c0047c0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2baa00640 1 -- 172.21.15.125:0/209616087 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4005310 con 0x7fc29c0047c0 2024-01-16T15:05:17.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2c2a00640 1 -- 172.21.15.125:0/209616087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0047c0 msgr2=0x7fc29c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/209616087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0047c0 0x7fc29c004c40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4001e20 tx=0x7fc29c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2c0a00640 1 -- 172.21.15.125:0/209616087 reap_dead start 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2c2a00640 1 -- 172.21.15.125:0/209616087 shutdown_connections 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.714+0000 7fc2c2a00640 1 -- 172.21.15.125:0/209616087 >> 172.21.15.125:0/209616087 conn(0x7fc2bc06a800 msgr2=0x7fc2bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.715+0000 7fc2c2a00640 1 -- 172.21.15.125:0/209616087 shutdown_connections 2024-01-16T15:05:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.715+0000 7fc2c2a00640 1 -- 172.21.15.125:0/209616087 wait complete. 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.716+0000 7fc2c2a00640 1 Processor -- start 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.717+0000 7fc2c2a00640 1 -- start start 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc29c0047c0 0x7fc2bc14b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc155c10 0x7fc2bc0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc156f60 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0cdfb0 con 0x7fc2bc155c10 2024-01-16T15:05:17.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0ce250 con 0x7fc29c0047c0 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59038/0 (socket says 172.21.15.125:59038) 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 -- 172.21.15.125:0/3480678621 learned_addr learned my addr 172.21.15.125:0/3480678621 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc29c0047c0 0x7fc2bc14b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2bb400640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc155c10 0x7fc2bc0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 -- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc29c0047c0 msgr2=0x7fc2bc14b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc29c0047c0 0x7fc2bc14b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 -- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc155c10 msgr2=0x7fc2bc0cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc155c10 0x7fc2bc0cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.718+0000 7fc2c0a00640 1 -- 172.21.15.125:0/3480678621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0ce530 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2c0a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4001200 tx=0x7fc2b400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b40074e0 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b4007640 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc1571b0 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4004a00 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.719+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc1f3240 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.720+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc1f33d0 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.721+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc2b4004b60 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.722+0000 7fc2b3e00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 0x7fc29406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.722+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2b408a680 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.723+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 0x7fc29406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.723+0000 7fc2bbe00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 0x7fc29406dff0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009a90 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.727+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2b40547d0 con 0x7fc2bc0cc6f0 2024-01-16T15:05:17.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.931+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2bc070550 con 0x7fc29406bb30 2024-01-16T15:05:17.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.934+0000 7fc2b3e00640 1 -- 172.21.15.125:0/3480678621 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc2ac013a20 con 0x7fc29406bb30 2024-01-16T15:05:17.913 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:05:17.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:05:17.913 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:05:17.913 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:05:17.913 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 msgr2=0x7fc29406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 0x7fc29406dff0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009a90 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 msgr2=0x7fc2bc0b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4001200 tx=0x7fc2b400d9d0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 shutdown_connections 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc29406bb30 0x7fc29406dff0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2bc0cc6f0 0x7fc2bc0b28a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc155c10 0x7fc2bc0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 --2- 172.21.15.125:0/3480678621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc29c0047c0 0x7fc2bc14b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 >> 172.21.15.125:0/3480678621 conn(0x7fc2bc06a800 msgr2=0x7fc2bc15e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 shutdown_connections 2024-01-16T15:05:17.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:17.937+0000 7fc2c2a00640 1 -- 172.21.15.125:0/3480678621 wait complete. 2024-01-16T15:05:17.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:05:18.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.164+0000 7fefec400640 1 Processor -- start 2024-01-16T15:05:18.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.164+0000 7fefec400640 1 -- start start 2024-01-16T15:05:18.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefec400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fefe40b3ae0 0x7fefe415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefec400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefe40cf290 0x7fefe40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefec400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe4070c40 con 0x7fefe40ae5a0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefec400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe4071450 con 0x7fefe40cf290 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefec400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe40cfc60 con 0x7fefe40b3ae0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefe9a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fefe40ae5a0 0x7fefe406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33046/0 (socket says 172.21.15.125:33046) 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefe9a00640 1 -- 172.21.15.125:0/1670599165 learned_addr learned my addr 172.21.15.125:0/1670599165 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefea400640 1 --2- 172.21.15.125:0/1670599165 >> v2:172.21.15.125:3300/0 conn(0x7fefe40cf290 0x7fefe40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.166+0000 7fefe9000640 1 --2- 172.21.15.125:0/1670599165 >> v2:172.21.15.137:3300/0 conn(0x7fefe40b3ae0 0x7fefe415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefe3e00640 1 -- 172.21.15.125:0/1670599165 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876873981 0 0) 0x7fefe4070c40 con 0x7fefe40ae5a0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefe3e00640 1 -- 172.21.15.125:0/1670599165 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4003520 con 0x7fefe40ae5a0 2024-01-16T15:05:18.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefea400640 1 -- 172.21.15.125:0/1670599165 >> v2:172.21.15.137:3300/0 conn(0x7fefe40b3ae0 msgr2=0x7fefe415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefea400640 1 --2- 172.21.15.125:0/1670599165 >> v2:172.21.15.137:3300/0 conn(0x7fefe40b3ae0 0x7fefe415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefea400640 1 -- 172.21.15.125:0/1670599165 >> v1:172.21.15.125:6789/0 conn(0x7fefe40ae5a0 legacy=0x7fefe406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.167+0000 7fefea400640 1 -- 172.21.15.125:0/1670599165 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe406c470 con 0x7fefe40cf290 2024-01-16T15:05:18.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.168+0000 7fefea400640 1 --2- 172.21.15.125:0/1670599165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40cf720 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fefe40cfea0 tx=0x7fefdc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921b21e19e309118 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.168+0000 7fefe3e00640 1 -- 172.21.15.125:0/1670599165 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc02cd00 con 0x7fefe40cf290 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.168+0000 7fefe3e00640 1 -- 172.21.15.125:0/1670599165 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc02ce60 con 0x7fefe40cf290 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.168+0000 7fefe3e00640 1 -- 172.21.15.125:0/1670599165 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc034660 con 0x7fefe40cf290 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 -- 172.21.15.125:0/1670599165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 msgr2=0x7fefe40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 --2- 172.21.15.125:0/1670599165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40cf720 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fefe40cfea0 tx=0x7fefdc02bf50 comp rx=0 tx=0).stop 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 -- 172.21.15.125:0/1670599165 shutdown_connections 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 --2- 172.21.15.125:0/1670599165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40cf720 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 --2- 172.21.15.125:0/1670599165 >> v2:172.21.15.137:3300/0 conn(0x7fefe40b3ae0 0x7fefe415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 -- 172.21.15.125:0/1670599165 >> 172.21.15.125:0/1670599165 conn(0x7fefe406a800 msgr2=0x7fefe406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 -- 172.21.15.125:0/1670599165 shutdown_connections 2024-01-16T15:05:18.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.169+0000 7fefec400640 1 -- 172.21.15.125:0/1670599165 wait complete. 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.171+0000 7fefec400640 1 Processor -- start 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.171+0000 7fefec400640 1 -- start start 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fefe40ae5a0 0x7fefe41f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe40b3ae0 0x7fefe414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4157320 con 0x7fefe40cf290 2024-01-16T15:05:18.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4157600 con 0x7fefe40b3ae0 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefec400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe41578e0 con 0x7fefe40ae5a0 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefea400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefea400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59060/0 (socket says 172.21.15.125:59060) 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.172+0000 7fefea400640 1 -- 172.21.15.125:0/1076716826 learned_addr learned my addr 172.21.15.125:0/1076716826 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefe9a00640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fefe40ae5a0 0x7fefe41f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefe9000640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe40b3ae0 0x7fefe414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 -- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fefe40ae5a0 msgr2=0x7fefe41f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fefe40ae5a0 0x7fefe41f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 -- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe40b3ae0 msgr2=0x7fefe414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe40b3ae0 0x7fefe414c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 -- 172.21.15.125:0/1076716826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc009a70 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefea400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009e80 tx=0x7fefdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc03d600 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc03d780 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc03db30 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.173+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe4157bc0 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.174+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe414cad0 con 0x7fefe40cf290 2024-01-16T15:05:18.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.175+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe40675d0 con 0x7fefe40cf290 2024-01-16T15:05:18.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.175+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fefdc03b040 con 0x7fefe40cf290 2024-01-16T15:05:18.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.176+0000 7fefe1600640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 0x7fefbc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.176+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefdc0aab10 con 0x7fefe40cf290 2024-01-16T15:05:18.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.177+0000 7fefe9a00640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 0x7fefbc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.177+0000 7fefe9a00640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 0x7fefbc06dff0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fefd40011e0 tx=0x7fefd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.181+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefdc077200 con 0x7fefe40cf290 2024-01-16T15:05:18.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.372+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fefe4161ae0 con 0x7fefe40cf290 2024-01-16T15:05:18.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.373+0000 7fefe1600640 1 -- 172.21.15.125:0/1076716826 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fefdc009450 con 0x7fefe40cf290 2024-01-16T15:05:18.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 msgr2=0x7fefbc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 0x7fefbc06dff0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fefd40011e0 tx=0x7fefd4009040 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 msgr2=0x7fefe40afca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009e80 tx=0x7fefdc004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 shutdown_connections 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fefbc06bb30 0x7fefbc06dff0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefe40cf290 0x7fefe40afca0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe40b3ae0 0x7fefe414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 --2- 172.21.15.125:0/1076716826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fefe40ae5a0 0x7fefe41f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.375+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 >> 172.21.15.125:0/1076716826 conn(0x7fefe406a800 msgr2=0x7fefe415d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.376+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 shutdown_connections 2024-01-16T15:05:18.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.376+0000 7fefec400640 1 -- 172.21.15.125:0/1076716826 wait complete. 2024-01-16T15:05:18.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:05:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.607+0000 7f694a000640 1 Processor -- start 2024-01-16T15:05:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.607+0000 7f694a000640 1 -- start start 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.608+0000 7f694a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6944072230 0x7f69440b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.608+0000 7f694a000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944071190 con 0x7f694415da10 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.608+0000 7f694a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944071470 con 0x7f6944150760 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.608+0000 7f694a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6944071750 con 0x7f6944072230 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6942a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f694415da10 0x7f69440b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33054/0 (socket says 172.21.15.125:33054) 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6942a00640 1 -- 172.21.15.125:0/2702714749 learned_addr learned my addr 172.21.15.125:0/2702714749 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6943e00640 1 --2- 172.21.15.125:0/2702714749 >> v2:172.21.15.137:3300/0 conn(0x7f6944072230 0x7f69440b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034150665 0 0) 0x7f6944071190 con 0x7f694415da10 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6924003520 con 0x7f694415da10 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.609+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979322381 0 0) 0x7f6924003520 con 0x7f694415da10 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 >> v2:172.21.15.137:3300/0 conn(0x7f6944072230 msgr2=0x7f69440b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 --2- 172.21.15.125:0/2702714749 >> v2:172.21.15.137:3300/0 conn(0x7f6944072230 0x7f69440b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 >> v1:172.21.15.137:6789/0 conn(0x7f6944150760 legacy=0x7f694415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944071a30 con 0x7f694415da10 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f692c001670 con 0x7f694415da10 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 >> v1:172.21.15.125:6789/0 conn(0x7f694415da10 legacy=0x7f69440b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69240039c0 0x7f6924003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f6924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6924005180 0x7f692400fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944071750 con 0x7f69240047c0 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924010460 con 0x7f6924005180 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6924010620 con 0x7f69240039c0 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f6924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6943400640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69240039c0 0x7f6924003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 -- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69240039c0 msgr2=0x7f6924003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6943e00640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6924005180 0x7f692400fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69240039c0 0x7f6924003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 -- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6924005180 msgr2=0x7f692400fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6924005180 0x7f692400fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6942a00640 1 -- 172.21.15.125:0/2702714749 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924003520 con 0x7f69240047c0 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.610+0000 7f6943e00640 1 -- 172.21.15.125:0/2702714749 reap_dead start 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.611+0000 7f6943400640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69240039c0 0x7f6924003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.611+0000 7f6942a00640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f6924004c40 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f692c00c0e0 tx=0x7f69240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c52e9f1dde40ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c005ad0 con 0x7f69240047c0 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c005c30 con 0x7f69240047c0 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f6942000640 1 -- 172.21.15.125:0/2702714749 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c02c7d0 con 0x7f69240047c0 2024-01-16T15:05:18.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f694a000640 1 -- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 msgr2=0x7f6924004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f694a000640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f6924004c40 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f692c00c0e0 tx=0x7f69240057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f694a000640 1 -- 172.21.15.125:0/2702714749 shutdown_connections 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f694a000640 1 --2- 172.21.15.125:0/2702714749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f6924004c40 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.612+0000 7f694a000640 1 -- 172.21.15.125:0/2702714749 >> 172.21.15.125:0/2702714749 conn(0x7f694406a800 msgr2=0x7f694406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.613+0000 7f694a000640 1 -- 172.21.15.125:0/2702714749 shutdown_connections 2024-01-16T15:05:18.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.613+0000 7f694a000640 1 -- 172.21.15.125:0/2702714749 wait complete. 2024-01-16T15:05:18.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.614+0000 7f694a000640 1 Processor -- start 2024-01-16T15:05:18.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.614+0000 7f694a000640 1 -- start start 2024-01-16T15:05:18.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 0x7f694414a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 0x7f69440cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694414c6f0 con 0x7f69240047c0 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694414a540 con 0x7f694414b1a0 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.615+0000 7f694a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694415b840 con 0x7f694415c600 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59084/0 (socket says 172.21.15.125:59084) 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 -- 172.21.15.125:0/3119429145 learned_addr learned my addr 172.21.15.125:0/3119429145 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6942a00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 0x7f694414a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943e00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 0x7f69440cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 -- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 msgr2=0x7f69440cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 0x7f69440cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 -- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 msgr2=0x7f694414a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 0x7f694414a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 -- 172.21.15.125:0/3119429145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694415bb20 con 0x7f69240047c0 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6942a00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 0x7f694414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943e00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 0x7f69440cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.616+0000 7f6943400640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f693400d7a0 tx=0x7f693400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.617+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934006d50 con 0x7f69240047c0 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.617+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934004950 con 0x7f69240047c0 2024-01-16T15:05:18.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.617+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934004d00 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.617+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694415be00 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.617+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69440ceaa0 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.618+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694406cd00 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.619+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f693400b8c0 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.620+0000 7f693b400640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 0x7f691c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.620+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6934088ae0 con 0x7f69240047c0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.620+0000 7f6942a00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 0x7f691c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.621+0000 7f6942a00640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 0x7f691c06dff0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f692c034000 tx=0x7f692c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.625+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69340551d0 con 0x7f69240047c0 2024-01-16T15:05:18.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.843+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f6944070570 con 0x7f691c06bb30 2024-01-16T15:05:18.828 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:05:18.850+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:18.828 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:05:18.850+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:18.828 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:05:18.850+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:18.828 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:05:18.850+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:18.829 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:05:18.850+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.885+0000 7f693b400640 1 -- 172.21.15.125:0/3119429145 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6944070570 con 0x7f691c06bb30 2024-01-16T15:05:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.887+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 msgr2=0x7f691c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.887+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 0x7f691c06dff0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f692c034000 tx=0x7f692c009030 comp rx=0 tx=0).stop 2024-01-16T15:05:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.887+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 msgr2=0x7f694414a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.887+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f693400d7a0 tx=0x7f693400dc60 comp rx=0 tx=0).stop 2024-01-16T15:05:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 shutdown_connections 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694415c600 0x7f69440cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694414b1a0 0x7f694414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f691c06bb30 0x7f691c06dff0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 --2- 172.21.15.125:0/3119429145 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69240047c0 0x7f694414a7b0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 >> 172.21.15.125:0/3119429145 conn(0x7f694406a800 msgr2=0x7f694414cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 shutdown_connections 2024-01-16T15:05:18.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:18.888+0000 7f694a000640 1 -- 172.21.15.125:0/3119429145 wait complete. 2024-01-16T15:05:18.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:19.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.098+0000 7fc4f4c00640 1 Processor -- start 2024-01-16T15:05:19.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.098+0000 7fc4f4c00640 1 -- start start 2024-01-16T15:05:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.099+0000 7fc4f4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4f013df80 0x7fc4f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.099+0000 7fc4f4c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f006c470 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.099+0000 7fc4f4c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f006c750 con 0x7fc4f013e360 2024-01-16T15:05:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.099+0000 7fc4f4c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f006ca30 con 0x7fc4f013df80 2024-01-16T15:05:19.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4f013df80 0x7fc4f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4f013df80 0x7fc4f015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59110/0 (socket says 172.21.15.125:59110) 2024-01-16T15:05:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ee800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc4f00cf9a0 0x7fc4f00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33058/0 (socket says 172.21.15.125:33058) 2024-01-16T15:05:19.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 -- 172.21.15.125:0/3806379571 learned_addr learned my addr 172.21.15.125:0/3806379571 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 -- 172.21.15.125:0/3806379571 >> v1:172.21.15.137:6789/0 conn(0x7fc4f013e360 legacy=0x7fc4f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 -- 172.21.15.125:0/3806379571 >> v1:172.21.15.125:6789/0 conn(0x7fc4f00cf9a0 legacy=0x7fc4f00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ede00640 1 -- 172.21.15.125:0/3806379571 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f006cd10 con 0x7fc4f013df80 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.100+0000 7fc4ed400640 1 -- 172.21.15.125:0/3806379571 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229492273 0 0) 0x7fc4f006c470 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4ede00640 1 --2- 172.21.15.125:0/3806379571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f013df80 0x7fc4f015bee0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009870 tx=0x7fc4e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3a4ea63f1cbde5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4ed400640 1 -- 172.21.15.125:0/3806379571 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0030860 con 0x7fc4f013df80 2024-01-16T15:05:19.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4ed400640 1 -- 172.21.15.125:0/3806379571 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e00309c0 con 0x7fc4f013df80 2024-01-16T15:05:19.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4ed400640 1 -- 172.21.15.125:0/3806379571 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0030d50 con 0x7fc4f013df80 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4f4c00640 1 -- 172.21.15.125:0/3806379571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f013df80 msgr2=0x7fc4f015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.102+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/3806379571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f013df80 0x7fc4f015bee0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009870 tx=0x7fc4e002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.103+0000 7fc4f4c00640 1 -- 172.21.15.125:0/3806379571 shutdown_connections 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.103+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/3806379571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f013df80 0x7fc4f015bee0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.103+0000 7fc4f4c00640 1 -- 172.21.15.125:0/3806379571 >> 172.21.15.125:0/3806379571 conn(0x7fc4f006a800 msgr2=0x7fc4f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.103+0000 7fc4f4c00640 1 -- 172.21.15.125:0/3806379571 shutdown_connections 2024-01-16T15:05:19.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.103+0000 7fc4f4c00640 1 -- 172.21.15.125:0/3806379571 wait complete. 2024-01-16T15:05:19.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.104+0000 7fc4f4c00640 1 Processor -- start 2024-01-16T15:05:19.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.104+0000 7fc4f4c00640 1 -- start start 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4f013df80 0x7fc4f01f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4f013e360 0x7fc4f00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0152160 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f01f3a50 con 0x7fc4f013df80 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.105+0000 7fc4f4c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f01f3cf0 con 0x7fc4f013e360 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59116/0 (socket says 172.21.15.125:59116) 2024-01-16T15:05:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 -- 172.21.15.125:0/1879545101 learned_addr learned my addr 172.21.15.125:0/1879545101 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ee800640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4f013e360 0x7fc4f00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4e5200640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4f013df80 0x7fc4f01f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 -- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4f013e360 msgr2=0x7fc4f00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4f013e360 0x7fc4f00b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 -- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4f013df80 msgr2=0x7fc4f01f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4f013df80 0x7fc4f01f3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 -- 172.21.15.125:0/1879545101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00094d0 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.106+0000 7fc4ede00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009870 tx=0x7fc4e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.107+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0039d60 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.107+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e0043400 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.107+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00437b0 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.107+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f01f3fd0 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.107+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f0152370 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.108+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0070f90 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.109+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 22) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc4e0043910 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.110+0000 7fc4e6a00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 0x7fc4c806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.110+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4e003e070 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.110+0000 7fc4e5200640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 0x7fc4c806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.111+0000 7fc4e5200640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 0x7fc4c806dff0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc4f00b44f0 tx=0x7fc4d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.114+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4e0077fc0 con 0x7fc4f00cf9a0 2024-01-16T15:05:19.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.317+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4f006abe0 con 0x7fc4c806bb30 2024-01-16T15:05:19.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.325+0000 7fc4e6a00640 1 -- 172.21.15.125:0/1879545101 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fc4f006abe0 con 0x7fc4c806bb30 2024-01-16T15:05:19.304 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.327+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 msgr2=0x7fc4c806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.327+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 0x7fc4c806dff0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc4f00b44f0 tx=0x7fc4d8009040 comp rx=0 tx=0).stop 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.327+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 msgr2=0x7fc4f00a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.327+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009870 tx=0x7fc4e0004730 comp rx=0 tx=0).stop 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 shutdown_connections 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4f013e360 0x7fc4f00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4f013df80 0x7fc4f01f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc4c806bb30 0x7fc4c806dff0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 --2- 172.21.15.125:0/1879545101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4f00cf9a0 0x7fc4f00a9570 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 >> 172.21.15.125:0/1879545101 conn(0x7fc4f006a800 msgr2=0x7fc4f0070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 shutdown_connections 2024-01-16T15:05:19.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:19.328+0000 7fc4f4c00640 1 -- 172.21.15.125:0/1879545101 wait complete. 2024-01-16T15:05:19.324 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3 2024-01-16T15:05:19.324 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:05:19.324 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:19.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:19.371 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:05:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:05:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.4687 s, 79.7 MB/s 2024-01-16T15:05:32.952 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:05:32.952 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:33.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:33.003 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:38.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:38.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:44.333 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:05:44.333 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:05:44.333 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2232 s, 95.7 MB/s 2024-01-16T15:05:44.335 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:05:44.335 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:44.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:44.383 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:05:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:05:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.5369 s, 93.1 MB/s 2024-01-16T15:05:56.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-16T15:05:56.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.260+0000 7fb620200640 1 Processor -- start 2024-01-16T15:05:56.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.260+0000 7fb620200640 1 -- start start 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb620200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 0x7fb6180cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb620200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb6180ce070 0x7fb618164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb620200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6180d00a0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb620200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6180b4010 con 0x7fb6180ce070 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb620200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180b42f0 con 0x7fb61815b2a0 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb61ce00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb6180cfaf0 0x7fb6180cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56072/0 (socket says 172.21.15.125:56072) 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb61ce00640 1 -- 172.21.15.125:0/644245604 learned_addr learned my addr 172.21.15.125:0/644245604 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb61e200640 1 --2- 172.21.15.125:0/644245604 >> v2:172.21.15.125:3300/0 conn(0x7fb6180ce070 0x7fb618164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.262+0000 7fb61d800640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 0x7fb6180cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb617e00640 1 -- 172.21.15.125:0/644245604 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572839491 0 0) 0x7fb6180d00a0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb61e200640 1 -- 172.21.15.125:0/644245604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 msgr2=0x7fb6180cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb61e200640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 0x7fb6180cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb61e200640 1 -- 172.21.15.125:0/644245604 >> v1:172.21.15.125:6789/0 conn(0x7fb6180cfaf0 legacy=0x7fb6180cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb61e200640 1 -- 172.21.15.125:0/644245604 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180b45d0 con 0x7fb6180ce070 2024-01-16T15:05:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.263+0000 7fb61d800640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 0x7fb6180cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.264+0000 7fb61e200640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180ce070 0x7fb618164f40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b118ac8ed171d12f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.264+0000 7fb617e00640 1 -- 172.21.15.125:0/644245604 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61002cc50 con 0x7fb6180ce070 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.264+0000 7fb617e00640 1 -- 172.21.15.125:0/644245604 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61002cdb0 con 0x7fb6180ce070 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.264+0000 7fb617e00640 1 -- 172.21.15.125:0/644245604 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610034660 con 0x7fb6180ce070 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 -- 172.21.15.125:0/644245604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180ce070 msgr2=0x7fb618164f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180ce070 0x7fb618164f40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61002bea0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 -- 172.21.15.125:0/644245604 shutdown_connections 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180ce070 0x7fb618164f40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 --2- 172.21.15.125:0/644245604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb61815b2a0 0x7fb6180cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 -- 172.21.15.125:0/644245604 >> 172.21.15.125:0/644245604 conn(0x7fb6180d1020 msgr2=0x7fb6180d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 -- 172.21.15.125:0/644245604 shutdown_connections 2024-01-16T15:05:56.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.265+0000 7fb620200640 1 -- 172.21.15.125:0/644245604 wait complete. 2024-01-16T15:05:56.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.267+0000 7fb620200640 1 Processor -- start 2024-01-16T15:05:56.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.267+0000 7fb620200640 1 -- start start 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6180ce070 0x7fb618150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 0x7fb61815dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61815cb10 con 0x7fb6180cfaf0 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61815cdf0 con 0x7fb61815b2a0 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb620200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61815d0d0 con 0x7fb6180ce070 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb61ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb61ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42584/0 (socket says 172.21.15.125:42584) 2024-01-16T15:05:56.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb61ce00640 1 -- 172.21.15.125:0/821854645 learned_addr learned my addr 172.21.15.125:0/821854645 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb61d800640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6180ce070 0x7fb618150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.268+0000 7fb61e200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 0x7fb61815dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 -- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6180ce070 msgr2=0x7fb618150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6180ce070 0x7fb618150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 -- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 msgr2=0x7fb61815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 0x7fb61815dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 -- 172.21.15.125:0/821854645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610009a70 con 0x7fb6180cfaf0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61e200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 0x7fb61815dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb61ce00640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb60000c4d0 tx=0x7fb60000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6000074e0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb600007660 con 0x7fb6180cfaf0 2024-01-16T15:05:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61815d3b0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600004a20 con 0x7fb6180cfaf0 2024-01-16T15:05:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.269+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6181508c0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.271+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6180675d0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.271+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 23) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb600004b80 con 0x7fb6180cfaf0 2024-01-16T15:05:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.273+0000 7fb615600640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 0x7fb5f006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.273+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fb6000879e0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.273+0000 7fb61d800640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 0x7fb5f006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.274+0000 7fb61d800640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 0x7fb5f006e080 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb608006fd0 tx=0x7fb608008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.278+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6000542a0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.467+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fb61806ec60 con 0x7fb6180cfaf0 2024-01-16T15:05:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.473+0000 7fb615600640 1 -- 172.21.15.125:0/821854645 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb6000578c0 con 0x7fb6180cfaf0 2024-01-16T15:05:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 msgr2=0x7fb5f006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 0x7fb5f006e080 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb608006fd0 tx=0x7fb608008040 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 msgr2=0x7fb618151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb60000c4d0 tx=0x7fb60000c990 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 shutdown_connections 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb61815b2a0 0x7fb61815dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb5f006bbc0 0x7fb5f006e080 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6180cfaf0 0x7fb618151690 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 --2- 172.21.15.125:0/821854645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6180ce070 0x7fb618150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.476+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 >> 172.21.15.125:0/821854645 conn(0x7fb6180d1020 msgr2=0x7fb6180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.477+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 shutdown_connections 2024-01-16T15:05:56.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.477+0000 7fb620200640 1 -- 172.21.15.125:0/821854645 wait complete. 2024-01-16T15:05:56.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:05:56.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.712+0000 7f5e2d600640 1 Processor -- start 2024-01-16T15:05:56.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.712+0000 7f5e2d600640 1 -- start start 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813df80 0x7f5e2815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e2d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e2813e360 0x7f5e280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e2d600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2806c470 con 0x7f5e2815f9c0 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e2d600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2806c750 con 0x7f5e2813e360 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e2d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2806ca30 con 0x7f5e2813df80 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e26000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e2813e360 0x7f5e280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e27400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5e2815f9c0 0x7f5e281408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41804/0 (socket says 172.21.15.125:41804) 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e27400640 1 -- 172.21.15.125:0/3487125391 learned_addr learned my addr 172.21.15.125:0/3487125391 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.714+0000 7f5e26a00640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813df80 0x7f5e2815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.715+0000 7f5e26000640 1 -- 172.21.15.125:0/3487125391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813df80 msgr2=0x7f5e2815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:56.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.715+0000 7f5e26000640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813df80 0x7f5e2815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.715+0000 7f5e26000640 1 -- 172.21.15.125:0/3487125391 >> v1:172.21.15.125:6789/0 conn(0x7f5e2815f9c0 legacy=0x7f5e281408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.715+0000 7f5e26000640 1 -- 172.21.15.125:0/3487125391 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2806cd10 con 0x7f5e2813e360 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.715+0000 7f5e25600640 1 -- 172.21.15.125:0/3487125391 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393809399 0 0) 0x7f5e2806c470 con 0x7f5e2815f9c0 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.716+0000 7f5e26000640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2813e360 0x7f5e280cf290 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5e100097a0 tx=0x7f5e1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b06b4c1de703583 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.716+0000 7f5e25600640 1 -- 172.21.15.125:0/3487125391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10031830 con 0x7f5e2813e360 2024-01-16T15:05:56.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.716+0000 7f5e25600640 1 -- 172.21.15.125:0/3487125391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10031990 con 0x7f5e2813e360 2024-01-16T15:05:56.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.716+0000 7f5e25600640 1 -- 172.21.15.125:0/3487125391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10031d20 con 0x7f5e2813e360 2024-01-16T15:05:56.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 -- 172.21.15.125:0/3487125391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2813e360 msgr2=0x7f5e280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2813e360 0x7f5e280cf290 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5e100097a0 tx=0x7f5e1002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 -- 172.21.15.125:0/3487125391 shutdown_connections 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2813e360 0x7f5e280cf290 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3487125391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813df80 0x7f5e2815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 -- 172.21.15.125:0/3487125391 >> 172.21.15.125:0/3487125391 conn(0x7f5e2806a800 msgr2=0x7f5e2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 -- 172.21.15.125:0/3487125391 shutdown_connections 2024-01-16T15:05:56.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.717+0000 7f5e2d600640 1 -- 172.21.15.125:0/3487125391 wait complete. 2024-01-16T15:05:56.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.719+0000 7f5e2d600640 1 Processor -- start 2024-01-16T15:05:56.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.719+0000 7f5e2d600640 1 -- start start 2024-01-16T15:05:56.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e2813df80 0x7f5e2814ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813e360 0x7f5e2814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28130580 con 0x7f5e2815f9c0 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e281302a0 con 0x7f5e2813df80 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e2d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2814af80 con 0x7f5e2813e360 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e27400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e27400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49768/0 (socket says 172.21.15.125:49768) 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.720+0000 7f5e27400640 1 -- 172.21.15.125:0/3293751060 learned_addr learned my addr 172.21.15.125:0/3293751060 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e26000640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813e360 0x7f5e2814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e26a00640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e2813df80 0x7f5e2814ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 -- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813e360 msgr2=0x7f5e2814f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813e360 0x7f5e2814f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 -- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e2813df80 msgr2=0x7f5e2814ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e2813df80 0x7f5e2814ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 -- 172.21.15.125:0/3293751060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10009400 con 0x7f5e2815f9c0 2024-01-16T15:05:56.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e27400640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000c7e0 tx=0x7f5e2000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000a640 con 0x7f5e2815f9c0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2000a7c0 con 0x7f5e2815f9c0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000ab70 con 0x7f5e2815f9c0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.721+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2814b260 con 0x7f5e2815f9c0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.722+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e28152930 con 0x7f5e2815f9c0 2024-01-16T15:05:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.723+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2814ade0 con 0x7f5e2815f9c0 2024-01-16T15:05:56.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.724+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 23) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5e2000acd0 con 0x7f5e2815f9c0 2024-01-16T15:05:56.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.725+0000 7f5e1ea00640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 0x7f5e0006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:56.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.725+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f5e20087a70 con 0x7f5e2815f9c0 2024-01-16T15:05:56.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.725+0000 7f5e26a00640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 0x7f5e0006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:56.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.726+0000 7f5e26a00640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 0x7f5e0006e080 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5e18006fd0 tx=0x7f5e18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:56.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.729+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e2814ade0 con 0x7f5e2815f9c0 2024-01-16T15:05:56.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.949+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e280ce850 con 0x7f5e0006bbc0 2024-01-16T15:05:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.974+0000 7f5e1ea00640 1 -- 172.21.15.125:0/3293751060 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5e1800f860 con 0x7f5e0006bbc0 2024-01-16T15:05:56.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 msgr2=0x7f5e0006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 0x7f5e0006e080 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5e18006fd0 tx=0x7f5e18008040 comp rx=0 tx=0).stop 2024-01-16T15:05:56.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 msgr2=0x7f5e2814fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000c7e0 tx=0x7f5e2000cca0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 shutdown_connections 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e0006bbc0 0x7f5e0006e080 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2815f9c0 0x7f5e2814fc10 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e2813e360 0x7f5e2814f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 --2- 172.21.15.125:0/3293751060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e2813df80 0x7f5e2814ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.977+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 >> 172.21.15.125:0/3293751060 conn(0x7f5e2806a800 msgr2=0x7f5e28153d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.978+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 shutdown_connections 2024-01-16T15:05:56.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:56.978+0000 7f5e2d600640 1 -- 172.21.15.125:0/3293751060 wait complete. 2024-01-16T15:05:56.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:57.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.201+0000 7f689b600640 1 Processor -- start 2024-01-16T15:05:57.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.201+0000 7f689b600640 1 -- start start 2024-01-16T15:05:57.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f689b600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f689413df80 0x7f689415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f689b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68940cf9a0 0x7f68940cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f689b600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689406c470 con 0x7f689413e360 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f689b600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689406c750 con 0x7f689413df80 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f689b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689406ca30 con 0x7f68940cf9a0 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f6898c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f689413df80 0x7f689415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f6893e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f689413e360 0x7f68940cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41812/0 (socket says 172.21.15.125:41812) 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f6893e00640 1 -- 172.21.15.125:0/907126409 learned_addr learned my addr 172.21.15.125:0/907126409 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.203+0000 7f6899600640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68940cf9a0 0x7f68940cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.204+0000 7f6898c00640 1 -- 172.21.15.125:0/907126409 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68940cf9a0 msgr2=0x7f68940cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.204+0000 7f6898c00640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68940cf9a0 0x7f68940cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.204+0000 7f6898c00640 1 -- 172.21.15.125:0/907126409 >> v1:172.21.15.125:6789/0 conn(0x7f689413e360 legacy=0x7f68940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.204+0000 7f6898c00640 1 -- 172.21.15.125:0/907126409 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689406cd10 con 0x7f689413df80 2024-01-16T15:05:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.204+0000 7f6893400640 1 -- 172.21.15.125:0/907126409 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917130637 0 0) 0x7f689406c470 con 0x7f689413e360 2024-01-16T15:05:57.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.205+0000 7f6898c00640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f689413df80 0x7f689415bee0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6884009d90 tx=0x7f688402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dc0b216b4db442 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:57.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.205+0000 7f6893400640 1 -- 172.21.15.125:0/907126409 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688402cc50 con 0x7f689413df80 2024-01-16T15:05:57.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.205+0000 7f6893400640 1 -- 172.21.15.125:0/907126409 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688402cdb0 con 0x7f689413df80 2024-01-16T15:05:57.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.205+0000 7f6893400640 1 -- 172.21.15.125:0/907126409 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884034660 con 0x7f689413df80 2024-01-16T15:05:57.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 -- 172.21.15.125:0/907126409 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f689413df80 msgr2=0x7f689415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f689413df80 0x7f689415bee0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6884009d90 tx=0x7f688402bea0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 -- 172.21.15.125:0/907126409 shutdown_connections 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68940cf9a0 0x7f68940cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 --2- 172.21.15.125:0/907126409 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f689413df80 0x7f689415bee0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 -- 172.21.15.125:0/907126409 >> 172.21.15.125:0/907126409 conn(0x7f689406a800 msgr2=0x7f689406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 -- 172.21.15.125:0/907126409 shutdown_connections 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.206+0000 7f689b600640 1 -- 172.21.15.125:0/907126409 wait complete. 2024-01-16T15:05:57.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.208+0000 7f689b600640 1 Processor -- start 2024-01-16T15:05:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.208+0000 7f689b600640 1 -- start start 2024-01-16T15:05:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f689413df80 0x7f6894150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f689413e360 0x7f689414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689414f8f0 con 0x7f68940cf9a0 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689414c460 con 0x7f689413e360 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f689b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689414c740 con 0x7f689413df80 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49776/0 (socket says 172.21.15.125:49776) 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 -- 172.21.15.125:0/2218045333 learned_addr learned my addr 172.21.15.125:0/2218045333 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6893e00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f689413df80 0x7f6894150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6899600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f689413e360 0x7f689414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 -- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f689413df80 msgr2=0x7f6894150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f689413df80 0x7f6894150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 -- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f689413e360 msgr2=0x7f689414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f689413e360 0x7f689414b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.209+0000 7f6898c00640 1 -- 172.21.15.125:0/2218045333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884009a70 con 0x7f68940cf9a0 2024-01-16T15:05:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f6898c00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6884037040 tx=0x7f6884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884040650 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68840407b0 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884040b60 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689414acc0 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.210+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689413dbd0 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.211+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689414ab00 con 0x7f68940cf9a0 2024-01-16T15:05:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.212+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 23) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6884040cc0 con 0x7f68940cf9a0 2024-01-16T15:05:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.213+0000 7f6890c00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 0x7f686c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.213+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f68840ab2f0 con 0x7f68940cf9a0 2024-01-16T15:05:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.213+0000 7f6893e00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 0x7f686c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:57.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.214+0000 7f6893e00640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 0x7f686c06e080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f689415b070 tx=0x7f687c007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.218+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6884077c10 con 0x7f68940cf9a0 2024-01-16T15:05:57.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.435+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f689415f9c0 con 0x7f686c06bbc0 2024-01-16T15:05:57.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.447+0000 7f6890c00640 1 -- 172.21.15.125:0/2218045333 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f687c0076a0 con 0x7f686c06bbc0 2024-01-16T15:05:57.426 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3 2024-01-16T15:05:57.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 msgr2=0x7f686c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 0x7f686c06e080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f689415b070 tx=0x7f687c007b50 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 msgr2=0x7f689413d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6884037040 tx=0x7f6884004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 shutdown_connections 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f689413e360 0x7f689414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f689413df80 0x7f6894150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686c06bbc0 0x7f686c06e080 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 --2- 172.21.15.125:0/2218045333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68940cf9a0 0x7f689413d410 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 >> 172.21.15.125:0/2218045333 conn(0x7f689406a800 msgr2=0x7f689415cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 shutdown_connections 2024-01-16T15:05:57.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:57.449+0000 7f689b600640 1 -- 172.21.15.125:0/2218045333 wait complete. 2024-01-16T15:05:57.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:57.446 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3) 2024-01-16T15:05:57.495 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3 2024-01-16T15:05:57.495 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:05:57.496 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:57.497 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:05:59.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:59.498 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3) 2024-01-16T15:05:59.562 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/05be8c30-d1bb-495c-b1c4-0db49be91fe3 2024-01-16T15:05:59.562 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:05:59.563 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:05:59.564 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:05:59.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96911 2024-01-16T15:05:59.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.805+0000 7f7fb0c00640 1 Processor -- start 2024-01-16T15:05:59.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.805+0000 7f7fb0c00640 1 -- start start 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.806+0000 7f7fb0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.806+0000 7f7fb0c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fac15da10 0x7f7fac0b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.806+0000 7f7fb0c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac071190 con 0x7f7fac0cf290 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7fb0c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac072610 con 0x7f7fac15da10 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7fb0c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac071470 con 0x7f7fac072230 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fac15da10 0x7f7fac0b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fac15da10 0x7f7fac0b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49780/0 (socket says 172.21.15.125:49780) 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 -- 172.21.15.125:0/1199045784 learned_addr learned my addr 172.21.15.125:0/1199045784 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7fa9e00640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 -- 172.21.15.125:0/1199045784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 msgr2=0x7f7fac150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 -- 172.21.15.125:0/1199045784 >> v1:172.21.15.137:6789/0 conn(0x7f7fac0cf290 legacy=0x7f7fac15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.807+0000 7f7faa800640 1 -- 172.21.15.125:0/1199045784 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac071750 con 0x7f7fac15da10 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.808+0000 7f7faa800640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac0b2bd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00bb10 tx=0x7f7f9c02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382fe111be040223 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.809+0000 7f7fa8a00640 1 -- 172.21.15.125:0/1199045784 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c02ee70 con 0x7f7fac15da10 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.809+0000 7f7fa8a00640 1 -- 172.21.15.125:0/1199045784 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c036440 con 0x7f7fac15da10 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.809+0000 7f7fa8a00640 1 -- 172.21.15.125:0/1199045784 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0367d0 con 0x7f7fac15da10 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.809+0000 7f7fb0c00640 1 -- 172.21.15.125:0/1199045784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 msgr2=0x7f7fac0b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.809+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac0b2bd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00bb10 tx=0x7f7f9c02e0c0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 -- 172.21.15.125:0/1199045784 shutdown_connections 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac0b2bd0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/1199045784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 -- 172.21.15.125:0/1199045784 >> 172.21.15.125:0/1199045784 conn(0x7f7fac06a800 msgr2=0x7f7fac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 -- 172.21.15.125:0/1199045784 shutdown_connections 2024-01-16T15:05:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.810+0000 7f7fb0c00640 1 -- 172.21.15.125:0/1199045784 wait complete. 2024-01-16T15:05:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.811+0000 7f7fb0c00640 1 Processor -- start 2024-01-16T15:05:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 -- start start 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fac0cf290 0x7f7fac0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cc1a0 con 0x7f7fac15da10 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cc480 con 0x7f7fac0cf290 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.812+0000 7f7fb0c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cc760 con 0x7f7fac072230 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49784/0 (socket says 172.21.15.125:49784) 2024-01-16T15:05:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 -- 172.21.15.125:0/3081006770 learned_addr learned my addr 172.21.15.125:0/3081006770 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 -- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 msgr2=0x7f7fac14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 -- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fac0cf290 msgr2=0x7f7fac0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fac0cf290 0x7f7fac0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 -- 172.21.15.125:0/3081006770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c00b770 con 0x7f7fac15da10 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.813+0000 7f7faa800640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00b2c0 tx=0x7f7f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.814+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c034990 con 0x7f7fac15da10 2024-01-16T15:05:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.814+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c034b10 con 0x7f7fac15da10 2024-01-16T15:05:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.814+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac0cca40 con 0x7f7fac15da10 2024-01-16T15:05:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.814+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c03f440 con 0x7f7fac15da10 2024-01-16T15:05:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.814+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac0cbe00 con 0x7f7fac15da10 2024-01-16T15:05:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.815+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac0ccf60 con 0x7f7fac15da10 2024-01-16T15:05:59.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.816+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 23) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7f9c03d070 con 0x7f7fac15da10 2024-01-16T15:05:59.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.817+0000 7f7fa2000640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 0x7f7f8406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.817+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7f9c0aac50 con 0x7f7fac15da10 2024-01-16T15:05:59.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.817+0000 7f7fa9e00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 0x7f7f8406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.818+0000 7f7fa9e00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 0x7f7f8406e080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4007e60 tx=0x7f7fa4007390 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:05:59.821+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fac0ccf60 con 0x7f7fac15da10 2024-01-16T15:06:00.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.049+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96911", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fac071d10 con 0x7f7f8406bbc0 2024-01-16T15:06:00.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.183+0000 7f7fa2000640 1 -- 172.21.15.125:0/3081006770 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7fa4007a50 con 0x7f7f8406bbc0 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 msgr2=0x7f7f8406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 0x7f7f8406e080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4007e60 tx=0x7f7fa4007390 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 msgr2=0x7f7fac15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00b2c0 tx=0x7f7f9c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 shutdown_connections 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f8406bbc0 0x7f7f8406e080 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fac15da10 0x7f7fac15bca0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fac0cf290 0x7f7fac0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 --2- 172.21.15.125:0/3081006770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fac072230 0x7f7fac14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.186+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 >> 172.21.15.125:0/3081006770 conn(0x7f7fac06a800 msgr2=0x7f7fac14f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.187+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 shutdown_connections 2024-01-16T15:06:00.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.187+0000 7f7fb0c00640 1 -- 172.21.15.125:0/3081006770 wait complete. 2024-01-16T15:06:00.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.190+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.190+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.190+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.190+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.190+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96912 2024-01-16T15:06:00.193 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.212+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.193 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.212+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.193 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.212+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.193 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.212+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.193 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:00.212+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.400+0000 7f07d4a00640 1 Processor -- start 2024-01-16T15:06:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.400+0000 7f07d4a00640 1 -- start start 2024-01-16T15:06:00.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.401+0000 7f07d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc143ae0 0x7f07cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.401+0000 7f07d4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07cc0cf7d0 0x7f07cc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.401+0000 7f07d4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f07cc15e1e0 0x7f07cc15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d4a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc0cfc30 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc0b1ee0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0b2780 con 0x7f07cc143ae0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07cc0cf7d0 0x7f07cc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07cc0cf7d0 0x7f07cc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49824/0 (socket says 172.21.15.125:49824) 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 -- 172.21.15.125:0/3654960279 learned_addr learned my addr 172.21.15.125:0/3654960279 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d2000640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc143ae0 0x7f07cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d2a00640 1 --2- 172.21.15.125:0/3654960279 >> v2:172.21.15.137:3300/0 conn(0x7f07cc15e1e0 0x7f07cc15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 -- 172.21.15.125:0/3654960279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc143ae0 msgr2=0x7f07cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc143ae0 0x7f07cc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 -- 172.21.15.125:0/3654960279 >> v2:172.21.15.137:3300/0 conn(0x7f07cc15e1e0 msgr2=0x7f07cc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:00.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 --2- 172.21.15.125:0/3654960279 >> v2:172.21.15.137:3300/0 conn(0x7f07cc15e1e0 0x7f07cc15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.402+0000 7f07d1600640 1 -- 172.21.15.125:0/3654960279 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc15e990 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d1600640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc0cf7d0 0x7f07cc15dca0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f07cc072160 tx=0x7f07b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7fdc207e9c0ce5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d0c00640 1 -- 172.21.15.125:0/3654960279 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4031870 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d0c00640 1 -- 172.21.15.125:0/3654960279 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b40319d0 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d0c00640 1 -- 172.21.15.125:0/3654960279 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4031d60 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d4a00640 1 -- 172.21.15.125:0/3654960279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc0cf7d0 msgr2=0x7f07cc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.404+0000 7f07d4a00640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc0cf7d0 0x7f07cc15dca0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f07cc072160 tx=0x7f07b402b910 comp rx=0 tx=0).stop 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 -- 172.21.15.125:0/3654960279 shutdown_connections 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 --2- 172.21.15.125:0/3654960279 >> v2:172.21.15.137:3300/0 conn(0x7f07cc15e1e0 0x7f07cc15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc0cf7d0 0x7f07cc15dca0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 --2- 172.21.15.125:0/3654960279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc143ae0 0x7f07cc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 -- 172.21.15.125:0/3654960279 >> 172.21.15.125:0/3654960279 conn(0x7f07cc06a800 msgr2=0x7f07cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:00.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 -- 172.21.15.125:0/3654960279 shutdown_connections 2024-01-16T15:06:00.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.405+0000 7f07d4a00640 1 -- 172.21.15.125:0/3654960279 wait complete. 2024-01-16T15:06:00.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.407+0000 7f07d4a00640 1 Processor -- start 2024-01-16T15:06:00.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.407+0000 7f07d4a00640 1 -- start start 2024-01-16T15:06:00.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc0cf7d0 0x7f07cc13ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07cc143ae0 0x7f07cc0aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc13ef40 con 0x7f07cc15e1e0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc140160 con 0x7f07cc143ae0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc1f3f00 con 0x7f07cc0cf7d0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49830/0 (socket says 172.21.15.125:49830) 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.408+0000 7f07d2a00640 1 -- 172.21.15.125:0/2793520620 learned_addr learned my addr 172.21.15.125:0/2793520620 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2000640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc0cf7d0 0x7f07cc13ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d1600640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07cc143ae0 0x7f07cc0aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 -- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc0cf7d0 msgr2=0x7f07cc13ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc0cf7d0 0x7f07cc13ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 -- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07cc143ae0 msgr2=0x7f07cc0aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07cc143ae0 0x7f07cc0aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 -- 172.21.15.125:0/2793520620 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40094d0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d2a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f07c400cc30 tx=0x7f07c400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c400ce40 con 0x7f07cc15e1e0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c4003990 con 0x7f07cc15e1e0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4003d20 con 0x7f07cc15e1e0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.409+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc1f41e0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.410+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc1f47a0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.411+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc1f49c0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.412+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f07c4003e80 con 0x7f07cc15e1e0 2024-01-16T15:06:00.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.413+0000 7f07ca000640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 0x7f07a406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.413+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f07c4088880 con 0x7f07cc15e1e0 2024-01-16T15:06:00.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.413+0000 7f07d2000640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 0x7f07a406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.414+0000 7f07d2000640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 0x7f07a406e1a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f07bc006fd0 tx=0x7f07bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.418+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07c40551a0 con 0x7f07cc15e1e0 2024-01-16T15:06:00.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.642+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96912", "target": ["mon-mgr", ""]}) v1 -- 0x7f07cc15e640 con 0x7f07a406bce0 2024-01-16T15:06:00.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.739+0000 7f07ca000640 1 -- 172.21.15.125:0/2793520620 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f07bc0094a0 con 0x7f07a406bce0 2024-01-16T15:06:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 msgr2=0x7f07a406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 0x7f07a406e1a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f07bc006fd0 tx=0x7f07bc008040 comp rx=0 tx=0).stop 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 msgr2=0x7f07cc0aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f07c400cc30 tx=0x7f07c400a5c0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 shutdown_connections 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07a406bce0 0x7f07a406e1a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07cc15e1e0 0x7f07cc0aeff0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07cc143ae0 0x7f07cc0aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 --2- 172.21.15.125:0/2793520620 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07cc0cf7d0 0x7f07cc13ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 >> 172.21.15.125:0/2793520620 conn(0x7f07cc06a800 msgr2=0x7f07cc06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 shutdown_connections 2024-01-16T15:06:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.743+0000 7f07d4a00640 1 -- 172.21.15.125:0/2793520620 wait complete. 2024-01-16T15:06:00.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96913 2024-01-16T15:06:00.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.978+0000 7f2ae6400640 1 Processor -- start 2024-01-16T15:06:00.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.978+0000 7f2ae6400640 1 -- start start 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 0x7f2ae00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ae00cf7d0 0x7f2ae015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 0x7f2ae015a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00cfc30 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00b1ee0 con 0x7f2ae0143ae0 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2ae6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00b2780 con 0x7f2ae015e1e0 2024-01-16T15:06:00.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.979+0000 7f2adea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ae00cf7d0 0x7f2ae015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ae00cf7d0 0x7f2ae015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49844/0 (socket says 172.21.15.125:49844) 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 -- 172.21.15.125:0/4235741631 learned_addr learned my addr 172.21.15.125:0/4235741631 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adf400640 1 --2- 172.21.15.125:0/4235741631 >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 0x7f2ae00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adfe00640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 0x7f2ae015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 -- 172.21.15.125:0/4235741631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 msgr2=0x7f2ae015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 0x7f2ae015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 -- 172.21.15.125:0/4235741631 >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 msgr2=0x7f2ae00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 --2- 172.21.15.125:0/4235741631 >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 0x7f2ae00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adea00640 1 -- 172.21.15.125:0/4235741631 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae015e990 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adfe00640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 0x7f2ae015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:00.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.980+0000 7f2adf400640 1 --2- 172.21.15.125:0/4235741631 >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 0x7f2ae00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.981+0000 7f2adea00640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae00cf7d0 0x7f2ae015dca0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cfea0 tx=0x7f2ac802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af282dcc313034e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.981+0000 7f2ade000640 1 -- 172.21.15.125:0/4235741631 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac802ee90 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.981+0000 7f2ade000640 1 -- 172.21.15.125:0/4235741631 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac8036440 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.981+0000 7f2ade000640 1 -- 172.21.15.125:0/4235741631 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80367d0 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 -- 172.21.15.125:0/4235741631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae00cf7d0 msgr2=0x7f2ae015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae00cf7d0 0x7f2ae015dca0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cfea0 tx=0x7f2ac802e0e0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 -- 172.21.15.125:0/4235741631 shutdown_connections 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae015e1e0 0x7f2ae015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 --2- 172.21.15.125:0/4235741631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae00cf7d0 0x7f2ae015dca0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 --2- 172.21.15.125:0/4235741631 >> v2:172.21.15.137:3300/0 conn(0x7f2ae0143ae0 0x7f2ae00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 -- 172.21.15.125:0/4235741631 >> 172.21.15.125:0/4235741631 conn(0x7f2ae006a800 msgr2=0x7f2ae006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 -- 172.21.15.125:0/4235741631 shutdown_connections 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.982+0000 7f2ae6400640 1 -- 172.21.15.125:0/4235741631 wait complete. 2024-01-16T15:06:00.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.984+0000 7f2ae6400640 1 Processor -- start 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.984+0000 7f2ae6400640 1 -- start start 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 0x7f2ae013d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 0x7f2ae0141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae013e5a0 con 0x7f2ae015e1e0 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae01f6380 con 0x7f2ae0143ae0 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2ae6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae01f6660 con 0x7f2ae00cf7d0 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2adfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2adfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49858/0 (socket says 172.21.15.125:49858) 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2adfe00640 1 -- 172.21.15.125:0/1630813428 learned_addr learned my addr 172.21.15.125:0/1630813428 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.985+0000 7f2adea00640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 0x7f2ae0141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adf400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 0x7f2ae013d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 -- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 msgr2=0x7f2ae013d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 0x7f2ae013d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 -- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 msgr2=0x7f2ae0141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 0x7f2ae0141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 -- 172.21.15.125:0/1630813428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac800b770 con 0x7f2ae015e1e0 2024-01-16T15:06:00.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adf400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 0x7f2ae013d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adea00640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 0x7f2ae0141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2adfe00640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000cd50 tx=0x7f2ad000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000f680 con 0x7f2ae015e1e0 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad000f800 con 0x7f2ae015e1e0 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae01f6940 con 0x7f2ae015e1e0 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000fbb0 con 0x7f2ae015e1e0 2024-01-16T15:06:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.986+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae013ddf0 con 0x7f2ae015e1e0 2024-01-16T15:06:00.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.987+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae013e710 con 0x7f2ae015e1e0 2024-01-16T15:06:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.988+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f2ad000b040 con 0x7f2ae015e1e0 2024-01-16T15:06:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.989+0000 7f2ad7400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 0x7f2ab806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.989+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2ad008b6f0 con 0x7f2ae015e1e0 2024-01-16T15:06:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.989+0000 7f2adf400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 0x7f2ab806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.990+0000 7f2adf400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 0x7f2ab806e1a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80088b0 tx=0x7f2ad8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:00.993+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ad0055830 con 0x7f2ae015e1e0 2024-01-16T15:06:01.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:01.220+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96913", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ae015e670 con 0x7f2ab806bce0 2024-01-16T15:06:04.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.119+0000 7f2ad7400640 1 -- 172.21.15.125:0/1630813428 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ad800c040 con 0x7f2ab806bce0 2024-01-16T15:06:04.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 msgr2=0x7f2ab806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 0x7f2ab806e1a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80088b0 tx=0x7f2ad8008040 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 msgr2=0x7f2ae0141b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000cd50 tx=0x7f2ad000a650 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 shutdown_connections 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ab806bce0 0x7f2ab806e1a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ae015e1e0 0x7f2ae0141b60 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ae0143ae0 0x7f2ae0141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 --2- 172.21.15.125:0/1630813428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ae00cf7d0 0x7f2ae013d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.122+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 >> 172.21.15.125:0/1630813428 conn(0x7f2ae006a800 msgr2=0x7f2ae015cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.123+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 shutdown_connections 2024-01-16T15:06:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.123+0000 7f2ae6400640 1 -- 172.21.15.125:0/1630813428 wait complete. 2024-01-16T15:06:04.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96914 2024-01-16T15:06:04.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.351+0000 7f337ba00640 1 Processor -- start 2024-01-16T15:06:04.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.351+0000 7f337ba00640 1 -- start start 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f337ba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 0x7f33740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f337ba00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3374070800 0x7f33740b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f337ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3374071a00 con 0x7f3374143a50 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f337ba00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3374143e30 con 0x7f3374070800 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f337ba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33740b24f0 con 0x7f33740cfad0 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f3379a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3374070800 0x7f33740b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f3379a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3374070800 0x7f33740b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49872/0 (socket says 172.21.15.125:49872) 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.353+0000 7f3379a00640 1 -- 172.21.15.125:0/1409469384 learned_addr learned my addr 172.21.15.125:0/1409469384 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3373e00640 1 --2- 172.21.15.125:0/1409469384 >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 0x7f33740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3379a00640 1 -- 172.21.15.125:0/1409469384 >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 msgr2=0x7f33740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3379a00640 1 --2- 172.21.15.125:0/1409469384 >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 0x7f33740702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3379a00640 1 -- 172.21.15.125:0/1409469384 >> v1:172.21.15.137:6789/0 conn(0x7f3374143a50 legacy=0x7f33740cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:04.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3379a00640 1 -- 172.21.15.125:0/1409469384 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740b1ee0 con 0x7f3374070800 2024-01-16T15:06:04.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.354+0000 7f3373e00640 1 --2- 172.21.15.125:0/1409469384 >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 0x7f33740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:04.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.355+0000 7f3379a00640 1 --2- 172.21.15.125:0/1409469384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374070800 0x7f33740b2980 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3374151660 tx=0x7f336c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3de8af43f76dcd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:04.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.355+0000 7f3373400640 1 -- 172.21.15.125:0/1409469384 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c02ca80 con 0x7f3374070800 2024-01-16T15:06:04.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.355+0000 7f3373400640 1 -- 172.21.15.125:0/1409469384 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c02cbe0 con 0x7f3374070800 2024-01-16T15:06:04.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.355+0000 7f3373400640 1 -- 172.21.15.125:0/1409469384 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0343f0 con 0x7f3374070800 2024-01-16T15:06:04.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 -- 172.21.15.125:0/1409469384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374070800 msgr2=0x7f33740b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:04.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 --2- 172.21.15.125:0/1409469384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374070800 0x7f33740b2980 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3374151660 tx=0x7f336c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 -- 172.21.15.125:0/1409469384 shutdown_connections 2024-01-16T15:06:04.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 --2- 172.21.15.125:0/1409469384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374070800 0x7f33740b2980 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 --2- 172.21.15.125:0/1409469384 >> v2:172.21.15.137:3300/0 conn(0x7f33740cfad0 0x7f33740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 -- 172.21.15.125:0/1409469384 >> 172.21.15.125:0/1409469384 conn(0x7f337406a800 msgr2=0x7f337406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:04.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 -- 172.21.15.125:0/1409469384 shutdown_connections 2024-01-16T15:06:04.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.356+0000 7f337ba00640 1 -- 172.21.15.125:0/1409469384 wait complete. 2024-01-16T15:06:04.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.358+0000 7f337ba00640 1 Processor -- start 2024-01-16T15:06:04.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.358+0000 7f337ba00640 1 -- start start 2024-01-16T15:06:04.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 0x7f337413de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33740cfad0 0x7f33741565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3374142be0 con 0x7f3374143a50 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337415a6e0 con 0x7f33740cfad0 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f337ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337415a9c0 con 0x7f3374070800 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f3379a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f3379a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49874/0 (socket says 172.21.15.125:49874) 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.359+0000 7f3379a00640 1 -- 172.21.15.125:0/999227988 learned_addr learned my addr 172.21.15.125:0/999227988 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3373e00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33740cfad0 0x7f33741565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379000640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 0x7f337413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 -- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 msgr2=0x7f337413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 0x7f337413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 -- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33740cfad0 msgr2=0x7f33741565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33740cfad0 0x7f33741565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 -- 172.21.15.125:0/999227988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c009b30 con 0x7f3374143a50 2024-01-16T15:06:04.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379000640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 0x7f337413de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3379a00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f336c009f40 tx=0x7f336c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c009510 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c03e450 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c03e7e0 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337415aca0 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.360+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337413e340 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.362+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337415aec0 con 0x7f3374143a50 2024-01-16T15:06:04.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.362+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f336c02cd50 con 0x7f3374143a50 2024-01-16T15:06:04.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.363+0000 7f3370c00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 0x7f334c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:04.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.363+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f336c0ab3b0 con 0x7f3374143a50 2024-01-16T15:06:04.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.364+0000 7f3379000640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 0x7f334c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:04.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.364+0000 7f3379000640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 0x7f334c06e1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f336400a810 tx=0x7f3364008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:04.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.368+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f336c077b70 con 0x7f3374143a50 2024-01-16T15:06:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:04.596+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f3374071180 con 0x7f334c06bce0 2024-01-16T15:06:09.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:09.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:09.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.106+0000 7f3370c00640 1 -- 172.21.15.125:0/999227988 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f336400ad90 con 0x7f334c06bce0 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 msgr2=0x7f334c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 0x7f334c06e1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f336400a810 tx=0x7f3364008040 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 msgr2=0x7f3374156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f336c009f40 tx=0x7f336c004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 shutdown_connections 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f334c06bce0 0x7f334c06e1a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3374143a50 0x7f3374156ae0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33740cfad0 0x7f33741565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 --2- 172.21.15.125:0/999227988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3374070800 0x7f337413de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.109+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 >> 172.21.15.125:0/999227988 conn(0x7f337406a800 msgr2=0x7f3374158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:09.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.110+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 shutdown_connections 2024-01-16T15:06:09.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.110+0000 7f337ba00640 1 -- 172.21.15.125:0/999227988 wait complete. 2024-01-16T15:06:09.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96915 2024-01-16T15:06:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.383+0000 7f3c5b000640 1 Processor -- start 2024-01-16T15:06:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.383+0000 7f3c5b000640 1 -- start start 2024-01-16T15:06:09.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c5b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 0x7f3c540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c5b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 0x7f3c540cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c5b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5415c5f0 con 0x7f3c5413df80 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c5b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5406c470 con 0x7f3c5415f9c0 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c5b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5406c750 con 0x7f3c5413e360 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c59000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 0x7f3c540cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.385+0000 7f3c53400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 0x7f3c540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c59000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 0x7f3c540cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:39340/0 (socket says 172.21.15.125:39340) 2024-01-16T15:06:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c59000640 1 -- 172.21.15.125:0/850036294 learned_addr learned my addr 172.21.15.125:0/850036294 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c53400640 1 -- 172.21.15.125:0/850036294 >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 msgr2=0x7f3c540cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c53400640 1 --2- 172.21.15.125:0/850036294 >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 0x7f3c540cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c53400640 1 -- 172.21.15.125:0/850036294 >> v1:172.21.15.137:6789/0 conn(0x7f3c5413df80 legacy=0x7f3c5415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.386+0000 7f3c53400640 1 -- 172.21.15.125:0/850036294 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5406ca30 con 0x7f3c5413e360 2024-01-16T15:06:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.388+0000 7f3c53400640 1 --2- 172.21.15.125:0/850036294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 0x7f3c540cf290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0097a0 tx=0x7f3c3c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5fe9b9bc53c44e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.388+0000 7f3c52a00640 1 -- 172.21.15.125:0/850036294 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c031830 con 0x7f3c5413e360 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.388+0000 7f3c52a00640 1 -- 172.21.15.125:0/850036294 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c031990 con 0x7f3c5413e360 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.388+0000 7f3c52a00640 1 -- 172.21.15.125:0/850036294 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c031d20 con 0x7f3c5413e360 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 -- 172.21.15.125:0/850036294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 msgr2=0x7f3c540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 --2- 172.21.15.125:0/850036294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 0x7f3c540cf290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0097a0 tx=0x7f3c3c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 -- 172.21.15.125:0/850036294 shutdown_connections 2024-01-16T15:06:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 --2- 172.21.15.125:0/850036294 >> v2:172.21.15.137:3300/0 conn(0x7f3c5415f9c0 0x7f3c540cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 --2- 172.21.15.125:0/850036294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5413e360 0x7f3c540cf290 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 -- 172.21.15.125:0/850036294 >> 172.21.15.125:0/850036294 conn(0x7f3c5406a800 msgr2=0x7f3c5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 -- 172.21.15.125:0/850036294 shutdown_connections 2024-01-16T15:06:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.389+0000 7f3c5b000640 1 -- 172.21.15.125:0/850036294 wait complete. 2024-01-16T15:06:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.391+0000 7f3c5b000640 1 Processor -- start 2024-01-16T15:06:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.391+0000 7f3c5b000640 1 -- start start 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c5413e360 0x7f3c54153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5415f9c0 0x7f3c54150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c541539b0 con 0x7f3c5413df80 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c54156bf0 con 0x7f3c5413e360 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c5b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5415aaa0 con 0x7f3c5415f9c0 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54446/0 (socket says 172.21.15.125:54446) 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.392+0000 7f3c53e00640 1 -- 172.21.15.125:0/487194210 learned_addr learned my addr 172.21.15.125:0/487194210 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c59000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5415f9c0 0x7f3c54150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53400640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c5413e360 0x7f3c54153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 -- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5415f9c0 msgr2=0x7f3c54150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5415f9c0 0x7f3c54150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 -- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c5413e360 msgr2=0x7f3c54153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c5413e360 0x7f3c54153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 -- 172.21.15.125:0/487194210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009400 con 0x7f3c5413df80 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c53e00640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c3c0 tx=0x7f3c4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.393+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440052d0 con 0x7f3c5413df80 2024-01-16T15:06:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.394+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c44005450 con 0x7f3c5413df80 2024-01-16T15:06:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.394+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44007260 con 0x7f3c5413df80 2024-01-16T15:06:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.394+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5415ad80 con 0x7f3c5413df80 2024-01-16T15:06:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.394+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5415b060 con 0x7f3c5413df80 2024-01-16T15:06:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.395+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c540675d0 con 0x7f3c5413df80 2024-01-16T15:06:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.396+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f3c440073c0 con 0x7f3c5413df80 2024-01-16T15:06:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.396+0000 7f3c4be00640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 0x7f3c2c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.397+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f3c44087ad0 con 0x7f3c5413df80 2024-01-16T15:06:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.397+0000 7f3c53400640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 0x7f3c2c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:09.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.398+0000 7f3c53400640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 0x7f3c2c06e1a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c036000 tx=0x7f3c3c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:09.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.402+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c44054170 con 0x7f3c5413df80 2024-01-16T15:06:09.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:09.630+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96915", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c5406cff0 con 0x7f3c2c06bce0 2024-01-16T15:06:14.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.098+0000 7f3c4be00640 1 -- 172.21.15.125:0/487194210 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c3c03a420 con 0x7f3c2c06bce0 2024-01-16T15:06:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 msgr2=0x7f3c2c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 0x7f3c2c06e1a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c036000 tx=0x7f3c3c03c040 comp rx=0 tx=0).stop 2024-01-16T15:06:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 msgr2=0x7f3c54154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c3c0 tx=0x7f3c4400c880 comp rx=0 tx=0).stop 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 shutdown_connections 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c5415f9c0 0x7f3c54150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c5413e360 0x7f3c54153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c2c06bce0 0x7f3c2c06e1a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 --2- 172.21.15.125:0/487194210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c5413df80 0x7f3c54154ca0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 >> 172.21.15.125:0/487194210 conn(0x7f3c5406a800 msgr2=0x7f3c5415cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.102+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 shutdown_connections 2024-01-16T15:06:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.103+0000 7f3c5b000640 1 -- 172.21.15.125:0/487194210 wait complete. 2024-01-16T15:06:14.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96916 2024-01-16T15:06:14.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.334+0000 7f3ecae00640 1 Processor -- start 2024-01-16T15:06:14.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.334+0000 7f3ecae00640 1 -- start start 2024-01-16T15:06:14.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.335+0000 7f3ecae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 0x7f3ec40cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.335+0000 7f3ecae00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3ec406d160 0x7f3ec406d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.335+0000 7f3ecae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec4070e50 con 0x7f3ec4143c40 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.335+0000 7f3ecae00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40723c0 con 0x7f3ec406d160 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.335+0000 7f3ecae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4144020 con 0x7f3ec40cc500 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3eca400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3ec406d160 0x7f3ec406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3ec9a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3ec4143c40 0x7f3ec4071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32858/0 (socket says 172.21.15.125:32858) 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3ec9a00640 1 -- 172.21.15.125:0/2517837908 learned_addr learned my addr 172.21.15.125:0/2517837908 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3ec9000640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 0x7f3ec40cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3eca400640 1 -- 172.21.15.125:0/2517837908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 msgr2=0x7f3ec40cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3eca400640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 0x7f3ec40cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3eca400640 1 -- 172.21.15.125:0/2517837908 >> v1:172.21.15.125:6789/0 conn(0x7f3ec4143c40 legacy=0x7f3ec4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3eca400640 1 -- 172.21.15.125:0/2517837908 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40ccea0 con 0x7f3ec406d160 2024-01-16T15:06:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3ec9000640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 0x7f3ec40cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.336+0000 7f3ec3e00640 1 -- 172.21.15.125:0/2517837908 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274279807 0 0) 0x7f3ec4070e50 con 0x7f3ec4143c40 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3eca400640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec406d160 0x7f3ec406d590 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009d90 tx=0x7f3ebc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606a5a1fcaddd6bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3ec3e00640 1 -- 172.21.15.125:0/2517837908 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc02cbc0 con 0x7f3ec406d160 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3ec3e00640 1 -- 172.21.15.125:0/2517837908 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc02cd20 con 0x7f3ec406d160 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3ec3e00640 1 -- 172.21.15.125:0/2517837908 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc034660 con 0x7f3ec406d160 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3ecae00640 1 -- 172.21.15.125:0/2517837908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec406d160 msgr2=0x7f3ec406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.338+0000 7f3ecae00640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec406d160 0x7f3ec406d590 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009d90 tx=0x7f3ebc02be10 comp rx=0 tx=0).stop 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 -- 172.21.15.125:0/2517837908 shutdown_connections 2024-01-16T15:06:14.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec406d160 0x7f3ec406d590 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 --2- 172.21.15.125:0/2517837908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec40cc500 0x7f3ec40cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 -- 172.21.15.125:0/2517837908 >> 172.21.15.125:0/2517837908 conn(0x7f3ec406a800 msgr2=0x7f3ec406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:14.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 -- 172.21.15.125:0/2517837908 shutdown_connections 2024-01-16T15:06:14.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.339+0000 7f3ecae00640 1 -- 172.21.15.125:0/2517837908 wait complete. 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.340+0000 7f3ecae00640 1 Processor -- start 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.341+0000 7f3ecae00640 1 -- start start 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.341+0000 7f3ecae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ec406d160 0x7f3ec413b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ecae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ecae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 0x7f3ec40a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ecae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40a95f0 con 0x7f3ec40cc500 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ecae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40aac00 con 0x7f3ec406d160 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ecae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40a9030 con 0x7f3ec4143c40 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54464/0 (socket says 172.21.15.125:54464) 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 -- 172.21.15.125:0/3097514227 learned_addr learned my addr 172.21.15.125:0/3097514227 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3eca400640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 0x7f3ec40a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 -- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 msgr2=0x7f3ec40a8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9a00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ec406d160 0x7f3ec413b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 0x7f3ec40a8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 -- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ec406d160 msgr2=0x7f3ec413b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ec406d160 0x7f3ec413b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3ec9000640 1 -- 172.21.15.125:0/3097514227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc009a70 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.342+0000 7f3eca400640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 0x7f3ec40a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ec9000640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00c7f0 tx=0x7f3eac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac007a30 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac007b90 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00ce40 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec40a9310 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.343+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec406d540 con 0x7f3ec40cc500 2024-01-16T15:06:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.344+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec40ab0e0 con 0x7f3ec40cc500 2024-01-16T15:06:14.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.345+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f3eac0113f0 con 0x7f3ec40cc500 2024-01-16T15:06:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.346+0000 7f3ec1600640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 0x7f3e9c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.346+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f3eac088880 con 0x7f3ec40cc500 2024-01-16T15:06:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.346+0000 7f3ec9a00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 0x7f3e9c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.347+0000 7f3ec9a00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 0x7f3e9c06e1a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4000f60 tx=0x7f3eb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.351+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ec40ab0e0 con 0x7f3ec40cc500 2024-01-16T15:06:14.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.575+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96916", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ec406c960 con 0x7f3e9c06bce0 2024-01-16T15:06:14.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.731+0000 7f3ec1600640 1 -- 172.21.15.125:0/3097514227 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3eb4001270 con 0x7f3e9c06bce0 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 msgr2=0x7f3e9c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 0x7f3e9c06e1a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4000f60 tx=0x7f3eb4009040 comp rx=0 tx=0).stop 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 msgr2=0x7f3ec40a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00c7f0 tx=0x7f3eac00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 shutdown_connections 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ec4143c40 0x7f3ec40a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e9c06bce0 0x7f3e9c06e1a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ec40cc500 0x7f3ec40a84c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 --2- 172.21.15.125:0/3097514227 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ec406d160 0x7f3ec413b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.734+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 >> 172.21.15.125:0/3097514227 conn(0x7f3ec406a800 msgr2=0x7f3ec406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.735+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 shutdown_connections 2024-01-16T15:06:14.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.735+0000 7f3ecae00640 1 -- 172.21.15.125:0/3097514227 wait complete. 2024-01-16T15:06:14.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96917 2024-01-16T15:06:14.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.967+0000 7f1a80000640 1 Processor -- start 2024-01-16T15:06:14.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.967+0000 7f1a80000640 1 -- start start 2024-01-16T15:06:14.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a80000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a7815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a80000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1a780cf290 0x7f1a780cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a80000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a78070c40 con 0x7f1a780ae5a0 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a80000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a78071450 con 0x7f1a780cf290 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a80000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a780cfc60 con 0x7f1a780b3ae0 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a7d600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1a780ae5a0 0x7f1a7806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32870/0 (socket says 172.21.15.125:32870) 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.969+0000 7f1a7d600640 1 -- 172.21.15.125:0/2202288678 learned_addr learned my addr 172.21.15.125:0/2202288678 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a7cc00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a7815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a7e000640 1 --2- 172.21.15.125:0/2202288678 >> v2:172.21.15.137:3300/0 conn(0x7f1a780cf290 0x7f1a780cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760710391 0 0) 0x7f1a78070c40 con 0x7f1a780ae5a0 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003520 con 0x7f1a780ae5a0 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004780044 0 0) 0x7f1a58003520 con 0x7f1a780ae5a0 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 msgr2=0x7f1a7815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a7815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 >> v2:172.21.15.137:3300/0 conn(0x7f1a780cf290 msgr2=0x7f1a780cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 --2- 172.21.15.125:0/2202288678 >> v2:172.21.15.137:3300/0 conn(0x7f1a780cf290 0x7f1a780cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7806c470 con 0x7f1a780ae5a0 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1a68001670 con 0x7f1a780ae5a0 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 >> v1:172.21.15.125:6789/0 conn(0x7f1a780ae5a0 legacy=0x7f1a7806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a58003970 0x7f1a58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a58004d00 0x7f1a58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a78071450 con 0x7f1a58004d00 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a780cfc60 con 0x7f1a58003970 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.970+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5800b980 con 0x7f1a780b3ae0 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a58004d00 0x7f1a58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7d600640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a58003970 0x7f1a58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7cc00640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 -- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 msgr2=0x7f1a580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:14.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a780b3ae0 0x7f1a580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 -- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a58003970 msgr2=0x7f1a58003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a58003970 0x7f1a58003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7e000640 1 -- 172.21.15.125:0/2202288678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58003520 con 0x7f1a58004d00 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.971+0000 7f1a7d600640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a58003970 0x7f1a58003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.972+0000 7f1a7e000640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a58004d00 0x7f1a58005180 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009fc0 tx=0x7f1a580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0d73a49e114b89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7002cc60 con 0x7f1a58004d00 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7000f040 con 0x7f1a58004d00 2024-01-16T15:06:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a77e00640 1 -- 172.21.15.125:0/2202288678 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a70034440 con 0x7f1a58004d00 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 -- 172.21.15.125:0/2202288678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a58004d00 msgr2=0x7f1a58005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 --2- 172.21.15.125:0/2202288678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a58004d00 0x7f1a58005180 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009fc0 tx=0x7f1a580057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a7e000640 1 -- 172.21.15.125:0/2202288678 reap_dead start 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 -- 172.21.15.125:0/2202288678 shutdown_connections 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 -- 172.21.15.125:0/2202288678 >> 172.21.15.125:0/2202288678 conn(0x7f1a7806a800 msgr2=0x7f1a7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:14.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 -- 172.21.15.125:0/2202288678 shutdown_connections 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.973+0000 7f1a80000640 1 -- 172.21.15.125:0/2202288678 wait complete. 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.975+0000 7f1a80000640 1 Processor -- start 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.975+0000 7f1a80000640 1 -- start start 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a58004d00 0x7f1a7815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a78157f80 0x7f1a781f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7815c4f0 con 0x7f1a7813e340 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a781501c0 con 0x7f1a78157f80 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a80000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a78151e80 con 0x7f1a58004d00 2024-01-16T15:06:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.976+0000 7f1a7cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54488/0 (socket says 172.21.15.125:54488) 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 -- 172.21.15.125:0/4072968651 learned_addr learned my addr 172.21.15.125:0/4072968651 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7d600640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a58004d00 0x7f1a7815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 -- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a58004d00 msgr2=0x7f1a7815ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7e000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a78157f80 0x7f1a781f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a58004d00 0x7f1a7815ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 -- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a78157f80 msgr2=0x7f1a781f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a78157f80 0x7f1a781f1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 -- 172.21.15.125:0/4072968651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a781509d0 con 0x7f1a7813e340 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a7cc00640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1a60011600 tx=0x7f1a60011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.977+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000f920 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.978+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6000fa80 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.978+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000fdf0 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.978+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a78150410 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.978+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a78152120 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.979+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a780675b0 con 0x7f1a7813e340 2024-01-16T15:06:14.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.980+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f1a60015460 con 0x7f1a7813e340 2024-01-16T15:06:14.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.981+0000 7f1a75600640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 0x7f1a5006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:14.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.981+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1a60088b80 con 0x7f1a7813e340 2024-01-16T15:06:14.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.981+0000 7f1a7d600640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 0x7f1a5006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:14.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.982+0000 7f1a7d600640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 0x7f1a5006e1d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1a68005ab0 tx=0x7f1a68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:14.985+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a60055280 con 0x7f1a7813e340 2024-01-16T15:06:15.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.212+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96917", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a78071170 con 0x7f1a5006bd10 2024-01-16T15:06:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.343+0000 7f1a75600640 1 -- 172.21.15.125:0/4072968651 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1a680109e0 con 0x7f1a5006bd10 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 msgr2=0x7f1a5006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 0x7f1a5006e1d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1a68005ab0 tx=0x7f1a68009040 comp rx=0 tx=0).stop 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 msgr2=0x7f1a7815bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1a60011600 tx=0x7f1a60011ac0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 shutdown_connections 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a78157f80 0x7f1a781f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1a5006bd10 0x7f1a5006e1d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a7813e340 0x7f1a7815bfb0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 --2- 172.21.15.125:0/4072968651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a58004d00 0x7f1a7815ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 >> 172.21.15.125:0/4072968651 conn(0x7f1a7806a800 msgr2=0x7f1a780b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:15.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.346+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 shutdown_connections 2024-01-16T15:06:15.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.347+0000 7f1a80000640 1 -- 172.21.15.125:0/4072968651 wait complete. 2024-01-16T15:06:15.345 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:06:15.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:06:15.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.572+0000 7f7195000640 1 Processor -- start 2024-01-16T15:06:15.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.572+0000 7f7195000640 1 -- start start 2024-01-16T15:06:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f7195000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f719015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f7195000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900b1ee0 con 0x7f7190143ae0 2024-01-16T15:06:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f7195000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900b2780 con 0x7f719015e1e0 2024-01-16T15:06:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f7195000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719015e990 con 0x7f71900cf9a0 2024-01-16T15:06:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f718de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7190143ae0 0x7f71900cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32882/0 (socket says 172.21.15.125:32882) 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f718de00640 1 -- 172.21.15.125:0/159303150 learned_addr learned my addr 172.21.15.125:0/159303150 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f718d400640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f719015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734480762 0 0) 0x7f71900b1ee0 con 0x7f7190143ae0 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.574+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7170003520 con 0x7f7190143ae0 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4281103954 0 0) 0x7f7170003520 con 0x7f7190143ae0 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 msgr2=0x7f719015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f719015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 >> v1:172.21.15.137:6789/0 conn(0x7f719015e1e0 legacy=0x7f71900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900b2130 con 0x7f7190143ae0 2024-01-16T15:06:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f71800019f0 con 0x7f7190143ae0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 >> v1:172.21.15.125:6789/0 conn(0x7f7190143ae0 legacy=0x7f71900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f71700039a0 0x7f7170003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f7170004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f7170005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719015e990 con 0x7f71700047c0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717000ba30 con 0x7f71700039a0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717000bbf0 con 0x7f71900cf9a0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718d400640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f7170004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.575+0000 7f718de00640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f71700039a0 0x7f7170003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718e800640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f7170005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718d400640 1 -- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 msgr2=0x7f7170005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718d400640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f71900cf9a0 0x7f7170005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718d400640 1 -- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f71700039a0 msgr2=0x7f7170003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718d400640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f71700039a0 0x7f7170003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.576+0000 7f718d400640 1 -- 172.21.15.125:0/159303150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7170003520 con 0x7f71700047c0 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.577+0000 7f718d400640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f7170004c40 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f7178009a90 tx=0x7f71700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292be8b8dec89e02 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.577+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802c680 con 0x7f71700047c0 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.577+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717802c7e0 con 0x7f71700047c0 2024-01-16T15:06:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.577+0000 7f718ca00640 1 -- 172.21.15.125:0/159303150 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802cb70 con 0x7f71700047c0 2024-01-16T15:06:15.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 -- 172.21.15.125:0/159303150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 msgr2=0x7f7170004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 --2- 172.21.15.125:0/159303150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f7170004c40 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f7178009a90 tx=0x7f71700057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f718e800640 1 -- 172.21.15.125:0/159303150 reap_dead start 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 -- 172.21.15.125:0/159303150 shutdown_connections 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 -- 172.21.15.125:0/159303150 >> 172.21.15.125:0/159303150 conn(0x7f719006a800 msgr2=0x7f719006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 -- 172.21.15.125:0/159303150 shutdown_connections 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.578+0000 7f7195000640 1 -- 172.21.15.125:0/159303150 wait complete. 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.580+0000 7f7195000640 1 Processor -- start 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.580+0000 7f7195000640 1 -- start start 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f719014f270 0x7f719013d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f719014e4f0 0x7f719014e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719013cee0 con 0x7f71700047c0 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719014eeb0 con 0x7f719014f270 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f7195000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719013bff0 con 0x7f719014e4f0 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54516/0 (socket says 172.21.15.125:54516) 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 -- 172.21.15.125:0/963794485 learned_addr learned my addr 172.21.15.125:0/963794485 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 -- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f719014e4f0 msgr2=0x7f719014e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f719014e4f0 0x7f719014e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 -- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f719014f270 msgr2=0x7f719013d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f719014f270 0x7f719013d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.581+0000 7f718de00640 1 -- 172.21.15.125:0/963794485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719013c2d0 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f718de00640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f7180007a50 tx=0x7f718000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180013420 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180013580 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180013910 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719013c5b0 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.582+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719013cb70 con 0x7f71700047c0 2024-01-16T15:06:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.584+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71900675d0 con 0x7f71700047c0 2024-01-16T15:06:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.584+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f7180013a70 con 0x7f71700047c0 2024-01-16T15:06:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.585+0000 7f7186000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 0x7f716806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.585+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7180088ff0 con 0x7f71700047c0 2024-01-16T15:06:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.586+0000 7f718d400640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 0x7f716806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.586+0000 7f718d400640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 0x7f716806e1a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7178036000 tx=0x7f717802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.590+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7180055690 con 0x7f71700047c0 2024-01-16T15:06:15.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.817+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f719015e640 con 0x7f716806bce0 2024-01-16T15:06:15.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.828+0000 7f7186000640 1 -- 172.21.15.125:0/963794485 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f71780040c0 con 0x7f716806bce0 2024-01-16T15:06:15.806 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:15.807 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 msgr2=0x7f716806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 0x7f716806e1a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7178036000 tx=0x7f717802e000 comp rx=0 tx=0).stop 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 msgr2=0x7f71900b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f7180007a50 tx=0x7f718000e530 comp rx=0 tx=0).stop 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 shutdown_connections 2024-01-16T15:06:15.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f719014e4f0 0x7f719014e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f719014f270 0x7f719013d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f716806bce0 0x7f716806e1a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 --2- 172.21.15.125:0/963794485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71700047c0 0x7f71900b1410 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 >> 172.21.15.125:0/963794485 conn(0x7f719006a800 msgr2=0x7f719015c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 shutdown_connections 2024-01-16T15:06:15.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:15.831+0000 7f7195000640 1 -- 172.21.15.125:0/963794485 wait complete. 2024-01-16T15:06:15.826 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:16.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:06:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.072+0000 7fcb7a200640 1 Processor -- start 2024-01-16T15:06:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.072+0000 7fcb7a200640 1 -- start start 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.073+0000 7fcb7a200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb740cf9a0 0x7fcb7415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.073+0000 7fcb7a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb7415e1e0 0x7fcb740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.073+0000 7fcb7a200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740b1ee0 con 0x7fcb74143ae0 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.073+0000 7fcb7a200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb740b2780 con 0x7fcb740cf9a0 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.073+0000 7fcb7a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7415e990 con 0x7fcb7415e1e0 2024-01-16T15:06:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb740cf9a0 0x7fcb7415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb740cf9a0 0x7fcb7415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60464/0 (socket says 172.21.15.125:60464) 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 -- 172.21.15.125:0/3676147777 learned_addr learned my addr 172.21.15.125:0/3676147777 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb73400640 1 --1- 172.21.15.125:0/3676147777 >> v1:172.21.15.125:6789/0 conn(0x7fcb74143ae0 0x7fcb740cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42692/0 (socket says 172.21.15.125:42692) 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 -- 172.21.15.125:0/3676147777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb7415e1e0 msgr2=0x7fcb740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 --2- 172.21.15.125:0/3676147777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb7415e1e0 0x7fcb740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 -- 172.21.15.125:0/3676147777 >> v1:172.21.15.125:6789/0 conn(0x7fcb74143ae0 legacy=0x7fcb740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb72a00640 1 -- 172.21.15.125:0/3676147777 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867951597 0 0) 0x7fcb740b1ee0 con 0x7fcb74143ae0 2024-01-16T15:06:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.074+0000 7fcb6be00640 1 -- 172.21.15.125:0/3676147777 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740b2130 con 0x7fcb740cf9a0 2024-01-16T15:06:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.075+0000 7fcb6be00640 1 --2- 172.21.15.125:0/3676147777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb740cf9a0 0x7fcb7415dca0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009870 tx=0x7fcb5c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6836bdc122ee4fee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.075+0000 7fcb72a00640 1 -- 172.21.15.125:0/3676147777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c030840 con 0x7fcb740cf9a0 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.075+0000 7fcb72a00640 1 -- 172.21.15.125:0/3676147777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0309a0 con 0x7fcb740cf9a0 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.075+0000 7fcb72a00640 1 -- 172.21.15.125:0/3676147777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c030d30 con 0x7fcb740cf9a0 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 -- 172.21.15.125:0/3676147777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb740cf9a0 msgr2=0x7fcb7415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3676147777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb740cf9a0 0x7fcb7415dca0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009870 tx=0x7fcb5c02b980 comp rx=0 tx=0).stop 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 -- 172.21.15.125:0/3676147777 shutdown_connections 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3676147777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb7415e1e0 0x7fcb740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3676147777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb740cf9a0 0x7fcb7415dca0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 -- 172.21.15.125:0/3676147777 >> 172.21.15.125:0/3676147777 conn(0x7fcb7406a800 msgr2=0x7fcb7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 -- 172.21.15.125:0/3676147777 shutdown_connections 2024-01-16T15:06:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.076+0000 7fcb7a200640 1 -- 172.21.15.125:0/3676147777 wait complete. 2024-01-16T15:06:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.078+0000 7fcb7a200640 1 Processor -- start 2024-01-16T15:06:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.078+0000 7fcb7a200640 1 -- start start 2024-01-16T15:06:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb740cf9a0 0x7fcb7413e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb74143ae0 0x7fcb740ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740ad8e0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740ace10 con 0x7fcb74143ae0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb7a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740ad040 con 0x7fcb740cf9a0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb73e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb73e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60480/0 (socket says 172.21.15.125:60480) 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.079+0000 7fcb73e00640 1 -- 172.21.15.125:0/3074761894 learned_addr learned my addr 172.21.15.125:0/3074761894 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73400640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb740cf9a0 0x7fcb7413e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb6be00640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb74143ae0 0x7fcb740ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 -- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb740cf9a0 msgr2=0x7fcb7413e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb740cf9a0 0x7fcb7413e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 -- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb74143ae0 msgr2=0x7fcb740ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb74143ae0 0x7fcb740ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 -- 172.21.15.125:0/3074761894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c0094d0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb73e00640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00b9a0 tx=0x7fcb6c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c00cba0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c00cd20 con 0x7fcb7415e1e0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c014680 con 0x7fcb7415e1e0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.080+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb740ad320 con 0x7fcb7415e1e0 2024-01-16T15:06:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.081+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7413f250 con 0x7fcb7415e1e0 2024-01-16T15:06:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.082+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7413f0b0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.082+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7fcb6c01d3b0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.083+0000 7fcb6b400640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 0x7fcb4c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.083+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcb6c088680 con 0x7fcb7415e1e0 2024-01-16T15:06:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.084+0000 7fcb73400640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 0x7fcb4c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.084+0000 7fcb73400640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 0x7fcb4c06e1d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcb64000f60 tx=0x7fcb64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.088+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb7413f0b0 con 0x7fcb7415e1e0 2024-01-16T15:06:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.315+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb7415e670 con 0x7fcb4c06bd10 2024-01-16T15:06:17.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.328+0000 7fcb6b400640 1 -- 172.21.15.125:0/3074761894 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fcb640013d0 con 0x7fcb4c06bd10 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:17.306 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.330+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 msgr2=0x7fcb4c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.330+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 0x7fcb4c06e1d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcb64000f60 tx=0x7fcb64009040 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.330+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 msgr2=0x7fcb740ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.330+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00b9a0 tx=0x7fcb6c00be60 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 shutdown_connections 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb4c06bd10 0x7fcb4c06e1d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7415e1e0 0x7fcb740ac8d0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb74143ae0 0x7fcb740ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 --2- 172.21.15.125:0/3074761894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb740cf9a0 0x7fcb7413e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 >> 172.21.15.125:0/3074761894 conn(0x7fcb7406a800 msgr2=0x7fcb7415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 shutdown_connections 2024-01-16T15:06:17.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:17.331+0000 7fcb7a200640 1 -- 172.21.15.125:0/3074761894 wait complete. 2024-01-16T15:06:17.327 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:18.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:06:18.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.574+0000 7f7c1ce00640 1 Processor -- start 2024-01-16T15:06:18.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.574+0000 7f7c1ce00640 1 -- start start 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c1ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7c180cf9a0 0x7f7c1815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7c180702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c1ce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180b1ee0 con 0x7f7c18143ae0 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c1ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c180b2780 con 0x7f7c180cf9a0 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c1ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1815e990 con 0x7f7c1815e1e0 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c15e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7c18143ae0 0x7f7c180cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42694/0 (socket says 172.21.15.125:42694) 2024-01-16T15:06:18.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c15e00640 1 -- 172.21.15.125:0/2904836812 learned_addr learned my addr 172.21.15.125:0/2904836812 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:18.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c16800640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7c180702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c0d200640 1 --2- 172.21.15.125:0/2904836812 >> v2:172.21.15.137:3300/0 conn(0x7f7c180cf9a0 0x7f7c1815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334163737 0 0) 0x7f7c180b1ee0 con 0x7f7c18143ae0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.576+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8003520 con 0x7f7c18143ae0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3321965195 0 0) 0x7f7bf8003520 con 0x7f7c18143ae0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 msgr2=0x7f7c180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7c180702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 >> v2:172.21.15.137:3300/0 conn(0x7f7c180cf9a0 msgr2=0x7f7c1815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 --2- 172.21.15.125:0/2904836812 >> v2:172.21.15.137:3300/0 conn(0x7f7c180cf9a0 0x7f7c1815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180b2130 con 0x7f7c18143ae0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7c08001670 con 0x7f7c18143ae0 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 >> v1:172.21.15.125:6789/0 conn(0x7f7c18143ae0 legacy=0x7f7c180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7bf8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bf80047c0 0x7f7bf8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7bf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180b2780 con 0x7f7bf8003970 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1815e990 con 0x7f7bf80047c0 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf800db90 con 0x7f7c1815e1e0 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.577+0000 7f7c15e00640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7bf8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c0d200640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bf80047c0 0x7f7bf8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c16800640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7bf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c15e00640 1 -- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 msgr2=0x7f7bf8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c15e00640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7bf8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c15e00640 1 -- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bf80047c0 msgr2=0x7f7bf8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c15e00640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bf80047c0 0x7f7bf8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c15e00640 1 -- 172.21.15.125:0/2904836812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8003520 con 0x7f7bf8003970 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.578+0000 7f7c16800640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c1815e1e0 0x7f7bf8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.579+0000 7f7c15e00640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7bf8003d90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c210 tx=0x7f7bf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409da3c9d862bd9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.579+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0802cae0 con 0x7f7bf8003970 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.579+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0802cc40 con 0x7f7bf8003970 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.579+0000 7f7c15400640 1 -- 172.21.15.125:0/2904836812 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080058d0 con 0x7f7bf8003970 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 -- 172.21.15.125:0/2904836812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 msgr2=0x7f7bf8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/2904836812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7bf8003d90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c210 tx=0x7f7bf80057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c16800640 1 -- 172.21.15.125:0/2904836812 reap_dead start 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 -- 172.21.15.125:0/2904836812 shutdown_connections 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 -- 172.21.15.125:0/2904836812 >> 172.21.15.125:0/2904836812 conn(0x7f7c1806a800 msgr2=0x7f7c1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 -- 172.21.15.125:0/2904836812 shutdown_connections 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.580+0000 7f7c1ce00640 1 -- 172.21.15.125:0/2904836812 wait complete. 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.582+0000 7f7c1ce00640 1 Processor -- start 2024-01-16T15:06:18.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.582+0000 7f7c1ce00640 1 -- start start 2024-01-16T15:06:18.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c180ac7f0 0x7f7c180aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c180aa910 0x7f7c1813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180ab370 con 0x7f7bf8003970 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180aadb0 con 0x7f7c180ac7f0 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c1ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180ab090 con 0x7f7c180aa910 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60494/0 (socket says 172.21.15.125:60494) 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 -- 172.21.15.125:0/1978204662 learned_addr learned my addr 172.21.15.125:0/1978204662 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c0d200640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c180ac7f0 0x7f7c180aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 -- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c180aa910 msgr2=0x7f7c1813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c180aa910 0x7f7c1813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 -- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c180ac7f0 msgr2=0x7f7c180aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c180ac7f0 0x7f7c180aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.583+0000 7f7c15e00640 1 -- 172.21.15.125:0/1978204662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180cd7e0 con 0x7f7bf8003970 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c15e00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c210 tx=0x7f7c180b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08039030 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c08004bf0 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0803c040 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c180cdac0 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.584+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c180b03d0 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.585+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c180b05c0 con 0x7f7bf8003970 2024-01-16T15:06:18.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.586+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f7c08032070 con 0x7f7bf8003970 2024-01-16T15:06:18.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.588+0000 7f7c0ea00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 0x7f7bf006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.588+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7c080a9ac0 con 0x7f7bf8003970 2024-01-16T15:06:18.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.588+0000 7f7c0d200640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 0x7f7bf006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.589+0000 7f7c0d200640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 0x7f7bf006e1d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7c1815c8d0 tx=0x7f7c00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.592+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c08076160 con 0x7f7bf8003970 2024-01-16T15:06:18.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.814+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c181f2370 con 0x7f7bf006bd10 2024-01-16T15:06:18.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.827+0000 7f7c0ea00640 1 -- 172.21.15.125:0/1978204662 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f7c181f2370 con 0x7f7bf006bd10 2024-01-16T15:06:18.805 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:06:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:18.806 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:18.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 msgr2=0x7f7bf006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 0x7f7bf006e1d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7c1815c8d0 tx=0x7f7c00008040 comp rx=0 tx=0).stop 2024-01-16T15:06:18.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 msgr2=0x7f7c180afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c210 tx=0x7f7c180b26f0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 shutdown_connections 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c180aa910 0x7f7c1813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c180ac7f0 0x7f7c180aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7bf006bd10 0x7f7bf006e1d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 --2- 172.21.15.125:0/1978204662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7bf8003970 0x7f7c180afe90 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 >> 172.21.15.125:0/1978204662 conn(0x7f7c1806a800 msgr2=0x7f7c1815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 shutdown_connections 2024-01-16T15:06:18.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:18.829+0000 7f7c1ce00640 1 -- 172.21.15.125:0/1978204662 wait complete. 2024-01-16T15:06:18.825 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:18.949 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:18.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:18.950 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:18.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:18.950 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:18.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:18.950 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:18.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:18.950 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:18.971+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:19.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:06:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.078+0000 7f7f85a00640 1 Processor -- start 2024-01-16T15:06:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.078+0000 7f7f85a00640 1 -- start start 2024-01-16T15:06:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f85a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f85a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7f800cf9a0 0x7f7f800cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f85a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8006c470 con 0x7f7f8013e360 2024-01-16T15:06:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f85a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8006c750 con 0x7f7f800cf9a0 2024-01-16T15:06:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f85a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8006ca30 con 0x7f7f8013df80 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f7ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f7f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7f800cf9a0 0x7f7f800cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f7ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:46962/0 (socket says 172.21.15.125:46962) 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.080+0000 7f7f7ea00640 1 -- 172.21.15.125:0/3039811102 learned_addr learned my addr 172.21.15.125:0/3039811102 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.081+0000 7f7f7f400640 1 -- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 msgr2=0x7f7f8015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.081+0000 7f7f7f400640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.081+0000 7f7f7f400640 1 -- 172.21.15.125:0/3039811102 >> v1:172.21.15.137:6789/0 conn(0x7f7f8013e360 legacy=0x7f7f800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.081+0000 7f7f7f400640 1 -- 172.21.15.125:0/3039811102 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8006cd10 con 0x7f7f800cf9a0 2024-01-16T15:06:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.081+0000 7f7f7ea00640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:20.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.082+0000 7f7f7f400640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f800cf9a0 0x7f7f800cd840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009d90 tx=0x7f7f7802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519ec614d7435b64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.083+0000 7f7f7e000640 1 -- 172.21.15.125:0/3039811102 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7802cc50 con 0x7f7f800cf9a0 2024-01-16T15:06:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.083+0000 7f7f7e000640 1 -- 172.21.15.125:0/3039811102 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7802cdb0 con 0x7f7f800cf9a0 2024-01-16T15:06:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.083+0000 7f7f7e000640 1 -- 172.21.15.125:0/3039811102 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78034660 con 0x7f7f800cf9a0 2024-01-16T15:06:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 -- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f800cf9a0 msgr2=0x7f7f800cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f800cf9a0 0x7f7f800cd840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009d90 tx=0x7f7f7802bea0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 -- 172.21.15.125:0/3039811102 shutdown_connections 2024-01-16T15:06:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f800cf9a0 0x7f7f800cd840 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 --2- 172.21.15.125:0/3039811102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f8013df80 0x7f7f8015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 -- 172.21.15.125:0/3039811102 >> 172.21.15.125:0/3039811102 conn(0x7f7f8006a800 msgr2=0x7f7f8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 -- 172.21.15.125:0/3039811102 shutdown_connections 2024-01-16T15:06:20.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.084+0000 7f7f85a00640 1 -- 172.21.15.125:0/3039811102 wait complete. 2024-01-16T15:06:20.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.086+0000 7f7f85a00640 1 Processor -- start 2024-01-16T15:06:20.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.087+0000 7f7f85a00640 1 -- start start 2024-01-16T15:06:20.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f800cf9a0 0x7f7f801f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f8013e360 0x7f7f8013a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f801f6160 con 0x7f7f8013df80 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8013ab80 con 0x7f7f8013e360 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f85a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8013ae60 con 0x7f7f800cf9a0 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f77e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f77e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60504/0 (socket says 172.21.15.125:60504) 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f77e00640 1 -- 172.21.15.125:0/1872461969 learned_addr learned my addr 172.21.15.125:0/1872461969 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f7ea00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f800cf9a0 0x7f7f801f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.088+0000 7f7f7f400640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f8013e360 0x7f7f8013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 -- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f800cf9a0 msgr2=0x7f7f801f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f800cf9a0 0x7f7f801f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 -- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f8013e360 msgr2=0x7f7f8013a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f8013e360 0x7f7f8013a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 -- 172.21.15.125:0/1872461969 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f700094d0 con 0x7f7f8013df80 2024-01-16T15:06:20.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f77e00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800c400 tx=0x7f7f6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f680052d0 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f68005450 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f78009a70 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68007260 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.089+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8013b420 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.091+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f800675d0 con 0x7f7f8013df80 2024-01-16T15:06:20.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.092+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 24) v1 ==== 90457+0+0 (secure 0 0 0) 0x7f7f680073c0 con 0x7f7f8013df80 2024-01-16T15:06:20.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.092+0000 7f7f76a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 0x7f7f5806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.092+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7f68087b30 con 0x7f7f8013df80 2024-01-16T15:06:20.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.093+0000 7f7f7ea00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 0x7f7f5806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.093+0000 7f7f7ea00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 0x7f7f5806e1a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009cf0 tx=0x7f7f70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.097+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f680541d0 con 0x7f7f8013df80 2024-01-16T15:06:20.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.284+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 25) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f7f680545a0 con 0x7f7f8013df80 2024-01-16T15:06:20.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:20.333+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f8015f9c0 con 0x7f7f5806bce0 2024-01-16T15:06:23.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.953+0000 7f7f76a00640 1 -- 172.21.15.125:0/1872461969 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f7f70009940 con 0x7f7f5806bce0 2024-01-16T15:06:23.931 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:23.932 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 msgr2=0x7f7f5806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 0x7f7f5806e1a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009cf0 tx=0x7f7f70008040 comp rx=0 tx=0).stop 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 msgr2=0x7f7f801391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800c400 tx=0x7f7f6800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 shutdown_connections 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7f8013e360 0x7f7f8013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7f5806bce0 0x7f7f5806e1a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f8013df80 0x7f7f801391a0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 --2- 172.21.15.125:0/1872461969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7f800cf9a0 0x7f7f801f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.957+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 >> 172.21.15.125:0/1872461969 conn(0x7f7f8006a800 msgr2=0x7f7f80070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:23.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.958+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 shutdown_connections 2024-01-16T15:06:23.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:23.958+0000 7f7f85a00640 1 -- 172.21.15.125:0/1872461969 wait complete. 2024-01-16T15:06:23.941 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:23.964+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:23.942 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:23.964+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:23.942 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:23.964+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:23.942 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:23.964+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:23.942 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:23.964+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:23.964 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:24.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:06:25.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.400+0000 7f0094800640 1 Processor -- start 2024-01-16T15:06:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.402+0000 7f0094800640 1 -- start start 2024-01-16T15:06:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c161a80 0x7f008c0c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 0x7f008c0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f008c0cf9c0 0x7f008c0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c161e60 con 0x7f008c0b2620 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c06bd90 con 0x7f008c0cf9c0 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.404+0000 7f0094800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0adcd0 con 0x7f008c161a80 2024-01-16T15:06:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.405+0000 7f0091e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c161a80 0x7f008c0c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.406+0000 7f0092800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f008c0cf9c0 0x7f008c0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.406+0000 7f0092800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f008c0cf9c0 0x7f008c0cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:38226/0 (socket says 172.21.15.125:38226) 2024-01-16T15:06:25.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.406+0000 7f0092800640 1 -- 172.21.15.125:0/450770786 learned_addr learned my addr 172.21.15.125:0/450770786 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:25.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.406+0000 7f0091400640 1 --2- 172.21.15.125:0/450770786 >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 0x7f008c0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0092800640 1 -- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c161a80 msgr2=0x7f008c0c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:25.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0092800640 1 --2- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c161a80 0x7f008c0c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0092800640 1 -- 172.21.15.125:0/450770786 >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 msgr2=0x7f008c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:25.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0092800640 1 --2- 172.21.15.125:0/450770786 >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 0x7f008c0c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0092800640 1 -- 172.21.15.125:0/450770786 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0d1e70 con 0x7f008c0cf9c0 2024-01-16T15:06:25.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.407+0000 7f0091400640 1 --2- 172.21.15.125:0/450770786 >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 0x7f008c0c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:25.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.409+0000 7f0092800640 1 --2- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c0cf9c0 0x7f008c0cfe10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f008c072170 tx=0x7f008402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3425e10cf51d6d8c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:25.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.410+0000 7f0090a00640 1 -- 172.21.15.125:0/450770786 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008402ca80 con 0x7f008c0cf9c0 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.410+0000 7f0090a00640 1 -- 172.21.15.125:0/450770786 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008402cbe0 con 0x7f008c0cf9c0 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.410+0000 7f0090a00640 1 -- 172.21.15.125:0/450770786 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840343f0 con 0x7f008c0cf9c0 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 -- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c0cf9c0 msgr2=0x7f008c0cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 --2- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c0cf9c0 0x7f008c0cfe10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f008c072170 tx=0x7f008402bee0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 -- 172.21.15.125:0/450770786 shutdown_connections 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 --2- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c0cf9c0 0x7f008c0cfe10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 --2- 172.21.15.125:0/450770786 >> v2:172.21.15.125:3300/0 conn(0x7f008c0b2620 0x7f008c0c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 --2- 172.21.15.125:0/450770786 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c161a80 0x7f008c0c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.412+0000 7f0094800640 1 -- 172.21.15.125:0/450770786 >> 172.21.15.125:0/450770786 conn(0x7f008c0d1020 msgr2=0x7f008c0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.413+0000 7f0094800640 1 -- 172.21.15.125:0/450770786 shutdown_connections 2024-01-16T15:06:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.413+0000 7f0094800640 1 -- 172.21.15.125:0/450770786 wait complete. 2024-01-16T15:06:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.416+0000 7f0094800640 1 Processor -- start 2024-01-16T15:06:25.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.417+0000 7f0094800640 1 -- start start 2024-01-16T15:06:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.418+0000 7f0094800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.418+0000 7f0094800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.418+0000 7f0094800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 0x7f008c0baed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.420+0000 7f0091400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.420+0000 7f0091400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:47008/0 (socket says 172.21.15.125:47008) 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.421+0000 7f0094800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1fb970 con 0x7f008c0b2620 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.421+0000 7f0094800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1fad60 con 0x7f008c161a80 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.421+0000 7f0094800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1fafd0 con 0x7f008c0cf9c0 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.421+0000 7f0091e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.422+0000 7f0091e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60522/0 (socket says 172.21.15.125:60522) 2024-01-16T15:06:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.422+0000 7f0091e00640 1 -- 172.21.15.125:0/362429077 learned_addr learned my addr 172.21.15.125:0/362429077 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0092800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 0x7f008c0baed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0091e00640 1 -- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 msgr2=0x7f008c0ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0091e00640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0091400640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0091e00640 1 -- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 msgr2=0x7f008c0baed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.424+0000 7f0091e00640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 0x7f008c0baed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.425+0000 7f0091e00640 1 -- 172.21.15.125:0/362429077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0084009b30 con 0x7f008c0b2620 2024-01-16T15:06:25.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.426+0000 7f0092800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 0x7f008c0baed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:25.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.429+0000 7f0091e00640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e8a0 tx=0x7f007c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:25.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.431+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c0094f0 con 0x7f008c0b2620 2024-01-16T15:06:25.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.431+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c00c940 con 0x7f008c0b2620 2024-01-16T15:06:25.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.431+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c00ccf0 con 0x7f008c0b2620 2024-01-16T15:06:25.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.431+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c1fb2b0 con 0x7f008c0b2620 2024-01-16T15:06:25.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.432+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c0c78a0 con 0x7f008c0b2620 2024-01-16T15:06:25.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.434+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c0675d0 con 0x7f008c0b2620 2024-01-16T15:06:25.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.439+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f007c013470 con 0x7f008c0b2620 2024-01-16T15:06:25.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.440+0000 7f008a000640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 0x7f006406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:25.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.440+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f007c088880 con 0x7f008c0b2620 2024-01-16T15:06:25.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.441+0000 7f0091400640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 0x7f006406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.443+0000 7f0091400640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 0x7f006406e2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0074006fd0 tx=0x7f0074008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.448+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f007c054f50 con 0x7f008c0b2620 2024-01-16T15:06:25.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:25.981+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f008c06bab0 con 0x7f006406be00 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.072+0000 7f008a000640 1 -- 172.21.15.125:0/362429077 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f00740094b0 con 0x7f006406be00 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "in-progress", 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:26.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_report": { 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "percentage completed": 0.0, 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "regfiles copied": "0/3", 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:26.051 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 msgr2=0x7f006406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 0x7f006406e2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0074006fd0 tx=0x7f0074008040 comp rx=0 tx=0).stop 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 msgr2=0x7f008c15d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e8a0 tx=0x7f007c00ed60 comp rx=0 tx=0).stop 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 shutdown_connections 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f008c161a80 0x7f008c0baed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f008c0cf9c0 0x7f008c0ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006406be00 0x7f006406e2c0 secure :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0074006fd0 tx=0x7f0074008040 comp rx=0 tx=0).stop 2024-01-16T15:06:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 --2- 172.21.15.125:0/362429077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f008c0b2620 0x7f008c15d110 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.079+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 >> 172.21.15.125:0/362429077 conn(0x7f008c0d1020 msgr2=0x7f008c0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.080+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 shutdown_connections 2024-01-16T15:06:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.080+0000 7f0094800640 1 -- 172.21.15.125:0/362429077 wait complete. 2024-01-16T15:06:26.098 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:06:26.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96917 2024-01-16T15:06:26.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.499+0000 7fa4a9c00640 1 Processor -- start 2024-01-16T15:06:26.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.499+0000 7fa4a9c00640 1 -- start start 2024-01-16T15:06:26.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4a4150760 0x7fa4a4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa4a415d300 0x7fa4a415d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 0x7fa4a40b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a415dc80 con 0x7fa4a4150760 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a4071190 con 0x7fa4a415d300 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a9c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4071470 con 0x7fa4a40b2bd0 2024-01-16T15:06:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4a4150760 0x7fa4a4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4a4150760 0x7fa4a4072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60538/0 (socket says 172.21.15.125:60538) 2024-01-16T15:06:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.502+0000 7fa4a2a00640 1 -- 172.21.15.125:0/1727953344 learned_addr learned my addr 172.21.15.125:0/1727953344 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.503+0000 7fa4a3400640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 0x7fa4a40b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.503+0000 7fa4a2000640 1 --2- 172.21.15.125:0/1727953344 >> v2:172.21.15.137:3300/0 conn(0x7fa4a415d300 0x7fa4a415d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a2a00640 1 -- 172.21.15.125:0/1727953344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 msgr2=0x7fa4a40b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a2a00640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 0x7fa4a40b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a2a00640 1 -- 172.21.15.125:0/1727953344 >> v2:172.21.15.137:3300/0 conn(0x7fa4a415d300 msgr2=0x7fa4a415d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a2a00640 1 --2- 172.21.15.125:0/1727953344 >> v2:172.21.15.137:3300/0 conn(0x7fa4a415d300 0x7fa4a415d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a2a00640 1 -- 172.21.15.125:0/1727953344 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4071750 con 0x7fa4a4150760 2024-01-16T15:06:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.504+0000 7fa4a3400640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 0x7fa4a40b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:26.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.506+0000 7fa4a2a00640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4150760 0x7fa4a4072230 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40b1ca0 tx=0x7fa49402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc86a319e4644216 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.506+0000 7fa4a1600640 1 -- 172.21.15.125:0/1727953344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49402f8e0 con 0x7fa4a4150760 2024-01-16T15:06:26.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.506+0000 7fa4a1600640 1 -- 172.21.15.125:0/1727953344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49402fa40 con 0x7fa4a4150760 2024-01-16T15:06:26.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.506+0000 7fa4a1600640 1 -- 172.21.15.125:0/1727953344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49402fdd0 con 0x7fa4a4150760 2024-01-16T15:06:26.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.507+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1727953344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4150760 msgr2=0x7fa4a4072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.507+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4150760 0x7fa4a4072230 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40b1ca0 tx=0x7fa49402db20 comp rx=0 tx=0).stop 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.509+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1727953344 shutdown_connections 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.509+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a40b2bd0 0x7fa4a40b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.509+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1727953344 >> v2:172.21.15.137:3300/0 conn(0x7fa4a415d300 0x7fa4a415d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.509+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1727953344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4150760 0x7fa4a4072230 secure :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40b1ca0 tx=0x7fa49402db20 comp rx=0 tx=0).stop 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.509+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1727953344 >> 172.21.15.125:0/1727953344 conn(0x7fa4a406a800 msgr2=0x7fa4a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.512+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1727953344 shutdown_connections 2024-01-16T15:06:26.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.515+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1727953344 wait complete. 2024-01-16T15:06:26.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.520+0000 7fa4a9c00640 1 Processor -- start 2024-01-16T15:06:26.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.521+0000 7fa4a9c00640 1 -- start start 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4a40b2bd0 0x7fa4a414a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a415d300 0x7fa4a40ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40cc7a0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41491e0 con 0x7fa4a40b2bd0 2024-01-16T15:06:26.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.524+0000 7fa4a9c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40cbe00 con 0x7fa4a415d300 2024-01-16T15:06:26.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.525+0000 7fa4a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.525+0000 7fa4a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60552/0 (socket says 172.21.15.125:60552) 2024-01-16T15:06:26.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.525+0000 7fa4a3400640 1 -- 172.21.15.125:0/1021127971 learned_addr learned my addr 172.21.15.125:0/1021127971 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:26.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 -- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a415d300 msgr2=0x7fa4a40ccbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:26.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a415d300 0x7fa4a40ccbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 -- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4a40b2bd0 msgr2=0x7fa4a414a190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:26.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4a40b2bd0 0x7fa4a414a190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 -- 172.21.15.125:0/1021127971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940094d0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.526+0000 7fa4a3400640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007cf0 tx=0x7fa49c00d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.527+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00f040 con 0x7fa4a4145fb0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.527+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c00dda0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.527+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c003b90 con 0x7fa4a4145fb0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.527+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a40cc0e0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.527+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a414a6d0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.531+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa474005250 con 0x7fa4a4145fb0 2024-01-16T15:06:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.552+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fa49c003cf0 con 0x7fa4a4145fb0 2024-01-16T15:06:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.554+0000 7fa49aa00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 0x7fa47c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.554+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fa49c089430 con 0x7fa4a4145fb0 2024-01-16T15:06:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.554+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa49c0b7f30 con 0x7fa4a4145fb0 2024-01-16T15:06:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.556+0000 7fa4a2a00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 0x7fa47c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:26.558+0000 7fa4a2a00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 0x7fa47c06e390 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa494038000 tx=0x7fa494008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.089+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96917", "target": ["mon-mgr", ""]}) v1 -- 0x7fa474000bc0 con 0x7fa47c06bed0 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.139+0000 7fa49aa00640 1 -- 172.21.15.125:0/1021127971 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fa474000bc0 con 0x7fa47c06bed0 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:06:27.117 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:27.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.147+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 msgr2=0x7fa47c06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.147+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 0x7fa47c06e390 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa494038000 tx=0x7fa494008030 comp rx=0 tx=0).stop 2024-01-16T15:06:27.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.147+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 msgr2=0x7fa4a40cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.147+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007cf0 tx=0x7fa49c00d490 comp rx=0 tx=0).stop 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.147+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 shutdown_connections 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.148+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa47c06bed0 0x7fa47c06e390 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.148+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a4145fb0 0x7fa4a40cd100 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.148+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4a415d300 0x7fa4a40ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.148+0000 7fa4a9c00640 1 --2- 172.21.15.125:0/1021127971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4a40b2bd0 0x7fa4a414a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.148+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 >> 172.21.15.125:0/1021127971 conn(0x7fa4a406a800 msgr2=0x7fa4a406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.149+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 shutdown_connections 2024-01-16T15:06:27.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.150+0000 7fa4a9c00640 1 -- 172.21.15.125:0/1021127971 wait complete. 2024-01-16T15:06:27.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.435+0000 7f13dd000640 1 Processor -- start 2024-01-16T15:06:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.435+0000 7f13dd000640 1 -- start start 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.436+0000 7f13dd000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13d80cfcf0 0x7f13d8160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13dd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d8160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13dd000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80d4160 con 0x7f13d80d3910 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13dd000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80ceff0 con 0x7f13d80cfcf0 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13dd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80cf410 con 0x7f13d80d00d0 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13d80cfcf0 0x7f13d8160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13d80cfcf0 0x7f13d8160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56054/0 (socket says 172.21.15.125:56054) 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 -- 172.21.15.125:0/795249944 learned_addr learned my addr 172.21.15.125:0/795249944 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5400640 1 --1- 172.21.15.125:0/795249944 >> v1:172.21.15.125:6789/0 conn(0x7f13d80d3910 0x7f13d80b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35286/0 (socket says 172.21.15.125:35286) 2024-01-16T15:06:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d6800640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d8160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 -- 172.21.15.125:0/795249944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 msgr2=0x7f13d8160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d8160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 -- 172.21.15.125:0/795249944 >> v1:172.21.15.125:6789/0 conn(0x7f13d80d3910 legacy=0x7f13d80b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d5e00640 1 -- 172.21.15.125:0/795249944 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80b2420 con 0x7f13d80cfcf0 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d6800640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d8160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.437+0000 7f13d4a00640 1 -- 172.21.15.125:0/795249944 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894731668 0 0) 0x7f13d80d4160 con 0x7f13d80d3910 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.438+0000 7f13d5e00640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80cfcf0 0x7f13d8160700 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009e80 tx=0x7f13d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac17acf3dc5b983f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13d4a00640 1 -- 172.21.15.125:0/795249944 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d002ca50 con 0x7f13d80cfcf0 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13d4a00640 1 -- 172.21.15.125:0/795249944 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d002cbb0 con 0x7f13d80cfcf0 2024-01-16T15:06:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13d4a00640 1 -- 172.21.15.125:0/795249944 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d00343f0 con 0x7f13d80cfcf0 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 -- 172.21.15.125:0/795249944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80cfcf0 msgr2=0x7f13d8160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80cfcf0 0x7f13d8160700 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009e80 tx=0x7f13d002beb0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 -- 172.21.15.125:0/795249944 shutdown_connections 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d8160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 --2- 172.21.15.125:0/795249944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80cfcf0 0x7f13d8160700 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 -- 172.21.15.125:0/795249944 >> 172.21.15.125:0/795249944 conn(0x7f13d809c530 msgr2=0x7f13d80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 -- 172.21.15.125:0/795249944 shutdown_connections 2024-01-16T15:06:27.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.439+0000 7f13dd000640 1 -- 172.21.15.125:0/795249944 wait complete. 2024-01-16T15:06:27.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.441+0000 7f13dd000640 1 Processor -- start 2024-01-16T15:06:27.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.441+0000 7f13dd000640 1 -- start start 2024-01-16T15:06:27.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d80cfcf0 0x7f13d80b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d80ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80b6a60 con 0x7f13d80d3910 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80caea0 con 0x7f13d80cfcf0 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13dd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80cb180 con 0x7f13d80d00d0 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13d6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13d6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56062/0 (socket says 172.21.15.125:56062) 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.442+0000 7f13d6800640 1 -- 172.21.15.125:0/2702223681 learned_addr learned my addr 172.21.15.125:0/2702223681 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d5400640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d80ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d5e00640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d80cfcf0 0x7f13d80b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 -- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 msgr2=0x7f13d80ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d80ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 -- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d80cfcf0 msgr2=0x7f13d80b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d80cfcf0 0x7f13d80b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 -- 172.21.15.125:0/2702223681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0009b30 con 0x7f13d80d3910 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d5400640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d80ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13d6800640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c7f0 tx=0x7f13c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8004220 con 0x7f13d80d3910 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c80043a0 con 0x7f13d80d3910 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8007860 con 0x7f13d80d3910 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d80cb460 con 0x7f13d80d3910 2024-01-16T15:06:27.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.443+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d80caa80 con 0x7f13d80d3910 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.444+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d80cb5f0 con 0x7f13d80d3910 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.445+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f13c8007bd0 con 0x7f13d80d3910 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.446+0000 7f13ce000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 0x7f13b006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.446+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f13c8088790 con 0x7f13d80d3910 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.446+0000 7f13d5e00640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 0x7f13b006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.447+0000 7f13d5e00640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 0x7f13b006e2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f13d0035000 tx=0x7f13d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.451+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13c8054dd0 con 0x7f13d80d3910 2024-01-16T15:06:27.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.694+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f13d80675d0 con 0x7f13d80d3910 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.695+0000 7f13ce000640 1 -- 172.21.15.125:0/2702223681 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2343 (secure 0 0 0) 0x7f13c801e940 con 0x7f13d80d3910 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 606, 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 808, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3239236564, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6842425344, 2024-01-16T15:06:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766251687936, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 9077502, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 1421497, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 3, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 7 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0, 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:27.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.697+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 msgr2=0x7f13b006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.697+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 0x7f13b006e2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f13d0035000 tx=0x7f13d0030000 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.697+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 msgr2=0x7f13d80b7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.697+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c7f0 tx=0x7f13c800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 shutdown_connections 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13b006be30 0x7f13b006e2f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13d80d3910 0x7f13d80b7170 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d80d00d0 0x7f13d80ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 --2- 172.21.15.125:0/2702223681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d80cfcf0 0x7f13d80b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 >> 172.21.15.125:0/2702223681 conn(0x7f13d809c530 msgr2=0x7f13d80b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 shutdown_connections 2024-01-16T15:06:27.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:27.698+0000 7f13dd000640 1 -- 172.21.15.125:0/2702223681 wait complete. 2024-01-16T15:06:28.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:28.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.926+0000 7f1cd8c00640 1 Processor -- start 2024-01-16T15:06:28.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.926+0000 7f1cd8c00640 1 -- start start 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd8c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cd414b2c0 0x7f1cd40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 0x7f1cd4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd8c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40cfbf0 con 0x7f1cd4070ea0 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd8c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd414b6a0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4071690 con 0x7f1cd40cf7d0 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cd414b2c0 0x7f1cd40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cd414b2c0 0x7f1cd40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56072/0 (socket says 172.21.15.125:56072) 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 -- 172.21.15.125:0/807321663 learned_addr learned my addr 172.21.15.125:0/807321663 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1ccbe00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 0x7f1cd4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 -- 172.21.15.125:0/807321663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 msgr2=0x7f1cd4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:28.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 0x7f1cd4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 -- 172.21.15.125:0/807321663 >> v1:172.21.15.137:6789/0 conn(0x7f1cd4070ea0 legacy=0x7f1cd40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:28.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1cd1e00640 1 -- 172.21.15.125:0/807321663 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4071970 con 0x7f1cd414b2c0 2024-01-16T15:06:28.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.928+0000 7f1ccbe00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 0x7f1cd4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:28.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd1e00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd40cf290 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009870 tx=0x7f1cc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e28e221c38934d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd1400640 1 -- 172.21.15.125:0/807321663 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4031640 con 0x7f1cd414b2c0 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd1400640 1 -- 172.21.15.125:0/807321663 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc40317a0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd1400640 1 -- 172.21.15.125:0/807321663 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4031b30 con 0x7f1cd414b2c0 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 -- 172.21.15.125:0/807321663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 msgr2=0x7f1cd40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd40cf290 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009870 tx=0x7f1cc402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 -- 172.21.15.125:0/807321663 shutdown_connections 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd40cf7d0 0x7f1cd4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/807321663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd40cf290 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.930+0000 7f1cd8c00640 1 -- 172.21.15.125:0/807321663 >> 172.21.15.125:0/807321663 conn(0x7f1cd406a800 msgr2=0x7f1cd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.931+0000 7f1cd8c00640 1 -- 172.21.15.125:0/807321663 shutdown_connections 2024-01-16T15:06:28.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.931+0000 7f1cd8c00640 1 -- 172.21.15.125:0/807321663 wait complete. 2024-01-16T15:06:28.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.933+0000 7f1cd8c00640 1 Processor -- start 2024-01-16T15:06:28.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.933+0000 7f1cd8c00640 1 -- start start 2024-01-16T15:06:28.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd4070ea0 0x7f1cd406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cd40cf7d0 0x7f1cd415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4146120 con 0x7f1cd414b2c0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4145df0 con 0x7f1cd40cf7d0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4164e50 con 0x7f1cd4070ea0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56074/0 (socket says 172.21.15.125:56074) 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd2800640 1 -- 172.21.15.125:0/2044544455 learned_addr learned my addr 172.21.15.125:0/2044544455 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.934+0000 7f1cd1e00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd4070ea0 0x7f1cd406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 -- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd4070ea0 msgr2=0x7f1cd406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd4070ea0 0x7f1cd406c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1ccbe00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cd40cf7d0 0x7f1cd415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 -- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cd40cf7d0 msgr2=0x7f1cd415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cd40cf7d0 0x7f1cd415e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 -- 172.21.15.125:0/2044544455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40094d0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd2800640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c780 tx=0x7f1ccc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc004b60 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc00ce90 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd41645b0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00a850 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.935+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd4164b70 con 0x7f1cd414b2c0 2024-01-16T15:06:28.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.937+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4146260 con 0x7f1cd414b2c0 2024-01-16T15:06:28.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.937+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f1ccc00a9b0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.939+0000 7f1cca000640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 0x7f1cac06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.939+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1ccc0888d0 con 0x7f1cd414b2c0 2024-01-16T15:06:28.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.939+0000 7f1cd1e00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 0x7f1cac06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.940+0000 7f1cd1e00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 0x7f1cac06e2c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4036000 tx=0x7f1cc403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:28.944+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ccc054fa0 con 0x7f1cd414b2c0 2024-01-16T15:06:29.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.184+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1cd40af840 con 0x7f1cd414b2c0 2024-01-16T15:06:29.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.184+0000 7f1cca000640 1 -- 172.21.15.125:0/2044544455 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2418 (secure 0 0 0) 0x7f1ccc0585c0 con 0x7f1cd414b2c0 2024-01-16T15:06:29.162 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:29.162 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:29.163 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 608, 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.164 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:29.165 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 808, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3239236564, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 6842425344, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766251687936, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 9076713, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 1411554, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 2, 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 5 2024-01-16T15:06:29.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:29.167 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:29.168 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:29.169 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:29.170 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.186+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 msgr2=0x7f1cac06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.186+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 0x7f1cac06e2c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4036000 tx=0x7f1cc403c040 comp rx=0 tx=0).stop 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.186+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 msgr2=0x7f1cd415e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.186+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c780 tx=0x7f1ccc00cc40 comp rx=0 tx=0).stop 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 shutdown_connections 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1cac06be00 0x7f1cac06e2c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd414b2c0 0x7f1cd415e5e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cd40cf7d0 0x7f1cd415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 --2- 172.21.15.125:0/2044544455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1cd4070ea0 0x7f1cd406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 >> 172.21.15.125:0/2044544455 conn(0x7f1cd406a800 msgr2=0x7f1cd40aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:29.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 shutdown_connections 2024-01-16T15:06:29.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:29.187+0000 7f1cd8c00640 1 -- 172.21.15.125:0/2044544455 wait complete. 2024-01-16T15:06:30.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.422+0000 7fcf3fe00640 1 Processor -- start 2024-01-16T15:06:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.422+0000 7fcf3fe00640 1 -- start start 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.423+0000 7fcf3fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcf3813df80 0x7fcf3815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.423+0000 7fcf3fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf3813e360 0x7fcf380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3fe00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3806c470 con 0x7fcf3815f9c0 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3fe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3806c750 con 0x7fcf3813e360 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3806ca30 con 0x7fcf3813df80 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf3813e360 0x7fcf380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf3813e360 0x7fcf380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56086/0 (socket says 172.21.15.125:56086) 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 -- 172.21.15.125:0/3859523004 learned_addr learned my addr 172.21.15.125:0/3859523004 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcf3815f9c0 0x7fcf381408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35300/0 (socket says 172.21.15.125:35300) 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3d400640 1 --2- 172.21.15.125:0/3859523004 >> v2:172.21.15.137:3300/0 conn(0x7fcf3813df80 0x7fcf3815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 -- 172.21.15.125:0/3859523004 >> v2:172.21.15.137:3300/0 conn(0x7fcf3813df80 msgr2=0x7fcf3815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 --2- 172.21.15.125:0/3859523004 >> v2:172.21.15.137:3300/0 conn(0x7fcf3813df80 0x7fcf3815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.424+0000 7fcf3ca00640 1 -- 172.21.15.125:0/3859523004 >> v1:172.21.15.125:6789/0 conn(0x7fcf3815f9c0 legacy=0x7fcf381408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.425+0000 7fcf3ca00640 1 -- 172.21.15.125:0/3859523004 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3806cd10 con 0x7fcf3813e360 2024-01-16T15:06:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.425+0000 7fcf37e00640 1 -- 172.21.15.125:0/3859523004 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966291468 0 0) 0x7fcf3806c470 con 0x7fcf3815f9c0 2024-01-16T15:06:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf3ca00640 1 --2- 172.21.15.125:0/3859523004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813e360 0x7fcf380cf290 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcf38072770 tx=0x7fcf2002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bf477e74e2cc3d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf37e00640 1 -- 172.21.15.125:0/3859523004 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20030860 con 0x7fcf3813e360 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf37e00640 1 -- 172.21.15.125:0/3859523004 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf200309c0 con 0x7fcf3813e360 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf37e00640 1 -- 172.21.15.125:0/3859523004 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20030d50 con 0x7fcf3813e360 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf3fe00640 1 -- 172.21.15.125:0/3859523004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813e360 msgr2=0x7fcf380cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.426+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/3859523004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813e360 0x7fcf380cf290 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcf38072770 tx=0x7fcf2002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 -- 172.21.15.125:0/3859523004 shutdown_connections 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/3859523004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813e360 0x7fcf380cf290 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/3859523004 >> v2:172.21.15.137:3300/0 conn(0x7fcf3813df80 0x7fcf3815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 -- 172.21.15.125:0/3859523004 >> 172.21.15.125:0/3859523004 conn(0x7fcf3806a800 msgr2=0x7fcf3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 -- 172.21.15.125:0/3859523004 shutdown_connections 2024-01-16T15:06:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.427+0000 7fcf3fe00640 1 -- 172.21.15.125:0/3859523004 wait complete. 2024-01-16T15:06:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.429+0000 7fcf3fe00640 1 Processor -- start 2024-01-16T15:06:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.429+0000 7fcf3fe00640 1 -- start start 2024-01-16T15:06:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf3813e360 0x7fcf38159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3815f9c0 0x7fcf3815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3814c5e0 con 0x7fcf3813df80 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3814ba80 con 0x7fcf3813e360 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3814bd40 con 0x7fcf3815f9c0 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.430+0000 7fcf3d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56094/0 (socket says 172.21.15.125:56094) 2024-01-16T15:06:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 -- 172.21.15.125:0/4009088604 learned_addr learned my addr 172.21.15.125:0/4009088604 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 -- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3815f9c0 msgr2=0x7fcf3815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3815f9c0 0x7fcf3815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3ca00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf3813e360 0x7fcf38159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 -- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf3813e360 msgr2=0x7fcf38159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf3813e360 0x7fcf38159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 -- 172.21.15.125:0/4009088604 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf200094d0 con 0x7fcf3813df80 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.431+0000 7fcf3d400640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800d5f0 tx=0x7fcf2800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.432+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280125f0 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.432+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28012750 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.432+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28012b00 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.432+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3814c020 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.432+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3814c7a0 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.433+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380675d0 con 0x7fcf3813df80 2024-01-16T15:06:30.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.434+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fcf28012c60 con 0x7fcf3813df80 2024-01-16T15:06:30.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.435+0000 7fcf35600640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 0x7fcf1006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.435+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcf28088a80 con 0x7fcf3813df80 2024-01-16T15:06:30.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.435+0000 7fcf3ca00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 0x7fcf1006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.435+0000 7fcf3ca00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 0x7fcf1006e2c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcf20036000 tx=0x7fcf2003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.439+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf28055300 con 0x7fcf3813df80 2024-01-16T15:06:30.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.698+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcf380702c0 con 0x7fcf3813df80 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.699+0000 7fcf35600640 1 -- 172.21.15.125:0/4009088604 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fcf28058920 con 0x7fcf3813df80 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.677 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 609, 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:30.678 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.679 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1315, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5369978754, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141828608, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952284672, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:30.680 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 178736913, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 178754640, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 49 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:30.681 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:30.682 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:30.683 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:30.684 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.701+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 msgr2=0x7fcf1006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.701+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 0x7fcf1006e2c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcf20036000 tx=0x7fcf2003b040 comp rx=0 tx=0).stop 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.701+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 msgr2=0x7fcf3813d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.701+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800d5f0 tx=0x7fcf2800dab0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 shutdown_connections 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3815f9c0 0x7fcf3815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf3813e360 0x7fcf38159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf1006be00 0x7fcf1006e2c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 --2- 172.21.15.125:0/4009088604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf3813df80 0x7fcf3813d960 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 >> 172.21.15.125:0/4009088604 conn(0x7fcf3806a800 msgr2=0x7fcf38153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 shutdown_connections 2024-01-16T15:06:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:30.702+0000 7fcf3fe00640 1 -- 172.21.15.125:0/4009088604 wait complete. 2024-01-16T15:06:31.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:31.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.939+0000 7f484bc00640 1 Processor -- start 2024-01-16T15:06:31.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.940+0000 7f484bc00640 1 -- start start 2024-01-16T15:06:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.941+0000 7f484bc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f484413df80 0x7f484415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.941+0000 7f484bc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484406c470 con 0x7f48440cf9a0 2024-01-16T15:06:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.941+0000 7f484bc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484406c750 con 0x7f484413e360 2024-01-16T15:06:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.941+0000 7f484bc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484406ca30 con 0x7f484413df80 2024-01-16T15:06:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4849c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f48440cf9a0 0x7f48440cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35306/0 (socket says 172.21.15.125:35306) 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4849c00640 1 -- 172.21.15.125:0/1570041717 learned_addr learned my addr 172.21.15.125:0/1570041717 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4849200640 1 --2- 172.21.15.125:0/1570041717 >> v2:172.21.15.137:3300/0 conn(0x7f484413df80 0x7f484415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004724874 0 0) 0x7f484406c470 con 0x7f48440cf9a0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4824003520 con 0x7f48440cf9a0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677473528 0 0) 0x7f4824003520 con 0x7f48440cf9a0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 >> v2:172.21.15.137:3300/0 conn(0x7f484413df80 msgr2=0x7f484415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 --2- 172.21.15.125:0/1570041717 >> v2:172.21.15.137:3300/0 conn(0x7f484413df80 0x7f484415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 >> v1:172.21.15.137:6789/0 conn(0x7f484413e360 legacy=0x7f48440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484406cd10 con 0x7f48440cf9a0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.942+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f483c0019f0 con 0x7f48440cf9a0 2024-01-16T15:06:31.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 >> v1:172.21.15.125:6789/0 conn(0x7f48440cf9a0 legacy=0x7f48440cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4824003970 0x7f4824003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f4824004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4824005060 0x7f482400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484406ca30 con 0x7f4824004390 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482400bfa0 con 0x7f4824005060 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482400c160 con 0x7f4824003970 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f4824004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 -- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4824003970 msgr2=0x7f4824003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4824003970 0x7f4824003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 -- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4824005060 msgr2=0x7f482400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4824005060 0x7f482400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f483bc00640 1 -- 172.21.15.125:0/1570041717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824003520 con 0x7f4824004390 2024-01-16T15:06:31.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.943+0000 7f4849c00640 1 -- 172.21.15.125:0/1570041717 reap_dead start 2024-01-16T15:06:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.944+0000 7f483bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f4824004860 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f482c001750 tx=0x7f48240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3b702bf3974c1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0052d0 con 0x7f4824004390 2024-01-16T15:06:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c02e050 con 0x7f4824004390 2024-01-16T15:06:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f4843e00640 1 -- 172.21.15.125:0/1570041717 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c003850 con 0x7f4824004390 2024-01-16T15:06:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 -- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 msgr2=0x7f4824004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f4824004860 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f482c001750 tx=0x7f48240057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 -- 172.21.15.125:0/1570041717 shutdown_connections 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 --2- 172.21.15.125:0/1570041717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f4824004860 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 -- 172.21.15.125:0/1570041717 >> 172.21.15.125:0/1570041717 conn(0x7f484406a800 msgr2=0x7f484406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.945+0000 7f484bc00640 1 -- 172.21.15.125:0/1570041717 shutdown_connections 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.946+0000 7f484bc00640 1 -- 172.21.15.125:0/1570041717 wait complete. 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.947+0000 7f484bc00640 1 Processor -- start 2024-01-16T15:06:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.947+0000 7f484bc00640 1 -- start start 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48440c4390 0x7f4844147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4844148420 0x7f4844149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48440c4810 con 0x7f4824004390 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48441488a0 con 0x7f4844148420 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.948+0000 7f484bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4844156810 con 0x7f48440c4390 2024-01-16T15:06:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56118/0 (socket says 172.21.15.125:56118) 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 -- 172.21.15.125:0/703320640 learned_addr learned my addr 172.21.15.125:0/703320640 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 -- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48440c4390 msgr2=0x7f4844147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48440c4390 0x7f4844147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 -- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4844148420 msgr2=0x7f4844149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4844148420 0x7f4844149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 -- 172.21.15.125:0/703320640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844148ac0 con 0x7f4824004390 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.949+0000 7f4849200640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f483400e7d0 tx=0x7f483400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.950+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483400ee80 con 0x7f4824004390 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.950+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483400c8e0 con 0x7f4824004390 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.950+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483400cc90 con 0x7f4824004390 2024-01-16T15:06:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.950+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4844148da0 con 0x7f4824004390 2024-01-16T15:06:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.950+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4844149360 con 0x7f4824004390 2024-01-16T15:06:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.951+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48440675d0 con 0x7f4824004390 2024-01-16T15:06:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.952+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f4834013470 con 0x7f4824004390 2024-01-16T15:06:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.953+0000 7f4841600640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 0x7f481c06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.953+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f4834088730 con 0x7f4824004390 2024-01-16T15:06:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.953+0000 7f483bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 0x7f481c06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:31.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.954+0000 7f483bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 0x7f481c06e2c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f482c033000 tx=0x7f482c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:31.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:31.958+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4834054e00 con 0x7f4824004390 2024-01-16T15:06:32.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.208+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f484406df00 con 0x7f4824004390 2024-01-16T15:06:32.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.209+0000 7f4841600640 1 -- 172.21.15.125:0/703320640 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f4834017990 con 0x7f4824004390 2024-01-16T15:06:32.187 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:32.187 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:32.188 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 611, 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:32.189 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:32.190 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:32.191 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1315, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5369978754, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141828608, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952284672, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 178736913, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 178754640, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 49 2024-01-16T15:06:32.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:32.193 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:32.194 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:32.195 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:32.196 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:32.197 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 msgr2=0x7f481c06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 0x7f481c06e2c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f482c033000 tx=0x7f482c02f000 comp rx=0 tx=0).stop 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 msgr2=0x7f48440c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f483400e7d0 tx=0x7f483400ec90 comp rx=0 tx=0).stop 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 shutdown_connections 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4844148420 0x7f4844149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f48440c4390 0x7f4844147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f481c06be00 0x7f481c06e2c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 --2- 172.21.15.125:0/703320640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4824004390 0x7f48440c3e50 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.212+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 >> 172.21.15.125:0/703320640 conn(0x7f484406a800 msgr2=0x7f48440af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:32.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.213+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 shutdown_connections 2024-01-16T15:06:32.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:32.213+0000 7f484bc00640 1 -- 172.21.15.125:0/703320640 wait complete. 2024-01-16T15:06:33.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:33.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.448+0000 7f0488400640 1 Processor -- start 2024-01-16T15:06:33.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.448+0000 7f0488400640 1 -- start start 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0488400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04800cf9a0 0x7f048015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0488400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048015e1e0 0x7f04800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0488400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800b1ee0 con 0x7f0480143ae0 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0488400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04800b2780 con 0x7f04800cf9a0 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0488400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048015e990 con 0x7f048015e1e0 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0485000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04800cf9a0 0x7f048015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0485000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04800cf9a0 0x7f048015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56134/0 (socket says 172.21.15.125:56134) 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0485000640 1 -- 172.21.15.125:0/3867247434 learned_addr learned my addr 172.21.15.125:0/3867247434 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0486400640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048015e1e0 0x7f04800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0485000640 1 -- 172.21.15.125:0/3867247434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048015e1e0 msgr2=0x7f04800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:33.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.450+0000 7f0485000640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048015e1e0 0x7f04800702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.451+0000 7f0485000640 1 -- 172.21.15.125:0/3867247434 >> v1:172.21.15.137:6789/0 conn(0x7f0480143ae0 legacy=0x7f04800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:33.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.451+0000 7f0485000640 1 -- 172.21.15.125:0/3867247434 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800b2130 con 0x7f04800cf9a0 2024-01-16T15:06:33.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f0485000640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04800cf9a0 0x7f048015dca0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f0468009870 tx=0x7f046802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894e52cac503e66b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f047fe00640 1 -- 172.21.15.125:0/3867247434 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468031640 con 0x7f04800cf9a0 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f047fe00640 1 -- 172.21.15.125:0/3867247434 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04680317a0 con 0x7f04800cf9a0 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f047fe00640 1 -- 172.21.15.125:0/3867247434 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468031b30 con 0x7f04800cf9a0 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f0488400640 1 -- 172.21.15.125:0/3867247434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04800cf9a0 msgr2=0x7f048015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.452+0000 7f0488400640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04800cf9a0 0x7f048015dca0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f0468009870 tx=0x7f046802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 -- 172.21.15.125:0/3867247434 shutdown_connections 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048015e1e0 0x7f04800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 --2- 172.21.15.125:0/3867247434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04800cf9a0 0x7f048015dca0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 -- 172.21.15.125:0/3867247434 >> 172.21.15.125:0/3867247434 conn(0x7f048006a800 msgr2=0x7f048006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:33.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 -- 172.21.15.125:0/3867247434 shutdown_connections 2024-01-16T15:06:33.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.453+0000 7f0488400640 1 -- 172.21.15.125:0/3867247434 wait complete. 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.455+0000 7f0488400640 1 Processor -- start 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.455+0000 7f0488400640 1 -- start start 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 0x7f048013fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048015e1e0 0x7f048006cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04801f30d0 con 0x7f0480143ae0 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480142220 con 0x7f048015e1e0 2024-01-16T15:06:33.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0488400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04801401e0 con 0x7f04800cf9a0 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56150/0 (socket says 172.21.15.125:56150) 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 -- 172.21.15.125:0/233366042 learned_addr learned my addr 172.21.15.125:0/233366042 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485a00640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 0x7f048013fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 -- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 msgr2=0x7f048013fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 0x7f048013fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 -- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048015e1e0 msgr2=0x7f048006cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0486400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048015e1e0 0x7f048006cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048015e1e0 0x7f048006cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485000640 1 -- 172.21.15.125:0/233366042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680094d0 con 0x7f0480143ae0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.456+0000 7f0485a00640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 0x7f048013fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f0485000640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0468009870 tx=0x7f04680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046803ac60 con 0x7f0480143ae0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046803ade0 con 0x7f0480143ae0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468044630 con 0x7f0480143ae0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04801404c0 con 0x7f0480143ae0 2024-01-16T15:06:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.457+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0480140a80 con 0x7f0480143ae0 2024-01-16T15:06:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.458+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04801f3210 con 0x7f0480143ae0 2024-01-16T15:06:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.459+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f046802f050 con 0x7f0480143ae0 2024-01-16T15:06:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.460+0000 7f047d600640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 0x7f045806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.460+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f04680aa7b0 con 0x7f0480143ae0 2024-01-16T15:06:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.460+0000 7f0485a00640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 0x7f045806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.461+0000 7f0485a00640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 0x7f045806e2c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0470004520 tx=0x7f0470009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.464+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04801f3210 con 0x7f0480143ae0 2024-01-16T15:06:33.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.721+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f04801f3210 con 0x7f0480143ae0 2024-01-16T15:06:33.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.722+0000 7f047d600640 1 -- 172.21.15.125:0/233366042 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f0468035990 con 0x7f0480143ae0 2024-01-16T15:06:33.700 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:33.701 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 612, 2024-01-16T15:06:33.702 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:33.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:33.704 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1315, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5369978754, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141828608, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952284672, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 178730425, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 178740055, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:33.705 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 46 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:33.706 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:33.707 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:33.708 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:33.709 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:33.710 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:33.710 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:33.710 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:33.710 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:33.710 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.724+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 msgr2=0x7f045806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 0x7f045806e2c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0470004520 tx=0x7f0470009040 comp rx=0 tx=0).stop 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 msgr2=0x7f0480141ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0468009870 tx=0x7f04680046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 shutdown_connections 2024-01-16T15:06:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048015e1e0 0x7f048006cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f045806be00 0x7f045806e2c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0480143ae0 0x7f0480141ce0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 --2- 172.21.15.125:0/233366042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04800cf9a0 0x7f048013fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 >> 172.21.15.125:0/233366042 conn(0x7f048006a800 msgr2=0x7f048015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 shutdown_connections 2024-01-16T15:06:33.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:33.725+0000 7f0488400640 1 -- 172.21.15.125:0/233366042 wait complete. 2024-01-16T15:06:34.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:35.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.109+0000 7f569e600640 1 Processor -- start 2024-01-16T15:06:35.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.109+0000 7f569e600640 1 -- start start 2024-01-16T15:06:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f569813df80 0x7f569815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f569813e360 0x7f56980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569815f9c0 0x7f56980cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569806c470 con 0x7f569813e360 2024-01-16T15:06:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569806c750 con 0x7f569813df80 2024-01-16T15:06:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f569e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569806ca30 con 0x7f569815f9c0 2024-01-16T15:06:35.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.112+0000 7f5696a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f569813e360 0x7f56980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f569813e360 0x7f56980cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56166/0 (socket says 172.21.15.125:56166) 2024-01-16T15:06:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 -- 172.21.15.125:0/4242573243 learned_addr learned my addr 172.21.15.125:0/4242573243 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5697400640 1 --2- 172.21.15.125:0/4242573243 >> v2:172.21.15.137:3300/0 conn(0x7f569813df80 0x7f569815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5697e00640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569815f9c0 0x7f56980cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 -- 172.21.15.125:0/4242573243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569815f9c0 msgr2=0x7f56980cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569815f9c0 0x7f56980cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 -- 172.21.15.125:0/4242573243 >> v2:172.21.15.137:3300/0 conn(0x7f569813df80 msgr2=0x7f569815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.113+0000 7f5696a00640 1 --2- 172.21.15.125:0/4242573243 >> v2:172.21.15.137:3300/0 conn(0x7f569813df80 0x7f569815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.114+0000 7f5696a00640 1 -- 172.21.15.125:0/4242573243 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569806cd10 con 0x7f569813e360 2024-01-16T15:06:35.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.116+0000 7f5696a00640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569813e360 0x7f56980cf290 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5698072770 tx=0x7f568002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37721ee36ccc146f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.117+0000 7f5696000640 1 -- 172.21.15.125:0/4242573243 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680031860 con 0x7f569813e360 2024-01-16T15:06:35.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.117+0000 7f5696000640 1 -- 172.21.15.125:0/4242573243 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56800319c0 con 0x7f569813e360 2024-01-16T15:06:35.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.117+0000 7f5696000640 1 -- 172.21.15.125:0/4242573243 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680031d50 con 0x7f569813e360 2024-01-16T15:06:35.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 -- 172.21.15.125:0/4242573243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569813e360 msgr2=0x7f56980cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569813e360 0x7f56980cf290 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5698072770 tx=0x7f568002b900 comp rx=0 tx=0).stop 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 -- 172.21.15.125:0/4242573243 shutdown_connections 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569815f9c0 0x7f56980cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 --2- 172.21.15.125:0/4242573243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569813e360 0x7f56980cf290 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 --2- 172.21.15.125:0/4242573243 >> v2:172.21.15.137:3300/0 conn(0x7f569813df80 0x7f569815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.118+0000 7f569e600640 1 -- 172.21.15.125:0/4242573243 >> 172.21.15.125:0/4242573243 conn(0x7f569806a800 msgr2=0x7f569806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.119+0000 7f569e600640 1 -- 172.21.15.125:0/4242573243 shutdown_connections 2024-01-16T15:06:35.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.119+0000 7f569e600640 1 -- 172.21.15.125:0/4242573243 wait complete. 2024-01-16T15:06:35.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.122+0000 7f569e600640 1 Processor -- start 2024-01-16T15:06:35.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.123+0000 7f569e600640 1 -- start start 2024-01-16T15:06:35.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f569813df80 0x7f5698153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569813e360 0x7f569813b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569813b6b0 con 0x7f569815f9c0 2024-01-16T15:06:35.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56981eed90 con 0x7f569813df80 2024-01-16T15:06:35.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.125+0000 7f569e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56981ef050 con 0x7f569813e360 2024-01-16T15:06:35.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.126+0000 7f5697e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.126+0000 7f5697e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56182/0 (socket says 172.21.15.125:56182) 2024-01-16T15:06:35.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.126+0000 7f5697e00640 1 -- 172.21.15.125:0/2412390458 learned_addr learned my addr 172.21.15.125:0/2412390458 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:35.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5696a00640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569813e360 0x7f569813b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5697e00640 1 -- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569813e360 msgr2=0x7f569813b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5697e00640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569813e360 0x7f569813b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5697e00640 1 -- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f569813df80 msgr2=0x7f5698153210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:35.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5697e00640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f569813df80 0x7f5698153210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.127+0000 7f5697e00640 1 -- 172.21.15.125:0/2412390458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800094d0 con 0x7f569815f9c0 2024-01-16T15:06:35.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.128+0000 7f5697e00640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f5690007e30 tx=0x7f569000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.135+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690014620 con 0x7f569815f9c0 2024-01-16T15:06:35.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.135+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569000ee90 con 0x7f569815f9c0 2024-01-16T15:06:35.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.135+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56900149a0 con 0x7f569815f9c0 2024-01-16T15:06:35.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.135+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56981ef330 con 0x7f569815f9c0 2024-01-16T15:06:35.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.135+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56981ef8f0 con 0x7f569815f9c0 2024-01-16T15:06:35.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.137+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56980675d0 con 0x7f569815f9c0 2024-01-16T15:06:35.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.140+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f5690014b00 con 0x7f569815f9c0 2024-01-16T15:06:35.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.151+0000 7f568f400640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 0x7f567006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.151+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f5690088e60 con 0x7f569815f9c0 2024-01-16T15:06:35.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.152+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5690059210 con 0x7f569815f9c0 2024-01-16T15:06:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.162+0000 7f5697400640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 0x7f567006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.165+0000 7f5697400640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 0x7f567006e2c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f56981f22c0 tx=0x7f5688009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.714+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f569806df00 con 0x7f569815f9c0 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.718+0000 7f568f400640 1 -- 172.21.15.125:0/2412390458 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f5690014db0 con 0x7f569815f9c0 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 614, 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1315, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5369978754, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141828608, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952284672, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 178735576, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 178745207, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 46 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.725+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 msgr2=0x7f567006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.725+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 0x7f567006e2c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f56981f22c0 tx=0x7f5688009190 comp rx=0 tx=0).stop 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 msgr2=0x7f5698152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f5690007e30 tx=0x7f569000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 shutdown_connections 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f567006be00 0x7f567006e2c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f569815f9c0 0x7f5698152990 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f569813e360 0x7f569813b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 --2- 172.21.15.125:0/2412390458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f569813df80 0x7f5698153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.726+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 >> 172.21.15.125:0/2412390458 conn(0x7f569806a800 msgr2=0x7f56980ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.727+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 shutdown_connections 2024-01-16T15:06:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:35.727+0000 7f569e600640 1 -- 172.21.15.125:0/2412390458 wait complete. 2024-01-16T15:06:36.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.076+0000 7fe550000640 1 Processor -- start 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.076+0000 7fe550000640 1 -- start start 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe550000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5480ae5a0 0x7fe54815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe550000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480cf290 0x7fe5480cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe550000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe548070c40 con 0x7fe5480b3ae0 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe550000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe548071450 con 0x7fe5480ae5a0 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe550000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480cfc60 con 0x7fe5480cf290 2024-01-16T15:06:37.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5480ae5a0 0x7fe54815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5480ae5a0 0x7fe54815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53686/0 (socket says 172.21.15.125:53686) 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 -- 172.21.15.125:0/3224936612 learned_addr learned my addr 172.21.15.125:0/3224936612 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54e000640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480cf290 0x7fe5480cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 -- 172.21.15.125:0/3224936612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480cf290 msgr2=0x7fe5480cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480cf290 0x7fe5480cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 -- 172.21.15.125:0/3224936612 >> v1:172.21.15.137:6789/0 conn(0x7fe5480b3ae0 legacy=0x7fe54806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.078+0000 7fe54d600640 1 -- 172.21.15.125:0/3224936612 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54806c470 con 0x7fe5480ae5a0 2024-01-16T15:06:37.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.079+0000 7fe54d600640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480ae5a0 0x7fe54815e430 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe538009870 tx=0x7fe53802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1adc580904be6ace server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe547e00640 1 -- 172.21.15.125:0/3224936612 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538030840 con 0x7fe5480ae5a0 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe547e00640 1 -- 172.21.15.125:0/3224936612 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5380309a0 con 0x7fe5480ae5a0 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe547e00640 1 -- 172.21.15.125:0/3224936612 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538030d30 con 0x7fe5480ae5a0 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 -- 172.21.15.125:0/3224936612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480ae5a0 msgr2=0x7fe54815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480ae5a0 0x7fe54815e430 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe538009870 tx=0x7fe53802b980 comp rx=0 tx=0).stop 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 -- 172.21.15.125:0/3224936612 shutdown_connections 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480cf290 0x7fe5480cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 --2- 172.21.15.125:0/3224936612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480ae5a0 0x7fe54815e430 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 -- 172.21.15.125:0/3224936612 >> 172.21.15.125:0/3224936612 conn(0x7fe54806a800 msgr2=0x7fe54806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 -- 172.21.15.125:0/3224936612 shutdown_connections 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.080+0000 7fe550000640 1 -- 172.21.15.125:0/3224936612 wait complete. 2024-01-16T15:06:37.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.082+0000 7fe550000640 1 Processor -- start 2024-01-16T15:06:37.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.082+0000 7fe550000640 1 -- start start 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480ae5a0 0x7fe548159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 0x7fe548154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481f2a30 con 0x7fe5480cf290 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481517d0 con 0x7fe5480b3ae0 2024-01-16T15:06:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe550000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481508b0 con 0x7fe5480ae5a0 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe54e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe54e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53690/0 (socket says 172.21.15.125:53690) 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe54e000640 1 -- 172.21.15.125:0/1316605246 learned_addr learned my addr 172.21.15.125:0/1316605246 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe54d600640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480ae5a0 0x7fe548159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.083+0000 7fe54cc00640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 0x7fe548154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 -- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480ae5a0 msgr2=0x7fe548159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480ae5a0 0x7fe548159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 -- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 msgr2=0x7fe548154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 0x7fe548154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 -- 172.21.15.125:0/1316605246 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380094d0 con 0x7fe5480cf290 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54cc00640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 0x7fe548154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe54e000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe54000cc30 tx=0x7fe54000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54000ce40 con 0x7fe5480cf290 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe540003990 con 0x7fe5480cf290 2024-01-16T15:06:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540003d20 con 0x7fe5480cf290 2024-01-16T15:06:37.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe548150fa0 con 0x7fe5480cf290 2024-01-16T15:06:37.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.084+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5481514f0 con 0x7fe5480cf290 2024-01-16T15:06:37.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.085+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5481f2b70 con 0x7fe5480cf290 2024-01-16T15:06:37.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.086+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fe540003e80 con 0x7fe5480cf290 2024-01-16T15:06:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.087+0000 7fe545600640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 0x7fe52006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.087+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fe540088b30 con 0x7fe5480cf290 2024-01-16T15:06:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.087+0000 7fe54d600640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 0x7fe52006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.088+0000 7fe54d600640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 0x7fe52006e2c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.092+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5481f2b70 con 0x7fe5480cf290 2024-01-16T15:06:37.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.332+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe5481f2b70 con 0x7fe5480cf290 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.333+0000 7fe545600640 1 -- 172.21.15.125:0/1316605246 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fe5481f2b70 con 0x7fe5480cf290 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:37.311 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:37.312 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 616, 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:37.313 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:37.314 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1349, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5512645434, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141857280, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952256000, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 199681266, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 190619957, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 48, 2024-01-16T15:06:37.315 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 51 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:37.316 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:37.317 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.318 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 65s)", 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:37.319 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 msgr2=0x7fe52006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 0x7fe52006e2c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538008030 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 msgr2=0x7fe54815d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe54000cc30 tx=0x7fe54000a5c0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 shutdown_connections 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52006be00 0x7fe52006e2c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5480cf290 0x7fe54815d820 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5480b3ae0 0x7fe548154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 --2- 172.21.15.125:0/1316605246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5480ae5a0 0x7fe548159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 >> 172.21.15.125:0/1316605246 conn(0x7fe54806a800 msgr2=0x7fe5480b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 shutdown_connections 2024-01-16T15:06:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:37.336+0000 7fe550000640 1 -- 172.21.15.125:0/1316605246 wait complete. 2024-01-16T15:06:38.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:38.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.720+0000 7f2ce9800640 1 Processor -- start 2024-01-16T15:06:38.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.721+0000 7f2ce9800640 1 -- start start 2024-01-16T15:06:38.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 0x7f2ce40cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce9800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2ce40d15b0 0x7f2ce40d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce9800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce41613e0 con 0x7f2ce415fa90 2024-01-16T15:06:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce9800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40b4b70 con 0x7f2ce4160e80 2024-01-16T15:06:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce9800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40d1f00 con 0x7f2ce40d15b0 2024-01-16T15:06:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2ce415fa90 0x7f2ce40b4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43282/0 (socket says 172.21.15.125:43282) 2024-01-16T15:06:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.725+0000 7f2ce2a00640 1 -- 172.21.15.125:0/833250898 learned_addr learned my addr 172.21.15.125:0/833250898 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.726+0000 7f2ce3400640 1 --2- 172.21.15.125:0/833250898 >> v2:172.21.15.137:3300/0 conn(0x7f2ce40d15b0 0x7f2ce40d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.726+0000 7f2ce2000640 1 --2- 172.21.15.125:0/833250898 >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 0x7f2ce40cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.727+0000 7f2ce3400640 1 -- 172.21.15.125:0/833250898 >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 msgr2=0x7f2ce40cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.728+0000 7f2ce3400640 1 --2- 172.21.15.125:0/833250898 >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 0x7f2ce40cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.728+0000 7f2ce3400640 1 -- 172.21.15.125:0/833250898 >> v1:172.21.15.125:6789/0 conn(0x7f2ce415fa90 legacy=0x7f2ce40b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.728+0000 7f2ce1600640 1 -- 172.21.15.125:0/833250898 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074698087 0 0) 0x7f2ce41613e0 con 0x7f2ce415fa90 2024-01-16T15:06:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.728+0000 7f2ce3400640 1 -- 172.21.15.125:0/833250898 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce4161630 con 0x7f2ce40d15b0 2024-01-16T15:06:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.728+0000 7f2ce2000640 1 --2- 172.21.15.125:0/833250898 >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 0x7f2ce40cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:38.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.730+0000 7f2ce3400640 1 --2- 172.21.15.125:0/833250898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce40d19c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40b2170 tx=0x7f2cd402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e1b989513106551 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.731+0000 7f2ce1600640 1 -- 172.21.15.125:0/833250898 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4030610 con 0x7f2ce40d15b0 2024-01-16T15:06:38.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.731+0000 7f2ce1600640 1 -- 172.21.15.125:0/833250898 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4030770 con 0x7f2ce40d15b0 2024-01-16T15:06:38.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.731+0000 7f2ce1600640 1 -- 172.21.15.125:0/833250898 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4030b00 con 0x7f2ce40d15b0 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.732+0000 7f2ce9800640 1 -- 172.21.15.125:0/833250898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 msgr2=0x7f2ce40d19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.732+0000 7f2ce9800640 1 --2- 172.21.15.125:0/833250898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce40d19c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40b2170 tx=0x7f2cd402b960 comp rx=0 tx=0).stop 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.732+0000 7f2ce9800640 1 -- 172.21.15.125:0/833250898 shutdown_connections 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.732+0000 7f2ce9800640 1 --2- 172.21.15.125:0/833250898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce40d19c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.732+0000 7f2ce9800640 1 --2- 172.21.15.125:0/833250898 >> v2:172.21.15.125:3300/0 conn(0x7f2ce4160e80 0x7f2ce40cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.733+0000 7f2ce9800640 1 -- 172.21.15.125:0/833250898 >> 172.21.15.125:0/833250898 conn(0x7f2ce409c540 msgr2=0x7f2ce40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.734+0000 7f2ce9800640 1 -- 172.21.15.125:0/833250898 shutdown_connections 2024-01-16T15:06:38.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.734+0000 7f2ce9800640 1 -- 172.21.15.125:0/833250898 wait complete. 2024-01-16T15:06:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.737+0000 7f2ce9800640 1 Processor -- start 2024-01-16T15:06:38.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.737+0000 7f2ce9800640 1 -- start start 2024-01-16T15:06:38.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce415fa90 0x7f2ce41f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41f78e0 con 0x7f2ce4160e80 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce415b2b0 con 0x7f2ce40d15b0 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.738+0000 7f2ce9800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce409d0c0 con 0x7f2ce415fa90 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.739+0000 7f2ce3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.740+0000 7f2ce2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce415fa90 0x7f2ce41f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.740+0000 7f2ce3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53696/0 (socket says 172.21.15.125:53696) 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.740+0000 7f2ce3400640 1 -- 172.21.15.125:0/1592101704 learned_addr learned my addr 172.21.15.125:0/1592101704 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:38.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.740+0000 7f2ce2a00640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 -- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce415fa90 msgr2=0x7f2ce41f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce415fa90 0x7f2ce41f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 -- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 msgr2=0x7f2ce415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce415ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 -- 172.21.15.125:0/1592101704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc009a70 con 0x7f2ce4160e80 2024-01-16T15:06:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.741+0000 7f2ce3400640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009860 tx=0x7f2cd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.742+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd403e070 con 0x7f2ce4160e80 2024-01-16T15:06:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.743+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4030e10 con 0x7f2ce4160e80 2024-01-16T15:06:38.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.743+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4039be0 con 0x7f2ce4160e80 2024-01-16T15:06:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.745+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd4009400 con 0x7f2ce4160e80 2024-01-16T15:06:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.745+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce41f4300 con 0x7f2ce4160e80 2024-01-16T15:06:38.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.747+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce40675d0 con 0x7f2ce4160e80 2024-01-16T15:06:38.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.750+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f2cd4039d40 con 0x7f2ce4160e80 2024-01-16T15:06:38.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.751+0000 7f2cdaa00640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 0x7f2cbc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.752+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2cd40aa500 con 0x7f2ce4160e80 2024-01-16T15:06:38.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.759+0000 7f2ce2a00640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 0x7f2cbc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.764+0000 7f2ce2a00640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 0x7f2cbc06e2c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:38.765+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cd4076b90 con 0x7f2ce4160e80 2024-01-16T15:06:39.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:39.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:39.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.349+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2ce40c6af0 con 0x7f2ce4160e80 2024-01-16T15:06:39.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.350+0000 7f2cdaa00640 1 -- 172.21.15.125:0/1592101704 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f2cd4034990 con 0x7f2ce4160e80 2024-01-16T15:06:39.331 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:39.331 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:39.332 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 618, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 1349, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 5512645434, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 11141857280, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 761952256000, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 190771425, 2024-01-16T15:06:39.334 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 189382534, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 45, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 48 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:39.335 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.336 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:39.337 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:39.338 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.366+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 msgr2=0x7f2cbc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.366+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 0x7f2cbc06e2c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007de0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 msgr2=0x7f2ce415e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009860 tx=0x7f2cd40046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 shutdown_connections 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2cbc06be00 0x7f2cbc06e2c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce4160e80 0x7f2ce415e0d0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce415fa90 0x7f2ce41f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 --2- 172.21.15.125:0/1592101704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce40d15b0 0x7f2ce415ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 >> 172.21.15.125:0/1592101704 conn(0x7f2ce409c540 msgr2=0x7f2ce40b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.367+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 shutdown_connections 2024-01-16T15:06:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:39.368+0000 7f2ce9800640 1 -- 172.21.15.125:0/1592101704 wait complete. 2024-01-16T15:06:40.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.594+0000 7fcac9600640 1 Processor -- start 2024-01-16T15:06:40.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.594+0000 7fcac9600640 1 -- start start 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcac40cf9a0 0x7fcac415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 0x7fcac40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac9600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40b1ee0 con 0x7fcac4143ae0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac40b2780 con 0x7fcac40cf9a0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac415e990 con 0x7fcac415e1e0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcac4143ae0 0x7fcac40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43286/0 (socket says 172.21.15.125:43286) 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac2a00640 1 -- 172.21.15.125:0/3858183567 learned_addr learned my addr 172.21.15.125:0/3858183567 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:40.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162515950 0 0) 0x7fcac40b1ee0 con 0x7fcac4143ae0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcabbe00640 1 --2- 172.21.15.125:0/3858183567 >> v2:172.21.15.137:3300/0 conn(0x7fcac40cf9a0 0x7fcac415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa4003520 con 0x7fcac4143ae0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.596+0000 7fcac3400640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 0x7fcac40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964898316 0 0) 0x7fcaa4003520 con 0x7fcac4143ae0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 msgr2=0x7fcac40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 0x7fcac40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 >> v2:172.21.15.137:3300/0 conn(0x7fcac40cf9a0 msgr2=0x7fcac415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 --2- 172.21.15.125:0/3858183567 >> v2:172.21.15.137:3300/0 conn(0x7fcac40cf9a0 0x7fcac415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40b2130 con 0x7fcac4143ae0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcab4001670 con 0x7fcac4143ae0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcabbe00640 1 --2- 172.21.15.125:0/3858183567 >> v2:172.21.15.137:3300/0 conn(0x7fcac40cf9a0 0x7fcac415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 >> v1:172.21.15.125:6789/0 conn(0x7fcac4143ae0 legacy=0x7fcac40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa4003970 0x7fcaa4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 0x7fcaa40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcaa4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40b2780 con 0x7fcaa4004d00 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac415e990 con 0x7fcaa4003970 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa400dc70 con 0x7fcac415e1e0 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.597+0000 7fcac3400640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcaa4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac2a00640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa4003970 0x7fcaa4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac3400640 1 -- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 msgr2=0x7fcaa40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac3400640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac415e1e0 0x7fcaa40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac3400640 1 -- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa4003970 msgr2=0x7fcaa4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac3400640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa4003970 0x7fcaa4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac3400640 1 -- 172.21.15.125:0/3858183567 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4003520 con 0x7fcaa4004d00 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.598+0000 7fcac2a00640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa4003970 0x7fcaa4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac3400640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcaa4005180 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fcabc007360 tx=0x7fcaa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e168ae1fba9ca0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc02cce0 con 0x7fcaa4004d00 2024-01-16T15:06:40.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc02ce40 con 0x7fcaa4004d00 2024-01-16T15:06:40.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac2000640 1 -- 172.21.15.125:0/3858183567 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc034660 con 0x7fcaa4004d00 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac9600640 1 -- 172.21.15.125:0/3858183567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 msgr2=0x7fcaa4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac9600640 1 --2- 172.21.15.125:0/3858183567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcaa4005180 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fcabc007360 tx=0x7fcaa40057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.599+0000 7fcac3400640 1 -- 172.21.15.125:0/3858183567 reap_dead start 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.600+0000 7fcac9600640 1 -- 172.21.15.125:0/3858183567 shutdown_connections 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.600+0000 7fcac9600640 1 -- 172.21.15.125:0/3858183567 >> 172.21.15.125:0/3858183567 conn(0x7fcac406a800 msgr2=0x7fcac406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.600+0000 7fcac9600640 1 -- 172.21.15.125:0/3858183567 shutdown_connections 2024-01-16T15:06:40.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.600+0000 7fcac9600640 1 -- 172.21.15.125:0/3858183567 wait complete. 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.601+0000 7fcac9600640 1 Processor -- start 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.601+0000 7fcac9600640 1 -- start start 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac41302f0 0x7fcac40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac4070800 0x7fcac415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41f2b00 con 0x7fcaa4004d00 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4070c80 con 0x7fcac4070800 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41ee540 con 0x7fcac41302f0 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53708/0 (socket says 172.21.15.125:53708) 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.602+0000 7fcac2a00640 1 -- 172.21.15.125:0/1922734664 learned_addr learned my addr 172.21.15.125:0/1922734664 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 -- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac41302f0 msgr2=0x7fcac40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac41302f0 0x7fcac40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 -- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac4070800 msgr2=0x7fcac415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac4070800 0x7fcac415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 -- 172.21.15.125:0/1922734664 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac41f1be0 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac2a00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcab4000f60 tx=0x7fcab400dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab40074e0 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab4007640 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4004a00 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac41f1e80 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.603+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41f2440 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.604+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac40675d0 con 0x7fcaa4004d00 2024-01-16T15:06:40.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.605+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fcab4004b60 con 0x7fcaa4004d00 2024-01-16T15:06:40.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.606+0000 7fcabaa00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 0x7fca9c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.607+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcab4088160 con 0x7fcaa4004d00 2024-01-16T15:06:40.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.607+0000 7fcabbe00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 0x7fca9c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.607+0000 7fcabbe00640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 0x7fca9c06e2f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcaac006fd0 tx=0x7fcaac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.611+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcab40547d0 con 0x7fcaa4004d00 2024-01-16T15:06:40.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.853+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcac40cf720 con 0x7fcaa4004d00 2024-01-16T15:06:40.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.854+0000 7fcabaa00640 1 -- 172.21.15.125:0/1922734664 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7fcab4019020 con 0x7fcaa4004d00 2024-01-16T15:06:40.833 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:40.833 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:40.833 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:40.834 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 619, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:40.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2152, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 8880758408, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 17307996160, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 755786117120, 2024-01-16T15:06:40.836 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 476593588, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 469971202, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 113, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 118 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:40.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:40.839 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 msgr2=0x7fca9c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 0x7fca9c06e2f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcaac006fd0 tx=0x7fcaac008040 comp rx=0 tx=0).stop 2024-01-16T15:06:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 msgr2=0x7fcac41ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcab4000f60 tx=0x7fcab400dd60 comp rx=0 tx=0).stop 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 shutdown_connections 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac4070800 0x7fcac415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcac41302f0 0x7fcac40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca9c06be30 0x7fca9c06e2f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 --2- 172.21.15.125:0/1922734664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaa4004d00 0x7fcac41ee000 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 >> 172.21.15.125:0/1922734664 conn(0x7fcac406a800 msgr2=0x7fcac41428f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 shutdown_connections 2024-01-16T15:06:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:40.857+0000 7fcac9600640 1 -- 172.21.15.125:0/1922734664 wait complete. 2024-01-16T15:06:41.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:42.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.081+0000 7f29ea600640 1 Processor -- start 2024-01-16T15:06:42.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.081+0000 7f29ea600640 1 -- start start 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29ea600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 0x7f29e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29ea600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29e415e3b0 0x7f29e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29ea600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40cfc30 con 0x7f29e40cf7d0 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29ea600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e40b1ee0 con 0x7f29e415e3b0 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29ea600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e40b2780 con 0x7f29e4143ae0 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.083+0000 7f29e3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29e415e3b0 0x7f29e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29e415e3b0 0x7f29e40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53718/0 (socket says 172.21.15.125:53718) 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 -- 172.21.15.125:0/1448638604 learned_addr learned my addr 172.21.15.125:0/1448638604 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3400640 1 --2- 172.21.15.125:0/1448638604 >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 0x7f29e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 -- 172.21.15.125:0/1448638604 >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 msgr2=0x7f29e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 --2- 172.21.15.125:0/1448638604 >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 0x7f29e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 -- 172.21.15.125:0/1448638604 >> v1:172.21.15.137:6789/0 conn(0x7f29e40cf7d0 legacy=0x7f29e415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:42.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3e00640 1 -- 172.21.15.125:0/1448638604 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e415e990 con 0x7f29e415e3b0 2024-01-16T15:06:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.084+0000 7f29e3400640 1 --2- 172.21.15.125:0/1448638604 >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 0x7f29e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.085+0000 7f29e3e00640 1 --2- 172.21.15.125:0/1448638604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e415e3b0 0x7f29e40702c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f29e40cfea0 tx=0x7f29dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d9cfd94454da1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.085+0000 7f29e2a00640 1 -- 172.21.15.125:0/1448638604 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc02ca80 con 0x7f29e415e3b0 2024-01-16T15:06:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.085+0000 7f29e2a00640 1 -- 172.21.15.125:0/1448638604 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29dc02cbe0 con 0x7f29e415e3b0 2024-01-16T15:06:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29e2a00640 1 -- 172.21.15.125:0/1448638604 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0343f0 con 0x7f29e415e3b0 2024-01-16T15:06:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 -- 172.21.15.125:0/1448638604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e415e3b0 msgr2=0x7f29e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 --2- 172.21.15.125:0/1448638604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e415e3b0 0x7f29e40702c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f29e40cfea0 tx=0x7f29dc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 -- 172.21.15.125:0/1448638604 shutdown_connections 2024-01-16T15:06:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 --2- 172.21.15.125:0/1448638604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e415e3b0 0x7f29e40702c0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 --2- 172.21.15.125:0/1448638604 >> v2:172.21.15.137:3300/0 conn(0x7f29e4143ae0 0x7f29e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 -- 172.21.15.125:0/1448638604 >> 172.21.15.125:0/1448638604 conn(0x7f29e406a800 msgr2=0x7f29e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 -- 172.21.15.125:0/1448638604 shutdown_connections 2024-01-16T15:06:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.086+0000 7f29ea600640 1 -- 172.21.15.125:0/1448638604 wait complete. 2024-01-16T15:06:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.088+0000 7f29ea600640 1 Processor -- start 2024-01-16T15:06:42.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.088+0000 7f29ea600640 1 -- start start 2024-01-16T15:06:42.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29e40cf7d0 0x7f29e4159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29e415e3b0 0x7f29e40a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40cde30 con 0x7f29e4143ae0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40ce110 con 0x7f29e415e3b0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29ea600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e41f4480 con 0x7f29e40cf7d0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29da800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53724/0 (socket says 172.21.15.125:53724) 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.089+0000 7f29da800640 1 -- 172.21.15.125:0/2832821384 learned_addr learned my addr 172.21.15.125:0/2832821384 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29e3e00640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29e415e3b0 0x7f29e40a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29e3400640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29e40cf7d0 0x7f29e4159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 -- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29e40cf7d0 msgr2=0x7f29e4159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29e40cf7d0 0x7f29e4159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 -- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29e415e3b0 msgr2=0x7f29e40a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29e415e3b0 0x7f29e40a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 -- 172.21.15.125:0/2832821384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc009b30 con 0x7f29e4143ae0 2024-01-16T15:06:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29da800640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c410 tx=0x7f29cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0035f0 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc003750 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e41f4760 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc003b40 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.090+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e40cd290 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.091+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e41f42e0 con 0x7f29e4143ae0 2024-01-16T15:06:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.092+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f29cc003ca0 con 0x7f29e4143ae0 2024-01-16T15:06:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.093+0000 7f29dbe00640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 0x7f29bc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.093+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f29cc087ee0 con 0x7f29e4143ae0 2024-01-16T15:06:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.093+0000 7f29e3400640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 0x7f29bc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.094+0000 7f29e3400640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 0x7f29bc06e2c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f29d4008850 tx=0x7f29d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.098+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29e41f42e0 con 0x7f29e4143ae0 2024-01-16T15:06:42.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.338+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f29e41f42e0 con 0x7f29e4143ae0 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.339+0000 7f29dbe00640 1 -- 172.21.15.125:0/2832821384 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f29e41f42e0 con 0x7f29e4143ae0 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:42.318 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:42.319 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 621, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:42.320 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2152, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 8880758408, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 17307996160, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 755786117120, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 476593588, 2024-01-16T15:06:42.321 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 469971202, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 113, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 118 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:42.322 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.323 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:42.324 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:42.325 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:42.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.341+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 msgr2=0x7f29bc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 0x7f29bc06e2c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f29d4008850 tx=0x7f29d4008040 comp rx=0 tx=0).stop 2024-01-16T15:06:42.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 msgr2=0x7f29e40a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c410 tx=0x7f29cc00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 shutdown_connections 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29e415e3b0 0x7f29e40a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f29bc06be00 0x7f29bc06e2c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29e4143ae0 0x7f29e40a8b90 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 --2- 172.21.15.125:0/2832821384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29e40cf7d0 0x7f29e4159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 >> 172.21.15.125:0/2832821384 conn(0x7f29e406a800 msgr2=0x7f29e415dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 shutdown_connections 2024-01-16T15:06:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:42.342+0000 7f29ea600640 1 -- 172.21.15.125:0/2832821384 wait complete. 2024-01-16T15:06:43.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:43.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.575+0000 7f34ab000640 1 Processor -- start 2024-01-16T15:06:43.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.575+0000 7f34ab000640 1 -- start start 2024-01-16T15:06:43.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.576+0000 7f34ab000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 0x7f34a40d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.576+0000 7f34ab000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40cb750 con 0x7f34a415b1f0 2024-01-16T15:06:43.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.576+0000 7f34ab000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40cba30 con 0x7f34a40d1aa0 2024-01-16T15:06:43.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.576+0000 7f34ab000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40cbec0 con 0x7f34a40b3310 2024-01-16T15:06:43.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f34a415b1f0 0x7f34a40b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43296/0 (socket says 172.21.15.125:43296) 2024-01-16T15:06:43.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a3e00640 1 -- 172.21.15.125:0/559378678 learned_addr learned my addr 172.21.15.125:0/559378678 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:43.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883952981 0 0) 0x7f34a40cb750 con 0x7f34a415b1f0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3484003520 con 0x7f34a415b1f0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a3400640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 0x7f34a40d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955710232 0 0) 0x7f3484003520 con 0x7f34a415b1f0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 msgr2=0x7f34a40d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 0x7f34a40d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 >> v1:172.21.15.137:6789/0 conn(0x7f34a40d1aa0 legacy=0x7f34a415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40d1ed0 con 0x7f34a415b1f0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.577+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3494001670 con 0x7f34a415b1f0 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 >> v1:172.21.15.125:6789/0 conn(0x7f34a415b1f0 legacy=0x7f34a40b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34840039a0 0x7f3484003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34840047c0 0x7f3484004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 0x7f3484005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40cbec0 con 0x7f34840047c0 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348400bb10 con 0x7f34840039a0 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348400bcd0 con 0x7f34a40b3310 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34840047c0 0x7f3484004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3e00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34840039a0 0x7f3484003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 -- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 msgr2=0x7f3484005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34a40b3310 0x7f3484005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 -- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34840039a0 msgr2=0x7f3484003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34840039a0 0x7f3484003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3400640 1 -- 172.21.15.125:0/559378678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484003520 con 0x7f34840047c0 2024-01-16T15:06:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.578+0000 7f34a3e00640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34840039a0 0x7f3484003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.579+0000 7f34a3400640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34840047c0 0x7f3484004c40 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f348c00ba30 tx=0x7f34840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27082eb47845d15 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c032680 con 0x7f34840047c0 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348c0327e0 con 0x7f34840047c0 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34a2a00640 1 -- 172.21.15.125:0/559378678 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c032b70 con 0x7f34840047c0 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 -- 172.21.15.125:0/559378678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34840047c0 msgr2=0x7f3484004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 --2- 172.21.15.125:0/559378678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34840047c0 0x7f3484004c40 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f348c00ba30 tx=0x7f34840057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34a9000640 1 -- 172.21.15.125:0/559378678 reap_dead start 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 -- 172.21.15.125:0/559378678 shutdown_connections 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 -- 172.21.15.125:0/559378678 >> 172.21.15.125:0/559378678 conn(0x7f34a406e7a0 msgr2=0x7f34a406eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 -- 172.21.15.125:0/559378678 shutdown_connections 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.580+0000 7f34ab000640 1 -- 172.21.15.125:0/559378678 wait complete. 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.582+0000 7f34ab000640 1 Processor -- start 2024-01-16T15:06:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.582+0000 7f34ab000640 1 -- start start 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34840047c0 0x7f34a40ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34a40c81d0 0x7f34a40b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4159b80 con 0x7f34a41591c0 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40c9610 con 0x7f34a40c81d0 2024-01-16T15:06:43.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34ab000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40c8a00 con 0x7f34840047c0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34a9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34a9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53748/0 (socket says 172.21.15.125:53748) 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.583+0000 7f34a9000640 1 -- 172.21.15.125:0/3619776563 learned_addr learned my addr 172.21.15.125:0/3619776563 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a3400640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34a40c81d0 0x7f34a40b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 -- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34840047c0 msgr2=0x7f34a40ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34840047c0 0x7f34a40ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 -- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34a40c81d0 msgr2=0x7f34a40b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34a40c81d0 0x7f34a40b6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 -- 172.21.15.125:0/3619776563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40c8ce0 con 0x7f34a41591c0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34a9000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f34a4071010 tx=0x7f349c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c005d40 con 0x7f34a41591c0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c00ced0 con 0x7f34a41591c0 2024-01-16T15:06:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00a850 con 0x7f34a41591c0 2024-01-16T15:06:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.584+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a40c8fc0 con 0x7f34a41591c0 2024-01-16T15:06:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.585+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a40b6650 con 0x7f34a41591c0 2024-01-16T15:06:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.586+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a40675d0 con 0x7f34a41591c0 2024-01-16T15:06:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.587+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f349c00a9b0 con 0x7f34a41591c0 2024-01-16T15:06:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.587+0000 7f349be00640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 0x7f347c06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.588+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f349c087700 con 0x7f34a41591c0 2024-01-16T15:06:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.588+0000 7f34a3e00640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 0x7f347c06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.588+0000 7f34a3e00640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 0x7f347c06e2c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3494007a10 tx=0x7f3494009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.592+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f349c053da0 con 0x7f34a41591c0 2024-01-16T15:06:43.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.852+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f34a40cfb80 con 0x7f34a41591c0 2024-01-16T15:06:43.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.853+0000 7f349be00640 1 -- 172.21.15.125:0/3619776563 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2409 (secure 0 0 0) 0x7f349c0573c0 con 0x7f34a41591c0 2024-01-16T15:06:43.831 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:43.831 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 622, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 2152, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 8880758408, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 17307996160, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 755786117120, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 306775146, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 292462792, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 73, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 74 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (26s)\n [========....................] (remaining: 65s)", 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.28571429848670959, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (26s)\n [==============..............] (remaining: 26s)", 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.5, 2024-01-16T15:06:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:43.838 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.855+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 msgr2=0x7f347c06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.855+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 0x7f347c06e2c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3494007a10 tx=0x7f3494009040 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.855+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 msgr2=0x7f34a4159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.855+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f34a4071010 tx=0x7f349c00cb90 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 shutdown_connections 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f347c06be00 0x7f347c06e2c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34a41591c0 0x7f34a4159640 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34a40c81d0 0x7f34a40b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 --2- 172.21.15.125:0/3619776563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34840047c0 0x7f34a40ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 >> 172.21.15.125:0/3619776563 conn(0x7f34a406e7a0 msgr2=0x7f34a40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 shutdown_connections 2024-01-16T15:06:43.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:43.856+0000 7f34ab000640 1 -- 172.21.15.125:0/3619776563 wait complete. 2024-01-16T15:06:44.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96918 2024-01-16T15:06:45.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.095+0000 7f6061e00640 1 Processor -- start 2024-01-16T15:06:45.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.095+0000 7f6061e00640 1 -- start start 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f6061e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f605c0cf9a0 0x7f605c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f6061e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c070c40 con 0x7f605c0b3ae0 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f6061e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c06c470 con 0x7f605c0ae5a0 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f6061e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c071450 con 0x7f605c0cf9a0 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f605a000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f605c0b3ae0 0x7f605c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43302/0 (socket says 172.21.15.125:43302) 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.097+0000 7f605a000640 1 -- 172.21.15.125:0/1734925053 learned_addr learned my addr 172.21.15.125:0/1734925053 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f605b400640 1 --2- 172.21.15.125:0/1734925053 >> v2:172.21.15.137:3300/0 conn(0x7f605c0cf9a0 0x7f605c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619739143 0 0) 0x7f605c070c40 con 0x7f605c0b3ae0 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c003520 con 0x7f605c0b3ae0 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4026701965 0 0) 0x7f603c003520 con 0x7f605c0b3ae0 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 >> v2:172.21.15.137:3300/0 conn(0x7f605c0cf9a0 msgr2=0x7f605c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 --2- 172.21.15.125:0/1734925053 >> v2:172.21.15.137:3300/0 conn(0x7f605c0cf9a0 0x7f605c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 >> v1:172.21.15.137:6789/0 conn(0x7f605c0ae5a0 legacy=0x7f605c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c06c750 con 0x7f605c0b3ae0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6044001670 con 0x7f605c0b3ae0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 >> v1:172.21.15.125:6789/0 conn(0x7f605c0b3ae0 legacy=0x7f605c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f603c003970 0x7f603c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c0047c0 0x7f603c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 0x7f603c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c071450 con 0x7f603c005180 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c00c030 con 0x7f603c003970 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.098+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c00c1f0 con 0x7f603c0047c0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 0x7f603c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605aa00640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f603c003970 0x7f603c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605a000640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c0047c0 0x7f603c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 -- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c0047c0 msgr2=0x7f603c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c0047c0 0x7f603c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 -- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f603c003970 msgr2=0x7f603c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f603c003970 0x7f603c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.099+0000 7f605b400640 1 -- 172.21.15.125:0/1734925053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c003520 con 0x7f603c005180 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.100+0000 7f605b400640 1 -- 172.21.15.125:0/1734925053 reap_dead start 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.100+0000 7f605b400640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 0x7f603c00b980 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f605c0aeb20 tx=0x7f603c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb14b61c4d5bf12 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.100+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605402cbd0 con 0x7f603c005180 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.100+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605400f040 con 0x7f603c005180 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.100+0000 7f6059600640 1 -- 172.21.15.125:0/1734925053 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054003990 con 0x7f603c005180 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 -- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 msgr2=0x7f603c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 0x7f603c00b980 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f605c0aeb20 tx=0x7f603c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 -- 172.21.15.125:0/1734925053 shutdown_connections 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 --2- 172.21.15.125:0/1734925053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c005180 0x7f603c00b980 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 -- 172.21.15.125:0/1734925053 >> 172.21.15.125:0/1734925053 conn(0x7f605c06a800 msgr2=0x7f605c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 -- 172.21.15.125:0/1734925053 shutdown_connections 2024-01-16T15:06:45.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.101+0000 7f6061e00640 1 -- 172.21.15.125:0/1734925053 wait complete. 2024-01-16T15:06:45.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.103+0000 7f6061e00640 1 Processor -- start 2024-01-16T15:06:45.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.103+0000 7f6061e00640 1 -- start start 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c005180 0x7f605c1f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 0x7f605c158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c15ef20 con 0x7f605c1537b0 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1f6b60 con 0x7f605c159b10 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f6061e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1f6e40 con 0x7f603c005180 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f605b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f605b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53774/0 (socket says 172.21.15.125:53774) 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.104+0000 7f605b400640 1 -- 172.21.15.125:0/836410813 learned_addr learned my addr 172.21.15.125:0/836410813 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:45.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605a000640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 0x7f605c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 -- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c005180 msgr2=0x7f605c1f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c005180 0x7f605c1f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 -- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 msgr2=0x7f605c158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 0x7f605c158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 -- 172.21.15.125:0/836410813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c1f7120 con 0x7f605c1537b0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605a000640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 0x7f605c158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f605b400640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f605402c640 tx=0x7f6054009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054035ee0 con 0x7f605c1537b0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6054040470 con 0x7f605c1537b0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054040820 con 0x7f605c1537b0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c15ebb0 con 0x7f605c1537b0 2024-01-16T15:06:45.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.105+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c158660 con 0x7f605c1537b0 2024-01-16T15:06:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.107+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c15ed40 con 0x7f605c1537b0 2024-01-16T15:06:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.107+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f6054040980 con 0x7f605c1537b0 2024-01-16T15:06:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.108+0000 7f6053400640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 0x7f603406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.108+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f60540ab6a0 con 0x7f605c1537b0 2024-01-16T15:06:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.109+0000 7f605aa00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 0x7f603406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.109+0000 7f605aa00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 0x7f603406e2c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f604c0014f0 tx=0x7f604c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.113+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6054077f60 con 0x7f605c1537b0 2024-01-16T15:06:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.337+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96918", "target": ["mon-mgr", ""]}) v1 -- 0x7f605c070550 con 0x7f603406be00 2024-01-16T15:06:45.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.518+0000 7f6053400640 1 -- 172.21.15.125:0/836410813 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f604c007ab0 con 0x7f603406be00 2024-01-16T15:06:45.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 msgr2=0x7f603406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 0x7f603406e2c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f604c0014f0 tx=0x7f604c009040 comp rx=0 tx=0).stop 2024-01-16T15:06:45.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 msgr2=0x7f605c14c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f605402c640 tx=0x7f6054009750 comp rx=0 tx=0).stop 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 shutdown_connections 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f603406be00 0x7f603406e2c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f605c1537b0 0x7f605c14c900 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605c159b10 0x7f605c158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 --2- 172.21.15.125:0/836410813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f603c005180 0x7f605c1f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.522+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 >> 172.21.15.125:0/836410813 conn(0x7f605c06a800 msgr2=0x7f605c06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.523+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 shutdown_connections 2024-01-16T15:06:45.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.523+0000 7f6061e00640 1 -- 172.21.15.125:0/836410813 wait complete. 2024-01-16T15:06:45.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96919 2024-01-16T15:06:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.749+0000 7f46e4e00640 1 Processor -- start 2024-01-16T15:06:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.749+0000 7f46e4e00640 1 -- start start 2024-01-16T15:06:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46e00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46e4e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e0071de0 con 0x7f46e00aede0 2024-01-16T15:06:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46e4e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00af270 con 0x7f46e00d0b60 2024-01-16T15:06:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46e4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e0072030 con 0x7f46e006a3e0 2024-01-16T15:06:45.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46de800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f46e00aede0 0x7f46e006f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43314/0 (socket says 172.21.15.125:43314) 2024-01-16T15:06:45.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.751+0000 7f46de800640 1 -- 172.21.15.125:0/92093222 learned_addr learned my addr 172.21.15.125:0/92093222 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.752+0000 7f46dde00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46e00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.752+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477426906 0 0) 0x7f46e0071de0 con 0x7f46e00aede0 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.752+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0003520 con 0x7f46e00aede0 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.752+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235417389 0 0) 0x7f46c0003520 con 0x7f46e00aede0 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 msgr2=0x7f46e00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46e00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 >> v1:172.21.15.137:6789/0 conn(0x7f46e00d0b60 legacy=0x7f46e006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e0072310 con 0x7f46e00aede0 2024-01-16T15:06:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f46d80019f0 con 0x7f46e00aede0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 >> v1:172.21.15.125:6789/0 conn(0x7f46e00aede0 legacy=0x7f46e006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00039a0 0x7f46c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e0072030 con 0x7f46c00047c0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c000ba30 con 0x7f46c00039a0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c000bbf0 con 0x7f46e006a3e0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dde00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00039a0 0x7f46c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46de800640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 -- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 msgr2=0x7f46c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 -- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00039a0 msgr2=0x7f46c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00039a0 0x7f46c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46dd400640 1 -- 172.21.15.125:0/92093222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0003520 con 0x7f46c00047c0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.753+0000 7f46de800640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e006a3e0 0x7f46c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46dd400640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f46e00d0fa0 tx=0x7f46c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27ee3e3c74ef450 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80052d0 con 0x7f46c00047c0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c802e050 con 0x7f46c00047c0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46dca00640 1 -- 172.21.15.125:0/92093222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80035f0 con 0x7f46c00047c0 2024-01-16T15:06:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46e4e00640 1 -- 172.21.15.125:0/92093222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46c00047c0 msgr2=0x7f46c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46e4e00640 1 --2- 172.21.15.125:0/92093222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f46e00d0fa0 tx=0x7f46c00057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.755+0000 7f46de800640 1 -- 172.21.15.125:0/92093222 reap_dead start 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.756+0000 7f46e4e00640 1 -- 172.21.15.125:0/92093222 shutdown_connections 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.756+0000 7f46e4e00640 1 -- 172.21.15.125:0/92093222 >> 172.21.15.125:0/92093222 conn(0x7f46e00cfef0 msgr2=0x7f46e00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.756+0000 7f46e4e00640 1 -- 172.21.15.125:0/92093222 shutdown_connections 2024-01-16T15:06:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.756+0000 7f46e4e00640 1 -- 172.21.15.125:0/92093222 wait complete. 2024-01-16T15:06:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.757+0000 7f46e4e00640 1 Processor -- start 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.758+0000 7f46e4e00640 1 -- start start 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.758+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 0x7f46e0156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.758+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.758+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e0156c90 0x7f46e013b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46e4e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e015bb60 con 0x7f46e00cc940 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46e4e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e014eef0 con 0x7f46c00047c0 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46e4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00ccdd0 con 0x7f46e0156c90 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53778/0 (socket says 172.21.15.125:53778) 2024-01-16T15:06:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 -- 172.21.15.125:0/317090553 learned_addr learned my addr 172.21.15.125:0/317090553 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dde00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 0x7f46e0156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46de800640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e0156c90 0x7f46e013b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 -- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e0156c90 msgr2=0x7f46e013b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e0156c90 0x7f46e013b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 -- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 msgr2=0x7f46e0156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 0x7f46e0156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dd400640 1 -- 172.21.15.125:0/317090553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e015b230 con 0x7f46e00cc940 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.759+0000 7f46dde00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 0x7f46e0156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46dd400640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f46c802bba0 tx=0x7f46c8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80076c0 con 0x7f46e00cc940 2024-01-16T15:06:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8007a30 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8006be0 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e015b510 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.760+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e015bff0 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.761+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e015bd10 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.762+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f46c8006d40 con 0x7f46e00cc940 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.763+0000 7f46d6000640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 0x7f46b806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.763+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f46c80aa410 con 0x7f46e00cc940 2024-01-16T15:06:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.763+0000 7f46dde00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 0x7f46b806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.764+0000 7f46dde00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 0x7f46b806e2c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009a00 tx=0x7f46d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.767+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46e015bd10 con 0x7f46e00cc940 2024-01-16T15:06:45.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:45.998+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96919", "target": ["mon-mgr", ""]}) v1 -- 0x7f46e006eb60 con 0x7f46b806be00 2024-01-16T15:06:46.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.130+0000 7f46d6000640 1 -- 172.21.15.125:0/317090553 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46d0009670 con 0x7f46b806be00 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 msgr2=0x7f46b806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 0x7f46b806e2c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009a00 tx=0x7f46d0008040 comp rx=0 tx=0).stop 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 msgr2=0x7f46e0156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f46c802bba0 tx=0x7f46c8001120 comp rx=0 tx=0).stop 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 shutdown_connections 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e0156c90 0x7f46e013b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806be00 0x7f46b806e2c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cc940 0x7f46e0156750 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 --2- 172.21.15.125:0/317090553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c00047c0 0x7f46e0156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 >> 172.21.15.125:0/317090553 conn(0x7f46e00cfef0 msgr2=0x7f46e00ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 shutdown_connections 2024-01-16T15:06:46.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.134+0000 7f46e4e00640 1 -- 172.21.15.125:0/317090553 wait complete. 2024-01-16T15:06:46.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969110 2024-01-16T15:06:46.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.363+0000 7fe4e2400640 1 Processor -- start 2024-01-16T15:06:46.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.363+0000 7fe4e2400640 1 -- start start 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe4dc15e1e0 0x7fe4dc15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc0cfc30 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc0b1ee0 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4e2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0b2780 con 0x7fe4dc143ae0 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.365+0000 7fe4daa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53792/0 (socket says 172.21.15.125:53792) 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 -- 172.21.15.125:0/179351920 learned_addr learned my addr 172.21.15.125:0/179351920 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4db400640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4dbe00640 1 --2- 172.21.15.125:0/179351920 >> v2:172.21.15.137:3300/0 conn(0x7fe4dc15e1e0 0x7fe4dc15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 -- 172.21.15.125:0/179351920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 msgr2=0x7fe4dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 -- 172.21.15.125:0/179351920 >> v2:172.21.15.137:3300/0 conn(0x7fe4dc15e1e0 msgr2=0x7fe4dc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 --2- 172.21.15.125:0/179351920 >> v2:172.21.15.137:3300/0 conn(0x7fe4dc15e1e0 0x7fe4dc15a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.366+0000 7fe4daa00640 1 -- 172.21.15.125:0/179351920 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc15e990 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.367+0000 7fe4daa00640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc072160 tx=0x7fe4c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98f4fbfb93fbd27 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.368+0000 7fe4da000640 1 -- 172.21.15.125:0/179351920 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4031870 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.368+0000 7fe4da000640 1 -- 172.21.15.125:0/179351920 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c40319d0 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.368+0000 7fe4da000640 1 -- 172.21.15.125:0/179351920 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4031d60 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.368+0000 7fe4e2400640 1 -- 172.21.15.125:0/179351920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc0cf7d0 msgr2=0x7fe4dc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc072160 tx=0x7fe4c402b910 comp rx=0 tx=0).stop 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 -- 172.21.15.125:0/179351920 shutdown_connections 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 --2- 172.21.15.125:0/179351920 >> v2:172.21.15.137:3300/0 conn(0x7fe4dc15e1e0 0x7fe4dc15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc15dca0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 --2- 172.21.15.125:0/179351920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 -- 172.21.15.125:0/179351920 >> 172.21.15.125:0/179351920 conn(0x7fe4dc06a800 msgr2=0x7fe4dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:46.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 -- 172.21.15.125:0/179351920 shutdown_connections 2024-01-16T15:06:46.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.369+0000 7fe4e2400640 1 -- 172.21.15.125:0/179351920 wait complete. 2024-01-16T15:06:46.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.371+0000 7fe4e2400640 1 Processor -- start 2024-01-16T15:06:46.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.371+0000 7fe4e2400640 1 -- start start 2024-01-16T15:06:46.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc1f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc13d530 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc13b220 con 0x7fe4dc0cf7d0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.372+0000 7fe4e2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc13d780 con 0x7fe4dc143ae0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53800/0 (socket says 172.21.15.125:53800) 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 -- 172.21.15.125:0/1068265588 learned_addr learned my addr 172.21.15.125:0/1068265588 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4db400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc1f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4daa00640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 -- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 msgr2=0x7fe4dc159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 -- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4dc0cf7d0 msgr2=0x7fe4dc1f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc1f0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 -- 172.21.15.125:0/1068265588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c40094d0 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.373+0000 7fe4dbe00640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4007e90 tx=0x7fe4d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.374+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4014420 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.374+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4014580 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.374+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4014930 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.374+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc13da60 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.374+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc13e020 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.375+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc0675d0 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.376+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fe4d4014a90 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.377+0000 7fe4d3400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 0x7fe4b406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.377+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fe4d4088e80 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.377+0000 7fe4db400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 0x7fe4b406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.378+0000 7fe4db400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 0x7fe4b406e2c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc006fd0 tx=0x7fe4cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.382+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d4055740 con 0x7fe4dc15e1e0 2024-01-16T15:06:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.610+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969110", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4dc15e640 con 0x7fe4b406be00 2024-01-16T15:06:46.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.746+0000 7fe4d3400640 1 -- 172.21.15.125:0/1068265588 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe4cc0094a0 con 0x7fe4b406be00 2024-01-16T15:06:46.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 msgr2=0x7fe4b406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 0x7fe4b406e2c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc006fd0 tx=0x7fe4cc008040 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 msgr2=0x7fe4dc13cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4007e90 tx=0x7fe4d400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 shutdown_connections 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe4b406be00 0x7fe4b406e2c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4dc15e1e0 0x7fe4dc13cc20 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4dc143ae0 0x7fe4dc159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 --2- 172.21.15.125:0/1068265588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4dc0cf7d0 0x7fe4dc1f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 >> 172.21.15.125:0/1068265588 conn(0x7fe4dc06a800 msgr2=0x7fe4dc15ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 shutdown_connections 2024-01-16T15:06:46.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.750+0000 7fe4e2400640 1 -- 172.21.15.125:0/1068265588 wait complete. 2024-01-16T15:06:46.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969111 2024-01-16T15:06:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.984+0000 7f259e800640 1 Processor -- start 2024-01-16T15:06:46.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.984+0000 7f259e800640 1 -- start start 2024-01-16T15:06:46.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f259e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f259815b090 0x7f25980b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f259e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980b2be0 0x7f259806b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f259e800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25980ce770 con 0x7f25980aeac0 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f259e800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25980ccd40 con 0x7f259815b090 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f259e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980ce180 con 0x7f25980b2be0 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f2597400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f259815b090 0x7f25980b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f2597400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f259815b090 0x7f25980b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54670/0 (socket says 172.21.15.125:54670) 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f2597400640 1 -- 172.21.15.125:0/2797865875 learned_addr learned my addr 172.21.15.125:0/2797865875 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.986+0000 7f2597e00640 1 --1- 172.21.15.125:0/2797865875 >> v1:172.21.15.125:6789/0 conn(0x7f25980aeac0 0x7f25980cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51866/0 (socket says 172.21.15.125:51866) 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.987+0000 7f258fe00640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980b2be0 0x7f259806b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.987+0000 7f2597400640 1 -- 172.21.15.125:0/2797865875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980b2be0 msgr2=0x7f259806b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:46.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.987+0000 7f2597400640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980b2be0 0x7f259806b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.987+0000 7f2597400640 1 -- 172.21.15.125:0/2797865875 >> v1:172.21.15.125:6789/0 conn(0x7f25980aeac0 legacy=0x7f25980cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.987+0000 7f2597400640 1 -- 172.21.15.125:0/2797865875 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980715a0 con 0x7f259815b090 2024-01-16T15:06:46.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.988+0000 7f2597400640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f259815b090 0x7f25980b26a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3194a30f6468c723 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.988+0000 7f2596a00640 1 -- 172.21.15.125:0/2797865875 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588030840 con 0x7f259815b090 2024-01-16T15:06:46.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.988+0000 7f2596a00640 1 -- 172.21.15.125:0/2797865875 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25880309a0 con 0x7f259815b090 2024-01-16T15:06:46.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.988+0000 7f2596a00640 1 -- 172.21.15.125:0/2797865875 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588030d30 con 0x7f259815b090 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 -- 172.21.15.125:0/2797865875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f259815b090 msgr2=0x7f25980b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f259815b090 0x7f25980b26a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258802b980 comp rx=0 tx=0).stop 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 -- 172.21.15.125:0/2797865875 shutdown_connections 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980b2be0 0x7f259806b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 --2- 172.21.15.125:0/2797865875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f259815b090 0x7f25980b26a0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 -- 172.21.15.125:0/2797865875 >> 172.21.15.125:0/2797865875 conn(0x7f2598070a80 msgr2=0x7f2598070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 -- 172.21.15.125:0/2797865875 shutdown_connections 2024-01-16T15:06:46.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.989+0000 7f259e800640 1 -- 172.21.15.125:0/2797865875 wait complete. 2024-01-16T15:06:46.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.991+0000 7f259e800640 1 Processor -- start 2024-01-16T15:06:46.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.991+0000 7f259e800640 1 -- start start 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980aeac0 0x7f259814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f259815b090 0x7f2598155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259814f510 con 0x7f25980b2be0 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259814fd90 con 0x7f259815b090 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f259e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2598150bf0 con 0x7f25980aeac0 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f258fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f258fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54684/0 (socket says 172.21.15.125:54684) 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.992+0000 7f258fe00640 1 -- 172.21.15.125:0/3540398043 learned_addr learned my addr 172.21.15.125:0/3540398043 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:46.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f2597e00640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f259815b090 0x7f2598155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f2597400640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980aeac0 0x7f259814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 -- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980aeac0 msgr2=0x7f259814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980aeac0 0x7f259814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 -- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f259815b090 msgr2=0x7f2598155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f259815b090 0x7f2598155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 -- 172.21.15.125:0/3540398043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880094d0 con 0x7f25980b2be0 2024-01-16T15:06:46.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258fe00640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f258000d7d0 tx=0x7f258000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.993+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258000b6c0 con 0x7f25980b2be0 2024-01-16T15:06:46.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.994+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258000b820 con 0x7f25980b2be0 2024-01-16T15:06:46.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.994+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258000bbb0 con 0x7f25980b2be0 2024-01-16T15:06:46.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.994+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2598153850 con 0x7f25980b2be0 2024-01-16T15:06:46.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.994+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259814ffe0 con 0x7f25980b2be0 2024-01-16T15:06:46.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.995+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2598150d60 con 0x7f25980b2be0 2024-01-16T15:06:46.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.996+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f258000bd10 con 0x7f25980b2be0 2024-01-16T15:06:46.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.997+0000 7f258f400640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 0x7f257006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.997+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2580088b60 con 0x7f25980b2be0 2024-01-16T15:06:46.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.997+0000 7f2597400640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 0x7f257006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:46.998+0000 7f2597400640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 0x7f257006e2c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2588036000 tx=0x7f2588008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.002+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25800553f0 con 0x7f25980b2be0 2024-01-16T15:06:47.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.355+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969111", "target": ["mon-mgr", ""]}) v1 -- 0x7f25980ce400 con 0x7f257006be00 2024-01-16T15:06:47.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.687+0000 7f258f400640 1 -- 172.21.15.125:0/3540398043 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2588030b90 con 0x7f257006be00 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.693+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 msgr2=0x7f257006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.693+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 0x7f257006e2c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2588036000 tx=0x7f2588008030 comp rx=0 tx=0).stop 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.693+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 msgr2=0x7f2598153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.693+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f258000d7d0 tx=0x7f258000dc90 comp rx=0 tx=0).stop 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.693+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 shutdown_connections 2024-01-16T15:06:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f259815b090 0x7f2598155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006be00 0x7f257006e2c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b2be0 0x7f2598153310 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 --2- 172.21.15.125:0/3540398043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980aeac0 0x7f259814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 >> 172.21.15.125:0/3540398043 conn(0x7f2598070a80 msgr2=0x7f259806a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 shutdown_connections 2024-01-16T15:06:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:47.694+0000 7f259e800640 1 -- 172.21.15.125:0/3540398043 wait complete. 2024-01-16T15:06:47.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969112 2024-01-16T15:06:48.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.110+0000 7f7b5cc00640 1 Processor -- start 2024-01-16T15:06:48.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.112+0000 7f7b5cc00640 1 -- start start 2024-01-16T15:06:48.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.115+0000 7f7b5cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b58070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.115+0000 7f7b5cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7b58070ea0 0x7f7b580cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.115+0000 7f7b5cc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b58071690 con 0x7f7b5814b2c0 2024-01-16T15:06:48.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.115+0000 7f7b5cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5814b6a0 con 0x7f7b58070ea0 2024-01-16T15:06:48.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.115+0000 7f7b5cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58071970 con 0x7f7b580cf9a0 2024-01-16T15:06:48.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.116+0000 7f7b55e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7b5814b2c0 0x7f7b580cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51870/0 (socket says 172.21.15.125:51870) 2024-01-16T15:06:48.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.116+0000 7f7b55e00640 1 -- 172.21.15.125:0/3798638426 learned_addr learned my addr 172.21.15.125:0/3798638426 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:48.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.116+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281142798 0 0) 0x7f7b58071690 con 0x7f7b5814b2c0 2024-01-16T15:06:48.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.116+0000 7f7b56800640 1 --2- 172.21.15.125:0/3798638426 >> v2:172.21.15.137:3300/0 conn(0x7f7b58070ea0 0x7f7b580cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.117+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b38003520 con 0x7f7b5814b2c0 2024-01-16T15:06:48.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377451584 0 0) 0x7f7b38003520 con 0x7f7b5814b2c0 2024-01-16T15:06:48.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 msgr2=0x7f7b58070960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:06:48.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b58070960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 >> v2:172.21.15.137:3300/0 conn(0x7f7b58070ea0 msgr2=0x7f7b580cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 --2- 172.21.15.125:0/3798638426 >> v2:172.21.15.137:3300/0 conn(0x7f7b58070ea0 0x7f7b580cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58071c50 con 0x7f7b5814b2c0 2024-01-16T15:06:48.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.118+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7b40001670 con 0x7f7b5814b2c0 2024-01-16T15:06:48.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.120+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 >> v1:172.21.15.125:6789/0 conn(0x7f7b5814b2c0 legacy=0x7f7b580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.121+0000 7f7b54a00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b38003970 0x7f7b38003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.121+0000 7f7b54a00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b38004320 0x7f7b38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.122+0000 7f7b54a00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b38005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.123+0000 7f7b55e00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b38003970 0x7f7b38003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.123+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5814b6a0 con 0x7f7b38003970 2024-01-16T15:06:48.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.123+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58071970 con 0x7f7b38004320 2024-01-16T15:06:48.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.123+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3800ba00 con 0x7f7b580cf9a0 2024-01-16T15:06:48.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.124+0000 7f7b56800640 1 --2- 172.21.15.125:0/3798638426 >> v2:172.21.15.137:3300/0 conn(0x7f7b58070ea0 0x7f7b580cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.124+0000 7f7b55400640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b38004320 0x7f7b38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.126+0000 7f7b56800640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b38005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55e00640 1 -- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 msgr2=0x7f7b38005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55e00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b38005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55e00640 1 -- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b38004320 msgr2=0x7f7b38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55e00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b38004320 0x7f7b38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55e00640 1 -- 172.21.15.125:0/3798638426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38003520 con 0x7f7b38003970 2024-01-16T15:06:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b55400640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b38004320 0x7f7b38004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.128+0000 7f7b56800640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b580cf9a0 0x7f7b38005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:48.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.130+0000 7f7b55e00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b38003970 0x7f7b38003d50 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005060 tx=0x7f7b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda3908924414606 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.131+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40005ad0 con 0x7f7b38003970 2024-01-16T15:06:48.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.131+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4002f070 con 0x7f7b38003970 2024-01-16T15:06:48.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.131+0000 7f7b54a00640 1 -- 172.21.15.125:0/3798638426 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40005e90 con 0x7f7b38003970 2024-01-16T15:06:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.132+0000 7f7b5cc00640 1 -- 172.21.15.125:0/3798638426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b38003970 msgr2=0x7f7b38003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.132+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/3798638426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b38003970 0x7f7b38003d50 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005060 tx=0x7f7b380057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.133+0000 7f7b56800640 1 -- 172.21.15.125:0/3798638426 reap_dead start 2024-01-16T15:06:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.133+0000 7f7b5cc00640 1 -- 172.21.15.125:0/3798638426 shutdown_connections 2024-01-16T15:06:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.133+0000 7f7b5cc00640 1 -- 172.21.15.125:0/3798638426 >> 172.21.15.125:0/3798638426 conn(0x7f7b5806a800 msgr2=0x7f7b5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.134+0000 7f7b5cc00640 1 -- 172.21.15.125:0/3798638426 shutdown_connections 2024-01-16T15:06:48.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.134+0000 7f7b5cc00640 1 -- 172.21.15.125:0/3798638426 wait complete. 2024-01-16T15:06:48.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.138+0000 7f7b5cc00640 1 Processor -- start 2024-01-16T15:06:48.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.139+0000 7f7b5cc00640 1 -- start start 2024-01-16T15:06:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.142+0000 7f7b5cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b38003970 0x7f7b58163cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.142+0000 7f7b5cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 0x7f7b58162770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.142+0000 7f7b5cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.143+0000 7f7b5cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58145690 con 0x7f7b581f7630 2024-01-16T15:06:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.143+0000 7f7b5cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58142f60 con 0x7f7b581f7ab0 2024-01-16T15:06:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.143+0000 7f7b5cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58143240 con 0x7f7b38003970 2024-01-16T15:06:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.143+0000 7f7b56800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.144+0000 7f7b56800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:35610/0 (socket says 172.21.15.125:35610) 2024-01-16T15:06:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.144+0000 7f7b56800640 1 -- 172.21.15.125:0/1054256910 learned_addr learned my addr 172.21.15.125:0/1054256910 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.145+0000 7f7b55400640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 0x7f7b58162770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.145+0000 7f7b55e00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b38003970 0x7f7b58163cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.146+0000 7f7b56800640 1 -- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b38003970 msgr2=0x7f7b58163cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.146+0000 7f7b56800640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b38003970 0x7f7b58163cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.146+0000 7f7b56800640 1 -- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 msgr2=0x7f7b58162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.146+0000 7f7b56800640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 0x7f7b58162770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.147+0000 7f7b56800640 1 -- 172.21.15.125:0/1054256910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58143520 con 0x7f7b581f7ab0 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.147+0000 7f7b55400640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 0x7f7b58162770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.147+0000 7f7b56800640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7b5806d650 tx=0x7f7b5000f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.148+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5000e040 con 0x7f7b581f7ab0 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.148+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5000f040 con 0x7f7b581f7ab0 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.148+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50003df0 con 0x7f7b581f7ab0 2024-01-16T15:06:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.148+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b58143800 con 0x7f7b581f7ab0 2024-01-16T15:06:48.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.148+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b581643d0 con 0x7f7b581f7ab0 2024-01-16T15:06:48.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.152+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28005250 con 0x7f7b581f7ab0 2024-01-16T15:06:48.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.159+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f7b50004030 con 0x7f7b581f7ab0 2024-01-16T15:06:48.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.160+0000 7f7b4e000640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 0x7f7b3006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.160+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7b50088d20 con 0x7f7b581f7ab0 2024-01-16T15:06:48.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.161+0000 7f7b55e00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 0x7f7b3006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.162+0000 7f7b55e00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 0x7f7b3006e340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7b40034000 tx=0x7f7b4000a070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.175+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b50055620 con 0x7f7b581f7ab0 2024-01-16T15:06:48.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:48.686+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969112", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b28000c80 con 0x7f7b3006be80 2024-01-16T15:06:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.034+0000 7f7b4e000640 1 -- 172.21.15.125:0/1054256910 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7b28000c80 con 0x7f7b3006be80 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 msgr2=0x7f7b3006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 0x7f7b3006e340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7b40034000 tx=0x7f7b4000a070 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 msgr2=0x7f7b58162cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7b5806d650 tx=0x7f7b5000f7b0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 shutdown_connections 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b581f7ab0 0x7f7b58162cb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b3006be80 0x7f7b3006e340 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b581f7630 0x7f7b58162770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 --2- 172.21.15.125:0/1054256910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b38003970 0x7f7b58163cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.045+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 >> 172.21.15.125:0/1054256910 conn(0x7f7b5806a800 msgr2=0x7f7b58149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.046+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 shutdown_connections 2024-01-16T15:06:49.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.046+0000 7f7b5cc00640 1 -- 172.21.15.125:0/1054256910 wait complete. 2024-01-16T15:06:49.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969113 2024-01-16T15:06:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.518+0000 7f1fe0800640 1 Processor -- start 2024-01-16T15:06:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.519+0000 7f1fe0800640 1 -- start start 2024-01-16T15:06:49.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.522+0000 7f1fe0800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fd814b2c0 0x7f1fd80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.522+0000 7f1fe0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.522+0000 7f1fe0800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd80cfbf0 con 0x7f1fd8070ea0 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.522+0000 7f1fe0800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd814b6a0 con 0x7f1fd814b2c0 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.522+0000 7f1fe0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8071690 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fd814b2c0 0x7f1fd80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fd814b2c0 0x7f1fd80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54700/0 (socket says 172.21.15.125:54700) 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 -- 172.21.15.125:0/701189678 learned_addr learned my addr 172.21.15.125:0/701189678 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 -- 172.21.15.125:0/701189678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 msgr2=0x7f1fd8070960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 --2- 172.21.15.125:0/701189678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd8070960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 -- 172.21.15.125:0/701189678 >> v1:172.21.15.125:6789/0 conn(0x7f1fd8070ea0 legacy=0x7f1fd80b1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:06:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.523+0000 7f1fdde00640 1 -- 172.21.15.125:0/701189678 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8071970 con 0x7f1fd814b2c0 2024-01-16T15:06:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.525+0000 7f1fdde00640 1 --2- 172.21.15.125:0/701189678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80cf290 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009870 tx=0x7f1fc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ccffa996196eaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.526+0000 7f1fdca00640 1 -- 172.21.15.125:0/701189678 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8031870 con 0x7f1fd814b2c0 2024-01-16T15:06:49.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.527+0000 7f1fdca00640 1 -- 172.21.15.125:0/701189678 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80319d0 con 0x7f1fd814b2c0 2024-01-16T15:06:49.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.527+0000 7f1fdca00640 1 -- 172.21.15.125:0/701189678 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8031d60 con 0x7f1fd814b2c0 2024-01-16T15:06:49.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.528+0000 7f1fe0800640 1 -- 172.21.15.125:0/701189678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 msgr2=0x7f1fd80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.528+0000 7f1fe0800640 1 --2- 172.21.15.125:0/701189678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80cf290 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009870 tx=0x7f1fc802b910 comp rx=0 tx=0).stop 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 -- 172.21.15.125:0/701189678 shutdown_connections 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 --2- 172.21.15.125:0/701189678 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd8070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 --2- 172.21.15.125:0/701189678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80cf290 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 -- 172.21.15.125:0/701189678 >> 172.21.15.125:0/701189678 conn(0x7f1fd806a800 msgr2=0x7f1fd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 -- 172.21.15.125:0/701189678 shutdown_connections 2024-01-16T15:06:49.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.529+0000 7f1fe0800640 1 -- 172.21.15.125:0/701189678 wait complete. 2024-01-16T15:06:49.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.532+0000 7f1fe0800640 1 Processor -- start 2024-01-16T15:06:49.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.532+0000 7f1fe0800640 1 -- start start 2024-01-16T15:06:49.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 0x7f1fd8154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8147de0 con 0x7f1fd814b2c0 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8147160 con 0x7f1fd8070ea0 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.534+0000 7f1fe0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8147440 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.535+0000 7f1fdde00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 0x7f1fd8154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.535+0000 7f1fdd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.535+0000 7f1fdd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:54812/0 (socket says 172.21.15.125:54812) 2024-01-16T15:06:49.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.535+0000 7f1fdd400640 1 -- 172.21.15.125:0/115558165 learned_addr learned my addr 172.21.15.125:0/115558165 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:49.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.536+0000 7f1fde800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.537+0000 7f1fdd400640 1 -- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 msgr2=0x7f1fd8154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.537+0000 7f1fdd400640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 0x7f1fd8154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.539+0000 7f1fdd400640 1 -- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 msgr2=0x7f1fd80af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.539+0000 7f1fdd400640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.539+0000 7f1fdd400640 1 -- 172.21.15.125:0/115558165 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80094d0 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.539+0000 7f1fde800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.539+0000 7f1fdde00640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 0x7f1fd8154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.543+0000 7f1fdd400640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b450 tx=0x7f1fc000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.544+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0011890 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.545+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc0011a10 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.545+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd8147720 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.545+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd8155510 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.545+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0011dc0 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.554+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f1fc001a400 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.555+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa8005250 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.556+0000 7f1fd6000640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 0x7f1fb006e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.556+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1fc00888e0 con 0x7f1fd80cf7d0 2024-01-16T15:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.556+0000 7f1fdde00640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 0x7f1fb006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.558+0000 7f1fdde00640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 0x7f1fb006e310 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8036000 tx=0x7f1fc803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:49.566+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fc0015080 con 0x7f1fd80cf7d0 2024-01-16T15:06:50.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.048+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969113", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fa8000c80 con 0x7f1fb006be50 2024-01-16T15:06:50.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.123+0000 7f1fd6000640 1 -- 172.21.15.125:0/115558165 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1fa8000c80 con 0x7f1fb006be50 2024-01-16T15:06:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 msgr2=0x7f1fb006e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 0x7f1fb006e310 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8036000 tx=0x7f1fc803c040 comp rx=0 tx=0).stop 2024-01-16T15:06:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 msgr2=0x7f1fd80aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b450 tx=0x7f1fc000b910 comp rx=0 tx=0).stop 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 shutdown_connections 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fb006be50 0x7f1fb006e310 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fd814b2c0 0x7f1fd80af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fd80cf7d0 0x7f1fd80aeb50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 --2- 172.21.15.125:0/115558165 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fd8070ea0 0x7f1fd8154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 >> 172.21.15.125:0/115558165 conn(0x7f1fd806a800 msgr2=0x7f1fd814a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 shutdown_connections 2024-01-16T15:06:50.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.125+0000 7f1fe0800640 1 -- 172.21.15.125:0/115558165 wait complete. 2024-01-16T15:06:50.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969114 2024-01-16T15:06:50.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.321+0000 7fab8e400640 1 Processor -- start 2024-01-16T15:06:50.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.322+0000 7fab8e400640 1 -- start start 2024-01-16T15:06:50.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab8e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab880cf9a0 0x7fab88070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab8e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab88070ea0 0x7fab880cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab8e400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab88071690 con 0x7fab8814b2c0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab8e400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8814b6a0 con 0x7fab880cf9a0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab8e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab88071970 con 0x7fab88070ea0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.323+0000 7fab7e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab880cf9a0 0x7fab88070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab880cf9a0 0x7fab88070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54718/0 (socket says 172.21.15.125:54718) 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 -- 172.21.15.125:0/958937577 learned_addr learned my addr 172.21.15.125:0/958937577 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab87400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fab8814b2c0 0x7fab880cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51882/0 (socket says 172.21.15.125:51882) 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab87e00640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab88070ea0 0x7fab880cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 -- 172.21.15.125:0/958937577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab88070ea0 msgr2=0x7fab880cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab88070ea0 0x7fab880cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 -- 172.21.15.125:0/958937577 >> v1:172.21.15.125:6789/0 conn(0x7fab8814b2c0 legacy=0x7fab880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab7e800640 1 -- 172.21.15.125:0/958937577 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88071c50 con 0x7fab880cf9a0 2024-01-16T15:06:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.324+0000 7fab86a00640 1 -- 172.21.15.125:0/958937577 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170326915 0 0) 0x7fab88071690 con 0x7fab8814b2c0 2024-01-16T15:06:50.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.325+0000 7fab7e800640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab880cf9a0 0x7fab88070960 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6960b982f53081 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.325+0000 7fab86a00640 1 -- 172.21.15.125:0/958937577 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70030840 con 0x7fab880cf9a0 2024-01-16T15:06:50.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.325+0000 7fab86a00640 1 -- 172.21.15.125:0/958937577 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab700309a0 con 0x7fab880cf9a0 2024-01-16T15:06:50.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.325+0000 7fab86a00640 1 -- 172.21.15.125:0/958937577 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70030d30 con 0x7fab880cf9a0 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 -- 172.21.15.125:0/958937577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab880cf9a0 msgr2=0x7fab88070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab880cf9a0 0x7fab88070960 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b980 comp rx=0 tx=0).stop 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 -- 172.21.15.125:0/958937577 shutdown_connections 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab88070ea0 0x7fab880cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 --2- 172.21.15.125:0/958937577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab880cf9a0 0x7fab88070960 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 -- 172.21.15.125:0/958937577 >> 172.21.15.125:0/958937577 conn(0x7fab8806a800 msgr2=0x7fab8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 -- 172.21.15.125:0/958937577 shutdown_connections 2024-01-16T15:06:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.326+0000 7fab8e400640 1 -- 172.21.15.125:0/958937577 wait complete. 2024-01-16T15:06:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.328+0000 7fab8e400640 1 Processor -- start 2024-01-16T15:06:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.328+0000 7fab8e400640 1 -- start start 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 0x7fab88144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab880cf9a0 0x7fab881647d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab88145780 con 0x7fab8814b2c0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab88144ee0 con 0x7fab88070ea0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab8e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab881451c0 con 0x7fab880cf9a0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54724/0 (socket says 172.21.15.125:54724) 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 -- 172.21.15.125:0/2655513051 learned_addr learned my addr 172.21.15.125:0/2655513051 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 0x7fab88144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 -- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab880cf9a0 msgr2=0x7fab881647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab7e800640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab880cf9a0 0x7fab881647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab880cf9a0 0x7fab881647d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 -- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 msgr2=0x7fab88144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 0x7fab88144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87e00640 1 -- 172.21.15.125:0/2655513051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700094d0 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.329+0000 7fab87400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 0x7fab88144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab87e00640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fab80007e40 tx=0x7fab8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80014620 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8000ee90 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab881454a0 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab800149e0 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.330+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab881f6b30 con 0x7fab8814b2c0 2024-01-16T15:06:50.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.331+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8813fb90 con 0x7fab8814b2c0 2024-01-16T15:06:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.332+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fab80014b40 con 0x7fab8814b2c0 2024-01-16T15:06:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.333+0000 7fab7fe00640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 0x7fab6006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.333+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fab80088ec0 con 0x7fab8814b2c0 2024-01-16T15:06:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.333+0000 7fab87400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 0x7fab6006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.334+0000 7fab87400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 0x7fab6006e2f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fab78000f60 tx=0x7fab78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.337+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab8813fb90 con 0x7fab8814b2c0 2024-01-16T15:06:50.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:50.639+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969114", "target": ["mon-mgr", ""]}) v1 -- 0x7fab88071330 con 0x7fab6006be30 2024-01-16T15:06:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.169+0000 7fab7fe00640 1 -- 172.21.15.125:0/2655513051 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab780012f0 con 0x7fab6006be30 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 msgr2=0x7fab6006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 0x7fab6006e2f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fab78000f60 tx=0x7fab78009040 comp rx=0 tx=0).stop 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 msgr2=0x7fab881f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fab80007e40 tx=0x7fab8000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 shutdown_connections 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006be30 0x7fab6006e2f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8814b2c0 0x7fab881f65f0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab880cf9a0 0x7fab881647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 --2- 172.21.15.125:0/2655513051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab88070ea0 0x7fab88144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.179+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 >> 172.21.15.125:0/2655513051 conn(0x7fab8806a800 msgr2=0x7fab88149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.180+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 shutdown_connections 2024-01-16T15:06:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.180+0000 7fab8e400640 1 -- 172.21.15.125:0/2655513051 wait complete. 2024-01-16T15:06:51.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969115 2024-01-16T15:06:51.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.597+0000 7f2d0e000640 1 Processor -- start 2024-01-16T15:06:51.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.598+0000 7f2d0e000640 1 -- start start 2024-01-16T15:06:51.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.600+0000 7f2d0e000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2d08150760 0x7f2d0815d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.600+0000 7f2d0e000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0815dc70 con 0x7f2d0815d840 2024-01-16T15:06:51.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.600+0000 7f2d0e000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d08150ba0 con 0x7f2d08072230 2024-01-16T15:06:51.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.600+0000 7f2d0e000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d08072610 con 0x7f2d08150760 2024-01-16T15:06:51.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.601+0000 7f2d07400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2d08072230 0x7f2d080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:40754/0 (socket says 172.21.15.125:40754) 2024-01-16T15:06:51.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.601+0000 7f2d07400640 1 -- 172.21.15.125:0/3237539763 learned_addr learned my addr 172.21.15.125:0/3237539763 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:51.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.603+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659318291 0 0) 0x7f2d08150ba0 con 0x7f2d08072230 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.603+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce8003520 con 0x7f2d08072230 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.603+0000 7f2d06a00640 1 --2- 172.21.15.125:0/3237539763 >> v2:172.21.15.137:3300/0 conn(0x7f2d08150760 0x7f2d0815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3320880495 0 0) 0x7f2ce8003520 con 0x7f2d08072230 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 >> v2:172.21.15.137:3300/0 conn(0x7f2d08150760 msgr2=0x7f2d0815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 --2- 172.21.15.125:0/3237539763 >> v2:172.21.15.137:3300/0 conn(0x7f2d08150760 0x7f2d0815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 >> v1:172.21.15.125:6789/0 conn(0x7f2d0815d840 legacy=0x7f2d080b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d08071190 con 0x7f2d08072230 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2cf0001670 con 0x7f2d08072230 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 >> v1:172.21.15.137:6789/0 conn(0x7f2d08072230 legacy=0x7f2d080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce8003970 0x7f2ce8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce80047c0 0x7f2ce8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 0x7f2ce800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08072610 con 0x7f2ce8005180 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce800c0d0 con 0x7f2ce8003970 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.604+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce800c290 con 0x7f2ce80047c0 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.606+0000 7f2d07e00640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 0x7f2ce800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.607+0000 7f2d06a00640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce80047c0 0x7f2ce8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.607+0000 7f2d07400640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce8003970 0x7f2ce8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.609+0000 7f2d07e00640 1 -- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce80047c0 msgr2=0x7f2ce8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.609+0000 7f2d07e00640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce80047c0 0x7f2ce8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.609+0000 7f2d07e00640 1 -- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce8003970 msgr2=0x7f2ce8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.609+0000 7f2d07e00640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce8003970 0x7f2ce8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.609+0000 7f2d07e00640 1 -- 172.21.15.125:0/3237539763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d08071190 con 0x7f2ce8005180 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.610+0000 7f2d07400640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ce8003970 0x7f2ce8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.613+0000 7f2d07e00640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 0x7f2ce800b980 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2d00001e50 tx=0x7f2ce80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6bd116572c583b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.614+0000 7f2d07e00640 1 -- 172.21.15.125:0/3237539763 reap_dead start 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.615+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00003b70 con 0x7f2ce8005180 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.615+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00003cf0 con 0x7f2ce8005180 2024-01-16T15:06:51.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.616+0000 7f2d0e000640 1 -- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 msgr2=0x7f2ce800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.617+0000 7f2d0e000640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 0x7f2ce800b980 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2d00001e50 tx=0x7f2ce80057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.617+0000 7f2d06000640 1 -- 172.21.15.125:0/3237539763 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00005310 con 0x7f2ce8005180 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.618+0000 7f2d0e000640 1 -- 172.21.15.125:0/3237539763 shutdown_connections 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.618+0000 7f2d0e000640 1 --2- 172.21.15.125:0/3237539763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ce8005180 0x7f2ce800b980 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.618+0000 7f2d0e000640 1 -- 172.21.15.125:0/3237539763 >> 172.21.15.125:0/3237539763 conn(0x7f2d0806a800 msgr2=0x7f2d0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.618+0000 7f2d0e000640 1 -- 172.21.15.125:0/3237539763 shutdown_connections 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.619+0000 7f2d0e000640 1 -- 172.21.15.125:0/3237539763 wait complete. 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.623+0000 7f2d0e000640 1 Processor -- start 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.623+0000 7f2d0e000640 1 -- start start 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce8005180 0x7f2d080cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 0x7f2d0815b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080ceac0 con 0x7f2d0813e960 2024-01-16T15:06:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080afd20 con 0x7f2d0813e4e0 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.625+0000 7f2d0e000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080cdeb0 con 0x7f2ce8005180 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.626+0000 7f2d06a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.626+0000 7f2d07e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 0x7f2d0815b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.626+0000 7f2d06a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:35644/0 (socket says 172.21.15.125:35644) 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.626+0000 7f2d06a00640 1 -- 172.21.15.125:0/1398358375 learned_addr learned my addr 172.21.15.125:0/1398358375 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.626+0000 7f2d07400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce8005180 0x7f2d080cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.628+0000 7f2d07e00640 1 -- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce8005180 msgr2=0x7f2d080cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.628+0000 7f2d07e00640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce8005180 0x7f2d080cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.628+0000 7f2d07e00640 1 -- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 msgr2=0x7f2d0815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.628+0000 7f2d07e00640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.628+0000 7f2d07e00640 1 -- 172.21.15.125:0/1398358375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080ce190 con 0x7f2d0813e960 2024-01-16T15:06:51.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.631+0000 7f2d06a00640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:51.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.633+0000 7f2d07e00640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 0x7f2d0815b1a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2d080cb0c0 tx=0x7f2d000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.633+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00003e60 con 0x7f2d0813e960 2024-01-16T15:06:51.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.634+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00007030 con 0x7f2d0813e960 2024-01-16T15:06:51.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.634+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d000073c0 con 0x7f2d0813e960 2024-01-16T15:06:51.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.635+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d080ce470 con 0x7f2d0813e960 2024-01-16T15:06:51.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.635+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0815b6e0 con 0x7f2d0813e960 2024-01-16T15:06:51.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.639+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd8005250 con 0x7f2d0813e960 2024-01-16T15:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.640+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f2d00007520 con 0x7f2d0813e960 2024-01-16T15:06:51.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.642+0000 7f2cff400640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 0x7f2ce006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.642+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2d000aa730 con 0x7f2d0813e960 2024-01-16T15:06:51.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.642+0000 7f2d07400640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 0x7f2ce006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.643+0000 7f2d07400640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 0x7f2ce006e340 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2d080a8870 tx=0x7f2cf0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:51.649+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d00076da0 con 0x7f2d0813e960 2024-01-16T15:06:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.140+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969115", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cd8000c80 con 0x7f2ce006be80 2024-01-16T15:06:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.653+0000 7f2cff400640 1 -- 172.21.15.125:0/1398358375 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2cd8000c80 con 0x7f2ce006be80 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 msgr2=0x7f2ce006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 0x7f2ce006e340 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2d080a8870 tx=0x7f2cf0009040 comp rx=0 tx=0).stop 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 msgr2=0x7f2d0815b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 0x7f2d0815b1a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2d080cb0c0 tx=0x7f2d000014a0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 shutdown_connections 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ce006be80 0x7f2ce006e340 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d0813e960 0x7f2d0815b1a0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d0813e4e0 0x7f2d0815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 --2- 172.21.15.125:0/1398358375 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ce8005180 0x7f2d080cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 >> 172.21.15.125:0/1398358375 conn(0x7f2d0806a800 msgr2=0x7f2d080b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 shutdown_connections 2024-01-16T15:06:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:52.657+0000 7f2d0e000640 1 -- 172.21.15.125:0/1398358375 wait complete. 2024-01-16T15:06:52.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969116 2024-01-16T15:06:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.142+0000 7f1863400640 1 Processor -- start 2024-01-16T15:06:53.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.142+0000 7f1863400640 1 -- start start 2024-01-16T15:06:53.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c162110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f185c06d3b0 0x7f185c162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0702c0 con 0x7f185c161d30 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c161b90 con 0x7f185c06d3b0 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1863400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c070840 con 0x7f185c0b3a50 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1860a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1860a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54742/0 (socket says 172.21.15.125:54742) 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.145+0000 7f1860a00640 1 -- 172.21.15.125:0/1515244535 learned_addr learned my addr 172.21.15.125:0/1515244535 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:53.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.146+0000 7f1861400640 1 --2- 172.21.15.125:0/1515244535 >> v2:172.21.15.137:3300/0 conn(0x7f185c06d3b0 0x7f185c162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.147+0000 7f1861400640 1 -- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 msgr2=0x7f185c162110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.148+0000 7f1861400640 1 --2- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c162110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.148+0000 7f1861400640 1 -- 172.21.15.125:0/1515244535 >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 msgr2=0x7f185c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.148+0000 7f1861400640 1 --2- 172.21.15.125:0/1515244535 >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.148+0000 7f1861400640 1 -- 172.21.15.125:0/1515244535 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c162b90 con 0x7f185c06d3b0 2024-01-16T15:06:53.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.150+0000 7f1860a00640 1 --2- 172.21.15.125:0/1515244535 >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:53.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.154+0000 7f1861400640 1 --2- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c06d3b0 0x7f185c162650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f185c0707b0 tx=0x7f184402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90a133f74c330bf1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.155+0000 7f185b400640 1 -- 172.21.15.125:0/1515244535 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184402e070 con 0x7f185c06d3b0 2024-01-16T15:06:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.155+0000 7f185b400640 1 -- 172.21.15.125:0/1515244535 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440315a0 con 0x7f185c06d3b0 2024-01-16T15:06:53.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.155+0000 7f185b400640 1 -- 172.21.15.125:0/1515244535 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844031930 con 0x7f185c06d3b0 2024-01-16T15:06:53.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.159+0000 7f1863400640 1 -- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c06d3b0 msgr2=0x7f185c162650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.159+0000 7f1863400640 1 --2- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c06d3b0 0x7f185c162650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f185c0707b0 tx=0x7f184402b840 comp rx=0 tx=0).stop 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.160+0000 7f1863400640 1 -- 172.21.15.125:0/1515244535 shutdown_connections 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.160+0000 7f1863400640 1 --2- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c06d3b0 0x7f185c162650 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.160+0000 7f1863400640 1 --2- 172.21.15.125:0/1515244535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c162110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.161+0000 7f1863400640 1 --2- 172.21.15.125:0/1515244535 >> v2:172.21.15.125:3300/0 conn(0x7f185c161d30 0x7f185c0b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.161+0000 7f1863400640 1 -- 172.21.15.125:0/1515244535 >> 172.21.15.125:0/1515244535 conn(0x7f185c0b0be0 msgr2=0x7f185c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.161+0000 7f1863400640 1 -- 172.21.15.125:0/1515244535 shutdown_connections 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.161+0000 7f1863400640 1 -- 172.21.15.125:0/1515244535 wait complete. 2024-01-16T15:06:53.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.163+0000 7f1863400640 1 Processor -- start 2024-01-16T15:06:53.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.163+0000 7f1863400640 1 -- start start 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c14a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c161d30 0x7f185c1e8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0adc20 con 0x7f185c06d3b0 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ad660 con 0x7f185c161d30 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1863400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ad940 con 0x7f185c0b3a50 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.165+0000 7f1860a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54756/0 (socket says 172.21.15.125:54756) 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 -- 172.21.15.125:0/3277238848 learned_addr learned my addr 172.21.15.125:0/3277238848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 -- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 msgr2=0x7f185c14a040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c14a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 -- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c161d30 msgr2=0x7f185c1e8780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c161d30 0x7f185c1e8780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.166+0000 7f1860a00640 1 -- 172.21.15.125:0/3277238848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0094d0 con 0x7f185c06d3b0 2024-01-16T15:06:53.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.167+0000 7f1860a00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f184c00e9b0 tx=0x7f184c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.167+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c00c9a0 con 0x7f185c06d3b0 2024-01-16T15:06:53.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.168+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c00cb00 con 0x7f185c06d3b0 2024-01-16T15:06:53.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.168+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1844009400 con 0x7f185c06d3b0 2024-01-16T15:06:53.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.168+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c00ce70 con 0x7f185c06d3b0 2024-01-16T15:06:53.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.168+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0a9920 con 0x7f185c06d3b0 2024-01-16T15:06:53.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.172+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c0a9b10 con 0x7f185c06d3b0 2024-01-16T15:06:53.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.178+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f184c013470 con 0x7f185c06d3b0 2024-01-16T15:06:53.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.181+0000 7f1858c00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 0x7f183406e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.182+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f184c088ae0 con 0x7f185c06d3b0 2024-01-16T15:06:53.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.185+0000 7f185be00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 0x7f183406e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.191+0000 7f185be00640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 0x7f183406e390 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f185400ab90 tx=0x7f1854009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.199+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f184c055110 con 0x7f185c06d3b0 2024-01-16T15:06:53.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:53.578+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969116", "target": ["mon-mgr", ""]}) v1 -- 0x7f185c0b41d0 con 0x7f183406bed0 2024-01-16T15:06:55.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.197+0000 7f1858c00640 1 -- 172.21.15.125:0/3277238848 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f185400a7a0 con 0x7f183406bed0 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 msgr2=0x7f183406e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 0x7f183406e390 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f185400ab90 tx=0x7f1854009250 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 msgr2=0x7f185c0abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f184c00e9b0 tx=0x7f184c00ee70 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 shutdown_connections 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f185c161d30 0x7f185c1e8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f185c0b3a50 0x7f185c14a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f183406bed0 0x7f183406e390 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 --2- 172.21.15.125:0/3277238848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f185c06d3b0 0x7f185c0abe60 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 >> 172.21.15.125:0/3277238848 conn(0x7f185c0b0be0 msgr2=0x7f185c15f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 shutdown_connections 2024-01-16T15:06:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.201+0000 7f1863400640 1 -- 172.21.15.125:0/3277238848 wait complete. 2024-01-16T15:06:55.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969117 2024-01-16T15:06:55.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.431+0000 7fbe5ea00640 1 Processor -- start 2024-01-16T15:06:55.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.431+0000 7fbe5ea00640 1 -- start start 2024-01-16T15:06:55.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.432+0000 7fbe5ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbe58143ae0 0x7fbe580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.432+0000 7fbe5ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe580cfc30 con 0x7fbe580cf7d0 2024-01-16T15:06:55.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.432+0000 7fbe5ea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe580b1ee0 con 0x7fbe5815e3b0 2024-01-16T15:06:55.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.432+0000 7fbe5ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe580b2780 con 0x7fbe58143ae0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.433+0000 7fbe4f200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbe580cf7d0 0x7fbe5815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51892/0 (socket says 172.21.15.125:51892) 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.433+0000 7fbe4f200640 1 -- 172.21.15.125:0/3643202304 learned_addr learned my addr 172.21.15.125:0/3643202304 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.433+0000 7fbe57e00640 1 --2- 172.21.15.125:0/3643202304 >> v2:172.21.15.137:3300/0 conn(0x7fbe58143ae0 0x7fbe580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.433+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128108698 0 0) 0x7fbe580cfc30 con 0x7fbe580cf7d0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.433+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38003520 con 0x7fbe580cf7d0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265990670 0 0) 0x7fbe38003520 con 0x7fbe580cf7d0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 >> v2:172.21.15.137:3300/0 conn(0x7fbe58143ae0 msgr2=0x7fbe580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 --2- 172.21.15.125:0/3643202304 >> v2:172.21.15.137:3300/0 conn(0x7fbe58143ae0 0x7fbe580cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 >> v1:172.21.15.137:6789/0 conn(0x7fbe5815e3b0 legacy=0x7fbe580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5815e990 con 0x7fbe580cf7d0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fbe40001670 con 0x7fbe580cf7d0 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 >> v1:172.21.15.125:6789/0 conn(0x7fbe580cf7d0 legacy=0x7fbe5815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe380039c0 0x7fbe38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe380047c0 0x7fbe38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 0x7fbe3800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580b2780 con 0x7fbe38005180 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3800e290 con 0x7fbe380047c0 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3800e450 con 0x7fbe380039c0 2024-01-16T15:06:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.434+0000 7fbe5ca00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 0x7fbe3800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe57e00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe380039c0 0x7fbe38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe5ca00640 1 -- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe380039c0 msgr2=0x7fbe38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe5ca00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe380039c0 0x7fbe38003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe4f200640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe380047c0 0x7fbe38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe5ca00640 1 -- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe380047c0 msgr2=0x7fbe38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe5ca00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe380047c0 0x7fbe38004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.435+0000 7fbe5ca00640 1 -- 172.21.15.125:0/3643202304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38003520 con 0x7fbe38005180 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.436+0000 7fbe5ca00640 1 -- 172.21.15.125:0/3643202304 reap_dead start 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.436+0000 7fbe5ca00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 0x7fbe3800db90 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fbe50001750 tx=0x7fbe380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651eb99bf3def01 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.436+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500052d0 con 0x7fbe38005180 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.436+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5002e050 con 0x7fbe38005180 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.436+0000 7fbe57400640 1 -- 172.21.15.125:0/3643202304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500035f0 con 0x7fbe38005180 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 msgr2=0x7fbe3800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 0x7fbe3800db90 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fbe50001750 tx=0x7fbe380057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3643202304 shutdown_connections 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3643202304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe38005180 0x7fbe3800db90 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3643202304 >> 172.21.15.125:0/3643202304 conn(0x7fbe5806a800 msgr2=0x7fbe5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3643202304 shutdown_connections 2024-01-16T15:06:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.437+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3643202304 wait complete. 2024-01-16T15:06:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.439+0000 7fbe5ea00640 1 Processor -- start 2024-01-16T15:06:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.439+0000 7fbe5ea00640 1 -- start start 2024-01-16T15:06:55.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe38005180 0x7fbe58159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe580c3ee0 0x7fbe580c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58159fd0 con 0x7fbe58140340 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58140d00 con 0x7fbe580c3ee0 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580c48a0 con 0x7fbe38005180 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54788/0 (socket says 172.21.15.125:54788) 2024-01-16T15:06:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 -- 172.21.15.125:0/3154205574 learned_addr learned my addr 172.21.15.125:0/3154205574 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:55.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe5ca00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe580c3ee0 0x7fbe580c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 -- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe38005180 msgr2=0x7fbe58159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe38005180 0x7fbe58159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 -- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe580c3ee0 msgr2=0x7fbe580c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe580c3ee0 0x7fbe580c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.440+0000 7fbe4f200640 1 -- 172.21.15.125:0/3154205574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58135250 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe4f200640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000d770 tx=0x7fbe4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400074e0 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe40007640 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400049e0 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe581343a0 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.441+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe58134920 con 0x7fbe58140340 2024-01-16T15:06:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.442+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe580675d0 con 0x7fbe58140340 2024-01-16T15:06:55.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.443+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fbe40004b40 con 0x7fbe58140340 2024-01-16T15:06:55.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.444+0000 7fbe54c00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 0x7fbe3006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.444+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fbe40087f90 con 0x7fbe58140340 2024-01-16T15:06:55.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.444+0000 7fbe57e00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 0x7fbe3006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.445+0000 7fbe57e00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 0x7fbe3006e2f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbe48007440 tx=0x7fbe48009750 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.448+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe40054620 con 0x7fbe58140340 2024-01-16T15:06:55.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.675+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969117", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe580b2410 con 0x7fbe3006be30 2024-01-16T15:06:55.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.819+0000 7fbe54c00640 1 -- 172.21.15.125:0/3154205574 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe48011040 con 0x7fbe3006be30 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 msgr2=0x7fbe3006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 0x7fbe3006e2f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbe48007440 tx=0x7fbe48009750 comp rx=0 tx=0).stop 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 msgr2=0x7fbe581407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000d770 tx=0x7fbe4000dc30 comp rx=0 tx=0).stop 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 shutdown_connections 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe580c3ee0 0x7fbe580c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbe3006be30 0x7fbe3006e2f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe58140340 0x7fbe581407c0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 --2- 172.21.15.125:0/3154205574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe38005180 0x7fbe58159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 >> 172.21.15.125:0/3154205574 conn(0x7fbe5806a800 msgr2=0x7fbe58070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:55.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 shutdown_connections 2024-01-16T15:06:55.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:55.822+0000 7fbe5ea00640 1 -- 172.21.15.125:0/3154205574 wait complete. 2024-01-16T15:06:55.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969118 2024-01-16T15:06:56.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.044+0000 7fadd8e00640 1 Processor -- start 2024-01-16T15:06:56.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.044+0000 7fadd8e00640 1 -- start start 2024-01-16T15:06:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.045+0000 7fadd8e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fadd415da10 0x7fadd4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.045+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.045+0000 7fadd8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd4071190 con 0x7fadd4150760 2024-01-16T15:06:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd8e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd4071470 con 0x7fadd415da10 2024-01-16T15:06:56.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4071750 con 0x7fadd40b2bd0 2024-01-16T15:06:56.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd1400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fadd415da10 0x7fadd4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd1400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fadd415da10 0x7fadd4072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54800/0 (socket says 172.21.15.125:54800) 2024-01-16T15:06:56.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd1400640 1 -- 172.21.15.125:0/3631476440 learned_addr learned my addr 172.21.15.125:0/3631476440 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:56.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.046+0000 7fadd2800640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.047+0000 7fadd1400640 1 -- 172.21.15.125:0/3631476440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 msgr2=0x7fadd40b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.047+0000 7fadd1400640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.047+0000 7fadd1400640 1 -- 172.21.15.125:0/3631476440 >> v1:172.21.15.137:6789/0 conn(0x7fadd4150760 legacy=0x7fadd415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.047+0000 7fadd1400640 1 -- 172.21.15.125:0/3631476440 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4071a30 con 0x7fadd415da10 2024-01-16T15:06:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.048+0000 7fadd1400640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4072230 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009e50 tx=0x7fadbc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5143e1c56470c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.048+0000 7fadd0a00640 1 -- 172.21.15.125:0/3631476440 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc02cb00 con 0x7fadd415da10 2024-01-16T15:06:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.048+0000 7fadd0a00640 1 -- 172.21.15.125:0/3631476440 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc02cc60 con 0x7fadd415da10 2024-01-16T15:06:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.048+0000 7fadd0a00640 1 -- 172.21.15.125:0/3631476440 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0343f0 con 0x7fadd415da10 2024-01-16T15:06:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 -- 172.21.15.125:0/3631476440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 msgr2=0x7fadd4072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4072230 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009e50 tx=0x7fadbc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 -- 172.21.15.125:0/3631476440 shutdown_connections 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 --2- 172.21.15.125:0/3631476440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4072230 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 -- 172.21.15.125:0/3631476440 >> 172.21.15.125:0/3631476440 conn(0x7fadd406a800 msgr2=0x7fadd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 -- 172.21.15.125:0/3631476440 shutdown_connections 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.049+0000 7fadd8e00640 1 -- 172.21.15.125:0/3631476440 wait complete. 2024-01-16T15:06:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.051+0000 7fadd8e00640 1 Processor -- start 2024-01-16T15:06:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.051+0000 7fadd8e00640 1 -- start start 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadd4150760 0x7fadd4146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd414e900 con 0x7fadd415da10 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd414ee20 con 0x7fadd4150760 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd414f100 con 0x7fadd40b2bd0 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54814/0 (socket says 172.21.15.125:54814) 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd2800640 1 -- 172.21.15.125:0/382711027 learned_addr learned my addr 172.21.15.125:0/382711027 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd1400640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadd4150760 0x7fadd4146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd1e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd2800640 1 -- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 msgr2=0x7fadd40cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.052+0000 7fadd2800640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd2800640 1 -- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadd4150760 msgr2=0x7fadd4146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd2800640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadd4150760 0x7fadd4146dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd2800640 1 -- 172.21.15.125:0/382711027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc009b30 con 0x7fadd415da10 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd1e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd2800640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00d800 tx=0x7fadcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00b6c0 con 0x7fadd415da10 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00b840 con 0x7fadd415da10 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd414f3e0 con 0x7fadd415da10 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00bbf0 con 0x7fadd415da10 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.053+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd414fbe0 con 0x7fadd415da10 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.054+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd414f900 con 0x7fadd415da10 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.055+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fadcc00bd50 con 0x7fadd415da10 2024-01-16T15:06:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.057+0000 7fadca000640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 0x7fadac06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.057+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fadcc088b30 con 0x7fadd415da10 2024-01-16T15:06:56.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.057+0000 7fadd1e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 0x7fadac06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.058+0000 7fadd1e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 0x7fadac06e2c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000f60 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.061+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadd414f900 con 0x7fadd415da10 2024-01-16T15:06:56.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.289+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969118", "target": ["mon-mgr", ""]}) v1 -- 0x7fadd4150b40 con 0x7fadac06be00 2024-01-16T15:06:56.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.440+0000 7fadca000640 1 -- 172.21.15.125:0/382711027 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fadc4001d30 con 0x7fadac06be00 2024-01-16T15:06:56.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 msgr2=0x7fadac06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 0x7fadac06e2c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000f60 tx=0x7fadc4009040 comp rx=0 tx=0).stop 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 msgr2=0x7fadd4147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00d800 tx=0x7fadcc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 shutdown_connections 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fadac06be00 0x7fadac06e2c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fadd415da10 0x7fadd4147310 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fadd4150760 0x7fadd4146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 --2- 172.21.15.125:0/382711027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fadd40b2bd0 0x7fadd40cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 >> 172.21.15.125:0/382711027 conn(0x7fadd406a800 msgr2=0x7fadd40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 shutdown_connections 2024-01-16T15:06:56.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.443+0000 7fadd8e00640 1 -- 172.21.15.125:0/382711027 wait complete. 2024-01-16T15:06:56.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969119 2024-01-16T15:06:56.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.675+0000 7f5976600640 1 Processor -- start 2024-01-16T15:06:56.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.675+0000 7f5976600640 1 -- start start 2024-01-16T15:06:56.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f5976600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5970072230 0x7f59700b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f5976600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5970071190 con 0x7f5970150760 2024-01-16T15:06:56.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f5976600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5970071470 con 0x7f597015da10 2024-01-16T15:06:56.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f5976600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5970071750 con 0x7f5970072230 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f596f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5970150760 0x7f597015d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34318/0 (socket says 172.21.15.125:34318) 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f596f400640 1 -- 172.21.15.125:0/2025261886 learned_addr learned my addr 172.21.15.125:0/2025261886 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f596fe00640 1 --2- 172.21.15.125:0/2025261886 >> v2:172.21.15.137:3300/0 conn(0x7f5970072230 0x7f59700b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.677+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090277923 0 0) 0x7f5970071190 con 0x7f5970150760 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5950003520 con 0x7f5970150760 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3430531891 0 0) 0x7f5950003520 con 0x7f5970150760 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 >> v2:172.21.15.137:3300/0 conn(0x7f5970072230 msgr2=0x7f59700b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 --2- 172.21.15.125:0/2025261886 >> v2:172.21.15.137:3300/0 conn(0x7f5970072230 0x7f59700b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 >> v1:172.21.15.137:6789/0 conn(0x7f597015da10 legacy=0x7f59700b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970071a30 con 0x7f5970150760 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5960001670 con 0x7f5970150760 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 >> v1:172.21.15.125:6789/0 conn(0x7f5970150760 legacy=0x7f597015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5950003970 0x7f5950003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f5950004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5950005180 0x7f595000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970071750 con 0x7f59500047c0 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595000c020 con 0x7f5950003970 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595000c1e0 con 0x7f5950005180 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.678+0000 7f596ea00640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f5950004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596f400640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5950003970 0x7f5950003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596ea00640 1 -- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5950005180 msgr2=0x7f595000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596ea00640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5950005180 0x7f595000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596ea00640 1 -- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5950003970 msgr2=0x7f5950003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596ea00640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5950003970 0x7f5950003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596ea00640 1 -- 172.21.15.125:0/2025261886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950003520 con 0x7f59500047c0 2024-01-16T15:06:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.679+0000 7f596fe00640 1 -- 172.21.15.125:0/2025261886 reap_dead start 2024-01-16T15:06:56.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.680+0000 7f596ea00640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f5950004c40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5958001750 tx=0x7f59500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5301fe6f83c7f679 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.680+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59580052d0 con 0x7f59500047c0 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.680+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595802e050 con 0x7f59500047c0 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.680+0000 7f596e000640 1 -- 172.21.15.125:0/2025261886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958003830 con 0x7f59500047c0 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 -- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 msgr2=0x7f5950004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f5950004c40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5958001750 tx=0x7f59500057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 -- 172.21.15.125:0/2025261886 shutdown_connections 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 --2- 172.21.15.125:0/2025261886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f5950004c40 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 -- 172.21.15.125:0/2025261886 >> 172.21.15.125:0/2025261886 conn(0x7f597006a800 msgr2=0x7f597006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 -- 172.21.15.125:0/2025261886 shutdown_connections 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.681+0000 7f5976600640 1 -- 172.21.15.125:0/2025261886 wait complete. 2024-01-16T15:06:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.683+0000 7f5976600640 1 Processor -- start 2024-01-16T15:06:56.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.683+0000 7f5976600640 1 -- start start 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f597014f980 0x7f5970157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5970157db0 0x7f5970158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597014fe00 con 0x7f59500047c0 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59700ce790 con 0x7f5970157db0 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f5976600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59700cea70 con 0x7f597014f980 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f596f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f596f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60922/0 (socket says 172.21.15.125:60922) 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.684+0000 7f596f400640 1 -- 172.21.15.125:0/3363540986 learned_addr learned my addr 172.21.15.125:0/3363540986 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 -- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f597014f980 msgr2=0x7f5970157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f597014f980 0x7f5970157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 -- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5970157db0 msgr2=0x7f5970158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5970157db0 0x7f5970158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 -- 172.21.15.125:0/3363540986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597012e570 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f596f400640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f596000e780 tx=0x7f596000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000ee30 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596000c8d0 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000cc80 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597012e850 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.685+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5970159720 con 0x7f59500047c0 2024-01-16T15:06:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.686+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970159940 con 0x7f59500047c0 2024-01-16T15:06:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.687+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f596001d460 con 0x7f59500047c0 2024-01-16T15:06:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.688+0000 7f5967400640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 0x7f594806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.688+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f5960088430 con 0x7f59500047c0 2024-01-16T15:06:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.689+0000 7f596ea00640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 0x7f594806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.689+0000 7f596ea00640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 0x7f594806e2c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5958033000 tx=0x7f595802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.693+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5970159940 con 0x7f59500047c0 2024-01-16T15:06:56.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:56.919+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969119", "target": ["mon-mgr", ""]}) v1 -- 0x7f597012e3f0 con 0x7f594806be00 2024-01-16T15:06:57.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.033+0000 7f5967400640 1 -- 172.21.15.125:0/3363540986 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5958005270 con 0x7f594806be00 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 msgr2=0x7f594806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 0x7f594806e2c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5958033000 tx=0x7f595802f000 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 msgr2=0x7f597014f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f596000e780 tx=0x7f596000ec40 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 shutdown_connections 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5970157db0 0x7f5970158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f597014f980 0x7f5970157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f594806be00 0x7f594806e2c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 --2- 172.21.15.125:0/3363540986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59500047c0 0x7f597014f440 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 >> 172.21.15.125:0/3363540986 conn(0x7f597006a800 msgr2=0x7f597006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 shutdown_connections 2024-01-16T15:06:57.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.036+0000 7f5976600640 1 -- 172.21.15.125:0/3363540986 wait complete. 2024-01-16T15:06:57.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone969120 2024-01-16T15:06:57.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.268+0000 7f9310800640 1 Processor -- start 2024-01-16T15:06:57.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.268+0000 7f9310800640 1 -- start start 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f9310800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9308143940 0x7f930806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f9310800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143d20 0x7f93080cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f9310800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930806ea10 con 0x7f9308146900 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f9310800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930806f3b0 con 0x7f9308143940 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f9310800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080ce720 con 0x7f9308143d20 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9308143940 0x7f930806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9308143940 0x7f930806f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60924/0 (socket says 172.21.15.125:60924) 2024-01-16T15:06:57.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 -- 172.21.15.125:0/2628721612 learned_addr learned my addr 172.21.15.125:0/2628721612 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 -- 172.21.15.125:0/2628721612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143d20 msgr2=0x7f93080cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 --2- 172.21.15.125:0/2628721612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143d20 0x7f93080cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 -- 172.21.15.125:0/2628721612 >> v1:172.21.15.137:6789/0 conn(0x7f9308146900 legacy=0x7f93080d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.270+0000 7f930de00640 1 -- 172.21.15.125:0/2628721612 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930806ecf0 con 0x7f9308143940 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.272+0000 7f930de00640 1 --2- 172.21.15.125:0/2628721612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143940 0x7f930806f7d0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009870 tx=0x7f92f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee84892744826111 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.272+0000 7f930ca00640 1 -- 172.21.15.125:0/2628721612 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f80317c0 con 0x7f9308143940 2024-01-16T15:06:57.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.272+0000 7f930ca00640 1 -- 172.21.15.125:0/2628721612 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f8031920 con 0x7f9308143940 2024-01-16T15:06:57.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.272+0000 7f930ca00640 1 -- 172.21.15.125:0/2628721612 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8031cb0 con 0x7f9308143940 2024-01-16T15:06:57.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 -- 172.21.15.125:0/2628721612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143940 msgr2=0x7f930806f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 --2- 172.21.15.125:0/2628721612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143940 0x7f930806f7d0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009870 tx=0x7f92f802b860 comp rx=0 tx=0).stop 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 -- 172.21.15.125:0/2628721612 shutdown_connections 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 --2- 172.21.15.125:0/2628721612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143d20 0x7f93080cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 --2- 172.21.15.125:0/2628721612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143940 0x7f930806f7d0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 -- 172.21.15.125:0/2628721612 >> 172.21.15.125:0/2628721612 conn(0x7f93080ccc70 msgr2=0x7f93080cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 -- 172.21.15.125:0/2628721612 shutdown_connections 2024-01-16T15:06:57.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.273+0000 7f9310800640 1 -- 172.21.15.125:0/2628721612 wait complete. 2024-01-16T15:06:57.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.275+0000 7f9310800640 1 Processor -- start 2024-01-16T15:06:57.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.275+0000 7f9310800640 1 -- start start 2024-01-16T15:06:57.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143940 0x7f93081381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 0x7f93080b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080c5600 con 0x7f9308143d20 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080b49f0 con 0x7f9308146900 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f9310800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080b4cd0 con 0x7f9308143940 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f930d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f930d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60926/0 (socket says 172.21.15.125:60926) 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f930d400640 1 -- 172.21.15.125:0/3334188421 learned_addr learned my addr 172.21.15.125:0/3334188421 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f930de00640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143940 0x7f93081381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.277+0000 7f930e800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 0x7f93080b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 -- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143940 msgr2=0x7f93081381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143940 0x7f93081381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 -- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 msgr2=0x7f93080b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 0x7f93080b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 -- 172.21.15.125:0/3334188421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80094d0 con 0x7f9308143d20 2024-01-16T15:06:57.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930e800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 0x7f93080b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:57.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f930d400640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f92f000b450 tx=0x7f92f000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0011890 con 0x7f9308143d20 2024-01-16T15:06:57.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f00119f0 con 0x7f9308143d20 2024-01-16T15:06:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0011da0 con 0x7f9308143d20 2024-01-16T15:06:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93080c4fb0 con 0x7f9308143d20 2024-01-16T15:06:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.278+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93080b59a0 con 0x7f9308143d20 2024-01-16T15:06:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.280+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93080675d0 con 0x7f9308143d20 2024-01-16T15:06:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.281+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f92f001a400 con 0x7f9308143d20 2024-01-16T15:06:57.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.281+0000 7f9306000640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 0x7f92e006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.282+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f92f0088940 con 0x7f9308143d20 2024-01-16T15:06:57.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.282+0000 7f930de00640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 0x7f92e006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.282+0000 7f930de00640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 0x7f92e006e2c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f92f8036000 tx=0x7f92f803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.287+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92f00bde90 con 0x7f9308143d20 2024-01-16T15:06:57.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.519+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone969120", "target": ["mon-mgr", ""]}) v1 -- 0x7f930806af20 con 0x7f92e006be00 2024-01-16T15:06:57.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.650+0000 7f9306000640 1 -- 172.21.15.125:0/3334188421 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f930806af20 con 0x7f92e006be00 2024-01-16T15:06:57.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.653+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 msgr2=0x7f92e006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.653+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 0x7f92e006e2c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f92f8036000 tx=0x7f92f803d040 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.653+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 msgr2=0x7f93080aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.653+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f92f000b450 tx=0x7f92f000b910 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 shutdown_connections 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9308146900 0x7f93080b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92e006be00 0x7f92e006e2c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9308143d20 0x7f93080aeb50 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 --2- 172.21.15.125:0/3334188421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9308143940 0x7f93081381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 >> 172.21.15.125:0/3334188421 conn(0x7f93080ccc70 msgr2=0x7f93080afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 shutdown_connections 2024-01-16T15:06:57.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.654+0000 7f9310800640 1 -- 172.21.15.125:0/3334188421 wait complete. 2024-01-16T15:06:57.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:57.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.880+0000 7f9f2ba00640 1 Processor -- start 2024-01-16T15:06:57.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.881+0000 7f9f2ba00640 1 -- start start 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9f240cd240 0x7f9f240b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f2406c6b0 0x7f9f24099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f24099fa0 0x7f9f2406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2409a430 con 0x7f9f2406c6b0 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2406cea0 con 0x7f9f240cd240 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.882+0000 7f9f2ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2406cbc0 con 0x7f9f24099fa0 2024-01-16T15:06:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f2406c6b0 0x7f9f24099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f2406c6b0 0x7f9f24099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60936/0 (socket says 172.21.15.125:60936) 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 -- 172.21.15.125:0/3635767368 learned_addr learned my addr 172.21.15.125:0/3635767368 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f29000640 1 --2- 172.21.15.125:0/3635767368 >> v2:172.21.15.137:3300/0 conn(0x7f9f240cd240 0x7f9f240b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f29a00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f24099fa0 0x7f9f2406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 -- 172.21.15.125:0/3635767368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f24099fa0 msgr2=0x7f9f2406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f24099fa0 0x7f9f2406b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 -- 172.21.15.125:0/3635767368 >> v2:172.21.15.137:3300/0 conn(0x7f9f240cd240 msgr2=0x7f9f240b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3635767368 >> v2:172.21.15.137:3300/0 conn(0x7f9f240cd240 0x7f9f240b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.883+0000 7f9f1bc00640 1 -- 172.21.15.125:0/3635767368 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2406f350 con 0x7f9f2406c6b0 2024-01-16T15:06:57.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.884+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f24099a60 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f9f24070a10 tx=0x7f9f0c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f984b1a907da0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.884+0000 7f9f23e00640 1 -- 172.21.15.125:0/3635767368 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c031860 con 0x7f9f2406c6b0 2024-01-16T15:06:57.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f23e00640 1 -- 172.21.15.125:0/3635767368 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c0319c0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f23e00640 1 -- 172.21.15.125:0/3635767368 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c031d50 con 0x7f9f2406c6b0 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3635767368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 msgr2=0x7f9f24099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f24099a60 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f9f24070a10 tx=0x7f9f0c02b900 comp rx=0 tx=0).stop 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3635767368 shutdown_connections 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f24099fa0 0x7f9f2406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3635767368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f24099a60 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3635767368 >> v2:172.21.15.137:3300/0 conn(0x7f9f240cd240 0x7f9f240b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3635767368 >> 172.21.15.125:0/3635767368 conn(0x7f9f240b2d00 msgr2=0x7f9f240b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3635767368 shutdown_connections 2024-01-16T15:06:57.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.885+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3635767368 wait complete. 2024-01-16T15:06:57.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.887+0000 7f9f2ba00640 1 Processor -- start 2024-01-16T15:06:57.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.887+0000 7f9f2ba00640 1 -- start start 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 0x7f9f24149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f240cd240 0x7f9f240aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240c92f0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240ca0d0 con 0x7f9f24099fa0 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.888+0000 7f9f2ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240c9010 con 0x7f9f240cd240 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60944/0 (socket says 172.21.15.125:60944) 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 -- 172.21.15.125:0/3127947535 learned_addr learned my addr 172.21.15.125:0/3127947535 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 0x7f9f24149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29a00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f240cd240 0x7f9f240aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 -- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f240cd240 msgr2=0x7f9f240aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f240cd240 0x7f9f240aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 -- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 msgr2=0x7f9f24149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 0x7f9f24149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 -- 172.21.15.125:0/3127947535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c0094d0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 0x7f9f24149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:57.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.889+0000 7f9f29000640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400ce40 tx=0x7f9f1400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.890+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14003b90 con 0x7f9f2406c6b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.890+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14003cf0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.890+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f240c9a80 con 0x7f9f2406c6b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.890+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14004240 con 0x7f9f2406c6b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.890+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f240b45a0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.891+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f240c98e0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.892+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f9f140043a0 con 0x7f9f2406c6b0 2024-01-16T15:06:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.893+0000 7f9f21600640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 0x7f9efc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.893+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f9f14012070 con 0x7f9f2406c6b0 2024-01-16T15:06:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.893+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 0x7f9efc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.894+0000 7f9f1bc00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 0x7f9efc06e2c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c036000 tx=0x7f9f0c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:57.897+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f240c98e0 con 0x7f9f2406c6b0 2024-01-16T15:06:58.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.157+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9f240c98e0 con 0x7f9f2406c6b0 2024-01-16T15:06:58.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.158+0000 7f9f21600640 1 -- 172.21.15.125:0/3127947535 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f9f240c98e0 con 0x7f9f2406c6b0 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:06:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 637, 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 72, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 3378, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 13960656925, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 28350361600, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 744743751680, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 398278511, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 409514358, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 95, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 112 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 46, 2024-01-16T15:06:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 5, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 5, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 7125 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:06:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 14, 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:06:10.134652+0000", 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": { 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "17e60f2c-8e84-4bef-8f8b-c4e2f846f3f2": { 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "15 ongoing+pending cloning jobs; Avg progress = 22.223% (39s)\n [======......................] (remaining: 2m)", 2024-01-16T15:06:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.22222666442394257, 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "f934dbe9-8121-4ed6-8ea8-2195897a585d": { 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "message": "4 ongoing cloning jobs; Avg progress = 83.335% (39s)\n [=======================.....] (remaining: 7s)", 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "progress": 0.83335000276565552, 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "add_to_ceph_s": true 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:06:58.143 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 msgr2=0x7f9efc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 0x7f9efc06e2c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c036000 tx=0x7f9f0c03c040 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 msgr2=0x7f9f240b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400ce40 tx=0x7f9f1400a660 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 shutdown_connections 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f240cd240 0x7f9f240aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f24099fa0 0x7f9f24149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9efc06be00 0x7f9efc06e2c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 --2- 172.21.15.125:0/3127947535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f2406c6b0 0x7f9f240b4060 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 >> 172.21.15.125:0/3127947535 conn(0x7f9f240b2d00 msgr2=0x7f9f240b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 shutdown_connections 2024-01-16T15:06:58.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.161+0000 7f9f2ba00640 1 -- 172.21.15.125:0/3127947535 wait complete. 2024-01-16T15:06:58.157 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-16T15:06:58.158 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.22222666442394257; onpen_p = 0.2857142984867096 2024-01-16T15:06:58.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96911 2024-01-16T15:06:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.393+0000 7f5456800640 1 Processor -- start 2024-01-16T15:06:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.394+0000 7f5456800640 1 -- start start 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f5456800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f54500cd600 0x7f5450070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f5456800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450071450 con 0x7f5450157c30 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f5456800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545015deb0 con 0x7f5450071030 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f5456800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545015ea30 con 0x7f54500cd600 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f544f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5450157c30 0x7f54501618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34320/0 (socket says 172.21.15.125:34320) 2024-01-16T15:06:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.395+0000 7f544f400640 1 -- 172.21.15.125:0/1873495683 learned_addr learned my addr 172.21.15.125:0/1873495683 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544ea00640 1 --2- 172.21.15.125:0/1873495683 >> v2:172.21.15.137:3300/0 conn(0x7f54500cd600 0x7f5450070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107268579 0 0) 0x7f5450071450 con 0x7f5450157c30 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5430003520 con 0x7f5450157c30 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91808753 0 0) 0x7f5430003520 con 0x7f5450157c30 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 >> v2:172.21.15.137:3300/0 conn(0x7f54500cd600 msgr2=0x7f5450070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 --2- 172.21.15.125:0/1873495683 >> v2:172.21.15.137:3300/0 conn(0x7f54500cd600 0x7f5450070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 >> v1:172.21.15.137:6789/0 conn(0x7f5450071030 legacy=0x7f545006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545015e190 con 0x7f5450157c30 2024-01-16T15:06:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5440001670 con 0x7f5450157c30 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 >> v1:172.21.15.125:6789/0 conn(0x7f5450157c30 legacy=0x7f54501618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 0x7f5430003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.396+0000 7f544e000640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544e000640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5430005180 0x7f543000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545015ea30 con 0x7f5430003970 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543000c020 con 0x7f5430005180 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543000c1e0 con 0x7f54300047c0 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 0x7f5430003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 -- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54300047c0 msgr2=0x7f5430004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 -- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5430005180 msgr2=0x7f543000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5430005180 0x7f543000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544f400640 1 -- 172.21.15.125:0/1873495683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5430003520 con 0x7f5430003970 2024-01-16T15:06:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.397+0000 7f544fe00640 1 -- 172.21.15.125:0/1873495683 reap_dead start 2024-01-16T15:06:58.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.398+0000 7f544f400640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 0x7f5430003d90 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f544000c190 tx=0x7f54300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13be9c203f01aff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440005ad0 con 0x7f5430003970 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5440005c30 con 0x7f5430003970 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f544e000640 1 -- 172.21.15.125:0/1873495683 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544002cae0 con 0x7f5430003970 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f5456800640 1 -- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 msgr2=0x7f5430003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f5456800640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 0x7f5430003d90 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f544000c190 tx=0x7f54300057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f5456800640 1 -- 172.21.15.125:0/1873495683 shutdown_connections 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f5456800640 1 --2- 172.21.15.125:0/1873495683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5430003970 0x7f5430003d90 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.399+0000 7f5456800640 1 -- 172.21.15.125:0/1873495683 >> 172.21.15.125:0/1873495683 conn(0x7f54500cf700 msgr2=0x7f54500cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.400+0000 7f5456800640 1 -- 172.21.15.125:0/1873495683 shutdown_connections 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.400+0000 7f5456800640 1 -- 172.21.15.125:0/1873495683 wait complete. 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.401+0000 7f5456800640 1 Processor -- start 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.401+0000 7f5456800640 1 -- start start 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5430003970 0x7f54500ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5450146de0 0x7f5450158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545015b6a0 0x7f545015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545015c060 con 0x7f5450146de0 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450147f40 con 0x7f545015b6a0 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.402+0000 7f5456800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54501592d0 con 0x7f5430003970 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5450146de0 0x7f5450158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5450146de0 0x7f5450158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60966/0 (socket says 172.21.15.125:60966) 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 -- 172.21.15.125:0/1825659607 learned_addr learned my addr 172.21.15.125:0/1825659607 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 -- 172.21.15.125:0/1825659607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5430003970 msgr2=0x7f54500ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5430003970 0x7f54500ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 -- 172.21.15.125:0/1825659607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545015b6a0 msgr2=0x7f545015bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f545015b6a0 0x7f545015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 -- 172.21.15.125:0/1825659607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450147610 con 0x7f5450146de0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.403+0000 7f544ea00640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5450146de0 0x7f5450158a80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f543800e860 tx=0x7f543800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.404+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438009600 con 0x7f5450146de0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.404+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543800c940 con 0x7f5450146de0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.404+0000 7f5456800640 1 -- 172.21.15.125:0/1825659607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54501478f0 con 0x7f5450146de0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.404+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800ccd0 con 0x7f5450146de0 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.404+0000 7f5456800640 1 -- 172.21.15.125:0/1825659607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450158fc0 con 0x7f5450146de0 2024-01-16T15:06:58.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.405+0000 7f5456800640 1 -- 172.21.15.125:0/1825659607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545006e9f0 con 0x7f5450146de0 2024-01-16T15:06:58.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.406+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f5438013470 con 0x7f5450146de0 2024-01-16T15:06:58.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.407+0000 7f5447400640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f542806be30 0x7f542806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.407+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f5438088940 con 0x7f5450146de0 2024-01-16T15:06:58.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.407+0000 7f544f400640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f542806be30 0x7f542806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.407+0000 7f544f400640 1 --2- 172.21.15.125:0/1825659607 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f542806be30 0x7f542806e2f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5440034000 tx=0x7f5440039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.411+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5438054fb0 con 0x7f5450146de0 2024-01-16T15:06:58.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.640+0000 7f5456800640 1 -- 172.21.15.125:0/1825659607 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96911", "target": ["mon-mgr", ""]}) v1 -- 0x7f545015da50 con 0x7f542806be30 2024-01-16T15:06:58.640 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:06:58.661+0000 7f00fb000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:06:58.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:06:58.662+0000 7f5447400640 1 -- 172.21.15.125:0/1825659607 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f545015da50 con 0x7f542806be30 2024-01-16T15:06:58.640 INFO:teuthology.orchestra.run.smithi125.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:06:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:00.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96912 2024-01-16T15:07:01.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.076+0000 7efcc2000640 1 Processor -- start 2024-01-16T15:07:01.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.077+0000 7efcc2000640 1 -- start start 2024-01-16T15:07:01.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.086+0000 7efcc2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc14b2c0 0x7efcbc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.086+0000 7efcc2000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efcbc071070 0x7efcbc0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcc2000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0cfbf0 con 0x7efcbc0cf7d0 2024-01-16T15:07:01.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcc2000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc14b6a0 con 0x7efcbc071070 2024-01-16T15:07:01.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcc2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc071690 con 0x7efcbc14b2c0 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcbb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc14b2c0 0x7efcbc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcbbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efcbc071070 0x7efcbc0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcbbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efcbc071070 0x7efcbc0cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:56292/0 (socket says 172.21.15.125:56292) 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.087+0000 7efcbbe00640 1 -- 172.21.15.125:0/2037943651 learned_addr learned my addr 172.21.15.125:0/2037943651 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.088+0000 7efcbbe00640 1 -- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc14b2c0 msgr2=0x7efcbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.088+0000 7efcbbe00640 1 --2- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc14b2c0 0x7efcbc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.088+0000 7efcbbe00640 1 -- 172.21.15.125:0/2037943651 >> v1:172.21.15.137:6789/0 conn(0x7efcbc0cf7d0 legacy=0x7efcbc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.088+0000 7efcbbe00640 1 -- 172.21.15.125:0/2037943651 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc071970 con 0x7efcbc071070 2024-01-16T15:07:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.092+0000 7efcbbe00640 1 --2- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc071070 0x7efcbc0cc160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009e50 tx=0x7efcb402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4af9e12e0a76f4cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.092+0000 7efcba000640 1 -- 172.21.15.125:0/2037943651 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb402ca80 con 0x7efcbc071070 2024-01-16T15:07:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.092+0000 7efcba000640 1 -- 172.21.15.125:0/2037943651 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb402cbe0 con 0x7efcbc071070 2024-01-16T15:07:01.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.092+0000 7efcba000640 1 -- 172.21.15.125:0/2037943651 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40343f0 con 0x7efcbc071070 2024-01-16T15:07:01.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 -- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc071070 msgr2=0x7efcbc0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 --2- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc071070 0x7efcbc0cc160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009e50 tx=0x7efcb402bee0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 -- 172.21.15.125:0/2037943651 shutdown_connections 2024-01-16T15:07:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 --2- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc071070 0x7efcbc0cc160 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 --2- 172.21.15.125:0/2037943651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc14b2c0 0x7efcbc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 -- 172.21.15.125:0/2037943651 >> 172.21.15.125:0/2037943651 conn(0x7efcbc06a800 msgr2=0x7efcbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 -- 172.21.15.125:0/2037943651 shutdown_connections 2024-01-16T15:07:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.094+0000 7efcc2000640 1 -- 172.21.15.125:0/2037943651 wait complete. 2024-01-16T15:07:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.099+0000 7efcc2000640 1 Processor -- start 2024-01-16T15:07:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.100+0000 7efcc2000640 1 -- start start 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc071070 0x7efcbc161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc0cf7d0 0x7efcbc1645d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcbc14b2c0 0x7efcbc1477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc143b70 con 0x7efcbc14b2c0 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc13fa20 con 0x7efcbc0cf7d0 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcc2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc142ef0 con 0x7efcbc071070 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcbbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcbc14b2c0 0x7efcbc1477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcbbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcbc14b2c0 0x7efcbc1477d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60970/0 (socket says 172.21.15.125:60970) 2024-01-16T15:07:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcbbe00640 1 -- 172.21.15.125:0/1064175682 learned_addr learned my addr 172.21.15.125:0/1064175682 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.101+0000 7efcbb400640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc071070 0x7efcbc161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 -- 172.21.15.125:0/1064175682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc071070 msgr2=0x7efcbc161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcbc071070 0x7efcbc161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 -- 172.21.15.125:0/1064175682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc0cf7d0 msgr2=0x7efcbc1645d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcbc0cf7d0 0x7efcbc1645d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 -- 172.21.15.125:0/1064175682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4009b30 con 0x7efcbc14b2c0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.102+0000 7efcbbe00640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcbc14b2c0 0x7efcbc1477d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009e50 tx=0x7efcb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.103+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4009510 con 0x7efcbc14b2c0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.103+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb403e450 con 0x7efcbc14b2c0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.103+0000 7efcc2000640 1 -- 172.21.15.125:0/1064175682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1431d0 con 0x7efcbc14b2c0 2024-01-16T15:07:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.103+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb403e800 con 0x7efcbc14b2c0 2024-01-16T15:07:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.103+0000 7efcc2000640 1 -- 172.21.15.125:0/1064175682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc143790 con 0x7efcbc14b2c0 2024-01-16T15:07:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.105+0000 7efcc2000640 1 -- 172.21.15.125:0/1064175682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc143cb0 con 0x7efcbc14b2c0 2024-01-16T15:07:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.105+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7efcb402cd50 con 0x7efcbc14b2c0 2024-01-16T15:07:01.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.108+0000 7efcb3400640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efc9406be00 0x7efc9406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.108+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7efcb40ab020 con 0x7efcbc14b2c0 2024-01-16T15:07:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.110+0000 7efcbb400640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efc9406be00 0x7efc9406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.112+0000 7efcbb400640 1 --2- 172.21.15.125:0/1064175682 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efc9406be00 0x7efc9406e2c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7efcac009a30 tx=0x7efcac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.118+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcbc143cb0 con 0x7efcbc14b2c0 2024-01-16T15:07:01.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.690+0000 7efcc2000640 1 -- 172.21.15.125:0/1064175682 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96912", "target": ["mon-mgr", ""]}) v1 -- 0x7efcbc0cbc40 con 0x7efc9406be00 2024-01-16T15:07:01.711 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:01.733+0000 7f00fb000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:01.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:01.733+0000 7efcb3400640 1 -- 172.21.15.125:0/1064175682 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7efcac009680 con 0x7efc9406be00 2024-01-16T15:07:01.713 INFO:teuthology.orchestra.run.smithi125.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:03.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96913 2024-01-16T15:07:03.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.987+0000 7fee4d600640 1 Processor -- start 2024-01-16T15:07:03.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.987+0000 7fee4d600640 1 -- start start 2024-01-16T15:07:03.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee4d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee480ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee4d600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480d0de0 con 0x7fee48160f80 2024-01-16T15:07:03.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee4d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480d10c0 con 0x7fee480b2a00 2024-01-16T15:07:03.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee4d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d13a0 con 0x7fee480ce8f0 2024-01-16T15:07:03.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee46a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fee48160f80 0x7fee48161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34322/0 (socket says 172.21.15.125:34322) 2024-01-16T15:07:03.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee46a00640 1 -- 172.21.15.125:0/1703959274 learned_addr learned my addr 172.21.15.125:0/1703959274 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:07:03.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339523217 0 0) 0x7fee480d0de0 con 0x7fee48160f80 2024-01-16T15:07:03.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee28003520 con 0x7fee48160f80 2024-01-16T15:07:03.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.989+0000 7fee3fe00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee480ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547813986 0 0) 0x7fee28003520 con 0x7fee48160f80 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 msgr2=0x7fee480ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee480ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 >> v1:172.21.15.137:6789/0 conn(0x7fee480b2a00 legacy=0x7fee480b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee480d1680 con 0x7fee48160f80 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fee38001670 con 0x7fee48160f80 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 >> v1:172.21.15.125:6789/0 conn(0x7fee48160f80 legacy=0x7fee48161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee280039a0 0x7fee28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee280047c0 0x7fee28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee28005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d13a0 con 0x7fee280039a0 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2800bb10 con 0x7fee280047c0 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2800bcd0 con 0x7fee480ce8f0 2024-01-16T15:07:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.990+0000 7fee46a00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee280039a0 0x7fee28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee3fe00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee280047c0 0x7fee28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee47400640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee28005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee46a00640 1 -- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 msgr2=0x7fee28005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee46a00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee480ce8f0 0x7fee28005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee46a00640 1 -- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee280047c0 msgr2=0x7fee28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee46a00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee280047c0 0x7fee28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.991+0000 7fee46a00640 1 -- 172.21.15.125:0/1703959274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28003520 con 0x7fee280039a0 2024-01-16T15:07:03.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.992+0000 7fee46a00640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee280039a0 0x7fee28003dc0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc4b9f39a0767d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.992+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38005ad0 con 0x7fee280039a0 2024-01-16T15:07:03.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.992+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee38005c50 con 0x7fee280039a0 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.992+0000 7fee46000640 1 -- 172.21.15.125:0/1703959274 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3802c810 con 0x7fee280039a0 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 -- 172.21.15.125:0/1703959274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee280039a0 msgr2=0x7fee28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 --2- 172.21.15.125:0/1703959274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee280039a0 0x7fee28003dc0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee280057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee47400640 1 -- 172.21.15.125:0/1703959274 reap_dead start 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 -- 172.21.15.125:0/1703959274 shutdown_connections 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 -- 172.21.15.125:0/1703959274 >> 172.21.15.125:0/1703959274 conn(0x7fee4809c550 msgr2=0x7fee480d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 -- 172.21.15.125:0/1703959274 shutdown_connections 2024-01-16T15:07:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.993+0000 7fee4d600640 1 -- 172.21.15.125:0/1703959274 wait complete. 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.995+0000 7fee4d600640 1 Processor -- start 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.995+0000 7fee4d600640 1 -- start start 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee280039a0 0x7fee4815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee480b0e90 0x7fee480b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee480c7840 0x7fee480c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480b1850 con 0x7fee480b0e90 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee48151fd0 con 0x7fee480c7840 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.996+0000 7fee4d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4815fa90 con 0x7fee280039a0 2024-01-16T15:07:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee480b0e90 0x7fee480b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee480b0e90 0x7fee480b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60994/0 (socket says 172.21.15.125:60994) 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 -- 172.21.15.125:0/1941121366 learned_addr learned my addr 172.21.15.125:0/1941121366 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 -- 172.21.15.125:0/1941121366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee280039a0 msgr2=0x7fee4815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fee280039a0 0x7fee4815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 -- 172.21.15.125:0/1941121366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee480c7840 msgr2=0x7fee480c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fee480c7840 0x7fee480c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 -- 172.21.15.125:0/1941121366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4815ee80 con 0x7fee480b0e90 2024-01-16T15:07:03.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.997+0000 7fee3fe00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee480b0e90 0x7fee480b1310 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fee3000f580 tx=0x7fee3000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.998+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee30014870 con 0x7fee480b0e90 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.998+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300149d0 con 0x7fee480b0e90 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.998+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee30014d80 con 0x7fee480b0e90 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.998+0000 7fee4d600640 1 -- 172.21.15.125:0/1941121366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4815f160 con 0x7fee480b0e90 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.998+0000 7fee4d600640 1 -- 172.21.15.125:0/1941121366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4815f720 con 0x7fee480b0e90 2024-01-16T15:07:03.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:03.999+0000 7fee4d600640 1 -- 172.21.15.125:0/1941121366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee480675d0 con 0x7fee480b0e90 2024-01-16T15:07:03.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.001+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fee30015410 con 0x7fee480b0e90 2024-01-16T15:07:03.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.001+0000 7fee3ea00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2006be00 0x7fee2006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.001+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fee30088f00 con 0x7fee480b0e90 2024-01-16T15:07:03.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.002+0000 7fee46a00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2006be00 0x7fee2006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.002+0000 7fee46a00640 1 --2- 172.21.15.125:0/1941121366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fee2006be00 0x7fee2006e2c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee38007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.007+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee300557a0 con 0x7fee480b0e90 2024-01-16T15:07:04.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.231+0000 7fee4d600640 1 -- 172.21.15.125:0/1941121366 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96913", "target": ["mon-mgr", ""]}) v1 -- 0x7fee480d1c40 con 0x7fee2006be00 2024-01-16T15:07:04.223 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:04.246+0000 7f00fb000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:04.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:04.246+0000 7fee3ea00640 1 -- 172.21.15.125:0/1941121366 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fee380014b0 con 0x7fee2006be00 2024-01-16T15:07:04.224 INFO:teuthology.orchestra.run.smithi125.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:06.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96914 2024-01-16T15:07:06.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.478+0000 7f667ec00640 1 Processor -- start 2024-01-16T15:07:06.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.478+0000 7f667ec00640 1 -- start start 2024-01-16T15:07:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.479+0000 7f667ec00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6678143ae0 0x7f66780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.479+0000 7f667ec00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780cfc30 con 0x7f667815e3b0 2024-01-16T15:07:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.479+0000 7f667ec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780b1ee0 con 0x7f66780cf7d0 2024-01-16T15:07:06.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.479+0000 7f667ec00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780b2780 con 0x7f6678143ae0 2024-01-16T15:07:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f667cc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f667815e3b0 0x7f66780702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34334/0 (socket says 172.21.15.125:34334) 2024-01-16T15:07:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f667cc00640 1 -- 172.21.15.125:0/1074053079 learned_addr learned my addr 172.21.15.125:0/1074053079 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:07:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677e00640 1 --2- 172.21.15.125:0/1074053079 >> v2:172.21.15.137:3300/0 conn(0x7f6678143ae0 0x7f66780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:06.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457286412 0 0) 0x7f66780cfc30 con 0x7f667815e3b0 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6658003520 con 0x7f667815e3b0 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3053603507 0 0) 0x7f6658003520 con 0x7f667815e3b0 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 >> v2:172.21.15.137:3300/0 conn(0x7f6678143ae0 msgr2=0x7f66780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 --2- 172.21.15.125:0/1074053079 >> v2:172.21.15.137:3300/0 conn(0x7f6678143ae0 0x7f66780cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 >> v1:172.21.15.137:6789/0 conn(0x7f66780cf7d0 legacy=0x7f667815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667815e990 con 0x7f667815e3b0 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.480+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f66700019f0 con 0x7f667815e3b0 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 >> v1:172.21.15.125:6789/0 conn(0x7f667815e3b0 legacy=0x7f66780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6658003970 0x7f6658003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f6658004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6658005180 0x7f665800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780b2780 con 0x7f66580047c0 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665800c020 con 0x7f6658005180 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665800c1e0 con 0x7f6658003970 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f6658004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 -- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6658003970 msgr2=0x7f6658003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6658003970 0x7f6658003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 -- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6658005180 msgr2=0x7f665800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6658005180 0x7f665800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f666f200640 1 -- 172.21.15.125:0/1074053079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658003520 con 0x7f66580047c0 2024-01-16T15:07:06.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.481+0000 7f667cc00640 1 -- 172.21.15.125:0/1074053079 reap_dead start 2024-01-16T15:07:06.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.482+0000 7f666f200640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f6658004c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6660001750 tx=0x7f66580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1cb33b10050941 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66600052d0 con 0x7f66580047c0 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666002e050 con 0x7f66580047c0 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f6677400640 1 -- 172.21.15.125:0/1074053079 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660003830 con 0x7f66580047c0 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 -- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 msgr2=0x7f6658004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f6658004c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6660001750 tx=0x7f66580057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 -- 172.21.15.125:0/1074053079 shutdown_connections 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 --2- 172.21.15.125:0/1074053079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f6658004c40 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 -- 172.21.15.125:0/1074053079 >> 172.21.15.125:0/1074053079 conn(0x7f667806a800 msgr2=0x7f667806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.483+0000 7f667ec00640 1 -- 172.21.15.125:0/1074053079 shutdown_connections 2024-01-16T15:07:06.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.484+0000 7f667ec00640 1 -- 172.21.15.125:0/1074053079 wait complete. 2024-01-16T15:07:06.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.485+0000 7f667ec00640 1 Processor -- start 2024-01-16T15:07:06.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.486+0000 7f667ec00640 1 -- start start 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.486+0000 7f667ec00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f66780c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.486+0000 7f667ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f667815c8e0 0x7f66780c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f667ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6678159790 0x7f6678159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f667ec00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c5b00 con 0x7f66580047c0 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f667ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667815a150 con 0x7f6678159790 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f667ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c4ef0 con 0x7f667815c8e0 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f66780c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f66780c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32780/0 (socket says 172.21.15.125:32780) 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 -- 172.21.15.125:0/2649754231 learned_addr learned my addr 172.21.15.125:0/2649754231 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 -- 172.21.15.125:0/2649754231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f667815c8e0 msgr2=0x7f66780c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:06.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f667815c8e0 0x7f66780c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 -- 172.21.15.125:0/2649754231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6678159790 msgr2=0x7f6678159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6678159790 0x7f6678159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.487+0000 7f6677e00640 1 -- 172.21.15.125:0/2649754231 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780c51d0 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f6677e00640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66580047c0 0x7f66780c4860 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f666800e7a0 tx=0x7f666800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800ee50 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666800c940 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f667ec00640 1 -- 172.21.15.125:0/2649754231 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780c54b0 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800ccf0 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.488+0000 7f667ec00640 1 -- 172.21.15.125:0/2649754231 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667815cd60 con 0x7f66580047c0 2024-01-16T15:07:06.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.489+0000 7f667ec00640 1 -- 172.21.15.125:0/2649754231 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780675d0 con 0x7f66580047c0 2024-01-16T15:07:06.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.491+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f6668013470 con 0x7f66580047c0 2024-01-16T15:07:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.492+0000 7f6674c00640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f665006be30 0x7f665006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.492+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f66680887b0 con 0x7f66580047c0 2024-01-16T15:07:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.492+0000 7f666f200640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f665006be30 0x7f665006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:06.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.493+0000 7f666f200640 1 --2- 172.21.15.125:0/2649754231 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f665006be30 0x7f665006e2f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6660033000 tx=0x7f666002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:06.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.497+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6668054e80 con 0x7f66580047c0 2024-01-16T15:07:06.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:06.727+0000 7f667ec00640 1 -- 172.21.15.125:0/2649754231 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7f66780b2410 con 0x7f665006be30 2024-01-16T15:07:08.932 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:08.954+0000 7f00fb000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:08.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:08.954+0000 7f6674c00640 1 -- 172.21.15.125:0/2649754231 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f66600052d0 con 0x7f665006be30 2024-01-16T15:07:08.932 INFO:teuthology.orchestra.run.smithi125.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:09.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:09.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:10.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96915 2024-01-16T15:07:11.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.203+0000 7fbd57200640 1 Processor -- start 2024-01-16T15:07:11.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.203+0000 7fbd57200640 1 -- start start 2024-01-16T15:07:11.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd57200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbd580ccb50 0x7fbd58070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd57200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbd5815bf10 0x7fbd58143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd57200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5815f4e0 con 0x7fbd580adf20 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd57200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5815f970 con 0x7fbd580ccb50 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd57200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd58071130 con 0x7fbd5815bf10 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd55400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbd580ccb50 0x7fbd58070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd55400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbd580ccb50 0x7fbd58070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34496/0 (socket says 172.21.15.125:34496) 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.205+0000 7fbd55400640 1 -- 172.21.15.125:0/3426794820 learned_addr learned my addr 172.21.15.125:0/3426794820 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.206+0000 7fbd56800640 1 --2- 172.21.15.125:0/3426794820 >> v2:172.21.15.137:3300/0 conn(0x7fbd5815bf10 0x7fbd58143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.206+0000 7fbd55400640 1 -- 172.21.15.125:0/3426794820 >> v2:172.21.15.137:3300/0 conn(0x7fbd5815bf10 msgr2=0x7fbd58143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.206+0000 7fbd55400640 1 --2- 172.21.15.125:0/3426794820 >> v2:172.21.15.137:3300/0 conn(0x7fbd5815bf10 0x7fbd58143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.206+0000 7fbd55400640 1 -- 172.21.15.125:0/3426794820 >> v1:172.21.15.137:6789/0 conn(0x7fbd580adf20 legacy=0x7fbd580cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:11.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.206+0000 7fbd55400640 1 -- 172.21.15.125:0/3426794820 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5815ebb0 con 0x7fbd580ccb50 2024-01-16T15:07:11.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.207+0000 7fbd55400640 1 --2- 172.21.15.125:0/3426794820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd58070a30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806ff20 tx=0x7fbd4002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739e5782f28f13b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.208+0000 7fbd54a00640 1 -- 172.21.15.125:0/3426794820 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd400317e0 con 0x7fbd580ccb50 2024-01-16T15:07:11.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.208+0000 7fbd54a00640 1 -- 172.21.15.125:0/3426794820 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd40031940 con 0x7fbd580ccb50 2024-01-16T15:07:11.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.208+0000 7fbd54a00640 1 -- 172.21.15.125:0/3426794820 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40031cd0 con 0x7fbd580ccb50 2024-01-16T15:07:11.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.208+0000 7fbd57200640 1 -- 172.21.15.125:0/3426794820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 msgr2=0x7fbd58070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.208+0000 7fbd57200640 1 --2- 172.21.15.125:0/3426794820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd58070a30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806ff20 tx=0x7fbd4002b880 comp rx=0 tx=0).stop 2024-01-16T15:07:11.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 -- 172.21.15.125:0/3426794820 shutdown_connections 2024-01-16T15:07:11.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 --2- 172.21.15.125:0/3426794820 >> v2:172.21.15.137:3300/0 conn(0x7fbd5815bf10 0x7fbd58143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 --2- 172.21.15.125:0/3426794820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd58070a30 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 -- 172.21.15.125:0/3426794820 >> 172.21.15.125:0/3426794820 conn(0x7fbd5806a7d0 msgr2=0x7fbd5806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 -- 172.21.15.125:0/3426794820 shutdown_connections 2024-01-16T15:07:11.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.209+0000 7fbd57200640 1 -- 172.21.15.125:0/3426794820 wait complete. 2024-01-16T15:07:11.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.210+0000 7fbd57200640 1 Processor -- start 2024-01-16T15:07:11.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.210+0000 7fbd57200640 1 -- start start 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd580adf20 0x7fbd5815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5815bf10 0x7fbd5815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581f40c0 con 0x7fbd580ccb50 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581f43a0 con 0x7fbd5815bf10 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.211+0000 7fbd57200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581f4680 con 0x7fbd580adf20 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34500/0 (socket says 172.21.15.125:34500) 2024-01-16T15:07:11.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 -- 172.21.15.125:0/3807775493 learned_addr learned my addr 172.21.15.125:0/3807775493 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 -- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd580adf20 msgr2=0x7fbd5815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd580adf20 0x7fbd5815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 -- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5815bf10 msgr2=0x7fbd5815db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5815bf10 0x7fbd5815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 -- 172.21.15.125:0/3807775493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd400094d0 con 0x7fbd580ccb50 2024-01-16T15:07:11.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.212+0000 7fbd55400640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbd4002b8c0 tx=0x7fbd400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.213+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40032880 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.213+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd40032a00 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.213+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40032db0 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.213+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581f4960 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.213+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5815a740 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.214+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd581f4e80 con 0x7fbd580ccb50 2024-01-16T15:07:11.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.215+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fbd4004d3f0 con 0x7fbd580ccb50 2024-01-16T15:07:11.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.216+0000 7fbd4e000640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 0x7fbd3006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.217+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fbd400aa340 con 0x7fbd580ccb50 2024-01-16T15:07:11.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.217+0000 7fbd55e00640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 0x7fbd3006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.217+0000 7fbd55e00640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 0x7fbd3006e2c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbd580b00f0 tx=0x7fbd48007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.221+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd581f4e80 con 0x7fbd580ccb50 2024-01-16T15:07:11.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:11.447+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96915", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5815c380 con 0x7fbd3006be00 2024-01-16T15:07:15.086 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:15.108+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:15.086 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:15.108+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:15.086 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:15.108+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:15.086 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:15.108+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:15.086 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:07:15.108+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.581+0000 7fbd4e000640 1 -- 172.21.15.125:0/3807775493 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbd48003fb0 con 0x7fbd3006be00 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 msgr2=0x7fbd3006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 0x7fbd3006e2c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbd580b00f0 tx=0x7fbd48007d30 comp rx=0 tx=0).stop 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 msgr2=0x7fbd5815d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbd4002b8c0 tx=0x7fbd400046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 shutdown_connections 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5815bf10 0x7fbd5815db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd3006be00 0x7fbd3006e2c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd580ccb50 0x7fbd5815d5e0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 --2- 172.21.15.125:0/3807775493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd580adf20 0x7fbd5815a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 >> 172.21.15.125:0/3807775493 conn(0x7fbd5806a7d0 msgr2=0x7fbd58071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 shutdown_connections 2024-01-16T15:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:15.585+0000 7fbd57200640 1 -- 172.21.15.125:0/3807775493 wait complete. 2024-01-16T15:07:17.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96916 2024-01-16T15:07:17.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.831+0000 7f3a3a600640 1 Processor -- start 2024-01-16T15:07:17.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.831+0000 7f3a3a600640 1 -- start start 2024-01-16T15:07:17.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a3a600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a3413e360 0x7f3a340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a3a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3415f9c0 0x7f3a340cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a3a600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3415c5f0 con 0x7f3a3413df80 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a3a600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3406c470 con 0x7f3a3413e360 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a3a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3406c750 con 0x7f3a3415f9c0 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a2a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a3413e360 0x7f3a340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a2a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a3413e360 0x7f3a340cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45970/0 (socket says 172.21.15.125:45970) 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.832+0000 7f3a2a800640 1 -- 172.21.15.125:0/955836559 learned_addr learned my addr 172.21.15.125:0/955836559 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.833+0000 7f3a2a800640 1 -- 172.21.15.125:0/955836559 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3415f9c0 msgr2=0x7f3a340cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.833+0000 7f3a2a800640 1 --2- 172.21.15.125:0/955836559 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3415f9c0 0x7f3a340cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.833+0000 7f3a2a800640 1 -- 172.21.15.125:0/955836559 >> v1:172.21.15.137:6789/0 conn(0x7f3a3413df80 legacy=0x7f3a3415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:17.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.833+0000 7f3a2a800640 1 -- 172.21.15.125:0/955836559 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3406ca30 con 0x7f3a3413e360 2024-01-16T15:07:17.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.834+0000 7f3a2a800640 1 --2- 172.21.15.125:0/955836559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3413e360 0x7f3a340cf290 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f3a3406ded0 tx=0x7f3a1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828be3b6d1f887b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.834+0000 7f3a32a00640 1 -- 172.21.15.125:0/955836559 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c031640 con 0x7f3a3413e360 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.834+0000 7f3a32a00640 1 -- 172.21.15.125:0/955836559 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c0317a0 con 0x7f3a3413e360 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.834+0000 7f3a32a00640 1 -- 172.21.15.125:0/955836559 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c031b30 con 0x7f3a3413e360 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 -- 172.21.15.125:0/955836559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3413e360 msgr2=0x7f3a340cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 --2- 172.21.15.125:0/955836559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3413e360 0x7f3a340cf290 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f3a3406ded0 tx=0x7f3a1c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 -- 172.21.15.125:0/955836559 shutdown_connections 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 --2- 172.21.15.125:0/955836559 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3415f9c0 0x7f3a340cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 --2- 172.21.15.125:0/955836559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3413e360 0x7f3a340cf290 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 -- 172.21.15.125:0/955836559 >> 172.21.15.125:0/955836559 conn(0x7f3a3406a800 msgr2=0x7f3a3406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 -- 172.21.15.125:0/955836559 shutdown_connections 2024-01-16T15:07:17.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.835+0000 7f3a3a600640 1 -- 172.21.15.125:0/955836559 wait complete. 2024-01-16T15:07:17.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.837+0000 7f3a3a600640 1 Processor -- start 2024-01-16T15:07:17.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.837+0000 7f3a3a600640 1 -- start start 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3413df80 0x7f3a340cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3413e360 0x7f3a341e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3413c400 con 0x7f3a3415f9c0 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3413b7f0 con 0x7f3a3413e360 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a3a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3413bad0 con 0x7f3a3413df80 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45982/0 (socket says 172.21.15.125:45982) 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 -- 172.21.15.125:0/3854533553 learned_addr learned my addr 172.21.15.125:0/3854533553 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:17.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 -- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3413df80 msgr2=0x7f3a340cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3413df80 0x7f3a340cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a2a800640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3413e360 0x7f3a341e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 -- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3413e360 msgr2=0x7f3a341e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3413e360 0x7f3a341e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.838+0000 7f3a33e00640 1 -- 172.21.15.125:0/3854533553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0094d0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a33e00640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b9e0 tx=0x7f3a2c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c00cb20 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c00cc80 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c014680 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3413bdb0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.839+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3413b3f0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.840+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a340675d0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.842+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3a2c01d3b0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.843+0000 7f3a2be00640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 0x7f3a0c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.843+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f3a2c0887c0 con 0x7f3a3415f9c0 2024-01-16T15:07:17.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.843+0000 7f3a33400640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 0x7f3a0c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.844+0000 7f3a33400640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 0x7f3a0c06e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009650 tx=0x7f3a24006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:17.848+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a2c054df0 con 0x7f3a3415f9c0 2024-01-16T15:07:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:18.075+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96916", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a3413b5e0 con 0x7f3a0c06be90 2024-01-16T15:07:19.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.198+0000 7f3a2be00640 1 -- 172.21.15.125:0/3854533553 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3a3413b5e0 con 0x7f3a0c06be90 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.201+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 msgr2=0x7f3a0c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.201+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 0x7f3a0c06e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009650 tx=0x7f3a24006ba0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.201+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 msgr2=0x7f3a341e6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.201+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b9e0 tx=0x7f3a2c00bea0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 shutdown_connections 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3a0c06be90 0x7f3a0c06e350 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a3415f9c0 0x7f3a341e6730 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3413e360 0x7f3a341e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 --2- 172.21.15.125:0/3854533553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3413df80 0x7f3a340cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 >> 172.21.15.125:0/3854533553 conn(0x7f3a3406a800 msgr2=0x7f3a341539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 shutdown_connections 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:19.202+0000 7f3a3a600640 1 -- 172.21.15.125:0/3854533553 wait complete. 2024-01-16T15:07:21.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96917 2024-01-16T15:07:21.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.441+0000 7fdabd800640 1 Processor -- start 2024-01-16T15:07:21.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.441+0000 7fdabd800640 1 -- start start 2024-01-16T15:07:21.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdabd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fdab80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdabd800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab80b2730 con 0x7fdab80cf9a0 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdabd800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab80d1ec0 con 0x7fdab80b41e0 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdabd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80b29d0 con 0x7fdab815d4b0 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdab7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdab80cf9a0 0x7fdab80cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59030/0 (socket says 172.21.15.125:59030) 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.442+0000 7fdab7400640 1 -- 172.21.15.125:0/1570556398 learned_addr learned my addr 172.21.15.125:0/1570556398 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab7e00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fdab80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429729601 0 0) 0x7fdab80b2730 con 0x7fdab80cf9a0 2024-01-16T15:07:21.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda98003520 con 0x7fdab80cf9a0 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800254738 0 0) 0x7fda98003520 con 0x7fdab80cf9a0 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 msgr2=0x7fdab80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fdab80d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 >> v1:172.21.15.137:6789/0 conn(0x7fdab80b41e0 legacy=0x7fdab80ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.443+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab80b2cb0 con 0x7fdab80cf9a0 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fdaa0001670 con 0x7fdab80cf9a0 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 >> v1:172.21.15.125:6789/0 conn(0x7fdab80cf9a0 legacy=0x7fdab80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda980039a0 0x7fda98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fda98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fda98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80b29d0 con 0x7fda980047c0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9800ba30 con 0x7fda980039a0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9800bbf0 con 0x7fdab815d4b0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab7400640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fda98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.444+0000 7fdab7e00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda980039a0 0x7fda98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdabce00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fda98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7400640 1 -- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 msgr2=0x7fda98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7400640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab815d4b0 0x7fda98005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7400640 1 -- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda980039a0 msgr2=0x7fda98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7400640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda980039a0 0x7fda98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7400640 1 -- 172.21.15.125:0/1570556398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98003520 con 0x7fda980047c0 2024-01-16T15:07:21.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.445+0000 7fdab7e00640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda980039a0 0x7fda98003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:21.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.446+0000 7fdab7400640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fda98004c40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000c240 tx=0x7fda980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615012b71abab9f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.446+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0005ad0 con 0x7fda980047c0 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.446+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa0005c30 con 0x7fda980047c0 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.446+0000 7fdab6a00640 1 -- 172.21.15.125:0/1570556398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa002c8b0 con 0x7fda980047c0 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 -- 172.21.15.125:0/1570556398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 msgr2=0x7fda98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 --2- 172.21.15.125:0/1570556398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fda98004c40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000c240 tx=0x7fda980057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabce00640 1 -- 172.21.15.125:0/1570556398 reap_dead start 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 -- 172.21.15.125:0/1570556398 shutdown_connections 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 -- 172.21.15.125:0/1570556398 >> 172.21.15.125:0/1570556398 conn(0x7fdab80d3000 msgr2=0x7fdab80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 -- 172.21.15.125:0/1570556398 shutdown_connections 2024-01-16T15:07:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.447+0000 7fdabd800640 1 -- 172.21.15.125:0/1570556398 wait complete. 2024-01-16T15:07:21.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.449+0000 7fdabd800640 1 Processor -- start 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.449+0000 7fdabd800640 1 -- start start 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdab80c7580 0x7fdab81f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab80c7a00 0x7fdab80c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80b4ae0 con 0x7fda980047c0 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80c6b30 con 0x7fdab80c7580 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdabd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab815abe0 con 0x7fdab80c7a00 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46002/0 (socket says 172.21.15.125:46002) 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 -- 172.21.15.125:0/3489768726 learned_addr learned my addr 172.21.15.125:0/3489768726 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 -- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab80c7a00 msgr2=0x7fdab80c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab80c7a00 0x7fdab80c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 -- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdab80c7580 msgr2=0x7fdab81f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdab80c7580 0x7fdab81f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.450+0000 7fdab7e00640 1 -- 172.21.15.125:0/3489768726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab815da10 con 0x7fda980047c0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdab7e00640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80116d0 tx=0x7fdaa8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa800f9c0 con 0x7fda980047c0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa800fb20 con 0x7fda980047c0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa800fed0 con 0x7fda980047c0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab81f3830 con 0x7fda980047c0 2024-01-16T15:07:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.451+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab80c6d80 con 0x7fda980047c0 2024-01-16T15:07:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.452+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab81f3d50 con 0x7fda980047c0 2024-01-16T15:07:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.453+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fdaa8015460 con 0x7fda980047c0 2024-01-16T15:07:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.454+0000 7fdaafe00640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 0x7fda9006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.455+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fdaa8089160 con 0x7fda980047c0 2024-01-16T15:07:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.455+0000 7fdab7400640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 0x7fda9006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.455+0000 7fdab7400640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 0x7fda9006e380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0034000 tx=0x7fdaa0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.459+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdaa8055730 con 0x7fda980047c0 2024-01-16T15:07:21.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:21.693+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96917", "target": ["mon-mgr", ""]}) v1 -- 0x7fdab815d890 con 0x7fda9006bec0 2024-01-16T15:07:25.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.399+0000 7fdaafe00640 1 -- 172.21.15.125:0/3489768726 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdaa0001430 con 0x7fda9006bec0 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 msgr2=0x7fda9006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 0x7fda9006e380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0034000 tx=0x7fdaa0007000 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 msgr2=0x7fdab815a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80116d0 tx=0x7fdaa8011b90 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 shutdown_connections 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdab80c7a00 0x7fdab80c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdab80c7580 0x7fdab81f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda9006bec0 0x7fda9006e380 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 --2- 172.21.15.125:0/3489768726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda980047c0 0x7fdab815a6a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 >> 172.21.15.125:0/3489768726 conn(0x7fdab80d3000 msgr2=0x7fdab80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 shutdown_connections 2024-01-16T15:07:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:25.402+0000 7fdabd800640 1 -- 172.21.15.125:0/3489768726 wait complete. 2024-01-16T15:07:27.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96918 2024-01-16T15:07:27.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.629+0000 7f3976e00640 1 Processor -- start 2024-01-16T15:07:27.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.629+0000 7f3976e00640 1 -- start start 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f39700adf20 0x7f3970070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f397015bf10 0x7f39700cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700cc980 0x7f3970143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397015f970 con 0x7f397015bf10 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397015f4e0 con 0x7f39700adf20 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.630+0000 7f3976e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700cce10 con 0x7f39700cc980 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f397015bf10 0x7f39700cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f397015bf10 0x7f39700cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46806/0 (socket says 172.21.15.125:46806) 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 -- 172.21.15.125:0/1292836484 learned_addr learned my addr 172.21.15.125:0/1292836484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 -- 172.21.15.125:0/1292836484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700cc980 msgr2=0x7f3970143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 --2- 172.21.15.125:0/1292836484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700cc980 0x7f3970143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 -- 172.21.15.125:0/1292836484 >> v2:172.21.15.137:3300/0 conn(0x7f39700adf20 msgr2=0x7f3970070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 --2- 172.21.15.125:0/1292836484 >> v2:172.21.15.137:3300/0 conn(0x7f39700adf20 0x7f3970070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.631+0000 7f3975000640 1 -- 172.21.15.125:0/1292836484 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970071130 con 0x7f397015bf10 2024-01-16T15:07:27.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.632+0000 7f3975000640 1 --2- 172.21.15.125:0/1292836484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700cc440 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f39700b4150 tx=0x7f395802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a4117634602ee0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f396fe00640 1 -- 172.21.15.125:0/1292836484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580317e0 con 0x7f397015bf10 2024-01-16T15:07:27.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f396fe00640 1 -- 172.21.15.125:0/1292836484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958031940 con 0x7f397015bf10 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f396fe00640 1 -- 172.21.15.125:0/1292836484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958031cd0 con 0x7f397015bf10 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 -- 172.21.15.125:0/1292836484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 msgr2=0x7f39700cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 --2- 172.21.15.125:0/1292836484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700cc440 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f39700b4150 tx=0x7f395802b880 comp rx=0 tx=0).stop 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 -- 172.21.15.125:0/1292836484 shutdown_connections 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 --2- 172.21.15.125:0/1292836484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700cc980 0x7f3970143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 --2- 172.21.15.125:0/1292836484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700cc440 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 --2- 172.21.15.125:0/1292836484 >> v2:172.21.15.137:3300/0 conn(0x7f39700adf20 0x7f3970070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 -- 172.21.15.125:0/1292836484 >> 172.21.15.125:0/1292836484 conn(0x7f397006a7d0 msgr2=0x7f397006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 -- 172.21.15.125:0/1292836484 shutdown_connections 2024-01-16T15:07:27.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.633+0000 7f3976e00640 1 -- 172.21.15.125:0/1292836484 wait complete. 2024-01-16T15:07:27.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.635+0000 7f3976e00640 1 Processor -- start 2024-01-16T15:07:27.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.636+0000 7f3976e00640 1 -- start start 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.636+0000 7f3976e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700adf20 0x7f39700c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.636+0000 7f3976e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 0x7f39700c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39700c6b70 con 0x7f397015bf10 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39700c4400 con 0x7f39700cc980 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700c46e0 con 0x7f39700adf20 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46816/0 (socket says 172.21.15.125:46816) 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 -- 172.21.15.125:0/558962752 learned_addr learned my addr 172.21.15.125:0/558962752 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3975000640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 0x7f39700c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 -- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700adf20 msgr2=0x7f39700c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3975a00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700adf20 0x7f39700c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700adf20 0x7f39700c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 -- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 msgr2=0x7f39700c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 0x7f39700c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3976400640 1 -- 172.21.15.125:0/558962752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580094d0 con 0x7f397015bf10 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.637+0000 7f3975000640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 0x7f39700c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f3976400640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f3968009fd0 tx=0x7f396800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968011830 con 0x7f397015bf10 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968011990 con 0x7f397015bf10 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968011d20 con 0x7f397015bf10 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39700c49c0 con 0x7f397015bf10 2024-01-16T15:07:27.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.638+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39700c4f80 con 0x7f397015bf10 2024-01-16T15:07:27.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.639+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39700cc730 con 0x7f397015bf10 2024-01-16T15:07:27.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.640+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f396801b460 con 0x7f397015bf10 2024-01-16T15:07:27.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.641+0000 7f396d600640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 0x7f394806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.641+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f3968088660 con 0x7f397015bf10 2024-01-16T15:07:27.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.641+0000 7f3975a00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 0x7f394806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.642+0000 7f3975a00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 0x7f394806e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3960007dc0 tx=0x7f39600072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.646+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3968058720 con 0x7f397015bf10 2024-01-16T15:07:27.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:27.869+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96918", "target": ["mon-mgr", ""]}) v1 -- 0x7f397015ee90 con 0x7f394806be90 2024-01-16T15:07:37.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.469+0000 7f396d600640 1 -- 172.21.15.125:0/558962752 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f39600079b0 con 0x7f394806be90 2024-01-16T15:07:37.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.471+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 msgr2=0x7f394806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.471+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 0x7f394806e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3960007dc0 tx=0x7f39600072f0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.471+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 msgr2=0x7f39700c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.471+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f3968009fd0 tx=0x7f396800b8e0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 shutdown_connections 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f394806be90 0x7f394806e350 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f397015bf10 0x7f39700c9230 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39700cc980 0x7f39700c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 --2- 172.21.15.125:0/558962752 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39700adf20 0x7f39700c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 >> 172.21.15.125:0/558962752 conn(0x7f397006a7d0 msgr2=0x7f39700715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 shutdown_connections 2024-01-16T15:07:37.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:37.472+0000 7f3976e00640 1 -- 172.21.15.125:0/558962752 wait complete. 2024-01-16T15:07:39.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96919 2024-01-16T15:07:39.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:39.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.726+0000 7f0804000640 1 Processor -- start 2024-01-16T15:07:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.726+0000 7f0804000640 1 -- start start 2024-01-16T15:07:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0804000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc0ce8f0 0x7f07fc0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0804000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07fc0b2a00 0x7f07fc0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0804000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07fc0d0de0 con 0x7f07fc160f80 2024-01-16T15:07:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0804000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07fc0d10c0 con 0x7f07fc0b2a00 2024-01-16T15:07:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0804000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0d13a0 con 0x7f07fc0ce8f0 2024-01-16T15:07:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0801600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f07fc160f80 0x7f07fc161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50884/0 (socket says 172.21.15.125:50884) 2024-01-16T15:07:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0801600640 1 -- 172.21.15.125:0/4070307845 learned_addr learned my addr 172.21.15.125:0/4070307845 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:07:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.728+0000 7f0800c00640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc0ce8f0 0x7f07fc0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082779554 0 0) 0x7f07fc0d0de0 con 0x7f07fc160f80 2024-01-16T15:07:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc003520 con 0x7f07fc160f80 2024-01-16T15:07:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579847641 0 0) 0x7f07dc003520 con 0x7f07fc160f80 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc0ce8f0 msgr2=0x7f07fc0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc0ce8f0 0x7f07fc0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 >> v2:172.21.15.125:3300/0 conn(0x7f07fc0b2a00 msgr2=0x7f07fc0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 --2- 172.21.15.125:0/4070307845 >> v2:172.21.15.125:3300/0 conn(0x7f07fc0b2a00 0x7f07fc0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0d1680 con 0x7f07fc160f80 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f07ec0019f0 con 0x7f07fc160f80 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 >> v1:172.21.15.125:6789/0 conn(0x7f07fc160f80 legacy=0x7f07fc161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc003970 0x7f07dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07dc0047c0 0x7f07dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07dc005180 0x7f07dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0d10c0 con 0x7f07dc0047c0 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0d13a0 con 0x7f07dc003970 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc00bff0 con 0x7f07dc005180 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.729+0000 7f0801600640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc003970 0x7f07dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07dc005180 0x7f07dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 -- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc003970 msgr2=0x7f07dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc003970 0x7f07dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 -- 172.21.15.125:0/4070307845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07dc0047c0 msgr2=0x7f07dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07dc0047c0 0x7f07dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.730+0000 7f0802000640 1 -- 172.21.15.125:0/4070307845 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc003520 con 0x7f07dc005180 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.731+0000 7f0802000640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07dc005180 0x7f07dc00b980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009860 tx=0x7f07dc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e28336919eddf50a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.731+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4032600 con 0x7f07dc005180 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f402e050 con 0x7f07dc005180 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f07fbe00640 1 -- 172.21.15.125:0/4070307845 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f40329c0 con 0x7f07dc005180 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f0804000640 1 -- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07dc005180 msgr2=0x7f07dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f0804000640 1 --2- 172.21.15.125:0/4070307845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07dc005180 0x7f07dc00b980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009860 tx=0x7f07dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f0802000640 1 -- 172.21.15.125:0/4070307845 reap_dead start 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f0804000640 1 -- 172.21.15.125:0/4070307845 shutdown_connections 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.732+0000 7f0804000640 1 -- 172.21.15.125:0/4070307845 >> 172.21.15.125:0/4070307845 conn(0x7f07fc09c550 msgr2=0x7f07fc0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.733+0000 7f0804000640 1 -- 172.21.15.125:0/4070307845 shutdown_connections 2024-01-16T15:07:39.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.733+0000 7f0804000640 1 -- 172.21.15.125:0/4070307845 wait complete. 2024-01-16T15:07:39.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.734+0000 7f0804000640 1 Processor -- start 2024-01-16T15:07:39.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.734+0000 7f0804000640 1 -- start start 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0b6d70 0x7f07fc0b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc1518b0 0x7f07fc06c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0cb680 con 0x7f07fc0b6d70 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0b8530 con 0x7f07dc005180 2024-01-16T15:07:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.735+0000 7f0804000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0b7330 con 0x7f07fc1518b0 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:54606/0 (socket says 172.21.15.125:54606) 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 -- 172.21.15.125:0/1308577294 learned_addr learned my addr 172.21.15.125:0/1308577294 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0802000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc1518b0 0x7f07fc06c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 -- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc1518b0 msgr2=0x7f07fc06c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc1518b0 0x7f07fc06c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 -- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0b6d70 msgr2=0x7f07fc0b7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.736+0000 7f0801600640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0b6d70 0x7f07fc0b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f0801600640 1 -- 172.21.15.125:0/1308577294 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0ca380 con 0x7f07dc005180 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f0801600640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007a20 tx=0x7f07ec00a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0133f0 con 0x7f07dc005180 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07fc0ca660 con 0x7f07dc005180 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec013590 con 0x7f07dc005180 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.737+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec013940 con 0x7f07dc005180 2024-01-16T15:07:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.738+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07fc0cac20 con 0x7f07dc005180 2024-01-16T15:07:39.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.739+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07fc0675d0 con 0x7f07dc005180 2024-01-16T15:07:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.741+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f07ec013aa0 con 0x7f07dc005180 2024-01-16T15:07:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.741+0000 7f07f9600640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 0x7f07d406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.742+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f07ec089a80 con 0x7f07dc005180 2024-01-16T15:07:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.742+0000 7f0800c00640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 0x7f07d406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.742+0000 7f0800c00640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 0x7f07d406e380 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006a90 tx=0x7f07e4006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:39.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.747+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07ec056140 con 0x7f07dc005180 2024-01-16T15:07:39.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:39.975+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96919", "target": ["mon-mgr", ""]}) v1 -- 0x7f07fc0d1c40 con 0x7f07d406bec0 2024-01-16T15:07:49.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.501+0000 7f07f9600640 1 -- 172.21.15.125:0/1308577294 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f07e40096a0 con 0x7f07d406bec0 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 msgr2=0x7f07d406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 0x7f07d406e380 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006a90 tx=0x7f07e4006a20 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 msgr2=0x7f07fc0cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007a20 tx=0x7f07ec00a730 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 shutdown_connections 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07fc1518b0 0x7f07fc06c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f07d406bec0 0x7f07d406e380 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0b6d70 0x7f07fc0b7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 --2- 172.21.15.125:0/1308577294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07dc005180 0x7f07fc0cb140 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 >> 172.21.15.125:0/1308577294 conn(0x7f07fc09c550 msgr2=0x7f07fc0cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 shutdown_connections 2024-01-16T15:07:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:49.505+0000 7f0804000640 1 -- 172.21.15.125:0/1308577294 wait complete. 2024-01-16T15:07:51.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969110 2024-01-16T15:07:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.752+0000 7f6385600640 1 Processor -- start 2024-01-16T15:07:51.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.752+0000 7f6385600640 1 -- start start 2024-01-16T15:07:51.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f6385600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f638013df80 0x7f638015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f6385600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f63800cf9a0 0x7f63800cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f6385600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638006c470 con 0x7f638013e360 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f6385600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638006c750 con 0x7f638013df80 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f6385600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638006ca30 con 0x7f63800cf9a0 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f637ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f638013df80 0x7f638015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f637ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f638013df80 0x7f638015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39536/0 (socket says 172.21.15.125:39536) 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f637ea00640 1 -- 172.21.15.125:0/2154500903 learned_addr learned my addr 172.21.15.125:0/2154500903 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.754+0000 7f637f400640 1 --2- 172.21.15.125:0/2154500903 >> v2:172.21.15.137:3300/0 conn(0x7f63800cf9a0 0x7f63800cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.755+0000 7f637ea00640 1 -- 172.21.15.125:0/2154500903 >> v2:172.21.15.137:3300/0 conn(0x7f63800cf9a0 msgr2=0x7f63800cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.755+0000 7f637ea00640 1 --2- 172.21.15.125:0/2154500903 >> v2:172.21.15.137:3300/0 conn(0x7f63800cf9a0 0x7f63800cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.755+0000 7f637ea00640 1 -- 172.21.15.125:0/2154500903 >> v1:172.21.15.137:6789/0 conn(0x7f638013e360 legacy=0x7f63800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.755+0000 7f637ea00640 1 -- 172.21.15.125:0/2154500903 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638006cd10 con 0x7f638013df80 2024-01-16T15:07:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.756+0000 7f637ea00640 1 --2- 172.21.15.125:0/2154500903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f638015bee0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f63800ccb10 tx=0x7f637002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2772498b62ac5096 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f637e000640 1 -- 172.21.15.125:0/2154500903 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370031870 con 0x7f638013df80 2024-01-16T15:07:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f637e000640 1 -- 172.21.15.125:0/2154500903 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63700319d0 con 0x7f638013df80 2024-01-16T15:07:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f637e000640 1 -- 172.21.15.125:0/2154500903 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370031d60 con 0x7f638013df80 2024-01-16T15:07:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f6385600640 1 -- 172.21.15.125:0/2154500903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 msgr2=0x7f638015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f6385600640 1 --2- 172.21.15.125:0/2154500903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f638015bee0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f63800ccb10 tx=0x7f637002b910 comp rx=0 tx=0).stop 2024-01-16T15:07:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.757+0000 7f6385600640 1 -- 172.21.15.125:0/2154500903 shutdown_connections 2024-01-16T15:07:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.758+0000 7f6385600640 1 --2- 172.21.15.125:0/2154500903 >> v2:172.21.15.137:3300/0 conn(0x7f63800cf9a0 0x7f63800cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.758+0000 7f6385600640 1 --2- 172.21.15.125:0/2154500903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f638015bee0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.758+0000 7f6385600640 1 -- 172.21.15.125:0/2154500903 >> 172.21.15.125:0/2154500903 conn(0x7f638006a800 msgr2=0x7f638006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.758+0000 7f6385600640 1 -- 172.21.15.125:0/2154500903 shutdown_connections 2024-01-16T15:07:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.758+0000 7f6385600640 1 -- 172.21.15.125:0/2154500903 wait complete. 2024-01-16T15:07:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.759+0000 7f6385600640 1 Processor -- start 2024-01-16T15:07:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.760+0000 7f6385600640 1 -- start start 2024-01-16T15:07:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.760+0000 7f6385600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 0x7f638013d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.760+0000 7f6385600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6385600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f638013e360 0x7f63800ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6385600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63801584a0 con 0x7f638013df80 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6385600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6380158cb0 con 0x7f63800cf9a0 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6385600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63801586f0 con 0x7f638013e360 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39550/0 (socket says 172.21.15.125:39550) 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 -- 172.21.15.125:0/912147573 learned_addr learned my addr 172.21.15.125:0/912147573 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f637ea00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 0x7f638013d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f637f400640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f638013e360 0x7f63800ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 -- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f638013e360 msgr2=0x7f63800ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f638013e360 0x7f63800ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 -- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 msgr2=0x7f638013d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 0x7f638013d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f6375e00640 1 -- 172.21.15.125:0/912147573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63700094d0 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.761+0000 7f637ea00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 0x7f638013d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6375e00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f636800c780 tx=0x7f636800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636800aa50 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636800ce90 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636800ae30 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63801589d0 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.762+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638013dd10 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.763+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6380071090 con 0x7f638013df80 2024-01-16T15:07:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.764+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6368012420 con 0x7f638013df80 2024-01-16T15:07:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.765+0000 7f6377400640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 0x7f635806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.765+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f6368088da0 con 0x7f638013df80 2024-01-16T15:07:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.765+0000 7f637ea00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 0x7f635806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.766+0000 7f637ea00640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 0x7f635806e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6370036000 tx=0x7f637003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.769+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6380071090 con 0x7f638013df80 2024-01-16T15:07:51.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:51.999+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969110", "target": ["mon-mgr", ""]}) v1 -- 0x7f6380158860 con 0x7f635806be90 2024-01-16T15:07:52.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.604+0000 7f6377400640 1 -- 172.21.15.125:0/912147573 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6380158860 con 0x7f635806be90 2024-01-16T15:07:52.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 msgr2=0x7f635806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 0x7f635806e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6370036000 tx=0x7f637003c040 comp rx=0 tx=0).stop 2024-01-16T15:07:52.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 msgr2=0x7f6380071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f636800c780 tx=0x7f636800cc40 comp rx=0 tx=0).stop 2024-01-16T15:07:52.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 shutdown_connections 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f638013e360 0x7f63800ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f635806be90 0x7f635806e350 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638013df80 0x7f6380071370 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 --2- 172.21.15.125:0/912147573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63800cf9a0 0x7f638013d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.607+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 >> 172.21.15.125:0/912147573 conn(0x7f638006a800 msgr2=0x7f638006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.608+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 shutdown_connections 2024-01-16T15:07:52.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:52.608+0000 7f6385600640 1 -- 172.21.15.125:0/912147573 wait complete. 2024-01-16T15:07:54.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969111 2024-01-16T15:07:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.846+0000 7f0edb000640 1 Processor -- start 2024-01-16T15:07:54.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.846+0000 7f0edb000640 1 -- start start 2024-01-16T15:07:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0edb000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0ed40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0edb000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ed40cf7d0 0x7f0ed4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0edb000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40cfbf0 con 0x7f0ed4070ea0 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0edb000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed414b6a0 con 0x7f0ed40cf7d0 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0edb000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4071690 con 0x7f0ed414b2c0 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0ed9000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0ed4070ea0 0x7f0ed40b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40130/0 (socket says 172.21.15.125:40130) 2024-01-16T15:07:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.848+0000 7f0ed9000640 1 -- 172.21.15.125:0/1160556134 learned_addr learned my addr 172.21.15.125:0/1160556134 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:07:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0ed40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed3400640 1 --2- 172.21.15.125:0/1160556134 >> v2:172.21.15.137:3300/0 conn(0x7f0ed40cf7d0 0x7f0ed4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265131810 0 0) 0x7f0ed40cfbf0 con 0x7f0ed4070ea0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb4003520 con 0x7f0ed4070ea0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188292029 0 0) 0x7f0eb4003520 con 0x7f0ed4070ea0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 msgr2=0x7f0ed40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0ed40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 >> v2:172.21.15.137:3300/0 conn(0x7f0ed40cf7d0 msgr2=0x7f0ed4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 --2- 172.21.15.125:0/1160556134 >> v2:172.21.15.137:3300/0 conn(0x7f0ed40cf7d0 0x7f0ed4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4071970 con 0x7f0ed4070ea0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0ecc0019f0 con 0x7f0ed4070ea0 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 >> v1:172.21.15.125:6789/0 conn(0x7f0ed4070ea0 legacy=0x7f0ed40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0eb4003970 0x7f0eb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0eb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0eb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed414b6a0 con 0x7f0eb40047c0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4071690 con 0x7f0eb4003970 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.849+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb400b980 con 0x7f0ed414b2c0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0eb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0eb4003970 0x7f0eb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed9000640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0eb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 -- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 msgr2=0x7f0eb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed414b2c0 0x7f0eb4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 -- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0eb4003970 msgr2=0x7f0eb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0eb4003970 0x7f0eb4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3400640 1 -- 172.21.15.125:0/1160556134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4003520 con 0x7f0eb40047c0 2024-01-16T15:07:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.850+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0eb4003970 0x7f0eb4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.852+0000 7f0ed3400640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0eb4004c40 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00ba30 tx=0x7f0eb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b47eaab6fe7866 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.852+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc032680 con 0x7f0eb40047c0 2024-01-16T15:07:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.852+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc0327e0 con 0x7f0eb40047c0 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.852+0000 7f0ed2a00640 1 -- 172.21.15.125:0/1160556134 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc032b70 con 0x7f0eb40047c0 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 -- 172.21.15.125:0/1160556134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 msgr2=0x7f0eb4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 --2- 172.21.15.125:0/1160556134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0eb4004c40 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00ba30 tx=0x7f0eb40057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0ed9000640 1 -- 172.21.15.125:0/1160556134 reap_dead start 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 -- 172.21.15.125:0/1160556134 shutdown_connections 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 -- 172.21.15.125:0/1160556134 >> 172.21.15.125:0/1160556134 conn(0x7f0ed406a800 msgr2=0x7f0ed406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 -- 172.21.15.125:0/1160556134 shutdown_connections 2024-01-16T15:07:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.853+0000 7f0edb000640 1 -- 172.21.15.125:0/1160556134 wait complete. 2024-01-16T15:07:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.855+0000 7f0edb000640 1 Processor -- start 2024-01-16T15:07:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.855+0000 7f0edb000640 1 -- start start 2024-01-16T15:07:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed4162340 0x7f0ed41627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed4164400 0x7f0ed4164890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4164dd0 con 0x7f0eb40047c0 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4159c60 con 0x7f0ed4164400 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0edb000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4162e70 con 0x7f0ed4162340 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39576/0 (socket says 172.21.15.125:39576) 2024-01-16T15:07:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 -- 172.21.15.125:0/2244724968 learned_addr learned my addr 172.21.15.125:0/2244724968 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 -- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed4162340 msgr2=0x7f0ed41627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed4162340 0x7f0ed41627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 -- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed4164400 msgr2=0x7f0ed4164890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed9000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed4164400 0x7f0ed4164890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed4164400 0x7f0ed4164890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.856+0000 7f0ed3e00640 1 -- 172.21.15.125:0/2244724968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4163150 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0ed3e00640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40116d0 tx=0x7f0ec4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec400f9c0 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec400fb20 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed4163430 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec400fed0 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.857+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed41450e0 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.858+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed40675d0 con 0x7f0eb40047c0 2024-01-16T15:07:54.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.859+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0ec4015460 con 0x7f0eb40047c0 2024-01-16T15:07:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.860+0000 7f0ecbe00640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 0x7f0eac06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.860+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f0ec4089080 con 0x7f0eb40047c0 2024-01-16T15:07:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.860+0000 7f0ed3400640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 0x7f0eac06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.861+0000 7f0ed3400640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 0x7f0eac06e380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00ba00 tx=0x7f0ebc032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:54.864+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ec4055790 con 0x7f0eb40047c0 2024-01-16T15:07:55.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:55.090+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969111", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ed40cbc40 con 0x7f0eac06bec0 2024-01-16T15:07:58.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.971+0000 7f0ecbe00640 1 -- 172.21.15.125:0/2244724968 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0ebc0329d0 con 0x7f0eac06bec0 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 msgr2=0x7f0eac06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 0x7f0eac06e380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00ba00 tx=0x7f0ebc032000 comp rx=0 tx=0).stop 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 msgr2=0x7f0ed4144ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40116d0 tx=0x7f0ec4011b90 comp rx=0 tx=0).stop 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 shutdown_connections 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed4164400 0x7f0ed4164890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed4162340 0x7f0ed41627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0eac06bec0 0x7f0eac06e380 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 --2- 172.21.15.125:0/2244724968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb40047c0 0x7f0ed4144ba0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.974+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 >> 172.21.15.125:0/2244724968 conn(0x7f0ed406a800 msgr2=0x7f0ed40ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:58.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.975+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 shutdown_connections 2024-01-16T15:07:58.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:07:58.975+0000 7f0edb000640 1 -- 172.21.15.125:0/2244724968 wait complete. 2024-01-16T15:08:00.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969112 2024-01-16T15:08:01.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.230+0000 7f8d83200640 1 Processor -- start 2024-01-16T15:08:01.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.230+0000 7f8d83200640 1 -- start start 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d83200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8d8406d6d0 0x7f8d840b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d83200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d8415a690 0x7f8d840ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d83200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d840b1e30 con 0x7f8d841505b0 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d83200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8415ab20 con 0x7f8d8415a690 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d83200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840b20a0 con 0x7f8d8406d6d0 2024-01-16T15:08:01.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d81e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8d841505b0 0x7f8d8406cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45132/0 (socket says 172.21.15.125:45132) 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d81e00640 1 -- 172.21.15.125:0/2480643583 learned_addr learned my addr 172.21.15.125:0/2480643583 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d82800640 1 --2- 172.21.15.125:0/2480643583 >> v2:172.21.15.125:3300/0 conn(0x7f8d8415a690 0x7f8d840ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d82800640 1 -- 172.21.15.125:0/2480643583 >> v2:172.21.15.137:3300/0 conn(0x7f8d8406d6d0 msgr2=0x7f8d840b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d82800640 1 --2- 172.21.15.125:0/2480643583 >> v2:172.21.15.137:3300/0 conn(0x7f8d8406d6d0 0x7f8d840b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d82800640 1 -- 172.21.15.125:0/2480643583 >> v1:172.21.15.125:6789/0 conn(0x7f8d841505b0 legacy=0x7f8d8406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d82800640 1 -- 172.21.15.125:0/2480643583 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840b2380 con 0x7f8d8415a690 2024-01-16T15:08:01.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.232+0000 7f8d81400640 1 -- 172.21.15.125:0/2480643583 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639508965 0 0) 0x7f8d840b1e30 con 0x7f8d841505b0 2024-01-16T15:08:01.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.233+0000 7f8d82800640 1 --2- 172.21.15.125:0/2480643583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d840ccc00 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8d840b3140 tx=0x7f8d7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fa4c17274c1484 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d81400640 1 -- 172.21.15.125:0/2480643583 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c02cc80 con 0x7f8d8415a690 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d81400640 1 -- 172.21.15.125:0/2480643583 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c02cde0 con 0x7f8d8415a690 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d81400640 1 -- 172.21.15.125:0/2480643583 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c034660 con 0x7f8d8415a690 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 -- 172.21.15.125:0/2480643583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 msgr2=0x7f8d840ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 --2- 172.21.15.125:0/2480643583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d840ccc00 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8d840b3140 tx=0x7f8d7c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 -- 172.21.15.125:0/2480643583 shutdown_connections 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 --2- 172.21.15.125:0/2480643583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d840ccc00 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 --2- 172.21.15.125:0/2480643583 >> v2:172.21.15.137:3300/0 conn(0x7f8d8406d6d0 0x7f8d840b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 -- 172.21.15.125:0/2480643583 >> 172.21.15.125:0/2480643583 conn(0x7f8d8406a7d0 msgr2=0x7f8d8406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:01.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 -- 172.21.15.125:0/2480643583 shutdown_connections 2024-01-16T15:08:01.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.234+0000 7f8d83200640 1 -- 172.21.15.125:0/2480643583 wait complete. 2024-01-16T15:08:01.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.236+0000 7f8d83200640 1 Processor -- start 2024-01-16T15:08:01.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.236+0000 7f8d83200640 1 -- start start 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8d8406d6d0 0x7f8d84071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8d841505b0 0x7f8d84138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8414bfd0 con 0x7f8d8415a690 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8414c220 con 0x7f8d841505b0 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d83200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8414c500 con 0x7f8d8406d6d0 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d82800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d82800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51470/0 (socket says 172.21.15.125:51470) 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.237+0000 7f8d82800640 1 -- 172.21.15.125:0/1562566103 learned_addr learned my addr 172.21.15.125:0/1562566103 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 -- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8d8406d6d0 msgr2=0x7f8d84071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8d8406d6d0 0x7f8d84071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 -- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8d841505b0 msgr2=0x7f8d84138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8d841505b0 0x7f8d84138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 -- 172.21.15.125:0/1562566103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c009a70 con 0x7f8d8415a690 2024-01-16T15:08:01.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d82800640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c037040 tx=0x7f8d7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c03e610 con 0x7f8d8415a690 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c03e770 con 0x7f8d8415a690 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c03eb20 con 0x7f8d8415a690 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8414c7e0 con 0x7f8d8415a690 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.238+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8414cda0 con 0x7f8d8415a690 2024-01-16T15:08:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.239+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d84067450 con 0x7f8d8415a690 2024-01-16T15:08:01.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.240+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8d7c03b050 con 0x7f8d8415a690 2024-01-16T15:08:01.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.241+0000 7f8d7aa00640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 0x7f8d5c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.241+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f8d7c0ab040 con 0x7f8d8415a690 2024-01-16T15:08:01.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.241+0000 7f8d81e00640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 0x7f8d5c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.242+0000 7f8d81e00640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 0x7f8d5c06e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d74007fd0 tx=0x7f8d74007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.246+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d7c077930 con 0x7f8d8415a690 2024-01-16T15:08:01.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:01.466+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969112", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d8406d560 con 0x7f8d5c06be90 2024-01-16T15:08:09.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:09.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:10.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.828+0000 7f8d7aa00640 1 -- 172.21.15.125:0/1562566103 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8d8406d560 con 0x7f8d5c06be90 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 msgr2=0x7f8d5c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 0x7f8d5c06e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d74007fd0 tx=0x7f8d74007920 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 msgr2=0x7f8d841588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c037040 tx=0x7f8d7c004a40 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 shutdown_connections 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d5c06be90 0x7f8d5c06e350 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d8415a690 0x7f8d841588e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8d841505b0 0x7f8d84138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 --2- 172.21.15.125:0/1562566103 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8d8406d6d0 0x7f8d84071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 >> 172.21.15.125:0/1562566103 conn(0x7f8d8406a7d0 msgr2=0x7f8d8406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:10.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 shutdown_connections 2024-01-16T15:08:10.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:10.831+0000 7f8d83200640 1 -- 172.21.15.125:0/1562566103 wait complete. 2024-01-16T15:08:12.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969113 2024-01-16T15:08:13.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.072+0000 7fc66bc00640 1 Processor -- start 2024-01-16T15:08:13.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.072+0000 7fc66bc00640 1 -- start start 2024-01-16T15:08:13.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc664143ae0 0x7fc6640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 0x7fc66415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc66415e1e0 0x7fc66415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640cfc30 con 0x7fc664143ae0 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640b1ee0 con 0x7fc66415e1e0 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc66bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640b2780 con 0x7fc6640cf7d0 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc669200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc664143ae0 0x7fc6640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc669200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc664143ae0 0x7fc6640cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57780/0 (socket says 172.21.15.125:57780) 2024-01-16T15:08:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.074+0000 7fc669200640 1 -- 172.21.15.125:0/2421757826 learned_addr learned my addr 172.21.15.125:0/2421757826 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc663e00640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 0x7fc66415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc669200640 1 -- 172.21.15.125:0/2421757826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 msgr2=0x7fc66415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc669200640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 0x7fc66415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc669200640 1 -- 172.21.15.125:0/2421757826 >> v2:172.21.15.137:3300/0 conn(0x7fc66415e1e0 msgr2=0x7fc66415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc669200640 1 --2- 172.21.15.125:0/2421757826 >> v2:172.21.15.137:3300/0 conn(0x7fc66415e1e0 0x7fc66415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc669200640 1 -- 172.21.15.125:0/2421757826 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66415e990 con 0x7fc664143ae0 2024-01-16T15:08:13.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.075+0000 7fc663e00640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 0x7fc66415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:13.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.076+0000 7fc669200640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc664143ae0 0x7fc6640cf290 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc664072160 tx=0x7fc65402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495fbc413302ffa6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:13.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.076+0000 7fc663400640 1 -- 172.21.15.125:0/2421757826 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654031870 con 0x7fc664143ae0 2024-01-16T15:08:13.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc663400640 1 -- 172.21.15.125:0/2421757826 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6540319d0 con 0x7fc664143ae0 2024-01-16T15:08:13.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc663400640 1 -- 172.21.15.125:0/2421757826 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654031d60 con 0x7fc664143ae0 2024-01-16T15:08:13.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 -- 172.21.15.125:0/2421757826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc664143ae0 msgr2=0x7fc6640cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc664143ae0 0x7fc6640cf290 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc664072160 tx=0x7fc65402b910 comp rx=0 tx=0).stop 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 -- 172.21.15.125:0/2421757826 shutdown_connections 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 --2- 172.21.15.125:0/2421757826 >> v2:172.21.15.137:3300/0 conn(0x7fc66415e1e0 0x7fc66415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6640cf7d0 0x7fc66415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 --2- 172.21.15.125:0/2421757826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc664143ae0 0x7fc6640cf290 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 -- 172.21.15.125:0/2421757826 >> 172.21.15.125:0/2421757826 conn(0x7fc66406a800 msgr2=0x7fc66406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 -- 172.21.15.125:0/2421757826 shutdown_connections 2024-01-16T15:08:13.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.077+0000 7fc66bc00640 1 -- 172.21.15.125:0/2421757826 wait complete. 2024-01-16T15:08:13.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.080+0000 7fc66bc00640 1 Processor -- start 2024-01-16T15:08:13.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.080+0000 7fc66bc00640 1 -- start start 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc664143ae0 0x7fc6641f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc66415e1e0 0x7fc66413b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6641f16b0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66413d6b0 con 0x7fc664143ae0 2024-01-16T15:08:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc66bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6641f10f0 con 0x7fc66415e1e0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc669200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc669200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57796/0 (socket says 172.21.15.125:57796) 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.081+0000 7fc669200640 1 -- 172.21.15.125:0/3896821054 learned_addr learned my addr 172.21.15.125:0/3896821054 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 -- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc66415e1e0 msgr2=0x7fc66413b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc66415e1e0 0x7fc66413b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 -- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc664143ae0 msgr2=0x7fc6641f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc664143ae0 0x7fc6641f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 -- 172.21.15.125:0/3896821054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540094d0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc669200640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc654000c00 tx=0x7fc6540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654031840 con 0x7fc6640cf7d0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6540443f0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6641f13d0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6540447a0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.082+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6641f0c80 con 0x7fc6640cf7d0 2024-01-16T15:08:13.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.084+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6640675d0 con 0x7fc6640cf7d0 2024-01-16T15:08:13.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.086+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc65404d450 con 0x7fc6640cf7d0 2024-01-16T15:08:13.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.087+0000 7fc660c00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 0x7fc63c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:13.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.087+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fc65402d040 con 0x7fc6640cf7d0 2024-01-16T15:08:13.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.088+0000 7fc663e00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 0x7fc63c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:13.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.088+0000 7fc663e00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 0x7fc63c06e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc64c006fd0 tx=0x7fc64c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:13.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.093+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc654077630 con 0x7fc6640cf7d0 2024-01-16T15:08:13.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:13.322+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969113", "target": ["mon-mgr", ""]}) v1 -- 0x7fc66415e640 con 0x7fc63c06be90 2024-01-16T15:08:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.268+0000 7fc660c00640 1 -- 172.21.15.125:0/3896821054 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc64c00ae40 con 0x7fc63c06be90 2024-01-16T15:08:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.273+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 msgr2=0x7fc63c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.273+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 0x7fc63c06e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc64c006fd0 tx=0x7fc64c008040 comp rx=0 tx=0).stop 2024-01-16T15:08:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.273+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 msgr2=0x7fc664159870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.273+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc654000c00 tx=0x7fc6540046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 shutdown_connections 2024-01-16T15:08:14.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc66415e1e0 0x7fc66413b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc664143ae0 0x7fc6641f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc63c06be90 0x7fc63c06e350 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 --2- 172.21.15.125:0/3896821054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6640cf7d0 0x7fc664159870 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.274+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 >> 172.21.15.125:0/3896821054 conn(0x7fc66406a800 msgr2=0x7fc6640b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:14.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.278+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 shutdown_connections 2024-01-16T15:08:14.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:14.278+0000 7fc66bc00640 1 -- 172.21.15.125:0/3896821054 wait complete. 2024-01-16T15:08:16.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969114 2024-01-16T15:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.535+0000 7fad5c000640 1 Processor -- start 2024-01-16T15:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.535+0000 7fad5c000640 1 -- start start 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad54143ae0 0x7fad540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad5415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fad5415e1e0 0x7fad5415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540cfc30 con 0x7fad54143ae0 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540b1ee0 con 0x7fad5415e1e0 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.536+0000 7fad5c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b2780 con 0x7fad540cf7d0 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad54143ae0 0x7fad540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad54143ae0 0x7fad540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57802/0 (socket says 172.21.15.125:57802) 2024-01-16T15:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 -- 172.21.15.125:0/2428032977 learned_addr learned my addr 172.21.15.125:0/2428032977 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 -- 172.21.15.125:0/2428032977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 msgr2=0x7fad5415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 --2- 172.21.15.125:0/2428032977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad5415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 -- 172.21.15.125:0/2428032977 >> v2:172.21.15.137:3300/0 conn(0x7fad5415e1e0 msgr2=0x7fad5415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 --2- 172.21.15.125:0/2428032977 >> v2:172.21.15.137:3300/0 conn(0x7fad5415e1e0 0x7fad5415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.537+0000 7fad59600640 1 -- 172.21.15.125:0/2428032977 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5415e990 con 0x7fad54143ae0 2024-01-16T15:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.538+0000 7fad59600640 1 --2- 172.21.15.125:0/2428032977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad54143ae0 0x7fad540cf290 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fad54072160 tx=0x7fad4402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1be578727fba03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.538+0000 7fad53e00640 1 -- 172.21.15.125:0/2428032977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44031860 con 0x7fad54143ae0 2024-01-16T15:08:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.538+0000 7fad53e00640 1 -- 172.21.15.125:0/2428032977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad440319c0 con 0x7fad54143ae0 2024-01-16T15:08:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.538+0000 7fad53e00640 1 -- 172.21.15.125:0/2428032977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44031d50 con 0x7fad54143ae0 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 -- 172.21.15.125:0/2428032977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad54143ae0 msgr2=0x7fad540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 --2- 172.21.15.125:0/2428032977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad54143ae0 0x7fad540cf290 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fad54072160 tx=0x7fad4402b900 comp rx=0 tx=0).stop 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 -- 172.21.15.125:0/2428032977 shutdown_connections 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 --2- 172.21.15.125:0/2428032977 >> v2:172.21.15.137:3300/0 conn(0x7fad5415e1e0 0x7fad5415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 --2- 172.21.15.125:0/2428032977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad5415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 --2- 172.21.15.125:0/2428032977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad54143ae0 0x7fad540cf290 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 -- 172.21.15.125:0/2428032977 >> 172.21.15.125:0/2428032977 conn(0x7fad5406a800 msgr2=0x7fad5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 -- 172.21.15.125:0/2428032977 shutdown_connections 2024-01-16T15:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.539+0000 7fad5c000640 1 -- 172.21.15.125:0/2428032977 wait complete. 2024-01-16T15:08:16.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.541+0000 7fad5c000640 1 Processor -- start 2024-01-16T15:08:16.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.541+0000 7fad5c000640 1 -- start start 2024-01-16T15:08:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad54141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad54143ae0 0x7fad540a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541e81d0 con 0x7fad5415e1e0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541e7a10 con 0x7fad54143ae0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad54142070 con 0x7fad540cf7d0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57818/0 (socket says 172.21.15.125:57818) 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.542+0000 7fad5a000640 1 -- 172.21.15.125:0/2856577344 learned_addr learned my addr 172.21.15.125:0/2856577344 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad58c00640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad54143ae0 0x7fad540a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad59600640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad54141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 -- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 msgr2=0x7fad54141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad54141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 -- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad54143ae0 msgr2=0x7fad540a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad54143ae0 0x7fad540a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 -- 172.21.15.125:0/2856577344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440094d0 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5a000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00b970 tx=0x7fad4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00cb40 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00ccc0 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad540a9610 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.543+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c014680 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.544+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540a85f0 con 0x7fad5415e1e0 2024-01-16T15:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.545+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad540b2bf0 con 0x7fad5415e1e0 2024-01-16T15:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.546+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fad4c01d3b0 con 0x7fad5415e1e0 2024-01-16T15:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.547+0000 7fad51600640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 0x7fad2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.547+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fad4c088760 con 0x7fad5415e1e0 2024-01-16T15:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.547+0000 7fad59600640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 0x7fad2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.547+0000 7fad59600640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 0x7fad2c06e350 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fad44036000 tx=0x7fad4403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.551+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad540b2bf0 con 0x7fad5415e1e0 2024-01-16T15:08:16.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:16.781+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969114", "target": ["mon-mgr", ""]}) v1 -- 0x7fad540b1a80 con 0x7fad2c06be90 2024-01-16T15:08:26.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.360+0000 7fad51600640 1 -- 172.21.15.125:0/2856577344 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad540b1a80 con 0x7fad2c06be90 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 msgr2=0x7fad2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 0x7fad2c06e350 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fad44036000 tx=0x7fad4403c040 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 msgr2=0x7fad540aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00b970 tx=0x7fad4c00be30 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 shutdown_connections 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2c06be90 0x7fad2c06e350 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad5415e1e0 0x7fad540aefc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad54143ae0 0x7fad540a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 --2- 172.21.15.125:0/2856577344 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad540cf7d0 0x7fad54141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 >> 172.21.15.125:0/2856577344 conn(0x7fad5406a800 msgr2=0x7fad540702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:26.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 shutdown_connections 2024-01-16T15:08:26.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:26.363+0000 7fad5c000640 1 -- 172.21.15.125:0/2856577344 wait complete. 2024-01-16T15:08:28.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969115 2024-01-16T15:08:28.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.603+0000 7f47a8a00640 1 Processor -- start 2024-01-16T15:08:28.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.603+0000 7f47a8a00640 1 -- start start 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.604+0000 7f47a8a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f47a40b2e00 0x7f47a4070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.604+0000 7f47a8a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47a406cb10 0x7f47a406cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.604+0000 7f47a8a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a40706a0 con 0x7f47a406c730 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.604+0000 7f47a8a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a40b3290 con 0x7f47a406cb10 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.604+0000 7f47a8a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a40708d0 con 0x7f47a40b2e00 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47a406cb10 0x7f47a406cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47a406cb10 0x7f47a406cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45104/0 (socket says 172.21.15.125:45104) 2024-01-16T15:08:28.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 -- 172.21.15.125:0/3065157770 learned_addr learned my addr 172.21.15.125:0/3065157770 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f479a800640 1 --2- 172.21.15.125:0/3065157770 >> v2:172.21.15.137:3300/0 conn(0x7f47a40b2e00 0x7f47a4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 -- 172.21.15.125:0/3065157770 >> v2:172.21.15.137:3300/0 conn(0x7f47a40b2e00 msgr2=0x7f47a4070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 --2- 172.21.15.125:0/3065157770 >> v2:172.21.15.137:3300/0 conn(0x7f47a40b2e00 0x7f47a4070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 -- 172.21.15.125:0/3065157770 >> v1:172.21.15.137:6789/0 conn(0x7f47a406c730 legacy=0x7f47a4071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.605+0000 7f47a3e00640 1 -- 172.21.15.125:0/3065157770 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4070bb0 con 0x7f47a406cb10 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.606+0000 7f47a3e00640 1 --2- 172.21.15.125:0/3065157770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406cb10 0x7f47a406cf40 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f47a40cc040 tx=0x7f479c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c984bbe3300a6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:28.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.606+0000 7f47a2a00640 1 -- 172.21.15.125:0/3065157770 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c031870 con 0x7f47a406cb10 2024-01-16T15:08:28.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a2a00640 1 -- 172.21.15.125:0/3065157770 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c0319d0 con 0x7f47a406cb10 2024-01-16T15:08:28.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a2a00640 1 -- 172.21.15.125:0/3065157770 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c031d60 con 0x7f47a406cb10 2024-01-16T15:08:28.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 -- 172.21.15.125:0/3065157770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406cb10 msgr2=0x7f47a406cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 --2- 172.21.15.125:0/3065157770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406cb10 0x7f47a406cf40 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f47a40cc040 tx=0x7f479c02b910 comp rx=0 tx=0).stop 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 -- 172.21.15.125:0/3065157770 shutdown_connections 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 --2- 172.21.15.125:0/3065157770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406cb10 0x7f47a406cf40 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 --2- 172.21.15.125:0/3065157770 >> v2:172.21.15.137:3300/0 conn(0x7f47a40b2e00 0x7f47a4070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 -- 172.21.15.125:0/3065157770 >> 172.21.15.125:0/3065157770 conn(0x7f47a406a7d0 msgr2=0x7f47a406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 -- 172.21.15.125:0/3065157770 shutdown_connections 2024-01-16T15:08:28.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.607+0000 7f47a8a00640 1 -- 172.21.15.125:0/3065157770 wait complete. 2024-01-16T15:08:28.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.609+0000 7f47a8a00640 1 Processor -- start 2024-01-16T15:08:28.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.609+0000 7f47a8a00640 1 -- start start 2024-01-16T15:08:28.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47a406cb10 0x7f47a41f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 0x7f47a415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4145600 con 0x7f47a406c730 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a41458e0 con 0x7f47a40b2e00 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4145bc0 con 0x7f47a406cb10 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.610+0000 7f47a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45120/0 (socket says 172.21.15.125:45120) 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 -- 172.21.15.125:0/1768251630 learned_addr learned my addr 172.21.15.125:0/1768251630 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:28.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f479a800640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47a406cb10 0x7f47a41f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3e00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 0x7f47a415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 -- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47a406cb10 msgr2=0x7f47a41f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47a406cb10 0x7f47a41f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 -- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 msgr2=0x7f47a415bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 0x7f47a415bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 -- 172.21.15.125:0/1768251630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0094d0 con 0x7f47a406c730 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3e00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 0x7f47a415bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.611+0000 7f47a3400640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f479400c430 tx=0x7f479400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.612+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794011790 con 0x7f47a406c730 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.612+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47940118f0 con 0x7f47a406c730 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.612+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794011ce0 con 0x7f47a406c730 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.612+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4145ea0 con 0x7f47a406c730 2024-01-16T15:08:28.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.612+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a41f8110 con 0x7f47a406c730 2024-01-16T15:08:28.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.613+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4146090 con 0x7f47a406c730 2024-01-16T15:08:28.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.614+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f479401c460 con 0x7f47a406c730 2024-01-16T15:08:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.614+0000 7f479be00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 0x7f477c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.615+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f4794088760 con 0x7f47a406c730 2024-01-16T15:08:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.615+0000 7f479a800640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 0x7f477c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.615+0000 7f479a800640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 0x7f477c06e350 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f478c0070b0 tx=0x7f478c007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:28.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.619+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4794054dc0 con 0x7f47a406c730 2024-01-16T15:08:28.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:28.847+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969115", "target": ["mon-mgr", ""]}) v1 -- 0x7f47a4145a50 con 0x7f477c06be90 2024-01-16T15:08:29.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:29.313+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f4794055190 con 0x7f47a406c730 2024-01-16T15:08:32.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.403+0000 7f479be00640 1 -- 172.21.15.125:0/1768251630 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f47a4145a50 con 0x7f477c06be90 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.411+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 msgr2=0x7f477c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.411+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 0x7f477c06e350 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f478c0070b0 tx=0x7f478c007040 comp rx=0 tx=0).stop 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.411+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 msgr2=0x7f47a41f8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.411+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f479400c430 tx=0x7f479400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 shutdown_connections 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47a40b2e00 0x7f47a415bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47a406cb10 0x7f47a41f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f477c06be90 0x7f477c06e350 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 --2- 172.21.15.125:0/1768251630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47a406c730 0x7f47a41f8370 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 >> 172.21.15.125:0/1768251630 conn(0x7f47a406a7d0 msgr2=0x7f47a40cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 shutdown_connections 2024-01-16T15:08:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:32.412+0000 7f47a8a00640 1 -- 172.21.15.125:0/1768251630 wait complete. 2024-01-16T15:08:34.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969116 2024-01-16T15:08:34.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.672+0000 7fd252400640 1 Processor -- start 2024-01-16T15:08:34.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.673+0000 7fd252400640 1 -- start start 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.674+0000 7fd252400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd24c13e360 0x7fd24c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.674+0000 7fd252400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c15f9c0 0x7fd24c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.674+0000 7fd252400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c15c5f0 con 0x7fd24c13df80 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.674+0000 7fd252400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c06c470 con 0x7fd24c13e360 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.674+0000 7fd252400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c06c750 con 0x7fd24c15f9c0 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd24c13e360 0x7fd24c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd24c13e360 0x7fd24c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45136/0 (socket says 172.21.15.125:45136) 2024-01-16T15:08:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 -- 172.21.15.125:0/2910810264 learned_addr learned my addr 172.21.15.125:0/2910810264 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24be00640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c15f9c0 0x7fd24c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 -- 172.21.15.125:0/2910810264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c15f9c0 msgr2=0x7fd24c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c15f9c0 0x7fd24c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 -- 172.21.15.125:0/2910810264 >> v1:172.21.15.137:6789/0 conn(0x7fd24c13df80 legacy=0x7fd24c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.675+0000 7fd24aa00640 1 -- 172.21.15.125:0/2910810264 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c06ca30 con 0x7fd24c13e360 2024-01-16T15:08:34.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.676+0000 7fd24aa00640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13e360 0x7fd24c0cf290 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd24c06ded0 tx=0x7fd23402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c97a61a7e0ff72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.676+0000 7fd24a000640 1 -- 172.21.15.125:0/2910810264 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234031830 con 0x7fd24c13e360 2024-01-16T15:08:34.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd24a000640 1 -- 172.21.15.125:0/2910810264 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234031990 con 0x7fd24c13e360 2024-01-16T15:08:34.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd24a000640 1 -- 172.21.15.125:0/2910810264 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234031d20 con 0x7fd24c13e360 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 -- 172.21.15.125:0/2910810264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13e360 msgr2=0x7fd24c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13e360 0x7fd24c0cf290 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd24c06ded0 tx=0x7fd23402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 -- 172.21.15.125:0/2910810264 shutdown_connections 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c15f9c0 0x7fd24c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 --2- 172.21.15.125:0/2910810264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13e360 0x7fd24c0cf290 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 -- 172.21.15.125:0/2910810264 >> 172.21.15.125:0/2910810264 conn(0x7fd24c06a800 msgr2=0x7fd24c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 -- 172.21.15.125:0/2910810264 shutdown_connections 2024-01-16T15:08:34.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.677+0000 7fd252400640 1 -- 172.21.15.125:0/2910810264 wait complete. 2024-01-16T15:08:34.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.679+0000 7fd252400640 1 Processor -- start 2024-01-16T15:08:34.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.679+0000 7fd252400640 1 -- start start 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c13e360 0x7fd24c14b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 0x7fd24c1f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c1f1db0 con 0x7fd24c13df80 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c13de10 con 0x7fd24c15f9c0 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd252400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c1f17f0 con 0x7fd24c13e360 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd24b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd24b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45148/0 (socket says 172.21.15.125:45148) 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.680+0000 7fd24b400640 1 -- 172.21.15.125:0/2972042765 learned_addr learned my addr 172.21.15.125:0/2972042765 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24aa00640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c13e360 0x7fd24c14b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24be00640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 0x7fd24c1f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 -- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c13e360 msgr2=0x7fd24c14b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c13e360 0x7fd24c14b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 -- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 msgr2=0x7fd24c1f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 0x7fd24c1f0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 -- 172.21.15.125:0/2972042765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2340094d0 con 0x7fd24c13df80 2024-01-16T15:08:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24be00640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 0x7fd24c1f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd24b400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00b3f0 tx=0x7fd23c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c011830 con 0x7fd24c13df80 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c011990 con 0x7fd24c13df80 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.681+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c011d40 con 0x7fd24c13df80 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.682+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c1f1ad0 con 0x7fd24c13df80 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.682+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c14aea0 con 0x7fd24c13df80 2024-01-16T15:08:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.683+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c14aa20 con 0x7fd24c13df80 2024-01-16T15:08:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.684+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fd23c01a400 con 0x7fd24c13df80 2024-01-16T15:08:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.684+0000 7fd243400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 0x7fd22406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.685+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fd23c088420 con 0x7fd24c13df80 2024-01-16T15:08:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.685+0000 7fd24aa00640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 0x7fd22406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.686+0000 7fd24aa00640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 0x7fd22406e2c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd234036000 tx=0x7fd23403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.689+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd24c14aa20 con 0x7fd24c13df80 2024-01-16T15:08:34.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:34.924+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969116", "target": ["mon-mgr", ""]}) v1 -- 0x7fd24c06cff0 con 0x7fd22406be00 2024-01-16T15:08:35.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.663+0000 7fd243400640 1 -- 172.21.15.125:0/2972042765 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd234031b80 con 0x7fd22406be00 2024-01-16T15:08:35.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 msgr2=0x7fd22406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:35.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 0x7fd22406e2c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd234036000 tx=0x7fd23403c040 comp rx=0 tx=0).stop 2024-01-16T15:08:35.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 msgr2=0x7fd24c13d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:35.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00b3f0 tx=0x7fd23c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 shutdown_connections 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd24c15f9c0 0x7fd24c1f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd24c13e360 0x7fd24c14b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd22406be00 0x7fd22406e2c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 --2- 172.21.15.125:0/2972042765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd24c13df80 0x7fd24c13d8d0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 >> 172.21.15.125:0/2972042765 conn(0x7fd24c06a800 msgr2=0x7fd24c15de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 shutdown_connections 2024-01-16T15:08:35.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:35.667+0000 7fd252400640 1 -- 172.21.15.125:0/2972042765 wait complete. 2024-01-16T15:08:37.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969117 2024-01-16T15:08:37.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.910+0000 7faa91400640 1 Processor -- start 2024-01-16T15:08:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.910+0000 7faa91400640 1 -- start start 2024-01-16T15:08:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.912+0000 7faa91400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.912+0000 7faa91400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faa8c0cf7d0 0x7faa8c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.912+0000 7faa91400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0cfc30 con 0x7faa8c15e1e0 2024-01-16T15:08:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.912+0000 7faa91400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c0b1ee0 con 0x7faa8c0cf7d0 2024-01-16T15:08:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.912+0000 7faa91400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0b2780 con 0x7faa8c143ae0 2024-01-16T15:08:37.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa81c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faa8c0cf7d0 0x7faa8c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:57260/0 (socket says 172.21.15.125:57260) 2024-01-16T15:08:37.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 -- 172.21.15.125:0/3443461768 learned_addr learned my addr 172.21.15.125:0/3443461768 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 -- 172.21.15.125:0/3443461768 >> v2:172.21.15.137:3300/0 conn(0x7faa8c0cf7d0 msgr2=0x7faa8c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 --2- 172.21.15.125:0/3443461768 >> v2:172.21.15.137:3300/0 conn(0x7faa8c0cf7d0 0x7faa8c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 -- 172.21.15.125:0/3443461768 >> v1:172.21.15.137:6789/0 conn(0x7faa8c15e1e0 legacy=0x7faa8c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.913+0000 7faa8a800640 1 -- 172.21.15.125:0/3443461768 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c15e990 con 0x7faa8c143ae0 2024-01-16T15:08:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.915+0000 7faa8a800640 1 --2- 172.21.15.125:0/3443461768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009870 tx=0x7faa7c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d02daddf8d310722 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.915+0000 7faa89e00640 1 -- 172.21.15.125:0/3443461768 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031640 con 0x7faa8c143ae0 2024-01-16T15:08:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.915+0000 7faa89e00640 1 -- 172.21.15.125:0/3443461768 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c0317a0 con 0x7faa8c143ae0 2024-01-16T15:08:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.915+0000 7faa89e00640 1 -- 172.21.15.125:0/3443461768 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031b30 con 0x7faa8c143ae0 2024-01-16T15:08:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 -- 172.21.15.125:0/3443461768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 msgr2=0x7faa8c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 --2- 172.21.15.125:0/3443461768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009870 tx=0x7faa7c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 -- 172.21.15.125:0/3443461768 shutdown_connections 2024-01-16T15:08:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 --2- 172.21.15.125:0/3443461768 >> v2:172.21.15.137:3300/0 conn(0x7faa8c0cf7d0 0x7faa8c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 --2- 172.21.15.125:0/3443461768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c0cf290 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 -- 172.21.15.125:0/3443461768 >> 172.21.15.125:0/3443461768 conn(0x7faa8c06a800 msgr2=0x7faa8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 -- 172.21.15.125:0/3443461768 shutdown_connections 2024-01-16T15:08:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.916+0000 7faa91400640 1 -- 172.21.15.125:0/3443461768 wait complete. 2024-01-16T15:08:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.918+0000 7faa91400640 1 Processor -- start 2024-01-16T15:08:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.918+0000 7faa91400640 1 -- start start 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa8c0cf7d0 0x7faa8c13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1390e0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c139780 con 0x7faa8c0cf7d0 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.919+0000 7faa91400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c13f600 con 0x7faa8c143ae0 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45260/0 (socket says 172.21.15.125:45260) 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 -- 172.21.15.125:0/2501191628 learned_addr learned my addr 172.21.15.125:0/2501191628 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8a800640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa8c0cf7d0 0x7faa8c13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa81c00640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 -- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 msgr2=0x7faa8c13d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c13d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 -- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa8c0cf7d0 msgr2=0x7faa8c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa8c0cf7d0 0x7faa8c13ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.920+0000 7faa8b200640 1 -- 172.21.15.125:0/2501191628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c0094d0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa8b200640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7faa8400b970 tx=0x7faa8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400cae0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8400cc40 con 0x7faa8c15e1e0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c13f8e0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84014680 con 0x7faa8c15e1e0 2024-01-16T15:08:37.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.921+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c13fea0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.922+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0b2bf0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.923+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7faa8401d3b0 con 0x7faa8c15e1e0 2024-01-16T15:08:37.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.924+0000 7faa83400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 0x7faa6406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.925+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7faa84088510 con 0x7faa8c15e1e0 2024-01-16T15:08:37.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.925+0000 7faa8a800640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 0x7faa6406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.925+0000 7faa8a800640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 0x7faa6406e2c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faa7c036000 tx=0x7faa7c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:37.929+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa8c0b2bf0 con 0x7faa8c15e1e0 2024-01-16T15:08:38.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:38.158+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969117", "target": ["mon-mgr", ""]}) v1 -- 0x7faa8c15e640 con 0x7faa6406be00 2024-01-16T15:08:40.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:40.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:44.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.429+0000 7faa83400640 1 -- 172.21.15.125:0/2501191628 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faa7c031a50 con 0x7faa6406be00 2024-01-16T15:08:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 msgr2=0x7faa6406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 0x7faa6406e2c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faa7c036000 tx=0x7faa7c03c040 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 msgr2=0x7faa8c1f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7faa8400b970 tx=0x7faa8400be30 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 shutdown_connections 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faa6406be00 0x7faa6406e2c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa8c15e1e0 0x7faa8c1f0120 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa8c143ae0 0x7faa8c13d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 --2- 172.21.15.125:0/2501191628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa8c0cf7d0 0x7faa8c13ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 >> 172.21.15.125:0/2501191628 conn(0x7faa8c06a800 msgr2=0x7faa8c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 shutdown_connections 2024-01-16T15:08:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:44.432+0000 7faa91400640 1 -- 172.21.15.125:0/2501191628 wait complete. 2024-01-16T15:08:46.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969118 2024-01-16T15:08:46.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.678+0000 7f64e4000640 1 Processor -- start 2024-01-16T15:08:46.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.678+0000 7f64e4000640 1 -- start start 2024-01-16T15:08:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.679+0000 7f64e4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.679+0000 7f64e4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64dc0d0560 0x7f64dc161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.679+0000 7f64e4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc0cf4c0 con 0x7f64dc0d2260 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.679+0000 7f64e4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc0d2ef0 con 0x7f64dc0d0560 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.679+0000 7f64e4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0c9c50 con 0x7f64dc0c9870 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.680+0000 7f64e0c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64dc0d0560 0x7f64dc161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.680+0000 7f64e1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.680+0000 7f64e1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:35396/0 (socket says 172.21.15.125:35396) 2024-01-16T15:08:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.680+0000 7f64e1600640 1 -- 172.21.15.125:0/1944581568 learned_addr learned my addr 172.21.15.125:0/1944581568 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.681+0000 7f64e1600640 1 -- 172.21.15.125:0/1944581568 >> v2:172.21.15.137:3300/0 conn(0x7f64dc0d0560 msgr2=0x7f64dc161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.681+0000 7f64e1600640 1 --2- 172.21.15.125:0/1944581568 >> v2:172.21.15.137:3300/0 conn(0x7f64dc0d0560 0x7f64dc161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.681+0000 7f64e1600640 1 -- 172.21.15.125:0/1944581568 >> v1:172.21.15.137:6789/0 conn(0x7f64dc0d2260 legacy=0x7f64dc0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.681+0000 7f64e1600640 1 -- 172.21.15.125:0/1944581568 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc0cf7a0 con 0x7f64dc0c9870 2024-01-16T15:08:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.683+0000 7f64e1600640 1 --2- 172.21.15.125:0/1944581568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009870 tx=0x7f64c402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf2065e4d3a4c43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.683+0000 7f64dbe00640 1 -- 172.21.15.125:0/1944581568 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4030840 con 0x7f64dc0c9870 2024-01-16T15:08:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.683+0000 7f64dbe00640 1 -- 172.21.15.125:0/1944581568 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c40309a0 con 0x7f64dc0c9870 2024-01-16T15:08:46.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.683+0000 7f64dbe00640 1 -- 172.21.15.125:0/1944581568 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4030d30 con 0x7f64dc0c9870 2024-01-16T15:08:46.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 -- 172.21.15.125:0/1944581568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 msgr2=0x7f64dc0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 --2- 172.21.15.125:0/1944581568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009870 tx=0x7f64c402b980 comp rx=0 tx=0).stop 2024-01-16T15:08:46.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 -- 172.21.15.125:0/1944581568 shutdown_connections 2024-01-16T15:08:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 --2- 172.21.15.125:0/1944581568 >> v2:172.21.15.137:3300/0 conn(0x7f64dc0d0560 0x7f64dc161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 --2- 172.21.15.125:0/1944581568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0c9870 0x7f64dc0ceb50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 -- 172.21.15.125:0/1944581568 >> 172.21.15.125:0/1944581568 conn(0x7f64dc09c560 msgr2=0x7f64dc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 -- 172.21.15.125:0/1944581568 shutdown_connections 2024-01-16T15:08:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.684+0000 7f64e4000640 1 -- 172.21.15.125:0/1944581568 wait complete. 2024-01-16T15:08:46.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.686+0000 7f64e4000640 1 Processor -- start 2024-01-16T15:08:46.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.686+0000 7f64e4000640 1 -- start start 2024-01-16T15:08:46.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0d0560 0x7f64dc15f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64dc0d2260 0x7f64dc15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc159da0 con 0x7f64dc0c9870 2024-01-16T15:08:46.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc151fc0 con 0x7f64dc0d2260 2024-01-16T15:08:46.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc159120 con 0x7f64dc0d0560 2024-01-16T15:08:46.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e1600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e1600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52416/0 (socket says 172.21.15.125:52416) 2024-01-16T15:08:46.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.687+0000 7f64e1600640 1 -- 172.21.15.125:0/4056775042 learned_addr learned my addr 172.21.15.125:0/4056775042 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e0c00640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0d0560 0x7f64dc15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e2000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64dc0d2260 0x7f64dc15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 -- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0d0560 msgr2=0x7f64dc15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0d0560 0x7f64dc15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 -- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64dc0d2260 msgr2=0x7f64dc15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64dc0d2260 0x7f64dc15fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 -- 172.21.15.125:0/4056775042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40094d0 con 0x7f64dc0c9870 2024-01-16T15:08:46.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.688+0000 7f64e1600640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f64c4000c00 tx=0x7f64c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.689+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4039a90 con 0x7f64dc0c9870 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.689+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4039bf0 con 0x7f64dc0c9870 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.689+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc159400 con 0x7f64dc0c9870 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.689+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4043410 con 0x7f64dc0c9870 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.689+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc1599c0 con 0x7f64dc0c9870 2024-01-16T15:08:46.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.690+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc0675d0 con 0x7f64dc0c9870 2024-01-16T15:08:46.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.691+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f64c404c3b0 con 0x7f64dc0c9870 2024-01-16T15:08:46.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.692+0000 7f64d9600640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 0x7f64b406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.692+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f64c40aaa40 con 0x7f64dc0c9870 2024-01-16T15:08:46.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.692+0000 7f64e0c00640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 0x7f64b406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.693+0000 7f64e0c00640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 0x7f64b406e2c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f64dc1602c0 tx=0x7f64cc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.697+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64c4077240 con 0x7f64dc0c9870 2024-01-16T15:08:46.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:46.946+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969118", "target": ["mon-mgr", ""]}) v1 -- 0x7f64dc0d0040 con 0x7f64b406be00 2024-01-16T15:08:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.215+0000 7f64d9600640 1 -- 172.21.15.125:0/4056775042 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f64cc00f050 con 0x7f64b406be00 2024-01-16T15:08:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 msgr2=0x7f64b406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 0x7f64b406e2c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f64dc1602c0 tx=0x7f64cc00b040 comp rx=0 tx=0).stop 2024-01-16T15:08:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 msgr2=0x7f64dc151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f64c4000c00 tx=0x7f64c4004730 comp rx=0 tx=0).stop 2024-01-16T15:08:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 shutdown_connections 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64dc0d2260 0x7f64dc15fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64dc0d0560 0x7f64dc15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64b406be00 0x7f64b406e2c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 --2- 172.21.15.125:0/4056775042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64dc0c9870 0x7f64dc151a80 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.219+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 >> 172.21.15.125:0/4056775042 conn(0x7f64dc09c560 msgr2=0x7f64dc0b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.220+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 shutdown_connections 2024-01-16T15:08:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:49.220+0000 7f64e4000640 1 -- 172.21.15.125:0/4056775042 wait complete. 2024-01-16T15:08:51.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969119 2024-01-16T15:08:51.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.445+0000 7f7600c00640 1 Processor -- start 2024-01-16T15:08:51.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.445+0000 7f7600c00640 1 -- start start 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.446+0000 7f7600c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75fc13e360 0x7f75fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.446+0000 7f7600c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc15f9c0 0x7f75fc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.446+0000 7f7600c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc15c5f0 con 0x7f75fc13df80 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.446+0000 7f7600c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75fc06c470 con 0x7f75fc13e360 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.446+0000 7f7600c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc06c750 con 0x7f75fc15f9c0 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f9e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f75fc13df80 0x7f75fc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36464/0 (socket says 172.21.15.125:36464) 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f9e00640 1 -- 172.21.15.125:0/1366210919 learned_addr learned my addr 172.21.15.125:0/1366210919 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:08:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f3e00640 1 --2- 172.21.15.125:0/1366210919 >> v2:172.21.15.125:3300/0 conn(0x7f75fc13e360 0x7f75fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f9400640 1 -- 172.21.15.125:0/1366210919 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271725423 0 0) 0x7f75fc15c5f0 con 0x7f75fc13df80 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f9400640 1 -- 172.21.15.125:0/1366210919 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003520 con 0x7f75fc13df80 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75fa800640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc15f9c0 0x7f75fc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f3e00640 1 -- 172.21.15.125:0/1366210919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc15f9c0 msgr2=0x7f75fc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f3e00640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc15f9c0 0x7f75fc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f3e00640 1 -- 172.21.15.125:0/1366210919 >> v1:172.21.15.125:6789/0 conn(0x7f75fc13df80 legacy=0x7f75fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.447+0000 7f75f3e00640 1 -- 172.21.15.125:0/1366210919 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc06ca30 con 0x7f75fc13e360 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f75f3e00640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0cf290 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f75e40097a0 tx=0x7f75e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1161a6c5fe91dab7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f75f9400640 1 -- 172.21.15.125:0/1366210919 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031830 con 0x7f75fc13e360 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f75f9400640 1 -- 172.21.15.125:0/1366210919 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4031990 con 0x7f75fc13e360 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f75f9400640 1 -- 172.21.15.125:0/1366210919 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031d20 con 0x7f75fc13e360 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f7600c00640 1 -- 172.21.15.125:0/1366210919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 msgr2=0x7f75fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f7600c00640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0cf290 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f75e40097a0 tx=0x7f75e402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f7600c00640 1 -- 172.21.15.125:0/1366210919 shutdown_connections 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.449+0000 7f7600c00640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc15f9c0 0x7f75fc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.450+0000 7f7600c00640 1 --2- 172.21.15.125:0/1366210919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0cf290 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.450+0000 7f7600c00640 1 -- 172.21.15.125:0/1366210919 >> 172.21.15.125:0/1366210919 conn(0x7f75fc06a800 msgr2=0x7f75fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.450+0000 7f7600c00640 1 -- 172.21.15.125:0/1366210919 shutdown_connections 2024-01-16T15:08:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.450+0000 7f7600c00640 1 -- 172.21.15.125:0/1366210919 wait complete. 2024-01-16T15:08:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.451+0000 7f7600c00640 1 Processor -- start 2024-01-16T15:08:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.451+0000 7f7600c00640 1 -- start start 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13e360 0x7f75fc0afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc15f9c0 0x7f75fc0b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc12f370 con 0x7f75fc13df80 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0ae3b0 con 0x7f75fc15f9c0 2024-01-16T15:08:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.452+0000 7f7600c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1f3790 con 0x7f75fc13e360 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52434/0 (socket says 172.21.15.125:52434) 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 -- 172.21.15.125:0/651305191 learned_addr learned my addr 172.21.15.125:0/651305191 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f3e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13e360 0x7f75fc0afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 -- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13e360 msgr2=0x7f75fc0afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13e360 0x7f75fc0afcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 -- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc15f9c0 msgr2=0x7f75fc0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75fa800640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc15f9c0 0x7f75fc0b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc15f9c0 0x7f75fc0b0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 -- 172.21.15.125:0/651305191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4009400 con 0x7f75fc13df80 2024-01-16T15:08:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.453+0000 7f75f9e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c400 tx=0x7f75ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:51.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.454+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec005ce0 con 0x7f75fc13df80 2024-01-16T15:08:51.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.454+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec005e40 con 0x7f75fc13df80 2024-01-16T15:08:51.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.454+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc1f3c20 con 0x7f75fc13df80 2024-01-16T15:08:51.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.454+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0072d0 con 0x7f75fc13df80 2024-01-16T15:08:51.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.454+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc1f2e60 con 0x7f75fc13df80 2024-01-16T15:08:51.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.455+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc0675d0 con 0x7f75fc13df80 2024-01-16T15:08:51.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.456+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f75ec0074b0 con 0x7f75fc13df80 2024-01-16T15:08:51.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.456+0000 7f75f2000640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 0x7f75d406e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:51.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.457+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f75ec087ce0 con 0x7f75fc13df80 2024-01-16T15:08:51.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.457+0000 7f75f3e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 0x7f75d406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:51.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.458+0000 7f75f3e00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 0x7f75d406e310 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f75e4036000 tx=0x7f75e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.461+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75ec054440 con 0x7f75fc13df80 2024-01-16T15:08:51.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:51.688+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969119", "target": ["mon-mgr", ""]}) v1 -- 0x7f75fc15eac0 con 0x7f75d406be50 2024-01-16T15:08:53.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.806+0000 7f75f2000640 1 -- 172.21.15.125:0/651305191 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f75fc15eac0 con 0x7f75d406be50 2024-01-16T15:08:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 msgr2=0x7f75d406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 0x7f75d406e310 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f75e4036000 tx=0x7f75e4008030 comp rx=0 tx=0).stop 2024-01-16T15:08:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 msgr2=0x7f75fc0ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c400 tx=0x7f75ec00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 shutdown_connections 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc15f9c0 0x7f75fc0b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13e360 0x7f75fc0afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406be50 0x7f75d406e310 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 --2- 172.21.15.125:0/651305191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13df80 0x7f75fc0ade70 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 >> 172.21.15.125:0/651305191 conn(0x7f75fc06a800 msgr2=0x7f75fc15a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 shutdown_connections 2024-01-16T15:08:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:53.809+0000 7f7600c00640 1 -- 172.21.15.125:0/651305191 wait complete. 2024-01-16T15:08:55.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone969120 2024-01-16T15:08:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.052+0000 7f2e2a800640 1 Processor -- start 2024-01-16T15:08:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.052+0000 7f2e2a800640 1 -- start start 2024-01-16T15:08:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e2415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e2a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e240cf9a0 0x7f2e240cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e2a800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2406c470 con 0x7f2e2413e360 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e2a800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2406c750 con 0x7f2e240cf9a0 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e2a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2406ca30 con 0x7f2e2413df80 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e240cf9a0 0x7f2e240cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e240cf9a0 0x7f2e240cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52444/0 (socket says 172.21.15.125:52444) 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 -- 172.21.15.125:0/1262286414 learned_addr learned my addr 172.21.15.125:0/1262286414 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23400640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e2415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 -- 172.21.15.125:0/1262286414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 msgr2=0x7f2e2415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e2415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 -- 172.21.15.125:0/1262286414 >> v1:172.21.15.137:6789/0 conn(0x7f2e2413e360 legacy=0x7f2e240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.054+0000 7f2e23e00640 1 -- 172.21.15.125:0/1262286414 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2406cd10 con 0x7f2e240cf9a0 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.055+0000 7f2e23e00640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e240cf9a0 0x7f2e240cd840 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009870 tx=0x7f2e1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e324f991946ffe9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.055+0000 7f2e22a00640 1 -- 172.21.15.125:0/1262286414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c031640 con 0x7f2e240cf9a0 2024-01-16T15:08:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.055+0000 7f2e22a00640 1 -- 172.21.15.125:0/1262286414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c0317a0 con 0x7f2e240cf9a0 2024-01-16T15:08:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.055+0000 7f2e22a00640 1 -- 172.21.15.125:0/1262286414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c031b30 con 0x7f2e240cf9a0 2024-01-16T15:08:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 -- 172.21.15.125:0/1262286414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e240cf9a0 msgr2=0x7f2e240cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e240cf9a0 0x7f2e240cd840 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009870 tx=0x7f2e1c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 -- 172.21.15.125:0/1262286414 shutdown_connections 2024-01-16T15:08:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e240cf9a0 0x7f2e240cd840 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1262286414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e2415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 -- 172.21.15.125:0/1262286414 >> 172.21.15.125:0/1262286414 conn(0x7f2e2406a800 msgr2=0x7f2e2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 -- 172.21.15.125:0/1262286414 shutdown_connections 2024-01-16T15:08:56.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.056+0000 7f2e2a800640 1 -- 172.21.15.125:0/1262286414 wait complete. 2024-01-16T15:08:56.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.058+0000 7f2e2a800640 1 Processor -- start 2024-01-16T15:08:56.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.058+0000 7f2e2a800640 1 -- start start 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e240cf9a0 0x7f2e24153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e240aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24071c70 con 0x7f2e2413e360 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e240a9000 con 0x7f2e240cf9a0 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e2a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e240a8660 con 0x7f2e2413df80 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e23e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e23e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52454/0 (socket says 172.21.15.125:52454) 2024-01-16T15:08:56.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.059+0000 7f2e23e00640 1 -- 172.21.15.125:0/1468276759 learned_addr learned my addr 172.21.15.125:0/1468276759 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e1a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e240aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23400640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e240cf9a0 0x7f2e24153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 -- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 msgr2=0x7f2e240aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e240aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 -- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e240cf9a0 msgr2=0x7f2e24153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e240cf9a0 0x7f2e24153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 -- 172.21.15.125:0/1468276759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c0094d0 con 0x7f2e2413e360 2024-01-16T15:08:56.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e23e00640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c02b930 tx=0x7f2e1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.060+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c03ac60 con 0x7f2e2413e360 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.061+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c03adc0 con 0x7f2e2413e360 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.061+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c044630 con 0x7f2e2413e360 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.061+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e240a8940 con 0x7f2e2413e360 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.061+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24153740 con 0x7f2e2413e360 2024-01-16T15:08:56.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.062+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240a9140 con 0x7f2e2413e360 2024-01-16T15:08:56.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.063+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f2e1c02f050 con 0x7f2e2413e360 2024-01-16T15:08:56.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.064+0000 7f2e1be00640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 0x7f2dfc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.064+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2e1c0aa3c0 con 0x7f2e2413e360 2024-01-16T15:08:56.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.064+0000 7f2e23400640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 0x7f2dfc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.065+0000 7f2e23400640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 0x7f2dfc06e2c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009a60 tx=0x7f2e14003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.068+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e240a9140 con 0x7f2e2413e360 2024-01-16T15:08:56.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:56.298+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone969120", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e24155130 con 0x7f2dfc06be00 2024-01-16T15:08:57.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.028+0000 7f2e1be00640 1 -- 172.21.15.125:0/1468276759 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2e24155130 con 0x7f2dfc06be00 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.030+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 msgr2=0x7f2dfc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.030+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 0x7f2dfc06e2c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009a60 tx=0x7f2e14003990 comp rx=0 tx=0).stop 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.030+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 msgr2=0x7f2e240ab3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.030+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c02b930 tx=0x7f2e1c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 shutdown_connections 2024-01-16T15:08:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2dfc06be00 0x7f2dfc06e2c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e2413e360 0x7f2e240ab3f0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e2413df80 0x7f2e240aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 --2- 172.21.15.125:0/1468276759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e240cf9a0 0x7f2e24153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 >> 172.21.15.125:0/1468276759 conn(0x7f2e2406a800 msgr2=0x7f2e240ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 shutdown_connections 2024-01-16T15:08:57.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:57.031+0000 7f2e2a800640 1 -- 172.21.15.125:0/1468276759 wait complete. 2024-01-16T15:08:59.030 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:08:59.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96911 2024-01-16T15:08:59.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.275+0000 7f7828200640 1 Processor -- start 2024-01-16T15:08:59.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.275+0000 7f7828200640 1 -- start start 2024-01-16T15:08:59.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7828200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78200723e0 0x7f7820166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7828200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78200ae8a0 con 0x7f78200ae3f0 2024-01-16T15:08:59.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7828200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782015fd40 con 0x7f782015b3f0 2024-01-16T15:08:59.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7828200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7820071930 con 0x7f78200723e0 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7826200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78200723e0 0x7f7820166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7824e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f78200ae3f0 0x7f7820071cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36974/0 (socket says 172.21.15.125:36974) 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.277+0000 7f7824e00640 1 -- 172.21.15.125:0/3881182426 learned_addr learned my addr 172.21.15.125:0/3881182426 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.278+0000 7f7826200640 1 -- 172.21.15.125:0/3881182426 >> v1:172.21.15.137:6789/0 conn(0x7f782015b3f0 legacy=0x7f78200adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.278+0000 7f7826200640 1 -- 172.21.15.125:0/3881182426 >> v1:172.21.15.125:6789/0 conn(0x7f78200ae3f0 legacy=0x7f7820071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.278+0000 7f7826200640 1 -- 172.21.15.125:0/3881182426 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782015ef80 con 0x7f78200723e0 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.278+0000 7f781fe00640 1 -- 172.21.15.125:0/3881182426 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015165634 0 0) 0x7f78200ae8a0 con 0x7f78200ae3f0 2024-01-16T15:08:59.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.279+0000 7f7826200640 1 --2- 172.21.15.125:0/3881182426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200723e0 0x7f7820166170 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f78180097a0 tx=0x7f781802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12a3ee0146633ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.279+0000 7f781fe00640 1 -- 172.21.15.125:0/3881182426 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818031830 con 0x7f78200723e0 2024-01-16T15:08:59.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.279+0000 7f781fe00640 1 -- 172.21.15.125:0/3881182426 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7818031990 con 0x7f78200723e0 2024-01-16T15:08:59.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.279+0000 7f781fe00640 1 -- 172.21.15.125:0/3881182426 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818031d20 con 0x7f78200723e0 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 -- 172.21.15.125:0/3881182426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200723e0 msgr2=0x7f7820166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 --2- 172.21.15.125:0/3881182426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200723e0 0x7f7820166170 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f78180097a0 tx=0x7f781802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 -- 172.21.15.125:0/3881182426 shutdown_connections 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 --2- 172.21.15.125:0/3881182426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200723e0 0x7f7820166170 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 -- 172.21.15.125:0/3881182426 >> 172.21.15.125:0/3881182426 conn(0x7f78200b0c00 msgr2=0x7f78200b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 -- 172.21.15.125:0/3881182426 shutdown_connections 2024-01-16T15:08:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.280+0000 7f7828200640 1 -- 172.21.15.125:0/3881182426 wait complete. 2024-01-16T15:08:59.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.282+0000 7f7828200640 1 Processor -- start 2024-01-16T15:08:59.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.282+0000 7f7828200640 1 -- start start 2024-01-16T15:08:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 0x7f782014d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f782015b3f0 0x7f782015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782009aa50 con 0x7f78200ae3f0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782014ded0 con 0x7f78200723e0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7828200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7820099e40 con 0x7f782015b3f0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54352/0 (socket says 172.21.15.125:54352) 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 -- 172.21.15.125:0/2493982022 learned_addr learned my addr 172.21.15.125:0/2493982022 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7825800640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 0x7f782014d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7826200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f782015b3f0 0x7f782015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 -- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f782015b3f0 msgr2=0x7f782015c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f782015b3f0 0x7f782015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 -- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 msgr2=0x7f782014d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 0x7f782014d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.283+0000 7f7824e00640 1 -- 172.21.15.125:0/2493982022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818009400 con 0x7f78200ae3f0 2024-01-16T15:08:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f7825800640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 0x7f782014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f7824e00640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f780800c4a0 tx=0x7f780800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78080074e0 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7808007640 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782009a120 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808004a20 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.284+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782009a6e0 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.285+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782009ac00 con 0x7f78200ae3f0 2024-01-16T15:08:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.286+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f7808004b80 con 0x7f78200ae3f0 2024-01-16T15:08:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.287+0000 7f781d600640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 0x7f77f806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.287+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f78080878b0 con 0x7f78200ae3f0 2024-01-16T15:08:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.287+0000 7f7825800640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 0x7f77f806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.288+0000 7f7825800640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 0x7f77f806e2c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7810007e90 tx=0x7f78100073a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.292+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f782009ac00 con 0x7f78200ae3f0 2024-01-16T15:08:59.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.520+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96911", "target": ["mon-mgr", ""]}) v1 -- 0x7f782015f820 con 0x7f77f806be00 2024-01-16T15:08:59.505 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:08:59.528+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:59.505 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:08:59.528+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:59.505 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:08:59.528+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:59.505 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:08:59.528+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:59.505 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:08:59.528+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.548+0000 7f781d600640 1 -- 172.21.15.125:0/2493982022 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f781000cdb0 con 0x7f77f806be00 2024-01-16T15:08:59.525 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:08:59.525 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:08:59.525 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:08:59.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:08:59.526 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 msgr2=0x7f77f806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 0x7f77f806e2c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7810007e90 tx=0x7f78100073a0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 msgr2=0x7f782009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f780800c4a0 tx=0x7f780800c960 comp rx=0 tx=0).stop 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 shutdown_connections 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f782015b3f0 0x7f782015c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f77f806be00 0x7f77f806e2c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78200ae3f0 0x7f782009aee0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 --2- 172.21.15.125:0/2493982022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78200723e0 0x7f782014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 >> 172.21.15.125:0/2493982022 conn(0x7f78200b0c00 msgr2=0x7f78200afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 shutdown_connections 2024-01-16T15:08:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.551+0000 7f7828200640 1 -- 172.21.15.125:0/2493982022 wait complete. 2024-01-16T15:08:59.546 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:08:59.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96912 2024-01-16T15:08:59.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.760+0000 7fd27ee00640 1 Processor -- start 2024-01-16T15:08:59.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.761+0000 7fd27ee00640 1 -- start start 2024-01-16T15:08:59.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.762+0000 7fd27ee00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2780adf20 0x7fd278070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.762+0000 7fd27ee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 0x7fd278143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.762+0000 7fd27ee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27815f4e0 con 0x7fd27815bf10 2024-01-16T15:08:59.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.762+0000 7fd27ee00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27815f970 con 0x7fd2780adf20 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.762+0000 7fd27ee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd278071130 con 0x7fd2780ccb50 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2780adf20 0x7fd278070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2780adf20 0x7fd278070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54386/0 (socket says 172.21.15.125:54386) 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 -- 172.21.15.125:0/4029340382 learned_addr learned my addr 172.21.15.125:0/4029340382 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27e400640 1 --2- 172.21.15.125:0/4029340382 >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 0x7fd278143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 -- 172.21.15.125:0/4029340382 >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 msgr2=0x7fd278143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 --2- 172.21.15.125:0/4029340382 >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 0x7fd278143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 -- 172.21.15.125:0/4029340382 >> v1:172.21.15.125:6789/0 conn(0x7fd27815bf10 legacy=0x7fd2780cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27da00640 1 -- 172.21.15.125:0/4029340382 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27815c390 con 0x7fd2780adf20 2024-01-16T15:08:59.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.763+0000 7fd27e400640 1 --2- 172.21.15.125:0/4029340382 >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 0x7fd278143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:59.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd27da00640 1 --2- 172.21.15.125:0/4029340382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2780adf20 0x7fd278070a30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd27806ff20 tx=0x7fd26802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68198dedab05422d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd277e00640 1 -- 172.21.15.125:0/4029340382 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26802ca80 con 0x7fd2780adf20 2024-01-16T15:08:59.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd277e00640 1 -- 172.21.15.125:0/4029340382 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26802cbe0 con 0x7fd2780adf20 2024-01-16T15:08:59.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd277e00640 1 -- 172.21.15.125:0/4029340382 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680343f0 con 0x7fd2780adf20 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd27ee00640 1 -- 172.21.15.125:0/4029340382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2780adf20 msgr2=0x7fd278070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.765+0000 7fd27ee00640 1 --2- 172.21.15.125:0/4029340382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2780adf20 0x7fd278070a30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd27806ff20 tx=0x7fd26802bee0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 -- 172.21.15.125:0/4029340382 shutdown_connections 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 --2- 172.21.15.125:0/4029340382 >> v2:172.21.15.137:3300/0 conn(0x7fd2780ccb50 0x7fd278143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 --2- 172.21.15.125:0/4029340382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2780adf20 0x7fd278070a30 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 -- 172.21.15.125:0/4029340382 >> 172.21.15.125:0/4029340382 conn(0x7fd27806a7d0 msgr2=0x7fd27806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 -- 172.21.15.125:0/4029340382 shutdown_connections 2024-01-16T15:08:59.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.766+0000 7fd27ee00640 1 -- 172.21.15.125:0/4029340382 wait complete. 2024-01-16T15:08:59.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.767+0000 7fd27ee00640 1 Processor -- start 2024-01-16T15:08:59.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 -- start start 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2780adf20 0x7fd2780c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 0x7fd27815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780c4710 con 0x7fd27815bf10 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780c49f0 con 0x7fd2780ccb50 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.768+0000 7fd27ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780c4cd0 con 0x7fd2780adf20 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54400/0 (socket says 172.21.15.125:54400) 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 -- 172.21.15.125:0/2712077970 learned_addr learned my addr 172.21.15.125:0/2712077970 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:08:59.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27d000640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 0x7fd27815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27da00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2780adf20 0x7fd2780c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 -- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2780adf20 msgr2=0x7fd2780c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2780adf20 0x7fd2780c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 -- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 msgr2=0x7fd27815a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 0x7fd27815a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27e400640 1 -- 172.21.15.125:0/2712077970 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009b30 con 0x7fd27815bf10 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.769+0000 7fd27d000640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 0x7fd27815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd27e400640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c7f0 tx=0x7fd27000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270007820 con 0x7fd27815bf10 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270007980 con 0x7fd27815bf10 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270007d70 con 0x7fd27815bf10 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2780c4fb0 con 0x7fd27815bf10 2024-01-16T15:08:59.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.770+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2780c5570 con 0x7fd27815bf10 2024-01-16T15:08:59.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.771+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd278067450 con 0x7fd27815bf10 2024-01-16T15:08:59.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.772+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fd2700113f0 con 0x7fd27815bf10 2024-01-16T15:08:59.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.773+0000 7fd275600640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 0x7fd25006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.773+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fd270088a90 con 0x7fd27815bf10 2024-01-16T15:08:59.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.773+0000 7fd27da00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 0x7fd25006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.774+0000 7fd27da00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 0x7fd25006e2c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd268035000 tx=0x7fd268030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:08:59.778+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd270055160 con 0x7fd27815bf10 2024-01-16T15:08:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.007+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96912", "target": ["mon-mgr", ""]}) v1 -- 0x7fd27815f170 con 0x7fd25006be00 2024-01-16T15:09:00.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.025+0000 7fd275600640 1 -- 172.21.15.125:0/2712077970 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd268009510 con 0x7fd25006be00 2024-01-16T15:09:00.002 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:09:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:00.003 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 msgr2=0x7fd25006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 0x7fd25006e2c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd268035000 tx=0x7fd268030000 comp rx=0 tx=0).stop 2024-01-16T15:09:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 msgr2=0x7fd27815c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c7f0 tx=0x7fd27000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 shutdown_connections 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd25006be00 0x7fd25006e2c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd27815bf10 0x7fd27815c580 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2780ccb50 0x7fd27815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 --2- 172.21.15.125:0/2712077970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2780adf20 0x7fd2780c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 >> 172.21.15.125:0/2712077970 conn(0x7fd27806a7d0 msgr2=0x7fd27815e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 shutdown_connections 2024-01-16T15:09:00.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.028+0000 7fd27ee00640 1 -- 172.21.15.125:0/2712077970 wait complete. 2024-01-16T15:09:00.023 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:00.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96913 2024-01-16T15:09:00.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.251+0000 7f13d7400640 1 Processor -- start 2024-01-16T15:09:00.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.251+0000 7f13d7400640 1 -- start start 2024-01-16T15:09:00.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13d00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f13d00cf7d0 0x7f13d0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d7400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d00cfbf0 con 0x7f13d0070ea0 2024-01-16T15:09:00.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d7400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d014b6a0 con 0x7f13d00cf7d0 2024-01-16T15:09:00.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0071690 con 0x7f13d014b2c0 2024-01-16T15:09:00.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d5400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f13d0070ea0 0x7f13d00b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36992/0 (socket says 172.21.15.125:36992) 2024-01-16T15:09:00.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.253+0000 7f13d5400640 1 -- 172.21.15.125:0/3537917014 learned_addr learned my addr 172.21.15.125:0/3537917014 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:00.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cfe00640 1 --2- 172.21.15.125:0/3537917014 >> v2:172.21.15.137:3300/0 conn(0x7f13d00cf7d0 0x7f13d0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13d4a00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13d00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245473229 0 0) 0x7f13d00cfbf0 con 0x7f13d0070ea0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b0003520 con 0x7f13d0070ea0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191865606 0 0) 0x7f13b0003520 con 0x7f13d0070ea0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 msgr2=0x7f13d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13d00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 >> v2:172.21.15.137:3300/0 conn(0x7f13d00cf7d0 msgr2=0x7f13d0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 --2- 172.21.15.125:0/3537917014 >> v2:172.21.15.137:3300/0 conn(0x7f13d00cf7d0 0x7f13d0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0071970 con 0x7f13d0070ea0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f13c80019f0 con 0x7f13d0070ea0 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 >> v1:172.21.15.125:6789/0 conn(0x7f13d0070ea0 legacy=0x7f13d00b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d014b6a0 con 0x7f13b0003970 2024-01-16T15:09:00.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0071690 con 0x7f13b0004d00 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.254+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b000b980 con 0x7f13d014b2c0 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13cfe00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d5400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 -- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 msgr2=0x7f13b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d014b2c0 0x7f13b00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 -- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b0004d00 msgr2=0x7f13b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.255+0000 7f13d4a00640 1 -- 172.21.15.125:0/3537917014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0003520 con 0x7f13b0003970 2024-01-16T15:09:00.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.256+0000 7f13d4a00640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13b0003d90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009ac0 tx=0x7f13b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac30f67ed480c89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.256+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c002c6b0 con 0x7f13b0003970 2024-01-16T15:09:00.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c002c810 con 0x7f13b0003970 2024-01-16T15:09:00.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13cf400640 1 -- 172.21.15.125:0/3537917014 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c002cba0 con 0x7f13b0003970 2024-01-16T15:09:00.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 -- 172.21.15.125:0/3537917014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 msgr2=0x7f13b0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 --2- 172.21.15.125:0/3537917014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13b0003d90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009ac0 tx=0x7f13b00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d5400640 1 -- 172.21.15.125:0/3537917014 reap_dead start 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 -- 172.21.15.125:0/3537917014 shutdown_connections 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 -- 172.21.15.125:0/3537917014 >> 172.21.15.125:0/3537917014 conn(0x7f13d006a800 msgr2=0x7f13d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 -- 172.21.15.125:0/3537917014 shutdown_connections 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.257+0000 7f13d7400640 1 -- 172.21.15.125:0/3537917014 wait complete. 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.259+0000 7f13d7400640 1 Processor -- start 2024-01-16T15:09:00.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.259+0000 7f13d7400640 1 -- start start 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 0x7f13d015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d0161e10 0x7f13d00a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0163020 con 0x7f13b0003970 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00ab050 con 0x7f13d0161e10 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d015e810 con 0x7f13d0164b40 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54428/0 (socket says 172.21.15.125:54428) 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.260+0000 7f13d4a00640 1 -- 172.21.15.125:0/1517641003 learned_addr learned my addr 172.21.15.125:0/1517641003 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:00.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13cfe00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 0x7f13d015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 -- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 msgr2=0x7f13d015e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 0x7f13d015e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 -- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d0161e10 msgr2=0x7f13d00a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d0161e10 0x7f13d00a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 -- 172.21.15.125:0/1517641003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0162780 con 0x7f13b0003970 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13cfe00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 0x7f13d015e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d4a00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f13c002c250 tx=0x7f13c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003810 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c0003970 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d0162a60 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.261+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003ce0 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.262+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d0163370 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.263+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d01631d0 con 0x7f13b0003970 2024-01-16T15:09:00.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.264+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f13c0047450 con 0x7f13b0003970 2024-01-16T15:09:00.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.264+0000 7f13ccc00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 0x7f13a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.265+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f13c00aa7f0 con 0x7f13b0003970 2024-01-16T15:09:00.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.265+0000 7f13cfe00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 0x7f13a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.265+0000 7f13cfe00640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 0x7f13a806e2f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f13d00b2090 tx=0x7f13b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.269+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13d01631d0 con 0x7f13b0003970 2024-01-16T15:09:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.499+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96913", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d00cbc40 con 0x7f13a806be30 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.516+0000 7f13ccc00640 1 -- 172.21.15.125:0/1517641003 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f13b8009650 con 0x7f13a806be30 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:00.493 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.518+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 msgr2=0x7f13a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.518+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 0x7f13a806e2f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f13d00b2090 tx=0x7f13b8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.518+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 msgr2=0x7f13d00aab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.518+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f13c002c250 tx=0x7f13c0004730 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 shutdown_connections 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d0161e10 0x7f13d00a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d0164b40 0x7f13d015e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13a806be30 0x7f13a806e2f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 --2- 172.21.15.125:0/1517641003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b0003970 0x7f13d00aab10 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 >> 172.21.15.125:0/1517641003 conn(0x7f13d006a800 msgr2=0x7f13d00af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 shutdown_connections 2024-01-16T15:09:00.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.519+0000 7f13d7400640 1 -- 172.21.15.125:0/1517641003 wait complete. 2024-01-16T15:09:00.513 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:00.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96914 2024-01-16T15:09:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.748+0000 7fa6b2800640 1 Processor -- start 2024-01-16T15:09:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.748+0000 7fa6b2800640 1 -- start start 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6b2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6ac0cd240 0x7fa6ac0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6b2800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac06cea0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6b2800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac06cbc0 con 0x7fa6ac0cd240 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6b2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac06f350 con 0x7fa6ac09a170 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6ab400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6ac0cd240 0x7fa6ac0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6ab400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6ac0cd240 0x7fa6ac0b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54436/0 (socket says 172.21.15.125:54436) 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.750+0000 7fa6ab400640 1 -- 172.21.15.125:0/1219624484 learned_addr learned my addr 172.21.15.125:0/1219624484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.751+0000 7fa6abe00640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.751+0000 7fa6ab400640 1 -- 172.21.15.125:0/1219624484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 msgr2=0x7fa6ac0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.751+0000 7fa6ab400640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.751+0000 7fa6ab400640 1 -- 172.21.15.125:0/1219624484 >> v1:172.21.15.137:6789/0 conn(0x7fa6ac06c6b0 legacy=0x7fa6ac099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.751+0000 7fa6ab400640 1 -- 172.21.15.125:0/1219624484 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac06f630 con 0x7fa6ac0cd240 2024-01-16T15:09:00.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.752+0000 7fa6ab400640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac0b3820 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5585b177a276d15d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6aa000640 1 -- 172.21.15.125:0/1219624484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c031640 con 0x7fa6ac0cd240 2024-01-16T15:09:00.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6aa000640 1 -- 172.21.15.125:0/1219624484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c0317a0 con 0x7fa6ac0cd240 2024-01-16T15:09:00.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6aa000640 1 -- 172.21.15.125:0/1219624484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c031b30 con 0x7fa6ac0cd240 2024-01-16T15:09:00.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 -- 172.21.15.125:0/1219624484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac0cd240 msgr2=0x7fa6ac0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac0b3820 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 -- 172.21.15.125:0/1219624484 shutdown_connections 2024-01-16T15:09:00.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1219624484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac0b3820 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 -- 172.21.15.125:0/1219624484 >> 172.21.15.125:0/1219624484 conn(0x7fa6ac0b2d00 msgr2=0x7fa6ac0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 -- 172.21.15.125:0/1219624484 shutdown_connections 2024-01-16T15:09:00.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.753+0000 7fa6b2800640 1 -- 172.21.15.125:0/1219624484 wait complete. 2024-01-16T15:09:00.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.755+0000 7fa6b2800640 1 Processor -- start 2024-01-16T15:09:00.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.755+0000 7fa6b2800640 1 -- start start 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac15ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0cafa0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0b6b80 con 0x7fa6ac0cd240 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6b2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0b6e60 con 0x7fa6ac09a170 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54448/0 (socket says 172.21.15.125:54448) 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.756+0000 7fa6ab400640 1 -- 172.21.15.125:0/1982031160 learned_addr learned my addr 172.21.15.125:0/1982031160 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 -- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 msgr2=0x7fa6ac14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 -- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ac0cd240 msgr2=0x7fa6ac15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac15ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 -- 172.21.15.125:0/1982031160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0094d0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6ab400640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c03ac60 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c03adc0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c044630 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.757+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac15e8e0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.758+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac15bb50 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.759+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac15eaa0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.759+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fa69c02f050 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.761+0000 7fa6a3400640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 0x7fa68406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.761+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fa69c0aa4a0 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.761+0000 7fa6aaa00640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 0x7fa68406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.762+0000 7fa6aaa00640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 0x7fa68406e2c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa694006fd0 tx=0x7fa694006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.765+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa69c076d60 con 0x7fa6ac06c6b0 2024-01-16T15:09:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:00.993+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6ac0cd620 con 0x7fa68406be00 2024-01-16T15:09:00.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.012+0000 7fa6a3400640 1 -- 172.21.15.125:0/1982031160 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fa694009280 con 0x7fa68406be00 2024-01-16T15:09:00.989 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:00.989 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:00.990 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "complete" 2024-01-16T15:09:00.990 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:00.990 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:00.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 msgr2=0x7fa68406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 0x7fa68406e2c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa694006fd0 tx=0x7fa694006bc0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 msgr2=0x7fa6ac14bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 shutdown_connections 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6ac0cd240 0x7fa6ac15ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6ac09a170 0x7fa6ac14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa68406be00 0x7fa68406e2c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 --2- 172.21.15.125:0/1982031160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6ac06c6b0 0x7fa6ac14bfe0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 >> 172.21.15.125:0/1982031160 conn(0x7fa6ac0b2d00 msgr2=0x7fa6ac099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 shutdown_connections 2024-01-16T15:09:00.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.015+0000 7fa6b2800640 1 -- 172.21.15.125:0/1982031160 wait complete. 2024-01-16T15:09:01.011 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:01.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96915 2024-01-16T15:09:01.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.246+0000 7fe967e00640 1 Processor -- start 2024-01-16T15:09:01.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.246+0000 7fe967e00640 1 -- start start 2024-01-16T15:09:01.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe967e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe960071200 0x7fe9600b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe967e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600b36f0 con 0x7fe96015e8d0 2024-01-16T15:09:01.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe967e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600cda60 con 0x7fe9600cd600 2024-01-16T15:09:01.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe967e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600aec70 con 0x7fe960071200 2024-01-16T15:09:01.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe965400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe96015e8d0 0x7fe96006a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36994/0 (socket says 172.21.15.125:36994) 2024-01-16T15:09:01.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.248+0000 7fe965400640 1 -- 172.21.15.125:0/3551473955 learned_addr learned my addr 172.21.15.125:0/3551473955 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:01.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe965e00640 1 --2- 172.21.15.125:0/3551473955 >> v2:172.21.15.137:3300/0 conn(0x7fe960071200 0x7fe9600b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888533156 0 0) 0x7fe9600b36f0 con 0x7fe96015e8d0 2024-01-16T15:09:01.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe940003520 con 0x7fe96015e8d0 2024-01-16T15:09:01.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe965e00640 1 -- 172.21.15.125:0/3551473955 >> v1:172.21.15.137:6789/0 conn(0x7fe9600cd600 legacy=0x7fe960070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208290554 0 0) 0x7fe940003520 con 0x7fe96015e8d0 2024-01-16T15:09:01.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe965e00640 1 -- 172.21.15.125:0/3551473955 >> v1:172.21.15.125:6789/0 conn(0x7fe96015e8d0 legacy=0x7fe96006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.249+0000 7fe965e00640 1 -- 172.21.15.125:0/3551473955 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600aef50 con 0x7fe960071200 2024-01-16T15:09:01.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.251+0000 7fe965e00640 1 --2- 172.21.15.125:0/3551473955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe9600b3e40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097a0 tx=0x7fe95802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d058f788c0660ece server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.251+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95802f050 con 0x7fe960071200 2024-01-16T15:09:01.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.251+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958031500 con 0x7fe960071200 2024-01-16T15:09:01.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.251+0000 7fe95fe00640 1 -- 172.21.15.125:0/3551473955 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950001670 con 0x7fe960071200 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 -- 172.21.15.125:0/3551473955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 msgr2=0x7fe9600b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 --2- 172.21.15.125:0/3551473955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe9600b3e40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097a0 tx=0x7fe95802b820 comp rx=0 tx=0).stop 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 -- 172.21.15.125:0/3551473955 shutdown_connections 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 --2- 172.21.15.125:0/3551473955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe9600b3e40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 -- 172.21.15.125:0/3551473955 >> 172.21.15.125:0/3551473955 conn(0x7fe9600cf700 msgr2=0x7fe9600cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 -- 172.21.15.125:0/3551473955 shutdown_connections 2024-01-16T15:09:01.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.252+0000 7fe967e00640 1 -- 172.21.15.125:0/3551473955 wait complete. 2024-01-16T15:09:01.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.254+0000 7fe967e00640 1 Processor -- start 2024-01-16T15:09:01.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.254+0000 7fe967e00640 1 -- start start 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe960147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9600cd600 0x7fe9601475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600a9ee0 con 0x7fe96015e8d0 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600a95b0 con 0x7fe960071200 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe967e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600a9820 con 0x7fe9600cd600 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe965e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe965e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54462/0 (socket says 172.21.15.125:54462) 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.255+0000 7fe965e00640 1 -- 172.21.15.125:0/3749091315 learned_addr learned my addr 172.21.15.125:0/3749091315 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:01.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe964a00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9600cd600 0x7fe9601475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965400640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe960147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 -- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9600cd600 msgr2=0x7fe9601475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9600cd600 0x7fe9601475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 -- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 msgr2=0x7fe960147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe960147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 -- 172.21.15.125:0/3749091315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958009400 con 0x7fe96015e8d0 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965400640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe960147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:01.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe965e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097a0 tx=0x7fe9580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958004230 con 0x7fe96015e8d0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958004390 con 0x7fe96015e8d0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95803d960 con 0x7fe96015e8d0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.256+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9600a9b00 con 0x7fe96015e8d0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.257+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960147ae0 con 0x7fe96015e8d0 2024-01-16T15:09:01.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.258+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9600675d0 con 0x7fe96015e8d0 2024-01-16T15:09:01.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.259+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 28) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fe95803a040 con 0x7fe96015e8d0 2024-01-16T15:09:01.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.260+0000 7fe95d600640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 0x7fe93806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.260+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fe9580aa110 con 0x7fe96015e8d0 2024-01-16T15:09:01.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.260+0000 7fe965400640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 0x7fe93806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.261+0000 7fe965400640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 0x7fe93806e2c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe950000f60 tx=0x7fe950009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.265+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe958076780 con 0x7fe96015e8d0 2024-01-16T15:09:01.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.338+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe9580dfe90 con 0x7fe96015e8d0 2024-01-16T15:09:01.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.491+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96915", "target": ["mon-mgr", ""]}) v1 -- 0x7fe96015ecb0 con 0x7fe93806be00 2024-01-16T15:09:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.507+0000 7fe95d600640 1 -- 172.21.15.125:0/3749091315 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fe9500058d0 con 0x7fe93806be00 2024-01-16T15:09:01.484 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:01.484 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:01.485 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.509+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 msgr2=0x7fe93806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.509+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 0x7fe93806e2c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe950000f60 tx=0x7fe950009040 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.509+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 msgr2=0x7fe9600c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.509+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097a0 tx=0x7fe9580046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 shutdown_connections 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe93806be00 0x7fe93806e2c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe96015e8d0 0x7fe9600c45e0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9600cd600 0x7fe9601475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 --2- 172.21.15.125:0/3749091315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe960071200 0x7fe960147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 >> 172.21.15.125:0/3749091315 conn(0x7fe9600cf700 msgr2=0x7fe96006a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 shutdown_connections 2024-01-16T15:09:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.510+0000 7fe967e00640 1 -- 172.21.15.125:0/3749091315 wait complete. 2024-01-16T15:09:01.504 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:01.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96916 2024-01-16T15:09:01.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.714+0000 7f1247000640 1 Processor -- start 2024-01-16T15:09:01.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.714+0000 7f1247000640 1 -- start start 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.715+0000 7f1247000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12400cf9a0 0x7f124015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.715+0000 7f1247000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400b1ee0 con 0x7f1240143ae0 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1247000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400b2780 con 0x7f124015e1e0 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1247000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f124015e990 con 0x7f12400cf9a0 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f123fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1240143ae0 0x7f12400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37006/0 (socket says 172.21.15.125:37006) 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f123fe00640 1 -- 172.21.15.125:0/2976383748 learned_addr learned my addr 172.21.15.125:0/2976383748 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:01.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1237200640 1 --2- 172.21.15.125:0/2976383748 >> v2:172.21.15.125:3300/0 conn(0x7f12400cf9a0 0x7f124015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1237200640 1 -- 172.21.15.125:0/2976383748 >> v1:172.21.15.137:6789/0 conn(0x7f124015e1e0 legacy=0x7f12400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:01.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1237200640 1 -- 172.21.15.125:0/2976383748 >> v1:172.21.15.125:6789/0 conn(0x7f1240143ae0 legacy=0x7f12400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.716+0000 7f1237200640 1 -- 172.21.15.125:0/2976383748 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12400b2130 con 0x7f12400cf9a0 2024-01-16T15:09:01.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.717+0000 7f1237200640 1 --2- 172.21.15.125:0/2976383748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f124015dca0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f12280097a0 tx=0x7f122802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cd14aab2de54a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.717+0000 7f123f400640 1 -- 172.21.15.125:0/2976383748 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228031830 con 0x7f12400cf9a0 2024-01-16T15:09:01.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f123f400640 1 -- 172.21.15.125:0/2976383748 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228031990 con 0x7f12400cf9a0 2024-01-16T15:09:01.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f123f400640 1 -- 172.21.15.125:0/2976383748 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228031d20 con 0x7f12400cf9a0 2024-01-16T15:09:01.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 -- 172.21.15.125:0/2976383748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 msgr2=0x7f124015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 --2- 172.21.15.125:0/2976383748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f124015dca0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f12280097a0 tx=0x7f122802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 -- 172.21.15.125:0/2976383748 shutdown_connections 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 --2- 172.21.15.125:0/2976383748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f124015dca0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 -- 172.21.15.125:0/2976383748 >> 172.21.15.125:0/2976383748 conn(0x7f124006a800 msgr2=0x7f124006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 -- 172.21.15.125:0/2976383748 shutdown_connections 2024-01-16T15:09:01.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.718+0000 7f1247000640 1 -- 172.21.15.125:0/2976383748 wait complete. 2024-01-16T15:09:01.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.720+0000 7f1247000640 1 Processor -- start 2024-01-16T15:09:01.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.721+0000 7f1247000640 1 -- start start 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1240143ae0 0x7f12401f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f124015e1e0 0x7f12400c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12400ad850 con 0x7f12400cf9a0 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12400ae220 con 0x7f1240143ae0 2024-01-16T15:09:01.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f1247000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12401f76a0 con 0x7f124015e1e0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54478/0 (socket says 172.21.15.125:54478) 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 -- 172.21.15.125:0/1276385539 learned_addr learned my addr 172.21.15.125:0/1276385539 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 -- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f124015e1e0 msgr2=0x7f12400c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f124015e1e0 0x7f12400c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 -- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1240143ae0 msgr2=0x7f12401f7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1240143ae0 0x7f12401f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.722+0000 7f123fe00640 1 -- 172.21.15.125:0/1276385539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009400 con 0x7f12400cf9a0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f123fe00640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f123000c4a0 tx=0x7f123000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230005ce0 con 0x7f12400cf9a0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1230005e40 con 0x7f12400cf9a0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230007270 con 0x7f12400cf9a0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12400acc40 con 0x7f12400cf9a0 2024-01-16T15:09:01.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.723+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12400ad200 con 0x7f12400cf9a0 2024-01-16T15:09:01.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.724+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12400675d0 con 0x7f12400cf9a0 2024-01-16T15:09:01.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.725+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1230007490 con 0x7f12400cf9a0 2024-01-16T15:09:01.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.726+0000 7f123cc00640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 0x7f121806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.726+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1230087c50 con 0x7f12400cf9a0 2024-01-16T15:09:01.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.726+0000 7f1237200640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 0x7f121806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.726+0000 7f1237200640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 0x7f121806e350 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1228036000 tx=0x7f1228030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.730+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1230054360 con 0x7f12400cf9a0 2024-01-16T15:09:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.940+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96916", "target": ["mon-mgr", ""]}) v1 -- 0x7f124015e670 con 0x7f121806be90 2024-01-16T15:09:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.956+0000 7f123cc00640 1 -- 172.21.15.125:0/1276385539 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f1228031b80 con 0x7f121806be90 2024-01-16T15:09:01.933 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:01.934 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:01.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 msgr2=0x7f121806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 0x7f121806e350 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1228036000 tx=0x7f1228030000 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 msgr2=0x7f12400adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f123000c4a0 tx=0x7f123000c960 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 shutdown_connections 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f124015e1e0 0x7f12400c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1240143ae0 0x7f12401f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f121806be90 0x7f121806e350 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 --2- 172.21.15.125:0/1276385539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12400cf9a0 0x7f12400adce0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 >> 172.21.15.125:0/1276385539 conn(0x7f124006a800 msgr2=0x7f124015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 shutdown_connections 2024-01-16T15:09:01.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:01.959+0000 7f1247000640 1 -- 172.21.15.125:0/1276385539 wait complete. 2024-01-16T15:09:01.953 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:01.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96917 2024-01-16T15:09:02.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.191+0000 7f4a64200640 1 Processor -- start 2024-01-16T15:09:02.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.191+0000 7f4a64200640 1 -- start start 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a64200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a5c13e360 0x7f4a5c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a64200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c15c5f0 con 0x7f4a5c13df80 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a64200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c06c470 con 0x7f4a5c15f9c0 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a64200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c06c750 con 0x7f4a5c13e360 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a60e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a5c13e360 0x7f4a5c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a60e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a5c13e360 0x7f4a5c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54492/0 (socket says 172.21.15.125:54492) 2024-01-16T15:09:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.193+0000 7f4a60e00640 1 -- 172.21.15.125:0/2872887142 learned_addr learned my addr 172.21.15.125:0/2872887142 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.194+0000 7f4a60e00640 1 -- 172.21.15.125:0/2872887142 >> v1:172.21.15.137:6789/0 conn(0x7f4a5c15f9c0 legacy=0x7f4a5c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.194+0000 7f4a60e00640 1 -- 172.21.15.125:0/2872887142 >> v1:172.21.15.125:6789/0 conn(0x7f4a5c13df80 legacy=0x7f4a5c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.194+0000 7f4a60e00640 1 -- 172.21.15.125:0/2872887142 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c06ca30 con 0x7f4a5c13e360 2024-01-16T15:09:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.195+0000 7f4a60e00640 1 --2- 172.21.15.125:0/2872887142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13e360 0x7f4a5c0cf290 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772cd9d766720148 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.195+0000 7f4a5be00640 1 -- 172.21.15.125:0/2872887142 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4402f850 con 0x7f4a5c13e360 2024-01-16T15:09:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.195+0000 7f4a5be00640 1 -- 172.21.15.125:0/2872887142 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4402f9b0 con 0x7f4a5c13e360 2024-01-16T15:09:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.195+0000 7f4a5be00640 1 -- 172.21.15.125:0/2872887142 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4402fd40 con 0x7f4a5c13e360 2024-01-16T15:09:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 -- 172.21.15.125:0/2872887142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13e360 msgr2=0x7f4a5c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 --2- 172.21.15.125:0/2872887142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13e360 0x7f4a5c0cf290 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402ba30 comp rx=0 tx=0).stop 2024-01-16T15:09:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 -- 172.21.15.125:0/2872887142 shutdown_connections 2024-01-16T15:09:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 --2- 172.21.15.125:0/2872887142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13e360 0x7f4a5c0cf290 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 -- 172.21.15.125:0/2872887142 >> 172.21.15.125:0/2872887142 conn(0x7f4a5c06a800 msgr2=0x7f4a5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:02.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 -- 172.21.15.125:0/2872887142 shutdown_connections 2024-01-16T15:09:02.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.196+0000 7f4a64200640 1 -- 172.21.15.125:0/2872887142 wait complete. 2024-01-16T15:09:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.198+0000 7f4a64200640 1 Processor -- start 2024-01-16T15:09:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.198+0000 7f4a64200640 1 -- start start 2024-01-16T15:09:02.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 0x7f4a5c14ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 0x7f4a5c14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c14ea20 con 0x7f4a5c13df80 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c14dd80 con 0x7f4a5c13e360 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a64200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c14e080 con 0x7f4a5c15f9c0 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a61800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a61800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54502/0 (socket says 172.21.15.125:54502) 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.199+0000 7f4a61800640 1 -- 172.21.15.125:0/106660222 learned_addr learned my addr 172.21.15.125:0/106660222 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a60e00640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 0x7f4a5c14ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a62200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 0x7f4a5c14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 -- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 msgr2=0x7f4a5c14f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 0x7f4a5c14f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 -- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 msgr2=0x7f4a5c14ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 0x7f4a5c14ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 -- 172.21.15.125:0/106660222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440094d0 con 0x7f4a5c13df80 2024-01-16T15:09:02.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a60e00640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 0x7f4a5c14ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a62200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 0x7f4a5c14f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a61800640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c390 tx=0x7f4a4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0052d0 con 0x7f4a5c13df80 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c005430 con 0x7f4a5c13df80 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c14e360 con 0x7f4a5c13df80 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c003830 con 0x7f4a5c13df80 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.200+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c1f1350 con 0x7f4a5c13df80 2024-01-16T15:09:02.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.201+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c14ec90 con 0x7f4a5c13df80 2024-01-16T15:09:02.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.203+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4a4c003990 con 0x7f4a5c13df80 2024-01-16T15:09:02.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.204+0000 7f4a59600640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 0x7f4a3406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.204+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f4a4c087da0 con 0x7f4a5c13df80 2024-01-16T15:09:02.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.204+0000 7f4a60e00640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 0x7f4a3406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.205+0000 7f4a60e00640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 0x7f4a3406e350 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4a44036000 tx=0x7f4a44039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.208+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a4c054400 con 0x7f4a5c13df80 2024-01-16T15:09:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.434+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96917", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a5c15f060 con 0x7f4a3406be90 2024-01-16T15:09:02.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.447+0000 7f4a59600640 1 -- 172.21.15.125:0/106660222 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f4a5c15f060 con 0x7f4a3406be90 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:02.426 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 msgr2=0x7f4a3406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 0x7f4a3406e350 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4a44036000 tx=0x7f4a44039040 comp rx=0 tx=0).stop 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 msgr2=0x7f4a5c1f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c390 tx=0x7f4a4c00c850 comp rx=0 tx=0).stop 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 shutdown_connections 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a5c15f9c0 0x7f4a5c14f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a5c13e360 0x7f4a5c14ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4a3406be90 0x7f4a3406e350 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 --2- 172.21.15.125:0/106660222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a5c13df80 0x7f4a5c1f0e10 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 >> 172.21.15.125:0/106660222 conn(0x7f4a5c06a800 msgr2=0x7f4a5c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:02.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 shutdown_connections 2024-01-16T15:09:02.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.450+0000 7f4a64200640 1 -- 172.21.15.125:0/106660222 wait complete. 2024-01-16T15:09:02.443 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:02.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96918 2024-01-16T15:09:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.672+0000 7f35f7800640 1 Processor -- start 2024-01-16T15:09:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.672+0000 7f35f7800640 1 -- start start 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.673+0000 7f35f7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35f0164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35f7800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b1ee0 con 0x7f35f00cf9a0 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35f7800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b2780 con 0x7f35f0143ae0 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35f7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f015e990 con 0x7f35f015e3b0 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35efe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f35f00cf9a0 0x7f35f015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37026/0 (socket says 172.21.15.125:37026) 2024-01-16T15:09:02.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35efe00640 1 -- 172.21.15.125:0/2898415009 learned_addr learned my addr 172.21.15.125:0/2898415009 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852084786 0 0) 0x7f35f00b1ee0 con 0x7f35f00cf9a0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003520 con 0x7f35f00cf9a0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.674+0000 7f35f5800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35f0164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376238520 0 0) 0x7f35d0003520 con 0x7f35f00cf9a0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 msgr2=0x7f35f0164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35f0164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 >> v1:172.21.15.137:6789/0 conn(0x7f35f0143ae0 legacy=0x7f35f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00b2130 con 0x7f35f00cf9a0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f35d8001670 con 0x7f35f00cf9a0 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 >> v1:172.21.15.125:6789/0 conn(0x7f35f00cf9a0 legacy=0x7f35f015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d0003fd0 0x7f35d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f015e990 con 0x7f35d0004d00 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d000bb40 con 0x7f35d0003fd0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d000df10 con 0x7f35f015e3b0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35f5800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35efe00640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d0003fd0 0x7f35d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.675+0000 7f35f4e00640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.676+0000 7f35f5800640 1 -- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 msgr2=0x7f35d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.676+0000 7f35f5800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f015e3b0 0x7f35d00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.676+0000 7f35f5800640 1 -- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d0003fd0 msgr2=0x7f35d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.676+0000 7f35f5800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d0003fd0 0x7f35d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.676+0000 7f35f5800640 1 -- 172.21.15.125:0/2898415009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0003520 con 0x7f35d0004d00 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f5800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35d0005180 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f35e8007360 tx=0x7f35d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d230b590af00d55c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e802ccd0 con 0x7f35d0004d00 2024-01-16T15:09:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e802ce30 con 0x7f35d0004d00 2024-01-16T15:09:02.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35ef400640 1 -- 172.21.15.125:0/2898415009 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8034660 con 0x7f35d0004d00 2024-01-16T15:09:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f7800640 1 -- 172.21.15.125:0/2898415009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 msgr2=0x7f35d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f7800640 1 --2- 172.21.15.125:0/2898415009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35d0005180 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f35e8007360 tx=0x7f35d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f5800640 1 -- 172.21.15.125:0/2898415009 reap_dead start 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f7800640 1 -- 172.21.15.125:0/2898415009 shutdown_connections 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.677+0000 7f35f7800640 1 -- 172.21.15.125:0/2898415009 >> 172.21.15.125:0/2898415009 conn(0x7f35f006a800 msgr2=0x7f35f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.678+0000 7f35f7800640 1 -- 172.21.15.125:0/2898415009 shutdown_connections 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.678+0000 7f35f7800640 1 -- 172.21.15.125:0/2898415009 wait complete. 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.679+0000 7f35f7800640 1 Processor -- start 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.679+0000 7f35f7800640 1 -- start start 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f0157260 0x7f35f01e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f01e85c0 0x7f35f01e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01e8f80 con 0x7f35d0004d00 2024-01-16T15:09:02.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01e6230 con 0x7f35f0157260 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01576e0 con 0x7f35f01e85c0 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54526/0 (socket says 172.21.15.125:54526) 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.680+0000 7f35f4e00640 1 -- 172.21.15.125:0/1913891579 learned_addr learned my addr 172.21.15.125:0/1913891579 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35efe00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f0157260 0x7f35f01e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f5800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f01e85c0 0x7f35f01e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 -- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f01e85c0 msgr2=0x7f35f01e8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f01e85c0 0x7f35f01e8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 -- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f0157260 msgr2=0x7f35f01e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f0157260 0x7f35f01e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 -- 172.21.15.125:0/1913891579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f01e6460 con 0x7f35d0004d00 2024-01-16T15:09:02.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35f4e00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f35e000e580 tx=0x7f35e000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e00052e0 con 0x7f35d0004d00 2024-01-16T15:09:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.681+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0005440 con 0x7f35d0004d00 2024-01-16T15:09:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.682+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f01e6740 con 0x7f35d0004d00 2024-01-16T15:09:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.682+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0003810 con 0x7f35d0004d00 2024-01-16T15:09:02.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.682+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f01e6d00 con 0x7f35d0004d00 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.683+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f006c3f0 con 0x7f35d0004d00 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.683+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f35e0003970 con 0x7f35d0004d00 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.684+0000 7f35ecc00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 0x7f35c806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.684+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f35e0088610 con 0x7f35d0004d00 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.685+0000 7f35efe00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 0x7f35c806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:02.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.685+0000 7f35efe00640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 0x7f35c806e380 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f35d8001c80 tx=0x7f35d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:02.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.688+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35e0054bf0 con 0x7f35d0004d00 2024-01-16T15:09:02.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.917+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96918", "target": ["mon-mgr", ""]}) v1 -- 0x7f35f00675d0 con 0x7f35c806bec0 2024-01-16T15:09:02.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.931+0000 7f35ecc00640 1 -- 172.21.15.125:0/1913891579 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f35f00675d0 con 0x7f35c806bec0 2024-01-16T15:09:02.908 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:02.908 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:02.909 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:02.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 msgr2=0x7f35c806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 0x7f35c806e380 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f35d8001c80 tx=0x7f35d8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 msgr2=0x7f35f013dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f35e000e580 tx=0x7f35e000ea40 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 shutdown_connections 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35f01e85c0 0x7f35f01e8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35f0157260 0x7f35f01e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f35c806bec0 0x7f35c806e380 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 --2- 172.21.15.125:0/1913891579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35d0004d00 0x7f35f013dad0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 >> 172.21.15.125:0/1913891579 conn(0x7f35f006a800 msgr2=0x7f35f0142db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 shutdown_connections 2024-01-16T15:09:02.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:02.934+0000 7f35f7800640 1 -- 172.21.15.125:0/1913891579 wait complete. 2024-01-16T15:09:02.929 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:02.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96919 2024-01-16T15:09:03.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.165+0000 7f2585c00640 1 Processor -- start 2024-01-16T15:09:03.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.165+0000 7f2585c00640 1 -- start start 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.166+0000 7f2585c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f258013df80 0x7f258015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f2585c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258006c470 con 0x7f25800cf9a0 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f2585c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258006c750 con 0x7f258013e360 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f2585c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258006ca30 con 0x7f258013df80 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f257f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f25800cf9a0 0x7f25800cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37036/0 (socket says 172.21.15.125:37036) 2024-01-16T15:09:03.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f257f400640 1 -- 172.21.15.125:0/2663944803 learned_addr learned my addr 172.21.15.125:0/2663944803 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f257ea00640 1 --2- 172.21.15.125:0/2663944803 >> v2:172.21.15.137:3300/0 conn(0x7f258013df80 0x7f258015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797880567 0 0) 0x7f258006c470 con 0x7f25800cf9a0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.167+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2560003520 con 0x7f25800cf9a0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2112092629 0 0) 0x7f2560003520 con 0x7f25800cf9a0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 >> v2:172.21.15.137:3300/0 conn(0x7f258013df80 msgr2=0x7f258015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 --2- 172.21.15.125:0/2663944803 >> v2:172.21.15.137:3300/0 conn(0x7f258013df80 0x7f258015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 >> v1:172.21.15.137:6789/0 conn(0x7f258013e360 legacy=0x7f25800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258006cd10 con 0x7f25800cf9a0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f25780019f0 con 0x7f25800cf9a0 2024-01-16T15:09:03.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 >> v1:172.21.15.125:6789/0 conn(0x7f25800cf9a0 legacy=0x7f25800cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2560003970 0x7f2560003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2560004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2560005180 0x7f256000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258006ca30 con 0x7f25600047c0 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256000c020 con 0x7f2560003970 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256000c1e0 con 0x7f2560005180 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.168+0000 7f2577e00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2560004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f2577e00640 1 -- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2560005180 msgr2=0x7f256000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f2577e00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2560005180 0x7f256000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f2577e00640 1 -- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2560003970 msgr2=0x7f2560003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f257ea00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2560003970 0x7f2560003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f2577e00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2560003970 0x7f2560003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f2577e00640 1 -- 172.21.15.125:0/2663944803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560003520 con 0x7f25600047c0 2024-01-16T15:09:03.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.169+0000 7f257f400640 1 -- 172.21.15.125:0/2663944803 reap_dead start 2024-01-16T15:09:03.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.170+0000 7f2577e00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2560004c40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2568001720 tx=0x7f25600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67efeff1d34eaaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.170+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25680052d0 con 0x7f25600047c0 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.170+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256802e050 con 0x7f25600047c0 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.170+0000 7f257e000640 1 -- 172.21.15.125:0/2663944803 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568003830 con 0x7f25600047c0 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 -- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 msgr2=0x7f2560004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2560004c40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2568001720 tx=0x7f25600057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 -- 172.21.15.125:0/2663944803 shutdown_connections 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 --2- 172.21.15.125:0/2663944803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2560004c40 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 -- 172.21.15.125:0/2663944803 >> 172.21.15.125:0/2663944803 conn(0x7f258006a800 msgr2=0x7f258006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 -- 172.21.15.125:0/2663944803 shutdown_connections 2024-01-16T15:09:03.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.171+0000 7f2585c00640 1 -- 172.21.15.125:0/2663944803 wait complete. 2024-01-16T15:09:03.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.173+0000 7f2585c00640 1 Processor -- start 2024-01-16T15:09:03.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.173+0000 7f2585c00640 1 -- start start 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2580150840 0x7f25801f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f258013d730 0x7f25800b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2580151850 con 0x7f25600047c0 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801f18b0 con 0x7f2580150840 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f2585c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2580150cd0 con 0x7f258013d730 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54536/0 (socket says 172.21.15.125:54536) 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 -- 172.21.15.125:0/1437501236 learned_addr learned my addr 172.21.15.125:0/1437501236 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:03.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 -- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f258013d730 msgr2=0x7f25800b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f258013d730 0x7f25800b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 -- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2580150840 msgr2=0x7f25801f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2580150840 0x7f25801f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.174+0000 7f257ea00640 1 -- 172.21.15.125:0/1437501236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580150fb0 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f257ea00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2570010790 tx=0x7f2570010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570009630 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257000eb60 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2580151290 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570010ea0 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.175+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2580156b80 con 0x7f25600047c0 2024-01-16T15:09:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.176+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25800675d0 con 0x7f25600047c0 2024-01-16T15:09:03.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.177+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f257001c400 con 0x7f25600047c0 2024-01-16T15:09:03.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.178+0000 7f2576a00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 0x7f255806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.179+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f2570088d20 con 0x7f25600047c0 2024-01-16T15:09:03.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.179+0000 7f2577e00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 0x7f255806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.179+0000 7f2577e00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 0x7f255806e350 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2568033000 tx=0x7f256802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.183+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2570055380 con 0x7f25600047c0 2024-01-16T15:09:03.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.411+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96919", "target": ["mon-mgr", ""]}) v1 -- 0x7f25800cf290 con 0x7f255806be90 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.427+0000 7f2576a00640 1 -- 172.21.15.125:0/1437501236 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f25680052b0 con 0x7f255806be90 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:03.404 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:03.405 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:03.405 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:03.405 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:03.405 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:03.405 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 msgr2=0x7f255806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 0x7f255806e350 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2568033000 tx=0x7f256802f000 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 msgr2=0x7f2580151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2570010790 tx=0x7f2570010c50 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 shutdown_connections 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f258013d730 0x7f25800b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2580150840 0x7f25801f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f255806be90 0x7f255806e350 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 --2- 172.21.15.125:0/1437501236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25600047c0 0x7f2580151e20 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.429+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 >> 172.21.15.125:0/1437501236 conn(0x7f258006a800 msgr2=0x7f25800cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.430+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 shutdown_connections 2024-01-16T15:09:03.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.430+0000 7f2585c00640 1 -- 172.21.15.125:0/1437501236 wait complete. 2024-01-16T15:09:03.424 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:03.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969110 2024-01-16T15:09:03.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.647+0000 7fa40ec00640 1 Processor -- start 2024-01-16T15:09:03.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.647+0000 7fa40ec00640 1 -- start start 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa40815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40ec00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa40815e1e0 0x7fa4080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40ec00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080b1ee0 con 0x7fa408143ae0 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40ec00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4080b2780 con 0x7fa40815e1e0 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40815e990 con 0x7fa4080cf9a0 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa407e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa408143ae0 0x7fa4080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37038/0 (socket says 172.21.15.125:37038) 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa407e00640 1 -- 172.21.15.125:0/4050686910 learned_addr learned my addr 172.21.15.125:0/4050686910 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:03.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa40cc00640 1 --2- 172.21.15.125:0/4050686910 >> v2:172.21.15.125:3300/0 conn(0x7fa40815e1e0 0x7fa4080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.649+0000 7fa407400640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa40815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.650+0000 7fa40cc00640 1 -- 172.21.15.125:0/4050686910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 msgr2=0x7fa40815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.650+0000 7fa40cc00640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa40815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.650+0000 7fa40cc00640 1 -- 172.21.15.125:0/4050686910 >> v1:172.21.15.125:6789/0 conn(0x7fa408143ae0 legacy=0x7fa4080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.650+0000 7fa40cc00640 1 -- 172.21.15.125:0/4050686910 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080b2130 con 0x7fa40815e1e0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.650+0000 7fa407400640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa40815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa40cc00640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080702c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa4000097a0 tx=0x7fa40002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b51a0abead135dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa406a00640 1 -- 172.21.15.125:0/4050686910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40002d040 con 0x7fa40815e1e0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa406a00640 1 -- 172.21.15.125:0/4050686910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400031810 con 0x7fa40815e1e0 2024-01-16T15:09:03.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa406a00640 1 -- 172.21.15.125:0/4050686910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400031ba0 con 0x7fa40815e1e0 2024-01-16T15:09:03.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa40ec00640 1 -- 172.21.15.125:0/4050686910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 msgr2=0x7fa4080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa40ec00640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080702c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa4000097a0 tx=0x7fa40002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa40ec00640 1 -- 172.21.15.125:0/4050686910 shutdown_connections 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.651+0000 7fa40ec00640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080702c0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.652+0000 7fa40ec00640 1 --2- 172.21.15.125:0/4050686910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa40815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.652+0000 7fa40ec00640 1 -- 172.21.15.125:0/4050686910 >> 172.21.15.125:0/4050686910 conn(0x7fa40806a800 msgr2=0x7fa40806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.652+0000 7fa40ec00640 1 -- 172.21.15.125:0/4050686910 shutdown_connections 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.652+0000 7fa40ec00640 1 -- 172.21.15.125:0/4050686910 wait complete. 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.653+0000 7fa40ec00640 1 Processor -- start 2024-01-16T15:09:03.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.653+0000 7fa40ec00640 1 -- start start 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa4081f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 0x7fa4080ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40813c8d0 con 0x7fa40815e1e0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4081f4070 con 0x7fa408143ae0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.654+0000 7fa40ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40813afc0 con 0x7fa4080cf9a0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54556/0 (socket says 172.21.15.125:54556) 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 -- 172.21.15.125:0/3167404366 learned_addr learned my addr 172.21.15.125:0/3167404366 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa407e00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa4081f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa407400640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 0x7fa4080ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 -- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 msgr2=0x7fa4081f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa4081f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 -- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 msgr2=0x7fa4080ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 0x7fa4080ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 -- 172.21.15.125:0/3167404366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400009400 con 0x7fa40815e1e0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa407400640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 0x7fa4080ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.655+0000 7fa40cc00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa4000097a0 tx=0x7fa4000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.656+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40003ace0 con 0x7fa40815e1e0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.656+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40003ae60 con 0x7fa40815e1e0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.656+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400044630 con 0x7fa40815e1e0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.656+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa408139f20 con 0x7fa40815e1e0 2024-01-16T15:09:03.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.656+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40813a4e0 con 0x7fa40815e1e0 2024-01-16T15:09:03.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.657+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4080675d0 con 0x7fa40815e1e0 2024-01-16T15:09:03.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.658+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa40004d450 con 0x7fa40815e1e0 2024-01-16T15:09:03.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.658+0000 7fa3ffe00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 0x7fa3e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.659+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fa4000aab10 con 0x7fa40815e1e0 2024-01-16T15:09:03.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.659+0000 7fa407e00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 0x7fa3e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.659+0000 7fa407e00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 0x7fa3e006e350 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8005c00 tx=0x7fa3f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.663+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa400077170 con 0x7fa40815e1e0 2024-01-16T15:09:03.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.887+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969110", "target": ["mon-mgr", ""]}) v1 -- 0x7fa40815e640 con 0x7fa3e006be90 2024-01-16T15:09:03.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.908+0000 7fa3ffe00640 1 -- 172.21.15.125:0/3167404366 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fa3f8001df0 con 0x7fa3e006be90 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:03.886 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:03.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 msgr2=0x7fa3e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 0x7fa3e006e350 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8005c00 tx=0x7fa3f8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:03.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 msgr2=0x7fa4080ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa4000097a0 tx=0x7fa4000046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 shutdown_connections 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa3e006be90 0x7fa3e006e350 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa40815e1e0 0x7fa4080ada50 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa408143ae0 0x7fa4080ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 --2- 172.21.15.125:0/3167404366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa4080cf9a0 0x7fa4081f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 >> 172.21.15.125:0/3167404366 conn(0x7fa40806a800 msgr2=0x7fa408070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 shutdown_connections 2024-01-16T15:09:03.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:03.911+0000 7fa40ec00640 1 -- 172.21.15.125:0/3167404366 wait complete. 2024-01-16T15:09:03.906 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:03.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969111 2024-01-16T15:09:04.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.135+0000 7fe0f7e00640 1 Processor -- start 2024-01-16T15:09:04.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.136+0000 7fe0f7e00640 1 -- start start 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f013df80 0x7fe0f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0f013e360 0x7fe0f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f7e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f006c470 con 0x7fe0f015f9c0 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f7e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f006c750 con 0x7fe0f013e360 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f006ca30 con 0x7fe0f013df80 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0f013e360 0x7fe0f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0f013e360 0x7fe0f00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54560/0 (socket says 172.21.15.125:54560) 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.137+0000 7fe0f4a00640 1 -- 172.21.15.125:0/3072311609 learned_addr learned my addr 172.21.15.125:0/3072311609 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.138+0000 7fe0f5400640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f013df80 0x7fe0f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.138+0000 7fe0f4a00640 1 -- 172.21.15.125:0/3072311609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f013df80 msgr2=0x7fe0f015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.138+0000 7fe0f4a00640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f013df80 0x7fe0f015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.138+0000 7fe0f4a00640 1 -- 172.21.15.125:0/3072311609 >> v1:172.21.15.137:6789/0 conn(0x7fe0f015f9c0 legacy=0x7fe0f01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:04.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.138+0000 7fe0f4a00640 1 -- 172.21.15.125:0/3072311609 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f006cd10 con 0x7fe0f013e360 2024-01-16T15:09:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.139+0000 7fe0f4a00640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f00cf290 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009870 tx=0x7fe0d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f2f5d9ea0af071 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.139+0000 7fe0efe00640 1 -- 172.21.15.125:0/3072311609 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8031640 con 0x7fe0f013e360 2024-01-16T15:09:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.139+0000 7fe0efe00640 1 -- 172.21.15.125:0/3072311609 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d80317a0 con 0x7fe0f013e360 2024-01-16T15:09:04.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.139+0000 7fe0efe00640 1 -- 172.21.15.125:0/3072311609 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8031b30 con 0x7fe0f013e360 2024-01-16T15:09:04.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 -- 172.21.15.125:0/3072311609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 msgr2=0x7fe0f00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f00cf290 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009870 tx=0x7fe0d802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 -- 172.21.15.125:0/3072311609 shutdown_connections 2024-01-16T15:09:04.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f00cf290 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/3072311609 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f013df80 0x7fe0f015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 -- 172.21.15.125:0/3072311609 >> 172.21.15.125:0/3072311609 conn(0x7fe0f006a800 msgr2=0x7fe0f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:04.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 -- 172.21.15.125:0/3072311609 shutdown_connections 2024-01-16T15:09:04.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.140+0000 7fe0f7e00640 1 -- 172.21.15.125:0/3072311609 wait complete. 2024-01-16T15:09:04.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.142+0000 7fe0f7e00640 1 Processor -- start 2024-01-16T15:09:04.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.142+0000 7fe0f7e00640 1 -- start start 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f013df80 0x7fe0f01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f015f9c0 0x7fe0f01f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00b3b90 con 0x7fe0f013e360 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f014de30 con 0x7fe0f013df80 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f014d9a0 con 0x7fe0f015f9c0 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54564/0 (socket says 172.21.15.125:54564) 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 -- 172.21.15.125:0/4059447191 learned_addr learned my addr 172.21.15.125:0/4059447191 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f5400640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f013df80 0x7fe0f01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 -- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f015f9c0 msgr2=0x7fe0f01f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f015f9c0 0x7fe0f01f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 -- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f013df80 msgr2=0x7fe0f01515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f013df80 0x7fe0f01515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.143+0000 7fe0f4a00640 1 -- 172.21.15.125:0/4059447191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d80094d0 con 0x7fe0f013e360 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0f4a00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d803ac60 con 0x7fe0f013e360 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d803adc0 con 0x7fe0f013e360 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8044630 con 0x7fe0f013e360 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f014c370 con 0x7fe0f013e360 2024-01-16T15:09:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.144+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f014d350 con 0x7fe0f013e360 2024-01-16T15:09:04.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.146+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f00675d0 con 0x7fe0f013e360 2024-01-16T15:09:04.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.146+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe0d802f050 con 0x7fe0f013e360 2024-01-16T15:09:04.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.147+0000 7fe0ed600640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 0x7fe0c806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.148+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fe0d80aa760 con 0x7fe0f013e360 2024-01-16T15:09:04.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.148+0000 7fe0f5400640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 0x7fe0c806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.148+0000 7fe0f5400640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 0x7fe0c806e380 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0006fd0 tx=0x7fe0e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.152+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0d8076d30 con 0x7fe0f013e360 2024-01-16T15:09:04.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.378+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969111", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0f00ce850 con 0x7fe0c806bec0 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.396+0000 7fe0ed600640 1 -- 172.21.15.125:0/4059447191 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fe0e0009480 con 0x7fe0c806bec0 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:04.374 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:04.374 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:04.374 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:04.374 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.398+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 msgr2=0x7fe0c806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.398+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 0x7fe0c806e380 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0006fd0 tx=0x7fe0e0008040 comp rx=0 tx=0).stop 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 msgr2=0x7fe0f014f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d80046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 shutdown_connections 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f015f9c0 0x7fe0f01f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0c806bec0 0x7fe0c806e380 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f013e360 0x7fe0f014f760 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 --2- 172.21.15.125:0/4059447191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f013df80 0x7fe0f01515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 >> 172.21.15.125:0/4059447191 conn(0x7fe0f006a800 msgr2=0x7fe0f015d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 shutdown_connections 2024-01-16T15:09:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.399+0000 7fe0f7e00640 1 -- 172.21.15.125:0/4059447191 wait complete. 2024-01-16T15:09:04.393 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:04.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969112 2024-01-16T15:09:04.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.619+0000 7f31cc800640 1 Processor -- start 2024-01-16T15:09:04.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.619+0000 7f31cc800640 1 -- start start 2024-01-16T15:09:04.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.621+0000 7f31cc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c4143ae0 0x7f31c40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.621+0000 7f31cc800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f31c40cf7d0 0x7f31c415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.621+0000 7f31cc800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c40cfc30 con 0x7f31c415e1e0 2024-01-16T15:09:04.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.621+0000 7f31cc800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c40b1ee0 con 0x7f31c40cf7d0 2024-01-16T15:09:04.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.621+0000 7f31cc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40b2780 con 0x7f31c4143ae0 2024-01-16T15:09:04.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.622+0000 7f31c9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f31c40cf7d0 0x7f31c415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.622+0000 7f31c9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f31c40cf7d0 0x7f31c415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:52614/0 (socket says 172.21.15.125:52614) 2024-01-16T15:09:04.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.622+0000 7f31c9400640 1 -- 172.21.15.125:0/2808650068 learned_addr learned my addr 172.21.15.125:0/2808650068 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:04.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.622+0000 7f31c9e00640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c4143ae0 0x7f31c40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.623+0000 7f31c9400640 1 -- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c4143ae0 msgr2=0x7f31c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:04.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.623+0000 7f31c9400640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c4143ae0 0x7f31c40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.623+0000 7f31c9400640 1 -- 172.21.15.125:0/2808650068 >> v1:172.21.15.137:6789/0 conn(0x7f31c415e1e0 legacy=0x7f31c415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.623+0000 7f31c9400640 1 -- 172.21.15.125:0/2808650068 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c415e990 con 0x7f31c40cf7d0 2024-01-16T15:09:04.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.624+0000 7f31c9400640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c40cf7d0 0x7f31c415dca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009870 tx=0x7f31ac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9feaa461cbf419e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.624+0000 7f31c8a00640 1 -- 172.21.15.125:0/2808650068 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac030840 con 0x7f31c40cf7d0 2024-01-16T15:09:04.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.624+0000 7f31c8a00640 1 -- 172.21.15.125:0/2808650068 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac0309a0 con 0x7f31c40cf7d0 2024-01-16T15:09:04.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.624+0000 7f31c8a00640 1 -- 172.21.15.125:0/2808650068 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac030d30 con 0x7f31c40cf7d0 2024-01-16T15:09:04.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 -- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c40cf7d0 msgr2=0x7f31c415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c40cf7d0 0x7f31c415dca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009870 tx=0x7f31ac02b980 comp rx=0 tx=0).stop 2024-01-16T15:09:04.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 -- 172.21.15.125:0/2808650068 shutdown_connections 2024-01-16T15:09:04.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c40cf7d0 0x7f31c415dca0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 --2- 172.21.15.125:0/2808650068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c4143ae0 0x7f31c40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 -- 172.21.15.125:0/2808650068 >> 172.21.15.125:0/2808650068 conn(0x7f31c406a800 msgr2=0x7f31c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:04.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 -- 172.21.15.125:0/2808650068 shutdown_connections 2024-01-16T15:09:04.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.625+0000 7f31cc800640 1 -- 172.21.15.125:0/2808650068 wait complete. 2024-01-16T15:09:04.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.627+0000 7f31cc800640 1 Processor -- start 2024-01-16T15:09:04.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.627+0000 7f31cc800640 1 -- start start 2024-01-16T15:09:04.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c40cf7d0 0x7f31c4159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c4143ae0 0x7f31c413a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4159eb0 con 0x7f31c415e1e0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40cc610 con 0x7f31c4143ae0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31cc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40cc860 con 0x7f31c40cf7d0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31ca800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31ca800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54570/0 (socket says 172.21.15.125:54570) 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.628+0000 7f31ca800640 1 -- 172.21.15.125:0/65734586 learned_addr learned my addr 172.21.15.125:0/65734586 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31c9e00640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c40cf7d0 0x7f31c4159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31c9400640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c4143ae0 0x7f31c413a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 -- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c40cf7d0 msgr2=0x7f31c4159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c40cf7d0 0x7f31c4159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 -- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c4143ae0 msgr2=0x7f31c413a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c4143ae0 0x7f31c413a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 -- 172.21.15.125:0/65734586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0094d0 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31ca800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c6b0 tx=0x7f31bc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc003b70 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc003cf0 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc003ea0 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c40ccb40 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.629+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c40cd100 con 0x7f31c415e1e0 2024-01-16T15:09:04.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.631+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c40cf5f0 con 0x7f31c415e1e0 2024-01-16T15:09:04.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.631+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f31bc005310 con 0x7f31c415e1e0 2024-01-16T15:09:04.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.632+0000 7f31c2000640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 0x7f319c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:04.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.632+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f31bc088580 con 0x7f31c415e1e0 2024-01-16T15:09:04.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.632+0000 7f31c9e00640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 0x7f319c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:04.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.633+0000 7f31c9e00640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 0x7f319c06e350 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009a10 tx=0x7f31b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:04.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.637+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31bc054c20 con 0x7f31c415e1e0 2024-01-16T15:09:04.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.862+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969112", "target": ["mon-mgr", ""]}) v1 -- 0x7f31c415e640 con 0x7f319c06be90 2024-01-16T15:09:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.881+0000 7f31c2000640 1 -- 172.21.15.125:0/65734586 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f31b4010870 con 0x7f319c06be90 2024-01-16T15:09:04.858 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:04.858 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:04.858 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:04.858 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:04.859 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 msgr2=0x7f319c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 0x7f319c06e350 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009a10 tx=0x7f31b4008040 comp rx=0 tx=0).stop 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 msgr2=0x7f31c41419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c6b0 tx=0x7f31bc00cb70 comp rx=0 tx=0).stop 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 shutdown_connections 2024-01-16T15:09:04.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f319c06be90 0x7f319c06e350 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c415e1e0 0x7f31c41419c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31c4143ae0 0x7f31c413a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 --2- 172.21.15.125:0/65734586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31c40cf7d0 0x7f31c4159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 >> 172.21.15.125:0/65734586 conn(0x7f31c406a800 msgr2=0x7f31c415ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 shutdown_connections 2024-01-16T15:09:04.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:04.884+0000 7f31cc800640 1 -- 172.21.15.125:0/65734586 wait complete. 2024-01-16T15:09:04.877 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:04.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969113 2024-01-16T15:09:05.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.102+0000 7f39d8400640 1 Processor -- start 2024-01-16T15:09:05.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.103+0000 7f39d8400640 1 -- start start 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d8400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d8400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f39d009a170 0x7f39d006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d8400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d00cfa70 con 0x7f39d0070490 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d8400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d00cff00 con 0x7f39d009a170 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d8400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d006fc70 con 0x7f39d015e880 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d6400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f39d009a170 0x7f39d006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d6400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f39d009a170 0x7f39d006e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54584/0 (socket says 172.21.15.125:54584) 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.104+0000 7f39d6400640 1 -- 172.21.15.125:0/2678063749 learned_addr learned my addr 172.21.15.125:0/2678063749 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.105+0000 7f39d5a00640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.105+0000 7f39d6400640 1 -- 172.21.15.125:0/2678063749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 msgr2=0x7f39d006ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.105+0000 7f39d6400640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d006ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.105+0000 7f39d6400640 1 -- 172.21.15.125:0/2678063749 >> v1:172.21.15.137:6789/0 conn(0x7f39d0070490 legacy=0x7f39d0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.105+0000 7f39d6400640 1 -- 172.21.15.125:0/2678063749 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00cf420 con 0x7f39d009a170 2024-01-16T15:09:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.106+0000 7f39d6400640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d009a170 0x7f39d006e7c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009e50 tx=0x7f39c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c79b66643bbc8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.106+0000 7f39cfe00640 1 -- 172.21.15.125:0/2678063749 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c802cb00 con 0x7f39d009a170 2024-01-16T15:09:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.106+0000 7f39cfe00640 1 -- 172.21.15.125:0/2678063749 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c802cc60 con 0x7f39d009a170 2024-01-16T15:09:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.106+0000 7f39cfe00640 1 -- 172.21.15.125:0/2678063749 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c80343f0 con 0x7f39d009a170 2024-01-16T15:09:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 -- 172.21.15.125:0/2678063749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d009a170 msgr2=0x7f39d006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d009a170 0x7f39d006e7c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009e50 tx=0x7f39c802bf60 comp rx=0 tx=0).stop 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 -- 172.21.15.125:0/2678063749 shutdown_connections 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d009a170 0x7f39d006e7c0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 --2- 172.21.15.125:0/2678063749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d006ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 -- 172.21.15.125:0/2678063749 >> 172.21.15.125:0/2678063749 conn(0x7f39d00b2d00 msgr2=0x7f39d00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 -- 172.21.15.125:0/2678063749 shutdown_connections 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.107+0000 7f39d8400640 1 -- 172.21.15.125:0/2678063749 wait complete. 2024-01-16T15:09:05.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.109+0000 7f39d8400640 1 Processor -- start 2024-01-16T15:09:05.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.110+0000 7f39d8400640 1 -- start start 2024-01-16T15:09:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d009a170 0x7f39d006f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d01f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00b70f0 con 0x7f39d0070490 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d01f3df0 con 0x7f39d009a170 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d8400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d01f3b10 con 0x7f39d015e880 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54590/0 (socket says 172.21.15.125:54590) 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d5a00640 1 -- 172.21.15.125:0/3506296064 learned_addr learned my addr 172.21.15.125:0/3506296064 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d6400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d01f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.111+0000 7f39d5000640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d009a170 0x7f39d006f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 -- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 msgr2=0x7f39d01f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d01f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 -- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d009a170 msgr2=0x7f39d006f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d009a170 0x7f39d006f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 -- 172.21.15.125:0/3506296064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8009b30 con 0x7f39d0070490 2024-01-16T15:09:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d5a00640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f39c000d800 tx=0x7f39c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c000b6c0 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c000b820 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c000bbb0 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d01f65e0 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.112+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d01f6ba0 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.114+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d00675d0 con 0x7f39d0070490 2024-01-16T15:09:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.115+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f39c000bd10 con 0x7f39d0070490 2024-01-16T15:09:05.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.115+0000 7f39cd600640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 0x7f39a806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.115+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f39c0088db0 con 0x7f39d0070490 2024-01-16T15:09:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.116+0000 7f39d5000640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 0x7f39a806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.116+0000 7f39d5000640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 0x7f39a806e350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.120+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39c0055410 con 0x7f39d0070490 2024-01-16T15:09:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.348+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969113", "target": ["mon-mgr", ""]}) v1 -- 0x7f39d00ccb30 con 0x7f39a806be90 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.364+0000 7f39cd600640 1 -- 172.21.15.125:0/3506296064 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f39b800bd80 con 0x7f39a806be90 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:05.342 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:05.343 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:05.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:05.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:05.343 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.367+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 msgr2=0x7f39a806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.367+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 0x7f39a806e350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.367+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 msgr2=0x7f39d006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.367+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f39c000d800 tx=0x7f39c000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.367+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 shutdown_connections 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39d015e880 0x7f39d01f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39d009a170 0x7f39d006f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f39a806be90 0x7f39a806e350 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 --2- 172.21.15.125:0/3506296064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39d0070490 0x7f39d006ef80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 >> 172.21.15.125:0/3506296064 conn(0x7f39d00b2d00 msgr2=0x7f39d006cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 shutdown_connections 2024-01-16T15:09:05.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.368+0000 7f39d8400640 1 -- 172.21.15.125:0/3506296064 wait complete. 2024-01-16T15:09:05.364 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:05.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969114 2024-01-16T15:09:05.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.603+0000 7fab90600640 1 Processor -- start 2024-01-16T15:09:05.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.603+0000 7fab90600640 1 -- start start 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.605+0000 7fab90600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8813df80 0x7fab8815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.605+0000 7fab90600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab8813e360 0x7fab880cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.605+0000 7fab90600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8806c470 con 0x7fab8815f9c0 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.605+0000 7fab90600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8806c750 con 0x7fab8813e360 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.605+0000 7fab90600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8806ca30 con 0x7fab8813df80 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab8813e360 0x7fab880cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab8813e360 0x7fab880cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54594/0 (socket says 172.21.15.125:54594) 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 -- 172.21.15.125:0/4189011367 learned_addr learned my addr 172.21.15.125:0/4189011367 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8dc00640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8813df80 0x7fab8815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 -- 172.21.15.125:0/4189011367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8813df80 msgr2=0x7fab8815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8813df80 0x7fab8815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 -- 172.21.15.125:0/4189011367 >> v1:172.21.15.137:6789/0 conn(0x7fab8815f9c0 legacy=0x7fab881408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.606+0000 7fab8d200640 1 -- 172.21.15.125:0/4189011367 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8806cd10 con 0x7fab8813e360 2024-01-16T15:09:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.607+0000 7fab8d200640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab880cf290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cda4144867e5124 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.607+0000 7fab87e00640 1 -- 172.21.15.125:0/4189011367 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70031640 con 0x7fab8813e360 2024-01-16T15:09:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.607+0000 7fab87e00640 1 -- 172.21.15.125:0/4189011367 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab700317a0 con 0x7fab8813e360 2024-01-16T15:09:05.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.607+0000 7fab87e00640 1 -- 172.21.15.125:0/4189011367 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70031b30 con 0x7fab8813e360 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 -- 172.21.15.125:0/4189011367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 msgr2=0x7fab880cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab880cf290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 -- 172.21.15.125:0/4189011367 shutdown_connections 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab880cf290 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 --2- 172.21.15.125:0/4189011367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8813df80 0x7fab8815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 -- 172.21.15.125:0/4189011367 >> 172.21.15.125:0/4189011367 conn(0x7fab8806a800 msgr2=0x7fab8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 -- 172.21.15.125:0/4189011367 shutdown_connections 2024-01-16T15:09:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.608+0000 7fab90600640 1 -- 172.21.15.125:0/4189011367 wait complete. 2024-01-16T15:09:05.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.610+0000 7fab90600640 1 Processor -- start 2024-01-16T15:09:05.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.610+0000 7fab90600640 1 -- start start 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab8813df80 0x7fab881515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8815f9c0 0x7fab881f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab880b3b90 con 0x7fab8813e360 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8814de30 con 0x7fab8813df80 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab90600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8814d9a0 con 0x7fab8815f9c0 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab8d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab8d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54598/0 (socket says 172.21.15.125:54598) 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.611+0000 7fab8d200640 1 -- 172.21.15.125:0/1340727565 learned_addr learned my addr 172.21.15.125:0/1340727565 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8dc00640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab8813df80 0x7fab881515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8e600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8815f9c0 0x7fab881f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 -- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8815f9c0 msgr2=0x7fab881f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8815f9c0 0x7fab881f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 -- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab8813df80 msgr2=0x7fab881515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab8813df80 0x7fab881515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 -- 172.21.15.125:0/1340727565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700094d0 con 0x7fab8813e360 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab8d200640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fab70000c00 tx=0x7fab700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7003ac60 con 0x7fab8813e360 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.612+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7003ade0 con 0x7fab8813e360 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.613+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70044630 con 0x7fab8813e360 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.613+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8814c370 con 0x7fab8813e360 2024-01-16T15:09:05.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.613+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8814d350 con 0x7fab8813e360 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.614+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab880675d0 con 0x7fab8813e360 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.615+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fab7002f050 con 0x7fab8813e360 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.615+0000 7fab85600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 0x7fab6006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.616+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fab700aa950 con 0x7fab8813e360 2024-01-16T15:09:05.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.616+0000 7fab8dc00640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 0x7fab6006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.616+0000 7fab8dc00640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 0x7fab6006e380 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.620+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab70076f80 con 0x7fab8813e360 2024-01-16T15:09:05.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.846+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969114", "target": ["mon-mgr", ""]}) v1 -- 0x7fab880ce850 con 0x7fab6006bec0 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.859+0000 7fab85600640 1 -- 172.21.15.125:0/1340727565 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fab7800c050 con 0x7fab6006bec0 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:05.837 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 msgr2=0x7fab6006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 0x7fab6006e380 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 msgr2=0x7fab8814f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fab70000c00 tx=0x7fab700046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 shutdown_connections 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab8815f9c0 0x7fab881f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab6006bec0 0x7fab6006e380 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab8813e360 0x7fab8814f760 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 --2- 172.21.15.125:0/1340727565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab8813df80 0x7fab881515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.861+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 >> 172.21.15.125:0/1340727565 conn(0x7fab8806a800 msgr2=0x7fab8815d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.862+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 shutdown_connections 2024-01-16T15:09:05.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:05.862+0000 7fab90600640 1 -- 172.21.15.125:0/1340727565 wait complete. 2024-01-16T15:09:05.856 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:05.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969115 2024-01-16T15:09:06.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.083+0000 7f8f71000640 1 Processor -- start 2024-01-16T15:09:06.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.083+0000 7f8f71000640 1 -- start start 2024-01-16T15:09:06.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.084+0000 7f8f71000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c143ae0 0x7f8f6c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.084+0000 7f8f71000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8f6c0cf7d0 0x7f8f6c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.084+0000 7f8f71000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c0cfc30 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.084+0000 7f8f71000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c0b1ee0 con 0x7f8f6c143ae0 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.084+0000 7f8f71000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c0b2780 con 0x7f8f6c0cf7d0 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c143ae0 0x7f8f6c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c143ae0 0x7f8f6c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54614/0 (socket says 172.21.15.125:54614) 2024-01-16T15:09:06.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 -- 172.21.15.125:0/2968537377 learned_addr learned my addr 172.21.15.125:0/2968537377 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:06.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 -- 172.21.15.125:0/2968537377 >> v2:172.21.15.137:3300/0 conn(0x7f8f6c0cf7d0 msgr2=0x7f8f6c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 --2- 172.21.15.125:0/2968537377 >> v2:172.21.15.137:3300/0 conn(0x7f8f6c0cf7d0 0x7f8f6c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 -- 172.21.15.125:0/2968537377 >> v1:172.21.15.137:6789/0 conn(0x7f8f6c15e1e0 legacy=0x7f8f6c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.085+0000 7f8f69e00640 1 -- 172.21.15.125:0/2968537377 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c15e990 con 0x7f8f6c143ae0 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.086+0000 7f8f69e00640 1 --2- 172.21.15.125:0/2968537377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c143ae0 0x7f8f6c0cf290 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0cb830 tx=0x7f8f5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcd4314baaa8bf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f69400640 1 -- 172.21.15.125:0/2968537377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031870 con 0x7f8f6c143ae0 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f69400640 1 -- 172.21.15.125:0/2968537377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0319d0 con 0x7f8f6c143ae0 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f69400640 1 -- 172.21.15.125:0/2968537377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031d60 con 0x7f8f6c143ae0 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 -- 172.21.15.125:0/2968537377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c143ae0 msgr2=0x7f8f6c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 --2- 172.21.15.125:0/2968537377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c143ae0 0x7f8f6c0cf290 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0cb830 tx=0x7f8f5c02b910 comp rx=0 tx=0).stop 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 -- 172.21.15.125:0/2968537377 shutdown_connections 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 --2- 172.21.15.125:0/2968537377 >> v2:172.21.15.137:3300/0 conn(0x7f8f6c0cf7d0 0x7f8f6c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 --2- 172.21.15.125:0/2968537377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c143ae0 0x7f8f6c0cf290 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 -- 172.21.15.125:0/2968537377 >> 172.21.15.125:0/2968537377 conn(0x7f8f6c06a800 msgr2=0x7f8f6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 -- 172.21.15.125:0/2968537377 shutdown_connections 2024-01-16T15:09:06.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.087+0000 7f8f71000640 1 -- 172.21.15.125:0/2968537377 wait complete. 2024-01-16T15:09:06.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.089+0000 7f8f71000640 1 Processor -- start 2024-01-16T15:09:06.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.089+0000 7f8f71000640 1 -- start start 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f6c0cf7d0 0x7f8f6c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6c143ae0 0x7f8f6c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c13af40 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c13a070 con 0x7f8f6c0cf7d0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f71000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cc610 con 0x7f8f6c143ae0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f6a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.090+0000 7f8f6a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54620/0 (socket says 172.21.15.125:54620) 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 -- 172.21.15.125:0/1947438175 learned_addr learned my addr 172.21.15.125:0/1947438175 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f69e00640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f6c0cf7d0 0x7f8f6c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f63e00640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6c143ae0 0x7f8f6c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 -- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6c143ae0 msgr2=0x7f8f6c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6c143ae0 0x7f8f6c0cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 -- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f6c0cf7d0 msgr2=0x7f8f6c0ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f6c0cf7d0 0x7f8f6c0ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 -- 172.21.15.125:0/1947438175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0094d0 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.091+0000 7f8f6a800640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400c780 tx=0x7f8f6400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.092+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64003d80 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.092+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6400ce90 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.092+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640052b0 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.092+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c0ae3a0 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.092+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c139d90 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.093+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c0b2bf0 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.094+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8f64005410 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.095+0000 7f8f62000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 0x7f8f4406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.095+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f8f64088090 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.095+0000 7f8f69e00640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 0x7f8f4406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.096+0000 7f8f69e00640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 0x7f8f4406e3a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c036000 tx=0x7f8f5c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.099+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f6c0b2bf0 con 0x7f8f6c15e1e0 2024-01-16T15:09:06.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.327+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969115", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f6c15e670 con 0x7f8f4406bee0 2024-01-16T15:09:06.380 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.345+0000 7f8f62000640 1 -- 172.21.15.125:0/1947438175 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f8f5c031cc0 con 0x7f8f4406bee0 2024-01-16T15:09:06.380 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:06.380 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:06.380 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:06.381 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 msgr2=0x7f8f4406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 0x7f8f4406e3a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c036000 tx=0x7f8f5c03c040 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 msgr2=0x7f8f6c1413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400c780 tx=0x7f8f6400cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 shutdown_connections 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f4406bee0 0x7f8f4406e3a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c15e1e0 0x7f8f6c1413b0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6c143ae0 0x7f8f6c0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 --2- 172.21.15.125:0/1947438175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f6c0cf7d0 0x7f8f6c0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 >> 172.21.15.125:0/1947438175 conn(0x7f8f6c06a800 msgr2=0x7f8f6c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 shutdown_connections 2024-01-16T15:09:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.348+0000 7f8f71000640 1 -- 172.21.15.125:0/1947438175 wait complete. 2024-01-16T15:09:06.384 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:06.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969116 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.629+0000 7f5a57a00640 1 Processor -- start 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.629+0000 7f5a57a00640 1 -- start start 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a57a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a5013df80 0x7f5a5015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a57a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a500cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a57a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5006c470 con 0x7f5a5013e360 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a57a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5006c750 con 0x7f5a5013df80 2024-01-16T15:09:06.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a57a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5006ca30 con 0x7f5a500cf9a0 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a55000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a5013df80 0x7f5a5015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a55000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a5013df80 0x7f5a5015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50792/0 (socket says 172.21.15.125:50792) 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.630+0000 7f5a55000640 1 -- 172.21.15.125:0/4159257311 learned_addr learned my addr 172.21.15.125:0/4159257311 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55a00640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a500cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55000640 1 -- 172.21.15.125:0/4159257311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 msgr2=0x7f5a500cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55000640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a500cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55000640 1 -- 172.21.15.125:0/4159257311 >> v1:172.21.15.137:6789/0 conn(0x7f5a5013e360 legacy=0x7f5a500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55000640 1 -- 172.21.15.125:0/4159257311 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5006cd10 con 0x7f5a5013df80 2024-01-16T15:09:06.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.631+0000 7f5a55a00640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a500cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.632+0000 7f5a55000640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013df80 0x7f5a5015bee0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009870 tx=0x7f5a4002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbacbcb8e484aa4c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.632+0000 7f5a4fe00640 1 -- 172.21.15.125:0/4159257311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40031640 con 0x7f5a5013df80 2024-01-16T15:09:06.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a4fe00640 1 -- 172.21.15.125:0/4159257311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a400317a0 con 0x7f5a5013df80 2024-01-16T15:09:06.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a4fe00640 1 -- 172.21.15.125:0/4159257311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40031b30 con 0x7f5a5013df80 2024-01-16T15:09:06.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 -- 172.21.15.125:0/4159257311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013df80 msgr2=0x7f5a5015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013df80 0x7f5a5015bee0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009870 tx=0x7f5a4002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 -- 172.21.15.125:0/4159257311 shutdown_connections 2024-01-16T15:09:06.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a500cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 --2- 172.21.15.125:0/4159257311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013df80 0x7f5a5015bee0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 -- 172.21.15.125:0/4159257311 >> 172.21.15.125:0/4159257311 conn(0x7f5a5006a800 msgr2=0x7f5a5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 -- 172.21.15.125:0/4159257311 shutdown_connections 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.633+0000 7f5a57a00640 1 -- 172.21.15.125:0/4159257311 wait complete. 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.635+0000 7f5a57a00640 1 Processor -- start 2024-01-16T15:09:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.635+0000 7f5a57a00640 1 -- start start 2024-01-16T15:09:06.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a50070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 0x7f5a50159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a50157110 con 0x7f5a5013e360 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a50157690 con 0x7f5a5013df80 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a57a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5013d410 con 0x7f5a500cf9a0 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.636+0000 7f5a55a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50798/0 (socket says 172.21.15.125:50798) 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 -- 172.21.15.125:0/2702702606 learned_addr learned my addr 172.21.15.125:0/2702702606 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55000640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a50070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a47c00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 0x7f5a50159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 -- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 msgr2=0x7f5a50070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a50070cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 -- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 msgr2=0x7f5a50159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 0x7f5a50159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 -- 172.21.15.125:0/2702702606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400094d0 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a47c00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 0x7f5a50159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55000640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a50070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a55a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800c5c0 tx=0x7f5a4800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.637+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48011800 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.638+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a48011980 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.638+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48011d30 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.638+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a50071770 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.638+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a50157320 con 0x7f5a5013e360 2024-01-16T15:09:06.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.639+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a50157820 con 0x7f5a5013e360 2024-01-16T15:09:06.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.640+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5a48022350 con 0x7f5a5013e360 2024-01-16T15:09:06.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.641+0000 7f5a4d600640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 0x7f5a2806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.641+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f5a480886b0 con 0x7f5a5013e360 2024-01-16T15:09:06.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.641+0000 7f5a55000640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 0x7f5a2806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.642+0000 7f5a55000640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 0x7f5a2806e350 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5a40036000 tx=0x7f5a4003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.646+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a48054f50 con 0x7f5a5013e360 2024-01-16T15:09:06.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.874+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969116", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a500cf500 con 0x7f5a2806be90 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.888+0000 7f5a4d600640 1 -- 172.21.15.125:0/2702702606 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f5a500cf500 con 0x7f5a2806be90 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:06.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:06.866 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.890+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 msgr2=0x7f5a2806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.890+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 0x7f5a2806e350 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5a40036000 tx=0x7f5a4003c040 comp rx=0 tx=0).stop 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.890+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 msgr2=0x7f5a50159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.890+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800c5c0 tx=0x7f5a4800ca80 comp rx=0 tx=0).stop 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 shutdown_connections 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a2806be90 0x7f5a2806e350 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a5013e360 0x7f5a50159aa0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a5013df80 0x7f5a50159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 --2- 172.21.15.125:0/2702702606 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a500cf9a0 0x7f5a50070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 >> 172.21.15.125:0/2702702606 conn(0x7f5a5006a800 msgr2=0x7f5a5015c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 shutdown_connections 2024-01-16T15:09:06.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:06.891+0000 7f5a57a00640 1 -- 172.21.15.125:0/2702702606 wait complete. 2024-01-16T15:09:06.889 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:06.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969117 2024-01-16T15:09:07.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.125+0000 7fd487200640 1 Processor -- start 2024-01-16T15:09:07.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.125+0000 7fd487200640 1 -- start start 2024-01-16T15:09:07.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.126+0000 7fd487200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd4800adf20 0x7fd480070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.126+0000 7fd487200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48015f4e0 con 0x7fd48015bf10 2024-01-16T15:09:07.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.126+0000 7fd487200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd480071130 con 0x7fd4800ccb50 2024-01-16T15:09:07.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.126+0000 7fd487200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48015f970 con 0x7fd4800adf20 2024-01-16T15:09:07.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.127+0000 7fd485400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd48015bf10 0x7fd4800cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49572/0 (socket says 172.21.15.125:49572) 2024-01-16T15:09:07.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.127+0000 7fd485400640 1 -- 172.21.15.125:0/3616487001 learned_addr learned my addr 172.21.15.125:0/3616487001 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.127+0000 7fd485e00640 1 --2- 172.21.15.125:0/3616487001 >> v2:172.21.15.137:3300/0 conn(0x7fd4800adf20 0x7fd480070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.127+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539565022 0 0) 0x7fd48015f4e0 con 0x7fd48015bf10 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.127+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd460003520 con 0x7fd48015bf10 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2631231161 0 0) 0x7fd460003520 con 0x7fd48015bf10 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 >> v2:172.21.15.137:3300/0 conn(0x7fd4800adf20 msgr2=0x7fd480070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 --2- 172.21.15.125:0/3616487001 >> v2:172.21.15.137:3300/0 conn(0x7fd4800adf20 0x7fd480070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 >> v1:172.21.15.137:6789/0 conn(0x7fd4800ccb50 legacy=0x7fd480143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48015c390 con 0x7fd48015bf10 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd468001670 con 0x7fd48015bf10 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 >> v1:172.21.15.125:6789/0 conn(0x7fd48015bf10 legacy=0x7fd4800cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460003970 0x7fd460003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460004390 0x7fd460004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 0x7fd46000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48015f970 con 0x7fd460005100 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46000c050 con 0x7fd460003970 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46000c210 con 0x7fd460004390 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd486800640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 0x7fd46000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.128+0000 7fd485e00640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460003970 0x7fd460003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd485400640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460004390 0x7fd460004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd486800640 1 -- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460004390 msgr2=0x7fd460004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd486800640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460004390 0x7fd460004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd486800640 1 -- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460003970 msgr2=0x7fd460003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd486800640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460003970 0x7fd460003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.129+0000 7fd486800640 1 -- 172.21.15.125:0/3616487001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd460003520 con 0x7fd460005100 2024-01-16T15:09:07.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.130+0000 7fd486800640 1 -- 172.21.15.125:0/3616487001 reap_dead start 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.130+0000 7fd486800640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 0x7fd46000b980 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd478001e80 tx=0x7fd4600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4577472c202d00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.130+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478003b70 con 0x7fd460005100 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478003cd0 con 0x7fd460005100 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd484a00640 1 -- 172.21.15.125:0/3616487001 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478005310 con 0x7fd460005100 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 -- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 msgr2=0x7fd46000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 0x7fd46000b980 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd478001e80 tx=0x7fd4600057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 -- 172.21.15.125:0/3616487001 shutdown_connections 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 --2- 172.21.15.125:0/3616487001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd460005100 0x7fd46000b980 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 -- 172.21.15.125:0/3616487001 >> 172.21.15.125:0/3616487001 conn(0x7fd48006a7d0 msgr2=0x7fd48006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 -- 172.21.15.125:0/3616487001 shutdown_connections 2024-01-16T15:09:07.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.131+0000 7fd487200640 1 -- 172.21.15.125:0/3616487001 wait complete. 2024-01-16T15:09:07.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.133+0000 7fd487200640 1 Processor -- start 2024-01-16T15:09:07.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.133+0000 7fd487200640 1 -- start start 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460005100 0x7fd48013b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4800b37d0 0x7fd48015e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48013c040 con 0x7fd4800b3350 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd480159fe0 con 0x7fd4800b37d0 2024-01-16T15:09:07.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.134+0000 7fd487200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48013bb40 con 0x7fd460005100 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50812/0 (socket says 172.21.15.125:50812) 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 -- 172.21.15.125:0/2252068758 learned_addr learned my addr 172.21.15.125:0/2252068758 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd486800640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4800b37d0 0x7fd48015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485e00640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460005100 0x7fd48013b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 -- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460005100 msgr2=0x7fd48013b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460005100 0x7fd48013b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 -- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4800b37d0 msgr2=0x7fd48015e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4800b37d0 0x7fd48015e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 -- 172.21.15.125:0/2252068758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48013c250 con 0x7fd4800b3350 2024-01-16T15:09:07.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.135+0000 7fd485400640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd46800e730 tx=0x7fd46800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.136+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46800c6c0 con 0x7fd4800b3350 2024-01-16T15:09:07.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.136+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46800c840 con 0x7fd4800b3350 2024-01-16T15:09:07.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.136+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48013c530 con 0x7fd4800b3350 2024-01-16T15:09:07.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.136+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46800cbf0 con 0x7fd4800b3350 2024-01-16T15:09:07.114 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.136+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48013caf0 con 0x7fd4800b3350 2024-01-16T15:09:07.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.137+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd480067450 con 0x7fd4800b3350 2024-01-16T15:09:07.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.138+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd46801d460 con 0x7fd4800b3350 2024-01-16T15:09:07.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.139+0000 7fd47e000640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 0x7fd45806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.139+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fd468088920 con 0x7fd4800b3350 2024-01-16T15:09:07.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.139+0000 7fd485e00640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 0x7fd45806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.140+0000 7fd485e00640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 0x7fd45806e350 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd48015ab50 tx=0x7fd470008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.143+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd468054f40 con 0x7fd4800b3350 2024-01-16T15:09:07.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.372+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969117", "target": ["mon-mgr", ""]}) v1 -- 0x7fd48015f170 con 0x7fd45806be90 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.388+0000 7fd47e000640 1 -- 172.21.15.125:0/2252068758 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fd4700096d0 con 0x7fd45806be90 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:07.365 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:07.366 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.390+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 msgr2=0x7fd45806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.390+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 0x7fd45806e350 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd48015ab50 tx=0x7fd470008040 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.390+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 msgr2=0x7fd48015ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.390+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd46800e730 tx=0x7fd46800ebf0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 shutdown_connections 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4800b37d0 0x7fd48015e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd45806be90 0x7fd45806e350 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4800b3350 0x7fd48015ddd0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 --2- 172.21.15.125:0/2252068758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd460005100 0x7fd48013b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 >> 172.21.15.125:0/2252068758 conn(0x7fd48006a7d0 msgr2=0x7fd4801445f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 shutdown_connections 2024-01-16T15:09:07.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.391+0000 7fd487200640 1 -- 172.21.15.125:0/2252068758 wait complete. 2024-01-16T15:09:07.385 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:07.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969118 2024-01-16T15:09:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.619+0000 7f82e6c00640 1 Processor -- start 2024-01-16T15:09:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.619+0000 7f82e6c00640 1 -- start start 2024-01-16T15:09:07.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.620+0000 7f82e6c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82e0160f80 0x7f82e0161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.620+0000 7f82e6c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f82e00ce8f0 0x7f82e00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.620+0000 7f82e6c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e01618a0 con 0x7f82e00cfbe0 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.620+0000 7f82e6c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e00d0de0 con 0x7f82e0160f80 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.620+0000 7f82e6c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e00d10c0 con 0x7f82e00ce8f0 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82e0160f80 0x7f82e0161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82e0160f80 0x7f82e0161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50814/0 (socket says 172.21.15.125:50814) 2024-01-16T15:09:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 -- 172.21.15.125:0/4288678796 learned_addr learned my addr 172.21.15.125:0/4288678796 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82df400640 1 --2- 172.21.15.125:0/4288678796 >> v2:172.21.15.137:3300/0 conn(0x7f82e00ce8f0 0x7f82e00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 -- 172.21.15.125:0/4288678796 >> v2:172.21.15.137:3300/0 conn(0x7f82e00ce8f0 msgr2=0x7f82e00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 --2- 172.21.15.125:0/4288678796 >> v2:172.21.15.137:3300/0 conn(0x7f82e00ce8f0 0x7f82e00ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 -- 172.21.15.125:0/4288678796 >> v1:172.21.15.125:6789/0 conn(0x7f82e00cfbe0 legacy=0x7f82e00d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.621+0000 7f82dfe00640 1 -- 172.21.15.125:0/4288678796 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00d13a0 con 0x7f82e0160f80 2024-01-16T15:09:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.622+0000 7f82dfe00640 1 --2- 172.21.15.125:0/4288678796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e0160f80 0x7f82e0161360 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f82e00b5d10 tx=0x7f82d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e553a72c18bccf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.623+0000 7f82dea00640 1 -- 172.21.15.125:0/4288678796 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0030860 con 0x7f82e0160f80 2024-01-16T15:09:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.623+0000 7f82dea00640 1 -- 172.21.15.125:0/4288678796 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00309c0 con 0x7f82e0160f80 2024-01-16T15:09:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.623+0000 7f82dea00640 1 -- 172.21.15.125:0/4288678796 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0030d50 con 0x7f82e0160f80 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.623+0000 7f82e6c00640 1 -- 172.21.15.125:0/4288678796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e0160f80 msgr2=0x7f82e0161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.623+0000 7f82e6c00640 1 --2- 172.21.15.125:0/4288678796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e0160f80 0x7f82e0161360 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f82e00b5d10 tx=0x7f82d002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 -- 172.21.15.125:0/4288678796 shutdown_connections 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 --2- 172.21.15.125:0/4288678796 >> v2:172.21.15.137:3300/0 conn(0x7f82e00ce8f0 0x7f82e00ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 --2- 172.21.15.125:0/4288678796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e0160f80 0x7f82e0161360 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 -- 172.21.15.125:0/4288678796 >> 172.21.15.125:0/4288678796 conn(0x7f82e009c550 msgr2=0x7f82e00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 -- 172.21.15.125:0/4288678796 shutdown_connections 2024-01-16T15:09:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.624+0000 7f82e6c00640 1 -- 172.21.15.125:0/4288678796 wait complete. 2024-01-16T15:09:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.626+0000 7f82e6c00640 1 Processor -- start 2024-01-16T15:09:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.626+0000 7f82e6c00640 1 -- start start 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82e00ce8f0 0x7f82e0151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e0160f80 0x7f82e015f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00cad70 con 0x7f82e00cfbe0 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00ca480 con 0x7f82e0160f80 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e6c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e015fed0 con 0x7f82e00ce8f0 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82df400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82df400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50822/0 (socket says 172.21.15.125:50822) 2024-01-16T15:09:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82df400640 1 -- 172.21.15.125:0/66422003 learned_addr learned my addr 172.21.15.125:0/66422003 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82e4c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e0160f80 0x7f82e015f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.627+0000 7f82dfe00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82e00ce8f0 0x7f82e0151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 -- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82e00ce8f0 msgr2=0x7f82e0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82e00ce8f0 0x7f82e0151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 -- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e0160f80 msgr2=0x7f82e015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e0160f80 0x7f82e015f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 -- 172.21.15.125:0/66422003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00094d0 con 0x7f82e00cfbe0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82df400640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c7f0 tx=0x7f82c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8004220 con 0x7f82e00cfbe0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8004380 con 0x7f82e00cfbe0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8007840 con 0x7f82e00cfbe0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e00ca720 con 0x7f82e00cfbe0 2024-01-16T15:09:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.628+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e01522f0 con 0x7f82e00cfbe0 2024-01-16T15:09:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.630+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e00d2160 con 0x7f82e00cfbe0 2024-01-16T15:09:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.630+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f82c8007bb0 con 0x7f82e00cfbe0 2024-01-16T15:09:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.632+0000 7f82d7e00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 0x7f82b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.632+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f82c8088470 con 0x7f82e00cfbe0 2024-01-16T15:09:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.632+0000 7f82dfe00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 0x7f82b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.633+0000 7f82dfe00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 0x7f82b806e350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f82d0036000 tx=0x7f82d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.636+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82c8054d90 con 0x7f82e00cfbe0 2024-01-16T15:09:07.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.863+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969118", "target": ["mon-mgr", ""]}) v1 -- 0x7f82e00d1c40 con 0x7f82b806be90 2024-01-16T15:09:07.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.873+0000 7f82d7e00640 1 -- 172.21.15.125:0/66422003 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f82d0044470 con 0x7f82b806be90 2024-01-16T15:09:07.850 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:07.850 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:07.851 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 msgr2=0x7f82b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 0x7f82b806e350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f82d0036000 tx=0x7f82d0008030 comp rx=0 tx=0).stop 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 msgr2=0x7f82e00c77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c7f0 tx=0x7f82c800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 shutdown_connections 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e0160f80 0x7f82e015f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f82b806be90 0x7f82b806e350 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82e00cfbe0 0x7f82e00c77c0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 --2- 172.21.15.125:0/66422003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82e00ce8f0 0x7f82e0151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 >> 172.21.15.125:0/66422003 conn(0x7f82e009c550 msgr2=0x7f82e006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 shutdown_connections 2024-01-16T15:09:07.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:07.876+0000 7f82e6c00640 1 -- 172.21.15.125:0/66422003 wait complete. 2024-01-16T15:09:07.871 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:07.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969119 2024-01-16T15:09:08.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.099+0000 7f4339400640 1 Processor -- start 2024-01-16T15:09:08.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.099+0000 7f4339400640 1 -- start start 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.100+0000 7f4339400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f433413e360 0x7f43340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.100+0000 7f4339400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f433415f9c0 0x7f43340cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.100+0000 7f4339400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433415c5f0 con 0x7f433413df80 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.100+0000 7f4339400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433406c470 con 0x7f433415f9c0 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.100+0000 7f4339400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433406c750 con 0x7f433413e360 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f433415f9c0 0x7f43340cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f433415f9c0 0x7f43340cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50838/0 (socket says 172.21.15.125:50838) 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 -- 172.21.15.125:0/4166344023 learned_addr learned my addr 172.21.15.125:0/4166344023 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f432be00640 1 --2- 172.21.15.125:0/4166344023 >> v2:172.21.15.137:3300/0 conn(0x7f433413e360 0x7f43340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 -- 172.21.15.125:0/4166344023 >> v2:172.21.15.137:3300/0 conn(0x7f433413e360 msgr2=0x7f43340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 --2- 172.21.15.125:0/4166344023 >> v2:172.21.15.137:3300/0 conn(0x7f433413e360 0x7f43340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 -- 172.21.15.125:0/4166344023 >> v1:172.21.15.137:6789/0 conn(0x7f433413df80 legacy=0x7f433415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.101+0000 7f4333200640 1 -- 172.21.15.125:0/4166344023 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433406ca30 con 0x7f433415f9c0 2024-01-16T15:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.102+0000 7f4333200640 1 --2- 172.21.15.125:0/4166344023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433415f9c0 0x7f43340cf7d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f43340ccae0 tx=0x7f432c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ec277d067ac25b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4331e00640 1 -- 172.21.15.125:0/4166344023 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c02ca70 con 0x7f433415f9c0 2024-01-16T15:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4331e00640 1 -- 172.21.15.125:0/4166344023 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c02cbd0 con 0x7f433415f9c0 2024-01-16T15:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4331e00640 1 -- 172.21.15.125:0/4166344023 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c0343f0 con 0x7f433415f9c0 2024-01-16T15:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 -- 172.21.15.125:0/4166344023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433415f9c0 msgr2=0x7f43340cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 --2- 172.21.15.125:0/4166344023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433415f9c0 0x7f43340cf7d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f43340ccae0 tx=0x7f432c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 -- 172.21.15.125:0/4166344023 shutdown_connections 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 --2- 172.21.15.125:0/4166344023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433415f9c0 0x7f43340cf7d0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 --2- 172.21.15.125:0/4166344023 >> v2:172.21.15.137:3300/0 conn(0x7f433413e360 0x7f43340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 -- 172.21.15.125:0/4166344023 >> 172.21.15.125:0/4166344023 conn(0x7f433406a800 msgr2=0x7f433406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 -- 172.21.15.125:0/4166344023 shutdown_connections 2024-01-16T15:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.103+0000 7f4339400640 1 -- 172.21.15.125:0/4166344023 wait complete. 2024-01-16T15:09:08.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.105+0000 7f4339400640 1 Processor -- start 2024-01-16T15:09:08.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.105+0000 7f4339400640 1 -- start start 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f433413e360 0x7f43340b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 0x7f43340b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340aaaa0 con 0x7f433413df80 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340a9a40 con 0x7f433415f9c0 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.106+0000 7f4339400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340a9ce0 con 0x7f433413e360 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50850/0 (socket says 172.21.15.125:50850) 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 -- 172.21.15.125:0/3832358034 learned_addr learned my addr 172.21.15.125:0/3832358034 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4333200640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 0x7f43340b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f432be00640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f433413e360 0x7f43340b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 -- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f433413e360 msgr2=0x7f43340b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f433413e360 0x7f43340b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 -- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 msgr2=0x7f43340b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 0x7f43340b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 -- 172.21.15.125:0/3832358034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c009b30 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4333200640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 0x7f43340b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.107+0000 7f4332800640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f432400b3e0 tx=0x7f432400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.108+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011830 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.108+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43240119b0 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.108+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340a9fc0 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.108+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011d60 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.108+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43340aa580 con 0x7f433413df80 2024-01-16T15:09:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.109+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43340675d0 con 0x7f433413df80 2024-01-16T15:09:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.110+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f432401a400 con 0x7f433413df80 2024-01-16T15:09:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.111+0000 7f432aa00640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 0x7f430c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.111+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f43240889b0 con 0x7f433413df80 2024-01-16T15:09:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.111+0000 7f432be00640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 0x7f430c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.112+0000 7f432be00640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 0x7f430c06e350 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.116+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4324054fd0 con 0x7f433413df80 2024-01-16T15:09:08.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.344+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969119", "target": ["mon-mgr", ""]}) v1 -- 0x7f433406cff0 con 0x7f430c06be90 2024-01-16T15:09:08.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.359+0000 7f432aa00640 1 -- 172.21.15.125:0/3832358034 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f431c009540 con 0x7f430c06be90 2024-01-16T15:09:08.336 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:08.337 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:08.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 msgr2=0x7f430c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 0x7f430c06e350 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 msgr2=0x7f433413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f432400b3e0 tx=0x7f432400b8a0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 shutdown_connections 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f433415f9c0 0x7f43340b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f433413e360 0x7f43340b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f430c06be90 0x7f430c06e350 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 --2- 172.21.15.125:0/3832358034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f433413df80 0x7f433413d940 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 >> 172.21.15.125:0/3832358034 conn(0x7f433406a800 msgr2=0x7f433415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 shutdown_connections 2024-01-16T15:09:08.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.362+0000 7f4339400640 1 -- 172.21.15.125:0/3832358034 wait complete. 2024-01-16T15:09:08.358 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:08.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone969120 2024-01-16T15:09:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.587+0000 7f9d89200640 1 Processor -- start 2024-01-16T15:09:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.587+0000 7f9d89200640 1 -- start start 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d89200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9d84144c60 0x7f9d840cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d89200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d84071a20 con 0x7f9d840cf780 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d89200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8406eef0 con 0x7f9d8406a3e0 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d89200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d840ccfd0 con 0x7f9d84144c60 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d81e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9d84144c60 0x7f9d840cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d81e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9d84144c60 0x7f9d840cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50866/0 (socket says 172.21.15.125:50866) 2024-01-16T15:09:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.589+0000 7f9d81e00640 1 -- 172.21.15.125:0/3523865902 learned_addr learned my addr 172.21.15.125:0/3523865902 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.590+0000 7f9d81e00640 1 -- 172.21.15.125:0/3523865902 >> v1:172.21.15.137:6789/0 conn(0x7f9d8406a3e0 legacy=0x7f9d84071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.590+0000 7f9d81e00640 1 -- 172.21.15.125:0/3523865902 >> v1:172.21.15.125:6789/0 conn(0x7f9d840cf780 legacy=0x7f9d8406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.590+0000 7f9d81e00640 1 -- 172.21.15.125:0/3523865902 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d840cd2b0 con 0x7f9d84144c60 2024-01-16T15:09:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.591+0000 7f9d81e00640 1 --2- 172.21.15.125:0/3523865902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d84144c60 0x7f9d840cc940 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009870 tx=0x7f9d7402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0e32bce011c010 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.592+0000 7f9d80a00640 1 -- 172.21.15.125:0/3523865902 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7402f840 con 0x7f9d84144c60 2024-01-16T15:09:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.592+0000 7f9d80a00640 1 -- 172.21.15.125:0/3523865902 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7402f9a0 con 0x7f9d84144c60 2024-01-16T15:09:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.592+0000 7f9d80a00640 1 -- 172.21.15.125:0/3523865902 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7402fd30 con 0x7f9d84144c60 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.592+0000 7f9d89200640 1 -- 172.21.15.125:0/3523865902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d84144c60 msgr2=0x7f9d840cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.592+0000 7f9d89200640 1 --2- 172.21.15.125:0/3523865902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d84144c60 0x7f9d840cc940 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009870 tx=0x7f9d7402ba20 comp rx=0 tx=0).stop 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.593+0000 7f9d89200640 1 -- 172.21.15.125:0/3523865902 shutdown_connections 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.593+0000 7f9d89200640 1 --2- 172.21.15.125:0/3523865902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d84144c60 0x7f9d840cc940 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.593+0000 7f9d89200640 1 -- 172.21.15.125:0/3523865902 >> 172.21.15.125:0/3523865902 conn(0x7f9d840cfef0 msgr2=0x7f9d840d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.593+0000 7f9d89200640 1 -- 172.21.15.125:0/3523865902 shutdown_connections 2024-01-16T15:09:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.593+0000 7f9d89200640 1 -- 172.21.15.125:0/3523865902 wait complete. 2024-01-16T15:09:08.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.594+0000 7f9d89200640 1 Processor -- start 2024-01-16T15:09:08.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.595+0000 7f9d89200640 1 -- start start 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.595+0000 7f9d89200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.595+0000 7f9d89200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d840cf780 0x7f9d8415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d89200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d84144c60 0x7f9d8415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d89200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d840cbdf0 con 0x7f9d8406a3e0 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d89200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84133680 con 0x7f9d840cf780 2024-01-16T15:09:08.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d89200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d840cb830 con 0x7f9d84144c60 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50882/0 (socket says 172.21.15.125:50882) 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 -- 172.21.15.125:0/350289434 learned_addr learned my addr 172.21.15.125:0/350289434 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 -- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d84144c60 msgr2=0x7f9d8415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d84144c60 0x7f9d8415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 -- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d840cf780 msgr2=0x7f9d8415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d840cf780 0x7f9d8415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 -- 172.21.15.125:0/350289434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740094d0 con 0x7f9d8406a3e0 2024-01-16T15:09:08.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.596+0000 7f9d81e00640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009870 tx=0x7f9d74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.597+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74037d10 con 0x7f9d8406a3e0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.597+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74037e70 con 0x7f9d8406a3e0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.597+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74041600 con 0x7f9d8406a3e0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.597+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d840cbb10 con 0x7f9d8406a3e0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.597+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8415af70 con 0x7f9d8406a3e0 2024-01-16T15:09:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.598+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d84130b20 con 0x7f9d8406a3e0 2024-01-16T15:09:08.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.599+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9d74041760 con 0x7f9d8406a3e0 2024-01-16T15:09:08.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.600+0000 7f9d7a000640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 0x7f9d5c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.600+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f9d7403c070 con 0x7f9d8406a3e0 2024-01-16T15:09:08.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.600+0000 7f9d81400640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 0x7f9d5c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.601+0000 7f9d81400640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 0x7f9d5c06e350 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c000f60 tx=0x7f9d6c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.604+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d84130b20 con 0x7f9d8406a3e0 2024-01-16T15:09:08.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.830+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone969120", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d840cfbc0 con 0x7f9d5c06be90 2024-01-16T15:09:08.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.843+0000 7f9d7a000640 1 -- 172.21.15.125:0/350289434 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f9d6c0012b0 con 0x7f9d5c06be90 2024-01-16T15:09:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:08.820 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:09:08.820 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:09:08.820 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 msgr2=0x7f9d5c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 0x7f9d5c06e350 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c000f60 tx=0x7f9d6c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 msgr2=0x7f9d840a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009870 tx=0x7f9d74004730 comp rx=0 tx=0).stop 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 shutdown_connections 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d84144c60 0x7f9d8415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d840cf780 0x7f9d8415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9d5c06be90 0x7f9d5c06e350 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 --2- 172.21.15.125:0/350289434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d8406a3e0 0x7f9d840a84c0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 >> 172.21.15.125:0/350289434 conn(0x7f9d840cfef0 msgr2=0x7f9d840d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 shutdown_connections 2024-01-16T15:09:08.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:08.846+0000 7f9d89200640 1 -- 172.21.15.125:0/350289434 wait complete. 2024-01-16T15:09:08.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:09:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.075+0000 7f21a7200640 1 Processor -- start 2024-01-16T15:09:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.075+0000 7f21a7200640 1 -- start start 2024-01-16T15:09:09.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 0x7f21a0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a7200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f21a00ccc10 0x7f21a006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a7200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a0070e50 con 0x7f21a00717d0 2024-01-16T15:09:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a7200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00723c0 con 0x7f21a00ccc10 2024-01-16T15:09:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0144020 con 0x7f21a0143c40 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f21a00ccc10 0x7f21a006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f21a00ccc10 0x7f21a006d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50890/0 (socket says 172.21.15.125:50890) 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 -- 172.21.15.125:0/761245942 learned_addr learned my addr 172.21.15.125:0/761245942 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a5e00640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 0x7f21a0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 -- 172.21.15.125:0/761245942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 msgr2=0x7f21a0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 0x7f21a0071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 -- 172.21.15.125:0/761245942 >> v1:172.21.15.125:6789/0 conn(0x7f21a00717d0 legacy=0x7f21a00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.077+0000 7f21a6800640 1 -- 172.21.15.125:0/761245942 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0071e00 con 0x7f21a00ccc10 2024-01-16T15:09:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.078+0000 7f21a5e00640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 0x7f21a0071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.079+0000 7f21a6800640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a00ccc10 0x7f21a006d160 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2198009e50 tx=0x7f219802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37172bc9c81bc233 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.079+0000 7f21a4a00640 1 -- 172.21.15.125:0/761245942 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219802ca80 con 0x7f21a00ccc10 2024-01-16T15:09:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.079+0000 7f21a4a00640 1 -- 172.21.15.125:0/761245942 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219802cbe0 con 0x7f21a00ccc10 2024-01-16T15:09:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.079+0000 7f21a4a00640 1 -- 172.21.15.125:0/761245942 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21980343f0 con 0x7f21a00ccc10 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 -- 172.21.15.125:0/761245942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a00ccc10 msgr2=0x7f21a006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a00ccc10 0x7f21a006d160 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2198009e50 tx=0x7f219802bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 -- 172.21.15.125:0/761245942 shutdown_connections 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a00ccc10 0x7f21a006d160 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 --2- 172.21.15.125:0/761245942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a0143c40 0x7f21a0071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 -- 172.21.15.125:0/761245942 >> 172.21.15.125:0/761245942 conn(0x7f21a006a800 msgr2=0x7f21a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 -- 172.21.15.125:0/761245942 shutdown_connections 2024-01-16T15:09:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.080+0000 7f21a7200640 1 -- 172.21.15.125:0/761245942 wait complete. 2024-01-16T15:09:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.081+0000 7f21a7200640 1 Processor -- start 2024-01-16T15:09:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.082+0000 7f21a7200640 1 -- start start 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21a00717d0 0x7f21a01f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a00ccc10 0x7f21a013c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00c3d70 con 0x7f21a0143c40 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00c3fc0 con 0x7f21a00717d0 2024-01-16T15:09:09.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00c42a0 con 0x7f21a00ccc10 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50906/0 (socket says 172.21.15.125:50906) 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 -- 172.21.15.125:0/2821709720 learned_addr learned my addr 172.21.15.125:0/2821709720 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 -- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a00ccc10 msgr2=0x7f21a013c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a00ccc10 0x7f21a013c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 -- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21a00717d0 msgr2=0x7f21a01f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21a00717d0 0x7f21a01f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.083+0000 7f21a6800640 1 -- 172.21.15.125:0/2821709720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198009b30 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f21a6800640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f2198009e50 tx=0x7f2198004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198009510 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219803e450 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219803e800 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a00c4580 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.084+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a00c4b40 con 0x7f21a0143c40 2024-01-16T15:09:09.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.085+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a00c4d60 con 0x7f21a0143c40 2024-01-16T15:09:09.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.086+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f219802cd50 con 0x7f21a0143c40 2024-01-16T15:09:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.087+0000 7f219e000640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 0x7f217806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.087+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f21980ab150 con 0x7f21a0143c40 2024-01-16T15:09:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.087+0000 7f21a5e00640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 0x7f217806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.088+0000 7f21a5e00640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 0x7f217806e350 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2190006fd0 tx=0x7f2190008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.091+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2198077ac0 con 0x7f21a0143c40 2024-01-16T15:09:09.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.305+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f21a006cc40 con 0x7f217806be90 2024-01-16T15:09:09.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.308+0000 7f219e000640 1 -- 172.21.15.125:0/2821709720 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f219000ae00 con 0x7f217806be90 2024-01-16T15:09:09.286 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:09:09.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.286 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:09:09.286 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:09.286 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:09:09.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 msgr2=0x7f217806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 0x7f217806e350 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2190006fd0 tx=0x7f2190008040 comp rx=0 tx=0).stop 2024-01-16T15:09:09.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 msgr2=0x7f21a013e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f2198009e50 tx=0x7f2198004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 shutdown_connections 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f217806be90 0x7f217806e350 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21a0143c40 0x7f21a013e130 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21a00ccc10 0x7f21a013c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 --2- 172.21.15.125:0/2821709720 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21a00717d0 0x7f21a01f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 >> 172.21.15.125:0/2821709720 conn(0x7f21a006a800 msgr2=0x7f21a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 shutdown_connections 2024-01-16T15:09:09.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.311+0000 7f21a7200640 1 -- 172.21.15.125:0/2821709720 wait complete. 2024-01-16T15:09:09.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:09:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.534+0000 7f0d5a000640 1 Processor -- start 2024-01-16T15:09:09.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.534+0000 7f0d5a000640 1 -- start start 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d5a000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d54143810 0x7f0d54071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d5a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d5415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d5a000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d540ae620 con 0x7f0d540ae170 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d5a000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d54143bf0 con 0x7f0d54143810 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d5a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5415e140 con 0x7f0d5415d770 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d58c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d54143810 0x7f0d54071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d53e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0d540ae170 0x7f0d5415e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49618/0 (socket says 172.21.15.125:49618) 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.536+0000 7f0d53e00640 1 -- 172.21.15.125:0/1867915518 learned_addr learned my addr 172.21.15.125:0/1867915518 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d58c00640 1 -- 172.21.15.125:0/1867915518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 msgr2=0x7f0d5415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d59600640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d5415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d58c00640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d5415dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d58c00640 1 -- 172.21.15.125:0/1867915518 >> v1:172.21.15.125:6789/0 conn(0x7f0d540ae170 legacy=0x7f0d5415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d53400640 1 -- 172.21.15.125:0/1867915518 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302950439 0 0) 0x7f0d540ae620 con 0x7f0d540ae170 2024-01-16T15:09:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.537+0000 7f0d58c00640 1 -- 172.21.15.125:0/1867915518 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5406c730 con 0x7f0d54143810 2024-01-16T15:09:09.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.538+0000 7f0d58c00640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d54143810 0x7f0d54071e40 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0d440097a0 tx=0x7f0d4402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e99a9a69a3bf0ba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d53400640 1 -- 172.21.15.125:0/1867915518 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4402f050 con 0x7f0d54143810 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d53400640 1 -- 172.21.15.125:0/1867915518 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44031700 con 0x7f0d54143810 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d53400640 1 -- 172.21.15.125:0/1867915518 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44031a90 con 0x7f0d54143810 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 -- 172.21.15.125:0/1867915518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d54143810 msgr2=0x7f0d54071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d54143810 0x7f0d54071e40 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0d440097a0 tx=0x7f0d4402b7a0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 -- 172.21.15.125:0/1867915518 shutdown_connections 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d5415dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 --2- 172.21.15.125:0/1867915518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d54143810 0x7f0d54071e40 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 -- 172.21.15.125:0/1867915518 >> 172.21.15.125:0/1867915518 conn(0x7f0d5406a7d0 msgr2=0x7f0d5406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 -- 172.21.15.125:0/1867915518 shutdown_connections 2024-01-16T15:09:09.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.539+0000 7f0d5a000640 1 -- 172.21.15.125:0/1867915518 wait complete. 2024-01-16T15:09:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.541+0000 7f0d5a000640 1 Processor -- start 2024-01-16T15:09:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.541+0000 7f0d5a000640 1 -- start start 2024-01-16T15:09:09.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d54143810 0x7f0d5412ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d540cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54145430 con 0x7f0d540ae170 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54145710 con 0x7f0d54143810 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d5a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d541459f0 con 0x7f0d5415d770 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d58c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d58c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50916/0 (socket says 172.21.15.125:50916) 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.542+0000 7f0d58c00640 1 -- 172.21.15.125:0/2735215509 learned_addr learned my addr 172.21.15.125:0/2735215509 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d53e00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d54143810 0x7f0d5412ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 -- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 msgr2=0x7f0d540cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d540cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 -- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d54143810 msgr2=0x7f0d5412ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d54143810 0x7f0d5412ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 -- 172.21.15.125:0/2735215509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44009400 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d58c00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0d44000c00 tx=0x7f0d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44004230 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44004390 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d54145cd0 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.543+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44033bd0 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.544+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d541450e0 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.545+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54067450 con 0x7f0d540ae170 2024-01-16T15:09:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.545+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0d44033d30 con 0x7f0d540ae170 2024-01-16T15:09:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.546+0000 7f0d50c00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 0x7f0d2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.546+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f0d440aa9d0 con 0x7f0d540ae170 2024-01-16T15:09:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.546+0000 7f0d53e00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 0x7f0d2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.547+0000 7f0d53e00640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 0x7f0d2c06e350 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c007fd0 tx=0x7f0d3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.550+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d44077030 con 0x7f0d540ae170 2024-01-16T15:09:09.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.752+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d5406ccd0 con 0x7f0d2c06be90 2024-01-16T15:09:09.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.777+0000 7f0d50c00640 1 -- 172.21.15.125:0/2735215509 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1029 (secure 0 0 0) 0x7f0d3c001c70 con 0x7f0d2c06be90 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96915" 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969119" 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969111" 2024-01-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969116" 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96914" 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969114" 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96916" 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969115" 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969120" 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969118" 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96918" 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969113" 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96919" 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96917" 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969112" 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969110" 2024-01-16T15:09:09.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96911" 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96912" 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone969117" 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone96913" 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:09.759 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:09:09.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 msgr2=0x7f0d2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 0x7f0d2c06e350 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c007fd0 tx=0x7f0d3c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:09.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 msgr2=0x7f0d541f20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0d44000c00 tx=0x7f0d440046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 shutdown_connections 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5415d770 0x7f0d540cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d54143810 0x7f0d5412ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0d2c06be90 0x7f0d2c06e350 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 --2- 172.21.15.125:0/2735215509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d540ae170 0x7f0d541f20f0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 >> 172.21.15.125:0/2735215509 conn(0x7f0d5406a7d0 msgr2=0x7f0d5415fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 shutdown_connections 2024-01-16T15:09:09.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.780+0000 7f0d5a000640 1 -- 172.21.15.125:0/2735215509 wait complete. 2024-01-16T15:09:09.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96915 2024-01-16T15:09:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.997+0000 7f88bfe00640 1 Processor -- start 2024-01-16T15:09:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.997+0000 7f88bfe00640 1 -- start start 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.998+0000 7f88bfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f88c00cc500 0x7f88c00cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.998+0000 7f88bfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c006d160 0x7f88c006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.998+0000 7f88bfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c0070e50 con 0x7f88c0143c40 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.998+0000 7f88bfe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c00723c0 con 0x7f88c00cc500 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.998+0000 7f88bfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00cced0 con 0x7f88c006d160 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f88c00cc500 0x7f88c00cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f88c00cc500 0x7f88c00cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50920/0 (socket says 172.21.15.125:50920) 2024-01-16T15:09:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 -- 172.21.15.125:0/410258767 learned_addr learned my addr 172.21.15.125:0/410258767 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88bf400640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c006d160 0x7f88c006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 -- 172.21.15.125:0/410258767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c006d160 msgr2=0x7f88c006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c006d160 0x7f88c006d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 -- 172.21.15.125:0/410258767 >> v1:172.21.15.137:6789/0 conn(0x7f88c0143c40 legacy=0x7f88c0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:09.999+0000 7f88b5e00640 1 -- 172.21.15.125:0/410258767 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0144020 con 0x7f88c00cc500 2024-01-16T15:09:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.000+0000 7f88b5e00640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c00cc500 0x7f88c00cc990 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f88c00b44a0 tx=0x7f88a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aea4075a6b56e8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.001+0000 7f88be000640 1 -- 172.21.15.125:0/410258767 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a802e070 con 0x7f88c00cc500 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.001+0000 7f88be000640 1 -- 172.21.15.125:0/410258767 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a80315a0 con 0x7f88c00cc500 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.001+0000 7f88be000640 1 -- 172.21.15.125:0/410258767 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8031930 con 0x7f88c00cc500 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.001+0000 7f88bfe00640 1 -- 172.21.15.125:0/410258767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c00cc500 msgr2=0x7f88c00cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.001+0000 7f88bfe00640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c00cc500 0x7f88c00cc990 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f88c00b44a0 tx=0x7f88a802b840 comp rx=0 tx=0).stop 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 -- 172.21.15.125:0/410258767 shutdown_connections 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c006d160 0x7f88c006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 --2- 172.21.15.125:0/410258767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c00cc500 0x7f88c00cc990 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 -- 172.21.15.125:0/410258767 >> 172.21.15.125:0/410258767 conn(0x7f88c006a800 msgr2=0x7f88c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 -- 172.21.15.125:0/410258767 shutdown_connections 2024-01-16T15:09:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.002+0000 7f88bfe00640 1 -- 172.21.15.125:0/410258767 wait complete. 2024-01-16T15:09:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.003+0000 7f88bfe00640 1 Processor -- start 2024-01-16T15:09:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.004+0000 7f88bfe00640 1 -- start start 2024-01-16T15:09:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.004+0000 7f88bfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.004+0000 7f88bfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c00cc500 0x7f88c013e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88c0143c40 0x7f88c013ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c014a000 con 0x7f88c006d160 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c014a2e0 con 0x7f88c0143c40 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c014a5c0 con 0x7f88c00cc500 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50922/0 (socket says 172.21.15.125:50922) 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 -- 172.21.15.125:0/2131890189 learned_addr learned my addr 172.21.15.125:0/2131890189 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 -- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c00cc500 msgr2=0x7f88c013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c00cc500 0x7f88c013e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 -- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88c0143c40 msgr2=0x7f88c013ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bf400640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88c0143c40 0x7f88c013ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88c0143c40 0x7f88c013ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.005+0000 7f88bea00640 1 -- 172.21.15.125:0/2131890189 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80094d0 con 0x7f88c006d160 2024-01-16T15:09:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88bea00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f88b000ba50 tx=0x7f88b000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000cbe0 con 0x7f88c006d160 2024-01-16T15:09:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b000cd80 con 0x7f88c006d160 2024-01-16T15:09:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c014a8a0 con 0x7f88c006d160 2024-01-16T15:09:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0014680 con 0x7f88c006d160 2024-01-16T15:09:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.006+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c0149bd0 con 0x7f88c006d160 2024-01-16T15:09:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.007+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c00ccd40 con 0x7f88c006d160 2024-01-16T15:09:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.008+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f88b00147e0 con 0x7f88c006d160 2024-01-16T15:09:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.009+0000 7f88b7400640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 0x7f889806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.009+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f88b0088670 con 0x7f88c006d160 2024-01-16T15:09:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.009+0000 7f88b5e00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 0x7f889806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.010+0000 7f88b5e00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 0x7f889806e350 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f88a8036000 tx=0x7f88a8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.013+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88b0054d80 con 0x7f88c006d160 2024-01-16T15:09:10.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:10.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:10.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.232+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96915", "target": ["mon-mgr", ""]}) v1 -- 0x7f88c006c960 con 0x7f889806be90 2024-01-16T15:09:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.259+0000 7f88b7400640 1 -- 172.21.15.125:0/2131890189 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f88a8031ae0 con 0x7f889806be90 2024-01-16T15:09:10.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 msgr2=0x7f889806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 0x7f889806e350 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f88a8036000 tx=0x7f88a8030000 comp rx=0 tx=0).stop 2024-01-16T15:09:10.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 msgr2=0x7f88c0149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f88b000ba50 tx=0x7f88b000bf10 comp rx=0 tx=0).stop 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 shutdown_connections 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88c0143c40 0x7f88c013ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88c00cc500 0x7f88c013e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f889806be90 0x7f889806e350 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 --2- 172.21.15.125:0/2131890189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88c006d160 0x7f88c0149690 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 >> 172.21.15.125:0/2131890189 conn(0x7f88c006a800 msgr2=0x7f88c00aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 shutdown_connections 2024-01-16T15:09:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.262+0000 7f88bfe00640 1 -- 172.21.15.125:0/2131890189 wait complete. 2024-01-16T15:09:10.241 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.264+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.241 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.264+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.241 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.264+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.241 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.264+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.241 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.264+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969119 2024-01-16T15:09:10.260 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.282+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.260 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.282+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.260 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.282+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.260 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.282+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.260 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:10.282+0000 7f00fd800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.502+0000 7faeb0200640 1 Processor -- start 2024-01-16T15:09:10.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.502+0000 7faeb0200640 1 -- start start 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeb0200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 0x7faea80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeb0200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea815e3b0 0x7faea80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeb0200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea80cfc30 con 0x7faea80cf7d0 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeb0200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea80b1ee0 con 0x7faea815e3b0 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeb0200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea80b2780 con 0x7faea8143ae0 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea815e3b0 0x7faea80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea815e3b0 0x7faea80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50974/0 (socket says 172.21.15.125:50974) 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 -- 172.21.15.125:0/2824050290 learned_addr learned my addr 172.21.15.125:0/2824050290 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faead800640 1 --2- 172.21.15.125:0/2824050290 >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 0x7faea80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 -- 172.21.15.125:0/2824050290 >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 msgr2=0x7faea80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 --2- 172.21.15.125:0/2824050290 >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 0x7faea80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.504+0000 7faeae200640 1 -- 172.21.15.125:0/2824050290 >> v1:172.21.15.137:6789/0 conn(0x7faea80cf7d0 legacy=0x7faea815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.505+0000 7faeae200640 1 -- 172.21.15.125:0/2824050290 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea815e990 con 0x7faea815e3b0 2024-01-16T15:09:10.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.505+0000 7faead800640 1 --2- 172.21.15.125:0/2824050290 >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 0x7faea80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.506+0000 7faeae200640 1 --2- 172.21.15.125:0/2824050290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea80702c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7faea80cfea0 tx=0x7faea002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7133f2d8ce4afa17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.506+0000 7faea7e00640 1 -- 172.21.15.125:0/2824050290 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea002ca80 con 0x7faea815e3b0 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.506+0000 7faea7e00640 1 -- 172.21.15.125:0/2824050290 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea002cbe0 con 0x7faea815e3b0 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.506+0000 7faea7e00640 1 -- 172.21.15.125:0/2824050290 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea00343f0 con 0x7faea815e3b0 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 -- 172.21.15.125:0/2824050290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 msgr2=0x7faea80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 --2- 172.21.15.125:0/2824050290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea80702c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7faea80cfea0 tx=0x7faea002bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 -- 172.21.15.125:0/2824050290 shutdown_connections 2024-01-16T15:09:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 --2- 172.21.15.125:0/2824050290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea80702c0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 --2- 172.21.15.125:0/2824050290 >> v2:172.21.15.137:3300/0 conn(0x7faea8143ae0 0x7faea80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 -- 172.21.15.125:0/2824050290 >> 172.21.15.125:0/2824050290 conn(0x7faea806a800 msgr2=0x7faea806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 -- 172.21.15.125:0/2824050290 shutdown_connections 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.507+0000 7faeb0200640 1 -- 172.21.15.125:0/2824050290 wait complete. 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.508+0000 7faeb0200640 1 Processor -- start 2024-01-16T15:09:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.509+0000 7faeb0200640 1 -- start start 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.509+0000 7faeb0200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faea80cf7d0 0x7faea81f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.509+0000 7faeb0200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea8143ae0 0x7faea80ab970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.509+0000 7faeb0200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeb0200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea80abeb0 con 0x7faea815e3b0 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeb0200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea81f11d0 con 0x7faea8143ae0 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeb0200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea81f14b0 con 0x7faea80cf7d0 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50986/0 (socket says 172.21.15.125:50986) 2024-01-16T15:09:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 -- 172.21.15.125:0/1582255410 learned_addr learned my addr 172.21.15.125:0/1582255410 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeace00640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea8143ae0 0x7faea80ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faead800640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faea80cf7d0 0x7faea81f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 -- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faea80cf7d0 msgr2=0x7faea81f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faea80cf7d0 0x7faea81f0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 -- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea8143ae0 msgr2=0x7faea80ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea8143ae0 0x7faea80ab970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.510+0000 7faeae200640 1 -- 172.21.15.125:0/1582255410 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0009b30 con 0x7faea815e3b0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faeae200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7faea0036040 tx=0x7faea0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0009510 con 0x7faea815e3b0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea003e450 con 0x7faea815e3b0 2024-01-16T15:09:10.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea003e800 con 0x7faea815e3b0 2024-01-16T15:09:10.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea81f1790 con 0x7faea815e3b0 2024-01-16T15:09:10.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.511+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea81314e0 con 0x7faea815e3b0 2024-01-16T15:09:10.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.512+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea80ad2c0 con 0x7faea815e3b0 2024-01-16T15:09:10.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.513+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7faea002cd50 con 0x7faea815e3b0 2024-01-16T15:09:10.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.514+0000 7faea5600640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 0x7fae8006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.514+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7faea00ab1b0 con 0x7faea815e3b0 2024-01-16T15:09:10.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.514+0000 7faead800640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 0x7fae8006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.515+0000 7faead800640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 0x7fae8006e350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faea815d6a0 tx=0x7fae98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.518+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faea80ad2c0 con 0x7faea815e3b0 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.723+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969119", "target": ["mon-mgr", ""]}) v1 -- 0x7faea80b2410 con 0x7fae8006be90 2024-01-16T15:09:10.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.765+0000 7faea5600640 1 -- 172.21.15.125:0/1582255410 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fae9800c040 con 0x7fae8006be90 2024-01-16T15:09:10.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 msgr2=0x7fae8006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 0x7fae8006e350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faea815d6a0 tx=0x7fae98008040 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 msgr2=0x7faea8130fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7faea0036040 tx=0x7faea0004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 shutdown_connections 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae8006be90 0x7fae8006e350 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea815e3b0 0x7faea8130fa0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea8143ae0 0x7faea80ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 --2- 172.21.15.125:0/1582255410 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faea80cf7d0 0x7faea81f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 >> 172.21.15.125:0/1582255410 conn(0x7faea806a800 msgr2=0x7faea815da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 shutdown_connections 2024-01-16T15:09:10.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.768+0000 7faeb0200640 1 -- 172.21.15.125:0/1582255410 wait complete. 2024-01-16T15:09:10.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969111 2024-01-16T15:09:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.985+0000 7fcbc1000640 1 Processor -- start 2024-01-16T15:09:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.985+0000 7fcbc1000640 1 -- start start 2024-01-16T15:09:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.986+0000 7fcbc1000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcbbc150760 0x7fcbbc15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.986+0000 7fcbc1000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc15dc70 con 0x7fcbbc072230 2024-01-16T15:09:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.986+0000 7fcbc1000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc150ba0 con 0x7fcbbc15d840 2024-01-16T15:09:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.986+0000 7fcbc1000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc072610 con 0x7fcbbc150760 2024-01-16T15:09:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb9e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcbbc072230 0x7fcbbc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49624/0 (socket says 172.21.15.125:49624) 2024-01-16T15:09:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb9e00640 1 -- 172.21.15.125:0/406320782 learned_addr learned my addr 172.21.15.125:0/406320782 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578814761 0 0) 0x7fcbbc15dc70 con 0x7fcbbc072230 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb9400640 1 --2- 172.21.15.125:0/406320782 >> v2:172.21.15.137:3300/0 conn(0x7fcbbc150760 0x7fcbbc15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c005730 con 0x7fcbbc072230 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223414656 0 0) 0x7fcb9c005730 con 0x7fcbbc072230 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 >> v2:172.21.15.137:3300/0 conn(0x7fcbbc150760 msgr2=0x7fcbbc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 --2- 172.21.15.125:0/406320782 >> v2:172.21.15.137:3300/0 conn(0x7fcbbc150760 0x7fcbbc15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.987+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 >> v1:172.21.15.137:6789/0 conn(0x7fcbbc15d840 legacy=0x7fcbbc0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc071190 con 0x7fcbbc072230 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcbac001670 con 0x7fcbbc072230 2024-01-16T15:09:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 >> v1:172.21.15.125:6789/0 conn(0x7fcbbc072230 legacy=0x7fcbbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c005b80 0x7fcb9c005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 0x7fcb9c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c007390 0x7fcb9c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc072610 con 0x7fcb9c0069d0 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c00e2c0 con 0x7fcb9c005b80 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c00e480 con 0x7fcb9c007390 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 0x7fcb9c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 -- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c007390 msgr2=0x7fcb9c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c007390 0x7fcb9c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 -- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c005b80 msgr2=0x7fcb9c005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c005b80 0x7fcb9c005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbb9400640 1 -- 172.21.15.125:0/406320782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c005730 con 0x7fcb9c0069d0 2024-01-16T15:09:10.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.988+0000 7fcbba800640 1 -- 172.21.15.125:0/406320782 reap_dead start 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.989+0000 7fcbb9400640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 0x7fcb9c006e50 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fcba400ba00 tx=0x7fcb9c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd156be1de7a357 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4032620 con 0x7fcb9c0069d0 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4032780 con 0x7fcb9c0069d0 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbb8a00640 1 -- 172.21.15.125:0/406320782 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4032b10 con 0x7fcb9c0069d0 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 -- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 msgr2=0x7fcb9c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 0x7fcb9c006e50 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fcba400ba00 tx=0x7fcb9c007a00 comp rx=0 tx=0).stop 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 -- 172.21.15.125:0/406320782 shutdown_connections 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 --2- 172.21.15.125:0/406320782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb9c0069d0 0x7fcb9c006e50 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 -- 172.21.15.125:0/406320782 >> 172.21.15.125:0/406320782 conn(0x7fcbbc06a800 msgr2=0x7fcbbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.990+0000 7fcbc1000640 1 -- 172.21.15.125:0/406320782 shutdown_connections 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.991+0000 7fcbc1000640 1 -- 172.21.15.125:0/406320782 wait complete. 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.992+0000 7fcbc1000640 1 Processor -- start 2024-01-16T15:09:10.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.992+0000 7fcbc1000640 1 -- start start 2024-01-16T15:09:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 0x7fcbbc1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbbc14e610 0x7fcbbc1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f4240 con 0x7fcbbc14ea90 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f3870 con 0x7fcb9c0069d0 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbc1000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f7330 con 0x7fcbbc14e610 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbba800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.993+0000 7fcbba800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50994/0 (socket says 172.21.15.125:50994) 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 -- 172.21.15.125:0/3443099561 learned_addr learned my addr 172.21.15.125:0/3443099561 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbb9e00640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 0x7fcbbc1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 -- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbbc14e610 msgr2=0x7fcbbc1f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbbc14e610 0x7fcbbc1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 -- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 msgr2=0x7fcbbc1f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 0x7fcbbc1f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 -- 172.21.15.125:0/3443099561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc1f7610 con 0x7fcbbc14ea90 2024-01-16T15:09:10.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbb9e00640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 0x7fcbbc1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbba800640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc06e190 tx=0x7fcbb400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4007070 con 0x7fcbbc14ea90 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb40071f0 con 0x7fcbbc14ea90 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.994+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc1f3500 con 0x7fcbbc14ea90 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.995+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40075a0 con 0x7fcbbc14ea90 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.995+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc1f6db0 con 0x7fcbbc14ea90 2024-01-16T15:09:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.996+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc1f39e0 con 0x7fcbbc14ea90 2024-01-16T15:09:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.997+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 29) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcbb4007700 con 0x7fcbbc14ea90 2024-01-16T15:09:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.997+0000 7fcbb2000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 0x7fcb9406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.998+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcbb4087c70 con 0x7fcbbc14ea90 2024-01-16T15:09:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.998+0000 7fcbb9e00640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 0x7fcb9406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:10.998+0000 7fcbb9e00640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 0x7fcb9406e350 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcbac001eb0 tx=0x7fcbac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.002+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbb40542d0 con 0x7fcbbc14ea90 2024-01-16T15:09:11.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.219+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969111", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbbc071a30 con 0x7fcb9406be90 2024-01-16T15:09:11.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.250+0000 7fcbb2000640 1 -- 172.21.15.125:0/3443099561 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcbac00d7f0 con 0x7fcb9406be90 2024-01-16T15:09:11.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 msgr2=0x7fcb9406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 0x7fcb9406e350 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcbac001eb0 tx=0x7fcbac009040 comp rx=0 tx=0).stop 2024-01-16T15:09:11.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 msgr2=0x7fcbbc1e44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc06e190 tx=0x7fcbb400c980 comp rx=0 tx=0).stop 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 shutdown_connections 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcb9406be90 0x7fcb9406e350 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbbc14ea90 0x7fcbbc1e44f0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbbc14e610 0x7fcbbc1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 --2- 172.21.15.125:0/3443099561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0069d0 0x7fcbbc1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 >> 172.21.15.125:0/3443099561 conn(0x7fcbbc06a800 msgr2=0x7fcbbc06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 shutdown_connections 2024-01-16T15:09:11.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.254+0000 7fcbc1000640 1 -- 172.21.15.125:0/3443099561 wait complete. 2024-01-16T15:09:11.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969116 2024-01-16T15:09:11.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.475+0000 7fe7cfc00640 1 Processor -- start 2024-01-16T15:09:11.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.475+0000 7fe7cfc00640 1 -- start start 2024-01-16T15:09:11.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 0x7fe7c815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7c813e360 0x7fe7c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c815f9c0 0x7fe7c80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c806c470 con 0x7fe7c813e360 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c806c750 con 0x7fe7c813df80 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7cfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c806ca30 con 0x7fe7c815f9c0 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7c7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7c813e360 0x7fe7c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7c7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7c813e360 0x7fe7c80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51004/0 (socket says 172.21.15.125:51004) 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.477+0000 7fe7c7e00640 1 -- 172.21.15.125:0/4031963977 learned_addr learned my addr 172.21.15.125:0/4031963977 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7cd200640 1 --2- 172.21.15.125:0/4031963977 >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 0x7fe7c815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7cdc00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c815f9c0 0x7fe7c80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7c7e00640 1 -- 172.21.15.125:0/4031963977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c815f9c0 msgr2=0x7fe7c80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c815f9c0 0x7fe7c80cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7c7e00640 1 -- 172.21.15.125:0/4031963977 >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 msgr2=0x7fe7c815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/4031963977 >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 0x7fe7c815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7c7e00640 1 -- 172.21.15.125:0/4031963977 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c806cd10 con 0x7fe7c813e360 2024-01-16T15:09:11.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.478+0000 7fe7cd200640 1 --2- 172.21.15.125:0/4031963977 >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 0x7fe7c815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:11.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.479+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813e360 0x7fe7c80cf290 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8072770 tx=0x7fe7b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c704afb7ed494fd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.479+0000 7fe7c7400640 1 -- 172.21.15.125:0/4031963977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0031870 con 0x7fe7c813e360 2024-01-16T15:09:11.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.479+0000 7fe7c7400640 1 -- 172.21.15.125:0/4031963977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b00319d0 con 0x7fe7c813e360 2024-01-16T15:09:11.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.479+0000 7fe7c7400640 1 -- 172.21.15.125:0/4031963977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0031d60 con 0x7fe7c813e360 2024-01-16T15:09:11.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 -- 172.21.15.125:0/4031963977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813e360 msgr2=0x7fe7c80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813e360 0x7fe7c80cf290 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8072770 tx=0x7fe7b002b910 comp rx=0 tx=0).stop 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 -- 172.21.15.125:0/4031963977 shutdown_connections 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c815f9c0 0x7fe7c80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/4031963977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813e360 0x7fe7c80cf290 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/4031963977 >> v2:172.21.15.137:3300/0 conn(0x7fe7c813df80 0x7fe7c815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 -- 172.21.15.125:0/4031963977 >> 172.21.15.125:0/4031963977 conn(0x7fe7c806a800 msgr2=0x7fe7c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 -- 172.21.15.125:0/4031963977 shutdown_connections 2024-01-16T15:09:11.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.480+0000 7fe7cfc00640 1 -- 172.21.15.125:0/4031963977 wait complete. 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.482+0000 7fe7cfc00640 1 Processor -- start 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.482+0000 7fe7cfc00640 1 -- start start 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c813e360 0x7fe7c815a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7c815f9c0 0x7fe7c81efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8153ce0 con 0x7fe7c813df80 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c80aff10 con 0x7fe7c815f9c0 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c80af390 con 0x7fe7c813e360 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cd200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cd200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51006/0 (socket says 172.21.15.125:51006) 2024-01-16T15:09:11.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.483+0000 7fe7cd200640 1 -- 172.21.15.125:0/2843403158 learned_addr learned my addr 172.21.15.125:0/2843403158 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c813e360 0x7fe7c815a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cdc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7c815f9c0 0x7fe7c81efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 -- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c813e360 msgr2=0x7fe7c815a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c813e360 0x7fe7c815a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 -- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7c815f9c0 msgr2=0x7fe7c81efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7c815f9c0 0x7fe7c81efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 -- 172.21.15.125:0/2843403158 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00094d0 con 0x7fe7c813df80 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cd200640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c820 tx=0x7fe7b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8007a30 con 0x7fe7c813df80 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b8007b90 con 0x7fe7c813df80 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.484+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c80af670 con 0x7fe7c813df80 2024-01-16T15:09:11.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.485+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800ce10 con 0x7fe7c813df80 2024-01-16T15:09:11.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.485+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c80afc30 con 0x7fe7c813df80 2024-01-16T15:09:11.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.486+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c80675d0 con 0x7fe7c813df80 2024-01-16T15:09:11.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.487+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fe7b80113f0 con 0x7fe7c813df80 2024-01-16T15:09:11.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.488+0000 7fe7c4c00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 0x7fe7a006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.488+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fe7b8088720 con 0x7fe7c813df80 2024-01-16T15:09:11.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.489+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 0x7fe7a006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.489+0000 7fe7c7e00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 0x7fe7a006e470 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036000 tx=0x7fe7b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.493+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7b8054fa0 con 0x7fe7c813df80 2024-01-16T15:09:11.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.716+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969116", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c81408d0 con 0x7fe7a006bfb0 2024-01-16T15:09:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.748+0000 7fe7c4c00640 1 -- 172.21.15.125:0/2843403158 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe7b003a420 con 0x7fe7a006bfb0 2024-01-16T15:09:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 msgr2=0x7fe7a006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 0x7fe7a006e470 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036000 tx=0x7fe7b003c040 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 msgr2=0x7fe7c81f3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c820 tx=0x7fe7b800cce0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 shutdown_connections 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7c815f9c0 0x7fe7c81efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7c813e360 0x7fe7c815a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe7a006bfb0 0x7fe7a006e470 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 --2- 172.21.15.125:0/2843403158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7c813df80 0x7fe7c81f3980 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 >> 172.21.15.125:0/2843403158 conn(0x7fe7c806a800 msgr2=0x7fe7c813fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.751+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 shutdown_connections 2024-01-16T15:09:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.752+0000 7fe7cfc00640 1 -- 172.21.15.125:0/2843403158 wait complete. 2024-01-16T15:09:11.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96914 2024-01-16T15:09:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.972+0000 7ffa43e00640 1 Processor -- start 2024-01-16T15:09:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.972+0000 7ffa43e00640 1 -- start start 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa3c150760 0x7ffa3c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c15d300 0x7ffa3c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 0x7ffa3c0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3c15dc80 con 0x7ffa3c150760 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3c071190 con 0x7ffa3c0b2bd0 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa43e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c071470 con 0x7ffa3c15d300 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa41400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa3c150760 0x7ffa3c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa41400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa3c150760 0x7ffa3c072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51016/0 (socket says 172.21.15.125:51016) 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.974+0000 7ffa41400640 1 -- 172.21.15.125:0/1772427532 learned_addr learned my addr 172.21.15.125:0/1772427532 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41e00640 1 --2- 172.21.15.125:0/1772427532 >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 0x7ffa3c0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa40a00640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c15d300 0x7ffa3c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41400640 1 -- 172.21.15.125:0/1772427532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c15d300 msgr2=0x7ffa3c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41400640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c15d300 0x7ffa3c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41400640 1 -- 172.21.15.125:0/1772427532 >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 msgr2=0x7ffa3c0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41400640 1 --2- 172.21.15.125:0/1772427532 >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 0x7ffa3c0b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41400640 1 -- 172.21.15.125:0/1772427532 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c071750 con 0x7ffa3c150760 2024-01-16T15:09:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.975+0000 7ffa41e00640 1 --2- 172.21.15.125:0/1772427532 >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 0x7ffa3c0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:11.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.976+0000 7ffa41400640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c150760 0x7ffa3c072230 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0cbc10 tx=0x7ffa2c02cfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87054d7cbe6df69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.976+0000 7ffa3be00640 1 -- 172.21.15.125:0/1772427532 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c02db80 con 0x7ffa3c150760 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.976+0000 7ffa3be00640 1 -- 172.21.15.125:0/1772427532 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c02dce0 con 0x7ffa3c150760 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.976+0000 7ffa3be00640 1 -- 172.21.15.125:0/1772427532 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c035610 con 0x7ffa3c150760 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 -- 172.21.15.125:0/1772427532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c150760 msgr2=0x7ffa3c072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c150760 0x7ffa3c072230 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0cbc10 tx=0x7ffa2c02cfe0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 -- 172.21.15.125:0/1772427532 shutdown_connections 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 --2- 172.21.15.125:0/1772427532 >> v2:172.21.15.137:3300/0 conn(0x7ffa3c0b2bd0 0x7ffa3c0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c15d300 0x7ffa3c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 --2- 172.21.15.125:0/1772427532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c150760 0x7ffa3c072230 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 -- 172.21.15.125:0/1772427532 >> 172.21.15.125:0/1772427532 conn(0x7ffa3c06a800 msgr2=0x7ffa3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:11.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 -- 172.21.15.125:0/1772427532 shutdown_connections 2024-01-16T15:09:11.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.977+0000 7ffa43e00640 1 -- 172.21.15.125:0/1772427532 wait complete. 2024-01-16T15:09:11.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.979+0000 7ffa43e00640 1 Processor -- start 2024-01-16T15:09:11.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.979+0000 7ffa43e00640 1 -- start start 2024-01-16T15:09:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa3c0b2bd0 0x7ffa3c1e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c150760 0x7ffa3c15b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c15ba40 con 0x7ffa3c15d300 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c1e4230 con 0x7ffa3c0b2bd0 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa43e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c1e4510 con 0x7ffa3c150760 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51030/0 (socket says 172.21.15.125:51030) 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 -- 172.21.15.125:0/2303281495 learned_addr learned my addr 172.21.15.125:0/2303281495 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa40a00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c150760 0x7ffa3c15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41400640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa3c0b2bd0 0x7ffa3c1e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 -- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c150760 msgr2=0x7ffa3c15b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c150760 0x7ffa3c15b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 -- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa3c0b2bd0 msgr2=0x7ffa3c1e3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa3c0b2bd0 0x7ffa3c1e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.981+0000 7ffa41e00640 1 -- 172.21.15.125:0/2303281495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c009b30 con 0x7ffa3c15d300 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa41e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c7f0 tx=0x7ffa3400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34007820 con 0x7ffa3c15d300 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa34007980 con 0x7ffa3c15d300 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c1e47f0 con 0x7ffa3c15d300 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34007d30 con 0x7ffa3c15d300 2024-01-16T15:09:11.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.982+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c1e4db0 con 0x7ffa3c15d300 2024-01-16T15:09:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.983+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3c0675d0 con 0x7ffa3c15d300 2024-01-16T15:09:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.984+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ffa340113f0 con 0x7ffa3c15d300 2024-01-16T15:09:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.986+0000 7ffa39600640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 0x7ffa1406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.986+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7ffa340888c0 con 0x7ffa3c15d300 2024-01-16T15:09:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.986+0000 7ffa41400640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 0x7ffa1406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.987+0000 7ffa41400640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 0x7ffa1406e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c036000 tx=0x7ffa2c00d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:11.990+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa34055240 con 0x7ffa3c15d300 2024-01-16T15:09:12.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.206+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96914", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa3c150b40 con 0x7ffa1406bfb0 2024-01-16T15:09:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.239+0000 7ffa39600640 1 -- 172.21.15.125:0/2303281495 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffa2c040470 con 0x7ffa1406bfb0 2024-01-16T15:09:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 msgr2=0x7ffa1406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 0x7ffa1406e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c036000 tx=0x7ffa2c00d040 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 msgr2=0x7ffa3c14f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c7f0 tx=0x7ffa3400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 shutdown_connections 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ffa1406bfb0 0x7ffa1406e470 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa3c15d300 0x7ffa3c14f300 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa3c150760 0x7ffa3c15b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 --2- 172.21.15.125:0/2303281495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa3c0b2bd0 0x7ffa3c1e3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 >> 172.21.15.125:0/2303281495 conn(0x7ffa3c06a800 msgr2=0x7ffa3c0cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 shutdown_connections 2024-01-16T15:09:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.242+0000 7ffa43e00640 1 -- 172.21.15.125:0/2303281495 wait complete. 2024-01-16T15:09:12.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:09:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.459+0000 7f9cc7a00640 1 Processor -- start 2024-01-16T15:09:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.459+0000 7f9cc7a00640 1 -- start start 2024-01-16T15:09:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cc7a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9cc013df80 0x7f9cc015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cc7a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc013e360 0x7f9cc00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cc7a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc006c470 con 0x7f9cc015f9c0 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cc7a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc006c750 con 0x7f9cc013e360 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cc7a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc006ca30 con 0x7f9cc013df80 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.460+0000 7f9cb7c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc013e360 0x7f9cc00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc013e360 0x7f9cc00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51044/0 (socket says 172.21.15.125:51044) 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3154285671 learned_addr learned my addr 172.21.15.125:0/3154285671 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cc5000640 1 --2- 172.21.15.125:0/3154285671 >> v2:172.21.15.137:3300/0 conn(0x7f9cc013df80 0x7f9cc015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3154285671 >> v2:172.21.15.137:3300/0 conn(0x7f9cc013df80 msgr2=0x7f9cc015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 --2- 172.21.15.125:0/3154285671 >> v2:172.21.15.137:3300/0 conn(0x7f9cc013df80 0x7f9cc015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3154285671 >> v1:172.21.15.125:6789/0 conn(0x7f9cc015f9c0 legacy=0x7f9cc01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.461+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3154285671 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc006cd10 con 0x7f9cc013e360 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.462+0000 7f9cb7c00640 1 --2- 172.21.15.125:0/3154285671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc00cf290 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0072770 tx=0x7f9ca802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6d878d4e0a6d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.462+0000 7f9cbfe00640 1 -- 172.21.15.125:0/3154285671 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8030860 con 0x7f9cc013e360 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.462+0000 7f9cbfe00640 1 -- 172.21.15.125:0/3154285671 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca80309c0 con 0x7f9cc013e360 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.462+0000 7f9cbfe00640 1 -- 172.21.15.125:0/3154285671 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8030d50 con 0x7f9cc013e360 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3154285671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 msgr2=0x7f9cc00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3154285671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc00cf290 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0072770 tx=0x7f9ca802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3154285671 shutdown_connections 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3154285671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc00cf290 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3154285671 >> v2:172.21.15.137:3300/0 conn(0x7f9cc013df80 0x7f9cc015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3154285671 >> 172.21.15.125:0/3154285671 conn(0x7f9cc006a800 msgr2=0x7f9cc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3154285671 shutdown_connections 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.463+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3154285671 wait complete. 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.465+0000 7f9cc7a00640 1 Processor -- start 2024-01-16T15:09:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.465+0000 7f9cc7a00640 1 -- start start 2024-01-16T15:09:12.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cc013df80 0x7f9cc01e6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 0x7f9cc0152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01e6880 con 0x7f9cc013e360 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01e5cb0 con 0x7f9cc013df80 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc7a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01e5f50 con 0x7f9cc015f9c0 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51052/0 (socket says 172.21.15.125:51052) 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3096444484 learned_addr learned my addr 172.21.15.125:0/3096444484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc5a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 0x7f9cc0152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 msgr2=0x7f9cc0152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 0x7f9cc0152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cc5000640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cc013df80 0x7f9cc01e6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.466+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cc013df80 msgr2=0x7f9cc01e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cb7c00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cc013df80 0x7f9cc01e6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cb7c00640 1 -- 172.21.15.125:0/3096444484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80094d0 con 0x7f9cc013e360 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cc5a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 0x7f9cc0152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cb7c00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9ca802b9e0 tx=0x7f9ca8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8030850 con 0x7f9cc013e360 2024-01-16T15:09:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca8043400 con 0x7f9cc013e360 2024-01-16T15:09:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc01e6230 con 0x7f9cc013e360 2024-01-16T15:09:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca80437b0 con 0x7f9cc013e360 2024-01-16T15:09:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.467+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc00b4250 con 0x7f9cc013e360 2024-01-16T15:09:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.468+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc01e69f0 con 0x7f9cc013e360 2024-01-16T15:09:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.469+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f9ca8043910 con 0x7f9cc013e360 2024-01-16T15:09:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.470+0000 7f9cbd600640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 0x7f9c9806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.471+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f9ca803e070 con 0x7f9cc013e360 2024-01-16T15:09:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.471+0000 7f9cc5000640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 0x7f9c9806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.472+0000 7f9cc5000640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 0x7f9c9806e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00b3b50 tx=0x7f9cb00083c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.475+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ca8078330 con 0x7f9cc013e360 2024-01-16T15:09:12.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.698+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cc01e60c0 con 0x7f9c9806bfb0 2024-01-16T15:09:12.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.722+0000 7f9cbd600640 1 -- 172.21.15.125:0/3096444484 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9cc01e60c0 con 0x7f9c9806bfb0 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:12.700 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:09:12.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 msgr2=0x7f9c9806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 0x7f9c9806e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00b3b50 tx=0x7f9cb00083c0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 msgr2=0x7f9cc0152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9ca802b9e0 tx=0x7f9ca8004730 comp rx=0 tx=0).stop 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 shutdown_connections 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cc015f9c0 0x7f9cc0152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9c9806bfb0 0x7f9c9806e470 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc013e360 0x7f9cc0152580 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 --2- 172.21.15.125:0/3096444484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cc013df80 0x7f9cc01e6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 >> 172.21.15.125:0/3096444484 conn(0x7f9cc006a800 msgr2=0x7f9cc00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 shutdown_connections 2024-01-16T15:09:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.725+0000 7f9cc7a00640 1 -- 172.21.15.125:0/3096444484 wait complete. 2024-01-16T15:09:12.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:09:12.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.958+0000 7f1c53200640 1 Processor -- start 2024-01-16T15:09:12.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.958+0000 7f1c53200640 1 -- start start 2024-01-16T15:09:12.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.959+0000 7f1c53200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c5415da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.959+0000 7f1c53200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1c540ce0c0 0x7f1c54070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.959+0000 7f1c53200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c540700f0 con 0x7f1c5415dfa0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.959+0000 7f1c53200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5415e3e0 con 0x7f1c540ce0c0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.959+0000 7f1c53200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540ce720 con 0x7f1c5413cda0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c49200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1c5415dfa0 0x7f1c540cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49650/0 (socket says 172.21.15.125:49650) 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c49200640 1 -- 172.21.15.125:0/2311025535 learned_addr learned my addr 172.21.15.125:0/2311025535 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c52800640 1 --2- 172.21.15.125:0/2311025535 >> v2:172.21.15.137:3300/0 conn(0x7f1c540ce0c0 0x7f1c54070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578622534 0 0) 0x7f1c540700f0 con 0x7f1c5415dfa0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51e00640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c5415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c34003520 con 0x7f1c5415dfa0 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922339181 0 0) 0x7f1c34003520 con 0x7f1c5415dfa0 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 msgr2=0x7f1c5415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c5415da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 >> v2:172.21.15.137:3300/0 conn(0x7f1c540ce0c0 msgr2=0x7f1c54070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 --2- 172.21.15.125:0/2311025535 >> v2:172.21.15.137:3300/0 conn(0x7f1c540ce0c0 0x7f1c54070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.960+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540cea00 con 0x7f1c5415dfa0 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1c3c001670 con 0x7f1c5415dfa0 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 >> v1:172.21.15.125:6789/0 conn(0x7f1c5415dfa0 legacy=0x7f1c540cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c34003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c34003fa0 0x7f1c340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c34004d00 0x7f1c34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5415e3e0 con 0x7f1c34003fa0 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540ce720 con 0x7f1c34004d00 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3400bb40 con 0x7f1c5413cda0 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c34003fa0 0x7f1c340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c51e00640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c34003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c52800640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c34004d00 0x7f1c34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 -- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 msgr2=0x7f1c34003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c5413cda0 0x7f1c34003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 -- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c34004d00 msgr2=0x7f1c34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c34004d00 0x7f1c34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c49200640 1 -- 172.21.15.125:0/2311025535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c34003520 con 0x7f1c34003fa0 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.961+0000 7f1c52800640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c34004d00 0x7f1c34005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.962+0000 7f1c49200640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c34003fa0 0x7f1c340047c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00c210 tx=0x7f1c340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf340220e245b56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c02cae0 con 0x7f1c34003fa0 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c02cc40 con 0x7f1c34003fa0 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c51400640 1 -- 172.21.15.125:0/2311025535 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c005af0 con 0x7f1c34003fa0 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 -- 172.21.15.125:0/2311025535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c34003fa0 msgr2=0x7f1c340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 --2- 172.21.15.125:0/2311025535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c34003fa0 0x7f1c340047c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00c210 tx=0x7f1c340057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c52800640 1 -- 172.21.15.125:0/2311025535 reap_dead start 2024-01-16T15:09:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 -- 172.21.15.125:0/2311025535 shutdown_connections 2024-01-16T15:09:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 -- 172.21.15.125:0/2311025535 >> 172.21.15.125:0/2311025535 conn(0x7f1c5406a3e0 msgr2=0x7f1c5406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 -- 172.21.15.125:0/2311025535 shutdown_connections 2024-01-16T15:09:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.963+0000 7f1c53200640 1 -- 172.21.15.125:0/2311025535 wait complete. 2024-01-16T15:09:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.965+0000 7f1c53200640 1 Processor -- start 2024-01-16T15:09:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.965+0000 7f1c53200640 1 -- start start 2024-01-16T15:09:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c34003fa0 0x7f1c54153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 0x7f1c540b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54133060 con 0x7f1c540ad8d0 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54132d80 con 0x7f1c540ad450 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c53200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540ac970 con 0x7f1c34003fa0 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c52800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c52800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51068/0 (socket says 172.21.15.125:51068) 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.966+0000 7f1c52800640 1 -- 172.21.15.125:0/4144174132 learned_addr learned my addr 172.21.15.125:0/4144174132 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c51e00640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c34003fa0 0x7f1c54153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c49200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 0x7f1c540b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 -- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c34003fa0 msgr2=0x7f1c54153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c34003fa0 0x7f1c54153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 -- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 msgr2=0x7f1c540b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 0x7f1c540b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 -- 172.21.15.125:0/4144174132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540acc50 con 0x7f1c540ad8d0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c49200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 0x7f1c540b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c52800640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406e600 tx=0x7f1c4c009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c007cf0 con 0x7f1c540ad8d0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c007e70 con 0x7f1c540ad8d0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c014620 con 0x7f1c540ad8d0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c540acf30 con 0x7f1c540ad8d0 2024-01-16T15:09:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.967+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5413d760 con 0x7f1c540ad8d0 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.969+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c54067450 con 0x7f1c540ad8d0 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.969+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1c4c01d3b0 con 0x7f1c540ad8d0 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.970+0000 7f1c4aa00640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 0x7f1c2c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.970+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1c4c0883b0 con 0x7f1c540ad8d0 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.971+0000 7f1c51e00640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 0x7f1c2c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.971+0000 7f1c51e00640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 0x7f1c2c06e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1c44006fd0 tx=0x7f1c44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:12.975+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c4c054cd0 con 0x7f1c540ad8d0 2024-01-16T15:09:13.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.185+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c54070620 con 0x7f1c2c06bfb0 2024-01-16T15:09:13.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.238+0000 7f1c4aa00640 1 -- 172.21.15.125:0/4144174132 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1c44004030 con 0x7f1c2c06bfb0 2024-01-16T15:09:13.215 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:13.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 msgr2=0x7f1c2c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 0x7f1c2c06e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1c44006fd0 tx=0x7f1c44008040 comp rx=0 tx=0).stop 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 msgr2=0x7f1c540b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406e600 tx=0x7f1c4c009d40 comp rx=0 tx=0).stop 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 shutdown_connections 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1c2c06bfb0 0x7f1c2c06e470 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c540ad8d0 0x7f1c540b3180 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c540ad450 0x7f1c540b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 --2- 172.21.15.125:0/4144174132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c34003fa0 0x7f1c54153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 >> 172.21.15.125:0/4144174132 conn(0x7f1c5406a3e0 msgr2=0x7f1c5406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:13.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 shutdown_connections 2024-01-16T15:09:13.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.241+0000 7f1c53200640 1 -- 172.21.15.125:0/4144174132 wait complete. 2024-01-16T15:09:13.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:09:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.470+0000 7f8ba8000640 1 Processor -- start 2024-01-16T15:09:13.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.470+0000 7f8ba8000640 1 -- start start 2024-01-16T15:09:13.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 0x7f8ba015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ba013e360 0x7f8ba00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba006c470 con 0x7f8ba013e360 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba006c750 con 0x7f8ba013df80 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.471+0000 7f8ba8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba006ca30 con 0x7f8ba015f9c0 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ba013e360 0x7f8ba00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ba013e360 0x7f8ba00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51076/0 (socket says 172.21.15.125:51076) 2024-01-16T15:09:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2222027935 learned_addr learned my addr 172.21.15.125:0/2222027935 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2222027935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 msgr2=0x7f8ba00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2222027935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2222027935 >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 msgr2=0x7f8ba015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2222027935 >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 0x7f8ba015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2222027935 >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 0x7f8ba015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2222027935 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba006cd10 con 0x7f8ba013e360 2024-01-16T15:09:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.472+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2222027935 >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 0x7f8ba015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.473+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2222027935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba00cf290 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0072770 tx=0x7f8b8802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51a99dd166feb8d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.474+0000 7f8b9fe00640 1 -- 172.21.15.125:0/2222027935 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031860 con 0x7f8ba013e360 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.474+0000 7f8b9fe00640 1 -- 172.21.15.125:0/2222027935 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b880319c0 con 0x7f8ba013e360 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.474+0000 7f8b9fe00640 1 -- 172.21.15.125:0/2222027935 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031d50 con 0x7f8ba013e360 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.474+0000 7f8ba8000640 1 -- 172.21.15.125:0/2222027935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 msgr2=0x7f8ba00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.474+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2222027935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba00cf290 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0072770 tx=0x7f8b8802b900 comp rx=0 tx=0).stop 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 -- 172.21.15.125:0/2222027935 shutdown_connections 2024-01-16T15:09:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2222027935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2222027935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba00cf290 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2222027935 >> v2:172.21.15.137:3300/0 conn(0x7f8ba013df80 0x7f8ba015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 -- 172.21.15.125:0/2222027935 >> 172.21.15.125:0/2222027935 conn(0x7f8ba006a800 msgr2=0x7f8ba006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:13.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 -- 172.21.15.125:0/2222027935 shutdown_connections 2024-01-16T15:09:13.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.475+0000 7f8ba8000640 1 -- 172.21.15.125:0/2222027935 wait complete. 2024-01-16T15:09:13.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.476+0000 7f8ba8000640 1 Processor -- start 2024-01-16T15:09:13.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.477+0000 7f8ba8000640 1 -- start start 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.477+0000 7f8ba8000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 0x7f8ba0153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.477+0000 7f8ba8000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.477+0000 7f8ba8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba013a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.477+0000 7f8ba8000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0153ec0 con 0x7f8ba013e360 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba8000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba013c200 con 0x7f8ba013df80 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba013ac20 con 0x7f8ba015f9c0 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51078/0 (socket says 172.21.15.125:51078) 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2520211313 learned_addr learned my addr 172.21.15.125:0/2520211313 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:13.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 0x7f8ba0153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 msgr2=0x7f8ba013a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba6000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba013a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba013a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 msgr2=0x7f8ba0153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 0x7f8ba0153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba4c00640 1 -- 172.21.15.125:0/2520211313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b880094d0 con 0x7f8ba013e360 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.478+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 0x7f8ba0153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8ba4c00640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8b88000c00 tx=0x7f8b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031830 con 0x7f8ba013e360 2024-01-16T15:09:13.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b880443f0 con 0x7f8ba013e360 2024-01-16T15:09:13.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880447a0 con 0x7f8ba013e360 2024-01-16T15:09:13.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba013af00 con 0x7f8ba013e360 2024-01-16T15:09:13.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.479+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba013b4c0 con 0x7f8ba013e360 2024-01-16T15:09:13.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.480+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba00675d0 con 0x7f8ba013e360 2024-01-16T15:09:13.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.481+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8b8804d450 con 0x7f8ba013e360 2024-01-16T15:09:13.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.482+0000 7f8b9d600640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 0x7f8b7806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.482+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f8b880aafd0 con 0x7f8ba013e360 2024-01-16T15:09:13.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.482+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 0x7f8b7806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.483+0000 7f8ba5600640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 0x7f8b7806e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006fd0 tx=0x7f8b90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.487+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b88077660 con 0x7f8ba013e360 2024-01-16T15:09:13.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.702+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ba01408d0 con 0x7f8b7806bfb0 2024-01-16T15:09:13.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.734+0000 7f8b9d600640 1 -- 172.21.15.125:0/2520211313 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8b9000c040 con 0x7f8b7806bfb0 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.736+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 msgr2=0x7f8b7806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.736+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 0x7f8b7806e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006fd0 tx=0x7f8b90008040 comp rx=0 tx=0).stop 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.736+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 msgr2=0x7f8ba013bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.736+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8b88000c00 tx=0x7f8b880046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 shutdown_connections 2024-01-16T15:09:13.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ba015f9c0 0x7f8ba013a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b7806bfb0 0x7f8b7806e470 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ba013e360 0x7f8ba013bcc0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 --2- 172.21.15.125:0/2520211313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ba013df80 0x7f8ba0153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 >> 172.21.15.125:0/2520211313 conn(0x7f8ba006a800 msgr2=0x7f8ba015aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 shutdown_connections 2024-01-16T15:09:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:13.737+0000 7f8ba8000640 1 -- 172.21.15.125:0/2520211313 wait complete. 2024-01-16T15:09:13.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969114 2024-01-16T15:09:13.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.019+0000 7fb13c600640 1 Processor -- start 2024-01-16T15:09:13.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.019+0000 7fb13c600640 1 -- start start 2024-01-16T15:09:13.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13c600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13c600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb13406c6b0 0x7fb1340cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13c600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13406cea0 con 0x7fb1340cd240 2024-01-16T15:09:13.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13c600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13406cbc0 con 0x7fb13406c6b0 2024-01-16T15:09:13.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13c600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13406f350 con 0x7fb13409a170 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb139200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb13a600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb13406c6b0 0x7fb1340cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb139200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56500/0 (socket says 172.21.15.125:56500) 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.021+0000 7fb139200640 1 -- 172.21.15.125:0/1621400467 learned_addr learned my addr 172.21.15.125:0/1621400467 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.022+0000 7fb139200640 1 -- 172.21.15.125:0/1621400467 >> v2:172.21.15.137:3300/0 conn(0x7fb13406c6b0 msgr2=0x7fb1340cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.022+0000 7fb139200640 1 --2- 172.21.15.125:0/1621400467 >> v2:172.21.15.137:3300/0 conn(0x7fb13406c6b0 0x7fb1340cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.022+0000 7fb139200640 1 -- 172.21.15.125:0/1621400467 >> v1:172.21.15.137:6789/0 conn(0x7fb1340cd240 legacy=0x7fb134099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.022+0000 7fb139200640 1 -- 172.21.15.125:0/1621400467 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13406f630 con 0x7fb13409a170 2024-01-16T15:09:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.023+0000 7fb139200640 1 --2- 172.21.15.125:0/1621400467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009870 tx=0x7fb11c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f085bba3f5c5b7d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.023+0000 7fb133e00640 1 -- 172.21.15.125:0/1621400467 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c030840 con 0x7fb13409a170 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.023+0000 7fb133e00640 1 -- 172.21.15.125:0/1621400467 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c0309a0 con 0x7fb13409a170 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.023+0000 7fb133e00640 1 -- 172.21.15.125:0/1621400467 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c030d30 con 0x7fb13409a170 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 -- 172.21.15.125:0/1621400467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 msgr2=0x7fb1340b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 --2- 172.21.15.125:0/1621400467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009870 tx=0x7fb11c02b980 comp rx=0 tx=0).stop 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 -- 172.21.15.125:0/1621400467 shutdown_connections 2024-01-16T15:09:14.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 --2- 172.21.15.125:0/1621400467 >> v2:172.21.15.137:3300/0 conn(0x7fb13406c6b0 0x7fb1340cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 --2- 172.21.15.125:0/1621400467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb1340b3820 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 -- 172.21.15.125:0/1621400467 >> 172.21.15.125:0/1621400467 conn(0x7fb1340b2d00 msgr2=0x7fb1340b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 -- 172.21.15.125:0/1621400467 shutdown_connections 2024-01-16T15:09:14.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.024+0000 7fb13c600640 1 -- 172.21.15.125:0/1621400467 wait complete. 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.026+0000 7fb13c600640 1 Processor -- start 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.026+0000 7fb13c600640 1 -- start start 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 0x7fb1340b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb13415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1340c6a30 con 0x7fb1340cd240 2024-01-16T15:09:14.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1341f3d70 con 0x7fb13406c6b0 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13c600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1341f4050 con 0x7fb13409a170 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51094/0 (socket says 172.21.15.125:51094) 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.027+0000 7fb13a600640 1 -- 172.21.15.125:0/2116291630 learned_addr learned my addr 172.21.15.125:0/2116291630 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb139200640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb13415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb139c00640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 0x7fb1340b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 -- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 msgr2=0x7fb13415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb13415d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 -- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 msgr2=0x7fb1340b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 0x7fb1340b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 -- 172.21.15.125:0/2116291630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0094d0 con 0x7fb1340cd240 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb139c00640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 0x7fb1340b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb13a600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00efc0 tx=0x7fb12c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.028+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c009ba0 con 0x7fb1340cd240 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.029+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c00f040 con 0x7fb1340cd240 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.029+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0146c0 con 0x7fb1340cd240 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.029+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1340c66c0 con 0x7fb1340cd240 2024-01-16T15:09:14.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.029+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1340cb2a0 con 0x7fb1340cd240 2024-01-16T15:09:14.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.030+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb134066760 con 0x7fb1340cd240 2024-01-16T15:09:14.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.031+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb12c014820 con 0x7fb1340cd240 2024-01-16T15:09:14.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.031+0000 7fb131600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 0x7fb10c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.032+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fb12c089370 con 0x7fb1340cd240 2024-01-16T15:09:14.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.032+0000 7fb139c00640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 0x7fb10c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.033+0000 7fb139c00640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 0x7fb10c06e470 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb13415fae0 tx=0x7fb124009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.036+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb12c0559a0 con 0x7fb1340cd240 2024-01-16T15:09:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.246+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969114", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1340c6540 con 0x7fb10c06bfb0 2024-01-16T15:09:14.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.264+0000 7fb131600640 1 -- 172.21.15.125:0/2116291630 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb124001cd0 con 0x7fb10c06bfb0 2024-01-16T15:09:14.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 msgr2=0x7fb10c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 0x7fb10c06e470 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb13415fae0 tx=0x7fb124009040 comp rx=0 tx=0).stop 2024-01-16T15:09:14.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 msgr2=0x7fb1341f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00efc0 tx=0x7fb12c00c490 comp rx=0 tx=0).stop 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 shutdown_connections 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.267+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb10c06bfb0 0x7fb10c06e470 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340cd240 0x7fb1341f3250 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb13409a170 0x7fb13415d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 --2- 172.21.15.125:0/2116291630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb13406c6b0 0x7fb1340b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 >> 172.21.15.125:0/2116291630 conn(0x7fb1340b2d00 msgr2=0x7fb134099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 shutdown_connections 2024-01-16T15:09:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.268+0000 7fb13c600640 1 -- 172.21.15.125:0/2116291630 wait complete. 2024-01-16T15:09:14.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96916 2024-01-16T15:09:14.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.474+0000 7fa57be00640 1 Processor -- start 2024-01-16T15:09:14.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.474+0000 7fa57be00640 1 -- start start 2024-01-16T15:09:14.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.475+0000 7fa57be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa57415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.475+0000 7fa57be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5740cf290 0x7fa5740cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.475+0000 7fa57be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa574070c40 con 0x7fa5740ae5a0 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.475+0000 7fa57be00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa574071450 con 0x7fa5740cf290 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.475+0000 7fa57be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cfc60 con 0x7fa5740b3ae0 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5740cf290 0x7fa5740cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5740cf290 0x7fa5740cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51104/0 (socket says 172.21.15.125:51104) 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 -- 172.21.15.125:0/2362567453 learned_addr learned my addr 172.21.15.125:0/2362567453 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa578a00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa57415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 -- 172.21.15.125:0/2362567453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 msgr2=0x7fa57415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa57415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 -- 172.21.15.125:0/2362567453 >> v1:172.21.15.137:6789/0 conn(0x7fa5740ae5a0 legacy=0x7fa57406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.476+0000 7fa579e00640 1 -- 172.21.15.125:0/2362567453 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57406c470 con 0x7fa5740cf290 2024-01-16T15:09:14.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.477+0000 7fa579e00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740cf290 0x7fa5740cf720 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009e50 tx=0x7fa56c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c385bf17f5786e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa573e00640 1 -- 172.21.15.125:0/2362567453 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c02cb00 con 0x7fa5740cf290 2024-01-16T15:09:14.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa573e00640 1 -- 172.21.15.125:0/2362567453 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c02cc60 con 0x7fa5740cf290 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa573e00640 1 -- 172.21.15.125:0/2362567453 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0343f0 con 0x7fa5740cf290 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 -- 172.21.15.125:0/2362567453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740cf290 msgr2=0x7fa5740cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740cf290 0x7fa5740cf720 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009e50 tx=0x7fa56c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 -- 172.21.15.125:0/2362567453 shutdown_connections 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740cf290 0x7fa5740cf720 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 --2- 172.21.15.125:0/2362567453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa57415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 -- 172.21.15.125:0/2362567453 >> 172.21.15.125:0/2362567453 conn(0x7fa57406a800 msgr2=0x7fa57406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 -- 172.21.15.125:0/2362567453 shutdown_connections 2024-01-16T15:09:14.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.478+0000 7fa57be00640 1 -- 172.21.15.125:0/2362567453 wait complete. 2024-01-16T15:09:14.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.481+0000 7fa57be00640 1 Processor -- start 2024-01-16T15:09:14.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.481+0000 7fa57be00640 1 -- start start 2024-01-16T15:09:14.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa5741f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5740cf290 0x7fa574159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5741537d0 con 0x7fa5740ae5a0 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa574156a00 con 0x7fa5740cf290 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa57be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5741f3dd0 con 0x7fa5740b3ae0 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa579400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa579400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51120/0 (socket says 172.21.15.125:51120) 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.483+0000 7fa579400640 1 -- 172.21.15.125:0/2571577721 learned_addr learned my addr 172.21.15.125:0/2571577721 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:14.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa578a00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa5741f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579e00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5740cf290 0x7fa574159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 -- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 msgr2=0x7fa5741f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa5741f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 -- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5740cf290 msgr2=0x7fa574159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5740cf290 0x7fa574159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 -- 172.21.15.125:0/2571577721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c009b30 con 0x7fa5740ae5a0 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.484+0000 7fa579400640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c430 tx=0x7fa56400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.486+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564011790 con 0x7fa5740ae5a0 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.486+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5640118f0 con 0x7fa5740ae5a0 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.486+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564011c60 con 0x7fa5740ae5a0 2024-01-16T15:09:14.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.486+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa574160070 con 0x7fa5740ae5a0 2024-01-16T15:09:14.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.488+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5741f6510 con 0x7fa5740ae5a0 2024-01-16T15:09:14.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.489+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5740675d0 con 0x7fa5740ae5a0 2024-01-16T15:09:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.491+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa56401c460 con 0x7fa5740ae5a0 2024-01-16T15:09:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.492+0000 7fa571600640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 0x7fa54c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.492+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7fa564087e10 con 0x7fa5740ae5a0 2024-01-16T15:09:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.494+0000 7fa578a00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 0x7fa54c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.495+0000 7fa578a00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 0x7fa54c06e470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cb650 tx=0x7fa55c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.499+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa564090060 con 0x7fa5740ae5a0 2024-01-16T15:09:14.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.704+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96916", "target": ["mon-mgr", ""]}) v1 -- 0x7fa574071170 con 0x7fa54c06bfb0 2024-01-16T15:09:14.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.744+0000 7fa571600640 1 -- 172.21.15.125:0/2571577721 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa55c00b200 con 0x7fa54c06bfb0 2024-01-16T15:09:14.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.746+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 msgr2=0x7fa54c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.746+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 0x7fa54c06e470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cb650 tx=0x7fa55c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:14.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.746+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 msgr2=0x7fa5741f3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.746+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c430 tx=0x7fa56400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 shutdown_connections 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa5740cf290 0x7fa574159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5740b3ae0 0x7fa5741f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa54c06bfb0 0x7fa54c06e470 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 --2- 172.21.15.125:0/2571577721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5740ae5a0 0x7fa5741f3890 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 >> 172.21.15.125:0/2571577721 conn(0x7fa57406a800 msgr2=0x7fa5740b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 shutdown_connections 2024-01-16T15:09:14.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:14.747+0000 7fa57be00640 1 -- 172.21.15.125:0/2571577721 wait complete. 2024-01-16T15:09:14.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969115 2024-01-16T15:09:14.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.022+0000 7fe4b0600640 1 Processor -- start 2024-01-16T15:09:14.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.022+0000 7fe4b0600640 1 -- start start 2024-01-16T15:09:15.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.024+0000 7fe4b0600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4a8160e10 0x7fe4a80d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.025+0000 7fe4b0600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80d4160 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.025+0000 7fe4b0600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80b2420 con 0x7fe4a80d3910 2024-01-16T15:09:15.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.025+0000 7fe4b0600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a80ceff0 con 0x7fe4a8160e10 2024-01-16T15:09:15.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.026+0000 7fe4ae600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4a8160e10 0x7fe4a80d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.026+0000 7fe4ae600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4a8160e10 0x7fe4a80d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51124/0 (socket says 172.21.15.125:51124) 2024-01-16T15:09:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.026+0000 7fe4ae600640 1 -- 172.21.15.125:0/2408411475 learned_addr learned my addr 172.21.15.125:0/2408411475 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.027+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799199788 0 0) 0x7fe4a80d4160 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.027+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe488003520 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.027+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345504868 0 0) 0x7fe488003520 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.027+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 >> v2:172.21.15.125:3300/0 conn(0x7fe4a8160e10 msgr2=0x7fe4a80d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.027+0000 7fe4a7e00640 1 --2- 172.21.15.125:0/2408411475 >> v2:172.21.15.125:3300/0 conn(0x7fe4a8160e10 0x7fe4a80d1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 >> v1:172.21.15.137:6789/0 conn(0x7fe4a80d3910 legacy=0x7fe4a80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80b2140 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fe498001670 con 0x7fe4a80cfcf0 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 >> v1:172.21.15.125:6789/0 conn(0x7fe4a80cfcf0 legacy=0x7fe4a8160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe488003970 0x7fe488003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 0x7fe488004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe488005180 0x7fe488014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80ceff0 con 0x7fe488003970 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488014930 con 0x7fe488005180 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe488014af0 con 0x7fe4880047c0 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4ae600640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe488005180 0x7fe488014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4ad200640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 0x7fe488004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.028+0000 7fe4adc00640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe488003970 0x7fe488003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ad200640 1 -- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe488005180 msgr2=0x7fe488014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ad200640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe488005180 0x7fe488014190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ad200640 1 -- 172.21.15.125:0/2408411475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe488003970 msgr2=0x7fe488003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ad200640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe488003970 0x7fe488003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ad200640 1 -- 172.21.15.125:0/2408411475 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80b2140 con 0x7fe4880047c0 2024-01-16T15:09:15.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.029+0000 7fe4ae600640 1 -- 172.21.15.125:0/2408411475 reap_dead start 2024-01-16T15:09:15.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.031+0000 7fe4ad200640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 0x7fe488004c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe490001750 tx=0x7fe4880057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4b3b98513e56594 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.032+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4900052d0 con 0x7fe4880047c0 2024-01-16T15:09:15.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.032+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49002e050 con 0x7fe4880047c0 2024-01-16T15:09:15.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.032+0000 7fe4a7e00640 1 -- 172.21.15.125:0/2408411475 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490003830 con 0x7fe4880047c0 2024-01-16T15:09:15.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.033+0000 7fe4b0600640 1 -- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 msgr2=0x7fe488004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.033+0000 7fe4b0600640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 0x7fe488004c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe490001750 tx=0x7fe4880057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.034+0000 7fe4b0600640 1 -- 172.21.15.125:0/2408411475 shutdown_connections 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.034+0000 7fe4b0600640 1 --2- 172.21.15.125:0/2408411475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4880047c0 0x7fe488004c40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.034+0000 7fe4b0600640 1 -- 172.21.15.125:0/2408411475 >> 172.21.15.125:0/2408411475 conn(0x7fe4a809c530 msgr2=0x7fe4a80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.035+0000 7fe4b0600640 1 -- 172.21.15.125:0/2408411475 shutdown_connections 2024-01-16T15:09:15.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.035+0000 7fe4b0600640 1 -- 172.21.15.125:0/2408411475 wait complete. 2024-01-16T15:09:15.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.037+0000 7fe4b0600640 1 Processor -- start 2024-01-16T15:09:15.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.038+0000 7fe4b0600640 1 -- start start 2024-01-16T15:09:15.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4880047c0 0x7fe4a815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4a80c7b00 0x7fe4a80b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80c8060 con 0x7fe4a80c7680 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80c8340 con 0x7fe4880047c0 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4b0600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80c8620 con 0x7fe4a80c7b00 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4ad200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4ad200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51134/0 (socket says 172.21.15.125:51134) 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4ad200640 1 -- 172.21.15.125:0/487646396 learned_addr learned my addr 172.21.15.125:0/487646396 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4ae600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4a80c7b00 0x7fe4a80b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.040+0000 7fe4adc00640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4880047c0 0x7fe4a815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 -- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4a80c7b00 msgr2=0x7fe4a80b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4a80c7b00 0x7fe4a80b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 -- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4880047c0 msgr2=0x7fe4a815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4880047c0 0x7fe4a815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 -- 172.21.15.125:0/487646396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80cb2e0 con 0x7fe4a80c7680 2024-01-16T15:09:15.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.041+0000 7fe4ad200640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fe490005040 tx=0x7fe490001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.042+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490007640 con 0x7fe4a80c7680 2024-01-16T15:09:15.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.043+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4900069e0 con 0x7fe4a80c7680 2024-01-16T15:09:15.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.043+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490006d70 con 0x7fe4a80c7680 2024-01-16T15:09:15.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.043+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a80caf00 con 0x7fe4a80c7680 2024-01-16T15:09:15.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.043+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a80c9550 con 0x7fe4a80c7680 2024-01-16T15:09:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.045+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80675d0 con 0x7fe4a80c7680 2024-01-16T15:09:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.047+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fe4900035f0 con 0x7fe4a80c7680 2024-01-16T15:09:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.048+0000 7fe4a5600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 0x7fe48006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.048+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7fe4900aa040 con 0x7fe4a80c7680 2024-01-16T15:09:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.050+0000 7fe4adc00640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 0x7fe48006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.052+0000 7fe4adc00640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 0x7fe48006e4c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe498000fb0 tx=0x7fe498009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.057+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe490076940 con 0x7fe4a80c7680 2024-01-16T15:09:15.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.355+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969115", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4a80c84b0 con 0x7fe48006c000 2024-01-16T15:09:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.382+0000 7fe4a5600640 1 -- 172.21.15.125:0/487646396 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe4a80c84b0 con 0x7fe48006c000 2024-01-16T15:09:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 msgr2=0x7fe48006e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 0x7fe48006e4c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe498000fb0 tx=0x7fe498009040 comp rx=0 tx=0).stop 2024-01-16T15:09:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 msgr2=0x7fe4a80b7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fe490005040 tx=0x7fe490001120 comp rx=0 tx=0).stop 2024-01-16T15:09:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 shutdown_connections 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4a80c7b00 0x7fe4a80b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe48006c000 0x7fe48006e4c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4a80c7680 0x7fe4a80b7d50 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 --2- 172.21.15.125:0/487646396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4880047c0 0x7fe4a815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.384+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 >> 172.21.15.125:0/487646396 conn(0x7fe4a809c530 msgr2=0x7fe4a80afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.385+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 shutdown_connections 2024-01-16T15:09:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.385+0000 7fe4b0600640 1 -- 172.21.15.125:0/487646396 wait complete. 2024-01-16T15:09:15.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969120 2024-01-16T15:09:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.587+0000 7f164e200640 1 Processor -- start 2024-01-16T15:09:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.587+0000 7f164e200640 1 -- start start 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f164e200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f164813df80 0x7f164815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f164e200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 0x7f16480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f164e200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164806c470 con 0x7f164815f9c0 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f164e200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164806c750 con 0x7f164813df80 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f164e200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164806ca30 con 0x7f164813e360 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f1647400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f164813df80 0x7f164815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f1647400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f164813df80 0x7f164815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51136/0 (socket says 172.21.15.125:51136) 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.589+0000 7f1647400640 1 -- 172.21.15.125:0/88674168 learned_addr learned my addr 172.21.15.125:0/88674168 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f1647400640 1 -- 172.21.15.125:0/88674168 >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 msgr2=0x7f16480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f163e800640 1 --2- 172.21.15.125:0/88674168 >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 0x7f16480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f1647400640 1 --2- 172.21.15.125:0/88674168 >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 0x7f16480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f1646a00640 1 -- 172.21.15.125:0/88674168 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395393730 0 0) 0x7f164806c470 con 0x7f164815f9c0 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f1647400640 1 -- 172.21.15.125:0/88674168 >> v1:172.21.15.125:6789/0 conn(0x7f164815f9c0 legacy=0x7f16481408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f1647400640 1 -- 172.21.15.125:0/88674168 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164806cd10 con 0x7f164813df80 2024-01-16T15:09:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.590+0000 7f163e800640 1 --2- 172.21.15.125:0/88674168 >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 0x7f16480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.591+0000 7f1647400640 1 --2- 172.21.15.125:0/88674168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164815bee0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f164806ded0 tx=0x7f163802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e182a6c70c92d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.591+0000 7f1646a00640 1 -- 172.21.15.125:0/88674168 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638030870 con 0x7f164813df80 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.591+0000 7f1646a00640 1 -- 172.21.15.125:0/88674168 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16380309d0 con 0x7f164813df80 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.591+0000 7f1646a00640 1 -- 172.21.15.125:0/88674168 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638030d60 con 0x7f164813df80 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 -- 172.21.15.125:0/88674168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 msgr2=0x7f164815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 --2- 172.21.15.125:0/88674168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164815bee0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f164806ded0 tx=0x7f163802b9b0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 -- 172.21.15.125:0/88674168 shutdown_connections 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 --2- 172.21.15.125:0/88674168 >> v2:172.21.15.137:3300/0 conn(0x7f164813e360 0x7f16480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 --2- 172.21.15.125:0/88674168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164815bee0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 -- 172.21.15.125:0/88674168 >> 172.21.15.125:0/88674168 conn(0x7f164806a800 msgr2=0x7f164806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 -- 172.21.15.125:0/88674168 shutdown_connections 2024-01-16T15:09:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.592+0000 7f164e200640 1 -- 172.21.15.125:0/88674168 wait complete. 2024-01-16T15:09:15.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.593+0000 7f164e200640 1 Processor -- start 2024-01-16T15:09:15.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.593+0000 7f164e200640 1 -- start start 2024-01-16T15:09:15.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f164813e360 0x7f164814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164815f9c0 0x7f164814f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1648153360 con 0x7f164813df80 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164813d650 con 0x7f164813e360 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.594+0000 7f164e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164813d930 con 0x7f164815f9c0 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51146/0 (socket says 172.21.15.125:51146) 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 -- 172.21.15.125:0/3229793711 learned_addr learned my addr 172.21.15.125:0/3229793711 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f163e800640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f164813e360 0x7f164814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 -- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164815f9c0 msgr2=0x7f164814f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647e00640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164815f9c0 0x7f164814f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164815f9c0 0x7f164814f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 -- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f164813e360 msgr2=0x7f164814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f164813e360 0x7f164814f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 -- 172.21.15.125:0/3229793711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16380094d0 con 0x7f164813df80 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.595+0000 7f1647400640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f1638004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.596+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638030840 con 0x7f164813df80 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.596+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1638039d90 con 0x7f164813df80 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.596+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638043630 con 0x7f164813df80 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.596+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164815c8f0 con 0x7f164813df80 2024-01-16T15:09:15.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.596+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164813dc10 con 0x7f164813df80 2024-01-16T15:09:15.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.597+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164815ce10 con 0x7f164813df80 2024-01-16T15:09:15.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.598+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f163802f3c0 con 0x7f164813df80 2024-01-16T15:09:15.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.598+0000 7f163fe00640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 0x7f162006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.599+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f16380aaac0 con 0x7f164813df80 2024-01-16T15:09:15.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.599+0000 7f163e800640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 0x7f162006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.599+0000 7f163e800640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 0x7f162006e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1648072020 tx=0x7f1630008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.603+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f164815ce10 con 0x7f164813df80 2024-01-16T15:09:15.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.811+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969120", "target": ["mon-mgr", ""]}) v1 -- 0x7f16480ce850 con 0x7f162006bfb0 2024-01-16T15:09:15.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.850+0000 7f163fe00640 1 -- 172.21.15.125:0/3229793711 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f163000c040 con 0x7f162006bfb0 2024-01-16T15:09:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 msgr2=0x7f162006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 0x7f162006e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1648072020 tx=0x7f1630008040 comp rx=0 tx=0).stop 2024-01-16T15:09:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 msgr2=0x7f164813c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f1638004730 comp rx=0 tx=0).stop 2024-01-16T15:09:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 shutdown_connections 2024-01-16T15:09:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164815f9c0 0x7f164814f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f164813e360 0x7f164814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f162006bfb0 0x7f162006e470 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1648072020 tx=0x7f1630008040 comp rx=0 tx=0).stop 2024-01-16T15:09:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 --2- 172.21.15.125:0/3229793711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f164813df80 0x7f164813c400 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.856+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 >> 172.21.15.125:0/3229793711 conn(0x7f164806a800 msgr2=0x7f16480afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.858+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 shutdown_connections 2024-01-16T15:09:15.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:15.858+0000 7f164e200640 1 -- 172.21.15.125:0/3229793711 wait complete. 2024-01-16T15:09:15.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969118 2024-01-16T15:09:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.165+0000 7fc177c00640 1 Processor -- start 2024-01-16T15:09:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.165+0000 7fc177c00640 1 -- start start 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc177c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17015d610 0x7fc1700ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc177c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc1700ced60 0x7fc1700d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc177c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1700d2170 con 0x7fc1700d1d20 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc177c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700d1390 con 0x7fc17015d610 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc177c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700b3400 con 0x7fc1700ced60 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.167+0000 7fc175200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17015d610 0x7fc1700ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc17015d610 0x7fc1700ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51152/0 (socket says 172.21.15.125:51152) 2024-01-16T15:09:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 -- 172.21.15.125:0/1877670399 learned_addr learned my addr 172.21.15.125:0/1877670399 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc16fe00640 1 --2- 172.21.15.125:0/1877670399 >> v2:172.21.15.137:3300/0 conn(0x7fc1700ced60 0x7fc1700d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc16f400640 1 -- 172.21.15.125:0/1877670399 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618636828 0 0) 0x7fc168001490 con 0x7fc1700d1d20 2024-01-16T15:09:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc16f400640 1 -- 172.21.15.125:0/1877670399 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc150003520 con 0x7fc1700d1d20 2024-01-16T15:09:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 -- 172.21.15.125:0/1877670399 >> v2:172.21.15.137:3300/0 conn(0x7fc1700ced60 msgr2=0x7fc1700d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 --2- 172.21.15.125:0/1877670399 >> v2:172.21.15.137:3300/0 conn(0x7fc1700ced60 0x7fc1700d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 -- 172.21.15.125:0/1877670399 >> v1:172.21.15.125:6789/0 conn(0x7fc1700d1d20 legacy=0x7fc1700cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.168+0000 7fc175200640 1 -- 172.21.15.125:0/1877670399 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700b27f0 con 0x7fc17015d610 2024-01-16T15:09:16.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.170+0000 7fc175200640 1 --2- 172.21.15.125:0/1877670399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700ce820 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc17008d4b0 tx=0x7fc16002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc276f361152de0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.170+0000 7fc16f400640 1 -- 172.21.15.125:0/1877670399 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160030870 con 0x7fc17015d610 2024-01-16T15:09:16.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.170+0000 7fc16f400640 1 -- 172.21.15.125:0/1877670399 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1600309d0 con 0x7fc17015d610 2024-01-16T15:09:16.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.170+0000 7fc16f400640 1 -- 172.21.15.125:0/1877670399 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160030d60 con 0x7fc17015d610 2024-01-16T15:09:16.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 -- 172.21.15.125:0/1877670399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 msgr2=0x7fc1700ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 --2- 172.21.15.125:0/1877670399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700ce820 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc17008d4b0 tx=0x7fc16002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 -- 172.21.15.125:0/1877670399 shutdown_connections 2024-01-16T15:09:16.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 --2- 172.21.15.125:0/1877670399 >> v2:172.21.15.137:3300/0 conn(0x7fc1700ced60 0x7fc1700d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 --2- 172.21.15.125:0/1877670399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700ce820 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.172+0000 7fc177c00640 1 -- 172.21.15.125:0/1877670399 >> 172.21.15.125:0/1877670399 conn(0x7fc17009c540 msgr2=0x7fc1700d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:16.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.173+0000 7fc177c00640 1 -- 172.21.15.125:0/1877670399 shutdown_connections 2024-01-16T15:09:16.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.173+0000 7fc177c00640 1 -- 172.21.15.125:0/1877670399 wait complete. 2024-01-16T15:09:16.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.176+0000 7fc177c00640 1 Processor -- start 2024-01-16T15:09:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.178+0000 7fc177c00640 1 -- start start 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1700ced60 0x7fc1701595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b6e40 con 0x7fc17015d610 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170070f30 con 0x7fc1700ced60 2024-01-16T15:09:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.180+0000 7fc177c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc170071210 con 0x7fc1700d1d20 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.181+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.181+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56574/0 (socket says 172.21.15.125:56574) 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.181+0000 7fc16fe00640 1 -- 172.21.15.125:0/646952976 learned_addr learned my addr 172.21.15.125:0/646952976 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.181+0000 7fc175c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.181+0000 7fc175200640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1700ced60 0x7fc1701595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 -- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1700ced60 msgr2=0x7fc1701595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1700ced60 0x7fc1701595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 -- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 msgr2=0x7fc1700cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 -- 172.21.15.125:0/646952976 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1600094d0 con 0x7fc1700d1d20 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc175c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.182+0000 7fc16fe00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc15800e870 tx=0x7fc15800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.183+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1580094f0 con 0x7fc1700d1d20 2024-01-16T15:09:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.183+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15800c940 con 0x7fc1700d1d20 2024-01-16T15:09:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.183+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800ccf0 con 0x7fc1700d1d20 2024-01-16T15:09:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.183+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700714f0 con 0x7fc1700d1d20 2024-01-16T15:09:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.184+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1700cb020 con 0x7fc1700d1d20 2024-01-16T15:09:16.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.188+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc158013470 con 0x7fc1700d1d20 2024-01-16T15:09:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.188+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1700b72d0 con 0x7fc1700d1d20 2024-01-16T15:09:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.189+0000 7fc16cc00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 0x7fc14806e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.189+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7fc158088670 con 0x7fc1700d1d20 2024-01-16T15:09:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.191+0000 7fc175200640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 0x7fc14806e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.192+0000 7fc175200640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 0x7fc14806e540 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc160036000 tx=0x7fc160008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.200+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1700b72d0 con 0x7fc1700d1d20 2024-01-16T15:09:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:16.452+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969118", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1700afe30 con 0x7fc14806c080 2024-01-16T15:09:18.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.956+0000 7fc16cc00640 1 -- 172.21.15.125:0/646952976 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc1700afe30 con 0x7fc14806c080 2024-01-16T15:09:18.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 msgr2=0x7fc14806e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 0x7fc14806e540 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc160036000 tx=0x7fc160008030 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 msgr2=0x7fc1700caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc15800e870 tx=0x7fc15800ed30 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 shutdown_connections 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc14806c080 0x7fc14806e540 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc17015d610 0x7fc1700cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1700d1d20 0x7fc1700caae0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 --2- 172.21.15.125:0/646952976 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1700ced60 0x7fc1701595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 >> 172.21.15.125:0/646952976 conn(0x7fc17009c540 msgr2=0x7fc1700d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 shutdown_connections 2024-01-16T15:09:18.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:18.961+0000 7fc177c00640 1 -- 172.21.15.125:0/646952976 wait complete. 2024-01-16T15:09:18.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96918 2024-01-16T15:09:19.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.178+0000 7fda16400640 1 Processor -- start 2024-01-16T15:09:19.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.179+0000 7fda16400640 1 -- start start 2024-01-16T15:09:19.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda16400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fda10166250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda16400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda100c6060 con 0x7fda10159040 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda16400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda100cf710 con 0x7fda1015c300 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda16400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda100c62d0 con 0x7fda100b34e0 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda0f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fda10159040 0x7fda100cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52470/0 (socket says 172.21.15.125:52470) 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.180+0000 7fda0f400640 1 -- 172.21.15.125:0/1787632632 learned_addr learned my addr 172.21.15.125:0/1787632632 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0fe00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fda10166250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720882793 0 0) 0x7fda100c6060 con 0x7fda10159040 2024-01-16T15:09:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f0003520 con 0x7fda10159040 2024-01-16T15:09:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407187025 0 0) 0x7fd9f0003520 con 0x7fda10159040 2024-01-16T15:09:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 msgr2=0x7fda10166250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fda10166250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 >> v1:172.21.15.137:6789/0 conn(0x7fda1015c300 legacy=0x7fda1015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100c65b0 con 0x7fda10159040 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fda00001670 con 0x7fda10159040 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 >> v1:172.21.15.125:6789/0 conn(0x7fda10159040 legacy=0x7fda100cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9f00039a0 0x7fd9f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fd9f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9f0004d00 0x7fd9f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda100c62d0 con 0x7fd9f0004d00 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f000ba60 con 0x7fd9f00039a0 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.181+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f000bc20 con 0x7fda100b34e0 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9f0004d00 0x7fd9f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0f400640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9f00039a0 0x7fd9f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda06800640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fd9f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 -- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 msgr2=0x7fd9f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fd9f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 -- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9f00039a0 msgr2=0x7fd9f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9f00039a0 0x7fd9f0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda0fe00640 1 -- 172.21.15.125:0/1787632632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f0003520 con 0x7fd9f0004d00 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.182+0000 7fda06800640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda100b34e0 0x7fd9f00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.183+0000 7fda0fe00640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9f0004d00 0x7fd9f0005180 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fda08009f90 tx=0x7fd9f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acb93564b43484 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.183+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0802cbd0 con 0x7fd9f0004d00 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.183+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0800f040 con 0x7fd9f0004d00 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.183+0000 7fda0ea00640 1 -- 172.21.15.125:0/1787632632 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08034440 con 0x7fd9f0004d00 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 -- 172.21.15.125:0/1787632632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9f0004d00 msgr2=0x7fd9f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 --2- 172.21.15.125:0/1787632632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9f0004d00 0x7fd9f0005180 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fda08009f90 tx=0x7fd9f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda0fe00640 1 -- 172.21.15.125:0/1787632632 reap_dead start 2024-01-16T15:09:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 -- 172.21.15.125:0/1787632632 shutdown_connections 2024-01-16T15:09:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 -- 172.21.15.125:0/1787632632 >> 172.21.15.125:0/1787632632 conn(0x7fda10066690 msgr2=0x7fda1006a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 -- 172.21.15.125:0/1787632632 shutdown_connections 2024-01-16T15:09:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.184+0000 7fda16400640 1 -- 172.21.15.125:0/1787632632 wait complete. 2024-01-16T15:09:19.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.186+0000 7fda16400640 1 Processor -- start 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.186+0000 7fda16400640 1 -- start start 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 0x7fda1015e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda10155970 0x7fda1015a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10157870 con 0x7fda100c9f70 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10154c60 con 0x7fda101535d0 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda16400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1015cb00 con 0x7fda10155970 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35720/0 (socket says 172.21.15.125:35720) 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 -- 172.21.15.125:0/719585986 learned_addr learned my addr 172.21.15.125:0/719585986 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda0f400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 0x7fda1015e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda0fe00640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda10155970 0x7fda1015a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 -- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda10155970 msgr2=0x7fda1015a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda10155970 0x7fda1015a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 -- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 msgr2=0x7fda1015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 0x7fda1015e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda06800640 1 -- 172.21.15.125:0/719585986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100ca400 con 0x7fda100c9f70 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.187+0000 7fda0f400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 0x7fda1015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda06800640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800d7d0 tx=0x7fd9f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8017070 con 0x7fda100c9f70 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f8005070 con 0x7fda100c9f70 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8005440 con 0x7fda100c9f70 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda100ca6e0 con 0x7fda100c9f70 2024-01-16T15:09:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.188+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda100caca0 con 0x7fda100c9f70 2024-01-16T15:09:19.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.189+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda10074a50 con 0x7fda100c9f70 2024-01-16T15:09:19.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.190+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd9f8007030 con 0x7fda100c9f70 2024-01-16T15:09:19.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.191+0000 7fda07e00640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 0x7fd9e806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.191+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7fd9f80881c0 con 0x7fda100c9f70 2024-01-16T15:09:19.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.191+0000 7fda0f400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 0x7fd9e806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.192+0000 7fda0f400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 0x7fd9e806e470 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fda00000f60 tx=0x7fda00000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.196+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9f80547f0 con 0x7fda100c9f70 2024-01-16T15:09:19.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.416+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96918", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1015c7c0 con 0x7fd9e806bfb0 2024-01-16T15:09:19.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.450+0000 7fda07e00640 1 -- 172.21.15.125:0/719585986 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda0000f7f0 con 0x7fd9e806bfb0 2024-01-16T15:09:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 msgr2=0x7fd9e806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 0x7fd9e806e470 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fda00000f60 tx=0x7fda00000ed0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 msgr2=0x7fda10155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800d7d0 tx=0x7fd9f800dc90 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 shutdown_connections 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda10155970 0x7fda1015a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd9e806bfb0 0x7fd9e806e470 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda100c9f70 0x7fda10155430 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 --2- 172.21.15.125:0/719585986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda101535d0 0x7fda1015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.453+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 >> 172.21.15.125:0/719585986 conn(0x7fda10066690 msgr2=0x7fda100cdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.454+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 shutdown_connections 2024-01-16T15:09:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.454+0000 7fda16400640 1 -- 172.21.15.125:0/719585986 wait complete. 2024-01-16T15:09:19.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969113 2024-01-16T15:09:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.679+0000 7f8212a00640 1 Processor -- start 2024-01-16T15:09:19.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.680+0000 7f8212a00640 1 -- start start 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.681+0000 7f8212a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f820c13e360 0x7f820c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.681+0000 7f8212a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f820c15f9c0 0x7f820c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.681+0000 7f8212a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c15c5f0 con 0x7f820c13df80 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.681+0000 7f8212a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c06c470 con 0x7f820c15f9c0 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.681+0000 7f8212a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c06c750 con 0x7f820c13e360 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f820c15f9c0 0x7f820c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f820c15f9c0 0x7f820c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35732/0 (socket says 172.21.15.125:35732) 2024-01-16T15:09:19.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 -- 172.21.15.125:0/2562416663 learned_addr learned my addr 172.21.15.125:0/2562416663 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f820b400640 1 --2- 172.21.15.125:0/2562416663 >> v2:172.21.15.137:3300/0 conn(0x7f820c13e360 0x7f820c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 -- 172.21.15.125:0/2562416663 >> v2:172.21.15.137:3300/0 conn(0x7f820c13e360 msgr2=0x7f820c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 --2- 172.21.15.125:0/2562416663 >> v2:172.21.15.137:3300/0 conn(0x7f820c13e360 0x7f820c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 -- 172.21.15.125:0/2562416663 >> v1:172.21.15.137:6789/0 conn(0x7f820c13df80 legacy=0x7f820c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.682+0000 7f8210a00640 1 -- 172.21.15.125:0/2562416663 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c06ca30 con 0x7f820c15f9c0 2024-01-16T15:09:19.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.683+0000 7f8210a00640 1 --2- 172.21.15.125:0/2562416663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c0cf7d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f820c0ccae0 tx=0x7f820402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b01c9ca38d151b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f820aa00640 1 -- 172.21.15.125:0/2562416663 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204031640 con 0x7f820c15f9c0 2024-01-16T15:09:19.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f820aa00640 1 -- 172.21.15.125:0/2562416663 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040317a0 con 0x7f820c15f9c0 2024-01-16T15:09:19.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f820aa00640 1 -- 172.21.15.125:0/2562416663 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204031b30 con 0x7f820c15f9c0 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 -- 172.21.15.125:0/2562416663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 msgr2=0x7f820c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 --2- 172.21.15.125:0/2562416663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c0cf7d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f820c0ccae0 tx=0x7f820402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 -- 172.21.15.125:0/2562416663 shutdown_connections 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 --2- 172.21.15.125:0/2562416663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c0cf7d0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 --2- 172.21.15.125:0/2562416663 >> v2:172.21.15.137:3300/0 conn(0x7f820c13e360 0x7f820c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 -- 172.21.15.125:0/2562416663 >> 172.21.15.125:0/2562416663 conn(0x7f820c06a800 msgr2=0x7f820c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.684+0000 7f8212a00640 1 -- 172.21.15.125:0/2562416663 shutdown_connections 2024-01-16T15:09:19.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.685+0000 7f8212a00640 1 -- 172.21.15.125:0/2562416663 wait complete. 2024-01-16T15:09:19.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.686+0000 7f8212a00640 1 Processor -- start 2024-01-16T15:09:19.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.686+0000 7f8212a00640 1 -- start start 2024-01-16T15:09:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820c13df80 0x7f820c1504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 0x7f820c156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c157c10 con 0x7f820c15f9c0 2024-01-16T15:09:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c157650 con 0x7f820c13e360 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.687+0000 7f8212a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c157930 con 0x7f820c13df80 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35736/0 (socket says 172.21.15.125:35736) 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 -- 172.21.15.125:0/1954117887 learned_addr learned my addr 172.21.15.125:0/1954117887 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f820be00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820c13df80 0x7f820c1504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f820b400640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 0x7f820c156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 -- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820c13df80 msgr2=0x7f820c1504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820c13df80 0x7f820c1504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 -- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 msgr2=0x7f820c156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 0x7f820c156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 -- 172.21.15.125:0/1954117887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040094d0 con 0x7f820c15f9c0 2024-01-16T15:09:19.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f820b400640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 0x7f820c156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:19.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.688+0000 7f8210a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f820402b930 tx=0x7f82040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.689+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820403ac60 con 0x7f820c15f9c0 2024-01-16T15:09:19.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.689+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820403adc0 con 0x7f820c15f9c0 2024-01-16T15:09:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.689+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204044630 con 0x7f820c15f9c0 2024-01-16T15:09:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.689+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c1590f0 con 0x7f820c15f9c0 2024-01-16T15:09:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.689+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c13d880 con 0x7f820c15f9c0 2024-01-16T15:09:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.690+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c159610 con 0x7f820c15f9c0 2024-01-16T15:09:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.691+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f820402f050 con 0x7f820c15f9c0 2024-01-16T15:09:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.692+0000 7f8203e00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 0x7f81e406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.692+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f82040aa930 con 0x7f820c15f9c0 2024-01-16T15:09:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.692+0000 7f820be00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 0x7f81e406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.693+0000 7f820be00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 0x7f81e406e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009f20 tx=0x7f81fc0094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.696+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f820c159610 con 0x7f820c15f9c0 2024-01-16T15:09:19.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.913+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969113", "target": ["mon-mgr", ""]}) v1 -- 0x7f820c165ec0 con 0x7f81e406bfb0 2024-01-16T15:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.944+0000 7f8203e00640 1 -- 172.21.15.125:0/1954117887 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f820c165ec0 con 0x7f81e406bfb0 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 msgr2=0x7f81e406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 0x7f81e406e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009f20 tx=0x7f81fc0094d0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 msgr2=0x7f820c1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f820402b930 tx=0x7f82040046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 shutdown_connections 2024-01-16T15:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f81e406bfb0 0x7f81e406e470 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f820c15f9c0 0x7f820c1385a0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f820c13e360 0x7f820c156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 --2- 172.21.15.125:0/1954117887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f820c13df80 0x7f820c1504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.947+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 >> 172.21.15.125:0/1954117887 conn(0x7f820c06a800 msgr2=0x7f820c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.948+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 shutdown_connections 2024-01-16T15:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:19.948+0000 7f8212a00640 1 -- 172.21.15.125:0/1954117887 wait complete. 2024-01-16T15:09:19.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96919 2024-01-16T15:09:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.174+0000 7f7c8fc00640 1 Processor -- start 2024-01-16T15:09:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.174+0000 7f7c8fc00640 1 -- start start 2024-01-16T15:09:20.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7c8814b2c0 0x7f7c880cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c880cf7d0 0x7f7c88070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c88070ea0 0x7f7c880b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8814b6a0 con 0x7f7c88070ea0 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c880cfbf0 con 0x7f7c8814b2c0 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88071690 con 0x7f7c880cf7d0 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8dc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c88070ea0 0x7f7c880b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8dc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c88070ea0 0x7f7c880b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35752/0 (socket says 172.21.15.125:35752) 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.176+0000 7f7c8dc00640 1 -- 172.21.15.125:0/3576899155 learned_addr learned my addr 172.21.15.125:0/3576899155 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:20.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c87e00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c880cf7d0 0x7f7c88070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8d200640 1 --2- 172.21.15.125:0/3576899155 >> v2:172.21.15.137:3300/0 conn(0x7f7c8814b2c0 0x7f7c880cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8dc00640 1 -- 172.21.15.125:0/3576899155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c880cf7d0 msgr2=0x7f7c88070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8dc00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c880cf7d0 0x7f7c88070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8dc00640 1 -- 172.21.15.125:0/3576899155 >> v2:172.21.15.137:3300/0 conn(0x7f7c8814b2c0 msgr2=0x7f7c880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8dc00640 1 --2- 172.21.15.125:0/3576899155 >> v2:172.21.15.137:3300/0 conn(0x7f7c8814b2c0 0x7f7c880cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.177+0000 7f7c8dc00640 1 -- 172.21.15.125:0/3576899155 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88071970 con 0x7f7c88070ea0 2024-01-16T15:09:20.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.178+0000 7f7c8dc00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c88070ea0 0x7f7c880b1ee0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f7c8806e650 tx=0x7f7c8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02cdb18211802f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.178+0000 7f7c87400640 1 -- 172.21.15.125:0/3576899155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80031870 con 0x7f7c88070ea0 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.178+0000 7f7c87400640 1 -- 172.21.15.125:0/3576899155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c800319d0 con 0x7f7c88070ea0 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.178+0000 7f7c87400640 1 -- 172.21.15.125:0/3576899155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80031d60 con 0x7f7c88070ea0 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 -- 172.21.15.125:0/3576899155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c88070ea0 msgr2=0x7f7c880b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c88070ea0 0x7f7c880b1ee0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f7c8806e650 tx=0x7f7c8002b910 comp rx=0 tx=0).stop 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 -- 172.21.15.125:0/3576899155 shutdown_connections 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c88070ea0 0x7f7c880b1ee0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/3576899155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c880cf7d0 0x7f7c88070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/3576899155 >> v2:172.21.15.137:3300/0 conn(0x7f7c8814b2c0 0x7f7c880cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 -- 172.21.15.125:0/3576899155 >> 172.21.15.125:0/3576899155 conn(0x7f7c8806a800 msgr2=0x7f7c8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 -- 172.21.15.125:0/3576899155 shutdown_connections 2024-01-16T15:09:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.179+0000 7f7c8fc00640 1 -- 172.21.15.125:0/3576899155 wait complete. 2024-01-16T15:09:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.181+0000 7f7c8fc00640 1 Processor -- start 2024-01-16T15:09:20.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.181+0000 7f7c8fc00640 1 -- start start 2024-01-16T15:09:20.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c88070ea0 0x7f7c88160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 0x7f7c880ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8813aa60 con 0x7f7c880cf7d0 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880ade40 con 0x7f7c88070ea0 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c8fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880ae120 con 0x7f7c8814b2c0 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35754/0 (socket says 172.21.15.125:35754) 2024-01-16T15:09:20.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.182+0000 7f7c87e00640 1 -- 172.21.15.125:0/2252999341 learned_addr learned my addr 172.21.15.125:0/2252999341 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c8d200640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c88070ea0 0x7f7c88160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c8dc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 0x7f7c880ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 -- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 msgr2=0x7f7c880ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 0x7f7c880ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 -- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c88070ea0 msgr2=0x7f7c88160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c88070ea0 0x7f7c88160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 -- 172.21.15.125:0/2252999341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800094d0 con 0x7f7c880cf7d0 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c8dc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 0x7f7c880ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c87e00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7c70007e60 tx=0x7f7c7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.183+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70014620 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.184+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7000ee90 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.184+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8813ad20 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.184+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c700149c0 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.184+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88161310 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.185+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c880ae340 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.186+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7c70014b20 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.186+0000 7f7c84c00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 0x7f7c6006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.187+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f7c70088f20 con 0x7f7c880cf7d0 2024-01-16T15:09:20.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.187+0000 7f7c8d200640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 0x7f7c6006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.187+0000 7f7c8d200640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 0x7f7c6006e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800a970 tx=0x7f7c78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.191+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c70055840 con 0x7f7c880cf7d0 2024-01-16T15:09:20.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.407+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96919", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c8813aba0 con 0x7f7c6006bfb0 2024-01-16T15:09:20.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.427+0000 7f7c84c00640 1 -- 172.21.15.125:0/2252999341 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7c7800c050 con 0x7f7c6006bfb0 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 msgr2=0x7f7c6006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 0x7f7c6006e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800a970 tx=0x7f7c78008040 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 msgr2=0x7f7c880ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7c70007e60 tx=0x7f7c7000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 shutdown_connections 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c8814b2c0 0x7f7c880ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7c6006bfb0 0x7f7c6006e470 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c880cf7d0 0x7f7c880ad2a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 --2- 172.21.15.125:0/2252999341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c88070ea0 0x7f7c88160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.430+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 >> 172.21.15.125:0/2252999341 conn(0x7f7c8806a800 msgr2=0x7f7c881481d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.431+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 shutdown_connections 2024-01-16T15:09:20.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.431+0000 7f7c8fc00640 1 -- 172.21.15.125:0/2252999341 wait complete. 2024-01-16T15:09:20.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96917 2024-01-16T15:09:20.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.656+0000 7f6d3f200640 1 Processor -- start 2024-01-16T15:09:20.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.656+0000 7f6d3f200640 1 -- start start 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 0x7f6d3815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d380cd530 0x7f6d3813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3f200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d380cdec0 con 0x7f6d380cd9b0 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d380ce1a0 con 0x7f6d380cd530 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380ce480 con 0x7f6d3813cfa0 2024-01-16T15:09:20.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6d380cd9b0 0x7f6d3815d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52486/0 (socket says 172.21.15.125:52486) 2024-01-16T15:09:20.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.658+0000 7f6d3e800640 1 -- 172.21.15.125:0/1863851366 learned_addr learned my addr 172.21.15.125:0/1863851366 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3d400640 1 --2- 172.21.15.125:0/1863851366 >> v2:172.21.15.137:3300/0 conn(0x7f6d380cd530 0x7f6d3813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3de00640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 0x7f6d3815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164588851 0 0) 0x7f6d380cdec0 con 0x7f6d380cd9b0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d18003520 con 0x7f6d380cd9b0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1861553587 0 0) 0x7f6d18003520 con 0x7f6d380cd9b0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 msgr2=0x7f6d3815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 0x7f6d3815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 >> v2:172.21.15.137:3300/0 conn(0x7f6d380cd530 msgr2=0x7f6d3813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 --2- 172.21.15.125:0/1863851366 >> v2:172.21.15.137:3300/0 conn(0x7f6d380cd530 0x7f6d3813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380ce760 con 0x7f6d380cd9b0 2024-01-16T15:09:20.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.659+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6d30001a10 con 0x7f6d380cd9b0 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 >> v1:172.21.15.125:6789/0 conn(0x7f6d380cd9b0 legacy=0x7f6d3815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 0x7f6d18003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d18004d00 0x7f6d18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380ce1a0 con 0x7f6d18003f40 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380ce480 con 0x7f6d18004d00 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1800dc40 con 0x7f6d3813cfa0 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3e800640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d18004d00 0x7f6d18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 -- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 msgr2=0x7f6d18003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d3813cfa0 0x7f6d18003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 -- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d18004d00 msgr2=0x7f6d18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d18004d00 0x7f6d18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3d400640 1 -- 172.21.15.125:0/1863851366 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18003520 con 0x7f6d18003f40 2024-01-16T15:09:20.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.660+0000 7f6d3e800640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d18004d00 0x7f6d18005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.661+0000 7f6d3d400640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d180047c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009ac0 tx=0x7f6d180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea5336b45140262 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002c710 con 0x7f6d18003f40 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2002c870 con 0x7f6d18003f40 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3ca00640 1 -- 172.21.15.125:0/1863851366 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002cc00 con 0x7f6d18003f40 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 -- 172.21.15.125:0/1863851366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 msgr2=0x7f6d180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 --2- 172.21.15.125:0/1863851366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d180047c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009ac0 tx=0x7f6d180057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3e800640 1 -- 172.21.15.125:0/1863851366 reap_dead start 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 -- 172.21.15.125:0/1863851366 shutdown_connections 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 -- 172.21.15.125:0/1863851366 >> 172.21.15.125:0/1863851366 conn(0x7f6d3806a800 msgr2=0x7f6d3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 -- 172.21.15.125:0/1863851366 shutdown_connections 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.662+0000 7f6d3f200640 1 -- 172.21.15.125:0/1863851366 wait complete. 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.664+0000 7f6d3f200640 1 Processor -- start 2024-01-16T15:09:20.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.664+0000 7f6d3f200640 1 -- start start 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d380c43c0 0x7f6d38158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d38148820 0x7f6d38148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38159c70 con 0x7f6d18003f40 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38146a00 con 0x7f6d380c43c0 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d381491e0 con 0x7f6d38148820 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35782/0 (socket says 172.21.15.125:35782) 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.665+0000 7f6d3de00640 1 -- 172.21.15.125:0/3571838327 learned_addr learned my addr 172.21.15.125:0/3571838327 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 -- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d38148820 msgr2=0x7f6d38148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d38148820 0x7f6d38148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 -- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d380c43c0 msgr2=0x7f6d38158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d380c43c0 0x7f6d38158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 -- 172.21.15.125:0/3571838327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d381593d0 con 0x7f6d18003f40 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3de00640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e7f0 tx=0x7f6d2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800c8e0 con 0x7f6d18003f40 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800ca40 con 0x7f6d18003f40 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800cdd0 con 0x7f6d18003f40 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.666+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d381596b0 con 0x7f6d18003f40 2024-01-16T15:09:20.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.667+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d38157960 con 0x7f6d18003f40 2024-01-16T15:09:20.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.668+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d38067450 con 0x7f6d18003f40 2024-01-16T15:09:20.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.669+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f6d28013470 con 0x7f6d18003f40 2024-01-16T15:09:20.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.669+0000 7f6d36000640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 0x7f6d1006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.670+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f6d280885b0 con 0x7f6d18003f40 2024-01-16T15:09:20.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.670+0000 7f6d3d400640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 0x7f6d1006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.670+0000 7f6d3d400640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 0x7f6d1006e4a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6d20036000 tx=0x7f6d2002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.674+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d28054e70 con 0x7f6d18003f40 2024-01-16T15:09:20.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.892+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96917", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d38146b40 con 0x7f6d1006bfe0 2024-01-16T15:09:20.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.930+0000 7f6d36000640 1 -- 172.21.15.125:0/3571838327 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6d2002da70 con 0x7f6d1006bfe0 2024-01-16T15:09:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 msgr2=0x7f6d1006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 0x7f6d1006e4a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6d20036000 tx=0x7f6d2002e000 comp rx=0 tx=0).stop 2024-01-16T15:09:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 msgr2=0x7f6d38157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e7f0 tx=0x7f6d2800ecb0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 shutdown_connections 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d38148820 0x7f6d38148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d380c43c0 0x7f6d38158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6d1006bfe0 0x7f6d1006e4a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 --2- 172.21.15.125:0/3571838327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18003f40 0x7f6d38157420 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.933+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 >> 172.21.15.125:0/3571838327 conn(0x7f6d3806a800 msgr2=0x7f6d38153bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.934+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 shutdown_connections 2024-01-16T15:09:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:20.934+0000 7f6d3f200640 1 -- 172.21.15.125:0/3571838327 wait complete. 2024-01-16T15:09:20.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969112 2024-01-16T15:09:21.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.149+0000 7f38d5600640 1 Processor -- start 2024-01-16T15:09:21.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.149+0000 7f38d5600640 1 -- start start 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38d5600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38d00cf9a0 0x7f38d0166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38d5600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d015c5f0 con 0x7f38d013df80 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38d5600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d006c470 con 0x7f38d013e360 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38d5600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d006c750 con 0x7f38d00cf9a0 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38cea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f38d013df80 0x7f38d015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52502/0 (socket says 172.21.15.125:52502) 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38cea00640 1 -- 172.21.15.125:0/3695007971 learned_addr learned my addr 172.21.15.125:0/3695007971 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:21.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.151+0000 7f38cf400640 1 --2- 172.21.15.125:0/3695007971 >> v2:172.21.15.125:3300/0 conn(0x7f38d00cf9a0 0x7f38d0166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.152+0000 7f38cf400640 1 -- 172.21.15.125:0/3695007971 >> v1:172.21.15.137:6789/0 conn(0x7f38d013e360 legacy=0x7f38d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.152+0000 7f38cf400640 1 -- 172.21.15.125:0/3695007971 >> v1:172.21.15.125:6789/0 conn(0x7f38d013df80 legacy=0x7f38d015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.152+0000 7f38cf400640 1 -- 172.21.15.125:0/3695007971 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d006ca30 con 0x7f38d00cf9a0 2024-01-16T15:09:21.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.153+0000 7f38cf400640 1 --2- 172.21.15.125:0/3695007971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d00cf9a0 0x7f38d0166030 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009d90 tx=0x7f38c802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2433db8700804e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.153+0000 7f38cd600640 1 -- 172.21.15.125:0/3695007971 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c802ccf0 con 0x7f38d00cf9a0 2024-01-16T15:09:21.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.153+0000 7f38cd600640 1 -- 172.21.15.125:0/3695007971 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c802ce50 con 0x7f38d00cf9a0 2024-01-16T15:09:21.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.153+0000 7f38cd600640 1 -- 172.21.15.125:0/3695007971 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8034660 con 0x7f38d00cf9a0 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 -- 172.21.15.125:0/3695007971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d00cf9a0 msgr2=0x7f38d0166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 --2- 172.21.15.125:0/3695007971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d00cf9a0 0x7f38d0166030 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009d90 tx=0x7f38c802bf40 comp rx=0 tx=0).stop 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 -- 172.21.15.125:0/3695007971 shutdown_connections 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 --2- 172.21.15.125:0/3695007971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d00cf9a0 0x7f38d0166030 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 -- 172.21.15.125:0/3695007971 >> 172.21.15.125:0/3695007971 conn(0x7f38d006a800 msgr2=0x7f38d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 -- 172.21.15.125:0/3695007971 shutdown_connections 2024-01-16T15:09:21.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.154+0000 7f38d5600640 1 -- 172.21.15.125:0/3695007971 wait complete. 2024-01-16T15:09:21.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.156+0000 7f38d5600640 1 Processor -- start 2024-01-16T15:09:21.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.156+0000 7f38d5600640 1 -- start start 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 0x7f38d0153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38d013e360 0x7f38d00aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d01537b0 con 0x7f38d013df80 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00a87f0 con 0x7f38d00cf9a0 2024-01-16T15:09:21.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38d5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00a84c0 con 0x7f38d013e360 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38ce000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38ce000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35798/0 (socket says 172.21.15.125:35798) 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38ce000640 1 -- 172.21.15.125:0/2520834475 learned_addr learned my addr 172.21.15.125:0/2520834475 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.157+0000 7f38cea00640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 0x7f38d0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38cf400640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38d013e360 0x7f38d00aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 -- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38d013e360 msgr2=0x7f38d00aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38d013e360 0x7f38d00aaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 -- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 msgr2=0x7f38d0153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 0x7f38d0153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 -- 172.21.15.125:0/2520834475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c8009a70 con 0x7f38d013df80 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38cea00640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 0x7f38d0153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38ce000640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c430 tx=0x7f38b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80052d0 con 0x7f38d013df80 2024-01-16T15:09:21.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b8005430 con 0x7f38d013df80 2024-01-16T15:09:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d0071520 con 0x7f38d013df80 2024-01-16T15:09:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8003830 con 0x7f38d013df80 2024-01-16T15:09:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.158+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d0071ae0 con 0x7f38d013df80 2024-01-16T15:09:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.159+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d00a8980 con 0x7f38d013df80 2024-01-16T15:09:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.160+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f38b8003990 con 0x7f38d013df80 2024-01-16T15:09:21.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.161+0000 7f38c6a00640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 0x7f38a806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.161+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f38b8087bc0 con 0x7f38d013df80 2024-01-16T15:09:21.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.162+0000 7f38cea00640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 0x7f38a806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.162+0000 7f38cea00640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 0x7f38a806e470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005880 tx=0x7f38c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.166+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38b8054480 con 0x7f38d013df80 2024-01-16T15:09:21.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.386+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969112", "target": ["mon-mgr", ""]}) v1 -- 0x7f38d006cff0 con 0x7f38a806bfb0 2024-01-16T15:09:21.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.416+0000 7f38c6a00640 1 -- 172.21.15.125:0/2520834475 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38c0001330 con 0x7f38a806bfb0 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 msgr2=0x7f38a806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 0x7f38a806e470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005880 tx=0x7f38c0009040 comp rx=0 tx=0).stop 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 msgr2=0x7f38d0070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c430 tx=0x7f38b800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 shutdown_connections 2024-01-16T15:09:21.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38d013e360 0x7f38d00aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f38a806bfb0 0x7f38a806e470 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38d013df80 0x7f38d0070cd0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 --2- 172.21.15.125:0/2520834475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38d00cf9a0 0x7f38d0153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.419+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 >> 172.21.15.125:0/2520834475 conn(0x7f38d006a800 msgr2=0x7f38d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.420+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 shutdown_connections 2024-01-16T15:09:21.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.420+0000 7f38d5600640 1 -- 172.21.15.125:0/2520834475 wait complete. 2024-01-16T15:09:21.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969110 2024-01-16T15:09:21.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.628+0000 7f686fc00640 1 Processor -- start 2024-01-16T15:09:21.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.629+0000 7f686fc00640 1 -- start start 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f68680cf9a0 0x7f6868070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f68680cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686fc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6868071690 con 0x7f686814b2c0 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686fc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686814b6a0 con 0x7f68680cf9a0 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6868071970 con 0x7f6868070ea0 2024-01-16T15:09:21.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686d200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f686814b2c0 0x7f68680cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52510/0 (socket says 172.21.15.125:52510) 2024-01-16T15:09:21.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.630+0000 7f686d200640 1 -- 172.21.15.125:0/3242869966 learned_addr learned my addr 172.21.15.125:0/3242869966 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:21.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f685fc00640 1 --2- 172.21.15.125:0/3242869966 >> v2:172.21.15.137:3300/0 conn(0x7f68680cf9a0 0x7f6868070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f686dc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f68680cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581796656 0 0) 0x7f6868071690 con 0x7f686814b2c0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6848003520 con 0x7f686814b2c0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839140915 0 0) 0x7f6848003520 con 0x7f686814b2c0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 msgr2=0x7f68680cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f68680cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 >> v2:172.21.15.137:3300/0 conn(0x7f68680cf9a0 msgr2=0x7f6868070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 --2- 172.21.15.125:0/3242869966 >> v2:172.21.15.137:3300/0 conn(0x7f68680cf9a0 0x7f6868070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6868071c50 con 0x7f686814b2c0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f68580019f0 con 0x7f686814b2c0 2024-01-16T15:09:21.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 >> v1:172.21.15.125:6789/0 conn(0x7f686814b2c0 legacy=0x7f68680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f6848003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848003ef0 0x7f68480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6848004d00 0x7f6848005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686814b6a0 con 0x7f6848004d00 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6868071970 con 0x7f6848003ef0 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.631+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684800b9d0 con 0x7f6868070ea0 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6848004d00 0x7f6848005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686d200640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f6848003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f685fc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848003ef0 0x7f68480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 -- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 msgr2=0x7f6848003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6868070ea0 0x7f6848003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 -- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848003ef0 msgr2=0x7f68480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848003ef0 0x7f68480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.632+0000 7f686dc00640 1 -- 172.21.15.125:0/3242869966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848003520 con 0x7f6848004d00 2024-01-16T15:09:21.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.633+0000 7f686dc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6848004d00 0x7f6848005180 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f686000ba30 tx=0x7f68480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313cc639d501b092 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860032640 con 0x7f6848004d00 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68600327a0 con 0x7f6848004d00 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f6867e00640 1 -- 172.21.15.125:0/3242869966 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860032b30 con 0x7f6848004d00 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 -- 172.21.15.125:0/3242869966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6848004d00 msgr2=0x7f6848005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 --2- 172.21.15.125:0/3242869966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6848004d00 0x7f6848005180 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f686000ba30 tx=0x7f68480057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686dc00640 1 -- 172.21.15.125:0/3242869966 reap_dead start 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 -- 172.21.15.125:0/3242869966 shutdown_connections 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 -- 172.21.15.125:0/3242869966 >> 172.21.15.125:0/3242869966 conn(0x7f686806a800 msgr2=0x7f686806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 -- 172.21.15.125:0/3242869966 shutdown_connections 2024-01-16T15:09:21.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.634+0000 7f686fc00640 1 -- 172.21.15.125:0/3242869966 wait complete. 2024-01-16T15:09:21.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.636+0000 7f686fc00640 1 Processor -- start 2024-01-16T15:09:21.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.636+0000 7f686fc00640 1 -- start start 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848004d00 0x7f6868161d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6868164990 0x7f6868163720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68681460d0 con 0x7f6868142dc0 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6868162e30 con 0x7f6868164990 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.637+0000 7f686fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6868162590 con 0x7f6848004d00 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35810/0 (socket says 172.21.15.125:35810) 2024-01-16T15:09:21.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 -- 172.21.15.125:0/2018577139 learned_addr learned my addr 172.21.15.125:0/2018577139 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 -- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848004d00 msgr2=0x7f6868161d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848004d00 0x7f6868161d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 -- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6868164990 msgr2=0x7f6868163720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6868164990 0x7f6868163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 -- 172.21.15.125:0/2018577139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6868162870 con 0x7f6868142dc0 2024-01-16T15:09:21.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.638+0000 7f685fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6850011600 tx=0x7f6850011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.639+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850015870 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.639+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68500159d0 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.639+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850015d40 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.639+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6868162b50 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.639+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68681622a0 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.640+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68680675d0 con 0x7f6868142dc0 2024-01-16T15:09:21.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.641+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f685000f490 con 0x7f6868142dc0 2024-01-16T15:09:21.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.642+0000 7f6865600640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 0x7f684006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.642+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f6850088aa0 con 0x7f6868142dc0 2024-01-16T15:09:21.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.642+0000 7f686d200640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 0x7f684006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.643+0000 7f686d200640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 0x7f684006e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6858005df0 tx=0x7f6858002110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.646+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6850055380 con 0x7f6868142dc0 2024-01-16T15:09:21.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.909+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969110", "target": ["mon-mgr", ""]}) v1 -- 0x7f6868071330 con 0x7f684006bfb0 2024-01-16T15:09:21.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.937+0000 7f6865600640 1 -- 172.21.15.125:0/2018577139 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6858004b60 con 0x7f684006bfb0 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.941+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 msgr2=0x7f684006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.941+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 0x7f684006e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6858005df0 tx=0x7f6858002110 comp rx=0 tx=0).stop 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 msgr2=0x7f686815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6850011600 tx=0x7f6850011ac0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 shutdown_connections 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6868164990 0x7f6868163720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f684006bfb0 0x7f684006e470 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6868142dc0 0x7f686815e910 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 --2- 172.21.15.125:0/2018577139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848004d00 0x7f6868161d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.942+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 >> 172.21.15.125:0/2018577139 conn(0x7f686806a800 msgr2=0x7f68680aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.943+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 shutdown_connections 2024-01-16T15:09:21.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:21.943+0000 7f686fc00640 1 -- 172.21.15.125:0/2018577139 wait complete. 2024-01-16T15:09:21.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96911 2024-01-16T15:09:22.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.163+0000 7fc20e600640 1 Processor -- start 2024-01-16T15:09:22.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.164+0000 7fc20e600640 1 -- start start 2024-01-16T15:09:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc208146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20e600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2080b2e10 con 0x7fc208071f00 2024-01-16T15:09:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20e600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20806cc20 con 0x7fc20806c7c0 2024-01-16T15:09:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20806cec0 con 0x7fc208071410 2024-01-16T15:09:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20dc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc208071f00 0x7fc208166cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52526/0 (socket says 172.21.15.125:52526) 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.165+0000 7fc20dc00640 1 -- 172.21.15.125:0/3679341987 learned_addr learned my addr 172.21.15.125:0/3679341987 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc20d200640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc208146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295610549 0 0) 0x7fc2080b2e10 con 0x7fc208071f00 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e8003520 con 0x7fc208071f00 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707087316 0 0) 0x7fc1e8003520 con 0x7fc208071f00 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 msgr2=0x7fc208146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc208146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 >> v1:172.21.15.137:6789/0 conn(0x7fc20806c7c0 legacy=0x7fc2080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20806d1a0 con 0x7fc208071f00 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc2000019f0 con 0x7fc208071f00 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 >> v1:172.21.15.125:6789/0 conn(0x7fc208071f00 legacy=0x7fc208166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc1e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1e8003f20 0x7fc1e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.166+0000 7fc207e00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1e8004d00 0x7fc1e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20806cec0 con 0x7fc1e8003f20 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e800ba60 con 0x7fc1e8004d00 2024-01-16T15:09:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e800bc20 con 0x7fc208071410 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1e8003f20 0x7fc1e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc20d200640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc1e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 -- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 msgr2=0x7fc1e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc208071410 0x7fc1e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 -- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1e8004d00 msgr2=0x7fc1e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1e8004d00 0x7fc1e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.167+0000 7fc1ffc00640 1 -- 172.21.15.125:0/3679341987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e8003520 con 0x7fc1e8003f20 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.168+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1e8003f20 0x7fc1e80047c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0001750 tx=0x7fc1e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc2ba85dfbd2ef9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.168+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00052d0 con 0x7fc1e8003f20 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.168+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f002e050 con 0x7fc1e8003f20 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.168+0000 7fc207e00640 1 -- 172.21.15.125:0/3679341987 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0003830 con 0x7fc1e8003f20 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 -- 172.21.15.125:0/3679341987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1e8003f20 msgr2=0x7fc1e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 --2- 172.21.15.125:0/3679341987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1e8003f20 0x7fc1e80047c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0001750 tx=0x7fc1e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20dc00640 1 -- 172.21.15.125:0/3679341987 reap_dead start 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 -- 172.21.15.125:0/3679341987 shutdown_connections 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 -- 172.21.15.125:0/3679341987 >> 172.21.15.125:0/3679341987 conn(0x7fc20806a800 msgr2=0x7fc20806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 -- 172.21.15.125:0/3679341987 shutdown_connections 2024-01-16T15:09:22.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.169+0000 7fc20e600640 1 -- 172.21.15.125:0/3679341987 wait complete. 2024-01-16T15:09:22.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.171+0000 7fc20e600640 1 Processor -- start 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.171+0000 7fc20e600640 1 -- start start 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1e8003f20 0x7fc20814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc208146f20 0x7fc2081473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20815cda0 con 0x7fc208149420 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20814c3b0 con 0x7fc208146f20 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2081498e0 con 0x7fc1e8003f20 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc1ffc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc1ffc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35824/0 (socket says 172.21.15.125:35824) 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc1ffc00640 1 -- 172.21.15.125:0/2617286345 learned_addr learned my addr 172.21.15.125:0/2617286345 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20d200640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1e8003f20 0x7fc20814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.172+0000 7fc20dc00640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc208146f20 0x7fc2081473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 -- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1e8003f20 msgr2=0x7fc20814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1e8003f20 0x7fc20814be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 -- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc208146f20 msgr2=0x7fc2081473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc208146f20 0x7fc2081473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 -- 172.21.15.125:0/2617286345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208149bc0 con 0x7fc208149420 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc1ffc00640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0008fd0 tx=0x7fc1f0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00076c0 con 0x7fc208149420 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f0007a30 con 0x7fc208149420 2024-01-16T15:09:22.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0006be0 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.173+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc208149ea0 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.174+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20814a460 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.174+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20814a5f0 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.175+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc1f0006d40 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.176+0000 7fc205600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 0x7fc1e006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.176+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fc1f00aa7d0 con 0x7fc208149420 2024-01-16T15:09:22.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.177+0000 7fc20d200640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 0x7fc1e006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.177+0000 7fc20d200640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 0x7fc1e006e4a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc20813fb50 tx=0x7fc1f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.181+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1f0076fb0 con 0x7fc208149420 2024-01-16T15:09:22.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.397+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96911", "target": ["mon-mgr", ""]}) v1 -- 0x7fc20815cda0 con 0x7fc1e006bfe0 2024-01-16T15:09:22.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.428+0000 7fc205600640 1 -- 172.21.15.125:0/2617286345 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc20815cda0 con 0x7fc1e006bfe0 2024-01-16T15:09:22.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 msgr2=0x7fc1e006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 0x7fc1e006e4a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc20813fb50 tx=0x7fc1f8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 msgr2=0x7fc2081e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0008fd0 tx=0x7fc1f0001120 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 shutdown_connections 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc208146f20 0x7fc2081473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc1e006bfe0 0x7fc1e006e4a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc208149420 0x7fc2081e4bc0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 --2- 172.21.15.125:0/2617286345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1e8003f20 0x7fc20814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.431+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 >> 172.21.15.125:0/2617286345 conn(0x7fc20806a800 msgr2=0x7fc2080afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.432+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 shutdown_connections 2024-01-16T15:09:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.432+0000 7fc20e600640 1 -- 172.21.15.125:0/2617286345 wait complete. 2024-01-16T15:09:22.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96912 2024-01-16T15:09:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.634+0000 7f8f6b800640 1 Processor -- start 2024-01-16T15:09:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.635+0000 7f8f6b800640 1 -- start start 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.636+0000 7f8f6b800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6414b2c0 0x7f8f640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.636+0000 7f8f6b800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8f640cf7d0 0x7f8f64070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.636+0000 7f8f6b800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640cfbf0 con 0x7f8f64070ea0 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.636+0000 7f8f6b800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6414b6a0 con 0x7f8f6414b2c0 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.636+0000 7f8f6b800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f64071690 con 0x7f8f640cf7d0 2024-01-16T15:09:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6414b2c0 0x7f8f640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6414b2c0 0x7f8f640cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35828/0 (socket says 172.21.15.125:35828) 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 -- 172.21.15.125:0/4288178144 learned_addr learned my addr 172.21.15.125:0/4288178144 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f69800640 1 --1- 172.21.15.125:0/4288178144 >> v1:172.21.15.125:6789/0 conn(0x7f8f64070ea0 0x7f8f640b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52540/0 (socket says 172.21.15.125:52540) 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 -- 172.21.15.125:0/4288178144 >> v2:172.21.15.137:3300/0 conn(0x7f8f640cf7d0 msgr2=0x7f8f64070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 --2- 172.21.15.125:0/4288178144 >> v2:172.21.15.137:3300/0 conn(0x7f8f640cf7d0 0x7f8f64070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 -- 172.21.15.125:0/4288178144 >> v1:172.21.15.125:6789/0 conn(0x7f8f64070ea0 legacy=0x7f8f640b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.637+0000 7f8f68e00640 1 -- 172.21.15.125:0/4288178144 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64071970 con 0x7f8f6414b2c0 2024-01-16T15:09:22.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.638+0000 7f8f68e00640 1 --2- 172.21.15.125:0/4288178144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6414b2c0 0x7f8f640cf290 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8f640cfea0 tx=0x7f8f5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b526866513f6147 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.638+0000 7f8f63e00640 1 -- 172.21.15.125:0/4288178144 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54030860 con 0x7f8f6414b2c0 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.638+0000 7f8f63e00640 1 -- 172.21.15.125:0/4288178144 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540309c0 con 0x7f8f6414b2c0 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.638+0000 7f8f63e00640 1 -- 172.21.15.125:0/4288178144 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54030d50 con 0x7f8f6414b2c0 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 -- 172.21.15.125:0/4288178144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6414b2c0 msgr2=0x7f8f640cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 --2- 172.21.15.125:0/4288178144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6414b2c0 0x7f8f640cf290 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8f640cfea0 tx=0x7f8f5402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 -- 172.21.15.125:0/4288178144 shutdown_connections 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 --2- 172.21.15.125:0/4288178144 >> v2:172.21.15.137:3300/0 conn(0x7f8f640cf7d0 0x7f8f64070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 --2- 172.21.15.125:0/4288178144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6414b2c0 0x7f8f640cf290 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 -- 172.21.15.125:0/4288178144 >> 172.21.15.125:0/4288178144 conn(0x7f8f6406a800 msgr2=0x7f8f6406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 -- 172.21.15.125:0/4288178144 shutdown_connections 2024-01-16T15:09:22.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.639+0000 7f8f6b800640 1 -- 172.21.15.125:0/4288178144 wait complete. 2024-01-16T15:09:22.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.641+0000 7f8f6b800640 1 Processor -- start 2024-01-16T15:09:22.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.641+0000 7f8f6b800640 1 -- start start 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f640cf7d0 0x7f8f64145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6414b2c0 0x7f8f640cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64144420 con 0x7f8f64070ea0 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6415e220 con 0x7f8f640cf7d0 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f6b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6415e470 con 0x7f8f6414b2c0 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35838/0 (socket says 172.21.15.125:35838) 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.642+0000 7f8f68e00640 1 -- 172.21.15.125:0/2250531240 learned_addr learned my addr 172.21.15.125:0/2250531240 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:22.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f69800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6414b2c0 0x7f8f640cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f5bc00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f640cf7d0 0x7f8f64145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 -- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6414b2c0 msgr2=0x7f8f640cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6414b2c0 0x7f8f640cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 -- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f640cf7d0 msgr2=0x7f8f64145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f640cf7d0 0x7f8f64145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 -- 172.21.15.125:0/2250531240 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540094d0 con 0x7f8f64070ea0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f68e00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000c00 tx=0x7f8f54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54030830 con 0x7f8f64070ea0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54039d90 con 0x7f8f64070ea0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6415e750 con 0x7f8f64070ea0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54043630 con 0x7f8f64070ea0 2024-01-16T15:09:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.643+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6415ed10 con 0x7f8f64070ea0 2024-01-16T15:09:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.644+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640675d0 con 0x7f8f64070ea0 2024-01-16T15:09:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.646+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8f5404c3b0 con 0x7f8f64070ea0 2024-01-16T15:09:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.646+0000 7f8f61600640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 0x7f8f3c06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.647+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8f540aace0 con 0x7f8f64070ea0 2024-01-16T15:09:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.647+0000 7f8f5bc00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 0x7f8f3c06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.647+0000 7f8f5bc00640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 0x7f8f3c06e4a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f6415dfa0 tx=0x7f8f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.651+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f540775c0 con 0x7f8f64070ea0 2024-01-16T15:09:22.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.901+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96912", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f640cbc40 con 0x7f8f3c06bfe0 2024-01-16T15:09:22.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.936+0000 7f8f61600640 1 -- 172.21.15.125:0/2250531240 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8f4c00adc0 con 0x7f8f3c06bfe0 2024-01-16T15:09:22.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.939+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 msgr2=0x7f8f3c06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.939+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 0x7f8f3c06e4a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f6415dfa0 tx=0x7f8f4c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:22.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.939+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 msgr2=0x7f8f6415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.939+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000c00 tx=0x7f8f54004730 comp rx=0 tx=0).stop 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 shutdown_connections 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f6414b2c0 0x7f8f640cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f640cf7d0 0x7f8f64145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f3c06bfe0 0x7f8f3c06e4a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 --2- 172.21.15.125:0/2250531240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f64070ea0 0x7f8f6415c240 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 >> 172.21.15.125:0/2250531240 conn(0x7f8f6406a800 msgr2=0x7f8f64148ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.940+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 shutdown_connections 2024-01-16T15:09:22.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:22.941+0000 7f8f6b800640 1 -- 172.21.15.125:0/2250531240 wait complete. 2024-01-16T15:09:22.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone969117 2024-01-16T15:09:23.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.156+0000 7f7a0d200640 1 Processor -- start 2024-01-16T15:09:23.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.156+0000 7f7a0d200640 1 -- start start 2024-01-16T15:09:23.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.157+0000 7f7a0d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f7a0815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.157+0000 7f7a0d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7a080cf290 0x7f7a080cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.157+0000 7f7a0d200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a08070c40 con 0x7f7a080ae5a0 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.157+0000 7f7a0d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a08071450 con 0x7f7a080cf290 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.157+0000 7f7a0d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080cfc60 con 0x7f7a080b3ae0 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.158+0000 7f7a05e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7a080ae5a0 0x7f7a0806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52552/0 (socket says 172.21.15.125:52552) 2024-01-16T15:09:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.158+0000 7f7a05e00640 1 -- 172.21.15.125:0/2539593522 learned_addr learned my addr 172.21.15.125:0/2539593522 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.158+0000 7f7a06800640 1 --2- 172.21.15.125:0/2539593522 >> v2:172.21.15.137:3300/0 conn(0x7f7a080cf290 0x7f7a080cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.158+0000 7f79ffe00640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f7a0815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.158+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250356052 0 0) 0x7f7a08070c40 con 0x7f7a080ae5a0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e8003520 con 0x7f7a080ae5a0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1399274153 0 0) 0x7f79e8003520 con 0x7f7a080ae5a0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 msgr2=0x7f7a0815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f7a0815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 >> v2:172.21.15.137:3300/0 conn(0x7f7a080cf290 msgr2=0x7f7a080cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 --2- 172.21.15.125:0/2539593522 >> v2:172.21.15.137:3300/0 conn(0x7f7a080cf290 0x7f7a080cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0806c470 con 0x7f7a080ae5a0 2024-01-16T15:09:23.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f79f8001670 con 0x7f7a080ae5a0 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 >> v1:172.21.15.125:6789/0 conn(0x7f7a080ae5a0 legacy=0x7f7a0806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79e8003970 0x7f79e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f79e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79e8004d00 0x7f79e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a08071450 con 0x7f79e8004d00 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080cfc60 con 0x7f79e8003970 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e800b980 con 0x7f7a080b3ae0 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.159+0000 7f7a06800640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79e8004d00 0x7f79e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a05e00640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79e8003970 0x7f79e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f79ffe00640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f79e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a06800640 1 -- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 msgr2=0x7f79e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a06800640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080b3ae0 0x7f79e80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a06800640 1 -- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79e8003970 msgr2=0x7f79e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a06800640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79e8003970 0x7f79e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.160+0000 7f7a06800640 1 -- 172.21.15.125:0/2539593522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8003520 con 0x7f79e8004d00 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.161+0000 7f7a06800640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79e8004d00 0x7f79e8005180 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009fc0 tx=0x7f79e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf975965d4abe99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.161+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0002cc60 con 0x7f79e8004d00 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.161+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0000f040 con 0x7f79e8004d00 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.161+0000 7f7a05400640 1 -- 172.21.15.125:0/2539593522 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00034440 con 0x7f79e8004d00 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 -- 172.21.15.125:0/2539593522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79e8004d00 msgr2=0x7f79e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2539593522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79e8004d00 0x7f79e8005180 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009fc0 tx=0x7f79e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a06800640 1 -- 172.21.15.125:0/2539593522 reap_dead start 2024-01-16T15:09:23.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 -- 172.21.15.125:0/2539593522 shutdown_connections 2024-01-16T15:09:23.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 -- 172.21.15.125:0/2539593522 >> 172.21.15.125:0/2539593522 conn(0x7f7a0806a800 msgr2=0x7f7a0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 -- 172.21.15.125:0/2539593522 shutdown_connections 2024-01-16T15:09:23.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.162+0000 7f7a0d200640 1 -- 172.21.15.125:0/2539593522 wait complete. 2024-01-16T15:09:23.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.164+0000 7f7a0d200640 1 Processor -- start 2024-01-16T15:09:23.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.164+0000 7f7a0d200640 1 -- start start 2024-01-16T15:09:23.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a0814f910 0x7f7a08152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0814fcf0 0x7f7a081f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081f3260 con 0x7f7a081f36f0 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081f6ca0 con 0x7f7a0814fcf0 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a0d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081f6f80 con 0x7f7a0814f910 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35854/0 (socket says 172.21.15.125:35854) 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 -- 172.21.15.125:0/2509148328 learned_addr learned my addr 172.21.15.125:0/2509148328 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f79ffe00640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0814fcf0 0x7f7a081f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 -- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a0814f910 msgr2=0x7f7a08152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a0814f910 0x7f7a08152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 -- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0814fcf0 msgr2=0x7f7a081f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0814fcf0 0x7f7a081f6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.165+0000 7f7a06800640 1 -- 172.21.15.125:0/2509148328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a081f7260 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f7a06800640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7a00037040 tx=0x7f7a00009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00034d50 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00003990 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00003d40 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a081f2ef0 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.166+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081f3ba0 con 0x7f7a081f36f0 2024-01-16T15:09:23.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.167+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a081f3e80 con 0x7f7a081f36f0 2024-01-16T15:09:23.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.168+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7a0003b050 con 0x7f7a081f36f0 2024-01-16T15:09:23.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.169+0000 7f79fe000640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 0x7f79e006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.169+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f7a000a9be0 con 0x7f7a081f36f0 2024-01-16T15:09:23.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.169+0000 7f7a05e00640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 0x7f79e006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.170+0000 7f7a05e00640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 0x7f79e006e4a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f79f8000f60 tx=0x7f79f8000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.173+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a081f3e80 con 0x7f7a081f36f0 2024-01-16T15:09:23.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.389+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone969117", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a08071170 con 0x7f79e006bfe0 2024-01-16T15:09:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.423+0000 7f79fe000640 1 -- 172.21.15.125:0/2509148328 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f79f80048d0 con 0x7f79e006bfe0 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.425+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 msgr2=0x7f79e006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.425+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 0x7f79e006e4a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f79f8000f60 tx=0x7f79f8000ed0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.425+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 msgr2=0x7f7a08159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.425+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7a00037040 tx=0x7f7a00009750 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.425+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 shutdown_connections 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f79e006bfe0 0x7f79e006e4a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a081f36f0 0x7f7a08159590 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0814fcf0 0x7f7a081f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 --2- 172.21.15.125:0/2509148328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a0814f910 0x7f7a08152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 >> 172.21.15.125:0/2509148328 conn(0x7f7a0806a800 msgr2=0x7f7a080af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 shutdown_connections 2024-01-16T15:09:23.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.426+0000 7f7a0d200640 1 -- 172.21.15.125:0/2509148328 wait complete. 2024-01-16T15:09:23.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96913 2024-01-16T15:09:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.637+0000 7f1186c00640 1 Processor -- start 2024-01-16T15:09:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.637+0000 7f1186c00640 1 -- start start 2024-01-16T15:09:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.638+0000 7f1186c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f118013df80 0x7f118015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.638+0000 7f1186c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118006c470 con 0x7f11800cf9a0 2024-01-16T15:09:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.638+0000 7f1186c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118006c750 con 0x7f118013e360 2024-01-16T15:09:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f1186c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118006ca30 con 0x7f118013df80 2024-01-16T15:09:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f1184c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f11800cf9a0 0x7f11800cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52564/0 (socket says 172.21.15.125:52564) 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f1184c00640 1 -- 172.21.15.125:0/4001272442 learned_addr learned my addr 172.21.15.125:0/4001272442 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287357487 0 0) 0x7f118006c470 con 0x7f11800cf9a0 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f117fe00640 1 --2- 172.21.15.125:0/4001272442 >> v2:172.21.15.137:3300/0 conn(0x7f118013df80 0x7f118015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.639+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1160003520 con 0x7f11800cf9a0 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3914515654 0 0) 0x7f1160003520 con 0x7f11800cf9a0 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 >> v2:172.21.15.137:3300/0 conn(0x7f118013df80 msgr2=0x7f118015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 --2- 172.21.15.125:0/4001272442 >> v2:172.21.15.137:3300/0 conn(0x7f118013df80 0x7f118015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 >> v1:172.21.15.137:6789/0 conn(0x7f118013e360 legacy=0x7f11800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118006cd10 con 0x7f11800cf9a0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f11780019f0 con 0x7f11800cf9a0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 >> v1:172.21.15.125:6789/0 conn(0x7f11800cf9a0 legacy=0x7f11800cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 0x7f1160003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11600047c0 0x7f1160004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600051d0 0x7f116000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118006ca30 con 0x7f11600039c0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116000c100 con 0x7f11600051d0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116000c2c0 con 0x7f11600047c0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 0x7f1160003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 -- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11600047c0 msgr2=0x7f1160004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11600047c0 0x7f1160004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 -- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600051d0 msgr2=0x7f116000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600051d0 0x7f116000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f117fe00640 1 -- 172.21.15.125:0/4001272442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160003520 con 0x7f11600039c0 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.640+0000 7f1184c00640 1 -- 172.21.15.125:0/4001272442 reap_dead start 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.641+0000 7f117fe00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 0x7f1160003dc0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f117000ba00 tx=0x7f11600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4dccbca441f5557 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170032630 con 0x7f11600039c0 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170032790 con 0x7f11600039c0 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f117f400640 1 -- 172.21.15.125:0/4001272442 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170032b20 con 0x7f11600039c0 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 -- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 msgr2=0x7f1160003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 0x7f1160003dc0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f117000ba00 tx=0x7f11600057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 -- 172.21.15.125:0/4001272442 shutdown_connections 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 --2- 172.21.15.125:0/4001272442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11600039c0 0x7f1160003dc0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 -- 172.21.15.125:0/4001272442 >> 172.21.15.125:0/4001272442 conn(0x7f118006a800 msgr2=0x7f118006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 -- 172.21.15.125:0/4001272442 shutdown_connections 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.642+0000 7f1186c00640 1 -- 172.21.15.125:0/4001272442 wait complete. 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.644+0000 7f1186c00640 1 Processor -- start 2024-01-16T15:09:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.644+0000 7f1186c00640 1 -- start start 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600039c0 0x7f11801f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f118013d8d0 0x7f1180156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11801f4f00 con 0x7f1180154520 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1180154ef0 con 0x7f11600039c0 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1186c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1180156e80 con 0x7f118013d8d0 2024-01-16T15:09:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1184c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1184c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35886/0 (socket says 172.21.15.125:35886) 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.645+0000 7f1184c00640 1 -- 172.21.15.125:0/893050097 learned_addr learned my addr 172.21.15.125:0/893050097 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 -- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f118013d8d0 msgr2=0x7f1180156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f118013d8d0 0x7f1180156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 -- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600039c0 msgr2=0x7f11801f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f117fe00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600039c0 0x7f11801f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600039c0 0x7f11801f3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 -- 172.21.15.125:0/893050097 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118013dd50 con 0x7f1180154520 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1184c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f118015b6f0 tx=0x7f117800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117800a640 con 0x7f1180154520 2024-01-16T15:09:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117800a7a0 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117800ab30 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.646+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11801f4660 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.647+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11801f4c20 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.648+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11800675d0 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.648+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f117800ac90 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.649+0000 7f117cc00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 0x7f115806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.649+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f1178087e40 con 0x7f1180154520 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.650+0000 7f117fe00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 0x7f115806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.650+0000 7f117fe00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 0x7f115806e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f11600057a0 tx=0x7f1170030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.654+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1178054720 con 0x7f1180154520 2024-01-16T15:09:23.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.903+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96913", "target": ["mon-mgr", ""]}) v1 -- 0x7f11800cf290 con 0x7f115806bfb0 2024-01-16T15:09:23.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.941+0000 7f117cc00640 1 -- 172.21.15.125:0/893050097 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1170032980 con 0x7f115806bfb0 2024-01-16T15:09:23.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.945+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 msgr2=0x7f115806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.945+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 0x7f115806e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f11600057a0 tx=0x7f1170030000 comp rx=0 tx=0).stop 2024-01-16T15:09:23.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.945+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 msgr2=0x7f11801549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.945+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f118015b6f0 tx=0x7f117800ccd0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 shutdown_connections 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f115806bfb0 0x7f115806e470 secure :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f11600057a0 tx=0x7f1170030000 comp rx=0 tx=0).stop 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1180154520 0x7f11801549b0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f118013d8d0 0x7f1180156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 --2- 172.21.15.125:0/893050097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11600039c0 0x7f11801f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.946+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 >> 172.21.15.125:0/893050097 conn(0x7f118006a800 msgr2=0x7f118006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.947+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 shutdown_connections 2024-01-16T15:09:23.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:23.947+0000 7f1186c00640 1 -- 172.21.15.125:0/893050097 wait complete. 2024-01-16T15:09:23.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:23.954 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:09:24.022 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T15:09:24.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:09:24.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.252+0000 7fa162e00640 1 Processor -- start 2024-01-16T15:09:24.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.252+0000 7fa162e00640 1 -- start start 2024-01-16T15:09:24.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa15c0ae170 0x7fa15c15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 0x7fa15c15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c143bf0 con 0x7fa15c0ae170 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c15e140 con 0x7fa15c15d770 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa162e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c06c730 con 0x7fa15c143810 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa161000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa15c0ae170 0x7fa15c15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa161000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa15c0ae170 0x7fa15c15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35900/0 (socket says 172.21.15.125:35900) 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.253+0000 7fa161000640 1 -- 172.21.15.125:0/3820602538 learned_addr learned my addr 172.21.15.125:0/3820602538 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161000640 1 -- 172.21.15.125:0/3820602538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 msgr2=0x7fa15c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161a00640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa162400640 1 --2- 172.21.15.125:0/3820602538 >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 0x7fa15c15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161000640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161000640 1 -- 172.21.15.125:0/3820602538 >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 msgr2=0x7fa15c15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161000640 1 --2- 172.21.15.125:0/3820602538 >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 0x7fa15c15dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa161000640 1 -- 172.21.15.125:0/3820602538 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0ae5f0 con 0x7fa15c0ae170 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.254+0000 7fa162400640 1 --2- 172.21.15.125:0/3820602538 >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 0x7fa15c15dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.255+0000 7fa161000640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c0ae170 0x7fa15c15e8a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0acc20 tx=0x7fa14402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3411d5f6ada5b576 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.255+0000 7fa15be00640 1 -- 172.21.15.125:0/3820602538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1440317e0 con 0x7fa15c0ae170 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.255+0000 7fa15be00640 1 -- 172.21.15.125:0/3820602538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa144031940 con 0x7fa15c0ae170 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.255+0000 7fa15be00640 1 -- 172.21.15.125:0/3820602538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa144031cd0 con 0x7fa15c0ae170 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 -- 172.21.15.125:0/3820602538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c0ae170 msgr2=0x7fa15c15e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c0ae170 0x7fa15c15e8a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0acc20 tx=0x7fa14402b880 comp rx=0 tx=0).stop 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 -- 172.21.15.125:0/3820602538 shutdown_connections 2024-01-16T15:09:24.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 --2- 172.21.15.125:0/3820602538 >> v2:172.21.15.137:3300/0 conn(0x7fa15c15d770 0x7fa15c15dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c0ae170 0x7fa15c15e8a0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 --2- 172.21.15.125:0/3820602538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 -- 172.21.15.125:0/3820602538 >> 172.21.15.125:0/3820602538 conn(0x7fa15c06a7d0 msgr2=0x7fa15c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 -- 172.21.15.125:0/3820602538 shutdown_connections 2024-01-16T15:09:24.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.256+0000 7fa162e00640 1 -- 172.21.15.125:0/3820602538 wait complete. 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.258+0000 7fa162e00640 1 Processor -- start 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.258+0000 7fa162e00640 1 -- start start 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa15c0ae170 0x7fa15c0b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0afa50 con 0x7fa15c15d770 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0b09e0 con 0x7fa15c0ae170 2024-01-16T15:09:24.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c1f3710 con 0x7fa15c143810 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35910/0 (socket says 172.21.15.125:35910) 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 -- 172.21.15.125:0/1368779910 learned_addr learned my addr 172.21.15.125:0/1368779910 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 -- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 msgr2=0x7fa15c15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 -- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa15c0ae170 msgr2=0x7fa15c0b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa15c0ae170 0x7fa15c0b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.259+0000 7fa162400640 1 -- 172.21.15.125:0/1368779910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1440094d0 con 0x7fa15c15d770 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa162400640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa15400cd50 tx=0x7fa15400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15400f680 con 0x7fa15c15d770 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15400f800 con 0x7fa15c15d770 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15400fbb0 con 0x7fa15c15d770 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c1f39f0 con 0x7fa15c15d770 2024-01-16T15:09:24.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.260+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c0aff80 con 0x7fa15c15d770 2024-01-16T15:09:24.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.261+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c1f3be0 con 0x7fa15c15d770 2024-01-16T15:09:24.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.262+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa15400b040 con 0x7fa15c15d770 2024-01-16T15:09:24.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.263+0000 7fa159600640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 0x7fa13406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.263+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fa154088f90 con 0x7fa15c15d770 2024-01-16T15:09:24.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.263+0000 7fa161a00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 0x7fa13406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.264+0000 7fa161a00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 0x7fa13406e470 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa14c006b10 tx=0x7fa14c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.271+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa15c1f3be0 con 0x7fa15c15d770 2024-01-16T15:09:24.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:24.509+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fa15c160540 con 0x7fa13406bfb0 2024-01-16T15:09:24.543 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T15:09:24.566+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:09:24.638 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:09:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.378+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fa15400fd10 con 0x7fa15c15d770 2024-01-16T15:09:27.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.592+0000 7fa159600640 1 -- 172.21.15.125:0/1368779910 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fa15c160540 con 0x7fa13406bfb0 2024-01-16T15:09:27.571 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:09:27.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.598+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 msgr2=0x7fa13406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.598+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 0x7fa13406e470 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa14c006b10 tx=0x7fa14c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:27.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.598+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 msgr2=0x7fa15c1f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.598+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa15400cd50 tx=0x7fa15400a650 comp rx=0 tx=0).stop 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 shutdown_connections 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa13406bfb0 0x7fa13406e470 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa15c15d770 0x7fa15c1f2c50 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa15c143810 0x7fa15c15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 --2- 172.21.15.125:0/1368779910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa15c0ae170 0x7fa15c0b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.599+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 >> 172.21.15.125:0/1368779910 conn(0x7fa15c06a7d0 msgr2=0x7fa15c13ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.600+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 shutdown_connections 2024-01-16T15:09:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:27.600+0000 7fa162e00640 1 -- 172.21.15.125:0/1368779910 wait complete. 2024-01-16T15:09:27.606 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:09:27.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:27.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:27.889 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:09:27.889 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:09:27.889 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:27.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:28.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.278+0000 7f20c2800640 1 Processor -- start 2024-01-16T15:09:28.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.279+0000 7f20c2800640 1 -- start start 2024-01-16T15:09:28.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20c2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 0x7f20bc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20c2800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc15d840 con 0x7f20bc06e380 2024-01-16T15:09:28.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20c2800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc072190 con 0x7f20bc0b30b0 2024-01-16T15:09:28.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20c2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc071490 con 0x7f20bc15d460 2024-01-16T15:09:28.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20baa00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f20bc06e380 0x7f20bc0b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40702/0 (socket says 172.21.15.125:40702) 2024-01-16T15:09:28.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20baa00640 1 -- 172.21.15.125:0/421461611 learned_addr learned my addr 172.21.15.125:0/421461611 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:28.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.281+0000 7f20bb400640 1 --2- 172.21.15.125:0/421461611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 0x7f20bc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.282+0000 7f20ba000640 1 -- 172.21.15.125:0/421461611 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976778577 0 0) 0x7f20bc072190 con 0x7f20bc0b30b0 2024-01-16T15:09:28.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.282+0000 7f20ba000640 1 -- 172.21.15.125:0/421461611 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209c003520 con 0x7f20bc0b30b0 2024-01-16T15:09:28.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.282+0000 7f20bb400640 1 -- 172.21.15.125:0/421461611 >> v1:172.21.15.137:6789/0 conn(0x7f20bc0b30b0 legacy=0x7f20bc06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.282+0000 7f20bb400640 1 -- 172.21.15.125:0/421461611 >> v1:172.21.15.125:6789/0 conn(0x7f20bc06e380 legacy=0x7f20bc0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.282+0000 7f20bb400640 1 -- 172.21.15.125:0/421461611 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc072470 con 0x7f20bc15d460 2024-01-16T15:09:28.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.284+0000 7f20bb400640 1 --2- 172.21.15.125:0/421461611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 0x7f20bc0ac8d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f20a40097d0 tx=0x7f20a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec17ffa086c2ca34 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.284+0000 7f20ba000640 1 -- 172.21.15.125:0/421461611 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4031830 con 0x7f20bc15d460 2024-01-16T15:09:28.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.284+0000 7f20ba000640 1 -- 172.21.15.125:0/421461611 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4031990 con 0x7f20bc15d460 2024-01-16T15:09:28.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.284+0000 7f20ba000640 1 -- 172.21.15.125:0/421461611 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4031d20 con 0x7f20bc15d460 2024-01-16T15:09:28.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.285+0000 7f20c2800640 1 -- 172.21.15.125:0/421461611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 msgr2=0x7f20bc0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.285+0000 7f20c2800640 1 --2- 172.21.15.125:0/421461611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 0x7f20bc0ac8d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f20a40097d0 tx=0x7f20a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.286+0000 7f20c2800640 1 -- 172.21.15.125:0/421461611 shutdown_connections 2024-01-16T15:09:28.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.286+0000 7f20c2800640 1 --2- 172.21.15.125:0/421461611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc15d460 0x7f20bc0ac8d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.286+0000 7f20c2800640 1 -- 172.21.15.125:0/421461611 >> 172.21.15.125:0/421461611 conn(0x7f20bc0b0be0 msgr2=0x7f20bc0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.286+0000 7f20c2800640 1 -- 172.21.15.125:0/421461611 shutdown_connections 2024-01-16T15:09:28.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.286+0000 7f20c2800640 1 -- 172.21.15.125:0/421461611 wait complete. 2024-01-16T15:09:28.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.288+0000 7f20c2800640 1 Processor -- start 2024-01-16T15:09:28.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.289+0000 7f20c2800640 1 -- start start 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.290+0000 7f20c2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20bc06e380 0x7f20bc0ac350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.290+0000 7f20c2800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 0x7f20bc0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.290+0000 7f20c2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20c2800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0ab2b0 con 0x7f20bc0b30b0 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20bb400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20bc06e380 0x7f20bc0ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:33258/0 (socket says 172.21.15.125:33258) 2024-01-16T15:09:28.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20bbe00640 1 -- 172.21.15.125:0/1180307637 learned_addr learned my addr 172.21.15.125:0/1180307637 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:28.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0ab590 con 0x7f20bc06e380 2024-01-16T15:09:28.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0ab870 con 0x7f20bc0aaa30 2024-01-16T15:09:28.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.291+0000 7f20baa00640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 0x7f20bc0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.292+0000 7f20baa00640 1 -- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 msgr2=0x7f20bc147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.292+0000 7f20baa00640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.293+0000 7f20baa00640 1 -- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20bc06e380 msgr2=0x7f20bc0ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.293+0000 7f20baa00640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20bc06e380 0x7f20bc0ac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.293+0000 7f20baa00640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4009400 con 0x7f20bc0b30b0 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.293+0000 7f20bbe00640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:28.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.294+0000 7f20baa00640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 0x7f20bc0aa4f0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f20b400cae0 tx=0x7f20b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.294+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400abe0 con 0x7f20bc0b30b0 2024-01-16T15:09:28.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.294+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b400ad40 con 0x7f20bc0b30b0 2024-01-16T15:09:28.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.295+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc0abb50 con 0x7f20bc0b30b0 2024-01-16T15:09:28.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.295+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc0ac890 con 0x7f20bc0b30b0 2024-01-16T15:09:28.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.296+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4012600 con 0x7f20bc0b30b0 2024-01-16T15:09:28.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.297+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c005250 con 0x7f20bc0b30b0 2024-01-16T15:09:28.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.298+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f20b4007070 con 0x7f20bc0b30b0 2024-01-16T15:09:28.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.300+0000 7f20b3400640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 0x7f209406e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.300+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f20b40885b0 con 0x7f20bc0b30b0 2024-01-16T15:09:28.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.301+0000 7f20bb400640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 0x7f209406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.302+0000 7f20bb400640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 0x7f209406e460 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f20a4036000 tx=0x7f20a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.309+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20b408e030 con 0x7f20bc0b30b0 2024-01-16T15:09:28.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.544+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f208c0057d0 con 0x7f20bc0b30b0 2024-01-16T15:09:28.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.547+0000 7f20b3400640 1 -- 172.21.15.125:0/1180307637 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13033 (secure 0 0 0) 0x7f20b4054ce0 con 0x7f20bc0b30b0 2024-01-16T15:09:28.524 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:28.524 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:27.581384+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/22640034":"2024-01-17T15:09:24.566983+0000","172.21.15.125:6836/22640034":"2024-01-17T15:09:24.566983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:28.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 msgr2=0x7f209406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 0x7f209406e460 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f20a4036000 tx=0x7f20a403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:28.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 msgr2=0x7f20bc0aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 0x7f20bc0aa4f0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f20b400cae0 tx=0x7f20b400a060 comp rx=0 tx=0).stop 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 shutdown_connections 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20bc0aaa30 0x7f20bc147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f209406bfa0 0x7f209406e460 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20bc0b30b0 0x7f20bc0aa4f0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 --2- 172.21.15.125:0/1180307637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20bc06e380 0x7f20bc0ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.549+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 >> 172.21.15.125:0/1180307637 conn(0x7f20bc0b0be0 msgr2=0x7f20bc15b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.550+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 shutdown_connections 2024-01-16T15:09:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.550+0000 7f20c2800640 1 -- 172.21.15.125:0/1180307637 wait complete. 2024-01-16T15:09:28.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:28.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:28.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.754+0000 7facab000640 1 Processor -- start 2024-01-16T15:09:28.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.755+0000 7facab000640 1 -- start start 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.756+0000 7facab000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 0x7faca415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.756+0000 7facab000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca4070c40 con 0x7faca40cf290 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.756+0000 7facab000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca406c470 con 0x7faca40ae5a0 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.756+0000 7facab000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca4071450 con 0x7faca40b3ae0 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca9000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faca40cf290 0x7faca40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40714/0 (socket says 172.21.15.125:40714) 2024-01-16T15:09:28.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca9000640 1 -- 172.21.15.125:0/2746881159 learned_addr learned my addr 172.21.15.125:0/2746881159 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca3400640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 0x7faca415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564992745 0 0) 0x7faca4070c40 con 0x7faca40cf290 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac84003520 con 0x7faca40cf290 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666141494 0 0) 0x7fac84003520 con 0x7faca40cf290 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 msgr2=0x7faca415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 0x7faca415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 >> v1:172.21.15.137:6789/0 conn(0x7faca40ae5a0 legacy=0x7faca406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca406c750 con 0x7faca40cf290 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fac9c0019f0 con 0x7faca40cf290 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 >> v1:172.21.15.125:6789/0 conn(0x7faca40cf290 legacy=0x7faca40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840039a0 0x7fac84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.757+0000 7faca2a00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac840047c0 0x7fac84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca2a00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 0x7fac84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca4071450 con 0x7fac840047c0 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8400bac0 con 0x7fac840039a0 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8400bc80 con 0x7faca40b3ae0 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac840047c0 0x7fac84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3e00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840039a0 0x7fac84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 -- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 msgr2=0x7fac84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca40b3ae0 0x7fac84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 -- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840039a0 msgr2=0x7fac84003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840039a0 0x7fac84003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3400640 1 -- 172.21.15.125:0/2746881159 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84003520 con 0x7fac840047c0 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.758+0000 7faca3e00640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840039a0 0x7fac84003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.759+0000 7faca3400640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac840047c0 0x7fac84004c40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009a90 tx=0x7fac840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2222134d4df30c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.759+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c02c6b0 con 0x7fac840047c0 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.759+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c02c810 con 0x7fac840047c0 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.759+0000 7faca2a00640 1 -- 172.21.15.125:0/2746881159 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c02cba0 con 0x7fac840047c0 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 -- 172.21.15.125:0/2746881159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac840047c0 msgr2=0x7fac84004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 --2- 172.21.15.125:0/2746881159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac840047c0 0x7fac84004c40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009a90 tx=0x7fac840057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7faca9000640 1 -- 172.21.15.125:0/2746881159 reap_dead start 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 -- 172.21.15.125:0/2746881159 shutdown_connections 2024-01-16T15:09:28.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 -- 172.21.15.125:0/2746881159 >> 172.21.15.125:0/2746881159 conn(0x7faca406a800 msgr2=0x7faca406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 -- 172.21.15.125:0/2746881159 shutdown_connections 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.760+0000 7facab000640 1 -- 172.21.15.125:0/2746881159 wait complete. 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.762+0000 7facab000640 1 Processor -- start 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.762+0000 7facab000640 1 -- start start 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840047c0 0x7faca415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca415c4f0 0x7faca41f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca41f6ca0 con 0x7faca41f6820 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca41f2b80 con 0x7fac840047c0 2024-01-16T15:09:28.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7facab000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca415cb20 con 0x7faca415c4f0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46282/0 (socket says 172.21.15.125:46282) 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 -- 172.21.15.125:0/4103509110 learned_addr learned my addr 172.21.15.125:0/4103509110 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca3e00640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840047c0 0x7faca415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca3400640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca415c4f0 0x7faca41f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 -- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca415c4f0 msgr2=0x7faca41f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca415c4f0 0x7faca41f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 -- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840047c0 msgr2=0x7faca415bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840047c0 0x7faca415bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.763+0000 7faca9000640 1 -- 172.21.15.125:0/4103509110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca415ce00 con 0x7faca41f6820 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7faca9000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7faca40cf670 tx=0x7fac9c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c00a610 con 0x7faca41f6820 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c00a770 con 0x7faca41f6820 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca4156950 con 0x7faca41f6820 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c00ab20 con 0x7faca41f6820 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.764+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca4156f10 con 0x7faca41f6820 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.765+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca415c8d0 con 0x7faca41f6820 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.766+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fac9c00ac80 con 0x7faca41f6820 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.767+0000 7fac9be00640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 0x7fac7c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.767+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fac9c0875d0 con 0x7faca41f6820 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.767+0000 7faca3e00640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 0x7fac7c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.768+0000 7faca3e00640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 0x7fac7c06e410 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fac94001750 tx=0x7fac94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.771+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac9c053c10 con 0x7faca41f6820 2024-01-16T15:09:28.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.945+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faca40675d0 con 0x7faca41f6820 2024-01-16T15:09:28.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.947+0000 7fac9be00640 1 -- 172.21.15.125:0/4103509110 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac9c057230 con 0x7faca41f6820 2024-01-16T15:09:28.924 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.949+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 msgr2=0x7fac7c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.949+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 0x7fac7c06e410 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fac94001750 tx=0x7fac94009040 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.949+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 msgr2=0x7faca414b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.949+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7faca40cf670 tx=0x7fac9c00ccd0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 shutdown_connections 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac7c06bf50 0x7fac7c06e410 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faca41f6820 0x7faca414b050 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faca415c4f0 0x7faca41f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 --2- 172.21.15.125:0/4103509110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac840047c0 0x7faca415bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 >> 172.21.15.125:0/4103509110 conn(0x7faca406a800 msgr2=0x7faca415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 shutdown_connections 2024-01-16T15:09:28.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:28.950+0000 7facab000640 1 -- 172.21.15.125:0/4103509110 wait complete. 2024-01-16T15:09:28.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:09:29.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.163+0000 7fc9d9000640 1 Processor -- start 2024-01-16T15:09:29.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.163+0000 7fc9d9000640 1 -- start start 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d9000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc9d4146650 0x7fc9d40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d9000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9d415e030 0x7fc9d415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d415dba0 con 0x7fc9d40726f0 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d9000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d40cd130 con 0x7fc9d415e030 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d9000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d406f800 con 0x7fc9d4146650 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9d415e030 0x7fc9d415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9d415e030 0x7fc9d415e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46286/0 (socket says 172.21.15.125:46286) 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d3e00640 1 -- 172.21.15.125:0/2919445617 learned_addr learned my addr 172.21.15.125:0/2919445617 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.165+0000 7fc9d3400640 1 --2- 172.21.15.125:0/2919445617 >> v2:172.21.15.137:3300/0 conn(0x7fc9d4146650 0x7fc9d40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.166+0000 7fc9d3e00640 1 -- 172.21.15.125:0/2919445617 >> v2:172.21.15.137:3300/0 conn(0x7fc9d4146650 msgr2=0x7fc9d40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.166+0000 7fc9d3e00640 1 --2- 172.21.15.125:0/2919445617 >> v2:172.21.15.137:3300/0 conn(0x7fc9d4146650 0x7fc9d40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.166+0000 7fc9d3e00640 1 -- 172.21.15.125:0/2919445617 >> v1:172.21.15.137:6789/0 conn(0x7fc9d40726f0 legacy=0x7fc9d40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.166+0000 7fc9d3e00640 1 -- 172.21.15.125:0/2919445617 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d406f240 con 0x7fc9d415e030 2024-01-16T15:09:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d3e00640 1 --2- 172.21.15.125:0/2919445617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d415e4e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4072c70 tx=0x7fc9cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7994d44a4da941fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d2000640 1 -- 172.21.15.125:0/2919445617 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc031870 con 0x7fc9d415e030 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d2000640 1 -- 172.21.15.125:0/2919445617 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc0319d0 con 0x7fc9d415e030 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d2000640 1 -- 172.21.15.125:0/2919445617 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc031d60 con 0x7fc9d415e030 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d9000640 1 -- 172.21.15.125:0/2919445617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 msgr2=0x7fc9d415e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d9000640 1 --2- 172.21.15.125:0/2919445617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d415e4e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4072c70 tx=0x7fc9cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d9000640 1 -- 172.21.15.125:0/2919445617 shutdown_connections 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d9000640 1 --2- 172.21.15.125:0/2919445617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d415e4e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.167+0000 7fc9d9000640 1 --2- 172.21.15.125:0/2919445617 >> v2:172.21.15.137:3300/0 conn(0x7fc9d4146650 0x7fc9d40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.168+0000 7fc9d9000640 1 -- 172.21.15.125:0/2919445617 >> 172.21.15.125:0/2919445617 conn(0x7fc9d40b1d40 msgr2=0x7fc9d40b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.168+0000 7fc9d9000640 1 -- 172.21.15.125:0/2919445617 shutdown_connections 2024-01-16T15:09:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.168+0000 7fc9d9000640 1 -- 172.21.15.125:0/2919445617 wait complete. 2024-01-16T15:09:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.169+0000 7fc9d9000640 1 Processor -- start 2024-01-16T15:09:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.169+0000 7fc9d9000640 1 -- start start 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9d40726f0 0x7fc9d41f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9d4146650 0x7fc9d41386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d40b4e90 con 0x7fc9d415e030 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d4138c30 con 0x7fc9d4146650 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.170+0000 7fc9d9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d40b3e80 con 0x7fc9d40726f0 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46288/0 (socket says 172.21.15.125:46288) 2024-01-16T15:09:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 -- 172.21.15.125:0/1664139452 learned_addr learned my addr 172.21.15.125:0/1664139452 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3400640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9d40726f0 0x7fc9d41f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d2a00640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9d4146650 0x7fc9d41386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 -- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9d40726f0 msgr2=0x7fc9d41f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9d40726f0 0x7fc9d41f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 -- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9d4146650 msgr2=0x7fc9d41386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9d4146650 0x7fc9d41386f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 -- 172.21.15.125:0/1664139452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc0094d0 con 0x7fc9d415e030 2024-01-16T15:09:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.171+0000 7fc9d3e00640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc02b950 tx=0x7fc9cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.172+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc03ac60 con 0x7fc9d415e030 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.172+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc03adc0 con 0x7fc9d415e030 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.172+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc044630 con 0x7fc9d415e030 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.172+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d40b4a00 con 0x7fc9d415e030 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.172+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d40b43b0 con 0x7fc9d415e030 2024-01-16T15:09:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.173+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d40b4b90 con 0x7fc9d415e030 2024-01-16T15:09:29.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.174+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc9cc044790 con 0x7fc9d415e030 2024-01-16T15:09:29.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.174+0000 7fc9cb400640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 0x7fc9ac06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.175+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc9cc02d040 con 0x7fc9d415e030 2024-01-16T15:09:29.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.175+0000 7fc9d3400640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 0x7fc9ac06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.175+0000 7fc9d3400640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 0x7fc9ac06e410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006fd0 tx=0x7fc9c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.179+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9cc0776d0 con 0x7fc9d415e030 2024-01-16T15:09:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.366+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc9d40afb30 con 0x7fc9d415e030 2024-01-16T15:09:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.367+0000 7fc9cb400640 1 -- 172.21.15.125:0/1664139452 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+120 (secure 0 0 0) 0x7fc9cc07acf0 con 0x7fc9d415e030 2024-01-16T15:09:29.343 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/22640034 2024-01-17T15:09:24.566983+0000 2024-01-16T15:09:29.344 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/22640034 2024-01-17T15:09:24.566983+0000 2024-01-16T15:09:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 msgr2=0x7fc9ac06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 0x7fc9ac06e410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006fd0 tx=0x7fc9c4008040 comp rx=0 tx=0).stop 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 msgr2=0x7fc9d4160120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc02b950 tx=0x7fc9cc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 shutdown_connections 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9ac06bf50 0x7fc9ac06e410 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9d415e030 0x7fc9d4160120 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9d4146650 0x7fc9d41386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 --2- 172.21.15.125:0/1664139452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9d40726f0 0x7fc9d41f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 >> 172.21.15.125:0/1664139452 conn(0x7fc9d40b1d40 msgr2=0x7fc9d41611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 shutdown_connections 2024-01-16T15:09:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.369+0000 7fc9d9000640 1 -- 172.21.15.125:0/1664139452 wait complete. 2024-01-16T15:09:29.363 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:09:29.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:29.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:29.448 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:09:29.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:29.448 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:09:29.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[63461]: fuse finished with error 0 and tester_r 0 2024-01-16T15:09:29.683 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:09:29.683 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:29.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.974+0000 7f69f3600640 1 Processor -- start 2024-01-16T15:09:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.974+0000 7f69f3600640 1 -- start start 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f3600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f69ec0b2ef0 0x7f69ec071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f3600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f3600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ec0cc5d0 con 0x7f69ec15aa60 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f3600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ec06d570 con 0x7f69ec0b2ef0 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f3600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec071cf0 con 0x7f69ec0718a0 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f0c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f69ec15aa60 0x7f69ec15ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40724/0 (socket says 172.21.15.125:40724) 2024-01-16T15:09:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.976+0000 7f69f0c00640 1 -- 172.21.15.125:0/1197569337 learned_addr learned my addr 172.21.15.125:0/1197569337 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69ebe00640 1 --2- 172.21.15.125:0/1197569337 >> v2:172.21.15.137:3300/0 conn(0x7f69ec0b2ef0 0x7f69ec071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69f1600640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569562888 0 0) 0x7f69ec0cc5d0 con 0x7f69ec15aa60 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc003520 con 0x7f69ec15aa60 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1513831773 0 0) 0x7f69cc003520 con 0x7f69ec15aa60 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 msgr2=0x7f69ec06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69ec06d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 >> v2:172.21.15.137:3300/0 conn(0x7f69ec0b2ef0 msgr2=0x7f69ec071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 --2- 172.21.15.125:0/1197569337 >> v2:172.21.15.137:3300/0 conn(0x7f69ec0b2ef0 0x7f69ec071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec0b3470 con 0x7f69ec15aa60 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f69dc0019f0 con 0x7f69ec15aa60 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 >> v1:172.21.15.125:6789/0 conn(0x7f69ec15aa60 legacy=0x7f69ec15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69cc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69cc003fa0 0x7f69cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc004d00 0x7f69cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec06d570 con 0x7f69cc003fa0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec071cf0 con 0x7f69cc004d00 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.977+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00dd50 con 0x7f69ec0718a0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69cc003fa0 0x7f69cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69f0c00640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69cc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69f1600640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc004d00 0x7f69cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 -- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 msgr2=0x7f69cc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec0718a0 0x7f69cc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 -- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc004d00 msgr2=0x7f69cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc004d00 0x7f69cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69ebe00640 1 -- 172.21.15.125:0/1197569337 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc003520 con 0x7f69cc003fa0 2024-01-16T15:09:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.978+0000 7f69f1600640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc004d00 0x7f69cc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.979+0000 7f69ebe00640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69cc003fa0 0x7f69cc0047c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f69d400ba30 tx=0x7f69cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e88db0c7ba870bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.979+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4032680 con 0x7f69cc003fa0 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.979+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d40327e0 con 0x7f69cc003fa0 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.979+0000 7f69eb400640 1 -- 172.21.15.125:0/1197569337 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4032b70 con 0x7f69cc003fa0 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 -- 172.21.15.125:0/1197569337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69cc003fa0 msgr2=0x7f69cc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 --2- 172.21.15.125:0/1197569337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69cc003fa0 0x7f69cc0047c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f69d400ba30 tx=0x7f69cc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f1600640 1 -- 172.21.15.125:0/1197569337 reap_dead start 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 -- 172.21.15.125:0/1197569337 shutdown_connections 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 -- 172.21.15.125:0/1197569337 >> 172.21.15.125:0/1197569337 conn(0x7f69ec0b1bf0 msgr2=0x7f69ec0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 -- 172.21.15.125:0/1197569337 shutdown_connections 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.980+0000 7f69f3600640 1 -- 172.21.15.125:0/1197569337 wait complete. 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.982+0000 7f69f3600640 1 Processor -- start 2024-01-16T15:09:29.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.982+0000 7f69f3600640 1 -- start start 2024-01-16T15:09:29.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 0x7f69ec148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 0x7f69ec14d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec15ca60 con 0x7f69ec148150 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec148710 con 0x7f69cc003fa0 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f3600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec15ccb0 con 0x7f69ec14b690 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46312/0 (socket says 172.21.15.125:46312) 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69ebe00640 1 -- 172.21.15.125:0/4153940816 learned_addr learned my addr 172.21.15.125:0/4153940816 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f0c00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 0x7f69ec148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.983+0000 7f69f1600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 0x7f69ec14d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 -- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 msgr2=0x7f69ec14d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 0x7f69ec14d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 -- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 msgr2=0x7f69ec148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 0x7f69ec148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 -- 172.21.15.125:0/4153940816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec15cf90 con 0x7f69ec148150 2024-01-16T15:09:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69f0c00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 0x7f69ec148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69f1600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 0x7f69ec14d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69ebe00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f69d402bf00 tx=0x7f69d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4030a60 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d4030bc0 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d403b420 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec15d270 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.984+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec14bb10 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.985+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec0675d0 con 0x7f69ec148150 2024-01-16T15:09:29.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.986+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f69d404d3f0 con 0x7f69ec148150 2024-01-16T15:09:29.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.988+0000 7f69e8c00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 0x7f69c406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.988+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69d40aada0 con 0x7f69ec148150 2024-01-16T15:09:29.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.988+0000 7f69f0c00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 0x7f69c406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.989+0000 7f69f0c00640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 0x7f69c406e410 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69dc0013d0 tx=0x7f69dc0020f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:29.992+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69d4077440 con 0x7f69ec148150 2024-01-16T15:09:30.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.184+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69ec14bdf0 con 0x7f69ec148150 2024-01-16T15:09:30.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.186+0000 7f69e8c00640 1 -- 172.21.15.125:0/4153940816 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13033 (secure 0 0 0) 0x7f69d407aa60 con 0x7f69ec148150 2024-01-16T15:09:30.163 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:30.163 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:27.581384+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/22640034":"2024-01-17T15:09:24.566983+0000","172.21.15.125:6836/22640034":"2024-01-17T15:09:24.566983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:30.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 msgr2=0x7f69c406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 0x7f69c406e410 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69dc0013d0 tx=0x7f69dc0020f0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 msgr2=0x7f69ec15c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f69d402bf00 tx=0x7f69d4004730 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 shutdown_connections 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69ec14b690 0x7f69ec14d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f69c406bf50 0x7f69c406e410 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ec148150 0x7f69ec15c100 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 --2- 172.21.15.125:0/4153940816 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69cc003fa0 0x7f69ec148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 >> 172.21.15.125:0/4153940816 conn(0x7f69ec0b1bf0 msgr2=0x7f69ec06a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 shutdown_connections 2024-01-16T15:09:30.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.189+0000 7f69f3600640 1 -- 172.21.15.125:0/4153940816 wait complete. 2024-01-16T15:09:30.183 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:30.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:30.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.417+0000 7f283b000640 1 Processor -- start 2024-01-16T15:09:30.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.417+0000 7f283b000640 1 -- start start 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f283b000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28340b3ae0 0x7f283415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f283b000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2834070c40 con 0x7f28340cf290 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f283b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283406c470 con 0x7f28340ae5a0 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f283b000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2834071450 con 0x7f28340b3ae0 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f2833400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28340b3ae0 0x7f283415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f2833400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28340b3ae0 0x7f283415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46322/0 (socket says 172.21.15.125:46322) 2024-01-16T15:09:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.419+0000 7f2833400640 1 -- 172.21.15.125:0/4146533425 learned_addr learned my addr 172.21.15.125:0/4146533425 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.420+0000 7f2833400640 1 -- 172.21.15.125:0/4146533425 >> v1:172.21.15.137:6789/0 conn(0x7f28340ae5a0 legacy=0x7f283406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.420+0000 7f2833400640 1 -- 172.21.15.125:0/4146533425 >> v1:172.21.15.125:6789/0 conn(0x7f28340cf290 legacy=0x7f28340b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.420+0000 7f2833400640 1 -- 172.21.15.125:0/4146533425 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283406c750 con 0x7f28340b3ae0 2024-01-16T15:09:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.421+0000 7f2833400640 1 --2- 172.21.15.125:0/4146533425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340b3ae0 0x7f283415e430 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c02bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994908c3ae9b3d6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.421+0000 7f2832a00640 1 -- 172.21.15.125:0/4146533425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c02d870 con 0x7f28340b3ae0 2024-01-16T15:09:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.421+0000 7f2832a00640 1 -- 172.21.15.125:0/4146533425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c02d9d0 con 0x7f28340b3ae0 2024-01-16T15:09:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.421+0000 7f2832a00640 1 -- 172.21.15.125:0/4146533425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c02dd60 con 0x7f28340b3ae0 2024-01-16T15:09:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 -- 172.21.15.125:0/4146533425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340b3ae0 msgr2=0x7f283415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 --2- 172.21.15.125:0/4146533425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340b3ae0 0x7f283415e430 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c02bab0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 -- 172.21.15.125:0/4146533425 shutdown_connections 2024-01-16T15:09:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 --2- 172.21.15.125:0/4146533425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340b3ae0 0x7f283415e430 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 -- 172.21.15.125:0/4146533425 >> 172.21.15.125:0/4146533425 conn(0x7f283406a800 msgr2=0x7f283406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 -- 172.21.15.125:0/4146533425 shutdown_connections 2024-01-16T15:09:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.422+0000 7f283b000640 1 -- 172.21.15.125:0/4146533425 wait complete. 2024-01-16T15:09:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.424+0000 7f283b000640 1 Processor -- start 2024-01-16T15:09:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.424+0000 7f283b000640 1 -- start start 2024-01-16T15:09:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28340ae5a0 0x7f28341459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28340b3ae0 0x7f2834157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834151020 con 0x7f28340cf290 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834152fe0 con 0x7f28340ae5a0 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f283b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2834158630 con 0x7f28340b3ae0 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f2839000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f2839000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46336/0 (socket says 172.21.15.125:46336) 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.425+0000 7f2839000640 1 -- 172.21.15.125:0/2792506983 learned_addr learned my addr 172.21.15.125:0/2792506983 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2833e00640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28340ae5a0 0x7f28341459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2833400640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28340b3ae0 0x7f2834157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 -- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28340b3ae0 msgr2=0x7f2834157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28340b3ae0 0x7f2834157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 -- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28340ae5a0 msgr2=0x7f28341459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28340ae5a0 0x7f28341459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 -- 172.21.15.125:0/2792506983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0094d0 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f2839000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f282c00c6b0 tx=0x7f282c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c003b70 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.426+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c003cf0 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.427+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c005310 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.427+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2834150680 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.427+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2834150c40 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.428+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834151160 con 0x7f28340cf290 2024-01-16T15:09:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.429+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f282c005470 con 0x7f28340cf290 2024-01-16T15:09:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.430+0000 7f282be00640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 0x7f280c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.430+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f282c088380 con 0x7f28340cf290 2024-01-16T15:09:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.430+0000 7f2833e00640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 0x7f280c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.430+0000 7f2833e00640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 0x7f280c06e3e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2824000f60 tx=0x7f2824009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.434+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2834151160 con 0x7f28340cf290 2024-01-16T15:09:30.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.626+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2834151160 con 0x7f28340cf290 2024-01-16T15:09:30.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.626+0000 7f282be00640 1 -- 172.21.15.125:0/2792506983 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2834151160 con 0x7f28340cf290 2024-01-16T15:09:30.603 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:09:30.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 msgr2=0x7f280c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 0x7f280c06e3e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2824000f60 tx=0x7f2824009040 comp rx=0 tx=0).stop 2024-01-16T15:09:30.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 msgr2=0x7f28340afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f282c00c6b0 tx=0x7f282c00cb70 comp rx=0 tx=0).stop 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 shutdown_connections 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f280c06bf20 0x7f280c06e3e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28340cf290 0x7f28340afaa0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28340b3ae0 0x7f2834157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 --2- 172.21.15.125:0/2792506983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28340ae5a0 0x7f28341459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 >> 172.21.15.125:0/2792506983 conn(0x7f283406a800 msgr2=0x7f28340b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 shutdown_connections 2024-01-16T15:09:30.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.629+0000 7f283b000640 1 -- 172.21.15.125:0/2792506983 wait complete. 2024-01-16T15:09:30.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:09:30.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.857+0000 7f9355e00640 1 Processor -- start 2024-01-16T15:09:30.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.857+0000 7f9355e00640 1 -- start start 2024-01-16T15:09:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.858+0000 7f9355e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.858+0000 7f9355e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.858+0000 7f9355e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93500b1ee0 con 0x7f9350143ae0 2024-01-16T15:09:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.858+0000 7f9355e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500b2780 con 0x7f935015e1e0 2024-01-16T15:09:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.858+0000 7f9355e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935015e990 con 0x7f93500cf9a0 2024-01-16T15:09:30.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.859+0000 7f934f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.859+0000 7f934e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.859+0000 7f934e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:33344/0 (socket says 172.21.15.125:33344) 2024-01-16T15:09:30.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.859+0000 7f934f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45490/0 (socket says 172.21.15.125:45490) 2024-01-16T15:09:30.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.859+0000 7f934e000640 1 -- 172.21.15.125:0/749857728 learned_addr learned my addr 172.21.15.125:0/749857728 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:30.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.860+0000 7f934e000640 1 -- 172.21.15.125:0/749857728 >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 msgr2=0x7f93500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.860+0000 7f934e000640 1 --2- 172.21.15.125:0/749857728 >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.860+0000 7f934e000640 1 -- 172.21.15.125:0/749857728 >> v1:172.21.15.137:6789/0 conn(0x7f9350143ae0 legacy=0x7f93500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.860+0000 7f934e000640 1 -- 172.21.15.125:0/749857728 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500b2130 con 0x7f93500cf9a0 2024-01-16T15:09:30.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.860+0000 7f934f400640 1 --2- 172.21.15.125:0/749857728 >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:30.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.861+0000 7f934e000640 1 --2- 172.21.15.125:0/749857728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9338009870 tx=0x7f933802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dfb8b4182f977e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.861+0000 7f934d600640 1 -- 172.21.15.125:0/749857728 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338030840 con 0x7f93500cf9a0 2024-01-16T15:09:30.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.861+0000 7f934d600640 1 -- 172.21.15.125:0/749857728 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93380309a0 con 0x7f93500cf9a0 2024-01-16T15:09:30.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f934d600640 1 -- 172.21.15.125:0/749857728 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338030d30 con 0x7f93500cf9a0 2024-01-16T15:09:30.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 -- 172.21.15.125:0/749857728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 msgr2=0x7f935015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 --2- 172.21.15.125:0/749857728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9338009870 tx=0x7f933802b980 comp rx=0 tx=0).stop 2024-01-16T15:09:30.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 -- 172.21.15.125:0/749857728 shutdown_connections 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 --2- 172.21.15.125:0/749857728 >> v2:172.21.15.137:3300/0 conn(0x7f935015e1e0 0x7f93500702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 --2- 172.21.15.125:0/749857728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f935015dca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 -- 172.21.15.125:0/749857728 >> 172.21.15.125:0/749857728 conn(0x7f935006a800 msgr2=0x7f935006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 -- 172.21.15.125:0/749857728 shutdown_connections 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.862+0000 7f9355e00640 1 -- 172.21.15.125:0/749857728 wait complete. 2024-01-16T15:09:30.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.864+0000 7f9355e00640 1 Processor -- start 2024-01-16T15:09:30.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.864+0000 7f9355e00640 1 -- start start 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f93500afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 0x7f93500cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b1930 con 0x7f935015e1e0 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b0640 con 0x7f9350143ae0 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f9355e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b14a0 con 0x7f93500cf9a0 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f934f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f934f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46344/0 (socket says 172.21.15.125:46344) 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.865+0000 7f934f400640 1 -- 172.21.15.125:0/1930715839 learned_addr learned my addr 172.21.15.125:0/1930715839 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934ea00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f93500afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 -- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 msgr2=0x7f93500afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934e000640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 0x7f93500cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f93500afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 -- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 msgr2=0x7f93500cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 0x7f93500cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 -- 172.21.15.125:0/1930715839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348009a70 con 0x7f935015e1e0 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934e000640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 0x7f93500cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:30.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f934f400640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f934800efc0 tx=0x7f934800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348009cd0 con 0x7f935015e1e0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934800f040 con 0x7f935015e1e0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348003990 con 0x7f935015e1e0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.866+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93380094d0 con 0x7f935015e1e0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.867+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93500b0b70 con 0x7f935015e1e0 2024-01-16T15:09:30.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.868+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93500b1610 con 0x7f935015e1e0 2024-01-16T15:09:30.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.869+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9348014480 con 0x7f935015e1e0 2024-01-16T15:09:30.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.870+0000 7f9347400640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 0x7f932806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.870+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9348089180 con 0x7f935015e1e0 2024-01-16T15:09:30.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.870+0000 7f934ea00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 0x7f932806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.871+0000 7f934ea00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 0x7f932806e3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f60 tx=0x7f9340009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:30.874+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9348055a40 con 0x7f935015e1e0 2024-01-16T15:09:31.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.074+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f93500675d0 con 0x7f935015e1e0 2024-01-16T15:09:31.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.075+0000 7f9347400640 1 -- 172.21.15.125:0/1930715839 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+120 (secure 0 0 0) 0x7f9348018080 con 0x7f935015e1e0 2024-01-16T15:09:31.052 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/22640034 2024-01-17T15:09:24.566983+0000 2024-01-16T15:09:31.052 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/22640034 2024-01-17T15:09:24.566983+0000 2024-01-16T15:09:31.053 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.077+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 msgr2=0x7f932806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.077+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 0x7f932806e3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f60 tx=0x7f9340009040 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 msgr2=0x7f93500cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f934800efc0 tx=0x7f934800c490 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 shutdown_connections 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f932806bf20 0x7f932806e3e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935015e1e0 0x7f93500cdba0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9350143ae0 0x7f93500cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 --2- 172.21.15.125:0/1930715839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93500cf9a0 0x7f93500afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 >> 172.21.15.125:0/1930715839 conn(0x7f935006a800 msgr2=0x7f9350070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 shutdown_connections 2024-01-16T15:09:31.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.078+0000 7f9355e00640 1 -- 172.21.15.125:0/1930715839 wait complete. 2024-01-16T15:09:31.072 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:09:31.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:31.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:31.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45115]: fuse finished with error 0 and tester_r 0 2024-01-16T15:09:31.167 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:09:31.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:31.168 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:09:31.355 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:09:31.355 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:31.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:31.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:09:31.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:31.411 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:09:31.463 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:09:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:09:31.465 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:09:31.465 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:31.521 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:31.521 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:09:31.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:31.521 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:09:31.572 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:09:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:09:31.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.811+0000 7f8559400640 1 Processor -- start 2024-01-16T15:09:31.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.811+0000 7f8559400640 1 -- start start 2024-01-16T15:09:31.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.812+0000 7f8559400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f855406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.812+0000 7f8559400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540b2e10 con 0x7f8554071410 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.812+0000 7f8559400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8554146bc0 con 0x7f8554071f00 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.812+0000 7f8559400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855406cec0 con 0x7f8554146770 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8553e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8554071410 0x7f85540717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40754/0 (socket says 172.21.15.125:40754) 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8553e00640 1 -- 172.21.15.125:0/2363342536 learned_addr learned my addr 172.21.15.125:0/2363342536 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675685809 0 0) 0x7f85540b2e10 con 0x7f8554071410 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534003520 con 0x7f8554071410 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8558a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f855406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4202921132 0 0) 0x7f8534003520 con 0x7f8554071410 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 msgr2=0x7f855406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f855406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 >> v1:172.21.15.137:6789/0 conn(0x7f8554071f00 legacy=0x7f8554164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.813+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855406d1a0 con 0x7f8554071410 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f85440019f0 con 0x7f8554071410 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 >> v1:172.21.15.125:6789/0 conn(0x7f8554071410 legacy=0x7f85540717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85340039a0 0x7f8534003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85340047c0 0x7f8534004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f853400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855406cec0 con 0x7f85340039a0 2024-01-16T15:09:31.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534005510 con 0x7f85340047c0 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853400bec0 con 0x7f8554146770 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85340039a0 0x7f8534003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8558a00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f853400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 -- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 msgr2=0x7f853400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8554146770 0x7f853400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 -- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85340047c0 msgr2=0x7f8534004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553400640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85340047c0 0x7f8534004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85340047c0 0x7f8534004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553e00640 1 -- 172.21.15.125:0/2363342536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534003520 con 0x7f85340039a0 2024-01-16T15:09:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.814+0000 7f8553400640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85340047c0 0x7f8534004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8553e00640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85340039a0 0x7f8534003dc0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f85440013d0 tx=0x7f85340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835fd6b6049584e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544034630 con 0x7f85340039a0 2024-01-16T15:09:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854402f070 con 0x7f85340039a0 2024-01-16T15:09:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8552a00640 1 -- 172.21.15.125:0/2363342536 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85440349d0 con 0x7f85340039a0 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8559400640 1 -- 172.21.15.125:0/2363342536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85340039a0 msgr2=0x7f8534003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8559400640 1 --2- 172.21.15.125:0/2363342536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85340039a0 0x7f8534003dc0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f85440013d0 tx=0x7f85340057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.816+0000 7f8558a00640 1 -- 172.21.15.125:0/2363342536 reap_dead start 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.817+0000 7f8559400640 1 -- 172.21.15.125:0/2363342536 shutdown_connections 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.817+0000 7f8559400640 1 -- 172.21.15.125:0/2363342536 >> 172.21.15.125:0/2363342536 conn(0x7f855406a800 msgr2=0x7f855406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.817+0000 7f8559400640 1 -- 172.21.15.125:0/2363342536 shutdown_connections 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.817+0000 7f8559400640 1 -- 172.21.15.125:0/2363342536 wait complete. 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.818+0000 7f8559400640 1 Processor -- start 2024-01-16T15:09:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.818+0000 7f8559400640 1 -- start start 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85340039a0 0x7f85540c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85541443a0 0x7f85540c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c4360 con 0x7f855413fc60 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855415d000 con 0x7f85541443a0 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.819+0000 7f8559400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8554143060 con 0x7f85340039a0 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46356/0 (socket says 172.21.15.125:46356) 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 -- 172.21.15.125:0/224822099 learned_addr learned my addr 172.21.15.125:0/224822099 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 -- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85340039a0 msgr2=0x7f85540c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8558a00640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85541443a0 0x7f85540c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553e00640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85340039a0 0x7f85540c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85340039a0 0x7f85540c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 -- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85541443a0 msgr2=0x7f85540c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85541443a0 0x7f85540c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 -- 172.21.15.125:0/224822099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554143300 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.820+0000 7f8553400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f853c000f60 tx=0x7f853c00d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.821+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c005480 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.821+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c003610 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.821+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c0039c0 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.821+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85541435e0 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.821+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8554143ba0 con 0x7f855413fc60 2024-01-16T15:09:31.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.822+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8554067450 con 0x7f855413fc60 2024-01-16T15:09:31.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.823+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f853c003b20 con 0x7f855413fc60 2024-01-16T15:09:31.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.823+0000 7f854be00640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 0x7f852c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.824+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f853c0884c0 con 0x7f855413fc60 2024-01-16T15:09:31.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.824+0000 7f8553e00640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 0x7f852c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.824+0000 7f8553e00640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 0x7f852c06e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8544034000 tx=0x7f8544009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:31.828+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f853c054d80 con 0x7f855413fc60 2024-01-16T15:09:32.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.041+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85540c44a0 con 0x7f855413fc60 2024-01-16T15:09:32.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.042+0000 7f854be00640 1 -- 172.21.15.125:0/224822099 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f853c0583a0 con 0x7f855413fc60 2024-01-16T15:09:32.019 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:32.019 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2392328978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2392328978},{"type":"v1","addr":"172.21.15.137:6834","nonce":2392328978}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5779,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2251602125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2251602125},{"type":"v1","addr":"172.21.15.137:6835","nonce":2251602125}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2951255872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2951255872},{"type":"v1","addr":"172.21.15.125:6835","nonce":2951255872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8076,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901757992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901757992},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901757992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[]} 2024-01-16T15:09:32.020 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 48 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 msgr2=0x7f852c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 0x7f852c06e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8544034000 tx=0x7f8544009030 comp rx=0 tx=0).stop 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 msgr2=0x7f855415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f853c000f60 tx=0x7f853c00d9f0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 shutdown_connections 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85541443a0 0x7f85540c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f852c06bf20 0x7f852c06e3e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f855413fc60 0x7f855415cac0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 --2- 172.21.15.125:0/224822099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85340039a0 0x7f85540c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.044+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 >> 172.21.15.125:0/224822099 conn(0x7f855406a800 msgr2=0x7f855415fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.045+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 shutdown_connections 2024-01-16T15:09:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.045+0000 7f8559400640 1 -- 172.21.15.125:0/224822099 wait complete. 2024-01-16T15:09:32.040 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.270+0000 7fb15ac00640 1 Processor -- start 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.270+0000 7fb15ac00640 1 -- start start 2024-01-16T15:09:32.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.271+0000 7fb15ac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1540cf9a0 0x7fb15415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.271+0000 7fb15ac00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540b1ee0 con 0x7fb154143ae0 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.271+0000 7fb15ac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540b2780 con 0x7fb15415e1e0 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.271+0000 7fb15ac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15415e990 con 0x7fb1540cf9a0 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb154143ae0 0x7fb1540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40764/0 (socket says 172.21.15.125:40764) 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153e00640 1 -- 172.21.15.125:0/2465998282 learned_addr learned my addr 172.21.15.125:0/2465998282 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153400640 1 --2- 172.21.15.125:0/2465998282 >> v2:172.21.15.125:3300/0 conn(0x7fb1540cf9a0 0x7fb15415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153400640 1 -- 172.21.15.125:0/2465998282 >> v1:172.21.15.137:6789/0 conn(0x7fb15415e1e0 legacy=0x7fb1540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153400640 1 -- 172.21.15.125:0/2465998282 >> v1:172.21.15.125:6789/0 conn(0x7fb154143ae0 legacy=0x7fb1540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.272+0000 7fb153400640 1 -- 172.21.15.125:0/2465998282 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540b2130 con 0x7fb1540cf9a0 2024-01-16T15:09:32.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb153400640 1 --2- 172.21.15.125:0/2465998282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1540cf9a0 0x7fb15415dca0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adae7590fa99ee42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb152a00640 1 -- 172.21.15.125:0/2465998282 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c031830 con 0x7fb1540cf9a0 2024-01-16T15:09:32.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb152a00640 1 -- 172.21.15.125:0/2465998282 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c031990 con 0x7fb1540cf9a0 2024-01-16T15:09:32.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb152a00640 1 -- 172.21.15.125:0/2465998282 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c031d20 con 0x7fb1540cf9a0 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb15ac00640 1 -- 172.21.15.125:0/2465998282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1540cf9a0 msgr2=0x7fb15415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.274+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2465998282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1540cf9a0 0x7fb15415dca0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.275+0000 7fb15ac00640 1 -- 172.21.15.125:0/2465998282 shutdown_connections 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.275+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2465998282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1540cf9a0 0x7fb15415dca0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.275+0000 7fb15ac00640 1 -- 172.21.15.125:0/2465998282 >> 172.21.15.125:0/2465998282 conn(0x7fb15406a800 msgr2=0x7fb15406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.275+0000 7fb15ac00640 1 -- 172.21.15.125:0/2465998282 shutdown_connections 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.275+0000 7fb15ac00640 1 -- 172.21.15.125:0/2465998282 wait complete. 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.276+0000 7fb15ac00640 1 Processor -- start 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.276+0000 7fb15ac00640 1 -- start start 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1540cf9a0 0x7fb1541365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 0x7fb1540a9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540ae5d0 con 0x7fb154143ae0 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540ab1f0 con 0x7fb15415e1e0 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.277+0000 7fb15ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540aa8e0 con 0x7fb1540cf9a0 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46376/0 (socket says 172.21.15.125:46376) 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 -- 172.21.15.125:0/2359571178 learned_addr learned my addr 172.21.15.125:0/2359571178 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb158c00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 0x7fb1540a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153e00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1540cf9a0 0x7fb1541365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 -- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1540cf9a0 msgr2=0x7fb1541365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1540cf9a0 0x7fb1541365c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 -- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 msgr2=0x7fb1540a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 0x7fb1540a9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb153400640 1 -- 172.21.15.125:0/2359571178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c009400 con 0x7fb154143ae0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.278+0000 7fb158c00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 0x7fb1540a9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb153400640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02bee0 tx=0x7fb13c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c03ba70 con 0x7fb154143ae0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c03bbf0 con 0x7fb154143ae0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1540aaba0 con 0x7fb154143ae0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0453f0 con 0x7fb154143ae0 2024-01-16T15:09:32.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.279+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1540a8780 con 0x7fb154143ae0 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.280+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1540ab360 con 0x7fb154143ae0 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.281+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb13c045550 con 0x7fb154143ae0 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.282+0000 7fb14be00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 0x7fb12c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.283+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb13c0aac10 con 0x7fb154143ae0 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.283+0000 7fb153e00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 0x7fb12c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.283+0000 7fb153e00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 0x7fb12c06e3e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb144005880 tx=0x7fb144009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.287+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb13c077370 con 0x7fb154143ae0 2024-01-16T15:09:32.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.477+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fb1540675d0 con 0x7fb154143ae0 2024-01-16T15:09:32.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.484+0000 7fb14be00640 1 -- 172.21.15.125:0/2359571178 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb13c07a990 con 0x7fb154143ae0 2024-01-16T15:09:32.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 msgr2=0x7fb12c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 0x7fb12c06e3e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb144005880 tx=0x7fb144009040 comp rx=0 tx=0).stop 2024-01-16T15:09:32.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 msgr2=0x7fb1540a93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02bee0 tx=0x7fb13c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 shutdown_connections 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb15415e1e0 0x7fb1540a9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb12c06bf20 0x7fb12c06e3e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb154143ae0 0x7fb1540a93e0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 --2- 172.21.15.125:0/2359571178 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1540cf9a0 0x7fb1541365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.487+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 >> 172.21.15.125:0/2359571178 conn(0x7fb15406a800 msgr2=0x7fb15415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.488+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 shutdown_connections 2024-01-16T15:09:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.488+0000 7fb15ac00640 1 -- 172.21.15.125:0/2359571178 wait complete. 2024-01-16T15:09:32.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:09:32.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.705+0000 7fad58800640 1 Processor -- start 2024-01-16T15:09:32.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.705+0000 7fad58800640 1 -- start start 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad58800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad500cf9a0 0x7fad5015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad58800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad50070c40 con 0x7fad500ae5a0 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad58800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5006c470 con 0x7fad500b3ae0 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad58800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad50071450 con 0x7fad500cf9a0 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad56800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad500cf9a0 0x7fad5015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad56800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad500cf9a0 0x7fad5015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46390/0 (socket says 172.21.15.125:46390) 2024-01-16T15:09:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad56800640 1 -- 172.21.15.125:0/3217811863 learned_addr learned my addr 172.21.15.125:0/3217811863 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.707+0000 7fad55e00640 1 --1- 172.21.15.125:0/3217811863 >> v1:172.21.15.125:6789/0 conn(0x7fad500ae5a0 0x7fad5006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40772/0 (socket says 172.21.15.125:40772) 2024-01-16T15:09:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.708+0000 7fad56800640 1 -- 172.21.15.125:0/3217811863 >> v1:172.21.15.137:6789/0 conn(0x7fad500b3ae0 legacy=0x7fad500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.708+0000 7fad56800640 1 -- 172.21.15.125:0/3217811863 >> v1:172.21.15.125:6789/0 conn(0x7fad500ae5a0 legacy=0x7fad5006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.708+0000 7fad56800640 1 -- 172.21.15.125:0/3217811863 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5006c750 con 0x7fad500cf9a0 2024-01-16T15:09:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.709+0000 7fad56800640 1 --2- 172.21.15.125:0/3217811863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015e430 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fad48009e50 tx=0x7fad4802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b2a277e2c8114d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.709+0000 7fad54a00640 1 -- 172.21.15.125:0/3217811863 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4802cbb0 con 0x7fad500cf9a0 2024-01-16T15:09:32.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.709+0000 7fad54a00640 1 -- 172.21.15.125:0/3217811863 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4802cd10 con 0x7fad500cf9a0 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.709+0000 7fad54a00640 1 -- 172.21.15.125:0/3217811863 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48034610 con 0x7fad500cf9a0 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 -- 172.21.15.125:0/3217811863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 msgr2=0x7fad5015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 --2- 172.21.15.125:0/3217811863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015e430 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fad48009e50 tx=0x7fad4802c010 comp rx=0 tx=0).stop 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 -- 172.21.15.125:0/3217811863 shutdown_connections 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 --2- 172.21.15.125:0/3217811863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015e430 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 -- 172.21.15.125:0/3217811863 >> 172.21.15.125:0/3217811863 conn(0x7fad5006a800 msgr2=0x7fad5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 -- 172.21.15.125:0/3217811863 shutdown_connections 2024-01-16T15:09:32.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.710+0000 7fad58800640 1 -- 172.21.15.125:0/3217811863 wait complete. 2024-01-16T15:09:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.712+0000 7fad58800640 1 Processor -- start 2024-01-16T15:09:32.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.713+0000 7fad58800640 1 -- start start 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad500ae5a0 0x7fad50156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad500b3ae0 0x7fad50149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50158af0 con 0x7fad500cf9a0 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50156d40 con 0x7fad500b3ae0 2024-01-16T15:09:32.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad58800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad50157b80 con 0x7fad500ae5a0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46402/0 (socket says 172.21.15.125:46402) 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 -- 172.21.15.125:0/2514779135 learned_addr learned my addr 172.21.15.125:0/2514779135 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad55e00640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad500ae5a0 0x7fad50156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 -- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad500ae5a0 msgr2=0x7fad50156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad500ae5a0 0x7fad50156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 -- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad500b3ae0 msgr2=0x7fad50149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad500b3ae0 0x7fad50149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.714+0000 7fad56800640 1 -- 172.21.15.125:0/2514779135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009b30 con 0x7fad500cf9a0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad56800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fad48036040 tx=0x7fad48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4803e450 con 0x7fad500cf9a0 2024-01-16T15:09:32.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4803e5b0 con 0x7fad500cf9a0 2024-01-16T15:09:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad50158010 con 0x7fad500cf9a0 2024-01-16T15:09:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4803e960 con 0x7fad500cf9a0 2024-01-16T15:09:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.715+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad50157250 con 0x7fad500cf9a0 2024-01-16T15:09:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.716+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad50157d10 con 0x7fad500cf9a0 2024-01-16T15:09:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.717+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fad4803eac0 con 0x7fad500cf9a0 2024-01-16T15:09:32.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.718+0000 7fad4e000640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 0x7fad2806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.718+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fad480ab700 con 0x7fad500cf9a0 2024-01-16T15:09:32.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.718+0000 7fad55e00640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 0x7fad2806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.719+0000 7fad55e00640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 0x7fad2806e3e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fad40007fd0 tx=0x7fad40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.723+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad48077d40 con 0x7fad500cf9a0 2024-01-16T15:09:32.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.898+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fad5006d2c0 con 0x7fad500cf9a0 2024-01-16T15:09:32.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.903+0000 7fad4e000640 1 -- 172.21.15.125:0/2514779135 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad4807b360 con 0x7fad500cf9a0 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 msgr2=0x7fad2806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 0x7fad2806e3e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fad40007fd0 tx=0x7fad40009040 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 msgr2=0x7fad5015fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fad48036040 tx=0x7fad48004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 shutdown_connections 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad2806bf20 0x7fad2806e3e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad500cf9a0 0x7fad5015fba0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad500b3ae0 0x7fad50149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 --2- 172.21.15.125:0/2514779135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad500ae5a0 0x7fad50156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.906+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 >> 172.21.15.125:0/2514779135 conn(0x7fad5006a800 msgr2=0x7fad5006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.907+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 shutdown_connections 2024-01-16T15:09:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:32.907+0000 7fad58800640 1 -- 172.21.15.125:0/2514779135 wait complete. 2024-01-16T15:09:32.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:09:33.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.115+0000 7f50d4000640 1 Processor -- start 2024-01-16T15:09:33.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.115+0000 7f50d4000640 1 -- start start 2024-01-16T15:09:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.117+0000 7f50d4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.117+0000 7f50d4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.117+0000 7f50d4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc06c470 con 0x7f50cc13e360 2024-01-16T15:09:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.117+0000 7f50d4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc06c750 con 0x7f50cc13df80 2024-01-16T15:09:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.117+0000 7f50d4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc06ca30 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.118+0000 7f50d1600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.118+0000 7f50d2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.118+0000 7f50d2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:33408/0 (socket says 172.21.15.125:33408) 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.118+0000 7f50d2000640 1 -- 172.21.15.125:0/1959592776 learned_addr learned my addr 172.21.15.125:0/1959592776 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.118+0000 7f50d1600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:45518/0 (socket says 172.21.15.125:45518) 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.119+0000 7f50d2000640 1 -- 172.21.15.125:0/1959592776 >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 msgr2=0x7f50cc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.119+0000 7f50d2000640 1 --2- 172.21.15.125:0/1959592776 >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.119+0000 7f50d2000640 1 -- 172.21.15.125:0/1959592776 >> v1:172.21.15.137:6789/0 conn(0x7f50cc13e360 legacy=0x7f50cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.119+0000 7f50d2000640 1 -- 172.21.15.125:0/1959592776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc06cd10 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.119+0000 7f50d1600640 1 --2- 172.21.15.125:0/1959592776 >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.120+0000 7f50d2000640 1 --2- 172.21.15.125:0/1959592776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d97ae712c1eee81 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.120+0000 7f50cbe00640 1 -- 172.21.15.125:0/1959592776 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4030840 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.120+0000 7f50cbe00640 1 -- 172.21.15.125:0/1959592776 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c40309a0 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.120+0000 7f50cbe00640 1 -- 172.21.15.125:0/1959592776 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4030d30 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 -- 172.21.15.125:0/1959592776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 msgr2=0x7f50cc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 --2- 172.21.15.125:0/1959592776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c402b980 comp rx=0 tx=0).stop 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 -- 172.21.15.125:0/1959592776 shutdown_connections 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 --2- 172.21.15.125:0/1959592776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc0cd840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 --2- 172.21.15.125:0/1959592776 >> v2:172.21.15.137:3300/0 conn(0x7f50cc13df80 0x7f50cc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 -- 172.21.15.125:0/1959592776 >> 172.21.15.125:0/1959592776 conn(0x7f50cc06a800 msgr2=0x7f50cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 -- 172.21.15.125:0/1959592776 shutdown_connections 2024-01-16T15:09:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.121+0000 7f50d4000640 1 -- 172.21.15.125:0/1959592776 wait complete. 2024-01-16T15:09:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.123+0000 7f50d4000640 1 Processor -- start 2024-01-16T15:09:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.124+0000 7f50d4000640 1 -- start start 2024-01-16T15:09:33.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 0x7f50cc0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc13de20 con 0x7f50cc13e360 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc12f9c0 con 0x7f50cc13df80 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc12fc10 con 0x7f50cc0cf9a0 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46414/0 (socket says 172.21.15.125:46414) 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d2000640 1 -- 172.21.15.125:0/2940611496 learned_addr learned my addr 172.21.15.125:0/2940611496 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d1600640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.125+0000 7f50d0c00640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 0x7f50cc0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 -- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 msgr2=0x7f50cc132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 -- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 msgr2=0x7f50cc0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 0x7f50cc0b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 -- 172.21.15.125:0/2940611496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc009a70 con 0x7f50cc13e360 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d0c00640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 0x7f50cc0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d2000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4039a90 con 0x7f50cc13e360 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4039c10 con 0x7f50cc13e360 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c40094d0 con 0x7f50cc13e360 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.126+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4043410 con 0x7f50cc13e360 2024-01-16T15:09:33.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.127+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc1304b0 con 0x7f50cc13e360 2024-01-16T15:09:33.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.128+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc130010 con 0x7f50cc13e360 2024-01-16T15:09:33.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.129+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f50c404c3b0 con 0x7f50cc13e360 2024-01-16T15:09:33.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.130+0000 7f50c9600640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 0x7f50a406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.130+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f50c40aaac0 con 0x7f50cc13e360 2024-01-16T15:09:33.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.130+0000 7f50d1600640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 0x7f50a406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.131+0000 7f50d1600640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 0x7f50a406e410 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f50bc007600 tx=0x7f50bc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.134+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50c4077070 con 0x7f50cc13e360 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.377+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7f50cc0675d0 con 0x7f50cc13e360 2024-01-16T15:09:33.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.633+0000 7f50c9600640 1 -- 172.21.15.125:0/2940611496 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v582) v1 ==== 166+0+0 (secure 0 0 0) 0x7f50c40480e0 con 0x7f50cc13e360 2024-01-16T15:09:33.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 msgr2=0x7f50a406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 0x7f50a406e410 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f50bc007600 tx=0x7f50bc00b040 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 msgr2=0x7f50cc0ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c4004730 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 shutdown_connections 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f50a406bf50 0x7f50a406e410 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50cc13e360 0x7f50cc0ada60 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50cc13df80 0x7f50cc0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 --2- 172.21.15.125:0/2940611496 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50cc0cf9a0 0x7f50cc132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.636+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 >> 172.21.15.125:0/2940611496 conn(0x7f50cc06a800 msgr2=0x7f50cc0ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.637+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 shutdown_connections 2024-01-16T15:09:33.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.637+0000 7f50d4000640 1 -- 172.21.15.125:0/2940611496 wait complete. 2024-01-16T15:09:33.638 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:09:33.641 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:09:33.641 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:09:33.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:33.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.874+0000 7fda46200640 1 Processor -- start 2024-01-16T15:09:33.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.874+0000 7fda46200640 1 -- start start 2024-01-16T15:09:33.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.875+0000 7fda46200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 0x7fda4015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.875+0000 7fda46200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda4013e360 0x7fda400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda46200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4006c470 con 0x7fda4015f9c0 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda46200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4006c750 con 0x7fda4013e360 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda46200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4006ca30 con 0x7fda4013df80 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda3fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fda4015f9c0 0x7fda401408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40778/0 (socket says 172.21.15.125:40778) 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda3fe00640 1 -- 172.21.15.125:0/2749923323 learned_addr learned my addr 172.21.15.125:0/2749923323 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda3f400640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 0x7fda4015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649892294 0 0) 0x7fda4006c470 con 0x7fda4015f9c0 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda20003520 con 0x7fda4015f9c0 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.876+0000 7fda36800640 1 --2- 172.21.15.125:0/2749923323 >> v2:172.21.15.137:3300/0 conn(0x7fda4013e360 0x7fda400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079407324 0 0) 0x7fda20003520 con 0x7fda4015f9c0 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 msgr2=0x7fda4015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 0x7fda4015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 >> v2:172.21.15.137:3300/0 conn(0x7fda4013e360 msgr2=0x7fda400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 --2- 172.21.15.125:0/2749923323 >> v2:172.21.15.137:3300/0 conn(0x7fda4013e360 0x7fda400cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4006cd10 con 0x7fda4015f9c0 2024-01-16T15:09:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fda38001670 con 0x7fda4015f9c0 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 >> v1:172.21.15.125:6789/0 conn(0x7fda4015f9c0 legacy=0x7fda401408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda20003970 0x7fda20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda200047c0 0x7fda20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 0x7fda20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4006c750 con 0x7fda200047c0 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4006ca30 con 0x7fda20003970 2024-01-16T15:09:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2000b980 con 0x7fda4013df80 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda200047c0 0x7fda20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 -- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 msgr2=0x7fda20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda4013df80 0x7fda20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 -- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda20003970 msgr2=0x7fda20003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda20003970 0x7fda20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.877+0000 7fda36800640 1 -- 172.21.15.125:0/2749923323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20003520 con 0x7fda200047c0 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.878+0000 7fda36800640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda200047c0 0x7fda20004c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fda28009ac0 tx=0x7fda200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292be1365aee6923 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2802c710 con 0x7fda200047c0 2024-01-16T15:09:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2802c870 con 0x7fda200047c0 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda3ea00640 1 -- 172.21.15.125:0/2749923323 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2802cc00 con 0x7fda200047c0 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 -- 172.21.15.125:0/2749923323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda200047c0 msgr2=0x7fda20004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 --2- 172.21.15.125:0/2749923323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda200047c0 0x7fda20004c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fda28009ac0 tx=0x7fda200057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda3fe00640 1 -- 172.21.15.125:0/2749923323 reap_dead start 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 -- 172.21.15.125:0/2749923323 shutdown_connections 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 -- 172.21.15.125:0/2749923323 >> 172.21.15.125:0/2749923323 conn(0x7fda4006a800 msgr2=0x7fda4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 -- 172.21.15.125:0/2749923323 shutdown_connections 2024-01-16T15:09:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.879+0000 7fda46200640 1 -- 172.21.15.125:0/2749923323 wait complete. 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.881+0000 7fda46200640 1 Processor -- start 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.881+0000 7fda46200640 1 -- start start 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda200047c0 0x7fda40153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 0x7fda4014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda400b3610 con 0x7fda40150110 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda400b2850 con 0x7fda401506d0 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda46200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda400b2b30 con 0x7fda200047c0 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda3fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda3fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46432/0 (socket says 172.21.15.125:46432) 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.882+0000 7fda3fe00640 1 -- 172.21.15.125:0/2572286355 learned_addr learned my addr 172.21.15.125:0/2572286355 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3f400640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda200047c0 0x7fda40153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda36800640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 0x7fda4014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 -- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda200047c0 msgr2=0x7fda40153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda200047c0 0x7fda40153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 -- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 msgr2=0x7fda4014fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 0x7fda4014fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 -- 172.21.15.125:0/2572286355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda400b2e10 con 0x7fda40150110 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda36800640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 0x7fda4014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda3fe00640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fda38000f60 tx=0x7fda3800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38007440 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380075c0 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda400b30f0 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.883+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38004a20 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.884+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda40131ac0 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.885+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda400675d0 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.885+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fda38004b80 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.886+0000 7fda37e00640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 0x7fda1806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.886+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fda38087ec0 con 0x7fda40150110 2024-01-16T15:09:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.886+0000 7fda3f400640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 0x7fda1806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.887+0000 7fda3f400640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 0x7fda1806e410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fda30004b30 tx=0x7fda3000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:33.891+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda38054500 con 0x7fda40150110 2024-01-16T15:09:34.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.080+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda4006df00 con 0x7fda40150110 2024-01-16T15:09:34.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.081+0000 7fda37e00640 1 -- 172.21.15.125:0/2572286355 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13033 (secure 0 0 0) 0x7fda38057b20 con 0x7fda40150110 2024-01-16T15:09:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:34.061 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:27.581384+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/22640034":"2024-01-17T15:09:24.566983+0000","172.21.15.125:6836/22640034":"2024-01-17T15:09:24.566983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 msgr2=0x7fda1806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 0x7fda1806e410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fda30004b30 tx=0x7fda3000d040 comp rx=0 tx=0).stop 2024-01-16T15:09:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 msgr2=0x7fda40131580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fda38000f60 tx=0x7fda3800c9e0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 shutdown_connections 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fda1806bf50 0x7fda1806e410 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda40150110 0x7fda40131580 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda401506d0 0x7fda4014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 --2- 172.21.15.125:0/2572286355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda200047c0 0x7fda40153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 >> 172.21.15.125:0/2572286355 conn(0x7fda4006a800 msgr2=0x7fda400b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 shutdown_connections 2024-01-16T15:09:34.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.084+0000 7fda46200640 1 -- 172.21.15.125:0/2572286355 wait complete. 2024-01-16T15:09:34.081 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:34.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:34.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.310+0000 7fdbeea00640 1 Processor -- start 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.310+0000 7fdbeea00640 1 -- start start 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.311+0000 7fdbeea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbe815e3b0 0x7fdbe8164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.311+0000 7fdbeea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80b1ee0 con 0x7fdbe80cf9a0 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.311+0000 7fdbeea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80b2780 con 0x7fdbe8143ae0 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.311+0000 7fdbeea00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe815e990 con 0x7fdbe815e3b0 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.311+0000 7fdbeca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbe815e3b0 0x7fdbe8164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.312+0000 7fdbeca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbe815e3b0 0x7fdbe8164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46446/0 (socket says 172.21.15.125:46446) 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.312+0000 7fdbeca00640 1 -- 172.21.15.125:0/197990649 learned_addr learned my addr 172.21.15.125:0/197990649 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.312+0000 7fdbeca00640 1 -- 172.21.15.125:0/197990649 >> v1:172.21.15.137:6789/0 conn(0x7fdbe8143ae0 legacy=0x7fdbe80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:34.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.312+0000 7fdbeca00640 1 -- 172.21.15.125:0/197990649 >> v1:172.21.15.125:6789/0 conn(0x7fdbe80cf9a0 legacy=0x7fdbe815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.312+0000 7fdbeca00640 1 -- 172.21.15.125:0/197990649 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80b2130 con 0x7fdbe815e3b0 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.313+0000 7fdbeca00640 1 --2- 172.21.15.125:0/197990649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe8164de0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009e50 tx=0x7fdbe002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e94a73e488eaac5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.313+0000 7fdbe7400640 1 -- 172.21.15.125:0/197990649 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe002cb10 con 0x7fdbe815e3b0 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.313+0000 7fdbe7400640 1 -- 172.21.15.125:0/197990649 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe002cc70 con 0x7fdbe815e3b0 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.313+0000 7fdbe7400640 1 -- 172.21.15.125:0/197990649 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00343f0 con 0x7fdbe815e3b0 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 -- 172.21.15.125:0/197990649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 msgr2=0x7fdbe8164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 --2- 172.21.15.125:0/197990649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe8164de0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009e50 tx=0x7fdbe002bf70 comp rx=0 tx=0).stop 2024-01-16T15:09:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 -- 172.21.15.125:0/197990649 shutdown_connections 2024-01-16T15:09:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 --2- 172.21.15.125:0/197990649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe8164de0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 -- 172.21.15.125:0/197990649 >> 172.21.15.125:0/197990649 conn(0x7fdbe806a800 msgr2=0x7fdbe806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 -- 172.21.15.125:0/197990649 shutdown_connections 2024-01-16T15:09:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.314+0000 7fdbeea00640 1 -- 172.21.15.125:0/197990649 wait complete. 2024-01-16T15:09:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.316+0000 7fdbeea00640 1 Processor -- start 2024-01-16T15:09:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.316+0000 7fdbeea00640 1 -- start start 2024-01-16T15:09:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 0x7fdbe80aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 0x7fdbe80b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe815c210 con 0x7fdbe815e3b0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80aef60 con 0x7fdbe8143ae0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80af190 con 0x7fdbe80cf9a0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46450/0 (socket says 172.21.15.125:46450) 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeca00640 1 -- 172.21.15.125:0/754243771 learned_addr learned my addr 172.21.15.125:0/754243771 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbdf200640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 0x7fdbe80b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbe7e00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 0x7fdbe80aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.317+0000 7fdbeca00640 1 -- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 msgr2=0x7fdbe80aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeca00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 0x7fdbe80aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeca00640 1 -- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 msgr2=0x7fdbe80b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeca00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 0x7fdbe80b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeca00640 1 -- 172.21.15.125:0/754243771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0009b30 con 0x7fdbe815e3b0 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbdf200640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 0x7fdbe80b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbe7e00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 0x7fdbe80aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeca00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009e50 tx=0x7fdbe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0009510 con 0x7fdbe815e3b0 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe003e450 con 0x7fdbe815e3b0 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe003e800 con 0x7fdbe815e3b0 2024-01-16T15:09:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe80af470 con 0x7fdbe815e3b0 2024-01-16T15:09:34.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.318+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe80afa30 con 0x7fdbe815e3b0 2024-01-16T15:09:34.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.320+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe80675d0 con 0x7fdbe815e3b0 2024-01-16T15:09:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.320+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fdbe003e960 con 0x7fdbe815e3b0 2024-01-16T15:09:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.321+0000 7fdbe4c00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 0x7fdbc006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.321+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdbe00ab670 con 0x7fdbe815e3b0 2024-01-16T15:09:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.322+0000 7fdbe7e00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 0x7fdbc006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.322+0000 7fdbe7e00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 0x7fdbc006e3e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8003eb0 tx=0x7fdbd8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.326+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbe0077d40 con 0x7fdbe815e3b0 2024-01-16T15:09:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.516+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbe806d840 con 0x7fdbe815e3b0 2024-01-16T15:09:34.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.517+0000 7fdbe4c00640 1 -- 172.21.15.125:0/754243771 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbe007b360 con 0x7fdbe815e3b0 2024-01-16T15:09:34.496 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 msgr2=0x7fdbc006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 0x7fdbc006e3e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8003eb0 tx=0x7fdbd8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 msgr2=0x7fdbe813e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009e50 tx=0x7fdbe0004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 shutdown_connections 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdbc006bf20 0x7fdbc006e3e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbe815e3b0 0x7fdbe813e320 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbe8143ae0 0x7fdbe80b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 --2- 172.21.15.125:0/754243771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbe80cf9a0 0x7fdbe80aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 >> 172.21.15.125:0/754243771 conn(0x7fdbe806a800 msgr2=0x7fdbe8142190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 shutdown_connections 2024-01-16T15:09:34.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.520+0000 7fdbeea00640 1 -- 172.21.15.125:0/754243771 wait complete. 2024-01-16T15:09:34.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:09:34.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.750+0000 7fe74a000640 1 Processor -- start 2024-01-16T15:09:34.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.750+0000 7fe74a000640 1 -- start start 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe74a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe74415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe74a000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7440cf290 0x7fe7440cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe74a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe744070c40 con 0x7fe7440ae5a0 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe74a000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe744071450 con 0x7fe7440cf290 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe74a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440cfc60 con 0x7fe7440b3ae0 2024-01-16T15:09:34.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe743e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7440cf290 0x7fe7440cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe743e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7440cf290 0x7fe7440cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46460/0 (socket says 172.21.15.125:46460) 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.752+0000 7fe743e00640 1 -- 172.21.15.125:0/3363053386 learned_addr learned my addr 172.21.15.125:0/3363053386 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.753+0000 7fe742a00640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe74415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.753+0000 7fe743e00640 1 -- 172.21.15.125:0/3363053386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 msgr2=0x7fe74415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.753+0000 7fe743e00640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe74415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.753+0000 7fe743e00640 1 -- 172.21.15.125:0/3363053386 >> v1:172.21.15.137:6789/0 conn(0x7fe7440ae5a0 legacy=0x7fe74406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:34.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.753+0000 7fe743e00640 1 -- 172.21.15.125:0/3363053386 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74406c470 con 0x7fe7440cf290 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.754+0000 7fe743e00640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe7440cf720 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009870 tx=0x7fe73c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5060cb18c3ec69d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe742000640 1 -- 172.21.15.125:0/3363053386 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c030840 con 0x7fe7440cf290 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe742000640 1 -- 172.21.15.125:0/3363053386 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c0309a0 con 0x7fe7440cf290 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe742000640 1 -- 172.21.15.125:0/3363053386 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c030d30 con 0x7fe7440cf290 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 -- 172.21.15.125:0/3363053386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 msgr2=0x7fe7440cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe7440cf720 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009870 tx=0x7fe73c02b980 comp rx=0 tx=0).stop 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 -- 172.21.15.125:0/3363053386 shutdown_connections 2024-01-16T15:09:34.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe7440cf720 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 --2- 172.21.15.125:0/3363053386 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe74415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 -- 172.21.15.125:0/3363053386 >> 172.21.15.125:0/3363053386 conn(0x7fe74406a800 msgr2=0x7fe74406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 -- 172.21.15.125:0/3363053386 shutdown_connections 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.755+0000 7fe74a000640 1 -- 172.21.15.125:0/3363053386 wait complete. 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.757+0000 7fe74a000640 1 Processor -- start 2024-01-16T15:09:34.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.757+0000 7fe74a000640 1 -- start start 2024-01-16T15:09:34.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7440ae5a0 0x7fe74414f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe744156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74415eb50 con 0x7fe7440cf290 2024-01-16T15:09:34.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744156c40 con 0x7fe7440ae5a0 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe74a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe744158220 con 0x7fe7440b3ae0 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe743e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.758+0000 7fe743e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46470/0 (socket says 172.21.15.125:46470) 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 -- 172.21.15.125:0/1571043700 learned_addr learned my addr 172.21.15.125:0/1571043700 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe742a00640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe744156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743400640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7440ae5a0 0x7fe74414f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 -- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 msgr2=0x7fe744156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe744156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 -- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7440ae5a0 msgr2=0x7fe74414f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7440ae5a0 0x7fe74414f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 -- 172.21.15.125:0/1571043700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c0094d0 con 0x7fe7440cf290 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.759+0000 7fe743e00640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe73c02b9c0 tx=0x7fe73c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.760+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c039b50 con 0x7fe7440cf290 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.760+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c039cb0 con 0x7fe7440cf290 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.760+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c043630 con 0x7fe7440cf290 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.760+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe744158500 con 0x7fe7440cf290 2024-01-16T15:09:34.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.760+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74415edf0 con 0x7fe7440cf290 2024-01-16T15:09:34.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.761+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74415efe0 con 0x7fe7440cf290 2024-01-16T15:09:34.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.762+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe73c042390 con 0x7fe7440cf290 2024-01-16T15:09:34.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.763+0000 7fe73b400640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 0x7fe71c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.763+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe73c0aa7d0 con 0x7fe7440cf290 2024-01-16T15:09:34.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.763+0000 7fe743400640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 0x7fe71c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.764+0000 7fe743400640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 0x7fe71c06e3e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe734004ea0 tx=0x7fe734009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:34.767+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe73c0770f0 con 0x7fe7440cf290 2024-01-16T15:09:35.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.028+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7fe7440675d0 con 0x7fe7440cf290 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.644+0000 7fe73b400640 1 -- 172.21.15.125:0/1571043700 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v584) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe73c07a710 con 0x7fe7440cf290 2024-01-16T15:09:35.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 msgr2=0x7fe71c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 0x7fe71c06e3e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe734004ea0 tx=0x7fe734009040 comp rx=0 tx=0).stop 2024-01-16T15:09:35.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 msgr2=0x7fe744152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe73c02b9c0 tx=0x7fe73c004730 comp rx=0 tx=0).stop 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 shutdown_connections 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe71c06bf20 0x7fe71c06e3e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7440cf290 0x7fe744152bb0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7440b3ae0 0x7fe744156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 --2- 172.21.15.125:0/1571043700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7440ae5a0 0x7fe74414f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 >> 172.21.15.125:0/1571043700 conn(0x7fe74406a800 msgr2=0x7fe74415a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.648+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 shutdown_connections 2024-01-16T15:09:35.627 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.649+0000 7fe74a000640 1 -- 172.21.15.125:0/1571043700 wait complete. 2024-01-16T15:09:35.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:09:35.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.889+0000 7f57f3e00640 1 Processor -- start 2024-01-16T15:09:35.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.889+0000 7f57f3e00640 1 -- start start 2024-01-16T15:09:35.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.890+0000 7f57f3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57ec0cf9a0 0x7f57ec15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f3e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec0b1ee0 con 0x7f57ec15e1e0 2024-01-16T15:09:35.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec0b2780 con 0x7f57ec143ae0 2024-01-16T15:09:35.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec15e990 con 0x7f57ec0cf9a0 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f57ec15e1e0 0x7f57ec0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40790/0 (socket says 172.21.15.125:40790) 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f1e00640 1 -- 172.21.15.125:0/1580751488 learned_addr learned my addr 172.21.15.125:0/1580751488 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57f0a00640 1 --2- 172.21.15.125:0/1580751488 >> v2:172.21.15.137:3300/0 conn(0x7f57ec0cf9a0 0x7f57ec15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.891+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194082980 0 0) 0x7f57ec0b1ee0 con 0x7f57ec15e1e0 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57cc005730 con 0x7f57ec15e1e0 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034761021 0 0) 0x7f57cc005730 con 0x7f57ec15e1e0 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 >> v2:172.21.15.137:3300/0 conn(0x7f57ec0cf9a0 msgr2=0x7f57ec15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 --2- 172.21.15.125:0/1580751488 >> v2:172.21.15.137:3300/0 conn(0x7f57ec0cf9a0 0x7f57ec15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 >> v1:172.21.15.137:6789/0 conn(0x7f57ec143ae0 legacy=0x7f57ec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec0b2130 con 0x7f57ec15e1e0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f57e4001670 con 0x7f57ec15e1e0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 >> v1:172.21.15.125:6789/0 conn(0x7f57ec15e1e0 legacy=0x7f57ec0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57cc005b80 0x7f57cc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc0069d0 0x7f57cc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 0x7f57cc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec15e990 con 0x7f57cc007390 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc00e230 con 0x7f57cc005b80 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc00e3f0 con 0x7f57cc0069d0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.892+0000 7f57f1e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 0x7f57cc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1400640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57cc005b80 0x7f57cc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f0a00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc0069d0 0x7f57cc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1e00640 1 -- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc0069d0 msgr2=0x7f57cc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc0069d0 0x7f57cc006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1e00640 1 -- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57cc005b80 msgr2=0x7f57cc005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57cc005b80 0x7f57cc005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f1e00640 1 -- 172.21.15.125:0/1580751488 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc005730 con 0x7f57cc007390 2024-01-16T15:09:35.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.893+0000 7f57f0a00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc0069d0 0x7f57cc006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.894+0000 7f57f1e00640 1 -- 172.21.15.125:0/1580751488 reap_dead start 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.894+0000 7f57f1e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 0x7f57cc00db90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f57e400c210 tx=0x7f57cc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1908c591b9a32d89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.894+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e402cae0 con 0x7f57cc007390 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.894+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e402cc40 con 0x7f57cc007390 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.894+0000 7f57ebe00640 1 -- 172.21.15.125:0/1580751488 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e40058d0 con 0x7f57cc007390 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 -- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 msgr2=0x7f57cc00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 0x7f57cc00db90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f57e400c210 tx=0x7f57cc007a00 comp rx=0 tx=0).stop 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 -- 172.21.15.125:0/1580751488 shutdown_connections 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 --2- 172.21.15.125:0/1580751488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57cc007390 0x7f57cc00db90 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 -- 172.21.15.125:0/1580751488 >> 172.21.15.125:0/1580751488 conn(0x7f57ec06a800 msgr2=0x7f57ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 -- 172.21.15.125:0/1580751488 shutdown_connections 2024-01-16T15:09:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.895+0000 7f57f3e00640 1 -- 172.21.15.125:0/1580751488 wait complete. 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.897+0000 7f57f3e00640 1 Processor -- start 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.897+0000 7f57f3e00640 1 -- start start 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc007390 0x7f57ec159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57ec0702c0 0x7f57ec070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec070c80 con 0x7f57ec13bb40 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec159db0 con 0x7f57ec0702c0 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec15d100 con 0x7f57cc007390 2024-01-16T15:09:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46498/0 (socket says 172.21.15.125:46498) 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 -- 172.21.15.125:0/190824671 learned_addr learned my addr 172.21.15.125:0/190824671 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f1e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57ec0702c0 0x7f57ec070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 -- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc007390 msgr2=0x7f57ec159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc007390 0x7f57ec159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 -- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57ec0702c0 msgr2=0x7f57ec070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57ec0702c0 0x7f57ec070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.898+0000 7f57f0a00640 1 -- 172.21.15.125:0/190824671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec15c860 con 0x7f57ec13bb40 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57f0a00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f57d40108b0 tx=0x7f57d4010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400eb60 con 0x7f57ec13bb40 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d400ecc0 con 0x7f57ec13bb40 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d401c400 con 0x7f57ec13bb40 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec15cb40 con 0x7f57ec13bb40 2024-01-16T15:09:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.899+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec0cc780 con 0x7f57ec13bb40 2024-01-16T15:09:35.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.901+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec0675d0 con 0x7f57ec13bb40 2024-01-16T15:09:35.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.901+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f57d401c560 con 0x7f57ec13bb40 2024-01-16T15:09:35.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.903+0000 7f57e9600640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 0x7f57c406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.903+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f57d4088b80 con 0x7f57ec13bb40 2024-01-16T15:09:35.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.903+0000 7f57f1400640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 0x7f57c406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.904+0000 7f57f1400640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 0x7f57c406e410 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001bf0 tx=0x7f57dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:35.907+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57d4055550 con 0x7f57ec13bb40 2024-01-16T15:09:36.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.095+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f57ec0cf400 con 0x7f57ec13bb40 2024-01-16T15:09:36.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.101+0000 7f57e9600640 1 -- 172.21.15.125:0/190824671 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f57d4058b70 con 0x7f57ec13bb40 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 msgr2=0x7f57c406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 0x7f57c406e410 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001bf0 tx=0x7f57dc009040 comp rx=0 tx=0).stop 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 msgr2=0x7f57ec15c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f57d40108b0 tx=0x7f57d4010d70 comp rx=0 tx=0).stop 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 shutdown_connections 2024-01-16T15:09:36.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57ec0702c0 0x7f57ec070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f57c406bf50 0x7f57c406e410 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57ec13bb40 0x7f57ec15c000 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 --2- 172.21.15.125:0/190824671 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57cc007390 0x7f57ec159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 >> 172.21.15.125:0/190824671 conn(0x7f57ec06a800 msgr2=0x7f57ec142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 shutdown_connections 2024-01-16T15:09:36.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.104+0000 7f57f3e00640 1 -- 172.21.15.125:0/190824671 wait complete. 2024-01-16T15:09:36.102 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:09:36.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:36.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:36.331 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:09:36.331 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:09:36.331 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:36.387 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:36.387 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:09:36.387 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:36.443 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:36.443 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:09:36.444 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:36.501 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:36.501 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:09:36.501 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:36.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:36.557 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:09:36.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:36.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.799+0000 7f4071600640 1 Processor -- start 2024-01-16T15:09:36.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.799+0000 7f4071600640 1 -- start start 2024-01-16T15:09:36.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.801+0000 7f4071600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.801+0000 7f4071600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f406c15f9c0 0x7f406c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.801+0000 7f4071600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c15c5f0 con 0x7f406c13df80 2024-01-16T15:09:36.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.801+0000 7f4071600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c06c470 con 0x7f406c15f9c0 2024-01-16T15:09:36.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.801+0000 7f4071600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c06c750 con 0x7f406c13e360 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:53648/0 (socket says 172.21.15.125:53648) 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 -- 172.21.15.125:0/1673634157 learned_addr learned my addr 172.21.15.125:0/1673634157 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f406b400640 1 --2- 172.21.15.125:0/1673634157 >> v2:172.21.15.137:3300/0 conn(0x7f406c15f9c0 0x7f406c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 -- 172.21.15.125:0/1673634157 >> v2:172.21.15.137:3300/0 conn(0x7f406c15f9c0 msgr2=0x7f406c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 --2- 172.21.15.125:0/1673634157 >> v2:172.21.15.137:3300/0 conn(0x7f406c15f9c0 0x7f406c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.802+0000 7f4063e00640 1 -- 172.21.15.125:0/1673634157 >> v1:172.21.15.137:6789/0 conn(0x7f406c13df80 legacy=0x7f406c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.803+0000 7f4063e00640 1 -- 172.21.15.125:0/1673634157 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c06ca30 con 0x7f406c13e360 2024-01-16T15:09:36.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.804+0000 7f4063e00640 1 --2- 172.21.15.125:0/1673634157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4054009870 tx=0x7f405402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1173972ab1c4c556 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.805+0000 7f406a000640 1 -- 172.21.15.125:0/1673634157 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054030840 con 0x7f406c13e360 2024-01-16T15:09:36.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.805+0000 7f406a000640 1 -- 172.21.15.125:0/1673634157 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40540309a0 con 0x7f406c13e360 2024-01-16T15:09:36.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.805+0000 7f406a000640 1 -- 172.21.15.125:0/1673634157 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054030d30 con 0x7f406c13e360 2024-01-16T15:09:36.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.805+0000 7f4071600640 1 -- 172.21.15.125:0/1673634157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 msgr2=0x7f406c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.805+0000 7f4071600640 1 --2- 172.21.15.125:0/1673634157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4054009870 tx=0x7f405402b980 comp rx=0 tx=0).stop 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 -- 172.21.15.125:0/1673634157 shutdown_connections 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 --2- 172.21.15.125:0/1673634157 >> v2:172.21.15.137:3300/0 conn(0x7f406c15f9c0 0x7f406c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 --2- 172.21.15.125:0/1673634157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c13e360 0x7f406c0cf290 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 -- 172.21.15.125:0/1673634157 >> 172.21.15.125:0/1673634157 conn(0x7f406c06a800 msgr2=0x7f406c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 -- 172.21.15.125:0/1673634157 shutdown_connections 2024-01-16T15:09:36.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.806+0000 7f4071600640 1 -- 172.21.15.125:0/1673634157 wait complete. 2024-01-16T15:09:36.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.808+0000 7f4071600640 1 Processor -- start 2024-01-16T15:09:36.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.808+0000 7f4071600640 1 -- start start 2024-01-16T15:09:36.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.809+0000 7f4071600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.809+0000 7f4071600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c13e360 0x7f406c150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.809+0000 7f4071600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c15f9c0 0x7f406c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f4071600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c151240 con 0x7f406c13df80 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f4071600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c153700 con 0x7f406c13e360 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f4071600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c12ef80 con 0x7f406c15f9c0 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38622/0 (socket says 172.21.15.125:38622) 2024-01-16T15:09:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 -- 172.21.15.125:0/4134737495 learned_addr learned my addr 172.21.15.125:0/4134737495 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:36.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 -- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c15f9c0 msgr2=0x7f406c155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c15f9c0 0x7f406c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 -- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c13e360 msgr2=0x7f406c150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c13e360 0x7f406c150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.810+0000 7f406aa00640 1 -- 172.21.15.125:0/4134737495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540094d0 con 0x7f406c13df80 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f406aa00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c400 tx=0x7f405c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c0052d0 con 0x7f406c13df80 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c005430 con 0x7f406c13df80 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c152af0 con 0x7f406c13df80 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c003830 con 0x7f406c13df80 2024-01-16T15:09:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.811+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c1530b0 con 0x7f406c13df80 2024-01-16T15:09:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.812+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c0675d0 con 0x7f406c13df80 2024-01-16T15:09:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.813+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f405c003990 con 0x7f406c13df80 2024-01-16T15:09:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.814+0000 7f4062a00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 0x7f404406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.814+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f405c087de0 con 0x7f406c13df80 2024-01-16T15:09:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.814+0000 7f4063e00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 0x7f404406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.815+0000 7f4063e00640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 0x7f404406e3e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4054036000 tx=0x7f405403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:36.818+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f405c054420 con 0x7f406c13df80 2024-01-16T15:09:37.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.044+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406c0af700 con 0x7f406c13df80 2024-01-16T15:09:37.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.045+0000 7f4062a00640 1 -- 172.21.15.125:0/4134737495 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f405c057a40 con 0x7f406c13df80 2024-01-16T15:09:37.024 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:37.024 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2392328978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2392328978},{"type":"v1","addr":"172.21.15.137:6834","nonce":2392328978}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5779,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2251602125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2251602125},{"type":"v1","addr":"172.21.15.137:6835","nonce":2251602125}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2951255872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2951255872},{"type":"v1","addr":"172.21.15.125:6835","nonce":2951255872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8076,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901757992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901757992},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901757992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[]} 2024-01-16T15:09:37.024 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 48 2024-01-16T15:09:37.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 msgr2=0x7f404406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 0x7f404406e3e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4054036000 tx=0x7f405403b040 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 msgr2=0x7f406c12bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c400 tx=0x7f405c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 shutdown_connections 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f406c15f9c0 0x7f406c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c13e360 0x7f406c150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f404406bf20 0x7f404406e3e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 --2- 172.21.15.125:0/4134737495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c13df80 0x7f406c12bf60 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 >> 172.21.15.125:0/4134737495 conn(0x7f406c06a800 msgr2=0x7f406c15d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.048+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 shutdown_connections 2024-01-16T15:09:37.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.049+0000 7f4071600640 1 -- 172.21.15.125:0/4134737495 wait complete. 2024-01-16T15:09:37.044 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:09:37.044 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:09:37.044 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:09:37.045 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:37.045 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:09:37.045 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:09:37.045 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:09:37.045 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:37.045 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:09:37.045 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:09:37.046 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:09:37.046 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:37.046 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:09:37.046 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:09:37.046 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:09:37.046 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:37.067 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:09:37.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:09:37.068 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:09:37.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:09:37.069 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:09:37.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:09:37.073 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:09:37.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:09:37.114 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:09:37.114 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:37.119 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:09:37.120 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:37.120 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:09:37.121 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:37.127 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:09:37.127 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:37.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:09:37.269 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:09:37.279 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T15:09:37.279 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:09:37.279 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T15:09:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.334+0000 7f9951800640 1 Processor -- start 2024-01-16T15:09:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.334+0000 7f9951800640 1 -- start start 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.335+0000 7f9951800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9951800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f994c0cf7d0 0x7f994c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9951800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 0x7f994c0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9951800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c14b6a0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9951800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0cfbf0 con 0x7f994c070ea0 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9951800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c071690 con 0x7f994c14b2c0 2024-01-16T15:09:37.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f994c0cf7d0 0x7f994c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f994c0cf7d0 0x7f994c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38644/0 (socket says 172.21.15.125:38644) 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 -- 172.21.15.125:0/2000825194 learned_addr learned my addr 172.21.15.125:0/2000825194 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f994aa00640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f994b400640 1 --2- 172.21.15.125:0/2000825194 >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 0x7f994c0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 -- 172.21.15.125:0/2000825194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 msgr2=0x7f994c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 -- 172.21.15.125:0/2000825194 >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 msgr2=0x7f994c0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 --2- 172.21.15.125:0/2000825194 >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 0x7f994c0b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f9943e00640 1 -- 172.21.15.125:0/2000825194 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c071970 con 0x7f994c0cf7d0 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f994aa00640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:37.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.336+0000 7f994b400640 1 --2- 172.21.15.125:0/2000825194 >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 0x7f994c0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:37.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.337+0000 7f9943e00640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c070960 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f994c06e2a0 tx=0x7f993402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6038731ab371678 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f994a000640 1 -- 172.21.15.125:0/2000825194 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934031870 con 0x7f994c0cf7d0 2024-01-16T15:09:37.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f994a000640 1 -- 172.21.15.125:0/2000825194 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99340319d0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f994a000640 1 -- 172.21.15.125:0/2000825194 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934031d60 con 0x7f994c0cf7d0 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 -- 172.21.15.125:0/2000825194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 msgr2=0x7f994c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c070960 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f994c06e2a0 tx=0x7f993402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 -- 172.21.15.125:0/2000825194 shutdown_connections 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 --2- 172.21.15.125:0/2000825194 >> v2:172.21.15.137:3300/0 conn(0x7f994c070ea0 0x7f994c0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c070960 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 --2- 172.21.15.125:0/2000825194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.338+0000 7f9951800640 1 -- 172.21.15.125:0/2000825194 >> 172.21.15.125:0/2000825194 conn(0x7f994c06a800 msgr2=0x7f994c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.339+0000 7f9951800640 1 -- 172.21.15.125:0/2000825194 shutdown_connections 2024-01-16T15:09:37.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.339+0000 7f9951800640 1 -- 172.21.15.125:0/2000825194 wait complete. 2024-01-16T15:09:37.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.340+0000 7f9951800640 1 Processor -- start 2024-01-16T15:09:37.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.340+0000 7f9951800640 1 -- start start 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f994c070ea0 0x7f994c0aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1638a0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1f81e0 con 0x7f994c070ea0 2024-01-16T15:09:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9951800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1459c0 con 0x7f994c14b2c0 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.341+0000 7f9943e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38654/0 (socket says 172.21.15.125:38654) 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 -- 172.21.15.125:0/1744307867 learned_addr learned my addr 172.21.15.125:0/1744307867 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f994aa00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f994c070ea0 0x7f994c0aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f994b400640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 -- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 msgr2=0x7f994c145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 -- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f994c070ea0 msgr2=0x7f994c0aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f994c070ea0 0x7f994c0aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.342+0000 7f9943e00640 1 -- 172.21.15.125:0/1744307867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340094d0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9943e00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f99340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993403aca0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993403ae20 con 0x7f994c0cf7d0 2024-01-16T15:09:37.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934044630 con 0x7f994c0cf7d0 2024-01-16T15:09:37.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c162f00 con 0x7f994c0cf7d0 2024-01-16T15:09:37.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.343+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c1634c0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.344+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0675d0 con 0x7f994c0cf7d0 2024-01-16T15:09:37.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.345+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f993404d450 con 0x7f994c0cf7d0 2024-01-16T15:09:37.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.346+0000 7f9942a00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 0x7f992406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.346+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f993402d040 con 0x7f994c0cf7d0 2024-01-16T15:09:37.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.346+0000 7f994aa00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 0x7f992406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.347+0000 7f994aa00640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 0x7f992406e3e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f993c006fd0 tx=0x7f993c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.350+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9934077460 con 0x7f994c0cf7d0 2024-01-16T15:09:37.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:37.547+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f994c1490c0 con 0x7f994c0cf7d0 2024-01-16T15:09:38.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.681+0000 7f9942a00640 1 -- 172.21.15.125:0/1744307867 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) v1 ==== 97+0+0 (secure 0 0 0) 0x7f993407aa80 con 0x7f994c0cf7d0 2024-01-16T15:09:38.659 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.684+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 msgr2=0x7f992406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.684+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 0x7f992406e3e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f993c006fd0 tx=0x7f993c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.684+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 msgr2=0x7f994c144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.684+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f99340046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 shutdown_connections 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f994c14b2c0 0x7f994c145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f992406bf20 0x7f992406e3e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994c0cf7d0 0x7f994c144f40 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 --2- 172.21.15.125:0/1744307867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f994c070ea0 0x7f994c0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 >> 172.21.15.125:0/1744307867 conn(0x7f994c06a800 msgr2=0x7f994c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 shutdown_connections 2024-01-16T15:09:38.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.685+0000 7f9951800640 1 -- 172.21.15.125:0/1744307867 wait complete. 2024-01-16T15:09:38.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:09:38.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.918+0000 7f7b32600640 1 Processor -- start 2024-01-16T15:09:38.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.918+0000 7f7b32600640 1 -- start start 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b32600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b2c06de50 0x7f7b2c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b32600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0abee0 con 0x7f7b2c070b20 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b32600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c06c800 con 0x7f7b2c15d4b0 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b32600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c070fd0 con 0x7f7b2c06de50 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b2be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b2c06de50 0x7f7b2c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b2be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7b2c06de50 0x7f7b2c0b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38662/0 (socket says 172.21.15.125:38662) 2024-01-16T15:09:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.920+0000 7f7b2be00640 1 -- 172.21.15.125:0/877572845 learned_addr learned my addr 172.21.15.125:0/877572845 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.921+0000 7f7b2be00640 1 -- 172.21.15.125:0/877572845 >> v1:172.21.15.137:6789/0 conn(0x7f7b2c15d4b0 legacy=0x7f7b2c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.921+0000 7f7b2be00640 1 -- 172.21.15.125:0/877572845 >> v1:172.21.15.125:6789/0 conn(0x7f7b2c070b20 legacy=0x7f7b2c0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.921+0000 7f7b2be00640 1 -- 172.21.15.125:0/877572845 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0b2110 con 0x7f7b2c06de50 2024-01-16T15:09:38.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.922+0000 7f7b2be00640 1 --2- 172.21.15.125:0/877572845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c06de50 0x7f7b2c0b3300 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009e50 tx=0x7f7b2402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca3b0cbeacbdac6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.922+0000 7f7b2aa00640 1 -- 172.21.15.125:0/877572845 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2402cb10 con 0x7f7b2c06de50 2024-01-16T15:09:38.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.922+0000 7f7b2aa00640 1 -- 172.21.15.125:0/877572845 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2402cc70 con 0x7f7b2c06de50 2024-01-16T15:09:38.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.922+0000 7f7b2aa00640 1 -- 172.21.15.125:0/877572845 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240343f0 con 0x7f7b2c06de50 2024-01-16T15:09:38.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 -- 172.21.15.125:0/877572845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c06de50 msgr2=0x7f7b2c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 --2- 172.21.15.125:0/877572845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c06de50 0x7f7b2c0b3300 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009e50 tx=0x7f7b2402bf70 comp rx=0 tx=0).stop 2024-01-16T15:09:38.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 -- 172.21.15.125:0/877572845 shutdown_connections 2024-01-16T15:09:38.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 --2- 172.21.15.125:0/877572845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c06de50 0x7f7b2c0b3300 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 -- 172.21.15.125:0/877572845 >> 172.21.15.125:0/877572845 conn(0x7f7b2c0b0c00 msgr2=0x7f7b2c0b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 -- 172.21.15.125:0/877572845 shutdown_connections 2024-01-16T15:09:38.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.923+0000 7f7b32600640 1 -- 172.21.15.125:0/877572845 wait complete. 2024-01-16T15:09:38.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.925+0000 7f7b32600640 1 Processor -- start 2024-01-16T15:09:38.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.925+0000 7f7b32600640 1 -- start start 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b2c06de50 0x7f7b2c146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b2c15d4b0 0x7f7b2c0adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c1470f0 con 0x7f7b2c070b20 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c15f720 con 0x7f7b2c06de50 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b32600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c15fa00 con 0x7f7b2c15d4b0 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b23e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b23e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38670/0 (socket says 172.21.15.125:38670) 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.926+0000 7f7b23e00640 1 -- 172.21.15.125:0/3736157118 learned_addr learned my addr 172.21.15.125:0/3736157118 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b2b400640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b2c06de50 0x7f7b2c146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b2be00640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b2c15d4b0 0x7f7b2c0adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 -- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b2c15d4b0 msgr2=0x7f7b2c0adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b2c15d4b0 0x7f7b2c0adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 -- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b2c06de50 msgr2=0x7f7b2c146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b2c06de50 0x7f7b2c146a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 -- 172.21.15.125:0/3736157118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24009b30 con 0x7f7b2c070b20 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23e00640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400c450 tx=0x7f7b1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b140052d0 con 0x7f7b2c070b20 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b14005430 con 0x7f7b2c070b20 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.927+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c1472d0 con 0x7f7b2c070b20 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.928+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14012870 con 0x7f7b2c070b20 2024-01-16T15:09:38.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.928+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c15ea60 con 0x7f7b2c070b20 2024-01-16T15:09:38.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.929+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c1477f0 con 0x7f7b2c070b20 2024-01-16T15:09:38.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.930+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f7b140129d0 con 0x7f7b2c070b20 2024-01-16T15:09:38.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.930+0000 7f7b23400640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 0x7f7b0406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.931+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b14088130 con 0x7f7b2c070b20 2024-01-16T15:09:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.931+0000 7f7b2b400640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 0x7f7b0406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.931+0000 7f7b2b400640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 0x7f7b0406e3e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c006fd0 tx=0x7f7b1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:38.935+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b2c1477f0 con 0x7f7b2c070b20 2024-01-16T15:09:39.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.197+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7b2c1477f0 con 0x7f7b2c070b20 2024-01-16T15:09:39.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.198+0000 7f7b23400640 1 -- 172.21.15.125:0/3736157118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7b2c1477f0 con 0x7f7b2c070b20 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.176 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.178 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.179 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.180 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.181 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.182 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T15:09:39.183 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.185 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:09:39.187 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T15:09:39.188 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 msgr2=0x7f7b0406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 0x7f7b0406e3e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c006fd0 tx=0x7f7b1c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 msgr2=0x7f7b2c0ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400c450 tx=0x7f7b1400c910 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 shutdown_connections 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b2c15d4b0 0x7f7b2c0adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b0406bf20 0x7f7b0406e3e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b2c070b20 0x7f7b2c0ad690 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 --2- 172.21.15.125:0/3736157118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b2c06de50 0x7f7b2c146a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 >> 172.21.15.125:0/3736157118 conn(0x7f7b2c0b0c00 msgr2=0x7f7b2c070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 shutdown_connections 2024-01-16T15:09:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.201+0000 7f7b32600640 1 -- 172.21.15.125:0/3736157118 wait complete. 2024-01-16T15:09:39.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:39.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.418+0000 7f688d000640 1 Processor -- start 2024-01-16T15:09:39.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.419+0000 7f688d000640 1 -- start start 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f688d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f688813cfa0 0x7f688815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f688d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68880cd530 0x7f688813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f688d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68880cdec0 con 0x7f68880cd9b0 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f688d000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68880ce1a0 con 0x7f68880cd530 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f688d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68880ce480 con 0x7f688813cfa0 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.420+0000 7f687e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68880cd530 0x7f688813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68880cd530 0x7f688813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38680/0 (socket says 172.21.15.125:38680) 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f6887e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f68880cd9b0 0x7f688815d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44518/0 (socket says 172.21.15.125:44518) 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 -- 172.21.15.125:0/1385650419 learned_addr learned my addr 172.21.15.125:0/1385650419 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:39.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f6887400640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f688813cfa0 0x7f688815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 -- 172.21.15.125:0/1385650419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f688813cfa0 msgr2=0x7f688815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f688813cfa0 0x7f688815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 -- 172.21.15.125:0/1385650419 >> v1:172.21.15.125:6789/0 conn(0x7f68880cd9b0 legacy=0x7f688815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f687e800640 1 -- 172.21.15.125:0/1385650419 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880ce760 con 0x7f68880cd530 2024-01-16T15:09:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.421+0000 7f6886a00640 1 -- 172.21.15.125:0/1385650419 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062414813 0 0) 0x7f68880cdec0 con 0x7f68880cd9b0 2024-01-16T15:09:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.422+0000 7f687e800640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cd530 0x7f688813f5a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6870009870 tx=0x7f687002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed82b24c090833bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.422+0000 7f6886a00640 1 -- 172.21.15.125:0/1385650419 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870031640 con 0x7f68880cd530 2024-01-16T15:09:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.422+0000 7f6886a00640 1 -- 172.21.15.125:0/1385650419 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68700317a0 con 0x7f68880cd530 2024-01-16T15:09:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.422+0000 7f6886a00640 1 -- 172.21.15.125:0/1385650419 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870031b30 con 0x7f68880cd530 2024-01-16T15:09:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 -- 172.21.15.125:0/1385650419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cd530 msgr2=0x7f688813f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cd530 0x7f688813f5a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6870009870 tx=0x7f687002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 -- 172.21.15.125:0/1385650419 shutdown_connections 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cd530 0x7f688813f5a0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 --2- 172.21.15.125:0/1385650419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f688813cfa0 0x7f688815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 -- 172.21.15.125:0/1385650419 >> 172.21.15.125:0/1385650419 conn(0x7f688806a800 msgr2=0x7f688806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 -- 172.21.15.125:0/1385650419 shutdown_connections 2024-01-16T15:09:39.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.423+0000 7f688d000640 1 -- 172.21.15.125:0/1385650419 wait complete. 2024-01-16T15:09:39.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.425+0000 7f688d000640 1 Processor -- start 2024-01-16T15:09:39.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.425+0000 7f688d000640 1 -- start start 2024-01-16T15:09:39.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68880cd530 0x7f688814fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68880cd9b0 0x7f688813b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688813bcd0 con 0x7f688813cfa0 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880ac8d0 con 0x7f68880cd9b0 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.426+0000 7f688d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688814ebd0 con 0x7f68880cd530 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38694/0 (socket says 172.21.15.125:38694) 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 -- 172.21.15.125:0/1593700387 learned_addr learned my addr 172.21.15.125:0/1593700387 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887400640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68880cd530 0x7f688814fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f687e800640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68880cd9b0 0x7f688813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 -- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68880cd530 msgr2=0x7f688814fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68880cd530 0x7f688814fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 -- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68880cd9b0 msgr2=0x7f688813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68880cd9b0 0x7f688813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 -- 172.21.15.125:0/1593700387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700094d0 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.427+0000 7f6887e00640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f688000c7e0 tx=0x7f688000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.428+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000a640 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.428+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000a7a0 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.428+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688814eeb0 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.428+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000ab30 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.428+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688814f470 con 0x7f688813cfa0 2024-01-16T15:09:39.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.429+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6888067450 con 0x7f688813cfa0 2024-01-16T15:09:39.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.430+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f688000ac90 con 0x7f688813cfa0 2024-01-16T15:09:39.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.431+0000 7f687fe00640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 0x7f686006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.431+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6880088130 con 0x7f688813cfa0 2024-01-16T15:09:39.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.431+0000 7f6887400640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 0x7f686006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.432+0000 7f6887400640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 0x7f686006e3e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6878006fd0 tx=0x7f6878008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.435+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6880054770 con 0x7f688813cfa0 2024-01-16T15:09:39.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.628+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68880aca10 con 0x7f688813cfa0 2024-01-16T15:09:39.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.630+0000 7f687fe00640 1 -- 172.21.15.125:0/1593700387 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+12906 (secure 0 0 0) 0x7f6880017070 con 0x7f688813cfa0 2024-01-16T15:09:39.608 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:39.608 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":82,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:38.679112+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 msgr2=0x7f686006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 0x7f686006e3e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6878006fd0 tx=0x7f6878008040 comp rx=0 tx=0).stop 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 msgr2=0x7f6888156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f688000c7e0 tx=0x7f688000cca0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 shutdown_connections 2024-01-16T15:09:39.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f686006bf20 0x7f686006e3e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f688813cfa0 0x7f6888156810 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68880cd9b0 0x7f688813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 --2- 172.21.15.125:0/1593700387 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68880cd530 0x7f688814fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 >> 172.21.15.125:0/1593700387 conn(0x7f688806a800 msgr2=0x7f6888153b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 shutdown_connections 2024-01-16T15:09:39.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.633+0000 7f688d000640 1 -- 172.21.15.125:0/1593700387 wait complete. 2024-01-16T15:09:39.630 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:39.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:39.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.857+0000 7fe92fc00640 1 Processor -- start 2024-01-16T15:09:39.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.857+0000 7fe92fc00640 1 -- start start 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92fc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9280cf9a0 0x7fe92815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92fc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9280b1ee0 con 0x7fe92815e1e0 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92fc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9280b2780 con 0x7fe928143ae0 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92fc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92815e990 con 0x7fe9280cf9a0 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe927e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9280cf9a0 0x7fe92815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92dc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe92815e1e0 0x7fe9280702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44534/0 (socket says 172.21.15.125:44534) 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe927e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9280cf9a0 0x7fe92815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38708/0 (socket says 172.21.15.125:38708) 2024-01-16T15:09:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.859+0000 7fe92dc00640 1 -- 172.21.15.125:0/3052236331 learned_addr learned my addr 172.21.15.125:0/3052236331 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:39.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.860+0000 7fe927e00640 1 -- 172.21.15.125:0/3052236331 >> v1:172.21.15.137:6789/0 conn(0x7fe928143ae0 legacy=0x7fe9280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:39.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.860+0000 7fe927e00640 1 -- 172.21.15.125:0/3052236331 >> v1:172.21.15.125:6789/0 conn(0x7fe92815e1e0 legacy=0x7fe9280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.860+0000 7fe927e00640 1 -- 172.21.15.125:0/3052236331 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9280b2130 con 0x7fe9280cf9a0 2024-01-16T15:09:39.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.860+0000 7fe927400640 1 -- 172.21.15.125:0/3052236331 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031576751 0 0) 0x7fe9280b1ee0 con 0x7fe92815e1e0 2024-01-16T15:09:39.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.861+0000 7fe927e00640 1 --2- 172.21.15.125:0/3052236331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe92815dca0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fe9100097a0 tx=0x7fe91002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f4f22404ba3ae2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.861+0000 7fe927400640 1 -- 172.21.15.125:0/3052236331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe910031830 con 0x7fe9280cf9a0 2024-01-16T15:09:39.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.861+0000 7fe927400640 1 -- 172.21.15.125:0/3052236331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe910031990 con 0x7fe9280cf9a0 2024-01-16T15:09:39.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe927400640 1 -- 172.21.15.125:0/3052236331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe910031d20 con 0x7fe9280cf9a0 2024-01-16T15:09:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 -- 172.21.15.125:0/3052236331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 msgr2=0x7fe92815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 --2- 172.21.15.125:0/3052236331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe92815dca0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fe9100097a0 tx=0x7fe91002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 -- 172.21.15.125:0/3052236331 shutdown_connections 2024-01-16T15:09:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 --2- 172.21.15.125:0/3052236331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe92815dca0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 -- 172.21.15.125:0/3052236331 >> 172.21.15.125:0/3052236331 conn(0x7fe92806a800 msgr2=0x7fe92806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 -- 172.21.15.125:0/3052236331 shutdown_connections 2024-01-16T15:09:39.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.862+0000 7fe92fc00640 1 -- 172.21.15.125:0/3052236331 wait complete. 2024-01-16T15:09:39.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.864+0000 7fe92fc00640 1 Processor -- start 2024-01-16T15:09:39.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.864+0000 7fe92fc00640 1 -- start start 2024-01-16T15:09:39.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe928143ae0 0x7fe9280c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 0x7fe92813a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9281419d0 con 0x7fe9280cf9a0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92813ace0 con 0x7fe928143ae0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c4550 con 0x7fe92815e1e0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.865+0000 7fe92d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38722/0 (socket says 172.21.15.125:38722) 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 -- 172.21.15.125:0/1647139617 learned_addr learned my addr 172.21.15.125:0/1647139617 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92dc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 0x7fe92813a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe927e00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe928143ae0 0x7fe9280c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 -- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 msgr2=0x7fe92813a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 0x7fe92813a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 -- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe928143ae0 msgr2=0x7fe9280c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe928143ae0 0x7fe9280c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 -- 172.21.15.125:0/1647139617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe910009400 con 0x7fe9280cf9a0 2024-01-16T15:09:39.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92dc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 0x7fe92813a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.866+0000 7fe92d200640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fe91800c430 tx=0x7fe91800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.867+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918011790 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.867+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9180118f0 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.867+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918011ce0 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.867+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9280c4830 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.867+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9280c4df0 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.868+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9280675d0 con 0x7fe9280cf9a0 2024-01-16T15:09:39.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.869+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe91801c460 con 0x7fe9280cf9a0 2024-01-16T15:09:39.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.870+0000 7fe924c00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 0x7fe90006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.870+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe918088510 con 0x7fe9280cf9a0 2024-01-16T15:09:39.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.870+0000 7fe927e00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 0x7fe90006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.871+0000 7fe927e00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 0x7fe90006e410 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe910036000 tx=0x7fe910008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:39.874+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe918054e30 con 0x7fe9280cf9a0 2024-01-16T15:09:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.064+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe928067c00 con 0x7fe9280cf9a0 2024-01-16T15:09:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.065+0000 7fe924c00640 1 -- 172.21.15.125:0/1647139617 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe918058450 con 0x7fe9280cf9a0 2024-01-16T15:09:40.043 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:09:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 msgr2=0x7fe90006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 0x7fe90006e410 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe910036000 tx=0x7fe910008030 comp rx=0 tx=0).stop 2024-01-16T15:09:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 msgr2=0x7fe928141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fe91800c430 tx=0x7fe91800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 shutdown_connections 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe92815e1e0 0x7fe92813a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe928143ae0 0x7fe9280c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe90006bf50 0x7fe90006e410 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 --2- 172.21.15.125:0/1647139617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9280cf9a0 0x7fe928141bd0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.067+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 >> 172.21.15.125:0/1647139617 conn(0x7fe92806a800 msgr2=0x7fe92806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.068+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 shutdown_connections 2024-01-16T15:09:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.068+0000 7fe92fc00640 1 -- 172.21.15.125:0/1647139617 wait complete. 2024-01-16T15:09:40.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:09:40.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.285+0000 7fbb2f800640 1 Processor -- start 2024-01-16T15:09:40.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.285+0000 7fbb2f800640 1 -- start start 2024-01-16T15:09:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2f800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb28143ae0 0x7fbb280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2f800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbb280cf7d0 0x7fbb2815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2f800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb280cfc30 con 0x7fbb2815e1e0 2024-01-16T15:09:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2f800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb280b1ee0 con 0x7fbb28143ae0 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2f800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb280b2780 con 0x7fbb280cf7d0 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb28143ae0 0x7fbb280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2ce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb28143ae0 0x7fbb280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38732/0 (socket says 172.21.15.125:38732) 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2ce00640 1 -- 172.21.15.125:0/956769552 learned_addr learned my addr 172.21.15.125:0/956769552 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb1fc00640 1 --2- 172.21.15.125:0/956769552 >> v2:172.21.15.137:3300/0 conn(0x7fbb280cf7d0 0x7fbb2815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2ce00640 1 -- 172.21.15.125:0/956769552 >> v2:172.21.15.137:3300/0 conn(0x7fbb280cf7d0 msgr2=0x7fbb2815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.287+0000 7fbb2ce00640 1 --2- 172.21.15.125:0/956769552 >> v2:172.21.15.137:3300/0 conn(0x7fbb280cf7d0 0x7fbb2815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.288+0000 7fbb2ce00640 1 -- 172.21.15.125:0/956769552 >> v1:172.21.15.137:6789/0 conn(0x7fbb2815e1e0 legacy=0x7fbb2815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.288+0000 7fbb2ce00640 1 -- 172.21.15.125:0/956769552 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2815e990 con 0x7fbb28143ae0 2024-01-16T15:09:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb2ce00640 1 --2- 172.21.15.125:0/956769552 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb28143ae0 0x7fbb280cf290 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fbb280cb830 tx=0x7fbb1802e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee3451f7a14f131 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb27e00640 1 -- 172.21.15.125:0/956769552 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1802ea90 con 0x7fbb28143ae0 2024-01-16T15:09:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb27e00640 1 -- 172.21.15.125:0/956769552 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1802ebf0 con 0x7fbb28143ae0 2024-01-16T15:09:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb27e00640 1 -- 172.21.15.125:0/956769552 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18009600 con 0x7fbb28143ae0 2024-01-16T15:09:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb2f800640 1 -- 172.21.15.125:0/956769552 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb28143ae0 msgr2=0x7fbb280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.289+0000 7fbb2f800640 1 --2- 172.21.15.125:0/956769552 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb28143ae0 0x7fbb280cf290 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fbb280cb830 tx=0x7fbb1802e0f0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 -- 172.21.15.125:0/956769552 shutdown_connections 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 --2- 172.21.15.125:0/956769552 >> v2:172.21.15.137:3300/0 conn(0x7fbb280cf7d0 0x7fbb2815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 --2- 172.21.15.125:0/956769552 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb28143ae0 0x7fbb280cf290 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 -- 172.21.15.125:0/956769552 >> 172.21.15.125:0/956769552 conn(0x7fbb2806a800 msgr2=0x7fbb2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 -- 172.21.15.125:0/956769552 shutdown_connections 2024-01-16T15:09:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.290+0000 7fbb2f800640 1 -- 172.21.15.125:0/956769552 wait complete. 2024-01-16T15:09:40.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.291+0000 7fbb2f800640 1 Processor -- start 2024-01-16T15:09:40.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.291+0000 7fbb2f800640 1 -- start start 2024-01-16T15:09:40.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb280cf7d0 0x7fbb280b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb28143ae0 0x7fbb280aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb280afa60 con 0x7fbb2815e1e0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb280aef60 con 0x7fbb280cf7d0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.292+0000 7fbb2f800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2813eeb0 con 0x7fbb28143ae0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38742/0 (socket says 172.21.15.125:38742) 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 -- 172.21.15.125:0/154029704 learned_addr learned my addr 172.21.15.125:0/154029704 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 -- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb28143ae0 msgr2=0x7fbb280aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb28143ae0 0x7fbb280aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 -- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb280cf7d0 msgr2=0x7fbb280b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb280cf7d0 0x7fbb280b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 -- 172.21.15.125:0/154029704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1800bd40 con 0x7fbb2815e1e0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb2d800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c440 tx=0x7fbb2000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.293+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011790 con 0x7fbb2815e1e0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.294+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb200118f0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.294+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb280af1c0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.294+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011ca0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.294+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb280af780 con 0x7fbb2815e1e0 2024-01-16T15:09:40.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.295+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb280b2bf0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.296+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fbb2001c460 con 0x7fbb2815e1e0 2024-01-16T15:09:40.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.296+0000 7fbb25600640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.297+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb20088450 con 0x7fbb2815e1e0 2024-01-16T15:09:40.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.297+0000 7fbb2ce00640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.297+0000 7fbb2ce00640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 0x7fbb0006e3e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbb18037000 tx=0x7fbb18030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.301+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb280b2bf0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:40.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:40.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.494+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbb280b2bf0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.495+0000 7fbb25600640 1 -- 172.21.15.125:0/154029704 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbb280b2bf0 con 0x7fbb2815e1e0 2024-01-16T15:09:40.472 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:40.473 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:09:40.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.497+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 msgr2=0x7fbb0006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.497+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 0x7fbb0006e3e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbb18037000 tx=0x7fbb18030000 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.497+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 msgr2=0x7fbb2813e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.497+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c440 tx=0x7fbb2000c900 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 shutdown_connections 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb2815e1e0 0x7fbb2813e970 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb28143ae0 0x7fbb280aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 --2- 172.21.15.125:0/154029704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb280cf7d0 0x7fbb280b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 >> 172.21.15.125:0/154029704 conn(0x7fbb2806a800 msgr2=0x7fbb2815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 shutdown_connections 2024-01-16T15:09:40.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.498+0000 7fbb2f800640 1 -- 172.21.15.125:0/154029704 wait complete. 2024-01-16T15:09:40.493 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:09:40.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:09:40.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.715+0000 7f239be00640 1 Processor -- start 2024-01-16T15:09:40.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.716+0000 7f239be00640 1 -- start start 2024-01-16T15:09:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.717+0000 7f239be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f239c146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.717+0000 7f239be00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0b2e10 con 0x7f239c06c7c0 2024-01-16T15:09:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.717+0000 7f239be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c06cc20 con 0x7f239c071f00 2024-01-16T15:09:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.717+0000 7f239be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c06cec0 con 0x7f239c071410 2024-01-16T15:09:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f2393e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f239c06c7c0 0x7f239c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44550/0 (socket says 172.21.15.125:44550) 2024-01-16T15:09:40.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f2393e00640 1 -- 172.21.15.125:0/4050615530 learned_addr learned my addr 172.21.15.125:0/4050615530 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f239aa00640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f239c146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924687639 0 0) 0x7f239c0b2e10 con 0x7f239c06c7c0 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c003520 con 0x7f239c06c7c0 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.718+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378821312 0 0) 0x7f237c003520 con 0x7f239c06c7c0 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 msgr2=0x7f239c146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f239c146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 >> v1:172.21.15.137:6789/0 conn(0x7f239c071f00 legacy=0x7f239c166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c06d1a0 con 0x7f239c06c7c0 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2384001670 con 0x7f239c06c7c0 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 >> v1:172.21.15.125:6789/0 conn(0x7f239c06c7c0 legacy=0x7f239c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f237c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f237c003f20 0x7f237c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f237c004d00 0x7f237c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c06cec0 con 0x7f237c004d00 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c00ba60 con 0x7f237c003f20 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c00bc20 con 0x7f239c071410 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f239b400640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f237c004d00 0x7f237c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.719+0000 7f2393e00640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f237c003f20 0x7f237c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239aa00640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f237c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239b400640 1 -- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 msgr2=0x7f237c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239b400640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c071410 0x7f237c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239b400640 1 -- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f237c003f20 msgr2=0x7f237c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239b400640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f237c003f20 0x7f237c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.720+0000 7f239b400640 1 -- 172.21.15.125:0/4050615530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c003520 con 0x7f237c004d00 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.721+0000 7f239b400640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f237c004d00 0x7f237c005180 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f2394001e50 tx=0x7f237c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fe0724c3187688 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.721+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003b70 con 0x7f237c004d00 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.721+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394003cd0 con 0x7f237c004d00 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.721+0000 7f239a000640 1 -- 172.21.15.125:0/4050615530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394005310 con 0x7f237c004d00 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 -- 172.21.15.125:0/4050615530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f237c004d00 msgr2=0x7f237c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 --2- 172.21.15.125:0/4050615530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f237c004d00 0x7f237c005180 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f2394001e50 tx=0x7f237c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239b400640 1 -- 172.21.15.125:0/4050615530 reap_dead start 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 -- 172.21.15.125:0/4050615530 shutdown_connections 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 -- 172.21.15.125:0/4050615530 >> 172.21.15.125:0/4050615530 conn(0x7f239c06a800 msgr2=0x7f239c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 -- 172.21.15.125:0/4050615530 shutdown_connections 2024-01-16T15:09:40.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.722+0000 7f239be00640 1 -- 172.21.15.125:0/4050615530 wait complete. 2024-01-16T15:09:40.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.724+0000 7f239be00640 1 Processor -- start 2024-01-16T15:09:40.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.724+0000 7f239be00640 1 -- start start 2024-01-16T15:09:40.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c15bf00 0x7f239c1e35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f239c1e4490 0x7f239c1e4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c1e4e60 con 0x7f239c13d820 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c13eb50 con 0x7f239c1e4490 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f239be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c13ca60 con 0x7f239c15bf00 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f2393e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f2393e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38770/0 (socket says 172.21.15.125:38770) 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.725+0000 7f2393e00640 1 -- 172.21.15.125:0/2887545220 learned_addr learned my addr 172.21.15.125:0/2887545220 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f239aa00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c15bf00 0x7f239c1e35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 -- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c15bf00 msgr2=0x7f239c1e35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f239b400640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f239c1e4490 0x7f239c1e4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c15bf00 0x7f239c1e35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 -- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f239c1e4490 msgr2=0x7f239c1e4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f239c1e4490 0x7f239c1e4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 -- 172.21.15.125:0/2887545220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c13cd40 con 0x7f239c13d820 2024-01-16T15:09:40.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2393e00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f23840072c0 tx=0x7f2384007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238400f8e0 con 0x7f239c13d820 2024-01-16T15:09:40.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238400fa40 con 0x7f239c13d820 2024-01-16T15:09:40.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238400fdf0 con 0x7f239c13d820 2024-01-16T15:09:40.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c13d020 con 0x7f239c13d820 2024-01-16T15:09:40.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.726+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c15c2e0 con 0x7f239c13d820 2024-01-16T15:09:40.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.728+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c13d540 con 0x7f239c13d820 2024-01-16T15:09:40.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.728+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f238401d460 con 0x7f239c13d820 2024-01-16T15:09:40.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.730+0000 7f2392a00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 0x7f237406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.730+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2384088820 con 0x7f239c13d820 2024-01-16T15:09:40.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.730+0000 7f239aa00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 0x7f237406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.731+0000 7f239aa00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 0x7f237406e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f239c145680 tx=0x7f238c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.734+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f239c13d540 con 0x7f239c13d820 2024-01-16T15:09:40.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:40.927+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f239c13d540 con 0x7f239c13d820 2024-01-16T15:09:41.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.705+0000 7f2392a00640 1 -- 172.21.15.125:0/2887545220 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f239c13d540 con 0x7f239c13d820 2024-01-16T15:09:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 msgr2=0x7f237406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 0x7f237406e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f239c145680 tx=0x7f238c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 msgr2=0x7f239c13e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f23840072c0 tx=0x7f2384007780 comp rx=0 tx=0).stop 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 shutdown_connections 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f239c1e4490 0x7f239c1e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f237406bf50 0x7f237406e410 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f239c13d820 0x7f239c13e610 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 --2- 172.21.15.125:0/2887545220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f239c15bf00 0x7f239c1e35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.709+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 >> 172.21.15.125:0/2887545220 conn(0x7f239c06a800 msgr2=0x7f239c0b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.710+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 shutdown_connections 2024-01-16T15:09:41.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.710+0000 7f239be00640 1 -- 172.21.15.125:0/2887545220 wait complete. 2024-01-16T15:09:41.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:09:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.928+0000 7f6e3d600640 1 Processor -- start 2024-01-16T15:09:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.928+0000 7f6e3d600640 1 -- start start 2024-01-16T15:09:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e38143b40 0x7f6e380cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e380ce620 0x7f6e380b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e38143f20 con 0x7f6e3806e800 2024-01-16T15:09:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3d600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3806fda0 con 0x7f6e380ce620 2024-01-16T15:09:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3806ed00 con 0x7f6e38143b40 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.929+0000 7f6e3cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e380ce620 0x7f6e380b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e380ce620 0x7f6e380b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38784/0 (socket says 172.21.15.125:38784) 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 -- 172.21.15.125:0/4081301903 learned_addr learned my addr 172.21.15.125:0/4081301903 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e37e00640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e38143b40 0x7f6e380cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 -- 172.21.15.125:0/4081301903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e38143b40 msgr2=0x7f6e380cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e38143b40 0x7f6e380cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 -- 172.21.15.125:0/4081301903 >> v1:172.21.15.137:6789/0 conn(0x7f6e3806e800 legacy=0x7f6e380cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.930+0000 7f6e3cc00640 1 -- 172.21.15.125:0/4081301903 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3806efe0 con 0x7f6e380ce620 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.931+0000 7f6e3cc00640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e380ce620 0x7f6e380b3040 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000d150 tx=0x7f6e3002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71efbe6c9dfd565 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e37400640 1 -- 172.21.15.125:0/4081301903 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3002e9e0 con 0x7f6e380ce620 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e37400640 1 -- 172.21.15.125:0/4081301903 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3002eb40 con 0x7f6e380ce620 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e37400640 1 -- 172.21.15.125:0/4081301903 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30009600 con 0x7f6e380ce620 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 -- 172.21.15.125:0/4081301903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e380ce620 msgr2=0x7f6e380b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e380ce620 0x7f6e380b3040 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000d150 tx=0x7f6e3002e040 comp rx=0 tx=0).stop 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 -- 172.21.15.125:0/4081301903 shutdown_connections 2024-01-16T15:09:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e380ce620 0x7f6e380b3040 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 --2- 172.21.15.125:0/4081301903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e38143b40 0x7f6e380cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 -- 172.21.15.125:0/4081301903 >> 172.21.15.125:0/4081301903 conn(0x7f6e380b1d40 msgr2=0x7f6e380b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 -- 172.21.15.125:0/4081301903 shutdown_connections 2024-01-16T15:09:41.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.932+0000 7f6e3d600640 1 -- 172.21.15.125:0/4081301903 wait complete. 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.934+0000 7f6e3d600640 1 Processor -- start 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.934+0000 7f6e3d600640 1 -- start start 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e380ce620 0x7f6e38159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e38143b40 0x7f6e3815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3815a050 con 0x7f6e3806e800 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3815b600 con 0x7f6e38143b40 2024-01-16T15:09:41.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.935+0000 7f6e3d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3815b8a0 con 0x7f6e380ce620 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38796/0 (socket says 172.21.15.125:38796) 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 -- 172.21.15.125:0/2270829562 learned_addr learned my addr 172.21.15.125:0/2270829562 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e3cc00640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e38143b40 0x7f6e3815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e2f200640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e380ce620 0x7f6e38159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 -- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e380ce620 msgr2=0x7f6e38159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e380ce620 0x7f6e38159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 -- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e38143b40 msgr2=0x7f6e3815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e38143b40 0x7f6e3815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.936+0000 7f6e37e00640 1 -- 172.21.15.125:0/2270829562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3000bd40 con 0x7f6e3806e800 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e37e00640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800d800 tx=0x7f6e2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800b6c0 con 0x7f6e3806e800 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800b820 con 0x7f6e3806e800 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800bbb0 con 0x7f6e3806e800 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3815bde0 con 0x7f6e3806e800 2024-01-16T15:09:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.937+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3815c3a0 con 0x7f6e3806e800 2024-01-16T15:09:41.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.938+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e38067b60 con 0x7f6e3806e800 2024-01-16T15:09:41.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.940+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6e2800bd10 con 0x7f6e3806e800 2024-01-16T15:09:41.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.941+0000 7f6e34c00640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 0x7f6e1006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.941+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6e28089510 con 0x7f6e3806e800 2024-01-16T15:09:41.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.941+0000 7f6e2f200640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 0x7f6e1006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.942+0000 7f6e2f200640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 0x7f6e1006e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6e380cd250 tx=0x7f6e20009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:41.945+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e28055ec0 con 0x7f6e3806e800 2024-01-16T15:09:42.113 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.135+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6e38141b50 con 0x7f6e3806e800 2024-01-16T15:09:42.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.711+0000 7f6e34c00640 1 -- 172.21.15.125:0/2270829562 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6e280594e0 con 0x7f6e3806e800 2024-01-16T15:09:42.690 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T15:09:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.716+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 msgr2=0x7f6e1006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.716+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 0x7f6e1006e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6e380cd250 tx=0x7f6e20009060 comp rx=0 tx=0).stop 2024-01-16T15:09:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 msgr2=0x7f6e380b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800d800 tx=0x7f6e2800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 shutdown_connections 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e38143b40 0x7f6e3815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e380ce620 0x7f6e38159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6e1006bf20 0x7f6e1006e3e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 --2- 172.21.15.125:0/2270829562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e3806e800 0x7f6e380b5140 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 >> 172.21.15.125:0/2270829562 conn(0x7f6e380b1d40 msgr2=0x7f6e380af740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 shutdown_connections 2024-01-16T15:09:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.717+0000 7f6e3d600640 1 -- 172.21.15.125:0/2270829562 wait complete. 2024-01-16T15:09:42.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:09:42.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.936+0000 7f2006200640 1 Processor -- start 2024-01-16T15:09:42.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.937+0000 7f2006200640 1 -- start start 2024-01-16T15:09:42.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f2006200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f20000b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f2006200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000b3400 con 0x7f20000d17e0 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f2006200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000b27f0 con 0x7f200015d610 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f2006200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20000d1390 con 0x7f20000d1c40 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f1ffea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f20000d17e0 0x7f20000b3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44556/0 (socket says 172.21.15.125:44556) 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.938+0000 7f1ffea00640 1 -- 172.21.15.125:0/2558840993 learned_addr learned my addr 172.21.15.125:0/2558840993 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1fffe00640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f20000b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178599373 0 0) 0x7f20000b3400 con 0x7f20000d17e0 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003520 con 0x7f20000d17e0 2024-01-16T15:09:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618297075 0 0) 0x7f1fe0003520 con 0x7f20000d17e0 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 msgr2=0x7f20000b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f20000b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 >> v1:172.21.15.137:6789/0 conn(0x7f200015d610 legacy=0x7f20000ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000b2ad0 con 0x7f20000d17e0 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1fe80019f0 con 0x7f20000d17e0 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 >> v1:172.21.15.125:6789/0 conn(0x7f20000d17e0 legacy=0x7f20000b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f1fe0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f1fe00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.939+0000 7f1ffe000640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fe0004d00 0x7f1fe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20000d1390 con 0x7f1fe00039a0 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000ba30 con 0x7f1fe0004d00 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000bbf0 con 0x7f20000d1c40 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f1fe0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1ffea00640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f1fe00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fffe00640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fe0004d00 0x7f1fe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 -- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 msgr2=0x7f1fe00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20000d1c40 0x7f1fe00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 -- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fe0004d00 msgr2=0x7f1fe0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1fe0004d00 0x7f1fe0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.940+0000 7f1fff400640 1 -- 172.21.15.125:0/2558840993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0003520 con 0x7f1fe00039a0 2024-01-16T15:09:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.941+0000 7f1fff400640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f1fe0003dc0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0001720 tx=0x7f1fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e857ed922805855 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00052d0 con 0x7f1fe00039a0 2024-01-16T15:09:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff002e050 con 0x7f1fe00039a0 2024-01-16T15:09:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f1ffe000640 1 -- 172.21.15.125:0/2558840993 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00035f0 con 0x7f1fe00039a0 2024-01-16T15:09:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 -- 172.21.15.125:0/2558840993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 msgr2=0x7f1fe0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 --2- 172.21.15.125:0/2558840993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f1fe0003dc0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0001720 tx=0x7f1fe00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f1fffe00640 1 -- 172.21.15.125:0/2558840993 reap_dead start 2024-01-16T15:09:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 -- 172.21.15.125:0/2558840993 shutdown_connections 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 -- 172.21.15.125:0/2558840993 >> 172.21.15.125:0/2558840993 conn(0x7f200009c540 msgr2=0x7f20000d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 -- 172.21.15.125:0/2558840993 shutdown_connections 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.942+0000 7f2006200640 1 -- 172.21.15.125:0/2558840993 wait complete. 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.944+0000 7f2006200640 1 Processor -- start 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.944+0000 7f2006200640 1 -- start start 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2000155850 0x7f2000152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2000158c40 0x7f20001590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2000155cd0 con 0x7f1fe00039a0 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2000159600 con 0x7f2000155850 2024-01-16T15:09:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f2006200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2000156980 con 0x7f2000158c40 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f1fff400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f1fff400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38808/0 (socket says 172.21.15.125:38808) 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.945+0000 7f1fff400640 1 -- 172.21.15.125:0/3066589109 learned_addr learned my addr 172.21.15.125:0/3066589109 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1ffea00640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2000155850 0x7f2000152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 -- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2000158c40 msgr2=0x7f20001590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fffe00640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2000158c40 0x7f20001590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2000158c40 0x7f20001590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 -- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2000155850 msgr2=0x7f2000152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2000155850 0x7f2000152820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 -- 172.21.15.125:0/3066589109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20001539e0 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1fff400640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0008fd0 tx=0x7f1ff0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00076c0 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0007a30 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2000153cc0 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.946+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0006be0 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.947+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2000154280 con 0x7f1fe00039a0 2024-01-16T15:09:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.947+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2000155c30 con 0x7f1fe00039a0 2024-01-16T15:09:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.949+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f1ff0006d40 con 0x7f1fe00039a0 2024-01-16T15:09:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.951+0000 7f1ff7400640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 0x7f1fd806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.951+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1ff00aa7f0 con 0x7f1fe00039a0 2024-01-16T15:09:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.951+0000 7f1ffea00640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 0x7f1fd806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.952+0000 7f1ffea00640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 0x7f1fd806e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80013d0 tx=0x7f1fe80020c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:42.955+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ff00770b0 con 0x7f1fe00039a0 2024-01-16T15:09:43.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.182+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f20000675d0 con 0x7f1fe00039a0 2024-01-16T15:09:43.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.762+0000 7f1ff7400640 1 -- 172.21.15.125:0/3066589109 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v50) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1ff007a6d0 con 0x7f1fe00039a0 2024-01-16T15:09:43.741 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 12 and data pool 13 2024-01-16T15:09:43.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.775+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 msgr2=0x7f1fd806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.775+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 0x7f1fd806e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80013d0 tx=0x7f1fe80020c0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.775+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 msgr2=0x7f2000153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.775+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0008fd0 tx=0x7f1ff0001120 comp rx=0 tx=0).stop 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 shutdown_connections 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2000158c40 0x7f20001590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2000155850 0x7f2000152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1fd806bf50 0x7f1fd806e410 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 --2- 172.21.15.125:0/3066589109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00039a0 0x7f2000153400 secure :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0008fd0 tx=0x7f1ff0001120 comp rx=0 tx=0).stop 2024-01-16T15:09:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.777+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 >> 172.21.15.125:0/3066589109 conn(0x7f200009c540 msgr2=0x7f20000c96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:43.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.778+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 shutdown_connections 2024-01-16T15:09:43.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.779+0000 7f2006200640 1 -- 172.21.15.125:0/3066589109 wait complete. 2024-01-16T15:09:43.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:43.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.987+0000 7f468c200640 1 Processor -- start 2024-01-16T15:09:43.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.988+0000 7f468c200640 1 -- start start 2024-01-16T15:09:43.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.989+0000 7f468c200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f468415d380 0x7f4684070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.989+0000 7f468c200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f46840b1f00 0x7f4684070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.989+0000 7f468c200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468406ded0 con 0x7f46840b38c0 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.989+0000 7f468c200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46840b2e20 con 0x7f468415d380 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.989+0000 7f468c200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468406c470 con 0x7f46840b1f00 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f468415d380 0x7f4684070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f468415d380 0x7f4684070410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38816/0 (socket says 172.21.15.125:38816) 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 -- 172.21.15.125:0/3532814109 learned_addr learned my addr 172.21.15.125:0/3532814109 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4688e00640 1 --1- 172.21.15.125:0/3532814109 >> v1:172.21.15.125:6789/0 conn(0x7f46840b38c0 0x7f4684160f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44564/0 (socket says 172.21.15.125:44564) 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f468a200640 1 --2- 172.21.15.125:0/3532814109 >> v2:172.21.15.137:3300/0 conn(0x7f46840b1f00 0x7f4684070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 -- 172.21.15.125:0/3532814109 >> v2:172.21.15.137:3300/0 conn(0x7f46840b1f00 msgr2=0x7f4684070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 --2- 172.21.15.125:0/3532814109 >> v2:172.21.15.137:3300/0 conn(0x7f46840b1f00 0x7f4684070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 -- 172.21.15.125:0/3532814109 >> v1:172.21.15.125:6789/0 conn(0x7f46840b38c0 legacy=0x7f4684160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.990+0000 7f4689800640 1 -- 172.21.15.125:0/3532814109 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46840b4240 con 0x7f468415d380 2024-01-16T15:09:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.991+0000 7f4689800640 1 --2- 172.21.15.125:0/3532814109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f468415d380 0x7f4684070410 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f46840b43d0 tx=0x7f467402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b390765dd457168d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.991+0000 7f4683e00640 1 -- 172.21.15.125:0/3532814109 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674030860 con 0x7f468415d380 2024-01-16T15:09:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f4683e00640 1 -- 172.21.15.125:0/3532814109 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46740309c0 con 0x7f468415d380 2024-01-16T15:09:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f4683e00640 1 -- 172.21.15.125:0/3532814109 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674030d50 con 0x7f468415d380 2024-01-16T15:09:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 -- 172.21.15.125:0/3532814109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f468415d380 msgr2=0x7f4684070410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 --2- 172.21.15.125:0/3532814109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f468415d380 0x7f4684070410 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f46840b43d0 tx=0x7f467402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 -- 172.21.15.125:0/3532814109 shutdown_connections 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 --2- 172.21.15.125:0/3532814109 >> v2:172.21.15.137:3300/0 conn(0x7f46840b1f00 0x7f4684070950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 --2- 172.21.15.125:0/3532814109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f468415d380 0x7f4684070410 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 -- 172.21.15.125:0/3532814109 >> 172.21.15.125:0/3532814109 conn(0x7f46840b0c00 msgr2=0x7f46840b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 -- 172.21.15.125:0/3532814109 shutdown_connections 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.992+0000 7f468c200640 1 -- 172.21.15.125:0/3532814109 wait complete. 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.994+0000 7f468c200640 1 Processor -- start 2024-01-16T15:09:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.994+0000 7f468c200640 1 -- start start 2024-01-16T15:09:43.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46840b1f00 0x7f46840ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f468415d380 0x7f46840ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46840ade70 con 0x7f46840b38c0 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841f3190 con 0x7f468415d380 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f468c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4684153d30 con 0x7f46840b1f00 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f4688e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f4688e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38830/0 (socket says 172.21.15.125:38830) 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.995+0000 7f4688e00640 1 -- 172.21.15.125:0/1138275892 learned_addr learned my addr 172.21.15.125:0/1138275892 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4689800640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46840b1f00 0x7f46840ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f468a200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f468415d380 0x7f46840ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 -- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46840b1f00 msgr2=0x7f46840ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46840b1f00 0x7f46840ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 -- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f468415d380 msgr2=0x7f46840ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f468415d380 0x7f46840ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 -- 172.21.15.125:0/1138275892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740094d0 con 0x7f46840b38c0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4688e00640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c4d0 tx=0x7f466c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.996+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c007480 con 0x7f46840b38c0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.997+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c0075e0 con 0x7f46840b38c0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.997+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c0049e0 con 0x7f46840b38c0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.997+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4684153f80 con 0x7f46840b38c0 2024-01-16T15:09:43.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.997+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4684154540 con 0x7f46840b38c0 2024-01-16T15:09:43.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.998+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4684067640 con 0x7f46840b38c0 2024-01-16T15:09:43.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:43.999+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f466c004b40 con 0x7f46840b38c0 2024-01-16T15:09:43.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.000+0000 7f4681600640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 0x7f465c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.000+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f466c0878d0 con 0x7f46840b38c0 2024-01-16T15:09:43.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.001+0000 7f4689800640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 0x7f465c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.001+0000 7f4689800640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 0x7f465c06e3e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4674036000 tx=0x7f4674008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.005+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f466c053f80 con 0x7f46840b38c0 2024-01-16T15:09:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.194+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46840b2bd0 con 0x7f46840b38c0 2024-01-16T15:09:44.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.197+0000 7f4681600640 1 -- 172.21.15.125:0/1138275892 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f466c018020 con 0x7f46840b38c0 2024-01-16T15:09:44.176 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:44.176 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:43.185089+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:40.982379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:09:42.187190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 msgr2=0x7f465c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 0x7f465c06e3e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4674036000 tx=0x7f4674008030 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 msgr2=0x7f46840b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c4d0 tx=0x7f466c00c990 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 shutdown_connections 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f468415d380 0x7f46840ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f465c06bf20 0x7f465c06e3e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840b38c0 0x7f46840b0270 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 --2- 172.21.15.125:0/1138275892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46840b1f00 0x7f46840ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 >> 172.21.15.125:0/1138275892 conn(0x7f46840b0c00 msgr2=0x7f46840706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:44.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 shutdown_connections 2024-01-16T15:09:44.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.200+0000 7f468c200640 1 -- 172.21.15.125:0/1138275892 wait complete. 2024-01-16T15:09:44.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:44.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.421+0000 7ff33fa00640 1 Processor -- start 2024-01-16T15:09:44.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.421+0000 7ff33fa00640 1 -- start start 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33fa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff338143ae0 0x7ff3380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3380cf7d0 0x7ff33815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33fa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3380cfc30 con 0x7ff33815e1e0 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33fa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3380b1ee0 con 0x7ff338143ae0 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3380b2780 con 0x7ff3380cf7d0 2024-01-16T15:09:44.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff338143ae0 0x7ff3380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff338143ae0 0x7ff3380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38842/0 (socket says 172.21.15.125:38842) 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.423+0000 7ff33d000640 1 -- 172.21.15.125:0/2126117466 learned_addr learned my addr 172.21.15.125:0/2126117466 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.424+0000 7ff33d000640 1 -- 172.21.15.125:0/2126117466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3380cf7d0 msgr2=0x7ff33815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.424+0000 7ff33d000640 1 --2- 172.21.15.125:0/2126117466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3380cf7d0 0x7ff33815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.424+0000 7ff33d000640 1 -- 172.21.15.125:0/2126117466 >> v1:172.21.15.137:6789/0 conn(0x7ff33815e1e0 legacy=0x7ff33815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.424+0000 7ff33d000640 1 -- 172.21.15.125:0/2126117466 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33815e990 con 0x7ff338143ae0 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.425+0000 7ff33d000640 1 --2- 172.21.15.125:0/2126117466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff338143ae0 0x7ff3380cf290 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff328009870 tx=0x7ff32802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30e5d5e1bf1e29f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.425+0000 7ff337e00640 1 -- 172.21.15.125:0/2126117466 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328031640 con 0x7ff338143ae0 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.425+0000 7ff337e00640 1 -- 172.21.15.125:0/2126117466 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3280317a0 con 0x7ff338143ae0 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.425+0000 7ff337e00640 1 -- 172.21.15.125:0/2126117466 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328031b30 con 0x7ff338143ae0 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 -- 172.21.15.125:0/2126117466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff338143ae0 msgr2=0x7ff3380cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 --2- 172.21.15.125:0/2126117466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff338143ae0 0x7ff3380cf290 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff328009870 tx=0x7ff32802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 -- 172.21.15.125:0/2126117466 shutdown_connections 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 --2- 172.21.15.125:0/2126117466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3380cf7d0 0x7ff33815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 --2- 172.21.15.125:0/2126117466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff338143ae0 0x7ff3380cf290 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 -- 172.21.15.125:0/2126117466 >> 172.21.15.125:0/2126117466 conn(0x7ff33806a800 msgr2=0x7ff33806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 -- 172.21.15.125:0/2126117466 shutdown_connections 2024-01-16T15:09:44.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.426+0000 7ff33fa00640 1 -- 172.21.15.125:0/2126117466 wait complete. 2024-01-16T15:09:44.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.428+0000 7ff33fa00640 1 Processor -- start 2024-01-16T15:09:44.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.428+0000 7ff33fa00640 1 -- start start 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.429+0000 7ff33fa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.429+0000 7ff33fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff338143ae0 0x7ff33815cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff33815e1e0 0x7ff3380a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33fa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338159fd0 con 0x7ff3380cf7d0 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33815d4b0 con 0x7ff33815e1e0 2024-01-16T15:09:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3381f1cb0 con 0x7ff338143ae0 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38856/0 (socket says 172.21.15.125:38856) 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 -- 172.21.15.125:0/1357596972 learned_addr learned my addr 172.21.15.125:0/1357596972 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 -- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff338143ae0 msgr2=0x7ff33815cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff338143ae0 0x7ff33815cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 -- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff33815e1e0 msgr2=0x7ff3380a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff33815e1e0 0x7ff3380a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 -- 172.21.15.125:0/1357596972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280094d0 con 0x7ff3380cf7d0 2024-01-16T15:09:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.430+0000 7ff33d000640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff328009870 tx=0x7ff3280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.431+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32803ac60 con 0x7ff3380cf7d0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.431+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32803ade0 con 0x7ff3380cf7d0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.431+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328044630 con 0x7ff3380cf7d0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.431+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3381f1f90 con 0x7ff3380cf7d0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.431+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3381f2550 con 0x7ff3380cf7d0 2024-01-16T15:09:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.432+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3380675d0 con 0x7ff3380cf7d0 2024-01-16T15:09:44.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.433+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff32802f050 con 0x7ff3380cf7d0 2024-01-16T15:09:44.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.434+0000 7ff335600640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 0x7ff31006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.434+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3280aaa40 con 0x7ff3380cf7d0 2024-01-16T15:09:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.434+0000 7ff32fc00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 0x7ff31006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.435+0000 7ff32fc00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 0x7ff31006e3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff338070d40 tx=0x7ff320006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.440+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff328077080 con 0x7ff3380cf7d0 2024-01-16T15:09:44.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.630+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff338070d70 con 0x7ff3380cf7d0 2024-01-16T15:09:44.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.633+0000 7ff335600640 1 -- 172.21.15.125:0/1357596972 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16621 (secure 0 0 0) 0x7ff32807a6a0 con 0x7ff3380cf7d0 2024-01-16T15:09:44.611 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:44.612 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:43.185089+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:40.982379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:09:42.187190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.635+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 msgr2=0x7ff31006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 0x7ff31006e3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff338070d40 tx=0x7ff320006c90 comp rx=0 tx=0).stop 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 msgr2=0x7ff338159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff328009870 tx=0x7ff3280046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 shutdown_connections 2024-01-16T15:09:44.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff33815e1e0 0x7ff3380a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff338143ae0 0x7ff33815cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff31006bf20 0x7ff31006e3e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 --2- 172.21.15.125:0/1357596972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3380cf7d0 0x7ff338159a90 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 >> 172.21.15.125:0/1357596972 conn(0x7ff33806a800 msgr2=0x7ff3380b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 shutdown_connections 2024-01-16T15:09:44.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.636+0000 7ff33fa00640 1 -- 172.21.15.125:0/1357596972 wait complete. 2024-01-16T15:09:44.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:09:44.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.859+0000 7f8e74600640 1 Processor -- start 2024-01-16T15:09:44.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.859+0000 7f8e74600640 1 -- start start 2024-01-16T15:09:44.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e74600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8e6c143ae0 0x7f8e6c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e74600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c0cfc30 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e74600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c0b1ee0 con 0x7f8e6c0cf7d0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e74600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0b2780 con 0x7f8e6c143ae0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e72600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8e6c15e3b0 0x7f8e6c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44580/0 (socket says 172.21.15.125:44580) 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e72600640 1 -- 172.21.15.125:0/1534576250 learned_addr learned my addr 172.21.15.125:0/1534576250 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e71c00640 1 --2- 172.21.15.125:0/1534576250 >> v2:172.21.15.137:3300/0 conn(0x7f8e6c143ae0 0x7f8e6c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.861+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421851881 0 0) 0x7f8e6c0cfc30 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c005730 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659613758 0 0) 0x7f8e4c005730 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 >> v2:172.21.15.137:3300/0 conn(0x7f8e6c143ae0 msgr2=0x7f8e6c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 --2- 172.21.15.125:0/1534576250 >> v2:172.21.15.137:3300/0 conn(0x7f8e6c143ae0 0x7f8e6c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 >> v1:172.21.15.137:6789/0 conn(0x7f8e6c0cf7d0 legacy=0x7f8e6c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c15e990 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8e64001670 con 0x7f8e6c15e3b0 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e71c00640 1 --2- 172.21.15.125:0/1534576250 >> v2:172.21.15.137:3300/0 conn(0x7f8e6c143ae0 0x7f8e6c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 >> v1:172.21.15.125:6789/0 conn(0x7f8e6c15e3b0 legacy=0x7f8e6c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c005b80 0x7f8e4c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 0x7f8e4c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e4c007390 0x7f8e4c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c0b2780 con 0x7f8e4c0069d0 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c00e2e0 con 0x7f8e4c007390 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c00e4a0 con 0x7f8e4c005b80 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.862+0000 7f8e71200640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 0x7f8e4c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71c00640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c005b80 0x7f8e4c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71200640 1 -- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c005b80 msgr2=0x7f8e4c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e72600640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e4c007390 0x7f8e4c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71200640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c005b80 0x7f8e4c005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71200640 1 -- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e4c007390 msgr2=0x7f8e4c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71200640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e4c007390 0x7f8e4c00db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e71200640 1 -- 172.21.15.125:0/1534576250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c005730 con 0x7f8e4c0069d0 2024-01-16T15:09:44.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.863+0000 7f8e72600640 1 -- 172.21.15.125:0/1534576250 reap_dead start 2024-01-16T15:09:44.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.864+0000 7f8e71200640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 0x7f8e4c006e50 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8e54001e50 tx=0x7f8e4c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d7b01a4d7734c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.864+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54003b70 con 0x7f8e4c0069d0 2024-01-16T15:09:44.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.864+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54003cd0 con 0x7f8e4c0069d0 2024-01-16T15:09:44.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.864+0000 7f8e6be00640 1 -- 172.21.15.125:0/1534576250 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54005310 con 0x7f8e4c0069d0 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 -- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 msgr2=0x7f8e4c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 0x7f8e4c006e50 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8e54001e50 tx=0x7f8e4c007a00 comp rx=0 tx=0).stop 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 -- 172.21.15.125:0/1534576250 shutdown_connections 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 --2- 172.21.15.125:0/1534576250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e4c0069d0 0x7f8e4c006e50 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 -- 172.21.15.125:0/1534576250 >> 172.21.15.125:0/1534576250 conn(0x7f8e6c06a800 msgr2=0x7f8e6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 -- 172.21.15.125:0/1534576250 shutdown_connections 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.865+0000 7f8e74600640 1 -- 172.21.15.125:0/1534576250 wait complete. 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.867+0000 7f8e74600640 1 Processor -- start 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.867+0000 7f8e74600640 1 -- start start 2024-01-16T15:09:44.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c0069d0 0x7f8e6c159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e6c0c6570 0x7f8e6c0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c0c4d00 con 0x7f8e6c0c5120 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c0c6f30 con 0x7f8e6c0c6570 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e74600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c159e40 con 0x7f8e4c0069d0 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38878/0 (socket says 172.21.15.125:38878) 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 -- 172.21.15.125:0/3322758276 learned_addr learned my addr 172.21.15.125:0/3322758276 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 -- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c0069d0 msgr2=0x7f8e6c159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c0069d0 0x7f8e6c159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 -- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e6c0c6570 msgr2=0x7f8e6c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e6c0c6570 0x7f8e6c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.868+0000 7f8e71200640 1 -- 172.21.15.125:0/3322758276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0c4080 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e71200640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402bf70 tx=0x7f8e540014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5403e070 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54003e40 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c0c4360 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5402f640 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.869+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c0c4920 con 0x7f8e6c0c5120 2024-01-16T15:09:44.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.870+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c0c4ab0 con 0x7f8e6c0c5120 2024-01-16T15:09:44.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.871+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8e5402f7a0 con 0x7f8e6c0c5120 2024-01-16T15:09:44.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.872+0000 7f8e69600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 0x7f8e4406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.873+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e540a9ee0 con 0x7f8e6c0c5120 2024-01-16T15:09:44.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.873+0000 7f8e71c00640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 0x7f8e4406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.873+0000 7f8e71c00640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 0x7f8e4406e3e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c006e80 tx=0x7f8e5c006df0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:44.877+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e54076610 con 0x7f8e6c0c5120 2024-01-16T15:09:45.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.103+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8e6c0675d0 con 0x7f8e6c0c5120 2024-01-16T15:09:45.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.781+0000 7f8e69600640 1 -- 172.21.15.125:0/3322758276 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8e54047030 con 0x7f8e6c0c5120 2024-01-16T15:09:45.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 msgr2=0x7f8e4406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 0x7f8e4406e3e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c006e80 tx=0x7f8e5c006df0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 msgr2=0x7f8e6c0c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402bf70 tx=0x7f8e540014a0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 shutdown_connections 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e6c0c6570 0x7f8e6c0c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8e4406bf20 0x7f8e4406e3e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e6c0c5120 0x7f8e6c0c55a0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 --2- 172.21.15.125:0/3322758276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e4c0069d0 0x7f8e6c159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.785+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 >> 172.21.15.125:0/3322758276 conn(0x7f8e6c06a800 msgr2=0x7f8e6c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:45.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.786+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 shutdown_connections 2024-01-16T15:09:45.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:45.786+0000 7f8e74600640 1 -- 172.21.15.125:0/3322758276 wait complete. 2024-01-16T15:09:45.784 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:09:45.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:09:45.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.012+0000 7f87e1a00640 1 Processor -- start 2024-01-16T15:09:45.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.012+0000 7f87e1a00640 1 -- start start 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87e1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc0cf9a0 0x7f87dc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87e1a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87dc15e1e0 0x7f87dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87e1a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0b1ee0 con 0x7f87dc143ae0 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87e1a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc0b2780 con 0x7f87dc15e1e0 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87e1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc15e990 con 0x7f87dc0cf9a0 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87db400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87dc15e1e0 0x7f87dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87db400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87dc15e1e0 0x7f87dc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38894/0 (socket says 172.21.15.125:38894) 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.014+0000 7f87db400640 1 -- 172.21.15.125:0/106098641 learned_addr learned my addr 172.21.15.125:0/106098641 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:45.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.015+0000 7f87db400640 1 -- 172.21.15.125:0/106098641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc0cf9a0 msgr2=0x7f87dc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.015+0000 7f87da000640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc0cf9a0 0x7f87dc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.015+0000 7f87db400640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc0cf9a0 0x7f87dc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.015+0000 7f87db400640 1 -- 172.21.15.125:0/106098641 >> v1:172.21.15.137:6789/0 conn(0x7f87dc143ae0 legacy=0x7f87dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:45.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.015+0000 7f87db400640 1 -- 172.21.15.125:0/106098641 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0b2130 con 0x7f87dc15e1e0 2024-01-16T15:09:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.016+0000 7f87db400640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc15e1e0 0x7f87dc0702c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a34223a80979a01 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:45.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.016+0000 7f87d9600640 1 -- 172.21.15.125:0/106098641 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc031640 con 0x7f87dc15e1e0 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.016+0000 7f87d9600640 1 -- 172.21.15.125:0/106098641 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc0317a0 con 0x7f87dc15e1e0 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.016+0000 7f87d9600640 1 -- 172.21.15.125:0/106098641 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc031b30 con 0x7f87dc15e1e0 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 -- 172.21.15.125:0/106098641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc15e1e0 msgr2=0x7f87dc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc15e1e0 0x7f87dc0702c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 -- 172.21.15.125:0/106098641 shutdown_connections 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc15e1e0 0x7f87dc0702c0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 --2- 172.21.15.125:0/106098641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc0cf9a0 0x7f87dc15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 -- 172.21.15.125:0/106098641 >> 172.21.15.125:0/106098641 conn(0x7f87dc06a800 msgr2=0x7f87dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 -- 172.21.15.125:0/106098641 shutdown_connections 2024-01-16T15:09:45.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.017+0000 7f87e1a00640 1 -- 172.21.15.125:0/106098641 wait complete. 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.019+0000 7f87e1a00640 1 Processor -- start 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.019+0000 7f87e1a00640 1 -- start start 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87dc143ae0 0x7f87dc0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 0x7f87dc0af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0c5590 con 0x7f87dc0cf9a0 2024-01-16T15:09:45.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0c59b0 con 0x7f87dc143ae0 2024-01-16T15:09:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87e1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0c4910 con 0x7f87dc15e1e0 2024-01-16T15:09:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87daa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87daa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38908/0 (socket says 172.21.15.125:38908) 2024-01-16T15:09:45.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87daa00640 1 -- 172.21.15.125:0/418586075 learned_addr learned my addr 172.21.15.125:0/418586075 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87da000640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87dc143ae0 0x7f87dc0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87daa00640 1 -- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 msgr2=0x7f87dc0af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.020+0000 7f87db400640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 0x7f87dc0af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87daa00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 0x7f87dc0af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87daa00640 1 -- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87dc143ae0 msgr2=0x7f87dc0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87daa00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87dc143ae0 0x7f87dc0c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87daa00640 1 -- 172.21.15.125:0/418586075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc0094d0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87db400640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 0x7f87dc0af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87daa00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7c0 tx=0x7f87d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d400aa50 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d400abb0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc0c4bf0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4012420 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.021+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc0c51b0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.023+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc0c5800 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.023+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f87d4012580 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.024+0000 7f87d2a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 0x7f87b406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.024+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87d4088ab0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.024+0000 7f87da000640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 0x7f87b406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.025+0000 7f87da000640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 0x7f87b406e3e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f87c4006fd0 tx=0x7f87c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.028+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87d4055150 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.260+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f87dc0675d0 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.784+0000 7f87d2a00640 1 -- 172.21.15.125:0/418586075 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7f87d4058770 con 0x7f87dc0cf9a0 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.787+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 msgr2=0x7f87b406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.787+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 0x7f87b406e3e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f87c4006fd0 tx=0x7f87c4008040 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.787+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 msgr2=0x7f87dc159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.787+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7c0 tx=0x7f87d400cc80 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 shutdown_connections 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87dc15e1e0 0x7f87dc0af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87dc143ae0 0x7f87dc0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f87b406bf20 0x7f87b406e3e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 --2- 172.21.15.125:0/418586075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87dc0cf9a0 0x7f87dc159cb0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 >> 172.21.15.125:0/418586075 conn(0x7f87dc06a800 msgr2=0x7f87dc0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 shutdown_connections 2024-01-16T15:09:46.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:46.788+0000 7f87e1a00640 1 -- 172.21.15.125:0/418586075 wait complete. 2024-01-16T15:09:46.787 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:09:46.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.016+0000 7f0a80c00640 1 Processor -- start 2024-01-16T15:09:46.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.016+0000 7f0a80c00640 1 -- start start 2024-01-16T15:09:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.017+0000 7f0a80c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0a7c13e360 0x7f0a7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.017+0000 7f0a80c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0a7c15f9c0 0x7f0a7c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.017+0000 7f0a80c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c15c5f0 con 0x7f0a7c13df80 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.017+0000 7f0a80c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c06c470 con 0x7f0a7c15f9c0 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.017+0000 7f0a80c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c06c750 con 0x7f0a7c13e360 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a7a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0a7c15f9c0 0x7f0a7c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a79e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0a7c13df80 0x7f0a7c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39520/0 (socket says 172.21.15.125:39520) 2024-01-16T15:09:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a79e00640 1 -- 172.21.15.125:0/1281696703 learned_addr learned my addr 172.21.15.125:0/1281696703 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a7a800640 1 -- 172.21.15.125:0/1281696703 >> v2:172.21.15.137:3300/0 conn(0x7f0a7c13e360 msgr2=0x7f0a7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a7a800640 1 --2- 172.21.15.125:0/1281696703 >> v2:172.21.15.137:3300/0 conn(0x7f0a7c13e360 0x7f0a7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a7a800640 1 -- 172.21.15.125:0/1281696703 >> v1:172.21.15.125:6789/0 conn(0x7f0a7c13df80 legacy=0x7f0a7c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.018+0000 7f0a7a800640 1 -- 172.21.15.125:0/1281696703 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c06ca30 con 0x7f0a7c15f9c0 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.019+0000 7f0a7a800640 1 --2- 172.21.15.125:0/1281696703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c15f9c0 0x7f0a7c0cf7d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c06ded0 tx=0x7f0a7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d13af7b3abc406 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a79400640 1 -- 172.21.15.125:0/1281696703 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74031830 con 0x7f0a7c15f9c0 2024-01-16T15:09:46.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a79400640 1 -- 172.21.15.125:0/1281696703 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74031990 con 0x7f0a7c15f9c0 2024-01-16T15:09:46.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a79400640 1 -- 172.21.15.125:0/1281696703 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74031d20 con 0x7f0a7c15f9c0 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 -- 172.21.15.125:0/1281696703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c15f9c0 msgr2=0x7f0a7c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 --2- 172.21.15.125:0/1281696703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c15f9c0 0x7f0a7c0cf7d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c06ded0 tx=0x7f0a7402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 -- 172.21.15.125:0/1281696703 shutdown_connections 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 --2- 172.21.15.125:0/1281696703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c15f9c0 0x7f0a7c0cf7d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 --2- 172.21.15.125:0/1281696703 >> v2:172.21.15.137:3300/0 conn(0x7f0a7c13e360 0x7f0a7c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.020+0000 7f0a80c00640 1 -- 172.21.15.125:0/1281696703 >> 172.21.15.125:0/1281696703 conn(0x7f0a7c06a800 msgr2=0x7f0a7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.021+0000 7f0a80c00640 1 -- 172.21.15.125:0/1281696703 shutdown_connections 2024-01-16T15:09:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.021+0000 7f0a80c00640 1 -- 172.21.15.125:0/1281696703 wait complete. 2024-01-16T15:09:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.023+0000 7f0a80c00640 1 Processor -- start 2024-01-16T15:09:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.023+0000 7f0a80c00640 1 -- start start 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 0x7f0a7c13cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a7c15f9c0 0x7f0a7c0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c151490 con 0x7f0a7c13e360 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c151770 con 0x7f0a7c13df80 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.024+0000 7f0a80c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c151a50 con 0x7f0a7c15f9c0 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39328/0 (socket says 172.21.15.125:39328) 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 -- 172.21.15.125:0/2013028755 learned_addr learned my addr 172.21.15.125:0/2013028755 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a79e00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 0x7f0a7c13cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 -- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a7c15f9c0 msgr2=0x7f0a7c0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a7a800640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a7c15f9c0 0x7f0a7c0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a7c15f9c0 0x7f0a7c0b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 -- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 msgr2=0x7f0a7c13cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 0x7f0a7c13cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a71200640 1 -- 172.21.15.125:0/2013028755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74009400 con 0x7f0a7c13e360 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.025+0000 7f0a79e00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 0x7f0a7c13cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a71200640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400b450 tx=0x7f0a6400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64011890 con 0x7f0a7c13e360 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a640119f0 con 0x7f0a7c13e360 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64011da0 con 0x7f0a7c13e360 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c151d30 con 0x7f0a7c13e360 2024-01-16T15:09:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.026+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c13d2e0 con 0x7f0a7c13e360 2024-01-16T15:09:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.027+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c0675d0 con 0x7f0a7c13e360 2024-01-16T15:09:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.028+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0a6401a400 con 0x7f0a7c13e360 2024-01-16T15:09:47.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.029+0000 7f0a72a00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 0x7f0a5406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.029+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a64088d30 con 0x7f0a7c13e360 2024-01-16T15:09:47.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.029+0000 7f0a79e00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 0x7f0a5406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.030+0000 7f0a79e00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 0x7f0a5406e3e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0012e0 tx=0x7f0a6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.034+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a640552b0 con 0x7f0a7c13e360 2024-01-16T15:09:47.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.261+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a7c06df00 con 0x7f0a7c13e360 2024-01-16T15:09:47.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.263+0000 7f0a72a00640 1 -- 172.21.15.125:0/2013028755 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f0a64015090 con 0x7f0a7c13e360 2024-01-16T15:09:47.241 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:47.241 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/4149833712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":4149833712},{"type":"v1","addr":"172.21.15.137:6834","nonce":4149833712}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":6235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2076499076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2076499076},{"type":"v1","addr":"172.21.15.137:6835","nonce":2076499076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8172,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3233056488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3233056488},{"type":"v1","addr":"172.21.15.125:6835","nonce":3233056488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:09:43.185893+0000","modified":"2024-01-16T15:09:46.262939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8181":{"gid":8181,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/589280594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":589280594},{"type":"v1","addr":"172.21.15.125:6837","nonce":589280594}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:09:47.241 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 54 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 msgr2=0x7f0a5406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 0x7f0a5406e3e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0012e0 tx=0x7f0a6c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 msgr2=0x7f0a7c13d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400b450 tx=0x7f0a6400b910 comp rx=0 tx=0).stop 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 shutdown_connections 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a7c15f9c0 0x7f0a7c0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0a5406bf20 0x7f0a5406e3e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7c13e360 0x7f0a7c13d670 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 --2- 172.21.15.125:0/2013028755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a7c13df80 0x7f0a7c13cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 >> 172.21.15.125:0/2013028755 conn(0x7f0a7c06a800 msgr2=0x7f0a7c0cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.265+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 shutdown_connections 2024-01-16T15:09:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.266+0000 7f0a80c00640 1 -- 172.21.15.125:0/2013028755 wait complete. 2024-01-16T15:09:47.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:47.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.492+0000 7f604e200640 1 Processor -- start 2024-01-16T15:09:47.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.492+0000 7f604e200640 1 -- start start 2024-01-16T15:09:47.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f604e200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f604813df80 0x7f604815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f604e200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604806c470 con 0x7f604813e360 2024-01-16T15:09:47.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f604e200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604806c750 con 0x7f60480cf9a0 2024-01-16T15:09:47.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f604e200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604806ca30 con 0x7f604813df80 2024-01-16T15:09:47.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f603fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f604813e360 0x7f60480cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39524/0 (socket says 172.21.15.125:39524) 2024-01-16T15:09:47.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f603fe00640 1 -- 172.21.15.125:0/1345173943 learned_addr learned my addr 172.21.15.125:0/1345173943 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f6047400640 1 --2- 172.21.15.125:0/1345173943 >> v2:172.21.15.137:3300/0 conn(0x7f604813df80 0x7f604815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.494+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764746909 0 0) 0x7f604806c470 con 0x7f604813e360 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003520 con 0x7f604813e360 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1524287857 0 0) 0x7f6028003520 con 0x7f604813e360 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 >> v2:172.21.15.137:3300/0 conn(0x7f604813df80 msgr2=0x7f604815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 --2- 172.21.15.125:0/1345173943 >> v2:172.21.15.137:3300/0 conn(0x7f604813df80 0x7f604815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 >> v1:172.21.15.137:6789/0 conn(0x7f60480cf9a0 legacy=0x7f60480cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604806cd10 con 0x7f604813e360 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6030001670 con 0x7f604813e360 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 >> v1:172.21.15.125:6789/0 conn(0x7f604813e360 legacy=0x7f60480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028003970 0x7f6028003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60280047c0 0x7f6028004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 0x7f602800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604806ca30 con 0x7f6028005180 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602800c020 con 0x7f60280047c0 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602800c1e0 con 0x7f6028003970 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.495+0000 7f6047e00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 0x7f602800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047400640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028003970 0x7f6028003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f603fe00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60280047c0 0x7f6028004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047e00640 1 -- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028003970 msgr2=0x7f6028003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047e00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028003970 0x7f6028003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047e00640 1 -- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60280047c0 msgr2=0x7f6028004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047e00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60280047c0 0x7f6028004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.496+0000 7f6047e00640 1 -- 172.21.15.125:0/1345173943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028003520 con 0x7f6028005180 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.497+0000 7f6047e00640 1 -- 172.21.15.125:0/1345173943 reap_dead start 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.497+0000 7f6047e00640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 0x7f602800b980 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6040001e20 tx=0x7f60280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4615bbab6189f65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.497+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040003b70 con 0x7f6028005180 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.497+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040003cd0 con 0x7f6028005180 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.497+0000 7f6046a00640 1 -- 172.21.15.125:0/1345173943 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040005310 con 0x7f6028005180 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 -- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 msgr2=0x7f602800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 0x7f602800b980 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6040001e20 tx=0x7f60280057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 -- 172.21.15.125:0/1345173943 shutdown_connections 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 --2- 172.21.15.125:0/1345173943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028005180 0x7f602800b980 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 -- 172.21.15.125:0/1345173943 >> 172.21.15.125:0/1345173943 conn(0x7f604806a800 msgr2=0x7f604806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 -- 172.21.15.125:0/1345173943 shutdown_connections 2024-01-16T15:09:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.498+0000 7f604e200640 1 -- 172.21.15.125:0/1345173943 wait complete. 2024-01-16T15:09:47.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.500+0000 7f604e200640 1 Processor -- start 2024-01-16T15:09:47.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.500+0000 7f604e200640 1 -- start start 2024-01-16T15:09:47.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 0x7f604813d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6048150fb0 0x7f60481f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60481505a0 con 0x7f6048151430 2024-01-16T15:09:47.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60481f1650 con 0x7f6048150fb0 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f604e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60481f1930 con 0x7f6028005180 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f6047e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f6047e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39350/0 (socket says 172.21.15.125:39350) 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.501+0000 7f6047e00640 1 -- 172.21.15.125:0/3883574640 learned_addr learned my addr 172.21.15.125:0/3883574640 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f603fe00640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6048150fb0 0x7f60481f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047400640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 0x7f604813d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 -- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 msgr2=0x7f604813d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 0x7f604813d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 -- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6048150fb0 msgr2=0x7f60481f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6048150fb0 0x7f60481f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 -- 172.21.15.125:0/3883574640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60481507b0 con 0x7f6048151430 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047400640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 0x7f604813d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f6047e00640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6040001f40 tx=0x7f60400014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040006d50 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040007030 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400073c0 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.502+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6048150a90 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.503+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604813dd10 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.504+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048150c80 con 0x7f6048151430 2024-01-16T15:09:47.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.504+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6040007520 con 0x7f6048151430 2024-01-16T15:09:47.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.505+0000 7f603f400640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 0x7f602006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.506+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60400aa3f0 con 0x7f6048151430 2024-01-16T15:09:47.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.506+0000 7f6047400640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 0x7f602006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.506+0000 7f6047400640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 0x7f602006e410 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6038009980 tx=0x7f6038008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.510+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6040076ba0 con 0x7f6048151430 2024-01-16T15:09:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.697+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60480675d0 con 0x7f6048151430 2024-01-16T15:09:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.700+0000 7f603f400640 1 -- 172.21.15.125:0/3883574640 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16608 (secure 0 0 0) 0x7f6040047020 con 0x7f6048151430 2024-01-16T15:09:47.678 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:47.678 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":86,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:09:44.760615+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:40.982379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:09:42.187190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 msgr2=0x7f602006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 0x7f602006e410 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6038009980 tx=0x7f6038008040 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 msgr2=0x7f60481f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6040001f40 tx=0x7f60400014a0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 shutdown_connections 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f602006bf50 0x7f602006e410 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6048151430 0x7f60481f1060 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6048150fb0 0x7f60481f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 --2- 172.21.15.125:0/3883574640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6028005180 0x7f604813d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 >> 172.21.15.125:0/3883574640 conn(0x7f604806a800 msgr2=0x7f604815d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 shutdown_connections 2024-01-16T15:09:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.703+0000 7f604e200640 1 -- 172.21.15.125:0/3883574640 wait complete. 2024-01-16T15:09:47.700 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:09:47.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:09:47.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.931+0000 7f8b90800640 1 Processor -- start 2024-01-16T15:09:47.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.931+0000 7f8b90800640 1 -- start start 2024-01-16T15:09:47.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.932+0000 7f8b90800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b88072230 0x7f8b88150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.932+0000 7f8b90800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8b8815da10 0x7f8b880b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.932+0000 7f8b90800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b88071190 con 0x7f8b880cf290 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.932+0000 7f8b90800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b88072610 con 0x7f8b88072230 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.932+0000 7f8b90800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b88071470 con 0x7f8b8815da10 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b88072230 0x7f8b88150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8d400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8b880cf290 0x7f8b8815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39530/0 (socket says 172.21.15.125:39530) 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8d400640 1 -- 172.21.15.125:0/1202696660 learned_addr learned my addr 172.21.15.125:0/1202696660 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8de00640 1 -- 172.21.15.125:0/1202696660 >> v2:172.21.15.137:3300/0 conn(0x7f8b8815da10 msgr2=0x7f8b880b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8de00640 1 --2- 172.21.15.125:0/1202696660 >> v2:172.21.15.137:3300/0 conn(0x7f8b8815da10 0x7f8b880b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8de00640 1 -- 172.21.15.125:0/1202696660 >> v1:172.21.15.125:6789/0 conn(0x7f8b880cf290 legacy=0x7f8b8815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.933+0000 7f8b8de00640 1 -- 172.21.15.125:0/1202696660 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88071750 con 0x7f8b88072230 2024-01-16T15:09:47.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.934+0000 7f8b8de00640 1 --2- 172.21.15.125:0/1202696660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b88150760 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8b880cb740 tx=0x7f8b7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8fb626d5c36be8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b8ca00640 1 -- 172.21.15.125:0/1202696660 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031830 con 0x7f8b88072230 2024-01-16T15:09:47.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b8ca00640 1 -- 172.21.15.125:0/1202696660 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78031990 con 0x7f8b88072230 2024-01-16T15:09:47.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b8ca00640 1 -- 172.21.15.125:0/1202696660 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031d20 con 0x7f8b88072230 2024-01-16T15:09:47.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 -- 172.21.15.125:0/1202696660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 msgr2=0x7f8b88150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 --2- 172.21.15.125:0/1202696660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b88150760 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8b880cb740 tx=0x7f8b7802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 -- 172.21.15.125:0/1202696660 shutdown_connections 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 --2- 172.21.15.125:0/1202696660 >> v2:172.21.15.137:3300/0 conn(0x7f8b8815da10 0x7f8b880b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 --2- 172.21.15.125:0/1202696660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b88150760 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 -- 172.21.15.125:0/1202696660 >> 172.21.15.125:0/1202696660 conn(0x7f8b8806a800 msgr2=0x7f8b8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 -- 172.21.15.125:0/1202696660 shutdown_connections 2024-01-16T15:09:47.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.935+0000 7f8b90800640 1 -- 172.21.15.125:0/1202696660 wait complete. 2024-01-16T15:09:47.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.937+0000 7f8b90800640 1 Processor -- start 2024-01-16T15:09:47.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.937+0000 7f8b90800640 1 -- start start 2024-01-16T15:09:47.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b880cf290 0x7f8b8814daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 0x7f8b8815f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8814e030 con 0x7f8b88072230 2024-01-16T15:09:47.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880ccf30 con 0x7f8b8815da10 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b90800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880cd1a0 con 0x7f8b880cf290 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b8de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b8de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39372/0 (socket says 172.21.15.125:39372) 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.938+0000 7f8b8de00640 1 -- 172.21.15.125:0/1130832709 learned_addr learned my addr 172.21.15.125:0/1130832709 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8d400640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b880cf290 0x7f8b8814daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8e800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 0x7f8b8815f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 -- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b880cf290 msgr2=0x7f8b8814daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b880cf290 0x7f8b8814daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 -- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 msgr2=0x7f8b8815f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 0x7f8b8815f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 -- 172.21.15.125:0/1130832709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78009400 con 0x7f8b88072230 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8e800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 0x7f8b8815f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b8de00640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009890 tx=0x7f8b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.939+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7803ba70 con 0x7f8b88072230 2024-01-16T15:09:47.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.940+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7803bbd0 con 0x7f8b88072230 2024-01-16T15:09:47.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.940+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780453f0 con 0x7f8b88072230 2024-01-16T15:09:47.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.940+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b880cd480 con 0x7f8b88072230 2024-01-16T15:09:47.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.940+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b880add70 con 0x7f8b88072230 2024-01-16T15:09:47.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.941+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b880675d0 con 0x7f8b88072230 2024-01-16T15:09:47.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.942+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8b78045550 con 0x7f8b88072230 2024-01-16T15:09:47.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.943+0000 7f8b86000640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 0x7f8b6006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.943+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b780aaac0 con 0x7f8b88072230 2024-01-16T15:09:47.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.943+0000 7f8b8d400640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 0x7f8b6006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.944+0000 7f8b8d400640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 0x7f8b6006e410 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b8814e380 tx=0x7f8b70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:47.947+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b78077160 con 0x7f8b88072230 2024-01-16T15:09:48.100 INFO:tasks.ceph.mds.c.smithi125.stderr:2024-01-16T15:09:48.122+0000 7f1050c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:09:48.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.135+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b880cd310 con 0x7f8b6006bf50 2024-01-16T15:09:48.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.162+0000 7f8b86000640 1 -- 172.21.15.125:0/1130832709 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286716 (secure 0 0 0) 0x7f8b880cd310 con 0x7f8b6006bf50 2024-01-16T15:09:48.144 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:48.145 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:09:48.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.167+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 msgr2=0x7f8b6006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.167+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 0x7f8b6006e410 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b8814e380 tx=0x7f8b70009040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 msgr2=0x7f8b880b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009890 tx=0x7f8b780046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 shutdown_connections 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815da10 0x7f8b8815f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b880cf290 0x7f8b8814daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8b6006bf50 0x7f8b6006e410 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 --2- 172.21.15.125:0/1130832709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b88072230 0x7f8b880b1fe0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 >> 172.21.15.125:0/1130832709 conn(0x7f8b8806a800 msgr2=0x7f8b8814e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 shutdown_connections 2024-01-16T15:09:48.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.168+0000 7f8b90800640 1 -- 172.21.15.125:0/1130832709 wait complete. 2024-01-16T15:09:48.165 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-01-16T15:09:46.226109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583124,"kb_used_data":9304,"kb_used_omap":12,"kb_used_meta":573683,"kb_avail":754391596,"statfs":{"total":773094113280,"available":772496994304,"internally_reserved":0,"allocated":9527296,"data_stored":4724469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":587451952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.517919"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.775925+0000","last_change":"2024-01-16T15:09:43.774102+0000","last_active":"2024-01-16T15:09:43.775925+0000","last_peered":"2024-01-16T15:09:43.775925+0000","last_clean":"2024-01-16T15:09:43.775925+0000","last_became_active":"2024-01-16T15:09:43.773782+0000","last_became_peered":"2024-01-16T15:09:43.773782+0000","last_unstale":"2024-01-16T15:09:43.775925+0000","last_undegraded":"2024-01-16T15:09:43.775925+0000","last_fullsized":"2024-01-16T15:09:43.775925+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:56.423756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785645+0000","last_change":"2024-01-16T15:09:43.778037+0000","last_active":"2024-01-16T15:09:43.785645+0000","last_peered":"2024-01-16T15:09:43.785645+0000","last_clean":"2024-01-16T15:09:43.785645+0000","last_became_active":"2024-01-16T15:09:43.777636+0000","last_became_peered":"2024-01-16T15:09:43.777636+0000","last_unstale":"2024-01-16T15:09:43.785645+0000","last_undegraded":"2024-01-16T15:09:43.785645+0000","last_fullsized":"2024-01-16T15:09:43.785645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778701+0000","last_change":"2024-01-16T15:09:43.775325+0000","last_active":"2024-01-16T15:09:43.778701+0000","last_peered":"2024-01-16T15:09:43.778701+0000","last_clean":"2024-01-16T15:09:43.778701+0000","last_became_active":"2024-01-16T15:09:43.775026+0000","last_became_peered":"2024-01-16T15:09:43.775026+0000","last_unstale":"2024-01-16T15:09:43.778701+0000","last_undegraded":"2024-01-16T15:09:43.778701+0000","last_fullsized":"2024-01-16T15:09:43.778701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:02.629184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.776584+0000","last_change":"2024-01-16T15:09:43.770829+0000","last_active":"2024-01-16T15:09:43.776584+0000","last_peered":"2024-01-16T15:09:43.776584+0000","last_clean":"2024-01-16T15:09:43.776584+0000","last_became_active":"2024-01-16T15:09:43.770508+0000","last_became_peered":"2024-01-16T15:09:43.770508+0000","last_unstale":"2024-01-16T15:09:43.776584+0000","last_undegraded":"2024-01-16T15:09:43.776584+0000","last_fullsized":"2024-01-16T15:09:43.776584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:47.331839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.758989+0000","last_change":"2024-01-16T15:09:43.757572+0000","last_active":"2024-01-16T15:09:43.758989+0000","last_peered":"2024-01-16T15:09:43.758989+0000","last_clean":"2024-01-16T15:09:43.758989+0000","last_became_active":"2024-01-16T15:09:43.756203+0000","last_became_peered":"2024-01-16T15:09:43.756203+0000","last_unstale":"2024-01-16T15:09:43.758989+0000","last_undegraded":"2024-01-16T15:09:43.758989+0000","last_fullsized":"2024-01-16T15:09:43.758989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:12.643360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773695+0000","last_change":"2024-01-16T15:09:43.772253+0000","last_active":"2024-01-16T15:09:43.773695+0000","last_peered":"2024-01-16T15:09:43.773695+0000","last_clean":"2024-01-16T15:09:43.773695+0000","last_became_active":"2024-01-16T15:09:43.771947+0000","last_became_peered":"2024-01-16T15:09:43.771947+0000","last_unstale":"2024-01-16T15:09:43.773695+0000","last_undegraded":"2024-01-16T15:09:43.773695+0000","last_fullsized":"2024-01-16T15:09:43.773695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774039+0000","last_change":"2024-01-16T15:09:43.769924+0000","last_active":"2024-01-16T15:09:43.774039+0000","last_peered":"2024-01-16T15:09:43.774039+0000","last_clean":"2024-01-16T15:09:43.774039+0000","last_became_active":"2024-01-16T15:09:43.769568+0000","last_became_peered":"2024-01-16T15:09:43.769568+0000","last_unstale":"2024-01-16T15:09:43.774039+0000","last_undegraded":"2024-01-16T15:09:43.774039+0000","last_fullsized":"2024-01-16T15:09:43.774039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:09.677709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773508+0000","last_change":"2024-01-16T15:09:43.772085+0000","last_active":"2024-01-16T15:09:43.773508+0000","last_peered":"2024-01-16T15:09:43.773508+0000","last_clean":"2024-01-16T15:09:43.773508+0000","last_became_active":"2024-01-16T15:09:43.771343+0000","last_became_peered":"2024-01-16T15:09:43.771343+0000","last_unstale":"2024-01-16T15:09:43.773508+0000","last_undegraded":"2024-01-16T15:09:43.773508+0000","last_fullsized":"2024-01-16T15:09:43.773508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782448+0000","last_change":"2024-01-16T15:09:43.781845+0000","last_active":"2024-01-16T15:09:43.782448+0000","last_peered":"2024-01-16T15:09:43.782448+0000","last_clean":"2024-01-16T15:09:43.782448+0000","last_became_active":"2024-01-16T15:09:43.781257+0000","last_became_peered":"2024-01-16T15:09:43.781257+0000","last_unstale":"2024-01-16T15:09:43.782448+0000","last_undegraded":"2024-01-16T15:09:43.782448+0000","last_fullsized":"2024-01-16T15:09:43.782448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:31.482814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779985+0000","last_change":"2024-01-16T15:09:43.777950+0000","last_active":"2024-01-16T15:09:43.779985+0000","last_peered":"2024-01-16T15:09:43.779985+0000","last_clean":"2024-01-16T15:09:43.779985+0000","last_became_active":"2024-01-16T15:09:43.777679+0000","last_became_peered":"2024-01-16T15:09:43.777679+0000","last_unstale":"2024-01-16T15:09:43.779985+0000","last_undegraded":"2024-01-16T15:09:43.779985+0000","last_fullsized":"2024-01-16T15:09:43.779985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:11.348680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767038+0000","last_change":"2024-01-16T15:09:43.762396+0000","last_active":"2024-01-16T15:09:43.767038+0000","last_peered":"2024-01-16T15:09:43.767038+0000","last_clean":"2024-01-16T15:09:43.767038+0000","last_became_active":"2024-01-16T15:09:43.762095+0000","last_became_peered":"2024-01-16T15:09:43.762095+0000","last_unstale":"2024-01-16T15:09:43.767038+0000","last_undegraded":"2024-01-16T15:09:43.767038+0000","last_fullsized":"2024-01-16T15:09:43.767038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:02:31.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779957+0000","last_change":"2024-01-16T15:09:43.778763+0000","last_active":"2024-01-16T15:09:43.779957+0000","last_peered":"2024-01-16T15:09:43.779957+0000","last_clean":"2024-01-16T15:09:43.779957+0000","last_became_active":"2024-01-16T15:09:43.778347+0000","last_became_peered":"2024-01-16T15:09:43.778347+0000","last_unstale":"2024-01-16T15:09:43.779957+0000","last_undegraded":"2024-01-16T15:09:43.779957+0000","last_fullsized":"2024-01-16T15:09:43.779957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:37.388699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.762244+0000","last_change":"2024-01-16T15:09:43.760825+0000","last_active":"2024-01-16T15:09:43.762244+0000","last_peered":"2024-01-16T15:09:43.762244+0000","last_clean":"2024-01-16T15:09:43.762244+0000","last_became_active":"2024-01-16T15:09:43.760469+0000","last_became_peered":"2024-01-16T15:09:43.760469+0000","last_unstale":"2024-01-16T15:09:43.762244+0000","last_undegraded":"2024-01-16T15:09:43.762244+0000","last_fullsized":"2024-01-16T15:09:43.762244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:59.113702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785400+0000","last_change":"2024-01-16T15:09:43.779898+0000","last_active":"2024-01-16T15:09:43.785400+0000","last_peered":"2024-01-16T15:09:43.785400+0000","last_clean":"2024-01-16T15:09:43.785400+0000","last_became_active":"2024-01-16T15:09:43.779597+0000","last_became_peered":"2024-01-16T15:09:43.779597+0000","last_unstale":"2024-01-16T15:09:43.785400+0000","last_undegraded":"2024-01-16T15:09:43.785400+0000","last_fullsized":"2024-01-16T15:09:43.785400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:01.741647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778006+0000","last_change":"2024-01-16T15:09:43.769398+0000","last_active":"2024-01-16T15:09:43.778006+0000","last_peered":"2024-01-16T15:09:43.778006+0000","last_clean":"2024-01-16T15:09:43.778006+0000","last_became_active":"2024-01-16T15:09:43.766691+0000","last_became_peered":"2024-01-16T15:09:43.766691+0000","last_unstale":"2024-01-16T15:09:43.778006+0000","last_undegraded":"2024-01-16T15:09:43.778006+0000","last_fullsized":"2024-01-16T15:09:43.778006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:02.629184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779292+0000","last_change":"2024-01-16T15:09:43.776862+0000","last_active":"2024-01-16T15:09:43.779292+0000","last_peered":"2024-01-16T15:09:43.779292+0000","last_clean":"2024-01-16T15:09:43.779292+0000","last_became_active":"2024-01-16T15:09:43.774835+0000","last_became_peered":"2024-01-16T15:09:43.774835+0000","last_unstale":"2024-01-16T15:09:43.779292+0000","last_undegraded":"2024-01-16T15:09:43.779292+0000","last_fullsized":"2024-01-16T15:09:43.779292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:19.955097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778142+0000","last_change":"2024-01-16T15:09:43.773016+0000","last_active":"2024-01-16T15:09:43.778142+0000","last_peered":"2024-01-16T15:09:43.778142+0000","last_clean":"2024-01-16T15:09:43.778142+0000","last_became_active":"2024-01-16T15:09:43.772699+0000","last_became_peered":"2024-01-16T15:09:43.772699+0000","last_unstale":"2024-01-16T15:09:43.778142+0000","last_undegraded":"2024-01-16T15:09:43.778142+0000","last_fullsized":"2024-01-16T15:09:43.778142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.783014+0000","last_change":"2024-01-16T15:09:43.775801+0000","last_active":"2024-01-16T15:09:43.783014+0000","last_peered":"2024-01-16T15:09:43.783014+0000","last_clean":"2024-01-16T15:09:43.783014+0000","last_became_active":"2024-01-16T15:09:43.775396+0000","last_became_peered":"2024-01-16T15:09:43.775396+0000","last_unstale":"2024-01-16T15:09:43.783014+0000","last_undegraded":"2024-01-16T15:09:43.783014+0000","last_fullsized":"2024-01-16T15:09:43.783014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:25.304917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.784732+0000","last_change":"2024-01-16T15:09:43.782812+0000","last_active":"2024-01-16T15:09:43.784732+0000","last_peered":"2024-01-16T15:09:43.784732+0000","last_clean":"2024-01-16T15:09:43.784732+0000","last_became_active":"2024-01-16T15:09:43.782518+0000","last_became_peered":"2024-01-16T15:09:43.782518+0000","last_unstale":"2024-01-16T15:09:43.784732+0000","last_undegraded":"2024-01-16T15:09:43.784732+0000","last_fullsized":"2024-01-16T15:09:43.784732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778507+0000","last_change":"2024-01-16T15:09:43.773259+0000","last_active":"2024-01-16T15:09:43.778507+0000","last_peered":"2024-01-16T15:09:43.778507+0000","last_clean":"2024-01-16T15:09:43.778507+0000","last_became_active":"2024-01-16T15:09:43.772826+0000","last_became_peered":"2024-01-16T15:09:43.772826+0000","last_unstale":"2024-01-16T15:09:43.778507+0000","last_undegraded":"2024-01-16T15:09:43.778507+0000","last_fullsized":"2024-01-16T15:09:43.778507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:23.256626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.787001+0000","last_change":"2024-01-16T15:09:43.785355+0000","last_active":"2024-01-16T15:09:43.787001+0000","last_peered":"2024-01-16T15:09:43.787001+0000","last_clean":"2024-01-16T15:09:43.787001+0000","last_became_active":"2024-01-16T15:09:43.784834+0000","last_became_peered":"2024-01-16T15:09:43.784834+0000","last_unstale":"2024-01-16T15:09:43.787001+0000","last_undegraded":"2024-01-16T15:09:43.787001+0000","last_fullsized":"2024-01-16T15:09:43.787001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:03:35.595574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772101+0000","last_change":"2024-01-16T15:09:43.767460+0000","last_active":"2024-01-16T15:09:43.772101+0000","last_peered":"2024-01-16T15:09:43.772101+0000","last_clean":"2024-01-16T15:09:43.772101+0000","last_became_active":"2024-01-16T15:09:43.767122+0000","last_became_peered":"2024-01-16T15:09:43.767122+0000","last_unstale":"2024-01-16T15:09:43.772101+0000","last_undegraded":"2024-01-16T15:09:43.772101+0000","last_fullsized":"2024-01-16T15:09:43.772101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:16.327868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.780822+0000","last_change":"2024-01-16T15:09:43.779826+0000","last_active":"2024-01-16T15:09:43.780822+0000","last_peered":"2024-01-16T15:09:43.780822+0000","last_clean":"2024-01-16T15:09:43.780822+0000","last_became_active":"2024-01-16T15:09:43.779564+0000","last_became_peered":"2024-01-16T15:09:43.779564+0000","last_unstale":"2024-01-16T15:09:43.780822+0000","last_undegraded":"2024-01-16T15:09:43.780822+0000","last_fullsized":"2024-01-16T15:09:43.780822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:31.043085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785980+0000","last_change":"2024-01-16T15:09:43.781216+0000","last_active":"2024-01-16T15:09:43.785980+0000","last_peered":"2024-01-16T15:09:43.785980+0000","last_clean":"2024-01-16T15:09:43.785980+0000","last_became_active":"2024-01-16T15:09:43.780920+0000","last_became_peered":"2024-01-16T15:09:43.780920+0000","last_unstale":"2024-01-16T15:09:43.785980+0000","last_undegraded":"2024-01-16T15:09:43.785980+0000","last_fullsized":"2024-01-16T15:09:43.785980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:19.098300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779399+0000","last_change":"2024-01-16T15:09:43.771225+0000","last_active":"2024-01-16T15:09:43.779399+0000","last_peered":"2024-01-16T15:09:43.779399+0000","last_clean":"2024-01-16T15:09:43.779399+0000","last_became_active":"2024-01-16T15:09:43.770963+0000","last_became_peered":"2024-01-16T15:09:43.770963+0000","last_unstale":"2024-01-16T15:09:43.779399+0000","last_undegraded":"2024-01-16T15:09:43.779399+0000","last_fullsized":"2024-01-16T15:09:43.779399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:04.751584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767934+0000","last_change":"2024-01-16T15:09:43.766897+0000","last_active":"2024-01-16T15:09:43.767934+0000","last_peered":"2024-01-16T15:09:43.767934+0000","last_clean":"2024-01-16T15:09:43.767934+0000","last_became_active":"2024-01-16T15:09:43.766365+0000","last_became_peered":"2024-01-16T15:09:43.766365+0000","last_unstale":"2024-01-16T15:09:43.767934+0000","last_undegraded":"2024-01-16T15:09:43.767934+0000","last_fullsized":"2024-01-16T15:09:43.767934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:22.072338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772984+0000","last_change":"2024-01-16T15:09:43.769605+0000","last_active":"2024-01-16T15:09:43.772984+0000","last_peered":"2024-01-16T15:09:43.772984+0000","last_clean":"2024-01-16T15:09:43.772984+0000","last_became_active":"2024-01-16T15:09:43.769346+0000","last_became_peered":"2024-01-16T15:09:43.769346+0000","last_unstale":"2024-01-16T15:09:43.772984+0000","last_undegraded":"2024-01-16T15:09:43.772984+0000","last_fullsized":"2024-01-16T15:09:43.772984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:16.918822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773263+0000","last_change":"2024-01-16T15:09:43.770542+0000","last_active":"2024-01-16T15:09:43.773263+0000","last_peered":"2024-01-16T15:09:43.773263+0000","last_clean":"2024-01-16T15:09:43.773263+0000","last_became_active":"2024-01-16T15:09:43.770081+0000","last_became_peered":"2024-01-16T15:09:43.770081+0000","last_unstale":"2024-01-16T15:09:43.773263+0000","last_undegraded":"2024-01-16T15:09:43.773263+0000","last_fullsized":"2024-01-16T15:09:43.773263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:17.550072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779903+0000","last_change":"2024-01-16T15:09:43.772406+0000","last_active":"2024-01-16T15:09:43.779903+0000","last_peered":"2024-01-16T15:09:43.779903+0000","last_clean":"2024-01-16T15:09:43.779903+0000","last_became_active":"2024-01-16T15:09:43.771755+0000","last_became_peered":"2024-01-16T15:09:43.771755+0000","last_unstale":"2024-01-16T15:09:43.779903+0000","last_undegraded":"2024-01-16T15:09:43.779903+0000","last_fullsized":"2024-01-16T15:09:43.779903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:03:35.595574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786904+0000","last_change":"2024-01-16T15:09:43.782157+0000","last_active":"2024-01-16T15:09:43.786904+0000","last_peered":"2024-01-16T15:09:43.786904+0000","last_clean":"2024-01-16T15:09:43.786904+0000","last_became_active":"2024-01-16T15:09:43.780596+0000","last_became_peered":"2024-01-16T15:09:43.780596+0000","last_unstale":"2024-01-16T15:09:43.786904+0000","last_undegraded":"2024-01-16T15:09:43.786904+0000","last_fullsized":"2024-01-16T15:09:43.786904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:16.563833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782882+0000","last_change":"2024-01-16T15:09:43.781969+0000","last_active":"2024-01-16T15:09:43.782882+0000","last_peered":"2024-01-16T15:09:43.782882+0000","last_clean":"2024-01-16T15:09:43.782882+0000","last_became_active":"2024-01-16T15:09:43.781636+0000","last_became_peered":"2024-01-16T15:09:43.781636+0000","last_unstale":"2024-01-16T15:09:43.782882+0000","last_undegraded":"2024-01-16T15:09:43.782882+0000","last_fullsized":"2024-01-16T15:09:43.782882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:53.684987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774810+0000","last_change":"2024-01-16T15:09:43.771219+0000","last_active":"2024-01-16T15:09:43.774810+0000","last_peered":"2024-01-16T15:09:43.774810+0000","last_clean":"2024-01-16T15:09:43.774810+0000","last_became_active":"2024-01-16T15:09:43.769133+0000","last_became_peered":"2024-01-16T15:09:43.769133+0000","last_unstale":"2024-01-16T15:09:43.774810+0000","last_undegraded":"2024-01-16T15:09:43.774810+0000","last_fullsized":"2024-01-16T15:09:43.774810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:02.316062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778695+0000","last_change":"2024-01-16T15:09:43.775716+0000","last_active":"2024-01-16T15:09:43.778695+0000","last_peered":"2024-01-16T15:09:43.778695+0000","last_clean":"2024-01-16T15:09:43.778695+0000","last_became_active":"2024-01-16T15:09:43.774133+0000","last_became_peered":"2024-01-16T15:09:43.774133+0000","last_unstale":"2024-01-16T15:09:43.778695+0000","last_undegraded":"2024-01-16T15:09:43.778695+0000","last_fullsized":"2024-01-16T15:09:43.778695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:34.688354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"85'4","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793573+0000","last_change":"2024-01-16T15:09:42.750178+0000","last_active":"2024-01-16T15:09:43.793573+0000","last_peered":"2024-01-16T15:09:43.793573+0000","last_clean":"2024-01-16T15:09:43.793573+0000","last_became_active":"2024-01-16T15:09:42.749455+0000","last_became_peered":"2024-01-16T15:09:42.749455+0000","last_unstale":"2024-01-16T15:09:43.793573+0000","last_undegraded":"2024-01-16T15:09:43.793573+0000","last_fullsized":"2024-01-16T15:09:43.793573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:11.632760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786244+0000","last_change":"2024-01-16T15:09:43.785576+0000","last_active":"2024-01-16T15:09:43.786244+0000","last_peered":"2024-01-16T15:09:43.786244+0000","last_clean":"2024-01-16T15:09:43.786244+0000","last_became_active":"2024-01-16T15:09:43.785207+0000","last_became_peered":"2024-01-16T15:09:43.785207+0000","last_unstale":"2024-01-16T15:09:43.786244+0000","last_undegraded":"2024-01-16T15:09:43.786244+0000","last_fullsized":"2024-01-16T15:09:43.786244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.405061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793014+0000","last_change":"2024-01-16T15:09:42.742043+0000","last_active":"2024-01-16T15:09:43.793014+0000","last_peered":"2024-01-16T15:09:43.793014+0000","last_clean":"2024-01-16T15:09:43.793014+0000","last_became_active":"2024-01-16T15:09:42.741705+0000","last_became_peered":"2024-01-16T15:09:42.741705+0000","last_unstale":"2024-01-16T15:09:43.793014+0000","last_undegraded":"2024-01-16T15:09:43.793014+0000","last_fullsized":"2024-01-16T15:09:43.793014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:15.120356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773244+0000","last_change":"2024-01-16T15:09:43.770366+0000","last_active":"2024-01-16T15:09:43.773244+0000","last_peered":"2024-01-16T15:09:43.773244+0000","last_clean":"2024-01-16T15:09:43.773244+0000","last_became_active":"2024-01-16T15:09:43.769726+0000","last_became_peered":"2024-01-16T15:09:43.769726+0000","last_unstale":"2024-01-16T15:09:43.773244+0000","last_undegraded":"2024-01-16T15:09:43.773244+0000","last_fullsized":"2024-01-16T15:09:43.773244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:45.696910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.797479+0000","last_change":"2024-01-16T15:09:42.736595+0000","last_active":"2024-01-16T15:09:43.797479+0000","last_peered":"2024-01-16T15:09:43.797479+0000","last_clean":"2024-01-16T15:09:43.797479+0000","last_became_active":"2024-01-16T15:09:42.736326+0000","last_became_peered":"2024-01-16T15:09:42.736326+0000","last_unstale":"2024-01-16T15:09:43.797479+0000","last_undegraded":"2024-01-16T15:09:43.797479+0000","last_fullsized":"2024-01-16T15:09:43.797479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:57:33.787581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778719+0000","last_change":"2024-01-16T15:09:43.775841+0000","last_active":"2024-01-16T15:09:43.778719+0000","last_peered":"2024-01-16T15:09:43.778719+0000","last_clean":"2024-01-16T15:09:43.778719+0000","last_became_active":"2024-01-16T15:09:43.775529+0000","last_became_peered":"2024-01-16T15:09:43.775529+0000","last_unstale":"2024-01-16T15:09:43.778719+0000","last_undegraded":"2024-01-16T15:09:43.778719+0000","last_fullsized":"2024-01-16T15:09:43.778719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:45.396536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.797209+0000","last_change":"2024-01-16T15:09:42.737705+0000","last_active":"2024-01-16T15:09:43.797209+0000","last_peered":"2024-01-16T15:09:43.797209+0000","last_clean":"2024-01-16T15:09:43.797209+0000","last_became_active":"2024-01-16T15:09:42.736886+0000","last_became_peered":"2024-01-16T15:09:42.736886+0000","last_unstale":"2024-01-16T15:09:43.797209+0000","last_undegraded":"2024-01-16T15:09:43.797209+0000","last_fullsized":"2024-01-16T15:09:43.797209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.908221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786940+0000","last_change":"2024-01-16T15:09:43.783088+0000","last_active":"2024-01-16T15:09:43.786940+0000","last_peered":"2024-01-16T15:09:43.786940+0000","last_clean":"2024-01-16T15:09:43.786940+0000","last_became_active":"2024-01-16T15:09:43.782811+0000","last_became_peered":"2024-01-16T15:09:43.782811+0000","last_unstale":"2024-01-16T15:09:43.786940+0000","last_undegraded":"2024-01-16T15:09:43.786940+0000","last_fullsized":"2024-01-16T15:09:43.786940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.953127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791626+0000","last_change":"2024-01-16T15:09:42.741198+0000","last_active":"2024-01-16T15:09:43.791626+0000","last_peered":"2024-01-16T15:09:43.791626+0000","last_clean":"2024-01-16T15:09:43.791626+0000","last_became_active":"2024-01-16T15:09:42.740824+0000","last_became_peered":"2024-01-16T15:09:42.740824+0000","last_unstale":"2024-01-16T15:09:43.791626+0000","last_undegraded":"2024-01-16T15:09:43.791626+0000","last_fullsized":"2024-01-16T15:09:43.791626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:22.877064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778308+0000","last_change":"2024-01-16T15:09:43.775602+0000","last_active":"2024-01-16T15:09:43.778308+0000","last_peered":"2024-01-16T15:09:43.778308+0000","last_clean":"2024-01-16T15:09:43.778308+0000","last_became_active":"2024-01-16T15:09:43.774088+0000","last_became_peered":"2024-01-16T15:09:43.774088+0000","last_unstale":"2024-01-16T15:09:43.778308+0000","last_undegraded":"2024-01-16T15:09:43.778308+0000","last_fullsized":"2024-01-16T15:09:43.778308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:12.844081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.790629+0000","last_change":"2024-01-16T15:09:42.744531+0000","last_active":"2024-01-16T15:09:43.790629+0000","last_peered":"2024-01-16T15:09:43.790629+0000","last_clean":"2024-01-16T15:09:43.790629+0000","last_became_active":"2024-01-16T15:09:42.744252+0000","last_became_peered":"2024-01-16T15:09:42.744252+0000","last_unstale":"2024-01-16T15:09:43.790629+0000","last_undegraded":"2024-01-16T15:09:43.790629+0000","last_fullsized":"2024-01-16T15:09:43.790629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:16.282677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772213+0000","last_change":"2024-01-16T15:09:43.770126+0000","last_active":"2024-01-16T15:09:43.772213+0000","last_peered":"2024-01-16T15:09:43.772213+0000","last_clean":"2024-01-16T15:09:43.772213+0000","last_became_active":"2024-01-16T15:09:43.769771+0000","last_became_peered":"2024-01-16T15:09:43.769771+0000","last_unstale":"2024-01-16T15:09:43.772213+0000","last_undegraded":"2024-01-16T15:09:43.772213+0000","last_fullsized":"2024-01-16T15:09:43.772213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:17.537253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779485+0000","last_change":"2024-01-16T15:09:42.742283+0000","last_active":"2024-01-16T15:09:43.779485+0000","last_peered":"2024-01-16T15:09:43.779485+0000","last_clean":"2024-01-16T15:09:43.779485+0000","last_became_active":"2024-01-16T15:09:42.741480+0000","last_became_peered":"2024-01-16T15:09:42.741480+0000","last_unstale":"2024-01-16T15:09:43.779485+0000","last_undegraded":"2024-01-16T15:09:43.779485+0000","last_fullsized":"2024-01-16T15:09:43.779485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:28.190177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.777986+0000","last_change":"2024-01-16T15:09:43.766483+0000","last_active":"2024-01-16T15:09:43.777986+0000","last_peered":"2024-01-16T15:09:43.777986+0000","last_clean":"2024-01-16T15:09:43.777986+0000","last_became_active":"2024-01-16T15:09:43.766173+0000","last_became_peered":"2024-01-16T15:09:43.766173+0000","last_unstale":"2024-01-16T15:09:43.777986+0000","last_undegraded":"2024-01-16T15:09:43.777986+0000","last_fullsized":"2024-01-16T15:09:43.777986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:17.537253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.743616+0000","last_change":"2024-01-16T15:09:42.742197+0000","last_active":"2024-01-16T15:09:43.743616+0000","last_peered":"2024-01-16T15:09:43.743616+0000","last_clean":"2024-01-16T15:09:43.743616+0000","last_became_active":"2024-01-16T15:09:42.741926+0000","last_became_peered":"2024-01-16T15:09:42.741926+0000","last_unstale":"2024-01-16T15:09:43.743616+0000","last_undegraded":"2024-01-16T15:09:43.743616+0000","last_fullsized":"2024-01-16T15:09:43.743616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:00.731046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778161+0000","last_change":"2024-01-16T15:09:43.770430+0000","last_active":"2024-01-16T15:09:43.778161+0000","last_peered":"2024-01-16T15:09:43.778161+0000","last_clean":"2024-01-16T15:09:43.778161+0000","last_became_active":"2024-01-16T15:09:43.770146+0000","last_became_peered":"2024-01-16T15:09:43.770146+0000","last_unstale":"2024-01-16T15:09:43.778161+0000","last_undegraded":"2024-01-16T15:09:43.778161+0000","last_fullsized":"2024-01-16T15:09:43.778161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:50.714447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767605+0000","last_change":"2024-01-16T15:09:42.749631+0000","last_active":"2024-01-16T15:09:43.767605+0000","last_peered":"2024-01-16T15:09:43.767605+0000","last_clean":"2024-01-16T15:09:43.767605+0000","last_became_active":"2024-01-16T15:09:42.749185+0000","last_became_peered":"2024-01-16T15:09:42.749185+0000","last_unstale":"2024-01-16T15:09:43.767605+0000","last_undegraded":"2024-01-16T15:09:43.767605+0000","last_fullsized":"2024-01-16T15:09:43.767605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:53.966397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778889+0000","last_change":"2024-01-16T15:09:43.773625+0000","last_active":"2024-01-16T15:09:43.778889+0000","last_peered":"2024-01-16T15:09:43.778889+0000","last_clean":"2024-01-16T15:09:43.778889+0000","last_became_active":"2024-01-16T15:09:43.773325+0000","last_became_peered":"2024-01-16T15:09:43.773325+0000","last_unstale":"2024-01-16T15:09:43.778889+0000","last_undegraded":"2024-01-16T15:09:43.778889+0000","last_fullsized":"2024-01-16T15:09:43.778889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:16.130957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767193+0000","last_change":"2024-01-16T15:09:42.748796+0000","last_active":"2024-01-16T15:09:43.767193+0000","last_peered":"2024-01-16T15:09:43.767193+0000","last_clean":"2024-01-16T15:09:43.767193+0000","last_became_active":"2024-01-16T15:09:42.748531+0000","last_became_peered":"2024-01-16T15:09:42.748531+0000","last_unstale":"2024-01-16T15:09:43.767193+0000","last_undegraded":"2024-01-16T15:09:43.767193+0000","last_fullsized":"2024-01-16T15:09:43.767193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:11.833480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.784371+0000","last_change":"2024-01-16T15:09:43.775073+0000","last_active":"2024-01-16T15:09:43.784371+0000","last_peered":"2024-01-16T15:09:43.784371+0000","last_clean":"2024-01-16T15:09:43.784371+0000","last_became_active":"2024-01-16T15:09:43.774712+0000","last_became_peered":"2024-01-16T15:09:43.774712+0000","last_unstale":"2024-01-16T15:09:43.784371+0000","last_undegraded":"2024-01-16T15:09:43.784371+0000","last_fullsized":"2024-01-16T15:09:43.784371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"85'3","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.794475+0000","last_change":"2024-01-16T15:09:42.749661+0000","last_active":"2024-01-16T15:09:43.794475+0000","last_peered":"2024-01-16T15:09:43.794475+0000","last_clean":"2024-01-16T15:09:43.794475+0000","last_became_active":"2024-01-16T15:09:42.749311+0000","last_became_peered":"2024-01-16T15:09:42.749311+0000","last_unstale":"2024-01-16T15:09:43.794475+0000","last_undegraded":"2024-01-16T15:09:43.794475+0000","last_fullsized":"2024-01-16T15:09:43.794475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:29.034433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772049+0000","last_change":"2024-01-16T15:09:43.768978+0000","last_active":"2024-01-16T15:09:43.772049+0000","last_peered":"2024-01-16T15:09:43.772049+0000","last_clean":"2024-01-16T15:09:43.772049+0000","last_became_active":"2024-01-16T15:09:43.767792+0000","last_became_peered":"2024-01-16T15:09:43.767792+0000","last_unstale":"2024-01-16T15:09:43.772049+0000","last_undegraded":"2024-01-16T15:09:43.772049+0000","last_fullsized":"2024-01-16T15:09:43.772049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.798215+0000","last_change":"2024-01-16T15:09:42.741203+0000","last_active":"2024-01-16T15:09:43.798215+0000","last_peered":"2024-01-16T15:09:43.798215+0000","last_clean":"2024-01-16T15:09:43.798215+0000","last_became_active":"2024-01-16T15:09:42.740893+0000","last_became_peered":"2024-01-16T15:09:42.740893+0000","last_unstale":"2024-01-16T15:09:43.798215+0000","last_undegraded":"2024-01-16T15:09:43.798215+0000","last_fullsized":"2024-01-16T15:09:43.798215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:24.802232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.780110+0000","last_change":"2024-01-16T15:09:43.770833+0000","last_active":"2024-01-16T15:09:43.780110+0000","last_peered":"2024-01-16T15:09:43.780110+0000","last_clean":"2024-01-16T15:09:43.780110+0000","last_became_active":"2024-01-16T15:09:43.768015+0000","last_became_peered":"2024-01-16T15:09:43.768015+0000","last_unstale":"2024-01-16T15:09:43.780110+0000","last_undegraded":"2024-01-16T15:09:43.780110+0000","last_fullsized":"2024-01-16T15:09:43.780110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:23.887665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793987+0000","last_change":"2024-01-16T15:09:42.733156+0000","last_active":"2024-01-16T15:09:43.793987+0000","last_peered":"2024-01-16T15:09:43.793987+0000","last_clean":"2024-01-16T15:09:43.793987+0000","last_became_active":"2024-01-16T15:09:42.732865+0000","last_became_peered":"2024-01-16T15:09:42.732865+0000","last_unstale":"2024-01-16T15:09:43.793987+0000","last_undegraded":"2024-01-16T15:09:43.793987+0000","last_fullsized":"2024-01-16T15:09:43.793987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:42.988000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772489+0000","last_change":"2024-01-16T15:09:43.770585+0000","last_active":"2024-01-16T15:09:43.772489+0000","last_peered":"2024-01-16T15:09:43.772489+0000","last_clean":"2024-01-16T15:09:43.772489+0000","last_became_active":"2024-01-16T15:09:43.770313+0000","last_became_peered":"2024-01-16T15:09:43.770313+0000","last_unstale":"2024-01-16T15:09:43.772489+0000","last_undegraded":"2024-01-16T15:09:43.772489+0000","last_fullsized":"2024-01-16T15:09:43.772489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:49.079185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.755089+0000","last_change":"2024-01-16T15:09:42.737907+0000","last_active":"2024-01-16T15:09:43.755089+0000","last_peered":"2024-01-16T15:09:43.755089+0000","last_clean":"2024-01-16T15:09:43.755089+0000","last_became_active":"2024-01-16T15:09:42.735956+0000","last_became_peered":"2024-01-16T15:09:42.735956+0000","last_unstale":"2024-01-16T15:09:43.755089+0000","last_undegraded":"2024-01-16T15:09:43.755089+0000","last_fullsized":"2024-01-16T15:09:43.755089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:51.370114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786970+0000","last_change":"2024-01-16T15:09:43.780144+0000","last_active":"2024-01-16T15:09:43.786970+0000","last_peered":"2024-01-16T15:09:43.786970+0000","last_clean":"2024-01-16T15:09:43.786970+0000","last_became_active":"2024-01-16T15:09:43.779866+0000","last_became_peered":"2024-01-16T15:09:43.779866+0000","last_unstale":"2024-01-16T15:09:43.786970+0000","last_undegraded":"2024-01-16T15:09:43.786970+0000","last_fullsized":"2024-01-16T15:09:43.786970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:04.561599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.757002+0000","last_change":"2024-01-16T15:09:42.746404+0000","last_active":"2024-01-16T15:09:43.757002+0000","last_peered":"2024-01-16T15:09:43.757002+0000","last_clean":"2024-01-16T15:09:43.757002+0000","last_became_active":"2024-01-16T15:09:42.745537+0000","last_became_peered":"2024-01-16T15:09:42.745537+0000","last_unstale":"2024-01-16T15:09:43.757002+0000","last_undegraded":"2024-01-16T15:09:43.757002+0000","last_fullsized":"2024-01-16T15:09:43.757002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.908221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779871+0000","last_change":"2024-01-16T15:09:43.770847+0000","last_active":"2024-01-16T15:09:43.779871+0000","last_peered":"2024-01-16T15:09:43.779871+0000","last_clean":"2024-01-16T15:09:43.779871+0000","last_became_active":"2024-01-16T15:09:43.770557+0000","last_became_peered":"2024-01-16T15:09:43.770557+0000","last_unstale":"2024-01-16T15:09:43.779871+0000","last_undegraded":"2024-01-16T15:09:43.779871+0000","last_fullsized":"2024-01-16T15:09:43.779871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:47.331839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.796649+0000","last_change":"2024-01-16T15:09:42.738582+0000","last_active":"2024-01-16T15:09:43.796649+0000","last_peered":"2024-01-16T15:09:43.796649+0000","last_clean":"2024-01-16T15:09:43.796649+0000","last_became_active":"2024-01-16T15:09:42.738241+0000","last_became_peered":"2024-01-16T15:09:42.738241+0000","last_unstale":"2024-01-16T15:09:43.796649+0000","last_undegraded":"2024-01-16T15:09:43.796649+0000","last_fullsized":"2024-01-16T15:09:43.796649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:15.120356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766619+0000","last_change":"2024-01-16T15:09:43.764653+0000","last_active":"2024-01-16T15:09:43.766619+0000","last_peered":"2024-01-16T15:09:43.766619+0000","last_clean":"2024-01-16T15:09:43.766619+0000","last_became_active":"2024-01-16T15:09:43.764128+0000","last_became_peered":"2024-01-16T15:09:43.764128+0000","last_unstale":"2024-01-16T15:09:43.766619+0000","last_undegraded":"2024-01-16T15:09:43.766619+0000","last_fullsized":"2024-01-16T15:09:43.766619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:50.714447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791232+0000","last_change":"2024-01-16T15:09:42.749449+0000","last_active":"2024-01-16T15:09:43.791232+0000","last_peered":"2024-01-16T15:09:43.791232+0000","last_clean":"2024-01-16T15:09:43.791232+0000","last_became_active":"2024-01-16T15:09:42.749106+0000","last_became_peered":"2024-01-16T15:09:42.749106+0000","last_unstale":"2024-01-16T15:09:43.791232+0000","last_undegraded":"2024-01-16T15:09:43.791232+0000","last_fullsized":"2024-01-16T15:09:43.791232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:45.690723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.775906+0000","last_change":"2024-01-16T15:09:43.773711+0000","last_active":"2024-01-16T15:09:43.775906+0000","last_peered":"2024-01-16T15:09:43.775906+0000","last_clean":"2024-01-16T15:09:43.775906+0000","last_became_active":"2024-01-16T15:09:43.773418+0000","last_became_peered":"2024-01-16T15:09:43.773418+0000","last_unstale":"2024-01-16T15:09:43.775906+0000","last_undegraded":"2024-01-16T15:09:43.775906+0000","last_fullsized":"2024-01-16T15:09:43.775906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:12.668028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.795503+0000","last_change":"2024-01-16T15:09:42.739030+0000","last_active":"2024-01-16T15:09:43.795503+0000","last_peered":"2024-01-16T15:09:43.795503+0000","last_clean":"2024-01-16T15:09:43.795503+0000","last_became_active":"2024-01-16T15:09:42.738742+0000","last_became_peered":"2024-01-16T15:09:42.738742+0000","last_unstale":"2024-01-16T15:09:43.795503+0000","last_undegraded":"2024-01-16T15:09:43.795503+0000","last_fullsized":"2024-01-16T15:09:43.795503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:38:39.018792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.771040+0000","last_change":"2024-01-16T15:09:43.764723+0000","last_active":"2024-01-16T15:09:43.771040+0000","last_peered":"2024-01-16T15:09:43.771040+0000","last_clean":"2024-01-16T15:09:43.771040+0000","last_became_active":"2024-01-16T15:09:43.764470+0000","last_became_peered":"2024-01-16T15:09:43.764470+0000","last_unstale":"2024-01-16T15:09:43.771040+0000","last_undegraded":"2024-01-16T15:09:43.771040+0000","last_fullsized":"2024-01-16T15:09:43.771040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778357+0000","last_change":"2024-01-16T15:09:42.742271+0000","last_active":"2024-01-16T15:09:43.778357+0000","last_peered":"2024-01-16T15:09:43.778357+0000","last_clean":"2024-01-16T15:09:43.778357+0000","last_became_active":"2024-01-16T15:09:42.741278+0000","last_became_peered":"2024-01-16T15:09:42.741278+0000","last_unstale":"2024-01-16T15:09:43.778357+0000","last_undegraded":"2024-01-16T15:09:43.778357+0000","last_fullsized":"2024-01-16T15:09:43.778357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:30.472214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.783271+0000","last_change":"2024-01-16T15:09:43.775824+0000","last_active":"2024-01-16T15:09:43.783271+0000","last_peered":"2024-01-16T15:09:43.783271+0000","last_clean":"2024-01-16T15:09:43.783271+0000","last_became_active":"2024-01-16T15:09:43.775451+0000","last_became_peered":"2024-01-16T15:09:43.775451+0000","last_unstale":"2024-01-16T15:09:43.783271+0000","last_undegraded":"2024-01-16T15:09:43.783271+0000","last_fullsized":"2024-01-16T15:09:43.783271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:04.561599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.794902+0000","last_change":"2024-01-16T15:09:42.741399+0000","last_active":"2024-01-16T15:09:43.794902+0000","last_peered":"2024-01-16T15:09:43.794902+0000","last_clean":"2024-01-16T15:09:43.794902+0000","last_became_active":"2024-01-16T15:09:42.741113+0000","last_became_peered":"2024-01-16T15:09:42.741113+0000","last_unstale":"2024-01-16T15:09:43.794902+0000","last_undegraded":"2024-01-16T15:09:43.794902+0000","last_fullsized":"2024-01-16T15:09:43.794902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:31.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773468+0000","last_change":"2024-01-16T15:09:43.770407+0000","last_active":"2024-01-16T15:09:43.773468+0000","last_peered":"2024-01-16T15:09:43.773468+0000","last_clean":"2024-01-16T15:09:43.773468+0000","last_became_active":"2024-01-16T15:09:43.770120+0000","last_became_peered":"2024-01-16T15:09:43.770120+0000","last_unstale":"2024-01-16T15:09:43.773468+0000","last_undegraded":"2024-01-16T15:09:43.773468+0000","last_fullsized":"2024-01-16T15:09:43.773468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:23.887665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772223+0000","last_change":"2024-01-16T15:09:42.731404+0000","last_active":"2024-01-16T15:09:43.772223+0000","last_peered":"2024-01-16T15:09:43.772223+0000","last_clean":"2024-01-16T15:09:43.772223+0000","last_became_active":"2024-01-16T15:09:42.730328+0000","last_became_peered":"2024-01-16T15:09:42.730328+0000","last_unstale":"2024-01-16T15:09:43.772223+0000","last_undegraded":"2024-01-16T15:09:43.772223+0000","last_fullsized":"2024-01-16T15:09:43.772223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:15.553232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778856+0000","last_change":"2024-01-16T15:09:43.773691+0000","last_active":"2024-01-16T15:09:43.778856+0000","last_peered":"2024-01-16T15:09:43.778856+0000","last_clean":"2024-01-16T15:09:43.778856+0000","last_became_active":"2024-01-16T15:09:43.773340+0000","last_became_peered":"2024-01-16T15:09:43.773340+0000","last_unstale":"2024-01-16T15:09:43.778856+0000","last_undegraded":"2024-01-16T15:09:43.778856+0000","last_fullsized":"2024-01-16T15:09:43.778856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:16.327868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772194+0000","last_change":"2024-01-16T15:09:42.750149+0000","last_active":"2024-01-16T15:09:43.772194+0000","last_peered":"2024-01-16T15:09:43.772194+0000","last_clean":"2024-01-16T15:09:43.772194+0000","last_became_active":"2024-01-16T15:09:42.749846+0000","last_became_peered":"2024-01-16T15:09:42.749846+0000","last_unstale":"2024-01-16T15:09:43.772194+0000","last_undegraded":"2024-01-16T15:09:43.772194+0000","last_fullsized":"2024-01-16T15:09:43.772194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:42.988000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782441+0000","last_change":"2024-01-16T15:09:43.781605+0000","last_active":"2024-01-16T15:09:43.782441+0000","last_peered":"2024-01-16T15:09:43.782441+0000","last_clean":"2024-01-16T15:09:43.782441+0000","last_became_active":"2024-01-16T15:09:43.781295+0000","last_became_peered":"2024-01-16T15:09:43.781295+0000","last_unstale":"2024-01-16T15:09:43.782441+0000","last_undegraded":"2024-01-16T15:09:43.782441+0000","last_fullsized":"2024-01-16T15:09:43.782441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:43.239501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.755407+0000","last_change":"2024-01-16T15:09:42.744253+0000","last_active":"2024-01-16T15:09:43.755407+0000","last_peered":"2024-01-16T15:09:43.755407+0000","last_clean":"2024-01-16T15:09:43.755407+0000","last_became_active":"2024-01-16T15:09:42.743273+0000","last_became_peered":"2024-01-16T15:09:42.743273+0000","last_unstale":"2024-01-16T15:09:43.755407+0000","last_undegraded":"2024-01-16T15:09:43.755407+0000","last_fullsized":"2024-01-16T15:09:43.755407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:44.385936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.770881+0000","last_change":"2024-01-16T15:09:43.767596+0000","last_active":"2024-01-16T15:09:43.770881+0000","last_peered":"2024-01-16T15:09:43.770881+0000","last_clean":"2024-01-16T15:09:43.770881+0000","last_became_active":"2024-01-16T15:09:43.767302+0000","last_became_peered":"2024-01-16T15:09:43.767302+0000","last_unstale":"2024-01-16T15:09:43.770881+0000","last_undegraded":"2024-01-16T15:09:43.770881+0000","last_fullsized":"2024-01-16T15:09:43.770881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:54.976998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791574+0000","last_change":"2024-01-16T15:09:42.740787+0000","last_active":"2024-01-16T15:09:43.791574+0000","last_peered":"2024-01-16T15:09:43.791574+0000","last_clean":"2024-01-16T15:09:43.791574+0000","last_became_active":"2024-01-16T15:09:42.740442+0000","last_became_peered":"2024-01-16T15:09:42.740442+0000","last_unstale":"2024-01-16T15:09:43.791574+0000","last_undegraded":"2024-01-16T15:09:43.791574+0000","last_fullsized":"2024-01-16T15:09:43.791574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:18.087700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778074+0000","last_change":"2024-01-16T15:09:43.774203+0000","last_active":"2024-01-16T15:09:43.778074+0000","last_peered":"2024-01-16T15:09:43.778074+0000","last_clean":"2024-01-16T15:09:43.778074+0000","last_became_active":"2024-01-16T15:09:43.773320+0000","last_became_peered":"2024-01-16T15:09:43.773320+0000","last_unstale":"2024-01-16T15:09:43.778074+0000","last_undegraded":"2024-01-16T15:09:43.778074+0000","last_fullsized":"2024-01-16T15:09:43.778074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:59.638845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.745258+0000","last_change":"2024-01-16T15:09:42.733464+0000","last_active":"2024-01-16T15:09:43.745258+0000","last_peered":"2024-01-16T15:09:43.745258+0000","last_clean":"2024-01-16T15:09:43.745258+0000","last_became_active":"2024-01-16T15:09:42.733147+0000","last_became_peered":"2024-01-16T15:09:42.733147+0000","last_unstale":"2024-01-16T15:09:43.745258+0000","last_undegraded":"2024-01-16T15:09:43.745258+0000","last_fullsized":"2024-01-16T15:09:43.745258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:24.294316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779677+0000","last_change":"2024-01-16T15:09:43.777690+0000","last_active":"2024-01-16T15:09:43.779677+0000","last_peered":"2024-01-16T15:09:43.779677+0000","last_clean":"2024-01-16T15:09:43.779677+0000","last_became_active":"2024-01-16T15:09:43.777376+0000","last_became_peered":"2024-01-16T15:09:43.777376+0000","last_unstale":"2024-01-16T15:09:43.779677+0000","last_undegraded":"2024-01-16T15:09:43.779677+0000","last_fullsized":"2024-01-16T15:09:43.779677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:29.980746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766935+0000","last_change":"2024-01-16T15:09:42.749866+0000","last_active":"2024-01-16T15:09:43.766935+0000","last_peered":"2024-01-16T15:09:43.766935+0000","last_clean":"2024-01-16T15:09:43.766935+0000","last_became_active":"2024-01-16T15:09:42.748923+0000","last_became_peered":"2024-01-16T15:09:42.748923+0000","last_unstale":"2024-01-16T15:09:43.766935+0000","last_undegraded":"2024-01-16T15:09:43.766935+0000","last_fullsized":"2024-01-16T15:09:43.766935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:52.674387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773002+0000","last_change":"2024-01-16T15:09:43.770934+0000","last_active":"2024-01-16T15:09:43.773002+0000","last_peered":"2024-01-16T15:09:43.773002+0000","last_clean":"2024-01-16T15:09:43.773002+0000","last_became_active":"2024-01-16T15:09:43.767425+0000","last_became_peered":"2024-01-16T15:09:43.767425+0000","last_unstale":"2024-01-16T15:09:43.773002+0000","last_undegraded":"2024-01-16T15:09:43.773002+0000","last_fullsized":"2024-01-16T15:09:43.773002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:46.701324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766703+0000","last_change":"2024-01-16T15:09:42.748671+0000","last_active":"2024-01-16T15:09:43.766703+0000","last_peered":"2024-01-16T15:09:43.766703+0000","last_clean":"2024-01-16T15:09:43.766703+0000","last_became_active":"2024-01-16T15:09:42.748369+0000","last_became_peered":"2024-01-16T15:09:42.748369+0000","last_unstale":"2024-01-16T15:09:43.766703+0000","last_undegraded":"2024-01-16T15:09:43.766703+0000","last_fullsized":"2024-01-16T15:09:43.766703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:18.944496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779288+0000","last_change":"2024-01-16T15:09:43.777930+0000","last_active":"2024-01-16T15:09:43.779288+0000","last_peered":"2024-01-16T15:09:43.779288+0000","last_clean":"2024-01-16T15:09:43.779288+0000","last_became_active":"2024-01-16T15:09:43.777611+0000","last_became_peered":"2024-01-16T15:09:43.777611+0000","last_unstale":"2024-01-16T15:09:43.779288+0000","last_undegraded":"2024-01-16T15:09:43.779288+0000","last_fullsized":"2024-01-16T15:09:43.779288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778454+0000","last_change":"2024-01-16T15:09:42.738154+0000","last_active":"2024-01-16T15:09:43.778454+0000","last_peered":"2024-01-16T15:09:43.778454+0000","last_clean":"2024-01-16T15:09:43.778454+0000","last_became_active":"2024-01-16T15:09:42.737866+0000","last_became_peered":"2024-01-16T15:09:42.737866+0000","last_unstale":"2024-01-16T15:09:43.778454+0000","last_undegraded":"2024-01-16T15:09:43.778454+0000","last_fullsized":"2024-01-16T15:09:43.778454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:24.802232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767606+0000","last_change":"2024-01-16T15:09:43.765099+0000","last_active":"2024-01-16T15:09:43.767606+0000","last_peered":"2024-01-16T15:09:43.767606+0000","last_clean":"2024-01-16T15:09:43.767606+0000","last_became_active":"2024-01-16T15:09:43.764820+0000","last_became_peered":"2024-01-16T15:09:43.764820+0000","last_unstale":"2024-01-16T15:09:43.767606+0000","last_undegraded":"2024-01-16T15:09:43.767606+0000","last_fullsized":"2024-01-16T15:09:43.767606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.744463+0000","last_change":"2024-01-16T15:09:42.742001+0000","last_active":"2024-01-16T15:09:43.744463+0000","last_peered":"2024-01-16T15:09:43.744463+0000","last_clean":"2024-01-16T15:09:43.744463+0000","last_became_active":"2024-01-16T15:09:42.741684+0000","last_became_peered":"2024-01-16T15:09:42.741684+0000","last_unstale":"2024-01-16T15:09:43.744463+0000","last_undegraded":"2024-01-16T15:09:43.744463+0000","last_fullsized":"2024-01-16T15:09:43.744463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:29.034433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772194+0000","last_change":"2024-01-16T15:09:43.769403+0000","last_active":"2024-01-16T15:09:43.772194+0000","last_peered":"2024-01-16T15:09:43.772194+0000","last_clean":"2024-01-16T15:09:43.772194+0000","last_became_active":"2024-01-16T15:09:43.768512+0000","last_became_peered":"2024-01-16T15:09:43.768512+0000","last_unstale":"2024-01-16T15:09:43.772194+0000","last_undegraded":"2024-01-16T15:09:43.772194+0000","last_fullsized":"2024-01-16T15:09:43.772194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:52.380714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766367+0000","last_change":"2024-01-16T15:09:42.743879+0000","last_active":"2024-01-16T15:09:43.766367+0000","last_peered":"2024-01-16T15:09:43.766367+0000","last_clean":"2024-01-16T15:09:43.766367+0000","last_became_active":"2024-01-16T15:09:42.743589+0000","last_became_peered":"2024-01-16T15:09:42.743589+0000","last_unstale":"2024-01-16T15:09:43.766367+0000","last_undegraded":"2024-01-16T15:09:43.766367+0000","last_fullsized":"2024-01-16T15:09:43.766367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:16.282677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774954+0000","last_change":"2024-01-16T15:09:43.771557+0000","last_active":"2024-01-16T15:09:43.774954+0000","last_peered":"2024-01-16T15:09:43.774954+0000","last_clean":"2024-01-16T15:09:43.774954+0000","last_became_active":"2024-01-16T15:09:43.771165+0000","last_became_peered":"2024-01-16T15:09:43.771165+0000","last_unstale":"2024-01-16T15:09:43.774954+0000","last_undegraded":"2024-01-16T15:09:43.774954+0000","last_fullsized":"2024-01-16T15:09:43.774954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:46.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.748930+0000","last_change":"2024-01-16T15:09:42.747369+0000","last_active":"2024-01-16T15:09:43.748930+0000","last_peered":"2024-01-16T15:09:43.748930+0000","last_clean":"2024-01-16T15:09:43.748930+0000","last_became_active":"2024-01-16T15:09:42.747090+0000","last_became_peered":"2024-01-16T15:09:42.747090+0000","last_unstale":"2024-01-16T15:09:43.748930+0000","last_undegraded":"2024-01-16T15:09:43.748930+0000","last_fullsized":"2024-01-16T15:09:43.748930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:42.228900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.771338+0000","last_change":"2024-01-16T15:09:43.767389+0000","last_active":"2024-01-16T15:09:43.771338+0000","last_peered":"2024-01-16T15:09:43.771338+0000","last_clean":"2024-01-16T15:09:43.771338+0000","last_became_active":"2024-01-16T15:09:43.767024+0000","last_became_peered":"2024-01-16T15:09:43.767024+0000","last_unstale":"2024-01-16T15:09:43.771338+0000","last_undegraded":"2024-01-16T15:09:43.771338+0000","last_fullsized":"2024-01-16T15:09:43.771338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:23.256626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793060+0000","last_change":"2024-01-16T15:09:42.736329+0000","last_active":"2024-01-16T15:09:43.793060+0000","last_peered":"2024-01-16T15:09:43.793060+0000","last_clean":"2024-01-16T15:09:43.793060+0000","last_became_active":"2024-01-16T15:09:42.736052+0000","last_became_peered":"2024-01-16T15:09:42.736052+0000","last_unstale":"2024-01-16T15:09:43.793060+0000","last_undegraded":"2024-01-16T15:09:43.793060+0000","last_fullsized":"2024-01-16T15:09:43.793060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:31.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":236,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-16T15:09:42.724894+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:09:42.724894+0000","last_peered":"2024-01-16T15:09:42.724894+0000","last_clean":"2024-01-16T15:09:42.724894+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:09:42.724894+0000","last_undegraded":"2024-01-16T15:09:42.724894+0000","last_fullsized":"2024-01-16T15:09:42.724894+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77124,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294716,"statfs":{"total":96636764160,"available":96557789184,"internally_reserved":0,"allocated":1626112,"data_stored":1029769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2729999999999999,"15min":1.3979999999999999},"min":{"1min":0.64600000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.5,"5min":8.5399999999999991,"15min":45.296999999999997},"last":1.411},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1479999999999999,"15min":1.3049999999999999},"min":{"1min":0.65600000000000003,"5min":0.55900000000000005,"15min":0.497},"max":{"1min":1.4650000000000001,"5min":2.8799999999999999,"15min":20.126999999999999},"last":1.5249999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.78300000000000003,"5min":0.56200000000000006,"15min":0.499},"max":{"1min":4.2919999999999998,"5min":7.4809999999999999,"15min":42.723999999999997},"last":1.27},{"interface":"front","average":{"1min":1.304,"5min":1.387,"15min":1.5660000000000001},"min":{"1min":0.752,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9430000000000001,"5min":5.1820000000000004,"15min":43.572000000000003},"last":1.367}]},{"osd":2,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.425,"15min":1.448},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.52100000000000002},"max":{"1min":1.649,"5min":14.858000000000001,"15min":35.308},"last":1.48},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.4139999999999999,"15min":1.472},"min":{"1min":0.76500000000000001,"5min":0.65800000000000003,"15min":0.503},"max":{"1min":2.6920000000000002,"5min":9.1899999999999995,"15min":23.594000000000001},"last":1.556}]},{"osd":3,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.603,"15min":1.601},"min":{"1min":0.65900000000000003,"5min":0.57999999999999996,"15min":0.51300000000000001},"max":{"1min":1.581,"5min":14.741,"15min":53.033999999999999},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.677,"15min":1.6060000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.60299999999999998},"max":{"1min":2.645,"5min":14.708,"15min":31.305},"last":1.819}]},{"osd":4,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.339},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.39600000000000002},"max":{"1min":2.028,"5min":2.2410000000000001,"15min":5.1870000000000003},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.4850000000000001,"15min":1.522},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":1.7110000000000001,"5min":14.948,"15min":36.374000000000002},"last":1.321}]},{"osd":5,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.488,"15min":1.4550000000000001},"min":{"1min":0.61799999999999999,"5min":0.55500000000000005,"15min":0.41499999999999998},"max":{"1min":2.593,"5min":9.1370000000000005,"15min":9.9399999999999995},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4319999999999999,"15min":1.474},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.46899999999999997},"max":{"1min":2.0350000000000001,"5min":2.9399999999999999,"15min":7.0170000000000003},"last":1.6120000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.671,"15min":1.5669999999999999},"min":{"1min":0.59899999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.2309999999999999,"5min":15.052,"15min":15.052},"last":1.651},{"interface":"front","average":{"1min":1.569,"5min":1.6539999999999999,"15min":1.625},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.49099999999999999},"max":{"1min":2.7360000000000002,"5min":8.4849999999999994,"15min":8.4849999999999994},"last":1.7589999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76544,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295296,"statfs":{"total":96636764160,"available":96558383104,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77330886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4319999999999999,"15min":1.6359999999999999},"min":{"1min":0.80900000000000005,"5min":0.65300000000000002,"15min":0.60099999999999998},"max":{"1min":1.657,"5min":9.1400000000000006,"15min":71.870999999999995},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.4370000000000001,"15min":1.744},"min":{"1min":0.82899999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.607,"5min":6.4059999999999997,"15min":71.846999999999994},"last":1.026}]},{"osd":1,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.603,"15min":1.3100000000000001},"min":{"1min":0.88400000000000001,"5min":0.70099999999999996,"15min":0.63900000000000001},"max":{"1min":1.7569999999999999,"5min":15.798,"15min":15.798},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.23,"5min":1.681,"15min":1.431},"min":{"1min":0.753,"5min":0.626,"15min":0.626},"max":{"1min":1.6679999999999999,"5min":24.556999999999999,"15min":24.556999999999999},"last":0.91300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.6919999999999999,"15min":1.4419999999999999},"min":{"1min":0.628,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.595,"5min":14.981,"15min":14.981},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.8240000000000001,"15min":1.5069999999999999},"min":{"1min":0.69799999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.619,"5min":14.175000000000001,"15min":14.175000000000001},"last":1.3109999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.4850000000000001,"15min":1.4910000000000001},"min":{"1min":0.78000000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":9.2170000000000005,"15min":14.827},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.7729999999999999,"15min":1.5820000000000001},"min":{"1min":0.81599999999999995,"5min":0.499,"15min":0.499},"max":{"1min":1.7629999999999999,"5min":21.890000000000001,"15min":21.890000000000001},"last":1.379}]},{"osd":4,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4710000000000001,"15min":1.4470000000000001},"min":{"1min":0.90200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.4980000000000002,"5min":10.901,"15min":10.901},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.327,"15min":1.444},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6910000000000001,"5min":4.9530000000000003,"15min":6.9000000000000004},"last":1.7629999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.4179999999999999,"15min":1.4199999999999999},"min":{"1min":0.52300000000000002,"5min":0.44600000000000001,"15min":0.38500000000000001},"max":{"1min":1.726,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.718},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5860000000000001,"15min":1.619},"min":{"1min":0.66600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":2.2400000000000002,"5min":9.2200000000000006,"15min":9.2200000000000006},"last":1.923}]},{"osd":7,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.536,"15min":1.631},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.927,"5min":6.4349999999999996,"15min":14.74},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.944,"15min":1.694},"min":{"1min":0.64600000000000002,"5min":0.54600000000000004,"15min":0.42599999999999999},"max":{"1min":1.788,"5min":14.494,"15min":14.494},"last":1.8919999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54432,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317408,"statfs":{"total":96636764160,"available":96581025792,"internally_reserved":0,"allocated":1060864,"data_stored":447810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.788,"15min":1.504},"min":{"1min":0.65600000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.5070000000000001,"5min":25.634,"15min":25.634},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.66600000000000004,"5min":0.59199999999999997,"15min":0.48299999999999998},"max":{"1min":4.2469999999999999,"5min":16.263000000000002,"15min":34.959000000000003},"last":0.82199999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.577,"15min":1.4219999999999999},"min":{"1min":0.71499999999999997,"5min":0.44,"15min":0.44},"max":{"1min":3.0139999999999998,"5min":26.378,"15min":26.378},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.3779999999999999,"15min":1.3220000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58299999999999996},"max":{"1min":4.3380000000000001,"5min":6.125,"15min":13.332000000000001},"last":1.073}]},{"osd":2,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.619,"15min":1.4139999999999999},"min":{"1min":0.68700000000000006,"5min":0.54700000000000004,"15min":0.42599999999999999},"max":{"1min":3.1179999999999999,"5min":25.495000000000001,"15min":25.495000000000001},"last":1.329},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.4870000000000001,"15min":1.4550000000000001},"min":{"1min":0.75900000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"max":{"1min":4.0229999999999997,"5min":6.7599999999999998,"15min":17.030000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.6919999999999999,"15min":1.6659999999999999},"min":{"1min":0.74399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0840000000000001,"5min":25.82,"15min":25.82},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.633,"5min":1.7450000000000001,"15min":1.6779999999999999},"min":{"1min":0.79800000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":4.1429999999999998,"5min":17.713999999999999,"15min":22.963999999999999},"last":1.169}]},{"osd":4,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2769999999999999,"15min":1.2889999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.378},"max":{"1min":6.1660000000000004,"5min":6.1660000000000004,"15min":8.6050000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.7549999999999999,"15min":1.5920000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.48299999999999998},"max":{"1min":1.7270000000000001,"5min":25.724,"15min":25.724},"last":1.554}]},{"osd":6,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":1.974,"15min":1.653},"min":{"1min":0.76700000000000002,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":7.6669999999999998,"5min":42.542999999999999,"15min":42.542999999999999},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.395,"15min":1.403},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.44600000000000001},"max":{"1min":6.093,"5min":6.093,"15min":10.015000000000001},"last":1.006}]},{"osd":7,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":2.1349999999999998,"15min":1.716},"min":{"1min":0.60399999999999998,"5min":0.47899999999999998,"15min":0.45700000000000002},"max":{"1min":1.712,"5min":26.484000000000002,"15min":26.484000000000002},"last":1.641},{"interface":"front","average":{"1min":1.587,"5min":1.6890000000000001,"15min":1.7010000000000001},"min":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.46999999999999997},"max":{"1min":5.0209999999999999,"5min":17.233000000000001,"15min":21.106000000000002},"last":1.385}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76544,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295296,"statfs":{"total":96636764160,"available":96558383104,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.415,"15min":1.333},"min":{"1min":0.65800000000000003,"5min":0.58199999999999996,"15min":0.44400000000000001},"max":{"1min":3.2919999999999998,"5min":7.1840000000000002,"15min":7.1840000000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4279999999999999,"15min":1.353},"min":{"1min":0.93700000000000006,"5min":0.60399999999999998,"15min":0.52800000000000002},"max":{"1min":3.1779999999999999,"5min":5.524,"15min":7.2190000000000003},"last":1.5549999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4550000000000001,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68799999999999994,"15min":0.57699999999999996},"max":{"1min":5.0019999999999998,"5min":8.9179999999999993,"15min":39.823999999999998},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.2450000000000001,"15min":1.3779999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.48399999999999999},"max":{"1min":1.734,"5min":6.9429999999999996,"15min":50.582000000000001},"last":1.0189999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.4350000000000001,"15min":1.341},"min":{"1min":0.5,"5min":0.5,"15min":0.39500000000000002},"max":{"1min":1.9219999999999999,"5min":10.984,"15min":10.984},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.454,"15min":1.393},"min":{"1min":0.90000000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.4780000000000002,"5min":6.8179999999999996,"15min":9.1509999999999998},"last":1.4830000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4059999999999999,"15min":1.359},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.54000000000000004},"max":{"1min":3.456,"5min":8.8979999999999997,"15min":11.971},"last":1.256},{"interface":"front","average":{"1min":1.268,"5min":1.5589999999999999,"15min":1.6479999999999999},"min":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.57899999999999996},"max":{"1min":1.6990000000000001,"5min":9.4329999999999998,"15min":27.626999999999999},"last":1.5740000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.597,"5min":4.2329999999999997,"15min":8.2599999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.6479999999999999,"15min":1.5369999999999999},"min":{"1min":0.72499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.7470000000000001,"5min":17.140999999999998,"15min":17.140999999999998},"last":1.129}]},{"osd":6,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.508,"15min":1.512},"min":{"1min":0.57099999999999995,"5min":0.48299999999999998,"15min":0.45300000000000001},"max":{"1min":1.7969999999999999,"5min":7.4509999999999996,"15min":7.4509999999999996},"last":1.3680000000000001},{"interface":"front","average":{"1min":2.073,"5min":1.865,"15min":1.641},"min":{"1min":1.26,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":12.324999999999999,"5min":25.643000000000001,"15min":25.643000000000001},"last":1.603}]},{"osd":7,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.7350000000000001,"15min":1.6060000000000001},"min":{"1min":0.94799999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.984,"5min":24.992999999999999,"15min":24.992999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.694,"15min":1.7809999999999999},"min":{"1min":0.78800000000000003,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.897,"5min":8.8369999999999997,"15min":19.001000000000001},"last":1.6299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.05,"15min":1.159},"min":{"1min":0.60699999999999998,"5min":0.374,"15min":0.313},"max":{"1min":1.1279999999999999,"5min":9.6229999999999993,"15min":38.423999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.224,"15min":1.1870000000000001},"min":{"1min":0.438,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.556,"5min":7.8300000000000001,"15min":20.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.4510000000000001,"15min":1.5960000000000001},"min":{"1min":0.57999999999999996,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.655,"5min":19.215,"15min":40.695999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.054,"5min":1.272,"15min":1.208},"min":{"1min":0.78300000000000003,"5min":0.46500000000000002,"15min":0.34100000000000003},"max":{"1min":1.3120000000000001,"5min":8.5449999999999999,"15min":9.9689999999999994},"last":0.93300000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.4079999999999999,"15min":1.526},"min":{"1min":0.755,"5min":0.436,"15min":0.436},"max":{"1min":1.599,"5min":10.391,"15min":59.009},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.6639999999999999,"15min":1.6379999999999999},"min":{"1min":0.72599999999999998,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":1.79,"5min":19.152000000000001,"15min":29.773},"last":1.153}]},{"osd":4,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.478,"15min":1.4870000000000001},"min":{"1min":0.85399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.671,"5min":7.4950000000000001,"15min":21.736000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.9570000000000001,"15min":1.9019999999999999},"min":{"1min":1.1040000000000001,"5min":0.622,"15min":0.622},"max":{"1min":1.7869999999999999,"5min":19.292000000000002,"15min":29.524000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.526,"15min":1.522},"min":{"1min":1.04,"5min":0.64500000000000002,"15min":0.54800000000000004},"max":{"1min":1.734,"5min":8.0739999999999998,"15min":9.8249999999999993},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.6899999999999999,"15min":1.5960000000000001},"min":{"1min":1.135,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8320000000000001,"5min":11.255000000000001,"15min":11.255000000000001},"last":1.0249999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.587,"15min":1.589},"min":{"1min":1.1359999999999999,"5min":0.81599999999999995,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":7.4560000000000004,"15min":20.459},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":2.0870000000000002,"15min":1.9990000000000001},"min":{"1min":1.264,"5min":0.67000000000000004,"15min":0.49399999999999999},"max":{"1min":1.9710000000000001,"5min":19.254999999999999,"15min":29.475000000000001},"last":1.4930000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.6140000000000001,"15min":1.637},"min":{"1min":1.244,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.9990000000000001,"5min":7.5339999999999998,"15min":20.109999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.611,"5min":2.1070000000000002,"15min":2.0099999999999998},"min":{"1min":1.169,"5min":0.65000000000000002,"15min":0.59999999999999998},"max":{"1min":2.1859999999999999,"5min":19.327999999999999,"15min":29.396999999999998},"last":1.5720000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1048576,"data_stored":446758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.1990000000000001,"15min":1.1659999999999999},"min":{"1min":0.55600000000000005,"5min":0.32500000000000001,"15min":0.28899999999999998},"max":{"1min":2.8919999999999999,"5min":24.155000000000001,"15min":25.335999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.3080000000000001,"15min":1.234},"min":{"1min":0.57699999999999996,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":2.952,"5min":25.199000000000002,"15min":25.199000000000002},"last":1.202}]},{"osd":1,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.2749999999999999,"15min":1.2549999999999999},"min":{"1min":0.64200000000000002,"5min":0.442,"15min":0.36199999999999999},"max":{"1min":1.6479999999999999,"5min":16.776,"15min":16.776},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.617,"15min":1.4770000000000001},"min":{"1min":0.78400000000000003,"5min":0.39100000000000001,"15min":0.318},"max":{"1min":3.2810000000000001,"5min":25.186,"15min":25.186},"last":0.89700000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5429999999999999,"15min":1.286},"min":{"1min":0.48499999999999999,"5min":0.41699999999999998,"15min":0.33900000000000002},"max":{"1min":1.712,"5min":37.470999999999997,"15min":37.470999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.167,"5min":1.2949999999999999,"15min":1.3799999999999999},"min":{"1min":0.629,"5min":0.54000000000000004,"15min":0.46500000000000002},"max":{"1min":1.506,"5min":6.5549999999999997,"15min":28.045999999999999},"last":1.131}]},{"osd":4,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.976,"15min":1.752},"min":{"1min":0.65200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":6.3630000000000004,"5min":25.062000000000001,"15min":25.062000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.8520000000000001,"15min":1.6479999999999999},"min":{"1min":0.92800000000000005,"5min":0.57599999999999996,"15min":0.502},"max":{"1min":1.869,"5min":37.307000000000002,"15min":37.307000000000002},"last":1.0169999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.5029999999999999,"15min":1.5740000000000001},"min":{"1min":0.72799999999999998,"5min":0.629,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":6.8849999999999998,"15min":19.327000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":2.0419999999999998,"15min":1.794},"min":{"1min":0.74299999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":3.5350000000000001,"5min":25.141999999999999,"15min":25.141999999999999},"last":1.351}]},{"osd":6,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.792,"15min":1.6379999999999999},"min":{"1min":1.1180000000000001,"5min":0.68400000000000005,"15min":0.56000000000000005},"max":{"1min":1.855,"5min":37.363,"15min":37.363},"last":1.097},{"interface":"front","average":{"1min":1.45,"5min":1.5920000000000001,"15min":1.5940000000000001},"min":{"1min":0.68899999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6230000000000002,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.377}]},{"osd":7,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":2.0790000000000002,"15min":1.869},"min":{"1min":0.93899999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.7530000000000001,"5min":25.106999999999999,"15min":25.106999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.899,"15min":1.7310000000000001},"min":{"1min":1.0920000000000001,"5min":0.72499999999999998,"15min":0.61499999999999999},"max":{"1min":1.9790000000000001,"5min":37.402999999999999,"15min":37.402999999999999},"last":1.2370000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64088,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307752,"statfs":{"total":96636764160,"available":96571138048,"internally_reserved":0,"allocated":1056768,"data_stored":446370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.431,"15min":1.194},"min":{"1min":0.36399999999999999,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":1.478,"5min":17.210999999999999,"15min":17.210999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.379,"15min":1.3100000000000001},"min":{"1min":0.65700000000000003,"5min":0.34300000000000003,"15min":0.26700000000000002},"max":{"1min":1.397,"5min":20.164000000000001,"15min":34.252000000000002},"last":4.3719999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.51,"15min":1.319},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.36099999999999999},"max":{"1min":3.6309999999999998,"5min":24.102,"15min":24.709},"last":1.476},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.605,"15min":1.452},"min":{"1min":0.78900000000000003,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":1.5309999999999999,"5min":21.795999999999999,"15min":30.128},"last":1.2929999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.9370000000000001,"15min":1.5349999999999999},"min":{"1min":0.63500000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.177,"5min":25.611000000000001,"15min":25.611000000000001},"last":3.3109999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.8300000000000001,"15min":1.4239999999999999},"min":{"1min":0.71499999999999997,"5min":0.41199999999999998,"15min":0.38900000000000001},"max":{"1min":1.6399999999999999,"5min":41.972000000000001,"15min":41.972000000000001},"last":4.6699999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.474,"15min":1.4419999999999999},"min":{"1min":0.751,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.3170000000000002,"5min":18.004999999999999,"15min":18.004999999999999},"last":3.2349999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":2.0030000000000001,"15min":1.655},"min":{"1min":1.004,"5min":0.67200000000000004,"15min":0.63700000000000001},"max":{"1min":7.3310000000000004,"5min":21.738,"15min":21.738},"last":5.8170000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4119999999999999,"5min":2.0569999999999999,"15min":1.7210000000000001},"min":{"1min":1.083,"5min":0.54900000000000004,"15min":0.46600000000000003},"max":{"1min":17.739999999999998,"5min":20.905999999999999,"15min":20.905999999999999},"last":4.1520000000000001},{"interface":"front","average":{"1min":2.258,"5min":1.9259999999999999,"15min":1.7410000000000001},"min":{"1min":0.79000000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":16.433,"5min":17.957999999999998,"15min":17.957999999999998},"last":5.258}]},{"osd":6,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":2.0489999999999999,"15min":1.7250000000000001},"min":{"1min":1.1830000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.881,"5min":21.832999999999998,"15min":21.832999999999998},"last":4.4089999999999998},{"interface":"front","average":{"1min":1.585,"5min":1.891,"15min":1.6220000000000001},"min":{"1min":1.2310000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.2669999999999999,"5min":20.509,"15min":20.509},"last":6.5350000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":2.0760000000000001,"15min":1.762},"min":{"1min":1.2270000000000001,"5min":0.75700000000000001,"15min":0.55600000000000005},"max":{"1min":2.379,"5min":20.669,"15min":20.669},"last":12.590999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.665,"15min":1.631},"min":{"1min":0.82399999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.794,"5min":18.039999999999999,"15min":18.039999999999999},"last":6.0229999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77132,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294708,"statfs":{"total":96636764160,"available":96557780992,"internally_reserved":0,"allocated":1638400,"data_stored":1035559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0980000000000001,"15min":1.1699999999999999},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.4119999999999999,"5min":10.683999999999999,"15min":26.751000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.996,"5min":1.355,"15min":1.181},"min":{"1min":0.434,"5min":0.434,"15min":0.35899999999999999},"max":{"1min":1.728,"5min":12.648999999999999,"15min":15.750999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.744,"15min":1.587},"min":{"1min":0.498,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":35.320999999999998,"15min":38.517000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.3680000000000001,"15min":1.288},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38800000000000001},"max":{"1min":1.841,"5min":11.720000000000001,"15min":24.402000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.8500000000000001,"15min":1.5},"min":{"1min":0.77200000000000002,"5min":0.45700000000000002,"15min":0.39100000000000001},"max":{"1min":3.4670000000000001,"5min":29.808,"15min":29.808},"last":0.621},{"interface":"front","average":{"1min":1.306,"5min":1.631,"15min":1.712},"min":{"1min":0.57999999999999996,"5min":0.51700000000000002,"15min":0.47799999999999998},"max":{"1min":2.0430000000000001,"5min":14.558,"15min":32.302},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.7170000000000001,"15min":1.5780000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":3.0899999999999999,"5min":9.6080000000000005,"15min":15.832000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.704,"15min":1.5429999999999999},"min":{"1min":0.997,"5min":0.61499999999999999,"15min":0.432},"max":{"1min":3.6829999999999998,"5min":14.685,"15min":19.062999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.579,"15min":1.7030000000000001},"min":{"1min":0.624,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.887,"5min":4.7949999999999999,"15min":26.800000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.7450000000000001,"15min":1.6140000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.0699999999999998,"5min":7.1550000000000002,"15min":15.906000000000001},"last":1.0469999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.75,"15min":1.6759999999999999},"min":{"1min":1.034,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":3.9790000000000001,"5min":14.772,"15min":20.844000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.502,"5min":1.5960000000000001,"15min":1.7050000000000001},"min":{"1min":1.123,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9910000000000001,"5min":4.4109999999999996,"15min":26.832000000000001},"last":0.86299999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.9470000000000001,"15min":1.704},"min":{"1min":0.69799999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.968,"5min":16.058,"15min":16.058},"last":0.754},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.9059999999999999,"15min":1.895},"min":{"1min":1.0629999999999999,"5min":0.75,"15min":0.47499999999999998},"max":{"1min":3.746,"5min":14.811,"15min":29.286999999999999},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:09:48.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:09:48.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.436+0000 7f06b5400640 1 Processor -- start 2024-01-16T15:09:48.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.436+0000 7f06b5400640 1 -- start start 2024-01-16T15:09:48.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06b5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f06b015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06b5400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f06b015e1e0 0x7f06b00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06b5400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b00b1ee0 con 0x7f06b0143ae0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06b5400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00b2780 con 0x7f06b015e1e0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06b5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b015e990 con 0x7f06b00cf9a0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06ae800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f06b0143ae0 0x7f06b00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39536/0 (socket says 172.21.15.125:39536) 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06ae800640 1 -- 172.21.15.125:0/3651655443 learned_addr learned my addr 172.21.15.125:0/3651655443 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06a5c00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f06b015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06af200640 1 --2- 172.21.15.125:0/3651655443 >> v2:172.21.15.137:3300/0 conn(0x7f06b015e1e0 0x7f06b00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611902904 0 0) 0x7f06b00b1ee0 con 0x7f06b0143ae0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.438+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0690003520 con 0x7f06b0143ae0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062141710 0 0) 0x7f0690003520 con 0x7f06b0143ae0 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 msgr2=0x7f06b015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f06b015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 >> v2:172.21.15.137:3300/0 conn(0x7f06b015e1e0 msgr2=0x7f06b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 --2- 172.21.15.125:0/3651655443 >> v2:172.21.15.137:3300/0 conn(0x7f06b015e1e0 0x7f06b00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00b2130 con 0x7f06b0143ae0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f06a0001670 con 0x7f06b0143ae0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 >> v1:172.21.15.125:6789/0 conn(0x7f06b0143ae0 legacy=0x7f06b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f0690003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690003ef0 0x7f06900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0690004d00 0x7f0690005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00b2780 con 0x7f0690004d00 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b015e990 con 0x7f0690003ef0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069000b980 con 0x7f06b00cf9a0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.439+0000 7f06af200640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0690004d00 0x7f0690005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06ae800640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f0690003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06a5c00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690003ef0 0x7f06900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06af200640 1 -- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 msgr2=0x7f0690003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06af200640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f0690003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06af200640 1 -- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690003ef0 msgr2=0x7f06900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06af200640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690003ef0 0x7f06900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06af200640 1 -- 172.21.15.125:0/3651655443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690003520 con 0x7f0690004d00 2024-01-16T15:09:48.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06ae800640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b00cf9a0 0x7f0690003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:48.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.440+0000 7f06a5c00640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690003ef0 0x7f06900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:48.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.441+0000 7f06af200640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0690004d00 0x7f0690005180 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009fc0 tx=0x7f06900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1590ad9f44ab267 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.441+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a802cc40 con 0x7f0690004d00 2024-01-16T15:09:48.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.441+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a800f040 con 0x7f0690004d00 2024-01-16T15:09:48.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.441+0000 7f06ade00640 1 -- 172.21.15.125:0/3651655443 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8034440 con 0x7f0690004d00 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.441+0000 7f06b5400640 1 -- 172.21.15.125:0/3651655443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0690004d00 msgr2=0x7f0690005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06b5400640 1 --2- 172.21.15.125:0/3651655443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0690004d00 0x7f0690005180 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009fc0 tx=0x7f06900057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06af200640 1 -- 172.21.15.125:0/3651655443 reap_dead start 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06b5400640 1 -- 172.21.15.125:0/3651655443 shutdown_connections 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06b5400640 1 -- 172.21.15.125:0/3651655443 >> 172.21.15.125:0/3651655443 conn(0x7f06b006a800 msgr2=0x7f06b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06b5400640 1 -- 172.21.15.125:0/3651655443 shutdown_connections 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.442+0000 7f06b5400640 1 -- 172.21.15.125:0/3651655443 wait complete. 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.443+0000 7f06b5400640 1 Processor -- start 2024-01-16T15:09:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 -- start start 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 0x7f06b0140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b013f430 0x7f06b0141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00ae4c0 con 0x7f06b00ae080 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b013fd20 con 0x7f0690004d00 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.444+0000 7f06b5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00ad2e0 con 0x7f06b013f430 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39390/0 (socket says 172.21.15.125:39390) 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 -- 172.21.15.125:0/518839028 learned_addr learned my addr 172.21.15.125:0/518839028 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06ae800640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 0x7f06b0140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 -- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b013f430 msgr2=0x7f06b0141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b013f430 0x7f06b0141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 -- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 msgr2=0x7f06b0140bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 0x7f06b0140bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 -- 172.21.15.125:0/518839028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00ad5c0 con 0x7f06b00ae080 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06ae800640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 0x7f06b0140bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.445+0000 7f06a5c00640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0698010830 tx=0x7f0698010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.446+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800eb60 con 0x7f06b00ae080 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.446+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800ecc0 con 0x7f06b00ae080 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.446+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069801c400 con 0x7f06b00ae080 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.446+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b00ad8a0 con 0x7f06b00ae080 2024-01-16T15:09:48.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.446+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b01597d0 con 0x7f06b00ae080 2024-01-16T15:09:48.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.447+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b00675d0 con 0x7f06b00ae080 2024-01-16T15:09:48.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.448+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f069801c560 con 0x7f06b00ae080 2024-01-16T15:09:48.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.449+0000 7f06a7400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 0x7f068806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.449+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0698088af0 con 0x7f06b00ae080 2024-01-16T15:09:48.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.449+0000 7f06ae800640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 0x7f068806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.450+0000 7f06ae800640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 0x7f068806e3e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f06a0001eb0 tx=0x7f06a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.454+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06980554c0 con 0x7f06b00ae080 2024-01-16T15:09:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.642+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06b015e640 con 0x7f068806bf20 2024-01-16T15:09:48.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.664+0000 7f06a7400640 1 -- 172.21.15.125:0/518839028 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286716 (secure 0 0 0) 0x7f06a000d040 con 0x7f068806bf20 2024-01-16T15:09:48.646 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:48.647 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 msgr2=0x7f068806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 0x7f068806e3e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f06a0001eb0 tx=0x7f06a0009040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 msgr2=0x7f06b013eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0698010830 tx=0x7f0698010cf0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 shutdown_connections 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06b013f430 0x7f06b0141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f068806bf20 0x7f068806e3e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00ae080 0x7f06b013eef0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 --2- 172.21.15.125:0/518839028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0690004d00 0x7f06b0140bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 >> 172.21.15.125:0/518839028 conn(0x7f06b006a800 msgr2=0x7f06b015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 shutdown_connections 2024-01-16T15:09:48.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.670+0000 7f06b5400640 1 -- 172.21.15.125:0/518839028 wait complete. 2024-01-16T15:09:48.666 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-01-16T15:09:48.226698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583124,"kb_used_data":9304,"kb_used_omap":12,"kb_used_meta":573683,"kb_avail":754391596,"statfs":{"total":773094113280,"available":772496994304,"internally_reserved":0,"allocated":9527296,"data_stored":4724469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":587451952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001529"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.775925+0000","last_change":"2024-01-16T15:09:43.774102+0000","last_active":"2024-01-16T15:09:43.775925+0000","last_peered":"2024-01-16T15:09:43.775925+0000","last_clean":"2024-01-16T15:09:43.775925+0000","last_became_active":"2024-01-16T15:09:43.773782+0000","last_became_peered":"2024-01-16T15:09:43.773782+0000","last_unstale":"2024-01-16T15:09:43.775925+0000","last_undegraded":"2024-01-16T15:09:43.775925+0000","last_fullsized":"2024-01-16T15:09:43.775925+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:56.423756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785645+0000","last_change":"2024-01-16T15:09:43.778037+0000","last_active":"2024-01-16T15:09:43.785645+0000","last_peered":"2024-01-16T15:09:43.785645+0000","last_clean":"2024-01-16T15:09:43.785645+0000","last_became_active":"2024-01-16T15:09:43.777636+0000","last_became_peered":"2024-01-16T15:09:43.777636+0000","last_unstale":"2024-01-16T15:09:43.785645+0000","last_undegraded":"2024-01-16T15:09:43.785645+0000","last_fullsized":"2024-01-16T15:09:43.785645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778701+0000","last_change":"2024-01-16T15:09:43.775325+0000","last_active":"2024-01-16T15:09:43.778701+0000","last_peered":"2024-01-16T15:09:43.778701+0000","last_clean":"2024-01-16T15:09:43.778701+0000","last_became_active":"2024-01-16T15:09:43.775026+0000","last_became_peered":"2024-01-16T15:09:43.775026+0000","last_unstale":"2024-01-16T15:09:43.778701+0000","last_undegraded":"2024-01-16T15:09:43.778701+0000","last_fullsized":"2024-01-16T15:09:43.778701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:02.629184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.776584+0000","last_change":"2024-01-16T15:09:43.770829+0000","last_active":"2024-01-16T15:09:43.776584+0000","last_peered":"2024-01-16T15:09:43.776584+0000","last_clean":"2024-01-16T15:09:43.776584+0000","last_became_active":"2024-01-16T15:09:43.770508+0000","last_became_peered":"2024-01-16T15:09:43.770508+0000","last_unstale":"2024-01-16T15:09:43.776584+0000","last_undegraded":"2024-01-16T15:09:43.776584+0000","last_fullsized":"2024-01-16T15:09:43.776584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:47.331839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.758989+0000","last_change":"2024-01-16T15:09:43.757572+0000","last_active":"2024-01-16T15:09:43.758989+0000","last_peered":"2024-01-16T15:09:43.758989+0000","last_clean":"2024-01-16T15:09:43.758989+0000","last_became_active":"2024-01-16T15:09:43.756203+0000","last_became_peered":"2024-01-16T15:09:43.756203+0000","last_unstale":"2024-01-16T15:09:43.758989+0000","last_undegraded":"2024-01-16T15:09:43.758989+0000","last_fullsized":"2024-01-16T15:09:43.758989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:12.643360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773695+0000","last_change":"2024-01-16T15:09:43.772253+0000","last_active":"2024-01-16T15:09:43.773695+0000","last_peered":"2024-01-16T15:09:43.773695+0000","last_clean":"2024-01-16T15:09:43.773695+0000","last_became_active":"2024-01-16T15:09:43.771947+0000","last_became_peered":"2024-01-16T15:09:43.771947+0000","last_unstale":"2024-01-16T15:09:43.773695+0000","last_undegraded":"2024-01-16T15:09:43.773695+0000","last_fullsized":"2024-01-16T15:09:43.773695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774039+0000","last_change":"2024-01-16T15:09:43.769924+0000","last_active":"2024-01-16T15:09:43.774039+0000","last_peered":"2024-01-16T15:09:43.774039+0000","last_clean":"2024-01-16T15:09:43.774039+0000","last_became_active":"2024-01-16T15:09:43.769568+0000","last_became_peered":"2024-01-16T15:09:43.769568+0000","last_unstale":"2024-01-16T15:09:43.774039+0000","last_undegraded":"2024-01-16T15:09:43.774039+0000","last_fullsized":"2024-01-16T15:09:43.774039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:09.677709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773508+0000","last_change":"2024-01-16T15:09:43.772085+0000","last_active":"2024-01-16T15:09:43.773508+0000","last_peered":"2024-01-16T15:09:43.773508+0000","last_clean":"2024-01-16T15:09:43.773508+0000","last_became_active":"2024-01-16T15:09:43.771343+0000","last_became_peered":"2024-01-16T15:09:43.771343+0000","last_unstale":"2024-01-16T15:09:43.773508+0000","last_undegraded":"2024-01-16T15:09:43.773508+0000","last_fullsized":"2024-01-16T15:09:43.773508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782448+0000","last_change":"2024-01-16T15:09:43.781845+0000","last_active":"2024-01-16T15:09:43.782448+0000","last_peered":"2024-01-16T15:09:43.782448+0000","last_clean":"2024-01-16T15:09:43.782448+0000","last_became_active":"2024-01-16T15:09:43.781257+0000","last_became_peered":"2024-01-16T15:09:43.781257+0000","last_unstale":"2024-01-16T15:09:43.782448+0000","last_undegraded":"2024-01-16T15:09:43.782448+0000","last_fullsized":"2024-01-16T15:09:43.782448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:31.482814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779985+0000","last_change":"2024-01-16T15:09:43.777950+0000","last_active":"2024-01-16T15:09:43.779985+0000","last_peered":"2024-01-16T15:09:43.779985+0000","last_clean":"2024-01-16T15:09:43.779985+0000","last_became_active":"2024-01-16T15:09:43.777679+0000","last_became_peered":"2024-01-16T15:09:43.777679+0000","last_unstale":"2024-01-16T15:09:43.779985+0000","last_undegraded":"2024-01-16T15:09:43.779985+0000","last_fullsized":"2024-01-16T15:09:43.779985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:11.348680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767038+0000","last_change":"2024-01-16T15:09:43.762396+0000","last_active":"2024-01-16T15:09:43.767038+0000","last_peered":"2024-01-16T15:09:43.767038+0000","last_clean":"2024-01-16T15:09:43.767038+0000","last_became_active":"2024-01-16T15:09:43.762095+0000","last_became_peered":"2024-01-16T15:09:43.762095+0000","last_unstale":"2024-01-16T15:09:43.767038+0000","last_undegraded":"2024-01-16T15:09:43.767038+0000","last_fullsized":"2024-01-16T15:09:43.767038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:02:31.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779957+0000","last_change":"2024-01-16T15:09:43.778763+0000","last_active":"2024-01-16T15:09:43.779957+0000","last_peered":"2024-01-16T15:09:43.779957+0000","last_clean":"2024-01-16T15:09:43.779957+0000","last_became_active":"2024-01-16T15:09:43.778347+0000","last_became_peered":"2024-01-16T15:09:43.778347+0000","last_unstale":"2024-01-16T15:09:43.779957+0000","last_undegraded":"2024-01-16T15:09:43.779957+0000","last_fullsized":"2024-01-16T15:09:43.779957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:37.388699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.762244+0000","last_change":"2024-01-16T15:09:43.760825+0000","last_active":"2024-01-16T15:09:43.762244+0000","last_peered":"2024-01-16T15:09:43.762244+0000","last_clean":"2024-01-16T15:09:43.762244+0000","last_became_active":"2024-01-16T15:09:43.760469+0000","last_became_peered":"2024-01-16T15:09:43.760469+0000","last_unstale":"2024-01-16T15:09:43.762244+0000","last_undegraded":"2024-01-16T15:09:43.762244+0000","last_fullsized":"2024-01-16T15:09:43.762244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:59.113702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785400+0000","last_change":"2024-01-16T15:09:43.779898+0000","last_active":"2024-01-16T15:09:43.785400+0000","last_peered":"2024-01-16T15:09:43.785400+0000","last_clean":"2024-01-16T15:09:43.785400+0000","last_became_active":"2024-01-16T15:09:43.779597+0000","last_became_peered":"2024-01-16T15:09:43.779597+0000","last_unstale":"2024-01-16T15:09:43.785400+0000","last_undegraded":"2024-01-16T15:09:43.785400+0000","last_fullsized":"2024-01-16T15:09:43.785400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:01.741647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778006+0000","last_change":"2024-01-16T15:09:43.769398+0000","last_active":"2024-01-16T15:09:43.778006+0000","last_peered":"2024-01-16T15:09:43.778006+0000","last_clean":"2024-01-16T15:09:43.778006+0000","last_became_active":"2024-01-16T15:09:43.766691+0000","last_became_peered":"2024-01-16T15:09:43.766691+0000","last_unstale":"2024-01-16T15:09:43.778006+0000","last_undegraded":"2024-01-16T15:09:43.778006+0000","last_fullsized":"2024-01-16T15:09:43.778006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:02.629184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779292+0000","last_change":"2024-01-16T15:09:43.776862+0000","last_active":"2024-01-16T15:09:43.779292+0000","last_peered":"2024-01-16T15:09:43.779292+0000","last_clean":"2024-01-16T15:09:43.779292+0000","last_became_active":"2024-01-16T15:09:43.774835+0000","last_became_peered":"2024-01-16T15:09:43.774835+0000","last_unstale":"2024-01-16T15:09:43.779292+0000","last_undegraded":"2024-01-16T15:09:43.779292+0000","last_fullsized":"2024-01-16T15:09:43.779292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:19.955097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778142+0000","last_change":"2024-01-16T15:09:43.773016+0000","last_active":"2024-01-16T15:09:43.778142+0000","last_peered":"2024-01-16T15:09:43.778142+0000","last_clean":"2024-01-16T15:09:43.778142+0000","last_became_active":"2024-01-16T15:09:43.772699+0000","last_became_peered":"2024-01-16T15:09:43.772699+0000","last_unstale":"2024-01-16T15:09:43.778142+0000","last_undegraded":"2024-01-16T15:09:43.778142+0000","last_fullsized":"2024-01-16T15:09:43.778142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.783014+0000","last_change":"2024-01-16T15:09:43.775801+0000","last_active":"2024-01-16T15:09:43.783014+0000","last_peered":"2024-01-16T15:09:43.783014+0000","last_clean":"2024-01-16T15:09:43.783014+0000","last_became_active":"2024-01-16T15:09:43.775396+0000","last_became_peered":"2024-01-16T15:09:43.775396+0000","last_unstale":"2024-01-16T15:09:43.783014+0000","last_undegraded":"2024-01-16T15:09:43.783014+0000","last_fullsized":"2024-01-16T15:09:43.783014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:25.304917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.784732+0000","last_change":"2024-01-16T15:09:43.782812+0000","last_active":"2024-01-16T15:09:43.784732+0000","last_peered":"2024-01-16T15:09:43.784732+0000","last_clean":"2024-01-16T15:09:43.784732+0000","last_became_active":"2024-01-16T15:09:43.782518+0000","last_became_peered":"2024-01-16T15:09:43.782518+0000","last_unstale":"2024-01-16T15:09:43.784732+0000","last_undegraded":"2024-01-16T15:09:43.784732+0000","last_fullsized":"2024-01-16T15:09:43.784732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778507+0000","last_change":"2024-01-16T15:09:43.773259+0000","last_active":"2024-01-16T15:09:43.778507+0000","last_peered":"2024-01-16T15:09:43.778507+0000","last_clean":"2024-01-16T15:09:43.778507+0000","last_became_active":"2024-01-16T15:09:43.772826+0000","last_became_peered":"2024-01-16T15:09:43.772826+0000","last_unstale":"2024-01-16T15:09:43.778507+0000","last_undegraded":"2024-01-16T15:09:43.778507+0000","last_fullsized":"2024-01-16T15:09:43.778507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:23.256626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.787001+0000","last_change":"2024-01-16T15:09:43.785355+0000","last_active":"2024-01-16T15:09:43.787001+0000","last_peered":"2024-01-16T15:09:43.787001+0000","last_clean":"2024-01-16T15:09:43.787001+0000","last_became_active":"2024-01-16T15:09:43.784834+0000","last_became_peered":"2024-01-16T15:09:43.784834+0000","last_unstale":"2024-01-16T15:09:43.787001+0000","last_undegraded":"2024-01-16T15:09:43.787001+0000","last_fullsized":"2024-01-16T15:09:43.787001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:03:35.595574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772101+0000","last_change":"2024-01-16T15:09:43.767460+0000","last_active":"2024-01-16T15:09:43.772101+0000","last_peered":"2024-01-16T15:09:43.772101+0000","last_clean":"2024-01-16T15:09:43.772101+0000","last_became_active":"2024-01-16T15:09:43.767122+0000","last_became_peered":"2024-01-16T15:09:43.767122+0000","last_unstale":"2024-01-16T15:09:43.772101+0000","last_undegraded":"2024-01-16T15:09:43.772101+0000","last_fullsized":"2024-01-16T15:09:43.772101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:16.327868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.780822+0000","last_change":"2024-01-16T15:09:43.779826+0000","last_active":"2024-01-16T15:09:43.780822+0000","last_peered":"2024-01-16T15:09:43.780822+0000","last_clean":"2024-01-16T15:09:43.780822+0000","last_became_active":"2024-01-16T15:09:43.779564+0000","last_became_peered":"2024-01-16T15:09:43.779564+0000","last_unstale":"2024-01-16T15:09:43.780822+0000","last_undegraded":"2024-01-16T15:09:43.780822+0000","last_fullsized":"2024-01-16T15:09:43.780822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:31.043085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.785980+0000","last_change":"2024-01-16T15:09:43.781216+0000","last_active":"2024-01-16T15:09:43.785980+0000","last_peered":"2024-01-16T15:09:43.785980+0000","last_clean":"2024-01-16T15:09:43.785980+0000","last_became_active":"2024-01-16T15:09:43.780920+0000","last_became_peered":"2024-01-16T15:09:43.780920+0000","last_unstale":"2024-01-16T15:09:43.785980+0000","last_undegraded":"2024-01-16T15:09:43.785980+0000","last_fullsized":"2024-01-16T15:09:43.785980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:19.098300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779399+0000","last_change":"2024-01-16T15:09:43.771225+0000","last_active":"2024-01-16T15:09:43.779399+0000","last_peered":"2024-01-16T15:09:43.779399+0000","last_clean":"2024-01-16T15:09:43.779399+0000","last_became_active":"2024-01-16T15:09:43.770963+0000","last_became_peered":"2024-01-16T15:09:43.770963+0000","last_unstale":"2024-01-16T15:09:43.779399+0000","last_undegraded":"2024-01-16T15:09:43.779399+0000","last_fullsized":"2024-01-16T15:09:43.779399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:04.751584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767934+0000","last_change":"2024-01-16T15:09:43.766897+0000","last_active":"2024-01-16T15:09:43.767934+0000","last_peered":"2024-01-16T15:09:43.767934+0000","last_clean":"2024-01-16T15:09:43.767934+0000","last_became_active":"2024-01-16T15:09:43.766365+0000","last_became_peered":"2024-01-16T15:09:43.766365+0000","last_unstale":"2024-01-16T15:09:43.767934+0000","last_undegraded":"2024-01-16T15:09:43.767934+0000","last_fullsized":"2024-01-16T15:09:43.767934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:22.072338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772984+0000","last_change":"2024-01-16T15:09:43.769605+0000","last_active":"2024-01-16T15:09:43.772984+0000","last_peered":"2024-01-16T15:09:43.772984+0000","last_clean":"2024-01-16T15:09:43.772984+0000","last_became_active":"2024-01-16T15:09:43.769346+0000","last_became_peered":"2024-01-16T15:09:43.769346+0000","last_unstale":"2024-01-16T15:09:43.772984+0000","last_undegraded":"2024-01-16T15:09:43.772984+0000","last_fullsized":"2024-01-16T15:09:43.772984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:16.918822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773263+0000","last_change":"2024-01-16T15:09:43.770542+0000","last_active":"2024-01-16T15:09:43.773263+0000","last_peered":"2024-01-16T15:09:43.773263+0000","last_clean":"2024-01-16T15:09:43.773263+0000","last_became_active":"2024-01-16T15:09:43.770081+0000","last_became_peered":"2024-01-16T15:09:43.770081+0000","last_unstale":"2024-01-16T15:09:43.773263+0000","last_undegraded":"2024-01-16T15:09:43.773263+0000","last_fullsized":"2024-01-16T15:09:43.773263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:17.550072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779903+0000","last_change":"2024-01-16T15:09:43.772406+0000","last_active":"2024-01-16T15:09:43.779903+0000","last_peered":"2024-01-16T15:09:43.779903+0000","last_clean":"2024-01-16T15:09:43.779903+0000","last_became_active":"2024-01-16T15:09:43.771755+0000","last_became_peered":"2024-01-16T15:09:43.771755+0000","last_unstale":"2024-01-16T15:09:43.779903+0000","last_undegraded":"2024-01-16T15:09:43.779903+0000","last_fullsized":"2024-01-16T15:09:43.779903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:03:35.595574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786904+0000","last_change":"2024-01-16T15:09:43.782157+0000","last_active":"2024-01-16T15:09:43.786904+0000","last_peered":"2024-01-16T15:09:43.786904+0000","last_clean":"2024-01-16T15:09:43.786904+0000","last_became_active":"2024-01-16T15:09:43.780596+0000","last_became_peered":"2024-01-16T15:09:43.780596+0000","last_unstale":"2024-01-16T15:09:43.786904+0000","last_undegraded":"2024-01-16T15:09:43.786904+0000","last_fullsized":"2024-01-16T15:09:43.786904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:16.563833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782882+0000","last_change":"2024-01-16T15:09:43.781969+0000","last_active":"2024-01-16T15:09:43.782882+0000","last_peered":"2024-01-16T15:09:43.782882+0000","last_clean":"2024-01-16T15:09:43.782882+0000","last_became_active":"2024-01-16T15:09:43.781636+0000","last_became_peered":"2024-01-16T15:09:43.781636+0000","last_unstale":"2024-01-16T15:09:43.782882+0000","last_undegraded":"2024-01-16T15:09:43.782882+0000","last_fullsized":"2024-01-16T15:09:43.782882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:53.684987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774810+0000","last_change":"2024-01-16T15:09:43.771219+0000","last_active":"2024-01-16T15:09:43.774810+0000","last_peered":"2024-01-16T15:09:43.774810+0000","last_clean":"2024-01-16T15:09:43.774810+0000","last_became_active":"2024-01-16T15:09:43.769133+0000","last_became_peered":"2024-01-16T15:09:43.769133+0000","last_unstale":"2024-01-16T15:09:43.774810+0000","last_undegraded":"2024-01-16T15:09:43.774810+0000","last_fullsized":"2024-01-16T15:09:43.774810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:02.316062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778695+0000","last_change":"2024-01-16T15:09:43.775716+0000","last_active":"2024-01-16T15:09:43.778695+0000","last_peered":"2024-01-16T15:09:43.778695+0000","last_clean":"2024-01-16T15:09:43.778695+0000","last_became_active":"2024-01-16T15:09:43.774133+0000","last_became_peered":"2024-01-16T15:09:43.774133+0000","last_unstale":"2024-01-16T15:09:43.778695+0000","last_undegraded":"2024-01-16T15:09:43.778695+0000","last_fullsized":"2024-01-16T15:09:43.778695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:34.688354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"85'4","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793573+0000","last_change":"2024-01-16T15:09:42.750178+0000","last_active":"2024-01-16T15:09:43.793573+0000","last_peered":"2024-01-16T15:09:43.793573+0000","last_clean":"2024-01-16T15:09:43.793573+0000","last_became_active":"2024-01-16T15:09:42.749455+0000","last_became_peered":"2024-01-16T15:09:42.749455+0000","last_unstale":"2024-01-16T15:09:43.793573+0000","last_undegraded":"2024-01-16T15:09:43.793573+0000","last_fullsized":"2024-01-16T15:09:43.793573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:11.632760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786244+0000","last_change":"2024-01-16T15:09:43.785576+0000","last_active":"2024-01-16T15:09:43.786244+0000","last_peered":"2024-01-16T15:09:43.786244+0000","last_clean":"2024-01-16T15:09:43.786244+0000","last_became_active":"2024-01-16T15:09:43.785207+0000","last_became_peered":"2024-01-16T15:09:43.785207+0000","last_unstale":"2024-01-16T15:09:43.786244+0000","last_undegraded":"2024-01-16T15:09:43.786244+0000","last_fullsized":"2024-01-16T15:09:43.786244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.405061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793014+0000","last_change":"2024-01-16T15:09:42.742043+0000","last_active":"2024-01-16T15:09:43.793014+0000","last_peered":"2024-01-16T15:09:43.793014+0000","last_clean":"2024-01-16T15:09:43.793014+0000","last_became_active":"2024-01-16T15:09:42.741705+0000","last_became_peered":"2024-01-16T15:09:42.741705+0000","last_unstale":"2024-01-16T15:09:43.793014+0000","last_undegraded":"2024-01-16T15:09:43.793014+0000","last_fullsized":"2024-01-16T15:09:43.793014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:15.120356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773244+0000","last_change":"2024-01-16T15:09:43.770366+0000","last_active":"2024-01-16T15:09:43.773244+0000","last_peered":"2024-01-16T15:09:43.773244+0000","last_clean":"2024-01-16T15:09:43.773244+0000","last_became_active":"2024-01-16T15:09:43.769726+0000","last_became_peered":"2024-01-16T15:09:43.769726+0000","last_unstale":"2024-01-16T15:09:43.773244+0000","last_undegraded":"2024-01-16T15:09:43.773244+0000","last_fullsized":"2024-01-16T15:09:43.773244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:45.696910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.797479+0000","last_change":"2024-01-16T15:09:42.736595+0000","last_active":"2024-01-16T15:09:43.797479+0000","last_peered":"2024-01-16T15:09:43.797479+0000","last_clean":"2024-01-16T15:09:43.797479+0000","last_became_active":"2024-01-16T15:09:42.736326+0000","last_became_peered":"2024-01-16T15:09:42.736326+0000","last_unstale":"2024-01-16T15:09:43.797479+0000","last_undegraded":"2024-01-16T15:09:43.797479+0000","last_fullsized":"2024-01-16T15:09:43.797479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:57:33.787581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778719+0000","last_change":"2024-01-16T15:09:43.775841+0000","last_active":"2024-01-16T15:09:43.778719+0000","last_peered":"2024-01-16T15:09:43.778719+0000","last_clean":"2024-01-16T15:09:43.778719+0000","last_became_active":"2024-01-16T15:09:43.775529+0000","last_became_peered":"2024-01-16T15:09:43.775529+0000","last_unstale":"2024-01-16T15:09:43.778719+0000","last_undegraded":"2024-01-16T15:09:43.778719+0000","last_fullsized":"2024-01-16T15:09:43.778719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:45.396536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.797209+0000","last_change":"2024-01-16T15:09:42.737705+0000","last_active":"2024-01-16T15:09:43.797209+0000","last_peered":"2024-01-16T15:09:43.797209+0000","last_clean":"2024-01-16T15:09:43.797209+0000","last_became_active":"2024-01-16T15:09:42.736886+0000","last_became_peered":"2024-01-16T15:09:42.736886+0000","last_unstale":"2024-01-16T15:09:43.797209+0000","last_undegraded":"2024-01-16T15:09:43.797209+0000","last_fullsized":"2024-01-16T15:09:43.797209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.908221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786940+0000","last_change":"2024-01-16T15:09:43.783088+0000","last_active":"2024-01-16T15:09:43.786940+0000","last_peered":"2024-01-16T15:09:43.786940+0000","last_clean":"2024-01-16T15:09:43.786940+0000","last_became_active":"2024-01-16T15:09:43.782811+0000","last_became_peered":"2024-01-16T15:09:43.782811+0000","last_unstale":"2024-01-16T15:09:43.786940+0000","last_undegraded":"2024-01-16T15:09:43.786940+0000","last_fullsized":"2024-01-16T15:09:43.786940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.953127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791626+0000","last_change":"2024-01-16T15:09:42.741198+0000","last_active":"2024-01-16T15:09:43.791626+0000","last_peered":"2024-01-16T15:09:43.791626+0000","last_clean":"2024-01-16T15:09:43.791626+0000","last_became_active":"2024-01-16T15:09:42.740824+0000","last_became_peered":"2024-01-16T15:09:42.740824+0000","last_unstale":"2024-01-16T15:09:43.791626+0000","last_undegraded":"2024-01-16T15:09:43.791626+0000","last_fullsized":"2024-01-16T15:09:43.791626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:22.877064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778308+0000","last_change":"2024-01-16T15:09:43.775602+0000","last_active":"2024-01-16T15:09:43.778308+0000","last_peered":"2024-01-16T15:09:43.778308+0000","last_clean":"2024-01-16T15:09:43.778308+0000","last_became_active":"2024-01-16T15:09:43.774088+0000","last_became_peered":"2024-01-16T15:09:43.774088+0000","last_unstale":"2024-01-16T15:09:43.778308+0000","last_undegraded":"2024-01-16T15:09:43.778308+0000","last_fullsized":"2024-01-16T15:09:43.778308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:12.844081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.790629+0000","last_change":"2024-01-16T15:09:42.744531+0000","last_active":"2024-01-16T15:09:43.790629+0000","last_peered":"2024-01-16T15:09:43.790629+0000","last_clean":"2024-01-16T15:09:43.790629+0000","last_became_active":"2024-01-16T15:09:42.744252+0000","last_became_peered":"2024-01-16T15:09:42.744252+0000","last_unstale":"2024-01-16T15:09:43.790629+0000","last_undegraded":"2024-01-16T15:09:43.790629+0000","last_fullsized":"2024-01-16T15:09:43.790629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:16.282677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772213+0000","last_change":"2024-01-16T15:09:43.770126+0000","last_active":"2024-01-16T15:09:43.772213+0000","last_peered":"2024-01-16T15:09:43.772213+0000","last_clean":"2024-01-16T15:09:43.772213+0000","last_became_active":"2024-01-16T15:09:43.769771+0000","last_became_peered":"2024-01-16T15:09:43.769771+0000","last_unstale":"2024-01-16T15:09:43.772213+0000","last_undegraded":"2024-01-16T15:09:43.772213+0000","last_fullsized":"2024-01-16T15:09:43.772213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:17.537253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779485+0000","last_change":"2024-01-16T15:09:42.742283+0000","last_active":"2024-01-16T15:09:43.779485+0000","last_peered":"2024-01-16T15:09:43.779485+0000","last_clean":"2024-01-16T15:09:43.779485+0000","last_became_active":"2024-01-16T15:09:42.741480+0000","last_became_peered":"2024-01-16T15:09:42.741480+0000","last_unstale":"2024-01-16T15:09:43.779485+0000","last_undegraded":"2024-01-16T15:09:43.779485+0000","last_fullsized":"2024-01-16T15:09:43.779485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:28.190177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.777986+0000","last_change":"2024-01-16T15:09:43.766483+0000","last_active":"2024-01-16T15:09:43.777986+0000","last_peered":"2024-01-16T15:09:43.777986+0000","last_clean":"2024-01-16T15:09:43.777986+0000","last_became_active":"2024-01-16T15:09:43.766173+0000","last_became_peered":"2024-01-16T15:09:43.766173+0000","last_unstale":"2024-01-16T15:09:43.777986+0000","last_undegraded":"2024-01-16T15:09:43.777986+0000","last_fullsized":"2024-01-16T15:09:43.777986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:17.537253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.743616+0000","last_change":"2024-01-16T15:09:42.742197+0000","last_active":"2024-01-16T15:09:43.743616+0000","last_peered":"2024-01-16T15:09:43.743616+0000","last_clean":"2024-01-16T15:09:43.743616+0000","last_became_active":"2024-01-16T15:09:42.741926+0000","last_became_peered":"2024-01-16T15:09:42.741926+0000","last_unstale":"2024-01-16T15:09:43.743616+0000","last_undegraded":"2024-01-16T15:09:43.743616+0000","last_fullsized":"2024-01-16T15:09:43.743616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:00.731046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778161+0000","last_change":"2024-01-16T15:09:43.770430+0000","last_active":"2024-01-16T15:09:43.778161+0000","last_peered":"2024-01-16T15:09:43.778161+0000","last_clean":"2024-01-16T15:09:43.778161+0000","last_became_active":"2024-01-16T15:09:43.770146+0000","last_became_peered":"2024-01-16T15:09:43.770146+0000","last_unstale":"2024-01-16T15:09:43.778161+0000","last_undegraded":"2024-01-16T15:09:43.778161+0000","last_fullsized":"2024-01-16T15:09:43.778161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:50.714447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767605+0000","last_change":"2024-01-16T15:09:42.749631+0000","last_active":"2024-01-16T15:09:43.767605+0000","last_peered":"2024-01-16T15:09:43.767605+0000","last_clean":"2024-01-16T15:09:43.767605+0000","last_became_active":"2024-01-16T15:09:42.749185+0000","last_became_peered":"2024-01-16T15:09:42.749185+0000","last_unstale":"2024-01-16T15:09:43.767605+0000","last_undegraded":"2024-01-16T15:09:43.767605+0000","last_fullsized":"2024-01-16T15:09:43.767605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:53.966397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778889+0000","last_change":"2024-01-16T15:09:43.773625+0000","last_active":"2024-01-16T15:09:43.778889+0000","last_peered":"2024-01-16T15:09:43.778889+0000","last_clean":"2024-01-16T15:09:43.778889+0000","last_became_active":"2024-01-16T15:09:43.773325+0000","last_became_peered":"2024-01-16T15:09:43.773325+0000","last_unstale":"2024-01-16T15:09:43.778889+0000","last_undegraded":"2024-01-16T15:09:43.778889+0000","last_fullsized":"2024-01-16T15:09:43.778889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:16.130957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767193+0000","last_change":"2024-01-16T15:09:42.748796+0000","last_active":"2024-01-16T15:09:43.767193+0000","last_peered":"2024-01-16T15:09:43.767193+0000","last_clean":"2024-01-16T15:09:43.767193+0000","last_became_active":"2024-01-16T15:09:42.748531+0000","last_became_peered":"2024-01-16T15:09:42.748531+0000","last_unstale":"2024-01-16T15:09:43.767193+0000","last_undegraded":"2024-01-16T15:09:43.767193+0000","last_fullsized":"2024-01-16T15:09:43.767193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:11.833480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.784371+0000","last_change":"2024-01-16T15:09:43.775073+0000","last_active":"2024-01-16T15:09:43.784371+0000","last_peered":"2024-01-16T15:09:43.784371+0000","last_clean":"2024-01-16T15:09:43.784371+0000","last_became_active":"2024-01-16T15:09:43.774712+0000","last_became_peered":"2024-01-16T15:09:43.774712+0000","last_unstale":"2024-01-16T15:09:43.784371+0000","last_undegraded":"2024-01-16T15:09:43.784371+0000","last_fullsized":"2024-01-16T15:09:43.784371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"85'3","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.794475+0000","last_change":"2024-01-16T15:09:42.749661+0000","last_active":"2024-01-16T15:09:43.794475+0000","last_peered":"2024-01-16T15:09:43.794475+0000","last_clean":"2024-01-16T15:09:43.794475+0000","last_became_active":"2024-01-16T15:09:42.749311+0000","last_became_peered":"2024-01-16T15:09:42.749311+0000","last_unstale":"2024-01-16T15:09:43.794475+0000","last_undegraded":"2024-01-16T15:09:43.794475+0000","last_fullsized":"2024-01-16T15:09:43.794475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:29.034433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772049+0000","last_change":"2024-01-16T15:09:43.768978+0000","last_active":"2024-01-16T15:09:43.772049+0000","last_peered":"2024-01-16T15:09:43.772049+0000","last_clean":"2024-01-16T15:09:43.772049+0000","last_became_active":"2024-01-16T15:09:43.767792+0000","last_became_peered":"2024-01-16T15:09:43.767792+0000","last_unstale":"2024-01-16T15:09:43.772049+0000","last_undegraded":"2024-01-16T15:09:43.772049+0000","last_fullsized":"2024-01-16T15:09:43.772049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.798215+0000","last_change":"2024-01-16T15:09:42.741203+0000","last_active":"2024-01-16T15:09:43.798215+0000","last_peered":"2024-01-16T15:09:43.798215+0000","last_clean":"2024-01-16T15:09:43.798215+0000","last_became_active":"2024-01-16T15:09:42.740893+0000","last_became_peered":"2024-01-16T15:09:42.740893+0000","last_unstale":"2024-01-16T15:09:43.798215+0000","last_undegraded":"2024-01-16T15:09:43.798215+0000","last_fullsized":"2024-01-16T15:09:43.798215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:24.802232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.780110+0000","last_change":"2024-01-16T15:09:43.770833+0000","last_active":"2024-01-16T15:09:43.780110+0000","last_peered":"2024-01-16T15:09:43.780110+0000","last_clean":"2024-01-16T15:09:43.780110+0000","last_became_active":"2024-01-16T15:09:43.768015+0000","last_became_peered":"2024-01-16T15:09:43.768015+0000","last_unstale":"2024-01-16T15:09:43.780110+0000","last_undegraded":"2024-01-16T15:09:43.780110+0000","last_fullsized":"2024-01-16T15:09:43.780110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:23.887665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793987+0000","last_change":"2024-01-16T15:09:42.733156+0000","last_active":"2024-01-16T15:09:43.793987+0000","last_peered":"2024-01-16T15:09:43.793987+0000","last_clean":"2024-01-16T15:09:43.793987+0000","last_became_active":"2024-01-16T15:09:42.732865+0000","last_became_peered":"2024-01-16T15:09:42.732865+0000","last_unstale":"2024-01-16T15:09:43.793987+0000","last_undegraded":"2024-01-16T15:09:43.793987+0000","last_fullsized":"2024-01-16T15:09:43.793987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:42.988000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772489+0000","last_change":"2024-01-16T15:09:43.770585+0000","last_active":"2024-01-16T15:09:43.772489+0000","last_peered":"2024-01-16T15:09:43.772489+0000","last_clean":"2024-01-16T15:09:43.772489+0000","last_became_active":"2024-01-16T15:09:43.770313+0000","last_became_peered":"2024-01-16T15:09:43.770313+0000","last_unstale":"2024-01-16T15:09:43.772489+0000","last_undegraded":"2024-01-16T15:09:43.772489+0000","last_fullsized":"2024-01-16T15:09:43.772489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:49.079185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.755089+0000","last_change":"2024-01-16T15:09:42.737907+0000","last_active":"2024-01-16T15:09:43.755089+0000","last_peered":"2024-01-16T15:09:43.755089+0000","last_clean":"2024-01-16T15:09:43.755089+0000","last_became_active":"2024-01-16T15:09:42.735956+0000","last_became_peered":"2024-01-16T15:09:42.735956+0000","last_unstale":"2024-01-16T15:09:43.755089+0000","last_undegraded":"2024-01-16T15:09:43.755089+0000","last_fullsized":"2024-01-16T15:09:43.755089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:51.370114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.786970+0000","last_change":"2024-01-16T15:09:43.780144+0000","last_active":"2024-01-16T15:09:43.786970+0000","last_peered":"2024-01-16T15:09:43.786970+0000","last_clean":"2024-01-16T15:09:43.786970+0000","last_became_active":"2024-01-16T15:09:43.779866+0000","last_became_peered":"2024-01-16T15:09:43.779866+0000","last_unstale":"2024-01-16T15:09:43.786970+0000","last_undegraded":"2024-01-16T15:09:43.786970+0000","last_fullsized":"2024-01-16T15:09:43.786970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:04.561599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.757002+0000","last_change":"2024-01-16T15:09:42.746404+0000","last_active":"2024-01-16T15:09:43.757002+0000","last_peered":"2024-01-16T15:09:43.757002+0000","last_clean":"2024-01-16T15:09:43.757002+0000","last_became_active":"2024-01-16T15:09:42.745537+0000","last_became_peered":"2024-01-16T15:09:42.745537+0000","last_unstale":"2024-01-16T15:09:43.757002+0000","last_undegraded":"2024-01-16T15:09:43.757002+0000","last_fullsized":"2024-01-16T15:09:43.757002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.908221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779871+0000","last_change":"2024-01-16T15:09:43.770847+0000","last_active":"2024-01-16T15:09:43.779871+0000","last_peered":"2024-01-16T15:09:43.779871+0000","last_clean":"2024-01-16T15:09:43.779871+0000","last_became_active":"2024-01-16T15:09:43.770557+0000","last_became_peered":"2024-01-16T15:09:43.770557+0000","last_unstale":"2024-01-16T15:09:43.779871+0000","last_undegraded":"2024-01-16T15:09:43.779871+0000","last_fullsized":"2024-01-16T15:09:43.779871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:47.331839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.796649+0000","last_change":"2024-01-16T15:09:42.738582+0000","last_active":"2024-01-16T15:09:43.796649+0000","last_peered":"2024-01-16T15:09:43.796649+0000","last_clean":"2024-01-16T15:09:43.796649+0000","last_became_active":"2024-01-16T15:09:42.738241+0000","last_became_peered":"2024-01-16T15:09:42.738241+0000","last_unstale":"2024-01-16T15:09:43.796649+0000","last_undegraded":"2024-01-16T15:09:43.796649+0000","last_fullsized":"2024-01-16T15:09:43.796649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:15.120356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766619+0000","last_change":"2024-01-16T15:09:43.764653+0000","last_active":"2024-01-16T15:09:43.766619+0000","last_peered":"2024-01-16T15:09:43.766619+0000","last_clean":"2024-01-16T15:09:43.766619+0000","last_became_active":"2024-01-16T15:09:43.764128+0000","last_became_peered":"2024-01-16T15:09:43.764128+0000","last_unstale":"2024-01-16T15:09:43.766619+0000","last_undegraded":"2024-01-16T15:09:43.766619+0000","last_fullsized":"2024-01-16T15:09:43.766619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:50.714447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791232+0000","last_change":"2024-01-16T15:09:42.749449+0000","last_active":"2024-01-16T15:09:43.791232+0000","last_peered":"2024-01-16T15:09:43.791232+0000","last_clean":"2024-01-16T15:09:43.791232+0000","last_became_active":"2024-01-16T15:09:42.749106+0000","last_became_peered":"2024-01-16T15:09:42.749106+0000","last_unstale":"2024-01-16T15:09:43.791232+0000","last_undegraded":"2024-01-16T15:09:43.791232+0000","last_fullsized":"2024-01-16T15:09:43.791232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:45.690723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.775906+0000","last_change":"2024-01-16T15:09:43.773711+0000","last_active":"2024-01-16T15:09:43.775906+0000","last_peered":"2024-01-16T15:09:43.775906+0000","last_clean":"2024-01-16T15:09:43.775906+0000","last_became_active":"2024-01-16T15:09:43.773418+0000","last_became_peered":"2024-01-16T15:09:43.773418+0000","last_unstale":"2024-01-16T15:09:43.775906+0000","last_undegraded":"2024-01-16T15:09:43.775906+0000","last_fullsized":"2024-01-16T15:09:43.775906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:12.668028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.795503+0000","last_change":"2024-01-16T15:09:42.739030+0000","last_active":"2024-01-16T15:09:43.795503+0000","last_peered":"2024-01-16T15:09:43.795503+0000","last_clean":"2024-01-16T15:09:43.795503+0000","last_became_active":"2024-01-16T15:09:42.738742+0000","last_became_peered":"2024-01-16T15:09:42.738742+0000","last_unstale":"2024-01-16T15:09:43.795503+0000","last_undegraded":"2024-01-16T15:09:43.795503+0000","last_fullsized":"2024-01-16T15:09:43.795503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:38:39.018792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.771040+0000","last_change":"2024-01-16T15:09:43.764723+0000","last_active":"2024-01-16T15:09:43.771040+0000","last_peered":"2024-01-16T15:09:43.771040+0000","last_clean":"2024-01-16T15:09:43.771040+0000","last_became_active":"2024-01-16T15:09:43.764470+0000","last_became_peered":"2024-01-16T15:09:43.764470+0000","last_unstale":"2024-01-16T15:09:43.771040+0000","last_undegraded":"2024-01-16T15:09:43.771040+0000","last_fullsized":"2024-01-16T15:09:43.771040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:21.464404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778357+0000","last_change":"2024-01-16T15:09:42.742271+0000","last_active":"2024-01-16T15:09:43.778357+0000","last_peered":"2024-01-16T15:09:43.778357+0000","last_clean":"2024-01-16T15:09:43.778357+0000","last_became_active":"2024-01-16T15:09:42.741278+0000","last_became_peered":"2024-01-16T15:09:42.741278+0000","last_unstale":"2024-01-16T15:09:43.778357+0000","last_undegraded":"2024-01-16T15:09:43.778357+0000","last_fullsized":"2024-01-16T15:09:43.778357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:30.472214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.783271+0000","last_change":"2024-01-16T15:09:43.775824+0000","last_active":"2024-01-16T15:09:43.783271+0000","last_peered":"2024-01-16T15:09:43.783271+0000","last_clean":"2024-01-16T15:09:43.783271+0000","last_became_active":"2024-01-16T15:09:43.775451+0000","last_became_peered":"2024-01-16T15:09:43.775451+0000","last_unstale":"2024-01-16T15:09:43.783271+0000","last_undegraded":"2024-01-16T15:09:43.783271+0000","last_fullsized":"2024-01-16T15:09:43.783271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:04.561599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.794902+0000","last_change":"2024-01-16T15:09:42.741399+0000","last_active":"2024-01-16T15:09:43.794902+0000","last_peered":"2024-01-16T15:09:43.794902+0000","last_clean":"2024-01-16T15:09:43.794902+0000","last_became_active":"2024-01-16T15:09:42.741113+0000","last_became_peered":"2024-01-16T15:09:42.741113+0000","last_unstale":"2024-01-16T15:09:43.794902+0000","last_undegraded":"2024-01-16T15:09:43.794902+0000","last_fullsized":"2024-01-16T15:09:43.794902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:31.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773468+0000","last_change":"2024-01-16T15:09:43.770407+0000","last_active":"2024-01-16T15:09:43.773468+0000","last_peered":"2024-01-16T15:09:43.773468+0000","last_clean":"2024-01-16T15:09:43.773468+0000","last_became_active":"2024-01-16T15:09:43.770120+0000","last_became_peered":"2024-01-16T15:09:43.770120+0000","last_unstale":"2024-01-16T15:09:43.773468+0000","last_undegraded":"2024-01-16T15:09:43.773468+0000","last_fullsized":"2024-01-16T15:09:43.773468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:23.887665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772223+0000","last_change":"2024-01-16T15:09:42.731404+0000","last_active":"2024-01-16T15:09:43.772223+0000","last_peered":"2024-01-16T15:09:43.772223+0000","last_clean":"2024-01-16T15:09:43.772223+0000","last_became_active":"2024-01-16T15:09:42.730328+0000","last_became_peered":"2024-01-16T15:09:42.730328+0000","last_unstale":"2024-01-16T15:09:43.772223+0000","last_undegraded":"2024-01-16T15:09:43.772223+0000","last_fullsized":"2024-01-16T15:09:43.772223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:15.553232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778856+0000","last_change":"2024-01-16T15:09:43.773691+0000","last_active":"2024-01-16T15:09:43.778856+0000","last_peered":"2024-01-16T15:09:43.778856+0000","last_clean":"2024-01-16T15:09:43.778856+0000","last_became_active":"2024-01-16T15:09:43.773340+0000","last_became_peered":"2024-01-16T15:09:43.773340+0000","last_unstale":"2024-01-16T15:09:43.778856+0000","last_undegraded":"2024-01-16T15:09:43.778856+0000","last_fullsized":"2024-01-16T15:09:43.778856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:16.327868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772194+0000","last_change":"2024-01-16T15:09:42.750149+0000","last_active":"2024-01-16T15:09:43.772194+0000","last_peered":"2024-01-16T15:09:43.772194+0000","last_clean":"2024-01-16T15:09:43.772194+0000","last_became_active":"2024-01-16T15:09:42.749846+0000","last_became_peered":"2024-01-16T15:09:42.749846+0000","last_unstale":"2024-01-16T15:09:43.772194+0000","last_undegraded":"2024-01-16T15:09:43.772194+0000","last_fullsized":"2024-01-16T15:09:43.772194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:42.988000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.782441+0000","last_change":"2024-01-16T15:09:43.781605+0000","last_active":"2024-01-16T15:09:43.782441+0000","last_peered":"2024-01-16T15:09:43.782441+0000","last_clean":"2024-01-16T15:09:43.782441+0000","last_became_active":"2024-01-16T15:09:43.781295+0000","last_became_peered":"2024-01-16T15:09:43.781295+0000","last_unstale":"2024-01-16T15:09:43.782441+0000","last_undegraded":"2024-01-16T15:09:43.782441+0000","last_fullsized":"2024-01-16T15:09:43.782441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:43.239501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.755407+0000","last_change":"2024-01-16T15:09:42.744253+0000","last_active":"2024-01-16T15:09:43.755407+0000","last_peered":"2024-01-16T15:09:43.755407+0000","last_clean":"2024-01-16T15:09:43.755407+0000","last_became_active":"2024-01-16T15:09:42.743273+0000","last_became_peered":"2024-01-16T15:09:42.743273+0000","last_unstale":"2024-01-16T15:09:43.755407+0000","last_undegraded":"2024-01-16T15:09:43.755407+0000","last_fullsized":"2024-01-16T15:09:43.755407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:44.385936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.770881+0000","last_change":"2024-01-16T15:09:43.767596+0000","last_active":"2024-01-16T15:09:43.770881+0000","last_peered":"2024-01-16T15:09:43.770881+0000","last_clean":"2024-01-16T15:09:43.770881+0000","last_became_active":"2024-01-16T15:09:43.767302+0000","last_became_peered":"2024-01-16T15:09:43.767302+0000","last_unstale":"2024-01-16T15:09:43.770881+0000","last_undegraded":"2024-01-16T15:09:43.770881+0000","last_fullsized":"2024-01-16T15:09:43.770881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:54.976998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.791574+0000","last_change":"2024-01-16T15:09:42.740787+0000","last_active":"2024-01-16T15:09:43.791574+0000","last_peered":"2024-01-16T15:09:43.791574+0000","last_clean":"2024-01-16T15:09:43.791574+0000","last_became_active":"2024-01-16T15:09:42.740442+0000","last_became_peered":"2024-01-16T15:09:42.740442+0000","last_unstale":"2024-01-16T15:09:43.791574+0000","last_undegraded":"2024-01-16T15:09:43.791574+0000","last_fullsized":"2024-01-16T15:09:43.791574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:18.087700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778074+0000","last_change":"2024-01-16T15:09:43.774203+0000","last_active":"2024-01-16T15:09:43.778074+0000","last_peered":"2024-01-16T15:09:43.778074+0000","last_clean":"2024-01-16T15:09:43.778074+0000","last_became_active":"2024-01-16T15:09:43.773320+0000","last_became_peered":"2024-01-16T15:09:43.773320+0000","last_unstale":"2024-01-16T15:09:43.778074+0000","last_undegraded":"2024-01-16T15:09:43.778074+0000","last_fullsized":"2024-01-16T15:09:43.778074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:59.638845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.745258+0000","last_change":"2024-01-16T15:09:42.733464+0000","last_active":"2024-01-16T15:09:43.745258+0000","last_peered":"2024-01-16T15:09:43.745258+0000","last_clean":"2024-01-16T15:09:43.745258+0000","last_became_active":"2024-01-16T15:09:42.733147+0000","last_became_peered":"2024-01-16T15:09:42.733147+0000","last_unstale":"2024-01-16T15:09:43.745258+0000","last_undegraded":"2024-01-16T15:09:43.745258+0000","last_fullsized":"2024-01-16T15:09:43.745258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:24.294316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779677+0000","last_change":"2024-01-16T15:09:43.777690+0000","last_active":"2024-01-16T15:09:43.779677+0000","last_peered":"2024-01-16T15:09:43.779677+0000","last_clean":"2024-01-16T15:09:43.779677+0000","last_became_active":"2024-01-16T15:09:43.777376+0000","last_became_peered":"2024-01-16T15:09:43.777376+0000","last_unstale":"2024-01-16T15:09:43.779677+0000","last_undegraded":"2024-01-16T15:09:43.779677+0000","last_fullsized":"2024-01-16T15:09:43.779677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:29.980746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766935+0000","last_change":"2024-01-16T15:09:42.749866+0000","last_active":"2024-01-16T15:09:43.766935+0000","last_peered":"2024-01-16T15:09:43.766935+0000","last_clean":"2024-01-16T15:09:43.766935+0000","last_became_active":"2024-01-16T15:09:42.748923+0000","last_became_peered":"2024-01-16T15:09:42.748923+0000","last_unstale":"2024-01-16T15:09:43.766935+0000","last_undegraded":"2024-01-16T15:09:43.766935+0000","last_fullsized":"2024-01-16T15:09:43.766935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:52.674387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.773002+0000","last_change":"2024-01-16T15:09:43.770934+0000","last_active":"2024-01-16T15:09:43.773002+0000","last_peered":"2024-01-16T15:09:43.773002+0000","last_clean":"2024-01-16T15:09:43.773002+0000","last_became_active":"2024-01-16T15:09:43.767425+0000","last_became_peered":"2024-01-16T15:09:43.767425+0000","last_unstale":"2024-01-16T15:09:43.773002+0000","last_undegraded":"2024-01-16T15:09:43.773002+0000","last_fullsized":"2024-01-16T15:09:43.773002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:46.701324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766703+0000","last_change":"2024-01-16T15:09:42.748671+0000","last_active":"2024-01-16T15:09:43.766703+0000","last_peered":"2024-01-16T15:09:43.766703+0000","last_clean":"2024-01-16T15:09:43.766703+0000","last_became_active":"2024-01-16T15:09:42.748369+0000","last_became_peered":"2024-01-16T15:09:42.748369+0000","last_unstale":"2024-01-16T15:09:43.766703+0000","last_undegraded":"2024-01-16T15:09:43.766703+0000","last_fullsized":"2024-01-16T15:09:43.766703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:18.944496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.779288+0000","last_change":"2024-01-16T15:09:43.777930+0000","last_active":"2024-01-16T15:09:43.779288+0000","last_peered":"2024-01-16T15:09:43.779288+0000","last_clean":"2024-01-16T15:09:43.779288+0000","last_became_active":"2024-01-16T15:09:43.777611+0000","last_became_peered":"2024-01-16T15:09:43.777611+0000","last_unstale":"2024-01-16T15:09:43.779288+0000","last_undegraded":"2024-01-16T15:09:43.779288+0000","last_fullsized":"2024-01-16T15:09:43.779288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:06.095964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.778454+0000","last_change":"2024-01-16T15:09:42.738154+0000","last_active":"2024-01-16T15:09:43.778454+0000","last_peered":"2024-01-16T15:09:43.778454+0000","last_clean":"2024-01-16T15:09:43.778454+0000","last_became_active":"2024-01-16T15:09:42.737866+0000","last_became_peered":"2024-01-16T15:09:42.737866+0000","last_unstale":"2024-01-16T15:09:43.778454+0000","last_undegraded":"2024-01-16T15:09:43.778454+0000","last_fullsized":"2024-01-16T15:09:43.778454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:24.802232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.767606+0000","last_change":"2024-01-16T15:09:43.765099+0000","last_active":"2024-01-16T15:09:43.767606+0000","last_peered":"2024-01-16T15:09:43.767606+0000","last_clean":"2024-01-16T15:09:43.767606+0000","last_became_active":"2024-01-16T15:09:43.764820+0000","last_became_peered":"2024-01-16T15:09:43.764820+0000","last_unstale":"2024-01-16T15:09:43.767606+0000","last_undegraded":"2024-01-16T15:09:43.767606+0000","last_fullsized":"2024-01-16T15:09:43.767606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:11.947840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.744463+0000","last_change":"2024-01-16T15:09:42.742001+0000","last_active":"2024-01-16T15:09:43.744463+0000","last_peered":"2024-01-16T15:09:43.744463+0000","last_clean":"2024-01-16T15:09:43.744463+0000","last_became_active":"2024-01-16T15:09:42.741684+0000","last_became_peered":"2024-01-16T15:09:42.741684+0000","last_unstale":"2024-01-16T15:09:43.744463+0000","last_undegraded":"2024-01-16T15:09:43.744463+0000","last_fullsized":"2024-01-16T15:09:43.744463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:29.034433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.772194+0000","last_change":"2024-01-16T15:09:43.769403+0000","last_active":"2024-01-16T15:09:43.772194+0000","last_peered":"2024-01-16T15:09:43.772194+0000","last_clean":"2024-01-16T15:09:43.772194+0000","last_became_active":"2024-01-16T15:09:43.768512+0000","last_became_peered":"2024-01-16T15:09:43.768512+0000","last_unstale":"2024-01-16T15:09:43.772194+0000","last_undegraded":"2024-01-16T15:09:43.772194+0000","last_fullsized":"2024-01-16T15:09:43.772194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:52.380714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.766367+0000","last_change":"2024-01-16T15:09:42.743879+0000","last_active":"2024-01-16T15:09:43.766367+0000","last_peered":"2024-01-16T15:09:43.766367+0000","last_clean":"2024-01-16T15:09:43.766367+0000","last_became_active":"2024-01-16T15:09:42.743589+0000","last_became_peered":"2024-01-16T15:09:42.743589+0000","last_unstale":"2024-01-16T15:09:43.766367+0000","last_undegraded":"2024-01-16T15:09:43.766367+0000","last_fullsized":"2024-01-16T15:09:43.766367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:16.282677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.774954+0000","last_change":"2024-01-16T15:09:43.771557+0000","last_active":"2024-01-16T15:09:43.774954+0000","last_peered":"2024-01-16T15:09:43.774954+0000","last_clean":"2024-01-16T15:09:43.774954+0000","last_became_active":"2024-01-16T15:09:43.771165+0000","last_became_peered":"2024-01-16T15:09:43.771165+0000","last_unstale":"2024-01-16T15:09:43.774954+0000","last_undegraded":"2024-01-16T15:09:43.774954+0000","last_fullsized":"2024-01-16T15:09:43.774954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:46.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.748930+0000","last_change":"2024-01-16T15:09:42.747369+0000","last_active":"2024-01-16T15:09:43.748930+0000","last_peered":"2024-01-16T15:09:43.748930+0000","last_clean":"2024-01-16T15:09:43.748930+0000","last_became_active":"2024-01-16T15:09:42.747090+0000","last_became_peered":"2024-01-16T15:09:42.747090+0000","last_unstale":"2024-01-16T15:09:43.748930+0000","last_undegraded":"2024-01-16T15:09:43.748930+0000","last_fullsized":"2024-01-16T15:09:43.748930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:42.228900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.771338+0000","last_change":"2024-01-16T15:09:43.767389+0000","last_active":"2024-01-16T15:09:43.771338+0000","last_peered":"2024-01-16T15:09:43.771338+0000","last_clean":"2024-01-16T15:09:43.771338+0000","last_became_active":"2024-01-16T15:09:43.767024+0000","last_became_peered":"2024-01-16T15:09:43.767024+0000","last_unstale":"2024-01-16T15:09:43.771338+0000","last_undegraded":"2024-01-16T15:09:43.771338+0000","last_fullsized":"2024-01-16T15:09:43.771338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:42.706590+0000","last_clean_scrub_stamp":"2024-01-16T15:09:42.706590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:23.256626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:43.793060+0000","last_change":"2024-01-16T15:09:42.736329+0000","last_active":"2024-01-16T15:09:43.793060+0000","last_peered":"2024-01-16T15:09:43.793060+0000","last_clean":"2024-01-16T15:09:43.793060+0000","last_became_active":"2024-01-16T15:09:42.736052+0000","last_became_peered":"2024-01-16T15:09:42.736052+0000","last_unstale":"2024-01-16T15:09:43.793060+0000","last_undegraded":"2024-01-16T15:09:43.793060+0000","last_fullsized":"2024-01-16T15:09:43.793060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:41.695990+0000","last_clean_scrub_stamp":"2024-01-16T15:09:41.695990+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:31.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":236,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-16T15:09:42.724894+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:09:42.724894+0000","last_peered":"2024-01-16T15:09:42.724894+0000","last_clean":"2024-01-16T15:09:42.724894+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:09:42.724894+0000","last_undegraded":"2024-01-16T15:09:42.724894+0000","last_fullsized":"2024-01-16T15:09:42.724894+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77124,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294716,"statfs":{"total":96636764160,"available":96557789184,"internally_reserved":0,"allocated":1626112,"data_stored":1029769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2729999999999999,"15min":1.3979999999999999},"min":{"1min":0.64600000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.5,"5min":8.5399999999999991,"15min":45.296999999999997},"last":1.411},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1479999999999999,"15min":1.3049999999999999},"min":{"1min":0.65600000000000003,"5min":0.55900000000000005,"15min":0.497},"max":{"1min":1.4650000000000001,"5min":2.8799999999999999,"15min":20.126999999999999},"last":1.5249999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.78300000000000003,"5min":0.56200000000000006,"15min":0.499},"max":{"1min":4.2919999999999998,"5min":7.4809999999999999,"15min":42.723999999999997},"last":1.27},{"interface":"front","average":{"1min":1.304,"5min":1.387,"15min":1.5660000000000001},"min":{"1min":0.752,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9430000000000001,"5min":5.1820000000000004,"15min":43.572000000000003},"last":1.367}]},{"osd":2,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.425,"15min":1.448},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.52100000000000002},"max":{"1min":1.649,"5min":14.858000000000001,"15min":35.308},"last":1.48},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.4139999999999999,"15min":1.472},"min":{"1min":0.76500000000000001,"5min":0.65800000000000003,"15min":0.503},"max":{"1min":2.6920000000000002,"5min":9.1899999999999995,"15min":23.594000000000001},"last":1.556}]},{"osd":3,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.603,"15min":1.601},"min":{"1min":0.65900000000000003,"5min":0.57999999999999996,"15min":0.51300000000000001},"max":{"1min":1.581,"5min":14.741,"15min":53.033999999999999},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.677,"15min":1.6060000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.60299999999999998},"max":{"1min":2.645,"5min":14.708,"15min":31.305},"last":1.819}]},{"osd":4,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.339},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.39600000000000002},"max":{"1min":2.028,"5min":2.2410000000000001,"15min":5.1870000000000003},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.4850000000000001,"15min":1.522},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":1.7110000000000001,"5min":14.948,"15min":36.374000000000002},"last":1.321}]},{"osd":5,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.488,"15min":1.4550000000000001},"min":{"1min":0.61799999999999999,"5min":0.55500000000000005,"15min":0.41499999999999998},"max":{"1min":2.593,"5min":9.1370000000000005,"15min":9.9399999999999995},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4319999999999999,"15min":1.474},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.46899999999999997},"max":{"1min":2.0350000000000001,"5min":2.9399999999999999,"15min":7.0170000000000003},"last":1.6120000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.671,"15min":1.5669999999999999},"min":{"1min":0.59899999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.2309999999999999,"5min":15.052,"15min":15.052},"last":1.651},{"interface":"front","average":{"1min":1.569,"5min":1.6539999999999999,"15min":1.625},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.49099999999999999},"max":{"1min":2.7360000000000002,"5min":8.4849999999999994,"15min":8.4849999999999994},"last":1.7589999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76544,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295296,"statfs":{"total":96636764160,"available":96558383104,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77330886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4319999999999999,"15min":1.6359999999999999},"min":{"1min":0.80900000000000005,"5min":0.65300000000000002,"15min":0.60099999999999998},"max":{"1min":1.657,"5min":9.1400000000000006,"15min":71.870999999999995},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.4370000000000001,"15min":1.744},"min":{"1min":0.82899999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.607,"5min":6.4059999999999997,"15min":71.846999999999994},"last":1.026}]},{"osd":1,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.603,"15min":1.3100000000000001},"min":{"1min":0.88400000000000001,"5min":0.70099999999999996,"15min":0.63900000000000001},"max":{"1min":1.7569999999999999,"5min":15.798,"15min":15.798},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.23,"5min":1.681,"15min":1.431},"min":{"1min":0.753,"5min":0.626,"15min":0.626},"max":{"1min":1.6679999999999999,"5min":24.556999999999999,"15min":24.556999999999999},"last":0.91300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.6919999999999999,"15min":1.4419999999999999},"min":{"1min":0.628,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.595,"5min":14.981,"15min":14.981},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.8240000000000001,"15min":1.5069999999999999},"min":{"1min":0.69799999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.619,"5min":14.175000000000001,"15min":14.175000000000001},"last":1.3109999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.4850000000000001,"15min":1.4910000000000001},"min":{"1min":0.78000000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":9.2170000000000005,"15min":14.827},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.7729999999999999,"15min":1.5820000000000001},"min":{"1min":0.81599999999999995,"5min":0.499,"15min":0.499},"max":{"1min":1.7629999999999999,"5min":21.890000000000001,"15min":21.890000000000001},"last":1.379}]},{"osd":4,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4710000000000001,"15min":1.4470000000000001},"min":{"1min":0.90200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.4980000000000002,"5min":10.901,"15min":10.901},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.327,"15min":1.444},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6910000000000001,"5min":4.9530000000000003,"15min":6.9000000000000004},"last":1.7629999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.4179999999999999,"15min":1.4199999999999999},"min":{"1min":0.52300000000000002,"5min":0.44600000000000001,"15min":0.38500000000000001},"max":{"1min":1.726,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.718},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5860000000000001,"15min":1.619},"min":{"1min":0.66600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":2.2400000000000002,"5min":9.2200000000000006,"15min":9.2200000000000006},"last":1.923}]},{"osd":7,"last update":"Tue Jan 16 15:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.536,"15min":1.631},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.927,"5min":6.4349999999999996,"15min":14.74},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.944,"15min":1.694},"min":{"1min":0.64600000000000002,"5min":0.54600000000000004,"15min":0.42599999999999999},"max":{"1min":1.788,"5min":14.494,"15min":14.494},"last":1.8919999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54432,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317408,"statfs":{"total":96636764160,"available":96581025792,"internally_reserved":0,"allocated":1060864,"data_stored":447810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.788,"15min":1.504},"min":{"1min":0.65600000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.5070000000000001,"5min":25.634,"15min":25.634},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.66600000000000004,"5min":0.59199999999999997,"15min":0.48299999999999998},"max":{"1min":4.2469999999999999,"5min":16.263000000000002,"15min":34.959000000000003},"last":0.82199999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.577,"15min":1.4219999999999999},"min":{"1min":0.71499999999999997,"5min":0.44,"15min":0.44},"max":{"1min":3.0139999999999998,"5min":26.378,"15min":26.378},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.3779999999999999,"15min":1.3220000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58299999999999996},"max":{"1min":4.3380000000000001,"5min":6.125,"15min":13.332000000000001},"last":1.073}]},{"osd":2,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.619,"15min":1.4139999999999999},"min":{"1min":0.68700000000000006,"5min":0.54700000000000004,"15min":0.42599999999999999},"max":{"1min":3.1179999999999999,"5min":25.495000000000001,"15min":25.495000000000001},"last":1.329},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.4870000000000001,"15min":1.4550000000000001},"min":{"1min":0.75900000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"max":{"1min":4.0229999999999997,"5min":6.7599999999999998,"15min":17.030000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.6919999999999999,"15min":1.6659999999999999},"min":{"1min":0.74399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0840000000000001,"5min":25.82,"15min":25.82},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.633,"5min":1.7450000000000001,"15min":1.6779999999999999},"min":{"1min":0.79800000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":4.1429999999999998,"5min":17.713999999999999,"15min":22.963999999999999},"last":1.169}]},{"osd":4,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2769999999999999,"15min":1.2889999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.378},"max":{"1min":6.1660000000000004,"5min":6.1660000000000004,"15min":8.6050000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.7549999999999999,"15min":1.5920000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.48299999999999998},"max":{"1min":1.7270000000000001,"5min":25.724,"15min":25.724},"last":1.554}]},{"osd":6,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":1.974,"15min":1.653},"min":{"1min":0.76700000000000002,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":7.6669999999999998,"5min":42.542999999999999,"15min":42.542999999999999},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.395,"15min":1.403},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.44600000000000001},"max":{"1min":6.093,"5min":6.093,"15min":10.015000000000001},"last":1.006}]},{"osd":7,"last update":"Tue Jan 16 15:09:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":2.1349999999999998,"15min":1.716},"min":{"1min":0.60399999999999998,"5min":0.47899999999999998,"15min":0.45700000000000002},"max":{"1min":1.712,"5min":26.484000000000002,"15min":26.484000000000002},"last":1.641},{"interface":"front","average":{"1min":1.587,"5min":1.6890000000000001,"15min":1.7010000000000001},"min":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.46999999999999997},"max":{"1min":5.0209999999999999,"5min":17.233000000000001,"15min":21.106000000000002},"last":1.385}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76544,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295296,"statfs":{"total":96636764160,"available":96558383104,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.415,"15min":1.333},"min":{"1min":0.65800000000000003,"5min":0.58199999999999996,"15min":0.44400000000000001},"max":{"1min":3.2919999999999998,"5min":7.1840000000000002,"15min":7.1840000000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4279999999999999,"15min":1.353},"min":{"1min":0.93700000000000006,"5min":0.60399999999999998,"15min":0.52800000000000002},"max":{"1min":3.1779999999999999,"5min":5.524,"15min":7.2190000000000003},"last":1.5549999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4550000000000001,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68799999999999994,"15min":0.57699999999999996},"max":{"1min":5.0019999999999998,"5min":8.9179999999999993,"15min":39.823999999999998},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.2450000000000001,"15min":1.3779999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.48399999999999999},"max":{"1min":1.734,"5min":6.9429999999999996,"15min":50.582000000000001},"last":1.0189999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.4350000000000001,"15min":1.341},"min":{"1min":0.5,"5min":0.5,"15min":0.39500000000000002},"max":{"1min":1.9219999999999999,"5min":10.984,"15min":10.984},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.454,"15min":1.393},"min":{"1min":0.90000000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.4780000000000002,"5min":6.8179999999999996,"15min":9.1509999999999998},"last":1.4830000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4059999999999999,"15min":1.359},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.54000000000000004},"max":{"1min":3.456,"5min":8.8979999999999997,"15min":11.971},"last":1.256},{"interface":"front","average":{"1min":1.268,"5min":1.5589999999999999,"15min":1.6479999999999999},"min":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.57899999999999996},"max":{"1min":1.6990000000000001,"5min":9.4329999999999998,"15min":27.626999999999999},"last":1.5740000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.597,"5min":4.2329999999999997,"15min":8.2599999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.6479999999999999,"15min":1.5369999999999999},"min":{"1min":0.72499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.7470000000000001,"5min":17.140999999999998,"15min":17.140999999999998},"last":1.129}]},{"osd":6,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.508,"15min":1.512},"min":{"1min":0.57099999999999995,"5min":0.48299999999999998,"15min":0.45300000000000001},"max":{"1min":1.7969999999999999,"5min":7.4509999999999996,"15min":7.4509999999999996},"last":1.3680000000000001},{"interface":"front","average":{"1min":2.073,"5min":1.865,"15min":1.641},"min":{"1min":1.26,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":12.324999999999999,"5min":25.643000000000001,"15min":25.643000000000001},"last":1.603}]},{"osd":7,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.7350000000000001,"15min":1.6060000000000001},"min":{"1min":0.94799999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.984,"5min":24.992999999999999,"15min":24.992999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.694,"15min":1.7809999999999999},"min":{"1min":0.78800000000000003,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.897,"5min":8.8369999999999997,"15min":19.001000000000001},"last":1.6299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1032192,"data_stored":439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.05,"15min":1.159},"min":{"1min":0.60699999999999998,"5min":0.374,"15min":0.313},"max":{"1min":1.1279999999999999,"5min":9.6229999999999993,"15min":38.423999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.224,"15min":1.1870000000000001},"min":{"1min":0.438,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.556,"5min":7.8300000000000001,"15min":20.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.4510000000000001,"15min":1.5960000000000001},"min":{"1min":0.57999999999999996,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.655,"5min":19.215,"15min":40.695999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.054,"5min":1.272,"15min":1.208},"min":{"1min":0.78300000000000003,"5min":0.46500000000000002,"15min":0.34100000000000003},"max":{"1min":1.3120000000000001,"5min":8.5449999999999999,"15min":9.9689999999999994},"last":0.93300000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.4079999999999999,"15min":1.526},"min":{"1min":0.755,"5min":0.436,"15min":0.436},"max":{"1min":1.599,"5min":10.391,"15min":59.009},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.6639999999999999,"15min":1.6379999999999999},"min":{"1min":0.72599999999999998,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":1.79,"5min":19.152000000000001,"15min":29.773},"last":1.153}]},{"osd":4,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.478,"15min":1.4870000000000001},"min":{"1min":0.85399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.671,"5min":7.4950000000000001,"15min":21.736000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.9570000000000001,"15min":1.9019999999999999},"min":{"1min":1.1040000000000001,"5min":0.622,"15min":0.622},"max":{"1min":1.7869999999999999,"5min":19.292000000000002,"15min":29.524000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.526,"15min":1.522},"min":{"1min":1.04,"5min":0.64500000000000002,"15min":0.54800000000000004},"max":{"1min":1.734,"5min":8.0739999999999998,"15min":9.8249999999999993},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.6899999999999999,"15min":1.5960000000000001},"min":{"1min":1.135,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8320000000000001,"5min":11.255000000000001,"15min":11.255000000000001},"last":1.0249999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.587,"15min":1.589},"min":{"1min":1.1359999999999999,"5min":0.81599999999999995,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":7.4560000000000004,"15min":20.459},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":2.0870000000000002,"15min":1.9990000000000001},"min":{"1min":1.264,"5min":0.67000000000000004,"15min":0.49399999999999999},"max":{"1min":1.9710000000000001,"5min":19.254999999999999,"15min":29.475000000000001},"last":1.4930000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.6140000000000001,"15min":1.637},"min":{"1min":1.244,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.9990000000000001,"5min":7.5339999999999998,"15min":20.109999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.611,"5min":2.1070000000000002,"15min":2.0099999999999998},"min":{"1min":1.169,"5min":0.65000000000000002,"15min":0.59999999999999998},"max":{"1min":2.1859999999999999,"5min":19.327999999999999,"15min":29.396999999999998},"last":1.5720000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1048576,"data_stored":446758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.1990000000000001,"15min":1.1659999999999999},"min":{"1min":0.55600000000000005,"5min":0.32500000000000001,"15min":0.28899999999999998},"max":{"1min":2.8919999999999999,"5min":24.155000000000001,"15min":25.335999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.3080000000000001,"15min":1.234},"min":{"1min":0.57699999999999996,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":2.952,"5min":25.199000000000002,"15min":25.199000000000002},"last":1.202}]},{"osd":1,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.2749999999999999,"15min":1.2549999999999999},"min":{"1min":0.64200000000000002,"5min":0.442,"15min":0.36199999999999999},"max":{"1min":1.6479999999999999,"5min":16.776,"15min":16.776},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.617,"15min":1.4770000000000001},"min":{"1min":0.78400000000000003,"5min":0.39100000000000001,"15min":0.318},"max":{"1min":3.2810000000000001,"5min":25.186,"15min":25.186},"last":0.89700000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5429999999999999,"15min":1.286},"min":{"1min":0.48499999999999999,"5min":0.41699999999999998,"15min":0.33900000000000002},"max":{"1min":1.712,"5min":37.470999999999997,"15min":37.470999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.167,"5min":1.2949999999999999,"15min":1.3799999999999999},"min":{"1min":0.629,"5min":0.54000000000000004,"15min":0.46500000000000002},"max":{"1min":1.506,"5min":6.5549999999999997,"15min":28.045999999999999},"last":1.131}]},{"osd":4,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.976,"15min":1.752},"min":{"1min":0.65200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":6.3630000000000004,"5min":25.062000000000001,"15min":25.062000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.8520000000000001,"15min":1.6479999999999999},"min":{"1min":0.92800000000000005,"5min":0.57599999999999996,"15min":0.502},"max":{"1min":1.869,"5min":37.307000000000002,"15min":37.307000000000002},"last":1.0169999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.5029999999999999,"15min":1.5740000000000001},"min":{"1min":0.72799999999999998,"5min":0.629,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":6.8849999999999998,"15min":19.327000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":2.0419999999999998,"15min":1.794},"min":{"1min":0.74299999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":3.5350000000000001,"5min":25.141999999999999,"15min":25.141999999999999},"last":1.351}]},{"osd":6,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.792,"15min":1.6379999999999999},"min":{"1min":1.1180000000000001,"5min":0.68400000000000005,"15min":0.56000000000000005},"max":{"1min":1.855,"5min":37.363,"15min":37.363},"last":1.097},{"interface":"front","average":{"1min":1.45,"5min":1.5920000000000001,"15min":1.5940000000000001},"min":{"1min":0.68899999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6230000000000002,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.377}]},{"osd":7,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":2.0790000000000002,"15min":1.869},"min":{"1min":0.93899999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.7530000000000001,"5min":25.106999999999999,"15min":25.106999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.899,"15min":1.7310000000000001},"min":{"1min":1.0920000000000001,"5min":0.72499999999999998,"15min":0.61499999999999999},"max":{"1min":1.9790000000000001,"5min":37.402999999999999,"15min":37.402999999999999},"last":1.2370000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64088,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307752,"statfs":{"total":96636764160,"available":96571138048,"internally_reserved":0,"allocated":1056768,"data_stored":446370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.431,"15min":1.194},"min":{"1min":0.36399999999999999,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":1.478,"5min":17.210999999999999,"15min":17.210999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.379,"15min":1.3100000000000001},"min":{"1min":0.65700000000000003,"5min":0.34300000000000003,"15min":0.26700000000000002},"max":{"1min":1.397,"5min":20.164000000000001,"15min":34.252000000000002},"last":4.3719999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.51,"15min":1.319},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.36099999999999999},"max":{"1min":3.6309999999999998,"5min":24.102,"15min":24.709},"last":1.476},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.605,"15min":1.452},"min":{"1min":0.78900000000000003,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":1.5309999999999999,"5min":21.795999999999999,"15min":30.128},"last":1.2929999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.9370000000000001,"15min":1.5349999999999999},"min":{"1min":0.63500000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.177,"5min":25.611000000000001,"15min":25.611000000000001},"last":3.3109999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.8300000000000001,"15min":1.4239999999999999},"min":{"1min":0.71499999999999997,"5min":0.41199999999999998,"15min":0.38900000000000001},"max":{"1min":1.6399999999999999,"5min":41.972000000000001,"15min":41.972000000000001},"last":4.6699999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.474,"15min":1.4419999999999999},"min":{"1min":0.751,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.3170000000000002,"5min":18.004999999999999,"15min":18.004999999999999},"last":3.2349999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":2.0030000000000001,"15min":1.655},"min":{"1min":1.004,"5min":0.67200000000000004,"15min":0.63700000000000001},"max":{"1min":7.3310000000000004,"5min":21.738,"15min":21.738},"last":5.8170000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4119999999999999,"5min":2.0569999999999999,"15min":1.7210000000000001},"min":{"1min":1.083,"5min":0.54900000000000004,"15min":0.46600000000000003},"max":{"1min":17.739999999999998,"5min":20.905999999999999,"15min":20.905999999999999},"last":4.1520000000000001},{"interface":"front","average":{"1min":2.258,"5min":1.9259999999999999,"15min":1.7410000000000001},"min":{"1min":0.79000000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":16.433,"5min":17.957999999999998,"15min":17.957999999999998},"last":5.258}]},{"osd":6,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":2.0489999999999999,"15min":1.7250000000000001},"min":{"1min":1.1830000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.881,"5min":21.832999999999998,"15min":21.832999999999998},"last":4.4089999999999998},{"interface":"front","average":{"1min":1.585,"5min":1.891,"15min":1.6220000000000001},"min":{"1min":1.2310000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.2669999999999999,"5min":20.509,"15min":20.509},"last":6.5350000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":2.0760000000000001,"15min":1.762},"min":{"1min":1.2270000000000001,"5min":0.75700000000000001,"15min":0.55600000000000005},"max":{"1min":2.379,"5min":20.669,"15min":20.669},"last":12.590999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.665,"15min":1.631},"min":{"1min":0.82399999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.794,"5min":18.039999999999999,"15min":18.039999999999999},"last":6.0229999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77132,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294708,"statfs":{"total":96636764160,"available":96557780992,"internally_reserved":0,"allocated":1638400,"data_stored":1035559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0980000000000001,"15min":1.1699999999999999},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.4119999999999999,"5min":10.683999999999999,"15min":26.751000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.996,"5min":1.355,"15min":1.181},"min":{"1min":0.434,"5min":0.434,"15min":0.35899999999999999},"max":{"1min":1.728,"5min":12.648999999999999,"15min":15.750999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.744,"15min":1.587},"min":{"1min":0.498,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":35.320999999999998,"15min":38.517000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.3680000000000001,"15min":1.288},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38800000000000001},"max":{"1min":1.841,"5min":11.720000000000001,"15min":24.402000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.8500000000000001,"15min":1.5},"min":{"1min":0.77200000000000002,"5min":0.45700000000000002,"15min":0.39100000000000001},"max":{"1min":3.4670000000000001,"5min":29.808,"15min":29.808},"last":0.621},{"interface":"front","average":{"1min":1.306,"5min":1.631,"15min":1.712},"min":{"1min":0.57999999999999996,"5min":0.51700000000000002,"15min":0.47799999999999998},"max":{"1min":2.0430000000000001,"5min":14.558,"15min":32.302},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.7170000000000001,"15min":1.5780000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":3.0899999999999999,"5min":9.6080000000000005,"15min":15.832000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.704,"15min":1.5429999999999999},"min":{"1min":0.997,"5min":0.61499999999999999,"15min":0.432},"max":{"1min":3.6829999999999998,"5min":14.685,"15min":19.062999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.579,"15min":1.7030000000000001},"min":{"1min":0.624,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.887,"5min":4.7949999999999999,"15min":26.800000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.7450000000000001,"15min":1.6140000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.0699999999999998,"5min":7.1550000000000002,"15min":15.906000000000001},"last":1.0469999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.75,"15min":1.6759999999999999},"min":{"1min":1.034,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":3.9790000000000001,"5min":14.772,"15min":20.844000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.502,"5min":1.5960000000000001,"15min":1.7050000000000001},"min":{"1min":1.123,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9910000000000001,"5min":4.4109999999999996,"15min":26.832000000000001},"last":0.86299999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.9470000000000001,"15min":1.704},"min":{"1min":0.69799999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.968,"5min":16.058,"15min":16.058},"last":0.754},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.9059999999999999,"15min":1.895},"min":{"1min":1.0629999999999999,"5min":0.75,"15min":0.47499999999999998},"max":{"1min":3.746,"5min":14.811,"15min":29.286999999999999},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:09:48.671 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:09:48.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:09:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.947+0000 7f4efac00640 1 Processor -- start 2024-01-16T15:09:48.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.947+0000 7f4efac00640 1 -- start start 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef40cf190 0x7f4ef40d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4ef40b07a0 0x7f4ef40cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efac00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40ae270 con 0x7f4ef40add80 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40b0c30 con 0x7f4ef40b07a0 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40b0f10 con 0x7f4ef40cf190 2024-01-16T15:09:48.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4ef8e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4ef40b07a0 0x7f4ef40cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efa200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4ef40add80 0x7f4ef40b1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39552/0 (socket says 172.21.15.125:39552) 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4ef8e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4ef40b07a0 0x7f4ef40cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39404/0 (socket says 172.21.15.125:39404) 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.949+0000 7f4efa200640 1 -- 172.21.15.125:0/1416009810 learned_addr learned my addr 172.21.15.125:0/1416009810 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef9800640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef40cf190 0x7f4ef40d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef8e00640 1 -- 172.21.15.125:0/1416009810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef40cf190 msgr2=0x7f4ef40d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef8e00640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef40cf190 0x7f4ef40d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef8e00640 1 -- 172.21.15.125:0/1416009810 >> v1:172.21.15.125:6789/0 conn(0x7f4ef40add80 legacy=0x7f4ef40b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef8e00640 1 -- 172.21.15.125:0/1416009810 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40b11f0 con 0x7f4ef40b07a0 2024-01-16T15:09:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.950+0000 7f4ef3e00640 1 -- 172.21.15.125:0/1416009810 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322251833 0 0) 0x7f4ef40ae270 con 0x7f4ef40add80 2024-01-16T15:09:48.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.951+0000 7f4ef8e00640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40b07a0 0x7f4ef40cf570 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0097a0 tx=0x7f4edc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac2eccafb84b0c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.951+0000 7f4ef3e00640 1 -- 172.21.15.125:0/1416009810 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc031780 con 0x7f4ef40b07a0 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.951+0000 7f4ef3e00640 1 -- 172.21.15.125:0/1416009810 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc0318e0 con 0x7f4ef40b07a0 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.951+0000 7f4ef3e00640 1 -- 172.21.15.125:0/1416009810 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc031c70 con 0x7f4ef40b07a0 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 -- 172.21.15.125:0/1416009810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40b07a0 msgr2=0x7f4ef40cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40b07a0 0x7f4ef40cf570 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0097a0 tx=0x7f4edc02b820 comp rx=0 tx=0).stop 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 -- 172.21.15.125:0/1416009810 shutdown_connections 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40b07a0 0x7f4ef40cf570 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 --2- 172.21.15.125:0/1416009810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef40cf190 0x7f4ef40d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 -- 172.21.15.125:0/1416009810 >> 172.21.15.125:0/1416009810 conn(0x7f4ef4099a60 msgr2=0x7f4ef4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 -- 172.21.15.125:0/1416009810 shutdown_connections 2024-01-16T15:09:48.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.952+0000 7f4efac00640 1 -- 172.21.15.125:0/1416009810 wait complete. 2024-01-16T15:09:48.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.954+0000 7f4efac00640 1 Processor -- start 2024-01-16T15:09:48.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.954+0000 7f4efac00640 1 -- start start 2024-01-16T15:09:48.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 0x7f4ef40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef414ffa0 0x7f4ef406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40b57e0 con 0x7f4ef40cf190 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40b5ac0 con 0x7f4ef40add80 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4efac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40b5da0 con 0x7f4ef414ffa0 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.955+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39418/0 (socket says 172.21.15.125:39418) 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 -- 172.21.15.125:0/3112910303 learned_addr learned my addr 172.21.15.125:0/3112910303 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef9800640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 0x7f4ef40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4efa200640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef414ffa0 0x7f4ef406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 -- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef414ffa0 msgr2=0x7f4ef406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef414ffa0 0x7f4ef406ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 -- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 msgr2=0x7f4ef40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 0x7f4ef40b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 -- 172.21.15.125:0/3112910303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc009400 con 0x7f4ef40cf190 2024-01-16T15:09:48.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef9800640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 0x7f4ef40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.956+0000 7f4ef8e00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4edc034040 tx=0x7f4edc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.957+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc032820 con 0x7f4ef40cf190 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.957+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc032980 con 0x7f4ef40cf190 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.957+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc032d30 con 0x7f4ef40cf190 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.957+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef40b6080 con 0x7f4ef40cf190 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.957+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef406f280 con 0x7f4ef40cf190 2024-01-16T15:09:48.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.958+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef406b580 con 0x7f4ef40cf190 2024-01-16T15:09:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.959+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4edc04e440 con 0x7f4ef40cf190 2024-01-16T15:09:48.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.960+0000 7f4ef1600640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 0x7f4ecc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.960+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4edc0aa980 con 0x7f4ef40cf190 2024-01-16T15:09:48.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.960+0000 7f4ef9800640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 0x7f4ecc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.961+0000 7f4ef9800640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 0x7f4ecc06e3e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4006fd0 tx=0x7f4ee4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:48.964+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ef406b580 con 0x7f4ef40cf190 2024-01-16T15:09:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.222+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4ef406b580 con 0x7f4ef40cf190 2024-01-16T15:09:49.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.228+0000 7f4ef1600640 1 -- 172.21.15.125:0/3112910303 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4ef406b580 con 0x7f4ef40cf190 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:09:49.206 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 msgr2=0x7f4ecc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 0x7f4ecc06e3e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4006fd0 tx=0x7f4ee4008040 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 msgr2=0x7f4ef40b5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4edc034040 tx=0x7f4edc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 shutdown_connections 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ef414ffa0 0x7f4ef406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ecc06bf20 0x7f4ecc06e3e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ef40cf190 0x7f4ef40b5160 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 --2- 172.21.15.125:0/3112910303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ef40add80 0x7f4ef40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 >> 172.21.15.125:0/3112910303 conn(0x7f4ef4099a60 msgr2=0x7f4ef40ae490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 shutdown_connections 2024-01-16T15:09:49.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.231+0000 7f4efac00640 1 -- 172.21.15.125:0/3112910303 wait complete. 2024-01-16T15:09:49.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:49.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.456+0000 7fce25200640 1 Processor -- start 2024-01-16T15:09:49.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.456+0000 7fce25200640 1 -- start start 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce25200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce25200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce2015e3b0 0x7fce200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce25200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200cfc30 con 0x7fce200cf7d0 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce25200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200b1ee0 con 0x7fce2015e3b0 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce25200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200b2780 con 0x7fce20143ae0 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce2015e3b0 0x7fce200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce2015e3b0 0x7fce200702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39430/0 (socket says 172.21.15.125:39430) 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 -- 172.21.15.125:0/1610594758 learned_addr learned my addr 172.21.15.125:0/1610594758 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1de00640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 -- 172.21.15.125:0/1610594758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 msgr2=0x7fce200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 -- 172.21.15.125:0/1610594758 >> v1:172.21.15.137:6789/0 conn(0x7fce200cf7d0 legacy=0x7fce2015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1e800640 1 -- 172.21.15.125:0/1610594758 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2015e990 con 0x7fce2015e3b0 2024-01-16T15:09:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.458+0000 7fce1de00640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:49.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.459+0000 7fce1e800640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce2015e3b0 0x7fce200702c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fce18009e50 tx=0x7fce1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6916b0f0fe7931 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce1d400640 1 -- 172.21.15.125:0/1610594758 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1802ca70 con 0x7fce2015e3b0 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce1d400640 1 -- 172.21.15.125:0/1610594758 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1802cbd0 con 0x7fce2015e3b0 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce1d400640 1 -- 172.21.15.125:0/1610594758 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce180343f0 con 0x7fce2015e3b0 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 -- 172.21.15.125:0/1610594758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce2015e3b0 msgr2=0x7fce200702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce2015e3b0 0x7fce200702c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fce18009e50 tx=0x7fce1802bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 -- 172.21.15.125:0/1610594758 shutdown_connections 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce2015e3b0 0x7fce200702c0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 --2- 172.21.15.125:0/1610594758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce20143ae0 0x7fce200cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.460+0000 7fce25200640 1 -- 172.21.15.125:0/1610594758 >> 172.21.15.125:0/1610594758 conn(0x7fce2006a800 msgr2=0x7fce2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.461+0000 7fce25200640 1 -- 172.21.15.125:0/1610594758 shutdown_connections 2024-01-16T15:09:49.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.461+0000 7fce25200640 1 -- 172.21.15.125:0/1610594758 wait complete. 2024-01-16T15:09:49.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.462+0000 7fce25200640 1 Processor -- start 2024-01-16T15:09:49.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.462+0000 7fce25200640 1 -- start start 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 0x7fce201f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce2015e3b0 0x7fce200afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce201f4ee0 con 0x7fce20143ae0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2013b500 con 0x7fce2015e3b0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce25200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2013e8f0 con 0x7fce200cf7d0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.463+0000 7fce17e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39434/0 (socket says 172.21.15.125:39434) 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 -- 172.21.15.125:0/3768800126 learned_addr learned my addr 172.21.15.125:0/3768800126 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce1de00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 0x7fce201f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce1e800640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce2015e3b0 0x7fce200afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 -- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 msgr2=0x7fce201f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 0x7fce201f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 -- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce2015e3b0 msgr2=0x7fce200afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce2015e3b0 0x7fce200afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 -- 172.21.15.125:0/3768800126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009b30 con 0x7fce20143ae0 2024-01-16T15:09:49.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce1de00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 0x7fce201f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:49.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.464+0000 7fce17e00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fce0800c430 tx=0x7fce0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.465+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08011790 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.465+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce080118f0 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.465+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08011ce0 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.465+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2013dd70 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.465+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2013e330 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.466+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2013b2d0 con 0x7fce20143ae0 2024-01-16T15:09:49.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.467+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fce0801c460 con 0x7fce20143ae0 2024-01-16T15:09:49.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.468+0000 7fce16000640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 0x7fcdf806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.468+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce08088570 con 0x7fce20143ae0 2024-01-16T15:09:49.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.468+0000 7fce1de00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 0x7fcdf806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.469+0000 7fce1de00640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 0x7fcdf806e410 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.472+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce2013b2d0 con 0x7fce20143ae0 2024-01-16T15:09:49.660 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.683+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce2013b2d0 con 0x7fce20143ae0 2024-01-16T15:09:49.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.684+0000 7fce16000640 1 -- 172.21.15.125:0/3768800126 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fce2013b2d0 con 0x7fce20143ae0 2024-01-16T15:09:49.662 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:09:49.662 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/4149833712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":4149833712},{"type":"v1","addr":"172.21.15.137:6834","nonce":4149833712}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":6235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2076499076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2076499076},{"type":"v1","addr":"172.21.15.137:6835","nonce":2076499076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8172,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3233056488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3233056488},{"type":"v1","addr":"172.21.15.125:6835","nonce":3233056488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:09:43.185893+0000","modified":"2024-01-16T15:09:46.262939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8181":{"gid":8181,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/589280594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":589280594},{"type":"v1","addr":"172.21.15.125:6837","nonce":589280594}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:09:49.662 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 54 2024-01-16T15:09:49.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 msgr2=0x7fcdf806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 0x7fcdf806e410 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10008040 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 msgr2=0x7fce201570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fce0800c430 tx=0x7fce0800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 shutdown_connections 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce2015e3b0 0x7fce200afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcdf806bf50 0x7fcdf806e410 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce20143ae0 0x7fce201570d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.686+0000 7fce25200640 1 --2- 172.21.15.125:0/3768800126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce200cf7d0 0x7fce201f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.687+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 >> 172.21.15.125:0/3768800126 conn(0x7fce2006a800 msgr2=0x7fce2015c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.687+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 shutdown_connections 2024-01-16T15:09:49.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:49.687+0000 7fce25200640 1 -- 172.21.15.125:0/3768800126 wait complete. 2024-01-16T15:09:50.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:09:43.185893+0000', 'modified': '2024-01-16T15:09:46.262939+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8181}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8181': {'gid': 8181, 'name': 'c', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/589280594', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6836', 'nonce': 589280594}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 589280594}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:09:50.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:09:50.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:09:50.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.323+0000 7f501d000640 1 Processor -- start 2024-01-16T15:09:50.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.323+0000 7f501d000640 1 -- start start 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f501d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50180cf9a0 0x7f5018070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f501d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 0x7f50180cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f501d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018071690 con 0x7f501814b2c0 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f501d000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501814b6a0 con 0x7f50180cf9a0 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f501d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5018071970 con 0x7f5018070ea0 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f5015e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f501814b2c0 0x7f50180cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39560/0 (socket says 172.21.15.125:39560) 2024-01-16T15:09:50.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.325+0000 7f5015e00640 1 -- 172.21.15.125:0/1858540155 learned_addr learned my addr 172.21.15.125:0/1858540155 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5016800640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 0x7f50180cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5015400640 1 --2- 172.21.15.125:0/1858540155 >> v2:172.21.15.137:3300/0 conn(0x7f50180cf9a0 0x7f5018070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57576511 0 0) 0x7f5018071690 con 0x7f501814b2c0 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003520 con 0x7f501814b2c0 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153665389 0 0) 0x7f4ff8003520 con 0x7f501814b2c0 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 msgr2=0x7f50180cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 0x7f50180cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 >> v2:172.21.15.137:3300/0 conn(0x7f50180cf9a0 msgr2=0x7f5018070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 --2- 172.21.15.125:0/1858540155 >> v2:172.21.15.137:3300/0 conn(0x7f50180cf9a0 0x7f5018070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5018071c50 con 0x7f501814b2c0 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5010001670 con 0x7f501814b2c0 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 >> v1:172.21.15.125:6789/0 conn(0x7f501814b2c0 legacy=0x7f50180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ff8003970 0x7f4ff8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 0x7f4ff80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f4ff8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501814b6a0 con 0x7f4ff8004d00 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018071970 con 0x7f4ff8003970 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.326+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff800b980 con 0x7f5018070ea0 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f4ff8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5015e00640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ff8003970 0x7f4ff8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 -- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 msgr2=0x7f4ff80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5018070ea0 0x7f4ff80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 -- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ff8003970 msgr2=0x7f4ff8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ff8003970 0x7f4ff8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.327+0000 7f5016800640 1 -- 172.21.15.125:0/1858540155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8003520 con 0x7f4ff8004d00 2024-01-16T15:09:50.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.328+0000 7f5016800640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f4ff8005180 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f5008009fc0 tx=0x7f4ff80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe0edf930cd941a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500802cc60 con 0x7f4ff8004d00 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500800f040 con 0x7f4ff8004d00 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f5014a00640 1 -- 172.21.15.125:0/1858540155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008034440 con 0x7f4ff8004d00 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 -- 172.21.15.125:0/1858540155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 msgr2=0x7f4ff8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 --2- 172.21.15.125:0/1858540155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f4ff8005180 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f5008009fc0 tx=0x7f4ff80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f5016800640 1 -- 172.21.15.125:0/1858540155 reap_dead start 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 -- 172.21.15.125:0/1858540155 shutdown_connections 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 -- 172.21.15.125:0/1858540155 >> 172.21.15.125:0/1858540155 conn(0x7f501806a800 msgr2=0x7f501806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 -- 172.21.15.125:0/1858540155 shutdown_connections 2024-01-16T15:09:50.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.329+0000 7f501d000640 1 -- 172.21.15.125:0/1858540155 wait complete. 2024-01-16T15:09:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.331+0000 7f501d000640 1 Processor -- start 2024-01-16T15:09:50.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.331+0000 7f501d000640 1 -- start start 2024-01-16T15:09:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 0x7f5018147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 0x7f501815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.311 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501815d3b0 con 0x7f4ff8004d00 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181481a0 con 0x7f50181478d0 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f501d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5018148480 con 0x7f501813ef90 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f5015e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f5015e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39460/0 (socket says 172.21.15.125:39460) 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.332+0000 7f5015e00640 1 -- 172.21.15.125:0/3007836615 learned_addr learned my addr 172.21.15.125:0/3007836615 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5016800640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 0x7f501815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015400640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 0x7f5018147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 -- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 msgr2=0x7f5018147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 0x7f5018147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 -- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 msgr2=0x7f501815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 0x7f501815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 -- 172.21.15.125:0/3007836615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501815d040 con 0x7f4ff8004d00 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015400640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 0x7f5018147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:50.312 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5016800640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 0x7f501815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f5015e00640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f501000d8e0 tx=0x7f501000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f500e000640 1 -- 172.21.15.125:0/3007836615 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100074e0 con 0x7f4ff8004d00 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f500e000640 1 -- 172.21.15.125:0/3007836615 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5010007640 con 0x7f4ff8004d00 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501813f410 con 0x7f4ff8004d00 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.333+0000 7f500e000640 1 -- 172.21.15.125:0/3007836615 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010004a20 con 0x7f4ff8004d00 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.334+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5018147f30 con 0x7f4ff8004d00 2024-01-16T15:09:50.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.336+0000 7f500e000640 1 -- 172.21.15.125:0/3007836615 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5010004b80 con 0x7f4ff8004d00 2024-01-16T15:09:50.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.337+0000 7f500e000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 0x7f4ff006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.337+0000 7f500e000640 1 -- 172.21.15.125:0/3007836615 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5010088410 con 0x7f4ff8004d00 2024-01-16T15:09:50.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.337+0000 7f5015400640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 0x7f4ff006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.338+0000 7f501fba0740 1 Processor -- start 2024-01-16T15:09:50.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.338+0000 7f501fba0740 1 -- start start 2024-01-16T15:09:50.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.338+0000 7f5015400640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 0x7f4ff006e3e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5018148760 tx=0x7f5000008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad210280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad20f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad16e880 con 0x562dad20f5e0 2024-01-16T15:09:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad17dae0 con 0x562dad20aba0 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dad17b1d0 con 0x562dad20fea0 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f5016800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f5016800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39470/0 (socket says 172.21.15.125:39470) 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f5016800640 1 -- 172.21.15.125:0/196143707 learned_addr learned my addr 172.21.15.125:0/196143707 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.339+0000 7f5015400640 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad20f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5015e00640 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad210280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 -- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 msgr2=0x562dad210280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad210280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 -- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 msgr2=0x562dad20f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad20f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 -- 172.21.15.125:0/196143707 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562dad13e860 con 0x562dad20f5e0 2024-01-16T15:09:50.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5016800640 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f500800e200 tx=0x7f500802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1151492199eeaf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5006a00640 1 -- 172.21.15.125:0/196143707 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500802cd20 con 0x562dad20f5e0 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5006a00640 1 -- 172.21.15.125:0/196143707 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500802ce80 con 0x562dad20f5e0 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.340+0000 7f5006a00640 1 -- 172.21.15.125:0/196143707 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008003bd0 con 0x562dad20f5e0 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 -- 172.21.15.125:0/196143707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 msgr2=0x562dad253560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f500800e200 tx=0x7f500802e070 comp rx=0 tx=0).stop 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 -- 172.21.15.125:0/196143707 shutdown_connections 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad253560 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad20f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 --2- 172.21.15.125:0/196143707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad210280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 -- 172.21.15.125:0/196143707 >> 172.21.15.125:0/196143707 conn(0x562dad1d1c70 msgr2=0x562dad255990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 -- 172.21.15.125:0/196143707 shutdown_connections 2024-01-16T15:09:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.341+0000 7f501fba0740 1 -- 172.21.15.125:0/196143707 wait complete. 2024-01-16T15:09:50.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.342+0000 7f501fba0740 1 -- start start 2024-01-16T15:09:50.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.342+0000 7f501fba0740 1 Processor -- start 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad23e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad231150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad176ac0 con 0x562dad20f5e0 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad1132f0 con 0x562dad20fea0 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dad130c70 con 0x562dad20aba0 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f501fba0740 10 client.0 _ll_register_callbacks cb 0x562dad1353b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39486/0 (socket says 172.21.15.125:39486) 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f5015400640 1 -- 172.21.15.125:0/2685075628 learned_addr learned my addr 172.21.15.125:0/2685075628 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f5015400640 1 -- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 msgr2=0x562dad23e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.343+0000 7f5015e00640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad23e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f5015400640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad23e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f5015400640 1 -- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 msgr2=0x562dad231150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f5015400640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad231150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f5015400640 1 -- 172.21.15.125:0/2685075628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff8003520 con 0x562dad20f5e0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f5015400640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5000015700 tx=0x7f50000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe9bdf3adc83699 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f4fff400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000004020 con 0x562dad20f5e0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5000004180 con 0x562dad20f5e0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562dad16f7c0 con 0x562dad20f5e0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5000004340 con 0x562dad20f5e0 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f501fba0740 10 client.8322 fetch_fsmap learned FSMap version 54 2024-01-16T15:09:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.344+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x562dad175140 con 0x562dad20f5e0 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562dad16f7c0 con 0x562dad20f5e0 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 54) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5000016930 con 0x562dad20f5e0 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 10 client.8322 fetch_fsmap finished waiting for FSMap version 54 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 10 client.8322 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 20 client.8322 populate_metadata read hostname 'smithi125' 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad22f1e0 0x562dad22fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 4 client.8322 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 --> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562dace776c0 con 0x562dad22f1e0 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f5015e00640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad22f1e0 0x562dad22fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f4fff400640 10 client.8322 ms_handle_connect on v2:172.21.15.125:6836/589280594 2024-01-16T15:09:50.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.345+0000 7f5015e00640 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad22f1e0 0x562dad22fb90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=602d1ddd635463c7 server_cookie=51a27f70c0310400 in_seq=0 out_seq=0 2024-01-16T15:09:50.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.346+0000 7f4fff400640 1 -- 172.21.15.125:0/2685075628 <== mds.0 v2:172.21.15.125:6836/589280594 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562dace776c0 con 0x562dad22f1e0 2024-01-16T15:09:50.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.346+0000 7f4fff400640 10 client.8322 handle_command_reply: tid=0 2024-01-16T15:09:50.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.346+0000 7f501fba0740 1 client.8322 shutdown 2024-01-16T15:09:50.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 msgr2=0x562dad22bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5000015700 tx=0x7f50000046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 shutdown_connections 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad22f1e0 0x562dad22fb90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad231150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22bb10 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 --2- 172.21.15.125:0/2685075628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad23e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f4fff400640 0 client.8322 ms_handle_reset on v2:172.21.15.125:6836/589280594 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 >> 172.21.15.125:0/2685075628 conn(0x562dad1d1c70 msgr2=0x562dad252b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 shutdown_connections 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 1 -- 172.21.15.125:0/2685075628 wait complete. 2024-01-16T15:09:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.347+0000 7f501fba0740 20 client.8322 trim_cache size 0 max 16384 2024-01-16T15:09:50.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.363+0000 7f501fba0740 1 Processor -- start 2024-01-16T15:09:50.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.363+0000 7f501fba0740 1 -- start start 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad2378b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x562daced9f20 con 0x562dad20f5e0 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad20d9b0 con 0x562dad20aba0 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dad254580 con 0x562dad20fea0 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39500/0 (socket says 172.21.15.125:39500) 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.364+0000 7f5015400640 1 -- 172.21.15.125:0/3701690433 learned_addr learned my addr 172.21.15.125:0/3701690433 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5016800640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015e00640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad2378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 -- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 msgr2=0x562dad234930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 -- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 msgr2=0x562dad2378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad2378b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 -- 172.21.15.125:0/3701690433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562dad112b80 con 0x562dad20f5e0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5016800640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5015400640 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5000015fd0 tx=0x7f5000021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149fcb0a3b176758 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5007400640 1 -- 172.21.15.125:0/3701690433 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000016420 con 0x562dad20f5e0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5007400640 1 -- 172.21.15.125:0/3701690433 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000016e40 con 0x562dad20f5e0 2024-01-16T15:09:50.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.365+0000 7f5007400640 1 -- 172.21.15.125:0/3701690433 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500003b450 con 0x562dad20f5e0 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 -- 172.21.15.125:0/3701690433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 msgr2=0x562dad20b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5000015fd0 tx=0x7f5000021000 comp rx=0 tx=0).stop 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 -- 172.21.15.125:0/3701690433 shutdown_connections 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad20b6e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 --2- 172.21.15.125:0/3701690433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20aba0 0x562dad2378b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 -- 172.21.15.125:0/3701690433 >> 172.21.15.125:0/3701690433 conn(0x562dad1d1c70 msgr2=0x562dad2062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 -- 172.21.15.125:0/3701690433 shutdown_connections 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.366+0000 7f501fba0740 1 -- 172.21.15.125:0/3701690433 wait complete. 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.367+0000 7f501fba0740 1 -- start start 2024-01-16T15:09:50.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.367+0000 7f501fba0740 1 Processor -- start 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad236850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad264580 con 0x562dad20f5e0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dad237340 con 0x562dad20fea0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dad2643d0 con 0x562dad20aba0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f501fba0740 10 client.0 _ll_register_callbacks cb 0x7f50000097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39512/0 (socket says 172.21.15.125:39512) 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 -- 172.21.15.125:0/2620339553 learned_addr learned my addr 172.21.15.125:0/2620339553 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015e00640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad236850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 -- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 msgr2=0x562dad236850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad236850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 -- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 msgr2=0x562dad234930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.368+0000 7f5015400640 1 -- 172.21.15.125:0/2620339553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562dad175140 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f5015400640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5000011010 tx=0x7f5000015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a840dbedd35651f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f4fff400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500003b920 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562dad16faa0 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f500003ba80 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f500003bc40 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f501fba0740 10 client.8328 fetch_fsmap learned FSMap version 54 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x562dad138170 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.369+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562dad16faa0 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 54) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5000039070 con 0x562dad20f5e0 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 10 client.8328 fetch_fsmap finished waiting for FSMap version 54 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 10 client.8328 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 20 client.8328 populate_metadata read hostname 'smithi125' 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad2b3540 0x562dad234e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 4 client.8328 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 --> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562dacecad10 con 0x562dad2b3540 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f5015e00640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad2b3540 0x562dad234e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f5015e00640 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad2b3540 0x562dad234e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c0d53b7edb615b server_cookie=d9435099bfb789dd in_seq=0 out_seq=0 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f4fff400640 10 client.8328 ms_handle_connect on v2:172.21.15.125:6836/589280594 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f4fff400640 1 -- 172.21.15.125:0/2620339553 <== mds.0 v2:172.21.15.125:6836/589280594 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562dacecad10 con 0x562dad2b3540 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f4fff400640 10 client.8328 handle_command_reply: tid=0 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.370+0000 7f501fba0740 1 client.8328 shutdown 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 msgr2=0x562dad22a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5000011010 tx=0x7f5000015000 comp rx=0 tx=0).stop 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 shutdown_connections 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:6836/589280594,v1:172.21.15.125:6837/589280594] conn(0x562dad2b3540 0x562dad234e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x562dad20fea0 0x562dad234930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f4fff400640 0 client.8328 ms_handle_reset on v2:172.21.15.125:6836/589280594 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x562dad20f5e0 0x562dad22a400 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 --2- 172.21.15.125:0/2620339553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x562dad20aba0 0x562dad236850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 >> 172.21.15.125:0/2620339553 conn(0x562dad1d1c70 msgr2=0x562dad209b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 shutdown_connections 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 1 -- 172.21.15.125:0/2620339553 wait complete. 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501fba0740 20 client.8328 trim_cache size 0 max 16384 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:09:50.353 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 8181, 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.6071823089999997, 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 54, 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 86, 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 86, 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 13.065669743000001 2024-01-16T15:09:50.354 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 msgr2=0x7f4ff006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 0x7f4ff006e3e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5018148760 tx=0x7f5000008040 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.371+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 msgr2=0x7f50181609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f501000d8e0 tx=0x7f501000dda0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 shutdown_connections 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50181478d0 0x7f501815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f501813ef90 0x7f5018147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ff006bf20 0x7f4ff006e3e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 --2- 172.21.15.125:0/3007836615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ff8004d00 0x7f50181609f0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 >> 172.21.15.125:0/3007836615 conn(0x7f501806a800 msgr2=0x7f50180b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 shutdown_connections 2024-01-16T15:09:50.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.372+0000 7f501d000640 1 -- 172.21.15.125:0/3007836615 wait complete. 2024-01-16T15:09:50.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:09:50.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.578+0000 7f83aa200640 1 Processor -- start 2024-01-16T15:09:50.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.578+0000 7f83aa200640 1 -- start start 2024-01-16T15:09:50.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.579+0000 7f83aa200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f83a40b4280 0x7f83a40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.579+0000 7f83aa200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83a40ce420 0x7f83a40ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.579+0000 7f83aa200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a4071ae0 con 0x7f83a40b3c10 2024-01-16T15:09:50.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.579+0000 7f83aa200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a40cdf90 con 0x7f83a40ce420 2024-01-16T15:09:50.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.579+0000 7f83aa200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a40b4ba0 con 0x7f83a40b4280 2024-01-16T15:09:50.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83a40ce420 0x7f83a40ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f83a40b3c10 0x7f83a40713d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39570/0 (socket says 172.21.15.125:39570) 2024-01-16T15:09:50.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a2a00640 1 -- 172.21.15.125:0/3245750311 learned_addr learned my addr 172.21.15.125:0/3245750311 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:50.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3400640 1 --2- 172.21.15.125:0/3245750311 >> v2:172.21.15.137:3300/0 conn(0x7f83a40b4280 0x7f83a40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3e00640 1 -- 172.21.15.125:0/3245750311 >> v2:172.21.15.137:3300/0 conn(0x7f83a40b4280 msgr2=0x7f83a40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3e00640 1 --2- 172.21.15.125:0/3245750311 >> v2:172.21.15.137:3300/0 conn(0x7f83a40b4280 0x7f83a40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3e00640 1 -- 172.21.15.125:0/3245750311 >> v1:172.21.15.125:6789/0 conn(0x7f83a40b3c10 legacy=0x7f83a40713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a2000640 1 -- 172.21.15.125:0/3245750311 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156221913 0 0) 0x7f838c001490 con 0x7f83a40b3c10 2024-01-16T15:09:50.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.580+0000 7f83a3e00640 1 -- 172.21.15.125:0/3245750311 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4072000 con 0x7f83a40ce420 2024-01-16T15:09:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.581+0000 7f83a3e00640 1 --2- 172.21.15.125:0/3245750311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40ce420 0x7f83a40ce8d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f83a4073fe0 tx=0x7f839402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8814afd56bf76b58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83a2000640 1 -- 172.21.15.125:0/3245750311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394031830 con 0x7f83a40ce420 2024-01-16T15:09:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83a2000640 1 -- 172.21.15.125:0/3245750311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8394031990 con 0x7f83a40ce420 2024-01-16T15:09:50.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83a2000640 1 -- 172.21.15.125:0/3245750311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394031d20 con 0x7f83a40ce420 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 -- 172.21.15.125:0/3245750311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40ce420 msgr2=0x7f83a40ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 --2- 172.21.15.125:0/3245750311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40ce420 0x7f83a40ce8d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f83a4073fe0 tx=0x7f839402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 -- 172.21.15.125:0/3245750311 shutdown_connections 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 --2- 172.21.15.125:0/3245750311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40ce420 0x7f83a40ce8d0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 --2- 172.21.15.125:0/3245750311 >> v2:172.21.15.137:3300/0 conn(0x7f83a40b4280 0x7f83a40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 -- 172.21.15.125:0/3245750311 >> 172.21.15.125:0/3245750311 conn(0x7f83a40cfef0 msgr2=0x7f83a40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.582+0000 7f83aa200640 1 -- 172.21.15.125:0/3245750311 shutdown_connections 2024-01-16T15:09:50.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.583+0000 7f83aa200640 1 -- 172.21.15.125:0/3245750311 wait complete. 2024-01-16T15:09:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.584+0000 7f83aa200640 1 Processor -- start 2024-01-16T15:09:50.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.584+0000 7f83aa200640 1 -- start start 2024-01-16T15:09:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83a40b4280 0x7f83a4140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83a40ce420 0x7f83a4157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4136e20 con 0x7f83a40b3c10 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a415bd50 con 0x7f83a40b4280 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83aa200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41f6e90 con 0x7f83a40ce420 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83a3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39528/0 (socket says 172.21.15.125:39528) 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.585+0000 7f83a3400640 1 -- 172.21.15.125:0/2422258988 learned_addr learned my addr 172.21.15.125:0/2422258988 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a2a00640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83a40b4280 0x7f83a4140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 -- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83a40ce420 msgr2=0x7f83a4157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83a40ce420 0x7f83a4157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 -- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83a40b4280 msgr2=0x7f83a4140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83a40b4280 0x7f83a4140290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 -- 172.21.15.125:0/2422258988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394009400 con 0x7f83a40b3c10 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f83a3400640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c820 tx=0x7f839c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c004020 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c004180 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.586+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c007830 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.587+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a41f7170 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.587+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a41f7730 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.588+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a40675d0 con 0x7f83a40b3c10 2024-01-16T15:09:50.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.589+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f839c007990 con 0x7f83a40b3c10 2024-01-16T15:09:50.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.590+0000 7f839b400640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 0x7f837c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.590+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f839c0884f0 con 0x7f83a40b3c10 2024-01-16T15:09:50.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.590+0000 7f83a2a00640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 0x7f837c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.591+0000 7f83a2a00640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 0x7f837c06e3e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f83a4167ee0 tx=0x7f838c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.594+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f839c054b90 con 0x7f83a40b3c10 2024-01-16T15:09:50.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.826+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f83a4141980 con 0x7f83a40b3c10 2024-01-16T15:09:50.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.827+0000 7f839b400640 1 -- 172.21.15.125:0/2422258988 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f839c0581b0 con 0x7f83a40b3c10 2024-01-16T15:09:50.805 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:09:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 msgr2=0x7f837c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 0x7f837c06e3e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f83a4167ee0 tx=0x7f838c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 msgr2=0x7f83a4135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c820 tx=0x7f839c00cce0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 shutdown_connections 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83a40ce420 0x7f83a4157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83a40b4280 0x7f83a4140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f837c06bf20 0x7f837c06e3e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 --2- 172.21.15.125:0/2422258988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a40b3c10 0x7f83a4135b20 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.829+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 >> 172.21.15.125:0/2422258988 conn(0x7f83a40cfef0 msgr2=0x7f83a4140f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.830+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 shutdown_connections 2024-01-16T15:09:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:50.830+0000 7f83aa200640 1 -- 172.21.15.125:0/2422258988 wait complete. 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:50.824 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:09:50.825 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:09:50.825 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:09:50.825 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:09:50.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:50.825 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:09:50.871 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:50.872 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 41858sec preferred_lft 41858sec 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:09:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.874 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:09:50.874 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:09:50.874 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:09:50.874 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:50.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:50.875 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:09:50.960 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:09:50.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:50.964 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:09:51.017 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:09:51.017 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:09:51.017 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:09:51.017 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:09:51.017 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:09:51.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:09:51.018 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:09:51.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:51.196 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:09:51.247 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:09:51.247 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:09:51.248 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:09:51.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:51.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:09:51.349 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:09:51.349 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:09:51.349 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:51.402 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:09:51.402 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:09:51.402 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:51.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:09:51.539 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:09:51.595 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:09:51.595 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:09:51.595 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:09:51.595 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:09:51.595 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:09:51.596 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:09:51.597 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:09:51.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:51.598 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:09:51.650 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:09:51.651 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:09:51.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:09:51.730 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:09:51.773 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:09:51.774 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:09:51.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:51.776 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:09:51.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:09:51.860+0000 7f5b1e960480 -1 init, newargv = 0x563db4b9c9a0 newargc=15 2024-01-16T15:09:51.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[70763]: starting ceph client 2024-01-16T15:09:51.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[70763]: starting fuse 2024-01-16T15:09:51.862 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:09:51.862 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:09:51.862 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:09:51.863 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:09:51.864 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:09:51.864 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:09:51.864 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:09:52.059 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.70763.asok 2024-01-16T15:09:52.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:09:52.077 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.70763.asok 2024-01-16T15:09:52.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:52.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.70763.asok status 2024-01-16T15:09:52.308 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "70763", 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:09:52.309 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 8346, 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 8346 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1526714220 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:09:52.310 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1526714220 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.8346 192.168.0.1:0/1526714220", 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/1526714220", 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 54, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 86, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:09:52.311 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:09:52.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:52.323 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:09:52.323 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:52.377 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:09:52.378 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:52.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:52.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:52.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:09:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.712+0000 7f347be00640 1 Processor -- start 2024-01-16T15:09:52.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.712+0000 7f347be00640 1 -- start start 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f347be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34740cd960 0x7f34740cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f347be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3474162f30 0x7f3474165320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f347be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740ce2a0 con 0x7f347415b2a0 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f347be00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34740d00a0 con 0x7f34740cd960 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f347be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34740b4010 con 0x7f3474162f30 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.713+0000 7f3478a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34740cd960 0x7f34740cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34740cd960 0x7f34740cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39536/0 (socket says 172.21.15.125:39536) 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 -- 172.21.15.125:0/3565051628 learned_addr learned my addr 172.21.15.125:0/3565051628 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 -- 172.21.15.125:0/3565051628 >> v2:172.21.15.137:3300/0 conn(0x7f3474162f30 msgr2=0x7f3474165320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3479e00640 1 --2- 172.21.15.125:0/3565051628 >> v2:172.21.15.137:3300/0 conn(0x7f3474162f30 0x7f3474165320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 --2- 172.21.15.125:0/3565051628 >> v2:172.21.15.137:3300/0 conn(0x7f3474162f30 0x7f3474165320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 -- 172.21.15.125:0/3565051628 >> v1:172.21.15.137:6789/0 conn(0x7f347415b2a0 legacy=0x7f34740cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.714+0000 7f3478a00640 1 -- 172.21.15.125:0/3565051628 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740b42f0 con 0x7f34740cd960 2024-01-16T15:09:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.715+0000 7f3478a00640 1 --2- 172.21.15.125:0/3565051628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34740cd960 0x7f34740cdd60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3474071f20 tx=0x7f345c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcdee70928c2889 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.715+0000 7f3473e00640 1 -- 172.21.15.125:0/3565051628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c031870 con 0x7f34740cd960 2024-01-16T15:09:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.715+0000 7f3473e00640 1 -- 172.21.15.125:0/3565051628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c0319d0 con 0x7f34740cd960 2024-01-16T15:09:52.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.715+0000 7f3473e00640 1 -- 172.21.15.125:0/3565051628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c031d60 con 0x7f34740cd960 2024-01-16T15:09:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 -- 172.21.15.125:0/3565051628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34740cd960 msgr2=0x7f34740cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 --2- 172.21.15.125:0/3565051628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34740cd960 0x7f34740cdd60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3474071f20 tx=0x7f345c02b910 comp rx=0 tx=0).stop 2024-01-16T15:09:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 -- 172.21.15.125:0/3565051628 shutdown_connections 2024-01-16T15:09:52.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 --2- 172.21.15.125:0/3565051628 >> v2:172.21.15.137:3300/0 conn(0x7f3474162f30 0x7f3474165320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 --2- 172.21.15.125:0/3565051628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34740cd960 0x7f34740cdd60 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 -- 172.21.15.125:0/3565051628 >> 172.21.15.125:0/3565051628 conn(0x7f34740d1020 msgr2=0x7f34740d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 -- 172.21.15.125:0/3565051628 shutdown_connections 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.716+0000 7f347be00640 1 -- 172.21.15.125:0/3565051628 wait complete. 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.718+0000 7f347be00640 1 Processor -- start 2024-01-16T15:09:52.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.718+0000 7f347be00640 1 -- start start 2024-01-16T15:09:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34740cd960 0x7f3474158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f347415b2a0 0x7f347415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34741597e0 con 0x7f3474162f30 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34740ca780 con 0x7f34740cd960 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f347be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34740c9e50 con 0x7f347415b2a0 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f3479e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f3479e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39542/0 (socket says 172.21.15.125:39542) 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.719+0000 7f3479e00640 1 -- 172.21.15.125:0/2135557502 learned_addr learned my addr 172.21.15.125:0/2135557502 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479400640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34740cd960 0x7f3474158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 -- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f347415b2a0 msgr2=0x7f347415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f347415b2a0 0x7f347415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 -- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34740cd960 msgr2=0x7f3474158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34740cd960 0x7f3474158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 -- 172.21.15.125:0/2135557502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c0094d0 con 0x7f3474162f30 2024-01-16T15:09:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3479e00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f346c00ccb0 tx=0x7f346c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c00ae80 con 0x7f3474162f30 2024-01-16T15:09:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c003990 con 0x7f3474162f30 2024-01-16T15:09:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34740ca130 con 0x7f3474162f30 2024-01-16T15:09:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c003d40 con 0x7f3474162f30 2024-01-16T15:09:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.720+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3474158c70 con 0x7f3474162f30 2024-01-16T15:09:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.721+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34740ca910 con 0x7f3474162f30 2024-01-16T15:09:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.723+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f346c003ea0 con 0x7f3474162f30 2024-01-16T15:09:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.724+0000 7f3471600640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 0x7f344c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.724+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f346c088b80 con 0x7f3474162f30 2024-01-16T15:09:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.724+0000 7f3479400640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 0x7f344c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.725+0000 7f3479400640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 0x7f344c06e3e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f347415e6f0 tx=0x7f3464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.729+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f346c055220 con 0x7f3474162f30 2024-01-16T15:09:52.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.962+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34740ceb50 con 0x7f3474162f30 2024-01-16T15:09:52.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.962+0000 7f3471600640 1 -- 172.21.15.125:0/2135557502 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f346c01d020 con 0x7f3474162f30 2024-01-16T15:09:52.940 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:09:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 msgr2=0x7f344c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 0x7f344c06e3e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f347415e6f0 tx=0x7f3464009040 comp rx=0 tx=0).stop 2024-01-16T15:09:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 msgr2=0x7f347415fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f346c00ccb0 tx=0x7f346c00a5c0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 shutdown_connections 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f344c06bf20 0x7f344c06e3e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3474162f30 0x7f347415fb00 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f347415b2a0 0x7f347415f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 --2- 172.21.15.125:0/2135557502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34740cd960 0x7f3474158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 >> 172.21.15.125:0/2135557502 conn(0x7f34740d1020 msgr2=0x7f347415a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 shutdown_connections 2024-01-16T15:09:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:52.965+0000 7f347be00640 1 -- 172.21.15.125:0/2135557502 wait complete. 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:09:52.962 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:09:52.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:52.962 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:09:53.020 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:09:53.020 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:09:53.021 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41845sec preferred_lft 41845sec 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.022 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:09:53.023 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:09:53.023 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:09:53.023 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:09:53.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.024 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:09:53.110 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:09:53.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.114 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:09:53.166 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:09:53.166 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:09:53.166 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:09:53.166 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:09:53.166 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:09:53.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:09:53.168 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:09:53.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.351 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:09:53.400 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:09:53.400 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:09:53.401 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:09:53.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:09:53.486 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:09:53.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:09:53.486 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:53.536 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:09:53.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:09:53.536 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:53.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:09:53.678 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:09:53.731 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:09:53.732 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:09:53.733 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:09:53.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.734 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:09:53.785 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:09:53.785 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:09:53.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:09:53.869 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:09:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:09:53.913 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:09:53.914 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:09:53.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:53.915 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:09:53.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:09:54.005+0000 7f5620e36480 -1 init, newargv = 0x5616f823a9e0 newargc=15 2024-01-16T15:09:53.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45904]: starting ceph client 2024-01-16T15:09:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45904]: starting fuse 2024-01-16T15:09:54.001 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:09:54.001 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:09:54.002 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:09:54.003 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:09:54.200 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.45904.asok 2024-01-16T15:09:54.205 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:09:54.217 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45904.asok 2024-01-16T15:09:54.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:54.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45904.asok status 2024-01-16T15:09:54.449 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:09:54.449 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "45904", 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 6295, 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:09:54.450 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 6295 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 149743074 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 149743074 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.6295 192.168.0.1:0/149743074", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/149743074", 2024-01-16T15:09:54.451 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 54, 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 86, 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:09:54.452 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:09:54.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:54.464 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:09:54.464 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:54.517 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:09:54.517 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:54.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:54.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:54.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:09:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.851+0000 7f7782c00640 1 Processor -- start 2024-01-16T15:09:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.852+0000 7f7782c00640 1 -- start start 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7782c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f777c15d3c0 0x7f777c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7782c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f777c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7782c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0b2c70 con 0x7f777c0cfa40 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7782c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777c0d03f0 con 0x7f777c15d3c0 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7782c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0cf6d0 con 0x7f777c0b4110 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7780e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f777c0cfa40 0x7f777c0cde90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39572/0 (socket says 172.21.15.125:39572) 2024-01-16T15:09:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f7780e00640 1 -- 172.21.15.125:0/1597073698 learned_addr learned my addr 172.21.15.125:0/1597073698 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.853+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262118115 0 0) 0x7f777c0b2c70 con 0x7f777c0cfa40 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c003520 con 0x7f777c0cfa40 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f7782200640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f777c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> v2:172.21.15.137:3300/0 conn(0x7f777c15d3c0 0x7f777c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379041057 0 0) 0x7f775c003520 con 0x7f777c0cfa40 2024-01-16T15:09:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 msgr2=0x7f777c0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f777c0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 >> v2:172.21.15.137:3300/0 conn(0x7f777c15d3c0 msgr2=0x7f777c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 --2- 172.21.15.125:0/1597073698 >> v2:172.21.15.137:3300/0 conn(0x7f777c15d3c0 0x7f777c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0cf3f0 con 0x7f777c0cfa40 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7764001670 con 0x7f777c0cfa40 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f7782200640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f777c0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 >> v1:172.21.15.125:6789/0 conn(0x7f777c0cfa40 legacy=0x7f777c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f775c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f775c0047c0 0x7f775c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f775c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0d03f0 con 0x7f775c003970 2024-01-16T15:09:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0cf6d0 con 0x7f775c0047c0 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c00bab0 con 0x7f777c0b4110 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.854+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> v2:172.21.15.137:3300/0 conn(0x7f777c15d3c0 0x7f777c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f775c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7782200640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f775c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 -- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 msgr2=0x7f775c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c0b4110 0x7f775c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 -- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f775c0047c0 msgr2=0x7f775c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f775c0047c0 0x7f775c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.855+0000 7f7781800640 1 -- 172.21.15.125:0/1597073698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c003520 con 0x7f775c003970 2024-01-16T15:09:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.856+0000 7f7781800640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f775c003d90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f776c006fd0 tx=0x7f775c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f19acada0ca5c72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.856+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c030660 con 0x7f775c003970 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.856+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c0307c0 con 0x7f775c003970 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.856+0000 7f777be00640 1 -- 172.21.15.125:0/1597073698 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c030b30 con 0x7f775c003970 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 -- 172.21.15.125:0/1597073698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 msgr2=0x7f775c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 --2- 172.21.15.125:0/1597073698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f775c003d90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f776c006fd0 tx=0x7f775c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782200640 1 -- 172.21.15.125:0/1597073698 reap_dead start 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 -- 172.21.15.125:0/1597073698 shutdown_connections 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 -- 172.21.15.125:0/1597073698 >> 172.21.15.125:0/1597073698 conn(0x7f777c0d3050 msgr2=0x7f777c0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 -- 172.21.15.125:0/1597073698 shutdown_connections 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.857+0000 7f7782c00640 1 -- 172.21.15.125:0/1597073698 wait complete. 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.859+0000 7f7782c00640 1 Processor -- start 2024-01-16T15:09:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.859+0000 7f7782c00640 1 -- start start 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c160250 0x7f777c0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c1606d0 0x7f777c0b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15f1b0 con 0x7f775c003970 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15f490 con 0x7f777c1606d0 2024-01-16T15:09:54.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7782c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15f770 con 0x7f777c160250 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7781800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7781800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39548/0 (socket says 172.21.15.125:39548) 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.860+0000 7f7781800640 1 -- 172.21.15.125:0/3415465316 learned_addr learned my addr 172.21.15.125:0/3415465316 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 -- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c160250 msgr2=0x7f777c0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c160250 0x7f777c0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 -- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c1606d0 msgr2=0x7f777c0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c1606d0 0x7f777c0b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 -- 172.21.15.125:0/3415465316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c15fa50 con 0x7f775c003970 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7781800640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f776c02bef0 tx=0x7f776c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0399e0 con 0x7f775c003970 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c03f040 con 0x7f775c003970 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c039d60 con 0x7f775c003970 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c15fd30 con 0x7f775c003970 2024-01-16T15:09:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.861+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c0cb5c0 con 0x7f775c003970 2024-01-16T15:09:54.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.863+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c08ba80 con 0x7f775c003970 2024-01-16T15:09:54.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.863+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f776c04c400 con 0x7f775c003970 2024-01-16T15:09:54.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.864+0000 7f7779600640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 0x7f775406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.864+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f776c0aaaf0 con 0x7f775c003970 2024-01-16T15:09:54.841 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.865+0000 7f7780e00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 0x7f775406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.865+0000 7f7780e00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 0x7f775406e410 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f777c15eda0 tx=0x7f77640014e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:54.869+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f776c0773b0 con 0x7f775c003970 2024-01-16T15:09:55.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.099+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f777c067450 con 0x7f775c003970 2024-01-16T15:09:55.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.857+0000 7f7779600640 1 -- 172.21.15.125:0/3415465316 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 136+0+0 (secure 0 0 0) 0x7f776c0340c0 con 0x7f775c003970 2024-01-16T15:09:55.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 msgr2=0x7f775406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 0x7f775406e410 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f777c15eda0 tx=0x7f77640014e0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 msgr2=0x7f777c156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f776c02bef0 tx=0x7f776c004730 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 shutdown_connections 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f777c1606d0 0x7f777c0b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f777c160250 0x7f777c0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f775406bf50 0x7f775406e410 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 --2- 172.21.15.125:0/3415465316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f775c003970 0x7f777c156520 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.860+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 >> 172.21.15.125:0/3415465316 conn(0x7f777c0d3050 msgr2=0x7f777c0b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.861+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 shutdown_connections 2024-01-16T15:09:55.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:55.861+0000 7f7782c00640 1 -- 172.21.15.125:0/3415465316 wait complete. 2024-01-16T15:09:55.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:09:56.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.089+0000 7faec8e00640 1 Processor -- start 2024-01-16T15:09:56.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.089+0000 7faec8e00640 1 -- start start 2024-01-16T15:09:56.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.090+0000 7faec8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 0x7faec4164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.090+0000 7faec8e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec40b1ee0 con 0x7faec40cf9a0 2024-01-16T15:09:56.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.090+0000 7faec8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec40b2780 con 0x7faec4143ae0 2024-01-16T15:09:56.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.090+0000 7faec8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec415e990 con 0x7faec415e3b0 2024-01-16T15:09:56.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.091+0000 7faeb9200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faec40cf9a0 0x7faec415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39578/0 (socket says 172.21.15.125:39578) 2024-01-16T15:09:56.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.091+0000 7faeb9200640 1 -- 172.21.15.125:0/4075381618 learned_addr learned my addr 172.21.15.125:0/4075381618 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.091+0000 7faec2800640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 0x7faec4164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.091+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194149725 0 0) 0x7faec40b1ee0 con 0x7faec40cf9a0 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.091+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea4003520 con 0x7faec40cf9a0 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047869032 0 0) 0x7faea4003520 con 0x7faec40cf9a0 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 msgr2=0x7faec4164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 0x7faec4164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 >> v1:172.21.15.137:6789/0 conn(0x7faec4143ae0 legacy=0x7faec40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec40b2130 con 0x7faec40cf9a0 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7faeac0019f0 con 0x7faec40cf9a0 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 >> v1:172.21.15.125:6789/0 conn(0x7faec40cf9a0 legacy=0x7faec415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faea4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea40047c0 0x7faea4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 0x7faea4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec415e990 con 0x7faea40039a0 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea400ba60 con 0x7faea40047c0 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea400de30 con 0x7faec415e3b0 2024-01-16T15:09:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.092+0000 7faec1e00640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faea4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.093+0000 7faec1e00640 1 -- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 msgr2=0x7faea4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.093+0000 7faec1e00640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec415e3b0 0x7faea4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.093+0000 7faec1e00640 1 -- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea40047c0 msgr2=0x7faea4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.093+0000 7faec1e00640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faea40047c0 0x7faea4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.093+0000 7faec1e00640 1 -- 172.21.15.125:0/4075381618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea4003520 con 0x7faea40039a0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec1e00640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faea4003dc0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7faeb4001720 tx=0x7faea40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9389c8c22b21d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40052d0 con 0x7faea40039a0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb402e050 con 0x7faea40039a0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec1400640 1 -- 172.21.15.125:0/4075381618 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40035f0 con 0x7faea40039a0 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec8e00640 1 -- 172.21.15.125:0/4075381618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 msgr2=0x7faea4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec8e00640 1 --2- 172.21.15.125:0/4075381618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faea4003dc0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7faeb4001720 tx=0x7faea40057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.094+0000 7faec2800640 1 -- 172.21.15.125:0/4075381618 reap_dead start 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.095+0000 7faec8e00640 1 -- 172.21.15.125:0/4075381618 shutdown_connections 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.095+0000 7faec8e00640 1 -- 172.21.15.125:0/4075381618 >> 172.21.15.125:0/4075381618 conn(0x7faec406a800 msgr2=0x7faec406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.095+0000 7faec8e00640 1 -- 172.21.15.125:0/4075381618 shutdown_connections 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.095+0000 7faec8e00640 1 -- 172.21.15.125:0/4075381618 wait complete. 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.096+0000 7faec8e00640 1 Processor -- start 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.096+0000 7faec8e00640 1 -- start start 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faec415fd50 0x7faec41601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec4158990 0x7faec4158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec4161530 con 0x7faea40039a0 2024-01-16T15:09:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec4159350 con 0x7faec415fd50 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.097+0000 7faec8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec41f3600 con 0x7faec4158990 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39570/0 (socket says 172.21.15.125:39570) 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 -- 172.21.15.125:0/3251686546 learned_addr learned my addr 172.21.15.125:0/3251686546 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faeb9200640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faec415fd50 0x7faec41601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec2800640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec4158990 0x7faec4158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 -- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec4158990 msgr2=0x7faec4158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec4158990 0x7faec4158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 -- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faec415fd50 msgr2=0x7faec41601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faec415fd50 0x7faec41601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.098+0000 7faec1e00640 1 -- 172.21.15.125:0/3251686546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec41608b0 con 0x7faea40039a0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faec1e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7faeb4008fd0 tx=0x7faeb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40076c0 con 0x7faea40039a0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb4007a30 con 0x7faea40039a0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4006be0 con 0x7faea40039a0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec4160b90 con 0x7faea40039a0 2024-01-16T15:09:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.099+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec4161150 con 0x7faea40039a0 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.100+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec406b140 con 0x7faea40039a0 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.101+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7faeb4006d40 con 0x7faea40039a0 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.102+0000 7faebaa00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 0x7fae9c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.102+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faeb40aa760 con 0x7faea40039a0 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.103+0000 7faeb9200640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 0x7fae9c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.103+0000 7faeb9200640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 0x7fae9c06e3e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faec415d130 tx=0x7faeac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.107+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faeb4077020 con 0x7faea40039a0 2024-01-16T15:09:56.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.327+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faec406c3f0 con 0x7fae9c06bf20 2024-01-16T15:09:56.306 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:09:56.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:09:56.306 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:09:56.306 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:09:56.306 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:09:56.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.329+0000 7faebaa00640 1 -- 172.21.15.125:0/3251686546 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7faec406c3f0 con 0x7fae9c06bf20 2024-01-16T15:09:56.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 msgr2=0x7fae9c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 0x7fae9c06e3e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faec415d130 tx=0x7faeac009040 comp rx=0 tx=0).stop 2024-01-16T15:09:56.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 msgr2=0x7faec413b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7faeb4008fd0 tx=0x7faeb4001120 comp rx=0 tx=0).stop 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 shutdown_connections 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faec4158990 0x7faec4158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faec415fd50 0x7faec41601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fae9c06bf20 0x7fae9c06e3e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 --2- 172.21.15.125:0/3251686546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea40039a0 0x7faec413b740 secure :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7faeb4008fd0 tx=0x7faeb4001120 comp rx=0 tx=0).stop 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.332+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 >> 172.21.15.125:0/3251686546 conn(0x7faec406a800 msgr2=0x7faec4140540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.333+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 shutdown_connections 2024-01-16T15:09:56.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.333+0000 7faec8e00640 1 -- 172.21.15.125:0/3251686546 wait complete. 2024-01-16T15:09:56.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:09:56.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.566+0000 7f42bfe00640 1 Processor -- start 2024-01-16T15:09:56.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.567+0000 7f42bfe00640 1 -- start start 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.568+0000 7f42bfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42c0143b40 0x7f42c00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.568+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c006e800 0x7f42c00cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.568+0000 7f42bfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c0143f20 con 0x7f42c00ce450 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.568+0000 7f42bfe00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c00ce880 con 0x7f42c0143b40 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.568+0000 7f42bfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c006fda0 con 0x7f42c006e800 2024-01-16T15:09:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42c0143b40 0x7f42c00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42c0143b40 0x7f42c00cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43724/0 (socket says 172.21.15.125:43724) 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 -- 172.21.15.125:0/4123272424 learned_addr learned my addr 172.21.15.125:0/4123272424 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42be000640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c006e800 0x7f42c00cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 -- 172.21.15.125:0/4123272424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c006e800 msgr2=0x7f42c00cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c006e800 0x7f42c00cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 -- 172.21.15.125:0/4123272424 >> v1:172.21.15.137:6789/0 conn(0x7f42c00ce450 legacy=0x7f42c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.569+0000 7f42bea00640 1 -- 172.21.15.125:0/4123272424 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c006ed00 con 0x7f42c0143b40 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.570+0000 7f42bea00640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c00cd4a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009e50 tx=0x7f42b002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15d47ed60c7c437 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bd600640 1 -- 172.21.15.125:0/4123272424 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002c9e0 con 0x7f42c0143b40 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bd600640 1 -- 172.21.15.125:0/4123272424 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b002cb40 con 0x7f42c0143b40 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bd600640 1 -- 172.21.15.125:0/4123272424 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b00343f0 con 0x7f42c0143b40 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 -- 172.21.15.125:0/4123272424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 msgr2=0x7f42c00cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c00cd4a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009e50 tx=0x7f42b002be40 comp rx=0 tx=0).stop 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 -- 172.21.15.125:0/4123272424 shutdown_connections 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c006e800 0x7f42c00cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 --2- 172.21.15.125:0/4123272424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c00cd4a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 -- 172.21.15.125:0/4123272424 >> 172.21.15.125:0/4123272424 conn(0x7f42c00b1d40 msgr2=0x7f42c00b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 -- 172.21.15.125:0/4123272424 shutdown_connections 2024-01-16T15:09:56.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.571+0000 7f42bfe00640 1 -- 172.21.15.125:0/4123272424 wait complete. 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.573+0000 7f42bfe00640 1 Processor -- start 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.573+0000 7f42bfe00640 1 -- start start 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 0x7f42c01e4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c00ce450 0x7f42c015c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c015c150 con 0x7f42c0143b40 2024-01-16T15:09:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c015b4d0 con 0x7f42c006e800 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c015b7b0 con 0x7f42c00ce450 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bf400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.574+0000 7f42bf400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43736/0 (socket says 172.21.15.125:43736) 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 -- 172.21.15.125:0/2984085477 learned_addr learned my addr 172.21.15.125:0/2984085477 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42be000640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c00ce450 0x7f42c015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bea00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 0x7f42c01e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 -- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c00ce450 msgr2=0x7f42c015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c00ce450 0x7f42c015c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 -- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 msgr2=0x7f42c01e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 0x7f42c01e4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 -- 172.21.15.125:0/2984085477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0009b30 con 0x7f42c0143b40 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bea00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 0x7f42c01e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42bf400640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f42b800b410 tx=0x7f42b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.575+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8004020 con 0x7f42c0143b40 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.576+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b8004180 con 0x7f42c0143b40 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.576+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8011630 con 0x7f42c0143b40 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.576+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c015ba90 con 0x7f42c0143b40 2024-01-16T15:09:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.576+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c015b0f0 con 0x7f42c0143b40 2024-01-16T15:09:56.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.577+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0130150 con 0x7f42c0143b40 2024-01-16T15:09:56.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.578+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f42b8011790 con 0x7f42c0143b40 2024-01-16T15:09:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.579+0000 7f42b7400640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 0x7f429806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.579+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42b8088570 con 0x7f42c0143b40 2024-01-16T15:09:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.579+0000 7f42bea00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 0x7f429806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.580+0000 7f42bea00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 0x7f429806e3e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f42b0035000 tx=0x7f42b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.584+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42b8054bb0 con 0x7f42c0143b40 2024-01-16T15:09:56.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.775+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f42c0067b60 con 0x7f42c0143b40 2024-01-16T15:09:56.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.776+0000 7f42b7400640 1 -- 172.21.15.125:0/2984085477 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f42b8015020 con 0x7f42c0143b40 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 msgr2=0x7f429806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 0x7f429806e3e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f42b0035000 tx=0x7f42b0030000 comp rx=0 tx=0).stop 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 msgr2=0x7f42c01f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f42b800b410 tx=0x7f42b800b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 shutdown_connections 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f429806bf20 0x7f429806e3e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42c0143b40 0x7f42c01f27f0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42c00ce450 0x7f42c015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 --2- 172.21.15.125:0/2984085477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42c006e800 0x7f42c01e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 >> 172.21.15.125:0/2984085477 conn(0x7f42c00b1d40 msgr2=0x7f42c00b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 shutdown_connections 2024-01-16T15:09:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:56.779+0000 7f42bfe00640 1 -- 172.21.15.125:0/2984085477 wait complete. 2024-01-16T15:09:56.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:09:56.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.015+0000 7f9e40800640 1 Processor -- start 2024-01-16T15:09:56.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.016+0000 7f9e40800640 1 -- start start 2024-01-16T15:09:56.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e40800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e38161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e40800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9e380ced50 0x7f9e380d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e40800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380cf1b0 con 0x7f9e380ce8f0 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e40800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e381618a0 con 0x7f9e380ced50 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e40800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380d0de0 con 0x7f9e38160f80 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e3d400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9e380ce8f0 0x7f9e380cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58100/0 (socket says 172.21.15.125:58100) 2024-01-16T15:09:56.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e3d400640 1 -- 172.21.15.125:0/1722249934 learned_addr learned my addr 172.21.15.125:0/1722249934 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e3e800640 1 --2- 172.21.15.125:0/1722249934 >> v2:172.21.15.137:3300/0 conn(0x7f9e380ced50 0x7f9e380d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e3de00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e38161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.017+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156740286 0 0) 0x7f9e380cf1b0 con 0x7f9e380ce8f0 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e18003520 con 0x7f9e380ce8f0 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2955995221 0 0) 0x7f9e18003520 con 0x7f9e380ce8f0 2024-01-16T15:09:56.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 msgr2=0x7f9e38161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e38161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 >> v2:172.21.15.137:3300/0 conn(0x7f9e380ced50 msgr2=0x7f9e380d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 --2- 172.21.15.125:0/1722249934 >> v2:172.21.15.137:3300/0 conn(0x7f9e380ced50 0x7f9e380d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380d10c0 con 0x7f9e380ce8f0 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f9e28001670 con 0x7f9e380ce8f0 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 >> v1:172.21.15.125:6789/0 conn(0x7f9e380ce8f0 legacy=0x7f9e380cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e18003970 0x7f9e18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381618a0 con 0x7f9e18003970 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380d0de0 con 0x7f9e18004d00 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1800b980 con 0x7f9e38160f80 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.018+0000 7f9e3de00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e18003970 0x7f9e18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3e800640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3d400640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3de00640 1 -- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 msgr2=0x7f9e180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3de00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3de00640 1 -- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18004d00 msgr2=0x7f9e18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3de00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3de00640 1 -- 172.21.15.125:0/1722249934 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18003520 con 0x7f9e18003970 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.019+0000 7f9e3d400640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e38160f80 0x7f9e180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.020+0000 7f9e3de00640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e18003970 0x7f9e18003d90 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009ac0 tx=0x7f9e180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfa5ea6c8d24575 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.020+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2002c6b0 con 0x7f9e18003970 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.020+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2002c810 con 0x7f9e18003970 2024-01-16T15:09:56.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.020+0000 7f9e3ca00640 1 -- 172.21.15.125:0/1722249934 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2002cba0 con 0x7f9e18003970 2024-01-16T15:09:57.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 -- 172.21.15.125:0/1722249934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e18003970 msgr2=0x7f9e18003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 --2- 172.21.15.125:0/1722249934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e18003970 0x7f9e18003d90 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009ac0 tx=0x7f9e180057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e3e800640 1 -- 172.21.15.125:0/1722249934 reap_dead start 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 -- 172.21.15.125:0/1722249934 shutdown_connections 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 -- 172.21.15.125:0/1722249934 >> 172.21.15.125:0/1722249934 conn(0x7f9e3809c550 msgr2=0x7f9e380d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 -- 172.21.15.125:0/1722249934 shutdown_connections 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.021+0000 7f9e40800640 1 -- 172.21.15.125:0/1722249934 wait complete. 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.023+0000 7f9e40800640 1 Processor -- start 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.023+0000 7f9e40800640 1 -- start start 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18003970 0x7f9e3815a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e381600b0 0x7f9e381f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38160530 con 0x7f9e381f3950 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3815df10 con 0x7f9e18003970 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e40800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3815e1f0 con 0x7f9e381600b0 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e3d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e3d400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43750/0 (socket says 172.21.15.125:43750) 2024-01-16T15:09:57.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.024+0000 7f9e3d400640 1 -- 172.21.15.125:0/2128601811 learned_addr learned my addr 172.21.15.125:0/2128601811 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3de00640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18003970 0x7f9e3815a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 -- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e381600b0 msgr2=0x7f9e381f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e381600b0 0x7f9e381f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 -- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18003970 msgr2=0x7f9e3815a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18003970 0x7f9e3815a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 -- 172.21.15.125:0/2128601811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3815e4d0 con 0x7f9e381f3950 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e3d400640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800d8b0 tx=0x7f9e2800dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280074e0 con 0x7f9e381f3950 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e28007640 con 0x7f9e381f3950 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280049e0 con 0x7f9e381f3950 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.025+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3815e7b0 con 0x7f9e381f3950 2024-01-16T15:09:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.026+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381f3dd0 con 0x7f9e381f3950 2024-01-16T15:09:57.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.027+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e380675d0 con 0x7f9e381f3950 2024-01-16T15:09:57.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.028+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9e28004b40 con 0x7f9e381f3950 2024-01-16T15:09:57.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.028+0000 7f9e36000640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 0x7f9e1006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.029+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e28088240 con 0x7f9e381f3950 2024-01-16T15:09:57.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.029+0000 7f9e3de00640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 0x7f9e1006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.029+0000 7f9e3de00640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 0x7f9e1006e410 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e20036000 tx=0x7f9e20030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.033+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e28054880 con 0x7f9e381f3950 2024-01-16T15:09:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.223+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f9e380d3920 con 0x7f9e381f3950 2024-01-16T15:09:57.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.231+0000 7f9e36000640 1 -- 172.21.15.125:0/2128601811 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v31)=0 v31) v1 ==== 131+0+0 (secure 0 0 0) 0x7f9e28019070 con 0x7f9e381f3950 2024-01-16T15:09:57.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 msgr2=0x7f9e1006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 0x7f9e1006e410 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e20036000 tx=0x7f9e20030000 comp rx=0 tx=0).stop 2024-01-16T15:09:57.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 msgr2=0x7f9e3815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800d8b0 tx=0x7f9e2800dd70 comp rx=0 tx=0).stop 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 shutdown_connections 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e381600b0 0x7f9e381f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9e1006bf50 0x7f9e1006e410 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e381f3950 0x7f9e3815fb70 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 --2- 172.21.15.125:0/2128601811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e18003970 0x7f9e3815a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 >> 172.21.15.125:0/2128601811 conn(0x7f9e3809c550 msgr2=0x7f9e380cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 shutdown_connections 2024-01-16T15:09:57.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.234+0000 7f9e40800640 1 -- 172.21.15.125:0/2128601811 wait complete. 2024-01-16T15:09:57.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:09:57.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.460+0000 7f513bc00640 1 Processor -- start 2024-01-16T15:09:57.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.460+0000 7f513bc00640 1 -- start start 2024-01-16T15:09:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.461+0000 7f513bc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5134143ae0 0x7f51340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.461+0000 7f513bc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51340cfc30 con 0x7f513415e3b0 2024-01-16T15:09:57.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f513bc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51340b1ee0 con 0x7f51340cf7d0 2024-01-16T15:09:57.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f513bc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51340b2780 con 0x7f5134143ae0 2024-01-16T15:09:57.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f5139c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f513415e3b0 0x7f51340702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58116/0 (socket says 172.21.15.125:58116) 2024-01-16T15:09:57.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f5139c00640 1 -- 172.21.15.125:0/76863810 learned_addr learned my addr 172.21.15.125:0/76863810 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:57.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228246386 0 0) 0x7f51340cfc30 con 0x7f513415e3b0 2024-01-16T15:09:57.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5114003520 con 0x7f513415e3b0 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.462+0000 7f5139200640 1 --2- 172.21.15.125:0/76863810 >> v2:172.21.15.137:3300/0 conn(0x7f5134143ae0 0x7f51340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470698360 0 0) 0x7f5114003520 con 0x7f513415e3b0 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 >> v2:172.21.15.137:3300/0 conn(0x7f5134143ae0 msgr2=0x7f51340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 --2- 172.21.15.125:0/76863810 >> v2:172.21.15.137:3300/0 conn(0x7f5134143ae0 0x7f51340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 >> v1:172.21.15.137:6789/0 conn(0x7f51340cf7d0 legacy=0x7f513415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513415e990 con 0x7f513415e3b0 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f512c0019f0 con 0x7f513415e3b0 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 >> v1:172.21.15.125:6789/0 conn(0x7f513415e3b0 legacy=0x7f51340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140039c0 0x7f5114003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 0x7f511400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340b2780 con 0x7f51140051d0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511400c120 con 0x7f51140047c0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511400c2e0 con 0x7f51140039c0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.463+0000 7f5139c00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 0x7f511400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5133e00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139200640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140039c0 0x7f5114003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139c00640 1 -- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140039c0 msgr2=0x7f5114003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139c00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140039c0 0x7f5114003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139c00640 1 -- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51140047c0 msgr2=0x7f5114004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139c00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.464+0000 7f5139c00640 1 -- 172.21.15.125:0/76863810 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114003520 con 0x7f51140051d0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f5139c00640 1 -- 172.21.15.125:0/76863810 reap_dead start 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f5139c00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 0x7f511400b980 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f512c00a750 tx=0x7f51140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27d20be9ff0bd6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c034630 con 0x7f51140051d0 2024-01-16T15:09:57.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c02f070 con 0x7f51140051d0 2024-01-16T15:09:57.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f5133400640 1 -- 172.21.15.125:0/76863810 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c034a10 con 0x7f51140051d0 2024-01-16T15:09:57.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f513bc00640 1 -- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 msgr2=0x7f511400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.465+0000 7f513bc00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 0x7f511400b980 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f512c00a750 tx=0x7f51140057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.466+0000 7f513bc00640 1 -- 172.21.15.125:0/76863810 shutdown_connections 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.466+0000 7f513bc00640 1 --2- 172.21.15.125:0/76863810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51140051d0 0x7f511400b980 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.466+0000 7f513bc00640 1 -- 172.21.15.125:0/76863810 >> 172.21.15.125:0/76863810 conn(0x7f513406a800 msgr2=0x7f513406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.466+0000 7f513bc00640 1 -- 172.21.15.125:0/76863810 shutdown_connections 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.466+0000 7f513bc00640 1 -- 172.21.15.125:0/76863810 wait complete. 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.467+0000 7f513bc00640 1 Processor -- start 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.467+0000 7f513bc00640 1 -- start start 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 0x7f51341f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5134133d90 0x7f51340ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134131660 con 0x7f51341311e0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340acb70 con 0x7f5134133d90 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.468+0000 7f513bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51340acdf0 con 0x7f51140051d0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43760/0 (socket says 172.21.15.125:43760) 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 -- 172.21.15.125:0/2404623167 learned_addr learned my addr 172.21.15.125:0/2404623167 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5133e00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5134133d90 0x7f51340ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139200640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 0x7f51341f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 -- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 msgr2=0x7f51341f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 0x7f51341f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 -- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5134133d90 msgr2=0x7f51340ab300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5134133d90 0x7f51340ab300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 -- 172.21.15.125:0/2404623167 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51340ad0d0 con 0x7f51341311e0 2024-01-16T15:09:57.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139200640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 0x7f51341f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.469+0000 7f5139c00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f512c02cb70 tx=0x7f51340b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.470+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c001930 con 0x7f51341311e0 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.470+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c004b60 con 0x7f51341311e0 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.470+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0347b0 con 0x7f51341311e0 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.470+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51340ad3b0 con 0x7f51341311e0 2024-01-16T15:09:57.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.470+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5134140000 con 0x7f51341311e0 2024-01-16T15:09:57.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.471+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51340ad8d0 con 0x7f51341311e0 2024-01-16T15:09:57.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.472+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f512c031030 con 0x7f51341311e0 2024-01-16T15:09:57.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.473+0000 7f5130c00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 0x7f510c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.473+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f512c0a9c20 con 0x7f51341311e0 2024-01-16T15:09:57.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.473+0000 7f5139200640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 0x7f510c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.474+0000 7f5139200640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 0x7f510c06e410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5124006fd0 tx=0x7f5124008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.477+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51340ad8d0 con 0x7f51341311e0 2024-01-16T15:09:57.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.667+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f51340ad8d0 con 0x7f51341311e0 2024-01-16T15:09:57.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.672+0000 7f5130c00640 1 -- 172.21.15.125:0/2404623167 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v32)=0 v32) v1 ==== 137+0+0 (secure 0 0 0) 0x7f51340ad8d0 con 0x7f51341311e0 2024-01-16T15:09:57.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.675+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 msgr2=0x7f510c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.675+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 0x7f510c06e410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5124006fd0 tx=0x7f5124008040 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.675+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 msgr2=0x7f51340702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.675+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f512c02cb70 tx=0x7f51340b26f0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 shutdown_connections 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f510c06bf50 0x7f510c06e410 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51341311e0 0x7f51340702c0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5134133d90 0x7f51340ab300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 --2- 172.21.15.125:0/2404623167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51140051d0 0x7f51341f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 >> 172.21.15.125:0/2404623167 conn(0x7f513406a800 msgr2=0x7f51340ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 shutdown_connections 2024-01-16T15:09:57.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.676+0000 7f513bc00640 1 -- 172.21.15.125:0/2404623167 wait complete. 2024-01-16T15:09:57.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:09:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.894+0000 7f5aae400640 1 Processor -- start 2024-01-16T15:09:57.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.895+0000 7f5aae400640 1 -- start start 2024-01-16T15:09:57.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.896+0000 7f5aae400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 0x7f5aa80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.896+0000 7f5aae400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5aa815f9c0 0x7f5aa80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.896+0000 7f5aae400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa815c5f0 con 0x7f5aa813df80 2024-01-16T15:09:57.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.896+0000 7f5aae400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa806c470 con 0x7f5aa815f9c0 2024-01-16T15:09:57.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.896+0000 7f5aae400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa806c750 con 0x7f5aa813e360 2024-01-16T15:09:57.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5aa813df80 0x7f5aa815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58126/0 (socket says 172.21.15.125:58126) 2024-01-16T15:09:57.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa7400640 1 -- 172.21.15.125:0/264079767 learned_addr learned my addr 172.21.15.125:0/264079767 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:57.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/264079767 >> v2:172.21.15.137:3300/0 conn(0x7f5aa815f9c0 0x7f5aa80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 0x7f5aa80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010754550 0 0) 0x7f5aa815c5f0 con 0x7f5aa813df80 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a88003520 con 0x7f5aa813df80 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090973114 0 0) 0x7f5a88003520 con 0x7f5aa813df80 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.897+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 msgr2=0x7f5aa80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 0x7f5aa80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 >> v2:172.21.15.137:3300/0 conn(0x7f5aa815f9c0 msgr2=0x7f5aa80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 --2- 172.21.15.125:0/264079767 >> v2:172.21.15.137:3300/0 conn(0x7f5aa815f9c0 0x7f5aa80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa806ca30 con 0x7f5aa813df80 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5a90001670 con 0x7f5aa813df80 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 >> v1:172.21.15.125:6789/0 conn(0x7f5aa813df80 legacy=0x7f5aa815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 0x7f5a88003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88003f40 0x7f5a880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a88004d00 0x7f5a88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa806c470 con 0x7f5a88004d00 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa806c750 con 0x7f5a88003f40 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8800ba30 con 0x7f5aa813e360 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a88004d00 0x7f5a88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88003f40 0x7f5a880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 -- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 msgr2=0x7f5a88003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa813e360 0x7f5a88003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 -- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88003f40 msgr2=0x7f5a880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88003f40 0x7f5a880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.898+0000 7f5aa7e00640 1 -- 172.21.15.125:0/264079767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a88003520 con 0x7f5a88004d00 2024-01-16T15:09:57.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.899+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88003f40 0x7f5a880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:57.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a88004d00 0x7f5a88005180 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009ac0 tx=0x7f5a880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf8acbc5648cf85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa002c710 con 0x7f5a88004d00 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa002c870 con 0x7f5a88004d00 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aa6000640 1 -- 172.21.15.125:0/264079767 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa002cc00 con 0x7f5a88004d00 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aae400640 1 -- 172.21.15.125:0/264079767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a88004d00 msgr2=0x7f5a88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aae400640 1 --2- 172.21.15.125:0/264079767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a88004d00 0x7f5a88005180 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009ac0 tx=0x7f5a880057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.900+0000 7f5aa7e00640 1 -- 172.21.15.125:0/264079767 reap_dead start 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.901+0000 7f5aae400640 1 -- 172.21.15.125:0/264079767 shutdown_connections 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.901+0000 7f5aae400640 1 -- 172.21.15.125:0/264079767 >> 172.21.15.125:0/264079767 conn(0x7f5aa806a800 msgr2=0x7f5aa806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.901+0000 7f5aae400640 1 -- 172.21.15.125:0/264079767 shutdown_connections 2024-01-16T15:09:57.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.901+0000 7f5aae400640 1 -- 172.21.15.125:0/264079767 wait complete. 2024-01-16T15:09:57.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.902+0000 7f5aae400640 1 Processor -- start 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 -- start start 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88004d00 0x7f5aa80ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa815a180 0x7f5aa815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa80ade10 con 0x7f5aa80719a0 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa815ab40 con 0x7f5a88004d00 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.903+0000 7f5aae400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa80ad190 con 0x7f5aa815a180 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43770/0 (socket says 172.21.15.125:43770) 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 -- 172.21.15.125:0/1350287051 learned_addr learned my addr 172.21.15.125:0/1350287051 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa7400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88004d00 0x7f5aa80ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa7e00640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa815a180 0x7f5aa815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 -- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa815a180 msgr2=0x7f5aa815a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa815a180 0x7f5aa815a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 -- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88004d00 msgr2=0x7f5aa80ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88004d00 0x7f5aa80ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.904+0000 7f5aa6a00640 1 -- 172.21.15.125:0/1350287051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa80ad470 con 0x7f5aa80719a0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5aa6a00640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009fd0 tx=0x7f5a9800f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98003b90 con 0x7f5aa80719a0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98003cf0 con 0x7f5aa80719a0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98004230 con 0x7f5aa80719a0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa80ad750 con 0x7f5aa80719a0 2024-01-16T15:09:57.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.905+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa80ace80 con 0x7f5aa80719a0 2024-01-16T15:09:57.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.906+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa8067000 con 0x7f5aa80719a0 2024-01-16T15:09:57.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.907+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5a98004390 con 0x7f5aa80719a0 2024-01-16T15:09:57.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.908+0000 7f5a9f400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 0x7f5a8006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.908+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a980890b0 con 0x7f5aa80719a0 2024-01-16T15:09:57.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.908+0000 7f5aa7400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 0x7f5a8006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.909+0000 7f5aa7400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 0x7f5a8006e410 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5a900017a0 tx=0x7f5a90001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:57.913+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a98055970 con 0x7f5aa80719a0 2024-01-16T15:09:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.125+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f5aa806cff0 con 0x7f5a8006bf50 2024-01-16T15:09:58.116 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:58.136+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:58.116 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:58.136+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:58.116 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:58.136+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:58.116 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:58.136+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:58.116 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:09:58.136+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:58.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.192+0000 7f5a9f400640 1 -- 172.21.15.125:0/1350287051 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5a9000e870 con 0x7f5a8006bf50 2024-01-16T15:09:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 msgr2=0x7f5a8006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 0x7f5a8006e410 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5a900017a0 tx=0x7f5a90001630 comp rx=0 tx=0).stop 2024-01-16T15:09:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 msgr2=0x7f5aa8151150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009fd0 tx=0x7f5a9800f5e0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 shutdown_connections 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aa815a180 0x7f5aa815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5a8006bf50 0x7f5a8006e410 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aa80719a0 0x7f5aa8151150 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 --2- 172.21.15.125:0/1350287051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a88004d00 0x7f5aa80ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 >> 172.21.15.125:0/1350287051 conn(0x7f5aa806a800 msgr2=0x7f5aa80cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 shutdown_connections 2024-01-16T15:09:58.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.195+0000 7f5aae400640 1 -- 172.21.15.125:0/1350287051 wait complete. 2024-01-16T15:09:58.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:09:58.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.407+0000 7fd5aee00640 1 Processor -- start 2024-01-16T15:09:58.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.408+0000 7fd5aee00640 1 -- start start 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.409+0000 7fd5aee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 0x7fd5a8166250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.409+0000 7fd5aee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80c6060 con 0x7fd5a815c300 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.409+0000 7fd5aee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80cf710 con 0x7fd5a8159040 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.409+0000 7fd5aee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80c62d0 con 0x7fd5a80b34e0 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd5a815c300 0x7fd5a815f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58128/0 (socket says 172.21.15.125:58128) 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a7400640 1 -- 172.21.15.125:0/3787009588 learned_addr learned my addr 172.21.15.125:0/3787009588 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5ace00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 0x7fd5a8166250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014482575 0 0) 0x7fd5a80c6060 con 0x7fd5a815c300 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd588003520 con 0x7fd5a815c300 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 444769534 0 0) 0x7fd588003520 con 0x7fd5a815c300 2024-01-16T15:09:58.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 msgr2=0x7fd5a8166250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 0x7fd5a8166250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 >> v1:172.21.15.137:6789/0 conn(0x7fd5a8159040 legacy=0x7fd5a80cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80c65b0 con 0x7fd5a815c300 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fd590001670 con 0x7fd5a815c300 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 >> v1:172.21.15.125:6789/0 conn(0x7fd5a815c300 legacy=0x7fd5a815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5880039a0 0x7fd588003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.410+0000 7fd5a6a00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a6a00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 0x7fd588005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80c62d0 con 0x7fd5880047c0 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58800ba60 con 0x7fd5880039a0 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58800bc20 con 0x7fd5a80b34e0 2024-01-16T15:09:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 -- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 msgr2=0x7fd588005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a80b34e0 0x7fd588005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 -- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5880039a0 msgr2=0x7fd588003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5880039a0 0x7fd588003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.411+0000 7fd5a7400640 1 -- 172.21.15.125:0/3787009588 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588003520 con 0x7fd5880047c0 2024-01-16T15:09:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.412+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c210 tx=0x7fd5880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafbfc1f9ede604e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.412+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590005ad0 con 0x7fd5880047c0 2024-01-16T15:09:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.412+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd590005c30 con 0x7fd5880047c0 2024-01-16T15:09:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.412+0000 7fd5a6a00640 1 -- 172.21.15.125:0/3787009588 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59002c870 con 0x7fd5880047c0 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 -- 172.21.15.125:0/3787009588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 msgr2=0x7fd588004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3787009588 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c210 tx=0x7fd5880057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5ace00640 1 -- 172.21.15.125:0/3787009588 reap_dead start 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 -- 172.21.15.125:0/3787009588 shutdown_connections 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 -- 172.21.15.125:0/3787009588 >> 172.21.15.125:0/3787009588 conn(0x7fd5a8066690 msgr2=0x7fd5a806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 -- 172.21.15.125:0/3787009588 shutdown_connections 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.413+0000 7fd5aee00640 1 -- 172.21.15.125:0/3787009588 wait complete. 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.415+0000 7fd5aee00640 1 Processor -- start 2024-01-16T15:09:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.415+0000 7fd5aee00640 1 -- start start 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a815af80 0x7fd5a8159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5a815b400 0x7fd5a81e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b0610 con 0x7fd5880047c0 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8159ee0 con 0x7fd5a815b400 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5aee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a815a1c0 con 0x7fd5a815af80 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43816/0 (socket says 172.21.15.125:43816) 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 -- 172.21.15.125:0/3355100052 learned_addr learned my addr 172.21.15.125:0/3355100052 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 -- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a815af80 msgr2=0x7fd5a8159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a815af80 0x7fd5a8159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 -- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5a815b400 msgr2=0x7fd5a81e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5a815b400 0x7fd5a81e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.416+0000 7fd5a7e00640 1 -- 172.21.15.125:0/3355100052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a815a4a0 con 0x7fd5880047c0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd5a7e00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c840 tx=0x7fd59800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598003b70 con 0x7fd5880047c0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598003cf0 con 0x7fd5880047c0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59800aa10 con 0x7fd5880047c0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a815a780 con 0x7fd5880047c0 2024-01-16T15:09:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.417+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a80c8960 con 0x7fd5880047c0 2024-01-16T15:09:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.418+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8074a50 con 0x7fd5880047c0 2024-01-16T15:09:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.419+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fd59800ab70 con 0x7fd5880047c0 2024-01-16T15:09:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.420+0000 7fd59fe00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 0x7fd58006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.420+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd598088410 con 0x7fd5880047c0 2024-01-16T15:09:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.420+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 0x7fd58006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.421+0000 7fd5a7400640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 0x7fd58006e3e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd590034000 tx=0x7fd590007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.424+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd598054d30 con 0x7fd5880047c0 2024-01-16T15:09:58.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.627+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5a815e580 con 0x7fd58006bf20 2024-01-16T15:09:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.634+0000 7fd59fe00640 1 -- 172.21.15.125:0/3355100052 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd5a815e580 con 0x7fd58006bf20 2024-01-16T15:09:58.611 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:09:58.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 msgr2=0x7fd58006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 0x7fd58006e3e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd590034000 tx=0x7fd590007000 comp rx=0 tx=0).stop 2024-01-16T15:09:58.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 msgr2=0x7fd5a80b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c840 tx=0x7fd59800cd00 comp rx=0 tx=0).stop 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 shutdown_connections 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5a815b400 0x7fd5a81e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5a815af80 0x7fd5a8159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd58006bf20 0x7fd58006e3e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 --2- 172.21.15.125:0/3355100052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5880047c0 0x7fd5a80b00d0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 >> 172.21.15.125:0/3355100052 conn(0x7fd5a8066690 msgr2=0x7fd5a80cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 shutdown_connections 2024-01-16T15:09:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.636+0000 7fd5aee00640 1 -- 172.21.15.125:0/3355100052 wait complete. 2024-01-16T15:09:58.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:09:58.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.848+0000 7f6fa8c00640 1 Processor -- start 2024-01-16T15:09:58.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.848+0000 7f6fa8c00640 1 -- start start 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa413cf10 0x7f6fa413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa40cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fa40cdc20 0x7f6fa415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa413ff60 con 0x7f6fa413cf10 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa40ce440 con 0x7f6fa40cdc20 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40ce720 con 0x7f6fa413fae0 2024-01-16T15:09:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa413cf10 0x7f6fa413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa413cf10 0x7f6fa413f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43832/0 (socket says 172.21.15.125:43832) 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 -- 172.21.15.125:0/811495282 learned_addr learned my addr 172.21.15.125:0/811495282 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 -- 172.21.15.125:0/811495282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 msgr2=0x7f6fa40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 --2- 172.21.15.125:0/811495282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa40cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 -- 172.21.15.125:0/811495282 >> v2:172.21.15.137:3300/0 conn(0x7f6fa40cdc20 msgr2=0x7f6fa415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 --2- 172.21.15.125:0/811495282 >> v2:172.21.15.137:3300/0 conn(0x7f6fa40cdc20 0x7f6fa415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.850+0000 7f6fa3400640 1 -- 172.21.15.125:0/811495282 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40cea00 con 0x7f6fa413cf10 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.851+0000 7f6fa3400640 1 --2- 172.21.15.125:0/811495282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa413cf10 0x7f6fa413f5a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4154a00 tx=0x7f6f9402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88451a1440541d1a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa2000640 1 -- 172.21.15.125:0/811495282 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9402ee90 con 0x7f6fa413cf10 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa2000640 1 -- 172.21.15.125:0/811495282 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f94036440 con 0x7f6fa413cf10 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa2000640 1 -- 172.21.15.125:0/811495282 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f940367d0 con 0x7f6fa413cf10 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 -- 172.21.15.125:0/811495282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa413cf10 msgr2=0x7f6fa413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/811495282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa413cf10 0x7f6fa413f5a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4154a00 tx=0x7f6f9402e0e0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 -- 172.21.15.125:0/811495282 shutdown_connections 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/811495282 >> v2:172.21.15.137:3300/0 conn(0x7f6fa40cdc20 0x7f6fa415d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/811495282 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/811495282 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa413cf10 0x7f6fa413f5a0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.852+0000 7f6fa8c00640 1 -- 172.21.15.125:0/811495282 >> 172.21.15.125:0/811495282 conn(0x7f6fa406a7d0 msgr2=0x7f6fa406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.853+0000 7f6fa8c00640 1 -- 172.21.15.125:0/811495282 shutdown_connections 2024-01-16T15:09:58.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.853+0000 7f6fa8c00640 1 -- 172.21.15.125:0/811495282 wait complete. 2024-01-16T15:09:58.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.854+0000 7f6fa8c00640 1 Processor -- start 2024-01-16T15:09:58.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.854+0000 7f6fa8c00640 1 -- start start 2024-01-16T15:09:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 0x7f6fa40ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa413efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40ac8d0 con 0x7f6fa40cdc20 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa413f4e0 con 0x7f6fa413cf10 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40ae3b0 con 0x7f6fa413fae0 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43834/0 (socket says 172.21.15.125:43834) 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.855+0000 7f6fa3400640 1 -- 172.21.15.125:0/1463036630 learned_addr learned my addr 172.21.15.125:0/1463036630 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 -- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 msgr2=0x7f6fa413efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa2a00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 0x7f6fa40ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa413efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa413efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 -- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 msgr2=0x7f6fa40ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 0x7f6fa40ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 -- 172.21.15.125:0/1463036630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9400b770 con 0x7f6fa40cdc20 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3e00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa2a00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 0x7f6fa40ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6fa3400640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6f9402e120 tx=0x7f6f94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.856+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94034b80 con 0x7f6fa40cdc20 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.857+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f94034d00 con 0x7f6fa40cdc20 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.857+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f940416c0 con 0x7f6fa40cdc20 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.857+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa40ae690 con 0x7f6fa40cdc20 2024-01-16T15:09:58.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.857+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa40ace00 con 0x7f6fa40cdc20 2024-01-16T15:09:58.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.858+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa4067450 con 0x7f6fa40cdc20 2024-01-16T15:09:58.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.859+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6f94041820 con 0x7f6fa40cdc20 2024-01-16T15:09:58.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.859+0000 7f6f9b400640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 0x7f6f7c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.860+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f940ab900 con 0x7f6fa40cdc20 2024-01-16T15:09:58.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.860+0000 7f6fa2a00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 0x7f6f7c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.837 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.860+0000 7f6fa2a00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 0x7f6f7c06e3e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40adcf0 tx=0x7f6f8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:58.864+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f940781c0 con 0x7f6fa40cdc20 2024-01-16T15:09:59.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.063+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fa40ce0b0 con 0x7f6f7c06bf20 2024-01-16T15:09:59.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.069+0000 7f6f9b400640 1 -- 172.21.15.125:0/1463036630 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f6f8c00a460 con 0x7f6f7c06bf20 2024-01-16T15:09:59.046 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 msgr2=0x7f6f7c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 0x7f6f7c06e3e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40adcf0 tx=0x7f6f8c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 msgr2=0x7f6fa40add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6f9402e120 tx=0x7f6f94004730 comp rx=0 tx=0).stop 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 shutdown_connections 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fa413fae0 0x7f6fa413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fa413cf10 0x7f6fa40ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f7c06bf20 0x7f6f7c06e3e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 --2- 172.21.15.125:0/1463036630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cdc20 0x7f6fa40add60 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 >> 172.21.15.125:0/1463036630 conn(0x7f6fa406a7d0 msgr2=0x7f6fa406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 shutdown_connections 2024-01-16T15:09:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:09:59.072+0000 7f6fa8c00640 1 -- 172.21.15.125:0/1463036630 wait complete. 2024-01-16T15:09:59.066 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:09:59.066 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:09:59.066 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:59.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:59.118 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:10.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:10.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:12.835 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:10:12.836 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:10:12.836 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6439 s, 78.7 MB/s 2024-01-16T15:10:12.837 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:10:12.837 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:10:12.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.878 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:23.941 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:10:23.941 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:10:23.941 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.9769 s, 97.8 MB/s 2024-01-16T15:10:23.943 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:10:23.943 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:10:23.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:23.996 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:10:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:10:36.243 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.1894 s, 88.1 MB/s 2024-01-16T15:10:36.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:10:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.487+0000 7fb44b200640 1 Processor -- start 2024-01-16T15:10:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.487+0000 7fb44b200640 1 -- start start 2024-01-16T15:10:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb44b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb44c15afb0 0x7fb44c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb44b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb44c13fae0 0x7fb44c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb44b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c13ff30 con 0x7fb44c13cfa0 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb44b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0cdc40 con 0x7fb44c15afb0 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb44b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0cdec0 con 0x7fb44c13fae0 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb441200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb44c15afb0 0x7fb44c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb441200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb44c15afb0 0x7fb44c13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48984/0 (socket says 172.21.15.125:48984) 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.489+0000 7fb441200640 1 -- 172.21.15.125:0/1704506724 learned_addr learned my addr 172.21.15.125:0/1704506724 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.490+0000 7fb441200640 1 -- 172.21.15.125:0/1704506724 >> v2:172.21.15.137:3300/0 conn(0x7fb44c13fae0 msgr2=0x7fb44c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.490+0000 7fb441200640 1 --2- 172.21.15.125:0/1704506724 >> v2:172.21.15.137:3300/0 conn(0x7fb44c13fae0 0x7fb44c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.490+0000 7fb441200640 1 -- 172.21.15.125:0/1704506724 >> v1:172.21.15.137:6789/0 conn(0x7fb44c13cfa0 legacy=0x7fb44c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:36.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.490+0000 7fb441200640 1 -- 172.21.15.125:0/1704506724 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0ce1a0 con 0x7fb44c15afb0 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb441200640 1 --2- 172.21.15.125:0/1704506724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c15afb0 0x7fb44c13f5a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb44c154a90 tx=0x7fb43402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3f04e9a9bdf7ed server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb449400640 1 -- 172.21.15.125:0/1704506724 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4340317e0 con 0x7fb44c15afb0 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb449400640 1 -- 172.21.15.125:0/1704506724 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb434031940 con 0x7fb44c15afb0 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb449400640 1 -- 172.21.15.125:0/1704506724 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434031cd0 con 0x7fb44c15afb0 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb44b200640 1 -- 172.21.15.125:0/1704506724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c15afb0 msgr2=0x7fb44c13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.491+0000 7fb44b200640 1 --2- 172.21.15.125:0/1704506724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c15afb0 0x7fb44c13f5a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb44c154a90 tx=0x7fb43402b880 comp rx=0 tx=0).stop 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 -- 172.21.15.125:0/1704506724 shutdown_connections 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 --2- 172.21.15.125:0/1704506724 >> v2:172.21.15.137:3300/0 conn(0x7fb44c13fae0 0x7fb44c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 --2- 172.21.15.125:0/1704506724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c15afb0 0x7fb44c13f5a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 -- 172.21.15.125:0/1704506724 >> 172.21.15.125:0/1704506724 conn(0x7fb44c06a800 msgr2=0x7fb44c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 -- 172.21.15.125:0/1704506724 shutdown_connections 2024-01-16T15:10:36.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.492+0000 7fb44b200640 1 -- 172.21.15.125:0/1704506724 wait complete. 2024-01-16T15:10:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.493+0000 7fb44b200640 1 Processor -- start 2024-01-16T15:10:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 -- start start 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb44c13fae0 0x7fb44c139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb44c15afb0 0x7fb44c13e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0af7e0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0b0360 con 0x7fb44c13fae0 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.494+0000 7fb44b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c13e9a0 con 0x7fb44c15afb0 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48986/0 (socket says 172.21.15.125:48986) 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 -- 172.21.15.125:0/3422216982 learned_addr learned my addr 172.21.15.125:0/3422216982 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 -- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb44c15afb0 msgr2=0x7fb44c13e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb44c15afb0 0x7fb44c13e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 -- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb44c13fae0 msgr2=0x7fb44c139890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb44c13fae0 0x7fb44c139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 -- 172.21.15.125:0/3422216982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340094d0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.495+0000 7fb449e00640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c410 tx=0x7fb43c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.496+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c0035f0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.496+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c003750 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.496+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c003b40 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.496+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c0afa30 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.496+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c0afff0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.497+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c067450 con 0x7fb44c13cfa0 2024-01-16T15:10:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.498+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb43c003ca0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.499+0000 7fb442a00640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 0x7fb42406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.499+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb43c0878b0 con 0x7fb44c13cfa0 2024-01-16T15:10:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.499+0000 7fb441200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 0x7fb42406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.500+0000 7fb441200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 0x7fb42406e470 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb434036000 tx=0x7fb434030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.504+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb43c054130 con 0x7fb44c13cfa0 2024-01-16T15:10:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.724+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb44c0cea40 con 0x7fb42406bfb0 2024-01-16T15:10:36.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.750+0000 7fb442a00640 1 -- 172.21.15.125:0/3422216982 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb434031c70 con 0x7fb42406bfb0 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 msgr2=0x7fb42406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 0x7fb42406e470 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb434036000 tx=0x7fb434030000 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 msgr2=0x7fb44c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c410 tx=0x7fb43c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 shutdown_connections 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb44c15afb0 0x7fb44c13e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb44c13fae0 0x7fb44c139890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb42406bfb0 0x7fb42406e470 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 --2- 172.21.15.125:0/3422216982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb44c13cfa0 0x7fb44c0cb1b0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 >> 172.21.15.125:0/3422216982 conn(0x7fb44c06a800 msgr2=0x7fb44c0714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 shutdown_connections 2024-01-16T15:10:36.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.753+0000 7fb44b200640 1 -- 172.21.15.125:0/3422216982 wait complete. 2024-01-16T15:10:36.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:10:36.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.967+0000 7f52cfc00640 1 Processor -- start 2024-01-16T15:10:36.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.967+0000 7f52cfc00640 1 -- start start 2024-01-16T15:10:36.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cfc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cfc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c80cfc30 con 0x7f52c80cf7d0 2024-01-16T15:10:36.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cfc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80b1ee0 con 0x7f52c8143ae0 2024-01-16T15:10:36.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b2780 con 0x7f52c815e3b0 2024-01-16T15:10:36.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:49070/0 (socket says 172.21.15.125:49070) 2024-01-16T15:10:36.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.969+0000 7f52cdc00640 1 -- 172.21.15.125:0/1119365541 learned_addr learned my addr 172.21.15.125:0/1119365541 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:36.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.970+0000 7f52cd200640 1 --2- 172.21.15.125:0/1119365541 >> v2:172.21.15.137:3300/0 conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.970+0000 7f52cdc00640 1 -- 172.21.15.125:0/1119365541 >> v2:172.21.15.137:3300/0 conn(0x7f52c8143ae0 msgr2=0x7f52c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:36.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.970+0000 7f52cdc00640 1 --2- 172.21.15.125:0/1119365541 >> v2:172.21.15.137:3300/0 conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.970+0000 7f52cdc00640 1 -- 172.21.15.125:0/1119365541 >> v1:172.21.15.137:6789/0 conn(0x7f52c80cf7d0 legacy=0x7f52c815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.970+0000 7f52cdc00640 1 -- 172.21.15.125:0/1119365541 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c815e990 con 0x7f52c815e3b0 2024-01-16T15:10:36.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.972+0000 7f52cdc00640 1 --2- 172.21.15.125:0/1119365541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d150 tx=0x7f52c002e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece7368ecbc475d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.973+0000 7f52c7e00640 1 -- 172.21.15.125:0/1119365541 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000b0d0 con 0x7f52c815e3b0 2024-01-16T15:10:36.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.973+0000 7f52c7e00640 1 -- 172.21.15.125:0/1119365541 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000b230 con 0x7f52c815e3b0 2024-01-16T15:10:36.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.973+0000 7f52c7e00640 1 -- 172.21.15.125:0/1119365541 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0009600 con 0x7f52c815e3b0 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.973+0000 7f52cfc00640 1 -- 172.21.15.125:0/1119365541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 msgr2=0x7f52c80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.973+0000 7f52cfc00640 1 --2- 172.21.15.125:0/1119365541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d150 tx=0x7f52c002e160 comp rx=0 tx=0).stop 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 -- 172.21.15.125:0/1119365541 shutdown_connections 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 --2- 172.21.15.125:0/1119365541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c815e3b0 0x7f52c80702c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 --2- 172.21.15.125:0/1119365541 >> v2:172.21.15.137:3300/0 conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 -- 172.21.15.125:0/1119365541 >> 172.21.15.125:0/1119365541 conn(0x7f52c806a800 msgr2=0x7f52c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:36.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 -- 172.21.15.125:0/1119365541 shutdown_connections 2024-01-16T15:10:36.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.974+0000 7f52cfc00640 1 -- 172.21.15.125:0/1119365541 wait complete. 2024-01-16T15:10:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.975+0000 7f52cfc00640 1 Processor -- start 2024-01-16T15:10:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.975+0000 7f52cfc00640 1 -- start start 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c80cf7d0 0x7f52c80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 0x7f52c80aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80ae4d0 con 0x7f52c8143ae0 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8135c20 con 0x7f52c815e3b0 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.976+0000 7f52cfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80aa8b0 con 0x7f52c80cf7d0 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42946/0 (socket says 172.21.15.125:42946) 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 -- 172.21.15.125:0/155711106 learned_addr learned my addr 172.21.15.125:0/155711106 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52cd200640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c80cf7d0 0x7f52c80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52cdc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 0x7f52c80aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 -- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c80cf7d0 msgr2=0x7f52c80adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c80cf7d0 0x7f52c80adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 -- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 msgr2=0x7f52c80aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 0x7f52c80aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52bfc00640 1 -- 172.21.15.125:0/155711106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c000bd40 con 0x7f52c8143ae0 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.977+0000 7f52cdc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 0x7f52c80aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52bfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c450 tx=0x7f52b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00052d0 con 0x7f52c8143ae0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0005430 con 0x7f52c8143ae0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0012870 con 0x7f52c8143ae0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c815d530 con 0x7f52c8143ae0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.978+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c80aa260 con 0x7f52c8143ae0 2024-01-16T15:10:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.979+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c80675d0 con 0x7f52c8143ae0 2024-01-16T15:10:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.980+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f52b00129d0 con 0x7f52c8143ae0 2024-01-16T15:10:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.981+0000 7f52c5600640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 0x7f52a006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.981+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f52b00880f0 con 0x7f52c8143ae0 2024-01-16T15:10:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.981+0000 7f52cd200640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 0x7f52a006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:36.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.982+0000 7f52cd200640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 0x7f52a006e470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009ad0 tx=0x7f52b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:36.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:36.985+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52b0054a10 con 0x7f52c8143ae0 2024-01-16T15:10:37.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.208+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f52c80b2410 con 0x7f52a006bfb0 2024-01-16T15:10:37.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.216+0000 7f52c5600640 1 -- 172.21.15.125:0/155711106 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f52b80119a0 con 0x7f52a006bfb0 2024-01-16T15:10:37.193 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:10:37.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.218+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 msgr2=0x7f52a006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.218+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 0x7f52a006e470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009ad0 tx=0x7f52b8008040 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.218+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 msgr2=0x7f52c81356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.218+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c450 tx=0x7f52b000c910 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 shutdown_connections 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52c815e3b0 0x7f52c80aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52a006bfb0 0x7f52a006e470 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52c8143ae0 0x7f52c81356e0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 --2- 172.21.15.125:0/155711106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52c80cf7d0 0x7f52c80adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 >> 172.21.15.125:0/155711106 conn(0x7f52c806a800 msgr2=0x7f52c806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 shutdown_connections 2024-01-16T15:10:37.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:37.219+0000 7f52cfc00640 1 -- 172.21.15.125:0/155711106 wait complete. 2024-01-16T15:10:37.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:37.213 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e) 2024-01-16T15:10:37.268 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:10:37.268 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:10:37.268 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:37.269 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:10:39.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:39.270 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e) 2024-01-16T15:10:39.333 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e 2024-01-16T15:10:39.333 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:10:39.333 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:39.334 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:10:39.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:10:39.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.571+0000 7f4a15e00640 1 Processor -- start 2024-01-16T15:10:39.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.572+0000 7f4a15e00640 1 -- start start 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a15e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f4a100717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a15e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1015e680 con 0x7f4a100ad880 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a15e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a10071cf0 con 0x7f4a1006bb00 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a15e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015dd50 con 0x7f4a100ac8d0 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a0ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4a100ad880 0x7f4a1006b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33444/0 (socket says 172.21.15.125:33444) 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.573+0000 7f4a0ea00640 1 -- 172.21.15.125:0/1526334221 learned_addr learned my addr 172.21.15.125:0/1526334221 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0f400640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f4a100717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440892541 0 0) 0x7f4a1015e680 con 0x7f4a100ad880 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f0003520 con 0x7f4a100ad880 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554276817 0 0) 0x7f49f0003520 con 0x7f4a100ad880 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 msgr2=0x7f4a100717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f4a100717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 >> v1:172.21.15.137:6789/0 conn(0x7f4a1006bb00 legacy=0x7f4a100b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1015e030 con 0x7f4a100ad880 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f4a00001670 con 0x7f4a100ad880 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 >> v1:172.21.15.125:6789/0 conn(0x7f4a100ad880 legacy=0x7f4a1006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f49f00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49f0003f70 0x7f49f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49f0004d00 0x7f49f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015dd50 con 0x7f49f0003f70 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f000bac0 con 0x7f49f0004d00 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f000bc80 con 0x7f4a100ac8d0 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.574+0000 7f4a0e000640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49f0003f70 0x7f49f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0ea00640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f49f00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0f400640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49f0004d00 0x7f49f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0e000640 1 -- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 msgr2=0x7f49f00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0e000640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a100ac8d0 0x7f49f00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0e000640 1 -- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49f0004d00 msgr2=0x7f49f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0e000640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49f0004d00 0x7f49f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0e000640 1 -- 172.21.15.125:0/1526334221 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f0003520 con 0x7f49f0003f70 2024-01-16T15:10:39.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.575+0000 7f4a0f400640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49f0004d00 0x7f49f0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.576+0000 7f4a0e000640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49f0003f70 0x7f49f00047c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f49f8001e50 tx=0x7f49f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60bf0573b740e50 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.576+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8003b70 con 0x7f49f0003f70 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.576+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f8003cd0 con 0x7f49f0003f70 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.576+0000 7f4a0d600640 1 -- 172.21.15.125:0/1526334221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8005310 con 0x7f49f0003f70 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 -- 172.21.15.125:0/1526334221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49f0003f70 msgr2=0x7f49f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 --2- 172.21.15.125:0/1526334221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49f0003f70 0x7f49f00047c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f49f8001e50 tx=0x7f49f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a0f400640 1 -- 172.21.15.125:0/1526334221 reap_dead start 2024-01-16T15:10:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 -- 172.21.15.125:0/1526334221 shutdown_connections 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 -- 172.21.15.125:0/1526334221 >> 172.21.15.125:0/1526334221 conn(0x7f4a10070a80 msgr2=0x7f4a10070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 -- 172.21.15.125:0/1526334221 shutdown_connections 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.577+0000 7f4a15e00640 1 -- 172.21.15.125:0/1526334221 wait complete. 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.579+0000 7f4a15e00640 1 Processor -- start 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.579+0000 7f4a15e00640 1 -- start start 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49f0003f70 0x7f4a101480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a100cc110 0x7f4a100cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015cfd0 con 0x7f4a10159020 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10158c00 con 0x7f4a100cc110 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a15e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10158260 con 0x7f49f0003f70 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a0f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a0f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42960/0 (socket says 172.21.15.125:42960) 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a0f400640 1 -- 172.21.15.125:0/2748686362 learned_addr learned my addr 172.21.15.125:0/2748686362 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a0e000640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a100cc110 0x7f4a100cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.580+0000 7f4a0ea00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49f0003f70 0x7f4a101480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 -- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49f0003f70 msgr2=0x7f4a101480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49f0003f70 0x7f4a101480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 -- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a100cc110 msgr2=0x7f4a100cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a100cc110 0x7f4a100cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 -- 172.21.15.125:0/2748686362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10158540 con 0x7f4a10159020 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a0f400640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f4a100acfb0 tx=0x7f4a08010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800e840 con 0x7f4a10159020 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0800e9a0 con 0x7f4a10159020 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800ed30 con 0x7f4a10159020 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.581+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a10158820 con 0x7f4a10159020 2024-01-16T15:10:39.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.582+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a100ce520 con 0x7f4a10159020 2024-01-16T15:10:39.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.583+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a10158d40 con 0x7f4a10159020 2024-01-16T15:10:39.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.584+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f4a08020410 con 0x7f4a10159020 2024-01-16T15:10:39.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.585+0000 7f4a07400640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 0x7f49e806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:39.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.585+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4a08087db0 con 0x7f4a10159020 2024-01-16T15:10:39.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.585+0000 7f4a0ea00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 0x7f49e806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:39.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.586+0000 7f4a0ea00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 0x7f49e806e470 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4a00001050 tx=0x7f4a00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:39.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.589+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a10158d40 con 0x7f4a10159020 2024-01-16T15:10:39.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.818+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a1015d340 con 0x7f49e806bfb0 2024-01-16T15:10:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.970+0000 7f4a07400640 1 -- 172.21.15.125:0/2748686362 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a1015d340 con 0x7f49e806bfb0 2024-01-16T15:10:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.973+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 msgr2=0x7f49e806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.973+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 0x7f49e806e470 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4a00001050 tx=0x7f4a00009040 comp rx=0 tx=0).stop 2024-01-16T15:10:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 msgr2=0x7f4a101594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f4a100acfb0 tx=0x7f4a08010c20 comp rx=0 tx=0).stop 2024-01-16T15:10:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 shutdown_connections 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f49e806bfb0 0x7f49e806e470 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a10159020 0x7f4a101594a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a100cc110 0x7f4a100cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 --2- 172.21.15.125:0/2748686362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49f0003f70 0x7f4a101480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 >> 172.21.15.125:0/2748686362 conn(0x7f4a10070a80 msgr2=0x7f4a1006ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 shutdown_connections 2024-01-16T15:10:39.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:39.974+0000 7f4a15e00640 1 -- 172.21.15.125:0/2748686362 wait complete. 2024-01-16T15:10:39.953 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.976+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.953 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.976+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.953 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.976+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.953 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.976+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.953 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.976+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.971 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:10:39.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:10:39.977 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.999+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.977 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.999+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.977 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.999+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.977 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.999+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:39.977 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:10:39.999+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:40.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.172+0000 7fad72c00640 1 Processor -- start 2024-01-16T15:10:40.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.172+0000 7fad72c00640 1 -- start start 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.173+0000 7fad72c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.173+0000 7fad72c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c15f4e0 con 0x7fad6c0ccb50 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.173+0000 7fad72c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c071130 con 0x7fad6c15bf10 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.173+0000 7fad72c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c15f970 con 0x7fad6c0adf20 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad72200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fad6c0ccb50 0x7fad6c143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33460/0 (socket says 172.21.15.125:33460) 2024-01-16T15:10:40.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad72200640 1 -- 172.21.15.125:0/2212510341 learned_addr learned my addr 172.21.15.125:0/2212510341 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:40.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad71800640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740634965 0 0) 0x7fad6c15f4e0 con 0x7fad6c0ccb50 2024-01-16T15:10:40.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c003520 con 0x7fad6c0ccb50 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590775847 0 0) 0x7fad4c003520 con 0x7fad6c0ccb50 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 msgr2=0x7fad6c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.174+0000 7fad6be00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad6c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 >> v1:172.21.15.137:6789/0 conn(0x7fad6c15bf10 legacy=0x7fad6c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c15c390 con 0x7fad6c0ccb50 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fad64001670 con 0x7fad6c0ccb50 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 >> v1:172.21.15.125:6789/0 conn(0x7fad6c0ccb50 legacy=0x7fad6c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad4c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad4c003f20 0x7fad4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad4c004d00 0x7fad4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c15f970 con 0x7fad4c004d00 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c00bb10 con 0x7fad4c003f20 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c00bcd0 con 0x7fad6c0adf20 2024-01-16T15:10:40.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad72200640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad4c004d00 0x7fad4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad71800640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad4c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.175+0000 7fad70e00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad4c003f20 0x7fad4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad72200640 1 -- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 msgr2=0x7fad4c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad72200640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c0adf20 0x7fad4c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad72200640 1 -- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad4c003f20 msgr2=0x7fad4c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad72200640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad4c003f20 0x7fad4c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad72200640 1 -- 172.21.15.125:0/2212510341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c003520 con 0x7fad4c004d00 2024-01-16T15:10:40.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.176+0000 7fad70e00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad4c003f20 0x7fad4c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:10:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72200640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad4c004d00 0x7fad4c005180 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c160 tx=0x7fad4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676744513bd0e4b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64005ad0 con 0x7fad4c004d00 2024-01-16T15:10:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64005c50 con 0x7fad4c004d00 2024-01-16T15:10:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad6be00640 1 -- 172.21.15.125:0/2212510341 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6402c870 con 0x7fad4c004d00 2024-01-16T15:10:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72c00640 1 -- 172.21.15.125:0/2212510341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad4c004d00 msgr2=0x7fad4c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72c00640 1 --2- 172.21.15.125:0/2212510341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad4c004d00 0x7fad4c005180 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c160 tx=0x7fad4c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72200640 1 -- 172.21.15.125:0/2212510341 reap_dead start 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72c00640 1 -- 172.21.15.125:0/2212510341 shutdown_connections 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.177+0000 7fad72c00640 1 -- 172.21.15.125:0/2212510341 >> 172.21.15.125:0/2212510341 conn(0x7fad6c06a7d0 msgr2=0x7fad6c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.178+0000 7fad72c00640 1 -- 172.21.15.125:0/2212510341 shutdown_connections 2024-01-16T15:10:40.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.178+0000 7fad72c00640 1 -- 172.21.15.125:0/2212510341 wait complete. 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.179+0000 7fad72c00640 1 Processor -- start 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.179+0000 7fad72c00640 1 -- start start 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 0x7fad6c153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c1eeda0 0x7fad6c15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c152510 con 0x7fad6c151fa0 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b0e90 con 0x7fad6c1eeda0 2024-01-16T15:10:40.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.180+0000 7fad72c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b1130 con 0x7fad6c145850 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43026/0 (socket says 172.21.15.125:43026) 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 -- 172.21.15.125:0/3730325137 learned_addr learned my addr 172.21.15.125:0/3730325137 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad71800640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 0x7fad6c153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad70e00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c1eeda0 0x7fad6c15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 -- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 msgr2=0x7fad6c153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 0x7fad6c153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 -- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c1eeda0 msgr2=0x7fad6c15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c1eeda0 0x7fad6c15d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 -- 172.21.15.125:0/3730325137 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c0b1410 con 0x7fad6c151fa0 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad71800640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 0x7fad6c153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72200640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fad64001930 tx=0x7fad6c15b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64001c30 con 0x7fad6c151fa0 2024-01-16T15:10:40.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6402c870 con 0x7fad6c151fa0 2024-01-16T15:10:40.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6c0b16f0 con 0x7fad6c151fa0 2024-01-16T15:10:40.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.181+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6402cbe0 con 0x7fad6c151fa0 2024-01-16T15:10:40.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.182+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c0b2410 con 0x7fad6c151fa0 2024-01-16T15:10:40.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.183+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c0ad380 con 0x7fad6c151fa0 2024-01-16T15:10:40.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.183+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fad64032070 con 0x7fad6c151fa0 2024-01-16T15:10:40.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.184+0000 7fad69600640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 0x7fad4406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.184+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fad6403c070 con 0x7fad6c151fa0 2024-01-16T15:10:40.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.185+0000 7fad71800640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 0x7fad4406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.185+0000 7fad71800640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 0x7fad4406e470 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007440 tx=0x7fad5c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.189+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad64075da0 con 0x7fad6c151fa0 2024-01-16T15:10:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.413+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fad6c0b12a0 con 0x7fad4406bfb0 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.422+0000 7fad69600640 1 -- 172.21.15.125:0/3730325137 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fad6c0b12a0 con 0x7fad4406bfb0 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:10:40.400 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:10:40.401 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:40.401 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:40.401 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:10:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 msgr2=0x7fad4406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 0x7fad4406e470 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007440 tx=0x7fad5c00b040 comp rx=0 tx=0).stop 2024-01-16T15:10:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 msgr2=0x7fad6c15da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fad64001930 tx=0x7fad6c15b490 comp rx=0 tx=0).stop 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 shutdown_connections 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad4406bfb0 0x7fad4406e470 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6c151fa0 0x7fad6c15da70 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad6c1eeda0 0x7fad6c15d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 --2- 172.21.15.125:0/3730325137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad6c145850 0x7fad6c153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 >> 172.21.15.125:0/3730325137 conn(0x7fad6c06a7d0 msgr2=0x7fad6c15a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 shutdown_connections 2024-01-16T15:10:40.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.425+0000 7fad72c00640 1 -- 172.21.15.125:0/3730325137 wait complete. 2024-01-16T15:10:40.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-16T15:10:40.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.695+0000 7f5b6ec00640 1 Processor -- start 2024-01-16T15:10:40.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.696+0000 7f5b6ec00640 1 -- start start 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6813e360 0x7f5b680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6ec00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b6815f9c0 0x7f5b680cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6ec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6815c5f0 con 0x7f5b6813df80 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6ec00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6806c470 con 0x7f5b6815f9c0 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6806c750 con 0x7f5b6813e360 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.697+0000 7f5b6cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b6815f9c0 0x7f5b680cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b6815f9c0 0x7f5b680cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43040/0 (socket says 172.21.15.125:43040) 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 -- 172.21.15.125:0/2586254053 learned_addr learned my addr 172.21.15.125:0/2586254053 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:40.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b5f200640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6813e360 0x7f5b680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 -- 172.21.15.125:0/2586254053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6813e360 msgr2=0x7f5b680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:40.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6813e360 0x7f5b680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 -- 172.21.15.125:0/2586254053 >> v1:172.21.15.137:6789/0 conn(0x7f5b6813df80 legacy=0x7f5b6815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.698+0000 7f5b6cc00640 1 -- 172.21.15.125:0/2586254053 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6806ca30 con 0x7f5b6815f9c0 2024-01-16T15:10:40.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.699+0000 7f5b6cc00640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6815f9c0 0x7f5b680cf7d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009e50 tx=0x7f5b6002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02c68bc26717df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b67400640 1 -- 172.21.15.125:0/2586254053 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6002cb00 con 0x7f5b6815f9c0 2024-01-16T15:10:40.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b67400640 1 -- 172.21.15.125:0/2586254053 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6002cc60 con 0x7f5b6815f9c0 2024-01-16T15:10:40.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b67400640 1 -- 172.21.15.125:0/2586254053 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b600343f0 con 0x7f5b6815f9c0 2024-01-16T15:10:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 -- 172.21.15.125:0/2586254053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6815f9c0 msgr2=0x7f5b680cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6815f9c0 0x7f5b680cf7d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009e50 tx=0x7f5b6002bf60 comp rx=0 tx=0).stop 2024-01-16T15:10:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 -- 172.21.15.125:0/2586254053 shutdown_connections 2024-01-16T15:10:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6815f9c0 0x7f5b680cf7d0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/2586254053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6813e360 0x7f5b680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.700+0000 7f5b6ec00640 1 -- 172.21.15.125:0/2586254053 >> 172.21.15.125:0/2586254053 conn(0x7f5b6806a800 msgr2=0x7f5b6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.701+0000 7f5b6ec00640 1 -- 172.21.15.125:0/2586254053 shutdown_connections 2024-01-16T15:10:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.701+0000 7f5b6ec00640 1 -- 172.21.15.125:0/2586254053 wait complete. 2024-01-16T15:10:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.703+0000 7f5b6ec00640 1 Processor -- start 2024-01-16T15:10:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.703+0000 7f5b6ec00640 1 -- start start 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b6813e360 0x7f5b680b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6815f9c0 0x7f5b680b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b681f0f90 con 0x7f5b6813df80 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b681f1270 con 0x7f5b6813e360 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b6ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b681f1550 con 0x7f5b6815f9c0 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b67e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b67e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43042/0 (socket says 172.21.15.125:43042) 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.704+0000 7f5b67e00640 1 -- 172.21.15.125:0/1860220031 learned_addr learned my addr 172.21.15.125:0/1860220031 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b6cc00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6815f9c0 0x7f5b680b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b5f200640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b6813e360 0x7f5b680b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 -- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6815f9c0 msgr2=0x7f5b680b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6815f9c0 0x7f5b680b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 -- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b6813e360 msgr2=0x7f5b680b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b6813e360 0x7f5b680b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 -- 172.21.15.125:0/1860220031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60009b30 con 0x7f5b6813df80 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b67e00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c400 tx=0x7f5b5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b580052d0 con 0x7f5b6813df80 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b58005430 con 0x7f5b6813df80 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b681f1830 con 0x7f5b6813df80 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003830 con 0x7f5b6813df80 2024-01-16T15:10:40.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.705+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6813dc70 con 0x7f5b6813df80 2024-01-16T15:10:40.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.707+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b681f1d50 con 0x7f5b6813df80 2024-01-16T15:10:40.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.708+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f5b58003990 con 0x7f5b6813df80 2024-01-16T15:10:40.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.709+0000 7f5b64c00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 0x7f5b4006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:40.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.709+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5b58087b50 con 0x7f5b6813df80 2024-01-16T15:10:40.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.709+0000 7f5b5f200640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 0x7f5b4006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.710+0000 7f5b5f200640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 0x7f5b4006e470 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5b681f4340 tx=0x7f5b50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:40.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.713+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b681f1d50 con 0x7f5b6813df80 2024-01-16T15:10:40.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:40.943+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b6806cff0 con 0x7f5b4006bfb0 2024-01-16T15:10:40.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:40.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:41.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:41.385+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f5b58054820 con 0x7f5b6813df80 2024-01-16T15:10:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.479+0000 7f5b64c00640 1 -- 172.21.15.125:0/1860220031 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b50009540 con 0x7f5b4006bfb0 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 msgr2=0x7f5b4006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 0x7f5b4006e470 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5b681f4340 tx=0x7f5b50008040 comp rx=0 tx=0).stop 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 msgr2=0x7f5b6813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c400 tx=0x7f5b5800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 shutdown_connections 2024-01-16T15:10:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b6815f9c0 0x7f5b680b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b6813e360 0x7f5b680b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5b4006bfb0 0x7f5b4006e470 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 --2- 172.21.15.125:0/1860220031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b6813df80 0x7f5b6813d730 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.482+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 >> 172.21.15.125:0/1860220031 conn(0x7f5b6806a800 msgr2=0x7f5b6815cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.483+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 shutdown_connections 2024-01-16T15:10:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:52.483+0000 7f5b6ec00640 1 -- 172.21.15.125:0/1860220031 wait complete. 2024-01-16T15:10:54.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:10:54.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.723+0000 7f317ee00640 1 Processor -- start 2024-01-16T15:10:54.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.723+0000 7f317ee00640 1 -- start start 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.724+0000 7f317ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f317813fcb0 0x7f31780cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317ee00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780ce440 con 0x7f317813cf10 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317ee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780ce720 con 0x7f31780cdc20 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31780cea00 con 0x7f317813fcb0 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317da00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f317813cf10 0x7f317813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37646/0 (socket says 172.21.15.125:37646) 2024-01-16T15:10:54.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317da00640 1 -- 172.21.15.125:0/524414787 learned_addr learned my addr 172.21.15.125:0/524414787 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307432058 0 0) 0x7f31780ce440 con 0x7f317813cf10 2024-01-16T15:10:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158003520 con 0x7f317813cf10 2024-01-16T15:10:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.725+0000 7f317d000640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f317813fcb0 0x7f31780cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151684841 0 0) 0x7f3158003520 con 0x7f317813cf10 2024-01-16T15:10:54.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f317813fcb0 msgr2=0x7f31780cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f317813fcb0 0x7f31780cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 >> v1:172.21.15.137:6789/0 conn(0x7f31780cdc20 legacy=0x7f31780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780b1d90 con 0x7f317813cf10 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3168001670 con 0x7f317813cf10 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 >> v1:172.21.15.125:6789/0 conn(0x7f317813cf10 legacy=0x7f317813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31580039a0 0x7f3158003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580047c0 0x7f3158004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3158005180 0x7f315800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780cea00 con 0x7f31580039a0 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315800c050 con 0x7f3158005180 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315800c210 con 0x7f31580047c0 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317d000640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f317813fcb0 0x7f31780cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31580039a0 0x7f3158003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 -- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580047c0 msgr2=0x7f3158004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580047c0 0x7f3158004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317e400640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3158005180 0x7f315800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 -- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3158005180 msgr2=0x7f315800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3158005180 0x7f315800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.726+0000 7f317da00640 1 -- 172.21.15.125:0/524414787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158003520 con 0x7f31580039a0 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.727+0000 7f317da00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31580039a0 0x7f3158003dc0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f316800c240 tx=0x7f31580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85454cf27c6aa9d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168005ad0 con 0x7f31580039a0 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3168005c30 con 0x7f31580039a0 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f3177e00640 1 -- 172.21.15.125:0/524414787 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316802c8b0 con 0x7f31580039a0 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317ee00640 1 -- 172.21.15.125:0/524414787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31580039a0 msgr2=0x7f3158003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317ee00640 1 --2- 172.21.15.125:0/524414787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31580039a0 0x7f3158003dc0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f316800c240 tx=0x7f31580057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317e400640 1 -- 172.21.15.125:0/524414787 reap_dead start 2024-01-16T15:10:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317ee00640 1 -- 172.21.15.125:0/524414787 shutdown_connections 2024-01-16T15:10:54.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317ee00640 1 -- 172.21.15.125:0/524414787 >> 172.21.15.125:0/524414787 conn(0x7f317806a7d0 msgr2=0x7f317806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:54.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.728+0000 7f317ee00640 1 -- 172.21.15.125:0/524414787 shutdown_connections 2024-01-16T15:10:54.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.729+0000 7f317ee00640 1 -- 172.21.15.125:0/524414787 wait complete. 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.730+0000 7f317ee00640 1 Processor -- start 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.730+0000 7f317ee00640 1 -- start start 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580039a0 0x7f31780acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31780ae240 0x7f317815d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317815d690 con 0x7f31781f6a70 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780ad1a0 con 0x7f31780ae240 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.731+0000 7f317ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31780ad480 con 0x7f31580039a0 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47798/0 (socket says 172.21.15.125:47798) 2024-01-16T15:10:54.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 -- 172.21.15.125:0/680640891 learned_addr learned my addr 172.21.15.125:0/680640891 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317da00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580039a0 0x7f31780acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317d000640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31780ae240 0x7f317815d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 -- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580039a0 msgr2=0x7f31780acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580039a0 0x7f31780acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 -- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31780ae240 msgr2=0x7f317815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31780ae240 0x7f317815d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 -- 172.21.15.125:0/680640891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780ad760 con 0x7f31781f6a70 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.732+0000 7f317e400640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3170000f80 tx=0x7f317000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.733+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170003b70 con 0x7f31781f6a70 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.733+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3170003cd0 con 0x7f31781f6a70 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.733+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170005290 con 0x7f31781f6a70 2024-01-16T15:10:54.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.733+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31780ada40 con 0x7f31781f6a70 2024-01-16T15:10:54.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.733+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31781f6ed0 con 0x7f31781f6a70 2024-01-16T15:10:54.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.734+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31780adf60 con 0x7f31781f6a70 2024-01-16T15:10:54.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.735+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f31700053f0 con 0x7f31781f6a70 2024-01-16T15:10:54.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.736+0000 7f3175600640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 0x7f315006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.736+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f31700882c0 con 0x7f31781f6a70 2024-01-16T15:10:54.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.736+0000 7f317da00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 0x7f315006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.737+0000 7f317da00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 0x7f315006e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f316800c240 tx=0x7f3168009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.741+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31780adf60 con 0x7f31781f6a70 2024-01-16T15:10:54.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:54.971+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f31780ce0b0 con 0x7f315006c100 2024-01-16T15:10:57.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.331+0000 7f3175600640 1 -- 172.21.15.125:0/680640891 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3168005da0 con 0x7f315006c100 2024-01-16T15:10:57.308 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 msgr2=0x7f315006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 0x7f315006e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f316800c240 tx=0x7f3168009030 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 msgr2=0x7f3178156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3170000f80 tx=0x7f317000dc00 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 shutdown_connections 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f315006c100 0x7f315006e5c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31781f6a70 0x7f3178156cb0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31780ae240 0x7f317815d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 --2- 172.21.15.125:0/680640891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31580039a0 0x7f31780acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 >> 172.21.15.125:0/680640891 conn(0x7f317806a7d0 msgr2=0x7f31781598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 shutdown_connections 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.336+0000 7f317ee00640 1 -- 172.21.15.125:0/680640891 wait complete. 2024-01-16T15:10:57.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:10:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.566+0000 7f1d82e00640 1 Processor -- start 2024-01-16T15:10:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.566+0000 7f1d82e00640 1 -- start start 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.567+0000 7f1d82e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c143ae0 0x7f1d7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.567+0000 7f1d82e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.567+0000 7f1d82e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1d7c15e1e0 0x7f1d7c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.567+0000 7f1d82e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cfc30 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d82e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b1ee0 con 0x7f1d7c15e1e0 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d82e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b2780 con 0x7f1d7c143ae0 2024-01-16T15:10:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40776/0 (socket says 172.21.15.125:40776) 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 -- 172.21.15.125:0/297963268 learned_addr learned my addr 172.21.15.125:0/297963268 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 -- 172.21.15.125:0/297963268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c143ae0 msgr2=0x7f1d7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 --2- 172.21.15.125:0/297963268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c143ae0 0x7f1d7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 -- 172.21.15.125:0/297963268 >> v2:172.21.15.137:3300/0 conn(0x7f1d7c15e1e0 msgr2=0x7f1d7c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 --2- 172.21.15.125:0/297963268 >> v2:172.21.15.137:3300/0 conn(0x7f1d7c15e1e0 0x7f1d7c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.568+0000 7f1d7b400640 1 -- 172.21.15.125:0/297963268 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c15e990 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.569+0000 7f1d7b400640 1 --2- 172.21.15.125:0/297963268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c072160 tx=0x7f1d6402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eea58349b1f2f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d7aa00640 1 -- 172.21.15.125:0/297963268 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6402f8e0 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d7aa00640 1 -- 172.21.15.125:0/297963268 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6402fa40 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d7aa00640 1 -- 172.21.15.125:0/297963268 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6402fdd0 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 -- 172.21.15.125:0/297963268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c0cf7d0 msgr2=0x7f1d7c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 --2- 172.21.15.125:0/297963268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c072160 tx=0x7f1d6402db20 comp rx=0 tx=0).stop 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 -- 172.21.15.125:0/297963268 shutdown_connections 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 --2- 172.21.15.125:0/297963268 >> v2:172.21.15.137:3300/0 conn(0x7f1d7c15e1e0 0x7f1d7c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 --2- 172.21.15.125:0/297963268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c0cf7d0 0x7f1d7c15dca0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 --2- 172.21.15.125:0/297963268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c143ae0 0x7f1d7c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 -- 172.21.15.125:0/297963268 >> 172.21.15.125:0/297963268 conn(0x7f1d7c06a800 msgr2=0x7f1d7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 -- 172.21.15.125:0/297963268 shutdown_connections 2024-01-16T15:10:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.570+0000 7f1d82e00640 1 -- 172.21.15.125:0/297963268 wait complete. 2024-01-16T15:10:57.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.572+0000 7f1d82e00640 1 Processor -- start 2024-01-16T15:10:57.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.572+0000 7f1d82e00640 1 -- start start 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0cf7d0 0x7f1d7c0cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 0x7f1d7c0b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1363e0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1366c0 con 0x7f1d7c15e1e0 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d82e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1369a0 con 0x7f1d7c0cf7d0 2024-01-16T15:10:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40782/0 (socket says 172.21.15.125:40782) 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.573+0000 7f1d7b400640 1 -- 172.21.15.125:0/636848833 learned_addr learned my addr 172.21.15.125:0/636848833 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7be00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0cf7d0 0x7f1d7c0cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d80e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 0x7f1d7c0b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 -- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0cf7d0 msgr2=0x7f1d7c0cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0cf7d0 0x7f1d7c0cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 -- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 msgr2=0x7f1d7c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 0x7f1d7c0b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 -- 172.21.15.125:0/636848833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640094d0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d80e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 0x7f1d7c0b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.574+0000 7f1d7b400640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1d64000c00 tx=0x7f1d640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.575+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64037d30 con 0x7f1d7c143ae0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.575+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64037e90 con 0x7f1d7c143ae0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.575+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c136c80 con 0x7f1d7c143ae0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.575+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64042690 con 0x7f1d7c143ae0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.575+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c0cb520 con 0x7f1d7c143ae0 2024-01-16T15:10:57.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.576+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c1371a0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.577+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f1d640427f0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.577+0000 7f1d73e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 0x7f1d5406e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.578+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1d64033070 con 0x7f1d7c143ae0 2024-01-16T15:10:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.578+0000 7f1d7be00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 0x7f1d5406e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.578+0000 7f1d7be00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 0x7f1d5406e5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c15d7d0 tx=0x7f1d6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.582+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d7c1371a0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.848+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1d7c1371a0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.849+0000 7f1d73e00640 1 -- 172.21.15.125:0/636848833 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1853 (secure 0 0 0) 0x7f1d7c1371a0 con 0x7f1d7c143ae0 2024-01-16T15:10:57.825 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:10:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 876, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 88, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:10:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:10:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 799, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222156819, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 7032090624, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766062022656, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 341, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 12368, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 3 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 55, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 6, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:10:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 6, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "c", 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 8181 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:10:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 16, 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:10:48.250760+0000", 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": {} 2024-01-16T15:10:57.831 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:10:57.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 msgr2=0x7f1d5406e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 0x7f1d5406e5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c15d7d0 tx=0x7f1d6c008040 comp rx=0 tx=0).stop 2024-01-16T15:10:57.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 msgr2=0x7f1d7c0b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1d64000c00 tx=0x7f1d640046c0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 shutdown_connections 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c15e1e0 0x7f1d7c0b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1d5406c100 0x7f1d5406e5c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d7c143ae0 0x7f1d7c0b0bc0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 --2- 172.21.15.125:0/636848833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0cf7d0 0x7f1d7c0cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 >> 172.21.15.125:0/636848833 conn(0x7f1d7c06a800 msgr2=0x7f1d7c15b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 shutdown_connections 2024-01-16T15:10:57.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:57.852+0000 7f1d82e00640 1 -- 172.21.15.125:0/636848833 wait complete. 2024-01-16T15:10:57.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:57.847 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e) 2024-01-16T15:10:57.902 INFO:teuthology.orchestra.run.smithi125.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/29f4d3e2-2338-4adf-ae67-a7fa15bc189e': No such file or directory 2024-01-16T15:10:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:10:57.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:10:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.129+0000 7fd031800640 1 Processor -- start 2024-01-16T15:10:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.129+0000 7fd031800640 1 -- start start 2024-01-16T15:10:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd031800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd02c0d1650 0x7fd02c0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd031800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0cec40 con 0x7fd02c0c6be0 2024-01-16T15:10:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd031800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0d0420 con 0x7fd02c0b34e0 2024-01-16T15:10:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd031800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c0d2010 con 0x7fd02c0d1650 2024-01-16T15:10:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd02b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd02c0d1650 0x7fd02c0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd02b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd02c0d1650 0x7fd02c0d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40796/0 (socket says 172.21.15.125:40796) 2024-01-16T15:10:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.131+0000 7fd02b400640 1 -- 172.21.15.125:0/2040749379 learned_addr learned my addr 172.21.15.125:0/2040749379 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.132+0000 7fd02b400640 1 -- 172.21.15.125:0/2040749379 >> v1:172.21.15.137:6789/0 conn(0x7fd02c0b34e0 legacy=0x7fd02c0b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.132+0000 7fd02b400640 1 -- 172.21.15.125:0/2040749379 >> v1:172.21.15.125:6789/0 conn(0x7fd02c0c6be0 legacy=0x7fd02c0c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.132+0000 7fd02aa00640 1 -- 172.21.15.125:0/2040749379 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472036747 0 0) 0x7fd02c0cec40 con 0x7fd02c0c6be0 2024-01-16T15:10:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.132+0000 7fd02b400640 1 -- 172.21.15.125:0/2040749379 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0d0e70 con 0x7fd02c0d1650 2024-01-16T15:10:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.133+0000 7fd02b400640 1 --2- 172.21.15.125:0/2040749379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0d1650 0x7fd02c0d1ad0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd014009870 tx=0x7fd01402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f4a12d93328e6d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.133+0000 7fd02aa00640 1 -- 172.21.15.125:0/2040749379 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd014030860 con 0x7fd02c0d1650 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.133+0000 7fd02aa00640 1 -- 172.21.15.125:0/2040749379 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0140309c0 con 0x7fd02c0d1650 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.133+0000 7fd02aa00640 1 -- 172.21.15.125:0/2040749379 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd014030d50 con 0x7fd02c0d1650 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 -- 172.21.15.125:0/2040749379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0d1650 msgr2=0x7fd02c0d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 --2- 172.21.15.125:0/2040749379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0d1650 0x7fd02c0d1ad0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd014009870 tx=0x7fd01402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 -- 172.21.15.125:0/2040749379 shutdown_connections 2024-01-16T15:10:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 --2- 172.21.15.125:0/2040749379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0d1650 0x7fd02c0d1ad0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 -- 172.21.15.125:0/2040749379 >> 172.21.15.125:0/2040749379 conn(0x7fd02c0d3090 msgr2=0x7fd02c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 -- 172.21.15.125:0/2040749379 shutdown_connections 2024-01-16T15:10:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.134+0000 7fd031800640 1 -- 172.21.15.125:0/2040749379 wait complete. 2024-01-16T15:10:58.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.136+0000 7fd031800640 1 Processor -- start 2024-01-16T15:10:58.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.136+0000 7fd031800640 1 -- start start 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd02c0c6be0 0x7fd02c15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd02c0d1650 0x7fd02c0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c15ac20 con 0x7fd02c0b34e0 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c15ae10 con 0x7fd02c0d1650 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd031800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c15b0f0 con 0x7fd02c0c6be0 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40798/0 (socket says 172.21.15.125:40798) 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 -- 172.21.15.125:0/3096002666 learned_addr learned my addr 172.21.15.125:0/3096002666 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd030e00640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd02c0d1650 0x7fd02c0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 -- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd02c0c6be0 msgr2=0x7fd02c15a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd02c0c6be0 0x7fd02c15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 -- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd02c0d1650 msgr2=0x7fd02c0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd02c0d1650 0x7fd02c0ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.138+0000 7fd02be00640 1 -- 172.21.15.125:0/3096002666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140094d0 con 0x7fd02c0b34e0 2024-01-16T15:10:58.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd02be00640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c430 tx=0x7fd01c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c011790 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c0118f0 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c011ce0 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c15b3d0 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.139+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c0b6790 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.140+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c0d1350 con 0x7fd02c0b34e0 2024-01-16T15:10:58.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.141+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fd01c01c460 con 0x7fd02c0b34e0 2024-01-16T15:10:58.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.143+0000 7fd023e00640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 0x7fd00406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.143+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd01c0888f0 con 0x7fd02c0b34e0 2024-01-16T15:10:58.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.143+0000 7fd02b400640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 0x7fd00406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.144+0000 7fd02b400640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 0x7fd00406e590 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd014036000 tx=0x7fd01403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.147+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd02c0d1350 con 0x7fd02c0b34e0 2024-01-16T15:10:58.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.363+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd02c15b260 con 0x7fd00406c0d0 2024-01-16T15:10:58.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.366+0000 7fd023e00640 1 -- 172.21.15.125:0/3096002666 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd02c15b260 con 0x7fd00406c0d0 2024-01-16T15:10:58.342 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:10:58.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:58.343 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:10:58.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:58.343 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:10:58.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 msgr2=0x7fd00406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 0x7fd00406e590 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd014036000 tx=0x7fd01403b040 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 msgr2=0x7fd02c160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c430 tx=0x7fd01c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 shutdown_connections 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd02c0d1650 0x7fd02c0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd02c0c6be0 0x7fd02c15a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd00406c0d0 0x7fd00406e590 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 --2- 172.21.15.125:0/3096002666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd02c0b34e0 0x7fd02c160000 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 >> 172.21.15.125:0/3096002666 conn(0x7fd02c0d3090 msgr2=0x7fd02c156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 shutdown_connections 2024-01-16T15:10:58.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.369+0000 7fd031800640 1 -- 172.21.15.125:0/3096002666 wait complete. 2024-01-16T15:10:58.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:10:58.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.595+0000 7f0918600640 1 Processor -- start 2024-01-16T15:10:58.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.596+0000 7f0918600640 1 -- start start 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.597+0000 7f0918600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f09100aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.597+0000 7f0918600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0910071de0 con 0x7f091006a3e0 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.597+0000 7f0918600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0910072030 con 0x7f09100d0b60 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.597+0000 7f0918600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09100acd60 con 0x7f09100ac8d0 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f0915c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f091006a3e0 0x7f091006a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50352/0 (socket says 172.21.15.125:50352) 2024-01-16T15:10:58.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f0915c00640 1 -- 172.21.15.125:0/1511397180 learned_addr learned my addr 172.21.15.125:0/1511397180 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144413028 0 0) 0x7f0910071de0 con 0x7f091006a3e0 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0003520 con 0x7f091006a3e0 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f0916600640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f09100aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310973443 0 0) 0x7f08f0003520 con 0x7f091006a3e0 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 msgr2=0x7f09100aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f09100aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.598+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 >> v1:172.21.15.137:6789/0 conn(0x7f09100d0b60 legacy=0x7f091006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910072310 con 0x7f091006a3e0 2024-01-16T15:10:58.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0900001670 con 0x7f091006a3e0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 >> v1:172.21.15.125:6789/0 conn(0x7f091006a3e0 legacy=0x7f091006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f08f00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f0003fd0 0x7f08f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0004d00 0x7f08f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09100acd60 con 0x7f08f0003fd0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f000bb40 con 0x7f08f0004d00 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f000bd00 con 0x7f09100ac8d0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f0915200640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f0003fd0 0x7f08f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f0915c00640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f08f00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f0916600640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0004d00 0x7f08f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.599+0000 7f0915200640 1 -- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 msgr2=0x7f08f00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.600+0000 7f0915200640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09100ac8d0 0x7f08f00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.600+0000 7f0915200640 1 -- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0004d00 msgr2=0x7f08f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.600+0000 7f0915200640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0004d00 0x7f08f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.600+0000 7f0915200640 1 -- 172.21.15.125:0/1511397180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f0003520 con 0x7f08f0003fd0 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0915200640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f0003fd0 0x7f08f00047c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f08f8001720 tx=0x7f08f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39726513a4cb3202 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80052d0 con 0x7f08f0003fd0 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f802e050 con 0x7f08f0003fd0 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f090fe00640 1 -- 172.21.15.125:0/1511397180 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8003830 con 0x7f08f0003fd0 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0918600640 1 -- 172.21.15.125:0/1511397180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f0003fd0 msgr2=0x7f08f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0918600640 1 --2- 172.21.15.125:0/1511397180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f0003fd0 0x7f08f00047c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f08f8001720 tx=0x7f08f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0916600640 1 -- 172.21.15.125:0/1511397180 reap_dead start 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0918600640 1 -- 172.21.15.125:0/1511397180 shutdown_connections 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.601+0000 7f0918600640 1 -- 172.21.15.125:0/1511397180 >> 172.21.15.125:0/1511397180 conn(0x7f09100cfef0 msgr2=0x7f09100d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.602+0000 7f0918600640 1 -- 172.21.15.125:0/1511397180 shutdown_connections 2024-01-16T15:10:58.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.602+0000 7f0918600640 1 -- 172.21.15.125:0/1511397180 wait complete. 2024-01-16T15:10:58.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.603+0000 7f0918600640 1 Processor -- start 2024-01-16T15:10:58.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.603+0000 7f0918600640 1 -- start start 2024-01-16T15:10:58.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0003fd0 0x7f0910158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0910154230 0x7f0910154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910159170 con 0x7f091013b7f0 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910154bd0 con 0x7f08f0003fd0 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.604+0000 7f0918600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09101587a0 con 0x7f0910154230 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40824/0 (socket says 172.21.15.125:40824) 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 -- 172.21.15.125:0/2964682830 learned_addr learned my addr 172.21.15.125:0/2964682830 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0915c00640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0003fd0 0x7f0910158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0915200640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0910154230 0x7f0910154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 -- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0910154230 msgr2=0x7f0910154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0910154230 0x7f0910154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 -- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0003fd0 msgr2=0x7f0910158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0003fd0 0x7f0910158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.605+0000 7f0916600640 1 -- 172.21.15.125:0/2964682830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910157b90 con 0x7f091013b7f0 2024-01-16T15:10:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f0916600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f0908009910 tx=0x7f0908009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908007de0 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090800f050 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908014650 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0910157e70 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.606+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0910158430 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.607+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09100675d0 con 0x7f091013b7f0 2024-01-16T15:10:58.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.608+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f09080147b0 con 0x7f091013b7f0 2024-01-16T15:10:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.609+0000 7f090d600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 0x7f08e806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.610+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0908088dd0 con 0x7f091013b7f0 2024-01-16T15:10:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.610+0000 7f0915c00640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 0x7f08e806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.610+0000 7f0915c00640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 0x7f08e806e5c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0900001c90 tx=0x7f0900009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:58.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.614+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0908055430 con 0x7f091013b7f0 2024-01-16T15:10:58.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.829+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f091006eb60 con 0x7f08e806c100 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.837+0000 7f090d600640 1 -- 172.21.15.125:0/2964682830 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f0900010860 con 0x7f08e806c100 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1clone1" 2024-01-16T15:10:58.814 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:58.815 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:10:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 msgr2=0x7f08e806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 0x7f08e806e5c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0900001c90 tx=0x7f0900009040 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 msgr2=0x7f091013bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f0908009910 tx=0x7f0908009dd0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 shutdown_connections 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08e806c100 0x7f08e806e5c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091013b7f0 0x7f091013bc70 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0910154230 0x7f0910154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 --2- 172.21.15.125:0/2964682830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08f0003fd0 0x7f0910158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 >> 172.21.15.125:0/2964682830 conn(0x7f09100cfef0 msgr2=0x7f091015a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 shutdown_connections 2024-01-16T15:10:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:58.841+0000 7f0918600640 1 -- 172.21.15.125:0/2964682830 wait complete. 2024-01-16T15:10:58.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:10:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.072+0000 7fbd7ac00640 1 Processor -- start 2024-01-16T15:10:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.072+0000 7fbd7ac00640 1 -- start start 2024-01-16T15:10:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd7ac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbd7413df80 0x7fbd7415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd7ac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbd740cf9a0 0x7fbd740cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd7ac00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7406c470 con 0x7fbd7413e360 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd7ac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7406c750 con 0x7fbd7413df80 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd7ac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7406ca30 con 0x7fbd740cf9a0 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd73400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbd7413e360 0x7fbd740cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50356/0 (socket says 172.21.15.125:50356) 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd73400640 1 -- 172.21.15.125:0/993603018 learned_addr learned my addr 172.21.15.125:0/993603018 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd73e00640 1 --2- 172.21.15.125:0/993603018 >> v2:172.21.15.125:3300/0 conn(0x7fbd7413df80 0x7fbd7415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.074+0000 7fbd78c00640 1 --2- 172.21.15.125:0/993603018 >> v2:172.21.15.137:3300/0 conn(0x7fbd740cf9a0 0x7fbd740cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.075+0000 7fbd73e00640 1 -- 172.21.15.125:0/993603018 >> v2:172.21.15.137:3300/0 conn(0x7fbd740cf9a0 msgr2=0x7fbd740cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.075+0000 7fbd73e00640 1 --2- 172.21.15.125:0/993603018 >> v2:172.21.15.137:3300/0 conn(0x7fbd740cf9a0 0x7fbd740cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.075+0000 7fbd73e00640 1 -- 172.21.15.125:0/993603018 >> v1:172.21.15.125:6789/0 conn(0x7fbd7413e360 legacy=0x7fbd740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.075+0000 7fbd72a00640 1 -- 172.21.15.125:0/993603018 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195967060 0 0) 0x7fbd7406c470 con 0x7fbd7413e360 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.075+0000 7fbd73e00640 1 -- 172.21.15.125:0/993603018 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7406cd10 con 0x7fbd7413df80 2024-01-16T15:10:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.076+0000 7fbd73e00640 1 --2- 172.21.15.125:0/993603018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413df80 0x7fbd7415bee0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbd740ccb10 tx=0x7fbd6402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9be5d649b098f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.076+0000 7fbd72a00640 1 -- 172.21.15.125:0/993603018 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6402cc80 con 0x7fbd7413df80 2024-01-16T15:10:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.076+0000 7fbd72a00640 1 -- 172.21.15.125:0/993603018 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6402cde0 con 0x7fbd7413df80 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.076+0000 7fbd72a00640 1 -- 172.21.15.125:0/993603018 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64034660 con 0x7fbd7413df80 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 -- 172.21.15.125:0/993603018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413df80 msgr2=0x7fbd7415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/993603018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413df80 0x7fbd7415bee0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbd740ccb10 tx=0x7fbd6402bed0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 -- 172.21.15.125:0/993603018 shutdown_connections 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/993603018 >> v2:172.21.15.137:3300/0 conn(0x7fbd740cf9a0 0x7fbd740cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/993603018 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413df80 0x7fbd7415bee0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 -- 172.21.15.125:0/993603018 >> 172.21.15.125:0/993603018 conn(0x7fbd7406a800 msgr2=0x7fbd7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 -- 172.21.15.125:0/993603018 shutdown_connections 2024-01-16T15:10:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.077+0000 7fbd7ac00640 1 -- 172.21.15.125:0/993603018 wait complete. 2024-01-16T15:10:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.079+0000 7fbd7ac00640 1 Processor -- start 2024-01-16T15:10:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.079+0000 7fbd7ac00640 1 -- start start 2024-01-16T15:10:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd740cf9a0 0x7fbd740b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd7413df80 0x7fbd74153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7413cf10 con 0x7fbd7413e360 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740b2cf0 con 0x7fbd7413df80 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd7ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740b2f40 con 0x7fbd740cf9a0 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd78c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd78c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40856/0 (socket says 172.21.15.125:40856) 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd78c00640 1 -- 172.21.15.125:0/4281640406 learned_addr learned my addr 172.21.15.125:0/4281640406 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.080+0000 7fbd73400640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd7413df80 0x7fbd74153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd73e00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd740cf9a0 0x7fbd740b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 -- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd740cf9a0 msgr2=0x7fbd740b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd740cf9a0 0x7fbd740b3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 -- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd7413df80 msgr2=0x7fbd74153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd7413df80 0x7fbd74153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 -- 172.21.15.125:0/4281640406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd64009a70 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd78c00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00d5a0 tx=0x7fbd6c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0125f0 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c012770 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c012b20 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd740b3220 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.081+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd740b37e0 con 0x7fbd7413e360 2024-01-16T15:10:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.082+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd740675d0 con 0x7fbd7413e360 2024-01-16T15:10:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.083+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fbd6c012c80 con 0x7fbd7413e360 2024-01-16T15:10:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.084+0000 7fbd6be00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 0x7fbd4c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.084+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbd6c088e40 con 0x7fbd7413e360 2024-01-16T15:10:59.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.084+0000 7fbd73e00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 0x7fbd4c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.085+0000 7fbd73e00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 0x7fbd4c06e590 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbd64036000 tx=0x7fbd64030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.089+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd6c0553c0 con 0x7fbd7413e360 2024-01-16T15:10:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.311+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd7415f9c0 con 0x7fbd4c06c0d0 2024-01-16T15:10:59.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.340+0000 7fbd6be00640 1 -- 172.21.15.125:0/4281640406 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fbd640345b0 con 0x7fbd4c06c0d0 2024-01-16T15:10:59.317 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:59.317 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:10:59.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:10:59.318 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:10:59.318 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:10:59.318 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:10:59.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.343+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 msgr2=0x7fbd4c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.343+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 0x7fbd4c06e590 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbd64036000 tx=0x7fbd64030000 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.343+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 msgr2=0x7fbd740adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.343+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00d5a0 tx=0x7fbd6c00da60 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 shutdown_connections 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbd4c06c0d0 0x7fbd4c06e590 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7413e360 0x7fbd740adb70 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd7413df80 0x7fbd74153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 --2- 172.21.15.125:0/4281640406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd740cf9a0 0x7fbd740b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 >> 172.21.15.125:0/4281640406 conn(0x7fbd7406a800 msgr2=0x7fbd740afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 shutdown_connections 2024-01-16T15:10:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.344+0000 7fbd7ac00640 1 -- 172.21.15.125:0/4281640406 wait complete. 2024-01-16T15:10:59.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:10:59.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.629+0000 7f4cdf800640 1 Processor -- start 2024-01-16T15:10:59.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.629+0000 7f4cdf800640 1 -- start start 2024-01-16T15:10:59.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdf800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd813df80 0x7f4cd815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdf800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4cd80cf9a0 0x7f4cd80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdf800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd806c470 con 0x7f4cd813e360 2024-01-16T15:10:59.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdf800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd806c750 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdf800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd806ca30 con 0x7f4cd813df80 2024-01-16T15:10:59.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cd7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4cd813e360 0x7f4cd80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50366/0 (socket says 172.21.15.125:50366) 2024-01-16T15:10:59.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cd7e00640 1 -- 172.21.15.125:0/901566049 learned_addr learned my addr 172.21.15.125:0/901566049 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdd800640 1 --2- 172.21.15.125:0/901566049 >> v2:172.21.15.125:3300/0 conn(0x7f4cd80cf9a0 0x7f4cd80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.631+0000 7f4cdce00640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd813df80 0x7f4cd815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856142483 0 0) 0x7f4cd806c470 con 0x7f4cd813e360 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb8003520 con 0x7f4cd813e360 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818279055 0 0) 0x7f4cb8003520 con 0x7f4cd813e360 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cdd800640 1 -- 172.21.15.125:0/901566049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd813df80 msgr2=0x7f4cd815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cdd800640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd813df80 0x7f4cd815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cdd800640 1 -- 172.21.15.125:0/901566049 >> v1:172.21.15.125:6789/0 conn(0x7f4cd813e360 legacy=0x7f4cd80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.632+0000 7f4cdd800640 1 -- 172.21.15.125:0/901566049 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd806cd10 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.633+0000 7f4cdd800640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd80cf9a0 0x7f4cd80cd840 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009d90 tx=0x7f4cd002bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65247f8069cd0afd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.633+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0001670 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd002c8a0 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cd7400640 1 -- 172.21.15.125:0/901566049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd002cc10 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 -- 172.21.15.125:0/901566049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd80cf9a0 msgr2=0x7f4cd80cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd80cf9a0 0x7f4cd80cd840 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009d90 tx=0x7f4cd002bd70 comp rx=0 tx=0).stop 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 -- 172.21.15.125:0/901566049 shutdown_connections 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd80cf9a0 0x7f4cd80cd840 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 --2- 172.21.15.125:0/901566049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd813df80 0x7f4cd815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 -- 172.21.15.125:0/901566049 >> 172.21.15.125:0/901566049 conn(0x7f4cd806a800 msgr2=0x7f4cd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 -- 172.21.15.125:0/901566049 shutdown_connections 2024-01-16T15:10:59.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.634+0000 7f4cdf800640 1 -- 172.21.15.125:0/901566049 wait complete. 2024-01-16T15:10:59.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.636+0000 7f4cdf800640 1 Processor -- start 2024-01-16T15:10:59.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.636+0000 7f4cdf800640 1 -- start start 2024-01-16T15:10:59.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd80cf9a0 0x7f4cd814bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 0x7f4cd8152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd814dad0 con 0x7f4cd813e360 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd81f3130 con 0x7f4cd813df80 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.637+0000 7f4cdf800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd8153940 con 0x7f4cd80cf9a0 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40884/0 (socket says 172.21.15.125:40884) 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 -- 172.21.15.125:0/336556879 learned_addr learned my addr 172.21.15.125:0/336556879 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cd7e00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 0x7f4cd8152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 -- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd80cf9a0 msgr2=0x7f4cd814bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdce00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd80cf9a0 0x7f4cd814bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd80cf9a0 0x7f4cd814bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 -- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 msgr2=0x7f4cd8152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 0x7f4cd8152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 -- 172.21.15.125:0/336556879 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0009a70 con 0x7f4cd813e360 2024-01-16T15:10:59.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cd7e00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 0x7f4cd8152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:59.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.638+0000 7f4cdd800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4cd002c380 tx=0x7f4cd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.639+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0007000 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.639+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd0009450 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.639+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd81530a0 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.639+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0004490 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.639+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd8153660 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.640+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd814dc10 con 0x7f4cd813e360 2024-01-16T15:10:59.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.641+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f4cd003b070 con 0x7f4cd813e360 2024-01-16T15:10:59.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.642+0000 7f4cd4c00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 0x7f4cb006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:59.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.642+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4cd00aac70 con 0x7f4cd813e360 2024-01-16T15:10:59.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.642+0000 7f4cdce00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 0x7f4cb006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:59.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.643+0000 7f4cdce00640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 0x7f4cb006e590 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009a10 tx=0x7f4cc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:59.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.646+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cd814dc10 con 0x7f4cd813e360 2024-01-16T15:10:59.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.871+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cd815f9c0 con 0x7f4cb006c0d0 2024-01-16T15:10:59.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.925+0000 7f4cd4c00640 1 -- 172.21.15.125:0/336556879 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4cc8010870 con 0x7f4cb006c0d0 2024-01-16T15:10:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:10:59.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.928+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 msgr2=0x7f4cb006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.928+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 0x7f4cb006e590 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009a10 tx=0x7f4cc8008040 comp rx=0 tx=0).stop 2024-01-16T15:10:59.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.928+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 msgr2=0x7f4cd8152a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:59.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.928+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4cd002c380 tx=0x7f4cd0004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 shutdown_connections 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4cb006c0d0 0x7f4cb006e590 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4cd813e360 0x7f4cd8152a90 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cd813df80 0x7f4cd8152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 --2- 172.21.15.125:0/336556879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4cd80cf9a0 0x7f4cd814bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 >> 172.21.15.125:0/336556879 conn(0x7f4cd806a800 msgr2=0x7f4cd80b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 shutdown_connections 2024-01-16T15:10:59.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:10:59.929+0000 7f4cdf800640 1 -- 172.21.15.125:0/336556879 wait complete. 2024-01-16T15:10:59.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:11:00.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.153+0000 7f9a56200640 1 Processor -- start 2024-01-16T15:11:00.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.153+0000 7f9a56200640 1 -- start start 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a56200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9a500ccb50 0x7f9a50070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a56200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5015f4e0 con 0x7f9a500adf20 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a56200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a50071130 con 0x7f9a5015bf10 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a56200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5015f970 con 0x7f9a500ccb50 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a54e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9a500adf20 0x7f9a500cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50374/0 (socket says 172.21.15.125:50374) 2024-01-16T15:11:00.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a54e00640 1 -- 172.21.15.125:0/4209765579 learned_addr learned my addr 172.21.15.125:0/4209765579 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:00.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.155+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244186491 0 0) 0x7f9a5015f4e0 con 0x7f9a500adf20 2024-01-16T15:11:00.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a30003520 con 0x7f9a500adf20 2024-01-16T15:11:00.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4fe00640 1 --2- 172.21.15.125:0/4209765579 >> v2:172.21.15.137:3300/0 conn(0x7f9a500ccb50 0x7f9a50070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113443117 0 0) 0x7f9a30003520 con 0x7f9a500adf20 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 >> v2:172.21.15.137:3300/0 conn(0x7f9a500ccb50 msgr2=0x7f9a50070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 --2- 172.21.15.125:0/4209765579 >> v2:172.21.15.137:3300/0 conn(0x7f9a500ccb50 0x7f9a50070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 >> v1:172.21.15.137:6789/0 conn(0x7f9a5015bf10 legacy=0x7f9a50143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5015ebb0 con 0x7f9a500adf20 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f9a40001670 con 0x7f9a500adf20 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 >> v1:172.21.15.125:6789/0 conn(0x7f9a500adf20 legacy=0x7f9a500cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a300047c0 0x7f9a30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a300052a0 0x7f9a3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5015f970 con 0x7f9a300039c0 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3000c1c0 con 0x7f9a300047c0 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3000c380 con 0x7f9a300052a0 2024-01-16T15:11:00.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.156+0000 7f9a54e00640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a4fe00640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a300047c0 0x7f9a30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a55800640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a300052a0 0x7f9a3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a54e00640 1 -- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a300052a0 msgr2=0x7f9a3000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a54e00640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a300052a0 0x7f9a3000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a54e00640 1 -- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a300047c0 msgr2=0x7f9a30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a54e00640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a300047c0 0x7f9a30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a54e00640 1 -- 172.21.15.125:0/4209765579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30003520 con 0x7f9a300039c0 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a55800640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a300052a0 0x7f9a3000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:00.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.157+0000 7f9a55800640 1 -- 172.21.15.125:0/4209765579 reap_dead start 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.158+0000 7f9a54e00640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a30003de0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000c0e0 tx=0x7f9a300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb1b324da0a68d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.158+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40005ad0 con 0x7f9a300039c0 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.158+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a40005c30 con 0x7f9a300039c0 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.158+0000 7f9a4f400640 1 -- 172.21.15.125:0/4209765579 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4002c7d0 con 0x7f9a300039c0 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 -- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 msgr2=0x7f9a30003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a30003de0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000c0e0 tx=0x7f9a300057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 -- 172.21.15.125:0/4209765579 shutdown_connections 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 --2- 172.21.15.125:0/4209765579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a30003de0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 -- 172.21.15.125:0/4209765579 >> 172.21.15.125:0/4209765579 conn(0x7f9a5006a7d0 msgr2=0x7f9a5006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 -- 172.21.15.125:0/4209765579 shutdown_connections 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.159+0000 7f9a56200640 1 -- 172.21.15.125:0/4209765579 wait complete. 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.161+0000 7f9a56200640 1 Processor -- start 2024-01-16T15:11:00.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.161+0000 7f9a56200640 1 -- start start 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a500b0280 0x7f9a5015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a5013b5d0 0x7f9a5013ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5015ac30 con 0x7f9a300039c0 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5013bf90 con 0x7f9a500b0280 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a56200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5013e690 con 0x7f9a5013b5d0 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a54e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a54e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40900/0 (socket says 172.21.15.125:40900) 2024-01-16T15:11:00.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a54e00640 1 -- 172.21.15.125:0/1376564994 learned_addr learned my addr 172.21.15.125:0/1376564994 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.162+0000 7f9a54e00640 1 -- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a5013b5d0 msgr2=0x7f9a5013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a54e00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a5013b5d0 0x7f9a5013ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a54e00640 1 -- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a500b0280 msgr2=0x7f9a5015a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a54e00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a500b0280 0x7f9a5015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a54e00640 1 -- 172.21.15.125:0/1376564994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5013da80 con 0x7f9a300039c0 2024-01-16T15:11:00.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a54e00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9a4002c370 tx=0x7f9a5015b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4003c070 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4002c9e0 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5013dd60 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4002cd70 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.163+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5013e320 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.164+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a50067450 con 0x7f9a300039c0 2024-01-16T15:11:00.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.165+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f9a40032070 con 0x7f9a300039c0 2024-01-16T15:11:00.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.166+0000 7f9a4cc00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 0x7f9a2806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.166+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a400a9a90 con 0x7f9a300039c0 2024-01-16T15:11:00.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.166+0000 7f9a4fe00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 0x7f9a2806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.167+0000 7f9a4fe00640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 0x7f9a2806e590 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9a38006d40 tx=0x7f9a38006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.171+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a40076060 con 0x7f9a300039c0 2024-01-16T15:11:00.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.389+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a5015c380 con 0x7f9a2806c0d0 2024-01-16T15:11:00.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.421+0000 7f9a4cc00640 1 -- 172.21.15.125:0/1376564994 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9a3800d4e0 con 0x7f9a2806c0d0 2024-01-16T15:11:00.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 msgr2=0x7f9a2806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 0x7f9a2806e590 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9a38006d40 tx=0x7f9a38006bd0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 msgr2=0x7f9a5013d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9a4002c370 tx=0x7f9a5015b490 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 shutdown_connections 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a5013b5d0 0x7f9a5013ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a500b0280 0x7f9a5015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9a2806c0d0 0x7f9a2806e590 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 --2- 172.21.15.125:0/1376564994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a300039c0 0x7f9a5013d4a0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 >> 172.21.15.125:0/1376564994 conn(0x7f9a5006a7d0 msgr2=0x7f9a5015c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 shutdown_connections 2024-01-16T15:11:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.425+0000 7f9a56200640 1 -- 172.21.15.125:0/1376564994 wait complete. 2024-01-16T15:11:00.405 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.429+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.405 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.429+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.405 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.429+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.405 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.429+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.405 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.429+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:11:00.425 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.449+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.426 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.449+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.426 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.449+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.426 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.449+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.426 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:00.449+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.626+0000 7ff89ba00640 1 Processor -- start 2024-01-16T15:11:00.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.626+0000 7ff89ba00640 1 -- start start 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff89ba00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff8940ceae0 0x7ff8940cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff89ba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff894141d30 0x7ff894142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff89ba00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89406b170 con 0x7ff8940cb860 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff89ba00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89415b500 con 0x7ff8940ceae0 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff89ba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89406bd10 con 0x7ff894141d30 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff893e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff8940ceae0 0x7ff8940cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff899000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff8940cb860 0x7ff89406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50376/0 (socket says 172.21.15.125:50376) 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff899000640 1 -- 172.21.15.125:0/301361211 learned_addr learned my addr 172.21.15.125:0/301361211 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff899a00640 1 --2- 172.21.15.125:0/301361211 >> v2:172.21.15.137:3300/0 conn(0x7ff894141d30 0x7ff894142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff893e00640 1 -- 172.21.15.125:0/301361211 >> v2:172.21.15.137:3300/0 conn(0x7ff894141d30 msgr2=0x7ff894142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.628+0000 7ff893e00640 1 --2- 172.21.15.125:0/301361211 >> v2:172.21.15.137:3300/0 conn(0x7ff894141d30 0x7ff894142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.629+0000 7ff893e00640 1 -- 172.21.15.125:0/301361211 >> v1:172.21.15.125:6789/0 conn(0x7ff8940cb860 legacy=0x7ff89406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.629+0000 7ff893e00640 1 -- 172.21.15.125:0/301361211 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940cf4b0 con 0x7ff8940ceae0 2024-01-16T15:11:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.630+0000 7ff893e00640 1 --2- 172.21.15.125:0/301361211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff8940cef70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ff894070ad0 tx=0x7ff87c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16dd95cf7fb6d596 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.630+0000 7ff893400640 1 -- 172.21.15.125:0/301361211 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c031830 con 0x7ff8940ceae0 2024-01-16T15:11:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff893400640 1 -- 172.21.15.125:0/301361211 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c031990 con 0x7ff8940ceae0 2024-01-16T15:11:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff893400640 1 -- 172.21.15.125:0/301361211 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c031d20 con 0x7ff8940ceae0 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 -- 172.21.15.125:0/301361211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 msgr2=0x7ff8940cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 --2- 172.21.15.125:0/301361211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff8940cef70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ff894070ad0 tx=0x7ff87c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 -- 172.21.15.125:0/301361211 shutdown_connections 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 --2- 172.21.15.125:0/301361211 >> v2:172.21.15.137:3300/0 conn(0x7ff894141d30 0x7ff894142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 --2- 172.21.15.125:0/301361211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff8940cef70 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 -- 172.21.15.125:0/301361211 >> 172.21.15.125:0/301361211 conn(0x7ff8940cf8c0 msgr2=0x7ff8940cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 -- 172.21.15.125:0/301361211 shutdown_connections 2024-01-16T15:11:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.631+0000 7ff89ba00640 1 -- 172.21.15.125:0/301361211 wait complete. 2024-01-16T15:11:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.633+0000 7ff89ba00640 1 Processor -- start 2024-01-16T15:11:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.633+0000 7ff89ba00640 1 -- start start 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 0x7ff894159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 0x7ff8941433b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89413b0b0 con 0x7ff8940ceae0 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff894159e60 con 0x7ff894141d30 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff89ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff894137e80 con 0x7ff8940cb860 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.634+0000 7ff893e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40962/0 (socket says 172.21.15.125:40962) 2024-01-16T15:11:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 -- 172.21.15.125:0/2484023943 learned_addr learned my addr 172.21.15.125:0/2484023943 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff899a00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 0x7ff8941433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff899000640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 0x7ff894159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 -- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 msgr2=0x7ff894159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 0x7ff894159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 -- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 msgr2=0x7ff8941433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 0x7ff8941433b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 -- 172.21.15.125:0/2484023943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c009400 con 0x7ff8940ceae0 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff899000640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 0x7ff894159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff899a00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 0x7ff8941433b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.635+0000 7ff893e00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff87c02bee0 tx=0x7ff87c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.636+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c03ba70 con 0x7ff8940ceae0 2024-01-16T15:11:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.636+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c03bbf0 con 0x7ff8940ceae0 2024-01-16T15:11:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.636+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff894138160 con 0x7ff8940ceae0 2024-01-16T15:11:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.636+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0433f0 con 0x7ff8940ceae0 2024-01-16T15:11:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.636+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff894138720 con 0x7ff8940ceae0 2024-01-16T15:11:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.637+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894144530 con 0x7ff8940ceae0 2024-01-16T15:11:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.638+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7ff87c043550 con 0x7ff8940ceae0 2024-01-16T15:11:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.639+0000 7ff890c00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 0x7ff86c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.639+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff87c0aa6a0 con 0x7ff8940ceae0 2024-01-16T15:11:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.639+0000 7ff899000640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 0x7ff86c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.640+0000 7ff899000640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 0x7ff86c06e590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff884005880 tx=0x7ff884009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.644+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff87c076e20 con 0x7ff8940ceae0 2024-01-16T15:11:00.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.863+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff89406ab20 con 0x7ff86c06c0d0 2024-01-16T15:11:00.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.894+0000 7ff890c00640 1 -- 172.21.15.125:0/2484023943 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff884001330 con 0x7ff86c06c0d0 2024-01-16T15:11:00.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.897+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 msgr2=0x7ff86c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.897+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 0x7ff86c06e590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff884005880 tx=0x7ff884009040 comp rx=0 tx=0).stop 2024-01-16T15:11:00.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.897+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 msgr2=0x7ff894142e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.897+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff87c02bee0 tx=0x7ff87c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 shutdown_connections 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff894141d30 0x7ff8941433b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff86c06c0d0 0x7ff86c06e590 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8940ceae0 0x7ff894142e70 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 --2- 172.21.15.125:0/2484023943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8940cb860 0x7ff894159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 >> 172.21.15.125:0/2484023943 conn(0x7ff8940cf8c0 msgr2=0x7ff8940cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 shutdown_connections 2024-01-16T15:11:00.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:00.898+0000 7ff89ba00640 1 -- 172.21.15.125:0/2484023943 wait complete. 2024-01-16T15:11:00.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:00.896 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:11:00.989 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T15:11:00.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:11:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.208+0000 7f1f80400640 1 Processor -- start 2024-01-16T15:11:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.208+0000 7f1f80400640 1 -- start start 2024-01-16T15:11:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f80400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1f7813df80 0x7f1f7815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f80400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780cf9a0 0x7f1f780cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f80400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7806c470 con 0x7f1f7813e360 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f80400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7806c750 con 0x7f1f780cf9a0 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f80400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7806ca30 con 0x7f1f7813df80 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f7e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780cf9a0 0x7f1f780cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f7e400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1f780cf9a0 0x7f1f780cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40976/0 (socket says 172.21.15.125:40976) 2024-01-16T15:11:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.210+0000 7f1f7e400640 1 -- 172.21.15.125:0/2074352293 learned_addr learned my addr 172.21.15.125:0/2074352293 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.211+0000 7f1f7da00640 1 --2- 172.21.15.125:0/2074352293 >> v2:172.21.15.137:3300/0 conn(0x7f1f7813df80 0x7f1f7815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.211+0000 7f1f7e400640 1 -- 172.21.15.125:0/2074352293 >> v2:172.21.15.137:3300/0 conn(0x7f1f7813df80 msgr2=0x7f1f7815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.211+0000 7f1f7e400640 1 --2- 172.21.15.125:0/2074352293 >> v2:172.21.15.137:3300/0 conn(0x7f1f7813df80 0x7f1f7815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.211+0000 7f1f7e400640 1 -- 172.21.15.125:0/2074352293 >> v1:172.21.15.137:6789/0 conn(0x7f1f7813e360 legacy=0x7f1f780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.211+0000 7f1f7e400640 1 -- 172.21.15.125:0/2074352293 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7806cd10 con 0x7f1f780cf9a0 2024-01-16T15:11:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.212+0000 7f1f7e400640 1 --2- 172.21.15.125:0/2074352293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780cf9a0 0x7f1f780cd840 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1f78072770 tx=0x7f1f7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a254dfaa5b15e9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.212+0000 7f1f77e00640 1 -- 172.21.15.125:0/2074352293 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7002ca80 con 0x7f1f780cf9a0 2024-01-16T15:11:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.212+0000 7f1f77e00640 1 -- 172.21.15.125:0/2074352293 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7002cbe0 con 0x7f1f780cf9a0 2024-01-16T15:11:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.212+0000 7f1f77e00640 1 -- 172.21.15.125:0/2074352293 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700343f0 con 0x7f1f780cf9a0 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 -- 172.21.15.125:0/2074352293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780cf9a0 msgr2=0x7f1f780cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 --2- 172.21.15.125:0/2074352293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780cf9a0 0x7f1f780cd840 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1f78072770 tx=0x7f1f7002bee0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 -- 172.21.15.125:0/2074352293 shutdown_connections 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 --2- 172.21.15.125:0/2074352293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f780cf9a0 0x7f1f780cd840 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 --2- 172.21.15.125:0/2074352293 >> v2:172.21.15.137:3300/0 conn(0x7f1f7813df80 0x7f1f7815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 -- 172.21.15.125:0/2074352293 >> 172.21.15.125:0/2074352293 conn(0x7f1f7806a800 msgr2=0x7f1f7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 -- 172.21.15.125:0/2074352293 shutdown_connections 2024-01-16T15:11:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.213+0000 7f1f80400640 1 -- 172.21.15.125:0/2074352293 wait complete. 2024-01-16T15:11:01.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.215+0000 7f1f80400640 1 Processor -- start 2024-01-16T15:11:01.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.215+0000 7f1f80400640 1 -- start start 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf9a0 0x7f1f78137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7813e360 0x7f1f7813cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78139660 con 0x7f1f7813df80 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781f1400 con 0x7f1f7813e360 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f80400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781f16e0 con 0x7f1f780cf9a0 2024-01-16T15:11:01.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40992/0 (socket says 172.21.15.125:40992) 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.216+0000 7f1f7d000640 1 -- 172.21.15.125:0/3074026222 learned_addr learned my addr 172.21.15.125:0/3074026222 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 -- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf9a0 msgr2=0x7f1f78137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf9a0 0x7f1f78137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 -- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7813e360 msgr2=0x7f1f7813cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7813e360 0x7f1f7813cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 -- 172.21.15.125:0/3074026222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70009b30 con 0x7f1f7813df80 2024-01-16T15:11:01.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f7d000640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c400 tx=0x7f1f6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60011790 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60011930 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60011ce0 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f781f19c0 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.217+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f78137e40 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.219+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f781f1ee0 con 0x7f1f7813df80 2024-01-16T15:11:01.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.219+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f1f6001c460 con 0x7f1f7813df80 2024-01-16T15:11:01.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.220+0000 7f1f75600640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 0x7f1f5006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.220+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1f600888b0 con 0x7f1f7813df80 2024-01-16T15:11:01.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.221+0000 7f1f7da00640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 0x7f1f5006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.221+0000 7f1f7da00640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 0x7f1f5006e590 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.225+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f781f1ee0 con 0x7f1f7813df80 2024-01-16T15:11:01.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:01.424+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7815f9c0 con 0x7f1f5006c0d0 2024-01-16T15:11:01.438 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T15:11:01.459+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:11:01.529 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:11:03.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:03.378+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f1f60055280 con 0x7f1f7813df80 2024-01-16T15:11:04.469 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:11:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.492+0000 7f1f75600640 1 -- 172.21.15.125:0/3074026222 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f1f6800ae20 con 0x7f1f5006c0d0 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.495+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 msgr2=0x7f1f5006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 0x7f1f5006e590 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 msgr2=0x7f1f7813c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c400 tx=0x7f1f6000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 shutdown_connections 2024-01-16T15:11:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7813e360 0x7f1f7813cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1f5006c0d0 0x7f1f5006e590 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f7813df80 0x7f1f7813c680 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 --2- 172.21.15.125:0/3074026222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f780cf9a0 0x7f1f78137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 >> 172.21.15.125:0/3074026222 conn(0x7f1f7806a800 msgr2=0x7f1f780b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 shutdown_connections 2024-01-16T15:11:04.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:04.496+0000 7f1f80400640 1 -- 172.21.15.125:0/3074026222 wait complete. 2024-01-16T15:11:04.500 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:11:04.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:04.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:04.695 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:11:04.696 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:04.696 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:04.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:05.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.056+0000 7fad26c00640 1 Processor -- start 2024-01-16T15:11:05.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.056+0000 7fad26c00640 1 -- start start 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad26c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fad2013df80 0x7fad2015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad26c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad2013e360 0x7fad200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad26c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2006c470 con 0x7fad2015f9c0 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad26c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2006c750 con 0x7fad2013e360 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad26c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2006ca30 con 0x7fad2013df80 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad2013e360 0x7fad200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad2013e360 0x7fad200cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41016/0 (socket says 172.21.15.125:41016) 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 -- 172.21.15.125:0/1800964685 learned_addr learned my addr 172.21.15.125:0/1800964685 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 -- 172.21.15.125:0/1800964685 >> v2:172.21.15.137:3300/0 conn(0x7fad2013df80 msgr2=0x7fad2015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:05.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 --2- 172.21.15.125:0/1800964685 >> v2:172.21.15.137:3300/0 conn(0x7fad2013df80 0x7fad2015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 -- 172.21.15.125:0/1800964685 >> v1:172.21.15.137:6789/0 conn(0x7fad2015f9c0 legacy=0x7fad201408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:05.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.058+0000 7fad1f400640 1 -- 172.21.15.125:0/1800964685 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2006cd10 con 0x7fad2013e360 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.059+0000 7fad1f400640 1 --2- 172.21.15.125:0/1800964685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad200cf290 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fad20072770 tx=0x7fad0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2172b9eb6076ac5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad1ea00640 1 -- 172.21.15.125:0/1800964685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08031870 con 0x7fad2013e360 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad1ea00640 1 -- 172.21.15.125:0/1800964685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad080319d0 con 0x7fad2013e360 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad1ea00640 1 -- 172.21.15.125:0/1800964685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08031d60 con 0x7fad2013e360 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 -- 172.21.15.125:0/1800964685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 msgr2=0x7fad200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 --2- 172.21.15.125:0/1800964685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad200cf290 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fad20072770 tx=0x7fad0802b910 comp rx=0 tx=0).stop 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 -- 172.21.15.125:0/1800964685 shutdown_connections 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 --2- 172.21.15.125:0/1800964685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad200cf290 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 --2- 172.21.15.125:0/1800964685 >> v2:172.21.15.137:3300/0 conn(0x7fad2013df80 0x7fad2015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.060+0000 7fad26c00640 1 -- 172.21.15.125:0/1800964685 >> 172.21.15.125:0/1800964685 conn(0x7fad2006a800 msgr2=0x7fad2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:05.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.061+0000 7fad26c00640 1 -- 172.21.15.125:0/1800964685 shutdown_connections 2024-01-16T15:11:05.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.061+0000 7fad26c00640 1 -- 172.21.15.125:0/1800964685 wait complete. 2024-01-16T15:11:05.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.062+0000 7fad26c00640 1 Processor -- start 2024-01-16T15:11:05.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.062+0000 7fad26c00640 1 -- start start 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 0x7fad2013cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 0x7fad200b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad201508d0 con 0x7fad2013e360 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2013d400 con 0x7fad2015f9c0 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad26c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2014fcc0 con 0x7fad2013df80 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.063+0000 7fad1f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41022/0 (socket says 172.21.15.125:41022) 2024-01-16T15:11:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 -- 172.21.15.125:0/2850953478 learned_addr learned my addr 172.21.15.125:0/2850953478 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad24c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 0x7fad200b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1fe00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 0x7fad2013cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 -- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 msgr2=0x7fad2013cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 0x7fad2013cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 -- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 msgr2=0x7fad200b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 0x7fad200b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 -- 172.21.15.125:0/2850953478 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080094d0 con 0x7fad2013e360 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1fe00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 0x7fad2013cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad24c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 0x7fad200b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad1f400640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fad0802b950 tx=0x7fad080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.064+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0803ad10 con 0x7fad2013e360 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.065+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0803ae70 con 0x7fad2013e360 2024-01-16T15:11:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.065+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08044610 con 0x7fad2013e360 2024-01-16T15:11:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.065+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2014ffa0 con 0x7fad2013e360 2024-01-16T15:11:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.065+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad20150560 con 0x7fad2013e360 2024-01-16T15:11:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.066+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad20150a80 con 0x7fad2013e360 2024-01-16T15:11:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.067+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fad08044770 con 0x7fad2013e360 2024-01-16T15:11:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.067+0000 7fad17e00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 0x7facf806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.068+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fad0802d040 con 0x7fad2013e360 2024-01-16T15:11:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.068+0000 7fad1fe00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 0x7facf806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.068+0000 7fad1fe00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 0x7facf806e620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fad10007eb0 tx=0x7fad100073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.072+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad20150a80 con 0x7fad2013e360 2024-01-16T15:11:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.262+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad20150a80 con 0x7fad2013e360 2024-01-16T15:11:05.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.264+0000 7fad17e00640 1 -- 172.21.15.125:0/2850953478 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13035 (secure 0 0 0) 0x7fad20150a80 con 0x7fad2013e360 2024-01-16T15:11:05.241 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:05.241 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:04.474760+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/589280594":"2024-01-17T15:11:01.459274+0000","172.21.15.125:6836/589280594":"2024-01-17T15:11:01.459274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 msgr2=0x7facf806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 0x7facf806e620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fad10007eb0 tx=0x7fad100073c0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 msgr2=0x7fad20150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fad0802b950 tx=0x7fad080046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 shutdown_connections 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad2015f9c0 0x7fad200b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7facf806c160 0x7facf806e620 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad2013e360 0x7fad20150d60 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 --2- 172.21.15.125:0/2850953478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad2013df80 0x7fad2013cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 >> 172.21.15.125:0/2850953478 conn(0x7fad2006a800 msgr2=0x7fad2015c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:05.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.267+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 shutdown_connections 2024-01-16T15:11:05.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.268+0000 7fad26c00640 1 -- 172.21.15.125:0/2850953478 wait complete. 2024-01-16T15:11:05.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:05.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:05.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.497+0000 7f5c71c00640 1 Processor -- start 2024-01-16T15:11:05.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.497+0000 7f5c71c00640 1 -- start start 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.498+0000 7f5c71c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c6c13df80 0x7f5c6c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.498+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c71c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c06c470 con 0x7f5c6c13e360 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c71c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6c06c750 con 0x7f5c6c13df80 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c71c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c06ca30 con 0x7f5c6c0cf9a0 2024-01-16T15:11:05.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c6c13df80 0x7f5c6c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c6c13df80 0x7f5c6c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41024/0 (socket says 172.21.15.125:41024) 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 -- 172.21.15.125:0/299094895 learned_addr learned my addr 172.21.15.125:0/299094895 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 -- 172.21.15.125:0/299094895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 msgr2=0x7f5c6c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/299094895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 -- 172.21.15.125:0/299094895 >> v1:172.21.15.125:6789/0 conn(0x7f5c6c13e360 legacy=0x7f5c6c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.499+0000 7f5c6aa00640 1 -- 172.21.15.125:0/299094895 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c06cd10 con 0x7f5c6c13df80 2024-01-16T15:11:05.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.500+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/299094895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13df80 0x7f5c6c15bee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009870 tx=0x7f5c5c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6090236cabcc6e28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c69600640 1 -- 172.21.15.125:0/299094895 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c030840 con 0x7f5c6c13df80 2024-01-16T15:11:05.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c69600640 1 -- 172.21.15.125:0/299094895 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c0309a0 con 0x7f5c6c13df80 2024-01-16T15:11:05.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c69600640 1 -- 172.21.15.125:0/299094895 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c030d30 con 0x7f5c6c13df80 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 -- 172.21.15.125:0/299094895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13df80 msgr2=0x7f5c6c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 --2- 172.21.15.125:0/299094895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13df80 0x7f5c6c15bee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009870 tx=0x7f5c5c02b980 comp rx=0 tx=0).stop 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 -- 172.21.15.125:0/299094895 shutdown_connections 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 --2- 172.21.15.125:0/299094895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 --2- 172.21.15.125:0/299094895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13df80 0x7f5c6c15bee0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.501+0000 7f5c71c00640 1 -- 172.21.15.125:0/299094895 >> 172.21.15.125:0/299094895 conn(0x7f5c6c06a800 msgr2=0x7f5c6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:05.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.502+0000 7f5c71c00640 1 -- 172.21.15.125:0/299094895 shutdown_connections 2024-01-16T15:11:05.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.502+0000 7f5c71c00640 1 -- 172.21.15.125:0/299094895 wait complete. 2024-01-16T15:11:05.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.503+0000 7f5c71c00640 1 Processor -- start 2024-01-16T15:11:05.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.503+0000 7f5c71c00640 1 -- start start 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 0x7f5c6c158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c135c90 con 0x7f5c6c13e360 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c15ac30 con 0x7f5c6c13df80 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.504+0000 7f5c71c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c071100 con 0x7f5c6c0cf9a0 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41028/0 (socket says 172.21.15.125:41028) 2024-01-16T15:11:05.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 -- 172.21.15.125:0/1461352845 learned_addr learned my addr 172.21.15.125:0/1461352845 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:05.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 -- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 msgr2=0x7f5c6c135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6a000640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 0x7f5c6c158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c135750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 -- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 msgr2=0x7f5c6c158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 0x7f5c6c158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 -- 172.21.15.125:0/1461352845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c0094d0 con 0x7f5c6c13e360 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c135750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6a000640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 0x7f5c6c158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.505+0000 7f5c6b400640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400ba10 tx=0x7f5c6400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.506+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400cb40 con 0x7f5c6c13e360 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.506+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400ccc0 con 0x7f5c6c13e360 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.506+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64014680 con 0x7f5c6c13e360 2024-01-16T15:11:05.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.506+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c0713e0 con 0x7f5c6c13e360 2024-01-16T15:11:05.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.506+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c0719a0 con 0x7f5c6c13e360 2024-01-16T15:11:05.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.507+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c0675d0 con 0x7f5c6c13e360 2024-01-16T15:11:05.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.508+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f5c6401d3b0 con 0x7f5c6c13e360 2024-01-16T15:11:05.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.509+0000 7f5c62a00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 0x7f5c4406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.509+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c640888d0 con 0x7f5c6c13e360 2024-01-16T15:11:05.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.509+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 0x7f5c4406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.510+0000 7f5c6aa00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 0x7f5c4406e620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c036000 tx=0x7f5c5c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.513+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c64054e80 con 0x7f5c6c13e360 2024-01-16T15:11:05.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.702+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c6c15b0f0 con 0x7f5c6c13e360 2024-01-16T15:11:05.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.703+0000 7f5c62a00640 1 -- 172.21.15.125:0/1461352845 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c64019020 con 0x7f5c6c13e360 2024-01-16T15:11:05.679 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:11:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.705+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 msgr2=0x7f5c4406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.705+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 0x7f5c4406e620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c036000 tx=0x7f5c5c03b040 comp rx=0 tx=0).stop 2024-01-16T15:11:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.705+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 msgr2=0x7f5c6c158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.705+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400ba10 tx=0x7f5c6400bed0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 shutdown_connections 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5c4406c160 0x7f5c4406e620 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c6c13e360 0x7f5c6c158ad0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c6c13df80 0x7f5c6c158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 --2- 172.21.15.125:0/1461352845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c6c0cf9a0 0x7f5c6c135750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 >> 172.21.15.125:0/1461352845 conn(0x7f5c6c06a800 msgr2=0x7f5c6c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 shutdown_connections 2024-01-16T15:11:05.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.706+0000 7f5c71c00640 1 -- 172.21.15.125:0/1461352845 wait complete. 2024-01-16T15:11:05.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:11:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.930+0000 7f1695000640 1 Processor -- start 2024-01-16T15:11:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.930+0000 7f1695000640 1 -- start start 2024-01-16T15:11:05.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.932+0000 7f1695000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f16900d0100 0x7f16900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.932+0000 7f1695000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.932+0000 7f1695000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900d0a10 con 0x7f1690160920 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.932+0000 7f1695000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900cf080 con 0x7f16900d0100 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.932+0000 7f1695000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c6be0 con 0x7f16900d0f40 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f16900d0100 0x7f16900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168f400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f1690160920 0x7f16900cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:58856/0 (socket says 172.21.15.125:58856) 2024-01-16T15:11:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:48620/0 (socket says 172.21.15.125:48620) 2024-01-16T15:11:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168f400640 1 -- 172.21.15.125:0/1533364441 learned_addr learned my addr 172.21.15.125:0/1533364441 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168e000640 1 -- 172.21.15.125:0/1533364441 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836412235 0 0) 0x7f16900d0a10 con 0x7f1690160920 2024-01-16T15:11:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.933+0000 7f168e000640 1 -- 172.21.15.125:0/1533364441 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1670003520 con 0x7f1690160920 2024-01-16T15:11:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.934+0000 7f168fe00640 1 -- 172.21.15.125:0/1533364441 >> v2:172.21.15.137:3300/0 conn(0x7f16900d0100 msgr2=0x7f16900b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.934+0000 7f168fe00640 1 --2- 172.21.15.125:0/1533364441 >> v2:172.21.15.137:3300/0 conn(0x7f16900d0100 0x7f16900b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.934+0000 7f168fe00640 1 -- 172.21.15.125:0/1533364441 >> v1:172.21.15.137:6789/0 conn(0x7f1690160920 legacy=0x7f16900cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.934+0000 7f168fe00640 1 -- 172.21.15.125:0/1533364441 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900ce7e0 con 0x7f16900d0f40 2024-01-16T15:11:05.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.935+0000 7f168fe00640 1 --2- 172.21.15.125:0/1533364441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f16880097a0 tx=0x7f168802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d65372dcab1204 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f168e000640 1 -- 172.21.15.125:0/1533364441 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688031830 con 0x7f16900d0f40 2024-01-16T15:11:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f168e000640 1 -- 172.21.15.125:0/1533364441 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688031990 con 0x7f16900d0f40 2024-01-16T15:11:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f168e000640 1 -- 172.21.15.125:0/1533364441 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688031d00 con 0x7f16900d0f40 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 -- 172.21.15.125:0/1533364441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 msgr2=0x7f16900d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 --2- 172.21.15.125:0/1533364441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f16880097a0 tx=0x7f168802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 -- 172.21.15.125:0/1533364441 shutdown_connections 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 --2- 172.21.15.125:0/1533364441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16900d0f40 0x7f16900d13d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 --2- 172.21.15.125:0/1533364441 >> v2:172.21.15.137:3300/0 conn(0x7f16900d0100 0x7f16900b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 -- 172.21.15.125:0/1533364441 >> 172.21.15.125:0/1533364441 conn(0x7f16900d3090 msgr2=0x7f16900d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.936+0000 7f1695000640 1 -- 172.21.15.125:0/1533364441 shutdown_connections 2024-01-16T15:11:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.937+0000 7f1695000640 1 -- 172.21.15.125:0/1533364441 wait complete. 2024-01-16T15:11:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.939+0000 7f1695000640 1 Processor -- start 2024-01-16T15:11:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.939+0000 7f1695000640 1 -- start start 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 0x7f16900c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1690160920 0x7f16900b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c7ce0 con 0x7f16900d0f40 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c7f30 con 0x7f16900d0100 2024-01-16T15:11:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f1695000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c8210 con 0x7f1690160920 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f168ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f168ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41034/0 (socket says 172.21.15.125:41034) 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.940+0000 7f168ea00640 1 -- 172.21.15.125:0/4092540142 learned_addr learned my addr 172.21.15.125:0/4092540142 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168f400640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 0x7f16900c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 -- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1690160920 msgr2=0x7f16900b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1690160920 0x7f16900b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 -- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 msgr2=0x7f16900c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 0x7f16900c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 -- 172.21.15.125:0/4092540142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009400 con 0x7f16900d0f40 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168f400640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 0x7f16900c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f168ea00640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f168000efc0 tx=0x7f168000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680009b80 con 0x7f16900d0f40 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000f040 con 0x7f16900d0f40 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16800146a0 con 0x7f16900d0f40 2024-01-16T15:11:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.941+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900c84f0 con 0x7f16900d0f40 2024-01-16T15:11:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.942+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169014d9a0 con 0x7f16900d0f40 2024-01-16T15:11:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.943+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1690067450 con 0x7f16900d0f40 2024-01-16T15:11:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.944+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f1680014800 con 0x7f16900d0f40 2024-01-16T15:11:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.945+0000 7f1687400640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 0x7f166806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.945+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1680089460 con 0x7f16900d0f40 2024-01-16T15:11:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.945+0000 7f168f400640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 0x7f166806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.946+0000 7f168f400640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 0x7f166806e620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f16780011e0 tx=0x7f1678009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:05.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:05.951+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1680055d80 con 0x7f16900d0f40 2024-01-16T15:11:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.153+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f16900d0640 con 0x7f16900d0f40 2024-01-16T15:11:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.154+0000 7f1687400640 1 -- 172.21.15.125:0/4092540142 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v91) v1 ==== 80+0+122 (secure 0 0 0) 0x7f16800593a0 con 0x7f16900d0f40 2024-01-16T15:11:06.131 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/589280594 2024-01-17T15:11:01.459274+0000 2024-01-16T15:11:06.131 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/589280594 2024-01-17T15:11:01.459274+0000 2024-01-16T15:11:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 msgr2=0x7f166806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 0x7f166806e620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f16780011e0 tx=0x7f1678009040 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 msgr2=0x7f169014d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f168000efc0 tx=0x7f168000c490 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 shutdown_connections 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1690160920 0x7f16900b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f166806c160 0x7f166806e620 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16900d0f40 0x7f169014d460 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 --2- 172.21.15.125:0/4092540142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16900d0100 0x7f16900c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 >> 172.21.15.125:0/4092540142 conn(0x7f16900d3090 msgr2=0x7f169006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 shutdown_connections 2024-01-16T15:11:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.157+0000 7f1695000640 1 -- 172.21.15.125:0/4092540142 wait complete. 2024-01-16T15:11:06.152 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:11:06.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:06.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:06.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[70763]: fuse finished with error 0 and tester_r 0 2024-01-16T15:11:06.244 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:11:06.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:06.244 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:11:06.486 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:11:06.486 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:06.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.796+0000 7fd561800640 1 Processor -- start 2024-01-16T15:11:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.796+0000 7fd561800640 1 -- start start 2024-01-16T15:11:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.797+0000 7fd561800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0ce8f0 0x7fd55c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.797+0000 7fd561800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd55c0b2a00 0x7fd55c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.798+0000 7fd561800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c0d0de0 con 0x7fd55c160f80 2024-01-16T15:11:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.798+0000 7fd561800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c0d10c0 con 0x7fd55c0b2a00 2024-01-16T15:11:06.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.798+0000 7fd561800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0d13a0 con 0x7fd55c0ce8f0 2024-01-16T15:11:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.798+0000 7fd55a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0ce8f0 0x7fd55c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.798+0000 7fd55b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd55c0b2a00 0x7fd55c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55aa00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fd55c160f80 0x7fd55c161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:34454/0 (socket says 172.21.15.125:34454) 2024-01-16T15:11:06.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55aa00640 1 -- 172.21.15.125:0/4066767587 learned_addr learned my addr 172.21.15.125:0/4066767587 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55b400640 1 -- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0ce8f0 msgr2=0x7fd55c0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55b400640 1 --2- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0ce8f0 0x7fd55c0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55b400640 1 -- 172.21.15.125:0/4066767587 >> v1:172.21.15.137:6789/0 conn(0x7fd55c160f80 legacy=0x7fd55c161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.799+0000 7fd55b400640 1 -- 172.21.15.125:0/4066767587 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c0d1680 con 0x7fd55c0b2a00 2024-01-16T15:11:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.801+0000 7fd55b400640 1 --2- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0b2a00 0x7fd55c0b2e60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd554009d90 tx=0x7fd55402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ab9359f001faac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.801+0000 7fd559600640 1 -- 172.21.15.125:0/4066767587 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55402cc50 con 0x7fd55c0b2a00 2024-01-16T15:11:06.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.801+0000 7fd559600640 1 -- 172.21.15.125:0/4066767587 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55402cdb0 con 0x7fd55c0b2a00 2024-01-16T15:11:06.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.801+0000 7fd559600640 1 -- 172.21.15.125:0/4066767587 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554034660 con 0x7fd55c0b2a00 2024-01-16T15:11:06.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 -- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0b2a00 msgr2=0x7fd55c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 --2- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0b2a00 0x7fd55c0b2e60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd554009d90 tx=0x7fd55402bea0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 -- 172.21.15.125:0/4066767587 shutdown_connections 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 --2- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0b2a00 0x7fd55c0b2e60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 --2- 172.21.15.125:0/4066767587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0ce8f0 0x7fd55c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 -- 172.21.15.125:0/4066767587 >> 172.21.15.125:0/4066767587 conn(0x7fd55c09c550 msgr2=0x7fd55c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 -- 172.21.15.125:0/4066767587 shutdown_connections 2024-01-16T15:11:06.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.802+0000 7fd561800640 1 -- 172.21.15.125:0/4066767587 wait complete. 2024-01-16T15:11:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.804+0000 7fd561800640 1 Processor -- start 2024-01-16T15:11:06.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.804+0000 7fd561800640 1 -- start start 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0b2a00 0x7fd55c15a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0ce8f0 0x7fd55c0b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0b8d20 con 0x7fd55c160f80 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0b7c80 con 0x7fd55c0ce8f0 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd561800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0b7f60 con 0x7fd55c0b2a00 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44986/0 (socket says 172.21.15.125:44986) 2024-01-16T15:11:06.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 -- 172.21.15.125:0/1145409779 learned_addr learned my addr 172.21.15.125:0/1145409779 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55a000640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0ce8f0 0x7fd55c0b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 -- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0b2a00 msgr2=0x7fd55c15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0b2a00 0x7fd55c15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 -- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0ce8f0 msgr2=0x7fd55c0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0ce8f0 0x7fd55c0b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.805+0000 7fd55b400640 1 -- 172.21.15.125:0/1145409779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554009a70 con 0x7fd55c160f80 2024-01-16T15:11:06.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd55b400640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd554009eb0 tx=0x7fd554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55403e610 con 0x7fd55c160f80 2024-01-16T15:11:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55403e770 con 0x7fd55c160f80 2024-01-16T15:11:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55403eb60 con 0x7fd55c160f80 2024-01-16T15:11:06.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c0b8240 con 0x7fd55c160f80 2024-01-16T15:11:06.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.806+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c0b8800 con 0x7fd55c160f80 2024-01-16T15:11:06.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.807+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c0b89f0 con 0x7fd55c160f80 2024-01-16T15:11:06.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.808+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd55403b050 con 0x7fd55c160f80 2024-01-16T15:11:06.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.809+0000 7fd552a00640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 0x7fd53406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:06.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.809+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd5540aabe0 con 0x7fd55c160f80 2024-01-16T15:11:06.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.809+0000 7fd55aa00640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 0x7fd53406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:06.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.810+0000 7fd55aa00640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 0x7fd53406e620 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd54c000f60 tx=0x7fd54c007a80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:06.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:06.814+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5540771c0 con 0x7fd55c160f80 2024-01-16T15:11:06.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.010+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd55c0675d0 con 0x7fd55c160f80 2024-01-16T15:11:06.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.012+0000 7fd552a00640 1 -- 172.21.15.125:0/1145409779 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13035 (secure 0 0 0) 0x7fd554009450 con 0x7fd55c160f80 2024-01-16T15:11:06.989 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:06.989 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:04.474760+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/589280594":"2024-01-17T15:11:01.459274+0000","172.21.15.125:6836/589280594":"2024-01-17T15:11:01.459274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 msgr2=0x7fd53406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 0x7fd53406e620 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd54c000f60 tx=0x7fd54c007a80 comp rx=0 tx=0).stop 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 msgr2=0x7fd55c0b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd554009eb0 tx=0x7fd554004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 shutdown_connections 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd53406c160 0x7fd53406e620 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd55c160f80 0x7fd55c0b7410 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd55c0ce8f0 0x7fd55c0b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 --2- 172.21.15.125:0/1145409779 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd55c0b2a00 0x7fd55c15a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 >> 172.21.15.125:0/1145409779 conn(0x7fd55c09c550 msgr2=0x7fd55c15b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 shutdown_connections 2024-01-16T15:11:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.015+0000 7fd561800640 1 -- 172.21.15.125:0/1145409779 wait complete. 2024-01-16T15:11:07.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:07.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:07.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.248+0000 7f7bc9200640 1 Processor -- start 2024-01-16T15:11:07.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.248+0000 7f7bc9200640 1 -- start start 2024-01-16T15:11:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.249+0000 7f7bc9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7bc40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc9200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc40cfc30 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc9200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc40b1ee0 con 0x7f7bc415e3b0 2024-01-16T15:11:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40b2780 con 0x7f7bc4143ae0 2024-01-16T15:11:07.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bbbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7bc40cf7d0 0x7f7bc415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42340/0 (socket says 172.21.15.125:42340) 2024-01-16T15:11:07.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bbbe00640 1 -- 172.21.15.125:0/4018580219 learned_addr learned my addr 172.21.15.125:0/4018580219 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc1e00640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7bc40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596705536 0 0) 0x7f7bc40cfc30 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.250+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba4003520 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730566453 0 0) 0x7f7ba4003520 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 msgr2=0x7f7bc40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7bc40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 >> v1:172.21.15.137:6789/0 conn(0x7f7bc415e3b0 legacy=0x7f7bc40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc415e990 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7bac001670 con 0x7f7bc40cf7d0 2024-01-16T15:11:07.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 >> v1:172.21.15.125:6789/0 conn(0x7f7bc40cf7d0 legacy=0x7f7bc415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ba40039a0 0x7f7ba4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7ba40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7ba4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40b2780 con 0x7f7ba4004d00 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba400ba30 con 0x7f7ba40039a0 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba400bba0 con 0x7f7bc4143ae0 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.251+0000 7f7bc2800640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7ba4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc1e00640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ba40039a0 0x7f7ba4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bbbe00640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7ba40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc2800640 1 -- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 msgr2=0x7f7ba40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc2800640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4143ae0 0x7f7ba40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc2800640 1 -- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ba40039a0 msgr2=0x7f7ba4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc2800640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ba40039a0 0x7f7ba4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.252+0000 7f7bc2800640 1 -- 172.21.15.125:0/4018580219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba4003520 con 0x7f7ba4004d00 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.253+0000 7f7bc2800640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7ba4005180 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc001f60 tx=0x7f7ba40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358ad9cd87989d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.253+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc02d6f0 con 0x7f7ba4004d00 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.253+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc02d870 con 0x7f7ba4004d00 2024-01-16T15:11:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.253+0000 7f7bc1400640 1 -- 172.21.15.125:0/4018580219 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc02dc20 con 0x7f7ba4004d00 2024-01-16T15:11:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 -- 172.21.15.125:0/4018580219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 msgr2=0x7f7ba4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 --2- 172.21.15.125:0/4018580219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7ba4005180 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc001f60 tx=0x7f7ba40057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc2800640 1 -- 172.21.15.125:0/4018580219 reap_dead start 2024-01-16T15:11:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 -- 172.21.15.125:0/4018580219 shutdown_connections 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 -- 172.21.15.125:0/4018580219 >> 172.21.15.125:0/4018580219 conn(0x7f7bc406a800 msgr2=0x7f7bc406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 -- 172.21.15.125:0/4018580219 shutdown_connections 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.254+0000 7f7bc9200640 1 -- 172.21.15.125:0/4018580219 wait complete. 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.256+0000 7f7bc9200640 1 Processor -- start 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.256+0000 7f7bc9200640 1 -- start start 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bc40cd060 0x7f7bc4137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4137d50 0x7f7bc4138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40ce070 con 0x7f7ba4004d00 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40cd4f0 con 0x7f7bc40cd060 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40cd7d0 con 0x7f7bc4137d50 2024-01-16T15:11:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45010/0 (socket says 172.21.15.125:45010) 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bc1e00640 1 -- 172.21.15.125:0/2360599095 learned_addr learned my addr 172.21.15.125:0/2360599095 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.257+0000 7f7bbbe00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bc40cd060 0x7f7bc4137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 -- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4137d50 msgr2=0x7f7bc4138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4137d50 0x7f7bc4138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 -- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bc40cd060 msgr2=0x7f7bc4137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bc40cd060 0x7f7bc4137810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 -- 172.21.15.125:0/2360599095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc40cdab0 con 0x7f7ba4004d00 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc1e00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4011600 tx=0x7f7bb4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4015870 con 0x7f7ba4004d00 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb40159d0 con 0x7f7ba4004d00 2024-01-16T15:11:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4015d40 con 0x7f7ba4004d00 2024-01-16T15:11:07.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc40cdd90 con 0x7f7ba4004d00 2024-01-16T15:11:07.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.258+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc40ceb40 con 0x7f7ba4004d00 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.260+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc40675d0 con 0x7f7ba4004d00 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.261+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f7bb400f490 con 0x7f7ba4004d00 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.262+0000 7f7bba000640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 0x7f7b9c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.262+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7bb4088b00 con 0x7f7ba4004d00 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.262+0000 7f7bbbe00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 0x7f7b9c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.263+0000 7f7bbbe00640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 0x7f7b9c06e620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7bac001050 tx=0x7f7bac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.266+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bb4055420 con 0x7f7ba4004d00 2024-01-16T15:11:07.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.455+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bc413b8b0 con 0x7f7ba4004d00 2024-01-16T15:11:07.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.456+0000 7f7bba000640 1 -- 172.21.15.125:0/2360599095 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bb4058a40 con 0x7f7ba4004d00 2024-01-16T15:11:07.433 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:11:07.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.458+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 msgr2=0x7f7b9c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.458+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 0x7f7b9c06e620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7bac001050 tx=0x7f7bac009040 comp rx=0 tx=0).stop 2024-01-16T15:11:07.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 msgr2=0x7f7bc4159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4011600 tx=0x7f7bb4011ac0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 shutdown_connections 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bc4137d50 0x7f7bc4138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bc40cd060 0x7f7bc4137810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7b9c06c160 0x7f7b9c06e620 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 --2- 172.21.15.125:0/2360599095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ba4004d00 0x7f7bc4159890 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 >> 172.21.15.125:0/2360599095 conn(0x7f7bc406a800 msgr2=0x7f7bc415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 shutdown_connections 2024-01-16T15:11:07.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.459+0000 7f7bc9200640 1 -- 172.21.15.125:0/2360599095 wait complete. 2024-01-16T15:11:07.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:11:07.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.689+0000 7effdfc00640 1 Processor -- start 2024-01-16T15:11:07.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.690+0000 7effdfc00640 1 -- start start 2024-01-16T15:11:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effd80ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdfc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7effd80ae750 0x7effd80ac590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdfc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8162100 con 0x7effd80cc1e0 2024-01-16T15:11:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdfc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd80ccf20 con 0x7effd80ae750 2024-01-16T15:11:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd80cd200 con 0x7effd8074520 2024-01-16T15:11:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdd200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7effd80cc1e0 0x7effd80cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42342/0 (socket says 172.21.15.125:42342) 2024-01-16T15:11:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.691+0000 7effdd200640 1 -- 172.21.15.125:0/1442435000 learned_addr learned my addr 172.21.15.125:0/1442435000 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effddc00640 1 --2- 172.21.15.125:0/1442435000 >> v2:172.21.15.137:3300/0 conn(0x7effd80ae750 0x7effd80ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7e00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effd80ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732125862 0 0) 0x7effd8162100 con 0x7effd80cc1e0 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb8003520 con 0x7effd80cc1e0 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2199515149 0 0) 0x7effb8003520 con 0x7effd80cc1e0 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 msgr2=0x7effd80ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effd80ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 >> v2:172.21.15.137:3300/0 conn(0x7effd80ae750 msgr2=0x7effd80ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 --2- 172.21.15.125:0/1442435000 >> v2:172.21.15.137:3300/0 conn(0x7effd80ae750 0x7effd80ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd806daa0 con 0x7effd80cc1e0 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7effc0001670 con 0x7effd80cc1e0 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 >> v1:172.21.15.125:6789/0 conn(0x7effd80cc1e0 legacy=0x7effd80cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effb8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.692+0000 7effd7400640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8003ef0 0x7effb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effd7400640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effb8004d00 0x7effb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd80ccf20 con 0x7effb8004d00 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd80cd200 con 0x7effb8003ef0 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb800ba60 con 0x7effd8074520 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effb8004d00 0x7effb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effdd200640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effb8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 -- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 msgr2=0x7effb8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd8074520 0x7effb8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 -- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8003ef0 msgr2=0x7effb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8003ef0 0x7effb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.693+0000 7effddc00640 1 -- 172.21.15.125:0/1442435000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8003520 con 0x7effb8004d00 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.694+0000 7effddc00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effb8004d00 0x7effb8005180 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7effd0009a90 tx=0x7effb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd48732c0cd6b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd002c6b0 con 0x7effb8004d00 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd002c810 con 0x7effb8004d00 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effd7400640 1 -- 172.21.15.125:0/1442435000 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd002cbc0 con 0x7effb8004d00 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 -- 172.21.15.125:0/1442435000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effb8004d00 msgr2=0x7effb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 --2- 172.21.15.125:0/1442435000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effb8004d00 0x7effb8005180 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7effd0009a90 tx=0x7effb80057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effddc00640 1 -- 172.21.15.125:0/1442435000 reap_dead start 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 -- 172.21.15.125:0/1442435000 shutdown_connections 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 -- 172.21.15.125:0/1442435000 >> 172.21.15.125:0/1442435000 conn(0x7effd80b0d70 msgr2=0x7effd80b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 -- 172.21.15.125:0/1442435000 shutdown_connections 2024-01-16T15:11:07.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.695+0000 7effdfc00640 1 -- 172.21.15.125:0/1442435000 wait complete. 2024-01-16T15:11:07.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.697+0000 7effdfc00640 1 Processor -- start 2024-01-16T15:11:07.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.697+0000 7effdfc00640 1 -- start start 2024-01-16T15:11:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8004d00 0x7effd81561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd80b2870 0x7effd80b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd80c73e0 con 0x7effd80c62f0 2024-01-16T15:11:07.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd80c67d0 con 0x7effb8004d00 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.698+0000 7effdfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd80c6ab0 con 0x7effd80b2870 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45032/0 (socket says 172.21.15.125:45032) 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 -- 172.21.15.125:0/2978046194 learned_addr learned my addr 172.21.15.125:0/2978046194 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effdd200640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8004d00 0x7effd81561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 -- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd80b2870 msgr2=0x7effd80b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd80b2870 0x7effd80b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 -- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8004d00 msgr2=0x7effd81561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8004d00 0x7effd81561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 -- 172.21.15.125:0/2978046194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd80c6d90 con 0x7effd80c62f0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.699+0000 7effd7e00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7effc800ef40 tx=0x7effc800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.700+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8010040 con 0x7effd80c62f0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.700+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8009e20 con 0x7effd80c62f0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.700+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800bea0 con 0x7effd80c62f0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.700+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd80c7070 con 0x7effd80c62f0 2024-01-16T15:11:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.700+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd80b2d20 con 0x7effd80c62f0 2024-01-16T15:11:07.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.701+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd80c7550 con 0x7effd80c62f0 2024-01-16T15:11:07.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.702+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7effc8003bd0 con 0x7effd80c62f0 2024-01-16T15:11:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.703+0000 7effd4c00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 0x7effb006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.703+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7effc8088e90 con 0x7effd80c62f0 2024-01-16T15:11:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.703+0000 7effdd200640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 0x7effb006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.703+0000 7effdd200640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 0x7effb006e620 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7effc0001eb0 tx=0x7effc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.707+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effc80554d0 con 0x7effd80c62f0 2024-01-16T15:11:07.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.913+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7effd80675d0 con 0x7effd80c62f0 2024-01-16T15:11:07.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.914+0000 7effd4c00640 1 -- 172.21.15.125:0/2978046194 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v91) v1 ==== 80+0+122 (secure 0 0 0) 0x7effc8058af0 con 0x7effd80c62f0 2024-01-16T15:11:07.890 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/589280594 2024-01-17T15:11:01.459274+0000 2024-01-16T15:11:07.891 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/589280594 2024-01-17T15:11:01.459274+0000 2024-01-16T15:11:07.891 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:11:07.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 msgr2=0x7effb006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 0x7effb006e620 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7effc0001eb0 tx=0x7effc0009040 comp rx=0 tx=0).stop 2024-01-16T15:11:07.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 msgr2=0x7effd81f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7effc800ef40 tx=0x7effc800b480 comp rx=0 tx=0).stop 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 shutdown_connections 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effd80b2870 0x7effd80b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7effb006c160 0x7effb006e620 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effd80c62f0 0x7effd81f26b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 --2- 172.21.15.125:0/2978046194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb8004d00 0x7effd81561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 >> 172.21.15.125:0/2978046194 conn(0x7effd80b0d70 msgr2=0x7effd80af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 shutdown_connections 2024-01-16T15:11:07.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:07.917+0000 7effdfc00640 1 -- 172.21.15.125:0/2978046194 wait complete. 2024-01-16T15:11:07.912 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:11:07.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:07.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:08.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45904]: fuse finished with error 0 and tester_r 0 2024-01-16T15:11:08.013 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:11:08.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:08.013 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:11:08.197 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:08.197 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:08.254 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:08.254 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:11:08.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:08.254 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:11:08.309 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:11:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:11:08.311 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:11:08.311 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:08.368 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:08.368 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:11:08.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:08.368 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:11:08.421 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:11:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:11:08.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.659+0000 7f4130a00640 1 Processor -- start 2024-01-16T15:11:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.659+0000 7f4130a00640 1 -- start start 2024-01-16T15:11:08.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.660+0000 7f4130a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f412815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.660+0000 7f4130a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41280b1ee0 con 0x7f412815e1e0 2024-01-16T15:11:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.660+0000 7f4130a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41280b2780 con 0x7f4128143ae0 2024-01-16T15:11:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.660+0000 7f4130a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412815e990 con 0x7f41280cf9a0 2024-01-16T15:11:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.661+0000 7f412ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f412815e1e0 0x7f41280702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42354/0 (socket says 172.21.15.125:42354) 2024-01-16T15:11:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.661+0000 7f412ea00640 1 -- 172.21.15.125:0/1438184695 learned_addr learned my addr 172.21.15.125:0/1438184695 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.661+0000 7f412d600640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f412815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.661+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162030872 0 0) 0x7f41280b1ee0 con 0x7f412815e1e0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.661+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108005730 con 0x7f412815e1e0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570223984 0 0) 0x7f41280b2780 con 0x7f4128143ae0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41280b1ee0 con 0x7f4128143ae0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516115349 0 0) 0x7f4108005730 con 0x7f412815e1e0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 msgr2=0x7f412815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f412815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 >> v1:172.21.15.137:6789/0 conn(0x7f4128143ae0 legacy=0x7f41280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41280b2130 con 0x7f412815e1e0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f41200019f0 con 0x7f412815e1e0 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 >> v1:172.21.15.125:6789/0 conn(0x7f412815e1e0 legacy=0x7f41280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4108005d40 0x7f4108006140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41080069d0 0x7f4108006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f4108007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412815e990 con 0x7f4108005d40 2024-01-16T15:11:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410800dc40 con 0x7f41080069d0 2024-01-16T15:11:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410800de00 con 0x7f41280cf9a0 2024-01-16T15:11:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412e000640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4108005d40 0x7f4108006140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412ea00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f4108007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412e000640 1 -- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 msgr2=0x7f4108007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.662+0000 7f412d600640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41080069d0 0x7f4108006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412e000640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f4108007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412e000640 1 -- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41080069d0 msgr2=0x7f4108006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412e000640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41080069d0 0x7f4108006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412e000640 1 -- 172.21.15.125:0/1438184695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108005730 con 0x7f4108005d40 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412ea00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41280cf9a0 0x7f4108007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.663+0000 7f412d600640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41080069d0 0x7f4108006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.664+0000 7f412e000640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4108005d40 0x7f4108006140 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f41180018d0 tx=0x7f4108007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94c63d3c72d749b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.664+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41180052d0 con 0x7f4108005d40 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.664+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118005430 con 0x7f4108005d40 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.664+0000 7f412cc00640 1 -- 172.21.15.125:0/1438184695 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118003810 con 0x7f4108005d40 2024-01-16T15:11:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 -- 172.21.15.125:0/1438184695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4108005d40 msgr2=0x7f4108006140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 --2- 172.21.15.125:0/1438184695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4108005d40 0x7f4108006140 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f41180018d0 tx=0x7f4108007a00 comp rx=0 tx=0).stop 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f412ea00640 1 -- 172.21.15.125:0/1438184695 reap_dead start 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 -- 172.21.15.125:0/1438184695 shutdown_connections 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 -- 172.21.15.125:0/1438184695 >> 172.21.15.125:0/1438184695 conn(0x7f412806a800 msgr2=0x7f412806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 -- 172.21.15.125:0/1438184695 shutdown_connections 2024-01-16T15:11:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.665+0000 7f4130a00640 1 -- 172.21.15.125:0/1438184695 wait complete. 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.667+0000 7f4130a00640 1 Processor -- start 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.667+0000 7f4130a00640 1 -- start start 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4108005d40 0x7f412813f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41281f62e0 0x7f4128131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128140760 con 0x7f4128140b80 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412813fae0 con 0x7f4108005d40 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f4130a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412813fdc0 con 0x7f41281f62e0 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45056/0 (socket says 172.21.15.125:45056) 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 -- 172.21.15.125:0/1233996997 learned_addr learned my addr 172.21.15.125:0/1233996997 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412e000640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4108005d40 0x7f412813f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 -- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41281f62e0 msgr2=0x7f4128131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41281f62e0 0x7f4128131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 -- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4108005d40 msgr2=0x7f412813f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4108005d40 0x7f412813f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.668+0000 7f412d600640 1 -- 172.21.15.125:0/1233996997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41281400a0 con 0x7f4128140b80 2024-01-16T15:11:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f412d600640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f41100108b0 tx=0x7f4110010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000eb60 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411000ecc0 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411001c400 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4128140380 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.669+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41281f6970 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.670+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41281408a0 con 0x7f4128140b80 2024-01-16T15:11:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.671+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f411001c560 con 0x7f4128140b80 2024-01-16T15:11:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.672+0000 7f4126000640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 0x7f410006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.672+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4110088c50 con 0x7f4128140b80 2024-01-16T15:11:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.672+0000 7f412e000640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 0x7f410006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.673+0000 7f412e000640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 0x7f410006e650 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4118034000 tx=0x7f4118031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:08.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.677+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41281408a0 con 0x7f4128140b80 2024-01-16T15:11:08.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.904+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41281408a0 con 0x7f4128140b80 2024-01-16T15:11:08.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.905+0000 7f4126000640 1 -- 172.21.15.125:0/1233996997 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f4110021e90 con 0x7f4128140b80 2024-01-16T15:11:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:08.881 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/4149833712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":4149833712},{"type":"v1","addr":"172.21.15.137:6834","nonce":4149833712}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":6235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2076499076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2076499076},{"type":"v1","addr":"172.21.15.137:6835","nonce":2076499076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8172,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3233056488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3233056488},{"type":"v1","addr":"172.21.15.125:6835","nonce":3233056488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/608622904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":608622904},{"type":"v1","addr":"172.21.15.125:6837","nonce":608622904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[]} 2024-01-16T15:11:08.881 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 57 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 msgr2=0x7f410006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 0x7f410006e650 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4118034000 tx=0x7f4118031000 comp rx=0 tx=0).stop 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 msgr2=0x7f41281f35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f41100108b0 tx=0x7f4110010d70 comp rx=0 tx=0).stop 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 shutdown_connections 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41281f62e0 0x7f4128131520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f410006c190 0x7f410006e650 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4128140b80 0x7f41281f35b0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 --2- 172.21.15.125:0/1233996997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4108005d40 0x7f412813f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:08.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.907+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 >> 172.21.15.125:0/1233996997 conn(0x7f412806a800 msgr2=0x7f412815b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:08.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.908+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 shutdown_connections 2024-01-16T15:11:08.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:08.908+0000 7f4130a00640 1 -- 172.21.15.125:0/1233996997 wait complete. 2024-01-16T15:11:08.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:11:09.116 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.139+0000 7fc811600640 1 Processor -- start 2024-01-16T15:11:09.117 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.140+0000 7fc811600640 1 -- start start 2024-01-16T15:11:09.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.141+0000 7fc811600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc80c0cd170 0x7fc80c071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.141+0000 7fc811600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c071690 con 0x7fc80c0cf9a0 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.141+0000 7fc811600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c071970 con 0x7fc80c14b2c0 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.141+0000 7fc811600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80c14b6a0 con 0x7fc80c0cd170 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc80c0cd170 0x7fc80c071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc80c0cd170 0x7fc80c071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45062/0 (socket says 172.21.15.125:45062) 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc801e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc80c0cf9a0 0x7fc80c070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42362/0 (socket says 172.21.15.125:42362) 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 -- 172.21.15.125:0/618900394 learned_addr learned my addr 172.21.15.125:0/618900394 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 -- 172.21.15.125:0/618900394 >> v1:172.21.15.137:6789/0 conn(0x7fc80c14b2c0 legacy=0x7fc80c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 -- 172.21.15.125:0/618900394 >> v1:172.21.15.125:6789/0 conn(0x7fc80c0cf9a0 legacy=0x7fc80c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80b400640 1 -- 172.21.15.125:0/618900394 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c071c50 con 0x7fc80c0cd170 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.142+0000 7fc80a000640 1 -- 172.21.15.125:0/618900394 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175987073 0 0) 0x7fc80c071690 con 0x7fc80c0cf9a0 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.143+0000 7fc80b400640 1 --2- 172.21.15.125:0/618900394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c071070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc804009e50 tx=0x7fc80402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180cb158f26c290c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc80a000640 1 -- 172.21.15.125:0/618900394 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80402cb20 con 0x7fc80c0cd170 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc80a000640 1 -- 172.21.15.125:0/618900394 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80402cc80 con 0x7fc80c0cd170 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc80a000640 1 -- 172.21.15.125:0/618900394 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040343f0 con 0x7fc80c0cd170 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc811600640 1 -- 172.21.15.125:0/618900394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 msgr2=0x7fc80c071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc811600640 1 --2- 172.21.15.125:0/618900394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c071070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc804009e50 tx=0x7fc80402bf80 comp rx=0 tx=0).stop 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc811600640 1 -- 172.21.15.125:0/618900394 shutdown_connections 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc811600640 1 --2- 172.21.15.125:0/618900394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c071070 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.144+0000 7fc811600640 1 -- 172.21.15.125:0/618900394 >> 172.21.15.125:0/618900394 conn(0x7fc80c06a800 msgr2=0x7fc80c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:09.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.145+0000 7fc811600640 1 -- 172.21.15.125:0/618900394 shutdown_connections 2024-01-16T15:11:09.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.145+0000 7fc811600640 1 -- 172.21.15.125:0/618900394 wait complete. 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.146+0000 7fc811600640 1 Processor -- start 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.146+0000 7fc811600640 1 -- start start 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80c0cf9a0 0x7fc80c143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 0x7fc80c1440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c144630 con 0x7fc80c0cd170 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c160ec0 con 0x7fc80c14b2c0 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc811600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c1638d0 con 0x7fc80c0cf9a0 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.147+0000 7fc80aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45078/0 (socket says 172.21.15.125:45078) 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 -- 172.21.15.125:0/3993026142 learned_addr learned my addr 172.21.15.125:0/3993026142 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc801e00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80c0cf9a0 0x7fc80c143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80b400640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 0x7fc80c1440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 -- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80c0cf9a0 msgr2=0x7fc80c143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80c0cf9a0 0x7fc80c143bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 -- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 msgr2=0x7fc80c1440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 0x7fc80c1440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 -- 172.21.15.125:0/3993026142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009b30 con 0x7fc80c0cd170 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80b400640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 0x7fc80c1440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.148+0000 7fc80aa00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b3f0 tx=0x7fc7fc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.149+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc011830 con 0x7fc80c0cd170 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.149+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc011990 con 0x7fc80c0cd170 2024-01-16T15:11:09.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.149+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc011d40 con 0x7fc80c0cd170 2024-01-16T15:11:09.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.149+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80c160be0 con 0x7fc80c0cd170 2024-01-16T15:11:09.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.149+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80c163dc0 con 0x7fc80c0cd170 2024-01-16T15:11:09.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.150+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80c0675d0 con 0x7fc80c0cd170 2024-01-16T15:11:09.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.151+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fc7fc01a400 con 0x7fc80c0cd170 2024-01-16T15:11:09.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.152+0000 7fc803400640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 0x7fc7e406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.152+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7fc088570 con 0x7fc80c0cd170 2024-01-16T15:11:09.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.152+0000 7fc801e00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 0x7fc7e406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.152+0000 7fc801e00640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 0x7fc7e406e620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0cb330 tx=0x7fc7f4007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.156+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7fc0bde90 con 0x7fc80c0cd170 2024-01-16T15:11:09.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.347+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fc80c06dc20 con 0x7fc80c0cd170 2024-01-16T15:11:09.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.353+0000 7fc803400640 1 -- 172.21.15.125:0/3993026142 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc7fc0150b0 con 0x7fc80c0cd170 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 msgr2=0x7fc7e406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 0x7fc7e406e620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0cb330 tx=0x7fc7f4007b50 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 msgr2=0x7fc80c1645c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b3f0 tx=0x7fc7fc00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 shutdown_connections 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc80c14b2c0 0x7fc80c1440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80c0cf9a0 0x7fc80c143bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc7e406c160 0x7fc7e406e620 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 --2- 172.21.15.125:0/3993026142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc80c0cd170 0x7fc80c1645c0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 >> 172.21.15.125:0/3993026142 conn(0x7fc80c06a800 msgr2=0x7fc80c06cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:09.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 shutdown_connections 2024-01-16T15:11:09.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.356+0000 7fc811600640 1 -- 172.21.15.125:0/3993026142 wait complete. 2024-01-16T15:11:09.352 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:11:09.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.583+0000 7f3b3de00640 1 Processor -- start 2024-01-16T15:11:09.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.583+0000 7f3b3de00640 1 -- start start 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b3de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b38161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b3de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b380ced50 0x7f3b380d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b3de00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380cf1b0 con 0x7f3b380ce8f0 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b3de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b381618a0 con 0x7f3b380ced50 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b3de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380d0de0 con 0x7f3b38160f80 2024-01-16T15:11:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b36000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b380ce8f0 0x7f3b380cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42378/0 (socket says 172.21.15.125:42378) 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.585+0000 7f3b36000640 1 -- 172.21.15.125:0/900544454 learned_addr learned my addr 172.21.15.125:0/900544454 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b36a00640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b38161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b37400640 1 --2- 172.21.15.125:0/900544454 >> v2:172.21.15.137:3300/0 conn(0x7f3b380ced50 0x7f3b380d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074503795 0 0) 0x7f3b380cf1b0 con 0x7f3b380ce8f0 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003520 con 0x7f3b380ce8f0 2024-01-16T15:11:09.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782483205 0 0) 0x7f3b18003520 con 0x7f3b380ce8f0 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 msgr2=0x7f3b38161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b38161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 >> v2:172.21.15.137:3300/0 conn(0x7f3b380ced50 msgr2=0x7f3b380d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 --2- 172.21.15.125:0/900544454 >> v2:172.21.15.137:3300/0 conn(0x7f3b380ced50 0x7f3b380d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b380d10c0 con 0x7f3b380ce8f0 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.586+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3b280019f0 con 0x7f3b380ce8f0 2024-01-16T15:11:09.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 >> v1:172.21.15.125:6789/0 conn(0x7f3b380ce8f0 legacy=0x7f3b380cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18003970 0x7f3b18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381618a0 con 0x7f3b18004d00 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380d0de0 con 0x7f3b18003970 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1800b980 con 0x7f3b38160f80 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b36a00640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18003970 0x7f3b18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b36000640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 -- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 msgr2=0x7f3b180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 -- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18003970 msgr2=0x7f3b18003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18003970 0x7f3b18003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b37400640 1 -- 172.21.15.125:0/900544454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18003520 con 0x7f3b18004d00 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.587+0000 7f3b36000640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38160f80 0x7f3b180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.588+0000 7f3b37400640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009ac0 tx=0x7f3b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780ec0ca40eea76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3002c710 con 0x7f3b18004d00 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3002c870 con 0x7f3b18004d00 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b35600640 1 -- 172.21.15.125:0/900544454 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3002cc00 con 0x7f3b18004d00 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b3de00640 1 -- 172.21.15.125:0/900544454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b18004d00 msgr2=0x7f3b18005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b3de00640 1 --2- 172.21.15.125:0/900544454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009ac0 tx=0x7f3b180057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b37400640 1 -- 172.21.15.125:0/900544454 reap_dead start 2024-01-16T15:11:09.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b3de00640 1 -- 172.21.15.125:0/900544454 shutdown_connections 2024-01-16T15:11:09.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b3de00640 1 -- 172.21.15.125:0/900544454 >> 172.21.15.125:0/900544454 conn(0x7f3b3809c550 msgr2=0x7f3b380d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:09.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.589+0000 7f3b3de00640 1 -- 172.21.15.125:0/900544454 shutdown_connections 2024-01-16T15:11:09.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.590+0000 7f3b3de00640 1 -- 172.21.15.125:0/900544454 wait complete. 2024-01-16T15:11:09.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.591+0000 7f3b3de00640 1 Processor -- start 2024-01-16T15:11:09.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.591+0000 7f3b3de00640 1 -- start start 2024-01-16T15:11:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18004d00 0x7f3b38151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38151010 0x7f3b380ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380b1850 con 0x7f3b380b0e90 2024-01-16T15:11:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381568f0 con 0x7f3b18004d00 2024-01-16T15:11:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.592+0000 7f3b3de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380caaf0 con 0x7f3b38151010 2024-01-16T15:11:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45094/0 (socket says 172.21.15.125:45094) 2024-01-16T15:11:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 -- 172.21.15.125:0/3778641492 learned_addr learned my addr 172.21.15.125:0/3778641492 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b36a00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18004d00 0x7f3b38151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b36000640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38151010 0x7f3b380ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 -- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38151010 msgr2=0x7f3b380ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38151010 0x7f3b380ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 -- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18004d00 msgr2=0x7f3b38151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18004d00 0x7f3b38151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 -- 172.21.15.125:0/3778641492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b380cd2d0 con 0x7f3b380b0e90 2024-01-16T15:11:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.593+0000 7f3b37400640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3b30037040 tx=0x7f3b30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.594+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3002c710 con 0x7f3b380b0e90 2024-01-16T15:11:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.594+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30003850 con 0x7f3b380b0e90 2024-01-16T15:11:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.594+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30003c00 con 0x7f3b380b0e90 2024-01-16T15:11:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.594+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b38155ce0 con 0x7f3b380b0e90 2024-01-16T15:11:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.594+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b381562a0 con 0x7f3b380b0e90 2024-01-16T15:11:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.595+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b380675d0 con 0x7f3b380b0e90 2024-01-16T15:11:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.596+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3b3003e440 con 0x7f3b380b0e90 2024-01-16T15:11:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.597+0000 7f3b2f400640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 0x7f3b1006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.597+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3b30000e40 con 0x7f3b380b0e90 2024-01-16T15:11:09.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.597+0000 7f3b36a00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 0x7f3b1006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.597+0000 7f3b36a00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 0x7f3b1006e620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009950 tx=0x7f3b20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.601+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b30076140 con 0x7f3b380b0e90 2024-01-16T15:11:09.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.793+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f3b380d3920 con 0x7f3b380b0e90 2024-01-16T15:11:09.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.799+0000 7f3b2f400640 1 -- 172.21.15.125:0/3778641492 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v34) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3b30079760 con 0x7f3b380b0e90 2024-01-16T15:11:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 msgr2=0x7f3b1006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 0x7f3b1006e620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009950 tx=0x7f3b20008040 comp rx=0 tx=0).stop 2024-01-16T15:11:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 msgr2=0x7f3b380b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3b30037040 tx=0x7f3b30004730 comp rx=0 tx=0).stop 2024-01-16T15:11:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 shutdown_connections 2024-01-16T15:11:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b1006c160 0x7f3b1006e620 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b380b0e90 0x7f3b380b1310 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b38151010 0x7f3b380ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 --2- 172.21.15.125:0/3778641492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b18004d00 0x7f3b38151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.802+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 >> 172.21.15.125:0/3778641492 conn(0x7f3b3809c550 msgr2=0x7f3b380b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.803+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 shutdown_connections 2024-01-16T15:11:09.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:09.803+0000 7f3b3de00640 1 -- 172.21.15.125:0/3778641492 wait complete. 2024-01-16T15:11:09.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:11:10.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.031+0000 7f375b200640 1 Processor -- start 2024-01-16T15:11:10.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.032+0000 7f375b200640 1 -- start start 2024-01-16T15:11:10.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.033+0000 7f375b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f375c0ccb50 0x7f375c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.033+0000 7f375b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f375c15bf10 0x7f375c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.033+0000 7f375b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c15f4e0 con 0x7f375c0adf20 2024-01-16T15:11:10.014 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.033+0000 7f375b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375c15f970 con 0x7f375c0ccb50 2024-01-16T15:11:10.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.033+0000 7f375b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375c071130 con 0x7f375c15bf10 2024-01-16T15:11:10.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f375c0ccb50 0x7f375c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f375c0ccb50 0x7f375c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45108/0 (socket says 172.21.15.125:45108) 2024-01-16T15:11:10.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 -- 172.21.15.125:0/1973935144 learned_addr learned my addr 172.21.15.125:0/1973935144 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:10.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f375a800640 1 --2- 172.21.15.125:0/1973935144 >> v2:172.21.15.137:3300/0 conn(0x7f375c15bf10 0x7f375c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 -- 172.21.15.125:0/1973935144 >> v2:172.21.15.137:3300/0 conn(0x7f375c15bf10 msgr2=0x7f375c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 --2- 172.21.15.125:0/1973935144 >> v2:172.21.15.137:3300/0 conn(0x7f375c15bf10 0x7f375c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 -- 172.21.15.125:0/1973935144 >> v1:172.21.15.137:6789/0 conn(0x7f375c0adf20 legacy=0x7f375c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.034+0000 7f3751200640 1 -- 172.21.15.125:0/1973935144 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c15ebb0 con 0x7f375c0ccb50 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.035+0000 7f3751200640 1 --2- 172.21.15.125:0/1973935144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c070a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f375c06ff20 tx=0x7f374c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4b59a1f0a78835 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.036+0000 7f3759400640 1 -- 172.21.15.125:0/1973935144 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c0317e0 con 0x7f375c0ccb50 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.036+0000 7f3759400640 1 -- 172.21.15.125:0/1973935144 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374c031940 con 0x7f375c0ccb50 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.036+0000 7f3759400640 1 -- 172.21.15.125:0/1973935144 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c031cd0 con 0x7f375c0ccb50 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.036+0000 7f375b200640 1 -- 172.21.15.125:0/1973935144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 msgr2=0x7f375c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.036+0000 7f375b200640 1 --2- 172.21.15.125:0/1973935144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c070a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f375c06ff20 tx=0x7f374c02b880 comp rx=0 tx=0).stop 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 -- 172.21.15.125:0/1973935144 shutdown_connections 2024-01-16T15:11:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 --2- 172.21.15.125:0/1973935144 >> v2:172.21.15.137:3300/0 conn(0x7f375c15bf10 0x7f375c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 --2- 172.21.15.125:0/1973935144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c070a30 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 -- 172.21.15.125:0/1973935144 >> 172.21.15.125:0/1973935144 conn(0x7f375c06a7d0 msgr2=0x7f375c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 -- 172.21.15.125:0/1973935144 shutdown_connections 2024-01-16T15:11:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.037+0000 7f375b200640 1 -- 172.21.15.125:0/1973935144 wait complete. 2024-01-16T15:11:10.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.038+0000 7f375b200640 1 Processor -- start 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.039+0000 7f375b200640 1 -- start start 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.039+0000 7f375b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f375c0adf20 0x7f375c134390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f375b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f375b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f375c15bf10 0x7f375c134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f375b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0aab30 con 0x7f375c0ccb50 2024-01-16T15:11:10.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f375b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0b2b40 con 0x7f375c15bf10 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f375b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c137c70 con 0x7f375c0adf20 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45122/0 (socket says 172.21.15.125:45122) 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 -- 172.21.15.125:0/1971746665 learned_addr learned my addr 172.21.15.125:0/1971746665 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 -- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f375c0adf20 msgr2=0x7f375c134390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f375c0adf20 0x7f375c134390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 -- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f375c15bf10 msgr2=0x7f375c134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f375c15bf10 0x7f375c134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.040+0000 7f3751200640 1 -- 172.21.15.125:0/1971746665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c0094d0 con 0x7f375c0ccb50 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f3751200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f374c02b8c0 tx=0x7f374c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c032880 con 0x7f375c0ccb50 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374c032a00 con 0x7f375c0ccb50 2024-01-16T15:11:10.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c032db0 con 0x7f375c0ccb50 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c0aa570 con 0x7f375c0ccb50 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.041+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c1348d0 con 0x7f375c0ccb50 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.042+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c0b28a0 con 0x7f375c0ccb50 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.043+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f374c04d3f0 con 0x7f375c0ccb50 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.044+0000 7f3752a00640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 0x7f373406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.044+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f374c0aa740 con 0x7f375c0ccb50 2024-01-16T15:11:10.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.044+0000 7f3759e00640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 0x7f373406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.045+0000 7f3759e00640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 0x7f373406e620 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f375c0ca9c0 tx=0x7f37440079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.048+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f375c0b28a0 con 0x7f375c0ccb50 2024-01-16T15:11:10.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.240+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f375c0b2c80 con 0x7f375c0ccb50 2024-01-16T15:11:10.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.245+0000 7f3752a00640 1 -- 172.21.15.125:0/1971746665 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f375c0b2c80 con 0x7f375c0ccb50 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 msgr2=0x7f373406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 0x7f373406e620 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f375c0ca9c0 tx=0x7f37440079b0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 msgr2=0x7f375c0b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f374c02b8c0 tx=0x7f374c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 shutdown_connections 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f375c15bf10 0x7f375c134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f373406c160 0x7f373406e620 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375c0ccb50 0x7f375c0b2f60 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 --2- 172.21.15.125:0/1971746665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f375c0adf20 0x7f375c134390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 >> 172.21.15.125:0/1971746665 conn(0x7f375c06a7d0 msgr2=0x7f375c15cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 shutdown_connections 2024-01-16T15:11:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.248+0000 7f375b200640 1 -- 172.21.15.125:0/1971746665 wait complete. 2024-01-16T15:11:10.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:11:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.477+0000 7f404da00640 1 Processor -- start 2024-01-16T15:11:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.477+0000 7f404da00640 1 -- start start 2024-01-16T15:11:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f404da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f404813cfa0 0x7f404815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f404da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40480cd530 0x7f404813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f404da00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40480cdec0 con 0x7f40480cd9b0 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f404da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40480ce1a0 con 0x7f404813cfa0 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f404da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40480ce480 con 0x7f40480cd530 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f4047e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f404813cfa0 0x7f404815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f4047e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f404813cfa0 0x7f404815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45128/0 (socket says 172.21.15.125:45128) 2024-01-16T15:11:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.479+0000 7f4047e00640 1 -- 172.21.15.125:0/713083009 learned_addr learned my addr 172.21.15.125:0/713083009 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.480+0000 7f4047400640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40480cd530 0x7f404813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.480+0000 7f4047e00640 1 -- 172.21.15.125:0/713083009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40480cd530 msgr2=0x7f404813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.480+0000 7f4047e00640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40480cd530 0x7f404813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.480+0000 7f4047e00640 1 -- 172.21.15.125:0/713083009 >> v1:172.21.15.137:6789/0 conn(0x7f40480cd9b0 legacy=0x7f404815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.480+0000 7f4047e00640 1 -- 172.21.15.125:0/713083009 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40480ce760 con 0x7f404813cfa0 2024-01-16T15:11:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.481+0000 7f4047e00640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f404813cfa0 0x7f404815afb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f403000bb10 tx=0x7f403002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed97c125d5b80cd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.481+0000 7f4046a00640 1 -- 172.21.15.125:0/713083009 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403002ede0 con 0x7f404813cfa0 2024-01-16T15:11:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.481+0000 7f4046a00640 1 -- 172.21.15.125:0/713083009 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030036440 con 0x7f404813cfa0 2024-01-16T15:11:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.481+0000 7f4046a00640 1 -- 172.21.15.125:0/713083009 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40300367d0 con 0x7f404813cfa0 2024-01-16T15:11:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 -- 172.21.15.125:0/713083009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f404813cfa0 msgr2=0x7f404815afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f404813cfa0 0x7f404815afb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f403000bb10 tx=0x7f403002e030 comp rx=0 tx=0).stop 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 -- 172.21.15.125:0/713083009 shutdown_connections 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40480cd530 0x7f404813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 --2- 172.21.15.125:0/713083009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f404813cfa0 0x7f404815afb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 -- 172.21.15.125:0/713083009 >> 172.21.15.125:0/713083009 conn(0x7f404806a800 msgr2=0x7f404806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 -- 172.21.15.125:0/713083009 shutdown_connections 2024-01-16T15:11:10.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.482+0000 7f404da00640 1 -- 172.21.15.125:0/713083009 wait complete. 2024-01-16T15:11:10.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.484+0000 7f404da00640 1 Processor -- start 2024-01-16T15:11:10.460 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.484+0000 7f404da00640 1 -- start start 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40480cd530 0x7f4048158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f404813cfa0 0x7f4048154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048158d40 con 0x7f40480cd9b0 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048159020 con 0x7f40480cd530 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4048159300 con 0x7f404813cfa0 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45142/0 (socket says 172.21.15.125:45142) 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047400640 1 -- 172.21.15.125:0/1779944677 learned_addr learned my addr 172.21.15.125:0/1779944677 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:10.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f404d000640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f404813cfa0 0x7f4048154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047400640 1 -- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f404813cfa0 msgr2=0x7f4048154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047e00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40480cd530 0x7f4048158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.485+0000 7f4047400640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f404813cfa0 0x7f4048154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f4047400640 1 -- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40480cd530 msgr2=0x7f4048158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f4047400640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40480cd530 0x7f4048158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f4047400640 1 -- 172.21.15.125:0/1779944677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403000b770 con 0x7f40480cd9b0 2024-01-16T15:11:10.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f4047400640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f403800ce40 tx=0x7f403800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038003b90 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4038003cf0 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40481595e0 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038004270 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.486+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4048159de0 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.487+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4048067450 con 0x7f40480cd9b0 2024-01-16T15:11:10.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.488+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f40380043d0 con 0x7f40480cd9b0 2024-01-16T15:11:10.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.490+0000 7f403fe00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 0x7f402006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:10.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.490+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4038012070 con 0x7f40480cd9b0 2024-01-16T15:11:10.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.490+0000 7f4047e00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 0x7f402006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:10.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.491+0000 7f4047e00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 0x7f402006e620 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4030038000 tx=0x7f4030008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:10.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.494+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40380559c0 con 0x7f40480cd9b0 2024-01-16T15:11:10.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:10.758+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f40480cbff0 con 0x7f40480cd9b0 2024-01-16T15:11:11.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:11.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:11.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.520+0000 7f403fe00640 1 -- 172.21.15.125:0/1779944677 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v654) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4038058fe0 con 0x7f40480cd9b0 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 msgr2=0x7f402006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 0x7f402006e620 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4030038000 tx=0x7f4030008040 comp rx=0 tx=0).stop 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 msgr2=0x7f40481f2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f403800ce40 tx=0x7f403800a660 comp rx=0 tx=0).stop 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 shutdown_connections 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f404813cfa0 0x7f4048154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f402006c160 0x7f402006e620 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40480cd9b0 0x7f40481f2ee0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 --2- 172.21.15.125:0/1779944677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40480cd530 0x7f4048158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.523+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 >> 172.21.15.125:0/1779944677 conn(0x7f404806a800 msgr2=0x7f40480717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:11.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.524+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 shutdown_connections 2024-01-16T15:11:11.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.524+0000 7f404da00640 1 -- 172.21.15.125:0/1779944677 wait complete. 2024-01-16T15:11:11.523 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:11:11.523 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:11:11.523 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:11:11.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.756+0000 7f31fa600640 1 Processor -- start 2024-01-16T15:11:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.756+0000 7f31fa600640 1 -- start start 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31fa600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31fa600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31f40cf290 0x7f31f40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31fa600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f4070c40 con 0x7f31f40b3ae0 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31fa600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f4071450 con 0x7f31f40cf290 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31fa600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cfc60 con 0x7f31f40ae5a0 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31f3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31f40cf290 0x7f31f40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31f3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31f40cf290 0x7f31f40cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45146/0 (socket says 172.21.15.125:45146) 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31f3e00640 1 -- 172.21.15.125:0/596297211 learned_addr learned my addr 172.21.15.125:0/596297211 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.758+0000 7f31f3400640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.759+0000 7f31f3e00640 1 -- 172.21.15.125:0/596297211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 msgr2=0x7f31f415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.759+0000 7f31f3e00640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.759+0000 7f31f3e00640 1 -- 172.21.15.125:0/596297211 >> v1:172.21.15.137:6789/0 conn(0x7f31f40b3ae0 legacy=0x7f31f406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.759+0000 7f31f3e00640 1 -- 172.21.15.125:0/596297211 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f406c470 con 0x7f31f40cf290 2024-01-16T15:11:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.760+0000 7f31f3e00640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40cf290 0x7f31f40cf720 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e50 tx=0x7f31ec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d550653fd0ae2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.760+0000 7f31f2a00640 1 -- 172.21.15.125:0/596297211 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec02cb00 con 0x7f31f40cf290 2024-01-16T15:11:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.760+0000 7f31f2a00640 1 -- 172.21.15.125:0/596297211 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec02cc60 con 0x7f31f40cf290 2024-01-16T15:11:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.760+0000 7f31f2a00640 1 -- 172.21.15.125:0/596297211 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0343f0 con 0x7f31f40cf290 2024-01-16T15:11:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 -- 172.21.15.125:0/596297211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40cf290 msgr2=0x7f31f40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40cf290 0x7f31f40cf720 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e50 tx=0x7f31ec02bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 -- 172.21.15.125:0/596297211 shutdown_connections 2024-01-16T15:11:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40cf290 0x7f31f40cf720 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 --2- 172.21.15.125:0/596297211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 -- 172.21.15.125:0/596297211 >> 172.21.15.125:0/596297211 conn(0x7f31f406a800 msgr2=0x7f31f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 -- 172.21.15.125:0/596297211 shutdown_connections 2024-01-16T15:11:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.761+0000 7f31fa600640 1 -- 172.21.15.125:0/596297211 wait complete. 2024-01-16T15:11:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.763+0000 7f31fa600640 1 Processor -- start 2024-01-16T15:11:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.763+0000 7f31fa600640 1 -- start start 2024-01-16T15:11:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f414c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f40cf290 0x7f31f415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f41f76c0 con 0x7f31f40b3ae0 2024-01-16T15:11:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f41f6a40 con 0x7f31f40cf290 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31fa600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f41f6d20 con 0x7f31f40ae5a0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45152/0 (socket says 172.21.15.125:45152) 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 -- 172.21.15.125:0/1520467353 learned_addr learned my addr 172.21.15.125:0/1520467353 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31f3400640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f414c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 -- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 msgr2=0x7f31f414c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f414c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 -- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f40cf290 msgr2=0x7f31f415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31f3e00640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f40cf290 0x7f31f415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.764+0000 7f31ebe00640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f40cf290 0x7f31f415fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31ebe00640 1 -- 172.21.15.125:0/1520467353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009b30 con 0x7f31f40b3ae0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31ebe00640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00c430 tx=0x7f31dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc011790 con 0x7f31f40b3ae0 2024-01-16T15:11:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc0118f0 con 0x7f31f40b3ae0 2024-01-16T15:11:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f41f7000 con 0x7f31f40b3ae0 2024-01-16T15:11:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc011ce0 con 0x7f31f40b3ae0 2024-01-16T15:11:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.765+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f414cc40 con 0x7f31f40b3ae0 2024-01-16T15:11:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.766+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f41f7800 con 0x7f31f40b3ae0 2024-01-16T15:11:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.767+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f31dc01c460 con 0x7f31f40b3ae0 2024-01-16T15:11:11.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.768+0000 7f31eb400640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 0x7f31cc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:11.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.768+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f31dc088930 con 0x7f31f40b3ae0 2024-01-16T15:11:11.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.768+0000 7f31f3400640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 0x7f31cc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:11.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.769+0000 7f31f3400640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 0x7f31cc06e620 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f31e4006fd0 tx=0x7f31e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:11.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.773+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31dc054f10 con 0x7f31f40b3ae0 2024-01-16T15:11:11.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.968+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31f40675d0 con 0x7f31f40b3ae0 2024-01-16T15:11:11.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.969+0000 7f31eb400640 1 -- 172.21.15.125:0/1520467353 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13035 (secure 0 0 0) 0x7f31dc058530 con 0x7f31f40b3ae0 2024-01-16T15:11:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:11.946 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:04.474760+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/589280594":"2024-01-17T15:11:01.459274+0000","172.21.15.125:6836/589280594":"2024-01-17T15:11:01.459274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.971+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 msgr2=0x7f31cc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.971+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 0x7f31cc06e620 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f31e4006fd0 tx=0x7f31e4008040 comp rx=0 tx=0).stop 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 msgr2=0x7f31f414e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00c430 tx=0x7f31dc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 shutdown_connections 2024-01-16T15:11:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f40cf290 0x7f31f415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c160 0x7f31cc06e620 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40b3ae0 0x7f31f414e450 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 --2- 172.21.15.125:0/1520467353 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f40ae5a0 0x7f31f414c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 >> 172.21.15.125:0/1520467353 conn(0x7f31f406a800 msgr2=0x7f31f415a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 shutdown_connections 2024-01-16T15:11:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:11.972+0000 7f31fa600640 1 -- 172.21.15.125:0/1520467353 wait complete. 2024-01-16T15:11:11.967 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:11.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:12.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.207+0000 7fb78a800640 1 Processor -- start 2024-01-16T15:11:12.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.208+0000 7fb78a800640 1 -- start start 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb78a800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb78413df80 0x7fb78415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb78a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb7840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb78a800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78406c470 con 0x7fb78415f9c0 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb78a800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78406c750 con 0x7fb78413df80 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb78a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78406ca30 con 0x7fb78413e360 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb783400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb78413df80 0x7fb78415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb783400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb78413df80 0x7fb78415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45154/0 (socket says 172.21.15.125:45154) 2024-01-16T15:11:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.209+0000 7fb783400640 1 -- 172.21.15.125:0/782333331 learned_addr learned my addr 172.21.15.125:0/782333331 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:12.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.210+0000 7fb77a800640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb7840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.210+0000 7fb783400640 1 -- 172.21.15.125:0/782333331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 msgr2=0x7fb7840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:12.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.210+0000 7fb783400640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb7840cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.210+0000 7fb783400640 1 -- 172.21.15.125:0/782333331 >> v1:172.21.15.137:6789/0 conn(0x7fb78415f9c0 legacy=0x7fb7841408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:12.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.210+0000 7fb783400640 1 -- 172.21.15.125:0/782333331 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78406cd10 con 0x7fb78413df80 2024-01-16T15:11:12.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.211+0000 7fb783400640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78413df80 0x7fb78415bee0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fb774009870 tx=0x7fb77402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd3cdca818a8978 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.211+0000 7fb782a00640 1 -- 172.21.15.125:0/782333331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774031640 con 0x7fb78413df80 2024-01-16T15:11:12.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.211+0000 7fb782a00640 1 -- 172.21.15.125:0/782333331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7740317a0 con 0x7fb78413df80 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.211+0000 7fb782a00640 1 -- 172.21.15.125:0/782333331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774031b30 con 0x7fb78413df80 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 -- 172.21.15.125:0/782333331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78413df80 msgr2=0x7fb78415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78413df80 0x7fb78415bee0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fb774009870 tx=0x7fb77402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 -- 172.21.15.125:0/782333331 shutdown_connections 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb7840cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 --2- 172.21.15.125:0/782333331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78413df80 0x7fb78415bee0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 -- 172.21.15.125:0/782333331 >> 172.21.15.125:0/782333331 conn(0x7fb78406a800 msgr2=0x7fb78406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 -- 172.21.15.125:0/782333331 shutdown_connections 2024-01-16T15:11:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.212+0000 7fb78a800640 1 -- 172.21.15.125:0/782333331 wait complete. 2024-01-16T15:11:12.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.214+0000 7fb78a800640 1 Processor -- start 2024-01-16T15:11:12.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.214+0000 7fb78a800640 1 -- start start 2024-01-16T15:11:12.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb78413df80 0x7fb7841e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb784150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb784154170 con 0x7fb78415f9c0 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb784150bd0 con 0x7fb78413df80 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb78a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7841534f0 con 0x7fb78413e360 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45160/0 (socket says 172.21.15.125:45160) 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 -- 172.21.15.125:0/2363384446 learned_addr learned my addr 172.21.15.125:0/2363384446 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783400640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb78413df80 0x7fb7841e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb77a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb784150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 -- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 msgr2=0x7fb784150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb784150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 -- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb78413df80 msgr2=0x7fb7841e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb78413df80 0x7fb7841e5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.215+0000 7fb783e00640 1 -- 172.21.15.125:0/2363384446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740094d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb783e00640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c780 tx=0x7fb77c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c004b60 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c00ce90 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c00a850 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7841537d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.216+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb784153d90 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.217+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78413b1d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.218+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fb77c00a9b0 con 0x7fb78415f9c0 2024-01-16T15:11:12.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.220+0000 7fb77be00640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 0x7fb75c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.220+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb77c088890 con 0x7fb78415f9c0 2024-01-16T15:11:12.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.220+0000 7fb783400640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 0x7fb75c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.220+0000 7fb783400640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 0x7fb75c06e620 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.224+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb78413b1d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.413+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb78413b1d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.414+0000 7fb77be00640 1 -- 172.21.15.125:0/2363384446 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb78413b1d0 con 0x7fb78415f9c0 2024-01-16T15:11:12.390 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:11:12.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.416+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 msgr2=0x7fb75c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.416+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 0x7fb75c06e620 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403c040 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.416+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 msgr2=0x7fb784152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.416+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c780 tx=0x7fb77c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 shutdown_connections 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb75c06c160 0x7fb75c06e620 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb78415f9c0 0x7fb784152cc0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb78413e360 0x7fb784150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 --2- 172.21.15.125:0/2363384446 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb78413df80 0x7fb7841e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 >> 172.21.15.125:0/2363384446 conn(0x7fb78406a800 msgr2=0x7fb784070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 shutdown_connections 2024-01-16T15:11:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.417+0000 7fb78a800640 1 -- 172.21.15.125:0/2363384446 wait complete. 2024-01-16T15:11:12.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:11:12.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.637+0000 7f77af200640 1 Processor -- start 2024-01-16T15:11:12.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.637+0000 7f77af200640 1 -- start start 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.638+0000 7f77af200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77b015afb0 0x7f77b013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.638+0000 7f77af200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77b013fae0 0x7f77b00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.638+0000 7f77af200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b013ff30 con 0x7f77b013cfa0 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77af200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b00cdc40 con 0x7f77b013fae0 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77af200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b00cdec0 con 0x7f77b015afb0 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ade00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f77b013cfa0 0x7f77b00cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42384/0 (socket says 172.21.15.125:42384) 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ade00640 1 -- 172.21.15.125:0/3072879093 learned_addr learned my addr 172.21.15.125:0/3072879093 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ae800640 1 --2- 172.21.15.125:0/3072879093 >> v2:172.21.15.125:3300/0 conn(0x7f77b013fae0 0x7f77b00b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ad400640 1 --2- 172.21.15.125:0/3072879093 >> v2:172.21.15.137:3300/0 conn(0x7f77b015afb0 0x7f77b013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77aca00640 1 -- 172.21.15.125:0/3072879093 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234957486 0 0) 0x7f77a8001110 con 0x7f77b013cfa0 2024-01-16T15:11:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ae800640 1 -- 172.21.15.125:0/3072879093 >> v2:172.21.15.137:3300/0 conn(0x7f77b015afb0 msgr2=0x7f77b013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ae800640 1 --2- 172.21.15.125:0/3072879093 >> v2:172.21.15.137:3300/0 conn(0x7f77b015afb0 0x7f77b013f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ae800640 1 -- 172.21.15.125:0/3072879093 >> v1:172.21.15.125:6789/0 conn(0x7f77b013cfa0 legacy=0x7f77b00cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.639+0000 7f77ae800640 1 -- 172.21.15.125:0/3072879093 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b00ce1a0 con 0x7f77b013fae0 2024-01-16T15:11:12.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77ae800640 1 --2- 172.21.15.125:0/3072879093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013fae0 0x7f77b00b2040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b3fd0 tx=0x7f77a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c10cf46f6ef0707 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77aca00640 1 -- 172.21.15.125:0/3072879093 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a002e070 con 0x7f77b013fae0 2024-01-16T15:11:12.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77aca00640 1 -- 172.21.15.125:0/3072879093 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a00315a0 con 0x7f77b013fae0 2024-01-16T15:11:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77aca00640 1 -- 172.21.15.125:0/3072879093 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0031930 con 0x7f77b013fae0 2024-01-16T15:11:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77af200640 1 -- 172.21.15.125:0/3072879093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013fae0 msgr2=0x7f77b00b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.641+0000 7f77af200640 1 --2- 172.21.15.125:0/3072879093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013fae0 0x7f77b00b2040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b3fd0 tx=0x7f77a002b840 comp rx=0 tx=0).stop 2024-01-16T15:11:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 -- 172.21.15.125:0/3072879093 shutdown_connections 2024-01-16T15:11:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 --2- 172.21.15.125:0/3072879093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013fae0 0x7f77b00b2040 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 --2- 172.21.15.125:0/3072879093 >> v2:172.21.15.137:3300/0 conn(0x7f77b015afb0 0x7f77b013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 -- 172.21.15.125:0/3072879093 >> 172.21.15.125:0/3072879093 conn(0x7f77b006a800 msgr2=0x7f77b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 -- 172.21.15.125:0/3072879093 shutdown_connections 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.642+0000 7f77af200640 1 -- 172.21.15.125:0/3072879093 wait complete. 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.643+0000 7f77af200640 1 Processor -- start 2024-01-16T15:11:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.643+0000 7f77af200640 1 -- start start 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77b013fae0 0x7f77b0132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77b015afb0 0x7f77b00712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0153e90 con 0x7f77b013cfa0 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0165d10 con 0x7f77b013fae0 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77af200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0165ff0 con 0x7f77b015afb0 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.644+0000 7f77ade00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45174/0 (socket says 172.21.15.125:45174) 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 -- 172.21.15.125:0/3768718760 learned_addr learned my addr 172.21.15.125:0/3768718760 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 -- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77b015afb0 msgr2=0x7f77b00712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77b015afb0 0x7f77b00712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 -- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77b013fae0 msgr2=0x7f77b0132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77b013fae0 0x7f77b0132620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 -- 172.21.15.125:0/3768718760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0009400 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77ade00640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c780 tx=0x7f77a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80072d0 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a8007450 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80047e0 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.645+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b01662d0 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.646+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b0166890 con 0x7f77b013cfa0 2024-01-16T15:11:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.647+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b0166ab0 con 0x7f77b013cfa0 2024-01-16T15:11:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.648+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f77a8004960 con 0x7f77b013cfa0 2024-01-16T15:11:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.649+0000 7f77a6000640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 0x7f778806e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.649+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f77a8087980 con 0x7f77b013cfa0 2024-01-16T15:11:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.649+0000 7f77ad400640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 0x7f778806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.649+0000 7f77ad400640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 0x7f778806e650 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f77980070b0 tx=0x7f7798007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:12.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.653+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77a8053f60 con 0x7f77b013cfa0 2024-01-16T15:11:12.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:12.910+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f77b013ff30 con 0x7f77b013cfa0 2024-01-16T15:11:13.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.532+0000 7f77a6000640 1 -- 172.21.15.125:0/3768718760 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v656) v1 ==== 158+0+0 (secure 0 0 0) 0x7f77b013ff30 con 0x7f77b013cfa0 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.535+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 msgr2=0x7f778806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.535+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 0x7f778806e650 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f77980070b0 tx=0x7f7798007040 comp rx=0 tx=0).stop 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.535+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 msgr2=0x7f77b0153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.535+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c780 tx=0x7f77a800cc40 comp rx=0 tx=0).stop 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 shutdown_connections 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77b015afb0 0x7f77b00712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77b013fae0 0x7f77b0132620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f778806c190 0x7f778806e650 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 --2- 172.21.15.125:0/3768718760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77b013cfa0 0x7f77b0153950 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 >> 172.21.15.125:0/3768718760 conn(0x7f77b006a800 msgr2=0x7f77b006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:13.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 shutdown_connections 2024-01-16T15:11:13.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.536+0000 7f77af200640 1 -- 172.21.15.125:0/3768718760 wait complete. 2024-01-16T15:11:13.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:11:13.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.770+0000 7f3e1b800640 1 Processor -- start 2024-01-16T15:11:13.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.771+0000 7f3e1b800640 1 -- start start 2024-01-16T15:11:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e14074520 0x7f3e140ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 0x7f3e140ac590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e14162100 con 0x7f3e14074520 2024-01-16T15:11:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e140ccf20 con 0x7f3e140ae750 2024-01-16T15:11:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e1b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140cd200 con 0x7f3e140cc1e0 2024-01-16T15:11:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e13e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e14074520 0x7f3e140ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e13e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e14074520 0x7f3e140ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45188/0 (socket says 172.21.15.125:45188) 2024-01-16T15:11:13.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e13e00640 1 -- 172.21.15.125:0/3776370905 learned_addr learned my addr 172.21.15.125:0/3776370905 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e18e00640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.772+0000 7f3e19800640 1 --2- 172.21.15.125:0/3776370905 >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 0x7f3e140ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e13e00640 1 -- 172.21.15.125:0/3776370905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 msgr2=0x7f3e140cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e13e00640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e13e00640 1 -- 172.21.15.125:0/3776370905 >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 msgr2=0x7f3e140ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e13e00640 1 --2- 172.21.15.125:0/3776370905 >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 0x7f3e140ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e13e00640 1 -- 172.21.15.125:0/3776370905 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1406daa0 con 0x7f3e14074520 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.773+0000 7f3e19800640 1 --2- 172.21.15.125:0/3776370905 >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 0x7f3e140ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e13e00640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e14074520 0x7f3e140ac050 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3e140716e0 tx=0x7f3dfc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3113d4dfd5aade64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:13.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e13400640 1 -- 172.21.15.125:0/3776370905 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc031740 con 0x7f3e14074520 2024-01-16T15:11:13.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e13400640 1 -- 172.21.15.125:0/3776370905 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc0318a0 con 0x7f3e14074520 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e13400640 1 -- 172.21.15.125:0/3776370905 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc031c30 con 0x7f3e14074520 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e1b800640 1 -- 172.21.15.125:0/3776370905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e14074520 msgr2=0x7f3e140ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.774+0000 7f3e1b800640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e14074520 0x7f3e140ac050 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3e140716e0 tx=0x7f3dfc02b7e0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 -- 172.21.15.125:0/3776370905 shutdown_connections 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 --2- 172.21.15.125:0/3776370905 >> v2:172.21.15.137:3300/0 conn(0x7f3e140ae750 0x7f3e140ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e14074520 0x7f3e140ac050 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 --2- 172.21.15.125:0/3776370905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 -- 172.21.15.125:0/3776370905 >> 172.21.15.125:0/3776370905 conn(0x7f3e140b0d70 msgr2=0x7f3e140b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 -- 172.21.15.125:0/3776370905 shutdown_connections 2024-01-16T15:11:13.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.775+0000 7f3e1b800640 1 -- 172.21.15.125:0/3776370905 wait complete. 2024-01-16T15:11:13.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.777+0000 7f3e1b800640 1 Processor -- start 2024-01-16T15:11:13.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.777+0000 7f3e1b800640 1 -- start start 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 0x7f3e14159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140c2c80 con 0x7f3e140ae750 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140c3800 con 0x7f3e14074520 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e1b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140c2ed0 con 0x7f3e140cc1e0 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45202/0 (socket says 172.21.15.125:45202) 2024-01-16T15:11:13.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 -- 172.21.15.125:0/2814898077 learned_addr learned my addr 172.21.15.125:0/2814898077 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e18e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 0x7f3e14159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e19800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 -- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 msgr2=0x7f3e140b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140b46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 -- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 msgr2=0x7f3e14159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 0x7f3e14159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e13e00640 1 -- 172.21.15.125:0/2814898077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0094d0 con 0x7f3e140ae750 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.778+0000 7f3e18e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 0x7f3e14159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e13e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc000c00 tx=0x7f3dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc033d60 con 0x7f3e140ae750 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc03d470 con 0x7f3e140ae750 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc03d820 con 0x7f3e140ae750 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e140c31b0 con 0x7f3e140ae750 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.779+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e141599b0 con 0x7f3e140ae750 2024-01-16T15:11:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.780+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e140c3970 con 0x7f3e140ae750 2024-01-16T15:11:13.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.781+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3dfc03a040 con 0x7f3e140ae750 2024-01-16T15:11:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.782+0000 7f3e10c00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 0x7f3dec06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.782+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3dfc0aa980 con 0x7f3e140ae750 2024-01-16T15:11:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.783+0000 7f3e18e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 0x7f3dec06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.783+0000 7f3e18e00640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 0x7f3dec06e650 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400b820 tx=0x7f3e04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.787+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3dfc076fc0 con 0x7f3e140ae750 2024-01-16T15:11:13.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.962+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3e140675d0 con 0x7f3e140ae750 2024-01-16T15:11:13.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.969+0000 7f3e10c00640 1 -- 172.21.15.125:0/2814898077 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3dfc0380c0 con 0x7f3e140ae750 2024-01-16T15:11:13.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 msgr2=0x7f3dec06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 0x7f3dec06e650 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400b820 tx=0x7f3e04008040 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 msgr2=0x7f3e140b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc000c00 tx=0x7f3dfc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 shutdown_connections 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e140cc1e0 0x7f3e140b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dec06c190 0x7f3dec06e650 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e140ae750 0x7f3e140b41a0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 --2- 172.21.15.125:0/2814898077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e14074520 0x7f3e14159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 >> 172.21.15.125:0/2814898077 conn(0x7f3e140b0d70 msgr2=0x7f3e1415d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 shutdown_connections 2024-01-16T15:11:13.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:13.972+0000 7f3e1b800640 1 -- 172.21.15.125:0/2814898077 wait complete. 2024-01-16T15:11:13.965 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:11:13.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:14.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:14.197 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:11:14.198 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:14.198 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:14.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:14.252 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:11:14.252 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:14.308 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:14.308 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:14.308 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:14.364 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:14.364 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:11:14.364 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:14.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:14.420 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:11:14.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.656+0000 7f3d34000640 1 Processor -- start 2024-01-16T15:11:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.657+0000 7f3d34000640 1 -- start start 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d34000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d2c143ae0 0x7f3d2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d34000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d34000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c0cfc30 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d34000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c0b1ee0 con 0x7f3d2c143ae0 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d34000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0b2780 con 0x7f3d2c15e3b0 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d31600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d2c143ae0 0x7f3d2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d31600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d2c143ae0 0x7f3d2c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45210/0 (socket says 172.21.15.125:45210) 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d31600640 1 -- 172.21.15.125:0/1765479774 learned_addr learned my addr 172.21.15.125:0/1765479774 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:14.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.658+0000 7f3d30c00640 1 --1- 172.21.15.125:0/1765479774 >> v1:172.21.15.125:6789/0 conn(0x7f3d2c0cf7d0 0x7f3d2c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42400/0 (socket says 172.21.15.125:42400) 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.659+0000 7f3d32000640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.659+0000 7f3d31600640 1 -- 172.21.15.125:0/1765479774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 msgr2=0x7f3d2c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.659+0000 7f3d31600640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.659+0000 7f3d31600640 1 -- 172.21.15.125:0/1765479774 >> v1:172.21.15.125:6789/0 conn(0x7f3d2c0cf7d0 legacy=0x7f3d2c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.659+0000 7f3d31600640 1 -- 172.21.15.125:0/1765479774 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c15e990 con 0x7f3d2c143ae0 2024-01-16T15:11:14.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.660+0000 7f3d31600640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c0cf290 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009870 tx=0x7f3d1c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fa252d455677f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:14.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.660+0000 7f3d2be00640 1 -- 172.21.15.125:0/1765479774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c030840 con 0x7f3d2c143ae0 2024-01-16T15:11:14.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.660+0000 7f3d2be00640 1 -- 172.21.15.125:0/1765479774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0309a0 con 0x7f3d2c143ae0 2024-01-16T15:11:14.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.660+0000 7f3d2be00640 1 -- 172.21.15.125:0/1765479774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c030d30 con 0x7f3d2c143ae0 2024-01-16T15:11:14.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 -- 172.21.15.125:0/1765479774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c143ae0 msgr2=0x7f3d2c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c0cf290 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009870 tx=0x7f3d1c02b980 comp rx=0 tx=0).stop 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 -- 172.21.15.125:0/1765479774 shutdown_connections 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 --2- 172.21.15.125:0/1765479774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c0cf290 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 -- 172.21.15.125:0/1765479774 >> 172.21.15.125:0/1765479774 conn(0x7f3d2c06a800 msgr2=0x7f3d2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 -- 172.21.15.125:0/1765479774 shutdown_connections 2024-01-16T15:11:14.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.661+0000 7f3d34000640 1 -- 172.21.15.125:0/1765479774 wait complete. 2024-01-16T15:11:14.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.663+0000 7f3d34000640 1 Processor -- start 2024-01-16T15:11:14.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.663+0000 7f3d34000640 1 -- start start 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c140280 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c159480 con 0x7f3d2c143ae0 2024-01-16T15:11:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d34000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c13f670 con 0x7f3d2c15e3b0 2024-01-16T15:11:14.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45216/0 (socket says 172.21.15.125:45216) 2024-01-16T15:11:14.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 -- 172.21.15.125:0/3378783109 learned_addr learned my addr 172.21.15.125:0/3378783109 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d32000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 -- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 msgr2=0x7f3d2c158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d30c00640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 -- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d2c143ae0 msgr2=0x7f3d2c1f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c1f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.664+0000 7f3d31600640 1 -- 172.21.15.125:0/3378783109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0094d0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d31600640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c000c00 tx=0x7f3d1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c039c90 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c039df0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c13f950 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c043640 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.665+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c13ff10 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.666+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c0675d0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.668+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3d1c0437a0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.669+0000 7f3d29600640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 0x7f3d0406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.669+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3d1c03e070 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.669+0000 7f3d30c00640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 0x7f3d0406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.670+0000 7f3d30c00640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 0x7f3d0406e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007fd0 tx=0x7f3d14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:14.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.674+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d1c078080 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.887+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d2c06b3f0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.888+0000 7f3d29600640 1 -- 172.21.15.125:0/3378783109 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f3d1c07b6a0 con 0x7f3d2c0cf7d0 2024-01-16T15:11:14.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:14.865 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/4149833712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":4149833712},{"type":"v1","addr":"172.21.15.137:6834","nonce":4149833712}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":6235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/2076499076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":2076499076},{"type":"v1","addr":"172.21.15.137:6835","nonce":2076499076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8172,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3233056488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3233056488},{"type":"v1","addr":"172.21.15.125:6835","nonce":3233056488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":8535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/608622904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":608622904},{"type":"v1","addr":"172.21.15.125:6837","nonce":608622904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[]} 2024-01-16T15:11:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 57 2024-01-16T15:11:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 msgr2=0x7f3d0406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 0x7f3d0406e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007fd0 tx=0x7f3d14009040 comp rx=0 tx=0).stop 2024-01-16T15:11:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 msgr2=0x7f3d2c13e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c000c00 tx=0x7f3d1c004730 comp rx=0 tx=0).stop 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 shutdown_connections 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d2c15e3b0 0x7f3d2c158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d2c143ae0 0x7f3d2c1f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d0406c160 0x7f3d0406e620 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 --2- 172.21.15.125:0/3378783109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d2c0cf7d0 0x7f3d2c13e6a0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 >> 172.21.15.125:0/3378783109 conn(0x7f3d2c06a800 msgr2=0x7f3d2c15ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 shutdown_connections 2024-01-16T15:11:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:14.891+0000 7f3d34000640 1 -- 172.21.15.125:0/3378783109 wait complete. 2024-01-16T15:11:14.886 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:11:14.886 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:11:14.887 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:11:14.887 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:14.887 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:11:14.887 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:11:14.887 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:11:14.887 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:14.887 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:11:14.887 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:11:14.888 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:11:14.888 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:14.888 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:11:14.888 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:11:14.888 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:11:14.888 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:14.911 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:11:14.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:11:14.914 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:11:14.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:11:14.924 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:11:14.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:11:14.926 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:11:14.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:11:14.975 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:11:14.975 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:14.977 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:11:14.977 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:14.987 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:11:14.987 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:14.987 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:11:14.988 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:14.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:11:15.130 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:11:15.131 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:11:15.138 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T15:11:15.139 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T15:11:15.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.192+0000 7fdef1200640 1 Processor -- start 2024-01-16T15:11:15.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.192+0000 7fdef1200640 1 -- start start 2024-01-16T15:11:15.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.193+0000 7fdef1200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdeec0cf9a0 0x7fdeec166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.193+0000 7fdef1200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec15c5f0 con 0x7fdeec13e360 2024-01-16T15:11:15.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.193+0000 7fdef1200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec06c470 con 0x7fdeec13df80 2024-01-16T15:11:15.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.193+0000 7fdef1200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec06c750 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.193+0000 7fdee9400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdeec13e360 0x7fdeec0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42420/0 (socket says 172.21.15.125:42420) 2024-01-16T15:11:15.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdee9400640 1 -- 172.21.15.125:0/3100006676 learned_addr learned my addr 172.21.15.125:0/3100006676 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:15.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdeea800640 1 --2- 172.21.15.125:0/3100006676 >> v2:172.21.15.137:3300/0 conn(0x7fdeec0cf9a0 0x7fdeec166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628388148 0 0) 0x7fdeec15c5f0 con 0x7fdeec13e360 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc003520 con 0x7fdeec13e360 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdeea800640 1 -- 172.21.15.125:0/3100006676 >> v1:172.21.15.137:6789/0 conn(0x7fdeec13df80 legacy=0x7fdeec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160686183 0 0) 0x7fdeec06c470 con 0x7fdeec13df80 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdeea800640 1 -- 172.21.15.125:0/3100006676 >> v1:172.21.15.125:6789/0 conn(0x7fdeec13e360 legacy=0x7fdeec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.194+0000 7fdeea800640 1 -- 172.21.15.125:0/3100006676 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec06ca30 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.195+0000 7fdeea800640 1 --2- 172.21.15.125:0/3100006676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec166030 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009d90 tx=0x7fdee402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6a639213b4b1eab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee402ccf0 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee402ce50 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdee8a00640 1 -- 172.21.15.125:0/3100006676 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4034660 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdef1200640 1 -- 172.21.15.125:0/3100006676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 msgr2=0x7fdeec166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdef1200640 1 --2- 172.21.15.125:0/3100006676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec166030 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009d90 tx=0x7fdee402bf40 comp rx=0 tx=0).stop 2024-01-16T15:11:15.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdef1200640 1 -- 172.21.15.125:0/3100006676 shutdown_connections 2024-01-16T15:11:15.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdef1200640 1 --2- 172.21.15.125:0/3100006676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec166030 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.196+0000 7fdef1200640 1 -- 172.21.15.125:0/3100006676 >> 172.21.15.125:0/3100006676 conn(0x7fdeec06a800 msgr2=0x7fdeec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:15.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.197+0000 7fdef1200640 1 -- 172.21.15.125:0/3100006676 shutdown_connections 2024-01-16T15:11:15.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.197+0000 7fdef1200640 1 -- 172.21.15.125:0/3100006676 wait complete. 2024-01-16T15:11:15.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.198+0000 7fdef1200640 1 Processor -- start 2024-01-16T15:11:15.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.198+0000 7fdef1200640 1 -- start start 2024-01-16T15:11:15.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdeec13df80 0x7fdeec14d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec14d850 con 0x7fdeec13e360 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15c850 con 0x7fdeec0cf9a0 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.199+0000 7fdef1200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15cb30 con 0x7fdeec13df80 2024-01-16T15:11:15.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45244/0 (socket says 172.21.15.125:45244) 2024-01-16T15:11:15.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 -- 172.21.15.125:0/1534427186 learned_addr learned my addr 172.21.15.125:0/1534427186 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:15.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdee9400640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdeec13df80 0x7fdeec14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdee9e00640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 -- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdeec13df80 msgr2=0x7fdeec14d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdeec13df80 0x7fdeec14d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 -- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 msgr2=0x7fdeec14f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec14f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdeea800640 1 -- 172.21.15.125:0/1534427186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4009a70 con 0x7fdeec13e360 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.200+0000 7fdee9e00640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec14f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdeea800640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fdee402dfd0 tx=0x7fdee4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee403d600 con 0x7fdeec13e360 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee403d760 con 0x7fdeec13e360 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec14fe80 con 0x7fdeec13e360 2024-01-16T15:11:15.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee403db10 con 0x7fdeec13e360 2024-01-16T15:11:15.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.201+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec150440 con 0x7fdeec13e360 2024-01-16T15:11:15.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.202+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec15ccc0 con 0x7fdeec13e360 2024-01-16T15:11:15.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.203+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fdee403b040 con 0x7fdeec13e360 2024-01-16T15:11:15.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.204+0000 7fdee2000640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 0x7fdec406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.204+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdee40aa7f0 con 0x7fdeec13e360 2024-01-16T15:11:15.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.204+0000 7fdee9e00640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 0x7fdec406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.205+0000 7fdee9e00640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 0x7fdec406e620 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdedc000f60 tx=0x7fdedc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.208+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdee4076e80 con 0x7fdeec13e360 2024-01-16T15:11:15.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:15.413+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdeec0675d0 con 0x7fdeec13e360 2024-01-16T15:11:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.568+0000 7fdee2000640 1 -- 172.21.15.125:0/1534427186 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdee4009450 con 0x7fdeec13e360 2024-01-16T15:11:16.545 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T15:11:16.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.571+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 msgr2=0x7fdec406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.571+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 0x7fdec406e620 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdedc000f60 tx=0x7fdedc009040 comp rx=0 tx=0).stop 2024-01-16T15:11:16.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.571+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 msgr2=0x7fdeec14ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.571+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fdee402dfd0 tx=0x7fdee4004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 shutdown_connections 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdec406c160 0x7fdec406e620 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeec13e360 0x7fdeec14ca90 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdeec13df80 0x7fdeec14d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 --2- 172.21.15.125:0/1534427186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdeec0cf9a0 0x7fdeec14f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 >> 172.21.15.125:0/1534427186 conn(0x7fdeec06a800 msgr2=0x7fdeec0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 shutdown_connections 2024-01-16T15:11:16.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.572+0000 7fdef1200640 1 -- 172.21.15.125:0/1534427186 wait complete. 2024-01-16T15:11:16.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:11:16.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.789+0000 7f316ea00640 1 Processor -- start 2024-01-16T15:11:16.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.789+0000 7f316ea00640 1 -- start start 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ea00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f316813df80 0x7f316815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316806c470 con 0x7f316815f9c0 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ea00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316806c750 con 0x7f316813df80 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316806ca30 con 0x7f316813e360 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f3167e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f316813df80 0x7f316815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ca00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f316815f9c0 0x7f31681408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39054/0 (socket says 172.21.15.125:39054) 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f316ca00640 1 -- 172.21.15.125:0/3530671493 learned_addr learned my addr 172.21.15.125:0/3530671493 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.791+0000 7f3167400640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:16.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.792+0000 7f3167e00640 1 -- 172.21.15.125:0/3530671493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 msgr2=0x7f31680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:16.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.792+0000 7f3167e00640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.792+0000 7f3167e00640 1 -- 172.21.15.125:0/3530671493 >> v1:172.21.15.125:6789/0 conn(0x7f316815f9c0 legacy=0x7f31681408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.792+0000 7f3167e00640 1 -- 172.21.15.125:0/3530671493 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316806cd10 con 0x7f316813df80 2024-01-16T15:11:16.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.792+0000 7f3166a00640 1 -- 172.21.15.125:0/3530671493 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657817634 0 0) 0x7f316806c470 con 0x7f316815f9c0 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.793+0000 7f3167e00640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f316815bee0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f31580097a0 tx=0x7f315802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6a92f53e028dc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.793+0000 7f3166a00640 1 -- 172.21.15.125:0/3530671493 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315802d040 con 0x7f316813df80 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.793+0000 7f3166a00640 1 -- 172.21.15.125:0/3530671493 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158031810 con 0x7f316813df80 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.793+0000 7f3166a00640 1 -- 172.21.15.125:0/3530671493 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158031ba0 con 0x7f316813df80 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 -- 172.21.15.125:0/3530671493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 msgr2=0x7f316815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f316815bee0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f31580097a0 tx=0x7f315802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 -- 172.21.15.125:0/3530671493 shutdown_connections 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 --2- 172.21.15.125:0/3530671493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f316815bee0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 -- 172.21.15.125:0/3530671493 >> 172.21.15.125:0/3530671493 conn(0x7f316806a800 msgr2=0x7f316806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:16.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 -- 172.21.15.125:0/3530671493 shutdown_connections 2024-01-16T15:11:16.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.794+0000 7f316ea00640 1 -- 172.21.15.125:0/3530671493 wait complete. 2024-01-16T15:11:16.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.795+0000 7f316ea00640 1 Processor -- start 2024-01-16T15:11:16.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.796+0000 7f316ea00640 1 -- start start 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31681536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316815f9c0 0x7f31680a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31681350d0 con 0x7f316813df80 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680a90b0 con 0x7f316815f9c0 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f316ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680adb60 con 0x7f316813e360 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49212/0 (socket says 172.21.15.125:49212) 2024-01-16T15:11:16.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 -- 172.21.15.125:0/2170381658 learned_addr learned my addr 172.21.15.125:0/2170381658 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:16.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 -- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 msgr2=0x7f31681536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31681536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 -- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316815f9c0 msgr2=0x7f31680a8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316815f9c0 0x7f31680a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.797+0000 7f3167e00640 1 -- 172.21.15.125:0/2170381658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009400 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f3167e00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f3158000c00 tx=0x7f31580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315803aa50 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315803abb0 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158004030 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31680ade40 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.798+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3168131460 con 0x7f316813df80 2024-01-16T15:11:16.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.799+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680ae360 con 0x7f316813df80 2024-01-16T15:11:16.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.800+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3158042070 con 0x7f316813df80 2024-01-16T15:11:16.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.801+0000 7f315fe00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 0x7f314006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:16.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.801+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f31580aa720 con 0x7f316813df80 2024-01-16T15:11:16.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.802+0000 7f3167400640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 0x7f314006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:16.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.802+0000 7f3167400640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 0x7f314006e620 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f31680cd110 tx=0x7f3150008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:16.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:16.806+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3158076e80 con 0x7f316813df80 2024-01-16T15:11:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.066+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f31680722f0 con 0x7f316813df80 2024-01-16T15:11:17.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.067+0000 7f315fe00640 1 -- 172.21.15.125:0/2170381658 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3158035090 con 0x7f316813df80 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.044 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.047 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.048 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:17.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.052 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:11:17.053 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T15:11:17.054 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:11:17.055 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T15:11:17.056 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 msgr2=0x7f314006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 0x7f314006e620 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f31680cd110 tx=0x7f3150008040 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 msgr2=0x7f31680ad580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f3158000c00 tx=0x7f31580046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 shutdown_connections 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316815f9c0 0x7f31680a8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316813e360 0x7f31681536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f314006c160 0x7f314006e620 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 --2- 172.21.15.125:0/2170381658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f316813df80 0x7f31680ad580 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 >> 172.21.15.125:0/2170381658 conn(0x7f316806a800 msgr2=0x7f31680b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 shutdown_connections 2024-01-16T15:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.070+0000 7f316ea00640 1 -- 172.21.15.125:0/2170381658 wait complete. 2024-01-16T15:11:17.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.293+0000 7f9577000640 1 Processor -- start 2024-01-16T15:11:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.293+0000 7f9577000640 1 -- start start 2024-01-16T15:11:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9577000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f9570166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9577000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957015c5f0 con 0x7f957013e360 2024-01-16T15:11:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9577000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957006c470 con 0x7f957013df80 2024-01-16T15:11:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9577000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957006c750 con 0x7f95700cf9a0 2024-01-16T15:11:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9567200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f957013e360 0x7f95700cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39060/0 (socket says 172.21.15.125:39060) 2024-01-16T15:11:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.295+0000 7f9567200640 1 -- 172.21.15.125:0/804039423 learned_addr learned my addr 172.21.15.125:0/804039423 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f9575000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f9570166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960855240 0 0) 0x7f957015c5f0 con 0x7f957013e360 2024-01-16T15:11:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9550003520 con 0x7f957013e360 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591507156 0 0) 0x7f9550003520 con 0x7f957013e360 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 msgr2=0x7f9570166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f9570166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 >> v1:172.21.15.137:6789/0 conn(0x7f957013df80 legacy=0x7f957015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957006ca30 con 0x7f957013e360 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f9558001670 con 0x7f957013e360 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 >> v1:172.21.15.125:6789/0 conn(0x7f957013e360 legacy=0x7f95700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95500039a0 0x7f9550003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f95500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f9550005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957006c750 con 0x7f9550004d00 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955000ba60 con 0x7f95500039a0 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.296+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955000bc20 con 0x7f95700cf9a0 2024-01-16T15:11:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f9550005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 -- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 msgr2=0x7f95500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f956fe00640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95500039a0 0x7f9550003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9567200640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f95500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95700cf9a0 0x7f95500047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 -- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95500039a0 msgr2=0x7f9550003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95500039a0 0x7f9550003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f9575000640 1 -- 172.21.15.125:0/804039423 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550003520 con 0x7f9550004d00 2024-01-16T15:11:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.297+0000 7f956fe00640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95500039a0 0x7f9550003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.298+0000 7f9575000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f9550005180 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f9570155d40 tx=0x7f95500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cfa62c5a123b32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.298+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956802cb90 con 0x7f9550004d00 2024-01-16T15:11:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.298+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956800f040 con 0x7f9550004d00 2024-01-16T15:11:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.298+0000 7f956f400640 1 -- 172.21.15.125:0/804039423 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568034440 con 0x7f9550004d00 2024-01-16T15:11:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 -- 172.21.15.125:0/804039423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 msgr2=0x7f9550005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 --2- 172.21.15.125:0/804039423 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f9550005180 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f9570155d40 tx=0x7f95500057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9575000640 1 -- 172.21.15.125:0/804039423 reap_dead start 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 -- 172.21.15.125:0/804039423 shutdown_connections 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 -- 172.21.15.125:0/804039423 >> 172.21.15.125:0/804039423 conn(0x7f957006a800 msgr2=0x7f957006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 -- 172.21.15.125:0/804039423 shutdown_connections 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.299+0000 7f9577000640 1 -- 172.21.15.125:0/804039423 wait complete. 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.301+0000 7f9577000640 1 Processor -- start 2024-01-16T15:11:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.301+0000 7f9577000640 1 -- start start 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.301+0000 7f9577000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.301+0000 7f9577000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9570159390 0x7f957013c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f9577000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9570150db0 0x7f9570151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f9577000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957015a4f0 con 0x7f9550004d00 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f9577000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9570151780 con 0x7f9570159390 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f9577000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9570159870 con 0x7f9570150db0 2024-01-16T15:11:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49228/0 (socket says 172.21.15.125:49228) 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 -- 172.21.15.125:0/3217678624 learned_addr learned my addr 172.21.15.125:0/3217678624 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 -- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9570150db0 msgr2=0x7f9570151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9570150db0 0x7f9570151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 -- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9570159390 msgr2=0x7f957013c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9570159390 0x7f957013c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 -- 172.21.15.125:0/3217678624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9570159b50 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.302+0000 7f956fe00640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f956000d7d0 tx=0x7f956000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.303+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560017070 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.303+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560005070 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.303+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9570159e30 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.303+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560005440 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.303+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95700b2bf0 con 0x7f9550004d00 2024-01-16T15:11:17.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.304+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95700675d0 con 0x7f9550004d00 2024-01-16T15:11:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.305+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f9560007030 con 0x7f9550004d00 2024-01-16T15:11:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.306+0000 7f956cc00640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 0x7f954806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.306+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f95600881f0 con 0x7f9550004d00 2024-01-16T15:11:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.306+0000 7f9567200640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 0x7f954806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.307+0000 7f9567200640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 0x7f954806e620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9558001eb0 tx=0x7f9558009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.311+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95600547d0 con 0x7f9550004d00 2024-01-16T15:11:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.503+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f957015b0f0 con 0x7f9550004d00 2024-01-16T15:11:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.505+0000 7f956cc00640 1 -- 172.21.15.125:0/3217678624 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+12906 (secure 0 0 0) 0x7f9560057df0 con 0x7f9550004d00 2024-01-16T15:11:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:17.482 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":96,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:16.565123+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 msgr2=0x7f954806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 0x7f954806e620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9558001eb0 tx=0x7f9558009040 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 msgr2=0x7f95700b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f956000d7d0 tx=0x7f956000dc90 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 shutdown_connections 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9570150db0 0x7f9570151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9570159390 0x7f957013c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f954806c160 0x7f954806e620 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 --2- 172.21.15.125:0/3217678624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9550004d00 0x7f95700b26b0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 >> 172.21.15.125:0/3217678624 conn(0x7f957006a800 msgr2=0x7f957015d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 shutdown_connections 2024-01-16T15:11:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.508+0000 7f9577000640 1 -- 172.21.15.125:0/3217678624 wait complete. 2024-01-16T15:11:17.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:17.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:17.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.727+0000 7efcd0800640 1 Processor -- start 2024-01-16T15:11:17.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.728+0000 7efcd0800640 1 -- start start 2024-01-16T15:11:17.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efcd0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efcc80d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efcd0800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80d1a30 con 0x7efcc8146730 2024-01-16T15:11:17.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efcd0800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80ce9e0 con 0x7efcc80cf100 2024-01-16T15:11:17.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efcd0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80cb6c0 con 0x7efcc80d1090 2024-01-16T15:11:17.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efccde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efcc8146730 0x7efcc80ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39068/0 (socket says 172.21.15.125:39068) 2024-01-16T15:11:17.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.729+0000 7efccde00640 1 -- 172.21.15.125:0/3898846910 learned_addr learned my addr 172.21.15.125:0/3898846910 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:17.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccd400640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efcc80d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557365507 0 0) 0x7efcc80d1a30 con 0x7efcc8146730 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8003520 con 0x7efcc8146730 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73463847 0 0) 0x7efca8003520 con 0x7efcc8146730 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 msgr2=0x7efcc80d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efcc80d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 >> v1:172.21.15.137:6789/0 conn(0x7efcc80cf100 legacy=0x7efcc80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8146b10 con 0x7efcc8146730 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7efcb8001670 con 0x7efcc8146730 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 >> v1:172.21.15.125:6789/0 conn(0x7efcc8146730 legacy=0x7efcc80ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efca80039a0 0x7efca8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efca8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca800ba30 con 0x7efca80039a0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca800bba0 con 0x7efca80047c0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.730+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca800bd60 con 0x7efcc80d1090 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efca80039a0 0x7efca8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccd400640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efcce800640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efca8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 -- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 msgr2=0x7efca8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efca8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 -- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efca80047c0 msgr2=0x7efca8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccde00640 1 -- 172.21.15.125:0/3898846910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8003520 con 0x7efca80039a0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efcce800640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcc80d1090 0x7efca8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.731+0000 7efccd400640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.732+0000 7efccde00640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efca80039a0 0x7efca8003dc0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7efcb8001eb0 tx=0x7efca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa5150ded7eff80 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.732+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8005ad0 con 0x7efca80039a0 2024-01-16T15:11:17.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8005c50 con 0x7efca80039a0 2024-01-16T15:11:17.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efccca00640 1 -- 172.21.15.125:0/3898846910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb802c7c0 con 0x7efca80039a0 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 -- 172.21.15.125:0/3898846910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efca80039a0 msgr2=0x7efca8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 --2- 172.21.15.125:0/3898846910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efca80039a0 0x7efca8003dc0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7efcb8001eb0 tx=0x7efca80057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcce800640 1 -- 172.21.15.125:0/3898846910 reap_dead start 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 -- 172.21.15.125:0/3898846910 shutdown_connections 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 -- 172.21.15.125:0/3898846910 >> 172.21.15.125:0/3898846910 conn(0x7efcc8066690 msgr2=0x7efcc806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 -- 172.21.15.125:0/3898846910 shutdown_connections 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.733+0000 7efcd0800640 1 -- 172.21.15.125:0/3898846910 wait complete. 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.735+0000 7efcd0800640 1 Processor -- start 2024-01-16T15:11:17.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.735+0000 7efcd0800640 1 -- start start 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efca80039a0 0x7efcc8144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcc80c9d00 0x7efcc80ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80ca6a0 con 0x7efcc80ada00 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc81454a0 con 0x7efcc80c9d00 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcd0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80ae770 con 0x7efca80039a0 2024-01-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcce800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcce800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49234/0 (socket says 172.21.15.125:49234) 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.736+0000 7efcce800640 1 -- 172.21.15.125:0/184743264 learned_addr learned my addr 172.21.15.125:0/184743264 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efccde00640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efca80039a0 0x7efcc8144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efccd400640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcc80c9d00 0x7efcc80ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 -- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efca80039a0 msgr2=0x7efcc8144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efca80039a0 0x7efcc8144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 -- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcc80c9d00 msgr2=0x7efcc80ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcc80c9d00 0x7efcc80ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 -- 172.21.15.125:0/184743264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc80aea50 con 0x7efcc80ada00 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcce800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7efcc806dfa0 tx=0x7efcc000d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc000b070 con 0x7efcc80ada00 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc0005310 con 0x7efcc80ada00 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc80aed30 con 0x7efcc80ada00 2024-01-16T15:11:17.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0007270 con 0x7efcc80ada00 2024-01-16T15:11:17.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.737+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc80af2f0 con 0x7efcc80ada00 2024-01-16T15:11:17.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.739+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc8074a50 con 0x7efcc80ada00 2024-01-16T15:11:17.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.740+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7efcc00073d0 con 0x7efcc80ada00 2024-01-16T15:11:17.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.740+0000 7efcc6000640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 0x7efca006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.740+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efcc0088300 con 0x7efcc80ada00 2024-01-16T15:11:17.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.741+0000 7efccde00640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 0x7efca006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.741+0000 7efccde00640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 0x7efca006e620 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.745+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcc00548e0 con 0x7efcc80ada00 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.919+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efcc81e27f0 con 0x7efcc80ada00 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.920+0000 7efcc6000640 1 -- 172.21.15.125:0/184743264 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcc81e27f0 con 0x7efcc80ada00 2024-01-16T15:11:17.897 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:11:17.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.922+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 msgr2=0x7efca006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.922+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 0x7efca006e620 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb8009030 comp rx=0 tx=0).stop 2024-01-16T15:11:17.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.922+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 msgr2=0x7efcc80ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.922+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7efcc806dfa0 tx=0x7efcc000d5e0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 shutdown_connections 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efca006c160 0x7efca006e620 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcc80ada00 0x7efcc80ade80 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcc80c9d00 0x7efcc80ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 --2- 172.21.15.125:0/184743264 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efca80039a0 0x7efcc8144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 >> 172.21.15.125:0/184743264 conn(0x7efcc8066690 msgr2=0x7efcc80b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 shutdown_connections 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:17.923+0000 7efcd0800640 1 -- 172.21.15.125:0/184743264 wait complete. 2024-01-16T15:11:17.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:11:18.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.144+0000 7fb028400640 1 Processor -- start 2024-01-16T15:11:18.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.145+0000 7fb028400640 1 -- start start 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb02015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb02013e360 0x7fb0200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 0x7fb0200cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02006c470 con 0x7fb02013e360 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02006c750 con 0x7fb02015f9c0 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb028400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02006ca30 con 0x7fb02013df80 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.146+0000 7fb025000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb02013e360 0x7fb0200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb02013e360 0x7fb0200cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49236/0 (socket says 172.21.15.125:49236) 2024-01-16T15:11:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 -- 172.21.15.125:0/2295490781 learned_addr learned my addr 172.21.15.125:0/2295490781 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025a00640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb02015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb026400640 1 --2- 172.21.15.125:0/2295490781 >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 0x7fb0200cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 -- 172.21.15.125:0/2295490781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 msgr2=0x7fb02015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb02015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 -- 172.21.15.125:0/2295490781 >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 msgr2=0x7fb0200cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 --2- 172.21.15.125:0/2295490781 >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 0x7fb0200cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb025000640 1 -- 172.21.15.125:0/2295490781 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02006cd10 con 0x7fb02013e360 2024-01-16T15:11:18.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.147+0000 7fb026400640 1 --2- 172.21.15.125:0/2295490781 >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 0x7fb0200cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:18.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.148+0000 7fb025000640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200cf290 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb02006e6d0 tx=0x7fb00802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5563965d24bc6c11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.148+0000 7fb01fe00640 1 -- 172.21.15.125:0/2295490781 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031860 con 0x7fb02013e360 2024-01-16T15:11:18.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.148+0000 7fb01fe00640 1 -- 172.21.15.125:0/2295490781 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080319c0 con 0x7fb02013e360 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.148+0000 7fb01fe00640 1 -- 172.21.15.125:0/2295490781 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031d50 con 0x7fb02013e360 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 -- 172.21.15.125:0/2295490781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 msgr2=0x7fb0200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200cf290 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb02006e6d0 tx=0x7fb00802b900 comp rx=0 tx=0).stop 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 -- 172.21.15.125:0/2295490781 shutdown_connections 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 --2- 172.21.15.125:0/2295490781 >> v2:172.21.15.137:3300/0 conn(0x7fb02015f9c0 0x7fb0200cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200cf290 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 --2- 172.21.15.125:0/2295490781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb02015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 -- 172.21.15.125:0/2295490781 >> 172.21.15.125:0/2295490781 conn(0x7fb02006a800 msgr2=0x7fb02006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 -- 172.21.15.125:0/2295490781 shutdown_connections 2024-01-16T15:11:18.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.149+0000 7fb028400640 1 -- 172.21.15.125:0/2295490781 wait complete. 2024-01-16T15:11:18.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.151+0000 7fb028400640 1 Processor -- start 2024-01-16T15:11:18.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.151+0000 7fb028400640 1 -- start start 2024-01-16T15:11:18.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb0201f4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 0x7fb02015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02013d320 con 0x7fb02013e360 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02015b0b0 con 0x7fb02015f9c0 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb028400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02015add0 con 0x7fb02013df80 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb025000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb025000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49252/0 (socket says 172.21.15.125:49252) 2024-01-16T15:11:18.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.152+0000 7fb025000640 1 -- 172.21.15.125:0/1507252885 learned_addr learned my addr 172.21.15.125:0/1507252885 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025a00640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb0201f4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb026400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 0x7fb02015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 -- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 msgr2=0x7fb0201f4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb0201f4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 -- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 msgr2=0x7fb02015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 0x7fb02015d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 -- 172.21.15.125:0/1507252885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0080094d0 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025a00640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb0201f4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb026400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 0x7fb02015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb025000640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fb00802b940 tx=0x7fb0080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.153+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00803ad30 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.154+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080443f0 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.154+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0080447a0 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.154+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0201f41e0 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.154+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0201f47a0 con 0x7fb02013e360 2024-01-16T15:11:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.155+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0201f4990 con 0x7fb02013e360 2024-01-16T15:11:18.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.156+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fb00804d450 con 0x7fb02013e360 2024-01-16T15:11:18.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.157+0000 7fb01d600640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 0x7faff806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.157+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0080aaf30 con 0x7fb02013e360 2024-01-16T15:11:18.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.157+0000 7fb025a00640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 0x7faff806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.158+0000 7fb025a00640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 0x7faff806e620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb010006fd0 tx=0x7fb010008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.161+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb008077510 con 0x7fb02013e360 2024-01-16T15:11:18.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.354+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb0200675d0 con 0x7fb02013e360 2024-01-16T15:11:18.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.355+0000 7fb01d600640 1 -- 172.21.15.125:0/1507252885 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb008035070 con 0x7fb02013e360 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:18.332 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 msgr2=0x7faff806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 0x7faff806e620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb010006fd0 tx=0x7fb010008040 comp rx=0 tx=0).stop 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 msgr2=0x7fb0200adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fb00802b940 tx=0x7fb0080046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 shutdown_connections 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb02015f9c0 0x7fb02015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7faff806c160 0x7faff806e620 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb02013e360 0x7fb0200adad0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 --2- 172.21.15.125:0/1507252885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb02013df80 0x7fb0201f4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.358+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 >> 172.21.15.125:0/1507252885 conn(0x7fb02006a800 msgr2=0x7fb0200cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.359+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 shutdown_connections 2024-01-16T15:11:18.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.359+0000 7fb028400640 1 -- 172.21.15.125:0/1507252885 wait complete. 2024-01-16T15:11:18.353 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:11:18.354 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:11:18.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.589+0000 7f8b03e00640 1 Processor -- start 2024-01-16T15:11:18.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.589+0000 7f8b03e00640 1 -- start start 2024-01-16T15:11:18.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8b04143c40 0x7f8b04071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b040ccc10 0x7f8b0406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b04070e50 con 0x7f8b040717d0 2024-01-16T15:11:18.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b040723c0 con 0x7f8b040ccc10 2024-01-16T15:11:18.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b04144020 con 0x7f8b04143c40 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.591+0000 7f8b03400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b040ccc10 0x7f8b0406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b040ccc10 0x7f8b0406d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49266/0 (socket says 172.21.15.125:49266) 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 -- 172.21.15.125:0/250199589 learned_addr learned my addr 172.21.15.125:0/250199589 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b02a00640 1 --2- 172.21.15.125:0/250199589 >> v2:172.21.15.137:3300/0 conn(0x7f8b04143c40 0x7f8b04071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 -- 172.21.15.125:0/250199589 >> v2:172.21.15.137:3300/0 conn(0x7f8b04143c40 msgr2=0x7f8b04071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 --2- 172.21.15.125:0/250199589 >> v2:172.21.15.137:3300/0 conn(0x7f8b04143c40 0x7f8b04071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 -- 172.21.15.125:0/250199589 >> v1:172.21.15.137:6789/0 conn(0x7f8b040717d0 legacy=0x7f8b040cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.592+0000 7f8b03400640 1 -- 172.21.15.125:0/250199589 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04071e00 con 0x7f8b040ccc10 2024-01-16T15:11:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.593+0000 7f8b03400640 1 --2- 172.21.15.125:0/250199589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b0406d160 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406e410 tx=0x7f8afc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03c66a6ba0097c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b01600640 1 -- 172.21.15.125:0/250199589 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc02c9f0 con 0x7f8b040ccc10 2024-01-16T15:11:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b01600640 1 -- 172.21.15.125:0/250199589 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc02cb50 con 0x7f8b040ccc10 2024-01-16T15:11:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b01600640 1 -- 172.21.15.125:0/250199589 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0343f0 con 0x7f8b040ccc10 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b03e00640 1 -- 172.21.15.125:0/250199589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 msgr2=0x7f8b0406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b03e00640 1 --2- 172.21.15.125:0/250199589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b0406d160 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406e410 tx=0x7f8afc02be50 comp rx=0 tx=0).stop 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.594+0000 7f8b03e00640 1 -- 172.21.15.125:0/250199589 shutdown_connections 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.595+0000 7f8b03e00640 1 --2- 172.21.15.125:0/250199589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b0406d160 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.595+0000 7f8b03e00640 1 --2- 172.21.15.125:0/250199589 >> v2:172.21.15.137:3300/0 conn(0x7f8b04143c40 0x7f8b04071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.595+0000 7f8b03e00640 1 -- 172.21.15.125:0/250199589 >> 172.21.15.125:0/250199589 conn(0x7f8b0406a800 msgr2=0x7f8b0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.595+0000 7f8b03e00640 1 -- 172.21.15.125:0/250199589 shutdown_connections 2024-01-16T15:11:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.595+0000 7f8b03e00640 1 -- 172.21.15.125:0/250199589 wait complete. 2024-01-16T15:11:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.596+0000 7f8b03e00640 1 Processor -- start 2024-01-16T15:11:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.596+0000 7f8b03e00640 1 -- start start 2024-01-16T15:11:18.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b040717d0 0x7f8b0413bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 0x7f8b040a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041f19c0 con 0x7f8b040ccc10 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041f0db0 con 0x7f8b040717d0 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.597+0000 7f8b03e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041f1090 con 0x7f8b04143c40 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49282/0 (socket says 172.21.15.125:49282) 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 -- 172.21.15.125:0/3276583228 learned_addr learned my addr 172.21.15.125:0/3276583228 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02a00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b040717d0 0x7f8b0413bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b03400640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 0x7f8b040a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 -- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 msgr2=0x7f8b040a8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 0x7f8b040a8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 -- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b040717d0 msgr2=0x7f8b0413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b040717d0 0x7f8b0413bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 -- 172.21.15.125:0/3276583228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009b30 con 0x7f8b040ccc10 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b03400640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 0x7f8b040a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.598+0000 7f8b02000640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00b3f0 tx=0x7f8aec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.599+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec011830 con 0x7f8b040ccc10 2024-01-16T15:11:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.599+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec011990 con 0x7f8b040ccc10 2024-01-16T15:11:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.599+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b041f1370 con 0x7f8b040ccc10 2024-01-16T15:11:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.599+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec011d40 con 0x7f8b040ccc10 2024-01-16T15:11:18.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.599+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0413c070 con 0x7f8b040ccc10 2024-01-16T15:11:18.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.600+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b041f1b70 con 0x7f8b040ccc10 2024-01-16T15:11:18.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.601+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f8aec01a400 con 0x7f8b040ccc10 2024-01-16T15:11:18.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.602+0000 7f8afaa00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 0x7f8adc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:18.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.602+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8aec0884b0 con 0x7f8b040ccc10 2024-01-16T15:11:18.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.602+0000 7f8b02a00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 0x7f8adc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:18.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.603+0000 7f8b02a00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 0x7f8adc06e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8af4006fd0 tx=0x7f8af4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.606+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b041f1b70 con 0x7f8b040ccc10 2024-01-16T15:11:18.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:18.801+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b04067450 con 0x7f8b040ccc10 2024-01-16T15:11:19.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.594+0000 7f8afaa00640 1 -- 172.21.15.125:0/3276583228 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v97) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8aec0150b0 con 0x7f8b040ccc10 2024-01-16T15:11:19.571 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 msgr2=0x7f8adc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 0x7f8adc06e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8af4006fd0 tx=0x7f8af4008040 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 msgr2=0x7f8b040a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00b3f0 tx=0x7f8aec00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 shutdown_connections 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b04143c40 0x7f8b040a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8adc06c160 0x7f8adc06e620 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b040ccc10 0x7f8b040a84c0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 --2- 172.21.15.125:0/3276583228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b040717d0 0x7f8b0413bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.598+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 >> 172.21.15.125:0/3276583228 conn(0x7f8b0406a800 msgr2=0x7f8b040afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.599+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 shutdown_connections 2024-01-16T15:11:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.599+0000 7f8b03e00640 1 -- 172.21.15.125:0/3276583228 wait complete. 2024-01-16T15:11:19.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.827+0000 7f4b78c00640 1 Processor -- start 2024-01-16T15:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.827+0000 7f4b78c00640 1 -- start start 2024-01-16T15:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b78c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b78c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b7415e450 0x7f4b7415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b78c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7415ba40 con 0x7f4b74071230 2024-01-16T15:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b78c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b741581a0 con 0x7f4b7415e450 2024-01-16T15:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b78c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7415eb10 con 0x7f4b74157dc0 2024-01-16T15:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b72800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b7415e450 0x7f4b7415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.829+0000 7f4b71e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b7415e450 0x7f4b7415bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:47816/0 (socket says 172.21.15.125:47816) 2024-01-16T15:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 -- 172.21.15.125:0/733309990 learned_addr learned my addr 172.21.15.125:0/733309990 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 -- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 msgr2=0x7f4b74070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 --2- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 -- 172.21.15.125:0/733309990 >> v1:172.21.15.137:6789/0 conn(0x7f4b74071230 legacy=0x7f4b7415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.830+0000 7f4b72800640 1 -- 172.21.15.125:0/733309990 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b74071860 con 0x7f4b7415e450 2024-01-16T15:11:19.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.832+0000 7f4b72800640 1 --2- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b7415e450 0x7f4b7415bed0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d150 tx=0x7f4b6c02e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdecbfc58c94913e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.833+0000 7f4b70a00640 1 -- 172.21.15.125:0/733309990 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c00b0d0 con 0x7f4b7415e450 2024-01-16T15:11:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.833+0000 7f4b70a00640 1 -- 172.21.15.125:0/733309990 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c00b230 con 0x7f4b7415e450 2024-01-16T15:11:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.833+0000 7f4b70a00640 1 -- 172.21.15.125:0/733309990 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c009600 con 0x7f4b7415e450 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.833+0000 7f4b78c00640 1 -- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b7415e450 msgr2=0x7f4b7415bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 --2- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b7415e450 0x7f4b7415bed0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d150 tx=0x7f4b6c02e160 comp rx=0 tx=0).stop 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 -- 172.21.15.125:0/733309990 shutdown_connections 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 --2- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b7415e450 0x7f4b7415bed0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 --2- 172.21.15.125:0/733309990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 -- 172.21.15.125:0/733309990 >> 172.21.15.125:0/733309990 conn(0x7f4b740b0be0 msgr2=0x7f4b740b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 -- 172.21.15.125:0/733309990 shutdown_connections 2024-01-16T15:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.834+0000 7f4b78c00640 1 -- 172.21.15.125:0/733309990 wait complete. 2024-01-16T15:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.836+0000 7f4b78c00640 1 Processor -- start 2024-01-16T15:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.836+0000 7f4b78c00640 1 -- start start 2024-01-16T15:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.837+0000 7f4b78c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74071230 0x7f4b740c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.837+0000 7f4b78c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.837+0000 7f4b78c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.837+0000 7f4b78c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741f3a80 con 0x7f4b7415e450 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.837+0000 7f4b78c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740c4510 con 0x7f4b74071230 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b78c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741f3cb0 con 0x7f4b74157dc0 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b72800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b72800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49298/0 (socket says 172.21.15.125:49298) 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b72800640 1 -- 172.21.15.125:0/172451784 learned_addr learned my addr 172.21.15.125:0/172451784 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b71e00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74071230 0x7f4b740c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.838+0000 7f4b71400640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 -- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 msgr2=0x7f4b74155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 -- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74071230 msgr2=0x7f4b740c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74071230 0x7f4b740c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 -- 172.21.15.125:0/172451784 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c00bd40 con 0x7f4b7415e450 2024-01-16T15:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b72800640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d150 tx=0x7f4b6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.839+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03f850 con 0x7f4b7415e450 2024-01-16T15:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.840+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c03f9b0 con 0x7f4b7415e450 2024-01-16T15:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.840+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03fd60 con 0x7f4b7415e450 2024-01-16T15:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.840+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b741f3f90 con 0x7f4b7415e450 2024-01-16T15:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.840+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b741f4550 con 0x7f4b7415e450 2024-01-16T15:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.841+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74067670 con 0x7f4b7415e450 2024-01-16T15:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.843+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4b6c052430 con 0x7f4b7415e450 2024-01-16T15:11:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.844+0000 7f4b6a000640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 0x7f4b4c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.844+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4b6c0abe20 con 0x7f4b7415e450 2024-01-16T15:11:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.844+0000 7f4b71e00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 0x7f4b4c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.845+0000 7f4b71e00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 0x7f4b4c06e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4b741f7100 tx=0x7f4b64009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:19.848+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b6c0787d0 con 0x7f4b7415e450 2024-01-16T15:11:20.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.041+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4b740b1760 con 0x7f4b7415e450 2024-01-16T15:11:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.611+0000 7f4b6a000640 1 -- 172.21.15.125:0/172451784 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v98) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4b6c0440d0 con 0x7f4b7415e450 2024-01-16T15:11:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.615+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 msgr2=0x7f4b4c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.615+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 0x7f4b4c06e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4b741f7100 tx=0x7f4b64009190 comp rx=0 tx=0).stop 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.615+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 msgr2=0x7f4b741f3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.615+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d150 tx=0x7f4b6c004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 shutdown_connections 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4c06c160 0x7f4b4c06e620 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7415e450 0x7f4b741f3540 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b74157dc0 0x7f4b74155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 --2- 172.21.15.125:0/172451784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74071230 0x7f4b740c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 >> 172.21.15.125:0/172451784 conn(0x7f4b740b0be0 msgr2=0x7f4b7406cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 shutdown_connections 2024-01-16T15:11:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.616+0000 7f4b78c00640 1 -- 172.21.15.125:0/172451784 wait complete. 2024-01-16T15:11:20.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:11:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.833+0000 7fe0e1800640 1 Processor -- start 2024-01-16T15:11:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.833+0000 7fe0e1800640 1 -- start start 2024-01-16T15:11:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc0cf7d0 0x7fe0dc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe0dc070ea0 0x7fe0dc0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc14b6a0 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc0cfbf0 con 0x7fe0dc070ea0 2024-01-16T15:11:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0e1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc071690 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0daa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0daa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49306/0 (socket says 172.21.15.125:49306) 2024-01-16T15:11:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.835+0000 7fe0daa00640 1 -- 172.21.15.125:0/3796035397 learned_addr learned my addr 172.21.15.125:0/3796035397 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0db400640 1 --2- 172.21.15.125:0/3796035397 >> v2:172.21.15.137:3300/0 conn(0x7fe0dc070ea0 0x7fe0dc0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0daa00640 1 -- 172.21.15.125:0/3796035397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc0cf7d0 msgr2=0x7fe0dc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0daa00640 1 --2- 172.21.15.125:0/3796035397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc0cf7d0 0x7fe0dc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0daa00640 1 -- 172.21.15.125:0/3796035397 >> v2:172.21.15.137:3300/0 conn(0x7fe0dc070ea0 msgr2=0x7fe0dc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0daa00640 1 --2- 172.21.15.125:0/3796035397 >> v2:172.21.15.137:3300/0 conn(0x7fe0dc070ea0 0x7fe0dc0b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.836+0000 7fe0daa00640 1 -- 172.21.15.125:0/3796035397 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc071970 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.837+0000 7fe0daa00640 1 --2- 172.21.15.125:0/3796035397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc06e2a0 tx=0x7fe0cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516f6a458d97a2b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.837+0000 7fe0da000640 1 -- 172.21.15.125:0/3796035397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc031870 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.837+0000 7fe0da000640 1 -- 172.21.15.125:0/3796035397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc0319d0 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.837+0000 7fe0da000640 1 -- 172.21.15.125:0/3796035397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc031d60 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 -- 172.21.15.125:0/3796035397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc14b2c0 msgr2=0x7fe0dc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 --2- 172.21.15.125:0/3796035397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc06e2a0 tx=0x7fe0cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 -- 172.21.15.125:0/3796035397 shutdown_connections 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 --2- 172.21.15.125:0/3796035397 >> v2:172.21.15.137:3300/0 conn(0x7fe0dc070ea0 0x7fe0dc0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 --2- 172.21.15.125:0/3796035397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc0cf7d0 0x7fe0dc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 --2- 172.21.15.125:0/3796035397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc14b2c0 0x7fe0dc0cf290 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 -- 172.21.15.125:0/3796035397 >> 172.21.15.125:0/3796035397 conn(0x7fe0dc06a800 msgr2=0x7fe0dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 -- 172.21.15.125:0/3796035397 shutdown_connections 2024-01-16T15:11:20.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.838+0000 7fe0e1800640 1 -- 172.21.15.125:0/3796035397 wait complete. 2024-01-16T15:11:20.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.840+0000 7fe0e1800640 1 Processor -- start 2024-01-16T15:11:20.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.840+0000 7fe0e1800640 1 -- start start 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 0x7fe0dc1646e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 0x7fe0dc1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15d730 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15cd90 con 0x7fe0dc070ea0 2024-01-16T15:11:20.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0e1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15d070 con 0x7fe0dc14b2c0 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0d1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0d1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49310/0 (socket says 172.21.15.125:49310) 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.841+0000 7fe0d1e00640 1 -- 172.21.15.125:0/2414310032 learned_addr learned my addr 172.21.15.125:0/2414310032 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0daa00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 0x7fe0dc1646e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0db400640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 0x7fe0dc1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 -- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 msgr2=0x7fe0dc1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 0x7fe0dc1419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 -- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 msgr2=0x7fe0dc1646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 0x7fe0dc1646e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 -- 172.21.15.125:0/2414310032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0094d0 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0daa00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 0x7fe0dc1646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0db400640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 0x7fe0dc1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:20.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d1e00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b4a0 tx=0x7fe0c400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4010610 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4010770 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4010b20 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.842+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc15d350 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.843+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc164e00 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.844+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc15d870 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.845+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe0c4010c80 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.846+0000 7fe0d3400640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 0x7fe0b406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.846+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe0c40883f0 con 0x7fe0dc0cf7d0 2024-01-16T15:11:20.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.846+0000 7fe0daa00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 0x7fe0b406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.847+0000 7fe0daa00640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 0x7fe0b406e620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc036000 tx=0x7fe0cc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:20.851+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0dc15d870 con 0x7fe0dc0cf7d0 2024-01-16T15:11:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.073+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe0dc15d4e0 con 0x7fe0dc0cf7d0 2024-01-16T15:11:21.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.614+0000 7fe0d3400640 1 -- 172.21.15.125:0/2414310032 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v59) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe0c40140c0 con 0x7fe0dc0cf7d0 2024-01-16T15:11:21.591 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 14 and data pool 15 2024-01-16T15:11:21.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 msgr2=0x7fe0b406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 0x7fe0b406e620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc036000 tx=0x7fe0cc03c040 comp rx=0 tx=0).stop 2024-01-16T15:11:21.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 msgr2=0x7fe0dc141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b4a0 tx=0x7fe0c400b960 comp rx=0 tx=0).stop 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 shutdown_connections 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0dc14b2c0 0x7fe0dc1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe0b406c160 0x7fe0b406e620 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0dc0cf7d0 0x7fe0dc141490 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 --2- 172.21.15.125:0/2414310032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0dc070ea0 0x7fe0dc1646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 >> 172.21.15.125:0/2414310032 conn(0x7fe0dc06a800 msgr2=0x7fe0dc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 shutdown_connections 2024-01-16T15:11:21.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.618+0000 7fe0e1800640 1 -- 172.21.15.125:0/2414310032 wait complete. 2024-01-16T15:11:21.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.821+0000 7fab6e800640 1 Processor -- start 2024-01-16T15:11:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.821+0000 7fab6e800640 1 -- start start 2024-01-16T15:11:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab6813df80 0x7fab6815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fab6815f9c0 0x7fab680cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6806c470 con 0x7fab6813df80 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6806c750 con 0x7fab6815f9c0 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab6e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6806ca30 con 0x7fab6813e360 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab67400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab6813df80 0x7fab6815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab67400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab6813df80 0x7fab6815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49314/0 (socket says 172.21.15.125:49314) 2024-01-16T15:11:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.823+0000 7fab67400640 1 -- 172.21.15.125:0/2046769857 learned_addr learned my addr 172.21.15.125:0/2046769857 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:21.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67400640 1 -- 172.21.15.125:0/2046769857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 msgr2=0x7fab680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:21.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab5e800640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67e00640 1 --2- 172.21.15.125:0/2046769857 >> v2:172.21.15.137:3300/0 conn(0x7fab6815f9c0 0x7fab680cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67400640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67400640 1 -- 172.21.15.125:0/2046769857 >> v2:172.21.15.137:3300/0 conn(0x7fab6815f9c0 msgr2=0x7fab680cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67400640 1 --2- 172.21.15.125:0/2046769857 >> v2:172.21.15.137:3300/0 conn(0x7fab6815f9c0 0x7fab680cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab67400640 1 -- 172.21.15.125:0/2046769857 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6806cd10 con 0x7fab6813df80 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.824+0000 7fab5e800640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.825+0000 7fab67400640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6815bee0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fab6806e6d0 tx=0x7fab5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dffba0b88225915 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.825+0000 7fab66a00640 1 -- 172.21.15.125:0/2046769857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58031870 con 0x7fab6813df80 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.825+0000 7fab66a00640 1 -- 172.21.15.125:0/2046769857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580319d0 con 0x7fab6813df80 2024-01-16T15:11:21.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.825+0000 7fab66a00640 1 -- 172.21.15.125:0/2046769857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58031d60 con 0x7fab6813df80 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 -- 172.21.15.125:0/2046769857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 msgr2=0x7fab6815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6815bee0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fab6806e6d0 tx=0x7fab5802b910 comp rx=0 tx=0).stop 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 -- 172.21.15.125:0/2046769857 shutdown_connections 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 --2- 172.21.15.125:0/2046769857 >> v2:172.21.15.137:3300/0 conn(0x7fab6815f9c0 0x7fab680cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 --2- 172.21.15.125:0/2046769857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6815bee0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 -- 172.21.15.125:0/2046769857 >> 172.21.15.125:0/2046769857 conn(0x7fab6806a800 msgr2=0x7fab6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 -- 172.21.15.125:0/2046769857 shutdown_connections 2024-01-16T15:11:21.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.826+0000 7fab6e800640 1 -- 172.21.15.125:0/2046769857 wait complete. 2024-01-16T15:11:21.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.828+0000 7fab6e800640 1 Processor -- start 2024-01-16T15:11:21.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.828+0000 7fab6e800640 1 -- start start 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab68146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 0x7fab6813c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68144770 con 0x7fab6813df80 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68147040 con 0x7fab6815f9c0 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab6e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6815d380 con 0x7fab6813e360 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab67400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab67400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49324/0 (socket says 172.21.15.125:49324) 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab67400640 1 -- 172.21.15.125:0/1602788265 learned_addr learned my addr 172.21.15.125:0/1602788265 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab67e00640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 0x7fab6813c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.829+0000 7fab5e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab68146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 -- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 msgr2=0x7fab68146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab68146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 -- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 msgr2=0x7fab6813c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 0x7fab6813c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 -- 172.21.15.125:0/1602788265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580094d0 con 0x7fab6813df80 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67e00640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 0x7fab6813c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab67400640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fab5802b950 tx=0x7fab580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58031860 con 0x7fab6813df80 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580443f0 con 0x7fab6813df80 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580447a0 con 0x7fab6813df80 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6815d660 con 0x7fab6813df80 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.830+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab68144390 con 0x7fab6813df80 2024-01-16T15:11:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.831+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab68144520 con 0x7fab6813df80 2024-01-16T15:11:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.832+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fab5804d450 con 0x7fab6813df80 2024-01-16T15:11:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.833+0000 7fab5fe00640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 0x7fab4006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.833+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab5802d040 con 0x7fab6813df80 2024-01-16T15:11:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.833+0000 7fab5e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 0x7fab4006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.834+0000 7fab5e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 0x7fab4006e620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fab6815f400 tx=0x7fab50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:21.837+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab580e0e90 con 0x7fab6813df80 2024-01-16T15:11:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.031+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab680675d0 con 0x7fab6813df80 2024-01-16T15:11:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.033+0000 7fab5fe00640 1 -- 172.21.15.125:0/1602788265 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fab58035070 con 0x7fab6813df80 2024-01-16T15:11:22.010 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:22.010 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:21.074762+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:18.855226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:20.091412+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 msgr2=0x7fab4006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 0x7fab4006e620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fab6815f400 tx=0x7fab50008040 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 msgr2=0x7fab6813b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fab5802b950 tx=0x7fab580046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 shutdown_connections 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab6815f9c0 0x7fab6813c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fab6813e360 0x7fab68146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fab4006c160 0x7fab4006e620 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 --2- 172.21.15.125:0/1602788265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab6813df80 0x7fab6813b9d0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 >> 172.21.15.125:0/1602788265 conn(0x7fab6806a800 msgr2=0x7fab6815c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 shutdown_connections 2024-01-16T15:11:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.036+0000 7fab6e800640 1 -- 172.21.15.125:0/1602788265 wait complete. 2024-01-16T15:11:22.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.260+0000 7fa2c3000640 1 Processor -- start 2024-01-16T15:11:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.260+0000 7fa2c3000640 1 -- start start 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c3000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa2bc13e360 0x7fa2bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c3000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc15c5f0 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c3000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc06c470 con 0x7fa2bc13df80 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c3000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc06c750 con 0x7fa2bc13e360 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c1000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa2bc15f9c0 0x7fa2bc0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39082/0 (socket says 172.21.15.125:39082) 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2c1000640 1 -- 172.21.15.125:0/2455204253 learned_addr learned my addr 172.21.15.125:0/2455204253 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2bb400640 1 --2- 172.21.15.125:0/2455204253 >> v2:172.21.15.137:3300/0 conn(0x7fa2bc13e360 0x7fa2bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706569139 0 0) 0x7fa2bc15c5f0 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.262+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c003520 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075319648 0 0) 0x7fa29c003520 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 >> v2:172.21.15.137:3300/0 conn(0x7fa2bc13e360 msgr2=0x7fa2bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 --2- 172.21.15.125:0/2455204253 >> v2:172.21.15.137:3300/0 conn(0x7fa2bc13e360 0x7fa2bc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 >> v1:172.21.15.137:6789/0 conn(0x7fa2bc13df80 legacy=0x7fa2bc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc06ca30 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa2b4001a10 con 0x7fa2bc15f9c0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 >> v1:172.21.15.125:6789/0 conn(0x7fa2bc15f9c0 legacy=0x7fa2bc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa29c003970 0x7fa29c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa29c0047c0 0x7fa29c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa29c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc06c750 con 0x7fa29c005180 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c00c020 con 0x7fa29c0047c0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c00c1e0 con 0x7fa29c003970 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.263+0000 7fa2c1000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa29c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2bbe00640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa29c003970 0x7fa29c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 -- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa29c003970 msgr2=0x7fa29c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa29c003970 0x7fa29c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 -- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa29c0047c0 msgr2=0x7fa29c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa29c0047c0 0x7fa29c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 -- 172.21.15.125:0/2455204253 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c003520 con 0x7fa29c005180 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2bbe00640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa29c003970 0x7fa29c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.264+0000 7fa2c1000640 1 -- 172.21.15.125:0/2455204253 reap_dead start 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2c1000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa29c00b980 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc15b6f0 tx=0x7fa29c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be09ec1002eddd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4034630 con 0x7fa29c005180 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b402f070 con 0x7fa29c005180 2024-01-16T15:11:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2baa00640 1 -- 172.21.15.125:0/2455204253 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4034990 con 0x7fa29c005180 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2c3000640 1 -- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 msgr2=0x7fa29c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.265+0000 7fa2c3000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa29c00b980 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc15b6f0 tx=0x7fa29c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.266+0000 7fa2c3000640 1 -- 172.21.15.125:0/2455204253 shutdown_connections 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.266+0000 7fa2c3000640 1 --2- 172.21.15.125:0/2455204253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa29c00b980 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.266+0000 7fa2c3000640 1 -- 172.21.15.125:0/2455204253 >> 172.21.15.125:0/2455204253 conn(0x7fa2bc06a800 msgr2=0x7fa2bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.266+0000 7fa2c3000640 1 -- 172.21.15.125:0/2455204253 shutdown_connections 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.266+0000 7fa2c3000640 1 -- 172.21.15.125:0/2455204253 wait complete. 2024-01-16T15:11:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.267+0000 7fa2c3000640 1 Processor -- start 2024-01-16T15:11:22.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.268+0000 7fa2c3000640 1 -- start start 2024-01-16T15:11:22.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.268+0000 7fa2c3000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2c3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2bc13db40 0x7fa2bc0cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2c3000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2bc0aa090 0x7fa2bc0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2c3000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc14b010 con 0x7fa29c005180 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2c3000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc14b220 con 0x7fa2bc0aa090 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2c3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc14b500 con 0x7fa2bc13db40 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49338/0 (socket says 172.21.15.125:49338) 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 -- 172.21.15.125:0/3920760142 learned_addr learned my addr 172.21.15.125:0/3920760142 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 -- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2bc13db40 msgr2=0x7fa2bc0cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2bc13db40 0x7fa2bc0cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 -- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2bc0aa090 msgr2=0x7fa2bc0cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2bc0aa090 0x7fa2bc0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 -- 172.21.15.125:0/3920760142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc14b7e0 con 0x7fa29c005180 2024-01-16T15:11:22.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.269+0000 7fa2bbe00640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000f60 tx=0x7fa2ac00d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.270+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac005480 con 0x7fa29c005180 2024-01-16T15:11:22.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.270+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac003610 con 0x7fa29c005180 2024-01-16T15:11:22.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.270+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc14bac0 con 0x7fa29c005180 2024-01-16T15:11:22.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.270+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac0039e0 con 0x7fa29c005180 2024-01-16T15:11:22.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.270+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc0cb840 con 0x7fa29c005180 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.271+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc0675d0 con 0x7fa29c005180 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.272+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa2ac003b40 con 0x7fa29c005180 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.273+0000 7fa2b3e00640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 0x7fa29406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.273+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2ac088860 con 0x7fa29c005180 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.273+0000 7fa2bb400640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 0x7fa29406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.273+0000 7fa2bb400640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 0x7fa29406e620 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4006a90 tx=0x7fa2a4006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.277+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2ac054ea0 con 0x7fa29c005180 2024-01-16T15:11:22.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.458+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2bc0a9e70 con 0x7fa29c005180 2024-01-16T15:11:22.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.460+0000 7fa2b3e00640 1 -- 172.21.15.125:0/3920760142 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16558 (secure 0 0 0) 0x7fa2ac001230 con 0x7fa29c005180 2024-01-16T15:11:22.436 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:22.437 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:22.355868+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:18.855226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:20.091412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:22.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.462+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 msgr2=0x7fa29406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.462+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 0x7fa29406e620 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4006a90 tx=0x7fa2a4006a20 comp rx=0 tx=0).stop 2024-01-16T15:11:22.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.462+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 msgr2=0x7fa2bc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.462+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000f60 tx=0x7fa2ac00d9f0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 shutdown_connections 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2bc0aa090 0x7fa2bc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2bc13db40 0x7fa2bc0cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa29406c160 0x7fa29406e620 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 --2- 172.21.15.125:0/3920760142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa29c005180 0x7fa2bc0cd840 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 >> 172.21.15.125:0/3920760142 conn(0x7fa2bc06a800 msgr2=0x7fa2bc154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 shutdown_connections 2024-01-16T15:11:22.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.463+0000 7fa2c3000640 1 -- 172.21.15.125:0/3920760142 wait complete. 2024-01-16T15:11:22.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:11:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.692+0000 7f4eac600640 1 Processor -- start 2024-01-16T15:11:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.692+0000 7f4eac600640 1 -- start start 2024-01-16T15:11:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4eac600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 0x7f4ea415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4eac600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ea40b3ae0 0x7f4ea406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4eac600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea4070c40 con 0x7f4ea406d120 2024-01-16T15:11:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4eac600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea4071450 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4eac600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406c470 con 0x7f4ea40ae5a0 2024-01-16T15:11:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.694+0000 7f4ea9200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ea40b3ae0 0x7f4ea406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 0x7f4ea415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4eaa600640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4ea406d120 0x7f4ea40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:58250/0 (socket says 172.21.15.125:58250) 2024-01-16T15:11:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4eaa600640 1 -- 172.21.15.125:0/1199466632 learned_addr learned my addr 172.21.15.125:0/1199466632 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9200640 1 -- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 msgr2=0x7f4ea415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9200640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 0x7f4ea415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9200640 1 -- 172.21.15.125:0/1199466632 >> v1:172.21.15.137:6789/0 conn(0x7f4ea406d120 legacy=0x7f4ea40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9200640 1 -- 172.21.15.125:0/1199466632 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea406c750 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.695+0000 7f4ea9c00640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 0x7f4ea415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:11:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.697+0000 7f4ea9200640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea406cbe0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009d90 tx=0x7f4e8c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b214989e0413d5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.697+0000 7f4ea3e00640 1 -- 172.21.15.125:0/1199466632 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c02cc50 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.697+0000 7f4ea3e00640 1 -- 172.21.15.125:0/1199466632 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c02cdb0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.697+0000 7f4ea3e00640 1 -- 172.21.15.125:0/1199466632 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c034660 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 -- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40b3ae0 msgr2=0x7f4ea406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea406cbe0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009d90 tx=0x7f4e8c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 -- 172.21.15.125:0/1199466632 shutdown_connections 2024-01-16T15:11:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea406cbe0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 --2- 172.21.15.125:0/1199466632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea40ae5a0 0x7f4ea415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 -- 172.21.15.125:0/1199466632 >> 172.21.15.125:0/1199466632 conn(0x7f4ea406a800 msgr2=0x7f4ea406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 -- 172.21.15.125:0/1199466632 shutdown_connections 2024-01-16T15:11:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.698+0000 7f4eac600640 1 -- 172.21.15.125:0/1199466632 wait complete. 2024-01-16T15:11:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.700+0000 7f4eac600640 1 Processor -- start 2024-01-16T15:11:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.700+0000 7f4eac600640 1 -- start start 2024-01-16T15:11:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea406d120 0x7f4ea414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 0x7f4ea40cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4158400 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4148570 con 0x7f4ea40ae5a0 2024-01-16T15:11:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eac600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4158120 con 0x7f4ea406d120 2024-01-16T15:11:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eaa600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eaa600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49346/0 (socket says 172.21.15.125:49346) 2024-01-16T15:11:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.701+0000 7f4eaa600640 1 -- 172.21.15.125:0/239835634 learned_addr learned my addr 172.21.15.125:0/239835634 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea9c00640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea406d120 0x7f4ea414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea9200640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 0x7f4ea40cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 -- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea406d120 msgr2=0x7f4ea414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea406d120 0x7f4ea414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 -- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 msgr2=0x7f4ea40cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 0x7f4ea40cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 -- 172.21.15.125:0/239835634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c009400 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea9200640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 0x7f4ea40cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eaa600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c420 tx=0x7f4e9400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94005ce0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94005e40 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e940072d0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.702+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c009a70 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.703+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea415b760 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.704+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea415b2c0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.705+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4e940074f0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.706+0000 7f4ea1600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 0x7f4e7c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.706+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e94087dc0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.706+0000 7f4ea9c00640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 0x7f4e7c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.707+0000 7f4ea9c00640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 0x7f4e7c06e620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009c60 tx=0x7f4e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.710+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e940544b0 con 0x7f4ea40b3ae0 2024-01-16T15:11:22.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:22.936+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4ea40675d0 con 0x7f4ea40b3ae0 2024-01-16T15:11:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.636+0000 7f4ea1600640 1 -- 172.21.15.125:0/239835634 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4e940170b0 con 0x7f4ea40b3ae0 2024-01-16T15:11:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 msgr2=0x7f4e7c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 0x7f4e7c06e620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009c60 tx=0x7f4e9c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 msgr2=0x7f4ea40cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c420 tx=0x7f4e9400c8e0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 shutdown_connections 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.639+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4e7c06c160 0x7f4e7c06e620 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ea40b3ae0 0x7f4ea40cbe00 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ea40ae5a0 0x7f4ea40cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 --2- 172.21.15.125:0/239835634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ea406d120 0x7f4ea414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 >> 172.21.15.125:0/239835634 conn(0x7f4ea406a800 msgr2=0x7f4ea406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 shutdown_connections 2024-01-16T15:11:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.640+0000 7f4eac600640 1 -- 172.21.15.125:0/239835634 wait complete. 2024-01-16T15:11:23.636 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:11:23.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:11:23.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.871+0000 7f19d1400640 1 Processor -- start 2024-01-16T15:11:23.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.872+0000 7f19d1400640 1 -- start start 2024-01-16T15:11:23.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.873+0000 7f19d1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 0x7f19cc06ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.873+0000 7f19d1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f19cc070490 0x7f19cc099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.873+0000 7f19d1400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0cfa70 con 0x7f19cc099fa0 2024-01-16T15:11:23.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.873+0000 7f19d1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc09a410 con 0x7f19cc070490 2024-01-16T15:11:23.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.873+0000 7f19d1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0cff00 con 0x7f19cc15e880 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19cb200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f19cc099fa0 0x7f19cc0cbb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39090/0 (socket says 172.21.15.125:39090) 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19cb200640 1 -- 172.21.15.125:0/357934064 learned_addr learned my addr 172.21.15.125:0/357934064 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9e00640 1 --2- 172.21.15.125:0/357934064 >> v2:172.21.15.137:3300/0 conn(0x7f19cc070490 0x7f19cc099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19ca800640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 0x7f19cc06ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039919213 0 0) 0x7f19cc0cfa70 con 0x7f19cc099fa0 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac003520 con 0x7f19cc099fa0 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387942419 0 0) 0x7f19ac003520 con 0x7f19cc099fa0 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 msgr2=0x7f19cc06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 0x7f19cc06ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 >> v2:172.21.15.137:3300/0 conn(0x7f19cc070490 msgr2=0x7f19cc099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.874+0000 7f19c9400640 1 --2- 172.21.15.125:0/357934064 >> v2:172.21.15.137:3300/0 conn(0x7f19cc070490 0x7f19cc099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc06fc70 con 0x7f19cc099fa0 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f19bc001670 con 0x7f19cc099fa0 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 >> v1:172.21.15.125:6789/0 conn(0x7f19cc099fa0 legacy=0x7f19cc0cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 0x7f19ac003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19ac003ef0 0x7f19ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac004d00 0x7f19ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc09a410 con 0x7f19ac003ef0 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0cff00 con 0x7f19ac004d00 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac00b9b0 con 0x7f19cc15e880 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19ac003ef0 0x7f19ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 -- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 msgr2=0x7f19ac003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc15e880 0x7f19ac003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 -- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac004d00 msgr2=0x7f19ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac004d00 0x7f19ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.875+0000 7f19c9e00640 1 -- 172.21.15.125:0/357934064 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac003520 con 0x7f19ac003ef0 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.876+0000 7f19c9e00640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19ac003ef0 0x7f19ac0047c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009ac0 tx=0x7f19ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b53f0523a4c58f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b402c710 con 0x7f19ac003ef0 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b402c870 con 0x7f19ac003ef0 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19c9400640 1 -- 172.21.15.125:0/357934064 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b402cc00 con 0x7f19ac003ef0 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 -- 172.21.15.125:0/357934064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19ac003ef0 msgr2=0x7f19ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 --2- 172.21.15.125:0/357934064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19ac003ef0 0x7f19ac0047c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009ac0 tx=0x7f19ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19cb200640 1 -- 172.21.15.125:0/357934064 reap_dead start 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 -- 172.21.15.125:0/357934064 shutdown_connections 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 -- 172.21.15.125:0/357934064 >> 172.21.15.125:0/357934064 conn(0x7f19cc0b2d00 msgr2=0x7f19cc0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 -- 172.21.15.125:0/357934064 shutdown_connections 2024-01-16T15:11:23.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.877+0000 7f19d1400640 1 -- 172.21.15.125:0/357934064 wait complete. 2024-01-16T15:11:23.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.879+0000 7f19d1400640 1 Processor -- start 2024-01-16T15:11:23.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.879+0000 7f19d1400640 1 -- start start 2024-01-16T15:11:23.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac003ef0 0x7f19cc0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc1554a0 0x7f19cc06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0b03f0 con 0x7f19cc0c86d0 2024-01-16T15:11:23.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0b06d0 con 0x7f19ac003ef0 2024-01-16T15:11:23.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.880+0000 7f19d1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0b09b0 con 0x7f19cc1554a0 2024-01-16T15:11:23.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49358/0 (socket says 172.21.15.125:49358) 2024-01-16T15:11:23.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 -- 172.21.15.125:0/996562102 learned_addr learned my addr 172.21.15.125:0/996562102 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:23.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19c9e00640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc1554a0 0x7f19cc06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19ca800640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac003ef0 0x7f19cc0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 -- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc1554a0 msgr2=0x7f19cc06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc1554a0 0x7f19cc06c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 -- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac003ef0 msgr2=0x7f19cc0afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac003ef0 0x7f19cc0afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 -- 172.21.15.125:0/996562102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0b0c90 con 0x7f19cc0c86d0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.881+0000 7f19cb200640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f19cc071b70 tx=0x7f19bc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.882+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0072d0 con 0x7f19cc0c86d0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.882+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc00ced0 con 0x7f19cc0c86d0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.882+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc007670 con 0x7f19cc0c86d0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.882+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0b0f70 con 0x7f19cc0c86d0 2024-01-16T15:11:23.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.882+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc155920 con 0x7f19cc0c86d0 2024-01-16T15:11:23.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.883+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc06cef0 con 0x7f19cc0c86d0 2024-01-16T15:11:23.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.884+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f19bc0047e0 con 0x7f19cc0c86d0 2024-01-16T15:11:23.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.885+0000 7f19c2a00640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 0x7f19a406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.885+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19bc087c40 con 0x7f19cc0c86d0 2024-01-16T15:11:23.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.885+0000 7f19ca800640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 0x7f19a406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.886+0000 7f19ca800640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 0x7f19a406e620 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004b30 tx=0x7f19c400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:23.890+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19cc06cef0 con 0x7f19cc0c86d0 2024-01-16T15:11:24.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.120+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f19cc06cef0 con 0x7f19cc0c86d0 2024-01-16T15:11:24.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.639+0000 7f19c2a00640 1 -- 172.21.15.125:0/996562102 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7f19cc06cef0 con 0x7f19cc0c86d0 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.643+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 msgr2=0x7f19a406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.643+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 0x7f19a406e620 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004b30 tx=0x7f19c400b040 comp rx=0 tx=0).stop 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.643+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 msgr2=0x7f19cc0b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.643+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f19cc071b70 tx=0x7f19bc00cb90 comp rx=0 tx=0).stop 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 shutdown_connections 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f19a406c160 0x7f19a406e620 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19cc0c86d0 0x7f19cc0b1490 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19cc1554a0 0x7f19cc06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 --2- 172.21.15.125:0/996562102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19ac003ef0 0x7f19cc0afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 >> 172.21.15.125:0/996562102 conn(0x7f19cc0b2d00 msgr2=0x7f19cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:24.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 shutdown_connections 2024-01-16T15:11:24.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.644+0000 7f19d1400640 1 -- 172.21.15.125:0/996562102 wait complete. 2024-01-16T15:11:24.642 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:11:24.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:24.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.869+0000 7fd89ec00640 1 Processor -- start 2024-01-16T15:11:24.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.869+0000 7fd89ec00640 1 -- start start 2024-01-16T15:11:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd898143ae0 0x7fd8980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8980cf7d0 0x7fd89815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd89815e1e0 0x7fd89815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8980cfc30 con 0x7fd89815e1e0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8980b1ee0 con 0x7fd898143ae0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980b2780 con 0x7fd8980cf7d0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd89815e1e0 0x7fd89815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd89815e1e0 0x7fd89815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49372/0 (socket says 172.21.15.125:49372) 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd89cc00640 1 -- 172.21.15.125:0/2056536751 learned_addr learned my addr 172.21.15.125:0/2056536751 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd897e00640 1 --2- 172.21.15.125:0/2056536751 >> v2:172.21.15.137:3300/0 conn(0x7fd898143ae0 0x7fd8980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.871+0000 7fd88f200640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8980cf7d0 0x7fd89815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.872+0000 7fd89cc00640 1 -- 172.21.15.125:0/2056536751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8980cf7d0 msgr2=0x7fd89815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.872+0000 7fd89cc00640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8980cf7d0 0x7fd89815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.872+0000 7fd89cc00640 1 -- 172.21.15.125:0/2056536751 >> v2:172.21.15.137:3300/0 conn(0x7fd898143ae0 msgr2=0x7fd8980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.872+0000 7fd89cc00640 1 --2- 172.21.15.125:0/2056536751 >> v2:172.21.15.137:3300/0 conn(0x7fd898143ae0 0x7fd8980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.872+0000 7fd89cc00640 1 -- 172.21.15.125:0/2056536751 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89815e990 con 0x7fd89815e1e0 2024-01-16T15:11:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.873+0000 7fd89cc00640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd89815e1e0 0x7fd89815a670 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd8980cfea0 tx=0x7fd89002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35b16cd1502d722 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.873+0000 7fd897400640 1 -- 172.21.15.125:0/2056536751 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89002ea80 con 0x7fd89815e1e0 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.873+0000 7fd897400640 1 -- 172.21.15.125:0/2056536751 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89002ebe0 con 0x7fd89815e1e0 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.873+0000 7fd897400640 1 -- 172.21.15.125:0/2056536751 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890009600 con 0x7fd89815e1e0 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 -- 172.21.15.125:0/2056536751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd89815e1e0 msgr2=0x7fd89815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd89815e1e0 0x7fd89815a670 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd8980cfea0 tx=0x7fd89002e0e0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 -- 172.21.15.125:0/2056536751 shutdown_connections 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd89815e1e0 0x7fd89815a670 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2056536751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8980cf7d0 0x7fd89815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2056536751 >> v2:172.21.15.137:3300/0 conn(0x7fd898143ae0 0x7fd8980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 -- 172.21.15.125:0/2056536751 >> 172.21.15.125:0/2056536751 conn(0x7fd89806a800 msgr2=0x7fd89806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 -- 172.21.15.125:0/2056536751 shutdown_connections 2024-01-16T15:11:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.874+0000 7fd89ec00640 1 -- 172.21.15.125:0/2056536751 wait complete. 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.875+0000 7fd89ec00640 1 Processor -- start 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.876+0000 7fd89ec00640 1 -- start start 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.876+0000 7fd89ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8980cf7d0 0x7fd898141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd89ec00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd89ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd89815e1e0 0x7fd8980cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd89ec00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd898142240 con 0x7fd898143ae0 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd89ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981f70a0 con 0x7fd8980cf7d0 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd89ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980c3f80 con 0x7fd89815e1e0 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49388/0 (socket says 172.21.15.125:49388) 2024-01-16T15:11:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 -- 172.21.15.125:0/2315987797 learned_addr learned my addr 172.21.15.125:0/2315987797 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 -- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd89815e1e0 msgr2=0x7fd8980cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd89815e1e0 0x7fd8980cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 -- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8980cf7d0 msgr2=0x7fd898141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8980cf7d0 0x7fd898141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.877+0000 7fd88f200640 1 -- 172.21.15.125:0/2315987797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89000bd40 con 0x7fd898143ae0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd88f200640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fd88000d830 tx=0x7fd88000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8800096a0 con 0x7fd898143ae0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880009800 con 0x7fd898143ae0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880009b90 con 0x7fd898143ae0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8980c4260 con 0x7fd898143ae0 2024-01-16T15:11:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.878+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8980c4820 con 0x7fd898143ae0 2024-01-16T15:11:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.879+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8980675d0 con 0x7fd898143ae0 2024-01-16T15:11:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.880+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd880014410 con 0x7fd898143ae0 2024-01-16T15:11:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.881+0000 7fd894c00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 0x7fd87006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.881+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd880088da0 con 0x7fd898143ae0 2024-01-16T15:11:24.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.881+0000 7fd897e00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 0x7fd87006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.882+0000 7fd897e00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 0x7fd87006e620 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd8980ce360 tx=0x7fd888008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:24.885+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd880055660 con 0x7fd898143ae0 2024-01-16T15:11:25.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.113+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd898070bf0 con 0x7fd898143ae0 2024-01-16T15:11:25.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.115+0000 7fd894c00640 1 -- 172.21.15.125:0/2315987797 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd88000b440 con 0x7fd898143ae0 2024-01-16T15:11:25.091 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:25.091 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6460,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2589810407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2589810407},{"type":"v1","addr":"172.21.15.137:6834","nonce":2589810407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6463,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1674315039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":1674315039},{"type":"v1","addr":"172.21.15.137:6835","nonce":1674315039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":8646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/995017134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":995017134},{"type":"v1","addr":"172.21.15.125:6836","nonce":995017134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:11:21.075580+0000","modified":"2024-01-16T15:11:24.121417+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/737412716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":737412716},{"type":"v1","addr":"172.21.15.125:6837","nonce":737412716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:11:25.092 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 63 2024-01-16T15:11:25.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 msgr2=0x7fd87006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 0x7fd87006e620 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd8980ce360 tx=0x7fd888008040 comp rx=0 tx=0).stop 2024-01-16T15:11:25.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 msgr2=0x7fd8981f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fd88000d830 tx=0x7fd88000dcf0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 shutdown_connections 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd89815e1e0 0x7fd8980cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd87006c160 0x7fd87006e620 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd898143ae0 0x7fd8981f6b60 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 --2- 172.21.15.125:0/2315987797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8980cf7d0 0x7fd898141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 >> 172.21.15.125:0/2315987797 conn(0x7fd89806a800 msgr2=0x7fd89815b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 shutdown_connections 2024-01-16T15:11:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.118+0000 7fd89ec00640 1 -- 172.21.15.125:0/2315987797 wait complete. 2024-01-16T15:11:25.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:25.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.338+0000 7ff8ed200640 1 Processor -- start 2024-01-16T15:11:25.314 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.338+0000 7ff8ed200640 1 -- start start 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8ed200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff8e813cfa0 0x7ff8e815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8ed200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 0x7ff8e813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8ed200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e80cdec0 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8ed200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80ce1a0 con 0x7ff8e813cfa0 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8ed200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80ce480 con 0x7ff8e80cd530 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8e7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff8e80cd9b0 0x7ff8e815d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39094/0 (socket says 172.21.15.125:39094) 2024-01-16T15:11:25.316 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.339+0000 7ff8e7e00640 1 -- 172.21.15.125:0/1140679911 learned_addr learned my addr 172.21.15.125:0/1140679911 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:25.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491629936 0 0) 0x7ff8e80cdec0 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8de800640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 0x7ff8e813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e7400640 1 --2- 172.21.15.125:0/1140679911 >> v2:172.21.15.137:3300/0 conn(0x7ff8e813cfa0 0x7ff8e815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c8003520 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 382576700 0 0) 0x7ff8c8003520 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 msgr2=0x7ff8e813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 0x7ff8e813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 >> v2:172.21.15.137:3300/0 conn(0x7ff8e813cfa0 msgr2=0x7ff8e815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 --2- 172.21.15.125:0/1140679911 >> v2:172.21.15.137:3300/0 conn(0x7ff8e813cfa0 0x7ff8e815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e80ce760 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff8e0001670 con 0x7ff8e80cd9b0 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 >> v1:172.21.15.125:6789/0 conn(0x7ff8e80cd9b0 legacy=0x7ff8e815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8c8003970 0x7ff8c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c80047c0 0x7ff8c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.340+0000 7ff8e6a00640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 0x7ff8c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80ce1a0 con 0x7ff8c8003970 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80ce480 con 0x7ff8c80047c0 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c800bb10 con 0x7ff8e80cd530 2024-01-16T15:11:25.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8c8003970 0x7ff8c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8de800640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c80047c0 0x7ff8c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 -- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 msgr2=0x7ff8c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80cd530 0x7ff8c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 -- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c80047c0 msgr2=0x7ff8c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c80047c0 0x7ff8c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8e7400640 1 -- 172.21.15.125:0/1140679911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c8003520 con 0x7ff8c8003970 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.341+0000 7ff8de800640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c80047c0 0x7ff8c8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.342+0000 7ff8e7400640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8c8003970 0x7ff8c8003d90 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800ba00 tx=0x7ff8c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e6dba34523f101 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.342+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8032630 con 0x7ff8c8003970 2024-01-16T15:11:25.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.342+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8032790 con 0x7ff8c8003970 2024-01-16T15:11:25.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.342+0000 7ff8e6a00640 1 -- 172.21.15.125:0/1140679911 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8032b20 con 0x7ff8c8003970 2024-01-16T15:11:25.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 -- 172.21.15.125:0/1140679911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8c8003970 msgr2=0x7ff8c8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 --2- 172.21.15.125:0/1140679911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8c8003970 0x7ff8c8003d90 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800ba00 tx=0x7ff8c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8e7e00640 1 -- 172.21.15.125:0/1140679911 reap_dead start 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 -- 172.21.15.125:0/1140679911 shutdown_connections 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 -- 172.21.15.125:0/1140679911 >> 172.21.15.125:0/1140679911 conn(0x7ff8e806a800 msgr2=0x7ff8e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 -- 172.21.15.125:0/1140679911 shutdown_connections 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.343+0000 7ff8ed200640 1 -- 172.21.15.125:0/1140679911 wait complete. 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.345+0000 7ff8ed200640 1 Processor -- start 2024-01-16T15:11:25.322 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.346+0000 7ff8ed200640 1 -- start start 2024-01-16T15:11:25.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c8003970 0x7ff8e81564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80c4010 0x7ff8e80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e81569e0 con 0x7ff8e8154f40 2024-01-16T15:11:25.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8155420 con 0x7ff8c8003970 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8ed200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8155700 con 0x7ff8e80c4010 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49396/0 (socket says 172.21.15.125:49396) 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 -- 172.21.15.125:0/3494721871 learned_addr learned my addr 172.21.15.125:0/3494721871 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8e7400640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c8003970 0x7ff8e81564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 -- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80c4010 msgr2=0x7ff8e80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80c4010 0x7ff8e80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 -- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c8003970 msgr2=0x7ff8e81564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c8003970 0x7ff8e81564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.347+0000 7ff8de800640 1 -- 172.21.15.125:0/3494721871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e81559e0 con 0x7ff8e8154f40 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8de800640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009f80 tx=0x7ff8d000f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d00145f0 con 0x7ff8e8154f40 2024-01-16T15:11:25.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d0014750 con 0x7ff8e8154f40 2024-01-16T15:11:25.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0014b00 con 0x7ff8e8154f40 2024-01-16T15:11:25.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e8155cc0 con 0x7ff8e8154f40 2024-01-16T15:11:25.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.348+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e8157320 con 0x7ff8e8154f40 2024-01-16T15:11:25.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.349+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e815b360 con 0x7ff8e8154f40 2024-01-16T15:11:25.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.350+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff8d0014c60 con 0x7ff8e8154f40 2024-01-16T15:11:25.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.351+0000 7ff8dfe00640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 0x7ff8c006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.351+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8d0089860 con 0x7ff8e8154f40 2024-01-16T15:11:25.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.351+0000 7ff8e7400640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 0x7ff8c006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.352+0000 7ff8e7400640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 0x7ff8c006e620 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80057a0 tx=0x7ff8d803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.356+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8d0055db0 con 0x7ff8e8154f40 2024-01-16T15:11:25.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.546+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e8067450 con 0x7ff8e8154f40 2024-01-16T15:11:25.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.549+0000 7ff8dfe00640 1 -- 172.21.15.125:0/3494721871 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16558 (secure 0 0 0) 0x7ff8d00593d0 con 0x7ff8e8154f40 2024-01-16T15:11:25.525 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:25.526 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:11:22.355868+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:18.855226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:20.091412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 msgr2=0x7ff8c006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 0x7ff8c006e620 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80057a0 tx=0x7ff8d803d040 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 msgr2=0x7ff8e8156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009f80 tx=0x7ff8d000f5e0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 shutdown_connections 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff8e80c4010 0x7ff8e80c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff8c006c160 0x7ff8c006e620 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e8154f40 0x7ff8e8156de0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 --2- 172.21.15.125:0/3494721871 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8c8003970 0x7ff8e81564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.551+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 >> 172.21.15.125:0/3494721871 conn(0x7ff8e806a800 msgr2=0x7ff8e814b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:25.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.552+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 shutdown_connections 2024-01-16T15:11:25.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.552+0000 7ff8ed200640 1 -- 172.21.15.125:0/3494721871 wait complete. 2024-01-16T15:11:25.547 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:11:25.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:11:25.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.780+0000 7efeb9000640 1 Processor -- start 2024-01-16T15:11:25.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.780+0000 7efeb9000640 1 -- start start 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb9000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efeb4099a60 0x7efeb4099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb9000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb40cfa70 con 0x7efeb415e880 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb409a3e0 con 0x7efeb406e7c0 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb9000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb40cff00 con 0x7efeb4099a60 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efeb415e880 0x7efeb406ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39106/0 (socket says 172.21.15.125:39106) 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.782+0000 7efeb1e00640 1 -- 172.21.15.125:0/912194621 learned_addr learned my addr 172.21.15.125:0/912194621 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efea9200640 1 --2- 172.21.15.125:0/912194621 >> v2:172.21.15.137:3300/0 conn(0x7efeb4099a60 0x7efeb4099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613169617 0 0) 0x7efeb40cfa70 con 0x7efeb415e880 2024-01-16T15:11:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe94003520 con 0x7efeb415e880 2024-01-16T15:11:25.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992590775 0 0) 0x7efe94003520 con 0x7efeb415e880 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 >> v2:172.21.15.137:3300/0 conn(0x7efeb4099a60 msgr2=0x7efeb4099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 --2- 172.21.15.125:0/912194621 >> v2:172.21.15.137:3300/0 conn(0x7efeb4099a60 0x7efeb4099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 >> v1:172.21.15.137:6789/0 conn(0x7efeb406e7c0 legacy=0x7efeb406ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb406fc70 con 0x7efeb415e880 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7efea4001670 con 0x7efeb415e880 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 >> v1:172.21.15.125:6789/0 conn(0x7efeb415e880 legacy=0x7efeb406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe940039c0 0x7efe94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe940047c0 0x7efe94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efe9400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40cff00 con 0x7efe94005180 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9400c060 con 0x7efe940047c0 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.783+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9400c220 con 0x7efe940039c0 2024-01-16T15:11:25.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efe9400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efea9200640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe940047c0 0x7efe94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb1e00640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe940039c0 0x7efe94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 -- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe940039c0 msgr2=0x7efe94003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe940039c0 0x7efe94003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 -- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe940047c0 msgr2=0x7efe94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe940047c0 0x7efe94004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.784+0000 7efeb2800640 1 -- 172.21.15.125:0/912194621 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94003520 con 0x7efe94005180 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.785+0000 7efeb2800640 1 -- 172.21.15.125:0/912194621 reap_dead start 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.785+0000 7efeb2800640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efe9400b980 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7efeac001e50 tx=0x7efe940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7510052f094a4ce8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.785+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac003b70 con 0x7efe94005180 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.785+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac003d10 con 0x7efe94005180 2024-01-16T15:11:25.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.785+0000 7efeb1400640 1 -- 172.21.15.125:0/912194621 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac005310 con 0x7efe94005180 2024-01-16T15:11:25.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 -- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 msgr2=0x7efe9400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.764 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efe9400b980 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7efeac001e50 tx=0x7efe940057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 -- 172.21.15.125:0/912194621 shutdown_connections 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 --2- 172.21.15.125:0/912194621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efe9400b980 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 -- 172.21.15.125:0/912194621 >> 172.21.15.125:0/912194621 conn(0x7efeb40b2d00 msgr2=0x7efeb40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 -- 172.21.15.125:0/912194621 shutdown_connections 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.786+0000 7efeb9000640 1 -- 172.21.15.125:0/912194621 wait complete. 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.788+0000 7efeb9000640 1 Processor -- start 2024-01-16T15:11:25.765 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.788+0000 7efeb9000640 1 -- start start 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efeb406b4f0 0x7efeb40c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efeb406b970 0x7efeb40c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb406cb10 con 0x7efe94005180 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb406bf00 con 0x7efeb406b970 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb406c1e0 con 0x7efeb406b4f0 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49410/0 (socket says 172.21.15.125:49410) 2024-01-16T15:11:25.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 -- 172.21.15.125:0/933000215 learned_addr learned my addr 172.21.15.125:0/933000215 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 -- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efeb406b4f0 msgr2=0x7efeb40c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efeb406b4f0 0x7efeb40c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 -- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efeb406b970 msgr2=0x7efeb40c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efeb406b970 0x7efeb40c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.789+0000 7efeb1e00640 1 -- 172.21.15.125:0/933000215 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb406c4c0 con 0x7efe94005180 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeb1e00640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7efea400d7a0 tx=0x7efea400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea40074e0 con 0x7efe94005180 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea4007640 con 0x7efe94005180 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea40049e0 con 0x7efe94005180 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb406c7a0 con 0x7efe94005180 2024-01-16T15:11:25.768 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.790+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb41f63c0 con 0x7efe94005180 2024-01-16T15:11:25.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.791+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb406c320 con 0x7efe94005180 2024-01-16T15:11:25.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.792+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7efea4004b40 con 0x7efe94005180 2024-01-16T15:11:25.769 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.793+0000 7efeaaa00640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 0x7efe8c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:25.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.793+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efea4088260 con 0x7efe94005180 2024-01-16T15:11:25.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.794+0000 7efea9200640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 0x7efe8c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:25.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.794+0000 7efea9200640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 0x7efe8c06e620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7efeb40cb110 tx=0x7efe9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:25.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.798+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efeb406c320 con 0x7efe94005180 2024-01-16T15:11:25.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.986+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efeb40cf700 con 0x7efe8c06c160 2024-01-16T15:11:25.963 INFO:tasks.ceph.mds.a.smithi125.stderr:2024-01-16T15:11:25.987+0000 7f72ce400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:11:25.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:25.995+0000 7efeaaa00640 1 -- 172.21.15.125:0/933000215 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+275394 (secure 0 0 0) 0x7efe9c009e40 con 0x7efe8c06c160 2024-01-16T15:11:25.973 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:25.977 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 msgr2=0x7efe8c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 0x7efe8c06e620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7efeb40cb110 tx=0x7efe9c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 msgr2=0x7efeb41579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7efea400d7a0 tx=0x7efea400dc60 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 shutdown_connections 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efeb406b970 0x7efeb40c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efeb406b4f0 0x7efeb40c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe8c06c160 0x7efe8c06e620 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 --2- 172.21.15.125:0/933000215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94005180 0x7efeb41579c0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 >> 172.21.15.125:0/933000215 conn(0x7efeb40b2d00 msgr2=0x7efeb40cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.001+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 shutdown_connections 2024-01-16T15:11:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.002+0000 7efeb9000640 1 -- 172.21.15.125:0/933000215 wait complete. 2024-01-16T15:11:25.997 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-01-16T15:11:24.261423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590480,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":164,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":60,"acting":60,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":542728,"kb_used_data":10492,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754431992,"statfs":{"total":773094113280,"available":772538359808,"internally_reserved":0,"allocated":10743808,"data_stored":5267492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":544853552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.664918"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365260+0000","last_change":"2024-01-16T15:11:21.386770+0000","last_active":"2024-01-16T15:11:22.365260+0000","last_peered":"2024-01-16T15:11:22.365260+0000","last_clean":"2024-01-16T15:11:22.365260+0000","last_became_active":"2024-01-16T15:11:21.385942+0000","last_became_peered":"2024-01-16T15:11:21.385942+0000","last_unstale":"2024-01-16T15:11:22.365260+0000","last_undegraded":"2024-01-16T15:11:22.365260+0000","last_fullsized":"2024-01-16T15:11:22.365260+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:53.149749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365803+0000","last_change":"2024-01-16T15:11:21.386478+0000","last_active":"2024-01-16T15:11:22.365803+0000","last_peered":"2024-01-16T15:11:22.365803+0000","last_clean":"2024-01-16T15:11:22.365803+0000","last_became_active":"2024-01-16T15:11:21.386037+0000","last_became_peered":"2024-01-16T15:11:21.386037+0000","last_unstale":"2024-01-16T15:11:22.365803+0000","last_undegraded":"2024-01-16T15:11:22.365803+0000","last_fullsized":"2024-01-16T15:11:22.365803+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:07.539617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365650+0000","last_change":"2024-01-16T15:11:21.384137+0000","last_active":"2024-01-16T15:11:22.365650+0000","last_peered":"2024-01-16T15:11:22.365650+0000","last_clean":"2024-01-16T15:11:22.365650+0000","last_became_active":"2024-01-16T15:11:21.383816+0000","last_became_peered":"2024-01-16T15:11:21.383816+0000","last_unstale":"2024-01-16T15:11:22.365650+0000","last_undegraded":"2024-01-16T15:11:22.365650+0000","last_fullsized":"2024-01-16T15:11:22.365650+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:11:33.092792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365080+0000","last_change":"2024-01-16T15:11:21.395324+0000","last_active":"2024-01-16T15:11:22.365080+0000","last_peered":"2024-01-16T15:11:22.365080+0000","last_clean":"2024-01-16T15:11:22.365080+0000","last_became_active":"2024-01-16T15:11:21.394570+0000","last_became_peered":"2024-01-16T15:11:21.394570+0000","last_unstale":"2024-01-16T15:11:22.365080+0000","last_undegraded":"2024-01-16T15:11:22.365080+0000","last_fullsized":"2024-01-16T15:11:22.365080+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:44.386586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366178+0000","last_change":"2024-01-16T15:11:21.391683+0000","last_active":"2024-01-16T15:11:22.366178+0000","last_peered":"2024-01-16T15:11:22.366178+0000","last_clean":"2024-01-16T15:11:22.366178+0000","last_became_active":"2024-01-16T15:11:21.391197+0000","last_became_peered":"2024-01-16T15:11:21.391197+0000","last_unstale":"2024-01-16T15:11:22.366178+0000","last_undegraded":"2024-01-16T15:11:22.366178+0000","last_fullsized":"2024-01-16T15:11:22.366178+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:11.683031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366082+0000","last_change":"2024-01-16T15:11:21.390053+0000","last_active":"2024-01-16T15:11:22.366082+0000","last_peered":"2024-01-16T15:11:22.366082+0000","last_clean":"2024-01-16T15:11:22.366082+0000","last_became_active":"2024-01-16T15:11:21.389763+0000","last_became_peered":"2024-01-16T15:11:21.389763+0000","last_unstale":"2024-01-16T15:11:22.366082+0000","last_undegraded":"2024-01-16T15:11:22.366082+0000","last_fullsized":"2024-01-16T15:11:22.366082+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:21.158872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.603594+0000","last_change":"2024-01-16T15:11:19.603594+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.603594+0000","last_undegraded":"2024-01-16T15:11:19.603594+0000","last_fullsized":"2024-01-16T15:11:19.603594+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366175+0000","last_change":"2024-01-16T15:11:20.644663+0000","last_active":"2024-01-16T15:11:22.366175+0000","last_peered":"2024-01-16T15:11:22.366175+0000","last_clean":"2024-01-16T15:11:22.366175+0000","last_became_active":"2024-01-16T15:11:20.644392+0000","last_became_peered":"2024-01-16T15:11:20.644392+0000","last_unstale":"2024-01-16T15:11:22.366175+0000","last_undegraded":"2024-01-16T15:11:22.366175+0000","last_fullsized":"2024-01-16T15:11:22.366175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:48.207643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.602913+0000","last_change":"2024-01-16T15:11:19.602913+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.602913+0000","last_undegraded":"2024-01-16T15:11:19.602913+0000","last_fullsized":"2024-01-16T15:11:19.602913+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.602686+0000","last_change":"2024-01-16T15:11:19.602686+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.602686+0000","last_undegraded":"2024-01-16T15:11:19.602686+0000","last_fullsized":"2024-01-16T15:11:19.602686+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.606166+0000","last_change":"2024-01-16T15:11:19.606166+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.606166+0000","last_undegraded":"2024-01-16T15:11:19.606166+0000","last_fullsized":"2024-01-16T15:11:19.606166+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365102+0000","last_change":"2024-01-16T15:11:21.392486+0000","last_active":"2024-01-16T15:11:22.365102+0000","last_peered":"2024-01-16T15:11:22.365102+0000","last_clean":"2024-01-16T15:11:22.365102+0000","last_became_active":"2024-01-16T15:11:21.392210+0000","last_became_peered":"2024-01-16T15:11:21.392210+0000","last_unstale":"2024-01-16T15:11:22.365102+0000","last_undegraded":"2024-01-16T15:11:22.365102+0000","last_fullsized":"2024-01-16T15:11:22.365102+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:13.164950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.602815+0000","last_change":"2024-01-16T15:11:19.602815+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.602815+0000","last_undegraded":"2024-01-16T15:11:19.602815+0000","last_fullsized":"2024-01-16T15:11:19.602815+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.607399+0000","last_change":"2024-01-16T15:11:19.607399+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.607399+0000","last_undegraded":"2024-01-16T15:11:19.607399+0000","last_fullsized":"2024-01-16T15:11:19.607399+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.606508+0000","last_change":"2024-01-16T15:11:19.606508+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.606508+0000","last_undegraded":"2024-01-16T15:11:19.606508+0000","last_fullsized":"2024-01-16T15:11:19.606508+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"99'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366863+0000","last_change":"2024-01-16T15:11:20.643870+0000","last_active":"2024-01-16T15:11:22.366863+0000","last_peered":"2024-01-16T15:11:22.366863+0000","last_clean":"2024-01-16T15:11:22.366863+0000","last_became_active":"2024-01-16T15:11:20.643537+0000","last_became_peered":"2024-01-16T15:11:20.643537+0000","last_unstale":"2024-01-16T15:11:22.366863+0000","last_undegraded":"2024-01-16T15:11:22.366863+0000","last_fullsized":"2024-01-16T15:11:22.366863+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:17.579148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367693+0000","last_change":"2024-01-16T15:11:21.390837+0000","last_active":"2024-01-16T15:11:22.367693+0000","last_peered":"2024-01-16T15:11:22.367693+0000","last_clean":"2024-01-16T15:11:22.367693+0000","last_became_active":"2024-01-16T15:11:21.390506+0000","last_became_peered":"2024-01-16T15:11:21.390506+0000","last_unstale":"2024-01-16T15:11:22.367693+0000","last_undegraded":"2024-01-16T15:11:22.367693+0000","last_fullsized":"2024-01-16T15:11:22.367693+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:05.575127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.606039+0000","last_change":"2024-01-16T15:11:19.606039+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.606039+0000","last_undegraded":"2024-01-16T15:11:19.606039+0000","last_fullsized":"2024-01-16T15:11:19.606039+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.605610+0000","last_change":"2024-01-16T15:11:19.605610+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.605610+0000","last_undegraded":"2024-01-16T15:11:19.605610+0000","last_fullsized":"2024-01-16T15:11:19.605610+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.606870+0000","last_change":"2024-01-16T15:11:19.606870+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.606870+0000","last_undegraded":"2024-01-16T15:11:19.606870+0000","last_fullsized":"2024-01-16T15:11:19.606870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366633+0000","last_change":"2024-01-16T15:11:21.385008+0000","last_active":"2024-01-16T15:11:22.366633+0000","last_peered":"2024-01-16T15:11:22.366633+0000","last_clean":"2024-01-16T15:11:22.366633+0000","last_became_active":"2024-01-16T15:11:21.383441+0000","last_became_peered":"2024-01-16T15:11:21.383441+0000","last_unstale":"2024-01-16T15:11:22.366633+0000","last_undegraded":"2024-01-16T15:11:22.366633+0000","last_fullsized":"2024-01-16T15:11:22.366633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:08.286858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.601994+0000","last_change":"2024-01-16T15:11:19.601994+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.601994+0000","last_undegraded":"2024-01-16T15:11:19.601994+0000","last_fullsized":"2024-01-16T15:11:19.601994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.602195+0000","last_change":"2024-01-16T15:11:19.602195+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.602195+0000","last_undegraded":"2024-01-16T15:11:19.602195+0000","last_fullsized":"2024-01-16T15:11:19.602195+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.601173+0000","last_change":"2024-01-16T15:11:19.601173+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.601173+0000","last_undegraded":"2024-01-16T15:11:19.601173+0000","last_fullsized":"2024-01-16T15:11:19.601173+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.601255+0000","last_change":"2024-01-16T15:11:19.601255+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.601255+0000","last_undegraded":"2024-01-16T15:11:19.601255+0000","last_fullsized":"2024-01-16T15:11:19.601255+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.601375+0000","last_change":"2024-01-16T15:11:19.601375+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.601375+0000","last_undegraded":"2024-01-16T15:11:19.601375+0000","last_fullsized":"2024-01-16T15:11:19.601375+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.601216+0000","last_change":"2024-01-16T15:11:19.601216+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.601216+0000","last_undegraded":"2024-01-16T15:11:19.601216+0000","last_fullsized":"2024-01-16T15:11:19.601216+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:19.580556+0000","last_change":"2024-01-16T15:11:19.580556+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.580556+0000","last_undegraded":"2024-01-16T15:11:19.580556+0000","last_fullsized":"2024-01-16T15:11:19.580556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:11:19.605440+0000","last_change":"2024-01-16T15:11:19.605440+0000","last_active":"2024-01-16T15:11:19.580556+0000","last_peered":"2024-01-16T15:11:19.580556+0000","last_clean":"2024-01-16T15:11:19.580556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:19.605440+0000","last_undegraded":"2024-01-16T15:11:19.605440+0000","last_fullsized":"2024-01-16T15:11:19.605440+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:11:20.595916+0000","last_change":"2024-01-16T15:11:20.595916+0000","last_active":"2024-01-16T15:11:20.595916+0000","last_peered":"2024-01-16T15:11:20.595916+0000","last_clean":"2024-01-16T15:11:20.595916+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:11:20.595916+0000","last_undegraded":"2024-01-16T15:11:20.595916+0000","last_fullsized":"2024-01-16T15:11:20.595916+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365158+0000","last_change":"2024-01-16T15:11:20.644061+0000","last_active":"2024-01-16T15:11:22.365158+0000","last_peered":"2024-01-16T15:11:22.365158+0000","last_clean":"2024-01-16T15:11:22.365158+0000","last_became_active":"2024-01-16T15:11:20.643678+0000","last_became_peered":"2024-01-16T15:11:20.643678+0000","last_unstale":"2024-01-16T15:11:22.365158+0000","last_undegraded":"2024-01-16T15:11:22.365158+0000","last_fullsized":"2024-01-16T15:11:22.365158+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:45.704299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.373443+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:11:21.373443+0000","last_peered":"2024-01-16T15:11:21.373443+0000","last_clean":"2024-01-16T15:11:21.373443+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:11:21.373443+0000","last_undegraded":"2024-01-16T15:11:21.373443+0000","last_fullsized":"2024-01-16T15:11:21.373443+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":40,"acting":40,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64852,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306988,"statfs":{"total":96636764160,"available":96570355712,"internally_reserved":0,"allocated":1839104,"data_stored":1123459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.194,"15min":1.4099999999999999},"min":{"1min":0.73899999999999999,"5min":0.60199999999999998,"15min":0.53400000000000003},"max":{"1min":1.6899999999999999,"5min":6.6369999999999996,"15min":45.296999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1619999999999999,"15min":1.3160000000000001},"min":{"1min":0.67200000000000004,"5min":0.65600000000000003,"15min":0.497},"max":{"1min":1.6180000000000001,"5min":2.8799999999999999,"15min":20.126999999999999},"last":1.3640000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3420000000000001,"15min":1.415},"min":{"1min":0.78700000000000003,"5min":0.56200000000000006,"15min":0.499},"max":{"1min":2.3969999999999998,"5min":7.4809999999999999,"15min":42.723999999999997},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.353,"15min":1.587},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.49099999999999999},"max":{"1min":1.871,"5min":2.5779999999999998,"15min":43.572000000000003},"last":1.2170000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.21,"15min":1.446},"min":{"1min":0.66700000000000004,"5min":0.55900000000000005,"15min":0.52100000000000002},"max":{"1min":1.657,"5min":3.7290000000000001,"15min":35.308},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.339,"15min":1.4790000000000001},"min":{"1min":0.84499999999999997,"5min":0.76100000000000001,"15min":0.503},"max":{"1min":2.0579999999999998,"5min":3.7589999999999999,"15min":23.594000000000001},"last":1.405}]},{"osd":3,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3600000000000001,"15min":1.6120000000000001},"min":{"1min":0.86799999999999999,"5min":0.65900000000000003,"15min":0.51300000000000001},"max":{"1min":1.851,"5min":6.5739999999999998,"15min":53.033999999999999},"last":0.995},{"interface":"front","average":{"1min":1.474,"5min":1.46,"15min":1.617},"min":{"1min":1.069,"5min":0.72799999999999998,"15min":0.60299999999999998},"max":{"1min":1.9139999999999999,"5min":8.8900000000000006,"15min":31.305},"last":1.2529999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.329,"15min":1.341},"min":{"1min":0.86899999999999999,"5min":0.49199999999999999,"15min":0.39600000000000002},"max":{"1min":2.1110000000000002,"5min":2.1819999999999999,"15min":5.1870000000000003},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.323,"15min":1.514},"min":{"1min":0.95899999999999996,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":2.161,"5min":2.9809999999999999,"15min":36.374000000000002},"last":0.85499999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.4339999999999999,"15min":1.474},"min":{"1min":1.036,"5min":0.61799999999999999,"15min":0.41499999999999998},"max":{"1min":2.25,"5min":4.1529999999999996,"15min":9.9399999999999995},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.635,"5min":1.5,"15min":1.4990000000000001},"min":{"1min":0.96499999999999997,"5min":0.54700000000000004,"15min":0.46899999999999997},"max":{"1min":3.1949999999999998,"5min":3.1949999999999998,"15min":7.0170000000000003},"last":1.4790000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.4970000000000001,"15min":1.5640000000000001},"min":{"1min":1.083,"5min":0.59899999999999998,"15min":0.53700000000000003},"max":{"1min":2.0419999999999998,"5min":3.5680000000000001,"15min":15.052},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.615,"15min":1.6319999999999999},"min":{"1min":1.2749999999999999,"5min":0.59399999999999997,"15min":0.49099999999999999},"max":{"1min":2.2090000000000001,"5min":2.7360000000000002,"15min":8.4849999999999994},"last":1.532}]}]},{"osd":6,"up_from":13,"seq":55834575030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76700,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295140,"statfs":{"total":96636764160,"available":96558223360,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77330886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.363,"15min":1.6399999999999999},"min":{"1min":0.626,"5min":0.61899999999999999,"15min":0.60099999999999998},"max":{"1min":1.6220000000000001,"5min":9.1400000000000006,"15min":71.870999999999995},"last":1.621},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.327,"15min":1.7410000000000001},"min":{"1min":0.71599999999999997,"5min":0.67200000000000004,"15min":0.501},"max":{"1min":2.2000000000000002,"5min":6.4059999999999997,"15min":71.846999999999994},"last":1.381}]},{"osd":1,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4530000000000001,"15min":1.3300000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.63900000000000001},"max":{"1min":3.4049999999999998,"5min":15.798,"15min":15.798},"last":1.462},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.528,"15min":1.4359999999999999},"min":{"1min":0.748,"5min":0.626,"15min":0.626},"max":{"1min":1.8979999999999999,"5min":24.556999999999999,"15min":24.556999999999999},"last":1.514}]},{"osd":2,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.651,"15min":1.4470000000000001},"min":{"1min":0.93500000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.3029999999999999,"5min":14.981,"15min":14.981},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.415,"5min":1.728,"15min":1.514},"min":{"1min":0.89900000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.7090000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":1.151}]},{"osd":3,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.367,"15min":1.488},"min":{"1min":1.044,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.1789999999999998,"5min":4.5410000000000004,"15min":14.827},"last":1.042},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.581},"min":{"1min":1.0669999999999999,"5min":0.499,"15min":0.499},"max":{"1min":2.032,"5min":21.890000000000001,"15min":21.890000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5169999999999999,"15min":1.4550000000000001},"min":{"1min":0.95899999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.1259999999999999,"5min":10.901,"15min":10.901},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.3520000000000001,"15min":1.444},"min":{"1min":1.014,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":6.9000000000000004},"last":1.569}]},{"osd":5,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3360000000000001,"15min":1.415},"min":{"1min":0.85999999999999999,"5min":0.44600000000000001,"15min":0.38500000000000001},"max":{"1min":2.1669999999999998,"5min":2.427,"15min":10.736000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.605,"15min":1.6240000000000001},"min":{"1min":1.3049999999999999,"5min":0.498,"15min":0.498},"max":{"1min":2.2130000000000001,"5min":9.2200000000000006,"15min":9.2200000000000006},"last":1.7210000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6669999999999998,"5min":1.776,"15min":1.7},"min":{"1min":1.2649999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":24.675000000000001,"5min":24.675000000000001,"15min":24.675000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6619999999999999,"15min":1.7030000000000001},"min":{"1min":1.0840000000000001,"5min":0.54600000000000004,"15min":0.42599999999999999},"max":{"1min":3.9580000000000002,"5min":13.991,"15min":14.494},"last":1.7709999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54556,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317284,"statfs":{"total":96636764160,"available":96580898816,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.7070000000000001,"15min":1.508},"min":{"1min":0.51600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.615,"5min":25.634,"15min":25.634},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.4219999999999999,"15min":1.4830000000000001},"min":{"1min":0.66900000000000004,"5min":0.49299999999999999,"15min":0.48299999999999998},"max":{"1min":1.6220000000000001,"5min":16.263000000000002,"15min":34.959000000000003},"last":1.71}]},{"osd":1,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.5329999999999999,"15min":1.4159999999999999},"min":{"1min":0.72899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.8460000000000001,"5min":26.378,"15min":26.378},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.3169999999999999,"15min":1.3140000000000001},"min":{"1min":0.61099999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.75,"5min":6.125,"15min":13.332000000000001},"last":1.022}]},{"osd":2,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.5700000000000001,"15min":1.3979999999999999},"min":{"1min":0.751,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":1.821,"5min":25.495000000000001,"15min":25.495000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.4399999999999999,"15min":1.4630000000000001},"min":{"1min":0.63800000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.012,"5min":6.3689999999999998,"15min":17.030000000000001},"last":1.2430000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.605,"15min":1.653},"min":{"1min":0.70099999999999996,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9379999999999999,"5min":25.82,"15min":25.82},"last":1.304},{"interface":"front","average":{"1min":1.419,"5min":1.409,"15min":1.6719999999999999},"min":{"1min":0.71699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8959999999999999,"5min":4.4269999999999996,"15min":22.963999999999999},"last":1.5469999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.266,"15min":1.288},"min":{"1min":0.58199999999999996,"5min":0.45500000000000002,"15min":0.378},"max":{"1min":1.964,"5min":6.1660000000000004,"15min":8.6050000000000004},"last":1.8},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.7430000000000001,"15min":1.5820000000000001},"min":{"1min":0.67200000000000004,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":2.0339999999999998,"5min":25.724,"15min":25.724},"last":1.871}]},{"osd":6,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.931,"15min":1.629},"min":{"1min":0.56499999999999995,"5min":0.46800000000000003,"15min":0.40100000000000002},"max":{"1min":1.7949999999999999,"5min":42.542999999999999,"15min":42.542999999999999},"last":1.349},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3700000000000001,"15min":1.389},"min":{"1min":0.65100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8500000000000001,"5min":6.093,"15min":10.015000000000001},"last":2.0590000000000002}]},{"osd":7,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.873,"15min":1.694},"min":{"1min":0.626,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.1179999999999999,"5min":26.484000000000002,"15min":26.484000000000002},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.46999999999999997},"max":{"1min":3.2509999999999999,"5min":17.233000000000001,"15min":21.106000000000002},"last":1.77}]}]},{"osd":4,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76704,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295136,"statfs":{"total":96636764160,"available":96558219264,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.54,"15min":1.3899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.44400000000000001},"max":{"1min":17.315999999999999,"5min":17.315999999999999,"15min":17.315999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":2.077,"5min":1.5309999999999999,"15min":1.415},"min":{"1min":0.79700000000000004,"5min":0.60399999999999998,"15min":0.52800000000000002},"max":{"1min":17.279,"5min":17.279,"15min":17.279},"last":1.1459999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.306,"15min":1.4419999999999999},"min":{"1min":0.75700000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.8080000000000001,"5min":6.9450000000000003,"15min":39.823999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.349,"5min":1.2190000000000001,"15min":1.389},"min":{"1min":0.80800000000000005,"5min":0.53700000000000003,"15min":0.48399999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":50.582000000000001},"last":0.93600000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3260000000000001,"15min":1.345},"min":{"1min":0.94099999999999995,"5min":0.5,"15min":0.39500000000000002},"max":{"1min":1.6180000000000001,"5min":7.319,"15min":10.984},"last":1.165},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.462,"15min":1.403},"min":{"1min":0.94999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.744,"5min":6.8179999999999996,"15min":9.1509999999999998},"last":1.288}]},{"osd":3,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.393,"15min":1.367},"min":{"1min":1.04,"5min":0.58199999999999996,"15min":0.54000000000000004},"max":{"1min":5.0199999999999996,"5min":8.8979999999999997,"15min":11.971},"last":1.113},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.4410000000000001,"15min":1.633},"min":{"1min":0.81399999999999995,"5min":0.61699999999999999,"15min":0.57899999999999996},"max":{"1min":2.0550000000000002,"5min":7.399,"15min":27.626999999999999},"last":1.0960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.369,"15min":1.377},"min":{"1min":0.88200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.619,"5min":4.2329999999999997,"15min":8.2599999999999998},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.536,"5min":1.661,"15min":1.518},"min":{"1min":0.98299999999999998,"5min":0.47599999999999998,"15min":0.46200000000000002},"max":{"1min":4.8520000000000003,"5min":17.140999999999998,"15min":17.140999999999998},"last":0.64100000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.46,"15min":1.484},"min":{"1min":0.74299999999999999,"5min":0.48299999999999998,"15min":0.45300000000000001},"max":{"1min":1.7529999999999999,"5min":7.4509999999999996,"15min":7.4509999999999996},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.444,"5min":1.885,"15min":1.621},"min":{"1min":0.88500000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.885,"5min":25.643000000000001,"15min":25.643000000000001},"last":1.0269999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.7769999999999999,"15min":1.591},"min":{"1min":1.103,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":4.9539999999999997,"5min":24.992999999999999,"15min":24.992999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.706,"15min":1.758},"min":{"1min":0.92900000000000005,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":2.0230000000000001,"5min":8.8369999999999997,"15min":19.001000000000001},"last":1.129}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64216,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307624,"statfs":{"total":96636764160,"available":96571006976,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98399999999999999,"15min":1.1479999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":1.8700000000000001,"5min":9.6229999999999993,"15min":38.423999999999999},"last":1.821},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1910000000000001,"15min":1.1950000000000001},"min":{"1min":0.51100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.893,"5min":7.8300000000000001,"15min":20.012},"last":1.1339999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.998,"15min":1.607},"min":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.603,"5min":2.4849999999999999,"15min":40.695999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.155,"5min":1.23,"15min":1.2},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":2.25,"5min":8.5449999999999999,"15min":9.9689999999999994},"last":1.0129999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3049999999999999,"15min":1.542},"min":{"1min":0.56200000000000006,"5min":0.46100000000000002,"15min":0.436},"max":{"1min":2.831,"5min":7.2809999999999997,"15min":59.009},"last":2.0790000000000002},{"interface":"front","average":{"1min":1.298,"5min":1.212,"15min":1.645},"min":{"1min":0.63600000000000001,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":2.2130000000000001,"5min":4.2279999999999998,"15min":29.773},"last":2.2029999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.474,"15min":1.484},"min":{"1min":0.83799999999999997,"5min":0.68799999999999994,"15min":0.59899999999999998},"max":{"1min":3.5049999999999999,"5min":7.4950000000000001,"15min":21.736000000000001},"last":1.865},{"interface":"front","average":{"1min":1.51,"5min":1.4870000000000001,"15min":1.901},"min":{"1min":0.76000000000000001,"5min":0.622,"15min":0.622},"max":{"1min":2.2919999999999998,"5min":9.1289999999999996,"15min":29.524000000000001},"last":1.7370000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.464,"15min":1.494},"min":{"1min":0.83399999999999996,"5min":0.64500000000000002,"15min":0.54800000000000004},"max":{"1min":2.3540000000000001,"5min":8.0739999999999998,"15min":9.8249999999999993},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.5660000000000001,"15min":1.573},"min":{"1min":0.94699999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.3969999999999998,"5min":11.255000000000001,"15min":11.255000000000001},"last":1.613}]},{"osd":6,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5780000000000001,"15min":1.589},"min":{"1min":0.88700000000000001,"5min":0.752,"15min":0.65900000000000003},"max":{"1min":2.9470000000000001,"5min":7.4560000000000004,"15min":20.459},"last":1.456},{"interface":"front","average":{"1min":1.702,"5min":1.6519999999999999,"15min":2.0049999999999999},"min":{"1min":0.86499999999999999,"5min":0.67000000000000004,"15min":0.49399999999999999},"max":{"1min":3.46,"5min":12.064,"15min":29.475000000000001},"last":1.2010000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.653,"15min":1.643},"min":{"1min":0.96899999999999997,"5min":0.73099999999999998,"15min":0.55300000000000005},"max":{"1min":3.7850000000000001,"5min":7.5339999999999998,"15min":20.109999999999999},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.617,"15min":2.0089999999999999},"min":{"1min":0.99199999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.0590000000000002,"5min":10.804,"15min":29.396999999999998},"last":1.9339999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.2250000000000001,"15min":1.143},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.5389999999999999,"5min":24.155000000000001,"15min":25.335999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0469999999999999,"15min":1.2150000000000001},"min":{"1min":0.60499999999999998,"5min":0.42299999999999999,"15min":0.27800000000000002},"max":{"1min":1.923,"5min":2.952,"15min":25.199000000000002},"last":1.119}]},{"osd":1,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2569999999999999,"15min":1.224},"min":{"1min":0.70099999999999996,"5min":0.59099999999999997,"15min":0.36199999999999999},"max":{"1min":1.4730000000000001,"5min":16.776,"15min":16.776},"last":1.284},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.4330000000000001,"15min":1.4390000000000001},"min":{"1min":0.749,"5min":0.48099999999999998,"15min":0.318},"max":{"1min":1.8240000000000001,"5min":25.186,"15min":25.186},"last":0.94699999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.4590000000000001,"15min":1.2569999999999999},"min":{"1min":0.72299999999999998,"5min":0.48499999999999999,"15min":0.33900000000000002},"max":{"1min":1.482,"5min":37.470999999999997,"15min":37.470999999999997},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2909999999999999,"15min":1.3520000000000001},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.46500000000000002},"max":{"1min":1.6100000000000001,"5min":6.5549999999999997,"15min":28.045999999999999},"last":1.391}]},{"osd":4,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0680000000000001,"5min":1.9319999999999999,"15min":1.7789999999999999},"min":{"1min":0.96399999999999997,"5min":0.65200000000000002,"15min":0.47499999999999998},"max":{"1min":16.681999999999999,"5min":25.062000000000001,"15min":25.062000000000001},"last":1.347},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.7509999999999999,"15min":1.6279999999999999},"min":{"1min":1.0369999999999999,"5min":0.83599999999999997,"15min":0.502},"max":{"1min":1.788,"5min":37.307000000000002,"15min":37.307000000000002},"last":1.323}]},{"osd":5,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5109999999999999,"15min":1.554},"min":{"1min":0.86099999999999999,"5min":0.72799999999999998,"15min":0.57299999999999995},"max":{"1min":2.0310000000000001,"5min":6.8849999999999998,"15min":19.327000000000002},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.7989999999999999,"15min":1.786},"min":{"1min":1.048,"5min":0.73699999999999999,"15min":0.52400000000000002},"max":{"1min":4.8659999999999997,"5min":25.141999999999999,"15min":25.141999999999999},"last":1.663}]},{"osd":6,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.8300000000000001,"15min":1.621},"min":{"1min":1.0600000000000001,"5min":0.99399999999999999,"15min":0.56000000000000005},"max":{"1min":1.9330000000000001,"5min":37.363,"15min":37.363},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.73,"5min":1.6359999999999999,"15min":1.583},"min":{"1min":1.1060000000000001,"5min":0.68899999999999995,"15min":0.58799999999999997},"max":{"1min":6.2599999999999998,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.476}]},{"osd":7,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.8169999999999999,"15min":1.845},"min":{"1min":1.0680000000000001,"5min":0.66100000000000003,"15min":0.45400000000000001},"max":{"1min":2.1899999999999999,"5min":25.106999999999999,"15min":25.106999999999999},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.9299999999999999,"15min":1.708},"min":{"1min":1.1719999999999999,"5min":0.88200000000000001,"15min":0.61499999999999999},"max":{"1min":2.0049999999999999,"5min":37.402999999999999,"15min":37.402999999999999},"last":1.518}]}]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64220,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307620,"statfs":{"total":96636764160,"available":96571002880,"internally_reserved":0,"allocated":1187840,"data_stored":507812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.2050000000000001,"15min":1.226},"min":{"1min":0.51600000000000001,"5min":0.36399999999999999,"15min":0.28799999999999998},"max":{"1min":14.106999999999999,"5min":14.106999999999999,"15min":17.210999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1319999999999999,"15min":1.339},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.26700000000000002},"max":{"1min":2.9089999999999998,"5min":5.6059999999999999,"15min":34.252000000000002},"last":0.627}]},{"osd":2,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1559999999999999,"15min":1.3220000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1219999999999999,"5min":6.1189999999999998,"15min":24.709},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.5589999999999999,"15min":1.494},"min":{"1min":0.46999999999999997,"5min":0.436,"15min":0.377},"max":{"1min":14.055,"5min":21.795999999999999,"15min":30.128},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.944,"15min":1.583},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.40300000000000002},"max":{"1min":13.967000000000001,"5min":25.611000000000001,"15min":25.611000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.4490000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":2.75,"5min":4.6699999999999999,"15min":41.972000000000001},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3460000000000001,"15min":1.4510000000000001},"min":{"1min":0.874,"5min":0.52500000000000002,"15min":0.44600000000000001},"max":{"1min":4.3799999999999999,"5min":4.5739999999999998,"15min":18.004999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":2.0529999999999999,"5min":2.008,"15min":1.724},"min":{"1min":0.74199999999999999,"5min":0.68100000000000005,"15min":0.63700000000000001},"max":{"1min":14.161,"5min":21.738,"15min":21.738},"last":0.96399999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.913,"15min":1.752},"min":{"1min":1.044,"5min":0.58699999999999997,"15min":0.46600000000000003},"max":{"1min":3.8130000000000002,"5min":17.739999999999998,"15min":20.905999999999999},"last":1.085},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.8480000000000001,"15min":1.762},"min":{"1min":0.98599999999999999,"5min":0.56100000000000005,"15min":0.48799999999999999},"max":{"1min":4.2489999999999997,"5min":16.433,"15min":17.957999999999998},"last":0.93200000000000005}]},{"osd":6,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.966,"15min":1.772},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":14.212,"5min":21.832999999999998,"15min":21.832999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.766,"15min":1.655},"min":{"1min":0.94199999999999995,"5min":0.63,"15min":0.59199999999999997},"max":{"1min":3.0990000000000002,"5min":9.4749999999999996,"15min":20.509},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.9670000000000001,"15min":1.804},"min":{"1min":1.008,"5min":0.75700000000000001,"15min":0.55600000000000005},"max":{"1min":2.9790000000000001,"5min":19.747,"15min":20.669},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.516,"5min":1.556,"15min":1.6399999999999999},"min":{"1min":0.82999999999999996,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":3.0390000000000001,"5min":6.2699999999999996,"15min":18.039999999999999},"last":1.0720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575029,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77268,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294572,"statfs":{"total":96636764160,"available":96557641728,"internally_reserved":0,"allocated":1777664,"data_stored":1097161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.98899999999999999,"15min":1.1539999999999999},"min":{"1min":0.44600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4490000000000001,"5min":10.683999999999999,"15min":26.751000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.123,"15min":1.1639999999999999},"min":{"1min":0.60399999999999998,"5min":0.433,"15min":0.35899999999999999},"max":{"1min":1.415,"5min":7.649,"15min":15.750999999999999},"last":0.80600000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.641,"15min":1.5649999999999999},"min":{"1min":0.52600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":35.320999999999998,"15min":38.517000000000003},"last":0.872},{"interface":"front","average":{"1min":1.107,"5min":1.292,"15min":1.2829999999999999},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.38800000000000001},"max":{"1min":1.8320000000000001,"5min":11.720000000000001,"15min":24.402000000000001},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.355,"15min":1.502},"min":{"1min":0.58499999999999996,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2789999999999999,"5min":14.677,"15min":29.808},"last":1.056},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.4450000000000001,"15min":1.6919999999999999},"min":{"1min":0.622,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.823,"5min":14.558,"15min":32.302},"last":1.359}]},{"osd":4,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.5629999999999999,"15min":1.5669999999999999},"min":{"1min":0.71899999999999997,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2000000000000002,"5min":9.6080000000000005,"15min":15.832000000000001},"last":1.276},{"interface":"front","average":{"1min":1.288,"5min":1.417,"15min":1.5349999999999999},"min":{"1min":0.68300000000000005,"5min":0.64400000000000002,"15min":0.432},"max":{"1min":1.8859999999999999,"5min":6.9820000000000002,"15min":19.062999999999999},"last":1.306}]},{"osd":5,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.462,"15min":1.6699999999999999},"min":{"1min":0.84699999999999998,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.7270000000000001,"5min":4.7949999999999999,"15min":26.800000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.6060000000000001,"15min":1.601},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.3559999999999999,"5min":7.1550000000000002,"15min":15.906000000000001},"last":1.1759999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.427,"15min":1.6639999999999999},"min":{"1min":0.93500000000000005,"5min":0.65700000000000003,"15min":0.60999999999999999},"max":{"1min":2.1739999999999999,"5min":4.7389999999999999,"15min":20.844000000000001},"last":1.575},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.488,"15min":1.6799999999999999},"min":{"1min":0.96699999999999997,"5min":0.73199999999999998,"15min":0.52800000000000002},"max":{"1min":1.9359999999999999,"5min":4.4109999999999996,"15min":26.832000000000001},"last":1.23}]},{"osd":7,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.742,"15min":1.6870000000000001},"min":{"1min":0.93999999999999995,"5min":0.69799999999999995,"15min":0.59599999999999997},"max":{"1min":2.484,"5min":16.058,"15min":16.058},"last":1.534},{"interface":"front","average":{"1min":1.353,"5min":1.611,"15min":1.8740000000000001},"min":{"1min":0.89100000000000001,"5min":0.70199999999999996,"15min":0.47499999999999998},"max":{"1min":2.0329999999999999,"5min":8.0609999999999999,"15min":29.286999999999999},"last":1.6479999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:11:26.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:11:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.273+0000 7ff679800640 1 Processor -- start 2024-01-16T15:11:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.273+0000 7ff679800640 1 -- start start 2024-01-16T15:11:26.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.274+0000 7ff679800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.274+0000 7ff679800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff679800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67415f9c0 0x7ff6740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff679800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67406c470 con 0x7ff67413df80 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff679800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67406c750 con 0x7ff67413e360 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff679800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406ca30 con 0x7ff67415f9c0 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff672a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff672a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49426/0 (socket says 172.21.15.125:49426) 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff672a00640 1 -- 172.21.15.125:0/2815236484 learned_addr learned my addr 172.21.15.125:0/2815236484 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff66be00640 1 --2- 172.21.15.125:0/2815236484 >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.275+0000 7ff673400640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67415f9c0 0x7ff6740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff672a00640 1 -- 172.21.15.125:0/2815236484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67415f9c0 msgr2=0x7ff6740cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff672a00640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67415f9c0 0x7ff6740cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff672a00640 1 -- 172.21.15.125:0/2815236484 >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 msgr2=0x7ff6740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff672a00640 1 --2- 172.21.15.125:0/2815236484 >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff672a00640 1 -- 172.21.15.125:0/2815236484 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67406cd10 con 0x7ff67413df80 2024-01-16T15:11:26.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.276+0000 7ff66be00640 1 --2- 172.21.15.125:0/2815236484 >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:26.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.277+0000 7ff672a00640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff67406ded0 tx=0x7ff66402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132d943ade075f61 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.277+0000 7ff672000640 1 -- 172.21.15.125:0/2815236484 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031870 con 0x7ff67413df80 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff672000640 1 -- 172.21.15.125:0/2815236484 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6640319d0 con 0x7ff67413df80 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff672000640 1 -- 172.21.15.125:0/2815236484 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031d60 con 0x7ff67413df80 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 -- 172.21.15.125:0/2815236484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67413df80 msgr2=0x7ff67415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff67406ded0 tx=0x7ff66402b910 comp rx=0 tx=0).stop 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 -- 172.21.15.125:0/2815236484 shutdown_connections 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67415f9c0 0x7ff6740cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 --2- 172.21.15.125:0/2815236484 >> v2:172.21.15.137:3300/0 conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 --2- 172.21.15.125:0/2815236484 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 -- 172.21.15.125:0/2815236484 >> 172.21.15.125:0/2815236484 conn(0x7ff67406a800 msgr2=0x7ff67406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 -- 172.21.15.125:0/2815236484 shutdown_connections 2024-01-16T15:11:26.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.278+0000 7ff679800640 1 -- 172.21.15.125:0/2815236484 wait complete. 2024-01-16T15:11:26.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.280+0000 7ff679800640 1 Processor -- start 2024-01-16T15:11:26.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.280+0000 7ff679800640 1 -- start start 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff67413df80 0x7ff6740af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67413e360 0x7ff674139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6740afa30 con 0x7ff67415f9c0 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67414f3a0 con 0x7ff67413df80 2024-01-16T15:11:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.281+0000 7ff679800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6740b02c0 con 0x7ff67413e360 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49442/0 (socket says 172.21.15.125:49442) 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 -- 172.21.15.125:0/4112337049 learned_addr learned my addr 172.21.15.125:0/4112337049 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff672a00640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff67413df80 0x7ff6740af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff66be00640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67413e360 0x7ff674139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 -- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67413e360 msgr2=0x7ff674139790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67413e360 0x7ff674139790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 -- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff67413df80 msgr2=0x7ff6740af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff67413df80 0x7ff6740af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 -- 172.21.15.125:0/4112337049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640094d0 con 0x7ff67415f9c0 2024-01-16T15:11:26.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.282+0000 7ff673400640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff66c007e30 tx=0x7ff66c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.283+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c014620 con 0x7ff67415f9c0 2024-01-16T15:11:26.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.283+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c00ee90 con 0x7ff67415f9c0 2024-01-16T15:11:26.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.283+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c0149c0 con 0x7ff67415f9c0 2024-01-16T15:11:26.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.283+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6741538e0 con 0x7ff67415f9c0 2024-01-16T15:11:26.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.283+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff674154180 con 0x7ff67415f9c0 2024-01-16T15:11:26.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.284+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6740675d0 con 0x7ff67415f9c0 2024-01-16T15:11:26.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.285+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff66c014b20 con 0x7ff67415f9c0 2024-01-16T15:11:26.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.287+0000 7ff66aa00640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 0x7ff64c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.287+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff66c089230 con 0x7ff67415f9c0 2024-01-16T15:11:26.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.287+0000 7ff672a00640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 0x7ff64c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.287+0000 7ff672a00640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 0x7ff64c06e650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff664036000 tx=0x7ff66403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.291+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff66c0bee90 con 0x7ff67415f9c0 2024-01-16T15:11:26.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.483+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6740cdf10 con 0x7ff64c06c190 2024-01-16T15:11:26.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.504+0000 7ff66aa00640 1 -- 172.21.15.125:0/4112337049 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287088 (secure 0 0 0) 0x7ff6740cdf10 con 0x7ff64c06c190 2024-01-16T15:11:26.484 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:26.486 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:11:26.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.509+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 msgr2=0x7ff64c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.509+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 0x7ff64c06e650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff664036000 tx=0x7ff66403c040 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.509+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 msgr2=0x7ff67413b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.509+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff66c007e30 tx=0x7ff66c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 shutdown_connections 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff64c06c190 0x7ff64c06e650 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff67415f9c0 0x7ff67413b020 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff67413e360 0x7ff674139790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 --2- 172.21.15.125:0/4112337049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff67413df80 0x7ff6740af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 >> 172.21.15.125:0/4112337049 conn(0x7ff67406a800 msgr2=0x7ff6740cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 shutdown_connections 2024-01-16T15:11:26.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.510+0000 7ff679800640 1 -- 172.21.15.125:0/4112337049 wait complete. 2024-01-16T15:11:26.505 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-01-16T15:11:26.262060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":543096,"kb_used_data":10884,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754431624,"statfs":{"total":773094113280,"available":772537982976,"internally_reserved":0,"allocated":11145216,"data_stored":5448369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":544853552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001827"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337800+0000","last_change":"2024-01-16T15:11:21.388911+0000","last_active":"2024-01-16T15:11:23.337800+0000","last_peered":"2024-01-16T15:11:23.337800+0000","last_clean":"2024-01-16T15:11:23.337800+0000","last_became_active":"2024-01-16T15:11:21.386105+0000","last_became_peered":"2024-01-16T15:11:21.386105+0000","last_unstale":"2024-01-16T15:11:23.337800+0000","last_undegraded":"2024-01-16T15:11:23.337800+0000","last_fullsized":"2024-01-16T15:11:23.337800+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:04:06.449937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338922+0000","last_change":"2024-01-16T15:11:21.389561+0000","last_active":"2024-01-16T15:11:23.338922+0000","last_peered":"2024-01-16T15:11:23.338922+0000","last_clean":"2024-01-16T15:11:23.338922+0000","last_became_active":"2024-01-16T15:11:21.389092+0000","last_became_peered":"2024-01-16T15:11:21.389092+0000","last_unstale":"2024-01-16T15:11:23.338922+0000","last_undegraded":"2024-01-16T15:11:23.338922+0000","last_fullsized":"2024-01-16T15:11:23.338922+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:14.744009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337038+0000","last_change":"2024-01-16T15:11:21.393233+0000","last_active":"2024-01-16T15:11:23.337038+0000","last_peered":"2024-01-16T15:11:23.337038+0000","last_clean":"2024-01-16T15:11:23.337038+0000","last_became_active":"2024-01-16T15:11:21.392651+0000","last_became_peered":"2024-01-16T15:11:21.392651+0000","last_unstale":"2024-01-16T15:11:23.337038+0000","last_undegraded":"2024-01-16T15:11:23.337038+0000","last_fullsized":"2024-01-16T15:11:23.337038+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:00.865427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337189+0000","last_change":"2024-01-16T15:11:21.391542+0000","last_active":"2024-01-16T15:11:23.337189+0000","last_peered":"2024-01-16T15:11:23.337189+0000","last_clean":"2024-01-16T15:11:23.337189+0000","last_became_active":"2024-01-16T15:11:21.391280+0000","last_became_peered":"2024-01-16T15:11:21.391280+0000","last_unstale":"2024-01-16T15:11:23.337189+0000","last_undegraded":"2024-01-16T15:11:23.337189+0000","last_fullsized":"2024-01-16T15:11:23.337189+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:28.464408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336200+0000","last_change":"2024-01-16T15:11:21.396230+0000","last_active":"2024-01-16T15:11:23.336200+0000","last_peered":"2024-01-16T15:11:23.336200+0000","last_clean":"2024-01-16T15:11:23.336200+0000","last_became_active":"2024-01-16T15:11:21.395414+0000","last_became_peered":"2024-01-16T15:11:21.395414+0000","last_unstale":"2024-01-16T15:11:23.336200+0000","last_undegraded":"2024-01-16T15:11:23.336200+0000","last_fullsized":"2024-01-16T15:11:23.336200+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:14.949265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365260+0000","last_change":"2024-01-16T15:11:21.386770+0000","last_active":"2024-01-16T15:11:22.365260+0000","last_peered":"2024-01-16T15:11:22.365260+0000","last_clean":"2024-01-16T15:11:22.365260+0000","last_became_active":"2024-01-16T15:11:21.385942+0000","last_became_peered":"2024-01-16T15:11:21.385942+0000","last_unstale":"2024-01-16T15:11:22.365260+0000","last_undegraded":"2024-01-16T15:11:22.365260+0000","last_fullsized":"2024-01-16T15:11:22.365260+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:53.149749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365803+0000","last_change":"2024-01-16T15:11:21.386478+0000","last_active":"2024-01-16T15:11:22.365803+0000","last_peered":"2024-01-16T15:11:22.365803+0000","last_clean":"2024-01-16T15:11:22.365803+0000","last_became_active":"2024-01-16T15:11:21.386037+0000","last_became_peered":"2024-01-16T15:11:21.386037+0000","last_unstale":"2024-01-16T15:11:22.365803+0000","last_undegraded":"2024-01-16T15:11:22.365803+0000","last_fullsized":"2024-01-16T15:11:22.365803+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:07.539617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337475+0000","last_change":"2024-01-16T15:11:21.391075+0000","last_active":"2024-01-16T15:11:23.337475+0000","last_peered":"2024-01-16T15:11:23.337475+0000","last_clean":"2024-01-16T15:11:23.337475+0000","last_became_active":"2024-01-16T15:11:21.387355+0000","last_became_peered":"2024-01-16T15:11:21.387355+0000","last_unstale":"2024-01-16T15:11:23.337475+0000","last_undegraded":"2024-01-16T15:11:23.337475+0000","last_fullsized":"2024-01-16T15:11:23.337475+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:21.158872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367696+0000","last_change":"2024-01-16T15:11:21.386723+0000","last_active":"2024-01-16T15:11:22.367696+0000","last_peered":"2024-01-16T15:11:22.367696+0000","last_clean":"2024-01-16T15:11:22.367696+0000","last_became_active":"2024-01-16T15:11:21.386196+0000","last_became_peered":"2024-01-16T15:11:21.386196+0000","last_unstale":"2024-01-16T15:11:22.367696+0000","last_undegraded":"2024-01-16T15:11:22.367696+0000","last_fullsized":"2024-01-16T15:11:22.367696+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:10.519253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336404+0000","last_change":"2024-01-16T15:11:21.393911+0000","last_active":"2024-01-16T15:11:23.336404+0000","last_peered":"2024-01-16T15:11:23.336404+0000","last_clean":"2024-01-16T15:11:23.336404+0000","last_became_active":"2024-01-16T15:11:21.393175+0000","last_became_peered":"2024-01-16T15:11:21.393175+0000","last_unstale":"2024-01-16T15:11:23.336404+0000","last_undegraded":"2024-01-16T15:11:23.336404+0000","last_fullsized":"2024-01-16T15:11:23.336404+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:49.223003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365650+0000","last_change":"2024-01-16T15:11:21.384137+0000","last_active":"2024-01-16T15:11:22.365650+0000","last_peered":"2024-01-16T15:11:22.365650+0000","last_clean":"2024-01-16T15:11:22.365650+0000","last_became_active":"2024-01-16T15:11:21.383816+0000","last_became_peered":"2024-01-16T15:11:21.383816+0000","last_unstale":"2024-01-16T15:11:22.365650+0000","last_undegraded":"2024-01-16T15:11:22.365650+0000","last_fullsized":"2024-01-16T15:11:22.365650+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:11:33.092792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367341+0000","last_change":"2024-01-16T15:11:21.384356+0000","last_active":"2024-01-16T15:11:22.367341+0000","last_peered":"2024-01-16T15:11:22.367341+0000","last_clean":"2024-01-16T15:11:22.367341+0000","last_became_active":"2024-01-16T15:11:21.382623+0000","last_became_peered":"2024-01-16T15:11:21.382623+0000","last_unstale":"2024-01-16T15:11:22.367341+0000","last_undegraded":"2024-01-16T15:11:22.367341+0000","last_fullsized":"2024-01-16T15:11:22.367341+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:03.751578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336294+0000","last_change":"2024-01-16T15:11:21.375853+0000","last_active":"2024-01-16T15:11:23.336294+0000","last_peered":"2024-01-16T15:11:23.336294+0000","last_clean":"2024-01-16T15:11:23.336294+0000","last_became_active":"2024-01-16T15:11:21.375116+0000","last_became_peered":"2024-01-16T15:11:21.375116+0000","last_unstale":"2024-01-16T15:11:23.336294+0000","last_undegraded":"2024-01-16T15:11:23.336294+0000","last_fullsized":"2024-01-16T15:11:23.336294+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:10.519253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364094+0000","last_change":"2024-01-16T15:11:21.400463+0000","last_active":"2024-01-16T15:11:22.364094+0000","last_peered":"2024-01-16T15:11:22.364094+0000","last_clean":"2024-01-16T15:11:22.364094+0000","last_became_active":"2024-01-16T15:11:21.399558+0000","last_became_peered":"2024-01-16T15:11:21.399558+0000","last_unstale":"2024-01-16T15:11:22.364094+0000","last_undegraded":"2024-01-16T15:11:22.364094+0000","last_fullsized":"2024-01-16T15:11:22.364094+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:35.124511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336096+0000","last_change":"2024-01-16T15:11:21.384957+0000","last_active":"2024-01-16T15:11:23.336096+0000","last_peered":"2024-01-16T15:11:23.336096+0000","last_clean":"2024-01-16T15:11:23.336096+0000","last_became_active":"2024-01-16T15:11:21.384640+0000","last_became_peered":"2024-01-16T15:11:21.384640+0000","last_unstale":"2024-01-16T15:11:23.336096+0000","last_undegraded":"2024-01-16T15:11:23.336096+0000","last_fullsized":"2024-01-16T15:11:23.336096+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:30:32.573661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365080+0000","last_change":"2024-01-16T15:11:21.395324+0000","last_active":"2024-01-16T15:11:22.365080+0000","last_peered":"2024-01-16T15:11:22.365080+0000","last_clean":"2024-01-16T15:11:22.365080+0000","last_became_active":"2024-01-16T15:11:21.394570+0000","last_became_peered":"2024-01-16T15:11:21.394570+0000","last_unstale":"2024-01-16T15:11:22.365080+0000","last_undegraded":"2024-01-16T15:11:22.365080+0000","last_fullsized":"2024-01-16T15:11:22.365080+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:44.386586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.339300+0000","last_change":"2024-01-16T15:11:21.391026+0000","last_active":"2024-01-16T15:11:23.339300+0000","last_peered":"2024-01-16T15:11:23.339300+0000","last_clean":"2024-01-16T15:11:23.339300+0000","last_became_active":"2024-01-16T15:11:21.386913+0000","last_became_peered":"2024-01-16T15:11:21.386913+0000","last_unstale":"2024-01-16T15:11:23.339300+0000","last_undegraded":"2024-01-16T15:11:23.339300+0000","last_fullsized":"2024-01-16T15:11:23.339300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:53.736917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338074+0000","last_change":"2024-01-16T15:11:21.398113+0000","last_active":"2024-01-16T15:11:23.338074+0000","last_peered":"2024-01-16T15:11:23.338074+0000","last_clean":"2024-01-16T15:11:23.338074+0000","last_became_active":"2024-01-16T15:11:21.397810+0000","last_became_peered":"2024-01-16T15:11:21.397810+0000","last_unstale":"2024-01-16T15:11:23.338074+0000","last_undegraded":"2024-01-16T15:11:23.338074+0000","last_fullsized":"2024-01-16T15:11:23.338074+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:08:27.830592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364969+0000","last_change":"2024-01-16T15:11:21.399184+0000","last_active":"2024-01-16T15:11:22.364969+0000","last_peered":"2024-01-16T15:11:22.364969+0000","last_clean":"2024-01-16T15:11:22.364969+0000","last_became_active":"2024-01-16T15:11:21.398422+0000","last_became_peered":"2024-01-16T15:11:21.398422+0000","last_unstale":"2024-01-16T15:11:22.364969+0000","last_undegraded":"2024-01-16T15:11:22.364969+0000","last_fullsized":"2024-01-16T15:11:22.364969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:32.447293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367665+0000","last_change":"2024-01-16T15:11:21.385042+0000","last_active":"2024-01-16T15:11:22.367665+0000","last_peered":"2024-01-16T15:11:22.367665+0000","last_clean":"2024-01-16T15:11:22.367665+0000","last_became_active":"2024-01-16T15:11:21.384772+0000","last_became_peered":"2024-01-16T15:11:21.384772+0000","last_unstale":"2024-01-16T15:11:22.367665+0000","last_undegraded":"2024-01-16T15:11:22.367665+0000","last_fullsized":"2024-01-16T15:11:22.367665+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:18.594508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363770+0000","last_change":"2024-01-16T15:11:21.389998+0000","last_active":"2024-01-16T15:11:22.363770+0000","last_peered":"2024-01-16T15:11:22.363770+0000","last_clean":"2024-01-16T15:11:22.363770+0000","last_became_active":"2024-01-16T15:11:21.389713+0000","last_became_peered":"2024-01-16T15:11:21.389713+0000","last_unstale":"2024-01-16T15:11:22.363770+0000","last_undegraded":"2024-01-16T15:11:22.363770+0000","last_fullsized":"2024-01-16T15:11:22.363770+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:46.348286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337971+0000","last_change":"2024-01-16T15:11:21.388914+0000","last_active":"2024-01-16T15:11:23.337971+0000","last_peered":"2024-01-16T15:11:23.337971+0000","last_clean":"2024-01-16T15:11:23.337971+0000","last_became_active":"2024-01-16T15:11:21.387022+0000","last_became_peered":"2024-01-16T15:11:21.387022+0000","last_unstale":"2024-01-16T15:11:23.337971+0000","last_undegraded":"2024-01-16T15:11:23.337971+0000","last_fullsized":"2024-01-16T15:11:23.337971+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:08.286858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.337918+0000","last_change":"2024-01-16T15:11:21.388167+0000","last_active":"2024-01-16T15:11:23.337918+0000","last_peered":"2024-01-16T15:11:23.337918+0000","last_clean":"2024-01-16T15:11:23.337918+0000","last_became_active":"2024-01-16T15:11:21.387765+0000","last_became_peered":"2024-01-16T15:11:21.387765+0000","last_unstale":"2024-01-16T15:11:23.337918+0000","last_undegraded":"2024-01-16T15:11:23.337918+0000","last_fullsized":"2024-01-16T15:11:23.337918+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:11.683031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366178+0000","last_change":"2024-01-16T15:11:21.391683+0000","last_active":"2024-01-16T15:11:22.366178+0000","last_peered":"2024-01-16T15:11:22.366178+0000","last_clean":"2024-01-16T15:11:22.366178+0000","last_became_active":"2024-01-16T15:11:21.391197+0000","last_became_peered":"2024-01-16T15:11:21.391197+0000","last_unstale":"2024-01-16T15:11:22.366178+0000","last_undegraded":"2024-01-16T15:11:22.366178+0000","last_fullsized":"2024-01-16T15:11:22.366178+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:11.683031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336711+0000","last_change":"2024-01-16T15:11:21.393105+0000","last_active":"2024-01-16T15:11:23.336711+0000","last_peered":"2024-01-16T15:11:23.336711+0000","last_clean":"2024-01-16T15:11:23.336711+0000","last_became_active":"2024-01-16T15:11:21.392785+0000","last_became_peered":"2024-01-16T15:11:21.392785+0000","last_unstale":"2024-01-16T15:11:23.336711+0000","last_undegraded":"2024-01-16T15:11:23.336711+0000","last_fullsized":"2024-01-16T15:11:23.336711+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:30:32.573661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.339248+0000","last_change":"2024-01-16T15:11:21.386339+0000","last_active":"2024-01-16T15:11:23.339248+0000","last_peered":"2024-01-16T15:11:23.339248+0000","last_clean":"2024-01-16T15:11:23.339248+0000","last_became_active":"2024-01-16T15:11:21.386056+0000","last_became_peered":"2024-01-16T15:11:21.386056+0000","last_unstale":"2024-01-16T15:11:23.339248+0000","last_undegraded":"2024-01-16T15:11:23.339248+0000","last_fullsized":"2024-01-16T15:11:23.339248+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:52.655180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366082+0000","last_change":"2024-01-16T15:11:21.390053+0000","last_active":"2024-01-16T15:11:22.366082+0000","last_peered":"2024-01-16T15:11:22.366082+0000","last_clean":"2024-01-16T15:11:22.366082+0000","last_became_active":"2024-01-16T15:11:21.389763+0000","last_became_peered":"2024-01-16T15:11:21.389763+0000","last_unstale":"2024-01-16T15:11:22.366082+0000","last_undegraded":"2024-01-16T15:11:22.366082+0000","last_fullsized":"2024-01-16T15:11:22.366082+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:21.158872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336391+0000","last_change":"2024-01-16T15:11:21.396321+0000","last_active":"2024-01-16T15:11:23.336391+0000","last_peered":"2024-01-16T15:11:23.336391+0000","last_clean":"2024-01-16T15:11:23.336391+0000","last_became_active":"2024-01-16T15:11:21.396009+0000","last_became_peered":"2024-01-16T15:11:21.396009+0000","last_unstale":"2024-01-16T15:11:23.336391+0000","last_undegraded":"2024-01-16T15:11:23.336391+0000","last_fullsized":"2024-01-16T15:11:23.336391+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:46.719660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338550+0000","last_change":"2024-01-16T15:11:21.388132+0000","last_active":"2024-01-16T15:11:23.338550+0000","last_peered":"2024-01-16T15:11:23.338550+0000","last_clean":"2024-01-16T15:11:23.338550+0000","last_became_active":"2024-01-16T15:11:21.387825+0000","last_became_peered":"2024-01-16T15:11:21.387825+0000","last_unstale":"2024-01-16T15:11:23.338550+0000","last_undegraded":"2024-01-16T15:11:23.338550+0000","last_fullsized":"2024-01-16T15:11:23.338550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:05.575127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363697+0000","last_change":"2024-01-16T15:11:21.392024+0000","last_active":"2024-01-16T15:11:22.363697+0000","last_peered":"2024-01-16T15:11:22.363697+0000","last_clean":"2024-01-16T15:11:22.363697+0000","last_became_active":"2024-01-16T15:11:21.391756+0000","last_became_peered":"2024-01-16T15:11:21.391756+0000","last_unstale":"2024-01-16T15:11:22.363697+0000","last_undegraded":"2024-01-16T15:11:22.363697+0000","last_fullsized":"2024-01-16T15:11:22.363697+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:39:32.237685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336645+0000","last_change":"2024-01-16T15:11:21.392043+0000","last_active":"2024-01-16T15:11:23.336645+0000","last_peered":"2024-01-16T15:11:23.336645+0000","last_clean":"2024-01-16T15:11:23.336645+0000","last_became_active":"2024-01-16T15:11:21.391763+0000","last_became_peered":"2024-01-16T15:11:21.391763+0000","last_unstale":"2024-01-16T15:11:23.336645+0000","last_undegraded":"2024-01-16T15:11:23.336645+0000","last_fullsized":"2024-01-16T15:11:23.336645+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336786+0000","last_change":"2024-01-16T15:11:21.394697+0000","last_active":"2024-01-16T15:11:23.336786+0000","last_peered":"2024-01-16T15:11:23.336786+0000","last_clean":"2024-01-16T15:11:23.336786+0000","last_became_active":"2024-01-16T15:11:21.392463+0000","last_became_peered":"2024-01-16T15:11:21.392463+0000","last_unstale":"2024-01-16T15:11:23.336786+0000","last_undegraded":"2024-01-16T15:11:23.336786+0000","last_fullsized":"2024-01-16T15:11:23.336786+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:46.348286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364953+0000","last_change":"2024-01-16T15:11:21.394428+0000","last_active":"2024-01-16T15:11:22.364953+0000","last_peered":"2024-01-16T15:11:22.364953+0000","last_clean":"2024-01-16T15:11:22.364953+0000","last_became_active":"2024-01-16T15:11:21.389734+0000","last_became_peered":"2024-01-16T15:11:21.389734+0000","last_unstale":"2024-01-16T15:11:22.364953+0000","last_undegraded":"2024-01-16T15:11:22.364953+0000","last_fullsized":"2024-01-16T15:11:22.364953+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:58.541508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"99'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.636567+0000","last_change":"2024-01-16T15:11:20.646531+0000","last_active":"2024-01-16T15:11:21.636567+0000","last_peered":"2024-01-16T15:11:21.636567+0000","last_clean":"2024-01-16T15:11:21.636567+0000","last_became_active":"2024-01-16T15:11:20.646251+0000","last_became_peered":"2024-01-16T15:11:20.646251+0000","last_unstale":"2024-01-16T15:11:21.636567+0000","last_undegraded":"2024-01-16T15:11:21.636567+0000","last_fullsized":"2024-01-16T15:11:21.636567+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:14.587574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338912+0000","last_change":"2024-01-16T15:11:21.391759+0000","last_active":"2024-01-16T15:11:23.338912+0000","last_peered":"2024-01-16T15:11:23.338912+0000","last_clean":"2024-01-16T15:11:23.338912+0000","last_became_active":"2024-01-16T15:11:21.391002+0000","last_became_peered":"2024-01-16T15:11:21.391002+0000","last_unstale":"2024-01-16T15:11:23.338912+0000","last_undegraded":"2024-01-16T15:11:23.338912+0000","last_fullsized":"2024-01-16T15:11:23.338912+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:44.386586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366175+0000","last_change":"2024-01-16T15:11:20.644663+0000","last_active":"2024-01-16T15:11:22.366175+0000","last_peered":"2024-01-16T15:11:22.366175+0000","last_clean":"2024-01-16T15:11:22.366175+0000","last_became_active":"2024-01-16T15:11:20.644392+0000","last_became_peered":"2024-01-16T15:11:20.644392+0000","last_unstale":"2024-01-16T15:11:22.366175+0000","last_undegraded":"2024-01-16T15:11:22.366175+0000","last_fullsized":"2024-01-16T15:11:22.366175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:48.207643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336053+0000","last_change":"2024-01-16T15:11:21.385686+0000","last_active":"2024-01-16T15:11:23.336053+0000","last_peered":"2024-01-16T15:11:23.336053+0000","last_clean":"2024-01-16T15:11:23.336053+0000","last_became_active":"2024-01-16T15:11:21.383675+0000","last_became_peered":"2024-01-16T15:11:21.383675+0000","last_unstale":"2024-01-16T15:11:23.336053+0000","last_undegraded":"2024-01-16T15:11:23.336053+0000","last_fullsized":"2024-01-16T15:11:23.336053+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:07.590472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"99'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.630028+0000","last_change":"2024-01-16T15:11:20.636848+0000","last_active":"2024-01-16T15:11:21.630028+0000","last_peered":"2024-01-16T15:11:21.630028+0000","last_clean":"2024-01-16T15:11:21.630028+0000","last_became_active":"2024-01-16T15:11:20.636520+0000","last_became_peered":"2024-01-16T15:11:20.636520+0000","last_unstale":"2024-01-16T15:11:21.630028+0000","last_undegraded":"2024-01-16T15:11:21.630028+0000","last_fullsized":"2024-01-16T15:11:21.630028+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:08:26.815232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.368196+0000","last_change":"2024-01-16T15:11:21.388900+0000","last_active":"2024-01-16T15:11:22.368196+0000","last_peered":"2024-01-16T15:11:22.368196+0000","last_clean":"2024-01-16T15:11:22.368196+0000","last_became_active":"2024-01-16T15:11:21.388629+0000","last_became_peered":"2024-01-16T15:11:21.388629+0000","last_unstale":"2024-01-16T15:11:22.368196+0000","last_undegraded":"2024-01-16T15:11:22.368196+0000","last_fullsized":"2024-01-16T15:11:22.368196+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:28.464408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"99'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.637303+0000","last_change":"2024-01-16T15:11:20.641230+0000","last_active":"2024-01-16T15:11:21.637303+0000","last_peered":"2024-01-16T15:11:21.637303+0000","last_clean":"2024-01-16T15:11:21.637303+0000","last_became_active":"2024-01-16T15:11:20.640466+0000","last_became_peered":"2024-01-16T15:11:20.640466+0000","last_unstale":"2024-01-16T15:11:21.637303+0000","last_undegraded":"2024-01-16T15:11:21.637303+0000","last_fullsized":"2024-01-16T15:11:21.637303+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:06.524256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.521549+0000","last_change":"2024-01-16T15:11:21.394816+0000","last_active":"2024-01-16T15:11:22.521549+0000","last_peered":"2024-01-16T15:11:22.521549+0000","last_clean":"2024-01-16T15:11:22.521549+0000","last_became_active":"2024-01-16T15:11:21.393455+0000","last_became_peered":"2024-01-16T15:11:21.393455+0000","last_unstale":"2024-01-16T15:11:22.521549+0000","last_undegraded":"2024-01-16T15:11:22.521549+0000","last_fullsized":"2024-01-16T15:11:22.521549+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:36.476097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"99'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.636132+0000","last_change":"2024-01-16T15:11:20.642493+0000","last_active":"2024-01-16T15:11:21.636132+0000","last_peered":"2024-01-16T15:11:21.636132+0000","last_clean":"2024-01-16T15:11:21.636132+0000","last_became_active":"2024-01-16T15:11:20.642215+0000","last_became_peered":"2024-01-16T15:11:20.642215+0000","last_unstale":"2024-01-16T15:11:21.636132+0000","last_undegraded":"2024-01-16T15:11:21.636132+0000","last_fullsized":"2024-01-16T15:11:21.636132+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:07.300060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365102+0000","last_change":"2024-01-16T15:11:21.392486+0000","last_active":"2024-01-16T15:11:22.365102+0000","last_peered":"2024-01-16T15:11:22.365102+0000","last_clean":"2024-01-16T15:11:22.365102+0000","last_became_active":"2024-01-16T15:11:21.392210+0000","last_became_peered":"2024-01-16T15:11:21.392210+0000","last_unstale":"2024-01-16T15:11:22.365102+0000","last_undegraded":"2024-01-16T15:11:22.365102+0000","last_fullsized":"2024-01-16T15:11:22.365102+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:13.164950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"99'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.628908+0000","last_change":"2024-01-16T15:11:20.634943+0000","last_active":"2024-01-16T15:11:21.628908+0000","last_peered":"2024-01-16T15:11:21.628908+0000","last_clean":"2024-01-16T15:11:21.628908+0000","last_became_active":"2024-01-16T15:11:20.634120+0000","last_became_peered":"2024-01-16T15:11:20.634120+0000","last_unstale":"2024-01-16T15:11:21.628908+0000","last_undegraded":"2024-01-16T15:11:21.628908+0000","last_fullsized":"2024-01-16T15:11:21.628908+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:52.134389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336224+0000","last_change":"2024-01-16T15:11:21.387551+0000","last_active":"2024-01-16T15:11:23.336224+0000","last_peered":"2024-01-16T15:11:23.336224+0000","last_clean":"2024-01-16T15:11:23.336224+0000","last_became_active":"2024-01-16T15:11:21.387254+0000","last_became_peered":"2024-01-16T15:11:21.387254+0000","last_unstale":"2024-01-16T15:11:23.336224+0000","last_undegraded":"2024-01-16T15:11:23.336224+0000","last_fullsized":"2024-01-16T15:11:23.336224+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:18.594508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336011+0000","last_change":"2024-01-16T15:11:20.642879+0000","last_active":"2024-01-16T15:11:23.336011+0000","last_peered":"2024-01-16T15:11:23.336011+0000","last_clean":"2024-01-16T15:11:23.336011+0000","last_became_active":"2024-01-16T15:11:20.641480+0000","last_became_peered":"2024-01-16T15:11:20.641480+0000","last_unstale":"2024-01-16T15:11:23.336011+0000","last_undegraded":"2024-01-16T15:11:23.336011+0000","last_fullsized":"2024-01-16T15:11:23.336011+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:35.460737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366327+0000","last_change":"2024-01-16T15:11:21.383516+0000","last_active":"2024-01-16T15:11:22.366327+0000","last_peered":"2024-01-16T15:11:22.366327+0000","last_clean":"2024-01-16T15:11:22.366327+0000","last_became_active":"2024-01-16T15:11:21.383118+0000","last_became_peered":"2024-01-16T15:11:21.383118+0000","last_unstale":"2024-01-16T15:11:22.366327+0000","last_undegraded":"2024-01-16T15:11:22.366327+0000","last_fullsized":"2024-01-16T15:11:22.366327+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:49.223003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.384859+0000","last_change":"2024-01-16T15:11:20.633112+0000","last_active":"2024-01-16T15:11:21.384859+0000","last_peered":"2024-01-16T15:11:21.384859+0000","last_clean":"2024-01-16T15:11:21.384859+0000","last_became_active":"2024-01-16T15:11:20.632637+0000","last_became_peered":"2024-01-16T15:11:20.632637+0000","last_unstale":"2024-01-16T15:11:21.384859+0000","last_undegraded":"2024-01-16T15:11:21.384859+0000","last_fullsized":"2024-01-16T15:11:21.384859+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:37.333862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.339424+0000","last_change":"2024-01-16T15:11:21.394229+0000","last_active":"2024-01-16T15:11:23.339424+0000","last_peered":"2024-01-16T15:11:23.339424+0000","last_clean":"2024-01-16T15:11:23.339424+0000","last_became_active":"2024-01-16T15:11:21.393967+0000","last_became_peered":"2024-01-16T15:11:21.393967+0000","last_unstale":"2024-01-16T15:11:23.339424+0000","last_undegraded":"2024-01-16T15:11:23.339424+0000","last_fullsized":"2024-01-16T15:11:23.339424+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:39:53.254044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.385127+0000","last_change":"2024-01-16T15:11:20.642701+0000","last_active":"2024-01-16T15:11:21.385127+0000","last_peered":"2024-01-16T15:11:21.385127+0000","last_clean":"2024-01-16T15:11:21.385127+0000","last_became_active":"2024-01-16T15:11:20.642414+0000","last_became_peered":"2024-01-16T15:11:20.642414+0000","last_unstale":"2024-01-16T15:11:21.385127+0000","last_undegraded":"2024-01-16T15:11:21.385127+0000","last_fullsized":"2024-01-16T15:11:21.385127+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:07.832971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367776+0000","last_change":"2024-01-16T15:11:21.389983+0000","last_active":"2024-01-16T15:11:22.367776+0000","last_peered":"2024-01-16T15:11:22.367776+0000","last_clean":"2024-01-16T15:11:22.367776+0000","last_became_active":"2024-01-16T15:11:21.389153+0000","last_became_peered":"2024-01-16T15:11:21.389153+0000","last_unstale":"2024-01-16T15:11:22.367776+0000","last_undegraded":"2024-01-16T15:11:22.367776+0000","last_fullsized":"2024-01-16T15:11:22.367776+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338292+0000","last_change":"2024-01-16T15:11:20.635955+0000","last_active":"2024-01-16T15:11:23.338292+0000","last_peered":"2024-01-16T15:11:23.338292+0000","last_clean":"2024-01-16T15:11:23.338292+0000","last_became_active":"2024-01-16T15:11:20.635672+0000","last_became_peered":"2024-01-16T15:11:20.635672+0000","last_unstale":"2024-01-16T15:11:23.338292+0000","last_undegraded":"2024-01-16T15:11:23.338292+0000","last_fullsized":"2024-01-16T15:11:23.338292+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:54.203415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336490+0000","last_change":"2024-01-16T15:11:21.389413+0000","last_active":"2024-01-16T15:11:23.336490+0000","last_peered":"2024-01-16T15:11:23.336490+0000","last_clean":"2024-01-16T15:11:23.336490+0000","last_became_active":"2024-01-16T15:11:21.388992+0000","last_became_peered":"2024-01-16T15:11:21.388992+0000","last_unstale":"2024-01-16T15:11:23.336490+0000","last_undegraded":"2024-01-16T15:11:23.336490+0000","last_fullsized":"2024-01-16T15:11:23.336490+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:55.218776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"99'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366863+0000","last_change":"2024-01-16T15:11:20.643870+0000","last_active":"2024-01-16T15:11:22.366863+0000","last_peered":"2024-01-16T15:11:22.366863+0000","last_clean":"2024-01-16T15:11:22.366863+0000","last_became_active":"2024-01-16T15:11:20.643537+0000","last_became_peered":"2024-01-16T15:11:20.643537+0000","last_unstale":"2024-01-16T15:11:22.366863+0000","last_undegraded":"2024-01-16T15:11:22.366863+0000","last_fullsized":"2024-01-16T15:11:22.366863+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:17.579148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367693+0000","last_change":"2024-01-16T15:11:21.390837+0000","last_active":"2024-01-16T15:11:22.367693+0000","last_peered":"2024-01-16T15:11:22.367693+0000","last_clean":"2024-01-16T15:11:22.367693+0000","last_became_active":"2024-01-16T15:11:21.390506+0000","last_became_peered":"2024-01-16T15:11:21.390506+0000","last_unstale":"2024-01-16T15:11:22.367693+0000","last_undegraded":"2024-01-16T15:11:22.367693+0000","last_fullsized":"2024-01-16T15:11:22.367693+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:05.575127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"99'6","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367893+0000","last_change":"2024-01-16T15:11:20.637123+0000","last_active":"2024-01-16T15:11:22.367893+0000","last_peered":"2024-01-16T15:11:22.367893+0000","last_clean":"2024-01-16T15:11:22.367893+0000","last_became_active":"2024-01-16T15:11:20.636760+0000","last_became_peered":"2024-01-16T15:11:20.636760+0000","last_unstale":"2024-01-16T15:11:22.367893+0000","last_undegraded":"2024-01-16T15:11:22.367893+0000","last_fullsized":"2024-01-16T15:11:22.367893+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:22.455273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336207+0000","last_change":"2024-01-16T15:11:21.378517+0000","last_active":"2024-01-16T15:11:23.336207+0000","last_peered":"2024-01-16T15:11:23.336207+0000","last_clean":"2024-01-16T15:11:23.336207+0000","last_became_active":"2024-01-16T15:11:21.377980+0000","last_became_peered":"2024-01-16T15:11:21.377980+0000","last_unstale":"2024-01-16T15:11:23.336207+0000","last_undegraded":"2024-01-16T15:11:23.336207+0000","last_fullsized":"2024-01-16T15:11:23.336207+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:03.751578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"99'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.637525+0000","last_change":"2024-01-16T15:11:20.639189+0000","last_active":"2024-01-16T15:11:21.637525+0000","last_peered":"2024-01-16T15:11:21.637525+0000","last_clean":"2024-01-16T15:11:21.637525+0000","last_became_active":"2024-01-16T15:11:20.638913+0000","last_became_peered":"2024-01-16T15:11:20.638913+0000","last_unstale":"2024-01-16T15:11:21.637525+0000","last_undegraded":"2024-01-16T15:11:21.637525+0000","last_fullsized":"2024-01-16T15:11:21.637525+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:37.333862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336646+0000","last_change":"2024-01-16T15:11:21.393940+0000","last_active":"2024-01-16T15:11:23.336646+0000","last_peered":"2024-01-16T15:11:23.336646+0000","last_clean":"2024-01-16T15:11:23.336646+0000","last_became_active":"2024-01-16T15:11:21.393345+0000","last_became_peered":"2024-01-16T15:11:21.393345+0000","last_unstale":"2024-01-16T15:11:23.336646+0000","last_undegraded":"2024-01-16T15:11:23.336646+0000","last_fullsized":"2024-01-16T15:11:23.336646+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:39:32.237685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.385826+0000","last_change":"2024-01-16T15:11:20.639698+0000","last_active":"2024-01-16T15:11:21.385826+0000","last_peered":"2024-01-16T15:11:21.385826+0000","last_clean":"2024-01-16T15:11:21.385826+0000","last_became_active":"2024-01-16T15:11:20.639426+0000","last_became_peered":"2024-01-16T15:11:20.639426+0000","last_unstale":"2024-01-16T15:11:21.385826+0000","last_undegraded":"2024-01-16T15:11:21.385826+0000","last_fullsized":"2024-01-16T15:11:21.385826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:45.244645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366978+0000","last_change":"2024-01-16T15:11:21.389222+0000","last_active":"2024-01-16T15:11:22.366978+0000","last_peered":"2024-01-16T15:11:22.366978+0000","last_clean":"2024-01-16T15:11:22.366978+0000","last_became_active":"2024-01-16T15:11:21.387838+0000","last_became_peered":"2024-01-16T15:11:21.387838+0000","last_unstale":"2024-01-16T15:11:22.366978+0000","last_undegraded":"2024-01-16T15:11:22.366978+0000","last_fullsized":"2024-01-16T15:11:22.366978+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:55.218776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366387+0000","last_change":"2024-01-16T15:11:20.640809+0000","last_active":"2024-01-16T15:11:22.366387+0000","last_peered":"2024-01-16T15:11:22.366387+0000","last_clean":"2024-01-16T15:11:22.366387+0000","last_became_active":"2024-01-16T15:11:20.640241+0000","last_became_peered":"2024-01-16T15:11:20.640241+0000","last_unstale":"2024-01-16T15:11:22.366387+0000","last_undegraded":"2024-01-16T15:11:22.366387+0000","last_fullsized":"2024-01-16T15:11:22.366387+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:34.062530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363676+0000","last_change":"2024-01-16T15:11:21.393790+0000","last_active":"2024-01-16T15:11:22.363676+0000","last_peered":"2024-01-16T15:11:22.363676+0000","last_clean":"2024-01-16T15:11:22.363676+0000","last_became_active":"2024-01-16T15:11:21.393524+0000","last_became_peered":"2024-01-16T15:11:21.393524+0000","last_unstale":"2024-01-16T15:11:22.363676+0000","last_undegraded":"2024-01-16T15:11:22.363676+0000","last_fullsized":"2024-01-16T15:11:22.363676+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:27.441060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"99'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.640137+0000","last_change":"2024-01-16T15:11:20.637213+0000","last_active":"2024-01-16T15:11:21.640137+0000","last_peered":"2024-01-16T15:11:21.640137+0000","last_clean":"2024-01-16T15:11:21.640137+0000","last_became_active":"2024-01-16T15:11:20.636937+0000","last_became_peered":"2024-01-16T15:11:20.636937+0000","last_unstale":"2024-01-16T15:11:21.640137+0000","last_undegraded":"2024-01-16T15:11:21.640137+0000","last_fullsized":"2024-01-16T15:11:21.640137+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:45.244645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336785+0000","last_change":"2024-01-16T15:11:21.393901+0000","last_active":"2024-01-16T15:11:23.336785+0000","last_peered":"2024-01-16T15:11:23.336785+0000","last_clean":"2024-01-16T15:11:23.336785+0000","last_became_active":"2024-01-16T15:11:21.393456+0000","last_became_peered":"2024-01-16T15:11:21.393456+0000","last_unstale":"2024-01-16T15:11:23.336785+0000","last_undegraded":"2024-01-16T15:11:23.336785+0000","last_fullsized":"2024-01-16T15:11:23.336785+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:03.751578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367378+0000","last_change":"2024-01-16T15:11:20.636443+0000","last_active":"2024-01-16T15:11:22.367378+0000","last_peered":"2024-01-16T15:11:22.367378+0000","last_clean":"2024-01-16T15:11:22.367378+0000","last_became_active":"2024-01-16T15:11:20.636117+0000","last_became_peered":"2024-01-16T15:11:20.636117+0000","last_unstale":"2024-01-16T15:11:22.367378+0000","last_undegraded":"2024-01-16T15:11:22.367378+0000","last_fullsized":"2024-01-16T15:11:22.367378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:13.933905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.521680+0000","last_change":"2024-01-16T15:11:21.399020+0000","last_active":"2024-01-16T15:11:22.521680+0000","last_peered":"2024-01-16T15:11:22.521680+0000","last_clean":"2024-01-16T15:11:22.521680+0000","last_became_active":"2024-01-16T15:11:21.398341+0000","last_became_peered":"2024-01-16T15:11:21.398341+0000","last_unstale":"2024-01-16T15:11:22.521680+0000","last_undegraded":"2024-01-16T15:11:22.521680+0000","last_fullsized":"2024-01-16T15:11:22.521680+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:08.104121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336258+0000","last_change":"2024-01-16T15:11:20.646913+0000","last_active":"2024-01-16T15:11:23.336258+0000","last_peered":"2024-01-16T15:11:23.336258+0000","last_clean":"2024-01-16T15:11:23.336258+0000","last_became_active":"2024-01-16T15:11:20.646455+0000","last_became_peered":"2024-01-16T15:11:20.646455+0000","last_unstale":"2024-01-16T15:11:23.336258+0000","last_undegraded":"2024-01-16T15:11:23.336258+0000","last_fullsized":"2024-01-16T15:11:23.336258+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:45.810115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.366633+0000","last_change":"2024-01-16T15:11:21.385008+0000","last_active":"2024-01-16T15:11:22.366633+0000","last_peered":"2024-01-16T15:11:22.366633+0000","last_clean":"2024-01-16T15:11:22.366633+0000","last_became_active":"2024-01-16T15:11:21.383441+0000","last_became_peered":"2024-01-16T15:11:21.383441+0000","last_unstale":"2024-01-16T15:11:22.366633+0000","last_undegraded":"2024-01-16T15:11:22.366633+0000","last_fullsized":"2024-01-16T15:11:22.366633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:08.286858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.386163+0000","last_change":"2024-01-16T15:11:20.641693+0000","last_active":"2024-01-16T15:11:21.386163+0000","last_peered":"2024-01-16T15:11:21.386163+0000","last_clean":"2024-01-16T15:11:21.386163+0000","last_became_active":"2024-01-16T15:11:20.640934+0000","last_became_peered":"2024-01-16T15:11:20.640934+0000","last_unstale":"2024-01-16T15:11:21.386163+0000","last_undegraded":"2024-01-16T15:11:21.386163+0000","last_fullsized":"2024-01-16T15:11:21.386163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:12.149590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364024+0000","last_change":"2024-01-16T15:11:21.398685+0000","last_active":"2024-01-16T15:11:22.364024+0000","last_peered":"2024-01-16T15:11:22.364024+0000","last_clean":"2024-01-16T15:11:22.364024+0000","last_became_active":"2024-01-16T15:11:21.397634+0000","last_became_peered":"2024-01-16T15:11:21.397634+0000","last_unstale":"2024-01-16T15:11:22.364024+0000","last_undegraded":"2024-01-16T15:11:22.364024+0000","last_fullsized":"2024-01-16T15:11:22.364024+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:04:06.449937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"99'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.629906+0000","last_change":"2024-01-16T15:11:20.636278+0000","last_active":"2024-01-16T15:11:21.629906+0000","last_peered":"2024-01-16T15:11:21.629906+0000","last_clean":"2024-01-16T15:11:21.629906+0000","last_became_active":"2024-01-16T15:11:20.635962+0000","last_became_peered":"2024-01-16T15:11:20.635962+0000","last_unstale":"2024-01-16T15:11:21.629906+0000","last_undegraded":"2024-01-16T15:11:21.629906+0000","last_fullsized":"2024-01-16T15:11:21.629906+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:02.736217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363893+0000","last_change":"2024-01-16T15:11:21.398193+0000","last_active":"2024-01-16T15:11:22.363893+0000","last_peered":"2024-01-16T15:11:22.363893+0000","last_clean":"2024-01-16T15:11:22.363893+0000","last_became_active":"2024-01-16T15:11:21.397916+0000","last_became_peered":"2024-01-16T15:11:21.397916+0000","last_unstale":"2024-01-16T15:11:22.363893+0000","last_undegraded":"2024-01-16T15:11:22.363893+0000","last_fullsized":"2024-01-16T15:11:22.363893+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:00.865427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.521428+0000","last_change":"2024-01-16T15:11:20.645798+0000","last_active":"2024-01-16T15:11:22.521428+0000","last_peered":"2024-01-16T15:11:22.521428+0000","last_clean":"2024-01-16T15:11:22.521428+0000","last_became_active":"2024-01-16T15:11:20.644421+0000","last_became_peered":"2024-01-16T15:11:20.644421+0000","last_unstale":"2024-01-16T15:11:22.521428+0000","last_undegraded":"2024-01-16T15:11:22.521428+0000","last_fullsized":"2024-01-16T15:11:22.521428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:17.707451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363343+0000","last_change":"2024-01-16T15:11:21.396721+0000","last_active":"2024-01-16T15:11:22.363343+0000","last_peered":"2024-01-16T15:11:22.363343+0000","last_clean":"2024-01-16T15:11:22.363343+0000","last_became_active":"2024-01-16T15:11:21.396314+0000","last_became_peered":"2024-01-16T15:11:21.396314+0000","last_unstale":"2024-01-16T15:11:22.363343+0000","last_undegraded":"2024-01-16T15:11:22.363343+0000","last_fullsized":"2024-01-16T15:11:22.363343+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:08.848331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.521418+0000","last_change":"2024-01-16T15:11:20.644604+0000","last_active":"2024-01-16T15:11:22.521418+0000","last_peered":"2024-01-16T15:11:22.521418+0000","last_clean":"2024-01-16T15:11:22.521418+0000","last_became_active":"2024-01-16T15:11:20.644280+0000","last_became_peered":"2024-01-16T15:11:20.644280+0000","last_unstale":"2024-01-16T15:11:22.521418+0000","last_undegraded":"2024-01-16T15:11:22.521418+0000","last_fullsized":"2024-01-16T15:11:22.521418+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:58.090611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.335579+0000","last_change":"2024-01-16T15:11:21.397688+0000","last_active":"2024-01-16T15:11:23.335579+0000","last_peered":"2024-01-16T15:11:23.335579+0000","last_clean":"2024-01-16T15:11:23.335579+0000","last_became_active":"2024-01-16T15:11:21.397192+0000","last_became_peered":"2024-01-16T15:11:21.397192+0000","last_unstale":"2024-01-16T15:11:23.335579+0000","last_undegraded":"2024-01-16T15:11:23.335579+0000","last_fullsized":"2024-01-16T15:11:23.335579+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:08.104121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.382793+0000","last_change":"2024-01-16T15:11:20.638439+0000","last_active":"2024-01-16T15:11:21.382793+0000","last_peered":"2024-01-16T15:11:21.382793+0000","last_clean":"2024-01-16T15:11:21.382793+0000","last_became_active":"2024-01-16T15:11:20.638135+0000","last_became_peered":"2024-01-16T15:11:20.638135+0000","last_unstale":"2024-01-16T15:11:21.382793+0000","last_undegraded":"2024-01-16T15:11:21.382793+0000","last_fullsized":"2024-01-16T15:11:21.382793+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:09.503892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.339500+0000","last_change":"2024-01-16T15:11:21.389378+0000","last_active":"2024-01-16T15:11:23.339500+0000","last_peered":"2024-01-16T15:11:23.339500+0000","last_clean":"2024-01-16T15:11:23.339500+0000","last_became_active":"2024-01-16T15:11:21.388239+0000","last_became_peered":"2024-01-16T15:11:21.388239+0000","last_unstale":"2024-01-16T15:11:23.339500+0000","last_undegraded":"2024-01-16T15:11:23.339500+0000","last_fullsized":"2024-01-16T15:11:23.339500+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:50.782075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.522003+0000","last_change":"2024-01-16T15:11:20.643583+0000","last_active":"2024-01-16T15:11:22.522003+0000","last_peered":"2024-01-16T15:11:22.522003+0000","last_clean":"2024-01-16T15:11:22.522003+0000","last_became_active":"2024-01-16T15:11:20.643237+0000","last_became_peered":"2024-01-16T15:11:20.643237+0000","last_unstale":"2024-01-16T15:11:22.522003+0000","last_undegraded":"2024-01-16T15:11:22.522003+0000","last_fullsized":"2024-01-16T15:11:22.522003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:41.424691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.338009+0000","last_change":"2024-01-16T15:11:21.398683+0000","last_active":"2024-01-16T15:11:23.338009+0000","last_peered":"2024-01-16T15:11:23.338009+0000","last_clean":"2024-01-16T15:11:23.338009+0000","last_became_active":"2024-01-16T15:11:21.398320+0000","last_became_peered":"2024-01-16T15:11:21.398320+0000","last_unstale":"2024-01-16T15:11:23.338009+0000","last_undegraded":"2024-01-16T15:11:23.338009+0000","last_fullsized":"2024-01-16T15:11:23.338009+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:35.124511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.383575+0000","last_change":"2024-01-16T15:11:20.644170+0000","last_active":"2024-01-16T15:11:21.383575+0000","last_peered":"2024-01-16T15:11:21.383575+0000","last_clean":"2024-01-16T15:11:21.383575+0000","last_became_active":"2024-01-16T15:11:20.643751+0000","last_became_peered":"2024-01-16T15:11:20.643751+0000","last_unstale":"2024-01-16T15:11:21.383575+0000","last_undegraded":"2024-01-16T15:11:21.383575+0000","last_fullsized":"2024-01-16T15:11:21.383575+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:27.449048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336406+0000","last_change":"2024-01-16T15:11:21.393502+0000","last_active":"2024-01-16T15:11:23.336406+0000","last_peered":"2024-01-16T15:11:23.336406+0000","last_clean":"2024-01-16T15:11:23.336406+0000","last_became_active":"2024-01-16T15:11:21.393155+0000","last_became_peered":"2024-01-16T15:11:21.393155+0000","last_unstale":"2024-01-16T15:11:23.336406+0000","last_undegraded":"2024-01-16T15:11:23.336406+0000","last_fullsized":"2024-01-16T15:11:23.336406+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:27.441060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.385397+0000","last_change":"2024-01-16T15:11:20.641251+0000","last_active":"2024-01-16T15:11:21.385397+0000","last_peered":"2024-01-16T15:11:21.385397+0000","last_clean":"2024-01-16T15:11:21.385397+0000","last_became_active":"2024-01-16T15:11:20.640749+0000","last_became_peered":"2024-01-16T15:11:20.640749+0000","last_unstale":"2024-01-16T15:11:21.385397+0000","last_undegraded":"2024-01-16T15:11:21.385397+0000","last_fullsized":"2024-01-16T15:11:21.385397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:43.371226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336106+0000","last_change":"2024-01-16T15:11:21.397816+0000","last_active":"2024-01-16T15:11:23.336106+0000","last_peered":"2024-01-16T15:11:23.336106+0000","last_clean":"2024-01-16T15:11:23.336106+0000","last_became_active":"2024-01-16T15:11:21.395376+0000","last_became_peered":"2024-01-16T15:11:21.395376+0000","last_unstale":"2024-01-16T15:11:23.336106+0000","last_undegraded":"2024-01-16T15:11:23.336106+0000","last_fullsized":"2024-01-16T15:11:23.336106+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:23.470633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336614+0000","last_change":"2024-01-16T15:11:20.639082+0000","last_active":"2024-01-16T15:11:23.336614+0000","last_peered":"2024-01-16T15:11:23.336614+0000","last_clean":"2024-01-16T15:11:23.336614+0000","last_became_active":"2024-01-16T15:11:20.638617+0000","last_became_peered":"2024-01-16T15:11:20.638617+0000","last_unstale":"2024-01-16T15:11:23.336614+0000","last_undegraded":"2024-01-16T15:11:23.336614+0000","last_fullsized":"2024-01-16T15:11:23.336614+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:17.707451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364372+0000","last_change":"2024-01-16T15:11:21.398852+0000","last_active":"2024-01-16T15:11:22.364372+0000","last_peered":"2024-01-16T15:11:22.364372+0000","last_clean":"2024-01-16T15:11:22.364372+0000","last_became_active":"2024-01-16T15:11:21.398412+0000","last_became_peered":"2024-01-16T15:11:21.398412+0000","last_unstale":"2024-01-16T15:11:22.364372+0000","last_undegraded":"2024-01-16T15:11:22.364372+0000","last_fullsized":"2024-01-16T15:11:22.364372+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:57:43.971649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.387188+0000","last_change":"2024-01-16T15:11:20.638229+0000","last_active":"2024-01-16T15:11:21.387188+0000","last_peered":"2024-01-16T15:11:21.387188+0000","last_clean":"2024-01-16T15:11:21.387188+0000","last_became_active":"2024-01-16T15:11:20.637097+0000","last_became_peered":"2024-01-16T15:11:20.637097+0000","last_unstale":"2024-01-16T15:11:21.387188+0000","last_undegraded":"2024-01-16T15:11:21.387188+0000","last_fullsized":"2024-01-16T15:11:21.387188+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:11:32.077432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.367098+0000","last_change":"2024-01-16T15:11:21.392426+0000","last_active":"2024-01-16T15:11:22.367098+0000","last_peered":"2024-01-16T15:11:22.367098+0000","last_clean":"2024-01-16T15:11:22.367098+0000","last_became_active":"2024-01-16T15:11:21.391149+0000","last_became_peered":"2024-01-16T15:11:21.391149+0000","last_unstale":"2024-01-16T15:11:22.367098+0000","last_undegraded":"2024-01-16T15:11:22.367098+0000","last_fullsized":"2024-01-16T15:11:22.367098+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:30:32.573661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.335641+0000","last_change":"2024-01-16T15:11:20.642571+0000","last_active":"2024-01-16T15:11:23.335641+0000","last_peered":"2024-01-16T15:11:23.335641+0000","last_clean":"2024-01-16T15:11:23.335641+0000","last_became_active":"2024-01-16T15:11:20.641588+0000","last_became_peered":"2024-01-16T15:11:20.641588+0000","last_unstale":"2024-01-16T15:11:23.335641+0000","last_undegraded":"2024-01-16T15:11:23.335641+0000","last_fullsized":"2024-01-16T15:11:23.335641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:22.641514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.363516+0000","last_change":"2024-01-16T15:11:21.395073+0000","last_active":"2024-01-16T15:11:22.363516+0000","last_peered":"2024-01-16T15:11:22.363516+0000","last_clean":"2024-01-16T15:11:22.363516+0000","last_became_active":"2024-01-16T15:11:21.394800+0000","last_became_peered":"2024-01-16T15:11:21.394800+0000","last_unstale":"2024-01-16T15:11:22.363516+0000","last_undegraded":"2024-01-16T15:11:22.363516+0000","last_fullsized":"2024-01-16T15:11:22.363516+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:07.590472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.336874+0000","last_change":"2024-01-16T15:11:20.629553+0000","last_active":"2024-01-16T15:11:23.336874+0000","last_peered":"2024-01-16T15:11:23.336874+0000","last_clean":"2024-01-16T15:11:23.336874+0000","last_became_active":"2024-01-16T15:11:20.629295+0000","last_became_peered":"2024-01-16T15:11:20.629295+0000","last_unstale":"2024-01-16T15:11:23.336874+0000","last_undegraded":"2024-01-16T15:11:23.336874+0000","last_fullsized":"2024-01-16T15:11:23.336874+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:34.062530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:23.335827+0000","last_change":"2024-01-16T15:11:21.396023+0000","last_active":"2024-01-16T15:11:23.335827+0000","last_peered":"2024-01-16T15:11:23.335827+0000","last_clean":"2024-01-16T15:11:23.335827+0000","last_became_active":"2024-01-16T15:11:21.395701+0000","last_became_peered":"2024-01-16T15:11:21.395701+0000","last_unstale":"2024-01-16T15:11:23.335827+0000","last_undegraded":"2024-01-16T15:11:23.335827+0000","last_fullsized":"2024-01-16T15:11:23.335827+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:35.077891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.385604+0000","last_change":"2024-01-16T15:11:20.637559+0000","last_active":"2024-01-16T15:11:21.385604+0000","last_peered":"2024-01-16T15:11:21.385604+0000","last_clean":"2024-01-16T15:11:21.385604+0000","last_became_active":"2024-01-16T15:11:20.637063+0000","last_became_peered":"2024-01-16T15:11:20.637063+0000","last_unstale":"2024-01-16T15:11:21.385604+0000","last_undegraded":"2024-01-16T15:11:21.385604+0000","last_fullsized":"2024-01-16T15:11:21.385604+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:07.300060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.364990+0000","last_change":"2024-01-16T15:11:21.395416+0000","last_active":"2024-01-16T15:11:22.364990+0000","last_peered":"2024-01-16T15:11:22.364990+0000","last_clean":"2024-01-16T15:11:22.364990+0000","last_became_active":"2024-01-16T15:11:21.395160+0000","last_became_peered":"2024-01-16T15:11:21.395160+0000","last_unstale":"2024-01-16T15:11:22.364990+0000","last_undegraded":"2024-01-16T15:11:22.364990+0000","last_fullsized":"2024-01-16T15:11:22.364990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:20.595916+0000","last_clean_scrub_stamp":"2024-01-16T15:11:20.595916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:30.465816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:11:22.365158+0000","last_change":"2024-01-16T15:11:20.644061+0000","last_active":"2024-01-16T15:11:22.365158+0000","last_peered":"2024-01-16T15:11:22.365158+0000","last_clean":"2024-01-16T15:11:22.365158+0000","last_became_active":"2024-01-16T15:11:20.643678+0000","last_became_peered":"2024-01-16T15:11:20.643678+0000","last_unstale":"2024-01-16T15:11:22.365158+0000","last_undegraded":"2024-01-16T15:11:22.365158+0000","last_fullsized":"2024-01-16T15:11:22.365158+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:19.580556+0000","last_clean_scrub_stamp":"2024-01-16T15:11:19.580556+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:45.704299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:11:21.373443+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:11:21.373443+0000","last_peered":"2024-01-16T15:11:21.373443+0000","last_clean":"2024-01-16T15:11:21.373443+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:11:21.373443+0000","last_undegraded":"2024-01-16T15:11:21.373443+0000","last_fullsized":"2024-01-16T15:11:21.373443+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64852,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306988,"statfs":{"total":96636764160,"available":96570355712,"internally_reserved":0,"allocated":1839104,"data_stored":1123459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.194,"15min":1.4099999999999999},"min":{"1min":0.73899999999999999,"5min":0.60199999999999998,"15min":0.53400000000000003},"max":{"1min":1.6899999999999999,"5min":6.6369999999999996,"15min":45.296999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1619999999999999,"15min":1.3160000000000001},"min":{"1min":0.67200000000000004,"5min":0.65600000000000003,"15min":0.497},"max":{"1min":1.6180000000000001,"5min":2.8799999999999999,"15min":20.126999999999999},"last":1.3640000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3420000000000001,"15min":1.415},"min":{"1min":0.78700000000000003,"5min":0.56200000000000006,"15min":0.499},"max":{"1min":2.3969999999999998,"5min":7.4809999999999999,"15min":42.723999999999997},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.353,"15min":1.587},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.49099999999999999},"max":{"1min":1.871,"5min":2.5779999999999998,"15min":43.572000000000003},"last":1.2170000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.21,"15min":1.446},"min":{"1min":0.66700000000000004,"5min":0.55900000000000005,"15min":0.52100000000000002},"max":{"1min":1.657,"5min":3.7290000000000001,"15min":35.308},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.339,"15min":1.4790000000000001},"min":{"1min":0.84499999999999997,"5min":0.76100000000000001,"15min":0.503},"max":{"1min":2.0579999999999998,"5min":3.7589999999999999,"15min":23.594000000000001},"last":1.405}]},{"osd":3,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3600000000000001,"15min":1.6120000000000001},"min":{"1min":0.86799999999999999,"5min":0.65900000000000003,"15min":0.51300000000000001},"max":{"1min":1.851,"5min":6.5739999999999998,"15min":53.033999999999999},"last":0.995},{"interface":"front","average":{"1min":1.474,"5min":1.46,"15min":1.617},"min":{"1min":1.069,"5min":0.72799999999999998,"15min":0.60299999999999998},"max":{"1min":1.9139999999999999,"5min":8.8900000000000006,"15min":31.305},"last":1.2529999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.329,"15min":1.341},"min":{"1min":0.86899999999999999,"5min":0.49199999999999999,"15min":0.39600000000000002},"max":{"1min":2.1110000000000002,"5min":2.1819999999999999,"15min":5.1870000000000003},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.323,"15min":1.514},"min":{"1min":0.95899999999999996,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":2.161,"5min":2.9809999999999999,"15min":36.374000000000002},"last":0.85499999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.4339999999999999,"15min":1.474},"min":{"1min":1.036,"5min":0.61799999999999999,"15min":0.41499999999999998},"max":{"1min":2.25,"5min":4.1529999999999996,"15min":9.9399999999999995},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.635,"5min":1.5,"15min":1.4990000000000001},"min":{"1min":0.96499999999999997,"5min":0.54700000000000004,"15min":0.46899999999999997},"max":{"1min":3.1949999999999998,"5min":3.1949999999999998,"15min":7.0170000000000003},"last":1.4790000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.4970000000000001,"15min":1.5640000000000001},"min":{"1min":1.083,"5min":0.59899999999999998,"15min":0.53700000000000003},"max":{"1min":2.0419999999999998,"5min":3.5680000000000001,"15min":15.052},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.615,"15min":1.6319999999999999},"min":{"1min":1.2749999999999999,"5min":0.59399999999999997,"15min":0.49099999999999999},"max":{"1min":2.2090000000000001,"5min":2.7360000000000002,"15min":8.4849999999999994},"last":1.532}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76752,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295088,"statfs":{"total":96636764160,"available":96558170112,"internally_reserved":0,"allocated":1245184,"data_stored":533631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77330886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.363,"15min":1.6399999999999999},"min":{"1min":0.626,"5min":0.61899999999999999,"15min":0.60099999999999998},"max":{"1min":1.6220000000000001,"5min":9.1400000000000006,"15min":71.870999999999995},"last":1.46},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.327,"15min":1.7410000000000001},"min":{"1min":0.71599999999999997,"5min":0.67200000000000004,"15min":0.501},"max":{"1min":2.2000000000000002,"5min":6.4059999999999997,"15min":71.846999999999994},"last":1.3260000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4530000000000001,"15min":1.3300000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.63900000000000001},"max":{"1min":3.4049999999999998,"5min":15.798,"15min":15.798},"last":1.234},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.528,"15min":1.4359999999999999},"min":{"1min":0.748,"5min":0.626,"15min":0.626},"max":{"1min":1.8979999999999999,"5min":24.556999999999999,"15min":24.556999999999999},"last":1.544}]},{"osd":2,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.651,"15min":1.4470000000000001},"min":{"1min":0.93500000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.3029999999999999,"5min":14.981,"15min":14.981},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.728,"15min":1.514},"min":{"1min":0.89900000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.7090000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":1.292}]},{"osd":3,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.367,"15min":1.488},"min":{"1min":1.044,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.1789999999999998,"5min":4.5410000000000004,"15min":14.827},"last":1.409},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.581},"min":{"1min":1.0669999999999999,"5min":0.499,"15min":0.499},"max":{"1min":2.032,"5min":21.890000000000001,"15min":21.890000000000001},"last":1.629}]},{"osd":4,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5169999999999999,"15min":1.4550000000000001},"min":{"1min":0.95899999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.1259999999999999,"5min":10.901,"15min":10.901},"last":1.248},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.3520000000000001,"15min":1.444},"min":{"1min":1.014,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":6.9000000000000004},"last":0.85099999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3360000000000001,"15min":1.415},"min":{"1min":0.85999999999999999,"5min":0.44600000000000001,"15min":0.38500000000000001},"max":{"1min":2.1669999999999998,"5min":2.427,"15min":10.736000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.605,"15min":1.6240000000000001},"min":{"1min":1.3049999999999999,"5min":0.498,"15min":0.498},"max":{"1min":2.2130000000000001,"5min":9.2200000000000006,"15min":9.2200000000000006},"last":1.363}]},{"osd":7,"last update":"Tue Jan 16 15:11:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6669999999999998,"5min":1.776,"15min":1.7},"min":{"1min":1.2649999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":24.675000000000001,"5min":24.675000000000001,"15min":24.675000000000001},"last":1.054},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6619999999999999,"15min":1.7030000000000001},"min":{"1min":1.0840000000000001,"5min":0.54600000000000004,"15min":0.42599999999999999},"max":{"1min":3.9580000000000002,"5min":13.991,"15min":14.494},"last":1.1539999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575030,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54604,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317236,"statfs":{"total":96636764160,"available":96580849664,"internally_reserved":0,"allocated":1241088,"data_stored":533049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.7070000000000001,"15min":1.508},"min":{"1min":0.51600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.615,"5min":25.634,"15min":25.634},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.139,"5min":1.4219999999999999,"15min":1.4830000000000001},"min":{"1min":0.66900000000000004,"5min":0.49299999999999999,"15min":0.48299999999999998},"max":{"1min":1.6220000000000001,"5min":16.263000000000002,"15min":34.959000000000003},"last":1.2030000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.5329999999999999,"15min":1.4159999999999999},"min":{"1min":0.72899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.8460000000000001,"5min":26.378,"15min":26.378},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.3169999999999999,"15min":1.3140000000000001},"min":{"1min":0.61099999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.75,"5min":6.125,"15min":13.332000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.5700000000000001,"15min":1.3979999999999999},"min":{"1min":0.751,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":1.821,"5min":25.495000000000001,"15min":25.495000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.4399999999999999,"15min":1.4630000000000001},"min":{"1min":0.63800000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.012,"5min":6.3689999999999998,"15min":17.030000000000001},"last":1.0900000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.605,"15min":1.653},"min":{"1min":0.70099999999999996,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9379999999999999,"5min":25.82,"15min":25.82},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.419,"5min":1.409,"15min":1.6719999999999999},"min":{"1min":0.71699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8959999999999999,"5min":4.4269999999999996,"15min":22.963999999999999},"last":1.298}]},{"osd":4,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.266,"15min":1.288},"min":{"1min":0.58199999999999996,"5min":0.45500000000000002,"15min":0.378},"max":{"1min":1.964,"5min":6.1660000000000004,"15min":8.6050000000000004},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.7430000000000001,"15min":1.5820000000000001},"min":{"1min":0.67200000000000004,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":2.0339999999999998,"5min":25.724,"15min":25.724},"last":1.534}]},{"osd":6,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.931,"15min":1.629},"min":{"1min":0.56499999999999995,"5min":0.46800000000000003,"15min":0.40100000000000002},"max":{"1min":1.7949999999999999,"5min":42.542999999999999,"15min":42.542999999999999},"last":1.405},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3700000000000001,"15min":1.389},"min":{"1min":0.65100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8500000000000001,"5min":6.093,"15min":10.015000000000001},"last":2.1230000000000002}]},{"osd":7,"last update":"Tue Jan 16 15:11:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.873,"15min":1.694},"min":{"1min":0.626,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.1179999999999999,"5min":26.484000000000002,"15min":26.484000000000002},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.46999999999999997},"max":{"1min":3.2509999999999999,"5min":17.233000000000001,"15min":21.106000000000002},"last":1.4990000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575030,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76768,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295072,"statfs":{"total":96636764160,"available":96558153728,"internally_reserved":0,"allocated":1257472,"data_stored":534715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.54,"15min":1.3899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.44400000000000001},"max":{"1min":17.315999999999999,"5min":17.315999999999999,"15min":17.315999999999999},"last":1.48},{"interface":"front","average":{"1min":2.077,"5min":1.5309999999999999,"15min":1.415},"min":{"1min":0.79700000000000004,"5min":0.60399999999999998,"15min":0.52800000000000002},"max":{"1min":17.279,"5min":17.279,"15min":17.279},"last":1.3899999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.306,"15min":1.4419999999999999},"min":{"1min":0.75700000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.8080000000000001,"5min":6.9450000000000003,"15min":39.823999999999998},"last":1.516},{"interface":"front","average":{"1min":1.349,"5min":1.2190000000000001,"15min":1.389},"min":{"1min":0.80800000000000005,"5min":0.53700000000000003,"15min":0.48399999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":50.582000000000001},"last":1.1639999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3260000000000001,"15min":1.345},"min":{"1min":0.94099999999999995,"5min":0.5,"15min":0.39500000000000002},"max":{"1min":1.6180000000000001,"5min":7.319,"15min":10.984},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.462,"15min":1.403},"min":{"1min":0.94999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.744,"5min":6.8179999999999996,"15min":9.1509999999999998},"last":1.4510000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.393,"15min":1.367},"min":{"1min":1.04,"5min":0.58199999999999996,"15min":0.54000000000000004},"max":{"1min":5.0199999999999996,"5min":8.8979999999999997,"15min":11.971},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.4410000000000001,"15min":1.633},"min":{"1min":0.81399999999999995,"5min":0.61699999999999999,"15min":0.57899999999999996},"max":{"1min":2.0550000000000002,"5min":7.399,"15min":27.626999999999999},"last":1.3100000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.369,"15min":1.377},"min":{"1min":0.88200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.619,"5min":4.2329999999999997,"15min":8.2599999999999998},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.661,"15min":1.518},"min":{"1min":0.98299999999999998,"5min":0.47599999999999998,"15min":0.46200000000000002},"max":{"1min":4.8520000000000003,"5min":17.140999999999998,"15min":17.140999999999998},"last":1.4219999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.46,"15min":1.484},"min":{"1min":0.74299999999999999,"5min":0.48299999999999998,"15min":0.45300000000000001},"max":{"1min":1.7529999999999999,"5min":7.4509999999999996,"15min":7.4509999999999996},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.885,"15min":1.621},"min":{"1min":0.88500000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.885,"5min":25.643000000000001,"15min":25.643000000000001},"last":1.2709999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:10:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.7769999999999999,"15min":1.591},"min":{"1min":1.103,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":4.9539999999999997,"5min":24.992999999999999,"15min":24.992999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.706,"15min":1.758},"min":{"1min":0.92900000000000005,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":2.0230000000000001,"5min":8.8369999999999997,"15min":19.001000000000001},"last":1.216}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64276,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307564,"statfs":{"total":96636764160,"available":96570945536,"internally_reserved":0,"allocated":1253376,"data_stored":534613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98399999999999999,"15min":1.1479999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":1.8700000000000001,"5min":9.6229999999999993,"15min":38.423999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1910000000000001,"15min":1.1950000000000001},"min":{"1min":0.51100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.893,"5min":7.8300000000000001,"15min":20.012},"last":0.83999999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.998,"15min":1.607},"min":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.603,"5min":2.4849999999999999,"15min":40.695999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.23,"15min":1.2},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":2.25,"5min":8.5449999999999999,"15min":9.9689999999999994},"last":0.95199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3049999999999999,"15min":1.542},"min":{"1min":0.56200000000000006,"5min":0.46100000000000002,"15min":0.436},"max":{"1min":2.831,"5min":7.2809999999999997,"15min":59.009},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.212,"15min":1.645},"min":{"1min":0.63600000000000001,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":2.2130000000000001,"5min":4.2279999999999998,"15min":29.773},"last":0.90500000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.474,"15min":1.484},"min":{"1min":0.83799999999999997,"5min":0.68799999999999994,"15min":0.59899999999999998},"max":{"1min":3.5049999999999999,"5min":7.4950000000000001,"15min":21.736000000000001},"last":1.149},{"interface":"front","average":{"1min":1.51,"5min":1.4870000000000001,"15min":1.901},"min":{"1min":0.76000000000000001,"5min":0.622,"15min":0.622},"max":{"1min":2.2919999999999998,"5min":9.1289999999999996,"15min":29.524000000000001},"last":1.27}]},{"osd":5,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.464,"15min":1.494},"min":{"1min":0.83399999999999996,"5min":0.64500000000000002,"15min":0.54800000000000004},"max":{"1min":2.3540000000000001,"5min":8.0739999999999998,"15min":9.8249999999999993},"last":1.21},{"interface":"front","average":{"1min":1.472,"5min":1.5660000000000001,"15min":1.573},"min":{"1min":0.94699999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.3969999999999998,"5min":11.255000000000001,"15min":11.255000000000001},"last":1.46}]},{"osd":6,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5780000000000001,"15min":1.589},"min":{"1min":0.88700000000000001,"5min":0.752,"15min":0.65900000000000003},"max":{"1min":2.9470000000000001,"5min":7.4560000000000004,"15min":20.459},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.6519999999999999,"15min":2.0049999999999999},"min":{"1min":0.86499999999999999,"5min":0.67000000000000004,"15min":0.49399999999999999},"max":{"1min":3.46,"5min":12.064,"15min":29.475000000000001},"last":1.548}]},{"osd":7,"last update":"Tue Jan 16 15:10:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.653,"15min":1.643},"min":{"1min":0.96899999999999997,"5min":0.73099999999999998,"15min":0.55300000000000005},"max":{"1min":3.7850000000000001,"5min":7.5339999999999998,"15min":20.109999999999999},"last":1.603},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.617,"15min":2.0089999999999999},"min":{"1min":0.99199999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.0590000000000002,"5min":10.804,"15min":29.396999999999998},"last":1.405}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64264,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307576,"statfs":{"total":96636764160,"available":96570957824,"internally_reserved":0,"allocated":1241088,"data_stored":533013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.2250000000000001,"15min":1.143},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.5389999999999999,"5min":24.155000000000001,"15min":25.335999999999999},"last":1.744},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0469999999999999,"15min":1.2150000000000001},"min":{"1min":0.60499999999999998,"5min":0.42299999999999999,"15min":0.27800000000000002},"max":{"1min":1.923,"5min":2.952,"15min":25.199000000000002},"last":0.82099999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2569999999999999,"15min":1.224},"min":{"1min":0.70099999999999996,"5min":0.59099999999999997,"15min":0.36199999999999999},"max":{"1min":1.4730000000000001,"5min":16.776,"15min":16.776},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.4330000000000001,"15min":1.4390000000000001},"min":{"1min":0.749,"5min":0.48099999999999998,"15min":0.318},"max":{"1min":1.8240000000000001,"5min":25.186,"15min":25.186},"last":1.2010000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.4590000000000001,"15min":1.2569999999999999},"min":{"1min":0.72299999999999998,"5min":0.48499999999999999,"15min":0.33900000000000002},"max":{"1min":1.482,"5min":37.470999999999997,"15min":37.470999999999997},"last":1.284},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2909999999999999,"15min":1.3520000000000001},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.46500000000000002},"max":{"1min":1.6100000000000001,"5min":6.5549999999999997,"15min":28.045999999999999},"last":1.5449999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0680000000000001,"5min":1.9319999999999999,"15min":1.7789999999999999},"min":{"1min":0.96399999999999997,"5min":0.65200000000000002,"15min":0.47499999999999998},"max":{"1min":16.681999999999999,"5min":25.062000000000001,"15min":25.062000000000001},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.7509999999999999,"15min":1.6279999999999999},"min":{"1min":1.0369999999999999,"5min":0.83599999999999997,"15min":0.502},"max":{"1min":1.788,"5min":37.307000000000002,"15min":37.307000000000002},"last":1.9279999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5109999999999999,"15min":1.554},"min":{"1min":0.86099999999999999,"5min":0.72799999999999998,"15min":0.57299999999999995},"max":{"1min":2.0310000000000001,"5min":6.8849999999999998,"15min":19.327000000000002},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.7989999999999999,"15min":1.786},"min":{"1min":1.048,"5min":0.73699999999999999,"15min":0.52400000000000002},"max":{"1min":4.8659999999999997,"5min":25.141999999999999,"15min":25.141999999999999},"last":1.8720000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.8300000000000001,"15min":1.621},"min":{"1min":1.0600000000000001,"5min":0.99399999999999999,"15min":0.56000000000000005},"max":{"1min":1.9330000000000001,"5min":37.363,"15min":37.363},"last":2.1539999999999999},{"interface":"front","average":{"1min":1.73,"5min":1.6359999999999999,"15min":1.583},"min":{"1min":1.1060000000000001,"5min":0.68899999999999995,"15min":0.58799999999999997},"max":{"1min":6.2599999999999998,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":2.056}]},{"osd":7,"last update":"Tue Jan 16 15:11:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.8169999999999999,"15min":1.845},"min":{"1min":1.0680000000000001,"5min":0.66100000000000003,"15min":0.45400000000000001},"max":{"1min":2.1899999999999999,"5min":25.106999999999999,"15min":25.106999999999999},"last":1.998},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.9299999999999999,"15min":1.708},"min":{"1min":1.1719999999999999,"5min":0.88200000000000001,"15min":0.61499999999999999},"max":{"1min":2.0049999999999999,"5min":37.402999999999999,"15min":37.402999999999999},"last":2.27}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64260,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307580,"statfs":{"total":96636764160,"available":96570961920,"internally_reserved":0,"allocated":1236992,"data_stored":532979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.2050000000000001,"15min":1.226},"min":{"1min":0.51600000000000001,"5min":0.36399999999999999,"15min":0.28799999999999998},"max":{"1min":14.106999999999999,"5min":14.106999999999999,"15min":17.210999999999999},"last":0.754},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1319999999999999,"15min":1.339},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.26700000000000002},"max":{"1min":2.9089999999999998,"5min":5.6059999999999999,"15min":34.252000000000002},"last":1.387}]},{"osd":2,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1559999999999999,"15min":1.3220000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1219999999999999,"5min":6.1189999999999998,"15min":24.709},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.5589999999999999,"15min":1.494},"min":{"1min":0.46999999999999997,"5min":0.436,"15min":0.377},"max":{"1min":14.055,"5min":21.795999999999999,"15min":30.128},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.944,"15min":1.583},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.40300000000000002},"max":{"1min":13.967000000000001,"5min":25.611000000000001,"15min":25.611000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.4490000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":2.75,"5min":4.6699999999999999,"15min":41.972000000000001},"last":1.482}]},{"osd":4,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3460000000000001,"15min":1.4510000000000001},"min":{"1min":0.874,"5min":0.52500000000000002,"15min":0.44600000000000001},"max":{"1min":4.3799999999999999,"5min":4.5739999999999998,"15min":18.004999999999999},"last":1.05},{"interface":"front","average":{"1min":2.0529999999999999,"5min":2.008,"15min":1.724},"min":{"1min":0.74199999999999999,"5min":0.68100000000000005,"15min":0.63700000000000001},"max":{"1min":14.161,"5min":21.738,"15min":21.738},"last":1.1499999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.913,"15min":1.752},"min":{"1min":1.044,"5min":0.58699999999999997,"15min":0.46600000000000003},"max":{"1min":3.8130000000000002,"5min":17.739999999999998,"15min":20.905999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.8480000000000001,"15min":1.762},"min":{"1min":0.98599999999999999,"5min":0.56100000000000005,"15min":0.48799999999999999},"max":{"1min":4.2489999999999997,"5min":16.433,"15min":17.957999999999998},"last":1.331}]},{"osd":6,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.966,"15min":1.772},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":14.212,"5min":21.832999999999998,"15min":21.832999999999998},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.766,"15min":1.655},"min":{"1min":0.94199999999999995,"5min":0.63,"15min":0.59199999999999997},"max":{"1min":3.0990000000000002,"5min":9.4749999999999996,"15min":20.509},"last":1.2370000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.9670000000000001,"15min":1.804},"min":{"1min":1.008,"5min":0.75700000000000001,"15min":0.55600000000000005},"max":{"1min":2.9790000000000001,"5min":19.747,"15min":20.669},"last":1.546},{"interface":"front","average":{"1min":1.516,"5min":1.556,"15min":1.6399999999999999},"min":{"1min":0.82999999999999996,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":3.0390000000000001,"5min":6.2699999999999996,"15min":18.039999999999999},"last":1.4350000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77320,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294520,"statfs":{"total":96636764160,"available":96557588480,"internally_reserved":0,"allocated":1830912,"data_stored":1122910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.98899999999999999,"15min":1.1539999999999999},"min":{"1min":0.44600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4490000000000001,"5min":10.683999999999999,"15min":26.751000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.123,"15min":1.1639999999999999},"min":{"1min":0.60399999999999998,"5min":0.433,"15min":0.35899999999999999},"max":{"1min":1.415,"5min":7.649,"15min":15.750999999999999},"last":0.80600000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.641,"15min":1.5649999999999999},"min":{"1min":0.52600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":35.320999999999998,"15min":38.517000000000003},"last":0.872},{"interface":"front","average":{"1min":1.107,"5min":1.292,"15min":1.2829999999999999},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.38800000000000001},"max":{"1min":1.8320000000000001,"5min":11.720000000000001,"15min":24.402000000000001},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.355,"15min":1.502},"min":{"1min":0.58499999999999996,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2789999999999999,"5min":14.677,"15min":29.808},"last":1.056},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.4450000000000001,"15min":1.6919999999999999},"min":{"1min":0.622,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.823,"5min":14.558,"15min":32.302},"last":1.359}]},{"osd":4,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.5629999999999999,"15min":1.5669999999999999},"min":{"1min":0.71899999999999997,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2000000000000002,"5min":9.6080000000000005,"15min":15.832000000000001},"last":1.276},{"interface":"front","average":{"1min":1.288,"5min":1.417,"15min":1.5349999999999999},"min":{"1min":0.68300000000000005,"5min":0.64400000000000002,"15min":0.432},"max":{"1min":1.8859999999999999,"5min":6.9820000000000002,"15min":19.062999999999999},"last":1.306}]},{"osd":5,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.462,"15min":1.6699999999999999},"min":{"1min":0.84699999999999998,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.7270000000000001,"5min":4.7949999999999999,"15min":26.800000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.6060000000000001,"15min":1.601},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.3559999999999999,"5min":7.1550000000000002,"15min":15.906000000000001},"last":1.1759999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.427,"15min":1.6639999999999999},"min":{"1min":0.93500000000000005,"5min":0.65700000000000003,"15min":0.60999999999999999},"max":{"1min":2.1739999999999999,"5min":4.7389999999999999,"15min":20.844000000000001},"last":1.575},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.488,"15min":1.6799999999999999},"min":{"1min":0.96699999999999997,"5min":0.73199999999999998,"15min":0.52800000000000002},"max":{"1min":1.9359999999999999,"5min":4.4109999999999996,"15min":26.832000000000001},"last":1.23}]},{"osd":7,"last update":"Tue Jan 16 15:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.742,"15min":1.6870000000000001},"min":{"1min":0.93999999999999995,"5min":0.69799999999999995,"15min":0.59599999999999997},"max":{"1min":2.484,"5min":16.058,"15min":16.058},"last":1.534},{"interface":"front","average":{"1min":1.353,"5min":1.611,"15min":1.8740000000000001},"min":{"1min":0.89100000000000001,"5min":0.70199999999999996,"15min":0.47499999999999998},"max":{"1min":2.0329999999999999,"5min":8.0609999999999999,"15min":29.286999999999999},"last":1.6479999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:11:26.509 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:11:26.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:11:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.796+0000 7f50c0800640 1 Processor -- start 2024-01-16T15:11:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.797+0000 7f50c0800640 1 -- start start 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.798+0000 7f50c0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f50b8070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.798+0000 7f50c0800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b815ba40 con 0x7f50b815e450 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.798+0000 7f50c0800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b815eb10 con 0x7f50b8071230 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.798+0000 7f50c0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81581a0 con 0x7f50b8157dc0 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50be800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f50b815e450 0x7f50b815bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40258/0 (socket says 172.21.15.125:40258) 2024-01-16T15:11:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50be800640 1 -- 172.21.15.125:0/2099254623 learned_addr learned my addr 172.21.15.125:0/2099254623 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50bde00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f50b8070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198831674 0 0) 0x7f50b815ba40 con 0x7f50b815e450 2024-01-16T15:11:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50b815e450 2024-01-16T15:11:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.799+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203941777 0 0) 0x7f5098003520 con 0x7f50b815e450 2024-01-16T15:11:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 msgr2=0x7f50b8070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f50b8070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 >> v1:172.21.15.137:6789/0 conn(0x7f50b8071230 legacy=0x7f50b815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8071860 con 0x7f50b815e450 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f50b00019f0 con 0x7f50b815e450 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 >> v1:172.21.15.125:6789/0 conn(0x7f50b815e450 legacy=0x7f50b815bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50980039a0 0x7f5098003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50980047c0 0x7f5098004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f5098005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81581a0 con 0x7f50980039a0 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509800dc40 con 0x7f50980047c0 2024-01-16T15:11:26.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509800de00 con 0x7f50b8157dc0 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50980039a0 0x7f5098003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50be800640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f5098005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bd400640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50980047c0 0x7f5098004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 -- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 msgr2=0x7f5098005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f5098005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 -- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50980047c0 msgr2=0x7f5098004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50980047c0 0x7f5098004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.800+0000 7f50bde00640 1 -- 172.21.15.125:0/2099254623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098003520 con 0x7f50980039a0 2024-01-16T15:11:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.801+0000 7f50be800640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50b8157dc0 0x7f5098005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50bde00640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50980039a0 0x7f5098003dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009a90 tx=0x7f50980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0ce56cf524ed2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a802c680 con 0x7f50980039a0 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a802c7e0 con 0x7f50980039a0 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50bca00640 1 -- 172.21.15.125:0/2099254623 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a802cb70 con 0x7f50980039a0 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50c0800640 1 -- 172.21.15.125:0/2099254623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50980039a0 msgr2=0x7f5098003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50c0800640 1 --2- 172.21.15.125:0/2099254623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50980039a0 0x7f5098003dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009a90 tx=0x7f50980057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.802+0000 7f50be800640 1 -- 172.21.15.125:0/2099254623 reap_dead start 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.803+0000 7f50c0800640 1 -- 172.21.15.125:0/2099254623 shutdown_connections 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.803+0000 7f50c0800640 1 -- 172.21.15.125:0/2099254623 >> 172.21.15.125:0/2099254623 conn(0x7f50b80b0be0 msgr2=0x7f50b80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.803+0000 7f50c0800640 1 -- 172.21.15.125:0/2099254623 shutdown_connections 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.803+0000 7f50c0800640 1 -- 172.21.15.125:0/2099254623 wait complete. 2024-01-16T15:11:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.804+0000 7f50c0800640 1 Processor -- start 2024-01-16T15:11:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.805+0000 7f50c0800640 1 -- start start 2024-01-16T15:11:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.805+0000 7f50c0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 0x7f50b80cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50c0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50b8153d70 0x7f50b80cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50c0800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50c0800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80ae540 con 0x7f50b80adb80 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50c0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80cdb80 con 0x7f50b8153d70 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50c0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80ce5f0 con 0x7f50980039a0 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57982/0 (socket says 172.21.15.125:57982) 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 -- 172.21.15.125:0/1008607499 learned_addr learned my addr 172.21.15.125:0/1008607499 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50bde00640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 0x7f50b80cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50bd400640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50b8153d70 0x7f50b80cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 -- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 msgr2=0x7f50b80cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 0x7f50b80cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 -- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50b8153d70 msgr2=0x7f50b80cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50b8153d70 0x7f50b80cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50be800640 1 -- 172.21.15.125:0/1008607499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b80ce8d0 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.806+0000 7f50bde00640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 0x7f50b80cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50be800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f50b80cfee0 tx=0x7f50b000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0004d60 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000bb80 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0007110 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b80cebb0 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.807+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b80cf170 con 0x7f50b80adb80 2024-01-16T15:11:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.808+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b80cf390 con 0x7f50b80adb80 2024-01-16T15:11:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.809+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f50b0007270 con 0x7f50b80adb80 2024-01-16T15:11:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.810+0000 7f50b6000640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 0x7f509006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:26.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.811+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50b0088d40 con 0x7f50b80adb80 2024-01-16T15:11:26.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.811+0000 7f50bde00640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 0x7f509006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:26.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.811+0000 7f50bde00640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 0x7f509006e650 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50a8036000 tx=0x7f50a8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:26.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:26.815+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50b0055320 con 0x7f50b80adb80 2024-01-16T15:11:27.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.079+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f50b8067670 con 0x7f50b80adb80 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.084+0000 7f50b6000640 1 -- 172.21.15.125:0/1008607499 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f50b0058940 con 0x7f50b80adb80 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T15:11:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:11:27.061 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 msgr2=0x7f509006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 0x7f509006e650 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50a8036000 tx=0x7f50a8030000 comp rx=0 tx=0).stop 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 msgr2=0x7f50b80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f50b80cfee0 tx=0x7f50b000b4a0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 shutdown_connections 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f509006c190 0x7f509006e650 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50b80adb80 0x7f50b80ae000 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50b8153d70 0x7f50b80cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 --2- 172.21.15.125:0/1008607499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50980039a0 0x7f50b80cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.086+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 >> 172.21.15.125:0/1008607499 conn(0x7f50b80b0be0 msgr2=0x7f50b815d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:27.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.087+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 shutdown_connections 2024-01-16T15:11:27.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.087+0000 7f50c0800640 1 -- 172.21.15.125:0/1008607499 wait complete. 2024-01-16T15:11:27.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:27.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.310+0000 7f1535400640 1 Processor -- start 2024-01-16T15:11:27.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.310+0000 7f1535400640 1 -- start start 2024-01-16T15:11:27.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.311+0000 7f1535400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1530160920 0x7f15300b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1535400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15300cfe70 0x7f1530161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1535400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300d0a10 con 0x7f15300cf9f0 2024-01-16T15:11:27.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1535400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300d1f50 con 0x7f15300cfe70 2024-01-16T15:11:27.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1535400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300cf080 con 0x7f1530160920 2024-01-16T15:11:27.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1534a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15300cfe70 0x7f1530161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:27.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1534a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15300cfe70 0x7f1530161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57992/0 (socket says 172.21.15.125:57992) 2024-01-16T15:11:27.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f1534a00640 1 -- 172.21.15.125:0/792185929 learned_addr learned my addr 172.21.15.125:0/792185929 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:27.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.312+0000 7f152fe00640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1530160920 0x7f15300b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:27.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.313+0000 7f1534a00640 1 -- 172.21.15.125:0/792185929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1530160920 msgr2=0x7f15300b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:27.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.313+0000 7f1534a00640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1530160920 0x7f15300b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.313+0000 7f1534a00640 1 -- 172.21.15.125:0/792185929 >> v1:172.21.15.137:6789/0 conn(0x7f15300cf9f0 legacy=0x7f1530161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.313+0000 7f1534a00640 1 -- 172.21.15.125:0/792185929 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300c6be0 con 0x7f15300cfe70 2024-01-16T15:11:27.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.314+0000 7f1534a00640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f1530161720 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f1528009e50 tx=0x7f152802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b43ee2ea4875165 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:27.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.315+0000 7f152ea00640 1 -- 172.21.15.125:0/792185929 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152802cb00 con 0x7f15300cfe70 2024-01-16T15:11:27.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.315+0000 7f152ea00640 1 -- 172.21.15.125:0/792185929 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152802cc60 con 0x7f15300cfe70 2024-01-16T15:11:27.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.315+0000 7f152ea00640 1 -- 172.21.15.125:0/792185929 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280343f0 con 0x7f15300cfe70 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.315+0000 7f1535400640 1 -- 172.21.15.125:0/792185929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 msgr2=0x7f1530161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.315+0000 7f1535400640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f1530161720 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f1528009e50 tx=0x7f152802bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 -- 172.21.15.125:0/792185929 shutdown_connections 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f1530161720 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 --2- 172.21.15.125:0/792185929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1530160920 0x7f15300b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 -- 172.21.15.125:0/792185929 >> 172.21.15.125:0/792185929 conn(0x7f15300d3090 msgr2=0x7f15300d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:27.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 -- 172.21.15.125:0/792185929 shutdown_connections 2024-01-16T15:11:27.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.316+0000 7f1535400640 1 -- 172.21.15.125:0/792185929 wait complete. 2024-01-16T15:11:27.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.318+0000 7f1535400640 1 Processor -- start 2024-01-16T15:11:27.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.318+0000 7f1535400640 1 -- start start 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15300cf9f0 0x7f15300ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1530160920 0x7f153006b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301f6950 con 0x7f15300cfe70 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300b8870 con 0x7f1530160920 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1535400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300b8b50 con 0x7f15300cf9f0 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f152f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f152f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57994/0 (socket says 172.21.15.125:57994) 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f152f400640 1 -- 172.21.15.125:0/3311295679 learned_addr learned my addr 172.21.15.125:0/3311295679 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.319+0000 7f1534a00640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1530160920 0x7f153006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:27.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 -- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15300cf9f0 msgr2=0x7f15300ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15300cf9f0 0x7f15300ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 -- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1530160920 msgr2=0x7f153006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1530160920 0x7f153006b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 -- 172.21.15.125:0/3311295679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1528009b30 con 0x7f15300cfe70 2024-01-16T15:11:27.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f152f400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f151800c400 tx=0x7f151800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15180052d0 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518005430 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15300b8e30 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518003830 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.320+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15300b84d0 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.321+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530067450 con 0x7f15300cfe70 2024-01-16T15:11:27.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.322+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f1518003990 con 0x7f15300cfe70 2024-01-16T15:11:27.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.324+0000 7f1527e00640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 0x7f150806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:27.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.324+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1518087f10 con 0x7f15300cfe70 2024-01-16T15:11:27.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.324+0000 7f152fe00640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 0x7f150806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:27.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.325+0000 7f152fe00640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 0x7f150806e620 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f15200098f0 tx=0x7f1520008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:27.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.328+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15180544f0 con 0x7f15300cfe70 2024-01-16T15:11:27.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.549+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15301f6a90 con 0x7f15300cfe70 2024-01-16T15:11:27.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.550+0000 7f1527e00640 1 -- 172.21.15.125:0/3311295679 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f1518017090 con 0x7f15300cfe70 2024-01-16T15:11:27.527 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:11:27.527 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6460,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2589810407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2589810407},{"type":"v1","addr":"172.21.15.137:6834","nonce":2589810407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6463,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1674315039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":1674315039},{"type":"v1","addr":"172.21.15.137:6835","nonce":1674315039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":8646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/995017134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":995017134},{"type":"v1","addr":"172.21.15.125:6836","nonce":995017134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:11:21.075580+0000","modified":"2024-01-16T15:11:24.121417+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/737412716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":737412716},{"type":"v1","addr":"172.21.15.125:6837","nonce":737412716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:11:27.527 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 63 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.552+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 msgr2=0x7f150806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.552+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 0x7f150806e620 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f15200098f0 tx=0x7f1520008040 comp rx=0 tx=0).stop 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.552+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 msgr2=0x7f15300b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.552+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f151800c400 tx=0x7f151800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 shutdown_connections 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1530160920 0x7f153006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f150806c160 0x7f150806e620 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15300cfe70 0x7f15300b7f90 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 --2- 172.21.15.125:0/3311295679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f15300cf9f0 0x7f15300ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 >> 172.21.15.125:0/3311295679 conn(0x7f15300d3090 msgr2=0x7f15300c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:27.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 shutdown_connections 2024-01-16T15:11:27.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:27.553+0000 7f1535400640 1 -- 172.21.15.125:0/3311295679 wait complete. 2024-01-16T15:11:27.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:11:21.075580+0000', 'modified': '2024-01-16T15:11:24.121417+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8649}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8649': {'gid': 8649, 'name': 'a', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/737412716', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6835', 'nonce': 737412716}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 737412716}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:11:27.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:11:27.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T15:11:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.192+0000 7fcf9d000640 1 Processor -- start 2024-01-16T15:11:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.193+0000 7fcf9d000640 1 -- start start 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.194+0000 7fcf9d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf9813df80 0x7fcf9815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf9d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9806c470 con 0x7fcf980cf9a0 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf9d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9806c750 con 0x7fcf9813e360 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf9d000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9806ca30 con 0x7fcf9813df80 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf96800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcf980cf9a0 0x7fcf980cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40272/0 (socket says 172.21.15.125:40272) 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf95e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf9813df80 0x7fcf9815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf96800640 1 -- 172.21.15.125:0/3715040637 learned_addr learned my addr 172.21.15.125:0/3715040637 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf95e00640 1 -- 172.21.15.125:0/3715040637 >> v1:172.21.15.137:6789/0 conn(0x7fcf9813e360 legacy=0x7fcf980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf95e00640 1 -- 172.21.15.125:0/3715040637 >> v1:172.21.15.125:6789/0 conn(0x7fcf980cf9a0 legacy=0x7fcf980cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf95e00640 1 -- 172.21.15.125:0/3715040637 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9806cd10 con 0x7fcf9813df80 2024-01-16T15:11:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.195+0000 7fcf94a00640 1 -- 172.21.15.125:0/3715040637 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372996206 0 0) 0x7fcf9806c470 con 0x7fcf980cf9a0 2024-01-16T15:11:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf95e00640 1 --2- 172.21.15.125:0/3715040637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf9815bee0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf8802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d8ecb991db286f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf94a00640 1 -- 172.21.15.125:0/3715040637 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8802cc60 con 0x7fcf9813df80 2024-01-16T15:11:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf94a00640 1 -- 172.21.15.125:0/3715040637 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8802cdc0 con 0x7fcf9813df80 2024-01-16T15:11:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf94a00640 1 -- 172.21.15.125:0/3715040637 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88034660 con 0x7fcf9813df80 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf9d000640 1 -- 172.21.15.125:0/3715040637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 msgr2=0x7fcf9815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.197+0000 7fcf9d000640 1 --2- 172.21.15.125:0/3715040637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf9815bee0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf8802beb0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.198+0000 7fcf9d000640 1 -- 172.21.15.125:0/3715040637 shutdown_connections 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.198+0000 7fcf9d000640 1 --2- 172.21.15.125:0/3715040637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf9815bee0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.198+0000 7fcf9d000640 1 -- 172.21.15.125:0/3715040637 >> 172.21.15.125:0/3715040637 conn(0x7fcf9806a800 msgr2=0x7fcf9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.198+0000 7fcf9d000640 1 -- 172.21.15.125:0/3715040637 shutdown_connections 2024-01-16T15:11:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.198+0000 7fcf9d000640 1 -- 172.21.15.125:0/3715040637 wait complete. 2024-01-16T15:11:28.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.199+0000 7fcf9d000640 1 Processor -- start 2024-01-16T15:11:28.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.200+0000 7fcf9d000640 1 -- start start 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.200+0000 7fcf9d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf980cf9a0 0x7fcf9814ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.200+0000 7fcf9d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf9d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf9813e360 0x7fcf980cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf9d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9814b060 con 0x7fcf9813df80 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf9d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98070990 con 0x7fcf9813e360 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf9d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98070be0 con 0x7fcf980cf9a0 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58020/0 (socket says 172.21.15.125:58020) 2024-01-16T15:11:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 -- 172.21.15.125:0/2832669906 learned_addr learned my addr 172.21.15.125:0/2832669906 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 -- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf980cf9a0 msgr2=0x7fcf9814ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf980cf9a0 0x7fcf9814ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 -- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf9813e360 msgr2=0x7fcf980cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf9813e360 0x7fcf980cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.201+0000 7fcf95400640 1 -- 172.21.15.125:0/2832669906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88009a70 con 0x7fcf9813df80 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf95400640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000c400 tx=0x7fcf8000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf8e000640 1 -- 172.21.15.125:0/2832669906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80011790 con 0x7fcf9813df80 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf8e000640 1 -- 172.21.15.125:0/2832669906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80011930 con 0x7fcf9813df80 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf98070ec0 con 0x7fcf9813df80 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf8e000640 1 -- 172.21.15.125:0/2832669906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80011ce0 con 0x7fcf9813df80 2024-01-16T15:11:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.202+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf98071480 con 0x7fcf9813df80 2024-01-16T15:11:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.204+0000 7fcf8e000640 1 -- 172.21.15.125:0/2832669906 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fcf8001c460 con 0x7fcf9813df80 2024-01-16T15:11:28.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.205+0000 7fcf8e000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 0x7fcf7006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.205+0000 7fcf8e000640 1 -- 172.21.15.125:0/2832669906 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf80088830 con 0x7fcf9813df80 2024-01-16T15:11:28.182 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.205+0000 7fcf95e00640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 0x7fcf7006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.206+0000 7fcf95e00640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 0x7fcf7006e650 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcf88036000 tx=0x7fcf8802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.206+0000 7fcf9fdb5740 1 Processor -- start 2024-01-16T15:11:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.206+0000 7fcf9fdb5740 1 -- start start 2024-01-16T15:11:28.184 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622143ba0 0x5576221480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5576221485e0 0x55762218c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576220a7880 con 0x557622148ea0 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576220b6ae0 con 0x557622143ba0 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576220b41d0 con 0x5576221485e0 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58034/0 (socket says 172.21.15.125:58034) 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.207+0000 7fcf95e00640 1 -- 172.21.15.125:0/2282674486 learned_addr learned my addr 172.21.15.125:0/2282674486 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95400640 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622143ba0 0x5576221480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 -- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5576221485e0 msgr2=0x55762218c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5576221485e0 0x55762218c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 -- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622143ba0 msgr2=0x5576221480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622143ba0 0x5576221480a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 -- 172.21.15.125:0/2282674486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557622077860 con 0x557622148ea0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf95e00640 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fcf8802c4c0 tx=0x7fcf88031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54e44d555dca1ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf86a00640 1 -- 172.21.15.125:0/2282674486 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88034e70 con 0x557622148ea0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf86a00640 1 -- 172.21.15.125:0/2282674486 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8804a410 con 0x557622148ea0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.208+0000 7fcf86a00640 1 -- 172.21.15.125:0/2282674486 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8804a780 con 0x557622148ea0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/2282674486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 msgr2=0x557622149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fcf8802c4c0 tx=0x7fcf88031000 comp rx=0 tx=0).stop 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/2282674486 shutdown_connections 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5576221485e0 0x55762218c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622143ba0 0x5576221480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/2282674486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x557622149280 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/2282674486 >> 172.21.15.125:0/2282674486 conn(0x55762210ac70 msgr2=0x55762218e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/2282674486 shutdown_connections 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.209+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/2282674486 wait complete. 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.210+0000 7fcf9fdb5740 1 -- start start 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.210+0000 7fcf9fdb5740 1 Processor -- start 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x557622193060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622148ea0 0x557622186fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576220afac0 con 0x5576221485e0 2024-01-16T15:11:28.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762204c2f0 con 0x557622148ea0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x557622069c70 con 0x557622143ba0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.211+0000 7fcf9fdb5740 10 client.0 _ll_register_callbacks cb 0x55762206e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58040/0 (socket says 172.21.15.125:58040) 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 -- 172.21.15.125:0/1942095400 learned_addr learned my addr 172.21.15.125:0/1942095400 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95e00640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x557622193060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf96800640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622148ea0 0x557622186fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 -- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 msgr2=0x557622193060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x557622193060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 -- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622148ea0 msgr2=0x557622186fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622148ea0 0x557622186fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 -- 172.21.15.125:0/1942095400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9806cd10 con 0x5576221485e0 2024-01-16T15:11:28.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf95400640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6dd54a69e7882c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf7f400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.212+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800703f0 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf8005cc80 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576220a8aa0 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf80060e30 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 10 client.8775 fetch_fsmap learned FSMap version 63 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x5576220ae140 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5576220a8aa0 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 63) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf80088400 con 0x5576221485e0 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 10 client.8775 fetch_fsmap finished waiting for FSMap version 63 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 10 client.8775 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 20 client.8775 populate_metadata read hostname 'smithi125' 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762216d760 0x55762216db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 4 client.8775 mds_command: new command op to 8649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:11:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.213+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 --> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576220a87c0 con 0x55762216d760 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.214+0000 7fcf95e00640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762216d760 0x55762216db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.214+0000 7fcf95e00640 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762216d760 0x55762216db40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb106a6199320577 server_cookie=87ed5214bb9c8e65 in_seq=0 out_seq=0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.214+0000 7fcf7f400640 10 client.8775 ms_handle_connect on v2:172.21.15.125:6835/737412716 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf7f400640 1 -- 172.21.15.125:0/1942095400 <== mds.0 v2:172.21.15.125:6835/737412716 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5576220a87c0 con 0x55762216d760 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf7f400640 10 client.8775 handle_command_reply: tid=0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 client.8775 shutdown 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 msgr2=0x557622196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80011f00 comp rx=0 tx=0).stop 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 shutdown_connections 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x557622148ea0 0x557622186fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5576221485e0 0x557622196990 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762216d760 0x55762216db40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/1942095400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x557622193060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 >> 172.21.15.125:0/1942095400 conn(0x55762210ac70 msgr2=0x55762218e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf7f400640 0 client.8775 ms_handle_reset on v2:172.21.15.125:6835/737412716 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 shutdown_connections 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.215+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/1942095400 wait complete. 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.216+0000 7fcf9fdb5740 20 client.8775 trim_cache size 0 max 16384 2024-01-16T15:11:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.232+0000 7fcf9fdb5740 1 Processor -- start 2024-01-16T15:11:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.232+0000 7fcf9fdb5740 1 -- start start 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221497c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x557621e12f20 con 0x557622148ea0 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576221b0de0 con 0x5576221485e0 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.233+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55762213f900 con 0x557622143ba0 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58056/0 (socket says 172.21.15.125:58056) 2024-01-16T15:11:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 -- 172.21.15.125:0/4116425515 learned_addr learned my addr 172.21.15.125:0/4116425515 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf95e00640 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 -- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 msgr2=0x5576221497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221497c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 -- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 msgr2=0x557622142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf95400640 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 -- 172.21.15.125:0/4116425515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55762218b4f0 con 0x557622148ea0 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.234+0000 7fcf96800640 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcf90001510 tx=0x7fcf900019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e4d8b3db0ce1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf87400640 1 -- 172.21.15.125:0/4116425515 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900074e0 con 0x557622148ea0 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf87400640 1 -- 172.21.15.125:0/4116425515 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90007640 con 0x557622148ea0 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf87400640 1 -- 172.21.15.125:0/4116425515 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90004a20 con 0x557622148ea0 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/4116425515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 msgr2=0x5576221430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcf90001510 tx=0x7fcf900019d0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/4116425515 shutdown_connections 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/4116425515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/4116425515 >> 172.21.15.125:0/4116425515 conn(0x55762210ac70 msgr2=0x557622188100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/4116425515 shutdown_connections 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.235+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/4116425515 wait complete. 2024-01-16T15:11:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.236+0000 7fcf9fdb5740 1 -- start start 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.236+0000 7fcf9fdb5740 1 Processor -- start 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576221f64c0 con 0x557622148ea0 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576221921d0 con 0x5576221485e0 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576221497c0 con 0x557622143ba0 2024-01-16T15:11:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf9fdb5740 10 client.0 _ll_register_callbacks cb 0x7fcf80003900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf96800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf96800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58066/0 (socket says 172.21.15.125:58066) 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.237+0000 7fcf96800640 1 -- 172.21.15.125:0/519201512 learned_addr learned my addr 172.21.15.125:0/519201512 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf95400640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf95e00640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 -- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 msgr2=0x5576221f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 -- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 msgr2=0x557622142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 -- 172.21.15.125:0/519201512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf900011c0 con 0x557622148ea0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf95e00640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf96800640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000f60 tx=0x7fcf90001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4011fb5fba485f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf7f400640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90020480 con 0x557622148ea0 2024-01-16T15:11:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf900205e0 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.238+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf8804a410 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf900207a0 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 10 client.8781 fetch_fsmap learned FSMap version 63 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x55762204bb80 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8804a410 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 63) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf90020900 con 0x557622148ea0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 10 client.8781 fetch_fsmap finished waiting for FSMap version 63 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 10 client.8781 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 20 client.8781 populate_metadata read hostname 'smithi125' 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762218ec70 0x55762218f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 4 client.8781 mds_command: new command op to 8649 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 --> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557621db06c0 con 0x55762218ec70 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.239+0000 7fcf95e00640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762218ec70 0x55762218f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf95e00640 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762218ec70 0x55762218f050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411c425cc821cd9f server_cookie=c12fc0bd4771c448 in_seq=0 out_seq=0 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf7f400640 10 client.8781 ms_handle_connect on v2:172.21.15.125:6835/737412716 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf7f400640 1 -- 172.21.15.125:0/519201512 <== mds.0 v2:172.21.15.125:6835/737412716 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x557621db06c0 con 0x55762218ec70 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf7f400640 10 client.8781 handle_command_reply: tid=0 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 client.8781 shutdown 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 msgr2=0x5576221430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000f60 tx=0x7fcf90001080 comp rx=0 tx=0).stop 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 shutdown_connections 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x557622148ea0 0x5576221430c0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5576221485e0 0x557622142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.125:6835/737412716,v1:172.21.15.125:6837/737412716] conn(0x55762218ec70 0x55762218f050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 --2- 172.21.15.125:0/519201512 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x557622143ba0 0x5576221f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 >> 172.21.15.125:0/519201512 conn(0x55762210ac70 msgr2=0x557622170700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.240+0000 7fcf7f400640 0 client.8781 ms_handle_reset on v2:172.21.15.125:6835/737412716 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 shutdown_connections 2024-01-16T15:11:28.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9fdb5740 1 -- 172.21.15.125:0/519201512 wait complete. 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9fdb5740 20 client.8781 trim_cache size 0 max 16384 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 8649, 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.6251107979999997, 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 63, 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 100, 2024-01-16T15:11:28.222 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 100, 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 13.081102886 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 msgr2=0x7fcf7006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 0x7fcf7006e650 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcf88036000 tx=0x7fcf8802e000 comp rx=0 tx=0).stop 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 msgr2=0x7fcf98071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000c400 tx=0x7fcf8000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 shutdown_connections 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf9813e360 0x7fcf980cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf7006c190 0x7fcf7006e650 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf9813df80 0x7fcf98071c80 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 --2- 172.21.15.125:0/2832669906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf980cf9a0 0x7fcf9814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 >> 172.21.15.125:0/2832669906 conn(0x7fcf9806a800 msgr2=0x7fcf9815dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.241+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 shutdown_connections 2024-01-16T15:11:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.242+0000 7fcf9d000640 1 -- 172.21.15.125:0/2832669906 wait complete. 2024-01-16T15:11:28.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:11:28.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.460+0000 7fa370600640 1 Processor -- start 2024-01-16T15:11:28.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.460+0000 7fa370600640 1 -- start start 2024-01-16T15:11:28.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.461+0000 7fa370600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3680cf9a0 0x7fa36815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.461+0000 7fa370600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680b1ee0 con 0x7fa36815e1e0 2024-01-16T15:11:28.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.461+0000 7fa370600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680b2780 con 0x7fa368143ae0 2024-01-16T15:11:28.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.461+0000 7fa370600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36815e990 con 0x7fa3680cf9a0 2024-01-16T15:11:28.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3680cf9a0 0x7fa36815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36e600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa36815e1e0 0x7fa3680702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40280/0 (socket says 172.21.15.125:40280) 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36e600640 1 -- 172.21.15.125:0/1362035652 learned_addr learned my addr 172.21.15.125:0/1362035652 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36d200640 1 -- 172.21.15.125:0/1362035652 >> v1:172.21.15.137:6789/0 conn(0x7fa368143ae0 legacy=0x7fa3680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36d200640 1 -- 172.21.15.125:0/1362035652 >> v1:172.21.15.125:6789/0 conn(0x7fa36815e1e0 legacy=0x7fa3680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa36d200640 1 -- 172.21.15.125:0/1362035652 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680b2130 con 0x7fa3680cf9a0 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.462+0000 7fa367e00640 1 -- 172.21.15.125:0/1362035652 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730345343 0 0) 0x7fa3680b1ee0 con 0x7fa36815e1e0 2024-01-16T15:11:28.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.463+0000 7fa36d200640 1 --2- 172.21.15.125:0/1362035652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3680cf9a0 0x7fa36815dca0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa3500097a0 tx=0x7fa35002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43e21d9054b2d83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa367e00640 1 -- 172.21.15.125:0/1362035652 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350031830 con 0x7fa3680cf9a0 2024-01-16T15:11:28.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa367e00640 1 -- 172.21.15.125:0/1362035652 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350031990 con 0x7fa3680cf9a0 2024-01-16T15:11:28.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa367e00640 1 -- 172.21.15.125:0/1362035652 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350031d20 con 0x7fa3680cf9a0 2024-01-16T15:11:28.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 -- 172.21.15.125:0/1362035652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3680cf9a0 msgr2=0x7fa36815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 --2- 172.21.15.125:0/1362035652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3680cf9a0 0x7fa36815dca0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa3500097a0 tx=0x7fa35002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 -- 172.21.15.125:0/1362035652 shutdown_connections 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 --2- 172.21.15.125:0/1362035652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3680cf9a0 0x7fa36815dca0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 -- 172.21.15.125:0/1362035652 >> 172.21.15.125:0/1362035652 conn(0x7fa36806a800 msgr2=0x7fa36806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 -- 172.21.15.125:0/1362035652 shutdown_connections 2024-01-16T15:11:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.464+0000 7fa370600640 1 -- 172.21.15.125:0/1362035652 wait complete. 2024-01-16T15:11:28.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.466+0000 7fa370600640 1 Processor -- start 2024-01-16T15:11:28.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.466+0000 7fa370600640 1 -- start start 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3680cf9a0 0x7fa368141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa36815e1e0 0x7fa36813a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3681419d0 con 0x7fa368143ae0 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36813ace0 con 0x7fa3680cf9a0 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa370600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680c4550 con 0x7fa36815e1e0 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa36d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa36d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58072/0 (socket says 172.21.15.125:58072) 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.467+0000 7fa36d200640 1 -- 172.21.15.125:0/4064232083 learned_addr learned my addr 172.21.15.125:0/4064232083 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 -- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa36815e1e0 msgr2=0x7fa36813a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa36815e1e0 0x7fa36813a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 -- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3680cf9a0 msgr2=0x7fa368141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3680cf9a0 0x7fa368141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 -- 172.21.15.125:0/4064232083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350009400 con 0x7fa368143ae0 2024-01-16T15:11:28.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa36d200640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098c0 tx=0x7fa3500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35003ba70 con 0x7fa368143ae0 2024-01-16T15:11:28.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35003bbf0 con 0x7fa368143ae0 2024-01-16T15:11:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3500453f0 con 0x7fa368143ae0 2024-01-16T15:11:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3680c4830 con 0x7fa368143ae0 2024-01-16T15:11:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.468+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3680c4df0 con 0x7fa368143ae0 2024-01-16T15:11:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.469+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3680675d0 con 0x7fa368143ae0 2024-01-16T15:11:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.470+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa35004e450 con 0x7fa368143ae0 2024-01-16T15:11:28.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.471+0000 7fa365600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 0x7fa34006e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.471+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3500aac30 con 0x7fa368143ae0 2024-01-16T15:11:28.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.472+0000 7fa36dc00640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 0x7fa34006e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.449 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.472+0000 7fa36dc00640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 0x7fa34006e6a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa3580043c0 tx=0x7fa358007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.475+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa350077210 con 0x7fa368143ae0 2024-01-16T15:11:28.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.709+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa368067c00 con 0x7fa368143ae0 2024-01-16T15:11:28.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.710+0000 7fa365600640 1 -- 172.21.15.125:0/4064232083 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa350035020 con 0x7fa368143ae0 2024-01-16T15:11:28.687 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:11:28.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 msgr2=0x7fa34006e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 0x7fa34006e6a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa3580043c0 tx=0x7fa358007d30 comp rx=0 tx=0).stop 2024-01-16T15:11:28.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 msgr2=0x7fa3680c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098c0 tx=0x7fa3500046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 shutdown_connections 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa36815e1e0 0x7fa36813a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa34006c1e0 0x7fa34006e6a0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa368143ae0 0x7fa3680c3e50 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 --2- 172.21.15.125:0/4064232083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3680cf9a0 0x7fa368141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 >> 172.21.15.125:0/4064232083 conn(0x7fa36806a800 msgr2=0x7fa36806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 shutdown_connections 2024-01-16T15:11:28.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:28.713+0000 7fa370600640 1 -- 172.21.15.125:0/4064232083 wait complete. 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:11:28.708 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:28.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:28.708 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:11:28.754 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:11:28.754 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:11:28.754 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:11:28.754 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.754 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 41760sec preferred_lft 41760sec 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:11:28.755 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:11:28.756 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:28.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:28.757 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:11:28.840 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:11:28.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:28.844 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:11:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:11:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:11:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:11:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:11:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:11:28.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:11:28.896 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:11:29.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.071 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:11:29.119 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:11:29.119 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:11:29.120 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:29.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:11:29.207 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:11:29.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:29.207 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:29.258 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:29.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:29.258 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:29.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:11:29.439 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:11:29.494 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:29.495 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:29.496 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:29.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.497 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:11:29.546 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:11:29.547 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:11:29.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:11:29.627 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:11:29.672 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:11:29.673 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:29.674 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:29.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.675 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:11:29.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:11:29.755+0000 7f5f1ae49480 -1 init, newargv = 0x55a0e4835ab0 newargc=15 2024-01-16T15:11:29.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[74204]: starting ceph client 2024-01-16T15:11:29.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[74204]: starting fuse 2024-01-16T15:11:29.758 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:11:29.758 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:11:29.758 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:11:29.759 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:11:29.760 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:11:29.957 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.74204.asok 2024-01-16T15:11:29.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:11:29.975 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74204.asok 2024-01-16T15:11:29.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74204.asok status 2024-01-16T15:11:30.211 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:11:30.211 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:11:30.211 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:11:30.211 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:11:30.211 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "74204", 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 8796, 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:11:30.212 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 8796 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 869555345 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:30.213 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 869555345 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.8796 192.168.0.1:0/869555345", 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/869555345", 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 63, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 100, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:11:30.214 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:11:30.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:30.226 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:11:30.226 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:30.279 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:11:30.279 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:30.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:30.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:30.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:11:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.974+0000 7f931da00640 1 Processor -- start 2024-01-16T15:11:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.974+0000 7f931da00640 1 -- start start 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f931da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f931814b2c0 0x7f93180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f931da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9318071070 0x7f93180cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f931da00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93180cfbf0 con 0x7f93180cf7d0 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f931da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931814b6a0 con 0x7f9318071070 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f931da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9318071690 con 0x7f931814b2c0 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f9317400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9318071070 0x7f93180cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f9317400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9318071070 0x7f93180cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58082/0 (socket says 172.21.15.125:58082) 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.976+0000 7f9317400640 1 -- 172.21.15.125:0/4226479045 learned_addr learned my addr 172.21.15.125:0/4226479045 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.977+0000 7f9316a00640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f931814b2c0 0x7f93180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.977+0000 7f9317400640 1 -- 172.21.15.125:0/4226479045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f931814b2c0 msgr2=0x7f93180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.977+0000 7f9317400640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f931814b2c0 0x7f93180cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.977+0000 7f9317400640 1 -- 172.21.15.125:0/4226479045 >> v1:172.21.15.137:6789/0 conn(0x7f93180cf7d0 legacy=0x7f9318070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.977+0000 7f9317400640 1 -- 172.21.15.125:0/4226479045 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318071970 con 0x7f9318071070 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.978+0000 7f9317400640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9318071070 0x7f93180cc160 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f9310009e50 tx=0x7f931002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f75712e30f4f94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.978+0000 7f9316000640 1 -- 172.21.15.125:0/4226479045 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931002ca70 con 0x7f9318071070 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.978+0000 7f9316000640 1 -- 172.21.15.125:0/4226479045 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931002cbd0 con 0x7f9318071070 2024-01-16T15:11:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.978+0000 7f9316000640 1 -- 172.21.15.125:0/4226479045 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93100343f0 con 0x7f9318071070 2024-01-16T15:11:30.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 -- 172.21.15.125:0/4226479045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9318071070 msgr2=0x7f93180cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9318071070 0x7f93180cc160 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f9310009e50 tx=0x7f931002bed0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 -- 172.21.15.125:0/4226479045 shutdown_connections 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9318071070 0x7f93180cc160 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 --2- 172.21.15.125:0/4226479045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f931814b2c0 0x7f93180cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 -- 172.21.15.125:0/4226479045 >> 172.21.15.125:0/4226479045 conn(0x7f931806a800 msgr2=0x7f931806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 -- 172.21.15.125:0/4226479045 shutdown_connections 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.979+0000 7f931da00640 1 -- 172.21.15.125:0/4226479045 wait complete. 2024-01-16T15:11:30.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.980+0000 7f931da00640 1 Processor -- start 2024-01-16T15:11:30.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.981+0000 7f931da00640 1 -- start start 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.981+0000 7f931da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9318071070 0x7f93181457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f931da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93180cf7d0 0x7f93180b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f931da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f931da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318145360 con 0x7f931814b2c0 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f931da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318144790 con 0x7f9318071070 2024-01-16T15:11:30.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f931da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9318144a30 con 0x7f93180cf7d0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58094/0 (socket says 172.21.15.125:58094) 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 -- 172.21.15.125:0/560058244 learned_addr learned my addr 172.21.15.125:0/560058244 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9316a00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9318071070 0x7f93181457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f930de00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93180cf7d0 0x7f93180b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 -- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93180cf7d0 msgr2=0x7f93180b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93180cf7d0 0x7f93180b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 -- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9318071070 msgr2=0x7f93181457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9318071070 0x7f93181457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.982+0000 7f9317400640 1 -- 172.21.15.125:0/560058244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9310009b30 con 0x7f931814b2c0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f9317400640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f9310009e50 tx=0x7f9310004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310009510 con 0x7f931814b2c0 2024-01-16T15:11:30.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931003e450 con 0x7f931814b2c0 2024-01-16T15:11:30.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931003e800 con 0x7f931814b2c0 2024-01-16T15:11:30.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9318144d10 con 0x7f931814b2c0 2024-01-16T15:11:30.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.983+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93180b0700 con 0x7f931814b2c0 2024-01-16T15:11:30.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.984+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9318145510 con 0x7f931814b2c0 2024-01-16T15:11:30.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.985+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f931003e960 con 0x7f931814b2c0 2024-01-16T15:11:30.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.986+0000 7f930f400640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 0x7f92f006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.986+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93100ab450 con 0x7f931814b2c0 2024-01-16T15:11:30.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.987+0000 7f9316a00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 0x7f92f006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.987+0000 7f9316a00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 0x7f92f006e620 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9308006fd0 tx=0x7f9308008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:30.991+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9318145510 con 0x7f931814b2c0 2024-01-16T15:11:31.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.219+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93180675d0 con 0x7f931814b2c0 2024-01-16T15:11:31.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.219+0000 7f930f400640 1 -- 172.21.15.125:0/560058244 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9310077d10 con 0x7f931814b2c0 2024-01-16T15:11:31.197 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:11:31.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 msgr2=0x7f92f006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 0x7f92f006e620 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9308006fd0 tx=0x7f9308008040 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 msgr2=0x7f931815fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f9310009e50 tx=0x7f9310004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 shutdown_connections 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f92f006c160 0x7f92f006e620 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f931814b2c0 0x7f931815fae0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93180cf7d0 0x7f93180b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 --2- 172.21.15.125:0/560058244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9318071070 0x7f93181457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 >> 172.21.15.125:0/560058244 conn(0x7f931806a800 msgr2=0x7f9318149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 shutdown_connections 2024-01-16T15:11:31.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:31.222+0000 7f931da00640 1 -- 172.21.15.125:0/560058244 wait complete. 2024-01-16T15:11:31.217 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:11:31.217 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:11:31.218 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:31.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.218 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:11:31.277 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41746sec preferred_lft 41746sec 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:11:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:11:31.279 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:31.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.280 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:11:31.366 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:11:31.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.370 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:11:31.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.419 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:11:31.419 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:11:31.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:11:31.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:11:31.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:11:31.420 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:11:31.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.598 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:11:31.646 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:11:31.646 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:11:31.647 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:31.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:11:31.732 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:11:31.732 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:31.732 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:31.783 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:31.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:11:31.922 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:31.974 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:11:31.974 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:11:31.974 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:11:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:31.976 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:31.977 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:31.977 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:31.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.977 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:11:32.027 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:11:32.027 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:11:32.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:11:32.112 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:11:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:32.159 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:32.160 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:32.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:32.161 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:11:32.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:11:32.245+0000 7f4535645480 -1 init, newargv = 0x55a80da02c70 newargc=15 2024-01-16T15:11:32.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46572]: starting ceph client 2024-01-16T15:11:32.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46572]: starting fuse 2024-01-16T15:11:32.245 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:11:32.246 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:11:32.246 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:11:32.247 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:11:32.444 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.46572.asok 2024-01-16T15:11:32.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:11:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:11:32.462 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46572.asok 2024-01-16T15:11:32.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:32.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46572.asok status 2024-01-16T15:11:32.689 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:11:32.689 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:11:32.689 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:11:32.689 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:11:32.689 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "46572", 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5996, 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5996 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:11:32.690 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3857271961 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3857271961 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5996 192.168.0.1:0/3857271961", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/3857271961", 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 63, 2024-01-16T15:11:32.691 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 100, 2024-01-16T15:11:32.692 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:11:32.692 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:11:32.692 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:11:32.692 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:32.704 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:11:32.704 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:32.754 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:11:32.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:32.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:32.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:32.844 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:11:33.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.088+0000 7f089c200640 1 Processor -- start 2024-01-16T15:11:33.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.088+0000 7f089c200640 1 -- start start 2024-01-16T15:11:33.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.090+0000 7f089c200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.090+0000 7f089c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.090+0000 7f089c200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08940cfc30 con 0x7f08940cf7d0 2024-01-16T15:11:33.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.090+0000 7f089c200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08940b1ee0 con 0x7f0894143ae0 2024-01-16T15:11:33.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.090+0000 7f089c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940b2780 con 0x7f089415e3b0 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f0899800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f089a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f089a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:40058/0 (socket says 172.21.15.125:40058) 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f0898e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f08940cf7d0 0x7f089415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:41474/0 (socket says 172.21.15.125:41474) 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f089a200640 1 -- 172.21.15.125:0/2301330190 learned_addr learned my addr 172.21.15.125:0/2301330190 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.091+0000 7f0899800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:50864/0 (socket says 172.21.15.125:50864) 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.092+0000 7f089a200640 1 -- 172.21.15.125:0/2301330190 >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 msgr2=0x7f08940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.092+0000 7f089a200640 1 --2- 172.21.15.125:0/2301330190 >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.092+0000 7f089a200640 1 -- 172.21.15.125:0/2301330190 >> v1:172.21.15.137:6789/0 conn(0x7f08940cf7d0 legacy=0x7f089415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.092+0000 7f089a200640 1 -- 172.21.15.125:0/2301330190 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089415e990 con 0x7f089415e3b0 2024-01-16T15:11:33.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.092+0000 7f0899800640 1 --2- 172.21.15.125:0/2301330190 >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:11:33.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.093+0000 7f089a200640 1 --2- 172.21.15.125:0/2301330190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d150 tx=0x7f088c02e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb047d5adb75a0ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f0893e00640 1 -- 172.21.15.125:0/2301330190 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00b0d0 con 0x7f089415e3b0 2024-01-16T15:11:33.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f0893e00640 1 -- 172.21.15.125:0/2301330190 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00b230 con 0x7f089415e3b0 2024-01-16T15:11:33.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f0893e00640 1 -- 172.21.15.125:0/2301330190 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c009600 con 0x7f089415e3b0 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 -- 172.21.15.125:0/2301330190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 msgr2=0x7f08940702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 --2- 172.21.15.125:0/2301330190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d150 tx=0x7f088c02e160 comp rx=0 tx=0).stop 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 -- 172.21.15.125:0/2301330190 shutdown_connections 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 --2- 172.21.15.125:0/2301330190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f08940702c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 --2- 172.21.15.125:0/2301330190 >> v2:172.21.15.137:3300/0 conn(0x7f0894143ae0 0x7f08940cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 -- 172.21.15.125:0/2301330190 >> 172.21.15.125:0/2301330190 conn(0x7f089406a800 msgr2=0x7f089406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 -- 172.21.15.125:0/2301330190 shutdown_connections 2024-01-16T15:11:33.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.094+0000 7f089c200640 1 -- 172.21.15.125:0/2301330190 wait complete. 2024-01-16T15:11:33.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.096+0000 7f089c200640 1 Processor -- start 2024-01-16T15:11:33.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.096+0000 7f089c200640 1 -- start start 2024-01-16T15:11:33.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 0x7f08940b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f089415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940af2a0 con 0x7f08940cf7d0 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940af550 con 0x7f0894143ae0 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f089c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940af830 con 0x7f089415e3b0 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f0899800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f0899800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58096/0 (socket says 172.21.15.125:58096) 2024-01-16T15:11:33.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.097+0000 7f0899800640 1 -- 172.21.15.125:0/1293691134 learned_addr learned my addr 172.21.15.125:0/1293691134 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:33.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f089a200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f089415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0898e00640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 0x7f08940b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 -- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 msgr2=0x7f089415c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f089415c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 -- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 msgr2=0x7f08940b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 0x7f08940b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 -- 172.21.15.125:0/1293691134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884009400 con 0x7f08940cf7d0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0898e00640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 0x7f08940b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0899800640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f088400ebe0 tx=0x7f088400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884009760 con 0x7f08940cf7d0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088400cc30 con 0x7f08940cf7d0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088400eee0 con 0x7f08940cf7d0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.098+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c00bd40 con 0x7f08940cf7d0 2024-01-16T15:11:33.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.099+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08940aef60 con 0x7f08940cf7d0 2024-01-16T15:11:33.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.100+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08940675d0 con 0x7f08940cf7d0 2024-01-16T15:11:33.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.101+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0884013470 con 0x7f08940cf7d0 2024-01-16T15:11:33.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.102+0000 7f0891600640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 0x7f086c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.102+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08840889a0 con 0x7f08940cf7d0 2024-01-16T15:11:33.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.102+0000 7f0898e00640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 0x7f086c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.103+0000 7f0898e00640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 0x7f086c06e620 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0894142510 tx=0x7f087c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.107+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0884055260 con 0x7f08940cf7d0 2024-01-16T15:11:33.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.338+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08940ae690 con 0x7f08940cf7d0 2024-01-16T15:11:33.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.712+0000 7f0891600640 1 -- 172.21.15.125:0/1293691134 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 136+0+0 (secure 0 0 0) 0x7f0884058880 con 0x7f08940cf7d0 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 msgr2=0x7f086c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 0x7f086c06e620 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0894142510 tx=0x7f087c009040 comp rx=0 tx=0).stop 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 msgr2=0x7f08940aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f088400ebe0 tx=0x7f088400c520 comp rx=0 tx=0).stop 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 shutdown_connections 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f089415e3b0 0x7f089415c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894143ae0 0x7f08940b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f086c06c160 0x7f086c06e620 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 --2- 172.21.15.125:0/1293691134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08940cf7d0 0x7f08940aea20 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.716+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 >> 172.21.15.125:0/1293691134 conn(0x7f089406a800 msgr2=0x7f089406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.717+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 shutdown_connections 2024-01-16T15:11:33.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.717+0000 7f089c200640 1 -- 172.21.15.125:0/1293691134 wait complete. 2024-01-16T15:11:33.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:11:33.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.941+0000 7ff6adc00640 1 Processor -- start 2024-01-16T15:11:33.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.941+0000 7ff6adc00640 1 -- start start 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.942+0000 7ff6adc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 0x7ff6a8150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6adc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff6a815da10 0x7ff6a80b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6adc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a8071190 con 0x7ff6a80cf290 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6adc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a8072610 con 0x7ff6a815da10 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6adc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8071470 con 0x7ff6a8072230 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff69de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff6a80cf290 0x7ff6a815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40296/0 (socket says 172.21.15.125:40296) 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff69de00640 1 -- 172.21.15.125:0/1072143195 learned_addr learned my addr 172.21.15.125:0/1072143195 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:33.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6a7400640 1 --2- 172.21.15.125:0/1072143195 >> v2:172.21.15.137:3300/0 conn(0x7ff6a815da10 0x7ff6a80b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6a6a00640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 0x7ff6a8150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173361522 0 0) 0x7ff6a8071190 con 0x7ff6a80cf290 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.943+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff688003520 con 0x7ff6a80cf290 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479565453 0 0) 0x7ff688003520 con 0x7ff6a80cf290 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 msgr2=0x7ff6a8150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 0x7ff6a8150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 >> v2:172.21.15.137:3300/0 conn(0x7ff6a815da10 msgr2=0x7ff6a80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 --2- 172.21.15.125:0/1072143195 >> v2:172.21.15.137:3300/0 conn(0x7ff6a815da10 0x7ff6a80b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8071750 con 0x7ff6a80cf290 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff690001670 con 0x7ff6a80cf290 2024-01-16T15:11:33.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 >> v1:172.21.15.125:6789/0 conn(0x7ff6a80cf290 legacy=0x7ff6a815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 0x7ff688003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff688003ef0 0x7ff6880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff688005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8072610 con 0x7ff688004d00 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8071470 con 0x7ff688003ef0 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68800b980 con 0x7ff6a8072230 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.944+0000 7ff6a7400640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff688005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.945+0000 7ff6a7400640 1 -- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 msgr2=0x7ff688003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.945+0000 7ff6a7400640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a8072230 0x7ff688003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.945+0000 7ff6a7400640 1 -- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff688003ef0 msgr2=0x7ff6880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.945+0000 7ff6a7400640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff688003ef0 0x7ff6880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.945+0000 7ff6a7400640 1 -- 172.21.15.125:0/1072143195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff688003520 con 0x7ff688004d00 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.946+0000 7ff6a7400640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff688005180 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7ff6a806e1b0 tx=0x7ff6880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35999c42995ded00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.946+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a002cbc0 con 0x7ff688004d00 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.946+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a000f040 con 0x7ff688004d00 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.946+0000 7ff6a6000640 1 -- 172.21.15.125:0/1072143195 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0034440 con 0x7ff688004d00 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 -- 172.21.15.125:0/1072143195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 msgr2=0x7ff688005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 --2- 172.21.15.125:0/1072143195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff688005180 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7ff6a806e1b0 tx=0x7ff6880057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6a7400640 1 -- 172.21.15.125:0/1072143195 reap_dead start 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 -- 172.21.15.125:0/1072143195 shutdown_connections 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 -- 172.21.15.125:0/1072143195 >> 172.21.15.125:0/1072143195 conn(0x7ff6a806a800 msgr2=0x7ff6a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 -- 172.21.15.125:0/1072143195 shutdown_connections 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.947+0000 7ff6adc00640 1 -- 172.21.15.125:0/1072143195 wait complete. 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.948+0000 7ff6adc00640 1 Processor -- start 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 -- start start 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6a8132630 0x7ff6a806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a806d000 0x7ff6a80cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a806dbf0 con 0x7ff688004d00 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8132cb0 con 0x7ff6a8132630 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.949+0000 7ff6adc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a814a4c0 con 0x7ff6a806d000 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58108/0 (socket says 172.21.15.125:58108) 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 -- 172.21.15.125:0/3739057052 learned_addr learned my addr 172.21.15.125:0/3739057052 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 -- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a806d000 msgr2=0x7ff6a80cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a806d000 0x7ff6a80cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 -- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6a8132630 msgr2=0x7ff6a806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6a8132630 0x7ff6a806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 -- 172.21.15.125:0/3739057052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a806d630 con 0x7ff688004d00 2024-01-16T15:11:33.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff6a6a00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7ff69800e7f0 tx=0x7ff69800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.950+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69800eea0 con 0x7ff688004d00 2024-01-16T15:11:33.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.951+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69800c8c0 con 0x7ff688004d00 2024-01-16T15:11:33.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.951+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69800cc50 con 0x7ff688004d00 2024-01-16T15:11:33.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.951+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a806d910 con 0x7ff688004d00 2024-01-16T15:11:33.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.951+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a80ce390 con 0x7ff688004d00 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.952+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a80675d0 con 0x7ff688004d00 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.953+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff698013470 con 0x7ff688004d00 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.954+0000 7ff69f400640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 0x7ff68006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.954+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6980887e0 con 0x7ff688004d00 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.954+0000 7ff69de00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 0x7ff68006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.955+0000 7ff69de00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 0x7ff68006e650 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff690000f60 tx=0x7ff6900015b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:33.958+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff698054f60 con 0x7ff688004d00 2024-01-16T15:11:34.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.171+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6a8071d10 con 0x7ff68006c190 2024-01-16T15:11:34.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.174+0000 7ff69f400640 1 -- 172.21.15.125:0/3739057052 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff6900047e0 con 0x7ff68006c190 2024-01-16T15:11:34.151 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:11:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:11:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:11:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:11:34.152 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:11:34.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 msgr2=0x7ff68006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 0x7ff68006e650 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff690000f60 tx=0x7ff6900015b0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 msgr2=0x7ff6a81320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7ff69800e7f0 tx=0x7ff69800ecb0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 shutdown_connections 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6a806d000 0x7ff6a80cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6a8132630 0x7ff6a806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff68006c190 0x7ff68006e650 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 --2- 172.21.15.125:0/3739057052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff688004d00 0x7ff6a81320f0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 >> 172.21.15.125:0/3739057052 conn(0x7ff6a806a800 msgr2=0x7ff6a81569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 shutdown_connections 2024-01-16T15:11:34.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.177+0000 7ff6adc00640 1 -- 172.21.15.125:0/3739057052 wait complete. 2024-01-16T15:11:34.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:11:34.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.409+0000 7f28c8000640 1 Processor -- start 2024-01-16T15:11:34.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.409+0000 7f28c8000640 1 -- start start 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.410+0000 7f28c8000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f28c00cf9a0 0x7f28c015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.410+0000 7f28c8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c015e1e0 0x7f28c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.410+0000 7f28c8000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c00b1ee0 con 0x7f28c0143ae0 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.410+0000 7f28c8000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c00b2780 con 0x7f28c00cf9a0 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.410+0000 7f28c8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c015e990 con 0x7f28c015e1e0 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f28c00cf9a0 0x7f28c015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c6000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c015e1e0 0x7f28c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f28c00cf9a0 0x7f28c015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:50898/0 (socket says 172.21.15.125:50898) 2024-01-16T15:11:34.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 -- 172.21.15.125:0/1696095239 learned_addr learned my addr 172.21.15.125:0/1696095239 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:34.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 -- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c015e1e0 msgr2=0x7f28c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:34.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 --2- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c015e1e0 0x7f28c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 -- 172.21.15.125:0/1696095239 >> v1:172.21.15.137:6789/0 conn(0x7f28c0143ae0 legacy=0x7f28c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.411+0000 7f28c4c00640 1 -- 172.21.15.125:0/1696095239 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c00b2130 con 0x7f28c00cf9a0 2024-01-16T15:11:34.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.413+0000 7f28c4c00640 1 --2- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c015dca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009e50 tx=0x7f28a802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa107ada97a40e4b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.413+0000 7f28bfe00640 1 -- 172.21.15.125:0/1696095239 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a802cb00 con 0x7f28c00cf9a0 2024-01-16T15:11:34.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28bfe00640 1 -- 172.21.15.125:0/1696095239 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a802cc60 con 0x7f28c00cf9a0 2024-01-16T15:11:34.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28bfe00640 1 -- 172.21.15.125:0/1696095239 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80343f0 con 0x7f28c00cf9a0 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 -- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 msgr2=0x7f28c015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 --2- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c015dca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009e50 tx=0x7f28a802bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 -- 172.21.15.125:0/1696095239 shutdown_connections 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 --2- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c015e1e0 0x7f28c00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 --2- 172.21.15.125:0/1696095239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c015dca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 -- 172.21.15.125:0/1696095239 >> 172.21.15.125:0/1696095239 conn(0x7f28c006a800 msgr2=0x7f28c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.414+0000 7f28c8000640 1 -- 172.21.15.125:0/1696095239 shutdown_connections 2024-01-16T15:11:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.415+0000 7f28c8000640 1 -- 172.21.15.125:0/1696095239 wait complete. 2024-01-16T15:11:34.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.416+0000 7f28c8000640 1 Processor -- start 2024-01-16T15:11:34.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.416+0000 7f28c8000640 1 -- start start 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c01424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c0143ae0 0x7f28c0159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0142a10 con 0x7f28c015e1e0 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0142cd0 con 0x7f28c00cf9a0 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.417+0000 7f28c8000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0142fb0 con 0x7f28c0143ae0 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58114/0 (socket says 172.21.15.125:58114) 2024-01-16T15:11:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 -- 172.21.15.125:0/582485452 learned_addr learned my addr 172.21.15.125:0/582485452 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:34.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 -- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c0143ae0 msgr2=0x7f28c0159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c5600640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c01424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c0143ae0 0x7f28c0159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 -- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 msgr2=0x7f28c01424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c01424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.418+0000 7f28c6000640 1 -- 172.21.15.125:0/582485452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8009b30 con 0x7f28c015e1e0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28c6000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e7f0 tx=0x7f28b800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8009510 con 0x7f28c015e1e0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b800ee40 con 0x7f28c015e1e0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c01f0f60 con 0x7f28c015e1e0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b800cb80 con 0x7f28c015e1e0 2024-01-16T15:11:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.419+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c0141010 con 0x7f28c015e1e0 2024-01-16T15:11:34.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.420+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c00675d0 con 0x7f28c015e1e0 2024-01-16T15:11:34.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.421+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f28b800cce0 con 0x7f28c015e1e0 2024-01-16T15:11:34.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.422+0000 7f28bd600640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 0x7f289806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.422+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28b80887c0 con 0x7f28c015e1e0 2024-01-16T15:11:34.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.422+0000 7f28c5600640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 0x7f289806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.423+0000 7f28c5600640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 0x7f289806e620 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28b0000f60 tx=0x7f28b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.426+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28b8055080 con 0x7f28c015e1e0 2024-01-16T15:11:34.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.620+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f28c00cec50 con 0x7f28c015e1e0 2024-01-16T15:11:34.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.620+0000 7f28bd600640 1 -- 172.21.15.125:0/582485452 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f28b80586a0 con 0x7f28c015e1e0 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 msgr2=0x7f289806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 0x7f289806e620 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28b0000f60 tx=0x7f28b0009040 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 msgr2=0x7f28c015b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e7f0 tx=0x7f28b800ebf0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 shutdown_connections 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f289806c160 0x7f289806e620 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28c015e1e0 0x7f28c015b870 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28c0143ae0 0x7f28c0159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 --2- 172.21.15.125:0/582485452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28c00cf9a0 0x7f28c01424d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.623+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 >> 172.21.15.125:0/582485452 conn(0x7f28c006a800 msgr2=0x7f28c006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.624+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 shutdown_connections 2024-01-16T15:11:34.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.624+0000 7f28c8000640 1 -- 172.21.15.125:0/582485452 wait complete. 2024-01-16T15:11:34.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:11:34.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.845+0000 7f3de9000640 1 Processor -- start 2024-01-16T15:11:34.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.845+0000 7f3de9000640 1 -- start start 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.846+0000 7f3de9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3de4166330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.846+0000 7f3de9000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40d00a0 con 0x7f3de415b2a0 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.846+0000 7f3de9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40b4010 con 0x7f3de40cd960 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.846+0000 7f3de9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de40b42f0 con 0x7f3de4163f40 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de1e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3de415b2a0 0x7f3de40cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40306/0 (socket says 172.21.15.125:40306) 2024-01-16T15:11:34.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de1e00640 1 -- 172.21.15.125:0/2182507637 learned_addr learned my addr 172.21.15.125:0/2182507637 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:34.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015804947 0 0) 0x7f3de40d00a0 con 0x7f3de415b2a0 2024-01-16T15:11:34.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc4003520 con 0x7f3de415b2a0 2024-01-16T15:11:34.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de2800640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3de4166330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.847+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716664117 0 0) 0x7f3dc4003520 con 0x7f3de415b2a0 2024-01-16T15:11:34.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 msgr2=0x7f3de4166330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3de4166330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 >> v1:172.21.15.137:6789/0 conn(0x7f3de40cd960 legacy=0x7f3de40cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40b45d0 con 0x7f3de415b2a0 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3dd4001670 con 0x7f3de415b2a0 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 >> v1:172.21.15.125:6789/0 conn(0x7f3de415b2a0 legacy=0x7f3de40cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3dc4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3dc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3dc4004d00 0x7f3dc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de40b42f0 con 0x7f3dc40039a0 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc400bb10 con 0x7f3dc4004d00 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc400bcd0 con 0x7f3de4163f40 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3dc4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de2800640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3dc4004d00 0x7f3dc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1400640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3dc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 -- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 msgr2=0x7f3dc40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3dc40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 -- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3dc4004d00 msgr2=0x7f3dc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3dc4004d00 0x7f3dc4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1e00640 1 -- 172.21.15.125:0/2182507637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc4003520 con 0x7f3dc40039a0 2024-01-16T15:11:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.848+0000 7f3de1400640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4163f40 0x7f3dc40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:34.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.850+0000 7f3de1e00640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3dc4003dc0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c240 tx=0x7f3dc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9f454bf282b896 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.850+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4005ad0 con 0x7f3dc40039a0 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.850+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd4005c30 con 0x7f3dc40039a0 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.850+0000 7f3de0a00640 1 -- 172.21.15.125:0/2182507637 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd402c8b0 con 0x7f3dc40039a0 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 -- 172.21.15.125:0/2182507637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 msgr2=0x7f3dc4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 --2- 172.21.15.125:0/2182507637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3dc4003dc0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c240 tx=0x7f3dc40057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de2800640 1 -- 172.21.15.125:0/2182507637 reap_dead start 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 -- 172.21.15.125:0/2182507637 shutdown_connections 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 -- 172.21.15.125:0/2182507637 >> 172.21.15.125:0/2182507637 conn(0x7f3de40d1020 msgr2=0x7f3de40d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 -- 172.21.15.125:0/2182507637 shutdown_connections 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.851+0000 7f3de9000640 1 -- 172.21.15.125:0/2182507637 wait complete. 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.852+0000 7f3de9000640 1 Processor -- start 2024-01-16T15:11:34.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.853+0000 7f3de9000640 1 -- start start 2024-01-16T15:11:34.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.853+0000 7f3de9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.853+0000 7f3de9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4158bb0 0x7f3de406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3de40c8990 0x7f3de415dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de9000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de40c8500 con 0x7f3dc40039a0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de406be20 con 0x7f3de40c8990 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de406c100 con 0x7f3de4158bb0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58128/0 (socket says 172.21.15.125:58128) 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 -- 172.21.15.125:0/4255646135 learned_addr learned my addr 172.21.15.125:0/4255646135 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 -- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4158bb0 msgr2=0x7f3de406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4158bb0 0x7f3de406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 -- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3de40c8990 msgr2=0x7f3de415dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de2800640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3de40c8990 0x7f3de415dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3de40c8990 0x7f3de415dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.854+0000 7f3de1e00640 1 -- 172.21.15.125:0/4255646135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40c8190 con 0x7f3dc40039a0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3de1e00640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f3dd402c450 tx=0x7f3dd4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4039070 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd402cce0 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3de9000640 1 -- 172.21.15.125:0/4255646135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de406c3e0 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd403c040 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.855+0000 7f3de9000640 1 -- 172.21.15.125:0/4255646135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de406ba30 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.856+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de40675d0 con 0x7f3dc40039a0 2024-01-16T15:11:34.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.857+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3dd4032070 con 0x7f3dc40039a0 2024-01-16T15:11:34.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.858+0000 7f3dda000640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 0x7f3dbc06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.858+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dd40a9a60 con 0x7f3dc40039a0 2024-01-16T15:11:34.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.858+0000 7f3de1400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 0x7f3dbc06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.836 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.859+0000 7f3de1400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 0x7f3dbc06e650 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3de406f780 tx=0x7f3dcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.839 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:34.862+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3dd4076320 con 0x7f3dc40039a0 2024-01-16T15:11:35.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.045+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f3de406c240 con 0x7f3dc40039a0 2024-01-16T15:11:35.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.052+0000 7f3dda000640 1 -- 172.21.15.125:0/4255646135 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v37)=0 v37) v1 ==== 131+0+0 (secure 0 0 0) 0x7f3de406c240 con 0x7f3dc40039a0 2024-01-16T15:11:35.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.055+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 msgr2=0x7f3dbc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.055+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 0x7f3dbc06e650 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3de406f780 tx=0x7f3dcc009040 comp rx=0 tx=0).stop 2024-01-16T15:11:35.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.055+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 msgr2=0x7f3de414c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.055+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f3dd402c450 tx=0x7f3dd4001a00 comp rx=0 tx=0).stop 2024-01-16T15:11:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.055+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 shutdown_connections 2024-01-16T15:11:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3de40c8990 0x7f3de415dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3de4158bb0 0x7f3de406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3dbc06c190 0x7f3dbc06e650 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 --2- 172.21.15.125:0/4255646135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dc40039a0 0x7f3de414c400 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 >> 172.21.15.125:0/4255646135 conn(0x7f3de40d1020 msgr2=0x7f3de406e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 shutdown_connections 2024-01-16T15:11:35.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.056+0000 7f3dd3400640 1 -- 172.21.15.125:0/4255646135 wait complete. 2024-01-16T15:11:35.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:11:35.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.280+0000 7f5937800640 1 Processor -- start 2024-01-16T15:11:35.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.280+0000 7f5937800640 1 -- start start 2024-01-16T15:11:35.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f5937800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5930143ae0 0x7f59300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f5937800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59300cf7d0 0x7f593015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f5937800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300cfc30 con 0x7f593015e1e0 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f5937800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300b1ee0 con 0x7f59300cf7d0 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f5937800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300b2780 con 0x7f5930143ae0 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59300cf7d0 0x7f593015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59300cf7d0 0x7f593015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58136/0 (socket says 172.21.15.125:58136) 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 -- 172.21.15.125:0/1242428686 learned_addr learned my addr 172.21.15.125:0/1242428686 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 -- 172.21.15.125:0/1242428686 >> v2:172.21.15.137:3300/0 conn(0x7f5930143ae0 msgr2=0x7f59300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 --2- 172.21.15.125:0/1242428686 >> v2:172.21.15.137:3300/0 conn(0x7f5930143ae0 0x7f59300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 -- 172.21.15.125:0/1242428686 >> v1:172.21.15.137:6789/0 conn(0x7f593015e1e0 legacy=0x7f593015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.282+0000 7f592fe00640 1 -- 172.21.15.125:0/1242428686 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593015e990 con 0x7f59300cf7d0 2024-01-16T15:11:35.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.283+0000 7f592fe00640 1 --2- 172.21.15.125:0/1242428686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59300cf7d0 0x7f593015dca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f59300cfea0 tx=0x7f591802db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ecb7914c5e47d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.284+0000 7f592f400640 1 -- 172.21.15.125:0/1242428686 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591802f8e0 con 0x7f59300cf7d0 2024-01-16T15:11:35.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.284+0000 7f592f400640 1 -- 172.21.15.125:0/1242428686 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591802fa40 con 0x7f59300cf7d0 2024-01-16T15:11:35.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.284+0000 7f592f400640 1 -- 172.21.15.125:0/1242428686 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591802fdd0 con 0x7f59300cf7d0 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.284+0000 7f5937800640 1 -- 172.21.15.125:0/1242428686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59300cf7d0 msgr2=0x7f593015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.284+0000 7f5937800640 1 --2- 172.21.15.125:0/1242428686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59300cf7d0 0x7f593015dca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f59300cfea0 tx=0x7f591802db20 comp rx=0 tx=0).stop 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 -- 172.21.15.125:0/1242428686 shutdown_connections 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 --2- 172.21.15.125:0/1242428686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59300cf7d0 0x7f593015dca0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 --2- 172.21.15.125:0/1242428686 >> v2:172.21.15.137:3300/0 conn(0x7f5930143ae0 0x7f59300cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 -- 172.21.15.125:0/1242428686 >> 172.21.15.125:0/1242428686 conn(0x7f593006a800 msgr2=0x7f593006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 -- 172.21.15.125:0/1242428686 shutdown_connections 2024-01-16T15:11:35.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.285+0000 7f5937800640 1 -- 172.21.15.125:0/1242428686 wait complete. 2024-01-16T15:11:35.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.287+0000 7f5937800640 1 Processor -- start 2024-01-16T15:11:35.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.287+0000 7f5937800640 1 -- start start 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59300cf7d0 0x7f59300cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f593015e1e0 0x7f593013eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59300cd910 con 0x7f5930143ae0 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593013fe70 con 0x7f59300cf7d0 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f5937800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59300ccfe0 con 0x7f593015e1e0 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f592fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f592fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58142/0 (socket says 172.21.15.125:58142) 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.288+0000 7f592fe00640 1 -- 172.21.15.125:0/3997351489 learned_addr learned my addr 172.21.15.125:0/3997351489 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f5934e00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59300cf7d0 0x7f59300cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 -- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f593015e1e0 msgr2=0x7f593013eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f593015e1e0 0x7f593013eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 -- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59300cf7d0 msgr2=0x7f59300cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59300cf7d0 0x7f59300cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 -- 172.21.15.125:0/3997351489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180094d0 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592fe00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f591802db60 tx=0x7f59180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918037cb0 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5918037e10 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180426b0 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.289+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59300cd2c0 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.290+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593013f2e0 con 0x7f5930143ae0 2024-01-16T15:11:35.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.291+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59300cdac0 con 0x7f5930143ae0 2024-01-16T15:11:35.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.292+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f5918042810 con 0x7f5930143ae0 2024-01-16T15:11:35.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.293+0000 7f592cc00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 0x7f590806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.293+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5918033070 con 0x7f5930143ae0 2024-01-16T15:11:35.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.293+0000 7f5934e00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 0x7f590806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.294+0000 7f5934e00640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 0x7f590806e670 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f592000a810 tx=0x7f5920008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.298+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59300cdac0 con 0x7f5930143ae0 2024-01-16T15:11:35.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.487+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f59300cdac0 con 0x7f5930143ae0 2024-01-16T15:11:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.492+0000 7f592cc00640 1 -- 172.21.15.125:0/3997351489 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v38)=0 v38) v1 ==== 137+0+0 (secure 0 0 0) 0x7f59300cdac0 con 0x7f5930143ae0 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.495+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 msgr2=0x7f590806e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.495+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 0x7f590806e670 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f592000a810 tx=0x7f5920008040 comp rx=0 tx=0).stop 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.495+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 msgr2=0x7f59300cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.495+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f591802db60 tx=0x7f59180046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 shutdown_connections 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f593015e1e0 0x7f593013eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f590806c1b0 0x7f590806e670 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5930143ae0 0x7f59300cc610 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 --2- 172.21.15.125:0/3997351489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59300cf7d0 0x7f59300cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 >> 172.21.15.125:0/3997351489 conn(0x7f593006a800 msgr2=0x7f59300702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 shutdown_connections 2024-01-16T15:11:35.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.496+0000 7f5937800640 1 -- 172.21.15.125:0/3997351489 wait complete. 2024-01-16T15:11:35.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:11:35.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.723+0000 7f8a0ce00640 1 Processor -- start 2024-01-16T15:11:35.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.723+0000 7f8a0ce00640 1 -- start start 2024-01-16T15:11:35.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.724+0000 7f8a0ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8a08150760 0x7f8a08072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.724+0000 7f8a0ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 0x7f8a0815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.724+0000 7f8a0ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0815dc80 con 0x7f8a080b2bd0 2024-01-16T15:11:35.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.724+0000 7f8a0ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a080b3770 con 0x7f8a08150760 2024-01-16T15:11:35.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.724+0000 7f8a0ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08071190 con 0x7f8a0815d300 2024-01-16T15:11:35.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.725+0000 7f8a05e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8a08150760 0x7f8a08072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.725+0000 7f89fd200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 0x7f8a0815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.725+0000 7f8a06800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8a080b2bd0 0x7f8a080b3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:41512/0 (socket says 172.21.15.125:41512) 2024-01-16T15:11:35.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.725+0000 7f8a06800640 1 -- 172.21.15.125:0/1575000574 learned_addr learned my addr 172.21.15.125:0/1575000574 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f8a05400640 1 -- 172.21.15.125:0/1575000574 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469937995 0 0) 0x7f8a0815dc80 con 0x7f8a080b2bd0 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f8a05400640 1 -- 172.21.15.125:0/1575000574 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e8003520 con 0x7f8a080b2bd0 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f89fd200640 1 -- 172.21.15.125:0/1575000574 >> v2:172.21.15.137:3300/0 conn(0x7f8a08150760 msgr2=0x7f8a08072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f89fd200640 1 --2- 172.21.15.125:0/1575000574 >> v2:172.21.15.137:3300/0 conn(0x7f8a08150760 0x7f8a08072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f89fd200640 1 -- 172.21.15.125:0/1575000574 >> v1:172.21.15.137:6789/0 conn(0x7f8a080b2bd0 legacy=0x7f8a080b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.726+0000 7f89fd200640 1 -- 172.21.15.125:0/1575000574 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08071470 con 0x7f8a0815d300 2024-01-16T15:11:35.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.727+0000 7f89fd200640 1 --2- 172.21.15.125:0/1575000574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 0x7f8a0815d740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89f00097a0 tx=0x7f89f002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2256325a08d9d43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.728+0000 7f8a05400640 1 -- 172.21.15.125:0/1575000574 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0031830 con 0x7f8a0815d300 2024-01-16T15:11:35.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.728+0000 7f8a05400640 1 -- 172.21.15.125:0/1575000574 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0031990 con 0x7f8a0815d300 2024-01-16T15:11:35.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.728+0000 7f8a05400640 1 -- 172.21.15.125:0/1575000574 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0031d20 con 0x7f8a0815d300 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 -- 172.21.15.125:0/1575000574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 msgr2=0x7f8a0815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/1575000574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 0x7f8a0815d740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89f00097a0 tx=0x7f89f002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 -- 172.21.15.125:0/1575000574 shutdown_connections 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/1575000574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a0815d300 0x7f8a0815d740 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/1575000574 >> v2:172.21.15.137:3300/0 conn(0x7f8a08150760 0x7f8a08072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 -- 172.21.15.125:0/1575000574 >> 172.21.15.125:0/1575000574 conn(0x7f8a0806a800 msgr2=0x7f8a0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 -- 172.21.15.125:0/1575000574 shutdown_connections 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.729+0000 7f8a0ce00640 1 -- 172.21.15.125:0/1575000574 wait complete. 2024-01-16T15:11:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.731+0000 7f8a0ce00640 1 Processor -- start 2024-01-16T15:11:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.731+0000 7f8a0ce00640 1 -- start start 2024-01-16T15:11:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a08150760 0x7f8a0813b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a0815d300 0x7f8a080aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0815c130 con 0x7f8a080b2bd0 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a080b1a10 con 0x7f8a0815d300 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a0ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08139a70 con 0x7f8a08150760 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58150/0 (socket says 172.21.15.125:58150) 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 -- 172.21.15.125:0/3427722694 learned_addr learned my addr 172.21.15.125:0/3427722694 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f89fd200640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a08150760 0x7f8a0813b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 -- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a08150760 msgr2=0x7f8a0813b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a08150760 0x7f8a0813b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 -- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a0815d300 msgr2=0x7f8a080aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a0815d300 0x7f8a080aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.732+0000 7f8a05e00640 1 -- 172.21.15.125:0/3427722694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f0009400 con 0x7f8a080b2bd0 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f8a05e00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f89f800e910 tx=0x7f89f800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800cc40 con 0x7f8a080b2bd0 2024-01-16T15:11:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f800cda0 con 0x7f8a080b2bd0 2024-01-16T15:11:35.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8011610 con 0x7f8a080b2bd0 2024-01-16T15:11:35.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a080b1730 con 0x7f8a080b2bd0 2024-01-16T15:11:35.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.733+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a08139700 con 0x7f8a080b2bd0 2024-01-16T15:11:35.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.734+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a080675d0 con 0x7f8a080b2bd0 2024-01-16T15:11:35.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.736+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f89f8011770 con 0x7f8a080b2bd0 2024-01-16T15:11:35.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.737+0000 7f89fea00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 0x7f89e006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.737+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89f8088b10 con 0x7f8a080b2bd0 2024-01-16T15:11:35.715 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.737+0000 7f89fd200640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 0x7f89e006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.738+0000 7f89fd200640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 0x7f89e006e620 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f89f0036000 tx=0x7f89f003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.741+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89f80550f0 con 0x7f8a080b2bd0 2024-01-16T15:11:35.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:35.958+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a08071d10 con 0x7f89e006c160 2024-01-16T15:11:35.943 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:35.965+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:35.943 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:35.965+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:35.943 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:35.965+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:35.943 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:35.965+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:35.943 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:11:35.965+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:35.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.012+0000 7f89fea00640 1 -- 172.21.15.125:0/3427722694 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f89f0031b80 con 0x7f89e006c160 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 msgr2=0x7f89e006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 0x7f89e006e620 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f89f0036000 tx=0x7f89f003c040 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 msgr2=0x7f8a080b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f89f800e910 tx=0x7f89f800edd0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 shutdown_connections 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a0815d300 0x7f8a080aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8a08150760 0x7f8a0813b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f89e006c160 0x7f89e006e620 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 --2- 172.21.15.125:0/3427722694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a080b2bd0 0x7f8a080b1150 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 >> 172.21.15.125:0/3427722694 conn(0x7f8a0806a800 msgr2=0x7f8a0815ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.015+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 shutdown_connections 2024-01-16T15:11:35.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.016+0000 7f8a0ce00640 1 -- 172.21.15.125:0/3427722694 wait complete. 2024-01-16T15:11:36.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:11:36.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.224+0000 7f06a5600640 1 Processor -- start 2024-01-16T15:11:36.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.224+0000 7f06a5600640 1 -- start start 2024-01-16T15:11:36.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 0x7f06a015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f06a013e360 0x7f06a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 0x7f06a00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a006c470 con 0x7f06a013e360 2024-01-16T15:11:36.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a006c750 con 0x7f06a013df80 2024-01-16T15:11:36.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f06a5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a006ca30 con 0x7f06a015f9c0 2024-01-16T15:11:36.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f069e000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f06a013e360 0x7f06a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f069e000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f06a013e360 0x7f06a00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58172/0 (socket says 172.21.15.125:58172) 2024-01-16T15:11:36.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.226+0000 7f069e000640 1 -- 172.21.15.125:0/3909605084 learned_addr learned my addr 172.21.15.125:0/3909605084 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069ea00640 1 --2- 172.21.15.125:0/3909605084 >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 0x7f06a015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069e000640 1 -- 172.21.15.125:0/3909605084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 msgr2=0x7f06a00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069f400640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 0x7f06a00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069e000640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 0x7f06a00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069e000640 1 -- 172.21.15.125:0/3909605084 >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 msgr2=0x7f06a015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069e000640 1 --2- 172.21.15.125:0/3909605084 >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 0x7f06a015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069e000640 1 -- 172.21.15.125:0/3909605084 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a006cd10 con 0x7f06a013e360 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069ea00640 1 --2- 172.21.15.125:0/3909605084 >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 0x7f06a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.227+0000 7f069f400640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 0x7f06a00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.228+0000 7f069e000640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a00cf290 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f06a0072770 tx=0x7f068802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7625c82f7ff816 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.228+0000 7f069d600640 1 -- 172.21.15.125:0/3909605084 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688031870 con 0x7f06a013e360 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.228+0000 7f069d600640 1 -- 172.21.15.125:0/3909605084 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06880319d0 con 0x7f06a013e360 2024-01-16T15:11:36.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.228+0000 7f069d600640 1 -- 172.21.15.125:0/3909605084 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688031d60 con 0x7f06a013e360 2024-01-16T15:11:36.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 -- 172.21.15.125:0/3909605084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 msgr2=0x7f06a00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a00cf290 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f06a0072770 tx=0x7f068802b910 comp rx=0 tx=0).stop 2024-01-16T15:11:36.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 -- 172.21.15.125:0/3909605084 shutdown_connections 2024-01-16T15:11:36.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a015f9c0 0x7f06a00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 --2- 172.21.15.125:0/3909605084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a00cf290 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 --2- 172.21.15.125:0/3909605084 >> v2:172.21.15.137:3300/0 conn(0x7f06a013df80 0x7f06a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 -- 172.21.15.125:0/3909605084 >> 172.21.15.125:0/3909605084 conn(0x7f06a006a800 msgr2=0x7f06a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:36.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 -- 172.21.15.125:0/3909605084 shutdown_connections 2024-01-16T15:11:36.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.229+0000 7f06a5600640 1 -- 172.21.15.125:0/3909605084 wait complete. 2024-01-16T15:11:36.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.231+0000 7f06a5600640 1 Processor -- start 2024-01-16T15:11:36.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.231+0000 7f06a5600640 1 -- start start 2024-01-16T15:11:36.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 0x7f06a0154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06a015f9c0 0x7f06a015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a01f4800 con 0x7f06a013e360 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a01f4c20 con 0x7f06a015f9c0 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f06a5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0154ab0 con 0x7f06a013df80 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f069e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f069e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58180/0 (socket says 172.21.15.125:58180) 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.232+0000 7f069e000640 1 -- 172.21.15.125:0/180165466 learned_addr learned my addr 172.21.15.125:0/180165466 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069ea00640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 0x7f06a0154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069f400640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06a015f9c0 0x7f06a015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 -- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 msgr2=0x7f06a0154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 0x7f06a0154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 -- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06a015f9c0 msgr2=0x7f06a015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06a015f9c0 0x7f06a015dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 -- 172.21.15.125:0/180165466 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880094d0 con 0x7f06a013e360 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069ea00640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 0x7f06a0154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f069e000640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f068802b950 tx=0x7f06880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.233+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688031860 con 0x7f06a013e360 2024-01-16T15:11:36.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.234+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06880443f0 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.234+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06880447a0 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.234+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a013d410 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.234+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01f4140 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.235+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a01f4940 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.236+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f068804d450 con 0x7f06a013e360 2024-01-16T15:11:36.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.237+0000 7f0696a00640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 0x7f067806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.237+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f068802d040 con 0x7f06a013e360 2024-01-16T15:11:36.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.237+0000 7f069ea00640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 0x7f067806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.238+0000 7f069ea00640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 0x7f067806e620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0690006fd0 tx=0x7f0690008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.241+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06a01f4940 con 0x7f06a013e360 2024-01-16T15:11:36.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.462+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a01408d0 con 0x7f067806c160 2024-01-16T15:11:36.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.469+0000 7f0696a00640 1 -- 172.21.15.125:0/180165466 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f069000c040 con 0x7f067806c160 2024-01-16T15:11:36.446 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:11:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 msgr2=0x7f067806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 0x7f067806e620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0690006fd0 tx=0x7f0690008040 comp rx=0 tx=0).stop 2024-01-16T15:11:36.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 msgr2=0x7f06a015d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f068802b950 tx=0x7f06880046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 shutdown_connections 2024-01-16T15:11:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.471+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06a015f9c0 0x7f06a015dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f067806c160 0x7f067806e620 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06a013e360 0x7f06a015d8a0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 --2- 172.21.15.125:0/180165466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06a013df80 0x7f06a0154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 >> 172.21.15.125:0/180165466 conn(0x7f06a006a800 msgr2=0x7f06a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 shutdown_connections 2024-01-16T15:11:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.472+0000 7f06a5600640 1 -- 172.21.15.125:0/180165466 wait complete. 2024-01-16T15:11:36.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:11:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.709+0000 7fa707e00640 1 Processor -- start 2024-01-16T15:11:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.709+0000 7fa707e00640 1 -- start start 2024-01-16T15:11:36.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa70806b580 0x7fa70806b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa70815e840 0x7fa70815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70815ab20 0x7fa7080cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70815b5e0 con 0x7fa70815e840 2024-01-16T15:11:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70806bea0 con 0x7fa70806b580 2024-01-16T15:11:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa707e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70815b300 con 0x7fa70815ab20 2024-01-16T15:11:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa706000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa70815e840 0x7fa70815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa706000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa70815e840 0x7fa70815a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36496/0 (socket says 172.21.15.125:36496) 2024-01-16T15:11:36.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.711+0000 7fa706000640 1 -- 172.21.15.125:0/2423843645 learned_addr learned my addr 172.21.15.125:0/2423843645 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:36.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706a00640 1 --2- 172.21.15.125:0/2423843645 >> v2:172.21.15.137:3300/0 conn(0x7fa70806b580 0x7fa70806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa707400640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70815ab20 0x7fa7080cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706000640 1 -- 172.21.15.125:0/2423843645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70815ab20 msgr2=0x7fa7080cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:36.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706000640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70815ab20 0x7fa7080cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706000640 1 -- 172.21.15.125:0/2423843645 >> v2:172.21.15.137:3300/0 conn(0x7fa70806b580 msgr2=0x7fa70806b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:36.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706000640 1 --2- 172.21.15.125:0/2423843645 >> v2:172.21.15.137:3300/0 conn(0x7fa70806b580 0x7fa70806b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.712+0000 7fa706000640 1 -- 172.21.15.125:0/2423843645 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70815dd60 con 0x7fa70815e840 2024-01-16T15:11:36.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.713+0000 7fa706000640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815e840 0x7fa70815a5e0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa7080718f0 tx=0x7fa6f002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffa35d607e1186e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.713+0000 7fa705600640 1 -- 172.21.15.125:0/2423843645 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f002ea80 con 0x7fa70815e840 2024-01-16T15:11:36.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa705600640 1 -- 172.21.15.125:0/2423843645 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f002ebe0 con 0x7fa70815e840 2024-01-16T15:11:36.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa705600640 1 -- 172.21.15.125:0/2423843645 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0009600 con 0x7fa70815e840 2024-01-16T15:11:36.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 -- 172.21.15.125:0/2423843645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815e840 msgr2=0x7fa70815a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815e840 0x7fa70815a5e0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa7080718f0 tx=0x7fa6f002e0e0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 -- 172.21.15.125:0/2423843645 shutdown_connections 2024-01-16T15:11:36.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70815ab20 0x7fa7080cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 --2- 172.21.15.125:0/2423843645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815e840 0x7fa70815a5e0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 --2- 172.21.15.125:0/2423843645 >> v2:172.21.15.137:3300/0 conn(0x7fa70806b580 0x7fa70806b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 -- 172.21.15.125:0/2423843645 >> 172.21.15.125:0/2423843645 conn(0x7fa7080b1ba0 msgr2=0x7fa7080b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.714+0000 7fa707e00640 1 -- 172.21.15.125:0/2423843645 shutdown_connections 2024-01-16T15:11:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.715+0000 7fa707e00640 1 -- 172.21.15.125:0/2423843645 wait complete. 2024-01-16T15:11:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.716+0000 7fa707e00640 1 Processor -- start 2024-01-16T15:11:36.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.716+0000 7fa707e00640 1 -- start start 2024-01-16T15:11:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70806b580 0x7fa708159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70815e840 0x7fa708141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70815a190 con 0x7fa70815ab20 2024-01-16T15:11:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081e4c20 con 0x7fa70815e840 2024-01-16T15:11:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.717+0000 7fa707e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa708141fc0 con 0x7fa70806b580 2024-01-16T15:11:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36512/0 (socket says 172.21.15.125:36512) 2024-01-16T15:11:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 -- 172.21.15.125:0/136993947 learned_addr learned my addr 172.21.15.125:0/136993947 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:11:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa707400640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70815e840 0x7fa708141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706a00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70806b580 0x7fa708159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 -- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70806b580 msgr2=0x7fa708159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70806b580 0x7fa708159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 -- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70815e840 msgr2=0x7fa708141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70815e840 0x7fa708141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.718+0000 7fa706000640 1 -- 172.21.15.125:0/136993947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f000bd40 con 0x7fa70815ab20 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa706000640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0038040 tx=0x7fa6f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f002ea80 con 0x7fa70815ab20 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f003f870 con 0x7fa70815ab20 2024-01-16T15:11:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f003fc20 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7081421f0 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.719+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7081427b0 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.721+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70806ee70 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.722+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa6f002ed50 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.723+0000 7fa6fea00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 0x7fa6e006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.723+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6f00ab850 con 0x7fa70815ab20 2024-01-16T15:11:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.723+0000 7fa706a00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 0x7fa6e006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:36.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.724+0000 7fa706a00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 0x7fa6e006e620 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8006fd0 tx=0x7fa6f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:36.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.728+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6f00780f0 con 0x7fa70815ab20 2024-01-16T15:11:36.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.946+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa70815afb0 con 0x7fa6e006c160 2024-01-16T15:11:36.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.956+0000 7fa6fea00640 1 -- 172.21.15.125:0/136993947 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa6f800f860 con 0x7fa6e006c160 2024-01-16T15:11:36.933 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:11:36.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.958+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 msgr2=0x7fa6e006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.958+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 0x7fa6e006e620 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8006fd0 tx=0x7fa6f8008040 comp rx=0 tx=0).stop 2024-01-16T15:11:36.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.958+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 msgr2=0x7fa7081e46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.958+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0038040 tx=0x7fa6f0004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:36.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 shutdown_connections 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa70815e840 0x7fa708141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa6e006c160 0x7fa6e006e620 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa70815ab20 0x7fa7081e46e0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 --2- 172.21.15.125:0/136993947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa70806b580 0x7fa708159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 >> 172.21.15.125:0/136993947 conn(0x7fa7080b1ba0 msgr2=0x7fa70806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 shutdown_connections 2024-01-16T15:11:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:11:36.959+0000 7fa707e00640 1 -- 172.21.15.125:0/136993947 wait complete. 2024-01-16T15:11:36.954 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:11:36.954 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:36.954 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:37.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:37.003 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:11:41.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:41.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:50.790 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:11:50.790 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:11:50.790 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6664 s, 78.6 MB/s 2024-01-16T15:11:50.792 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:11:50.792 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:50.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:50.843 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:12:02.582 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:12:02.582 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:12:02.582 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.6344 s, 92.3 MB/s 2024-01-16T15:12:02.584 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:12:02.584 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:02.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:02.636 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:12:11.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:11.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:13.638 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:12:13.638 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:12:13.638 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.9478 s, 98.1 MB/s 2024-01-16T15:12:13.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:12:13.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.881+0000 7f7d3d000640 1 Processor -- start 2024-01-16T15:12:13.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.881+0000 7f7d3d000640 1 -- start start 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d38071410 0x7f7d38146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d380717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7d38071d30 0x7f7d380b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3806cc20 con 0x7f7d38071410 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3806cec0 con 0x7f7d38071d30 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3806d1a0 con 0x7f7d3806c7c0 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d37400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d38071410 0x7f7d38146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d37400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d38071410 0x7f7d38146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39902/0 (socket says 172.21.15.125:39902) 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d37400640 1 -- 172.21.15.125:0/407843385 learned_addr learned my addr 172.21.15.125:0/407843385 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d37e00640 1 --2- 172.21.15.125:0/407843385 >> v2:172.21.15.137:3300/0 conn(0x7f7d38071d30 0x7f7d380b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.883+0000 7f7d2fe00640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d380717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.884+0000 7f7d37400640 1 -- 172.21.15.125:0/407843385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 msgr2=0x7f7d380717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.884+0000 7f7d37400640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d380717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.884+0000 7f7d37400640 1 -- 172.21.15.125:0/407843385 >> v2:172.21.15.137:3300/0 conn(0x7f7d38071d30 msgr2=0x7f7d380b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.884+0000 7f7d37400640 1 --2- 172.21.15.125:0/407843385 >> v2:172.21.15.137:3300/0 conn(0x7f7d38071d30 0x7f7d380b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.884+0000 7f7d37400640 1 -- 172.21.15.125:0/407843385 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3806d480 con 0x7f7d38071410 2024-01-16T15:12:13.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.885+0000 7f7d37400640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d38146770 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f7d38146ec0 tx=0x7f7d2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94a6380a06e918b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.885+0000 7f7d36a00640 1 -- 172.21.15.125:0/407843385 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28031870 con 0x7f7d38071410 2024-01-16T15:12:13.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d36a00640 1 -- 172.21.15.125:0/407843385 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d280319d0 con 0x7f7d38071410 2024-01-16T15:12:13.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d36a00640 1 -- 172.21.15.125:0/407843385 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28031d60 con 0x7f7d38071410 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 -- 172.21.15.125:0/407843385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 msgr2=0x7f7d38146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d38146770 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f7d38146ec0 tx=0x7f7d2802b910 comp rx=0 tx=0).stop 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 -- 172.21.15.125:0/407843385 shutdown_connections 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 --2- 172.21.15.125:0/407843385 >> v2:172.21.15.137:3300/0 conn(0x7f7d38071d30 0x7f7d380b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d380717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 --2- 172.21.15.125:0/407843385 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d38146770 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 -- 172.21.15.125:0/407843385 >> 172.21.15.125:0/407843385 conn(0x7f7d3806a800 msgr2=0x7f7d3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 -- 172.21.15.125:0/407843385 shutdown_connections 2024-01-16T15:12:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.886+0000 7f7d3d000640 1 -- 172.21.15.125:0/407843385 wait complete. 2024-01-16T15:12:13.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.888+0000 7f7d3d000640 1 Processor -- start 2024-01-16T15:12:13.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.888+0000 7f7d3d000640 1 -- start start 2024-01-16T15:12:13.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d3815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38071d30 0x7f7d380c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c5eb0 con 0x7f7d38071410 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815d030 con 0x7f7d38071d30 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c52a0 con 0x7f7d3806c7c0 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.889+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39906/0 (socket says 172.21.15.125:39906) 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 -- 172.21.15.125:0/3805308305 learned_addr learned my addr 172.21.15.125:0/3805308305 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d37400640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d3815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d37e00640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38071d30 0x7f7d380c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 -- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 msgr2=0x7f7d3815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d3815caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 -- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38071d30 msgr2=0x7f7d380c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38071d30 0x7f7d380c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 -- 172.21.15.125:0/3805308305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280094d0 con 0x7f7d38071410 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.890+0000 7f7d2fe00640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d800 tx=0x7f7d2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.891+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000b6c0 con 0x7f7d38071410 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.891+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000b820 con 0x7f7d38071410 2024-01-16T15:12:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.891+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380c5580 con 0x7f7d38071410 2024-01-16T15:12:13.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.891+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000bbb0 con 0x7f7d38071410 2024-01-16T15:12:13.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.891+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d380c5b40 con 0x7f7d38071410 2024-01-16T15:12:13.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.892+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38067450 con 0x7f7d38071410 2024-01-16T15:12:13.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.893+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 35) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f7d2000bd10 con 0x7f7d38071410 2024-01-16T15:12:13.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.894+0000 7f7d2f400640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 0x7f7d1006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.894+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d20088ed0 con 0x7f7d38071410 2024-01-16T15:12:13.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.894+0000 7f7d37400640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 0x7f7d1006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.894+0000 7f7d37400640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 0x7f7d1006e6b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d2803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:13.898+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d20055500 con 0x7f7d38071410 2024-01-16T15:12:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.119+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d38072150 con 0x7f7d1006c1f0 2024-01-16T15:12:14.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.146+0000 7f7d2f400640 1 -- 172.21.15.125:0/3805308305 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7d28031bc0 con 0x7f7d1006c1f0 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 msgr2=0x7f7d1006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 0x7f7d1006e6b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d2803c040 comp rx=0 tx=0).stop 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 msgr2=0x7f7d380c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d800 tx=0x7f7d2000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 shutdown_connections 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38071d30 0x7f7d380c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7d1006c1f0 0x7f7d1006e6b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d38071410 0x7f7d380c6b70 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 --2- 172.21.15.125:0/3805308305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d3806c7c0 0x7f7d3815caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 >> 172.21.15.125:0/3805308305 conn(0x7f7d3806a800 msgr2=0x7f7d380afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 shutdown_connections 2024-01-16T15:12:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.150+0000 7f7d3d000640 1 -- 172.21.15.125:0/3805308305 wait complete. 2024-01-16T15:12:14.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:12:14.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.385+0000 7f7585600640 1 Processor -- start 2024-01-16T15:12:14.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.385+0000 7f7585600640 1 -- start start 2024-01-16T15:12:14.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.386+0000 7f7585600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f75800ae5a0 0x7f758015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.386+0000 7f7585600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75800cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.386+0000 7f7585600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7580070c40 con 0x7f75800b3ae0 2024-01-16T15:12:14.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.386+0000 7f7585600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7580071450 con 0x7f75800ae5a0 2024-01-16T15:12:14.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.386+0000 7f7585600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800cfc60 con 0x7f75800cf290 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f7577e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f75800b3ae0 0x7f758006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34856/0 (socket says 172.21.15.125:34856) 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f7577e00640 1 -- 172.21.15.125:0/3365025256 learned_addr learned my addr 172.21.15.125:0/3365025256 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f757f400640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75800cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f757ea00640 1 --2- 172.21.15.125:0/3365025256 >> v2:172.21.15.137:3300/0 conn(0x7f75800ae5a0 0x7f758015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836272332 0 0) 0x7f7580070c40 con 0x7f75800b3ae0 2024-01-16T15:12:14.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.387+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7560003520 con 0x7f75800b3ae0 2024-01-16T15:12:14.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422026165 0 0) 0x7f7560003520 con 0x7f75800b3ae0 2024-01-16T15:12:14.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 msgr2=0x7f75800cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75800cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 >> v2:172.21.15.137:3300/0 conn(0x7f75800ae5a0 msgr2=0x7f758015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 --2- 172.21.15.125:0/3365025256 >> v2:172.21.15.137:3300/0 conn(0x7f75800ae5a0 0x7f758015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758006c470 con 0x7f75800b3ae0 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7568001670 con 0x7f75800b3ae0 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 >> v1:172.21.15.125:6789/0 conn(0x7f75800b3ae0 legacy=0x7f758006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7560003970 0x7f7560003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560004d00 0x7f7560005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580071450 con 0x7f7560003970 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75800cfc60 con 0x7f7560004d00 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756000ba00 con 0x7f75800cf290 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.388+0000 7f757ea00640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7560003970 0x7f7560003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f7577e00640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757f400640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560004d00 0x7f7560005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757ea00640 1 -- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 msgr2=0x7f75600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757ea00640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75800cf290 0x7f75600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757ea00640 1 -- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560004d00 msgr2=0x7f7560005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757ea00640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560004d00 0x7f7560005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757ea00640 1 -- 172.21.15.125:0/3365025256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7560003520 con 0x7f7560003970 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.389+0000 7f757f400640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560004d00 0x7f7560005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.390+0000 7f757ea00640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7560003970 0x7f7560003d90 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7570009ac0 tx=0x7f75600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd145db613db4ffd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.390+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757002c710 con 0x7f7560003970 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.390+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757002c870 con 0x7f7560003970 2024-01-16T15:12:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.390+0000 7f757e000640 1 -- 172.21.15.125:0/3365025256 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757002cc00 con 0x7f7560003970 2024-01-16T15:12:14.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 -- 172.21.15.125:0/3365025256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7560003970 msgr2=0x7f7560003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 --2- 172.21.15.125:0/3365025256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7560003970 0x7f7560003d90 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7570009ac0 tx=0x7f75600057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f757f400640 1 -- 172.21.15.125:0/3365025256 reap_dead start 2024-01-16T15:12:14.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 -- 172.21.15.125:0/3365025256 shutdown_connections 2024-01-16T15:12:14.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 -- 172.21.15.125:0/3365025256 >> 172.21.15.125:0/3365025256 conn(0x7f758006a800 msgr2=0x7f758006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 -- 172.21.15.125:0/3365025256 shutdown_connections 2024-01-16T15:12:14.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.391+0000 7f7585600640 1 -- 172.21.15.125:0/3365025256 wait complete. 2024-01-16T15:12:14.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.393+0000 7f7585600640 1 Processor -- start 2024-01-16T15:12:14.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.393+0000 7f7585600640 1 -- start start 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560003970 0x7f75800cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 0x7f75801459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758015fd30 con 0x7f758015f370 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580146330 con 0x7f7560003970 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f7585600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800cbf30 con 0x7f7580148e10 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f757f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f757f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39936/0 (socket says 172.21.15.125:39936) 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f757f400640 1 -- 172.21.15.125:0/2012238815 learned_addr learned my addr 172.21.15.125:0/2012238815 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.394+0000 7f757ea00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560003970 0x7f75800cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7577e00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 0x7f75801459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 -- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 msgr2=0x7f75801459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 0x7f75801459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 -- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560003970 msgr2=0x7f75800cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560003970 0x7f75800cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 -- 172.21.15.125:0/2012238815 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800cc210 con 0x7f758015f370 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7577e00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 0x7f75801459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f757f400640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f75800b4170 tx=0x7f757800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578017400 con 0x7f758015f370 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578011040 con 0x7f758015f370 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75800cc4f0 con 0x7f758015f370 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75780177a0 con 0x7f758015f370 2024-01-16T15:12:14.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.395+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75800ccab0 con 0x7f758015f370 2024-01-16T15:12:14.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.396+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75800675d0 con 0x7f758015f370 2024-01-16T15:12:14.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.398+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 35) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f75780203b0 con 0x7f758015f370 2024-01-16T15:12:14.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.398+0000 7f7576a00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 0x7f755806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.399+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7578088920 con 0x7f758015f370 2024-01-16T15:12:14.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.399+0000 7f757ea00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 0x7f755806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.399+0000 7f757ea00640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 0x7f755806e6b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7570036000 tx=0x7f7570030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.403+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75780551b0 con 0x7f758015f370 2024-01-16T15:12:14.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.623+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7580071170 con 0x7f755806c1f0 2024-01-16T15:12:14.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.628+0000 7f7576a00640 1 -- 172.21.15.125:0/2012238815 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f757002cab0 con 0x7f755806c1f0 2024-01-16T15:12:14.605 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 msgr2=0x7f755806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 0x7f755806e6b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7570036000 tx=0x7f7570030000 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 msgr2=0x7f758015f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f75800b4170 tx=0x7f757800f4a0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 shutdown_connections 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f755806c1f0 0x7f755806e6b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758015f370 0x7f758015f7f0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7580148e10 0x7f75801459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 --2- 172.21.15.125:0/2012238815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7560003970 0x7f75800cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 >> 172.21.15.125:0/2012238815 conn(0x7f758006a800 msgr2=0x7f7580159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.631+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 shutdown_connections 2024-01-16T15:12:14.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:14.632+0000 7f7585600640 1 -- 172.21.15.125:0/2012238815 wait complete. 2024-01-16T15:12:14.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:14.627 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a) 2024-01-16T15:12:14.685 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:12:14.686 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:12:14.686 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:12:14.687 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:12:16.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:16.688 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a) 2024-01-16T15:12:16.753 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/cc9616d3-bf68-4625-86b4-b99ce55d389a 2024-01-16T15:12:16.753 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:12:16.753 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:12:16.754 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:12:16.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone57941 2024-01-16T15:12:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.996+0000 7f813cc00640 1 Processor -- start 2024-01-16T15:12:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.997+0000 7f813cc00640 1 -- start start 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.998+0000 7f813cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f81380cf9a0 0x7f813815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.998+0000 7f813cc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380b1ee0 con 0x7f813815e1e0 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f813cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380b2780 con 0x7f8138143ae0 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f813cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813815e990 con 0x7f81380cf9a0 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f8136800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f813815e1e0 0x7f81380702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36382/0 (socket says 172.21.15.125:36382) 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f8136800640 1 -- 172.21.15.125:0/1217526839 learned_addr learned my addr 172.21.15.125:0/1217526839 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f812d200640 1 --2- 172.21.15.125:0/1217526839 >> v2:172.21.15.137:3300/0 conn(0x7f81380cf9a0 0x7f813815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525128051 0 0) 0x7f81380b1ee0 con 0x7f813815e1e0 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:16.999+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8118003520 con 0x7f813815e1e0 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567610286 0 0) 0x7f8118003520 con 0x7f813815e1e0 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 >> v2:172.21.15.137:3300/0 conn(0x7f81380cf9a0 msgr2=0x7f813815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 --2- 172.21.15.125:0/1217526839 >> v2:172.21.15.137:3300/0 conn(0x7f81380cf9a0 0x7f813815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 >> v1:172.21.15.137:6789/0 conn(0x7f8138143ae0 legacy=0x7f81380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380b2130 con 0x7f813815e1e0 2024-01-16T15:12:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8130001670 con 0x7f813815e1e0 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 >> v1:172.21.15.125:6789/0 conn(0x7f813815e1e0 legacy=0x7f81380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81180039c0 0x7f8118003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81180047c0 0x7f8118004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f811800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813815e990 con 0x7f8118005180 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811800c080 con 0x7f81180039c0 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811800c240 con 0x7f81180047c0 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f812d200640 1 --2- 172.21.15.125:0/1217526839 >> v2:172.21.15.137:3300/0 conn(0x7f81380cf9a0 0x7f813815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.000+0000 7f8136800640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f811800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8135e00640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81180039c0 0x7f8118003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8136800640 1 -- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81180047c0 msgr2=0x7f8118004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8136800640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81180047c0 0x7f8118004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8136800640 1 -- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81180039c0 msgr2=0x7f8118003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8136800640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81180039c0 0x7f8118003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.001+0000 7f8136800640 1 -- 172.21.15.125:0/1217526839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8118003520 con 0x7f8118005180 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.002+0000 7f8136800640 1 -- 172.21.15.125:0/1217526839 reap_dead start 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.002+0000 7f8136800640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f811800b980 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f813000c0e0 tx=0x7f81180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d356bf9e7b3dc88 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.002+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130005ad0 con 0x7f8118005180 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.002+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130005c70 con 0x7f8118005180 2024-01-16T15:12:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.002+0000 7f8135400640 1 -- 172.21.15.125:0/1217526839 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813002c810 con 0x7f8118005180 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 -- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 msgr2=0x7f811800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f811800b980 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f813000c0e0 tx=0x7f81180057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 -- 172.21.15.125:0/1217526839 shutdown_connections 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 --2- 172.21.15.125:0/1217526839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f811800b980 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 -- 172.21.15.125:0/1217526839 >> 172.21.15.125:0/1217526839 conn(0x7f813806a800 msgr2=0x7f813806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 -- 172.21.15.125:0/1217526839 shutdown_connections 2024-01-16T15:12:16.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.003+0000 7f813cc00640 1 -- 172.21.15.125:0/1217526839 wait complete. 2024-01-16T15:12:16.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.005+0000 7f813cc00640 1 Processor -- start 2024-01-16T15:12:16.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.005+0000 7f813cc00640 1 -- start start 2024-01-16T15:12:16.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f813814c040 0x7f813814eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f813813bf60 0x7f81380af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813814c480 con 0x7f8118005180 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813814efe0 con 0x7f813813bf60 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f813cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813815d4a0 con 0x7f813814c040 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39000/0 (socket says 172.21.15.125:39000) 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 -- 172.21.15.125:0/1900997280 learned_addr learned my addr 172.21.15.125:0/1900997280 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 -- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f813814c040 msgr2=0x7f813814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f813814c040 0x7f813814eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 -- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f813813bf60 msgr2=0x7f81380af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f813813bf60 0x7f81380af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.006+0000 7f8135e00640 1 -- 172.21.15.125:0/1900997280 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813813e290 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f8135e00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f812800e840 tx=0x7f812800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128003b80 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128003d00 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128005310 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813813e570 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.007+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813813df40 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.008+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813813e0d0 con 0x7f8118005180 2024-01-16T15:12:16.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.009+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 35) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f8128005470 con 0x7f8118005180 2024-01-16T15:12:16.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.010+0000 7f812ea00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 0x7f811006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.010+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8128088aa0 con 0x7f8118005180 2024-01-16T15:12:16.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.010+0000 7f812d200640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 0x7f811006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.987 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.011+0000 7f812d200640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 0x7f811006e6e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8120006a90 tx=0x7f8120006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.015+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8128055020 con 0x7f8118005180 2024-01-16T15:12:17.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.240+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone57941", "target": ["mon-mgr", ""]}) v1 -- 0x7f813815e670 con 0x7f811006c220 2024-01-16T15:12:17.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.364+0000 7f812ea00640 1 -- 172.21.15.125:0/1900997280 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8120009810 con 0x7f811006c220 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 msgr2=0x7f811006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 0x7f811006e6e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8120006a90 tx=0x7f8120006a20 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 msgr2=0x7f813815ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f812800e840 tx=0x7f812800ed00 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 shutdown_connections 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f813813bf60 0x7f81380af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f813814c040 0x7f813814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f811006c220 0x7f811006e6e0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 --2- 172.21.15.125:0/1900997280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8118005180 0x7f813815ce80 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 >> 172.21.15.125:0/1900997280 conn(0x7f813806a800 msgr2=0x7f81380cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 shutdown_connections 2024-01-16T15:12:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.367+0000 7f813cc00640 1 -- 172.21.15.125:0/1900997280 wait complete. 2024-01-16T15:12:17.349 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.373+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.349 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.373+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.349 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.373+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.349 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.373+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.349 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.373+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone57942 2024-01-16T15:12:17.377 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.401+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.377 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.401+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.377 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.401+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.377 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.401+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.377 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:17.401+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.578+0000 7f3c36200640 1 Processor -- start 2024-01-16T15:12:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.578+0000 7f3c36200640 1 -- start start 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c36200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c300cf9a0 0x7f3c30166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c36200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3015c5f0 con 0x7f3c3013df80 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c36200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3006c470 con 0x7f3c3013e360 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c36200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3006c750 con 0x7f3c300cf9a0 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c2f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3c3013df80 0x7f3c3015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36388/0 (socket says 172.21.15.125:36388) 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c2f400640 1 -- 172.21.15.125:0/1259034432 learned_addr learned my addr 172.21.15.125:0/1259034432 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.580+0000 7f3c2fe00640 1 --2- 172.21.15.125:0/1259034432 >> v2:172.21.15.125:3300/0 conn(0x7f3c300cf9a0 0x7f3c30166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.581+0000 7f3c2e000640 1 -- 172.21.15.125:0/1259034432 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665470732 0 0) 0x7f3c3015c5f0 con 0x7f3c3013df80 2024-01-16T15:12:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.581+0000 7f3c2e000640 1 -- 172.21.15.125:0/1259034432 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c10003520 con 0x7f3c3013df80 2024-01-16T15:12:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.581+0000 7f3c2fe00640 1 -- 172.21.15.125:0/1259034432 >> v1:172.21.15.137:6789/0 conn(0x7f3c3013e360 legacy=0x7f3c300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.581+0000 7f3c2fe00640 1 -- 172.21.15.125:0/1259034432 >> v1:172.21.15.125:6789/0 conn(0x7f3c3013df80 legacy=0x7f3c3015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.581+0000 7f3c2fe00640 1 -- 172.21.15.125:0/1259034432 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3006ca30 con 0x7f3c300cf9a0 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.582+0000 7f3c2fe00640 1 --2- 172.21.15.125:0/1259034432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c300cf9a0 0x7f3c30166030 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009d90 tx=0x7f3c2802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff74d04a1e5e2d35 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.582+0000 7f3c2e000640 1 -- 172.21.15.125:0/1259034432 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2802cc60 con 0x7f3c300cf9a0 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.582+0000 7f3c2e000640 1 -- 172.21.15.125:0/1259034432 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2802cdc0 con 0x7f3c300cf9a0 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.582+0000 7f3c2e000640 1 -- 172.21.15.125:0/1259034432 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28034640 con 0x7f3c300cf9a0 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 -- 172.21.15.125:0/1259034432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c300cf9a0 msgr2=0x7f3c30166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 --2- 172.21.15.125:0/1259034432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c300cf9a0 0x7f3c30166030 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009d90 tx=0x7f3c2802beb0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 -- 172.21.15.125:0/1259034432 shutdown_connections 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 --2- 172.21.15.125:0/1259034432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c300cf9a0 0x7f3c30166030 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 -- 172.21.15.125:0/1259034432 >> 172.21.15.125:0/1259034432 conn(0x7f3c3006a800 msgr2=0x7f3c3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 -- 172.21.15.125:0/1259034432 shutdown_connections 2024-01-16T15:12:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.583+0000 7f3c36200640 1 -- 172.21.15.125:0/1259034432 wait complete. 2024-01-16T15:12:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.585+0000 7f3c36200640 1 Processor -- start 2024-01-16T15:12:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.585+0000 7f3c36200640 1 -- start start 2024-01-16T15:12:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 0x7f3c300adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c3013e360 0x7f3c30132880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c300ae2e0 con 0x7f3c3013df80 2024-01-16T15:12:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3012f800 con 0x7f3c300cf9a0 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c36200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3012fae0 con 0x7f3c3013e360 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c2ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c2ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39040/0 (socket says 172.21.15.125:39040) 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c2ea00640 1 -- 172.21.15.125:0/167300049 learned_addr learned my addr 172.21.15.125:0/167300049 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c2f400640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 0x7f3c300adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.586+0000 7f3c2fe00640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c3013e360 0x7f3c30132880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 -- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c3013e360 msgr2=0x7f3c30132880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c3013e360 0x7f3c30132880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 -- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 msgr2=0x7f3c300adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 0x7f3c300adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 -- 172.21.15.125:0/167300049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28009a70 con 0x7f3c3013df80 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2f400640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 0x7f3c300adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c2ea00640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c410 tx=0x7f3c1800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18011790 con 0x7f3c3013df80 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c180118f0 con 0x7f3c3013df80 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18011ce0 con 0x7f3c3013df80 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3012fdc0 con 0x7f3c3013df80 2024-01-16T15:12:17.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.587+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c30130380 con 0x7f3c3013df80 2024-01-16T15:12:17.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.588+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c300675d0 con 0x7f3c3013df80 2024-01-16T15:12:17.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.589+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 35) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f3c18005080 con 0x7f3c3013df80 2024-01-16T15:12:17.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.590+0000 7f3c27400640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 0x7f3c0806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.590+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c18088590 con 0x7f3c3013df80 2024-01-16T15:12:17.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.591+0000 7f3c2f400640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 0x7f3c0806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.591+0000 7f3c2f400640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 0x7f3c0806e6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3c200011e0 tx=0x7f3c20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.595+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c18058530 con 0x7f3c3013df80 2024-01-16T15:12:17.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.820+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone57942", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c3006cff0 con 0x7f3c0806c1f0 2024-01-16T15:12:17.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.956+0000 7f3c27400640 1 -- 172.21.15.125:0/167300049 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c20001350 con 0x7f3c0806c1f0 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.959+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 msgr2=0x7f3c0806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.959+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 0x7f3c0806e6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3c200011e0 tx=0x7f3c20009040 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.959+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 msgr2=0x7f3c3015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.959+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c410 tx=0x7f3c1800c8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 shutdown_connections 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c3013e360 0x7f3c30132880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c0806c1f0 0x7f3c0806e6b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c3013df80 0x7f3c3015a9b0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 --2- 172.21.15.125:0/167300049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c300cf9a0 0x7f3c300adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 >> 172.21.15.125:0/167300049 conn(0x7f3c3006a800 msgr2=0x7f3c300af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 shutdown_connections 2024-01-16T15:12:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:17.960+0000 7f3c36200640 1 -- 172.21.15.125:0/167300049 wait complete. 2024-01-16T15:12:17.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone57943 2024-01-16T15:12:18.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.240+0000 7ff65a400640 1 Processor -- start 2024-01-16T15:12:18.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.240+0000 7ff65a400640 1 -- start start 2024-01-16T15:12:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.242+0000 7ff65a400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff65415b2a0 0x7ff6540cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.242+0000 7ff65a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6540d00a0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.242+0000 7ff65a400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6540b4010 con 0x7ff6540ce070 2024-01-16T15:12:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.242+0000 7ff65a400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6540b42f0 con 0x7ff65415b2a0 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.242+0000 7ff653400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff65415b2a0 0x7ff6540cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff652a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff6540cfaf0 0x7ff6540cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36392/0 (socket says 172.21.15.125:36392) 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff653400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff65415b2a0 0x7ff6540cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39056/0 (socket says 172.21.15.125:39056) 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff652a00640 1 -- 172.21.15.125:0/2588901343 learned_addr learned my addr 172.21.15.125:0/2588901343 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff653400640 1 -- 172.21.15.125:0/2588901343 >> v1:172.21.15.137:6789/0 conn(0x7ff6540ce070 legacy=0x7ff654164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff653400640 1 -- 172.21.15.125:0/2588901343 >> v1:172.21.15.125:6789/0 conn(0x7ff6540cfaf0 legacy=0x7ff6540cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.243+0000 7ff653400640 1 -- 172.21.15.125:0/2588901343 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540b45d0 con 0x7ff65415b2a0 2024-01-16T15:12:18.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.244+0000 7ff653400640 1 --2- 172.21.15.125:0/2588901343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff65415b2a0 0x7ff6540cf5b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d90 tx=0x7ff64402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7357305ef8ca32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.245+0000 7ff652000640 1 -- 172.21.15.125:0/2588901343 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64402cc60 con 0x7ff65415b2a0 2024-01-16T15:12:18.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.245+0000 7ff652000640 1 -- 172.21.15.125:0/2588901343 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64402cdc0 con 0x7ff65415b2a0 2024-01-16T15:12:18.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.245+0000 7ff652000640 1 -- 172.21.15.125:0/2588901343 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff644034660 con 0x7ff65415b2a0 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 -- 172.21.15.125:0/2588901343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff65415b2a0 msgr2=0x7ff6540cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 --2- 172.21.15.125:0/2588901343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff65415b2a0 0x7ff6540cf5b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d90 tx=0x7ff64402beb0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 -- 172.21.15.125:0/2588901343 shutdown_connections 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 --2- 172.21.15.125:0/2588901343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff65415b2a0 0x7ff6540cf5b0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 -- 172.21.15.125:0/2588901343 >> 172.21.15.125:0/2588901343 conn(0x7ff6540d1020 msgr2=0x7ff6540d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 -- 172.21.15.125:0/2588901343 shutdown_connections 2024-01-16T15:12:18.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.246+0000 7ff65a400640 1 -- 172.21.15.125:0/2588901343 wait complete. 2024-01-16T15:12:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.248+0000 7ff65a400640 1 Processor -- start 2024-01-16T15:12:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.248+0000 7ff65a400640 1 -- start start 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.249+0000 7ff65a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6540ce070 0x7ff654157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.249+0000 7ff65a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff65a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff65415b2a0 0x7ff6541f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff65a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654154da0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff65a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65406bbf0 con 0x7ff65415b2a0 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff65a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65406bed0 con 0x7ff6540ce070 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff652a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff652a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39058/0 (socket says 172.21.15.125:39058) 2024-01-16T15:12:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff652a00640 1 -- 172.21.15.125:0/208328863 learned_addr learned my addr 172.21.15.125:0/208328863 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff653400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6540ce070 0x7ff654157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff653e00640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff65415b2a0 0x7ff6541f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.250+0000 7ff652a00640 1 -- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6540ce070 msgr2=0x7ff654157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff652a00640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6540ce070 0x7ff654157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff652a00640 1 -- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff65415b2a0 msgr2=0x7ff6541f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff652a00640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff65415b2a0 0x7ff6541f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff652a00640 1 -- 172.21.15.125:0/208328863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644009a70 con 0x7ff6540cfaf0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff652a00640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c4a0 tx=0x7ff63c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c005ce0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c005e40 con 0x7ff6540cfaf0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c007270 con 0x7ff6540cfaf0 2024-01-16T15:12:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65406c1b0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.251+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65406c490 con 0x7ff6540cfaf0 2024-01-16T15:12:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.253+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6541551f0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.254+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 35) v1 ==== 90916+0+0 (secure 0 0 0) 0x7ff63c0073d0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.254+0000 7ff64b400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 0x7ff62c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.255+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff63c087b10 con 0x7ff6540cfaf0 2024-01-16T15:12:18.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.255+0000 7ff653400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 0x7ff62c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.256+0000 7ff653400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 0x7ff62c06e6b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff644036000 tx=0x7ff644030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.259+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6541551f0 con 0x7ff6540cfaf0 2024-01-16T15:12:18.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:18.488+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone57943", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6540b6de0 con 0x7ff62c06c1f0 2024-01-16T15:12:19.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:19.387+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 36) v1 ==== 91018+0+0 (secure 0 0 0) 0x7ff63c054850 con 0x7ff6540cfaf0 2024-01-16T15:12:20.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.339+0000 7ff64b400640 1 -- 172.21.15.125:0/208328863 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff64402cb80 con 0x7ff62c06c1f0 2024-01-16T15:12:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 msgr2=0x7ff62c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 0x7ff62c06e6b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff644036000 tx=0x7ff644030000 comp rx=0 tx=0).stop 2024-01-16T15:12:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 msgr2=0x7ff65406b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c4a0 tx=0x7ff63c00c960 comp rx=0 tx=0).stop 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 shutdown_connections 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff65415b2a0 0x7ff6541f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff62c06c1f0 0x7ff62c06e6b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6540cfaf0 0x7ff65406b4f0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 --2- 172.21.15.125:0/208328863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff6540ce070 0x7ff654157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 >> 172.21.15.125:0/208328863 conn(0x7ff6540d1020 msgr2=0x7ff6540b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.342+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 shutdown_connections 2024-01-16T15:12:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.343+0000 7ff65a400640 1 -- 172.21.15.125:0/208328863 wait complete. 2024-01-16T15:12:20.340 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:12:20.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone57943 2024-01-16T15:12:20.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.560+0000 7f6ec7200640 1 Processor -- start 2024-01-16T15:12:20.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.560+0000 7f6ec7200640 1 -- start start 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6ec813cf10 0x7f6ec813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813fae0 0x7f6ec80cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ec80cdc20 0x7f6ec815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec813ff60 con 0x7f6ec80cdc20 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec80ce440 con 0x7f6ec813cf10 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ce720 con 0x7f6ec813fae0 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ec80cdc20 0x7f6ec815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ec80cdc20 0x7f6ec815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39070/0 (socket says 172.21.15.125:39070) 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec6800640 1 -- 172.21.15.125:0/1967125916 learned_addr learned my addr 172.21.15.125:0/1967125916 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ec5e00640 1 --2- 172.21.15.125:0/1967125916 >> v2:172.21.15.137:3300/0 conn(0x7f6ec813cf10 0x7f6ec813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.562+0000 7f6ebd200640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813fae0 0x7f6ec80cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.563+0000 7f6ec6800640 1 -- 172.21.15.125:0/1967125916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813fae0 msgr2=0x7f6ec80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.563+0000 7f6ec6800640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813fae0 0x7f6ec80cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.563+0000 7f6ec6800640 1 -- 172.21.15.125:0/1967125916 >> v2:172.21.15.137:3300/0 conn(0x7f6ec813cf10 msgr2=0x7f6ec813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:20.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.563+0000 7f6ec6800640 1 --2- 172.21.15.125:0/1967125916 >> v2:172.21.15.137:3300/0 conn(0x7f6ec813cf10 0x7f6ec813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.563+0000 7f6ec6800640 1 -- 172.21.15.125:0/1967125916 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80cea00 con 0x7f6ec80cdc20 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.564+0000 7f6ec6800640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec815d9b0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806e390 tx=0x7f6ec002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530e6fade553ae17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.564+0000 7f6ec5400640 1 -- 172.21.15.125:0/1967125916 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec002ec80 con 0x7f6ec80cdc20 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.564+0000 7f6ec5400640 1 -- 172.21.15.125:0/1967125916 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec002ede0 con 0x7f6ec80cdc20 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.564+0000 7f6ec5400640 1 -- 172.21.15.125:0/1967125916 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0036610 con 0x7f6ec80cdc20 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 -- 172.21.15.125:0/1967125916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 msgr2=0x7f6ec815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec815d9b0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806e390 tx=0x7f6ec002e0e0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 -- 172.21.15.125:0/1967125916 shutdown_connections 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec815d9b0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 --2- 172.21.15.125:0/1967125916 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813fae0 0x7f6ec80cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 --2- 172.21.15.125:0/1967125916 >> v2:172.21.15.137:3300/0 conn(0x7f6ec813cf10 0x7f6ec813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 -- 172.21.15.125:0/1967125916 >> 172.21.15.125:0/1967125916 conn(0x7f6ec806a7d0 msgr2=0x7f6ec806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 -- 172.21.15.125:0/1967125916 shutdown_connections 2024-01-16T15:12:20.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.565+0000 7f6ec7200640 1 -- 172.21.15.125:0/1967125916 wait complete. 2024-01-16T15:12:20.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.567+0000 7f6ec7200640 1 Processor -- start 2024-01-16T15:12:20.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.567+0000 7f6ec7200640 1 -- start start 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813cf10 0x7f6ec8150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ec813fae0 0x7f6ec81364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8152020 con 0x7f6ec80cdc20 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8150dd0 con 0x7f6ec813fae0 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec81369f0 con 0x7f6ec813cf10 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39072/0 (socket says 172.21.15.125:39072) 2024-01-16T15:12:20.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 -- 172.21.15.125:0/2627016834 learned_addr learned my addr 172.21.15.125:0/2627016834 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:20.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 -- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813cf10 msgr2=0x7f6ec8150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813cf10 0x7f6ec8150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 -- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ec813fae0 msgr2=0x7f6ec81364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ec813fae0 0x7f6ec81364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.568+0000 7f6ec5e00640 1 -- 172.21.15.125:0/2627016834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0009b30 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ec5e00640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800d800 tx=0x7f6eb800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8009620 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8009780 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec8070950 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8009b10 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.569+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec8070f10 con 0x7f6ec80cdc20 2024-01-16T15:12:20.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.570+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec8067450 con 0x7f6ec80cdc20 2024-01-16T15:12:20.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.571+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 36) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f6eb8009c70 con 0x7f6ec80cdc20 2024-01-16T15:12:20.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.572+0000 7f6ebea00640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 0x7f6ea006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.572+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6eb8088b40 con 0x7f6ec80cdc20 2024-01-16T15:12:20.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.572+0000 7f6ebd200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 0x7f6ea006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.573+0000 7f6ebd200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 0x7f6ea006e800 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.576+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6eb8055400 con 0x7f6ec80cdc20 2024-01-16T15:12:20.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:20.804+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone57943", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ec80ce0b0 con 0x7f6ea006c340 2024-01-16T15:12:25.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.205+0000 7f6ebea00640 1 -- 172.21.15.125:0/2627016834 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f6eb000be20 con 0x7f6ea006c340 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:12:25.182 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:12:25.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:12:25.183 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.208+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 msgr2=0x7f6ea006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.208+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 0x7f6ea006e800 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.208+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 msgr2=0x7f6ec81526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.208+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800d800 tx=0x7f6eb800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 shutdown_connections 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ec813fae0 0x7f6ec81364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ec813cf10 0x7f6ec8150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6ea006c340 0x7f6ea006e800 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 --2- 172.21.15.125:0/2627016834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ec80cdc20 0x7f6ec81526f0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 >> 172.21.15.125:0/2627016834 conn(0x7f6ec806a7d0 msgr2=0x7f6ec80b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 shutdown_connections 2024-01-16T15:12:25.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.209+0000 7f6ec7200640 1 -- 172.21.15.125:0/2627016834 wait complete. 2024-01-16T15:12:25.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone57941 2024-01-16T15:12:25.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.443+0000 7f6465800640 1 Processor -- start 2024-01-16T15:12:25.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.443+0000 7f6465800640 1 -- start start 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.444+0000 7f6465800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 0x7f64600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.444+0000 7f6465800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64600cfc30 con 0x7f64600cf7d0 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f6465800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64600b1ee0 con 0x7f646015e3b0 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f6465800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64600b2780 con 0x7f6460143ae0 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f6455e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64600cf7d0 0x7f646015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36398/0 (socket says 172.21.15.125:36398) 2024-01-16T15:12:25.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f6455e00640 1 -- 172.21.15.125:0/3566977546 learned_addr learned my addr 172.21.15.125:0/3566977546 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f645ea00640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 0x7f64600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27769038 0 0) 0x7f64600cfc30 con 0x7f64600cf7d0 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.445+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003520 con 0x7f64600cf7d0 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3914529839 0 0) 0x7f6440003520 con 0x7f64600cf7d0 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 msgr2=0x7f64600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 0x7f64600cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 >> v1:172.21.15.137:6789/0 conn(0x7f646015e3b0 legacy=0x7f64600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646015e990 con 0x7f64600cf7d0 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f6448001670 con 0x7f64600cf7d0 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 >> v1:172.21.15.125:6789/0 conn(0x7f64600cf7d0 legacy=0x7f646015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64400039a0 0x7f6440003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 0x7f64400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6440004d00 0x7f6440005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64600b2780 con 0x7f64400039a0 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644000ba30 con 0x7f6440004d00 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644000bbf0 con 0x7f6460143ae0 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.446+0000 7f645ea00640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64400039a0 0x7f6440003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645ea00640 1 -- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 msgr2=0x7f64400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645ea00640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6460143ae0 0x7f64400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645ea00640 1 -- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6440004d00 msgr2=0x7f6440005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645f400640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6440004d00 0x7f6440005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645ea00640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6440004d00 0x7f6440005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.447+0000 7f645ea00640 1 -- 172.21.15.125:0/3566977546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440003520 con 0x7f64400039a0 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.448+0000 7f645ea00640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64400039a0 0x7f6440003dc0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6450009a90 tx=0x7f64400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb1039c5e07e8bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.448+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645002c6b0 con 0x7f64400039a0 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.448+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645002c810 con 0x7f64400039a0 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.448+0000 7f645e000640 1 -- 172.21.15.125:0/3566977546 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645002cba0 con 0x7f64400039a0 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 -- 172.21.15.125:0/3566977546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64400039a0 msgr2=0x7f6440003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 --2- 172.21.15.125:0/3566977546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64400039a0 0x7f6440003dc0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6450009a90 tx=0x7f64400057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f645f400640 1 -- 172.21.15.125:0/3566977546 reap_dead start 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 -- 172.21.15.125:0/3566977546 shutdown_connections 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 -- 172.21.15.125:0/3566977546 >> 172.21.15.125:0/3566977546 conn(0x7f646006a800 msgr2=0x7f646006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:25.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 -- 172.21.15.125:0/3566977546 shutdown_connections 2024-01-16T15:12:25.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.449+0000 7f6465800640 1 -- 172.21.15.125:0/3566977546 wait complete. 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.451+0000 7f6465800640 1 Processor -- start 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.451+0000 7f6465800640 1 -- start start 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64400039a0 0x7f646015ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646013c0d0 0x7f646015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646013b7c0 con 0x7f646013bc50 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646015b200 con 0x7f646013c0d0 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6465800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646013b490 con 0x7f64400039a0 2024-01-16T15:12:25.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39094/0 (socket says 172.21.15.125:39094) 2024-01-16T15:12:25.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 -- 172.21.15.125:0/1232009494 learned_addr learned my addr 172.21.15.125:0/1232009494 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f645f400640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646013c0d0 0x7f646015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f645ea00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64400039a0 0x7f646015ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 -- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64400039a0 msgr2=0x7f646015ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64400039a0 0x7f646015ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 -- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646013c0d0 msgr2=0x7f646015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646013c0d0 0x7f646015acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.452+0000 7f6455e00640 1 -- 172.21.15.125:0/1232009494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64600aebc0 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6455e00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6448001eb0 tx=0x7f644800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64480074e0 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6448007640 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64600aeea0 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448004a20 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.453+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64600af180 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.454+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64600675d0 con 0x7f646013bc50 2024-01-16T15:12:25.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.455+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 36) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f6448004b80 con 0x7f646013bc50 2024-01-16T15:12:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.456+0000 7f6457400640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 0x7f643806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.456+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6448087e00 con 0x7f646013bc50 2024-01-16T15:12:25.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.456+0000 7f645ea00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 0x7f643806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.457+0000 7f645ea00640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 0x7f643806e800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6450036000 tx=0x7f6450030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.437 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.461+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64480543a0 con 0x7f646013bc50 2024-01-16T15:12:25.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:25.691+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone57941", "target": ["mon-mgr", ""]}) v1 -- 0x7f64600b2410 con 0x7f643806c340 2024-01-16T15:12:30.185 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:30.208+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:30.186 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:30.208+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:30.186 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:30.208+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:30.186 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:30.208+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:30.186 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:12:30.208+0000 7f0102800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:30.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.379+0000 7f6457400640 1 -- 172.21.15.125:0/1232009494 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6450004020 con 0x7f643806c340 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 msgr2=0x7f643806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 0x7f643806e800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6450036000 tx=0x7f6450030000 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 msgr2=0x7f646015a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6448001eb0 tx=0x7f644800cc00 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 shutdown_connections 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646013c0d0 0x7f646015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f643806c340 0x7f643806e800 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f646013bc50 0x7f646015a780 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 --2- 172.21.15.125:0/1232009494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64400039a0 0x7f646015ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.382+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 >> 172.21.15.125:0/1232009494 conn(0x7f646006a800 msgr2=0x7f64600cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:30.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.383+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 shutdown_connections 2024-01-16T15:12:30.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:30.383+0000 7f6465800640 1 -- 172.21.15.125:0/1232009494 wait complete. 2024-01-16T15:12:32.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone57942 2024-01-16T15:12:32.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.621+0000 7f01cd000640 1 Processor -- start 2024-01-16T15:12:32.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.622+0000 7f01cd000640 1 -- start start 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.623+0000 7f01cd000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01c80c4700 0x7f01c80b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.623+0000 7f01cd000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c806bd90 con 0x7f01c8161a80 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.623+0000 7f01cd000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c80adcd0 con 0x7f01c80b2620 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.623+0000 7f01cd000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c8161e60 con 0x7f01c80c4700 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01c80c4700 0x7f01c80b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01c8161a80 0x7f01c80c3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52382/0 (socket says 172.21.15.125:52382) 2024-01-16T15:12:32.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5e00640 1 -- 172.21.15.125:0/1098984329 learned_addr learned my addr 172.21.15.125:0/1098984329 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:12:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5400640 1 -- 172.21.15.125:0/1098984329 >> v1:172.21.15.137:6789/0 conn(0x7f01c80b2620 legacy=0x7f01c80b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5400640 1 -- 172.21.15.125:0/1098984329 >> v1:172.21.15.125:6789/0 conn(0x7f01c8161a80 legacy=0x7f01c80c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c5400640 1 -- 172.21.15.125:0/1098984329 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c80d1e70 con 0x7f01c80c4700 2024-01-16T15:12:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.624+0000 7f01c4a00640 1 -- 172.21.15.125:0/1098984329 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208580397 0 0) 0x7f01c806bd90 con 0x7f01c8161a80 2024-01-16T15:12:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.625+0000 7f01c5400640 1 --2- 172.21.15.125:0/1098984329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80c4700 0x7f01c80b0400 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f01b00097a0 tx=0x7f01b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e462ac8882f54f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01c4a00640 1 -- 172.21.15.125:0/1098984329 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b002e070 con 0x7f01c80c4700 2024-01-16T15:12:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01c4a00640 1 -- 172.21.15.125:0/1098984329 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00315a0 con 0x7f01c80c4700 2024-01-16T15:12:32.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01c4a00640 1 -- 172.21.15.125:0/1098984329 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0031930 con 0x7f01c80c4700 2024-01-16T15:12:32.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 -- 172.21.15.125:0/1098984329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80c4700 msgr2=0x7f01c80b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:32.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 --2- 172.21.15.125:0/1098984329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80c4700 0x7f01c80b0400 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f01b00097a0 tx=0x7f01b002b840 comp rx=0 tx=0).stop 2024-01-16T15:12:32.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 -- 172.21.15.125:0/1098984329 shutdown_connections 2024-01-16T15:12:32.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 --2- 172.21.15.125:0/1098984329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80c4700 0x7f01c80b0400 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:32.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 -- 172.21.15.125:0/1098984329 >> 172.21.15.125:0/1098984329 conn(0x7f01c80d1020 msgr2=0x7f01c80d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:32.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.626+0000 7f01cd000640 1 -- 172.21.15.125:0/1098984329 shutdown_connections 2024-01-16T15:12:32.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.627+0000 7f01cd000640 1 -- 172.21.15.125:0/1098984329 wait complete. 2024-01-16T15:12:32.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.628+0000 7f01cd000640 1 Processor -- start 2024-01-16T15:12:32.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.628+0000 7f01cd000640 1 -- start start 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01c80c4700 0x7f01c814e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01c8161a80 0x7f01c80c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80c8850 con 0x7f01c80b2620 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80c61b0 con 0x7f01c80c4700 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.629+0000 7f01cd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80c5ed0 con 0x7f01c8161a80 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54628/0 (socket says 172.21.15.125:54628) 2024-01-16T15:12:32.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 -- 172.21.15.125:0/1836231694 learned_addr learned my addr 172.21.15.125:0/1836231694 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c6800640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01c8161a80 0x7f01c80c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5400640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01c80c4700 0x7f01c814e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 -- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01c8161a80 msgr2=0x7f01c80c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01c8161a80 0x7f01c80c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 -- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01c80c4700 msgr2=0x7f01c814e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01c80c4700 0x7f01c814e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.630+0000 7f01c5e00640 1 -- 172.21.15.125:0/1836231694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009400 con 0x7f01c80b2620 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01c5e00640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cc10 tx=0x7f01b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b800ce20 con 0x7f01c80b2620 2024-01-16T15:12:32.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8012400 con 0x7f01c80b2620 2024-01-16T15:12:32.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80127b0 con 0x7f01c80b2620 2024-01-16T15:12:32.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c80c8190 con 0x7f01c80b2620 2024-01-16T15:12:32.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.631+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c814e830 con 0x7f01c80b2620 2024-01-16T15:12:32.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.632+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c80c8990 con 0x7f01c80b2620 2024-01-16T15:12:32.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.633+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f01b8007070 con 0x7f01c80b2620 2024-01-16T15:12:32.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.634+0000 7f01be000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 0x7f01a006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:32.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.635+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01b8088b80 con 0x7f01c80b2620 2024-01-16T15:12:32.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.635+0000 7f01c5400640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 0x7f01a006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:32.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.635+0000 7f01c5400640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 0x7f01a006e860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f01b0036000 tx=0x7f01b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:32.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.640+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01c80c8990 con 0x7f01c80b2620 2024-01-16T15:12:32.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:12:32.873+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone57942", "target": ["mon-mgr", ""]}) v1 -- 0x7f01c806bab0 con 0x7f01a006c3a0 2024-01-16T15:12:41.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:41.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:11.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:11.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:13:29.410+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f01b80554d0 con 0x7f01c80b2620 2024-01-16T15:13:41.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:41.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:12.040 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:12.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.651+0000 7f01be000640 1 -- 172.21.15.125:0/1836231694 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f01b0004420 con 0x7f01a006c3a0 2024-01-16T15:14:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.654+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 msgr2=0x7f01a006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.654+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 0x7f01a006e860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f01b0036000 tx=0x7f01b0008030 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.654+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 msgr2=0x7f01c80cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.654+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cc10 tx=0x7f01b800a430 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 shutdown_connections 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01c8161a80 0x7f01c80c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01c80c4700 0x7f01c814e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f01a006c3a0 0x7f01a006e860 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 --2- 172.21.15.125:0/1836231694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01c80b2620 0x7f01c80cc620 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 >> 172.21.15.125:0/1836231694 conn(0x7f01c80d1020 msgr2=0x7f01c8160080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 shutdown_connections 2024-01-16T15:14:30.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:30.655+0000 7f01cd000640 1 -- 172.21.15.125:0/1836231694 wait complete. 2024-01-16T15:14:32.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone57943 2024-01-16T15:14:32.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.895+0000 7f08be800640 1 Processor -- start 2024-01-16T15:14:32.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.895+0000 7f08be800640 1 -- start start 2024-01-16T15:14:32.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08be800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f08b80ce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08be800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b815ed70 con 0x7f08b806e700 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08be800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b815f050 con 0x7f08b8144b20 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08be800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b815f330 con 0x7f08b806aaf0 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08bca00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f08b806e700 0x7f08b806a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48898/0 (socket says 172.21.15.125:48898) 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08bca00640 1 -- 172.21.15.125:0/2009830713 learned_addr learned my addr 172.21.15.125:0/2009830713 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.897+0000 7f08bde00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f08b80ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799107469 0 0) 0x7f08b815ed70 con 0x7f08b806e700 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0898003520 con 0x7f08b806e700 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582312350 0 0) 0x7f0898003520 con 0x7f08b806e700 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 msgr2=0x7f08b80ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f08b80ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 >> v1:172.21.15.137:6789/0 conn(0x7f08b8144b20 legacy=0x7f08b8071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b815f610 con 0x7f08b806e700 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f08a0001670 con 0x7f08b806e700 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 >> v1:172.21.15.125:6789/0 conn(0x7f08b806e700 legacy=0x7f08b806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08980039a0 0x7f0898003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980047c0 0x7f0898004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f0898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b815f330 con 0x7f08980039a0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089800ba60 con 0x7f08980047c0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089800bc20 con 0x7f08b806aaf0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.898+0000 7f08bd400640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08980039a0 0x7f0898003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bde00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f0898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bca00640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980047c0 0x7f0898004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bd400640 1 -- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 msgr2=0x7f0898005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bd400640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b806aaf0 0x7f0898005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bd400640 1 -- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980047c0 msgr2=0x7f0898004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bd400640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980047c0 0x7f0898004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.899+0000 7f08bd400640 1 -- 172.21.15.125:0/2009830713 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898003520 con 0x7f08980039a0 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.900+0000 7f08bd400640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08980039a0 0x7f0898003dc0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f08a8001720 tx=0x7f08980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ede9f60b80434b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.900+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a80052d0 con 0x7f08980039a0 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.900+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a802e050 con 0x7f08980039a0 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.900+0000 7f08b7e00640 1 -- 172.21.15.125:0/2009830713 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a80035f0 con 0x7f08980039a0 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 -- 172.21.15.125:0/2009830713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08980039a0 msgr2=0x7f0898003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 --2- 172.21.15.125:0/2009830713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08980039a0 0x7f0898003dc0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f08a8001720 tx=0x7f08980057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08bde00640 1 -- 172.21.15.125:0/2009830713 reap_dead start 2024-01-16T15:14:32.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 -- 172.21.15.125:0/2009830713 shutdown_connections 2024-01-16T15:14:32.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 -- 172.21.15.125:0/2009830713 >> 172.21.15.125:0/2009830713 conn(0x7f08b806b4f0 msgr2=0x7f08b806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:32.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 -- 172.21.15.125:0/2009830713 shutdown_connections 2024-01-16T15:14:32.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.901+0000 7f08be800640 1 -- 172.21.15.125:0/2009830713 wait complete. 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.903+0000 7f08be800640 1 Processor -- start 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.903+0000 7f08be800640 1 -- start start 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 0x7f08b81414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b80cb260 0x7f08b814a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b814ba30 con 0x7f08b814a6a0 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b8141a10 con 0x7f08980039a0 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08be800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b814ab20 con 0x7f08b80cb260 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08bde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50298/0 (socket says 172.21.15.125:50298) 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08bde00640 1 -- 172.21.15.125:0/353478186 learned_addr learned my addr 172.21.15.125:0/353478186 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:32.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.904+0000 7f08bd400640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 0x7f08b81414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 -- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b80cb260 msgr2=0x7f08b814a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bca00640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b80cb260 0x7f08b814a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b80cb260 0x7f08b814a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 -- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 msgr2=0x7f08b81414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 0x7f08b81414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 -- 172.21.15.125:0/353478186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b814adb0 con 0x7f08b814a6a0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bd400640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 0x7f08b81414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08bde00640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f08b815eb50 tx=0x7f08b00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000e070 con 0x7f08b814a6a0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b0011ed0 con 0x7f08b814a6a0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b814b090 con 0x7f08b814a6a0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0003990 con 0x7f08b814a6a0 2024-01-16T15:14:32.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.905+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b814b650 con 0x7f08b814a6a0 2024-01-16T15:14:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.906+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b814b810 con 0x7f08b814a6a0 2024-01-16T15:14:32.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.907+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f08b0003b10 con 0x7f08b814a6a0 2024-01-16T15:14:32.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.909+0000 7f08b5600640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 0x7f089006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:32.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.909+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08b00890a0 con 0x7f08b814a6a0 2024-01-16T15:14:32.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.909+0000 7f08bd400640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 0x7f089006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:32.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.910+0000 7f08bd400640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 0x7f089006e7d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f08a8033000 tx=0x7f08a8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:32.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:32.914+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08b0055640 con 0x7f08b814a6a0 2024-01-16T15:14:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:33.141+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone57943", "target": ["mon-mgr", ""]}) v1 -- 0x7f08b814bb70 con 0x7f089006c310 2024-01-16T15:14:35.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.617+0000 7f08b5600640 1 -- 172.21.15.125:0/353478186 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f08b814bb70 con 0x7f089006c310 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.620+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 msgr2=0x7f089006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.620+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 0x7f089006e7d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f08a8033000 tx=0x7f08a8009030 comp rx=0 tx=0).stop 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 msgr2=0x7f08b814be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f08b815eb50 tx=0x7f08b00115a0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 shutdown_connections 2024-01-16T15:14:35.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f089006c310 0x7f089006e7d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08b814a6a0 0x7f08b814be50 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08b80cb260 0x7f08b814a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 --2- 172.21.15.125:0/353478186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08980039a0 0x7f08b81414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 >> 172.21.15.125:0/353478186 conn(0x7f08b806b4f0 msgr2=0x7f08b80b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 shutdown_connections 2024-01-16T15:14:35.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:35.621+0000 7f08be800640 1 -- 172.21.15.125:0/353478186 wait complete. 2024-01-16T15:14:37.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone57941 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.867+0000 7f8f89800640 1 Processor -- start 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.867+0000 7f8f89800640 1 -- start start 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.868+0000 7f8f89800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f84071290 0x7f8f84071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.868+0000 7f8f89800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.868+0000 7f8f89800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f84071bb0 con 0x7f8f8415eb60 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.868+0000 7f8f89800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840ccb10 con 0x7f8f84071290 2024-01-16T15:14:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.868+0000 7f8f89800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415f620 con 0x7f8f8415f010 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f84071290 0x7f8f84071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f8415eb60 0x7f8f840cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43968/0 (socket says 172.21.15.125:43968) 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83400640 1 -- 172.21.15.125:0/746911272 learned_addr learned my addr 172.21.15.125:0/746911272 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f88e00640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83e00640 1 -- 172.21.15.125:0/746911272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 msgr2=0x7f8f840b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83e00640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83e00640 1 -- 172.21.15.125:0/746911272 >> v1:172.21.15.125:6789/0 conn(0x7f8f8415eb60 legacy=0x7f8f840cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f83e00640 1 -- 172.21.15.125:0/746911272 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8415f900 con 0x7f8f84071290 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f82a00640 1 -- 172.21.15.125:0/746911272 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118155716 0 0) 0x7f8f84071bb0 con 0x7f8f8415eb60 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.869+0000 7f8f88e00640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.870+0000 7f8f83e00640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f84071670 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f8f740097a0 tx=0x7f8f7402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee423dc0c4d491d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f82a00640 1 -- 172.21.15.125:0/746911272 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7402d040 con 0x7f8f84071290 2024-01-16T15:14:37.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f82a00640 1 -- 172.21.15.125:0/746911272 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74031810 con 0x7f8f84071290 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f82a00640 1 -- 172.21.15.125:0/746911272 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74031ba0 con 0x7f8f84071290 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 -- 172.21.15.125:0/746911272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 msgr2=0x7f8f84071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f84071670 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f8f740097a0 tx=0x7f8f7402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 -- 172.21.15.125:0/746911272 shutdown_connections 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 --2- 172.21.15.125:0/746911272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f84071670 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 -- 172.21.15.125:0/746911272 >> 172.21.15.125:0/746911272 conn(0x7f8f840b0ed0 msgr2=0x7f8f840cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 -- 172.21.15.125:0/746911272 shutdown_connections 2024-01-16T15:14:37.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.871+0000 7f8f89800640 1 -- 172.21.15.125:0/746911272 wait complete. 2024-01-16T15:14:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.873+0000 7f8f89800640 1 Processor -- start 2024-01-16T15:14:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.873+0000 7f8f89800640 1 -- start start 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8415eb60 0x7f8f840acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840ae480 con 0x7f8f84071290 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840ad800 con 0x7f8f8415eb60 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f89800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840adae0 con 0x7f8f8415f010 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f83e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f83e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38304/0 (socket says 172.21.15.125:38304) 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.874+0000 7f8f83e00640 1 -- 172.21.15.125:0/88959177 learned_addr learned my addr 172.21.15.125:0/88959177 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83400640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8415eb60 0x7f8f840acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 -- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 msgr2=0x7f8f840ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f88e00640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 -- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8415eb60 msgr2=0x7f8f840acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8415eb60 0x7f8f840acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 -- 172.21.15.125:0/88959177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74009400 con 0x7f8f84071290 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f83e00640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f8f740098c0 tx=0x7f8f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.875+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7403ad60 con 0x7f8f84071290 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.876+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74044400 con 0x7f8f84071290 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.876+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740447b0 con 0x7f8f84071290 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.876+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f840addc0 con 0x7f8f84071290 2024-01-16T15:14:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.876+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f841f0e00 con 0x7f8f84071290 2024-01-16T15:14:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.877+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f840b20e0 con 0x7f8f84071290 2024-01-16T15:14:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.878+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f8f7404d440 con 0x7f8f84071290 2024-01-16T15:14:37.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.879+0000 7f8f7be00640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 0x7f8f5c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.879+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f740aaf10 con 0x7f8f84071290 2024-01-16T15:14:37.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.879+0000 7f8f83400640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 0x7f8f5c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.880+0000 7f8f83400640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 0x7f8f5c06e7d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c007fd0 tx=0x7f8f6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:37.884+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f740774b0 con 0x7f8f84071290 2024-01-16T15:14:38.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.110+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone57941", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f840cd000 con 0x7f8f5c06c310 2024-01-16T15:14:38.102 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:38.119+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:38.102 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:38.119+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:38.102 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:38.119+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:38.102 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:38.119+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:38.102 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:38.119+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:38.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.136+0000 7f8f7be00640 1 -- 172.21.15.125:0/88959177 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f8f6c001350 con 0x7f8f5c06c310 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:14:38.119 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:14:38.120 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:38.120 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:38.120 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 msgr2=0x7f8f5c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 0x7f8f5c06e7d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c007fd0 tx=0x7f8f6c009040 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 msgr2=0x7f8f840aac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f8f740098c0 tx=0x7f8f740046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 shutdown_connections 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f8415f010 0x7f8f840ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8415eb60 0x7f8f840acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8f5c06c310 0x7f8f5c06e7d0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 --2- 172.21.15.125:0/88959177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f84071290 0x7f8f840aac00 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 >> 172.21.15.125:0/88959177 conn(0x7f8f840b0ed0 msgr2=0x7f8f8415de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.138+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 shutdown_connections 2024-01-16T15:14:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.139+0000 7f8f89800640 1 -- 172.21.15.125:0/88959177 wait complete. 2024-01-16T15:14:38.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone57942 2024-01-16T15:14:38.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.347+0000 7fca87e00640 1 Processor -- start 2024-01-16T15:14:38.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.347+0000 7fca87e00640 1 -- start start 2024-01-16T15:14:38.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.348+0000 7fca87e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fca880cfde0 0x7fca880b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.348+0000 7fca87e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880d1c60 con 0x7fca8815d4b0 2024-01-16T15:14:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.348+0000 7fca87e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880b2730 con 0x7fca880cf9a0 2024-01-16T15:14:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.348+0000 7fca87e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca880d1ec0 con 0x7fca880cfde0 2024-01-16T15:14:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.349+0000 7fca86a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fca8815d4b0 0x7fca880d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43974/0 (socket says 172.21.15.125:43974) 2024-01-16T15:14:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.349+0000 7fca86a00640 1 -- 172.21.15.125:0/3600315439 learned_addr learned my addr 172.21.15.125:0/3600315439 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.349+0000 7fca87400640 1 --2- 172.21.15.125:0/3600315439 >> v2:172.21.15.137:3300/0 conn(0x7fca880cfde0 0x7fca880b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.349+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463856384 0 0) 0x7fca880d1c60 con 0x7fca8815d4b0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.349+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68003520 con 0x7fca8815d4b0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464072791 0 0) 0x7fca68003520 con 0x7fca8815d4b0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 >> v2:172.21.15.137:3300/0 conn(0x7fca880cfde0 msgr2=0x7fca880b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 --2- 172.21.15.125:0/3600315439 >> v2:172.21.15.137:3300/0 conn(0x7fca880cfde0 0x7fca880b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 >> v1:172.21.15.137:6789/0 conn(0x7fca880cf9a0 legacy=0x7fca880b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880b29d0 con 0x7fca8815d4b0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fca78001670 con 0x7fca8815d4b0 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 >> v1:172.21.15.125:6789/0 conn(0x7fca8815d4b0 legacy=0x7fca880d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca680047c0 0x7fca68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 0x7fca6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca880d1ec0 con 0x7fca68005180 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6800c080 con 0x7fca680039c0 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6800c240 con 0x7fca680047c0 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca87400640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 0x7fca6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca86a00640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca87400640 1 -- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca680047c0 msgr2=0x7fca68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca87400640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca680047c0 0x7fca68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca87400640 1 -- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca680039c0 msgr2=0x7fca68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.350+0000 7fca87400640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.351+0000 7fca87400640 1 -- 172.21.15.125:0/3600315439 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68003520 con 0x7fca68005180 2024-01-16T15:14:38.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.351+0000 7fca87400640 1 -- 172.21.15.125:0/3600315439 reap_dead start 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca87400640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 0x7fca6800b980 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fca80009fc0 tx=0x7fca680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ed58ddd718c475 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8002cc60 con 0x7fca68005180 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8000f040 con 0x7fca68005180 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca86000640 1 -- 172.21.15.125:0/3600315439 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80003990 con 0x7fca68005180 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca87e00640 1 -- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 msgr2=0x7fca6800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.352+0000 7fca87e00640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 0x7fca6800b980 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fca80009fc0 tx=0x7fca680057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.353+0000 7fca87e00640 1 -- 172.21.15.125:0/3600315439 shutdown_connections 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.353+0000 7fca87e00640 1 --2- 172.21.15.125:0/3600315439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca68005180 0x7fca6800b980 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.353+0000 7fca87e00640 1 -- 172.21.15.125:0/3600315439 >> 172.21.15.125:0/3600315439 conn(0x7fca880d3000 msgr2=0x7fca880d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.353+0000 7fca87e00640 1 -- 172.21.15.125:0/3600315439 shutdown_connections 2024-01-16T15:14:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.353+0000 7fca87e00640 1 -- 172.21.15.125:0/3600315439 wait complete. 2024-01-16T15:14:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.354+0000 7fca87e00640 1 Processor -- start 2024-01-16T15:14:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.355+0000 7fca87e00640 1 -- start start 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.355+0000 7fca87e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca68005180 0x7fca880a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca8814be70 0x7fca8814e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca880a0e10 con 0x7fca8814d480 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8815c230 con 0x7fca8814be70 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8814cff0 con 0x7fca68005180 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38322/0 (socket says 172.21.15.125:38322) 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 -- 172.21.15.125:0/4247054700 learned_addr learned my addr 172.21.15.125:0/4247054700 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca86a00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca68005180 0x7fca880a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 -- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca68005180 msgr2=0x7fca880a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca68005180 0x7fca880a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca7fe00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca8814be70 0x7fca8814e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 -- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca8814be70 msgr2=0x7fca8814e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca8814be70 0x7fca8814e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.356+0000 7fca87400640 1 -- 172.21.15.125:0/4247054700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8814ee50 con 0x7fca8814d480 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca87400640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fca8002c6d0 tx=0x7fca80009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80035d60 con 0x7fca8814d480 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca80035ee0 con 0x7fca8814d480 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80003bb0 con 0x7fca8814d480 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8814c3e0 con 0x7fca8814d480 2024-01-16T15:14:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.357+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8814c9a0 con 0x7fca8814d480 2024-01-16T15:14:38.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.358+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca88067450 con 0x7fca8814d480 2024-01-16T15:14:38.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.359+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fca80003d10 con 0x7fca8814d480 2024-01-16T15:14:38.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.360+0000 7fca7ea00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 0x7fca6006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.360+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca800ab6f0 con 0x7fca8814d480 2024-01-16T15:14:38.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.360+0000 7fca86a00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 0x7fca6006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.361+0000 7fca86a00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 0x7fca6006e7d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fca780016d0 tx=0x7fca78000f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.365+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca80077c90 con 0x7fca8814d480 2024-01-16T15:14:38.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.594+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone57942", "target": ["mon-mgr", ""]}) v1 -- 0x7fca880d08e0 con 0x7fca6006c310 2024-01-16T15:14:38.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.616+0000 7fca7ea00640 1 -- 172.21.15.125:0/4247054700 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fca880d08e0 con 0x7fca6006c310 2024-01-16T15:14:38.599 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:14:38.599 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:38.600 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 msgr2=0x7fca6006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 0x7fca6006e7d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fca780016d0 tx=0x7fca78000f30 comp rx=0 tx=0).stop 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 msgr2=0x7fca8814d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fca8002c6d0 tx=0x7fca80009750 comp rx=0 tx=0).stop 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 shutdown_connections 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fca6006c310 0x7fca6006e7d0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca8814d480 0x7fca8814d900 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca8814be70 0x7fca8814e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 --2- 172.21.15.125:0/4247054700 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca68005180 0x7fca880a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 >> 172.21.15.125:0/4247054700 conn(0x7fca880d3000 msgr2=0x7fca8815c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 shutdown_connections 2024-01-16T15:14:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.619+0000 7fca87e00640 1 -- 172.21.15.125:0/4247054700 wait complete. 2024-01-16T15:14:38.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone57943 2024-01-16T15:14:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.835+0000 7f5cf6400640 1 Processor -- start 2024-01-16T15:14:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.836+0000 7f5cf6400640 1 -- start start 2024-01-16T15:14:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cf6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5cf00ce8f0 0x7f5cf00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cf6400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cf00b2a00 0x7f5cf00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cf6400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf00d0de0 con 0x7f5cf0160f80 2024-01-16T15:14:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cf6400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf00d10c0 con 0x7f5cf00b2a00 2024-01-16T15:14:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cf6400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf00d13a0 con 0x7f5cf00ce8f0 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cf00b2a00 0x7f5cf00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cf00b2a00 0x7f5cf00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38324/0 (socket says 172.21.15.125:38324) 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 -- 172.21.15.125:0/909420183 learned_addr learned my addr 172.21.15.125:0/909420183 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5ce6800640 1 --2- 172.21.15.125:0/909420183 >> v2:172.21.15.137:3300/0 conn(0x7f5cf00ce8f0 0x7f5cf00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 -- 172.21.15.125:0/909420183 >> v2:172.21.15.137:3300/0 conn(0x7f5cf00ce8f0 msgr2=0x7f5cf00ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 --2- 172.21.15.125:0/909420183 >> v2:172.21.15.137:3300/0 conn(0x7f5cf00ce8f0 0x7f5cf00ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 -- 172.21.15.125:0/909420183 >> v1:172.21.15.137:6789/0 conn(0x7f5cf0160f80 legacy=0x7f5cf0161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.837+0000 7f5cefe00640 1 -- 172.21.15.125:0/909420183 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf00d1680 con 0x7f5cf00b2a00 2024-01-16T15:14:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.838+0000 7f5cefe00640 1 --2- 172.21.15.125:0/909420183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf00b2a00 0x7f5cf00b2e90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00b5d10 tx=0x7f5ce802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b1b5723f3ad330 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5ceea00640 1 -- 172.21.15.125:0/909420183 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce802ca80 con 0x7f5cf00b2a00 2024-01-16T15:14:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5ceea00640 1 -- 172.21.15.125:0/909420183 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce802cbe0 con 0x7f5cf00b2a00 2024-01-16T15:14:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5ceea00640 1 -- 172.21.15.125:0/909420183 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80343f0 con 0x7f5cf00b2a00 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 -- 172.21.15.125:0/909420183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf00b2a00 msgr2=0x7f5cf00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 --2- 172.21.15.125:0/909420183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf00b2a00 0x7f5cf00b2e90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00b5d10 tx=0x7f5ce802bee0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 -- 172.21.15.125:0/909420183 shutdown_connections 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 --2- 172.21.15.125:0/909420183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf00b2a00 0x7f5cf00b2e90 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 --2- 172.21.15.125:0/909420183 >> v2:172.21.15.137:3300/0 conn(0x7f5cf00ce8f0 0x7f5cf00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 -- 172.21.15.125:0/909420183 >> 172.21.15.125:0/909420183 conn(0x7f5cf009c550 msgr2=0x7f5cf00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.839+0000 7f5cf6400640 1 -- 172.21.15.125:0/909420183 shutdown_connections 2024-01-16T15:14:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.840+0000 7f5cf6400640 1 -- 172.21.15.125:0/909420183 wait complete. 2024-01-16T15:14:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.841+0000 7f5cf6400640 1 Processor -- start 2024-01-16T15:14:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.841+0000 7f5cf6400640 1 -- start start 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cf00b2a00 0x7f5cf015e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cf00ce8f0 0x7f5cf00b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf0151240 con 0x7f5cf0160f80 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf015ed80 con 0x7f5cf00b2a00 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cf6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf015f060 con 0x7f5cf00ce8f0 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38336/0 (socket says 172.21.15.125:38336) 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.842+0000 7f5cefe00640 1 -- 172.21.15.125:0/1326726777 learned_addr learned my addr 172.21.15.125:0/1326726777 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cef400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cf00b2a00 0x7f5cf015e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5ce6800640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cf00ce8f0 0x7f5cf00b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 -- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cf00ce8f0 msgr2=0x7f5cf00b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cf00ce8f0 0x7f5cf00b5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 -- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cf00b2a00 msgr2=0x7f5cf015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cf00b2a00 0x7f5cf015e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 -- 172.21.15.125:0/1326726777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8009b30 con 0x7f5cf0160f80 2024-01-16T15:14:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cefe00640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8036040 tx=0x7f5ce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8009510 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce803e450 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce803e800 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.843+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf015f340 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.844+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf015f900 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.845+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf006cea0 con 0x7f5cf0160f80 2024-01-16T15:14:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.845+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f5ce802cd50 con 0x7f5cf0160f80 2024-01-16T15:14:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.846+0000 7f5ce7e00640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 0x7f5cc806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.846+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ce80ab650 con 0x7f5cf0160f80 2024-01-16T15:14:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.846+0000 7f5cef400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 0x7f5cc806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.847+0000 7f5cef400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 0x7f5cc806e7d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0004ea0 tx=0x7f5ce0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:38.851+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ce8077ca0 con 0x7f5cf0160f80 2024-01-16T15:14:39.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:39.074+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone57943", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cf00d1c40 con 0x7f5cc806c310 2024-01-16T15:14:39.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:39.421+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f5ce8078070 con 0x7f5cf0160f80 2024-01-16T15:14:40.175 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.192+0000 7f5ce7e00640 1 -- 172.21.15.125:0/1326726777 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f5ce0007db0 con 0x7f5cc806c310 2024-01-16T15:14:40.175 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:14:40.175 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:14:40.175 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "canceled", 2024-01-16T15:14:40.175 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "failure": { 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "errno": "4", 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:40.176 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 msgr2=0x7f5cc806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 0x7f5cc806e7d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0004ea0 tx=0x7f5ce0009040 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 msgr2=0x7f5cf00b6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8036040 tx=0x7f5ce8004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 shutdown_connections 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5cc806c310 0x7f5cc806e7d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cf0160f80 0x7f5cf00b6370 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5cf00ce8f0 0x7f5cf00b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 --2- 172.21.15.125:0/1326726777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5cf00b2a00 0x7f5cf015e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.195+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 >> 172.21.15.125:0/1326726777 conn(0x7f5cf009c550 msgr2=0x7f5cf0161360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.196+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 shutdown_connections 2024-01-16T15:14:40.179 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.196+0000 7f5cf6400640 1 -- 172.21.15.125:0/1326726777 wait complete. 2024-01-16T15:14:40.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:14:40.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.424+0000 7f47bfc00640 1 Processor -- start 2024-01-16T15:14:40.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.424+0000 7f47bfc00640 1 -- start start 2024-01-16T15:14:40.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.425+0000 7f47bfc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f47b8143ae0 0x7f47b80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.425+0000 7f47bfc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b80cfc30 con 0x7f47b80cf7d0 2024-01-16T15:14:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.425+0000 7f47bfc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b80b1ee0 con 0x7f47b815e3b0 2024-01-16T15:14:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.425+0000 7f47bfc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b80b2780 con 0x7f47b8143ae0 2024-01-16T15:14:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47afc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f47b80cf7d0 0x7f47b815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43982/0 (socket says 172.21.15.125:43982) 2024-01-16T15:14:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47afc00640 1 -- 172.21.15.125:0/973188574 learned_addr learned my addr 172.21.15.125:0/973188574 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47bd200640 1 --2- 172.21.15.125:0/973188574 >> v2:172.21.15.137:3300/0 conn(0x7f47b8143ae0 0x7f47b80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082286782 0 0) 0x7f47b80cfc30 con 0x7f47b80cf7d0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003520 con 0x7f47b80cf7d0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520281519 0 0) 0x7f4798003520 con 0x7f47b80cf7d0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 >> v2:172.21.15.137:3300/0 conn(0x7f47b8143ae0 msgr2=0x7f47b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 --2- 172.21.15.125:0/973188574 >> v2:172.21.15.137:3300/0 conn(0x7f47b8143ae0 0x7f47b80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 >> v1:172.21.15.137:6789/0 conn(0x7f47b815e3b0 legacy=0x7f47b80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b815e990 con 0x7f47b80cf7d0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.426+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f47a0001670 con 0x7f47b80cf7d0 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 >> v1:172.21.15.125:6789/0 conn(0x7f47b80cf7d0 legacy=0x7f47b815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4798003970 0x7f4798003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 0x7f4798004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4798005180 0x7f479800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80b2780 con 0x7f47980047c0 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479800c020 con 0x7f4798005180 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479800c1e0 con 0x7f4798003970 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 0x7f4798004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 -- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4798003970 msgr2=0x7f4798003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4798003970 0x7f4798003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 -- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4798005180 msgr2=0x7f479800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47bdc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4798005180 0x7f479800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4798005180 0x7f479800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47afc00640 1 -- 172.21.15.125:0/973188574 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798003520 con 0x7f47980047c0 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47bdc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4798005180 0x7f479800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.427+0000 7f47bdc00640 1 -- 172.21.15.125:0/973188574 reap_dead start 2024-01-16T15:14:40.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47afc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 0x7f4798004c40 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c190 tx=0x7f47980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bbfa9c199802e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0005ad0 con 0x7f47980047c0 2024-01-16T15:14:40.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a0005c50 con 0x7f47980047c0 2024-01-16T15:14:40.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47b7e00640 1 -- 172.21.15.125:0/973188574 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a002c8a0 con 0x7f47980047c0 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47bfc00640 1 -- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 msgr2=0x7f4798004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47bfc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 0x7f4798004c40 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c190 tx=0x7f47980057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47bfc00640 1 -- 172.21.15.125:0/973188574 shutdown_connections 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47bfc00640 1 --2- 172.21.15.125:0/973188574 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47980047c0 0x7f4798004c40 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.429+0000 7f47bfc00640 1 -- 172.21.15.125:0/973188574 >> 172.21.15.125:0/973188574 conn(0x7f47b806a800 msgr2=0x7f47b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.430+0000 7f47bfc00640 1 -- 172.21.15.125:0/973188574 shutdown_connections 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.430+0000 7f47bfc00640 1 -- 172.21.15.125:0/973188574 wait complete. 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.431+0000 7f47bfc00640 1 Processor -- start 2024-01-16T15:14:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.431+0000 7f47bfc00640 1 -- start start 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47980047c0 0x7f47b80a9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47b80aa2d0 0x7f47b8141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b81ee9d0 con 0x7f47b81f1fc0 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b81f2450 con 0x7f47b80aa2d0 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.432+0000 7f47bfc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b81f2730 con 0x7f47980047c0 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38364/0 (socket says 172.21.15.125:38364) 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 -- 172.21.15.125:0/3655310069 learned_addr learned my addr 172.21.15.125:0/3655310069 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 -- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47980047c0 msgr2=0x7f47b80a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47bd200640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47980047c0 0x7f47b80a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47bdc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47b80aa2d0 0x7f47b8141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47980047c0 0x7f47b80a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 -- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47b80aa2d0 msgr2=0x7f47b8141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47b80aa2d0 0x7f47b8141500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.433+0000 7f47afc00640 1 -- 172.21.15.125:0/3655310069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b81f2a10 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47afc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c190 tx=0x7f47b80b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0044070 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a003a050 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b81f2cf0 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a00049e0 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.434+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b8142450 con 0x7f47b81f1fc0 2024-01-16T15:14:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.435+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b81eeb10 con 0x7f47b81f1fc0 2024-01-16T15:14:40.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.436+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f47a0032070 con 0x7f47b81f1fc0 2024-01-16T15:14:40.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.437+0000 7f47b5600640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 0x7f479006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.437+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47a00aa010 con 0x7f47b81f1fc0 2024-01-16T15:14:40.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.437+0000 7f47bd200640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 0x7f479006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.438+0000 7f47bd200640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 0x7f479006e860 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f47b80cf600 tx=0x7f47a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.442+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47a0076a50 con 0x7f47b81f1fc0 2024-01-16T15:14:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.704+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f47b80675d0 con 0x7f47b81f1fc0 2024-01-16T15:14:40.687 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.704+0000 7f47b5600640 1 -- 172.21.15.125:0/3655310069 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1855 (secure 0 0 0) 0x7f47a007a070 con 0x7f47b81f1fc0 2024-01-16T15:14:40.687 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "health": { 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "HEALTH_OK", 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "checks": {}, 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "mutes": [] 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "election_epoch": 4, 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum": [ 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: 0, 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: 1, 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: 2 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:14:40.688 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_names": [ 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "a", 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "b", 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "c" 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "quorum_age": 1099, 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": { 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 1, 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "min_mon_release_name": "reef", 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "num_mons": 3 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap": { 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 100, 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "num_osds": 8, 2024-01-16T15:14:40.689 INFO:teuthology.orchestra.run.smithi125.stdout: "num_up_osds": 8, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_up_since": 1705416993, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "num_in_osds": 8, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_in_since": 1705416984, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "num_remapped_pgs": 0 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "pgmap": { 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "pgs_by_state": [ 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "state_name": "active+clean", 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "count": 97 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pgs": 97, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "num_pools": 3, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "num_objects": 799, 2024-01-16T15:14:40.690 INFO:teuthology.orchestra.run.smithi125.stdout: "data_bytes": 3222417561, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_used": 7012548608, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_avail": 766081564672, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "bytes_total": 773094113280, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "read_bytes_sec": 938, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "write_bytes_sec": 12454, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "read_op_per_sec": 0, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "write_op_per_sec": 3 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "fsmap": { 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 64, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 7, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "max": 1, 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: "by_rank": [ 2024-01-16T15:14:40.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "filesystem_id": 7, 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "rank": 0, 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "a", 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "status": "up:active", 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "gid": 8649 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "up:standby": 3 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "mgrmap": { 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "available": true, 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "num_standbys": 1, 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "modules": [ 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "iostat", 2024-01-16T15:14:40.692 INFO:teuthology.orchestra.run.smithi125.stdout: "nfs", 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "restful" 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "servicemap": { 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 19, 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2024-01-16T15:12:28.285720+0000", 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "services": {} 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout: "progress_events": {} 2024-01-16T15:14:40.693 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 msgr2=0x7f479006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 0x7f479006e860 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f47b80cf600 tx=0x7f47a8008040 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 msgr2=0x7f47b813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c190 tx=0x7f47b80b26f0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 shutdown_connections 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47b80aa2d0 0x7f47b8141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f479006c3a0 0x7f479006e860 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b81f1fc0 0x7f47b813f880 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 --2- 172.21.15.125:0/3655310069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47980047c0 0x7f47b80a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.707+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 >> 172.21.15.125:0/3655310069 conn(0x7f47b806a800 msgr2=0x7f47b8070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.708+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 shutdown_connections 2024-01-16T15:14:40.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.708+0000 7f47bfc00640 1 -- 172.21.15.125:0/3655310069 wait complete. 2024-01-16T15:14:40.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:40.710 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone57941/cc9616d3-bf68-4625-86b4-b99ce55d389a) 2024-01-16T15:14:40.768 INFO:teuthology.orchestra.run.smithi125.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone57941/cc9616d3-bf68-4625-86b4-b99ce55d389a': No such file or directory 2024-01-16T15:14:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:14:40.769 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:14:40.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.996+0000 7fac50a00640 1 Processor -- start 2024-01-16T15:14:40.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.996+0000 7fac50a00640 1 -- start start 2024-01-16T15:14:40.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.997+0000 7fac50a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac4c143c40 0x7fac4c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.997+0000 7fac50a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 0x7fac4c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac50a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac50a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c0723c0 con 0x7fac4c143c40 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac50a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c070e50 con 0x7fac4c0717d0 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac50a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0cce40 con 0x7fac4c0cca40 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac4c143c40 0x7fac4c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac4c143c40 0x7fac4c071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38380/0 (socket says 172.21.15.125:38380) 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 -- 172.21.15.125:0/2333635083 learned_addr learned my addr 172.21.15.125:0/2333635083 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac42800640 1 --2- 172.21.15.125:0/2333635083 >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 0x7fac4c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4be00640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 -- 172.21.15.125:0/2333635083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 msgr2=0x7fac4c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c15eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 -- 172.21.15.125:0/2333635083 >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 msgr2=0x7fac4c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 --2- 172.21.15.125:0/2333635083 >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 0x7fac4c0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac4b400640 1 -- 172.21.15.125:0/2333635083 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c144020 con 0x7fac4c143c40 2024-01-16T15:14:40.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.998+0000 7fac42800640 1 --2- 172.21.15.125:0/2333635083 >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 0x7fac4c0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:40.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:40.999+0000 7fac4b400640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c071290 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0b44a0 tx=0x7fac3c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26b2a9169c68d5b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac4aa00640 1 -- 172.21.15.125:0/2333635083 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0317d0 con 0x7fac4c143c40 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac4aa00640 1 -- 172.21.15.125:0/2333635083 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c031930 con 0x7fac4c143c40 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac4aa00640 1 -- 172.21.15.125:0/2333635083 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031cc0 con 0x7fac4c143c40 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 -- 172.21.15.125:0/2333635083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 msgr2=0x7fac4c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c071290 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0b44a0 tx=0x7fac3c02b870 comp rx=0 tx=0).stop 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 -- 172.21.15.125:0/2333635083 shutdown_connections 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 --2- 172.21.15.125:0/2333635083 >> v2:172.21.15.137:3300/0 conn(0x7fac4c0717d0 0x7fac4c0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.000+0000 7fac50a00640 1 --2- 172.21.15.125:0/2333635083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c071290 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.001+0000 7fac50a00640 1 -- 172.21.15.125:0/2333635083 >> 172.21.15.125:0/2333635083 conn(0x7fac4c06a800 msgr2=0x7fac4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.001+0000 7fac50a00640 1 -- 172.21.15.125:0/2333635083 shutdown_connections 2024-01-16T15:14:40.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.001+0000 7fac50a00640 1 -- 172.21.15.125:0/2333635083 wait complete. 2024-01-16T15:14:40.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.003+0000 7fac50a00640 1 Processor -- start 2024-01-16T15:14:40.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.003+0000 7fac50a00640 1 -- start start 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 0x7fac4c13c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0b2e50 con 0x7fac4c143c40 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0ada50 con 0x7fac4c0717d0 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac50a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0ace40 con 0x7fac4c0cca40 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.004+0000 7fac4be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38382/0 (socket says 172.21.15.125:38382) 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 -- 172.21.15.125:0/2571340226 learned_addr learned my addr 172.21.15.125:0/2571340226 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4b400640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 0x7fac4c13c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac42800640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 -- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 msgr2=0x7fac4c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 -- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 msgr2=0x7fac4c13c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 0x7fac4c13c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 -- 172.21.15.125:0/2571340226 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0094d0 con 0x7fac4c143c40 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4b400640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 0x7fac4c13c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac42800640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac4be00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fac4400cd70 tx=0x7fac4400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.005+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44003b90 con 0x7fac4c143c40 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.006+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44003cf0 con 0x7fac4c143c40 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.006+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44004240 con 0x7fac4c143c40 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.006+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c0ad120 con 0x7fac4c143c40 2024-01-16T15:14:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.006+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c0ad6e0 con 0x7fac4c143c40 2024-01-16T15:14:40.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.007+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c067450 con 0x7fac4c143c40 2024-01-16T15:14:40.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.008+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fac440043a0 con 0x7fac4c143c40 2024-01-16T15:14:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.009+0000 7fac43e00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 0x7fac2406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.009+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac44012070 con 0x7fac4c143c40 2024-01-16T15:14:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.009+0000 7fac4b400640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 0x7fac2406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.010+0000 7fac4b400640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 0x7fac2406e860 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.013+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac44055950 con 0x7fac4c143c40 2024-01-16T15:14:41.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.211+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fac4c06c960 con 0x7fac2406c3a0 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.214+0000 7fac43e00640 1 -- 172.21.15.125:0/2571340226 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fac3c031aa0 con 0x7fac2406c3a0 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:41.197 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 msgr2=0x7fac2406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 0x7fac2406e860 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03d040 comp rx=0 tx=0).stop 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 msgr2=0x7fac4c15a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fac4400cd70 tx=0x7fac4400a660 comp rx=0 tx=0).stop 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 shutdown_connections 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fac2406c3a0 0x7fac2406e860 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4c143c40 0x7fac4c15a5f0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac4c0cca40 0x7fac4c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 --2- 172.21.15.125:0/2571340226 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac4c0717d0 0x7fac4c13c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.216+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 >> 172.21.15.125:0/2571340226 conn(0x7fac4c06a800 msgr2=0x7fac4c06d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.217+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 shutdown_connections 2024-01-16T15:14:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.217+0000 7fac50a00640 1 -- 172.21.15.125:0/2571340226 wait complete. 2024-01-16T15:14:41.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:14:41.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.438+0000 7f1561400640 1 Processor -- start 2024-01-16T15:14:41.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.438+0000 7f1561400640 1 -- start start 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1561400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 0x7f155c0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1561400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f155c0ced60 0x7f155c0d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1561400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0d2170 con 0x7f155c0d1d20 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1561400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0d1390 con 0x7f155c0ced60 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1561400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0b3400 con 0x7f155c15d610 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f155c0ced60 0x7f155c0d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f155c0ced60 0x7f155c0d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38390/0 (socket says 172.21.15.125:38390) 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 -- 172.21.15.125:0/2321905502 learned_addr learned my addr 172.21.15.125:0/2321905502 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 -- 172.21.15.125:0/2321905502 >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 msgr2=0x7f155c0ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f155a800640 1 --2- 172.21.15.125:0/2321905502 >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 0x7f155c0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 --2- 172.21.15.125:0/2321905502 >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 0x7f155c0ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 -- 172.21.15.125:0/2321905502 >> v1:172.21.15.125:6789/0 conn(0x7f155c0d1d20 legacy=0x7f155c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f1553e00640 1 -- 172.21.15.125:0/2321905502 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0b27f0 con 0x7f155c0ced60 2024-01-16T15:14:41.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.440+0000 7f155a800640 1 --2- 172.21.15.125:0/2321905502 >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 0x7f155c0ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.441+0000 7f1553e00640 1 --2- 172.21.15.125:0/2321905502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c0ced60 0x7f155c0d17e0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f155c0b5dc0 tx=0x7f154402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175d5f7488d68ecb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.441+0000 7f1559e00640 1 -- 172.21.15.125:0/2321905502 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544030870 con 0x7f155c0ced60 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.441+0000 7f1559e00640 1 -- 172.21.15.125:0/2321905502 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15440309d0 con 0x7f155c0ced60 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.441+0000 7f1559e00640 1 -- 172.21.15.125:0/2321905502 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544030d60 con 0x7f155c0ced60 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 -- 172.21.15.125:0/2321905502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c0ced60 msgr2=0x7f155c0d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 --2- 172.21.15.125:0/2321905502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c0ced60 0x7f155c0d17e0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f155c0b5dc0 tx=0x7f154402b9b0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 -- 172.21.15.125:0/2321905502 shutdown_connections 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 --2- 172.21.15.125:0/2321905502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c0ced60 0x7f155c0d17e0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 --2- 172.21.15.125:0/2321905502 >> v2:172.21.15.137:3300/0 conn(0x7f155c15d610 0x7f155c0ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 -- 172.21.15.125:0/2321905502 >> 172.21.15.125:0/2321905502 conn(0x7f155c09c540 msgr2=0x7f155c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 -- 172.21.15.125:0/2321905502 shutdown_connections 2024-01-16T15:14:41.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.442+0000 7f1561400640 1 -- 172.21.15.125:0/2321905502 wait complete. 2024-01-16T15:14:41.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.443+0000 7f1561400640 1 Processor -- start 2024-01-16T15:14:41.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.444+0000 7f1561400640 1 -- start start 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.444+0000 7f1561400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f155c0ced60 0x7f155c1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.444+0000 7f1561400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f155c0d1d20 0x7f155c0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.444+0000 7f1561400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f1561400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b7680 con 0x7f155c15d610 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f1561400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b7960 con 0x7f155c0d1d20 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f1561400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b7c40 con 0x7f155c0ced60 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38392/0 (socket says 172.21.15.125:38392) 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 -- 172.21.15.125:0/1456669094 learned_addr learned my addr 172.21.15.125:0/1456669094 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f1553e00640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f155c0d1d20 0x7f155c0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155a800640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f155c0ced60 0x7f155c1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 -- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f155c0ced60 msgr2=0x7f155c1515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f155c0ced60 0x7f155c1515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 -- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f155c0d1d20 msgr2=0x7f155c0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f155c0d1d20 0x7f155c0ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 -- 172.21.15.125:0/1456669094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15440094d0 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.445+0000 7f155b200640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f155400c7a0 tx=0x7f155400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.446+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554003b70 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.446+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554003cf0 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.446+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554003ea0 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.446+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c0b7f20 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.446+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c151b30 con 0x7f155c15d610 2024-01-16T15:14:41.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.447+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c0b8440 con 0x7f155c15d610 2024-01-16T15:14:41.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.448+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f155400aa90 con 0x7f155c15d610 2024-01-16T15:14:41.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.449+0000 7f1552a00640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 0x7f153406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.449+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1554088760 con 0x7f155c15d610 2024-01-16T15:14:41.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.449+0000 7f155a800640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 0x7f153406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.450+0000 7f155a800640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 0x7f153406e860 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f154c008850 tx=0x7f154c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.454+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f155c0b8440 con 0x7f155c15d610 2024-01-16T15:14:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.671+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f155c151cc0 con 0x7f153406c3a0 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.682+0000 7f1552a00640 1 -- 172.21.15.125:0/1456669094 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+185 (secure 0 0 0) 0x7f155c151cc0 con 0x7f153406c3a0 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:41.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone57941" 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone57943" 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone57942" 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:41.666 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.684+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 msgr2=0x7f153406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.684+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 0x7f153406e860 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f154c008850 tx=0x7f154c008040 comp rx=0 tx=0).stop 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.684+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 msgr2=0x7f155c1f30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.684+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f155400c7a0 tx=0x7f155400cc60 comp rx=0 tx=0).stop 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 shutdown_connections 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f153406c3a0 0x7f153406e860 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c15d610 0x7f155c1f30d0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f155c0d1d20 0x7f155c0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 --2- 172.21.15.125:0/1456669094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f155c0ced60 0x7f155c1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 >> 172.21.15.125:0/1456669094 conn(0x7f155c09c540 msgr2=0x7f155c15fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 shutdown_connections 2024-01-16T15:14:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.685+0000 7f1561400640 1 -- 172.21.15.125:0/1456669094 wait complete. 2024-01-16T15:14:41.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:14:41.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.904+0000 7fc85b600640 1 Processor -- start 2024-01-16T15:14:41.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.904+0000 7fc85b600640 1 -- start start 2024-01-16T15:14:41.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc85b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8540cf9a0 0x7fc85415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc85b600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc85415e1e0 0x7fc8540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc85b600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8540b1ee0 con 0x7fc854143ae0 2024-01-16T15:14:41.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc85b600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8540b2780 con 0x7fc85415e1e0 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc85b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85415e990 con 0x7fc8540cf9a0 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc859600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc85415e1e0 0x7fc8540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc859600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc85415e1e0 0x7fc8540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38396/0 (socket says 172.21.15.125:38396) 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc859600640 1 -- 172.21.15.125:0/1860722653 learned_addr learned my addr 172.21.15.125:0/1860722653 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.905+0000 7fc858c00640 1 --1- 172.21.15.125:0/1860722653 >> v1:172.21.15.125:6789/0 conn(0x7fc854143ae0 0x7fc8540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44002/0 (socket says 172.21.15.125:44002) 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.906+0000 7fc84bc00640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8540cf9a0 0x7fc85415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.906+0000 7fc859600640 1 -- 172.21.15.125:0/1860722653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8540cf9a0 msgr2=0x7fc85415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.906+0000 7fc859600640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8540cf9a0 0x7fc85415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.906+0000 7fc859600640 1 -- 172.21.15.125:0/1860722653 >> v1:172.21.15.125:6789/0 conn(0x7fc854143ae0 legacy=0x7fc8540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.906+0000 7fc859600640 1 -- 172.21.15.125:0/1860722653 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540b2130 con 0x7fc85415e1e0 2024-01-16T15:14:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.907+0000 7fc859600640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc85415e1e0 0x7fc8540702c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00d150 tx=0x7fc84c02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4efdf952cc33a91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.907+0000 7fc853e00640 1 -- 172.21.15.125:0/1860722653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c00b0d0 con 0x7fc85415e1e0 2024-01-16T15:14:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.907+0000 7fc853e00640 1 -- 172.21.15.125:0/1860722653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c00b230 con 0x7fc85415e1e0 2024-01-16T15:14:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.907+0000 7fc853e00640 1 -- 172.21.15.125:0/1860722653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c009600 con 0x7fc85415e1e0 2024-01-16T15:14:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 -- 172.21.15.125:0/1860722653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc85415e1e0 msgr2=0x7fc8540702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc85415e1e0 0x7fc8540702c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00d150 tx=0x7fc84c02e160 comp rx=0 tx=0).stop 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 -- 172.21.15.125:0/1860722653 shutdown_connections 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc85415e1e0 0x7fc8540702c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 --2- 172.21.15.125:0/1860722653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8540cf9a0 0x7fc85415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 -- 172.21.15.125:0/1860722653 >> 172.21.15.125:0/1860722653 conn(0x7fc85406a800 msgr2=0x7fc85406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 -- 172.21.15.125:0/1860722653 shutdown_connections 2024-01-16T15:14:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.908+0000 7fc85b600640 1 -- 172.21.15.125:0/1860722653 wait complete. 2024-01-16T15:14:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.910+0000 7fc85b600640 1 Processor -- start 2024-01-16T15:14:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.910+0000 7fc85b600640 1 -- start start 2024-01-16T15:14:41.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8540cf9a0 0x7fc8540ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc85415e1e0 0x7fc8541f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541e4ef0 con 0x7fc854143ae0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541f3010 con 0x7fc8540cf9a0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc85b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541e43b0 con 0x7fc85415e1e0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38398/0 (socket says 172.21.15.125:38398) 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 -- 172.21.15.125:0/2485041241 learned_addr learned my addr 172.21.15.125:0/2485041241 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc858c00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8540cf9a0 0x7fc8540ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 -- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc85415e1e0 msgr2=0x7fc8541f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc85415e1e0 0x7fc8541f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 -- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8540cf9a0 msgr2=0x7fc8540ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8540cf9a0 0x7fc8540ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.912+0000 7fc84bc00640 1 -- 172.21.15.125:0/2485041241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c00bd40 con 0x7fc854143ae0 2024-01-16T15:14:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc84bc00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00d800 tx=0x7fc83c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00b6c0 con 0x7fc854143ae0 2024-01-16T15:14:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c00b820 con 0x7fc854143ae0 2024-01-16T15:14:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8541e4650 con 0x7fc854143ae0 2024-01-16T15:14:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00bbb0 con 0x7fc854143ae0 2024-01-16T15:14:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.913+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8541e4c10 con 0x7fc854143ae0 2024-01-16T15:14:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.914+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8541f31c0 con 0x7fc854143ae0 2024-01-16T15:14:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.915+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fc83c00bd10 con 0x7fc854143ae0 2024-01-16T15:14:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.916+0000 7fc851600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 0x7fc82c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.916+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc83c089800 con 0x7fc854143ae0 2024-01-16T15:14:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.917+0000 7fc858c00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 0x7fc82c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.917+0000 7fc858c00640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 0x7fc82c06e860 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc844007fd0 tx=0x7fc844009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:41.920+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc83c0560c0 con 0x7fc854143ae0 2024-01-16T15:14:42.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:42.122+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc85415e640 con 0x7fc82c06c3a0 2024-01-16T15:14:42.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:42.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:45.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.194+0000 7fc851600640 1 -- 172.21.15.125:0/2485041241 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc8440012b0 con 0x7fc82c06c3a0 2024-01-16T15:14:45.177 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:45.177 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:14:45.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:14:45.178 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:14:45.178 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:14:45.178 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:14:45.180 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 msgr2=0x7fc82c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 0x7fc82c06e860 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc844007fd0 tx=0x7fc844009040 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 msgr2=0x7fc8541419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00d800 tx=0x7fc83c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 shutdown_connections 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc85415e1e0 0x7fc8541f34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc82c06c3a0 0x7fc82c06e860 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc854143ae0 0x7fc8541419a0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 --2- 172.21.15.125:0/2485041241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8540cf9a0 0x7fc8540ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 >> 172.21.15.125:0/2485041241 conn(0x7fc85406a800 msgr2=0x7fc85406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 shutdown_connections 2024-01-16T15:14:45.181 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.198+0000 7fc85b600640 1 -- 172.21.15.125:0/2485041241 wait complete. 2024-01-16T15:14:45.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:14:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.436+0000 7fad9b200640 1 Processor -- start 2024-01-16T15:14:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.437+0000 7fad9b200640 1 -- start start 2024-01-16T15:14:45.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.438+0000 7fad9b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940b3ae0 0x7fad9415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.438+0000 7fad9b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad940cf290 0x7fad940cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.438+0000 7fad9b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94070c40 con 0x7fad940ae5a0 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.438+0000 7fad9b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad94071450 con 0x7fad940cf290 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.438+0000 7fad9b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cfc60 con 0x7fad940b3ae0 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad940cf290 0x7fad940cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad940cf290 0x7fad940cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38414/0 (socket says 172.21.15.125:38414) 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 -- 172.21.15.125:0/448911065 learned_addr learned my addr 172.21.15.125:0/448911065 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad8b200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940b3ae0 0x7fad9415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 -- 172.21.15.125:0/448911065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940b3ae0 msgr2=0x7fad9415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940b3ae0 0x7fad9415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 -- 172.21.15.125:0/448911065 >> v1:172.21.15.137:6789/0 conn(0x7fad940ae5a0 legacy=0x7fad9406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.439+0000 7fad99200640 1 -- 172.21.15.125:0/448911065 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9406c470 con 0x7fad940cf290 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.440+0000 7fad99200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940cf290 0x7fad940cf720 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e50 tx=0x7fad8c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46e5ae2e06bd902 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad93400640 1 -- 172.21.15.125:0/448911065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c02cb00 con 0x7fad940cf290 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad93400640 1 -- 172.21.15.125:0/448911065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c02cc60 con 0x7fad940cf290 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad93400640 1 -- 172.21.15.125:0/448911065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c0343f0 con 0x7fad940cf290 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 -- 172.21.15.125:0/448911065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940cf290 msgr2=0x7fad940cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940cf290 0x7fad940cf720 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e50 tx=0x7fad8c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 -- 172.21.15.125:0/448911065 shutdown_connections 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940cf290 0x7fad940cf720 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 --2- 172.21.15.125:0/448911065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940b3ae0 0x7fad9415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 -- 172.21.15.125:0/448911065 >> 172.21.15.125:0/448911065 conn(0x7fad9406a800 msgr2=0x7fad9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 -- 172.21.15.125:0/448911065 shutdown_connections 2024-01-16T15:14:45.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.441+0000 7fad9b200640 1 -- 172.21.15.125:0/448911065 wait complete. 2024-01-16T15:14:45.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.443+0000 7fad9b200640 1 Processor -- start 2024-01-16T15:14:45.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.443+0000 7fad9b200640 1 -- start start 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940ae5a0 0x7fad941f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad940cf290 0x7fad94157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad941577e0 con 0x7fad940b3ae0 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94156e40 con 0x7fad940cf290 2024-01-16T15:14:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad9b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad94157120 con 0x7fad940ae5a0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad8b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad8b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38422/0 (socket says 172.21.15.125:38422) 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.444+0000 7fad8b200640 1 -- 172.21.15.125:0/3752086237 learned_addr learned my addr 172.21.15.125:0/3752086237 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad99200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad940cf290 0x7fad94157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad93e00640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940ae5a0 0x7fad941f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 -- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940ae5a0 msgr2=0x7fad941f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940ae5a0 0x7fad941f24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 -- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad940cf290 msgr2=0x7fad94157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad940cf290 0x7fad94157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 -- 172.21.15.125:0/3752086237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c009b30 con 0x7fad940b3ae0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad8b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00d800 tx=0x7fad7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00b6c0 con 0x7fad940b3ae0 2024-01-16T15:14:45.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c00b820 con 0x7fad940b3ae0 2024-01-16T15:14:45.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00bbb0 con 0x7fad940b3ae0 2024-01-16T15:14:45.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.445+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad94157400 con 0x7fad940b3ae0 2024-01-16T15:14:45.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.446+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad941506e0 con 0x7fad940b3ae0 2024-01-16T15:14:45.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.447+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94150900 con 0x7fad940b3ae0 2024-01-16T15:14:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.448+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fad7c00bd10 con 0x7fad940b3ae0 2024-01-16T15:14:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.448+0000 7fad90c00640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 0x7fad6c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.449+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad7c088c90 con 0x7fad940b3ae0 2024-01-16T15:14:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.449+0000 7fad93e00640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 0x7fad6c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.449+0000 7fad93e00640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 0x7fad6c06e860 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fad84003d90 tx=0x7fad84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.453+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad94150900 con 0x7fad940b3ae0 2024-01-16T15:14:45.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.676+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad94071170 con 0x7fad6c06c3a0 2024-01-16T15:14:45.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.738+0000 7fad90c00640 1 -- 172.21.15.125:0/3752086237 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad8400d4c0 con 0x7fad6c06c3a0 2024-01-16T15:14:45.720 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 msgr2=0x7fad6c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 0x7fad6c06e860 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fad84003d90 tx=0x7fad84009040 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 msgr2=0x7fad941501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00d800 tx=0x7fad7c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 shutdown_connections 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fad940cf290 0x7fad94157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fad6c06c3a0 0x7fad6c06e860 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad940b3ae0 0x7fad941501a0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 --2- 172.21.15.125:0/3752086237 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fad940ae5a0 0x7fad941f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.741+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 >> 172.21.15.125:0/3752086237 conn(0x7fad9406a800 msgr2=0x7fad9415a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.742+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 shutdown_connections 2024-01-16T15:14:45.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.742+0000 7fad9b200640 1 -- 172.21.15.125:0/3752086237 wait complete. 2024-01-16T15:14:45.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:14:45.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.967+0000 7f7e2ae00640 1 Processor -- start 2024-01-16T15:14:45.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.967+0000 7f7e2ae00640 1 -- start start 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e24071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2ae00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7e240ae170 0x7f7e2415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2ae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2406c730 con 0x7f7e2415d770 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2ae00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e24143bf0 con 0x7f7e240ae170 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240ae5f0 con 0x7f7e24143810 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2a400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7e2415d770 0x7f7e2415dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44016/0 (socket says 172.21.15.125:44016) 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e2a400640 1 -- 172.21.15.125:0/176202391 learned_addr learned my addr 172.21.15.125:0/176202391 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e29a00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e24071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.969+0000 7f7e29000640 1 --2- 172.21.15.125:0/176202391 >> v2:172.21.15.137:3300/0 conn(0x7f7e240ae170 0x7f7e2415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455048488 0 0) 0x7f7e2406c730 con 0x7f7e2415d770 2024-01-16T15:14:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04003520 con 0x7f7e2415d770 2024-01-16T15:14:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002061371 0 0) 0x7f7e04003520 con 0x7f7e2415d770 2024-01-16T15:14:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 msgr2=0x7f7e24071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e24071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 >> v2:172.21.15.137:3300/0 conn(0x7f7e240ae170 msgr2=0x7f7e2415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 --2- 172.21.15.125:0/176202391 >> v2:172.21.15.137:3300/0 conn(0x7f7e240ae170 0x7f7e2415e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240724d0 con 0x7f7e2415d770 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7e14001670 con 0x7f7e2415d770 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 >> v1:172.21.15.125:6789/0 conn(0x7f7e2415d770 legacy=0x7f7e2415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e04003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e04003ef0 0x7f7e040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e04004d00 0x7f7e04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24143bf0 con 0x7f7e04004d00 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240ae5f0 con 0x7f7e04003ef0 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.970+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400b9d0 con 0x7f7e24143810 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e04004d00 0x7f7e04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e29a00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e04003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e29000640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e04003ef0 0x7f7e040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 -- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 msgr2=0x7f7e04003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e24143810 0x7f7e04003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 -- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e04003ef0 msgr2=0x7f7e040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e04003ef0 0x7f7e040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.971+0000 7f7e2a400640 1 -- 172.21.15.125:0/176202391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04003520 con 0x7f7e04004d00 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.972+0000 7f7e2a400640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e04004d00 0x7f7e04005180 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c240 tx=0x7f7e040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1143d2bfb2733d37 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.972+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14005ad0 con 0x7f7e04004d00 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.972+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14005c30 con 0x7f7e04004d00 2024-01-16T15:14:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.972+0000 7f7e23e00640 1 -- 172.21.15.125:0/176202391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1402caf0 con 0x7f7e04004d00 2024-01-16T15:14:45.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 -- 172.21.15.125:0/176202391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e04004d00 msgr2=0x7f7e04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/176202391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e04004d00 0x7f7e04005180 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c240 tx=0x7f7e040057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2a400640 1 -- 172.21.15.125:0/176202391 reap_dead start 2024-01-16T15:14:45.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 -- 172.21.15.125:0/176202391 shutdown_connections 2024-01-16T15:14:45.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 -- 172.21.15.125:0/176202391 >> 172.21.15.125:0/176202391 conn(0x7f7e2406a7d0 msgr2=0x7f7e2406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 -- 172.21.15.125:0/176202391 shutdown_connections 2024-01-16T15:14:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.973+0000 7f7e2ae00640 1 -- 172.21.15.125:0/176202391 wait complete. 2024-01-16T15:14:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.975+0000 7f7e2ae00640 1 Processor -- start 2024-01-16T15:14:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.975+0000 7f7e2ae00640 1 -- start start 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e04004d00 0x7f7e24152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e240b0c80 0x7f7e240b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240afc20 con 0x7f7e2415c610 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240afec0 con 0x7f7e240b0c80 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e2ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240b01a0 con 0x7f7e04004d00 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e29000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e29000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38432/0 (socket says 172.21.15.125:38432) 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.976+0000 7f7e29000640 1 -- 172.21.15.125:0/3532747213 learned_addr learned my addr 172.21.15.125:0/3532747213 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 -- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e04004d00 msgr2=0x7f7e24152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e2a400640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e240b0c80 0x7f7e240b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29a00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e04004d00 0x7f7e24152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e04004d00 0x7f7e24152560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 -- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e240b0c80 msgr2=0x7f7e240b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e240b0c80 0x7f7e240b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 -- 172.21.15.125:0/3532747213 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240b0480 con 0x7f7e2415c610 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e29000640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c011600 tx=0x7f7e0c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00f920 con 0x7f7e2415c610 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00fa80 con 0x7f7e2415c610 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.977+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00fdf0 con 0x7f7e2415c610 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.978+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240b0760 con 0x7f7e2415c610 2024-01-16T15:14:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.978+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2413e6d0 con 0x7f7e2415c610 2024-01-16T15:14:45.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.979+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e24067450 con 0x7f7e2415c610 2024-01-16T15:14:45.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.980+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f7e0c015460 con 0x7f7e2415c610 2024-01-16T15:14:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.980+0000 7f7e21600640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 0x7f7dfc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.981+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e0c089000 con 0x7f7e2415c610 2024-01-16T15:14:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.981+0000 7f7e29a00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 0x7f7dfc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.981+0000 7f7e29a00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 0x7f7dfc06e860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007380 tx=0x7f7e1c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:45.985+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e0c055580 con 0x7f7e2415c610 2024-01-16T15:14:46.183 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.200+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e2406cfb0 con 0x7f7dfc06c3a0 2024-01-16T15:14:46.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.228+0000 7f7e21600640 1 -- 172.21.15.125:0/3532747213 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e1c009c90 con 0x7f7dfc06c3a0 2024-01-16T15:14:46.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 msgr2=0x7f7dfc06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 0x7f7dfc06e860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007380 tx=0x7f7e1c00b040 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 msgr2=0x7f7e2413e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c011600 tx=0x7f7e0c011ac0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 shutdown_connections 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e240b0c80 0x7f7e240b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7dfc06c3a0 0x7f7dfc06e860 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e2415c610 0x7f7e2413e190 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 --2- 172.21.15.125:0/3532747213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e04004d00 0x7f7e24152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 >> 172.21.15.125:0/3532747213 conn(0x7f7e2406a7d0 msgr2=0x7f7e24070930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 shutdown_connections 2024-01-16T15:14:46.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.232+0000 7f7e2ae00640 1 -- 172.21.15.125:0/3532747213 wait complete. 2024-01-16T15:14:46.216 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.234+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.216 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.234+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.216 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.234+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.216 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.234+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.217 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.234+0000 7f00fec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone57941 2024-01-16T15:14:46.237 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.254+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.237 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.254+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.237 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.254+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.237 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.254+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.237 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:14:46.254+0000 7f00fce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.435+0000 7f0d06c00640 1 Processor -- start 2024-01-16T15:14:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.435+0000 7f0d06c00640 1 -- start start 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d000cfa40 0x7f0d000cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0d000cde90 0x7f0d000ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d000b2c70 con 0x7f0d000cfa40 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d000d03c0 con 0x7f0d000cde90 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d06c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d000cf6d0 con 0x7f0d0015d3c0 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d000cfa40 0x7f0d000cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d000cfa40 0x7f0d000cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38470/0 (socket says 172.21.15.125:38470) 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 -- 172.21.15.125:0/3511023181 learned_addr learned my addr 172.21.15.125:0/3511023181 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 -- 172.21.15.125:0/3511023181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 msgr2=0x7f0d000d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 --2- 172.21.15.125:0/3511023181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 -- 172.21.15.125:0/3511023181 >> v2:172.21.15.137:3300/0 conn(0x7f0d000cde90 msgr2=0x7f0d000ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 --2- 172.21.15.125:0/3511023181 >> v2:172.21.15.137:3300/0 conn(0x7f0d000cde90 0x7f0d000ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.437+0000 7f0d04e00640 1 -- 172.21.15.125:0/3511023181 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d000cf3f0 con 0x7f0d000cfa40 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.438+0000 7f0d04e00640 1 --2- 172.21.15.125:0/3511023181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cfa40 0x7f0d000cfe80 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f0d0008b6f0 tx=0x7f0ce802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddff6bc1e00dcd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.439+0000 7f0cffe00640 1 -- 172.21.15.125:0/3511023181 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031870 con 0x7f0d000cfa40 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.439+0000 7f0cffe00640 1 -- 172.21.15.125:0/3511023181 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80319d0 con 0x7f0d000cfa40 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.439+0000 7f0cffe00640 1 -- 172.21.15.125:0/3511023181 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031d60 con 0x7f0d000cfa40 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.439+0000 7f0d06c00640 1 -- 172.21.15.125:0/3511023181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cfa40 msgr2=0x7f0d000cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.439+0000 7f0d06c00640 1 --2- 172.21.15.125:0/3511023181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cfa40 0x7f0d000cfe80 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f0d0008b6f0 tx=0x7f0ce802b910 comp rx=0 tx=0).stop 2024-01-16T15:14:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 -- 172.21.15.125:0/3511023181 shutdown_connections 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 --2- 172.21.15.125:0/3511023181 >> v2:172.21.15.137:3300/0 conn(0x7f0d000cde90 0x7f0d000ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 --2- 172.21.15.125:0/3511023181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cfa40 0x7f0d000cfe80 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 --2- 172.21.15.125:0/3511023181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 -- 172.21.15.125:0/3511023181 >> 172.21.15.125:0/3511023181 conn(0x7f0d000d3050 msgr2=0x7f0d000d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 -- 172.21.15.125:0/3511023181 shutdown_connections 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.440+0000 7f0d06c00640 1 -- 172.21.15.125:0/3511023181 wait complete. 2024-01-16T15:14:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.441+0000 7f0d06c00640 1 Processor -- start 2024-01-16T15:14:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.441+0000 7f0d06c00640 1 -- start start 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d000cfa40 0x7f0d00154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d000c9b60 con 0x7f0d000cde90 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d00154d60 con 0x7f0d000cfa40 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.442+0000 7f0d06c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d000c9890 con 0x7f0d0015d3c0 2024-01-16T15:14:46.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38480/0 (socket says 172.21.15.125:38480) 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 -- 172.21.15.125:0/816266126 learned_addr learned my addr 172.21.15.125:0/816266126 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 -- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 msgr2=0x7f0d000c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 -- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d000cfa40 msgr2=0x7f0d00154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d000cfa40 0x7f0d00154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 -- 172.21.15.125:0/816266126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80094d0 con 0x7f0d000cde90 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0d05800640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ba50 tx=0x7f0cf000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.443+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000cbe0 con 0x7f0d000cde90 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.444+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000cd60 con 0x7f0d000cde90 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.444+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0015e770 con 0x7f0d000cde90 2024-01-16T15:14:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.444+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0014680 con 0x7f0d000cde90 2024-01-16T15:14:46.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.444+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0015ed30 con 0x7f0d000cde90 2024-01-16T15:14:46.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.445+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0008ba80 con 0x7f0d000cde90 2024-01-16T15:14:46.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.448+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0cf00147e0 con 0x7f0d000cde90 2024-01-16T15:14:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.449+0000 7f0cfd600640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 0x7f0cd806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.449+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cf00889c0 con 0x7f0d000cde90 2024-01-16T15:14:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.449+0000 7f0d04e00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 0x7f0cd806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.450+0000 7f0d04e00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 0x7f0cd806e890 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8036000 tx=0x7f0ce8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.436 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.454+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cf0054fa0 con 0x7f0d000cde90 2024-01-16T15:14:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.669+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone57941", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d000b2900 con 0x7f0cd806c3d0 2024-01-16T15:14:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.704+0000 7f0cfd600640 1 -- 172.21.15.125:0/816266126 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0ce8031d30 con 0x7f0cd806c3d0 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 msgr2=0x7f0cd806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 0x7f0cd806e890 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8036000 tx=0x7f0ce8030000 comp rx=0 tx=0).stop 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 msgr2=0x7f0d00160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ba50 tx=0x7f0cf000bf10 comp rx=0 tx=0).stop 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 shutdown_connections 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d0015d3c0 0x7f0d000c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d000cfa40 0x7f0d00154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0cd806c3d0 0x7f0cd806e890 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 --2- 172.21.15.125:0/816266126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d000cde90 0x7f0d00160270 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 >> 172.21.15.125:0/816266126 conn(0x7f0d000d3050 msgr2=0x7f0d000b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.707+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 shutdown_connections 2024-01-16T15:14:46.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.708+0000 7f0d06c00640 1 -- 172.21.15.125:0/816266126 wait complete. 2024-01-16T15:14:46.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone57943 2024-01-16T15:14:46.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.903+0000 7fe553200640 1 Processor -- start 2024-01-16T15:14:46.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.903+0000 7fe553200640 1 -- start start 2024-01-16T15:14:46.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.904+0000 7fe553200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe54c13df80 0x7fe54c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe553200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe54c13e360 0x7fe54c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe553200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c15f9c0 0x7fe54c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe553200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c06c470 con 0x7fe54c13df80 2024-01-16T15:14:46.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe553200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c06c750 con 0x7fe54c13e360 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe553200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c06ca30 con 0x7fe54c15f9c0 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe54c13df80 0x7fe54c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe54c13df80 0x7fe54c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36816/0 (socket says 172.21.15.125:36816) 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 -- 172.21.15.125:0/1085629348 learned_addr learned my addr 172.21.15.125:0/1085629348 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 -- 172.21.15.125:0/1085629348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c15f9c0 msgr2=0x7fe54c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 --2- 172.21.15.125:0/1085629348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c15f9c0 0x7fe54c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 -- 172.21.15.125:0/1085629348 >> v2:172.21.15.137:3300/0 conn(0x7fe54c13e360 msgr2=0x7fe54c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 --2- 172.21.15.125:0/1085629348 >> v2:172.21.15.137:3300/0 conn(0x7fe54c13e360 0x7fe54c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.905+0000 7fe54be00640 1 -- 172.21.15.125:0/1085629348 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c06cd10 con 0x7fe54c13df80 2024-01-16T15:14:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.906+0000 7fe54be00640 1 --2- 172.21.15.125:0/1085629348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c13df80 0x7fe54c15bee0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06ded0 tx=0x7fe53c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e44f2c1c82b077 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.907+0000 7fe54aa00640 1 -- 172.21.15.125:0/1085629348 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c031860 con 0x7fe54c13df80 2024-01-16T15:14:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.907+0000 7fe54aa00640 1 -- 172.21.15.125:0/1085629348 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c0319c0 con 0x7fe54c13df80 2024-01-16T15:14:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.907+0000 7fe54aa00640 1 -- 172.21.15.125:0/1085629348 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c031d50 con 0x7fe54c13df80 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.907+0000 7fe553200640 1 -- 172.21.15.125:0/1085629348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c13df80 msgr2=0x7fe54c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.907+0000 7fe553200640 1 --2- 172.21.15.125:0/1085629348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c13df80 0x7fe54c15bee0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06ded0 tx=0x7fe53c02b900 comp rx=0 tx=0).stop 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 -- 172.21.15.125:0/1085629348 shutdown_connections 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 --2- 172.21.15.125:0/1085629348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c15f9c0 0x7fe54c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 --2- 172.21.15.125:0/1085629348 >> v2:172.21.15.137:3300/0 conn(0x7fe54c13e360 0x7fe54c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 --2- 172.21.15.125:0/1085629348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c13df80 0x7fe54c15bee0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 -- 172.21.15.125:0/1085629348 >> 172.21.15.125:0/1085629348 conn(0x7fe54c06a800 msgr2=0x7fe54c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 -- 172.21.15.125:0/1085629348 shutdown_connections 2024-01-16T15:14:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.908+0000 7fe553200640 1 -- 172.21.15.125:0/1085629348 wait complete. 2024-01-16T15:14:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.909+0000 7fe553200640 1 Processor -- start 2024-01-16T15:14:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.909+0000 7fe553200640 1 -- start start 2024-01-16T15:14:46.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c13df80 0x7fe54c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c13e360 0x7fe54c139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c147e60 con 0x7fe54c15f9c0 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c155d70 con 0x7fe54c13df80 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.910+0000 7fe553200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c13ba40 con 0x7fe54c13e360 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36830/0 (socket says 172.21.15.125:36830) 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 -- 172.21.15.125:0/2031184564 learned_addr learned my addr 172.21.15.125:0/2031184564 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe54be00640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c13df80 0x7fe54c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe54b400640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c13e360 0x7fe54c139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 -- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c13e360 msgr2=0x7fe54c139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c13e360 0x7fe54c139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 -- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c13df80 msgr2=0x7fe54c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c13df80 0x7fe54c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.911+0000 7fe551200640 1 -- 172.21.15.125:0/2031184564 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0094d0 con 0x7fe54c15f9c0 2024-01-16T15:14:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe551200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b9e0 tx=0x7fe54400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400cb20 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54400cc80 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544014680 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c15a940 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.912+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c15af00 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.913+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0675d0 con 0x7fe54c15f9c0 2024-01-16T15:14:46.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.914+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fe54401d3b0 con 0x7fe54c15f9c0 2024-01-16T15:14:46.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.915+0000 7fe543e00640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 0x7fe52406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.915+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5440889d0 con 0x7fe54c15f9c0 2024-01-16T15:14:46.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.915+0000 7fe54be00640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 0x7fe52406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.915+0000 7fe54be00640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 0x7fe52406e860 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe53c036000 tx=0x7fe53c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:46.919+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe544054fb0 con 0x7fe54c15f9c0 2024-01-16T15:14:47.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.119+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone57943", "target": ["mon-mgr", ""]}) v1 -- 0x7fe54c1408d0 con 0x7fe52406c3a0 2024-01-16T15:14:47.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.148+0000 7fe543e00640 1 -- 172.21.15.125:0/2031184564 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe53c031da0 con 0x7fe52406c3a0 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.150+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 msgr2=0x7fe52406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.150+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 0x7fe52406e860 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe53c036000 tx=0x7fe53c03c040 comp rx=0 tx=0).stop 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.150+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 msgr2=0x7fe54c1399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.150+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b9e0 tx=0x7fe54400bea0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 shutdown_connections 2024-01-16T15:14:47.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe52406c3a0 0x7fe52406e860 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54c15f9c0 0x7fe54c1399c0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe54c13e360 0x7fe54c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 --2- 172.21.15.125:0/2031184564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c13df80 0x7fe54c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 >> 172.21.15.125:0/2031184564 conn(0x7fe54c06a800 msgr2=0x7fe54c0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 shutdown_connections 2024-01-16T15:14:47.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.151+0000 7fe553200640 1 -- 172.21.15.125:0/2031184564 wait complete. 2024-01-16T15:14:47.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone57942 2024-01-16T15:14:47.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.359+0000 7fbdc7e00640 1 Processor -- start 2024-01-16T15:14:47.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.359+0000 7fbdc7e00640 1 -- start start 2024-01-16T15:14:47.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 0x7fbdc80b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbdc80b4140 0x7fbdc80d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc80c7bf0 con 0x7fbdc80d1b90 2024-01-16T15:14:47.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc80d2020 con 0x7fbdc80b4140 2024-01-16T15:14:47.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80d0420 con 0x7fbdc80b34e0 2024-01-16T15:14:47.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbdc80d1b90 0x7fbdc80cdc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52450/0 (socket says 172.21.15.125:52450) 2024-01-16T15:14:47.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc7400640 1 -- 172.21.15.125:0/4155456141 learned_addr learned my addr 172.21.15.125:0/4155456141 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:47.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.361+0000 7fbdc6000640 1 --2- 172.21.15.125:0/4155456141 >> v2:172.21.15.125:3300/0 conn(0x7fbdc80b4140 0x7fbdc80d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 0x7fbdc80b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6000640 1 -- 172.21.15.125:0/4155456141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 msgr2=0x7fbdc80b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6000640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 0x7fbdc80b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6000640 1 -- 172.21.15.125:0/4155456141 >> v1:172.21.15.125:6789/0 conn(0x7fbdc80d1b90 legacy=0x7fbdc80cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6000640 1 -- 172.21.15.125:0/4155456141 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80d0e70 con 0x7fbdc80b4140 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 0x7fbdc80b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.362+0000 7fbdc5600640 1 -- 172.21.15.125:0/4155456141 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415727322 0 0) 0x7fbdc80c7bf0 con 0x7fbdc80d1b90 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.363+0000 7fbdc6000640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80b4140 0x7fbdc80d1650 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00097a0 tx=0x7fbdb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b89a3f264ee725 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.363+0000 7fbdc5600640 1 -- 172.21.15.125:0/4155456141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0031830 con 0x7fbdc80b4140 2024-01-16T15:14:47.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.363+0000 7fbdc5600640 1 -- 172.21.15.125:0/4155456141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb0031990 con 0x7fbdc80b4140 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.363+0000 7fbdc5600640 1 -- 172.21.15.125:0/4155456141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0031d20 con 0x7fbdc80b4140 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 -- 172.21.15.125:0/4155456141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80b4140 msgr2=0x7fbdc80d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80b4140 0x7fbdc80d1650 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00097a0 tx=0x7fbdb002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 -- 172.21.15.125:0/4155456141 shutdown_connections 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80b4140 0x7fbdc80d1650 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/4155456141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b34e0 0x7fbdc80b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 -- 172.21.15.125:0/4155456141 >> 172.21.15.125:0/4155456141 conn(0x7fbdc80d3090 msgr2=0x7fbdc80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 -- 172.21.15.125:0/4155456141 shutdown_connections 2024-01-16T15:14:47.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.364+0000 7fbdc7e00640 1 -- 172.21.15.125:0/4155456141 wait complete. 2024-01-16T15:14:47.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.365+0000 7fbdc7e00640 1 Processor -- start 2024-01-16T15:14:47.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.365+0000 7fbdc7e00640 1 -- start start 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 0x7fbdc8154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 0x7fbdc80b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80b7340 con 0x7fbdc80d1b90 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80b7590 con 0x7fbdc80b34e0 2024-01-16T15:14:47.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.366+0000 7fbdc7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80b7870 con 0x7fbdc80b4140 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36860/0 (socket says 172.21.15.125:36860) 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 -- 172.21.15.125:0/2047742896 learned_addr learned my addr 172.21.15.125:0/2047742896 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 0x7fbdc8154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc6000640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 0x7fbdc80b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 -- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 msgr2=0x7fbdc80b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 0x7fbdc80b6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 -- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 msgr2=0x7fbdc8154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 0x7fbdc8154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 -- 172.21.15.125:0/2047742896 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0009400 con 0x7fbdc80d1b90 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc6000640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 0x7fbdc80b6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 0x7fbdc8154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:47.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.367+0000 7fbdc7400640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001eb0 tx=0x7fbdc000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.368+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc00074e0 con 0x7fbdc80d1b90 2024-01-16T15:14:47.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.368+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc0007640 con 0x7fbdc80d1b90 2024-01-16T15:14:47.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.368+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0004a00 con 0x7fbdc80d1b90 2024-01-16T15:14:47.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.368+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc80b7b50 con 0x7fbdc80d1b90 2024-01-16T15:14:47.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.368+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc80b8110 con 0x7fbdc80d1b90 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.369+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc80b8300 con 0x7fbdc80d1b90 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.370+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 39) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fbdc0004b60 con 0x7fbdc80d1b90 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.370+0000 7fbdbf400640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 0x7fbda006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.371+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdc0088810 con 0x7fbdc80d1b90 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.371+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 0x7fbda006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.371+0000 7fbdc6a00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 0x7fbda006e890 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbdc815c970 tx=0x7fbdb8007550 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.375+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdc0054df0 con 0x7fbdc80d1b90 2024-01-16T15:14:47.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.486+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fbdc00551c0 con 0x7fbdc80d1b90 2024-01-16T15:14:47.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.582+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone57942", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdc80b79e0 con 0x7fbda006c3d0 2024-01-16T15:14:47.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.619+0000 7fbdbf400640 1 -- 172.21.15.125:0/2047742896 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbdc80b79e0 con 0x7fbda006c3d0 2024-01-16T15:14:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.621+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 msgr2=0x7fbda006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.621+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 0x7fbda006e890 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbdc815c970 tx=0x7fbdb8007550 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.621+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 msgr2=0x7fbdc80b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.621+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001eb0 tx=0x7fbdc000c870 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 shutdown_connections 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fbda006c3d0 0x7fbda006e890 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc80d1b90 0x7fbdc80b6b60 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdc80b4140 0x7fbdc80b6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 --2- 172.21.15.125:0/2047742896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdc80b34e0 0x7fbdc8154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 >> 172.21.15.125:0/2047742896 conn(0x7fbdc80d3090 msgr2=0x7fbdc8156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 shutdown_connections 2024-01-16T15:14:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.622+0000 7fbdc7e00640 1 -- 172.21.15.125:0/2047742896 wait complete. 2024-01-16T15:14:47.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:47.624 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:14:47.672 INFO:teuthology.orchestra.run.smithi125.stdout:2 2024-01-16T15:14:47.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:14:47.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.883+0000 7f70a6600640 1 Processor -- start 2024-01-16T15:14:47.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.883+0000 7f70a6600640 1 -- start start 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.884+0000 7f70a6600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f70a013fcb0 0x7f70a00cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.884+0000 7f70a6600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70a00cdc20 0x7f70a00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.884+0000 7f70a6600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00ce440 con 0x7f70a013cf10 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.884+0000 7f70a6600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a00ce720 con 0x7f70a00cdc20 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.884+0000 7f70a6600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a00cea00 con 0x7f70a013fcb0 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f70a013cf10 0x7f70a013f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52454/0 (socket says 172.21.15.125:52454) 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5200640 1 -- 172.21.15.125:0/3687839070 learned_addr learned my addr 172.21.15.125:0/3687839070 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5c00640 1 --2- 172.21.15.125:0/3687839070 >> v2:172.21.15.125:3300/0 conn(0x7f70a00cdc20 0x7f70a00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f7097c00640 1 --2- 172.21.15.125:0/3687839070 >> v2:172.21.15.137:3300/0 conn(0x7f70a013fcb0 0x7f70a00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f709fe00640 1 -- 172.21.15.125:0/3687839070 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123585809 0 0) 0x7f70a00ce440 con 0x7f70a013cf10 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f709fe00640 1 -- 172.21.15.125:0/3687839070 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7080005730 con 0x7f70a013cf10 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5c00640 1 -- 172.21.15.125:0/3687839070 >> v2:172.21.15.137:3300/0 conn(0x7f70a013fcb0 msgr2=0x7f70a00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5c00640 1 --2- 172.21.15.125:0/3687839070 >> v2:172.21.15.137:3300/0 conn(0x7f70a013fcb0 0x7f70a00cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5c00640 1 -- 172.21.15.125:0/3687839070 >> v1:172.21.15.125:6789/0 conn(0x7f70a013cf10 legacy=0x7f70a013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.885+0000 7f70a5c00640 1 -- 172.21.15.125:0/3687839070 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00b1d90 con 0x7f70a00cdc20 2024-01-16T15:14:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.886+0000 7f70a5c00640 1 --2- 172.21.15.125:0/3687839070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a00b2590 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f70a0157f90 tx=0x7f709802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89db7390a594b238 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f709fe00640 1 -- 172.21.15.125:0/3687839070 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709802cc70 con 0x7f70a00cdc20 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f709fe00640 1 -- 172.21.15.125:0/3687839070 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709802cdd0 con 0x7f70a00cdc20 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f709fe00640 1 -- 172.21.15.125:0/3687839070 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098034660 con 0x7f70a00cdc20 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 -- 172.21.15.125:0/3687839070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 msgr2=0x7f70a00b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 --2- 172.21.15.125:0/3687839070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a00b2590 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f70a0157f90 tx=0x7f709802bec0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 -- 172.21.15.125:0/3687839070 shutdown_connections 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 --2- 172.21.15.125:0/3687839070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a00b2590 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 --2- 172.21.15.125:0/3687839070 >> v2:172.21.15.137:3300/0 conn(0x7f70a013fcb0 0x7f70a00cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 -- 172.21.15.125:0/3687839070 >> 172.21.15.125:0/3687839070 conn(0x7f70a006a7d0 msgr2=0x7f70a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:47.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.887+0000 7f70a6600640 1 -- 172.21.15.125:0/3687839070 shutdown_connections 2024-01-16T15:14:47.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.888+0000 7f70a6600640 1 -- 172.21.15.125:0/3687839070 wait complete. 2024-01-16T15:14:47.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.889+0000 7f70a6600640 1 Processor -- start 2024-01-16T15:14:47.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.889+0000 7f70a6600640 1 -- start start 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70a013cf10 0x7f70a014ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70a013fcb0 0x7f70a0139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a014d940 con 0x7f70a00cdc20 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a0137fc0 con 0x7f70a013fcb0 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a01382a0 con 0x7f70a013cf10 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36878/0 (socket says 172.21.15.125:36878) 2024-01-16T15:14:47.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.890+0000 7f70a5200640 1 -- 172.21.15.125:0/1039647812 learned_addr learned my addr 172.21.15.125:0/1039647812 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 -- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70a013cf10 msgr2=0x7f70a014ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70a013cf10 0x7f70a014ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 -- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70a013fcb0 msgr2=0x7f70a0139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70a013fcb0 0x7f70a0139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 -- 172.21.15.125:0/1039647812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098009a70 con 0x7f70a00cdc20 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a5200640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f709000c500 tx=0x7f709000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090005ce0 con 0x7f70a00cdc20 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090005e40 con 0x7f70a00cdc20 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a0138580 con 0x7f70a00cdc20 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090007330 con 0x7f70a00cdc20 2024-01-16T15:14:47.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.891+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a0138b40 con 0x7f70a00cdc20 2024-01-16T15:14:47.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.892+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0067450 con 0x7f70a00cdc20 2024-01-16T15:14:47.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.893+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7090007550 con 0x7f70a00cdc20 2024-01-16T15:14:47.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.895+0000 7f709d600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 0x7f707806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:47.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.895+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70900881f0 con 0x7f70a00cdc20 2024-01-16T15:14:47.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.895+0000 7f7097c00640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 0x7f707806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:47.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.896+0000 7f7097c00640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 0x7f707806e980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7088006fd0 tx=0x7f7088008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:47.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:47.900+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7090054800 con 0x7f70a00cdc20 2024-01-16T15:14:48.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:48.114+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f70a00ccae0 con 0x7f707806c4c0 2024-01-16T15:14:48.117 INFO:tasks.ceph.mon.a.smithi125.stderr:2024-01-16T15:14:48.134+0000 7ffb7dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:14:48.195 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:14:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:49.513+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f7090054bd0 con 0x7f70a00cdc20 2024-01-16T15:14:51.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.174+0000 7f709d600640 1 -- 172.21.15.125:0/1039647812 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f70a00ccae0 con 0x7f707806c4c0 2024-01-16T15:14:51.158 INFO:teuthology.orchestra.run.smithi125.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:14:51.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 msgr2=0x7f707806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 0x7f707806e980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7088006fd0 tx=0x7f7088008040 comp rx=0 tx=0).stop 2024-01-16T15:14:51.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 msgr2=0x7f70a01f2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f709000c500 tx=0x7f709000c9c0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 shutdown_connections 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70a013fcb0 0x7f70a0139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70a013cf10 0x7f70a014ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f707806c4c0 0x7f707806e980 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 --2- 172.21.15.125:0/1039647812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70a00cdc20 0x7f70a01f2bf0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.180+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 >> 172.21.15.125:0/1039647812 conn(0x7f70a006a7d0 msgr2=0x7f70a00af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.181+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 shutdown_connections 2024-01-16T15:14:51.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.181+0000 7f70a6600640 1 -- 172.21.15.125:0/1039647812 wait complete. 2024-01-16T15:14:51.192 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:14:51.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:51.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:51.389 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:14:51.390 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:14:51.390 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:51.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.709+0000 7fa529200640 1 Processor -- start 2024-01-16T15:14:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.709+0000 7fa529200640 1 -- start start 2024-01-16T15:14:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.710+0000 7fa529200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa52413e360 0x7fa5240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa529200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 0x7fa5240cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa529200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52415c5f0 con 0x7fa52413df80 2024-01-16T15:14:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa529200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52406c470 con 0x7fa52413e360 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa529200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52406c750 con 0x7fa52415f9c0 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa52413e360 0x7fa5240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa52413e360 0x7fa5240cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36896/0 (socket says 172.21.15.125:36896) 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 -- 172.21.15.125:0/3030018884 learned_addr learned my addr 172.21.15.125:0/3030018884 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa522800640 1 --2- 172.21.15.125:0/3030018884 >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 0x7fa5240cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 -- 172.21.15.125:0/3030018884 >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 msgr2=0x7fa5240cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 --2- 172.21.15.125:0/3030018884 >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 0x7fa5240cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.711+0000 7fa519200640 1 -- 172.21.15.125:0/3030018884 >> v1:172.21.15.125:6789/0 conn(0x7fa52413df80 legacy=0x7fa52415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.712+0000 7fa519200640 1 -- 172.21.15.125:0/3030018884 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52406ca30 con 0x7fa52413e360 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.712+0000 7fa522800640 1 --2- 172.21.15.125:0/3030018884 >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 0x7fa5240cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa519200640 1 --2- 172.21.15.125:0/3030018884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52413e360 0x7fa5240cf290 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ccae0 tx=0x7fa50c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db31126378c15ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa521400640 1 -- 172.21.15.125:0/3030018884 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c030870 con 0x7fa52413e360 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa521400640 1 -- 172.21.15.125:0/3030018884 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c0309d0 con 0x7fa52413e360 2024-01-16T15:14:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa521400640 1 -- 172.21.15.125:0/3030018884 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c030d60 con 0x7fa52413e360 2024-01-16T15:14:51.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa529200640 1 -- 172.21.15.125:0/3030018884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52413e360 msgr2=0x7fa5240cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.713+0000 7fa529200640 1 --2- 172.21.15.125:0/3030018884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52413e360 0x7fa5240cf290 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ccae0 tx=0x7fa50c02b9b0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 -- 172.21.15.125:0/3030018884 shutdown_connections 2024-01-16T15:14:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 --2- 172.21.15.125:0/3030018884 >> v2:172.21.15.137:3300/0 conn(0x7fa52415f9c0 0x7fa5240cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 --2- 172.21.15.125:0/3030018884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52413e360 0x7fa5240cf290 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 -- 172.21.15.125:0/3030018884 >> 172.21.15.125:0/3030018884 conn(0x7fa52406a800 msgr2=0x7fa52406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 -- 172.21.15.125:0/3030018884 shutdown_connections 2024-01-16T15:14:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.714+0000 7fa529200640 1 -- 172.21.15.125:0/3030018884 wait complete. 2024-01-16T15:14:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.715+0000 7fa529200640 1 Processor -- start 2024-01-16T15:14:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 -- start start 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52413df80 0x7fa524150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa52413e360 0x7fa5241527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524153310 con 0x7fa52415f9c0 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5241535f0 con 0x7fa52413e360 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.716+0000 7fa529200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5241538d0 con 0x7fa52413df80 2024-01-16T15:14:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36898/0 (socket says 172.21.15.125:36898) 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 -- 172.21.15.125:0/107329932 learned_addr learned my addr 172.21.15.125:0/107329932 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa521e00640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52413df80 0x7fa524150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 -- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52413df80 msgr2=0x7fa524150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52413df80 0x7fa524150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 -- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa52413e360 msgr2=0x7fa5241527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa52413e360 0x7fa5241527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 -- 172.21.15.125:0/107329932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0094d0 con 0x7fa52415f9c0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.717+0000 7fa522800640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00cea0 tx=0x7fa51c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.718+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c003b90 con 0x7fa52415f9c0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.718+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c003d10 con 0x7fa52415f9c0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.718+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c004260 con 0x7fa52415f9c0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.718+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa524153bb0 con 0x7fa52415f9c0 2024-01-16T15:14:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.718+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5241e4de0 con 0x7fa52415f9c0 2024-01-16T15:14:51.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.719+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5240a84c0 con 0x7fa52415f9c0 2024-01-16T15:14:51.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.720+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa51c0043c0 con 0x7fa52415f9c0 2024-01-16T15:14:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.721+0000 7fa51aa00640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 0x7fa4fc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.721+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa51c0890f0 con 0x7fa52415f9c0 2024-01-16T15:14:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.721+0000 7fa521e00640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 0x7fa4fc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.721+0000 7fa521e00640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 0x7fa4fc06e8f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f60 tx=0x7fa514009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:51.725+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa51c0559b0 con 0x7fa52415f9c0 2024-01-16T15:14:51.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.009+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5240675d0 con 0x7fa52415f9c0 2024-01-16T15:14:51.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.010+0000 7fa51aa00640 1 -- 172.21.15.125:0/107329932 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+13036 (secure 0 0 0) 0x7fa51c058fd0 con 0x7fa52415f9c0 2024-01-16T15:14:51.993 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:51.993 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":103,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:14:51.154789+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/737412716":"2024-01-17T15:14:48.135036+0000","172.21.15.125:6835/737412716":"2024-01-17T15:14:48.135036+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:51.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 msgr2=0x7fa4fc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 0x7fa4fc06e8f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f60 tx=0x7fa514009040 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 msgr2=0x7fa524152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00cea0 tx=0x7fa51c00cf90 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 shutdown_connections 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa4fc06c430 0x7fa4fc06e8f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa52415f9c0 0x7fa524152d30 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa52413e360 0x7fa5241527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 --2- 172.21.15.125:0/107329932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52413df80 0x7fa524150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 >> 172.21.15.125:0/107329932 conn(0x7fa52406a800 msgr2=0x7fa52406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 shutdown_connections 2024-01-16T15:14:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.013+0000 7fa529200640 1 -- 172.21.15.125:0/107329932 wait complete. 2024-01-16T15:14:52.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:52.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.221+0000 7f8005e00640 1 Processor -- start 2024-01-16T15:14:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.221+0000 7f8005e00640 1 -- start start 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f8005e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f800015b3f0 0x7f80000adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f8005e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80000723e0 0x7f8000070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f8005e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80000ae8a0 con 0x7f80000ae220 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f8005e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000ae6b0 con 0x7f800015b3f0 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f8005e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800015fd40 con 0x7f80000723e0 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f7ff5e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f80000ae220 0x7f8000071cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52482/0 (socket says 172.21.15.125:52482) 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f7ff5e00640 1 -- 172.21.15.125:0/2991012797 learned_addr learned my addr 172.21.15.125:0/2991012797 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:52.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.222+0000 7f7ffea00640 1 --2- 172.21.15.125:0/2991012797 >> v2:172.21.15.125:3300/0 conn(0x7f800015b3f0 0x7f80000adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7fff400640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80000723e0 0x7f8000070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffe000640 1 -- 172.21.15.125:0/2991012797 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710201504 0 0) 0x7f80000ae8a0 con 0x7f80000ae220 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffe000640 1 -- 172.21.15.125:0/2991012797 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f80000ae220 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffea00640 1 -- 172.21.15.125:0/2991012797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80000723e0 msgr2=0x7f8000070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffea00640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80000723e0 0x7f8000070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffea00640 1 -- 172.21.15.125:0/2991012797 >> v1:172.21.15.125:6789/0 conn(0x7f80000ae220 legacy=0x7f8000071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.223+0000 7f7ffea00640 1 -- 172.21.15.125:0/2991012797 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800015ef80 con 0x7f800015b3f0 2024-01-16T15:14:52.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f7ffea00640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800015b3f0 0x7f80000adce0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097a0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f267ce4d3116cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f7ffe000640 1 -- 172.21.15.125:0/2991012797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0031830 con 0x7f800015b3f0 2024-01-16T15:14:52.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f7ffe000640 1 -- 172.21.15.125:0/2991012797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0031990 con 0x7f800015b3f0 2024-01-16T15:14:52.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f7ffe000640 1 -- 172.21.15.125:0/2991012797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0031d20 con 0x7f800015b3f0 2024-01-16T15:14:52.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f8005e00640 1 -- 172.21.15.125:0/2991012797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800015b3f0 msgr2=0x7f80000adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.225+0000 7f8005e00640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800015b3f0 0x7f80000adce0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097a0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 -- 172.21.15.125:0/2991012797 shutdown_connections 2024-01-16T15:14:52.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80000723e0 0x7f8000070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 --2- 172.21.15.125:0/2991012797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800015b3f0 0x7f80000adce0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 -- 172.21.15.125:0/2991012797 >> 172.21.15.125:0/2991012797 conn(0x7f80000b0c00 msgr2=0x7f80000b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 -- 172.21.15.125:0/2991012797 shutdown_connections 2024-01-16T15:14:52.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.226+0000 7f8005e00640 1 -- 172.21.15.125:0/2991012797 wait complete. 2024-01-16T15:14:52.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.227+0000 7f8005e00640 1 Processor -- start 2024-01-16T15:14:52.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.227+0000 7f8005e00640 1 -- start start 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 0x7f8000151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f800015b3f0 0x7f800015e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f63a0 con 0x7f80000ae220 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f6c40 con 0x7f80000723e0 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.228+0000 7f8005e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f3b50 con 0x7f800015b3f0 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36920/0 (socket says 172.21.15.125:36920) 2024-01-16T15:14:52.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 -- 172.21.15.125:0/1932384274 learned_addr learned my addr 172.21.15.125:0/1932384274 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ffea00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 0x7f8000151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 -- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f800015b3f0 msgr2=0x7f800015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7fff400640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f800015b3f0 0x7f800015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f800015b3f0 0x7f800015e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 -- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 msgr2=0x7f8000151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 0x7f8000151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ff5e00640 1 -- 172.21.15.125:0/1932384274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0009400 con 0x7f80000ae220 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.229+0000 7f7ffea00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 0x7f8000151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f7ff5e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800cbc0 tx=0x7f7fe800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe800ad40 con 0x7f80000ae220 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8012400 con 0x7f80000ae220 2024-01-16T15:14:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe800aea0 con 0x7f80000ae220 2024-01-16T15:14:52.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80001f65f0 con 0x7f80000ae220 2024-01-16T15:14:52.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.230+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80001f70d0 con 0x7f80000ae220 2024-01-16T15:14:52.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.231+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80001f6db0 con 0x7f80000ae220 2024-01-16T15:14:52.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.232+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f7fe8007070 con 0x7f80000ae220 2024-01-16T15:14:52.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.233+0000 7f7ff7e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 0x7f7fd806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.233+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7fe8088a10 con 0x7f80000ae220 2024-01-16T15:14:52.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.233+0000 7f7ffea00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 0x7f7fd806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.234+0000 7f7ffea00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 0x7f7fd806e8f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0036000 tx=0x7f7ff003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.237+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fe80bee90 con 0x7f80000ae220 2024-01-16T15:14:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.427+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8000067640 con 0x7f80000ae220 2024-01-16T15:14:52.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.428+0000 7f7ff7e00640 1 -- 172.21.15.125:0/1932384274 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fe8017070 con 0x7f80000ae220 2024-01-16T15:14:52.410 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:14:52.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 msgr2=0x7f7fd806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 0x7f7fd806e8f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0036000 tx=0x7f7ff003c040 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 msgr2=0x7f800015db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800cbc0 tx=0x7f7fe800a430 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 shutdown_connections 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f800015b3f0 0x7f800015e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7fd806c430 0x7f7fd806e8f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80000ae220 0x7f800015db10 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 --2- 172.21.15.125:0/1932384274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80000723e0 0x7f8000151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 >> 172.21.15.125:0/1932384274 conn(0x7f80000b0c00 msgr2=0x7f8000070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 shutdown_connections 2024-01-16T15:14:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.431+0000 7f8005e00640 1 -- 172.21.15.125:0/1932384274 wait complete. 2024-01-16T15:14:52.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:14:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.657+0000 7fc76f200640 1 Processor -- start 2024-01-16T15:14:52.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.658+0000 7fc76f200640 1 -- start start 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc768143ae0 0x7fc7680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76f200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc76815e3b0 0x7fc7680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76f200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7680cfc30 con 0x7fc7680cf7d0 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76f200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680b1ee0 con 0x7fc76815e3b0 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680b2780 con 0x7fc768143ae0 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc76815e3b0 0x7fc7680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc76815e3b0 0x7fc7680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36928/0 (socket says 172.21.15.125:36928) 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.659+0000 7fc76d200640 1 -- 172.21.15.125:0/521229888 learned_addr learned my addr 172.21.15.125:0/521229888 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:52.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.660+0000 7fc767e00640 1 --2- 172.21.15.125:0/521229888 >> v2:172.21.15.137:3300/0 conn(0x7fc768143ae0 0x7fc7680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.660+0000 7fc76d200640 1 -- 172.21.15.125:0/521229888 >> v2:172.21.15.137:3300/0 conn(0x7fc768143ae0 msgr2=0x7fc7680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.660+0000 7fc76d200640 1 --2- 172.21.15.125:0/521229888 >> v2:172.21.15.137:3300/0 conn(0x7fc768143ae0 0x7fc7680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.660+0000 7fc76d200640 1 -- 172.21.15.125:0/521229888 >> v1:172.21.15.137:6789/0 conn(0x7fc7680cf7d0 legacy=0x7fc76815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.660+0000 7fc76d200640 1 -- 172.21.15.125:0/521229888 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76815e990 con 0x7fc76815e3b0 2024-01-16T15:14:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.661+0000 7fc76d200640 1 --2- 172.21.15.125:0/521229888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc7680702c0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cfea0 tx=0x7fc76002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f1d69e18be4f9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.661+0000 7fc766a00640 1 -- 172.21.15.125:0/521229888 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76002ea90 con 0x7fc76815e3b0 2024-01-16T15:14:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.661+0000 7fc766a00640 1 -- 172.21.15.125:0/521229888 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76002ebf0 con 0x7fc76815e3b0 2024-01-16T15:14:52.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.661+0000 7fc766a00640 1 -- 172.21.15.125:0/521229888 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760009600 con 0x7fc76815e3b0 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 -- 172.21.15.125:0/521229888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 msgr2=0x7fc7680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 --2- 172.21.15.125:0/521229888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc7680702c0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cfea0 tx=0x7fc76002e0f0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 -- 172.21.15.125:0/521229888 shutdown_connections 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 --2- 172.21.15.125:0/521229888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc7680702c0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 --2- 172.21.15.125:0/521229888 >> v2:172.21.15.137:3300/0 conn(0x7fc768143ae0 0x7fc7680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 -- 172.21.15.125:0/521229888 >> 172.21.15.125:0/521229888 conn(0x7fc76806a800 msgr2=0x7fc76806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 -- 172.21.15.125:0/521229888 shutdown_connections 2024-01-16T15:14:52.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.662+0000 7fc76f200640 1 -- 172.21.15.125:0/521229888 wait complete. 2024-01-16T15:14:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.664+0000 7fc76f200640 1 Processor -- start 2024-01-16T15:14:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.664+0000 7fc76f200640 1 -- start start 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7680cf7d0 0x7fc768159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 0x7fc7680ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680adf70 con 0x7fc76815e3b0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680ad360 con 0x7fc7680cf7d0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680ad640 con 0x7fc768143ae0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36936/0 (socket says 172.21.15.125:36936) 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.665+0000 7fc76d200640 1 -- 172.21.15.125:0/1416207892 learned_addr learned my addr 172.21.15.125:0/1416207892 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc767e00640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7680cf7d0 0x7fc768159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc767400640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 0x7fc7680ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 -- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 msgr2=0x7fc7680ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 0x7fc7680ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 -- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7680cf7d0 msgr2=0x7fc768159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7680cf7d0 0x7fc768159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 -- 172.21.15.125:0/1416207892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76000bd40 con 0x7fc76815e3b0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc767400640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 0x7fc7680ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76d200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc760038040 tx=0x7fc760004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76002ea90 con 0x7fc76815e3b0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc760036820 con 0x7fc76815e3b0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760036bd0 con 0x7fc76815e3b0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.666+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7680ad920 con 0x7fc76815e3b0 2024-01-16T15:14:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.667+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7680ae400 con 0x7fc76815e3b0 2024-01-16T15:14:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.667+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7680ae0e0 con 0x7fc76815e3b0 2024-01-16T15:14:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.668+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc760036d30 con 0x7fc76815e3b0 2024-01-16T15:14:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.669+0000 7fc75fe00640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 0x7fc74006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.669+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7600abe70 con 0x7fc76815e3b0 2024-01-16T15:14:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.670+0000 7fc767e00640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 0x7fc74006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.670+0000 7fc767e00640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 0x7fc74006e8f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc750006fd0 tx=0x7fc750008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.674+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc760078360 con 0x7fc76815e3b0 2024-01-16T15:14:52.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.872+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7680ae0b0 con 0x7fc76815e3b0 2024-01-16T15:14:52.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.873+0000 7fc75fe00640 1 -- 172.21.15.125:0/1416207892 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v103) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc760008370 con 0x7fc76815e3b0 2024-01-16T15:14:52.855 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/737412716 2024-01-17T15:14:48.135036+0000 2024-01-16T15:14:52.855 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/737412716 2024-01-17T15:14:48.135036+0000 2024-01-16T15:14:52.856 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 msgr2=0x7fc74006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 0x7fc74006e8f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc750006fd0 tx=0x7fc750008040 comp rx=0 tx=0).stop 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 msgr2=0x7fc768141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc760038040 tx=0x7fc760004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 shutdown_connections 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc74006c430 0x7fc74006e8f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc76815e3b0 0x7fc768141570 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc768143ae0 0x7fc7680ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 --2- 172.21.15.125:0/1416207892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7680cf7d0 0x7fc768159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.875+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 >> 172.21.15.125:0/1416207892 conn(0x7fc76806a800 msgr2=0x7fc768140bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.876+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 shutdown_connections 2024-01-16T15:14:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:52.876+0000 7fc76f200640 1 -- 172.21.15.125:0/1416207892 wait complete. 2024-01-16T15:14:52.874 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:14:52.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:52.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:14:52.952 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:14:52.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:52.952 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:14:52.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[74204]: fuse finished with error 0 and tester_r 0 2024-01-16T15:14:53.161 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:14:53.161 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:53.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.445+0000 7f6fbd800640 1 Processor -- start 2024-01-16T15:14:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.446+0000 7f6fbd800640 1 -- start start 2024-01-16T15:14:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fbd800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fb815a110 0x7f6fb806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fbd800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 0x7f6fb806dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fbd800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80b3880 con 0x7f6fb8066300 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fbd800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80b46e0 con 0x7f6fb815a110 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fbd800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb806d900 con 0x7f6fb806bb00 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fb6a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fb815a110 0x7f6fb806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fb6000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6fb8066300 0x7f6fb806b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52496/0 (socket says 172.21.15.125:52496) 2024-01-16T15:14:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.447+0000 7f6fb6000640 1 -- 172.21.15.125:0/74015373 learned_addr learned my addr 172.21.15.125:0/74015373 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb7400640 1 --2- 172.21.15.125:0/74015373 >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 0x7f6fb806dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb6a00640 1 -- 172.21.15.125:0/74015373 >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 msgr2=0x7f6fb806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/74015373 >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 0x7f6fb806dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb6a00640 1 -- 172.21.15.125:0/74015373 >> v1:172.21.15.125:6789/0 conn(0x7f6fb8066300 legacy=0x7f6fb806b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb6a00640 1 -- 172.21.15.125:0/74015373 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb815a4f0 con 0x7f6fb815a110 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb5600640 1 -- 172.21.15.125:0/74015373 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727719386 0 0) 0x7f6fb80b3880 con 0x7f6fb8066300 2024-01-16T15:14:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.448+0000 7f6fb7400640 1 --2- 172.21.15.125:0/74015373 >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 0x7f6fb806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.449+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/74015373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb815a110 0x7f6fb806d3c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80aef40 tx=0x7f6fa802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffc7496aea5cac5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.449+0000 7f6fb5600640 1 -- 172.21.15.125:0/74015373 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8031830 con 0x7f6fb815a110 2024-01-16T15:14:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.449+0000 7f6fb5600640 1 -- 172.21.15.125:0/74015373 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8031990 con 0x7f6fb815a110 2024-01-16T15:14:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.449+0000 7f6fb5600640 1 -- 172.21.15.125:0/74015373 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8031d20 con 0x7f6fb815a110 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 -- 172.21.15.125:0/74015373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb815a110 msgr2=0x7f6fb806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 --2- 172.21.15.125:0/74015373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb815a110 0x7f6fb806d3c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80aef40 tx=0x7f6fa802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 -- 172.21.15.125:0/74015373 shutdown_connections 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 --2- 172.21.15.125:0/74015373 >> v2:172.21.15.137:3300/0 conn(0x7f6fb806bb00 0x7f6fb806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 --2- 172.21.15.125:0/74015373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb815a110 0x7f6fb806d3c0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.450+0000 7f6fbd800640 1 -- 172.21.15.125:0/74015373 >> 172.21.15.125:0/74015373 conn(0x7f6fb8070b10 msgr2=0x7f6fb8070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.451+0000 7f6fbd800640 1 -- 172.21.15.125:0/74015373 shutdown_connections 2024-01-16T15:14:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.451+0000 7f6fbd800640 1 -- 172.21.15.125:0/74015373 wait complete. 2024-01-16T15:14:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.453+0000 7f6fbd800640 1 Processor -- start 2024-01-16T15:14:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.453+0000 7f6fbd800640 1 -- start start 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 0x7f6fb8144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb815a110 0x7f6fb8150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb8145870 con 0x7f6fb806bb00 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81481d0 con 0x7f6fb815a110 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.454+0000 7f6fbd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80cded0 con 0x7f6fb8066300 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36952/0 (socket says 172.21.15.125:36952) 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 -- 172.21.15.125:0/15988542 learned_addr learned my addr 172.21.15.125:0/15988542 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 0x7f6fb8144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb7400640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb815a110 0x7f6fb8150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 -- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 msgr2=0x7f6fb8144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 0x7f6fb8144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 -- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb815a110 msgr2=0x7f6fb8150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb815a110 0x7f6fb8150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 -- 172.21.15.125:0/15988542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8009400 con 0x7f6fb806bb00 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 0x7f6fb8144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.455+0000 7f6fb6000640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c4d0 tx=0x7f6fa000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.456+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00074e0 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.456+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0007660 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.456+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0004a20 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.456+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb8148470 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.456+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb80ce1b0 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.457+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb8148990 con 0x7f6fb806bb00 2024-01-16T15:14:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.458+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6fa0004b80 con 0x7f6fb806bb00 2024-01-16T15:14:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.459+0000 7f6faea00640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 0x7f6f9006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.459+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fa0087c50 con 0x7f6fb806bb00 2024-01-16T15:14:53.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.459+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 0x7f6f9006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.460+0000 7f6fb6a00640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 0x7f6f9006e8f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036000 tx=0x7f6fa8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.463+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fb8148990 con 0x7f6fb806bb00 2024-01-16T15:14:53.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.655+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fb8148990 con 0x7f6fb806bb00 2024-01-16T15:14:53.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.657+0000 7f6faea00640 1 -- 172.21.15.125:0/15988542 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f6fb8148990 con 0x7f6fb806bb00 2024-01-16T15:14:53.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:53.640 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":103,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:14:51.154789+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/737412716":"2024-01-17T15:14:48.135036+0000","172.21.15.125:6835/737412716":"2024-01-17T15:14:48.135036+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 msgr2=0x7f6f9006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 0x7f6f9006e8f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036000 tx=0x7f6fa8008030 comp rx=0 tx=0).stop 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 msgr2=0x7f6fb814fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c4d0 tx=0x7f6fa000c990 comp rx=0 tx=0).stop 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 shutdown_connections 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb815a110 0x7f6fb8150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f9006c430 0x7f6f9006e8f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb806bb00 0x7f6fb814fac0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 --2- 172.21.15.125:0/15988542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb8066300 0x7f6fb8144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.659+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 >> 172.21.15.125:0/15988542 conn(0x7f6fb8070b10 msgr2=0x7f6fb806a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.660+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 shutdown_connections 2024-01-16T15:14:53.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.660+0000 7f6fbd800640 1 -- 172.21.15.125:0/15988542 wait complete. 2024-01-16T15:14:53.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:53.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.883+0000 7f54cce00640 1 Processor -- start 2024-01-16T15:14:53.866 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.884+0000 7f54cce00640 1 -- start start 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.885+0000 7f54cce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f54c813df80 0x7f54c815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.885+0000 7f54cce00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54c813e360 0x7f54c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.885+0000 7f54cce00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c806c470 con 0x7f54c815f9c0 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.885+0000 7f54cce00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c806c750 con 0x7f54c813e360 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.885+0000 7f54cce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c806ca30 con 0x7f54c813df80 2024-01-16T15:14:53.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54c813e360 0x7f54c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c6800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f54c815f9c0 0x7f54c81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52498/0 (socket says 172.21.15.125:52498) 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c6800640 1 -- 172.21.15.125:0/359238954 learned_addr learned my addr 172.21.15.125:0/359238954 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5e00640 1 --2- 172.21.15.125:0/359238954 >> v2:172.21.15.137:3300/0 conn(0x7f54c813df80 0x7f54c815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5400640 1 -- 172.21.15.125:0/359238954 >> v2:172.21.15.137:3300/0 conn(0x7f54c813df80 msgr2=0x7f54c815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5400640 1 --2- 172.21.15.125:0/359238954 >> v2:172.21.15.137:3300/0 conn(0x7f54c813df80 0x7f54c815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5400640 1 -- 172.21.15.125:0/359238954 >> v1:172.21.15.125:6789/0 conn(0x7f54c815f9c0 legacy=0x7f54c81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.886+0000 7f54c5400640 1 -- 172.21.15.125:0/359238954 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c806cd10 con 0x7f54c813e360 2024-01-16T15:14:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.887+0000 7f54c5400640 1 --2- 172.21.15.125:0/359238954 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813e360 0x7f54c80cf290 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f54c8072770 tx=0x7f54b002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec86fe0d730a1d28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.887+0000 7f54c4a00640 1 -- 172.21.15.125:0/359238954 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b002cc70 con 0x7f54c813e360 2024-01-16T15:14:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54c4a00640 1 -- 172.21.15.125:0/359238954 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b002cdd0 con 0x7f54c813e360 2024-01-16T15:14:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54c4a00640 1 -- 172.21.15.125:0/359238954 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0034660 con 0x7f54c813e360 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 -- 172.21.15.125:0/359238954 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813e360 msgr2=0x7f54c80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 --2- 172.21.15.125:0/359238954 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813e360 0x7f54c80cf290 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f54c8072770 tx=0x7f54b002bec0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 -- 172.21.15.125:0/359238954 shutdown_connections 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 --2- 172.21.15.125:0/359238954 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813e360 0x7f54c80cf290 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 --2- 172.21.15.125:0/359238954 >> v2:172.21.15.137:3300/0 conn(0x7f54c813df80 0x7f54c815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 -- 172.21.15.125:0/359238954 >> 172.21.15.125:0/359238954 conn(0x7f54c806a800 msgr2=0x7f54c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 -- 172.21.15.125:0/359238954 shutdown_connections 2024-01-16T15:14:53.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.888+0000 7f54cce00640 1 -- 172.21.15.125:0/359238954 wait complete. 2024-01-16T15:14:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.890+0000 7f54cce00640 1 Processor -- start 2024-01-16T15:14:53.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.890+0000 7f54cce00640 1 -- start start 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f54c813e360 0x7f54c813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54c815f9c0 0x7f54c813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c814d8d0 con 0x7f54c813df80 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c814e170 con 0x7f54c813e360 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.891+0000 7f54cce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c814f8b0 con 0x7f54c815f9c0 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36980/0 (socket says 172.21.15.125:36980) 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 -- 172.21.15.125:0/2647799519 learned_addr learned my addr 172.21.15.125:0/2647799519 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5400640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f54c813e360 0x7f54c813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c6800640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54c815f9c0 0x7f54c813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 -- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54c815f9c0 msgr2=0x7f54c813d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54c815f9c0 0x7f54c813d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 -- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f54c813e360 msgr2=0x7f54c813d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f54c813e360 0x7f54c813d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 -- 172.21.15.125:0/2647799519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0009a70 con 0x7f54c813df80 2024-01-16T15:14:53.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.892+0000 7f54c5e00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f54b800d620 tx=0x7f54b800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.893+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b800b620 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.893+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b800b780 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.893+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b800bb30 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.893+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c815dfc0 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.893+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c814de00 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.894+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80675d0 con 0x7f54c813df80 2024-01-16T15:14:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.895+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f54b800bc90 con 0x7f54c813df80 2024-01-16T15:14:53.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.896+0000 7f54be000640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 0x7f54a006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.896+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f54b8088f70 con 0x7f54c813df80 2024-01-16T15:14:53.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.896+0000 7f54c5400640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 0x7f54a006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.897+0000 7f54c5400640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 0x7f54a006e8f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f54b0036000 tx=0x7f54b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:53.900+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54b8055540 con 0x7f54c813df80 2024-01-16T15:14:54.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.081+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54c806df00 con 0x7f54c813df80 2024-01-16T15:14:54.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.081+0000 7f54be000640 1 -- 172.21.15.125:0/2647799519 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54b8058b60 con 0x7f54c813df80 2024-01-16T15:14:54.065 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 msgr2=0x7f54a006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 0x7f54a006e8f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f54b0036000 tx=0x7f54b0030000 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 msgr2=0x7f54c815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f54b800d620 tx=0x7f54b800dae0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 shutdown_connections 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f54c815f9c0 0x7f54c813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f54c813e360 0x7f54c813d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f54a006c430 0x7f54a006e8f0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 --2- 172.21.15.125:0/2647799519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54c813df80 0x7f54c815d740 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 >> 172.21.15.125:0/2647799519 conn(0x7f54c806a800 msgr2=0x7f54c80b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 shutdown_connections 2024-01-16T15:14:54.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.084+0000 7f54cce00640 1 -- 172.21.15.125:0/2647799519 wait complete. 2024-01-16T15:14:54.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:14:54.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.304+0000 7f84eae00640 1 Processor -- start 2024-01-16T15:14:54.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.304+0000 7f84eae00640 1 -- start start 2024-01-16T15:14:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84eae00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f84e4160890 0x7f84e40d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84eae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 0x7f84e40d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84eae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40cf6a0 con 0x7f84e40c6be0 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84eae00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e40d1430 con 0x7f84e4160890 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84eae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40b4370 con 0x7f84e4160c70 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84ea400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f84e40c6be0 0x7f84e4161110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52504/0 (socket says 172.21.15.125:52504) 2024-01-16T15:14:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.306+0000 7f84ea400640 1 -- 172.21.15.125:0/1271525415 learned_addr learned my addr 172.21.15.125:0/1271525415 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:54.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e9a00640 1 --2- 172.21.15.125:0/1271525415 >> v2:172.21.15.137:3300/0 conn(0x7f84e4160890 0x7f84e40d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e9000640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 0x7f84e40d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926852170 0 0) 0x7f84e40cf6a0 con 0x7f84e40c6be0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c4003520 con 0x7f84e40c6be0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255176295 0 0) 0x7f84c4003520 con 0x7f84e40c6be0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 msgr2=0x7f84e40d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 0x7f84e40d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 >> v2:172.21.15.137:3300/0 conn(0x7f84e4160890 msgr2=0x7f84e40d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 --2- 172.21.15.125:0/1271525415 >> v2:172.21.15.137:3300/0 conn(0x7f84e4160890 0x7f84e40d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40ce7c0 con 0x7f84e40c6be0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f84dc001670 con 0x7f84e40c6be0 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 >> v1:172.21.15.125:6789/0 conn(0x7f84e40c6be0 legacy=0x7f84e4161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84c4003970 0x7f84c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84c40047c0 0x7f84c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 0x7f84c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40d1430 con 0x7f84c40047c0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40b4370 con 0x7f84c4003970 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c400bab0 con 0x7f84e4160c70 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.307+0000 7f84e9000640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84c40047c0 0x7f84c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9a00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84c4003970 0x7f84c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9000640 1 -- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 msgr2=0x7f84c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9000640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84e4160c70 0x7f84c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9000640 1 -- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84c4003970 msgr2=0x7f84c4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9000640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84c4003970 0x7f84c4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9000640 1 -- 172.21.15.125:0/1271525415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4003520 con 0x7f84c40047c0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.308+0000 7f84e9a00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84c4003970 0x7f84c4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.309+0000 7f84e9000640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84c40047c0 0x7f84c4004c40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00ba00 tx=0x7f84c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e51d7650503e12 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.309+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc032640 con 0x7f84c40047c0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.309+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc0327a0 con 0x7f84c40047c0 2024-01-16T15:14:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.309+0000 7f84e3e00640 1 -- 172.21.15.125:0/1271525415 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc032b30 con 0x7f84c40047c0 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 -- 172.21.15.125:0/1271525415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84c40047c0 msgr2=0x7f84c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 --2- 172.21.15.125:0/1271525415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84c40047c0 0x7f84c4004c40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00ba00 tx=0x7f84c40057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84ea400640 1 -- 172.21.15.125:0/1271525415 reap_dead start 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 -- 172.21.15.125:0/1271525415 shutdown_connections 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 -- 172.21.15.125:0/1271525415 >> 172.21.15.125:0/1271525415 conn(0x7f84e40d3090 msgr2=0x7f84e40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 -- 172.21.15.125:0/1271525415 shutdown_connections 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.310+0000 7f84eae00640 1 -- 172.21.15.125:0/1271525415 wait complete. 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.311+0000 7f84eae00640 1 Processor -- start 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 -- start start 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84c40047c0 0x7f84e415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84e40cc090 0x7f84e40cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4143250 con 0x7f84e40c5fc0 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40ccdc0 con 0x7f84e40cc090 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.312+0000 7f84eae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40cd0a0 con 0x7f84c40047c0 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37004/0 (socket says 172.21.15.125:37004) 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 -- 172.21.15.125:0/792502420 learned_addr learned my addr 172.21.15.125:0/792502420 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9a00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84c40047c0 0x7f84e415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 -- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84c40047c0 msgr2=0x7f84e415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84c40047c0 0x7f84e415ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 -- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84e40cc090 msgr2=0x7f84e40cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84ea400640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84e40cc090 0x7f84e40cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84e40cc090 0x7f84e40cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.313+0000 7f84e9000640 1 -- 172.21.15.125:0/792502420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e41434c0 con 0x7f84e40c5fc0 2024-01-16T15:14:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84e9000640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f84cc02bec0 tx=0x7f84cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc004220 con 0x7f84e40c5fc0 2024-01-16T15:14:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc0043a0 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc03ba30 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e40c6980 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.314+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e40cca50 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.315+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4143650 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.316+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f84cc03bb90 con 0x7f84e40c5fc0 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.317+0000 7f84e1600640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 0x7f84bc06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.317+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f84cc0ab030 con 0x7f84e40c5fc0 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.317+0000 7f84e9a00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 0x7f84bc06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.318+0000 7f84e9a00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 0x7f84bc06e920 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f84e4159910 tx=0x7f84d400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.321+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84cc0779f0 con 0x7f84e40c5fc0 2024-01-16T15:14:54.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.526+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f84e4143650 con 0x7f84e40c5fc0 2024-01-16T15:14:54.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.527+0000 7f84e1600640 1 -- 172.21.15.125:0/792502420 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v103) v1 ==== 80+0+122 (secure 0 0 0) 0x7f84cc036080 con 0x7f84e40c5fc0 2024-01-16T15:14:54.510 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/737412716 2024-01-17T15:14:48.135036+0000 2024-01-16T15:14:54.510 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6835/737412716 2024-01-17T15:14:48.135036+0000 2024-01-16T15:14:54.510 INFO:teuthology.orchestra.run.smithi125.stderr:listed 2 entries 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 msgr2=0x7f84bc06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 0x7f84bc06e920 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f84e4159910 tx=0x7f84d400b040 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 msgr2=0x7f84e40c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f84cc02bec0 tx=0x7f84cc004730 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 shutdown_connections 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84e40cc090 0x7f84e40cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f84bc06c460 0x7f84bc06e920 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e40c5fc0 0x7f84e40c6440 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 --2- 172.21.15.125:0/792502420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84c40047c0 0x7f84e415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 >> 172.21.15.125:0/792502420 conn(0x7f84e40d3090 msgr2=0x7f84e406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 shutdown_connections 2024-01-16T15:14:54.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:54.530+0000 7f84eae00640 1 -- 172.21.15.125:0/792502420 wait complete. 2024-01-16T15:14:54.533 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:14:54.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:54.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:14:54.628 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46572]: fuse finished with error 0 and tester_r 0 2024-01-16T15:14:54.631 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:14:54.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:54.632 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:14:54.811 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:14:54.811 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:54.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:14:54.868 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:14:54.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:54.868 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:14:54.919 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:14:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:14:54.921 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:14:54.921 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:54.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:14:54.979 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:14:54.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:54.979 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:14:55.032 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:14:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:14:55.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:14:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.270+0000 7fbab9c00640 1 Processor -- start 2024-01-16T15:14:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.270+0000 7fbab9c00640 1 -- start start 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.271+0000 7fbab9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab4143ae0 0x7fbab40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.271+0000 7fbab9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab415e3b0 0x7fbab40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.271+0000 7fbab9c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40cfc30 con 0x7fbab40cf7d0 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.271+0000 7fbab9c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab40b1ee0 con 0x7fbab4143ae0 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.271+0000 7fbab9c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b2780 con 0x7fbab415e3b0 2024-01-16T15:14:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab4143ae0 0x7fbab40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab4143ae0 0x7fbab40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37016/0 (socket says 172.21.15.125:37016) 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 -- 172.21.15.125:0/3409000505 learned_addr learned my addr 172.21.15.125:0/3409000505 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 -- 172.21.15.125:0/3409000505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab415e3b0 msgr2=0x7fbab40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 --2- 172.21.15.125:0/3409000505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab415e3b0 0x7fbab40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 -- 172.21.15.125:0/3409000505 >> v1:172.21.15.137:6789/0 conn(0x7fbab40cf7d0 legacy=0x7fbab415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.272+0000 7fbab2a00640 1 -- 172.21.15.125:0/3409000505 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab415e990 con 0x7fbab4143ae0 2024-01-16T15:14:55.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.273+0000 7fbab2a00640 1 --2- 172.21.15.125:0/3409000505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab4143ae0 0x7fbab40cf290 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400bb10 tx=0x7fbaa402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1ffaebbca86f85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab2000640 1 -- 172.21.15.125:0/3409000505 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa402ee70 con 0x7fbab4143ae0 2024-01-16T15:14:55.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab2000640 1 -- 172.21.15.125:0/3409000505 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa4036440 con 0x7fbab4143ae0 2024-01-16T15:14:55.256 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab2000640 1 -- 172.21.15.125:0/3409000505 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa40367d0 con 0x7fbab4143ae0 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 -- 172.21.15.125:0/3409000505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab4143ae0 msgr2=0x7fbab40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 --2- 172.21.15.125:0/3409000505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab4143ae0 0x7fbab40cf290 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400bb10 tx=0x7fbaa402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 -- 172.21.15.125:0/3409000505 shutdown_connections 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 --2- 172.21.15.125:0/3409000505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab415e3b0 0x7fbab40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 --2- 172.21.15.125:0/3409000505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab4143ae0 0x7fbab40cf290 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 -- 172.21.15.125:0/3409000505 >> 172.21.15.125:0/3409000505 conn(0x7fbab406a800 msgr2=0x7fbab406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 -- 172.21.15.125:0/3409000505 shutdown_connections 2024-01-16T15:14:55.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.274+0000 7fbab9c00640 1 -- 172.21.15.125:0/3409000505 wait complete. 2024-01-16T15:14:55.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.276+0000 7fbab9c00640 1 Processor -- start 2024-01-16T15:14:55.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.276+0000 7fbab9c00640 1 -- start start 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 0x7fbab4137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 0x7fbab40aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40aef60 con 0x7fbab415e3b0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4159f40 con 0x7fbab40cf7d0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab9c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40af1e0 con 0x7fbab4143ae0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37020/0 (socket says 172.21.15.125:37020) 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.277+0000 7fbab3400640 1 -- 172.21.15.125:0/4125605340 learned_addr learned my addr 172.21.15.125:0/4125605340 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab2a00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 0x7fbab4137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbaabe00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 0x7fbab40aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 -- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 msgr2=0x7fbab40aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 0x7fbab40aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 -- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 msgr2=0x7fbab4137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 0x7fbab4137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 -- 172.21.15.125:0/4125605340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa400b770 con 0x7fbab415e3b0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab2a00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 0x7fbab4137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbaabe00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 0x7fbab40aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab3400640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b970 tx=0x7fbaac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00cb40 con 0x7fbab415e3b0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac00ccc0 con 0x7fbab415e3b0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac014680 con 0x7fbab415e3b0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.278+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab40af4c0 con 0x7fbab415e3b0 2024-01-16T15:14:55.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.279+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab40afa80 con 0x7fbab415e3b0 2024-01-16T15:14:55.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.280+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab415e1d0 con 0x7fbab415e3b0 2024-01-16T15:14:55.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.281+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fbaac01d3b0 con 0x7fbab415e3b0 2024-01-16T15:14:55.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.281+0000 7fbaaaa00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 0x7fba8c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.282+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbaac088a10 con 0x7fbab415e3b0 2024-01-16T15:14:55.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.282+0000 7fbab2a00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 0x7fba8c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.282+0000 7fbab2a00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 0x7fba8c06e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4038000 tx=0x7fbaa4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.286+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaac054fe0 con 0x7fbab415e3b0 2024-01-16T15:14:55.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.512+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab40675d0 con 0x7fbab415e3b0 2024-01-16T15:14:55.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.513+0000 7fbaaaa00640 1 -- 172.21.15.125:0/4125605340 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fbaac019070 con 0x7fbab415e3b0 2024-01-16T15:14:55.496 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:55.497 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":66,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6460,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2589810407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2589810407},{"type":"v1","addr":"172.21.15.137:6834","nonce":2589810407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6463,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1674315039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":1674315039},{"type":"v1","addr":"172.21.15.137:6835","nonce":1674315039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":8646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/995017134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":995017134},{"type":"v1","addr":"172.21.15.125:6836","nonce":995017134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/752965027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":752965027},{"type":"v1","addr":"172.21.15.125:6837","nonce":752965027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[]} 2024-01-16T15:14:55.497 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 66 2024-01-16T15:14:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 msgr2=0x7fba8c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 0x7fba8c06e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4038000 tx=0x7fbaa4008040 comp rx=0 tx=0).stop 2024-01-16T15:14:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 msgr2=0x7fbab4159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b970 tx=0x7fbaac00be30 comp rx=0 tx=0).stop 2024-01-16T15:14:55.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 shutdown_connections 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fba8c06c430 0x7fba8c06e8f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab415e3b0 0x7fbab4159a00 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbab4143ae0 0x7fbab40aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 --2- 172.21.15.125:0/4125605340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbab40cf7d0 0x7fbab4137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 >> 172.21.15.125:0/4125605340 conn(0x7fbab406a800 msgr2=0x7fbab415c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.516+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 shutdown_connections 2024-01-16T15:14:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.517+0000 7fbab9c00640 1 -- 172.21.15.125:0/4125605340 wait complete. 2024-01-16T15:14:55.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:14:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.739+0000 7f6fb7800640 1 Processor -- start 2024-01-16T15:14:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.739+0000 7f6fb7800640 1 -- start start 2024-01-16T15:14:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb7800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6fb0150760 0x7f6fb0072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb7800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fb015d300 0x7f6fb015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb7800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb015dc80 con 0x7f6fb00b2bd0 2024-01-16T15:14:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb7800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb00b3710 con 0x7f6fb015d300 2024-01-16T15:14:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb7800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb0071190 con 0x7f6fb0150760 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fafe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fb015d300 0x7f6fb015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fafe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fb015d300 0x7f6fb015d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37022/0 (socket says 172.21.15.125:37022) 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fafe00640 1 -- 172.21.15.125:0/2239863953 learned_addr learned my addr 172.21.15.125:0/2239863953 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.741+0000 7f6fb4e00640 1 --2- 172.21.15.125:0/2239863953 >> v2:172.21.15.137:3300/0 conn(0x7f6fb0150760 0x7f6fb0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.742+0000 7f6fafe00640 1 -- 172.21.15.125:0/2239863953 >> v2:172.21.15.137:3300/0 conn(0x7f6fb0150760 msgr2=0x7f6fb0072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.742+0000 7f6fafe00640 1 --2- 172.21.15.125:0/2239863953 >> v2:172.21.15.137:3300/0 conn(0x7f6fb0150760 0x7f6fb0072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.742+0000 7f6fafe00640 1 -- 172.21.15.125:0/2239863953 >> v1:172.21.15.137:6789/0 conn(0x7f6fb00b2bd0 legacy=0x7f6fb00b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.742+0000 7f6fafe00640 1 -- 172.21.15.125:0/2239863953 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0071470 con 0x7f6fb015d300 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.743+0000 7f6fafe00640 1 --2- 172.21.15.125:0/2239863953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb015d740 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6fb006e420 tx=0x7f6f9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc07d5c2fcdb31b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.743+0000 7f6faf400640 1 -- 172.21.15.125:0/2239863953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9802ca80 con 0x7f6fb015d300 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.743+0000 7f6faf400640 1 -- 172.21.15.125:0/2239863953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9802cbe0 con 0x7f6fb015d300 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.743+0000 7f6faf400640 1 -- 172.21.15.125:0/2239863953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f980343f0 con 0x7f6fb015d300 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 -- 172.21.15.125:0/2239863953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 msgr2=0x7f6fb015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2239863953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb015d740 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6fb006e420 tx=0x7f6f9802bee0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 -- 172.21.15.125:0/2239863953 shutdown_connections 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2239863953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb015d740 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2239863953 >> v2:172.21.15.137:3300/0 conn(0x7f6fb0150760 0x7f6fb0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 -- 172.21.15.125:0/2239863953 >> 172.21.15.125:0/2239863953 conn(0x7f6fb006a800 msgr2=0x7f6fb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 -- 172.21.15.125:0/2239863953 shutdown_connections 2024-01-16T15:14:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.744+0000 7f6fb7800640 1 -- 172.21.15.125:0/2239863953 wait complete. 2024-01-16T15:14:55.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.745+0000 7f6fb7800640 1 Processor -- start 2024-01-16T15:14:55.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.746+0000 7f6fb7800640 1 -- start start 2024-01-16T15:14:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb00b2bd0 0x7f6fb0150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb0150760 0x7f6fb0148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cd560 con 0x7f6fb015d300 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cc8e0 con 0x7f6fb00b2bd0 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00ccbc0 con 0x7f6fb0150760 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37032/0 (socket says 172.21.15.125:37032) 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 -- 172.21.15.125:0/2842307436 learned_addr learned my addr 172.21.15.125:0/2842307436 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fafe00640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb0150760 0x7f6fb0148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 -- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb0150760 msgr2=0x7f6fb0148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb0150760 0x7f6fb0148a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 -- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb00b2bd0 msgr2=0x7f6fb0150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb4e00640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb00b2bd0 0x7f6fb0150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb00b2bd0 0x7f6fb0150da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.747+0000 7f6fb5800640 1 -- 172.21.15.125:0/2842307436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f98009b30 con 0x7f6fb015d300 2024-01-16T15:14:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6fb5800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800b490 tx=0x7f6fa800b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8011830 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa80119b0 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8011d60 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00ccea0 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.748+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb01e33b0 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.749+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb00cd6a0 con 0x7f6fb015d300 2024-01-16T15:14:55.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.750+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6fa801a400 con 0x7f6fb015d300 2024-01-16T15:14:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.751+0000 7f6facc00640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 0x7f6f8806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.751+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fa8088bd0 con 0x7f6fb015d300 2024-01-16T15:14:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.751+0000 7f6fb4e00640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 0x7f6f8806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.752+0000 7f6fb4e00640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 0x7f6f8806e8f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006fd0 tx=0x7f6fa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.755+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fa8055140 con 0x7f6fb015d300 2024-01-16T15:14:55.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.946+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f6fb00675d0 con 0x7f6fb015d300 2024-01-16T15:14:55.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.951+0000 7f6facc00640 1 -- 172.21.15.125:0/2842307436 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6fa8015020 con 0x7f6fb015d300 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.954+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 msgr2=0x7f6f8806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.954+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 0x7f6f8806e8f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006fd0 tx=0x7f6fa0008040 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.954+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 msgr2=0x7f6fb00cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.954+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800b490 tx=0x7f6fa800b950 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 shutdown_connections 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6f8806c430 0x7f6f8806e8f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fb015d300 0x7f6fb00cbe40 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6fb0150760 0x7f6fb0148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 --2- 172.21.15.125:0/2842307436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb00b2bd0 0x7f6fb0150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 >> 172.21.15.125:0/2842307436 conn(0x7f6fb006a800 msgr2=0x7f6fb0159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 shutdown_connections 2024-01-16T15:14:55.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:55.955+0000 7f6fb7800640 1 -- 172.21.15.125:0/2842307436 wait complete. 2024-01-16T15:14:55.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:14:56.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.171+0000 7f0f92600640 1 Processor -- start 2024-01-16T15:14:56.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.171+0000 7f0f92600640 1 -- start start 2024-01-16T15:14:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.172+0000 7f0f92600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0f8c15da10 0x7f0f8c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.172+0000 7f0f92600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 0x7f0f8c0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.172+0000 7f0f92600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c071190 con 0x7f0f8c150760 2024-01-16T15:14:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.172+0000 7f0f92600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c071470 con 0x7f0f8c15da10 2024-01-16T15:14:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.172+0000 7f0f92600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c071750 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.173+0000 7f0f8be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 0x7f0f8c0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.173+0000 7f0f82800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0f8c15da10 0x7f0f8c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.173+0000 7f0f8b400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f0f8c150760 0x7f0f8c15d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:33218/0 (socket says 172.21.15.125:33218) 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.173+0000 7f0f8b400640 1 -- 172.21.15.125:0/937558002 learned_addr learned my addr 172.21.15.125:0/937558002 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.174+0000 7f0f8be00640 1 -- 172.21.15.125:0/937558002 >> v2:172.21.15.137:3300/0 conn(0x7f0f8c15da10 msgr2=0x7f0f8c072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.174+0000 7f0f8be00640 1 --2- 172.21.15.125:0/937558002 >> v2:172.21.15.137:3300/0 conn(0x7f0f8c15da10 0x7f0f8c072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.174+0000 7f0f8be00640 1 -- 172.21.15.125:0/937558002 >> v1:172.21.15.137:6789/0 conn(0x7f0f8c150760 legacy=0x7f0f8c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.174+0000 7f0f8be00640 1 -- 172.21.15.125:0/937558002 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c071a30 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.175+0000 7f0f8be00640 1 --2- 172.21.15.125:0/937558002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 0x7f0f8c0b3060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400aea0 tx=0x7f0f8402cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=719d5def5e907284 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f8aa00640 1 -- 172.21.15.125:0/937558002 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8402db40 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f8aa00640 1 -- 172.21.15.125:0/937558002 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8402dca0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f8aa00640 1 -- 172.21.15.125:0/937558002 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84008660 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f92600640 1 -- 172.21.15.125:0/937558002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 msgr2=0x7f0f8c0b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f92600640 1 --2- 172.21.15.125:0/937558002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 0x7f0f8c0b3060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400aea0 tx=0x7f0f8402cfa0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f92600640 1 -- 172.21.15.125:0/937558002 shutdown_connections 2024-01-16T15:14:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f92600640 1 --2- 172.21.15.125:0/937558002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c0b2bd0 0x7f0f8c0b3060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.176+0000 7f0f92600640 1 --2- 172.21.15.125:0/937558002 >> v2:172.21.15.137:3300/0 conn(0x7f0f8c15da10 0x7f0f8c072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.177+0000 7f0f92600640 1 -- 172.21.15.125:0/937558002 >> 172.21.15.125:0/937558002 conn(0x7f0f8c06a800 msgr2=0x7f0f8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.177+0000 7f0f92600640 1 -- 172.21.15.125:0/937558002 shutdown_connections 2024-01-16T15:14:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.177+0000 7f0f92600640 1 -- 172.21.15.125:0/937558002 wait complete. 2024-01-16T15:14:56.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.178+0000 7f0f92600640 1 Processor -- start 2024-01-16T15:14:56.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.178+0000 7f0f92600640 1 -- start start 2024-01-16T15:14:56.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f8c150760 0x7f0f8c0adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c15da10 0x7f0f8c0cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1f2fd0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c151e40 con 0x7f0f8c150760 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.179+0000 7f0f92600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c152120 con 0x7f0f8c15da10 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37048/0 (socket says 172.21.15.125:37048) 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 -- 172.21.15.125:0/3482549301 learned_addr learned my addr 172.21.15.125:0/3482549301 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8be00640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c15da10 0x7f0f8c0cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f82800640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f8c150760 0x7f0f8c0adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 -- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c15da10 msgr2=0x7f0f8c0cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c15da10 0x7f0f8c0cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 -- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f8c150760 msgr2=0x7f0f8c0adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f8c150760 0x7f0f8c0adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.180+0000 7f0f8b400640 1 -- 172.21.15.125:0/3482549301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8400ab80 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f8b400640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00c4a0 tx=0x7f0f7c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c005ce0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c005e40 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c007270 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c152400 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.181+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1529c0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.182+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c152bb0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.183+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f0f7c007490 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.184+0000 7f0f83e00640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 0x7f0f6406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.184+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0f7c087f40 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.184+0000 7f0f82800640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 0x7f0f6406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.185+0000 7f0f82800640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 0x7f0f6406e8f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c15bc30 tx=0x7f0f74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.188+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f7c054560 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.380+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f0f8c0675d0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.385+0000 7f0f83e00640 1 -- 172.21.15.125:0/3482549301 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v40) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0f7c0170b0 con 0x7f0f8c0b2bd0 2024-01-16T15:14:56.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 msgr2=0x7f0f6406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 0x7f0f6406e8f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c15bc30 tx=0x7f0f74008040 comp rx=0 tx=0).stop 2024-01-16T15:14:56.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 msgr2=0x7f0f8c1f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00c4a0 tx=0x7f0f7c00c960 comp rx=0 tx=0).stop 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 shutdown_connections 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f8c15da10 0x7f0f8c0cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f8c150760 0x7f0f8c0adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f6406c430 0x7f0f6406e8f0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 --2- 172.21.15.125:0/3482549301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8c0b2bd0 0x7f0f8c1f2a90 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.388+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 >> 172.21.15.125:0/3482549301 conn(0x7f0f8c06a800 msgr2=0x7f0f8c0ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.389+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 shutdown_connections 2024-01-16T15:14:56.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.389+0000 7f0f92600640 1 -- 172.21.15.125:0/3482549301 wait complete. 2024-01-16T15:14:56.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:14:56.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.606+0000 7f0b0bc00640 1 Processor -- start 2024-01-16T15:14:56.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.607+0000 7f0b0bc00640 1 -- start start 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.608+0000 7f0b0bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0b0415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.608+0000 7f0b0bc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0b0413e360 0x7f0b040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.608+0000 7f0b0bc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0406c470 con 0x7f0b0415f9c0 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.608+0000 7f0b0bc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0406c750 con 0x7f0b0413e360 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.608+0000 7f0b0bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0406ca30 con 0x7f0b0413df80 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b09c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0b0415f9c0 0x7f0b041408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45230/0 (socket says 172.21.15.125:45230) 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b09c00640 1 -- 172.21.15.125:0/3182929510 learned_addr learned my addr 172.21.15.125:0/3182929510 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b09200640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0b0415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464957458 0 0) 0x7f0b0406c470 con 0x7f0b0415f9c0 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0afbc00640 1 --2- 172.21.15.125:0/3182929510 >> v2:172.21.15.137:3300/0 conn(0x7f0b0413e360 0x7f0b040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae4003520 con 0x7f0b0415f9c0 2024-01-16T15:14:56.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743642579 0 0) 0x7f0ae4003520 con 0x7f0b0415f9c0 2024-01-16T15:14:56.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 msgr2=0x7f0b0415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.609+0000 7f0b03e00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0b0415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 >> v2:172.21.15.137:3300/0 conn(0x7f0b0413e360 msgr2=0x7f0b040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 --2- 172.21.15.125:0/3182929510 >> v2:172.21.15.137:3300/0 conn(0x7f0b0413e360 0x7f0b040cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0406cd10 con 0x7f0b0415f9c0 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0afc001670 con 0x7f0b0415f9c0 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 >> v1:172.21.15.125:6789/0 conn(0x7f0b0415f9c0 legacy=0x7f0b041408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4003970 0x7f0ae4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0ae40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ae4004d00 0x7f0ae4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0406c750 con 0x7f0ae4004d00 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0406ca30 con 0x7f0ae4003970 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae400b980 con 0x7f0b0413df80 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09c00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ae4004d00 0x7f0ae4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09200640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4003970 0x7f0ae4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0afbc00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0ae40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09c00640 1 -- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 msgr2=0x7f0ae40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09c00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0ae40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09c00640 1 -- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4003970 msgr2=0x7f0ae4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.610+0000 7f0b09c00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4003970 0x7f0ae4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.611+0000 7f0b09c00640 1 -- 172.21.15.125:0/3182929510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae4003520 con 0x7f0ae4004d00 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.611+0000 7f0afbc00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413df80 0x7f0ae40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.611+0000 7f0b09200640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4003970 0x7f0ae4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b09c00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ae4004d00 0x7f0ae4005180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c2c0 tx=0x7f0ae40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca104fb6c18e4b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc005ad0 con 0x7f0ae4004d00 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc005c30 con 0x7f0ae4004d00 2024-01-16T15:14:56.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b03e00640 1 -- 172.21.15.125:0/3182929510 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc02c8b0 con 0x7f0ae4004d00 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b0bc00640 1 -- 172.21.15.125:0/3182929510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ae4004d00 msgr2=0x7f0ae4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/3182929510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ae4004d00 0x7f0ae4005180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c2c0 tx=0x7f0ae40057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.612+0000 7f0b09c00640 1 -- 172.21.15.125:0/3182929510 reap_dead start 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.613+0000 7f0b0bc00640 1 -- 172.21.15.125:0/3182929510 shutdown_connections 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.613+0000 7f0b0bc00640 1 -- 172.21.15.125:0/3182929510 >> 172.21.15.125:0/3182929510 conn(0x7f0b0406a800 msgr2=0x7f0b0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.613+0000 7f0b0bc00640 1 -- 172.21.15.125:0/3182929510 shutdown_connections 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.613+0000 7f0b0bc00640 1 -- 172.21.15.125:0/3182929510 wait complete. 2024-01-16T15:14:56.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.614+0000 7f0b0bc00640 1 Processor -- start 2024-01-16T15:14:56.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.614+0000 7f0b0bc00640 1 -- start start 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4004d00 0x7f0b041f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413d8c0 0x7f0b041509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b041534a0 con 0x7f0b04150f10 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b041f3050 con 0x7f0ae4004d00 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.615+0000 7f0b0bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04152890 con 0x7f0b0413d8c0 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33790/0 (socket says 172.21.15.125:33790) 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 -- 172.21.15.125:0/749804072 learned_addr learned my addr 172.21.15.125:0/749804072 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09200640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4004d00 0x7f0b041f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0afbc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413d8c0 0x7f0b041509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 -- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413d8c0 msgr2=0x7f0b041509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413d8c0 0x7f0b041509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 -- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4004d00 msgr2=0x7f0b041f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4004d00 0x7f0b041f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 -- 172.21.15.125:0/749804072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04152b70 con 0x7f0b04150f10 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.616+0000 7f0b09c00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0afc000c00 tx=0x7f0afc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.617+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc03c040 con 0x7f0b04150f10 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.617+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc02ccc0 con 0x7f0b04150f10 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.617+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc007270 con 0x7f0b04150f10 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.617+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b04152e50 con 0x7f0b04150f10 2024-01-16T15:14:56.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.617+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b04153930 con 0x7f0b04150f10 2024-01-16T15:14:56.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.618+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b040675d0 con 0x7f0b04150f10 2024-01-16T15:14:56.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.619+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f0afc032070 con 0x7f0b04150f10 2024-01-16T15:14:56.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.620+0000 7f0b01600640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 0x7f0adc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.621+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0afc0a9280 con 0x7f0b04150f10 2024-01-16T15:14:56.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.621+0000 7f0b09200640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 0x7f0adc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.621+0000 7f0b09200640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 0x7f0adc06e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0b0406dd20 tx=0x7f0af400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.625+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0afc075ba0 con 0x7f0b04150f10 2024-01-16T15:14:56.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.817+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0b041535e0 con 0x7f0b04150f10 2024-01-16T15:14:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.822+0000 7f0b01600640 1 -- 172.21.15.125:0/749804072 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0afc046070 con 0x7f0b04150f10 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 msgr2=0x7f0adc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 0x7f0adc06e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0b0406dd20 tx=0x7f0af400b040 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 msgr2=0x7f0b040af090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0afc000c00 tx=0x7f0afc001a00 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 shutdown_connections 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0adc06c430 0x7f0adc06e8f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b04150f10 0x7f0b040af090 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b0413d8c0 0x7f0b041509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 --2- 172.21.15.125:0/749804072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae4004d00 0x7f0b041f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 >> 172.21.15.125:0/749804072 conn(0x7f0b0406a800 msgr2=0x7f0b040b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 shutdown_connections 2024-01-16T15:14:56.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:56.825+0000 7f0b0bc00640 1 -- 172.21.15.125:0/749804072 wait complete. 2024-01-16T15:14:56.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:14:57.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.055+0000 7f70f0800640 1 Processor -- start 2024-01-16T15:14:57.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.055+0000 7f70f0800640 1 -- start start 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70f0800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70e8071200 0x7f70e806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70f0800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e815deb0 con 0x7f70e8157c30 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70f0800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e815e190 con 0x7f70e80cd600 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70f0800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e815ea30 con 0x7f70e8071200 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ede00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f70e8157c30 0x7f70e81618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45246/0 (socket says 172.21.15.125:45246) 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ede00640 1 -- 172.21.15.125:0/4030127713 learned_addr learned my addr 172.21.15.125:0/4030127713 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:57.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ee800640 1 --2- 172.21.15.125:0/4030127713 >> v2:172.21.15.125:3300/0 conn(0x7f70e8071200 0x7f70e806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ee800640 1 -- 172.21.15.125:0/4030127713 >> v1:172.21.15.137:6789/0 conn(0x7f70e80cd600 legacy=0x7f70e8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:57.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ee800640 1 -- 172.21.15.125:0/4030127713 >> v1:172.21.15.125:6789/0 conn(0x7f70e8157c30 legacy=0x7f70e81618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.057+0000 7f70ee800640 1 -- 172.21.15.125:0/4030127713 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e815e470 con 0x7f70e8071200 2024-01-16T15:14:57.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70ee800640 1 --2- 172.21.15.125:0/4030127713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e8071200 0x7f70e806dcb0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009d90 tx=0x7f70e002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dda94ecdb8bdfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70eca00640 1 -- 172.21.15.125:0/4030127713 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e002cc60 con 0x7f70e8071200 2024-01-16T15:14:57.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70eca00640 1 -- 172.21.15.125:0/4030127713 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e002cdc0 con 0x7f70e8071200 2024-01-16T15:14:57.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70eca00640 1 -- 172.21.15.125:0/4030127713 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0034660 con 0x7f70e8071200 2024-01-16T15:14:57.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70f0800640 1 -- 172.21.15.125:0/4030127713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e8071200 msgr2=0x7f70e806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.059+0000 7f70f0800640 1 --2- 172.21.15.125:0/4030127713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e8071200 0x7f70e806dcb0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009d90 tx=0x7f70e002beb0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.060+0000 7f70f0800640 1 -- 172.21.15.125:0/4030127713 shutdown_connections 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.060+0000 7f70f0800640 1 --2- 172.21.15.125:0/4030127713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e8071200 0x7f70e806dcb0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.060+0000 7f70f0800640 1 -- 172.21.15.125:0/4030127713 >> 172.21.15.125:0/4030127713 conn(0x7f70e80cf700 msgr2=0x7f70e80cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.060+0000 7f70f0800640 1 -- 172.21.15.125:0/4030127713 shutdown_connections 2024-01-16T15:14:57.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.060+0000 7f70f0800640 1 -- 172.21.15.125:0/4030127713 wait complete. 2024-01-16T15:14:57.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.061+0000 7f70f0800640 1 Processor -- start 2024-01-16T15:14:57.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.062+0000 7f70f0800640 1 -- start start 2024-01-16T15:14:57.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.062+0000 7f70f0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70e8071200 0x7f70e81430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70f0800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70f0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70e8157c30 0x7f70e81e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70f0800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8143630 con 0x7f70e80cd600 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70f0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8154540 con 0x7f70e8071200 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70f0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8154820 con 0x7f70e8157c30 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33822/0 (socket says 172.21.15.125:33822) 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 -- 172.21.15.125:0/3772782728 learned_addr learned my addr 172.21.15.125:0/3772782728 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ede00640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70e8071200 0x7f70e81430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ee800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70e8157c30 0x7f70e81e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 -- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70e8157c30 msgr2=0x7f70e81e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70e8157c30 0x7f70e81e92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 -- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70e8071200 msgr2=0x7f70e81430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70e8071200 0x7f70e81430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.063+0000 7f70ed400640 1 -- 172.21.15.125:0/3772782728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e0009a70 con 0x7f70e80cd600 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70ed400640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c430 tx=0x7f70d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0011790 con 0x7f70e80cd600 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d00118f0 con 0x7f70e80cd600 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e8154b00 con 0x7f70e80cd600 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0011ce0 con 0x7f70e80cd600 2024-01-16T15:14:57.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.064+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e81550c0 con 0x7f70e80cd600 2024-01-16T15:14:57.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.065+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e80675d0 con 0x7f70e80cd600 2024-01-16T15:14:57.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.066+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f70d001c460 con 0x7f70e80cd600 2024-01-16T15:14:57.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.067+0000 7f70e6000640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 0x7f70c006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.067+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f70d0088a30 con 0x7f70e80cd600 2024-01-16T15:14:57.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.067+0000 7f70ede00640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 0x7f70c006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.068+0000 7f70ede00640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 0x7f70c006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f70d8005880 tx=0x7f70d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.071+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70d0055000 con 0x7f70e80cd600 2024-01-16T15:14:57.317 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:57.335+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f70e8071ae0 con 0x7f70e80cd600 2024-01-16T15:14:58.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.208+0000 7f70e6000640 1 -- 172.21.15.125:0/3772782728 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v792) v1 ==== 155+0+0 (secure 0 0 0) 0x7f70d0058620 con 0x7f70e80cd600 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 msgr2=0x7f70c006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 0x7f70c006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f70d8005880 tx=0x7f70d8009040 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 msgr2=0x7f70e81555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c430 tx=0x7f70d000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 shutdown_connections 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70e8157c30 0x7f70e81e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70c006c430 0x7f70c006e8f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80cd600 0x7f70e81555e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 --2- 172.21.15.125:0/3772782728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70e8071200 0x7f70e81430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 >> 172.21.15.125:0/3772782728 conn(0x7f70e80cf700 msgr2=0x7f70e806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 shutdown_connections 2024-01-16T15:14:58.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.212+0000 7f70f0800640 1 -- 172.21.15.125:0/3772782728 wait complete. 2024-01-16T15:14:58.217 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:14:58.217 INFO:tasks.cephfs_test_runner:Starting test: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:14:58.217 INFO:tasks.cephfs_test_runner:test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:14:58.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.450+0000 7f3002000640 1 Processor -- start 2024-01-16T15:14:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.451+0000 7f3002000640 1 -- start start 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f3002000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2ffc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f3002000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc15c5f0 con 0x7f2ffc13df80 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f3002000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc06c470 con 0x7f2ffc15f9c0 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f3002000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06c750 con 0x7f2ffc13e360 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f2ffb400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2ffc13df80 0x7f2ffc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45252/0 (socket says 172.21.15.125:45252) 2024-01-16T15:14:58.435 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.452+0000 7f2ffb400640 1 -- 172.21.15.125:0/1804642623 learned_addr learned my addr 172.21.15.125:0/1804642623 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746656235 0 0) 0x7f2ffc15c5f0 con 0x7f2ffc13df80 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fdc003520 con 0x7f2ffc13df80 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ff2800640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2ffc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683123346 0 0) 0x7f2fdc003520 con 0x7f2ffc13df80 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 msgr2=0x7f2ffc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2ffc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 >> v1:172.21.15.137:6789/0 conn(0x7f2ffc15f9c0 legacy=0x7f2ffc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc06ca30 con 0x7f2ffc13df80 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2fec001670 con 0x7f2ffc13df80 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 >> v1:172.21.15.125:6789/0 conn(0x7f2ffc13df80 legacy=0x7f2ffc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2fdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2fdc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06c750 con 0x7f2fdc0039a0 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00bae0 con 0x7f2fdc0047c0 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.453+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00bca0 con 0x7f2ffc13e360 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2fdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ff2800640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 -- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 msgr2=0x7f2fdc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffbe00640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2fdc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc13e360 0x7f2fdc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 -- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fdc0047c0 msgr2=0x7f2fdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.439 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ffb400640 1 -- 172.21.15.125:0/1804642623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc003520 con 0x7f2fdc0039a0 2024-01-16T15:14:58.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.454+0000 7f2ff2800640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.455+0000 7f2ffb400640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2fdc003dc0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00c240 tx=0x7f2fdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d70f79c516a5e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec005ad0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec005c30 con 0x7f2fdc0039a0 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f2ffaa00640 1 -- 172.21.15.125:0/1804642623 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec02c8b0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f3002000640 1 -- 172.21.15.125:0/1804642623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 msgr2=0x7f2fdc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f3002000640 1 --2- 172.21.15.125:0/1804642623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2fdc003dc0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00c240 tx=0x7f2fdc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f2ffbe00640 1 -- 172.21.15.125:0/1804642623 reap_dead start 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f3002000640 1 -- 172.21.15.125:0/1804642623 shutdown_connections 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.456+0000 7f3002000640 1 -- 172.21.15.125:0/1804642623 >> 172.21.15.125:0/1804642623 conn(0x7f2ffc06a800 msgr2=0x7f2ffc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.457+0000 7f3002000640 1 -- 172.21.15.125:0/1804642623 shutdown_connections 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.457+0000 7f3002000640 1 -- 172.21.15.125:0/1804642623 wait complete. 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.458+0000 7f3002000640 1 Processor -- start 2024-01-16T15:14:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.458+0000 7f3002000640 1 -- start start 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc0707e0 0x7f2ffc15a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc070c40 0x7f2ffc15a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc072140 con 0x7f2fdc0039a0 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc14da30 con 0x7f2ffc0707e0 2024-01-16T15:14:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.459+0000 7f3002000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc135bf0 con 0x7f2ffc070c40 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33838/0 (socket says 172.21.15.125:33838) 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 -- 172.21.15.125:0/672633692 learned_addr learned my addr 172.21.15.125:0/672633692 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ff2800640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc0707e0 0x7f2ffc15a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffbe00640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc070c40 0x7f2ffc15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 -- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc070c40 msgr2=0x7f2ffc15a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc070c40 0x7f2ffc15a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 -- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc0707e0 msgr2=0x7f2ffc15a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc0707e0 0x7f2ffc15a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 -- 172.21.15.125:0/672633692 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0714c0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.460+0000 7f2ffb400640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f2fec001930 tx=0x7f2fec001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.461+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec039070 con 0x7f2fdc0039a0 2024-01-16T15:14:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.461+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec02ccc0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.461+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc0717a0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.461+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec03c040 con 0x7f2fdc0039a0 2024-01-16T15:14:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.461+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc071d60 con 0x7f2fdc0039a0 2024-01-16T15:14:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.462+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc0675d0 con 0x7f2fdc0039a0 2024-01-16T15:14:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.463+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2fec032070 con 0x7f2fdc0039a0 2024-01-16T15:14:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.464+0000 7f2ff3e00640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 0x7f2fd406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.464+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2fec0a9980 con 0x7f2fdc0039a0 2024-01-16T15:14:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.464+0000 7f2ff2800640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 0x7f2fd406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.465+0000 7f2ff2800640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 0x7f2fd406e920 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc15f010 tx=0x7f2fe4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.468+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fec076240 con 0x7f2fdc0039a0 2024-01-16T15:14:58.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.664+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ffc06df00 con 0x7f2fdc0039a0 2024-01-16T15:14:58.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.666+0000 7f2ff3e00640 1 -- 172.21.15.125:0/672633692 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f2fec079860 con 0x7f2fdc0039a0 2024-01-16T15:14:58.649 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:14:58.649 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":103,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:14:51.154789+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/737412716":"2024-01-17T15:14:48.135036+0000","172.21.15.125:6835/737412716":"2024-01-17T15:14:48.135036+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.668+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 msgr2=0x7f2fd406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.668+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 0x7f2fd406e920 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc15f010 tx=0x7f2fe4008040 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.668+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 msgr2=0x7f2ffc1356b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.668+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f2fec001930 tx=0x7f2fec001a00 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 shutdown_connections 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc070c40 0x7f2ffc15a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc0707e0 0x7f2ffc15a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fd406c460 0x7f2fd406e920 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 --2- 172.21.15.125:0/672633692 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fdc0039a0 0x7f2ffc1356b0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 >> 172.21.15.125:0/672633692 conn(0x7f2ffc06a800 msgr2=0x7f2ffc15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 shutdown_connections 2024-01-16T15:14:58.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.669+0000 7f3002000640 1 -- 172.21.15.125:0/672633692 wait complete. 2024-01-16T15:14:58.671 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:58.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:58.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.892+0000 7efe7c800640 1 Processor -- start 2024-01-16T15:14:58.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.892+0000 7efe7c800640 1 -- start start 2024-01-16T15:14:58.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.893+0000 7efe7c800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efe740b3ae0 0x7efe7415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.893+0000 7efe7c800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe74070c40 con 0x7efe740ae5a0 2024-01-16T15:14:58.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe7c800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7406c470 con 0x7efe740cf290 2024-01-16T15:14:58.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe7c800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe74071450 con 0x7efe740b3ae0 2024-01-16T15:14:58.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe79e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efe740ae5a0 0x7efe7406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45260/0 (socket says 172.21.15.125:45260) 2024-01-16T15:14:58.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe79e00640 1 -- 172.21.15.125:0/1440555457 learned_addr learned my addr 172.21.15.125:0/1440555457 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:58.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe79400640 1 --2- 172.21.15.125:0/1440555457 >> v2:172.21.15.137:3300/0 conn(0x7efe740b3ae0 0x7efe7415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997101078 0 0) 0x7efe74070c40 con 0x7efe740ae5a0 2024-01-16T15:14:58.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.894+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe54003520 con 0x7efe740ae5a0 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206475782 0 0) 0x7efe54003520 con 0x7efe740ae5a0 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 >> v2:172.21.15.137:3300/0 conn(0x7efe740b3ae0 msgr2=0x7efe7415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 --2- 172.21.15.125:0/1440555457 >> v2:172.21.15.137:3300/0 conn(0x7efe740b3ae0 0x7efe7415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 >> v1:172.21.15.137:6789/0 conn(0x7efe740cf290 legacy=0x7efe740b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7406c750 con 0x7efe740ae5a0 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7efe64001670 con 0x7efe740ae5a0 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 >> v1:172.21.15.125:6789/0 conn(0x7efe740ae5a0 legacy=0x7efe7406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe540039c0 0x7efe54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe540047c0 0x7efe54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 0x7efe5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe74071450 con 0x7efe54005180 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5400c060 con 0x7efe540047c0 2024-01-16T15:14:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5400c220 con 0x7efe540039c0 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.895+0000 7efe7a800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 0x7efe5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe79400640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe540047c0 0x7efe54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe79e00640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe540039c0 0x7efe54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe7a800640 1 -- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe540039c0 msgr2=0x7efe54003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe7a800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe540039c0 0x7efe54003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe7a800640 1 -- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe540047c0 msgr2=0x7efe54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe7a800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe540047c0 0x7efe54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe7a800640 1 -- 172.21.15.125:0/1440555457 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54003520 con 0x7efe54005180 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.896+0000 7efe79e00640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe540039c0 0x7efe54003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.897+0000 7efe7a800640 1 -- 172.21.15.125:0/1440555457 reap_dead start 2024-01-16T15:14:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.897+0000 7efe7a800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 0x7efe5400b980 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001e50 tx=0x7efe540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170f6b9a8e858377 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.897+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c003b70 con 0x7efe54005180 2024-01-16T15:14:58.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.897+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c003cf0 con 0x7efe54005180 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.897+0000 7efe78a00640 1 -- 172.21.15.125:0/1440555457 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c005310 con 0x7efe54005180 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 -- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 msgr2=0x7efe5400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 0x7efe5400b980 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001e50 tx=0x7efe540057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 -- 172.21.15.125:0/1440555457 shutdown_connections 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 --2- 172.21.15.125:0/1440555457 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe54005180 0x7efe5400b980 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 -- 172.21.15.125:0/1440555457 >> 172.21.15.125:0/1440555457 conn(0x7efe7406a800 msgr2=0x7efe7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 -- 172.21.15.125:0/1440555457 shutdown_connections 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.898+0000 7efe7c800640 1 -- 172.21.15.125:0/1440555457 wait complete. 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.900+0000 7efe7c800640 1 Processor -- start 2024-01-16T15:14:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.900+0000 7efe7c800640 1 -- start start 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 0x7efe7415bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe740cc450 0x7efe740cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740cea30 con 0x7efe740ce5b0 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740cce10 con 0x7efe740cc450 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7c800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe74156c50 con 0x7efe54005180 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33854/0 (socket says 172.21.15.125:33854) 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 -- 172.21.15.125:0/213417910 learned_addr learned my addr 172.21.15.125:0/213417910 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79e00640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 0x7efe7415bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe7a800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe740cc450 0x7efe740cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 -- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 msgr2=0x7efe7415bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 0x7efe7415bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 -- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe740cc450 msgr2=0x7efe740cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe740cc450 0x7efe740cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79400640 1 -- 172.21.15.125:0/213417910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe741583b0 con 0x7efe740ce5b0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.901+0000 7efe79e00640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 0x7efe7415bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe79400640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7efe5c011540 tx=0x7efe5c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c015930 con 0x7efe740ce5b0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c015a90 con 0x7efe740ce5b0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c015e20 con 0x7efe740ce5b0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7415c2d0 con 0x7efe740ce5b0 2024-01-16T15:14:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.902+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7415c890 con 0x7efe740ce5b0 2024-01-16T15:14:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.903+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740675d0 con 0x7efe740ce5b0 2024-01-16T15:14:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.904+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7efe5c00f460 con 0x7efe740ce5b0 2024-01-16T15:14:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.905+0000 7efe72000640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 0x7efe4c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.905+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efe5c089270 con 0x7efe740ce5b0 2024-01-16T15:14:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.905+0000 7efe79e00640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 0x7efe4c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.906+0000 7efe79e00640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 0x7efe4c06e920 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efe64001eb0 tx=0x7efe64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:58.909+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe5c0557e0 con 0x7efe740ce5b0 2024-01-16T15:14:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.101+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe74158fa0 con 0x7efe740ce5b0 2024-01-16T15:14:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.102+0000 7efe72000640 1 -- 172.21.15.125:0/213417910 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe5c0220a0 con 0x7efe740ce5b0 2024-01-16T15:14:59.085 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 msgr2=0x7efe4c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 0x7efe4c06e920 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efe64001eb0 tx=0x7efe64009040 comp rx=0 tx=0).stop 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 msgr2=0x7efe741f24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7efe5c011540 tx=0x7efe5c011a00 comp rx=0 tx=0).stop 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 shutdown_connections 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe740cc450 0x7efe740cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7efe4c06c460 0x7efe4c06e920 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe740ce5b0 0x7efe741f24e0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 --2- 172.21.15.125:0/213417910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efe54005180 0x7efe7415bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 >> 172.21.15.125:0/213417910 conn(0x7efe7406a800 msgr2=0x7efe740706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 shutdown_connections 2024-01-16T15:14:59.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.105+0000 7efe7c800640 1 -- 172.21.15.125:0/213417910 wait complete. 2024-01-16T15:14:59.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled' 2024-01-16T15:14:59.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.337+0000 7f3fc2600640 1 Processor -- start 2024-01-16T15:14:59.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.337+0000 7f3fc2600640 1 -- start start 2024-01-16T15:14:59.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.338+0000 7f3fc2600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3fbc15e3b0 0x7f3fbc164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.338+0000 7f3fc2600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0b1ee0 con 0x7f3fbc143ae0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fc2600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0b2780 con 0x7f3fbc0cf9a0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fc2600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc15e990 con 0x7f3fbc15e3b0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fbb400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3fbc143ae0 0x7f3fbc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45276/0 (socket says 172.21.15.125:45276) 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fbb400640 1 -- 172.21.15.125:0/2984074898 learned_addr learned my addr 172.21.15.125:0/2984074898 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374832064 0 0) 0x7f3fbc0b1ee0 con 0x7f3fbc143ae0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c003520 con 0x7f3fbc143ae0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.339+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/2984074898 >> v2:172.21.15.137:3300/0 conn(0x7f3fbc15e3b0 0x7f3fbc164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1742076153 0 0) 0x7f3f9c003520 con 0x7f3fbc143ae0 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 >> v2:172.21.15.137:3300/0 conn(0x7f3fbc15e3b0 msgr2=0x7f3fbc164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 --2- 172.21.15.125:0/2984074898 >> v2:172.21.15.137:3300/0 conn(0x7f3fbc15e3b0 0x7f3fbc164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 >> v1:172.21.15.137:6789/0 conn(0x7f3fbc0cf9a0 legacy=0x7f3fbc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0b2130 con 0x7f3fbc143ae0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3fac0019f0 con 0x7f3fbc143ae0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 >> v1:172.21.15.125:6789/0 conn(0x7f3fbc143ae0 legacy=0x7f3fbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c0039c0 0x7f3f9c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 0x7f3f9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc15e990 con 0x7f3f9c0051d0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c00c120 con 0x7f3f9c0047c0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c00c2e0 con 0x7f3f9c0039c0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 0x7f3f9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.340+0000 7f3fbaa00640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbb400640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c0039c0 0x7f3f9c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbbe00640 1 -- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c0039c0 msgr2=0x7f3f9c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f9c0039c0 0x7f3f9c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbbe00640 1 -- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0047c0 msgr2=0x7f3f9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.341+0000 7f3fbbe00640 1 -- 172.21.15.125:0/2984074898 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c003520 con 0x7f3f9c0051d0 2024-01-16T15:14:59.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.342+0000 7f3fbbe00640 1 -- 172.21.15.125:0/2984074898 reap_dead start 2024-01-16T15:14:59.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.342+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 0x7f3f9c00b980 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400ba30 tx=0x7f3f9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32788e5ac7b26e60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.342+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4004020 con 0x7f3f9c0051d0 2024-01-16T15:14:59.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.342+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4004180 con 0x7f3f9c0051d0 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.342+0000 7f3fba000640 1 -- 172.21.15.125:0/2984074898 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4003610 con 0x7f3f9c0051d0 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 -- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 msgr2=0x7f3f9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 0x7f3f9c00b980 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400ba30 tx=0x7f3f9c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 -- 172.21.15.125:0/2984074898 shutdown_connections 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 --2- 172.21.15.125:0/2984074898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f9c0051d0 0x7f3f9c00b980 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 -- 172.21.15.125:0/2984074898 >> 172.21.15.125:0/2984074898 conn(0x7f3fbc06a800 msgr2=0x7f3fbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 -- 172.21.15.125:0/2984074898 shutdown_connections 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.343+0000 7f3fc2600640 1 -- 172.21.15.125:0/2984074898 wait complete. 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.345+0000 7f3fc2600640 1 Processor -- start 2024-01-16T15:14:59.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.345+0000 7f3fc2600640 1 -- start start 2024-01-16T15:14:59.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0051d0 0x7f3fbc13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.330 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fbc139050 0x7f3fbc0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc139530 con 0x7f3fbc15b720 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc139810 con 0x7f3f9c0051d0 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fc2600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc139af0 con 0x7f3fbc139050 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33862/0 (socket says 172.21.15.125:33862) 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 -- 172.21.15.125:0/957224463 learned_addr learned my addr 172.21.15.125:0/957224463 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbbe00640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fbc139050 0x7f3fbc0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbb400640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0051d0 0x7f3fbc13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 -- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fbc139050 msgr2=0x7f3fbc0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fbc139050 0x7f3fbc0cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 -- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0051d0 msgr2=0x7f3fbc13ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0051d0 0x7f3fbc13ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.346+0000 7f3fbaa00640 1 -- 172.21.15.125:0/957224463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc139dd0 con 0x7f3fbc15b720 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fbaa00640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400e410 tx=0x7f3fa400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40052e0 con 0x7f3fbc15b720 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa4005440 con 0x7f3fbc15b720 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40037f0 con 0x7f3fbc15b720 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc159640 con 0x7f3fbc15b720 2024-01-16T15:14:59.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.347+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0ada40 con 0x7f3fbc15b720 2024-01-16T15:14:59.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.348+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0cd8c0 con 0x7f3fbc15b720 2024-01-16T15:14:59.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.349+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f3fa4003950 con 0x7f3fbc15b720 2024-01-16T15:14:59.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.351+0000 7f3fb3400640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 0x7f3f9406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.351+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3fa4088040 con 0x7f3fbc15b720 2024-01-16T15:14:59.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.351+0000 7f3fbb400640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 0x7f3f9406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.352+0000 7f3fbb400640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 0x7f3f9406e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001a10 tx=0x7f3fac001270 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.355+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3fbc0cd8c0 con 0x7f3fbc15b720 2024-01-16T15:14:59.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:14:59.623+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]} v 0) v1 -- 0x7f3fbc0ada40 con 0x7f3fbc15b720 2024-01-16T15:15:00.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.221+0000 7f3fb3400640 1 -- 172.21.15.125:0/957224463 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]}]=0 v794) v1 ==== 158+0+0 (secure 0 0 0) 0x7f3fbc0ada40 con 0x7f3fbc15b720 2024-01-16T15:15:00.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.224+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 msgr2=0x7f3f9406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.224+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 0x7f3f9406e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001a10 tx=0x7f3fac001270 comp rx=0 tx=0).stop 2024-01-16T15:15:00.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 msgr2=0x7f3fbc0ad320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400e410 tx=0x7f3fa400e8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 shutdown_connections 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fbc139050 0x7f3fbc0cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f9406c430 0x7f3f9406e8f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fbc15b720 0x7f3fbc0ad320 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 --2- 172.21.15.125:0/957224463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f9c0051d0 0x7f3fbc13ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 >> 172.21.15.125:0/957224463 conn(0x7f3fbc06a800 msgr2=0x7f3fbc0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 shutdown_connections 2024-01-16T15:15:00.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.225+0000 7f3fc2600640 1 -- 172.21.15.125:0/957224463 wait complete. 2024-01-16T15:15:00.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:00.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.457+0000 7fecaac00640 1 Processor -- start 2024-01-16T15:15:00.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.457+0000 7fecaac00640 1 -- start start 2024-01-16T15:15:00.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7fecaac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7fecaac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feca415f9c0 0x7feca40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7fecaac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca415c5f0 con 0x7feca413df80 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7fecaac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca406c470 con 0x7feca415f9c0 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7fecaac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca406c750 con 0x7feca413e360 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7feca3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.459+0000 7feca8c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feca415f9c0 0x7feca40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:39440/0 (socket says 172.21.15.125:39440) 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 -- 172.21.15.125:0/1154521691 learned_addr learned my addr 172.21.15.125:0/1154521691 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3e00640 1 --1- 172.21.15.125:0/1154521691 >> v1:172.21.15.137:6789/0 conn(0x7feca413df80 0x7feca415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:59816/0 (socket says 172.21.15.125:59816) 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca2a00640 1 -- 172.21.15.125:0/1154521691 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360969817 0 0) 0x7feca415c5f0 con 0x7feca413df80 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca2a00640 1 -- 172.21.15.125:0/1154521691 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec84003520 con 0x7feca413df80 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 -- 172.21.15.125:0/1154521691 >> v2:172.21.15.137:3300/0 conn(0x7feca415f9c0 msgr2=0x7feca40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 --2- 172.21.15.125:0/1154521691 >> v2:172.21.15.137:3300/0 conn(0x7feca415f9c0 0x7feca40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 -- 172.21.15.125:0/1154521691 >> v1:172.21.15.137:6789/0 conn(0x7feca413df80 legacy=0x7feca415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.460+0000 7feca3400640 1 -- 172.21.15.125:0/1154521691 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca406ca30 con 0x7feca413e360 2024-01-16T15:15:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.462+0000 7feca3400640 1 --2- 172.21.15.125:0/1154521691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416ce612a337428e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.462+0000 7feca2a00640 1 -- 172.21.15.125:0/1154521691 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c030840 con 0x7feca413e360 2024-01-16T15:15:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.462+0000 7feca2a00640 1 -- 172.21.15.125:0/1154521691 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c0309a0 con 0x7feca413e360 2024-01-16T15:15:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.462+0000 7feca2a00640 1 -- 172.21.15.125:0/1154521691 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c030d10 con 0x7feca413e360 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 -- 172.21.15.125:0/1154521691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 msgr2=0x7feca40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 --2- 172.21.15.125:0/1154521691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02b980 comp rx=0 tx=0).stop 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 -- 172.21.15.125:0/1154521691 shutdown_connections 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 --2- 172.21.15.125:0/1154521691 >> v2:172.21.15.137:3300/0 conn(0x7feca415f9c0 0x7feca40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 --2- 172.21.15.125:0/1154521691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413e360 0x7feca40cf290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 -- 172.21.15.125:0/1154521691 >> 172.21.15.125:0/1154521691 conn(0x7feca406a800 msgr2=0x7feca406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 -- 172.21.15.125:0/1154521691 shutdown_connections 2024-01-16T15:15:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.463+0000 7fecaac00640 1 -- 172.21.15.125:0/1154521691 wait complete. 2024-01-16T15:15:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.465+0000 7fecaac00640 1 Processor -- start 2024-01-16T15:15:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.465+0000 7fecaac00640 1 -- start start 2024-01-16T15:15:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413df80 0x7feca415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca413e360 0x7feca4152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca415aa70 con 0x7feca415f9c0 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca413a120 con 0x7feca413e360 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7fecaac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca413a400 con 0x7feca413df80 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.466+0000 7feca8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33868/0 (socket says 172.21.15.125:33868) 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 -- 172.21.15.125:0/1044408079 learned_addr learned my addr 172.21.15.125:0/1044408079 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca3e00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413df80 0x7feca415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca3400640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca413e360 0x7feca4152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 -- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413df80 msgr2=0x7feca415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413df80 0x7feca415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 -- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca413e360 msgr2=0x7feca4152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca413e360 0x7feca4152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.451 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 -- 172.21.15.125:0/1044408079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0094d0 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7feca8c00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00efc0 tx=0x7fec9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.467+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c014480 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.468+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c00f040 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.468+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c014820 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.468+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca413a6e0 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.468+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca413aca0 con 0x7feca415f9c0 2024-01-16T15:15:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.469+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca413ae90 con 0x7feca415f9c0 2024-01-16T15:15:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.470+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fec9c014980 con 0x7feca415f9c0 2024-01-16T15:15:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.471+0000 7fec9be00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 0x7fec7c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.471+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fec9c0896c0 con 0x7feca415f9c0 2024-01-16T15:15:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.471+0000 7feca3e00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 0x7fec7c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.472+0000 7feca3e00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 0x7fec7c06e8f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fec94000f60 tx=0x7fec94007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.475+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec9c055c30 con 0x7feca415f9c0 2024-01-16T15:15:00.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.664+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7feca40675d0 con 0x7feca415f9c0 2024-01-16T15:15:00.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.671+0000 7fec9be00640 1 -- 172.21.15.125:0/1044408079 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fec9c018070 con 0x7feca415f9c0 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.673+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 msgr2=0x7fec7c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.673+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 0x7fec7c06e8f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fec94000f60 tx=0x7fec94007b50 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.673+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 msgr2=0x7feca40cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.673+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00efc0 tx=0x7fec9c00c490 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 shutdown_connections 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fec7c06c430 0x7fec7c06e8f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feca415f9c0 0x7feca40cdec0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca413e360 0x7feca4152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 --2- 172.21.15.125:0/1044408079 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca413df80 0x7feca415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 >> 172.21.15.125:0/1044408079 conn(0x7feca406a800 msgr2=0x7feca415cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 shutdown_connections 2024-01-16T15:15:00.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:00.674+0000 7fecaac00640 1 -- 172.21.15.125:0/1044408079 wait complete. 2024-01-16T15:15:00.676 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:15:00.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:00.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:00.897 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:15:00.897 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:15:00.897 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:00.953 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:00.954 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:15:00.954 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:01.009 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:01.009 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:15:01.009 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:01.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:01.067 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:15:01.067 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:01.123 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:01.123 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:15:01.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:01.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.356+0000 7f4b13800640 1 Processor -- start 2024-01-16T15:15:01.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.356+0000 7f4b13800640 1 -- start start 2024-01-16T15:15:01.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.357+0000 7f4b13800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13df80 0x7f4b0c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.357+0000 7f4b13800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.357+0000 7f4b13800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c06c470 con 0x7f4b0c13e360 2024-01-16T15:15:01.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.357+0000 7f4b13800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c06c750 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.357+0000 7f4b13800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c06ca30 con 0x7f4b0c13df80 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.358+0000 7f4b11800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.358+0000 7f4b11800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:55156/0 (socket says 172.21.15.125:55156) 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.358+0000 7f4b11800640 1 -- 172.21.15.125:0/2314335424 learned_addr learned my addr 172.21.15.125:0/2314335424 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.358+0000 7f4b0be00640 1 --1- 172.21.15.125:0/2314335424 >> v1:172.21.15.137:6789/0 conn(0x7f4b0c13e360 0x7f4b0c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:59818/0 (socket says 172.21.15.125:59818) 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.358+0000 7f4b10e00640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13df80 0x7f4b0c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.359+0000 7f4b11800640 1 -- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13df80 msgr2=0x7f4b0c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.359+0000 7f4b11800640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13df80 0x7f4b0c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.359+0000 7f4b11800640 1 -- 172.21.15.125:0/2314335424 >> v1:172.21.15.137:6789/0 conn(0x7f4b0c13e360 legacy=0x7f4b0c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.359+0000 7f4b11800640 1 -- 172.21.15.125:0/2314335424 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c06cd10 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.360+0000 7f4b11800640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009e50 tx=0x7f4b0402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=870adfdd99b56497 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.360+0000 7f4b0b400640 1 -- 172.21.15.125:0/2314335424 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0402cb00 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.360+0000 7f4b0b400640 1 -- 172.21.15.125:0/2314335424 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0402cc60 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.360+0000 7f4b0b400640 1 -- 172.21.15.125:0/2314335424 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b040343f0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 -- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c0cf9a0 msgr2=0x7f4b0c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009e50 tx=0x7f4b0402bf60 comp rx=0 tx=0).stop 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 -- 172.21.15.125:0/2314335424 shutdown_connections 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0cd840 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 --2- 172.21.15.125:0/2314335424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13df80 0x7f4b0c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 -- 172.21.15.125:0/2314335424 >> 172.21.15.125:0/2314335424 conn(0x7f4b0c06a800 msgr2=0x7f4b0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 -- 172.21.15.125:0/2314335424 shutdown_connections 2024-01-16T15:15:01.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.361+0000 7f4b13800640 1 -- 172.21.15.125:0/2314335424 wait complete. 2024-01-16T15:15:01.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.363+0000 7f4b13800640 1 Processor -- start 2024-01-16T15:15:01.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.363+0000 7f4b13800640 1 -- start start 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c13df80 0x7f4b0c0aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13e360 0x7f4b0c0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0a8e40 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0a9120 con 0x7f4b0c13df80 2024-01-16T15:15:01.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b13800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0a9400 con 0x7f4b0c13e360 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33876/0 (socket says 172.21.15.125:33876) 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 -- 172.21.15.125:0/2449042600 learned_addr learned my addr 172.21.15.125:0/2449042600 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 -- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13e360 msgr2=0x7f4b0c0af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13e360 0x7f4b0c0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 -- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c13df80 msgr2=0x7f4b0c0aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c13df80 0x7f4b0c0aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.364+0000 7f4b10e00640 1 -- 172.21.15.125:0/2449042600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009b30 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b10e00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00e9b0 tx=0x7f4afc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc009670 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc00c9a0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c0a96e0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00cd50 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.365+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c150cb0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.366+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c0675d0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.367+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4afc013470 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.368+0000 7f4b08c00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 0x7f4ae406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.368+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4afc088cc0 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.368+0000 7f4b0be00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 0x7f4ae406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.369+0000 7f4b0be00640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 0x7f4ae406e8f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c154ec0 tx=0x7f4af40079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.372+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4afc055290 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.598+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b0c15e870 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.599+0000 7f4b08c00640 1 -- 172.21.15.125:0/2449042600 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f4afc021e90 con 0x7f4b0c0cf9a0 2024-01-16T15:15:01.582 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:01.582 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":66,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6460,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6834/2589810407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2589810407},{"type":"v1","addr":"172.21.15.137:6834","nonce":2589810407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6463,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1674315039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6833","nonce":1674315039},{"type":"v1","addr":"172.21.15.137:6835","nonce":1674315039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":8646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/995017134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":995017134},{"type":"v1","addr":"172.21.15.125:6836","nonce":995017134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/752965027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":752965027},{"type":"v1","addr":"172.21.15.125:6837","nonce":752965027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[]} 2024-01-16T15:15:01.582 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 66 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.601+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 msgr2=0x7f4ae406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.601+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 0x7f4ae406e8f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c154ec0 tx=0x7f4af40079c0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.601+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 msgr2=0x7f4b0c0a87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.601+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00e9b0 tx=0x7f4afc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 shutdown_connections 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b0c13e360 0x7f4b0c0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b0c13df80 0x7f4b0c0aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4ae406c430 0x7f4ae406e8f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 --2- 172.21.15.125:0/2449042600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b0c0cf9a0 0x7f4b0c0a87d0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 >> 172.21.15.125:0/2449042600 conn(0x7f4b0c06a800 msgr2=0x7f4b0c15abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 shutdown_connections 2024-01-16T15:15:01.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.602+0000 7f4b13800640 1 -- 172.21.15.125:0/2449042600 wait complete. 2024-01-16T15:15:01.603 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:15:01.603 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:15:01.603 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:15:01.603 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:01.603 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:15:01.603 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:15:01.604 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:15:01.604 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:01.604 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:15:01.604 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:15:01.604 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:15:01.604 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:01.604 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:15:01.604 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:15:01.605 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:15:01.605 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:01.625 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:15:01.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:15:01.626 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:15:01.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:15:01.629 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:15:01.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:15:01.632 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:15:01.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:15:01.673 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:15:01.674 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:01.679 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:15:01.680 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:01.680 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:15:01.680 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:01.686 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:15:01.686 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:01.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:15:01.826 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2024-01-16T15:15:01.827 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2024-01-16T15:15:01.838 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-16T15:15:01.840 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-16T15:15:01.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.882+0000 7f806ee00640 1 Processor -- start 2024-01-16T15:15:01.865 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.883+0000 7f806ee00640 1 -- start start 2024-01-16T15:15:01.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 0x7f806815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f806813e360 0x7f80680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f806815f9c0 0x7f80680cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806806c470 con 0x7f806813e360 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806806c750 con 0x7f806815f9c0 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806806ca30 con 0x7f806813df80 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f8067400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f806813e360 0x7f80680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f8067400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f806813e360 0x7f80680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33912/0 (socket says 172.21.15.125:33912) 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f8067400640 1 -- 172.21.15.125:0/724978474 learned_addr learned my addr 172.21.15.125:0/724978474 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f806ce00640 1 --2- 172.21.15.125:0/724978474 >> v2:172.21.15.137:3300/0 conn(0x7f806815f9c0 0x7f80680cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.884+0000 7f8067e00640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 0x7f806815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067400640 1 -- 172.21.15.125:0/724978474 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 msgr2=0x7f806815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067400640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 0x7f806815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067400640 1 -- 172.21.15.125:0/724978474 >> v2:172.21.15.137:3300/0 conn(0x7f806815f9c0 msgr2=0x7f80680cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067400640 1 --2- 172.21.15.125:0/724978474 >> v2:172.21.15.137:3300/0 conn(0x7f806815f9c0 0x7f80680cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.868 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067400640 1 -- 172.21.15.125:0/724978474 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806806cd10 con 0x7f806813e360 2024-01-16T15:15:01.869 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.885+0000 7f8067e00640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 0x7f806815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f8067400640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813e360 0x7f80680cf290 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f806806e6d0 tx=0x7f805002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937caca46e6477e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f8066a00640 1 -- 172.21.15.125:0/724978474 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050031870 con 0x7f806813e360 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f8066a00640 1 -- 172.21.15.125:0/724978474 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80500319d0 con 0x7f806813e360 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f8066a00640 1 -- 172.21.15.125:0/724978474 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050031d60 con 0x7f806813e360 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f806ee00640 1 -- 172.21.15.125:0/724978474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813e360 msgr2=0x7f80680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.886+0000 7f806ee00640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813e360 0x7f80680cf290 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f806806e6d0 tx=0x7f805002b910 comp rx=0 tx=0).stop 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 -- 172.21.15.125:0/724978474 shutdown_connections 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 --2- 172.21.15.125:0/724978474 >> v2:172.21.15.137:3300/0 conn(0x7f806815f9c0 0x7f80680cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813e360 0x7f80680cf290 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 --2- 172.21.15.125:0/724978474 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806813df80 0x7f806815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 -- 172.21.15.125:0/724978474 >> 172.21.15.125:0/724978474 conn(0x7f806806a800 msgr2=0x7f806806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 -- 172.21.15.125:0/724978474 shutdown_connections 2024-01-16T15:15:01.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.887+0000 7f806ee00640 1 -- 172.21.15.125:0/724978474 wait complete. 2024-01-16T15:15:01.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.888+0000 7f806ee00640 1 Processor -- start 2024-01-16T15:15:01.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.889+0000 7f806ee00640 1 -- start start 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f806813e360 0x7f8068139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806815f9c0 0x7f806813e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806813a450 con 0x7f806813df80 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681f1740 con 0x7f806813e360 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f806ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80681f0b90 con 0x7f806815f9c0 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33924/0 (socket says 172.21.15.125:33924) 2024-01-16T15:15:01.873 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 -- 172.21.15.125:0/1450900518 learned_addr learned my addr 172.21.15.125:0/1450900518 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 -- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806815f9c0 msgr2=0x7f806813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806815f9c0 0x7f806813e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 -- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f806813e360 msgr2=0x7f8068139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f806813e360 0x7f8068139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 -- 172.21.15.125:0/1450900518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80500094d0 con 0x7f806813df80 2024-01-16T15:15:01.874 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.890+0000 7f8067e00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f805800d7d0 tx=0x7f805800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.891+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058004030 con 0x7f806813df80 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.891+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058004190 con 0x7f806813df80 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.891+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80680b37d0 con 0x7f806813df80 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.891+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058004520 con 0x7f806813df80 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.891+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80681f1080 con 0x7f806813df80 2024-01-16T15:15:01.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.892+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80681f19b0 con 0x7f806813df80 2024-01-16T15:15:01.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.893+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f805800b640 con 0x7f806813df80 2024-01-16T15:15:01.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.894+0000 7f805fe00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 0x7f804006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.894+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8058088bc0 con 0x7f806813df80 2024-01-16T15:15:01.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.894+0000 7f8067400640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 0x7f804006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.895+0000 7f8067400640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 0x7f804006e8f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8050036000 tx=0x7f8050030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:01.899+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8058055130 con 0x7f806813df80 2024-01-16T15:15:02.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.101+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f80680675d0 con 0x7f806813df80 2024-01-16T15:15:02.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.492+0000 7f805fe00640 1 -- 172.21.15.125:0/1450900518 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8058058750 con 0x7f806813df80 2024-01-16T15:15:02.476 INFO:teuthology.orchestra.run.smithi125.stderr: removed all blocklist entries 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.496+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 msgr2=0x7f804006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.496+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 0x7f804006e8f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8050036000 tx=0x7f8050030000 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.496+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 msgr2=0x7f8068138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.496+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f805800d7d0 tx=0x7f805800dc90 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 shutdown_connections 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f806815f9c0 0x7f806813e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f806813e360 0x7f8068139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f804006c430 0x7f804006e8f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 --2- 172.21.15.125:0/1450900518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f806813df80 0x7f8068138e30 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 >> 172.21.15.125:0/1450900518 conn(0x7f806806a800 msgr2=0x7f80680ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 shutdown_connections 2024-01-16T15:15:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.497+0000 7f806ee00640 1 -- 172.21.15.125:0/1450900518 wait complete. 2024-01-16T15:15:02.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:15:02.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.712+0000 7f3e46600640 1 Processor -- start 2024-01-16T15:15:02.695 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.712+0000 7f3e46600640 1 -- start start 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e46600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3e40157c30 0x7f3e400cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e46600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e40071200 0x7f3e4006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e46600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4015deb0 con 0x7f3e401618f0 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e46600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4015ea30 con 0x7f3e40071200 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e46600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4006d860 con 0x7f3e40157c30 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e3ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3e401618f0 0x7f3e40070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45300/0 (socket says 172.21.15.125:45300) 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e3ea00640 1 -- 172.21.15.125:0/3606975328 learned_addr learned my addr 172.21.15.125:0/3606975328 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:02.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e3fe00640 1 --2- 172.21.15.125:0/3606975328 >> v2:172.21.15.125:3300/0 conn(0x7f3e40071200 0x7f3e4006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.714+0000 7f3e3f400640 1 --2- 172.21.15.125:0/3606975328 >> v2:172.21.15.137:3300/0 conn(0x7f3e40157c30 0x7f3e400cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3e000640 1 -- 172.21.15.125:0/3606975328 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581030887 0 0) 0x7f3e4015deb0 con 0x7f3e401618f0 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3e000640 1 -- 172.21.15.125:0/3606975328 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e20003520 con 0x7f3e401618f0 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3fe00640 1 -- 172.21.15.125:0/3606975328 >> v2:172.21.15.137:3300/0 conn(0x7f3e40157c30 msgr2=0x7f3e400cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3fe00640 1 --2- 172.21.15.125:0/3606975328 >> v2:172.21.15.137:3300/0 conn(0x7f3e40157c30 0x7f3e400cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3fe00640 1 -- 172.21.15.125:0/3606975328 >> v1:172.21.15.125:6789/0 conn(0x7f3e401618f0 legacy=0x7f3e40070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.715+0000 7f3e3fe00640 1 -- 172.21.15.125:0/3606975328 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4015e190 con 0x7f3e40071200 2024-01-16T15:15:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.716+0000 7f3e3fe00640 1 --2- 172.21.15.125:0/3606975328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40071200 0x7f3e4006dcb0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3e400723f0 tx=0x7f3e3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce7fa921047c527 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.716+0000 7f3e3e000640 1 -- 172.21.15.125:0/3606975328 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38031830 con 0x7f3e40071200 2024-01-16T15:15:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.716+0000 7f3e3e000640 1 -- 172.21.15.125:0/3606975328 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e38031990 con 0x7f3e40071200 2024-01-16T15:15:02.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.716+0000 7f3e3e000640 1 -- 172.21.15.125:0/3606975328 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38031d20 con 0x7f3e40071200 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 -- 172.21.15.125:0/3606975328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40071200 msgr2=0x7f3e4006dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 --2- 172.21.15.125:0/3606975328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40071200 0x7f3e4006dcb0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3e400723f0 tx=0x7f3e3802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 -- 172.21.15.125:0/3606975328 shutdown_connections 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 --2- 172.21.15.125:0/3606975328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40071200 0x7f3e4006dcb0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 --2- 172.21.15.125:0/3606975328 >> v2:172.21.15.137:3300/0 conn(0x7f3e40157c30 0x7f3e400cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 -- 172.21.15.125:0/3606975328 >> 172.21.15.125:0/3606975328 conn(0x7f3e400cf700 msgr2=0x7f3e400cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 -- 172.21.15.125:0/3606975328 shutdown_connections 2024-01-16T15:15:02.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.717+0000 7f3e46600640 1 -- 172.21.15.125:0/3606975328 wait complete. 2024-01-16T15:15:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.718+0000 7f3e46600640 1 Processor -- start 2024-01-16T15:15:02.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.719+0000 7f3e46600640 1 -- start start 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e40071200 0x7f3e401582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e401618f0 0x7f3e401498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40144be0 con 0x7f3e40157c30 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40144ec0 con 0x7f3e401618f0 2024-01-16T15:15:02.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e46600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e401451a0 con 0x7f3e40071200 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33938/0 (socket says 172.21.15.125:33938) 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 -- 172.21.15.125:0/3283278791 learned_addr learned my addr 172.21.15.125:0/3283278791 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 -- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e40071200 msgr2=0x7f3e401582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e40071200 0x7f3e401582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 -- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e401618f0 msgr2=0x7f3e401498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3fe00640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e401618f0 0x7f3e401498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e401618f0 0x7f3e401498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.720+0000 7f3e3ea00640 1 -- 172.21.15.125:0/3283278791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38009400 con 0x7f3e40157c30 2024-01-16T15:15:02.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e3ea00640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c500 tx=0x7f3e2800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28005ce0 con 0x7f3e40157c30 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28005e40 con 0x7f3e40157c30 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007330 con 0x7f3e40157c30 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e40145480 con 0x7f3e40157c30 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.721+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e401448d0 con 0x7f3e40157c30 2024-01-16T15:15:02.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.722+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e40146e70 con 0x7f3e40157c30 2024-01-16T15:15:02.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.723+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f3e28007490 con 0x7f3e40157c30 2024-01-16T15:15:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.724+0000 7f3e37400640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 0x7f3e1806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.724+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3e280880f0 con 0x7f3e40157c30 2024-01-16T15:15:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.724+0000 7f3e3f400640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 0x7f3e1806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.725+0000 7f3e3f400640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 0x7f3e1806e920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3e300070b0 tx=0x7f3e30007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.728+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e40146e70 con 0x7f3e40157c30 2024-01-16T15:15:02.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.990+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3e40146e70 con 0x7f3e40157c30 2024-01-16T15:15:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.990+0000 7f3e37400640 1 -- 172.21.15.125:0/3283278791 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3e40146e70 con 0x7f3e40157c30 2024-01-16T15:15:02.973 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_dump": [ 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.a", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlrHgIAxAAttQrjecGcE0+P66v75Ph7A==", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.b", 2024-01-16T15:15:02.974 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxBs/IxAABIbjuVlA5bQkz0N7w6i/lA==", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.c", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlXQlVExAAM8kG1Bj1e7xBT+mei9DMsw==", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mds.d", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDsmKZlxmNRMRAAxrxPJubIENPrzQH6yN0VJw==", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.0", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDxmKZlitLZGhAAPL1ROgi0fQqZzg+gXCktpQ==", 2024-01-16T15:15:02.976 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.1", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD0mKZlRtJPNBAACt9yrsrPxK4mXYiHJsJFqg==", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.979 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.2", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD4mKZl960jERAA3POvRNmJHuJmhPSr73v03Q==", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.3", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQD7mKZlUhREGxAA9eFMz6ABVS/X9nW+fU17MQ==", 2024-01-16T15:15:02.980 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.4", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQACmaZlDDlfIRAAD806BU+GTEsRRSCIDamIKQ==", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.5", 2024-01-16T15:15:02.981 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAFmaZlSMpMLBAAcQSIfmx0hJfcZ/wT1SEj6g==", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.6", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAJmaZluNvuCBAAGapWaElYI7tNd155pd6Www==", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.982 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "osd.7", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQAMmaZlXzKiEhAA+WVGAejLOoe2nthOlV81xA==", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile osd", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.0", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA==", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow r", 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.1", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlC0EuFxAAEYfGVzHddoumrlvHVDfmkg==", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.2", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZldv9NDRAAU1APsKPvKVO6jHwBdY0ZgQ==", 2024-01-16T15:15:02.984 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.3", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDtmKZlAb6wIRAALYlkQp0tIZiLOmpLok0dLw==", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow r", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow rw", 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow rwx" 2024-01-16T15:15:02.985 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "client.admin", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDqmKZlWG4nMxAAArEys0nIySqcnu17gSXJzQ==", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr": "allow *", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow *", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.x", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZl5rQmMxAA/OTMVHWjYDDDWQ7JDGdYUw==", 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "entity": "mgr.y", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "AQDrmKZlwSecJxAAonPo8vE8C0s3bbfy0j20Fg==", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "caps": { 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mds": "allow *", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": "allow *" 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2024-01-16T15:15:02.987 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:15:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 msgr2=0x7f3e1806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 0x7f3e1806e920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3e300070b0 tx=0x7f3e30007040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 msgr2=0x7f3e40144390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c500 tx=0x7f3e2800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 shutdown_connections 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e401618f0 0x7f3e401498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e1806c460 0x7f3e1806e920 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e40157c30 0x7f3e40144390 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 --2- 172.21.15.125:0/3283278791 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e40071200 0x7f3e401582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.993+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 >> 172.21.15.125:0/3283278791 conn(0x7f3e400cf700 msgr2=0x7f3e40160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.994+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 shutdown_connections 2024-01-16T15:15:02.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:02.994+0000 7f3e46600640 1 -- 172.21.15.125:0/3283278791 wait complete. 2024-01-16T15:15:02.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:03.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.216+0000 7fa059c00640 1 Processor -- start 2024-01-16T15:15:03.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.216+0000 7fa059c00640 1 -- start start 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa059c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0540cf9a0 0x7fa054166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa059c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05415c5f0 con 0x7fa05413e360 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa059c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05406c470 con 0x7fa05413df80 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa059c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05406c750 con 0x7fa0540cf9a0 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa04be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa05413e360 0x7fa0540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45306/0 (socket says 172.21.15.125:45306) 2024-01-16T15:15:03.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa04be00640 1 -- 172.21.15.125:0/2874096643 learned_addr learned my addr 172.21.15.125:0/2874096643 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.218+0000 7fa053400640 1 --2- 172.21.15.125:0/2874096643 >> v2:172.21.15.137:3300/0 conn(0x7fa0540cf9a0 0x7fa054166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234024549 0 0) 0x7fa05415c5f0 con 0x7fa05413e360 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa034003520 con 0x7fa05413e360 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826395154 0 0) 0x7fa034003520 con 0x7fa05413e360 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 >> v2:172.21.15.137:3300/0 conn(0x7fa0540cf9a0 msgr2=0x7fa054166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 --2- 172.21.15.125:0/2874096643 >> v2:172.21.15.137:3300/0 conn(0x7fa0540cf9a0 0x7fa054166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 >> v1:172.21.15.137:6789/0 conn(0x7fa05413df80 legacy=0x7fa05415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05406ca30 con 0x7fa05413e360 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa03c001670 con 0x7fa05413e360 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 >> v1:172.21.15.125:6789/0 conn(0x7fa05413e360 legacy=0x7fa0540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0340039c0 0x7fa034003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0340047c0 0x7fa034004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa03400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05406c750 con 0x7fa034005180 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03400c080 con 0x7fa0340039c0 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.219+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03400c240 con 0x7fa0340047c0 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa03400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa052a00640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0340039c0 0x7fa034003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa04be00640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0340047c0 0x7fa034004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 -- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0340047c0 msgr2=0x7fa034004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0340047c0 0x7fa034004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 -- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0340039c0 msgr2=0x7fa034003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0340039c0 0x7fa034003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa053400640 1 -- 172.21.15.125:0/2874096643 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034003520 con 0x7fa034005180 2024-01-16T15:15:03.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.220+0000 7fa052a00640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0340039c0 0x7fa034003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:03.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.221+0000 7fa053400640 1 -- 172.21.15.125:0/2874096643 reap_dead start 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.221+0000 7fa053400640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa03400b980 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009f90 tx=0x7fa0340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ecbe478fd24d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.221+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c02cc20 con 0x7fa034005180 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.221+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c00f040 con 0x7fa034005180 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.221+0000 7fa052000640 1 -- 172.21.15.125:0/2874096643 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c003990 con 0x7fa034005180 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 -- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 msgr2=0x7fa03400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa03400b980 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009f90 tx=0x7fa0340057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 -- 172.21.15.125:0/2874096643 shutdown_connections 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 --2- 172.21.15.125:0/2874096643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa03400b980 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 -- 172.21.15.125:0/2874096643 >> 172.21.15.125:0/2874096643 conn(0x7fa05406a800 msgr2=0x7fa05406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 -- 172.21.15.125:0/2874096643 shutdown_connections 2024-01-16T15:15:03.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.222+0000 7fa059c00640 1 -- 172.21.15.125:0/2874096643 wait complete. 2024-01-16T15:15:03.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.224+0000 7fa059c00640 1 Processor -- start 2024-01-16T15:15:03.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.224+0000 7fa059c00640 1 -- start start 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 0x7fa054071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 0x7fa0540b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05413de00 con 0x7fa034005180 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05414f070 con 0x7fa054135520 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa059c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05414f350 con 0x7fa05414e900 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa052a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa052a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33962/0 (socket says 172.21.15.125:33962) 2024-01-16T15:15:03.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.225+0000 7fa052a00640 1 -- 172.21.15.125:0/2195638327 learned_addr learned my addr 172.21.15.125:0/2195638327 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 -- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 msgr2=0x7fa054071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa053400640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 0x7fa0540b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa04be00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 0x7fa054071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 0x7fa054071d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 -- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 msgr2=0x7fa0540b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 0x7fa0540b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 -- 172.21.15.125:0/2195638327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054071520 con 0x7fa034005180 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa04be00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 0x7fa054071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa053400640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 0x7fa0540b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa052a00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fa04400d770 tx=0x7fa04400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044005270 con 0x7fa034005180 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0440053d0 con 0x7fa034005180 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa054071800 con 0x7fa034005180 2024-01-16T15:15:03.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04400b8d0 con 0x7fa034005180 2024-01-16T15:15:03.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.226+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa054072260 con 0x7fa034005180 2024-01-16T15:15:03.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.228+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0540675d0 con 0x7fa034005180 2024-01-16T15:15:03.211 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.228+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa04400ba30 con 0x7fa034005180 2024-01-16T15:15:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.229+0000 7fa04aa00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 0x7fa02c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.229+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa044088a40 con 0x7fa034005180 2024-01-16T15:15:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.229+0000 7fa04be00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 0x7fa02c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.213 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.230+0000 7fa04be00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 0x7fa02c06e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0017a0 tx=0x7fa03c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.233+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0440bee90 con 0x7fa034005180 2024-01-16T15:15:03.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.425+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa05406df00 con 0x7fa034005180 2024-01-16T15:15:03.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.427+0000 7fa04aa00640 1 -- 172.21.15.125:0/2195638327 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+12907 (secure 0 0 0) 0x7fa044054fb0 con 0x7fa034005180 2024-01-16T15:15:03.410 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:03.410 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":107,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:02.506856+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:03.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 msgr2=0x7fa02c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 0x7fa02c06e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0017a0 tx=0x7fa03c009040 comp rx=0 tx=0).stop 2024-01-16T15:15:03.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 msgr2=0x7fa054150a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fa04400d770 tx=0x7fa04400dc30 comp rx=0 tx=0).stop 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 shutdown_connections 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa054135520 0x7fa0540b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa05414e900 0x7fa054071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa02c06c430 0x7fa02c06e8f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 --2- 172.21.15.125:0/2195638327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa034005180 0x7fa054150a30 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 >> 172.21.15.125:0/2195638327 conn(0x7fa05406a800 msgr2=0x7fa05415cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 shutdown_connections 2024-01-16T15:15:03.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.429+0000 7fa059c00640 1 -- 172.21.15.125:0/2195638327 wait complete. 2024-01-16T15:15:03.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:03.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.662+0000 7f90a4400640 1 Processor -- start 2024-01-16T15:15:03.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.662+0000 7f90a4400640 1 -- start start 2024-01-16T15:15:03.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c15e880 0x7f909c06ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a4400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f909c09a170 0x7f909c06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a4400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0cfa70 con 0x7f909c070490 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a4400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c0cff00 con 0x7f909c09a170 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c06fc70 con 0x7f909c15e880 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a2400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f909c09a170 0x7f909c06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a2400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f909c09a170 0x7f909c06e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33972/0 (socket says 172.21.15.125:33972) 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.664+0000 7f90a2400640 1 -- 172.21.15.125:0/3902007633 learned_addr learned my addr 172.21.15.125:0/3902007633 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.665+0000 7f90a1a00640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c15e880 0x7f909c06ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.665+0000 7f90a2400640 1 -- 172.21.15.125:0/3902007633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c15e880 msgr2=0x7f909c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.665+0000 7f90a2400640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c15e880 0x7f909c06ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.665+0000 7f90a2400640 1 -- 172.21.15.125:0/3902007633 >> v1:172.21.15.137:6789/0 conn(0x7f909c070490 legacy=0x7f909c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.665+0000 7f90a2400640 1 -- 172.21.15.125:0/3902007633 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0cf420 con 0x7f909c09a170 2024-01-16T15:15:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.666+0000 7f90a2400640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c09a170 0x7f909c06e7c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9094009870 tx=0x7f909402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8240bb6ec936b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.666+0000 7f909be00640 1 -- 172.21.15.125:0/3902007633 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094030840 con 0x7f909c09a170 2024-01-16T15:15:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.666+0000 7f909be00640 1 -- 172.21.15.125:0/3902007633 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940309a0 con 0x7f909c09a170 2024-01-16T15:15:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.666+0000 7f909be00640 1 -- 172.21.15.125:0/3902007633 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094030d30 con 0x7f909c09a170 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 -- 172.21.15.125:0/3902007633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c09a170 msgr2=0x7f909c06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c09a170 0x7f909c06e7c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9094009870 tx=0x7f909402b980 comp rx=0 tx=0).stop 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 -- 172.21.15.125:0/3902007633 shutdown_connections 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c09a170 0x7f909c06e7c0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 --2- 172.21.15.125:0/3902007633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c15e880 0x7f909c06ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 -- 172.21.15.125:0/3902007633 >> 172.21.15.125:0/3902007633 conn(0x7f909c0b2d00 msgr2=0x7f909c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 -- 172.21.15.125:0/3902007633 shutdown_connections 2024-01-16T15:15:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.667+0000 7f90a4400640 1 -- 172.21.15.125:0/3902007633 wait complete. 2024-01-16T15:15:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.669+0000 7f90a4400640 1 Processor -- start 2024-01-16T15:15:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.669+0000 7f90a4400640 1 -- start start 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c070490 0x7f909c15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f909c09a170 0x7f909c1f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0b4060 con 0x7f909c15e880 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1f0fa0 con 0x7f909c09a170 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1f1280 con 0x7f909c070490 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33984/0 (socket says 172.21.15.125:33984) 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.670+0000 7f90a2400640 1 -- 172.21.15.125:0/1854434848 learned_addr learned my addr 172.21.15.125:0/1854434848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a1a00640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c070490 0x7f909c15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a1000640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f909c09a170 0x7f909c1f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 -- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c070490 msgr2=0x7f909c15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c070490 0x7f909c15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 -- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f909c09a170 msgr2=0x7f909c1f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f909c09a170 0x7f909c1f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 -- 172.21.15.125:0/1854434848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940094d0 con 0x7f909c15e880 2024-01-16T15:15:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f90a2400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f909402b9c0 tx=0x7f9094004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.671+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094039c90 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.672+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094039df0 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.672+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1f1560 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.672+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094043640 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.672+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c1f2170 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.673+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c1f1a80 con 0x7f909c15e880 2024-01-16T15:15:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.674+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f90940437a0 con 0x7f909c15e880 2024-01-16T15:15:03.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.675+0000 7f9099600640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 0x7f907406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.675+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f909403e070 con 0x7f909c15e880 2024-01-16T15:15:03.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.676+0000 7f90a1a00640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 0x7f907406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.676+0000 7f90a1a00640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 0x7f907406e8f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f908c006fd0 tx=0x7f908c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.680+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9094078100 con 0x7f909c15e880 2024-01-16T15:15:03.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.870+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f909c0675d0 con 0x7f909c15e880 2024-01-16T15:15:03.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.870+0000 7f9099600640 1 -- 172.21.15.125:0/1854434848 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90940340a0 con 0x7f909c15e880 2024-01-16T15:15:03.853 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 msgr2=0x7f907406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 0x7f907406e8f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f908c006fd0 tx=0x7f908c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 msgr2=0x7f909c0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f909402b9c0 tx=0x7f9094004730 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 shutdown_connections 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f907406c430 0x7f907406e8f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f909c15e880 0x7f909c0caa80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f909c09a170 0x7f909c1f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 --2- 172.21.15.125:0/1854434848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f909c070490 0x7f909c15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.873+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 >> 172.21.15.125:0/1854434848 conn(0x7f909c0b2d00 msgr2=0x7f909c0b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.874+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 shutdown_connections 2024-01-16T15:15:03.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:03.874+0000 7f90a4400640 1 -- 172.21.15.125:0/1854434848 wait complete. 2024-01-16T15:15:03.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:15:04.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.095+0000 7fcd4f000640 1 Processor -- start 2024-01-16T15:15:04.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.095+0000 7fcd4f000640 1 -- start start 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd4f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd48166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd4f000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4815c5f0 con 0x7fcd4813df80 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd4f000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4806c470 con 0x7fcd4813e360 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd4f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4806c750 con 0x7fcd480cf9a0 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd47e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcd4813df80 0x7fcd4815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45314/0 (socket says 172.21.15.125:45314) 2024-01-16T15:15:04.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.097+0000 7fcd47e00640 1 -- 172.21.15.125:0/1025185340 learned_addr learned my addr 172.21.15.125:0/1025185340 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:04.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd4d000640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd48166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369021963 0 0) 0x7fcd4815c5f0 con 0x7fcd4813df80 2024-01-16T15:15:04.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28003520 con 0x7fcd4813df80 2024-01-16T15:15:04.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343540905 0 0) 0x7fcd28003520 con 0x7fcd4813df80 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 msgr2=0x7fcd48166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd48166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 >> v1:172.21.15.137:6789/0 conn(0x7fcd4813e360 legacy=0x7fcd480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4806ca30 con 0x7fcd4813df80 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcd38001670 con 0x7fcd4813df80 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 >> v1:172.21.15.125:6789/0 conn(0x7fcd4813df80 legacy=0x7fcd4815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd280039a0 0x7fcd28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.098+0000 7fcd47400640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd28004d00 0x7fcd28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4806c750 con 0x7fcd280039a0 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800ba90 con 0x7fcd28004d00 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800bc50 con 0x7fcd480cf9a0 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd280039a0 0x7fcd28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd3f200640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 -- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 msgr2=0x7fcd280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd480cf9a0 0x7fcd280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd4d000640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd28004d00 0x7fcd28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 -- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd28004d00 msgr2=0x7fcd28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd28004d00 0x7fcd28005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.099+0000 7fcd47e00640 1 -- 172.21.15.125:0/1025185340 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28003520 con 0x7fcd280039a0 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.100+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd280039a0 0x7fcd28003dc0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fcd38001eb0 tx=0x7fcd280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9baf498fb605fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38005ad0 con 0x7fcd280039a0 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd38005c30 con 0x7fcd280039a0 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd47400640 1 -- 172.21.15.125:0/1025185340 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3802c7c0 con 0x7fcd280039a0 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 -- 172.21.15.125:0/1025185340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd280039a0 msgr2=0x7fcd28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1025185340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd280039a0 0x7fcd28003dc0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fcd38001eb0 tx=0x7fcd280057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4d000640 1 -- 172.21.15.125:0/1025185340 reap_dead start 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 -- 172.21.15.125:0/1025185340 shutdown_connections 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 -- 172.21.15.125:0/1025185340 >> 172.21.15.125:0/1025185340 conn(0x7fcd4806a800 msgr2=0x7fcd4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 -- 172.21.15.125:0/1025185340 shutdown_connections 2024-01-16T15:15:04.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.101+0000 7fcd4f000640 1 -- 172.21.15.125:0/1025185340 wait complete. 2024-01-16T15:15:04.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.103+0000 7fcd4f000640 1 Processor -- start 2024-01-16T15:15:04.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.103+0000 7fcd4f000640 1 -- start start 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd280039a0 0x7fcd480ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 0x7fcd481ef520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481581b0 con 0x7fcd48157050 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480ae360 con 0x7fcd481ef070 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd4f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48157530 con 0x7fcd280039a0 2024-01-16T15:15:04.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd3f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd3f200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33996/0 (socket says 172.21.15.125:33996) 2024-01-16T15:15:04.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.104+0000 7fcd3f200640 1 -- 172.21.15.125:0/1302862771 learned_addr learned my addr 172.21.15.125:0/1302862771 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd280039a0 0x7fcd480ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd4d000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 0x7fcd481ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 -- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd280039a0 msgr2=0x7fcd480ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd280039a0 0x7fcd480ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 -- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 msgr2=0x7fcd481ef520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 0x7fcd481ef520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 -- 172.21.15.125:0/1302862771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48157810 con 0x7fcd48157050 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd4d000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 0x7fcd481ef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd3f200640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000e580 tx=0x7fcd3000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd300052e0 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd30005440 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30003810 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.105+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd48157af0 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.106+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd48153d70 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.107+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd481582f0 con 0x7fcd48157050 2024-01-16T15:15:04.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.108+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fcd30003970 con 0x7fcd48157050 2024-01-16T15:15:04.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.108+0000 7fcd44c00640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 0x7fcd2006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.109+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcd300889a0 con 0x7fcd48157050 2024-01-16T15:15:04.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.109+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 0x7fcd2006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.110+0000 7fcd47e00640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 0x7fcd2006e8f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcd38034000 tx=0x7fcd38009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.113+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd481582f0 con 0x7fcd48157050 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.308+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcd481582f0 con 0x7fcd48157050 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.309+0000 7fcd44c00640 1 -- 172.21.15.125:0/1302862771 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcd481582f0 con 0x7fcd48157050 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 msgr2=0x7fcd2006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 0x7fcd2006e8f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcd38034000 tx=0x7fcd38009030 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 msgr2=0x7fcd48153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000e580 tx=0x7fcd3000ea40 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 shutdown_connections 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd481ef070 0x7fcd481ef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcd2006c430 0x7fcd2006e8f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd48157050 0x7fcd48153830 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 --2- 172.21.15.125:0/1302862771 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd280039a0 0x7fcd480ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 >> 172.21.15.125:0/1302862771 conn(0x7fcd4806a800 msgr2=0x7fcd480afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 shutdown_connections 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.312+0000 7fcd4f000640 1 -- 172.21.15.125:0/1302862771 wait complete. 2024-01-16T15:15:04.312 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:15:04.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:15:04.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.544+0000 7fd024400640 1 Processor -- start 2024-01-16T15:15:04.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.544+0000 7fd024400640 1 -- start start 2024-01-16T15:15:04.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.546+0000 7fd024400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd01c0cf9a0 0x7fd01c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.546+0000 7fd024400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.546+0000 7fd024400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c0b1ee0 con 0x7fd01c143ae0 2024-01-16T15:15:04.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.546+0000 7fd024400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c0b2780 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.529 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.546+0000 7fd024400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c15e990 con 0x7fd01c15e1e0 2024-01-16T15:15:04.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd021000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd01c0cf9a0 0x7fd01c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd022400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd022400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:39632/0 (socket says 172.21.15.125:39632) 2024-01-16T15:15:04.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd022400640 1 -- 172.21.15.125:0/406739941 learned_addr learned my addr 172.21.15.125:0/406739941 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:04.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd021000640 1 -- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 msgr2=0x7fd01c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd021000640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd021000640 1 -- 172.21.15.125:0/406739941 >> v1:172.21.15.137:6789/0 conn(0x7fd01c143ae0 legacy=0x7fd01c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd021000640 1 -- 172.21.15.125:0/406739941 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c0b2130 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.547+0000 7fd022400640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:04.531 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.549+0000 7fd021000640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c0cf9a0 0x7fd01c15dca0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd00400ba40 tx=0x7fd00402e080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ff3ef104fda57c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.549+0000 7fd01be00640 1 -- 172.21.15.125:0/406739941 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00402ee30 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.549+0000 7fd01be00640 1 -- 172.21.15.125:0/406739941 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd004036440 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.549+0000 7fd01be00640 1 -- 172.21.15.125:0/406739941 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040367d0 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 -- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c0cf9a0 msgr2=0x7fd01c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c0cf9a0 0x7fd01c15dca0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd00400ba40 tx=0x7fd00402e080 comp rx=0 tx=0).stop 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 -- 172.21.15.125:0/406739941 shutdown_connections 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c15e1e0 0x7fd01c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 --2- 172.21.15.125:0/406739941 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c0cf9a0 0x7fd01c15dca0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 -- 172.21.15.125:0/406739941 >> 172.21.15.125:0/406739941 conn(0x7fd01c06a800 msgr2=0x7fd01c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 -- 172.21.15.125:0/406739941 shutdown_connections 2024-01-16T15:15:04.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.550+0000 7fd024400640 1 -- 172.21.15.125:0/406739941 wait complete. 2024-01-16T15:15:04.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.552+0000 7fd024400640 1 Processor -- start 2024-01-16T15:15:04.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.552+0000 7fd024400640 1 -- start start 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c0cf9a0 0x7fd01c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c15e1e0 0x7fd01c13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c13f870 con 0x7fd01c143ae0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c13ecf0 con 0x7fd01c15e1e0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd024400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c13efd0 con 0x7fd01c0cf9a0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd021000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd021000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33998/0 (socket says 172.21.15.125:33998) 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd021000640 1 -- 172.21.15.125:0/887530268 learned_addr learned my addr 172.21.15.125:0/887530268 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd021a00640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c0cf9a0 0x7fd01c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.553+0000 7fd022400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c15e1e0 0x7fd01c13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 -- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c0cf9a0 msgr2=0x7fd01c0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c0cf9a0 0x7fd01c0aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 -- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c15e1e0 msgr2=0x7fd01c13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c15e1e0 0x7fd01c13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 -- 172.21.15.125:0/887530268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd014009b30 con 0x7fd01c143ae0 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd021000640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd004039040 tx=0x7fd0040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00403f850 con 0x7fd01c143ae0 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00403f9d0 con 0x7fd01c143ae0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00403fd80 con 0x7fd01c143ae0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00400b6a0 con 0x7fd01c143ae0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.554+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c13f590 con 0x7fd01c143ae0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.555+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01c06e0d0 con 0x7fd01c143ae0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.556+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd00403d070 con 0x7fd01c143ae0 2024-01-16T15:15:04.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.557+0000 7fd019600640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 0x7fcff406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.557+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd0040aaf30 con 0x7fd01c143ae0 2024-01-16T15:15:04.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.558+0000 7fd021a00640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 0x7fcff406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.558+0000 7fd021a00640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 0x7fcff406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000f60 tx=0x7fd00c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.562+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd004077910 con 0x7fd01c143ae0 2024-01-16T15:15:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:04.755+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd01c0675d0 con 0x7fd01c143ae0 2024-01-16T15:15:05.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.523+0000 7fd019600640 1 -- 172.21.15.125:0/887530268 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd0040430b0 con 0x7fd01c143ae0 2024-01-16T15:15:05.506 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2024-01-16T15:15:05.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.526+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 msgr2=0x7fcff406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.526+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 0x7fcff406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000f60 tx=0x7fd00c009040 comp rx=0 tx=0).stop 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 msgr2=0x7fd01c13e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd004039040 tx=0x7fd0040046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 shutdown_connections 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd01c15e1e0 0x7fd01c13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcff406c430 0x7fcff406e8f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd01c143ae0 0x7fd01c13e0a0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 --2- 172.21.15.125:0/887530268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd01c0cf9a0 0x7fd01c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.527+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 >> 172.21.15.125:0/887530268 conn(0x7fd01c06a800 msgr2=0x7fd01c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.528+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 shutdown_connections 2024-01-16T15:15:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.528+0000 7fd024400640 1 -- 172.21.15.125:0/887530268 wait complete. 2024-01-16T15:15:05.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:15:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.762+0000 7f8bd9200640 1 Processor -- start 2024-01-16T15:15:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.762+0000 7f8bd9200640 1 -- start start 2024-01-16T15:15:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd9200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8bd415b1f0 0x7f8bd40cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd9200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8bd40d1710 0x7f8bd40b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd9200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd415b5d0 con 0x7f8bd40cbc00 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd9200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd40ce890 con 0x7f8bd40d1710 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd9200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd40cd7f0 con 0x7f8bd415b1f0 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd1400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8bd40cbc00 0x7f8bd40d1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45320/0 (socket says 172.21.15.125:45320) 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd1400640 1 -- 172.21.15.125:0/2356146221 learned_addr learned my addr 172.21.15.125:0/2356146221 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.764+0000 7f8bd2800640 1 --2- 172.21.15.125:0/2356146221 >> v2:172.21.15.125:3300/0 conn(0x7f8bd40d1710 0x7f8bd40b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd1e00640 1 --2- 172.21.15.125:0/2356146221 >> v2:172.21.15.137:3300/0 conn(0x7f8bd415b1f0 0x7f8bd40cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd0a00640 1 -- 172.21.15.125:0/2356146221 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037761467 0 0) 0x7f8bd415b5d0 con 0x7f8bd40cbc00 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd0a00640 1 -- 172.21.15.125:0/2356146221 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4003520 con 0x7f8bd40cbc00 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd2800640 1 -- 172.21.15.125:0/2356146221 >> v2:172.21.15.137:3300/0 conn(0x7f8bd415b1f0 msgr2=0x7f8bd40cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd2800640 1 --2- 172.21.15.125:0/2356146221 >> v2:172.21.15.137:3300/0 conn(0x7f8bd415b1f0 0x7f8bd40cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd2800640 1 -- 172.21.15.125:0/2356146221 >> v1:172.21.15.125:6789/0 conn(0x7f8bd40cbc00 legacy=0x7f8bd40d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.765+0000 7f8bd2800640 1 -- 172.21.15.125:0/2356146221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40cdad0 con 0x7f8bd40d1710 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.766+0000 7f8bd2800640 1 --2- 172.21.15.125:0/2356146221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd40b3f00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f8bd406b1c0 tx=0x7f8bcc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f73b77fce642ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.766+0000 7f8bd0a00640 1 -- 172.21.15.125:0/2356146221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc02e070 con 0x7f8bd40d1710 2024-01-16T15:15:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.766+0000 7f8bd0a00640 1 -- 172.21.15.125:0/2356146221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc0315a0 con 0x7f8bd40d1710 2024-01-16T15:15:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd0a00640 1 -- 172.21.15.125:0/2356146221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc031930 con 0x7f8bd40d1710 2024-01-16T15:15:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 -- 172.21.15.125:0/2356146221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 msgr2=0x7f8bd40b3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 --2- 172.21.15.125:0/2356146221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd40b3f00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f8bd406b1c0 tx=0x7f8bcc02b840 comp rx=0 tx=0).stop 2024-01-16T15:15:05.750 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 -- 172.21.15.125:0/2356146221 shutdown_connections 2024-01-16T15:15:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 --2- 172.21.15.125:0/2356146221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd40b3f00 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 --2- 172.21.15.125:0/2356146221 >> v2:172.21.15.137:3300/0 conn(0x7f8bd415b1f0 0x7f8bd40cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 -- 172.21.15.125:0/2356146221 >> 172.21.15.125:0/2356146221 conn(0x7f8bd4066900 msgr2=0x7f8bd406a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 -- 172.21.15.125:0/2356146221 shutdown_connections 2024-01-16T15:15:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.767+0000 7f8bd9200640 1 -- 172.21.15.125:0/2356146221 wait complete. 2024-01-16T15:15:05.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.769+0000 7f8bd9200640 1 Processor -- start 2024-01-16T15:15:05.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.769+0000 7f8bd9200640 1 -- start start 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bd40cbc00 0x7f8bd415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bd415b1f0 0x7f8bd4159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd4157cd0 con 0x7f8bd40d1710 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40adc30 con 0x7f8bd415b1f0 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd9200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40adf10 con 0x7f8bd40cbc00 2024-01-16T15:15:05.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd1400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34020/0 (socket says 172.21.15.125:34020) 2024-01-16T15:15:05.754 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.770+0000 7f8bd1400640 1 -- 172.21.15.125:0/4082928509 learned_addr learned my addr 172.21.15.125:0/4082928509 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1e00640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bd40cbc00 0x7f8bd415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd2800640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bd415b1f0 0x7f8bd4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 -- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bd40cbc00 msgr2=0x7f8bd415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bd40cbc00 0x7f8bd415e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 -- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bd415b1f0 msgr2=0x7f8bd4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bd415b1f0 0x7f8bd4159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 -- 172.21.15.125:0/4082928509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc009400 con 0x7f8bd40d1710 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd1400640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00cb90 tx=0x7f8bbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00ad40 con 0x7f8bd40d1710 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc012400 con 0x7f8bd40d1710 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00aea0 con 0x7f8bd40d1710 2024-01-16T15:15:05.755 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.771+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd40ae1f0 con 0x7f8bd40d1710 2024-01-16T15:15:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.772+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd40ae7b0 con 0x7f8bd40d1710 2024-01-16T15:15:05.756 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.772+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd4067670 con 0x7f8bd40d1710 2024-01-16T15:15:05.757 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.774+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8bbc0127f0 con 0x7f8bd40d1710 2024-01-16T15:15:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.775+0000 7f8bca000640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 0x7f8bac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.775+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8bbc089280 con 0x7f8bd40d1710 2024-01-16T15:15:05.758 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.775+0000 7f8bd1e00640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 0x7f8bac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.776+0000 7f8bd1e00640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 0x7f8bac06e8f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009a30 tx=0x7f8bc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.761 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.779+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bbc0558a0 con 0x7f8bd40d1710 2024-01-16T15:15:05.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:05.975+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8bd406f460 con 0x7f8bd40d1710 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.529+0000 7f8bca000640 1 -- 172.21.15.125:0/4082928509 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8bbc0170d0 con 0x7f8bd40d1710 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 msgr2=0x7f8bac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 0x7f8bac06e8f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009a30 tx=0x7f8bc4008040 comp rx=0 tx=0).stop 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 msgr2=0x7f8bd415c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00cb90 tx=0x7f8bbc00a430 comp rx=0 tx=0).stop 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 shutdown_connections 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.532+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bd415b1f0 0x7f8bd4159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bac06c430 0x7f8bac06e8f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bd40d1710 0x7f8bd415c8e0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 --2- 172.21.15.125:0/4082928509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bd40cbc00 0x7f8bd415e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 >> 172.21.15.125:0/4082928509 conn(0x7f8bd4066900 msgr2=0x7f8bd406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 shutdown_connections 2024-01-16T15:15:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.533+0000 7f8bd9200640 1 -- 172.21.15.125:0/4082928509 wait complete. 2024-01-16T15:15:06.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:15:06.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.747+0000 7f52e1800640 1 Processor -- start 2024-01-16T15:15:06.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.747+0000 7f52e1800640 1 -- start start 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52e1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52e1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc06dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52e1800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc15deb0 con 0x7f52dc1618f0 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52e1800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc15ea30 con 0x7f52dc157c30 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52e1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc06d860 con 0x7f52dc071200 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52daa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52da000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f52dc1618f0 0x7f52dc070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51874/0 (socket says 172.21.15.125:51874) 2024-01-16T15:15:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52da000640 1 -- 172.21.15.125:0/2340964347 learned_addr learned my addr 172.21.15.125:0/2340964347 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.749+0000 7f52db400640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc06dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52daa00640 1 -- 172.21.15.125:0/2340964347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 msgr2=0x7f52dc06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52daa00640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc06dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52daa00640 1 -- 172.21.15.125:0/2340964347 >> v1:172.21.15.125:6789/0 conn(0x7f52dc1618f0 legacy=0x7f52dc070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52daa00640 1 -- 172.21.15.125:0/2340964347 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc15e190 con 0x7f52dc157c30 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52db400640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.750+0000 7f52d9600640 1 -- 172.21.15.125:0/2340964347 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650168494 0 0) 0x7f52dc15deb0 con 0x7f52dc1618f0 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.751+0000 7f52daa00640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6ec7ca9c4312f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.751+0000 7f52d9600640 1 -- 172.21.15.125:0/2340964347 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc02d040 con 0x7f52dc157c30 2024-01-16T15:15:06.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.751+0000 7f52d9600640 1 -- 172.21.15.125:0/2340964347 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc031810 con 0x7f52dc157c30 2024-01-16T15:15:06.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.751+0000 7f52d9600640 1 -- 172.21.15.125:0/2340964347 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc031ba0 con 0x7f52dc157c30 2024-01-16T15:15:06.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 -- 172.21.15.125:0/2340964347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 msgr2=0x7f52dc0cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 -- 172.21.15.125:0/2340964347 shutdown_connections 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 --2- 172.21.15.125:0/2340964347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 -- 172.21.15.125:0/2340964347 >> 172.21.15.125:0/2340964347 conn(0x7f52dc0cf700 msgr2=0x7f52dc0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 -- 172.21.15.125:0/2340964347 shutdown_connections 2024-01-16T15:15:06.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.752+0000 7f52e1800640 1 -- 172.21.15.125:0/2340964347 wait complete. 2024-01-16T15:15:06.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.754+0000 7f52e1800640 1 Processor -- start 2024-01-16T15:15:06.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.754+0000 7f52e1800640 1 -- start start 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52dc1618f0 0x7f52dc0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc15c8c0 con 0x7f52dc157c30 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc15bcb0 con 0x7f52dc1618f0 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52e1800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc15bf90 con 0x7f52dc071200 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52da000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52da000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49012/0 (socket says 172.21.15.125:49012) 2024-01-16T15:15:06.738 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52da000640 1 -- 172.21.15.125:0/848695419 learned_addr learned my addr 172.21.15.125:0/848695419 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.755+0000 7f52daa00640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52db400640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52dc1618f0 0x7f52dc0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 -- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 msgr2=0x7f52dc15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 -- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52dc1618f0 msgr2=0x7f52dc0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52dc1618f0 0x7f52dc0cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 -- 172.21.15.125:0/848695419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc009400 con 0x7f52dc157c30 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52da000640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f52c400c4a0 tx=0x7f52c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c4005ce0 con 0x7f52dc157c30 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c4005e40 con 0x7f52dc157c30 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c40072b0 con 0x7f52dc157c30 2024-01-16T15:15:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc15c270 con 0x7f52dc157c30 2024-01-16T15:15:06.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.756+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc155f80 con 0x7f52dc157c30 2024-01-16T15:15:06.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.757+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc15ca70 con 0x7f52dc157c30 2024-01-16T15:15:06.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.759+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f52c4007410 con 0x7f52dc157c30 2024-01-16T15:15:06.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.760+0000 7f52d2a00640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 0x7f52b406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.760+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f52c4088010 con 0x7f52dc157c30 2024-01-16T15:15:06.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.760+0000 7f52daa00640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 0x7f52b406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.761+0000 7f52daa00640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 0x7f52b406e920 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f52cc036000 tx=0x7f52cc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.765+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52dc15ca70 con 0x7f52dc157c30 2024-01-16T15:15:06.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:06.980+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f52dc06d3d0 con 0x7f52dc157c30 2024-01-16T15:15:07.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.573+0000 7f52d2a00640 1 -- 172.21.15.125:0/848695419 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v69) v1 ==== 174+0+0 (secure 0 0 0) 0x7f52c4017070 con 0x7f52dc157c30 2024-01-16T15:15:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 16 and data pool 17 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.579+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 msgr2=0x7f52b406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.579+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 0x7f52b406e920 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f52cc036000 tx=0x7f52cc03c040 comp rx=0 tx=0).stop 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.579+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 msgr2=0x7f52dc0cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.579+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f52c400c4a0 tx=0x7f52c400c960 comp rx=0 tx=0).stop 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 shutdown_connections 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52dc1618f0 0x7f52dc0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f52b406c460 0x7f52b406e920 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52dc157c30 0x7f52dc0cd600 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 --2- 172.21.15.125:0/848695419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52dc071200 0x7f52dc15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 >> 172.21.15.125:0/848695419 conn(0x7f52dc0cf700 msgr2=0x7f52dc0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 shutdown_connections 2024-01-16T15:15:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.581+0000 7f52e1800640 1 -- 172.21.15.125:0/848695419 wait complete. 2024-01-16T15:15:07.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.787+0000 7fa17f600640 1 Processor -- start 2024-01-16T15:15:07.770 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.787+0000 7fa17f600640 1 -- start start 2024-01-16T15:15:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.788+0000 7fa17f600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa1780d17e0 0x7fa1780d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.771 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.788+0000 7fa17f600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1780d2180 con 0x7fa1780b3890 2024-01-16T15:15:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa17f600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1780b3400 con 0x7fa17815d610 2024-01-16T15:15:07.772 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa17f600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1780d1390 con 0x7fa1780d17e0 2024-01-16T15:15:07.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa17d600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa1780b3890 0x7fa1780b3ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51878/0 (socket says 172.21.15.125:51878) 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa17d600640 1 -- 172.21.15.125:0/3664837693 learned_addr learned my addr 172.21.15.125:0/3664837693 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa177e00640 1 --2- 172.21.15.125:0/3664837693 >> v2:172.21.15.137:3300/0 conn(0x7fa1780d17e0 0x7fa1780d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876265236 0 0) 0x7fa1780d2180 con 0x7fa1780b3890 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.789+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa158003520 con 0x7fa1780b3890 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607660818 0 0) 0x7fa158003520 con 0x7fa1780b3890 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 >> v2:172.21.15.137:3300/0 conn(0x7fa1780d17e0 msgr2=0x7fa1780d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 --2- 172.21.15.125:0/3664837693 >> v2:172.21.15.137:3300/0 conn(0x7fa1780d17e0 0x7fa1780d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 >> v1:172.21.15.137:6789/0 conn(0x7fa17815d610 legacy=0x7fa1780ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1780b27f0 con 0x7fa1780b3890 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa1700019f0 con 0x7fa1780b3890 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 >> v1:172.21.15.125:6789/0 conn(0x7fa1780b3890 legacy=0x7fa1780b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 0x7fa158003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158005180 0x7fa15800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.774 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d1390 con 0x7fa158003970 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15800c020 con 0x7fa158005180 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15800c1e0 con 0x7fa1580047c0 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa17cc00640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 0x7fa158003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.790+0000 7fa177e00640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17d600640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158005180 0x7fa15800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17cc00640 1 -- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1580047c0 msgr2=0x7fa158004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17cc00640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17cc00640 1 -- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158005180 msgr2=0x7fa15800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17cc00640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158005180 0x7fa15800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17cc00640 1 -- 172.21.15.125:0/3664837693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158003520 con 0x7fa158003970 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17d600640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158005180 0x7fa15800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.791+0000 7fa17d600640 1 -- 172.21.15.125:0/3664837693 reap_dead start 2024-01-16T15:15:07.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.792+0000 7fa17cc00640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 0x7fa158003d90 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fa168001750 tx=0x7fa1580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8ee859ac3597fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.792+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680052d0 con 0x7fa158003970 2024-01-16T15:15:07.776 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.792+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16802e050 con 0x7fa158003970 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.792+0000 7fa177400640 1 -- 172.21.15.125:0/3664837693 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168003830 con 0x7fa158003970 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 -- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 msgr2=0x7fa158003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 0x7fa158003d90 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fa168001750 tx=0x7fa1580057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 -- 172.21.15.125:0/3664837693 shutdown_connections 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 --2- 172.21.15.125:0/3664837693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa158003970 0x7fa158003d90 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 -- 172.21.15.125:0/3664837693 >> 172.21.15.125:0/3664837693 conn(0x7fa17809c540 msgr2=0x7fa1780d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 -- 172.21.15.125:0/3664837693 shutdown_connections 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.793+0000 7fa17f600640 1 -- 172.21.15.125:0/3664837693 wait complete. 2024-01-16T15:15:07.777 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.795+0000 7fa17f600640 1 Processor -- start 2024-01-16T15:15:07.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.795+0000 7fa17f600640 1 -- start start 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158003970 0x7fa1780cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa178159980 0x7fa1780cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780b8a00 con 0x7fa17815e9f0 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780b8c80 con 0x7fa158003970 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17815df10 con 0x7fa178159980 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49020/0 (socket says 172.21.15.125:49020) 2024-01-16T15:15:07.779 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 -- 172.21.15.125:0/1170274012 learned_addr learned my addr 172.21.15.125:0/1170274012 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:07.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 -- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa178159980 msgr2=0x7fa1780cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa178159980 0x7fa1780cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 -- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158003970 msgr2=0x7fa1780cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158003970 0x7fa1780cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.796+0000 7fa17d600640 1 -- 172.21.15.125:0/1170274012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17815e1f0 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa17d600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fa170001580 tx=0x7fa17000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170004d60 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17000bb80 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170007070 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17815e4d0 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.797+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1780cc910 con 0x7fa17815e9f0 2024-01-16T15:15:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.798+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1780675d0 con 0x7fa17815e9f0 2024-01-16T15:15:07.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.799+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa1700071d0 con 0x7fa17815e9f0 2024-01-16T15:15:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.800+0000 7fa174c00640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 0x7fa15006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.800+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa170088d40 con 0x7fa17815e9f0 2024-01-16T15:15:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.800+0000 7fa17cc00640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 0x7fa15006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.801+0000 7fa17cc00640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 0x7fa15006e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa168033000 tx=0x7fa16802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.804+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1700552b0 con 0x7fa17815e9f0 2024-01-16T15:15:07.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.992+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17806c8d0 con 0x7fa17815e9f0 2024-01-16T15:15:07.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.993+0000 7fa174c00640 1 -- 172.21.15.125:0/1170274012 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fa1700588d0 con 0x7fa17815e9f0 2024-01-16T15:15:07.977 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:07.977 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":110,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:06.981948+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:07.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 msgr2=0x7fa15006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 0x7fa15006e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa168033000 tx=0x7fa16802f000 comp rx=0 tx=0).stop 2024-01-16T15:15:07.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 msgr2=0x7fa17815ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fa170001580 tx=0x7fa17000b4a0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 shutdown_connections 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa15006c460 0x7fa15006e920 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17815e9f0 0x7fa17815ee70 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa178159980 0x7fa1780cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 --2- 172.21.15.125:0/1170274012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa158003970 0x7fa1780cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 >> 172.21.15.125:0/1170274012 conn(0x7fa17809c540 msgr2=0x7fa1780d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 shutdown_connections 2024-01-16T15:15:07.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:07.996+0000 7fa17f600640 1 -- 172.21.15.125:0/1170274012 wait complete. 2024-01-16T15:15:07.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:08.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.208+0000 7f8dd8e00640 1 Processor -- start 2024-01-16T15:15:08.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.208+0000 7f8dd8e00640 1 -- start start 2024-01-16T15:15:08.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.209+0000 7f8dd8e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8dd415afb0 0x7f8dd413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.209+0000 7f8dd8e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd413ff30 con 0x7f8dd413cfa0 2024-01-16T15:15:08.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.209+0000 7f8dd8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd40cdc40 con 0x7f8dd413fae0 2024-01-16T15:15:08.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.209+0000 7f8dd8e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd40cdec0 con 0x7f8dd415afb0 2024-01-16T15:15:08.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd3400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8dd413cfa0 0x7f8dd40cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51880/0 (socket says 172.21.15.125:51880) 2024-01-16T15:15:08.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd3400640 1 -- 172.21.15.125:0/3280180163 learned_addr learned my addr 172.21.15.125:0/3280180163 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:08.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458306060 0 0) 0x7f8dc4001110 con 0x7f8dd413cfa0 2024-01-16T15:15:08.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2a00640 1 --2- 172.21.15.125:0/3280180163 >> v2:172.21.15.137:3300/0 conn(0x7f8dd415afb0 0x7f8dd413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.194 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4005730 con 0x7f8dd413cfa0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061590396 0 0) 0x7f8db4005730 con 0x7f8dd413cfa0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 >> v2:172.21.15.137:3300/0 conn(0x7f8dd415afb0 msgr2=0x7f8dd413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 --2- 172.21.15.125:0/3280180163 >> v2:172.21.15.137:3300/0 conn(0x7f8dd415afb0 0x7f8dd413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 >> v1:172.21.15.137:6789/0 conn(0x7f8dd413fae0 legacy=0x7f8dd40b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40ce1a0 con 0x7f8dd413cfa0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f8dc4001800 con 0x7f8dd413cfa0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 >> v1:172.21.15.125:6789/0 conn(0x7f8dd413cfa0 legacy=0x7f8dd40cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.210+0000 7f8dd2000640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8db4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd2000640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db40069d0 0x7f8db4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd2000640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db4007390 0x7f8db400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd40cdec0 con 0x7f8db4005b80 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db400e330 con 0x7f8db4007390 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db400e4f0 con 0x7f8db40069d0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8db4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 -- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db40069d0 msgr2=0x7f8db4006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db40069d0 0x7f8db4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 -- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db4007390 msgr2=0x7f8db400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db4007390 0x7f8db400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3400640 1 -- 172.21.15.125:0/3280180163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4005730 con 0x7f8db4005b80 2024-01-16T15:15:08.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.211+0000 7f8dd3e00640 1 -- 172.21.15.125:0/3280180163 reap_dead start 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.212+0000 7f8dd3400640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8db4005fa0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400c240 tx=0x7f8db4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4ae7c21641f3a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.212+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4005ad0 con 0x7f8db4005b80 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.212+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc4005c50 con 0x7f8db4005b80 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.212+0000 7f8dd2000640 1 -- 172.21.15.125:0/3280180163 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc402c8b0 con 0x7f8db4005b80 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 -- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 msgr2=0x7f8db4005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8db4005fa0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400c240 tx=0x7f8db4007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 -- 172.21.15.125:0/3280180163 shutdown_connections 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/3280180163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8db4005fa0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 -- 172.21.15.125:0/3280180163 >> 172.21.15.125:0/3280180163 conn(0x7f8dd406a800 msgr2=0x7f8dd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 -- 172.21.15.125:0/3280180163 shutdown_connections 2024-01-16T15:15:08.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.213+0000 7f8dd8e00640 1 -- 172.21.15.125:0/3280180163 wait complete. 2024-01-16T15:15:08.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.215+0000 7f8dd8e00640 1 Processor -- start 2024-01-16T15:15:08.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.215+0000 7f8dd8e00640 1 -- start start 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 0x7f8dd4153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8dd4158cb0 0x7f8dd4159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd4159670 con 0x7f8db4005b80 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd413c620 con 0x7f8dd413a230 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd41f4c40 con 0x7f8dd4158cb0 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49046/0 (socket says 172.21.15.125:49046) 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.216+0000 7f8dd3400640 1 -- 172.21.15.125:0/1405647549 learned_addr learned my addr 172.21.15.125:0/1405647549 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd2a00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 0x7f8dd4153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 -- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8dd4158cb0 msgr2=0x7f8dd4159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8dd4158cb0 0x7f8dd4159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 -- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 msgr2=0x7f8dd4153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 0x7f8dd4153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 -- 172.21.15.125:0/1405647549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd413cab0 con 0x7f8db4005b80 2024-01-16T15:15:08.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd2a00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 0x7f8dd4153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:08.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd3400640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4001ac0 tx=0x7f8dc402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc403c040 con 0x7f8db4005b80 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc402cd80 con 0x7f8db4005b80 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd413ba10 con 0x7f8db4005b80 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4007270 con 0x7f8db4005b80 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.217+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd413bfd0 con 0x7f8db4005b80 2024-01-16T15:15:08.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.219+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd4067450 con 0x7f8db4005b80 2024-01-16T15:15:08.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.220+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8dc4032070 con 0x7f8db4005b80 2024-01-16T15:15:08.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.221+0000 7f8dcb400640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 0x7f8dac06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.221+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dc40a99b0 con 0x7f8db4005b80 2024-01-16T15:15:08.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.221+0000 7f8dd2a00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 0x7f8dac06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.222+0000 7f8dd2a00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 0x7f8dac06e920 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40aea00 tx=0x7f8dbc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.225+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dc40762d0 con 0x7f8db4005b80 2024-01-16T15:15:08.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.417+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd413ff30 con 0x7f8db4005b80 2024-01-16T15:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.420+0000 7f8dcb400640 1 -- 172.21.15.125:0/1405647549 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f8dd413ff30 con 0x7f8db4005b80 2024-01-16T15:15:08.403 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:08.403 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":110,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:06.981948+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:08.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 msgr2=0x7f8dac06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 0x7f8dac06e920 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40aea00 tx=0x7f8dbc00b040 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 msgr2=0x7f8dd41f4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4001ac0 tx=0x7f8dc402e070 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 shutdown_connections 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8dd4158cb0 0x7f8dd4159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8dd413a230 0x7f8dd4153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8dac06c460 0x7f8dac06e920 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 --2- 172.21.15.125:0/1405647549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db4005b80 0x7f8dd41f4700 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 >> 172.21.15.125:0/1405647549 conn(0x7f8dd406a800 msgr2=0x7f8dd406cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 shutdown_connections 2024-01-16T15:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.422+0000 7f8dd8e00640 1 -- 172.21.15.125:0/1405647549 wait complete. 2024-01-16T15:15:08.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:15:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.654+0000 7f8484600640 1 Processor -- start 2024-01-16T15:15:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.655+0000 7f8484600640 1 -- start start 2024-01-16T15:15:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.656+0000 7f8484600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.656+0000 7f8484600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.656+0000 7f8484600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0cfc30 con 0x7f847c0cf7d0 2024-01-16T15:15:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.656+0000 7f8484600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c0b1ee0 con 0x7f847c15e3b0 2024-01-16T15:15:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.656+0000 7f8484600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0b2780 con 0x7f847c143ae0 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8482600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8482600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:56000/0 (socket says 172.21.15.125:56000) 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481200640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f847c0cf7d0 0x7f847c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:51376/0 (socket says 172.21.15.125:51376) 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:35482/0 (socket says 172.21.15.125:35482) 2024-01-16T15:15:08.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8482600640 1 -- 172.21.15.125:0/2224474417 learned_addr learned my addr 172.21.15.125:0/2224474417 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 -- 172.21.15.125:0/2224474417 >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 msgr2=0x7f847c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 --2- 172.21.15.125:0/2224474417 >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 -- 172.21.15.125:0/2224474417 >> v1:172.21.15.137:6789/0 conn(0x7f847c0cf7d0 legacy=0x7f847c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.657+0000 7f8481c00640 1 -- 172.21.15.125:0/2224474417 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c15e990 con 0x7f847c143ae0 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.658+0000 7f8482600640 1 --2- 172.21.15.125:0/2224474417 >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.659+0000 7f8481c00640 1 --2- 172.21.15.125:0/2224474417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f846c0097a0 tx=0x7f846c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d07ce3c4acaacaa5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.659+0000 7f847be00640 1 -- 172.21.15.125:0/2224474417 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c02d040 con 0x7f847c143ae0 2024-01-16T15:15:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.659+0000 7f847be00640 1 -- 172.21.15.125:0/2224474417 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c031810 con 0x7f847c143ae0 2024-01-16T15:15:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.659+0000 7f847be00640 1 -- 172.21.15.125:0/2224474417 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c031ba0 con 0x7f847c143ae0 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 -- 172.21.15.125:0/2224474417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 msgr2=0x7f847c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 --2- 172.21.15.125:0/2224474417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f846c0097a0 tx=0x7f846c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 -- 172.21.15.125:0/2224474417 shutdown_connections 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 --2- 172.21.15.125:0/2224474417 >> v2:172.21.15.137:3300/0 conn(0x7f847c15e3b0 0x7f847c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 --2- 172.21.15.125:0/2224474417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c143ae0 0x7f847c0cf290 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 -- 172.21.15.125:0/2224474417 >> 172.21.15.125:0/2224474417 conn(0x7f847c06a800 msgr2=0x7f847c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 -- 172.21.15.125:0/2224474417 shutdown_connections 2024-01-16T15:15:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.660+0000 7f8484600640 1 -- 172.21.15.125:0/2224474417 wait complete. 2024-01-16T15:15:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.662+0000 7f8484600640 1 Processor -- start 2024-01-16T15:15:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.662+0000 7f8484600640 1 -- start start 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 0x7f847c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c15e3b0 0x7f847c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c14b210 con 0x7f847c0cf7d0 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c158f50 con 0x7f847c143ae0 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8484600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c159230 con 0x7f847c15e3b0 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8481c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8481c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49058/0 (socket says 172.21.15.125:49058) 2024-01-16T15:15:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8481c00640 1 -- 172.21.15.125:0/3967749115 learned_addr learned my addr 172.21.15.125:0/3967749115 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.663+0000 7f8482600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c15e3b0 0x7f847c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481200640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 0x7f847c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 -- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c15e3b0 msgr2=0x7f847c0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c15e3b0 0x7f847c0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 -- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 msgr2=0x7f847c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 0x7f847c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 -- 172.21.15.125:0/3967749115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474009b30 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481200640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 0x7f847c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.664+0000 7f8481c00640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.665+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c006e10 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.665+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c03ad60 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.665+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c044610 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.665+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c009400 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.665+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c1597f0 con 0x7f847c0cf7d0 2024-01-16T15:15:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.666+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c0675d0 con 0x7f847c0cf7d0 2024-01-16T15:15:08.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.668+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f846c04d450 con 0x7f847c0cf7d0 2024-01-16T15:15:08.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.669+0000 7f8479600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 0x7f845406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.669+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f846c0aad70 con 0x7f847c0cf7d0 2024-01-16T15:15:08.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.669+0000 7f8481200640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 0x7f845406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.670+0000 7f8481200640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 0x7f845406e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f847c158b20 tx=0x7f8464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.673+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f846c0772e0 con 0x7f847c0cf7d0 2024-01-16T15:15:08.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:08.904+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f847c15a7a0 con 0x7f847c0cf7d0 2024-01-16T15:15:09.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.589+0000 7f8479600640 1 -- 172.21.15.125:0/3967749115 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f846c07a900 con 0x7f847c0cf7d0 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 msgr2=0x7f845406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 0x7f845406e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f847c158b20 tx=0x7f8464009040 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 msgr2=0x7f847c14acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 shutdown_connections 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.592+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f847c15e3b0 0x7f847c0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f847c143ae0 0x7f847c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f845406c430 0x7f845406e8f0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 --2- 172.21.15.125:0/3967749115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f847c0cf7d0 0x7f847c14acd0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 >> 172.21.15.125:0/3967749115 conn(0x7f847c06a800 msgr2=0x7f847c15c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 shutdown_connections 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.593+0000 7f8484600640 1 -- 172.21.15.125:0/3967749115 wait complete. 2024-01-16T15:15:09.596 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:15:09.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:15:09.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.825+0000 7f2b9ea00640 1 Processor -- start 2024-01-16T15:15:09.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.825+0000 7f2b9ea00640 1 -- start start 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.826+0000 7f2b9ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b98070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.826+0000 7f2b9ea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9815f4e0 con 0x7f2b9815bf10 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.826+0000 7f2b9ea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9815c3c0 con 0x7f2b980adf20 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.826+0000 7f2b9ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9815f970 con 0x7f2b980ccb50 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.827+0000 7f2b9e000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2b9815bf10 0x7f2b98143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51888/0 (socket says 172.21.15.125:51888) 2024-01-16T15:15:09.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.827+0000 7f2b9e000640 1 -- 172.21.15.125:0/710894206 learned_addr learned my addr 172.21.15.125:0/710894206 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.827+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b98070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.827+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051093541 0 0) 0x7f2b9815f4e0 con 0x7f2b9815bf10 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.827+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78003520 con 0x7f2b9815bf10 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3316756438 0 0) 0x7f2b78003520 con 0x7f2b9815bf10 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 msgr2=0x7f2b98070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b98070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 >> v1:172.21.15.137:6789/0 conn(0x7f2b980adf20 legacy=0x7f2b980cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98071130 con 0x7f2b9815bf10 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2b90001670 con 0x7f2b9815bf10 2024-01-16T15:15:09.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 >> v1:172.21.15.125:6789/0 conn(0x7f2b9815bf10 legacy=0x7f2b98143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b780039a0 0x7f2b78003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b78004d00 0x7f2b78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9815f970 con 0x7f2b780039a0 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7800bab0 con 0x7f2b78004d00 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7800bc70 con 0x7f2b980ccb50 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b780039a0 0x7f2b78003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9e000640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b78004d00 0x7f2b78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 -- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 msgr2=0x7f2b780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 -- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b78004d00 msgr2=0x7f2b78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b78004d00 0x7f2b78005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.828+0000 7f2b9d600640 1 -- 172.21.15.125:0/710894206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78003520 con 0x7f2b780039a0 2024-01-16T15:15:09.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.829+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980ccb50 0x7f2b780047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:09.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9d600640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b780039a0 0x7f2b78003dc0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f2b88001e50 tx=0x7f2b780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8494ffed120a950c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88003b70 con 0x7f2b780039a0 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88003cd0 con 0x7f2b780039a0 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b97e00640 1 -- 172.21.15.125:0/710894206 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88005310 con 0x7f2b780039a0 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9ea00640 1 -- 172.21.15.125:0/710894206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b780039a0 msgr2=0x7f2b78003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/710894206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b780039a0 0x7f2b78003dc0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f2b88001e50 tx=0x7f2b780057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9e000640 1 -- 172.21.15.125:0/710894206 reap_dead start 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9ea00640 1 -- 172.21.15.125:0/710894206 shutdown_connections 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.830+0000 7f2b9ea00640 1 -- 172.21.15.125:0/710894206 >> 172.21.15.125:0/710894206 conn(0x7f2b9806a7d0 msgr2=0x7f2b9806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.831+0000 7f2b9ea00640 1 -- 172.21.15.125:0/710894206 shutdown_connections 2024-01-16T15:15:09.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.831+0000 7f2b9ea00640 1 -- 172.21.15.125:0/710894206 wait complete. 2024-01-16T15:15:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.832+0000 7f2b9ea00640 1 Processor -- start 2024-01-16T15:15:09.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.832+0000 7f2b9ea00640 1 -- start start 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b780039a0 0x7f2b9815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980b00a0 0x7f2b980b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980b0a60 con 0x7f2b9814c250 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814a3d0 con 0x7f2b780039a0 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.833+0000 7f2b9ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814c9c0 con 0x7f2b980b00a0 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49076/0 (socket says 172.21.15.125:49076) 2024-01-16T15:15:09.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 -- 172.21.15.125:0/3754538298 learned_addr learned my addr 172.21.15.125:0/3754538298 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:09.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9e000640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980b00a0 0x7f2b980b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9d600640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b780039a0 0x7f2b9815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 -- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980b00a0 msgr2=0x7f2b980b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980b00a0 0x7f2b980b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 -- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b780039a0 msgr2=0x7f2b9815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b780039a0 0x7f2b9815d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 -- 172.21.15.125:0/3754538298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9814cca0 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.834+0000 7f2b9cc00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f2b80010880 tx=0x7f2b80010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.835+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000eb60 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.835+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8000ecc0 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.835+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8001c400 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.835+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9814cf80 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.835+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9813ca50 con 0x7f2b9814c250 2024-01-16T15:15:09.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.836+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9814a1c0 con 0x7f2b9814c250 2024-01-16T15:15:09.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.837+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2b8001c560 con 0x7f2b9814c250 2024-01-16T15:15:09.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.838+0000 7f2b95600640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 0x7f2b7006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.838+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b800890a0 con 0x7f2b9814c250 2024-01-16T15:15:09.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.838+0000 7f2b9d600640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 0x7f2b7006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.839+0000 7f2b9d600640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 0x7f2b7006e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2b88034000 tx=0x7f2b88009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:09.843+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b800556c0 con 0x7f2b9814c250 2024-01-16T15:15:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.071+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2b98067450 con 0x7f2b9814c250 2024-01-16T15:15:10.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.602+0000 7f2b95600640 1 -- 172.21.15.125:0/3754538298 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2b80058ce0 con 0x7f2b9814c250 2024-01-16T15:15:10.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 msgr2=0x7f2b7006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 0x7f2b7006e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2b88034000 tx=0x7f2b88009030 comp rx=0 tx=0).stop 2024-01-16T15:15:10.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 msgr2=0x7f2b9813c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f2b80010880 tx=0x7f2b80010d40 comp rx=0 tx=0).stop 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 shutdown_connections 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b980b00a0 0x7f2b980b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b7006c430 0x7f2b7006e8f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b9814c250 0x7f2b9813c510 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 --2- 172.21.15.125:0/3754538298 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b780039a0 0x7f2b9815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 >> 172.21.15.125:0/3754538298 conn(0x7f2b9806a7d0 msgr2=0x7f2b9815dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 shutdown_connections 2024-01-16T15:15:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.606+0000 7f2b9ea00640 1 -- 172.21.15.125:0/3754538298 wait complete. 2024-01-16T15:15:10.610 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:15:10.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.835+0000 7f00e4400640 1 Processor -- start 2024-01-16T15:15:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.835+0000 7f00e4400640 1 -- start start 2024-01-16T15:15:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 0x7f00dc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e4400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00dc0cf9a0 0x7f00dc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e4400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc06c470 con 0x7f00dc13e360 2024-01-16T15:15:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e4400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc06c750 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc06ca30 con 0x7f00dc13df80 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e2400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00dc0cf9a0 0x7f00dc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e2400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00dc0cf9a0 0x7f00dc0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49080/0 (socket says 172.21.15.125:49080) 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e2400640 1 -- 172.21.15.125:0/4221462473 learned_addr learned my addr 172.21.15.125:0/4221462473 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.837+0000 7f00e1a00640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 0x7f00dc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.838+0000 7f00e2400640 1 -- 172.21.15.125:0/4221462473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 msgr2=0x7f00dc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.838+0000 7f00e2400640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 0x7f00dc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.838+0000 7f00e2400640 1 -- 172.21.15.125:0/4221462473 >> v1:172.21.15.137:6789/0 conn(0x7f00dc13e360 legacy=0x7f00dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.838+0000 7f00e2400640 1 -- 172.21.15.125:0/4221462473 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc06cd10 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.838+0000 7f00e1a00640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 0x7f00dc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:10.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.839+0000 7f00e2400640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc0cf9a0 0x7f00dc0cd840 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af272140a7565eff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00dbe00640 1 -- 172.21.15.125:0/4221462473 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d402cb00 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00dbe00640 1 -- 172.21.15.125:0/4221462473 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d402cc60 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00dbe00640 1 -- 172.21.15.125:0/4221462473 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d40343f0 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 -- 172.21.15.125:0/4221462473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc0cf9a0 msgr2=0x7f00dc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc0cf9a0 0x7f00dc0cd840 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d402bf60 comp rx=0 tx=0).stop 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 -- 172.21.15.125:0/4221462473 shutdown_connections 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc0cf9a0 0x7f00dc0cd840 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 --2- 172.21.15.125:0/4221462473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc13df80 0x7f00dc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 -- 172.21.15.125:0/4221462473 >> 172.21.15.125:0/4221462473 conn(0x7f00dc06a800 msgr2=0x7f00dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.840+0000 7f00e4400640 1 -- 172.21.15.125:0/4221462473 shutdown_connections 2024-01-16T15:15:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.841+0000 7f00e4400640 1 -- 172.21.15.125:0/4221462473 wait complete. 2024-01-16T15:15:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.842+0000 7f00e4400640 1 Processor -- start 2024-01-16T15:15:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.842+0000 7f00e4400640 1 -- start start 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc0cf9a0 0x7f00dc0ab410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00dc13df80 0x7f00dc0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc0cbd10 con 0x7f00dc13e360 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc154200 con 0x7f00dc13df80 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.843+0000 7f00e4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc0aa650 con 0x7f00dc0cf9a0 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49086/0 (socket says 172.21.15.125:49086) 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 -- 172.21.15.125:0/1527562949 learned_addr learned my addr 172.21.15.125:0/1527562949 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e1a00640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc0cf9a0 0x7f00dc0ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e1000640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00dc13df80 0x7f00dc0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 -- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc0cf9a0 msgr2=0x7f00dc0ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc0cf9a0 0x7f00dc0ab410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 -- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00dc13df80 msgr2=0x7f00dc0cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00dc13df80 0x7f00dc0cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 -- 172.21.15.125:0/1527562949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4009b30 con 0x7f00dc13e360 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.844+0000 7f00e2400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.845+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4009510 con 0x7f00dc13e360 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.845+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d403e450 con 0x7f00dc13e360 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.845+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d403e800 con 0x7f00dc13e360 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.845+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc0aa930 con 0x7f00dc13e360 2024-01-16T15:15:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.845+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc0aaef0 con 0x7f00dc13e360 2024-01-16T15:15:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.846+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc0ab0e0 con 0x7f00dc13e360 2024-01-16T15:15:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.847+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f00d403e960 con 0x7f00dc13e360 2024-01-16T15:15:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.848+0000 7f00d9600640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 0x7f00b406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.848+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00d40ab810 con 0x7f00dc13e360 2024-01-16T15:15:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.848+0000 7f00e1a00640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 0x7f00b406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.849+0000 7f00e1a00640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 0x7f00b406e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.835 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:10.852+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00d4077de0 con 0x7f00dc13e360 2024-01-16T15:15:11.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.077+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00dc0675d0 con 0x7f00dc13e360 2024-01-16T15:15:11.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.078+0000 7f00d9600640 1 -- 172.21.15.125:0/1527562949 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f00d407b400 con 0x7f00dc13e360 2024-01-16T15:15:11.061 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:11.061 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1181037452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1181037452},{"type":"v1","addr":"172.21.15.137:6833","nonce":1181037452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1991371319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1991371319},{"type":"v1","addr":"172.21.15.137:6835","nonce":1991371319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":9156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/13212755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":13212755},{"type":"v1","addr":"172.21.15.125:6835","nonce":13212755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:06.982766+0000","modified":"2024-01-16T15:15:10.072697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9162":{"gid":9162,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/2099236429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2099236429},{"type":"v1","addr":"172.21.15.125:6837","nonce":2099236429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:15:11.061 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 73 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 msgr2=0x7f00b406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 0x7f00b406e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc008040 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 msgr2=0x7f00dc0ac420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d4004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 shutdown_connections 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f00b406c430 0x7f00b406e8f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00dc13e360 0x7f00dc0ac420 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00dc13df80 0x7f00dc0cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 --2- 172.21.15.125:0/1527562949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00dc0cf9a0 0x7f00dc0ab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.080+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 >> 172.21.15.125:0/1527562949 conn(0x7f00dc06a800 msgr2=0x7f00dc15c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.081+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 shutdown_connections 2024-01-16T15:15:11.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.081+0000 7f00e4400640 1 -- 172.21.15.125:0/1527562949 wait complete. 2024-01-16T15:15:11.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:11.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.307+0000 7f36a2200640 1 Processor -- start 2024-01-16T15:15:11.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.307+0000 7f36a2200640 1 -- start start 2024-01-16T15:15:11.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.308+0000 7f36a2200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f369c143ae0 0x7f369c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.308+0000 7f36a2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c0cf7d0 0x7f369c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f36a2200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c0cfc30 con 0x7f369c15e1e0 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f36a2200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c0b1ee0 con 0x7f369c143ae0 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f36a2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0b2780 con 0x7f369c0cf7d0 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f369c143ae0 0x7f369c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f369c143ae0 0x7f369c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49090/0 (socket says 172.21.15.125:49090) 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 -- 172.21.15.125:0/3489365685 learned_addr learned my addr 172.21.15.125:0/3489365685 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f3692800640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c0cf7d0 0x7f369c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 -- 172.21.15.125:0/3489365685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c0cf7d0 msgr2=0x7f369c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c0cf7d0 0x7f369c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 -- 172.21.15.125:0/3489365685 >> v1:172.21.15.137:6789/0 conn(0x7f369c15e1e0 legacy=0x7f369c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.309+0000 7f369b400640 1 -- 172.21.15.125:0/3489365685 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c15e990 con 0x7f369c143ae0 2024-01-16T15:15:11.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.310+0000 7f369b400640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c143ae0 0x7f369c0cf290 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f3684009870 tx=0x7f368402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988e6e887bbe34fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f369aa00640 1 -- 172.21.15.125:0/3489365685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684031640 con 0x7f369c143ae0 2024-01-16T15:15:11.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f369aa00640 1 -- 172.21.15.125:0/3489365685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36840317a0 con 0x7f369c143ae0 2024-01-16T15:15:11.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f369aa00640 1 -- 172.21.15.125:0/3489365685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684031b30 con 0x7f369c143ae0 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 -- 172.21.15.125:0/3489365685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c143ae0 msgr2=0x7f369c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c143ae0 0x7f369c0cf290 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f3684009870 tx=0x7f368402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 -- 172.21.15.125:0/3489365685 shutdown_connections 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c0cf7d0 0x7f369c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 --2- 172.21.15.125:0/3489365685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c143ae0 0x7f369c0cf290 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.311+0000 7f36a2200640 1 -- 172.21.15.125:0/3489365685 >> 172.21.15.125:0/3489365685 conn(0x7f369c06a800 msgr2=0x7f369c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.312+0000 7f36a2200640 1 -- 172.21.15.125:0/3489365685 shutdown_connections 2024-01-16T15:15:11.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.312+0000 7f36a2200640 1 -- 172.21.15.125:0/3489365685 wait complete. 2024-01-16T15:15:11.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.313+0000 7f36a2200640 1 Processor -- start 2024-01-16T15:15:11.296 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.313+0000 7f36a2200640 1 -- start start 2024-01-16T15:15:11.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c143ae0 0x7f369c165e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f369c15e1e0 0x7f369c0a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c165ac0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0abcb0 con 0x7f369c15e1e0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.314+0000 7f36a2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c133bd0 con 0x7f369c143ae0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49092/0 (socket says 172.21.15.125:49092) 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 -- 172.21.15.125:0/198091013 learned_addr learned my addr 172.21.15.125:0/198091013 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f3692800640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c143ae0 0x7f369c165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 -- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c143ae0 msgr2=0x7f369c165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c143ae0 0x7f369c165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369be00640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f369c15e1e0 0x7f369c0a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 -- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f369c15e1e0 msgr2=0x7f369c0a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f369c15e1e0 0x7f369c0a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.315+0000 7f369b400640 1 -- 172.21.15.125:0/198091013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840094d0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f369b400640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f3684009870 tx=0x7f36840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368403ac60 con 0x7f369c0cf7d0 2024-01-16T15:15:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368403adc0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c164f40 con 0x7f369c0cf7d0 2024-01-16T15:15:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684044630 con 0x7f369c0cf7d0 2024-01-16T15:15:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.316+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c165500 con 0x7f369c0cf7d0 2024-01-16T15:15:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.317+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c0abeb0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.318+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f368402f050 con 0x7f369c0cf7d0 2024-01-16T15:15:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.319+0000 7f3693e00640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 0x7f367406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.319+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36840aa900 con 0x7f369c0cf7d0 2024-01-16T15:15:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.319+0000 7f3692800640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 0x7f367406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.320+0000 7f3692800640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 0x7f367406e8f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f369c06cb60 tx=0x7f368c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.323+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3684076ed0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.517+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f369c0675d0 con 0x7f369c0cf7d0 2024-01-16T15:15:11.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.520+0000 7f3693e00640 1 -- 172.21.15.125:0/198091013 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16575 (secure 0 0 0) 0x7f3684031e60 con 0x7f369c0cf7d0 2024-01-16T15:15:11.504 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:11.504 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:08.572685+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 msgr2=0x7f367406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 0x7f367406e8f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f369c06cb60 tx=0x7f368c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 msgr2=0x7f369c133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f3684009870 tx=0x7f36840046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 shutdown_connections 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f369c15e1e0 0x7f369c0a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f369c143ae0 0x7f369c165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f367406c430 0x7f367406e8f0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 --2- 172.21.15.125:0/198091013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f369c0cf7d0 0x7f369c133690 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 >> 172.21.15.125:0/198091013 conn(0x7f369c06a800 msgr2=0x7f369c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 shutdown_connections 2024-01-16T15:15:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.523+0000 7f36a2200640 1 -- 172.21.15.125:0/198091013 wait complete. 2024-01-16T15:15:11.526 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:15:11.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:11.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.739+0000 7f9369a00640 1 Processor -- start 2024-01-16T15:15:11.723 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.739+0000 7f9369a00640 1 -- start start 2024-01-16T15:15:11.724 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9369a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f936413df80 0x7f936415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9369a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f93640cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9369a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936406c470 con 0x7f936413e360 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9369a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936406c750 con 0x7f936413df80 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9369a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936406ca30 con 0x7f93640cf9a0 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9362000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f936413e360 0x7f93640cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51900/0 (socket says 172.21.15.125:51900) 2024-01-16T15:15:11.725 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.741+0000 7f9362000640 1 -- 172.21.15.125:0/2837752275 learned_addr learned my addr 172.21.15.125:0/2837752275 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:11.726 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9363400640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f93640cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775441916 0 0) 0x7f936406c470 con 0x7f936413e360 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9362a00640 1 --2- 172.21.15.125:0/2837752275 >> v2:172.21.15.137:3300/0 conn(0x7f936413df80 0x7f936415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9344003520 con 0x7f936413e360 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819460123 0 0) 0x7f9344003520 con 0x7f936413e360 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 msgr2=0x7f93640cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f93640cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 >> v2:172.21.15.137:3300/0 conn(0x7f936413df80 msgr2=0x7f936415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 --2- 172.21.15.125:0/2837752275 >> v2:172.21.15.137:3300/0 conn(0x7f936413df80 0x7f936415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936406cd10 con 0x7f936413e360 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f934c001670 con 0x7f936413e360 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 >> v1:172.21.15.125:6789/0 conn(0x7f936413e360 legacy=0x7f93640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9344003970 0x7f9344003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93440047c0 0x7f9344004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f9344005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936406c750 con 0x7f93440047c0 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936406ca30 con 0x7f9344003970 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.742+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934400b980 con 0x7f93640cf9a0 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93440047c0 0x7f9344004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362a00640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9344003970 0x7f9344003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9363400640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f9344005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 -- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 msgr2=0x7f9344005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93640cf9a0 0x7f9344005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 -- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9344003970 msgr2=0x7f9344003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9344003970 0x7f9344003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.743+0000 7f9362000640 1 -- 172.21.15.125:0/2837752275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344003520 con 0x7f93440047c0 2024-01-16T15:15:11.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.744+0000 7f9362000640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93440047c0 0x7f9344004c40 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f934c00c210 tx=0x7f93440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459712236bf4b6be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c005ad0 con 0x7f93440047c0 2024-01-16T15:15:11.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c005c30 con 0x7f93440047c0 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9361600640 1 -- 172.21.15.125:0/2837752275 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c02c8d0 con 0x7f93440047c0 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 -- 172.21.15.125:0/2837752275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93440047c0 msgr2=0x7f9344004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 --2- 172.21.15.125:0/2837752275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93440047c0 0x7f9344004c40 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f934c00c210 tx=0x7f93440057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9363400640 1 -- 172.21.15.125:0/2837752275 reap_dead start 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 -- 172.21.15.125:0/2837752275 shutdown_connections 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 -- 172.21.15.125:0/2837752275 >> 172.21.15.125:0/2837752275 conn(0x7f936406a800 msgr2=0x7f936406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 -- 172.21.15.125:0/2837752275 shutdown_connections 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.745+0000 7f9369a00640 1 -- 172.21.15.125:0/2837752275 wait complete. 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.747+0000 7f9369a00640 1 Processor -- start 2024-01-16T15:15:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.747+0000 7f9369a00640 1 -- start start 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93440047c0 0x7f9364071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 0x7f936413cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936413d310 con 0x7f9364070530 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93640a8830 con 0x7f936414b390 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.748+0000 7f9369a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9364070fb0 con 0x7f93440047c0 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49106/0 (socket says 172.21.15.125:49106) 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 -- 172.21.15.125:0/3756818619 learned_addr learned my addr 172.21.15.125:0/3756818619 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9362000640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 0x7f936413cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 -- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93440047c0 msgr2=0x7f9364071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93440047c0 0x7f9364071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 -- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 msgr2=0x7f936413cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 0x7f936413cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 -- 172.21.15.125:0/3756818619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364071290 con 0x7f9364070530 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9362000640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 0x7f936413cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.749+0000 7f9363400640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f93640cfea0 tx=0x7f935c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.750+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c004500 con 0x7f9364070530 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.750+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c003610 con 0x7f9364070530 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.750+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9364071570 con 0x7f9364070530 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.750+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0039e0 con 0x7f9364070530 2024-01-16T15:15:11.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.750+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93640709e0 con 0x7f9364070530 2024-01-16T15:15:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.751+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93640675d0 con 0x7f9364070530 2024-01-16T15:15:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.752+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f935c003b40 con 0x7f9364070530 2024-01-16T15:15:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.754+0000 7f935aa00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 0x7f933c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.754+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f935c088a80 con 0x7f9364070530 2024-01-16T15:15:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.754+0000 7f9362a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 0x7f933c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.755+0000 7f9362a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 0x7f933c06e8f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f936414b8e0 tx=0x7f9354009750 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.758+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f935c054ff0 con 0x7f9364070530 2024-01-16T15:15:11.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.945+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f936415f9c0 con 0x7f933c06c430 2024-01-16T15:15:11.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.968+0000 7f935aa00640 1 -- 172.21.15.125:0/3756818619 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286888 (secure 0 0 0) 0x7f935400f050 con 0x7f933c06c430 2024-01-16T15:15:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:11.957 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:15:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 msgr2=0x7f933c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 0x7f933c06e8f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f936414b8e0 tx=0x7f9354009750 comp rx=0 tx=0).stop 2024-01-16T15:15:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 msgr2=0x7f93641f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f93640cfea0 tx=0x7f935c011920 comp rx=0 tx=0).stop 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 shutdown_connections 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f933c06c430 0x7f933c06e8f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9364070530 0x7f93641f5f60 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f936414b390 0x7f936413cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 --2- 172.21.15.125:0/3756818619 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f93440047c0 0x7f9364071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 >> 172.21.15.125:0/3756818619 conn(0x7f936406a800 msgr2=0x7f936415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.974+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 shutdown_connections 2024-01-16T15:15:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:11.975+0000 7f9369a00640 1 -- 172.21.15.125:0/3756818619 wait complete. 2024-01-16T15:15:11.976 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2024-01-16T15:15:10.347198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":587136,"kb_used_data":11980,"kb_used_omap":12,"kb_used_meta":575027,"kb_avail":754387584,"statfs":{"total":773094113280,"available":772492886016,"internally_reserved":0,"allocated":12267520,"data_stored":5941369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":588828208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.821207"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667912+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:15:08.667912+0000","last_peered":"2024-01-16T15:15:08.667912+0000","last_clean":"2024-01-16T15:15:08.667912+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:15:08.667912+0000","last_undegraded":"2024-01-16T15:15:08.667912+0000","last_fullsized":"2024-01-16T15:15:08.667912+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:15.148113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596297+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:15:08.596297+0000","last_peered":"2024-01-16T15:15:08.596297+0000","last_clean":"2024-01-16T15:15:08.596297+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:15:08.596297+0000","last_undegraded":"2024-01-16T15:15:08.596297+0000","last_fullsized":"2024-01-16T15:15:08.596297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595647+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:15:08.595647+0000","last_peered":"2024-01-16T15:15:08.595647+0000","last_clean":"2024-01-16T15:15:08.595647+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:15:08.595647+0000","last_undegraded":"2024-01-16T15:15:08.595647+0000","last_fullsized":"2024-01-16T15:15:08.595647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667166+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:15:08.667166+0000","last_peered":"2024-01-16T15:15:08.667166+0000","last_clean":"2024-01-16T15:15:08.667166+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:15:08.667166+0000","last_undegraded":"2024-01-16T15:15:08.667166+0000","last_fullsized":"2024-01-16T15:15:08.667166+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:02:11.876145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900668+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:15:08.900668+0000","last_peered":"2024-01-16T15:15:08.900668+0000","last_clean":"2024-01-16T15:15:08.900668+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:15:08.900668+0000","last_undegraded":"2024-01-16T15:15:08.900668+0000","last_fullsized":"2024-01-16T15:15:08.900668+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:46.608210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899195+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:15:08.899195+0000","last_peered":"2024-01-16T15:15:08.899195+0000","last_clean":"2024-01-16T15:15:08.899195+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:15:08.899195+0000","last_undegraded":"2024-01-16T15:15:08.899195+0000","last_fullsized":"2024-01-16T15:15:08.899195+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:00.714461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900016+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:15:08.900016+0000","last_peered":"2024-01-16T15:15:08.900016+0000","last_clean":"2024-01-16T15:15:08.900016+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:15:08.900016+0000","last_undegraded":"2024-01-16T15:15:08.900016+0000","last_fullsized":"2024-01-16T15:15:08.900016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593293+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:15:08.593293+0000","last_peered":"2024-01-16T15:15:08.593293+0000","last_clean":"2024-01-16T15:15:08.593293+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:15:08.593293+0000","last_undegraded":"2024-01-16T15:15:08.593293+0000","last_fullsized":"2024-01-16T15:15:08.593293+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667929+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:15:08.667929+0000","last_peered":"2024-01-16T15:15:08.667929+0000","last_clean":"2024-01-16T15:15:08.667929+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:15:08.667929+0000","last_undegraded":"2024-01-16T15:15:08.667929+0000","last_fullsized":"2024-01-16T15:15:08.667929+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:25.063859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667765+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:15:08.667765+0000","last_peered":"2024-01-16T15:15:08.667765+0000","last_clean":"2024-01-16T15:15:08.667765+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:15:08.667765+0000","last_undegraded":"2024-01-16T15:15:08.667765+0000","last_fullsized":"2024-01-16T15:15:08.667765+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:45.318654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595739+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:15:08.595739+0000","last_peered":"2024-01-16T15:15:08.595739+0000","last_clean":"2024-01-16T15:15:08.595739+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:15:08.595739+0000","last_undegraded":"2024-01-16T15:15:08.595739+0000","last_fullsized":"2024-01-16T15:15:08.595739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667165+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:15:08.667165+0000","last_peered":"2024-01-16T15:15:08.667165+0000","last_clean":"2024-01-16T15:15:08.667165+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:15:08.667165+0000","last_undegraded":"2024-01-16T15:15:08.667165+0000","last_fullsized":"2024-01-16T15:15:08.667165+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:32.928994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594296+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:15:08.594296+0000","last_peered":"2024-01-16T15:15:08.594296+0000","last_clean":"2024-01-16T15:15:08.594296+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:15:08.594296+0000","last_undegraded":"2024-01-16T15:15:08.594296+0000","last_fullsized":"2024-01-16T15:15:08.594296+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594829+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:15:08.594829+0000","last_peered":"2024-01-16T15:15:08.594829+0000","last_clean":"2024-01-16T15:15:08.594829+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:15:08.594829+0000","last_undegraded":"2024-01-16T15:15:08.594829+0000","last_fullsized":"2024-01-16T15:15:08.594829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899050+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:15:08.899050+0000","last_peered":"2024-01-16T15:15:08.899050+0000","last_clean":"2024-01-16T15:15:08.899050+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:15:08.899050+0000","last_undegraded":"2024-01-16T15:15:08.899050+0000","last_fullsized":"2024-01-16T15:15:08.899050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:28.316281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666498+0000","last_change":"2024-01-16T15:15:07.589974+0000","last_active":"2024-01-16T15:15:08.666498+0000","last_peered":"2024-01-16T15:15:08.666498+0000","last_clean":"2024-01-16T15:15:08.666498+0000","last_became_active":"2024-01-16T15:15:07.584841+0000","last_became_peered":"2024-01-16T15:15:07.584841+0000","last_unstale":"2024-01-16T15:15:08.666498+0000","last_undegraded":"2024-01-16T15:15:08.666498+0000","last_fullsized":"2024-01-16T15:15:08.666498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:34.639802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596489+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:15:08.596489+0000","last_peered":"2024-01-16T15:15:08.596489+0000","last_clean":"2024-01-16T15:15:08.596489+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:15:08.596489+0000","last_undegraded":"2024-01-16T15:15:08.596489+0000","last_fullsized":"2024-01-16T15:15:08.596489+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899772+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:15:08.899772+0000","last_peered":"2024-01-16T15:15:08.899772+0000","last_clean":"2024-01-16T15:15:08.899772+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:15:08.899772+0000","last_undegraded":"2024-01-16T15:15:08.899772+0000","last_fullsized":"2024-01-16T15:15:08.899772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595572+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:15:08.595572+0000","last_peered":"2024-01-16T15:15:08.595572+0000","last_clean":"2024-01-16T15:15:08.595572+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:15:08.595572+0000","last_undegraded":"2024-01-16T15:15:08.595572+0000","last_fullsized":"2024-01-16T15:15:08.595572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595453+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:15:08.595453+0000","last_peered":"2024-01-16T15:15:08.595453+0000","last_clean":"2024-01-16T15:15:08.595453+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:15:08.595453+0000","last_undegraded":"2024-01-16T15:15:08.595453+0000","last_fullsized":"2024-01-16T15:15:08.595453+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595897+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:15:08.595897+0000","last_peered":"2024-01-16T15:15:08.595897+0000","last_clean":"2024-01-16T15:15:08.595897+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:15:08.595897+0000","last_undegraded":"2024-01-16T15:15:08.595897+0000","last_fullsized":"2024-01-16T15:15:08.595897+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591507+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:15:08.591507+0000","last_peered":"2024-01-16T15:15:08.591507+0000","last_clean":"2024-01-16T15:15:08.591507+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:15:08.591507+0000","last_undegraded":"2024-01-16T15:15:08.591507+0000","last_fullsized":"2024-01-16T15:15:08.591507+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594834+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:15:08.594834+0000","last_peered":"2024-01-16T15:15:08.594834+0000","last_clean":"2024-01-16T15:15:08.594834+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:15:08.594834+0000","last_undegraded":"2024-01-16T15:15:08.594834+0000","last_fullsized":"2024-01-16T15:15:08.594834+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591019+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:15:08.591019+0000","last_peered":"2024-01-16T15:15:08.591019+0000","last_clean":"2024-01-16T15:15:08.591019+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:15:08.591019+0000","last_undegraded":"2024-01-16T15:15:08.591019+0000","last_fullsized":"2024-01-16T15:15:08.591019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596616+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:15:08.596616+0000","last_peered":"2024-01-16T15:15:08.596616+0000","last_clean":"2024-01-16T15:15:08.596616+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:15:08.596616+0000","last_undegraded":"2024-01-16T15:15:08.596616+0000","last_fullsized":"2024-01-16T15:15:08.596616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667834+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:15:08.667834+0000","last_peered":"2024-01-16T15:15:08.667834+0000","last_clean":"2024-01-16T15:15:08.667834+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:15:08.667834+0000","last_undegraded":"2024-01-16T15:15:08.667834+0000","last_fullsized":"2024-01-16T15:15:08.667834+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:06.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899902+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:15:08.899902+0000","last_peered":"2024-01-16T15:15:08.899902+0000","last_clean":"2024-01-16T15:15:08.899902+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:15:08.899902+0000","last_undegraded":"2024-01-16T15:15:08.899902+0000","last_fullsized":"2024-01-16T15:15:08.899902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:21:39.656818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592049+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:15:08.592049+0000","last_peered":"2024-01-16T15:15:08.592049+0000","last_clean":"2024-01-16T15:15:08.592049+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:15:08.592049+0000","last_undegraded":"2024-01-16T15:15:08.592049+0000","last_fullsized":"2024-01-16T15:15:08.592049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595053+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:15:08.595053+0000","last_peered":"2024-01-16T15:15:08.595053+0000","last_clean":"2024-01-16T15:15:08.595053+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:15:08.595053+0000","last_undegraded":"2024-01-16T15:15:08.595053+0000","last_fullsized":"2024-01-16T15:15:08.595053+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899461+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:15:08.899461+0000","last_peered":"2024-01-16T15:15:08.899461+0000","last_clean":"2024-01-16T15:15:08.899461+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:15:08.899461+0000","last_undegraded":"2024-01-16T15:15:08.899461+0000","last_fullsized":"2024-01-16T15:15:08.899461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666804+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:15:08.666804+0000","last_peered":"2024-01-16T15:15:08.666804+0000","last_clean":"2024-01-16T15:15:08.666804+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:15:08.666804+0000","last_undegraded":"2024-01-16T15:15:08.666804+0000","last_fullsized":"2024-01-16T15:15:08.666804+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:50.738078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898883+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:15:08.898883+0000","last_peered":"2024-01-16T15:15:08.898883+0000","last_clean":"2024-01-16T15:15:08.898883+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:15:08.898883+0000","last_undegraded":"2024-01-16T15:15:08.898883+0000","last_fullsized":"2024-01-16T15:15:08.898883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:08.692285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899768+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:15:08.899768+0000","last_peered":"2024-01-16T15:15:08.899768+0000","last_clean":"2024-01-16T15:15:08.899768+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:15:08.899768+0000","last_undegraded":"2024-01-16T15:15:08.899768+0000","last_fullsized":"2024-01-16T15:15:08.899768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:12.183950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899701+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:15:08.899701+0000","last_peered":"2024-01-16T15:15:08.899701+0000","last_clean":"2024-01-16T15:15:08.899701+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:15:08.899701+0000","last_undegraded":"2024-01-16T15:15:08.899701+0000","last_fullsized":"2024-01-16T15:15:08.899701+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900332+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:15:08.900332+0000","last_peered":"2024-01-16T15:15:08.900332+0000","last_clean":"2024-01-16T15:15:08.900332+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:15:08.900332+0000","last_undegraded":"2024-01-16T15:15:08.900332+0000","last_fullsized":"2024-01-16T15:15:08.900332+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594174+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:15:08.594174+0000","last_peered":"2024-01-16T15:15:08.594174+0000","last_clean":"2024-01-16T15:15:08.594174+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:15:08.594174+0000","last_undegraded":"2024-01-16T15:15:08.594174+0000","last_fullsized":"2024-01-16T15:15:08.594174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899386+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:15:08.899386+0000","last_peered":"2024-01-16T15:15:08.899386+0000","last_clean":"2024-01-16T15:15:08.899386+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:15:08.899386+0000","last_undegraded":"2024-01-16T15:15:08.899386+0000","last_fullsized":"2024-01-16T15:15:08.899386+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:48.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900859+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:15:08.900859+0000","last_peered":"2024-01-16T15:15:08.900859+0000","last_clean":"2024-01-16T15:15:08.900859+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:15:08.900859+0000","last_undegraded":"2024-01-16T15:15:08.900859+0000","last_fullsized":"2024-01-16T15:15:08.900859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:22.852738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595887+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:15:08.595887+0000","last_peered":"2024-01-16T15:15:08.595887+0000","last_clean":"2024-01-16T15:15:08.595887+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:15:08.595887+0000","last_undegraded":"2024-01-16T15:15:08.595887+0000","last_fullsized":"2024-01-16T15:15:08.595887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900429+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:15:08.900429+0000","last_peered":"2024-01-16T15:15:08.900429+0000","last_clean":"2024-01-16T15:15:08.900429+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:15:08.900429+0000","last_undegraded":"2024-01-16T15:15:08.900429+0000","last_fullsized":"2024-01-16T15:15:08.900429+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:07.677232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666987+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:15:08.666987+0000","last_peered":"2024-01-16T15:15:08.666987+0000","last_clean":"2024-01-16T15:15:08.666987+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:15:08.666987+0000","last_undegraded":"2024-01-16T15:15:08.666987+0000","last_fullsized":"2024-01-16T15:15:08.666987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:08.656641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899104+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:15:08.899104+0000","last_peered":"2024-01-16T15:15:08.899104+0000","last_clean":"2024-01-16T15:15:08.899104+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:15:08.899104+0000","last_undegraded":"2024-01-16T15:15:08.899104+0000","last_fullsized":"2024-01-16T15:15:08.899104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595386+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:15:08.595386+0000","last_peered":"2024-01-16T15:15:08.595386+0000","last_clean":"2024-01-16T15:15:08.595386+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:15:08.595386+0000","last_undegraded":"2024-01-16T15:15:08.595386+0000","last_fullsized":"2024-01-16T15:15:08.595386+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591606+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:15:08.591606+0000","last_peered":"2024-01-16T15:15:08.591606+0000","last_clean":"2024-01-16T15:15:08.591606+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:15:08.591606+0000","last_undegraded":"2024-01-16T15:15:08.591606+0000","last_fullsized":"2024-01-16T15:15:08.591606+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666549+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:15:08.666549+0000","last_peered":"2024-01-16T15:15:08.666549+0000","last_clean":"2024-01-16T15:15:08.666549+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:15:08.666549+0000","last_undegraded":"2024-01-16T15:15:08.666549+0000","last_fullsized":"2024-01-16T15:15:08.666549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:19.703629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594989+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:15:08.594989+0000","last_peered":"2024-01-16T15:15:08.594989+0000","last_clean":"2024-01-16T15:15:08.594989+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:15:08.594989+0000","last_undegraded":"2024-01-16T15:15:08.594989+0000","last_fullsized":"2024-01-16T15:15:08.594989+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898691+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:15:08.898691+0000","last_peered":"2024-01-16T15:15:08.898691+0000","last_clean":"2024-01-16T15:15:08.898691+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:15:08.898691+0000","last_undegraded":"2024-01-16T15:15:08.898691+0000","last_fullsized":"2024-01-16T15:15:08.898691+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666658+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:15:08.666658+0000","last_peered":"2024-01-16T15:15:08.666658+0000","last_clean":"2024-01-16T15:15:08.666658+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:15:08.666658+0000","last_undegraded":"2024-01-16T15:15:08.666658+0000","last_fullsized":"2024-01-16T15:15:08.666658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:43.538921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899448+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:15:08.899448+0000","last_peered":"2024-01-16T15:15:08.899448+0000","last_clean":"2024-01-16T15:15:08.899448+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:15:08.899448+0000","last_undegraded":"2024-01-16T15:15:08.899448+0000","last_fullsized":"2024-01-16T15:15:08.899448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900968+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:15:08.900968+0000","last_peered":"2024-01-16T15:15:08.900968+0000","last_clean":"2024-01-16T15:15:08.900968+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:15:08.900968+0000","last_undegraded":"2024-01-16T15:15:08.900968+0000","last_fullsized":"2024-01-16T15:15:08.900968+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:27.301228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900506+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:15:08.900506+0000","last_peered":"2024-01-16T15:15:08.900506+0000","last_clean":"2024-01-16T15:15:08.900506+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:15:08.900506+0000","last_undegraded":"2024-01-16T15:15:08.900506+0000","last_fullsized":"2024-01-16T15:15:08.900506+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592226+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:15:08.592226+0000","last_peered":"2024-01-16T15:15:08.592226+0000","last_clean":"2024-01-16T15:15:08.592226+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:15:08.592226+0000","last_undegraded":"2024-01-16T15:15:08.592226+0000","last_fullsized":"2024-01-16T15:15:08.592226+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592152+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:15:08.592152+0000","last_peered":"2024-01-16T15:15:08.592152+0000","last_clean":"2024-01-16T15:15:08.592152+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:15:08.592152+0000","last_undegraded":"2024-01-16T15:15:08.592152+0000","last_fullsized":"2024-01-16T15:15:08.592152+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667269+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:15:08.667269+0000","last_peered":"2024-01-16T15:15:08.667269+0000","last_clean":"2024-01-16T15:15:08.667269+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:15:08.667269+0000","last_undegraded":"2024-01-16T15:15:08.667269+0000","last_fullsized":"2024-01-16T15:15:08.667269+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899725+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:15:08.899725+0000","last_peered":"2024-01-16T15:15:08.899725+0000","last_clean":"2024-01-16T15:15:08.899725+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:15:08.899725+0000","last_undegraded":"2024-01-16T15:15:08.899725+0000","last_fullsized":"2024-01-16T15:15:08.899725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667336+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:15:08.667336+0000","last_peered":"2024-01-16T15:15:08.667336+0000","last_clean":"2024-01-16T15:15:08.667336+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:15:08.667336+0000","last_undegraded":"2024-01-16T15:15:08.667336+0000","last_fullsized":"2024-01-16T15:15:08.667336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:32.472941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591569+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:15:08.591569+0000","last_peered":"2024-01-16T15:15:08.591569+0000","last_clean":"2024-01-16T15:15:08.591569+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:15:08.591569+0000","last_undegraded":"2024-01-16T15:15:08.591569+0000","last_fullsized":"2024-01-16T15:15:08.591569+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899098+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:15:08.899098+0000","last_peered":"2024-01-16T15:15:08.899098+0000","last_clean":"2024-01-16T15:15:08.899098+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:15:08.899098+0000","last_undegraded":"2024-01-16T15:15:08.899098+0000","last_fullsized":"2024-01-16T15:15:08.899098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:40.359437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591753+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:15:08.591753+0000","last_peered":"2024-01-16T15:15:08.591753+0000","last_clean":"2024-01-16T15:15:08.591753+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:15:08.591753+0000","last_undegraded":"2024-01-16T15:15:08.591753+0000","last_fullsized":"2024-01-16T15:15:08.591753+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898761+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:15:08.898761+0000","last_peered":"2024-01-16T15:15:08.898761+0000","last_clean":"2024-01-16T15:15:08.898761+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:15:08.898761+0000","last_undegraded":"2024-01-16T15:15:08.898761+0000","last_fullsized":"2024-01-16T15:15:08.898761+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591171+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:15:08.591171+0000","last_peered":"2024-01-16T15:15:08.591171+0000","last_clean":"2024-01-16T15:15:08.591171+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:15:08.591171+0000","last_undegraded":"2024-01-16T15:15:08.591171+0000","last_fullsized":"2024-01-16T15:15:08.591171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899740+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:15:08.899740+0000","last_peered":"2024-01-16T15:15:08.899740+0000","last_clean":"2024-01-16T15:15:08.899740+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:15:08.899740+0000","last_undegraded":"2024-01-16T15:15:08.899740+0000","last_fullsized":"2024-01-16T15:15:08.899740+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:11.168897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.901277+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:15:08.901277+0000","last_peered":"2024-01-16T15:15:08.901277+0000","last_clean":"2024-01-16T15:15:08.901277+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:15:08.901277+0000","last_undegraded":"2024-01-16T15:15:08.901277+0000","last_fullsized":"2024-01-16T15:15:08.901277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:21.652051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667982+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:15:08.667982+0000","last_peered":"2024-01-16T15:15:08.667982+0000","last_clean":"2024-01-16T15:15:08.667982+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:15:08.667982+0000","last_undegraded":"2024-01-16T15:15:08.667982+0000","last_fullsized":"2024-01-16T15:15:08.667982+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:53.963781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899763+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:15:08.899763+0000","last_peered":"2024-01-16T15:15:08.899763+0000","last_clean":"2024-01-16T15:15:08.899763+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:15:08.899763+0000","last_undegraded":"2024-01-16T15:15:08.899763+0000","last_fullsized":"2024-01-16T15:15:08.899763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:45.884405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.901110+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:15:08.901110+0000","last_peered":"2024-01-16T15:15:08.901110+0000","last_clean":"2024-01-16T15:15:08.901110+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:15:08.901110+0000","last_undegraded":"2024-01-16T15:15:08.901110+0000","last_fullsized":"2024-01-16T15:15:08.901110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:59.699408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594406+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:15:08.594406+0000","last_peered":"2024-01-16T15:15:08.594406+0000","last_clean":"2024-01-16T15:15:08.594406+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:15:08.594406+0000","last_undegraded":"2024-01-16T15:15:08.594406+0000","last_fullsized":"2024-01-16T15:15:08.594406+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899491+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:15:08.899491+0000","last_peered":"2024-01-16T15:15:08.899491+0000","last_clean":"2024-01-16T15:15:08.899491+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:15:08.899491+0000","last_undegraded":"2024-01-16T15:15:08.899491+0000","last_fullsized":"2024-01-16T15:15:08.899491+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595955+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:15:08.595955+0000","last_peered":"2024-01-16T15:15:08.595955+0000","last_clean":"2024-01-16T15:15:08.595955+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:15:08.595955+0000","last_undegraded":"2024-01-16T15:15:08.595955+0000","last_fullsized":"2024-01-16T15:15:08.595955+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899087+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:15:08.899087+0000","last_peered":"2024-01-16T15:15:08.899087+0000","last_clean":"2024-01-16T15:15:08.899087+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:15:08.899087+0000","last_undegraded":"2024-01-16T15:15:08.899087+0000","last_fullsized":"2024-01-16T15:15:08.899087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594902+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:15:08.594902+0000","last_peered":"2024-01-16T15:15:08.594902+0000","last_clean":"2024-01-16T15:15:08.594902+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:15:08.594902+0000","last_undegraded":"2024-01-16T15:15:08.594902+0000","last_fullsized":"2024-01-16T15:15:08.594902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899359+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:15:08.899359+0000","last_peered":"2024-01-16T15:15:08.899359+0000","last_clean":"2024-01-16T15:15:08.899359+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:15:08.899359+0000","last_undegraded":"2024-01-16T15:15:08.899359+0000","last_fullsized":"2024-01-16T15:15:08.899359+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666900+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:15:08.666900+0000","last_peered":"2024-01-16T15:15:08.666900+0000","last_clean":"2024-01-16T15:15:08.666900+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:15:08.666900+0000","last_undegraded":"2024-01-16T15:15:08.666900+0000","last_fullsized":"2024-01-16T15:15:08.666900+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.557222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899591+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:15:08.899591+0000","last_peered":"2024-01-16T15:15:08.899591+0000","last_clean":"2024-01-16T15:15:08.899591+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:15:08.899591+0000","last_undegraded":"2024-01-16T15:15:08.899591+0000","last_fullsized":"2024-01-16T15:15:08.899591+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593311+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:15:08.593311+0000","last_peered":"2024-01-16T15:15:08.593311+0000","last_clean":"2024-01-16T15:15:08.593311+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:15:08.593311+0000","last_undegraded":"2024-01-16T15:15:08.593311+0000","last_fullsized":"2024-01-16T15:15:08.593311+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899170+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:15:08.899170+0000","last_peered":"2024-01-16T15:15:08.899170+0000","last_clean":"2024-01-16T15:15:08.899170+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:15:08.899170+0000","last_undegraded":"2024-01-16T15:15:08.899170+0000","last_fullsized":"2024-01-16T15:15:08.899170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591490+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:15:08.591490+0000","last_peered":"2024-01-16T15:15:08.591490+0000","last_clean":"2024-01-16T15:15:08.591490+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:15:08.591490+0000","last_undegraded":"2024-01-16T15:15:08.591490+0000","last_fullsized":"2024-01-16T15:15:08.591490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666665+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:15:08.666665+0000","last_peered":"2024-01-16T15:15:08.666665+0000","last_clean":"2024-01-16T15:15:08.666665+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:15:08.666665+0000","last_undegraded":"2024-01-16T15:15:08.666665+0000","last_fullsized":"2024-01-16T15:15:08.666665+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:02.498301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900030+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:15:08.900030+0000","last_peered":"2024-01-16T15:15:08.900030+0000","last_clean":"2024-01-16T15:15:08.900030+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:15:08.900030+0000","last_undegraded":"2024-01-16T15:15:08.900030+0000","last_fullsized":"2024-01-16T15:15:08.900030+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592106+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:15:08.592106+0000","last_peered":"2024-01-16T15:15:08.592106+0000","last_clean":"2024-01-16T15:15:08.592106+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:15:08.592106+0000","last_undegraded":"2024-01-16T15:15:08.592106+0000","last_fullsized":"2024-01-16T15:15:08.592106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667983+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:15:08.667983+0000","last_peered":"2024-01-16T15:15:08.667983+0000","last_clean":"2024-01-16T15:15:08.667983+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:15:08.667983+0000","last_undegraded":"2024-01-16T15:15:08.667983+0000","last_fullsized":"2024-01-16T15:15:08.667983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:40.476189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900018+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:15:08.900018+0000","last_peered":"2024-01-16T15:15:08.900018+0000","last_clean":"2024-01-16T15:15:08.900018+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:15:08.900018+0000","last_undegraded":"2024-01-16T15:15:08.900018+0000","last_fullsized":"2024-01-16T15:15:08.900018+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667295+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:15:08.667295+0000","last_peered":"2024-01-16T15:15:08.667295+0000","last_clean":"2024-01-16T15:15:08.667295+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:15:08.667295+0000","last_undegraded":"2024-01-16T15:15:08.667295+0000","last_fullsized":"2024-01-16T15:15:08.667295+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:29.732315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667263+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:15:08.667263+0000","last_peered":"2024-01-16T15:15:08.667263+0000","last_clean":"2024-01-16T15:15:08.667263+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:15:08.667263+0000","last_undegraded":"2024-01-16T15:15:08.667263+0000","last_fullsized":"2024-01-16T15:15:08.667263+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:41.055270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899061+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:15:08.899061+0000","last_peered":"2024-01-16T15:15:08.899061+0000","last_clean":"2024-01-16T15:15:08.899061+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:15:08.899061+0000","last_undegraded":"2024-01-16T15:15:08.899061+0000","last_fullsized":"2024-01-16T15:15:08.899061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592262+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:15:08.592262+0000","last_peered":"2024-01-16T15:15:08.592262+0000","last_clean":"2024-01-16T15:15:08.592262+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:15:08.592262+0000","last_undegraded":"2024-01-16T15:15:08.592262+0000","last_fullsized":"2024-01-16T15:15:08.592262+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594866+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:15:08.594866+0000","last_peered":"2024-01-16T15:15:08.594866+0000","last_clean":"2024-01-16T15:15:08.594866+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:15:08.594866+0000","last_undegraded":"2024-01-16T15:15:08.594866+0000","last_fullsized":"2024-01-16T15:15:08.594866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899345+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:15:08.899345+0000","last_peered":"2024-01-16T15:15:08.899345+0000","last_clean":"2024-01-16T15:15:08.899345+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:15:08.899345+0000","last_undegraded":"2024-01-16T15:15:08.899345+0000","last_fullsized":"2024-01-16T15:15:08.899345+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595719+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:15:08.595719+0000","last_peered":"2024-01-16T15:15:08.595719+0000","last_clean":"2024-01-16T15:15:08.595719+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:15:08.595719+0000","last_undegraded":"2024-01-16T15:15:08.595719+0000","last_fullsized":"2024-01-16T15:15:08.595719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595117+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:15:08.595117+0000","last_peered":"2024-01-16T15:15:08.595117+0000","last_clean":"2024-01-16T15:15:08.595117+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:15:08.595117+0000","last_undegraded":"2024-01-16T15:15:08.595117+0000","last_fullsized":"2024-01-16T15:15:08.595117+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593661+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:15:08.593661+0000","last_peered":"2024-01-16T15:15:08.593661+0000","last_clean":"2024-01-16T15:15:08.593661+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:15:08.593661+0000","last_undegraded":"2024-01-16T15:15:08.593661+0000","last_fullsized":"2024-01-16T15:15:08.593661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900147+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:15:08.900147+0000","last_peered":"2024-01-16T15:15:08.900147+0000","last_clean":"2024-01-16T15:15:08.900147+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:15:08.900147+0000","last_undegraded":"2024-01-16T15:15:08.900147+0000","last_fullsized":"2024-01-16T15:15:08.900147+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666857+0000","last_change":"2024-01-16T15:15:07.592852+0000","last_active":"2024-01-16T15:15:08.666857+0000","last_peered":"2024-01-16T15:15:08.666857+0000","last_clean":"2024-01-16T15:15:08.666857+0000","last_became_active":"2024-01-16T15:15:07.592163+0000","last_became_peered":"2024-01-16T15:15:07.592163+0000","last_unstale":"2024-01-16T15:15:08.666857+0000","last_undegraded":"2024-01-16T15:15:08.666857+0000","last_fullsized":"2024-01-16T15:15:08.666857+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:15.876527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666780+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:15:08.666780+0000","last_peered":"2024-01-16T15:15:08.666780+0000","last_clean":"2024-01-16T15:15:08.666780+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:15:08.666780+0000","last_undegraded":"2024-01-16T15:15:08.666780+0000","last_fullsized":"2024-01-16T15:15:08.666780+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:10.684617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591079+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:15:08.591079+0000","last_peered":"2024-01-16T15:15:08.591079+0000","last_clean":"2024-01-16T15:15:08.591079+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:15:08.591079+0000","last_undegraded":"2024-01-16T15:15:08.591079+0000","last_fullsized":"2024-01-16T15:15:08.591079+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899955+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:15:08.899955+0000","last_peered":"2024-01-16T15:15:08.899955+0000","last_clean":"2024-01-16T15:15:08.899955+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:15:08.899955+0000","last_undegraded":"2024-01-16T15:15:08.899955+0000","last_fullsized":"2024-01-16T15:15:08.899955+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":265,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:15:07.562660+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:15:07.562660+0000","last_peered":"2024-01-16T15:15:07.562660+0000","last_clean":"2024-01-16T15:15:07.562660+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:15:07.562660+0000","last_undegraded":"2024-01-16T15:15:07.562660+0000","last_fullsized":"2024-01-16T15:15:07.562660+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575075,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64976,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306864,"statfs":{"total":96636764160,"available":96570228736,"internally_reserved":0,"allocated":1970176,"data_stored":1180863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.111,"15min":1.4239999999999999},"min":{"1min":0.82999999999999996,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.5489999999999999,"5min":3.3969999999999998,"15min":45.296999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.226,"15min":1.347},"min":{"1min":0.80100000000000005,"5min":0.627,"15min":0.497},"max":{"1min":1.7410000000000001,"5min":3.4990000000000001,"15min":20.126999999999999},"last":1.2589999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2509999999999999,"15min":1.444},"min":{"1min":0.94699999999999995,"5min":0.69699999999999995,"15min":0.499},"max":{"1min":2.2210000000000001,"5min":3.431,"15min":42.723999999999997},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.413,"15min":1.6299999999999999},"min":{"1min":0.98299999999999998,"5min":0.73099999999999998,"15min":0.49099999999999999},"max":{"1min":2.4620000000000002,"5min":3.6539999999999999,"15min":43.572000000000003},"last":1.617}]},{"osd":2,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2050000000000001,"15min":1.4550000000000001},"min":{"1min":0.91000000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.1640000000000001,"5min":2.5910000000000002,"15min":35.308},"last":1.532},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.242,"15min":1.47},"min":{"1min":0.83899999999999997,"5min":0.70099999999999996,"15min":0.503},"max":{"1min":1.8260000000000001,"5min":2.2080000000000002,"15min":23.594000000000001},"last":1.153}]},{"osd":3,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.427,"15min":1.671},"min":{"1min":0.91900000000000004,"5min":0.73899999999999999,"15min":0.51300000000000001},"max":{"1min":2.2799999999999998,"5min":12.331,"15min":53.033999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.339,"15min":1.5940000000000001},"min":{"1min":0.94699999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0230000000000001,"5min":2.1389999999999998,"15min":31.305},"last":1.369}]},{"osd":4,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.381,"15min":1.343},"min":{"1min":0.96599999999999997,"5min":0.77100000000000002,"15min":0.39600000000000002},"max":{"1min":2.226,"5min":3.5710000000000002,"15min":5.1870000000000003},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3140000000000001,"15min":1.514},"min":{"1min":0.89700000000000002,"5min":0.58299999999999996,"15min":0.41099999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":36.374000000000002},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4099999999999999,"15min":1.466},"min":{"1min":0.77700000000000002,"5min":0.60299999999999998,"15min":0.45700000000000002},"max":{"1min":1.9379999999999999,"5min":2.5369999999999999,"15min":9.9399999999999995},"last":1.466},{"interface":"front","average":{"1min":1.583,"5min":1.589,"15min":1.5349999999999999},"min":{"1min":1.0449999999999999,"5min":0.76400000000000001,"15min":0.53500000000000003},"max":{"1min":2.3860000000000001,"5min":4.7400000000000002,"15min":7.0170000000000003},"last":1.498}]},{"osd":6,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.506,"15min":1.5629999999999999},"min":{"1min":1.306,"5min":0.73199999999999998,"15min":0.53700000000000003},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":15.052},"last":1.659},{"interface":"front","average":{"1min":1.599,"5min":1.5720000000000001,"15min":1.6060000000000001},"min":{"1min":0.89700000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1280000000000001,"5min":2.782,"15min":8.4849999999999994},"last":1.575}]}]},{"osd":6,"up_from":13,"seq":55834575076,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81048,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290792,"statfs":{"total":96636764160,"available":96553771008,"internally_reserved":0,"allocated":1384448,"data_stored":596249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81590726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.242,"15min":1.677},"min":{"1min":0.86399999999999999,"5min":0.626,"15min":0.61899999999999999},"max":{"1min":1.637,"5min":1.9359999999999999,"15min":71.870999999999995},"last":1.399},{"interface":"front","average":{"1min":1.302,"5min":1.3140000000000001,"15min":1.754},"min":{"1min":0.82899999999999996,"5min":0.64600000000000002,"15min":0.501},"max":{"1min":1.79,"5min":2.2000000000000002,"15min":71.846999999999994},"last":1.6419999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1839999999999999,"15min":1.3460000000000001},"min":{"1min":0.88400000000000001,"5min":0.67700000000000005,"15min":0.63900000000000001},"max":{"1min":1.6779999999999999,"5min":3.4049999999999998,"15min":15.798},"last":1.339},{"interface":"front","average":{"1min":1.268,"5min":1.21,"15min":1.4550000000000001},"min":{"1min":0.86599999999999999,"5min":0.70999999999999996,"15min":0.626},"max":{"1min":1.5680000000000001,"5min":1.8979999999999999,"15min":24.556999999999999},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3080000000000001,"15min":1.4610000000000001},"min":{"1min":0.96799999999999997,"5min":0.84199999999999997,"15min":0.40000000000000002},"max":{"1min":1.8819999999999999,"5min":2.3029999999999999,"15min":14.981},"last":1.76},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.373,"15min":1.5369999999999999},"min":{"1min":0.99399999999999999,"5min":0.873,"15min":0.57299999999999995},"max":{"1min":1.7490000000000001,"5min":5.1909999999999998,"15min":14.175000000000001},"last":1.679}]},{"osd":3,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.339,"15min":1.4750000000000001},"min":{"1min":0.876,"5min":0.82899999999999996,"15min":0.51100000000000001},"max":{"1min":1.6319999999999999,"5min":2.1789999999999998,"15min":14.827},"last":1.454},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.401,"15min":1.583},"min":{"1min":0.94299999999999995,"5min":0.753,"15min":0.499},"max":{"1min":1.776,"5min":2.032,"15min":21.890000000000001},"last":1.5069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.325,"15min":1.4199999999999999},"min":{"1min":0.73499999999999999,"5min":0.627,"15min":0.45000000000000001},"max":{"1min":1.661,"5min":2.1259999999999999,"15min":10.901},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.391,"15min":1.4239999999999999},"min":{"1min":0.68000000000000005,"5min":0.60299999999999998,"15min":0.48899999999999999},"max":{"1min":2.0859999999999999,"5min":2.1560000000000001,"15min":6.9000000000000004},"last":1.8720000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3069999999999999,"15min":1.3799999999999999},"min":{"1min":0.78800000000000003,"5min":0.55300000000000005,"15min":0.44600000000000001},"max":{"1min":1.752,"5min":2.1669999999999998,"15min":10.736000000000001},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.512,"15min":1.5920000000000001},"min":{"1min":0.80600000000000005,"5min":0.77400000000000002,"15min":0.498},"max":{"1min":1.9490000000000001,"5min":2.2130000000000001,"15min":9.2200000000000006},"last":1.7}]},{"osd":7,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.7250000000000001,"15min":1.6699999999999999},"min":{"1min":0.83699999999999997,"5min":0.64700000000000002,"15min":0.53200000000000003},"max":{"1min":1.988,"5min":24.675000000000001,"15min":24.675000000000001},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.516,"15min":1.6799999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.54600000000000004},"max":{"1min":1.827,"5min":5.2469999999999999,"15min":14.494},"last":1.7430000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76892,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294948,"statfs":{"total":96636764160,"available":96558026752,"internally_reserved":0,"allocated":1388544,"data_stored":596783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0960000000000001,"15min":1.5209999999999999},"min":{"1min":0.77300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.667,"5min":2.0830000000000002,"15min":25.634},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.2390000000000001,"15min":1.51},"min":{"1min":0.90100000000000002,"5min":0.49299999999999999,"15min":0.48299999999999998},"max":{"1min":1.6830000000000001,"5min":8.7880000000000003,"15min":34.959000000000003},"last":1.4590000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2170000000000001,"15min":1.4019999999999999},"min":{"1min":0.85799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.6910000000000001,"5min":1.8460000000000001,"15min":26.378},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.292,"5min":1.254,"15min":1.3520000000000001},"min":{"1min":1.022,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.542,"5min":5.3940000000000001,"15min":13.332000000000001},"last":1.012}]},{"osd":2,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.337,"15min":1.427},"min":{"1min":0.98499999999999999,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":1.502,"5min":5.2110000000000003,"15min":25.495000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.373,"15min":1.4970000000000001},"min":{"1min":1.0629999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.794,"5min":5.3070000000000004,"15min":17.030000000000001},"last":1.1739999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.343,"15min":1.6619999999999999},"min":{"1min":0.97199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.536,"5min":1.994,"15min":25.82},"last":1.075},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.349,"15min":1.681},"min":{"1min":1.03,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.698,"5min":1.8959999999999999,"15min":22.963999999999999},"last":1.2889999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.214,"15min":1.2789999999999999},"min":{"1min":0.86499999999999999,"5min":0.46700000000000003,"15min":0.378},"max":{"1min":1.7050000000000001,"5min":1.964,"15min":8.6050000000000004},"last":1.369},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.4870000000000001,"15min":1.5920000000000001},"min":{"1min":1.093,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":1.76,"5min":2.1930000000000001,"15min":25.724},"last":1.3400000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.601},"min":{"1min":0.87,"5min":0.46800000000000003,"15min":0.40100000000000002},"max":{"1min":1.7689999999999999,"5min":1.9750000000000001,"15min":42.542999999999999},"last":1.518},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2929999999999999,"15min":1.3799999999999999},"min":{"1min":0.996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.6120000000000001,"5min":2.1230000000000002,"15min":10.015000000000001},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.47,"15min":1.6910000000000001},"min":{"1min":1.1719999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7490000000000001,"5min":2.141,"15min":26.484000000000002},"last":1.423},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.494,"15min":1.6799999999999999},"min":{"1min":1.1519999999999999,"5min":0.60699999999999998,"15min":0.46999999999999997},"max":{"1min":1.8320000000000001,"5min":3.2509999999999999,"15min":21.106000000000002},"last":1.575}]}]},{"osd":4,"up_from":13,"seq":55834575075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81032,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290808,"statfs":{"total":96636764160,"available":96553787392,"internally_reserved":0,"allocated":1368064,"data_stored":589879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4710000000000001,"15min":1.4059999999999999},"min":{"1min":0.84199999999999997,"5min":0.57899999999999996,"15min":0.44400000000000001},"max":{"1min":2.0640000000000001,"5min":17.315999999999999,"15min":17.315999999999999},"last":1.966},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5,"15min":1.4359999999999999},"min":{"1min":0.93100000000000005,"5min":0.79700000000000004,"15min":0.52800000000000002},"max":{"1min":1.9390000000000001,"5min":17.279,"15min":17.279},"last":1.867}]},{"osd":1,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2789999999999999,"15min":1.476},"min":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.498},"max":{"1min":1.669,"5min":6.1849999999999996,"15min":39.823999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.1899999999999999,"15min":1.3899999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.48399999999999999},"max":{"1min":1.8160000000000001,"5min":4.2809999999999997,"15min":50.582000000000001},"last":1.534}]},{"osd":2,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.167,"15min":1.3029999999999999},"min":{"1min":0.87,"5min":0.73299999999999998,"15min":0.39500000000000002},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":10.984},"last":0.877},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.83799999999999997,"5min":0.81999999999999995,"15min":0.53500000000000003},"max":{"1min":1.8169999999999999,"5min":1.8360000000000001,"15min":9.1509999999999998},"last":1.0129999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.387,"15min":1.3919999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.54000000000000004},"max":{"1min":1.639,"5min":9.8350000000000009,"15min":11.971},"last":1.222},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.5840000000000001},"min":{"1min":1.0269999999999999,"5min":0.75,"15min":0.57899999999999996},"max":{"1min":1.657,"5min":2.0550000000000002,"15min":27.626999999999999},"last":1.0740000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.286,"15min":1.353},"min":{"1min":0.97199999999999998,"5min":0.81200000000000006,"15min":0.40000000000000002},"max":{"1min":1.8839999999999999,"5min":1.992,"15min":8.2599999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.393,"15min":1.4590000000000001},"min":{"1min":0.998,"5min":0.64100000000000001,"15min":0.46200000000000002},"max":{"1min":1.742,"5min":4.8520000000000003,"15min":17.140999999999998},"last":1.74}]},{"osd":6,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.329,"15min":1.4359999999999999},"min":{"1min":0.91500000000000004,"5min":0.74299999999999999,"15min":0.45300000000000001},"max":{"1min":1.758,"5min":2.0139999999999998,"15min":7.4509999999999996},"last":1.29},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4610000000000001,"15min":1.5980000000000001},"min":{"1min":1.161,"5min":0.88500000000000001,"15min":0.44900000000000001},"max":{"1min":1.986,"5min":2.093,"15min":25.643000000000001},"last":1.47}]},{"osd":7,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4790000000000001,"15min":1.554},"min":{"1min":1.1830000000000001,"5min":0.86299999999999999,"15min":0.53300000000000003},"max":{"1min":1.8680000000000001,"5min":4.9539999999999997,"15min":24.992999999999999},"last":1.421},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5129999999999999,"15min":1.6990000000000001},"min":{"1min":1.121,"5min":0.92900000000000005,"15min":0.50800000000000001},"max":{"1min":1.9119999999999999,"5min":2.0230000000000001,"15min":19.001000000000001},"last":1.629}]}]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68580,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303260,"statfs":{"total":96636764160,"available":96566538240,"internally_reserved":0,"allocated":1396736,"data_stored":597861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.91900000000000004,"15min":1.161},"min":{"1min":0.60099999999999998,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":1.115,"5min":1.925,"15min":38.423999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.98599999999999999,"15min":1.1899999999999999},"min":{"1min":0.77000000000000002,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":1.3420000000000001,"5min":2.9729999999999999,"15min":20.012},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0069999999999999,"15min":1.629},"min":{"1min":0.70799999999999996,"5min":0.44900000000000001,"15min":0.39300000000000002},"max":{"1min":1.302,"5min":3.1150000000000002,"15min":40.695999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.071,"15min":1.212},"min":{"1min":0.72999999999999998,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":1.5029999999999999,"5min":2.25,"15min":9.9689999999999994},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.141,"15min":1.5529999999999999},"min":{"1min":0.81200000000000006,"5min":0.46100000000000002,"15min":0.436},"max":{"1min":1.794,"5min":3.0670000000000002,"15min":59.009},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.216,"15min":1.6779999999999999},"min":{"1min":0.97999999999999998,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.4510000000000001,"5min":3.4279999999999999,"15min":29.773},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3240000000000001,"15min":1.466},"min":{"1min":1.036,"5min":0.71799999999999997,"15min":0.59899999999999998},"max":{"1min":1.627,"5min":3.5049999999999999,"15min":21.736000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3919999999999999,"15min":1.9199999999999999},"min":{"1min":1.073,"5min":0.64000000000000001,"15min":0.622},"max":{"1min":1.71,"5min":3.8250000000000002,"15min":29.524000000000001},"last":0.94499999999999995}]},{"osd":5,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.341,"15min":1.49},"min":{"1min":1.133,"5min":0.70999999999999996,"15min":0.54800000000000004},"max":{"1min":2.0139999999999998,"5min":2.3540000000000001,"15min":9.8249999999999993},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.403,"15min":1.581},"min":{"1min":1.083,"5min":0.71299999999999997,"15min":0.60799999999999998},"max":{"1min":1.7589999999999999,"5min":2.907,"15min":11.255000000000001},"last":1.2689999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4850000000000001,"15min":1.603},"min":{"1min":1.0509999999999999,"5min":0.752,"15min":0.65900000000000003},"max":{"1min":1.7789999999999999,"5min":6.4480000000000004,"15min":20.459},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.538,"15min":2.028},"min":{"1min":1.1870000000000001,"5min":0.81299999999999994,"15min":0.49399999999999999},"max":{"1min":1.857,"5min":3.9039999999999999,"15min":29.475000000000001},"last":1.042}]},{"osd":7,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4810000000000001,"15min":1.637},"min":{"1min":1.097,"5min":0.73099999999999998,"15min":0.55300000000000005},"max":{"1min":1.853,"5min":3.7850000000000001,"15min":20.109999999999999},"last":1.008},{"interface":"front","average":{"1min":1.46,"5min":1.5329999999999999,"15min":2.0270000000000001},"min":{"1min":1.1719999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.9350000000000001,"5min":3.9590000000000001,"15min":29.396999999999998},"last":0.99199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575075,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1380352,"data_stored":596179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88500000000000001,"15min":1.109},"min":{"1min":0.68999999999999995,"5min":0.314,"15min":0.314},"max":{"1min":1.482,"5min":1.744,"15min":25.335999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66100000000000003,"5min":0.42299999999999999,"15min":0.27800000000000002},"max":{"1min":1.4299999999999999,"5min":2.4449999999999998,"15min":25.199000000000002},"last":1.157}]},{"osd":1,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.018,"15min":1.1919999999999999},"min":{"1min":0.80000000000000004,"5min":0.53500000000000003,"15min":0.36799999999999999},"max":{"1min":1.55,"5min":1.7989999999999999,"15min":16.776},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1419999999999999,"15min":1.4179999999999999},"min":{"1min":0.82999999999999996,"5min":0.46500000000000002,"15min":0.318},"max":{"1min":1.478,"5min":4.734,"15min":25.186},"last":1.0489999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0640000000000001,"15min":1.246},"min":{"1min":0.83399999999999996,"5min":0.52600000000000002,"15min":0.33900000000000002},"max":{"1min":1.528,"5min":1.7430000000000001,"15min":37.470999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.218,"15min":1.329},"min":{"1min":0.96299999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.494,"5min":2.3220000000000001,"15min":28.045999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.496,"15min":1.764},"min":{"1min":1.165,"5min":0.65200000000000002,"15min":0.47499999999999998},"max":{"1min":1.615,"5min":16.681999999999999,"15min":25.062000000000001},"last":1.28},{"interface":"front","average":{"1min":1.367,"5min":1.3120000000000001,"15min":1.611},"min":{"1min":1.018,"5min":0.746,"15min":0.502},"max":{"1min":1.6930000000000001,"5min":1.964,"15min":37.307000000000002},"last":1.448}]},{"osd":5,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3959999999999999,"15min":1.5289999999999999},"min":{"1min":1.1779999999999999,"5min":0.83099999999999996,"15min":0.57299999999999995},"max":{"1min":1.708,"5min":2.0310000000000001,"15min":19.327000000000002},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.448,"15min":1.7649999999999999},"min":{"1min":1.089,"5min":0.77400000000000002,"15min":0.52400000000000002},"max":{"1min":1.73,"5min":4.8659999999999997,"15min":25.141999999999999},"last":1.5189999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.401,"15min":1.615},"min":{"1min":1.141,"5min":0.81999999999999995,"15min":0.56000000000000005},"max":{"1min":1.909,"5min":2.1539999999999999,"15min":37.363},"last":1.581},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.506,"15min":1.55},"min":{"1min":1.1799999999999999,"5min":0.86499999999999999,"15min":0.58799999999999997},"max":{"1min":1.6659999999999999,"5min":6.2599999999999998,"15min":6.9630000000000001},"last":1.4910000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.484,"15min":1.8200000000000001},"min":{"1min":1.268,"5min":0.70099999999999996,"15min":0.45400000000000001},"max":{"1min":1.794,"5min":2.1899999999999999,"15min":25.106999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.458,"15min":1.6839999999999999},"min":{"1min":1.2230000000000001,"5min":0.78400000000000003,"15min":0.61499999999999999},"max":{"1min":1.9950000000000001,"5min":2.27,"15min":37.402999999999999},"last":1.724}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68572,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303268,"statfs":{"total":96636764160,"available":96566546432,"internally_reserved":0,"allocated":1388544,"data_stored":596303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97499999999999998,"15min":1.2110000000000001},"min":{"1min":0.65200000000000002,"5min":0.40799999999999997,"15min":0.32100000000000001},"max":{"1min":1.3520000000000001,"5min":14.106999999999999,"15min":17.210999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":1.3580000000000001},"min":{"1min":0.81299999999999994,"5min":0.35899999999999999,"15min":0.26700000000000002},"max":{"1min":1.589,"5min":4.3719999999999999,"15min":34.252000000000002},"last":0.89200000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93500000000000005,"15min":1.3340000000000001},"min":{"1min":0.76300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5209999999999999,"5min":2.1219999999999999,"15min":24.709},"last":1.071},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2070000000000001,"15min":1.498},"min":{"1min":0.85299999999999998,"5min":0.436,"15min":0.42599999999999999},"max":{"1min":1.518,"5min":14.055,"15min":30.128},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.335,"15min":1.5980000000000001},"min":{"1min":0.90200000000000002,"5min":0.56000000000000005,"15min":0.40300000000000002},"max":{"1min":1.6519999999999999,"5min":13.967000000000001,"15min":25.611000000000001},"last":1.194},{"interface":"front","average":{"1min":1.089,"5min":1.119,"15min":1.4450000000000001},"min":{"1min":0.80800000000000005,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":1.4259999999999999,"5min":4.6699999999999999,"15min":41.972000000000001},"last":1.405}]},{"osd":4,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.262,"15min":1.446},"min":{"1min":1.004,"5min":0.52500000000000002,"15min":0.44600000000000001},"max":{"1min":1.5920000000000001,"5min":4.3799999999999999,"15min":18.004999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5569999999999999,"15min":1.7330000000000001},"min":{"1min":1.161,"5min":0.74199999999999999,"15min":0.63700000000000001},"max":{"1min":1.7090000000000001,"5min":14.161,"15min":21.738},"last":1.49}]},{"osd":5,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.5629999999999999,"15min":1.774},"min":{"1min":1.0660000000000001,"5min":0.83599999999999997,"15min":0.46600000000000003},"max":{"1min":1.7,"5min":9.0380000000000003,"15min":20.905999999999999},"last":1.012},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.548,"15min":1.7909999999999999},"min":{"1min":1.1120000000000001,"5min":0.56100000000000005,"15min":0.48799999999999999},"max":{"1min":1.7949999999999999,"5min":8.9969999999999999,"15min":17.957999999999998},"last":1.268}]},{"osd":6,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.571,"15min":1.764},"min":{"1min":1.1359999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9890000000000001,"5min":14.212,"15min":21.832999999999998},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.4630000000000001,"15min":1.643},"min":{"1min":1.143,"5min":0.66700000000000004,"15min":0.59199999999999997},"max":{"1min":1.649,"5min":6.5350000000000001,"15min":20.509},"last":1.3200000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.591,"15min":1.7909999999999999},"min":{"1min":1.1910000000000001,"5min":0.78100000000000003,"15min":0.55600000000000005},"max":{"1min":1.7010000000000001,"5min":12.590999999999999,"15min":20.669},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4810000000000001,"15min":1.6459999999999999},"min":{"1min":1.073,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.6910000000000001,"5min":6.0229999999999997,"15min":18.039999999999999},"last":1.593}]}]},{"osd":0,"up_from":13,"seq":55834575075,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81636,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290204,"statfs":{"total":96636764160,"available":96553168896,"internally_reserved":0,"allocated":1990656,"data_stored":1187252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86599999999999999,"15min":1.141},"min":{"1min":0.52600000000000002,"5min":0.377,"15min":0.28799999999999998},"max":{"1min":1.1919999999999999,"5min":1.921,"15min":26.751000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.998,"15min":1.1879999999999999},"min":{"1min":0.76000000000000001,"5min":0.45000000000000001,"15min":0.42199999999999999},"max":{"1min":1.589,"5min":1.589,"15min":15.750999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.012,"15min":1.5660000000000001},"min":{"1min":0.57999999999999996,"5min":0.46500000000000002,"15min":0.33500000000000002},"max":{"1min":1.395,"5min":1.7569999999999999,"15min":38.517000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.3140000000000001},"min":{"1min":0.877,"5min":0.48799999999999999,"15min":0.38800000000000001},"max":{"1min":1.546,"5min":2.2349999999999999,"15min":24.402000000000001},"last":0.63500000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0629999999999999,"15min":1.5109999999999999},"min":{"1min":0.66200000000000003,"5min":0.39600000000000002,"15min":0.33100000000000002},"max":{"1min":1.8919999999999999,"5min":2.2789999999999999,"15min":29.808},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.2969999999999999,"15min":1.72},"min":{"1min":0.81799999999999995,"5min":0.53000000000000003,"15min":0.41699999999999998},"max":{"1min":1.8,"5min":1.829,"15min":32.302},"last":0.79400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3460000000000001,"15min":1.577},"min":{"1min":1.1259999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.9419999999999999,"5min":2.2000000000000002,"15min":15.832000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.276,"15min":1.54},"min":{"1min":1.004,"5min":0.68300000000000005,"15min":0.432},"max":{"1min":1.6140000000000001,"5min":2.3490000000000002,"15min":19.062999999999999},"last":1.48}]},{"osd":5,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4470000000000001,"15min":1.6679999999999999},"min":{"1min":1.103,"5min":0.84699999999999998,"15min":0.56899999999999995},"max":{"1min":1.663,"5min":2.3730000000000002,"15min":26.800000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.494,"15min":1.6180000000000001},"min":{"1min":1.272,"5min":0.78500000000000003,"15min":0.60699999999999998},"max":{"1min":2.077,"5min":6.5430000000000001,"15min":15.906000000000001},"last":1.22}]},{"osd":6,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3380000000000001,"15min":1.663},"min":{"1min":0.96799999999999997,"5min":0.66000000000000003,"15min":0.60999999999999999},"max":{"1min":1.7470000000000001,"5min":2.1739999999999999,"15min":20.844000000000001},"last":1.026},{"interface":"front","average":{"1min":1.359,"5min":1.5009999999999999,"15min":1.6899999999999999},"min":{"1min":0.84999999999999998,"5min":0.80700000000000005,"15min":0.52800000000000002},"max":{"1min":1.7330000000000001,"5min":7.1849999999999996,"15min":26.832000000000001},"last":0.86499999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5109999999999999,"15min":1.7150000000000001},"min":{"1min":1.141,"5min":0.84199999999999997,"15min":0.59599999999999997},"max":{"1min":1.702,"5min":5.2450000000000001,"15min":16.058},"last":1.274},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4299999999999999,"15min":1.869},"min":{"1min":1.0980000000000001,"5min":0.68799999999999994,"15min":0.47499999999999998},"max":{"1min":1.881,"5min":2.5579999999999998,"15min":29.286999999999999},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:11.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:12.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.248+0000 7f7946800640 1 Processor -- start 2024-01-16T15:15:12.231 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.248+0000 7f7946800640 1 -- start start 2024-01-16T15:15:12.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7946800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79400ae5a0 0x7f794015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7946800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 0x7f794006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7946800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940070c40 con 0x7f794006d120 2024-01-16T15:15:12.233 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7946800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7940071450 con 0x7f79400ae5a0 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7946800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794006c470 con 0x7f79400b3ae0 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79400ae5a0 0x7f794015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79400ae5a0 0x7f794015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49110/0 (socket says 172.21.15.125:49110) 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 -- 172.21.15.125:0/418870724 learned_addr learned my addr 172.21.15.125:0/418870724 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f7937e00640 1 --2- 172.21.15.125:0/418870724 >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 0x7f794006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 -- 172.21.15.125:0/418870724 >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 msgr2=0x7f794006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 --2- 172.21.15.125:0/418870724 >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 0x7f794006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.250+0000 7f793f400640 1 -- 172.21.15.125:0/418870724 >> v1:172.21.15.137:6789/0 conn(0x7f794006d120 legacy=0x7f79400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.251+0000 7f793f400640 1 -- 172.21.15.125:0/418870724 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794006c750 con 0x7f79400ae5a0 2024-01-16T15:15:12.235 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.251+0000 7f7937e00640 1 --2- 172.21.15.125:0/418870724 >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 0x7f794006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.252+0000 7f793f400640 1 --2- 172.21.15.125:0/418870724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f794015e430 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f79400cfea0 tx=0x7f793002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cb546306f6b926 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.252+0000 7f793ea00640 1 -- 172.21.15.125:0/418870724 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930031870 con 0x7f79400ae5a0 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.252+0000 7f793ea00640 1 -- 172.21.15.125:0/418870724 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79300319d0 con 0x7f79400ae5a0 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.252+0000 7f793ea00640 1 -- 172.21.15.125:0/418870724 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930031d60 con 0x7f79400ae5a0 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 -- 172.21.15.125:0/418870724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 msgr2=0x7f794015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 --2- 172.21.15.125:0/418870724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f794015e430 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f79400cfea0 tx=0x7f793002b910 comp rx=0 tx=0).stop 2024-01-16T15:15:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 -- 172.21.15.125:0/418870724 shutdown_connections 2024-01-16T15:15:12.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 --2- 172.21.15.125:0/418870724 >> v2:172.21.15.137:3300/0 conn(0x7f79400b3ae0 0x7f794006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 --2- 172.21.15.125:0/418870724 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f794015e430 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 -- 172.21.15.125:0/418870724 >> 172.21.15.125:0/418870724 conn(0x7f794006a800 msgr2=0x7f794006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 -- 172.21.15.125:0/418870724 shutdown_connections 2024-01-16T15:15:12.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.253+0000 7f7946800640 1 -- 172.21.15.125:0/418870724 wait complete. 2024-01-16T15:15:12.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.255+0000 7f7946800640 1 Processor -- start 2024-01-16T15:15:12.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.255+0000 7f7946800640 1 -- start start 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f794006d120 0x7f79401f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79400b3ae0 0x7f79400cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79401f32f0 con 0x7f79400ae5a0 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79401f2d10 con 0x7f794006d120 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7946800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7940156020 con 0x7f79400b3ae0 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49124/0 (socket says 172.21.15.125:49124) 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 -- 172.21.15.125:0/1739533952 learned_addr learned my addr 172.21.15.125:0/1739533952 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f793f400640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f794006d120 0x7f79401f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 -- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79400b3ae0 msgr2=0x7f79400cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79400b3ae0 0x7f79400cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 -- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f794006d120 msgr2=0x7f79401f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f794006d120 0x7f79401f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.256+0000 7f7937e00640 1 -- 172.21.15.125:0/1739533952 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79300094d0 con 0x7f79400ae5a0 2024-01-16T15:15:12.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7937e00640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f792800c820 tx=0x7f792800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928007a30 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928007b90 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792800ce10 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79400ce2e0 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.257+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79400ce5c0 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.258+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79400675d0 con 0x7f79400ae5a0 2024-01-16T15:15:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.259+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f79280113f0 con 0x7f79400ae5a0 2024-01-16T15:15:12.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.260+0000 7f7937400640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 0x7f791806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.260+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7928088be0 con 0x7f79400ae5a0 2024-01-16T15:15:12.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.260+0000 7f793f400640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 0x7f791806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.261+0000 7f793f400640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 0x7f791806e8f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7930036000 tx=0x7f793003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.264+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7928055150 con 0x7f79400ae5a0 2024-01-16T15:15:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.450+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7940070550 con 0x7f791806c430 2024-01-16T15:15:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.467+0000 7f7937400640 1 -- 172.21.15.125:0/1739533952 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286888 (secure 0 0 0) 0x7f793003a420 con 0x7f791806c430 2024-01-16T15:15:12.454 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:15:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 msgr2=0x7f791806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 0x7f791806e8f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7930036000 tx=0x7f793003c040 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 msgr2=0x7f7940155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f792800c820 tx=0x7f792800cce0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 shutdown_connections 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79400b3ae0 0x7f79400cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f791806c430 0x7f791806e8f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79400ae5a0 0x7f7940155ae0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 --2- 172.21.15.125:0/1739533952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f794006d120 0x7f79401f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 >> 172.21.15.125:0/1739533952 conn(0x7f794006a800 msgr2=0x7f79400cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 shutdown_connections 2024-01-16T15:15:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.472+0000 7f7946800640 1 -- 172.21.15.125:0/1739533952 wait complete. 2024-01-16T15:15:12.474 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-01-16T15:15:12.348115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":587136,"kb_used_data":11980,"kb_used_omap":12,"kb_used_meta":575027,"kb_avail":754387584,"statfs":{"total":773094113280,"available":772492886016,"internally_reserved":0,"allocated":12267520,"data_stored":5941369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":588828208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002111"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667912+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:15:08.667912+0000","last_peered":"2024-01-16T15:15:08.667912+0000","last_clean":"2024-01-16T15:15:08.667912+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:15:08.667912+0000","last_undegraded":"2024-01-16T15:15:08.667912+0000","last_fullsized":"2024-01-16T15:15:08.667912+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:15.148113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596297+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:15:08.596297+0000","last_peered":"2024-01-16T15:15:08.596297+0000","last_clean":"2024-01-16T15:15:08.596297+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:15:08.596297+0000","last_undegraded":"2024-01-16T15:15:08.596297+0000","last_fullsized":"2024-01-16T15:15:08.596297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595647+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:15:08.595647+0000","last_peered":"2024-01-16T15:15:08.595647+0000","last_clean":"2024-01-16T15:15:08.595647+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:15:08.595647+0000","last_undegraded":"2024-01-16T15:15:08.595647+0000","last_fullsized":"2024-01-16T15:15:08.595647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667166+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:15:08.667166+0000","last_peered":"2024-01-16T15:15:08.667166+0000","last_clean":"2024-01-16T15:15:08.667166+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:15:08.667166+0000","last_undegraded":"2024-01-16T15:15:08.667166+0000","last_fullsized":"2024-01-16T15:15:08.667166+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:02:11.876145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900668+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:15:08.900668+0000","last_peered":"2024-01-16T15:15:08.900668+0000","last_clean":"2024-01-16T15:15:08.900668+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:15:08.900668+0000","last_undegraded":"2024-01-16T15:15:08.900668+0000","last_fullsized":"2024-01-16T15:15:08.900668+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:46.608210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899195+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:15:08.899195+0000","last_peered":"2024-01-16T15:15:08.899195+0000","last_clean":"2024-01-16T15:15:08.899195+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:15:08.899195+0000","last_undegraded":"2024-01-16T15:15:08.899195+0000","last_fullsized":"2024-01-16T15:15:08.899195+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:00.714461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900016+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:15:08.900016+0000","last_peered":"2024-01-16T15:15:08.900016+0000","last_clean":"2024-01-16T15:15:08.900016+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:15:08.900016+0000","last_undegraded":"2024-01-16T15:15:08.900016+0000","last_fullsized":"2024-01-16T15:15:08.900016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593293+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:15:08.593293+0000","last_peered":"2024-01-16T15:15:08.593293+0000","last_clean":"2024-01-16T15:15:08.593293+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:15:08.593293+0000","last_undegraded":"2024-01-16T15:15:08.593293+0000","last_fullsized":"2024-01-16T15:15:08.593293+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667929+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:15:08.667929+0000","last_peered":"2024-01-16T15:15:08.667929+0000","last_clean":"2024-01-16T15:15:08.667929+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:15:08.667929+0000","last_undegraded":"2024-01-16T15:15:08.667929+0000","last_fullsized":"2024-01-16T15:15:08.667929+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:25.063859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667765+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:15:08.667765+0000","last_peered":"2024-01-16T15:15:08.667765+0000","last_clean":"2024-01-16T15:15:08.667765+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:15:08.667765+0000","last_undegraded":"2024-01-16T15:15:08.667765+0000","last_fullsized":"2024-01-16T15:15:08.667765+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:45.318654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595739+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:15:08.595739+0000","last_peered":"2024-01-16T15:15:08.595739+0000","last_clean":"2024-01-16T15:15:08.595739+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:15:08.595739+0000","last_undegraded":"2024-01-16T15:15:08.595739+0000","last_fullsized":"2024-01-16T15:15:08.595739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667165+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:15:08.667165+0000","last_peered":"2024-01-16T15:15:08.667165+0000","last_clean":"2024-01-16T15:15:08.667165+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:15:08.667165+0000","last_undegraded":"2024-01-16T15:15:08.667165+0000","last_fullsized":"2024-01-16T15:15:08.667165+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:32.928994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594296+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:15:08.594296+0000","last_peered":"2024-01-16T15:15:08.594296+0000","last_clean":"2024-01-16T15:15:08.594296+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:15:08.594296+0000","last_undegraded":"2024-01-16T15:15:08.594296+0000","last_fullsized":"2024-01-16T15:15:08.594296+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594829+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:15:08.594829+0000","last_peered":"2024-01-16T15:15:08.594829+0000","last_clean":"2024-01-16T15:15:08.594829+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:15:08.594829+0000","last_undegraded":"2024-01-16T15:15:08.594829+0000","last_fullsized":"2024-01-16T15:15:08.594829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899050+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:15:08.899050+0000","last_peered":"2024-01-16T15:15:08.899050+0000","last_clean":"2024-01-16T15:15:08.899050+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:15:08.899050+0000","last_undegraded":"2024-01-16T15:15:08.899050+0000","last_fullsized":"2024-01-16T15:15:08.899050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:28.316281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666498+0000","last_change":"2024-01-16T15:15:07.589974+0000","last_active":"2024-01-16T15:15:08.666498+0000","last_peered":"2024-01-16T15:15:08.666498+0000","last_clean":"2024-01-16T15:15:08.666498+0000","last_became_active":"2024-01-16T15:15:07.584841+0000","last_became_peered":"2024-01-16T15:15:07.584841+0000","last_unstale":"2024-01-16T15:15:08.666498+0000","last_undegraded":"2024-01-16T15:15:08.666498+0000","last_fullsized":"2024-01-16T15:15:08.666498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:34.639802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596489+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:15:08.596489+0000","last_peered":"2024-01-16T15:15:08.596489+0000","last_clean":"2024-01-16T15:15:08.596489+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:15:08.596489+0000","last_undegraded":"2024-01-16T15:15:08.596489+0000","last_fullsized":"2024-01-16T15:15:08.596489+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899772+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:15:08.899772+0000","last_peered":"2024-01-16T15:15:08.899772+0000","last_clean":"2024-01-16T15:15:08.899772+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:15:08.899772+0000","last_undegraded":"2024-01-16T15:15:08.899772+0000","last_fullsized":"2024-01-16T15:15:08.899772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595572+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:15:08.595572+0000","last_peered":"2024-01-16T15:15:08.595572+0000","last_clean":"2024-01-16T15:15:08.595572+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:15:08.595572+0000","last_undegraded":"2024-01-16T15:15:08.595572+0000","last_fullsized":"2024-01-16T15:15:08.595572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595453+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:15:08.595453+0000","last_peered":"2024-01-16T15:15:08.595453+0000","last_clean":"2024-01-16T15:15:08.595453+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:15:08.595453+0000","last_undegraded":"2024-01-16T15:15:08.595453+0000","last_fullsized":"2024-01-16T15:15:08.595453+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595897+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:15:08.595897+0000","last_peered":"2024-01-16T15:15:08.595897+0000","last_clean":"2024-01-16T15:15:08.595897+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:15:08.595897+0000","last_undegraded":"2024-01-16T15:15:08.595897+0000","last_fullsized":"2024-01-16T15:15:08.595897+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591507+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:15:08.591507+0000","last_peered":"2024-01-16T15:15:08.591507+0000","last_clean":"2024-01-16T15:15:08.591507+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:15:08.591507+0000","last_undegraded":"2024-01-16T15:15:08.591507+0000","last_fullsized":"2024-01-16T15:15:08.591507+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594834+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:15:08.594834+0000","last_peered":"2024-01-16T15:15:08.594834+0000","last_clean":"2024-01-16T15:15:08.594834+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:15:08.594834+0000","last_undegraded":"2024-01-16T15:15:08.594834+0000","last_fullsized":"2024-01-16T15:15:08.594834+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591019+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:15:08.591019+0000","last_peered":"2024-01-16T15:15:08.591019+0000","last_clean":"2024-01-16T15:15:08.591019+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:15:08.591019+0000","last_undegraded":"2024-01-16T15:15:08.591019+0000","last_fullsized":"2024-01-16T15:15:08.591019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.596616+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:15:08.596616+0000","last_peered":"2024-01-16T15:15:08.596616+0000","last_clean":"2024-01-16T15:15:08.596616+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:15:08.596616+0000","last_undegraded":"2024-01-16T15:15:08.596616+0000","last_fullsized":"2024-01-16T15:15:08.596616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667834+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:15:08.667834+0000","last_peered":"2024-01-16T15:15:08.667834+0000","last_clean":"2024-01-16T15:15:08.667834+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:15:08.667834+0000","last_undegraded":"2024-01-16T15:15:08.667834+0000","last_fullsized":"2024-01-16T15:15:08.667834+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:06.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899902+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:15:08.899902+0000","last_peered":"2024-01-16T15:15:08.899902+0000","last_clean":"2024-01-16T15:15:08.899902+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:15:08.899902+0000","last_undegraded":"2024-01-16T15:15:08.899902+0000","last_fullsized":"2024-01-16T15:15:08.899902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:21:39.656818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592049+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:15:08.592049+0000","last_peered":"2024-01-16T15:15:08.592049+0000","last_clean":"2024-01-16T15:15:08.592049+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:15:08.592049+0000","last_undegraded":"2024-01-16T15:15:08.592049+0000","last_fullsized":"2024-01-16T15:15:08.592049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595053+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:15:08.595053+0000","last_peered":"2024-01-16T15:15:08.595053+0000","last_clean":"2024-01-16T15:15:08.595053+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:15:08.595053+0000","last_undegraded":"2024-01-16T15:15:08.595053+0000","last_fullsized":"2024-01-16T15:15:08.595053+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899461+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:15:08.899461+0000","last_peered":"2024-01-16T15:15:08.899461+0000","last_clean":"2024-01-16T15:15:08.899461+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:15:08.899461+0000","last_undegraded":"2024-01-16T15:15:08.899461+0000","last_fullsized":"2024-01-16T15:15:08.899461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666804+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:15:08.666804+0000","last_peered":"2024-01-16T15:15:08.666804+0000","last_clean":"2024-01-16T15:15:08.666804+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:15:08.666804+0000","last_undegraded":"2024-01-16T15:15:08.666804+0000","last_fullsized":"2024-01-16T15:15:08.666804+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:50.738078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898883+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:15:08.898883+0000","last_peered":"2024-01-16T15:15:08.898883+0000","last_clean":"2024-01-16T15:15:08.898883+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:15:08.898883+0000","last_undegraded":"2024-01-16T15:15:08.898883+0000","last_fullsized":"2024-01-16T15:15:08.898883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:08.692285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899768+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:15:08.899768+0000","last_peered":"2024-01-16T15:15:08.899768+0000","last_clean":"2024-01-16T15:15:08.899768+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:15:08.899768+0000","last_undegraded":"2024-01-16T15:15:08.899768+0000","last_fullsized":"2024-01-16T15:15:08.899768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:12.183950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899701+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:15:08.899701+0000","last_peered":"2024-01-16T15:15:08.899701+0000","last_clean":"2024-01-16T15:15:08.899701+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:15:08.899701+0000","last_undegraded":"2024-01-16T15:15:08.899701+0000","last_fullsized":"2024-01-16T15:15:08.899701+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900332+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:15:08.900332+0000","last_peered":"2024-01-16T15:15:08.900332+0000","last_clean":"2024-01-16T15:15:08.900332+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:15:08.900332+0000","last_undegraded":"2024-01-16T15:15:08.900332+0000","last_fullsized":"2024-01-16T15:15:08.900332+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594174+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:15:08.594174+0000","last_peered":"2024-01-16T15:15:08.594174+0000","last_clean":"2024-01-16T15:15:08.594174+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:15:08.594174+0000","last_undegraded":"2024-01-16T15:15:08.594174+0000","last_fullsized":"2024-01-16T15:15:08.594174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899386+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:15:08.899386+0000","last_peered":"2024-01-16T15:15:08.899386+0000","last_clean":"2024-01-16T15:15:08.899386+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:15:08.899386+0000","last_undegraded":"2024-01-16T15:15:08.899386+0000","last_fullsized":"2024-01-16T15:15:08.899386+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:48.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900859+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:15:08.900859+0000","last_peered":"2024-01-16T15:15:08.900859+0000","last_clean":"2024-01-16T15:15:08.900859+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:15:08.900859+0000","last_undegraded":"2024-01-16T15:15:08.900859+0000","last_fullsized":"2024-01-16T15:15:08.900859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:22.852738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595887+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:15:08.595887+0000","last_peered":"2024-01-16T15:15:08.595887+0000","last_clean":"2024-01-16T15:15:08.595887+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:15:08.595887+0000","last_undegraded":"2024-01-16T15:15:08.595887+0000","last_fullsized":"2024-01-16T15:15:08.595887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900429+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:15:08.900429+0000","last_peered":"2024-01-16T15:15:08.900429+0000","last_clean":"2024-01-16T15:15:08.900429+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:15:08.900429+0000","last_undegraded":"2024-01-16T15:15:08.900429+0000","last_fullsized":"2024-01-16T15:15:08.900429+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:07.677232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666987+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:15:08.666987+0000","last_peered":"2024-01-16T15:15:08.666987+0000","last_clean":"2024-01-16T15:15:08.666987+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:15:08.666987+0000","last_undegraded":"2024-01-16T15:15:08.666987+0000","last_fullsized":"2024-01-16T15:15:08.666987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:08.656641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899104+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:15:08.899104+0000","last_peered":"2024-01-16T15:15:08.899104+0000","last_clean":"2024-01-16T15:15:08.899104+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:15:08.899104+0000","last_undegraded":"2024-01-16T15:15:08.899104+0000","last_fullsized":"2024-01-16T15:15:08.899104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595386+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:15:08.595386+0000","last_peered":"2024-01-16T15:15:08.595386+0000","last_clean":"2024-01-16T15:15:08.595386+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:15:08.595386+0000","last_undegraded":"2024-01-16T15:15:08.595386+0000","last_fullsized":"2024-01-16T15:15:08.595386+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591606+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:15:08.591606+0000","last_peered":"2024-01-16T15:15:08.591606+0000","last_clean":"2024-01-16T15:15:08.591606+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:15:08.591606+0000","last_undegraded":"2024-01-16T15:15:08.591606+0000","last_fullsized":"2024-01-16T15:15:08.591606+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666549+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:15:08.666549+0000","last_peered":"2024-01-16T15:15:08.666549+0000","last_clean":"2024-01-16T15:15:08.666549+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:15:08.666549+0000","last_undegraded":"2024-01-16T15:15:08.666549+0000","last_fullsized":"2024-01-16T15:15:08.666549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:19.703629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594989+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:15:08.594989+0000","last_peered":"2024-01-16T15:15:08.594989+0000","last_clean":"2024-01-16T15:15:08.594989+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:15:08.594989+0000","last_undegraded":"2024-01-16T15:15:08.594989+0000","last_fullsized":"2024-01-16T15:15:08.594989+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898691+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:15:08.898691+0000","last_peered":"2024-01-16T15:15:08.898691+0000","last_clean":"2024-01-16T15:15:08.898691+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:15:08.898691+0000","last_undegraded":"2024-01-16T15:15:08.898691+0000","last_fullsized":"2024-01-16T15:15:08.898691+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666658+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:15:08.666658+0000","last_peered":"2024-01-16T15:15:08.666658+0000","last_clean":"2024-01-16T15:15:08.666658+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:15:08.666658+0000","last_undegraded":"2024-01-16T15:15:08.666658+0000","last_fullsized":"2024-01-16T15:15:08.666658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:43.538921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899448+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:15:08.899448+0000","last_peered":"2024-01-16T15:15:08.899448+0000","last_clean":"2024-01-16T15:15:08.899448+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:15:08.899448+0000","last_undegraded":"2024-01-16T15:15:08.899448+0000","last_fullsized":"2024-01-16T15:15:08.899448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900968+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:15:08.900968+0000","last_peered":"2024-01-16T15:15:08.900968+0000","last_clean":"2024-01-16T15:15:08.900968+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:15:08.900968+0000","last_undegraded":"2024-01-16T15:15:08.900968+0000","last_fullsized":"2024-01-16T15:15:08.900968+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:27.301228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900506+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:15:08.900506+0000","last_peered":"2024-01-16T15:15:08.900506+0000","last_clean":"2024-01-16T15:15:08.900506+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:15:08.900506+0000","last_undegraded":"2024-01-16T15:15:08.900506+0000","last_fullsized":"2024-01-16T15:15:08.900506+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592226+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:15:08.592226+0000","last_peered":"2024-01-16T15:15:08.592226+0000","last_clean":"2024-01-16T15:15:08.592226+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:15:08.592226+0000","last_undegraded":"2024-01-16T15:15:08.592226+0000","last_fullsized":"2024-01-16T15:15:08.592226+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592152+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:15:08.592152+0000","last_peered":"2024-01-16T15:15:08.592152+0000","last_clean":"2024-01-16T15:15:08.592152+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:15:08.592152+0000","last_undegraded":"2024-01-16T15:15:08.592152+0000","last_fullsized":"2024-01-16T15:15:08.592152+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667269+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:15:08.667269+0000","last_peered":"2024-01-16T15:15:08.667269+0000","last_clean":"2024-01-16T15:15:08.667269+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:15:08.667269+0000","last_undegraded":"2024-01-16T15:15:08.667269+0000","last_fullsized":"2024-01-16T15:15:08.667269+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899725+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:15:08.899725+0000","last_peered":"2024-01-16T15:15:08.899725+0000","last_clean":"2024-01-16T15:15:08.899725+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:15:08.899725+0000","last_undegraded":"2024-01-16T15:15:08.899725+0000","last_fullsized":"2024-01-16T15:15:08.899725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667336+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:15:08.667336+0000","last_peered":"2024-01-16T15:15:08.667336+0000","last_clean":"2024-01-16T15:15:08.667336+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:15:08.667336+0000","last_undegraded":"2024-01-16T15:15:08.667336+0000","last_fullsized":"2024-01-16T15:15:08.667336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:32.472941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591569+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:15:08.591569+0000","last_peered":"2024-01-16T15:15:08.591569+0000","last_clean":"2024-01-16T15:15:08.591569+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:15:08.591569+0000","last_undegraded":"2024-01-16T15:15:08.591569+0000","last_fullsized":"2024-01-16T15:15:08.591569+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899098+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:15:08.899098+0000","last_peered":"2024-01-16T15:15:08.899098+0000","last_clean":"2024-01-16T15:15:08.899098+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:15:08.899098+0000","last_undegraded":"2024-01-16T15:15:08.899098+0000","last_fullsized":"2024-01-16T15:15:08.899098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:40.359437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591753+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:15:08.591753+0000","last_peered":"2024-01-16T15:15:08.591753+0000","last_clean":"2024-01-16T15:15:08.591753+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:15:08.591753+0000","last_undegraded":"2024-01-16T15:15:08.591753+0000","last_fullsized":"2024-01-16T15:15:08.591753+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.898761+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:15:08.898761+0000","last_peered":"2024-01-16T15:15:08.898761+0000","last_clean":"2024-01-16T15:15:08.898761+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:15:08.898761+0000","last_undegraded":"2024-01-16T15:15:08.898761+0000","last_fullsized":"2024-01-16T15:15:08.898761+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591171+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:15:08.591171+0000","last_peered":"2024-01-16T15:15:08.591171+0000","last_clean":"2024-01-16T15:15:08.591171+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:15:08.591171+0000","last_undegraded":"2024-01-16T15:15:08.591171+0000","last_fullsized":"2024-01-16T15:15:08.591171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899740+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:15:08.899740+0000","last_peered":"2024-01-16T15:15:08.899740+0000","last_clean":"2024-01-16T15:15:08.899740+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:15:08.899740+0000","last_undegraded":"2024-01-16T15:15:08.899740+0000","last_fullsized":"2024-01-16T15:15:08.899740+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:11.168897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.901277+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:15:08.901277+0000","last_peered":"2024-01-16T15:15:08.901277+0000","last_clean":"2024-01-16T15:15:08.901277+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:15:08.901277+0000","last_undegraded":"2024-01-16T15:15:08.901277+0000","last_fullsized":"2024-01-16T15:15:08.901277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:21.652051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667982+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:15:08.667982+0000","last_peered":"2024-01-16T15:15:08.667982+0000","last_clean":"2024-01-16T15:15:08.667982+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:15:08.667982+0000","last_undegraded":"2024-01-16T15:15:08.667982+0000","last_fullsized":"2024-01-16T15:15:08.667982+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:53.963781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899763+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:15:08.899763+0000","last_peered":"2024-01-16T15:15:08.899763+0000","last_clean":"2024-01-16T15:15:08.899763+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:15:08.899763+0000","last_undegraded":"2024-01-16T15:15:08.899763+0000","last_fullsized":"2024-01-16T15:15:08.899763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:45.884405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.901110+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:15:08.901110+0000","last_peered":"2024-01-16T15:15:08.901110+0000","last_clean":"2024-01-16T15:15:08.901110+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:15:08.901110+0000","last_undegraded":"2024-01-16T15:15:08.901110+0000","last_fullsized":"2024-01-16T15:15:08.901110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:59.699408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594406+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:15:08.594406+0000","last_peered":"2024-01-16T15:15:08.594406+0000","last_clean":"2024-01-16T15:15:08.594406+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:15:08.594406+0000","last_undegraded":"2024-01-16T15:15:08.594406+0000","last_fullsized":"2024-01-16T15:15:08.594406+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899491+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:15:08.899491+0000","last_peered":"2024-01-16T15:15:08.899491+0000","last_clean":"2024-01-16T15:15:08.899491+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:15:08.899491+0000","last_undegraded":"2024-01-16T15:15:08.899491+0000","last_fullsized":"2024-01-16T15:15:08.899491+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595955+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:15:08.595955+0000","last_peered":"2024-01-16T15:15:08.595955+0000","last_clean":"2024-01-16T15:15:08.595955+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:15:08.595955+0000","last_undegraded":"2024-01-16T15:15:08.595955+0000","last_fullsized":"2024-01-16T15:15:08.595955+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899087+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:15:08.899087+0000","last_peered":"2024-01-16T15:15:08.899087+0000","last_clean":"2024-01-16T15:15:08.899087+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:15:08.899087+0000","last_undegraded":"2024-01-16T15:15:08.899087+0000","last_fullsized":"2024-01-16T15:15:08.899087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594902+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:15:08.594902+0000","last_peered":"2024-01-16T15:15:08.594902+0000","last_clean":"2024-01-16T15:15:08.594902+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:15:08.594902+0000","last_undegraded":"2024-01-16T15:15:08.594902+0000","last_fullsized":"2024-01-16T15:15:08.594902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899359+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:15:08.899359+0000","last_peered":"2024-01-16T15:15:08.899359+0000","last_clean":"2024-01-16T15:15:08.899359+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:15:08.899359+0000","last_undegraded":"2024-01-16T15:15:08.899359+0000","last_fullsized":"2024-01-16T15:15:08.899359+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666900+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:15:08.666900+0000","last_peered":"2024-01-16T15:15:08.666900+0000","last_clean":"2024-01-16T15:15:08.666900+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:15:08.666900+0000","last_undegraded":"2024-01-16T15:15:08.666900+0000","last_fullsized":"2024-01-16T15:15:08.666900+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.557222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899591+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:15:08.899591+0000","last_peered":"2024-01-16T15:15:08.899591+0000","last_clean":"2024-01-16T15:15:08.899591+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:15:08.899591+0000","last_undegraded":"2024-01-16T15:15:08.899591+0000","last_fullsized":"2024-01-16T15:15:08.899591+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593311+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:15:08.593311+0000","last_peered":"2024-01-16T15:15:08.593311+0000","last_clean":"2024-01-16T15:15:08.593311+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:15:08.593311+0000","last_undegraded":"2024-01-16T15:15:08.593311+0000","last_fullsized":"2024-01-16T15:15:08.593311+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899170+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:15:08.899170+0000","last_peered":"2024-01-16T15:15:08.899170+0000","last_clean":"2024-01-16T15:15:08.899170+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:15:08.899170+0000","last_undegraded":"2024-01-16T15:15:08.899170+0000","last_fullsized":"2024-01-16T15:15:08.899170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591490+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:15:08.591490+0000","last_peered":"2024-01-16T15:15:08.591490+0000","last_clean":"2024-01-16T15:15:08.591490+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:15:08.591490+0000","last_undegraded":"2024-01-16T15:15:08.591490+0000","last_fullsized":"2024-01-16T15:15:08.591490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666665+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:15:08.666665+0000","last_peered":"2024-01-16T15:15:08.666665+0000","last_clean":"2024-01-16T15:15:08.666665+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:15:08.666665+0000","last_undegraded":"2024-01-16T15:15:08.666665+0000","last_fullsized":"2024-01-16T15:15:08.666665+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:02.498301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900030+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:15:08.900030+0000","last_peered":"2024-01-16T15:15:08.900030+0000","last_clean":"2024-01-16T15:15:08.900030+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:15:08.900030+0000","last_undegraded":"2024-01-16T15:15:08.900030+0000","last_fullsized":"2024-01-16T15:15:08.900030+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592106+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:15:08.592106+0000","last_peered":"2024-01-16T15:15:08.592106+0000","last_clean":"2024-01-16T15:15:08.592106+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:15:08.592106+0000","last_undegraded":"2024-01-16T15:15:08.592106+0000","last_fullsized":"2024-01-16T15:15:08.592106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667983+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:15:08.667983+0000","last_peered":"2024-01-16T15:15:08.667983+0000","last_clean":"2024-01-16T15:15:08.667983+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:15:08.667983+0000","last_undegraded":"2024-01-16T15:15:08.667983+0000","last_fullsized":"2024-01-16T15:15:08.667983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:40.476189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900018+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:15:08.900018+0000","last_peered":"2024-01-16T15:15:08.900018+0000","last_clean":"2024-01-16T15:15:08.900018+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:15:08.900018+0000","last_undegraded":"2024-01-16T15:15:08.900018+0000","last_fullsized":"2024-01-16T15:15:08.900018+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667295+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:15:08.667295+0000","last_peered":"2024-01-16T15:15:08.667295+0000","last_clean":"2024-01-16T15:15:08.667295+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:15:08.667295+0000","last_undegraded":"2024-01-16T15:15:08.667295+0000","last_fullsized":"2024-01-16T15:15:08.667295+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:29.732315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.667263+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:15:08.667263+0000","last_peered":"2024-01-16T15:15:08.667263+0000","last_clean":"2024-01-16T15:15:08.667263+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:15:08.667263+0000","last_undegraded":"2024-01-16T15:15:08.667263+0000","last_fullsized":"2024-01-16T15:15:08.667263+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:41.055270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899061+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:15:08.899061+0000","last_peered":"2024-01-16T15:15:08.899061+0000","last_clean":"2024-01-16T15:15:08.899061+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:15:08.899061+0000","last_undegraded":"2024-01-16T15:15:08.899061+0000","last_fullsized":"2024-01-16T15:15:08.899061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.592262+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:15:08.592262+0000","last_peered":"2024-01-16T15:15:08.592262+0000","last_clean":"2024-01-16T15:15:08.592262+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:15:08.592262+0000","last_undegraded":"2024-01-16T15:15:08.592262+0000","last_fullsized":"2024-01-16T15:15:08.592262+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.594866+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:15:08.594866+0000","last_peered":"2024-01-16T15:15:08.594866+0000","last_clean":"2024-01-16T15:15:08.594866+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:15:08.594866+0000","last_undegraded":"2024-01-16T15:15:08.594866+0000","last_fullsized":"2024-01-16T15:15:08.594866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899345+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:15:08.899345+0000","last_peered":"2024-01-16T15:15:08.899345+0000","last_clean":"2024-01-16T15:15:08.899345+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:15:08.899345+0000","last_undegraded":"2024-01-16T15:15:08.899345+0000","last_fullsized":"2024-01-16T15:15:08.899345+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595719+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:15:08.595719+0000","last_peered":"2024-01-16T15:15:08.595719+0000","last_clean":"2024-01-16T15:15:08.595719+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:15:08.595719+0000","last_undegraded":"2024-01-16T15:15:08.595719+0000","last_fullsized":"2024-01-16T15:15:08.595719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.595117+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:15:08.595117+0000","last_peered":"2024-01-16T15:15:08.595117+0000","last_clean":"2024-01-16T15:15:08.595117+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:15:08.595117+0000","last_undegraded":"2024-01-16T15:15:08.595117+0000","last_fullsized":"2024-01-16T15:15:08.595117+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.593661+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:15:08.593661+0000","last_peered":"2024-01-16T15:15:08.593661+0000","last_clean":"2024-01-16T15:15:08.593661+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:15:08.593661+0000","last_undegraded":"2024-01-16T15:15:08.593661+0000","last_fullsized":"2024-01-16T15:15:08.593661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.900147+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:15:08.900147+0000","last_peered":"2024-01-16T15:15:08.900147+0000","last_clean":"2024-01-16T15:15:08.900147+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:15:08.900147+0000","last_undegraded":"2024-01-16T15:15:08.900147+0000","last_fullsized":"2024-01-16T15:15:08.900147+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666857+0000","last_change":"2024-01-16T15:15:07.592852+0000","last_active":"2024-01-16T15:15:08.666857+0000","last_peered":"2024-01-16T15:15:08.666857+0000","last_clean":"2024-01-16T15:15:08.666857+0000","last_became_active":"2024-01-16T15:15:07.592163+0000","last_became_peered":"2024-01-16T15:15:07.592163+0000","last_unstale":"2024-01-16T15:15:08.666857+0000","last_undegraded":"2024-01-16T15:15:08.666857+0000","last_fullsized":"2024-01-16T15:15:08.666857+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:15.876527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.666780+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:15:08.666780+0000","last_peered":"2024-01-16T15:15:08.666780+0000","last_clean":"2024-01-16T15:15:08.666780+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:15:08.666780+0000","last_undegraded":"2024-01-16T15:15:08.666780+0000","last_fullsized":"2024-01-16T15:15:08.666780+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:10.684617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.591079+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:15:08.591079+0000","last_peered":"2024-01-16T15:15:08.591079+0000","last_clean":"2024-01-16T15:15:08.591079+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:15:08.591079+0000","last_undegraded":"2024-01-16T15:15:08.591079+0000","last_fullsized":"2024-01-16T15:15:08.591079+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:15:08.899955+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:15:08.899955+0000","last_peered":"2024-01-16T15:15:08.899955+0000","last_clean":"2024-01-16T15:15:08.899955+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:15:08.899955+0000","last_undegraded":"2024-01-16T15:15:08.899955+0000","last_fullsized":"2024-01-16T15:15:08.899955+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":265,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:15:07.562660+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:15:07.562660+0000","last_peered":"2024-01-16T15:15:07.562660+0000","last_clean":"2024-01-16T15:15:07.562660+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:15:07.562660+0000","last_undegraded":"2024-01-16T15:15:07.562660+0000","last_fullsized":"2024-01-16T15:15:07.562660+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575075,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64976,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306864,"statfs":{"total":96636764160,"available":96570228736,"internally_reserved":0,"allocated":1970176,"data_stored":1180863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.111,"15min":1.4239999999999999},"min":{"1min":0.82999999999999996,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.5489999999999999,"5min":3.3969999999999998,"15min":45.296999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.226,"15min":1.347},"min":{"1min":0.80100000000000005,"5min":0.627,"15min":0.497},"max":{"1min":1.7410000000000001,"5min":3.4990000000000001,"15min":20.126999999999999},"last":1.2589999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2509999999999999,"15min":1.444},"min":{"1min":0.94699999999999995,"5min":0.69699999999999995,"15min":0.499},"max":{"1min":2.2210000000000001,"5min":3.431,"15min":42.723999999999997},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.413,"15min":1.6299999999999999},"min":{"1min":0.98299999999999998,"5min":0.73099999999999998,"15min":0.49099999999999999},"max":{"1min":2.4620000000000002,"5min":3.6539999999999999,"15min":43.572000000000003},"last":1.617}]},{"osd":2,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2050000000000001,"15min":1.4550000000000001},"min":{"1min":0.91000000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.1640000000000001,"5min":2.5910000000000002,"15min":35.308},"last":1.532},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.242,"15min":1.47},"min":{"1min":0.83899999999999997,"5min":0.70099999999999996,"15min":0.503},"max":{"1min":1.8260000000000001,"5min":2.2080000000000002,"15min":23.594000000000001},"last":1.153}]},{"osd":3,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.427,"15min":1.671},"min":{"1min":0.91900000000000004,"5min":0.73899999999999999,"15min":0.51300000000000001},"max":{"1min":2.2799999999999998,"5min":12.331,"15min":53.033999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.339,"15min":1.5940000000000001},"min":{"1min":0.94699999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0230000000000001,"5min":2.1389999999999998,"15min":31.305},"last":1.369}]},{"osd":4,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.381,"15min":1.343},"min":{"1min":0.96599999999999997,"5min":0.77100000000000002,"15min":0.39600000000000002},"max":{"1min":2.226,"5min":3.5710000000000002,"15min":5.1870000000000003},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3140000000000001,"15min":1.514},"min":{"1min":0.89700000000000002,"5min":0.58299999999999996,"15min":0.41099999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":36.374000000000002},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4099999999999999,"15min":1.466},"min":{"1min":0.77700000000000002,"5min":0.60299999999999998,"15min":0.45700000000000002},"max":{"1min":1.9379999999999999,"5min":2.5369999999999999,"15min":9.9399999999999995},"last":1.466},{"interface":"front","average":{"1min":1.583,"5min":1.589,"15min":1.5349999999999999},"min":{"1min":1.0449999999999999,"5min":0.76400000000000001,"15min":0.53500000000000003},"max":{"1min":2.3860000000000001,"5min":4.7400000000000002,"15min":7.0170000000000003},"last":1.498}]},{"osd":6,"last update":"Tue Jan 16 15:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.506,"15min":1.5629999999999999},"min":{"1min":1.306,"5min":0.73199999999999998,"15min":0.53700000000000003},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":15.052},"last":1.659},{"interface":"front","average":{"1min":1.599,"5min":1.5720000000000001,"15min":1.6060000000000001},"min":{"1min":0.89700000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1280000000000001,"5min":2.782,"15min":8.4849999999999994},"last":1.575}]}]},{"osd":6,"up_from":13,"seq":55834575076,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81048,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290792,"statfs":{"total":96636764160,"available":96553771008,"internally_reserved":0,"allocated":1384448,"data_stored":596249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81590726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.242,"15min":1.677},"min":{"1min":0.86399999999999999,"5min":0.626,"15min":0.61899999999999999},"max":{"1min":1.637,"5min":1.9359999999999999,"15min":71.870999999999995},"last":1.399},{"interface":"front","average":{"1min":1.302,"5min":1.3140000000000001,"15min":1.754},"min":{"1min":0.82899999999999996,"5min":0.64600000000000002,"15min":0.501},"max":{"1min":1.79,"5min":2.2000000000000002,"15min":71.846999999999994},"last":1.6419999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1839999999999999,"15min":1.3460000000000001},"min":{"1min":0.88400000000000001,"5min":0.67700000000000005,"15min":0.63900000000000001},"max":{"1min":1.6779999999999999,"5min":3.4049999999999998,"15min":15.798},"last":1.339},{"interface":"front","average":{"1min":1.268,"5min":1.21,"15min":1.4550000000000001},"min":{"1min":0.86599999999999999,"5min":0.70999999999999996,"15min":0.626},"max":{"1min":1.5680000000000001,"5min":1.8979999999999999,"15min":24.556999999999999},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3080000000000001,"15min":1.4610000000000001},"min":{"1min":0.96799999999999997,"5min":0.84199999999999997,"15min":0.40000000000000002},"max":{"1min":1.8819999999999999,"5min":2.3029999999999999,"15min":14.981},"last":1.76},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.373,"15min":1.5369999999999999},"min":{"1min":0.99399999999999999,"5min":0.873,"15min":0.57299999999999995},"max":{"1min":1.7490000000000001,"5min":5.1909999999999998,"15min":14.175000000000001},"last":1.679}]},{"osd":3,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.339,"15min":1.4750000000000001},"min":{"1min":0.876,"5min":0.82899999999999996,"15min":0.51100000000000001},"max":{"1min":1.6319999999999999,"5min":2.1789999999999998,"15min":14.827},"last":1.454},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.401,"15min":1.583},"min":{"1min":0.94299999999999995,"5min":0.753,"15min":0.499},"max":{"1min":1.776,"5min":2.032,"15min":21.890000000000001},"last":1.5069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.325,"15min":1.4199999999999999},"min":{"1min":0.73499999999999999,"5min":0.627,"15min":0.45000000000000001},"max":{"1min":1.661,"5min":2.1259999999999999,"15min":10.901},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.391,"15min":1.4239999999999999},"min":{"1min":0.68000000000000005,"5min":0.60299999999999998,"15min":0.48899999999999999},"max":{"1min":2.0859999999999999,"5min":2.1560000000000001,"15min":6.9000000000000004},"last":1.8720000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3069999999999999,"15min":1.3799999999999999},"min":{"1min":0.78800000000000003,"5min":0.55300000000000005,"15min":0.44600000000000001},"max":{"1min":1.752,"5min":2.1669999999999998,"15min":10.736000000000001},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.512,"15min":1.5920000000000001},"min":{"1min":0.80600000000000005,"5min":0.77400000000000002,"15min":0.498},"max":{"1min":1.9490000000000001,"5min":2.2130000000000001,"15min":9.2200000000000006},"last":1.7}]},{"osd":7,"last update":"Tue Jan 16 15:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.7250000000000001,"15min":1.6699999999999999},"min":{"1min":0.83699999999999997,"5min":0.64700000000000002,"15min":0.53200000000000003},"max":{"1min":1.988,"5min":24.675000000000001,"15min":24.675000000000001},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.516,"15min":1.6799999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.54600000000000004},"max":{"1min":1.827,"5min":5.2469999999999999,"15min":14.494},"last":1.7430000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76892,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294948,"statfs":{"total":96636764160,"available":96558026752,"internally_reserved":0,"allocated":1388544,"data_stored":596783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0960000000000001,"15min":1.5209999999999999},"min":{"1min":0.77300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.667,"5min":2.0830000000000002,"15min":25.634},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.2390000000000001,"15min":1.51},"min":{"1min":0.90100000000000002,"5min":0.49299999999999999,"15min":0.48299999999999998},"max":{"1min":1.6830000000000001,"5min":8.7880000000000003,"15min":34.959000000000003},"last":1.4590000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2170000000000001,"15min":1.4019999999999999},"min":{"1min":0.85799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.6910000000000001,"5min":1.8460000000000001,"15min":26.378},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.292,"5min":1.254,"15min":1.3520000000000001},"min":{"1min":1.022,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.542,"5min":5.3940000000000001,"15min":13.332000000000001},"last":1.012}]},{"osd":2,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.337,"15min":1.427},"min":{"1min":0.98499999999999999,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":1.502,"5min":5.2110000000000003,"15min":25.495000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.373,"15min":1.4970000000000001},"min":{"1min":1.0629999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.794,"5min":5.3070000000000004,"15min":17.030000000000001},"last":1.1739999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.343,"15min":1.6619999999999999},"min":{"1min":0.97199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.536,"5min":1.994,"15min":25.82},"last":1.075},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.349,"15min":1.681},"min":{"1min":1.03,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.698,"5min":1.8959999999999999,"15min":22.963999999999999},"last":1.2889999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.214,"15min":1.2789999999999999},"min":{"1min":0.86499999999999999,"5min":0.46700000000000003,"15min":0.378},"max":{"1min":1.7050000000000001,"5min":1.964,"15min":8.6050000000000004},"last":1.369},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.4870000000000001,"15min":1.5920000000000001},"min":{"1min":1.093,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":1.76,"5min":2.1930000000000001,"15min":25.724},"last":1.3400000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.601},"min":{"1min":0.87,"5min":0.46800000000000003,"15min":0.40100000000000002},"max":{"1min":1.7689999999999999,"5min":1.9750000000000001,"15min":42.542999999999999},"last":1.518},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2929999999999999,"15min":1.3799999999999999},"min":{"1min":0.996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.6120000000000001,"5min":2.1230000000000002,"15min":10.015000000000001},"last":1.48}]},{"osd":7,"last update":"Tue Jan 16 15:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.47,"15min":1.6910000000000001},"min":{"1min":1.1719999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7490000000000001,"5min":2.141,"15min":26.484000000000002},"last":1.423},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.494,"15min":1.6799999999999999},"min":{"1min":1.1519999999999999,"5min":0.60699999999999998,"15min":0.46999999999999997},"max":{"1min":1.8320000000000001,"5min":3.2509999999999999,"15min":21.106000000000002},"last":1.575}]}]},{"osd":4,"up_from":13,"seq":55834575075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81032,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290808,"statfs":{"total":96636764160,"available":96553787392,"internally_reserved":0,"allocated":1368064,"data_stored":589879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4710000000000001,"15min":1.4059999999999999},"min":{"1min":0.84199999999999997,"5min":0.57899999999999996,"15min":0.44400000000000001},"max":{"1min":2.0640000000000001,"5min":17.315999999999999,"15min":17.315999999999999},"last":1.966},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5,"15min":1.4359999999999999},"min":{"1min":0.93100000000000005,"5min":0.79700000000000004,"15min":0.52800000000000002},"max":{"1min":1.9390000000000001,"5min":17.279,"15min":17.279},"last":1.867}]},{"osd":1,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2789999999999999,"15min":1.476},"min":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.498},"max":{"1min":1.669,"5min":6.1849999999999996,"15min":39.823999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.1899999999999999,"15min":1.3899999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.48399999999999999},"max":{"1min":1.8160000000000001,"5min":4.2809999999999997,"15min":50.582000000000001},"last":1.534}]},{"osd":2,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.167,"15min":1.3029999999999999},"min":{"1min":0.87,"5min":0.73299999999999998,"15min":0.39500000000000002},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":10.984},"last":0.877},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.83799999999999997,"5min":0.81999999999999995,"15min":0.53500000000000003},"max":{"1min":1.8169999999999999,"5min":1.8360000000000001,"15min":9.1509999999999998},"last":1.0129999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.387,"15min":1.3919999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.54000000000000004},"max":{"1min":1.639,"5min":9.8350000000000009,"15min":11.971},"last":1.222},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.5840000000000001},"min":{"1min":1.0269999999999999,"5min":0.75,"15min":0.57899999999999996},"max":{"1min":1.657,"5min":2.0550000000000002,"15min":27.626999999999999},"last":1.0740000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.286,"15min":1.353},"min":{"1min":0.97199999999999998,"5min":0.81200000000000006,"15min":0.40000000000000002},"max":{"1min":1.8839999999999999,"5min":1.992,"15min":8.2599999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.393,"15min":1.4590000000000001},"min":{"1min":0.998,"5min":0.64100000000000001,"15min":0.46200000000000002},"max":{"1min":1.742,"5min":4.8520000000000003,"15min":17.140999999999998},"last":1.74}]},{"osd":6,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.329,"15min":1.4359999999999999},"min":{"1min":0.91500000000000004,"5min":0.74299999999999999,"15min":0.45300000000000001},"max":{"1min":1.758,"5min":2.0139999999999998,"15min":7.4509999999999996},"last":1.29},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4610000000000001,"15min":1.5980000000000001},"min":{"1min":1.161,"5min":0.88500000000000001,"15min":0.44900000000000001},"max":{"1min":1.986,"5min":2.093,"15min":25.643000000000001},"last":1.47}]},{"osd":7,"last update":"Tue Jan 16 15:15:05 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4790000000000001,"15min":1.554},"min":{"1min":1.1830000000000001,"5min":0.86299999999999999,"15min":0.53300000000000003},"max":{"1min":1.8680000000000001,"5min":4.9539999999999997,"15min":24.992999999999999},"last":1.421},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5129999999999999,"15min":1.6990000000000001},"min":{"1min":1.121,"5min":0.92900000000000005,"15min":0.50800000000000001},"max":{"1min":1.9119999999999999,"5min":2.0230000000000001,"15min":19.001000000000001},"last":1.629}]}]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68580,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303260,"statfs":{"total":96636764160,"available":96566538240,"internally_reserved":0,"allocated":1396736,"data_stored":597861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.91900000000000004,"15min":1.161},"min":{"1min":0.60099999999999998,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":1.115,"5min":1.925,"15min":38.423999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.98599999999999999,"15min":1.1899999999999999},"min":{"1min":0.77000000000000002,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":1.3420000000000001,"5min":2.9729999999999999,"15min":20.012},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0069999999999999,"15min":1.629},"min":{"1min":0.70799999999999996,"5min":0.44900000000000001,"15min":0.39300000000000002},"max":{"1min":1.302,"5min":3.1150000000000002,"15min":40.695999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.071,"15min":1.212},"min":{"1min":0.72999999999999998,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":1.5029999999999999,"5min":2.25,"15min":9.9689999999999994},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.141,"15min":1.5529999999999999},"min":{"1min":0.81200000000000006,"5min":0.46100000000000002,"15min":0.436},"max":{"1min":1.794,"5min":3.0670000000000002,"15min":59.009},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.216,"15min":1.6779999999999999},"min":{"1min":0.97999999999999998,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.4510000000000001,"5min":3.4279999999999999,"15min":29.773},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3240000000000001,"15min":1.466},"min":{"1min":1.036,"5min":0.71799999999999997,"15min":0.59899999999999998},"max":{"1min":1.627,"5min":3.5049999999999999,"15min":21.736000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3919999999999999,"15min":1.9199999999999999},"min":{"1min":1.073,"5min":0.64000000000000001,"15min":0.622},"max":{"1min":1.71,"5min":3.8250000000000002,"15min":29.524000000000001},"last":0.94499999999999995}]},{"osd":5,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.341,"15min":1.49},"min":{"1min":1.133,"5min":0.70999999999999996,"15min":0.54800000000000004},"max":{"1min":2.0139999999999998,"5min":2.3540000000000001,"15min":9.8249999999999993},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.403,"15min":1.581},"min":{"1min":1.083,"5min":0.71299999999999997,"15min":0.60799999999999998},"max":{"1min":1.7589999999999999,"5min":2.907,"15min":11.255000000000001},"last":1.2689999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4850000000000001,"15min":1.603},"min":{"1min":1.0509999999999999,"5min":0.752,"15min":0.65900000000000003},"max":{"1min":1.7789999999999999,"5min":6.4480000000000004,"15min":20.459},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.538,"15min":2.028},"min":{"1min":1.1870000000000001,"5min":0.81299999999999994,"15min":0.49399999999999999},"max":{"1min":1.857,"5min":3.9039999999999999,"15min":29.475000000000001},"last":1.042}]},{"osd":7,"last update":"Tue Jan 16 15:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4810000000000001,"15min":1.637},"min":{"1min":1.097,"5min":0.73099999999999998,"15min":0.55300000000000005},"max":{"1min":1.853,"5min":3.7850000000000001,"15min":20.109999999999999},"last":1.008},{"interface":"front","average":{"1min":1.46,"5min":1.5329999999999999,"15min":2.0270000000000001},"min":{"1min":1.1719999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.9350000000000001,"5min":3.9590000000000001,"15min":29.396999999999998},"last":0.99199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575075,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1380352,"data_stored":596179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88500000000000001,"15min":1.109},"min":{"1min":0.68999999999999995,"5min":0.314,"15min":0.314},"max":{"1min":1.482,"5min":1.744,"15min":25.335999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66100000000000003,"5min":0.42299999999999999,"15min":0.27800000000000002},"max":{"1min":1.4299999999999999,"5min":2.4449999999999998,"15min":25.199000000000002},"last":1.157}]},{"osd":1,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.018,"15min":1.1919999999999999},"min":{"1min":0.80000000000000004,"5min":0.53500000000000003,"15min":0.36799999999999999},"max":{"1min":1.55,"5min":1.7989999999999999,"15min":16.776},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1419999999999999,"15min":1.4179999999999999},"min":{"1min":0.82999999999999996,"5min":0.46500000000000002,"15min":0.318},"max":{"1min":1.478,"5min":4.734,"15min":25.186},"last":1.0489999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0640000000000001,"15min":1.246},"min":{"1min":0.83399999999999996,"5min":0.52600000000000002,"15min":0.33900000000000002},"max":{"1min":1.528,"5min":1.7430000000000001,"15min":37.470999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.218,"15min":1.329},"min":{"1min":0.96299999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.494,"5min":2.3220000000000001,"15min":28.045999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.496,"15min":1.764},"min":{"1min":1.165,"5min":0.65200000000000002,"15min":0.47499999999999998},"max":{"1min":1.615,"5min":16.681999999999999,"15min":25.062000000000001},"last":1.28},{"interface":"front","average":{"1min":1.367,"5min":1.3120000000000001,"15min":1.611},"min":{"1min":1.018,"5min":0.746,"15min":0.502},"max":{"1min":1.6930000000000001,"5min":1.964,"15min":37.307000000000002},"last":1.448}]},{"osd":5,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3959999999999999,"15min":1.5289999999999999},"min":{"1min":1.1779999999999999,"5min":0.83099999999999996,"15min":0.57299999999999995},"max":{"1min":1.708,"5min":2.0310000000000001,"15min":19.327000000000002},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.448,"15min":1.7649999999999999},"min":{"1min":1.089,"5min":0.77400000000000002,"15min":0.52400000000000002},"max":{"1min":1.73,"5min":4.8659999999999997,"15min":25.141999999999999},"last":1.5189999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.401,"15min":1.615},"min":{"1min":1.141,"5min":0.81999999999999995,"15min":0.56000000000000005},"max":{"1min":1.909,"5min":2.1539999999999999,"15min":37.363},"last":1.581},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.506,"15min":1.55},"min":{"1min":1.1799999999999999,"5min":0.86499999999999999,"15min":0.58799999999999997},"max":{"1min":1.6659999999999999,"5min":6.2599999999999998,"15min":6.9630000000000001},"last":1.4910000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.484,"15min":1.8200000000000001},"min":{"1min":1.268,"5min":0.70099999999999996,"15min":0.45400000000000001},"max":{"1min":1.794,"5min":2.1899999999999999,"15min":25.106999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.458,"15min":1.6839999999999999},"min":{"1min":1.2230000000000001,"5min":0.78400000000000003,"15min":0.61499999999999999},"max":{"1min":1.9950000000000001,"5min":2.27,"15min":37.402999999999999},"last":1.724}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68572,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303268,"statfs":{"total":96636764160,"available":96566546432,"internally_reserved":0,"allocated":1388544,"data_stored":596303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97499999999999998,"15min":1.2110000000000001},"min":{"1min":0.65200000000000002,"5min":0.40799999999999997,"15min":0.32100000000000001},"max":{"1min":1.3520000000000001,"5min":14.106999999999999,"15min":17.210999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":1.3580000000000001},"min":{"1min":0.81299999999999994,"5min":0.35899999999999999,"15min":0.26700000000000002},"max":{"1min":1.589,"5min":4.3719999999999999,"15min":34.252000000000002},"last":0.89200000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93500000000000005,"15min":1.3340000000000001},"min":{"1min":0.76300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5209999999999999,"5min":2.1219999999999999,"15min":24.709},"last":1.071},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2070000000000001,"15min":1.498},"min":{"1min":0.85299999999999998,"5min":0.436,"15min":0.42599999999999999},"max":{"1min":1.518,"5min":14.055,"15min":30.128},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.335,"15min":1.5980000000000001},"min":{"1min":0.90200000000000002,"5min":0.56000000000000005,"15min":0.40300000000000002},"max":{"1min":1.6519999999999999,"5min":13.967000000000001,"15min":25.611000000000001},"last":1.194},{"interface":"front","average":{"1min":1.089,"5min":1.119,"15min":1.4450000000000001},"min":{"1min":0.80800000000000005,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":1.4259999999999999,"5min":4.6699999999999999,"15min":41.972000000000001},"last":1.405}]},{"osd":4,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.262,"15min":1.446},"min":{"1min":1.004,"5min":0.52500000000000002,"15min":0.44600000000000001},"max":{"1min":1.5920000000000001,"5min":4.3799999999999999,"15min":18.004999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5569999999999999,"15min":1.7330000000000001},"min":{"1min":1.161,"5min":0.74199999999999999,"15min":0.63700000000000001},"max":{"1min":1.7090000000000001,"5min":14.161,"15min":21.738},"last":1.49}]},{"osd":5,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.5629999999999999,"15min":1.774},"min":{"1min":1.0660000000000001,"5min":0.83599999999999997,"15min":0.46600000000000003},"max":{"1min":1.7,"5min":9.0380000000000003,"15min":20.905999999999999},"last":1.012},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.548,"15min":1.7909999999999999},"min":{"1min":1.1120000000000001,"5min":0.56100000000000005,"15min":0.48799999999999999},"max":{"1min":1.7949999999999999,"5min":8.9969999999999999,"15min":17.957999999999998},"last":1.268}]},{"osd":6,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.571,"15min":1.764},"min":{"1min":1.1359999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9890000000000001,"5min":14.212,"15min":21.832999999999998},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.4630000000000001,"15min":1.643},"min":{"1min":1.143,"5min":0.66700000000000004,"15min":0.59199999999999997},"max":{"1min":1.649,"5min":6.5350000000000001,"15min":20.509},"last":1.3200000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.591,"15min":1.7909999999999999},"min":{"1min":1.1910000000000001,"5min":0.78100000000000003,"15min":0.55600000000000005},"max":{"1min":1.7010000000000001,"5min":12.590999999999999,"15min":20.669},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4810000000000001,"15min":1.6459999999999999},"min":{"1min":1.073,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.6910000000000001,"5min":6.0229999999999997,"15min":18.039999999999999},"last":1.593}]}]},{"osd":0,"up_from":13,"seq":55834575075,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81636,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290204,"statfs":{"total":96636764160,"available":96553168896,"internally_reserved":0,"allocated":1990656,"data_stored":1187252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86599999999999999,"15min":1.141},"min":{"1min":0.52600000000000002,"5min":0.377,"15min":0.28799999999999998},"max":{"1min":1.1919999999999999,"5min":1.921,"15min":26.751000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.998,"15min":1.1879999999999999},"min":{"1min":0.76000000000000001,"5min":0.45000000000000001,"15min":0.42199999999999999},"max":{"1min":1.589,"5min":1.589,"15min":15.750999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.012,"15min":1.5660000000000001},"min":{"1min":0.57999999999999996,"5min":0.46500000000000002,"15min":0.33500000000000002},"max":{"1min":1.395,"5min":1.7569999999999999,"15min":38.517000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.3140000000000001},"min":{"1min":0.877,"5min":0.48799999999999999,"15min":0.38800000000000001},"max":{"1min":1.546,"5min":2.2349999999999999,"15min":24.402000000000001},"last":0.63500000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0629999999999999,"15min":1.5109999999999999},"min":{"1min":0.66200000000000003,"5min":0.39600000000000002,"15min":0.33100000000000002},"max":{"1min":1.8919999999999999,"5min":2.2789999999999999,"15min":29.808},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.2969999999999999,"15min":1.72},"min":{"1min":0.81799999999999995,"5min":0.53000000000000003,"15min":0.41699999999999998},"max":{"1min":1.8,"5min":1.829,"15min":32.302},"last":0.79400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3460000000000001,"15min":1.577},"min":{"1min":1.1259999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.9419999999999999,"5min":2.2000000000000002,"15min":15.832000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.276,"15min":1.54},"min":{"1min":1.004,"5min":0.68300000000000005,"15min":0.432},"max":{"1min":1.6140000000000001,"5min":2.3490000000000002,"15min":19.062999999999999},"last":1.48}]},{"osd":5,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4470000000000001,"15min":1.6679999999999999},"min":{"1min":1.103,"5min":0.84699999999999998,"15min":0.56899999999999995},"max":{"1min":1.663,"5min":2.3730000000000002,"15min":26.800000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.494,"15min":1.6180000000000001},"min":{"1min":1.272,"5min":0.78500000000000003,"15min":0.60699999999999998},"max":{"1min":2.077,"5min":6.5430000000000001,"15min":15.906000000000001},"last":1.22}]},{"osd":6,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3380000000000001,"15min":1.663},"min":{"1min":0.96799999999999997,"5min":0.66000000000000003,"15min":0.60999999999999999},"max":{"1min":1.7470000000000001,"5min":2.1739999999999999,"15min":20.844000000000001},"last":1.026},{"interface":"front","average":{"1min":1.359,"5min":1.5009999999999999,"15min":1.6899999999999999},"min":{"1min":0.84999999999999998,"5min":0.80700000000000005,"15min":0.52800000000000002},"max":{"1min":1.7330000000000001,"5min":7.1849999999999996,"15min":26.832000000000001},"last":0.86499999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5109999999999999,"15min":1.7150000000000001},"min":{"1min":1.141,"5min":0.84199999999999997,"15min":0.59599999999999997},"max":{"1min":1.702,"5min":5.2450000000000001,"15min":16.058},"last":1.274},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4299999999999999,"15min":1.869},"min":{"1min":1.0980000000000001,"5min":0.68799999999999994,"15min":0.47499999999999998},"max":{"1min":1.881,"5min":2.5579999999999998,"15min":29.286999999999999},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:12.478 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:15:12.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:15:12.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:12.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:12.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.745+0000 7f3ccea00640 1 Processor -- start 2024-01-16T15:15:12.728 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.745+0000 7f3ccea00640 1 -- start start 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3ccea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3cc8141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3ccea00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc813e440 con 0x7f3cc815f100 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3ccea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc80ce1d0 con 0x7f3cc806c850 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3ccea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80ce4b0 con 0x7f3cc815ed20 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3cc7400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3cc815f100 0x7f3cc80cd780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51912/0 (socket says 172.21.15.125:51912) 2024-01-16T15:15:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.747+0000 7f3cc7400640 1 -- 172.21.15.125:0/592210143 learned_addr learned my addr 172.21.15.125:0/592210143 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:12.732 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3cc8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243535662 0 0) 0x7f3cc813e440 con 0x7f3cc815f100 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca8003520 con 0x7f3cc815f100 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003555693 0 0) 0x7f3ca8003520 con 0x7f3cc815f100 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 msgr2=0x7f3cc8141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3cc8141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 >> v1:172.21.15.137:6789/0 conn(0x7f3cc806c850 legacy=0x7f3cc806cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc80ce790 con 0x7f3cc815f100 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3cb0001670 con 0x7f3cc815f100 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 >> v1:172.21.15.125:6789/0 conn(0x7f3cc815f100 legacy=0x7f3cc80cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca80039a0 0x7f3ca8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3ca80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.748+0000 7f3cc6a00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ca8004d00 0x7f3ca8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.733 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80ce4b0 con 0x7f3ca80039a0 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca800bb10 con 0x7f3ca8004d00 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca800bcd0 con 0x7f3cc815ed20 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca80039a0 0x7f3ca8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7400640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3ca80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 -- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 msgr2=0x7f3ca80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3ccca00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ca8004d00 0x7f3ca8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cc815ed20 0x7f3ca80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 -- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ca8004d00 msgr2=0x7f3ca8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ca8004d00 0x7f3ca8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3cc7e00640 1 -- 172.21.15.125:0/592210143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8003520 con 0x7f3ca80039a0 2024-01-16T15:15:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.749+0000 7f3ccca00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ca8004d00 0x7f3ca8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:12.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.750+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca80039a0 0x7f3ca8003dc0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009ac0 tx=0x7f3ca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb92f2f7ead1f148 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb802c6b0 con 0x7f3ca80039a0 2024-01-16T15:15:12.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb802c810 con 0x7f3ca80039a0 2024-01-16T15:15:12.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3cc6a00640 1 -- 172.21.15.125:0/592210143 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb802cba0 con 0x7f3ca80039a0 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 -- 172.21.15.125:0/592210143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca80039a0 msgr2=0x7f3ca8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 --2- 172.21.15.125:0/592210143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca80039a0 0x7f3ca8003dc0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009ac0 tx=0x7f3ca80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccca00640 1 -- 172.21.15.125:0/592210143 reap_dead start 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 -- 172.21.15.125:0/592210143 shutdown_connections 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 -- 172.21.15.125:0/592210143 >> 172.21.15.125:0/592210143 conn(0x7f3cc80b0be0 msgr2=0x7f3cc80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 -- 172.21.15.125:0/592210143 shutdown_connections 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.751+0000 7f3ccea00640 1 -- 172.21.15.125:0/592210143 wait complete. 2024-01-16T15:15:12.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.753+0000 7f3ccea00640 1 Processor -- start 2024-01-16T15:15:12.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.753+0000 7f3ccea00640 1 -- start start 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 0x7f3cc80712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 0x7f3cc815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc8071d20 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80c3ef0 con 0x7f3cc80717e0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80c41d0 con 0x7f3ca80039a0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3cc7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3cc7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49142/0 (socket says 172.21.15.125:49142) 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3cc7400640 1 -- 172.21.15.125:0/2952896905 learned_addr learned my addr 172.21.15.125:0/2952896905 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 0x7f3cc80712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.755+0000 7f3ccca00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 0x7f3cc815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 -- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 msgr2=0x7f3cc80712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 0x7f3cc80712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 -- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 msgr2=0x7f3cc815bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 0x7f3cc815bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 -- 172.21.15.125:0/2952896905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc815f6c0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 0x7f3cc80712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3ccca00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 0x7f3cc815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cc7400640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000d810 tx=0x7f3cb000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0007560 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb00047e0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc80c44b0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0004b70 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.756+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc813d800 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.758+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc80c49d0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.758+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f3cb0004cd0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.759+0000 7f3cbfe00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 0x7f3ca006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.759+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cb00887c0 con 0x7f3cc80c4cb0 2024-01-16T15:15:12.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.760+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 0x7f3ca006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.743 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.760+0000 7f3cc7e00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 0x7f3ca006e920 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8036000 tx=0x7f3cb8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:12.764+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cc80c49d0 con 0x7f3cc80c4cb0 2024-01-16T15:15:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.028+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3cc80c49d0 con 0x7f3cc80c4cb0 2024-01-16T15:15:13.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.033+0000 7f3cbfe00640 1 -- 172.21.15.125:0/2952896905 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3cc80c49d0 con 0x7f3cc80c4cb0 2024-01-16T15:15:13.016 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2024-01-16T15:15:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDtmKZl/+iFAhAAkB8YC129DKd3UxBF2OJetA== 2024-01-16T15:15:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: caps mds = "allow" 2024-01-16T15:15:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: caps mon = "allow r" 2024-01-16T15:15:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:13.017 INFO:teuthology.orchestra.run.smithi125.stderr:updated caps for client.0 2024-01-16T15:15:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.035+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 msgr2=0x7f3ca006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.035+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 0x7f3ca006e920 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8036000 tx=0x7f3cb8030000 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.035+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 msgr2=0x7f3cc815b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.035+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000d810 tx=0x7f3cb000dcd0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 shutdown_connections 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cc80717e0 0x7f3cc815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3ca006c460 0x7f3ca006e920 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cc80c4cb0 0x7f3cc815b730 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 --2- 172.21.15.125:0/2952896905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ca80039a0 0x7f3cc80712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 >> 172.21.15.125:0/2952896905 conn(0x7f3cc80b0be0 msgr2=0x7f3cc80afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 shutdown_connections 2024-01-16T15:15:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.036+0000 7f3ccea00640 1 -- 172.21.15.125:0/2952896905 wait complete. 2024-01-16T15:15:13.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:13.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.249+0000 7fca05000640 1 Processor -- start 2024-01-16T15:15:13.232 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.249+0000 7fca05000640 1 -- start start 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.250+0000 7fca05000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fca0015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.250+0000 7fca05000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fca0013e360 0x7fca000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.250+0000 7fca05000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0006c470 con 0x7fca0015f9c0 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.250+0000 7fca05000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0006c750 con 0x7fca0013e360 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.250+0000 7fca05000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006ca30 con 0x7fca0013df80 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fe800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fca0015f9c0 0x7fca001408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51928/0 (socket says 172.21.15.125:51928) 2024-01-16T15:15:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fe800640 1 -- 172.21.15.125:0/2107959456 learned_addr learned my addr 172.21.15.125:0/2107959456 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fca0015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2107959456 >> v2:172.21.15.137:3300/0 conn(0x7fca0013e360 0x7fca000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167330179 0 0) 0x7fca0006c470 con 0x7fca0015f9c0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0003520 con 0x7fca0015f9c0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752392734 0 0) 0x7fc9e0003520 con 0x7fca0015f9c0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 msgr2=0x7fca0015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fca0015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 >> v2:172.21.15.137:3300/0 conn(0x7fca0013e360 msgr2=0x7fca000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 --2- 172.21.15.125:0/2107959456 >> v2:172.21.15.137:3300/0 conn(0x7fca0013e360 0x7fca000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0006cd10 con 0x7fca0015f9c0 2024-01-16T15:15:13.237 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc9f80019f0 con 0x7fca0015f9c0 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.251+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 >> v1:172.21.15.125:6789/0 conn(0x7fca0015f9c0 legacy=0x7fca001408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fc9e0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0003ef0 0x7fc9e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9e0004d00 0x7fc9e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006c750 con 0x7fc9e0004d00 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006ca30 con 0x7fc9e0003ef0 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e000b980 con 0x7fca0013df80 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9e0004d00 0x7fc9e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fc9e0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0003ef0 0x7fc9e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 -- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 msgr2=0x7fc9e0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0013df80 0x7fc9e0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 -- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0003ef0 msgr2=0x7fc9e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0003ef0 0x7fc9e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fe800640 1 -- 172.21.15.125:0/2107959456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0003520 con 0x7fc9e0004d00 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.252+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0003ef0 0x7fc9e00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.253+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9e0004d00 0x7fc9e0005180 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800a8d0 tx=0x7fc9e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2768151926e28ff2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8034630 con 0x7fc9e0004d00 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f802f070 con 0x7fc9e0004d00 2024-01-16T15:15:13.239 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fc9fca00640 1 -- 172.21.15.125:0/2107959456 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80349d0 con 0x7fc9e0004d00 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 -- 172.21.15.125:0/2107959456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9e0004d00 msgr2=0x7fc9e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 --2- 172.21.15.125:0/2107959456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9e0004d00 0x7fc9e0005180 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800a8d0 tx=0x7fc9e00057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fc9fe800640 1 -- 172.21.15.125:0/2107959456 reap_dead start 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 -- 172.21.15.125:0/2107959456 shutdown_connections 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 -- 172.21.15.125:0/2107959456 >> 172.21.15.125:0/2107959456 conn(0x7fca0006a800 msgr2=0x7fca0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 -- 172.21.15.125:0/2107959456 shutdown_connections 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.254+0000 7fca05000640 1 -- 172.21.15.125:0/2107959456 wait complete. 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.256+0000 7fca05000640 1 Processor -- start 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.256+0000 7fca05000640 1 -- start start 2024-01-16T15:15:13.243 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 0x7fca00152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 0x7fca000afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000a8af0 con 0x7fca000ab000 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca00153260 con 0x7fc9e0004d00 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fca05000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca000ab9c0 con 0x7fca000af620 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49164/0 (socket says 172.21.15.125:49164) 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fc9fd400640 1 -- 172.21.15.125:0/2987826290 learned_addr learned my addr 172.21.15.125:0/2987826290 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.257+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 0x7fca00152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 0x7fca000afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 -- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 msgr2=0x7fca000afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 0x7fca000afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 -- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 msgr2=0x7fca00152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 0x7fca00152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 -- 172.21.15.125:0/2987826290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca000b0910 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 0x7fca00152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fe800640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 0x7fca000afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9fd400640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8010a00 tx=0x7fc9e8010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800ec20 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e800ed80 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e801c620 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca000b0350 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.258+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca000a8710 con 0x7fca000ab000 2024-01-16T15:15:13.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.259+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca000675d0 con 0x7fca000ab000 2024-01-16T15:15:13.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.260+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc9e801c780 con 0x7fca000ab000 2024-01-16T15:15:13.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.261+0000 7fc9f6000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 0x7fc9d806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.261+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9e80891b0 con 0x7fca000ab000 2024-01-16T15:15:13.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.261+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 0x7fc9d806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.262+0000 7fc9fde00640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 0x7fc9d806e8f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009a00 tx=0x7fc9f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.266+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9e80557d0 con 0x7fca000ab000 2024-01-16T15:15:13.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.494+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca0015ec70 con 0x7fca000ab000 2024-01-16T15:15:13.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.495+0000 7fc9f6000640 1 -- 172.21.15.125:0/2987826290 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fc9e8058df0 con 0x7fca000ab000 2024-01-16T15:15:13.478 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:13.478 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1181037452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1181037452},{"type":"v1","addr":"172.21.15.137:6833","nonce":1181037452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1991371319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1991371319},{"type":"v1","addr":"172.21.15.137:6835","nonce":1991371319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":9156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/13212755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":13212755},{"type":"v1","addr":"172.21.15.125:6835","nonce":13212755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:06.982766+0000","modified":"2024-01-16T15:15:10.072697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9162":{"gid":9162,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/2099236429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2099236429},{"type":"v1","addr":"172.21.15.125:6837","nonce":2099236429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:15:13.479 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 73 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 msgr2=0x7fc9d806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 0x7fc9d806e8f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009a00 tx=0x7fc9f0008040 comp rx=0 tx=0).stop 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 msgr2=0x7fca000ab480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8010a00 tx=0x7fc9e8010ec0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 shutdown_connections 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca000af620 0x7fca000afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc9d806c430 0x7fc9d806e8f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca000ab000 0x7fca000ab480 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 --2- 172.21.15.125:0/2987826290 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9e0004d00 0x7fca00152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 >> 172.21.15.125:0/2987826290 conn(0x7fca0006a800 msgr2=0x7fca0015a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.497+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 shutdown_connections 2024-01-16T15:15:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:13.498+0000 7fca05000640 1 -- 172.21.15.125:0/2987826290 wait complete. 2024-01-16T15:15:13.488 INFO:tasks.ceph.mds.c.smithi125.stderr:2024-01-16T15:15:13.505+0000 7f9edc600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:15:13.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:15:06.982766+0000', 'modified': '2024-01-16T15:15:10.072697+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9162}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9162': {'gid': 9162, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/2099236429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6836', 'nonce': 2099236429}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 2099236429}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:15:13.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:15:13.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:15:14.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.136+0000 7f1667000640 1 Processor -- start 2024-01-16T15:15:14.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.136+0000 7f1667000640 1 -- start start 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1667000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1660157c30 0x7f16600cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1667000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166015deb0 con 0x7f1660071200 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1667000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166015e190 con 0x7f16601618f0 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1667000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166015ea30 con 0x7f1660157c30 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1665000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1660071200 0x7f166006dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51932/0 (socket says 172.21.15.125:51932) 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f1665000640 1 -- 172.21.15.125:0/1257955993 learned_addr learned my addr 172.21.15.125:0/1257955993 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f165fe00640 1 --2- 172.21.15.125:0/1257955993 >> v2:172.21.15.137:3300/0 conn(0x7f1660157c30 0x7f16600cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.138+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756821324 0 0) 0x7f166015deb0 con 0x7f1660071200 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1640003520 con 0x7f1660071200 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786971556 0 0) 0x7f1640003520 con 0x7f1660071200 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 >> v2:172.21.15.137:3300/0 conn(0x7f1660157c30 msgr2=0x7f16600cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 --2- 172.21.15.125:0/1257955993 >> v2:172.21.15.137:3300/0 conn(0x7f1660157c30 0x7f16600cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 >> v1:172.21.15.137:6789/0 conn(0x7f16601618f0 legacy=0x7f1660070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166015e470 con 0x7f1660071200 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1658001670 con 0x7f1660071200 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 >> v1:172.21.15.125:6789/0 conn(0x7f1660071200 legacy=0x7f166006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16400039c0 0x7f1640003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 0x7f1640004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640005180 0x7f164000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166015ea30 con 0x7f16400047c0 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164000c080 con 0x7f1640005180 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164000c240 con 0x7f16400039c0 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.139+0000 7f165f400640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 0x7f1640004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165fe00640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16400039c0 0x7f1640003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f1665000640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640005180 0x7f164000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165f400640 1 -- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16400039c0 msgr2=0x7f1640003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165f400640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16400039c0 0x7f1640003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165f400640 1 -- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640005180 msgr2=0x7f164000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165f400640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640005180 0x7f164000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165f400640 1 -- 172.21.15.125:0/1257955993 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640003520 con 0x7f16400047c0 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f165fe00640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16400039c0 0x7f1640003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f1665000640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640005180 0x7f164000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.140+0000 7f1665000640 1 -- 172.21.15.125:0/1257955993 reap_dead start 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.141+0000 7f165f400640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 0x7f1640004c40 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f1648001750 tx=0x7f16400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c92cf75e2c1ba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.141+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16480052d0 con 0x7f16400047c0 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.141+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164802e050 con 0x7f16400047c0 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.141+0000 7f165ea00640 1 -- 172.21.15.125:0/1257955993 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648003830 con 0x7f16400047c0 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 -- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 msgr2=0x7f1640004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 0x7f1640004c40 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f1648001750 tx=0x7f16400057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 -- 172.21.15.125:0/1257955993 shutdown_connections 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 --2- 172.21.15.125:0/1257955993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16400047c0 0x7f1640004c40 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 -- 172.21.15.125:0/1257955993 >> 172.21.15.125:0/1257955993 conn(0x7f16600cf700 msgr2=0x7f16600cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 -- 172.21.15.125:0/1257955993 shutdown_connections 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.142+0000 7f1667000640 1 -- 172.21.15.125:0/1257955993 wait complete. 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.144+0000 7f1667000640 1 Processor -- start 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.144+0000 7f1667000640 1 -- start start 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 0x7f16601570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1660158070 0x7f166015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16601f4b60 con 0x7f166015b1b0 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16601f3fe0 con 0x7f16400047c0 2024-01-16T15:15:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1667000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16601f42c0 con 0x7f1660158070 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49184/0 (socket says 172.21.15.125:49184) 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f1665000640 1 -- 172.21.15.125:0/2332767294 learned_addr learned my addr 172.21.15.125:0/2332767294 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f165fe00640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 0x7f16601570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 -- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1660158070 msgr2=0x7f166015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.145+0000 7f165f400640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1660158070 0x7f166015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1660158070 0x7f166015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 -- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 msgr2=0x7f16601570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 0x7f16601570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 -- 172.21.15.125:0/2332767294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16601f45a0 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f165fe00640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 0x7f16601570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1665000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f166006b630 tx=0x7f165800d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1657e00640 1 -- 172.21.15.125:0/2332767294 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658007440 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1657e00640 1 -- 172.21.15.125:0/2332767294 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16580075a0 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1657e00640 1 -- 172.21.15.125:0/2332767294 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580049e0 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16601f4880 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.146+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16601584d0 con 0x7f166015b1b0 2024-01-16T15:15:14.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.148+0000 7f1657e00640 1 -- 172.21.15.125:0/2332767294 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f1658004b40 con 0x7f166015b1b0 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.149+0000 7f1657e00640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 0x7f163806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.149+0000 7f1657e00640 1 -- 172.21.15.125:0/2332767294 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1658088160 con 0x7f166015b1b0 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.150+0000 7f165fe00640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 0x7f163806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.150+0000 7f1669dc5740 1 Processor -- start 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.150+0000 7f1669dc5740 1 -- start start 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.150+0000 7f165fe00640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 0x7f163806e8f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f16600695b0 tx=0x7f165000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac076ea0 0x5616ac077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0760a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616abfd5880 con 0x5616ac0765e0 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616abfe4ae0 con 0x5616ac071ba0 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616abfe21d0 con 0x5616ac076ea0 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49192/0 (socket says 172.21.15.125:49192) 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.151+0000 7f1665000640 1 -- 172.21.15.125:0/3745172984 learned_addr learned my addr 172.21.15.125:0/3745172984 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 -- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac076ea0 msgr2=0x5616ac077280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac076ea0 0x5616ac077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 -- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 msgr2=0x5616ac0760a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f165f400640 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0760a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 -- 172.21.15.125:0/3745172984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616abfa5860 con 0x5616ac0765e0 2024-01-16T15:15:14.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1665000640 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f1658054950 tx=0x7f1658058510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2958f0b67778a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1654c00640 1 -- 172.21.15.125:0/3745172984 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580883f0 con 0x5616ac0765e0 2024-01-16T15:15:14.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1654c00640 1 -- 172.21.15.125:0/3745172984 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16580567f0 con 0x5616ac0765e0 2024-01-16T15:15:14.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.152+0000 7f1654c00640 1 -- 172.21.15.125:0/3745172984 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580602c0 con 0x5616ac0765e0 2024-01-16T15:15:14.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 -- 172.21.15.125:0/3745172984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 msgr2=0x5616ac0ba560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f1658054950 tx=0x7f1658058510 comp rx=0 tx=0).stop 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 -- 172.21.15.125:0/3745172984 shutdown_connections 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0ba560 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0760a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 --2- 172.21.15.125:0/3745172984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac076ea0 0x5616ac077280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 -- 172.21.15.125:0/3745172984 >> 172.21.15.125:0/3745172984 conn(0x5616ac038c70 msgr2=0x5616ac0bc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 -- 172.21.15.125:0/3745172984 shutdown_connections 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.153+0000 7f1669dc5740 1 -- 172.21.15.125:0/3745172984 wait complete. 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.154+0000 7f1669dc5740 1 -- start start 2024-01-16T15:15:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.154+0000 7f1669dc5740 1 Processor -- start 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac09c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac09fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616abfddac0 con 0x5616ac071ba0 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616abf7a2f0 con 0x5616ac076ea0 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616abf97c70 con 0x5616ac0765e0 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f1669dc5740 10 client.0 _ll_register_callbacks cb 0x5616abf9c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f165fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f165fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49198/0 (socket says 172.21.15.125:49198) 2024-01-16T15:15:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.155+0000 7f165fe00640 1 -- 172.21.15.125:0/2591178347 learned_addr learned my addr 172.21.15.125:0/2591178347 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 -- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 msgr2=0x5616ac09c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac09c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 -- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 msgr2=0x5616ac09fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f1665000640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac09fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac09fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 -- 172.21.15.125:0/2591178347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16601584d0 con 0x5616ac071ba0 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f1665000640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac09fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f165fe00640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f1650015d90 tx=0x7f1650009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8239f3dd4cc4087 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f164d600640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165000e050 con 0x5616ac071ba0 2024-01-16T15:15:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1650016d00 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.156+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616abfd6aa0 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1650021400 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 10 client.9294 fetch_fsmap learned FSMap version 73 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5616abfdc140 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5616abfd6aa0 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1650021560 con 0x5616ac071ba0 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 10 client.9294 fetch_fsmap finished waiting for FSMap version 73 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 10 client.9294 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 20 client.9294 populate_metadata read hostname 'smithi125' 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac0d57b0 0x5616ac0d9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 4 client.9294 mds_command: new command op to 9162 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 --> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616abfd67c0 con 0x5616ac0d57b0 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.157+0000 7f165f400640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac0d57b0 0x5616ac0d9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.158+0000 7f164d600640 10 client.9294 ms_handle_connect on v2:172.21.15.125:6836/2099236429 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.158+0000 7f165f400640 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac0d57b0 0x5616ac0d9550 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36718bfc577567a2 server_cookie=dc1e8ec7c3e91017 in_seq=0 out_seq=0 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f164d600640 1 -- 172.21.15.125:0/2591178347 <== mds.0 v2:172.21.15.125:6836/2099236429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616abfd67c0 con 0x5616ac0d57b0 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f164d600640 10 client.9294 handle_command_reply: tid=0 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 client.9294 shutdown 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 msgr2=0x5616ac0c1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f1650015d90 tx=0x7f1650009750 comp rx=0 tx=0).stop 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 shutdown_connections 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac0d57b0 0x5616ac0d9550 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac09fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac09c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 --2- 172.21.15.125:0/2591178347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac071ba0 0x5616ac0c1a70 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f164d600640 0 client.9294 ms_handle_reset on v2:172.21.15.125:6836/2099236429 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 >> 172.21.15.125:0/2591178347 conn(0x5616ac038c70 msgr2=0x5616ac077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 shutdown_connections 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 1 -- 172.21.15.125:0/2591178347 wait complete. 2024-01-16T15:15:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.159+0000 7f1669dc5740 20 client.9294 trim_cache size 0 max 16384 2024-01-16T15:15:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.176+0000 7f1669dc5740 1 Processor -- start 2024-01-16T15:15:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.176+0000 7f1669dc5740 1 -- start start 2024-01-16T15:15:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac071ba0 0x5616ac09f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac0bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616abd40f20 con 0x5616ac0765e0 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ac070b80 con 0x5616ac076ea0 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616ac0bc470 con 0x5616ac071ba0 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f165f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f165f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49202/0 (socket says 172.21.15.125:49202) 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.177+0000 7f165f400640 1 -- 172.21.15.125:0/1582196615 learned_addr learned my addr 172.21.15.125:0/1582196615 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 -- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac071ba0 msgr2=0x5616ac09f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac071ba0 0x5616ac09f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 -- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 msgr2=0x5616ac0bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac0bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 -- 172.21.15.125:0/1582196615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616abf79b80 con 0x5616ac0765e0 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f165f400640 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f16480015c0 tx=0x7f164802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c7fe0a16cfdd1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f1655600640 1 -- 172.21.15.125:0/1582196615 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648044b00 con 0x5616ac0765e0 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f1655600640 1 -- 172.21.15.125:0/1582196615 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1648044c60 con 0x5616ac0765e0 2024-01-16T15:15:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.178+0000 7f1655600640 1 -- 172.21.15.125:0/1582196615 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648044290 con 0x5616ac0765e0 2024-01-16T15:15:14.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 -- 172.21.15.125:0/1582196615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 msgr2=0x5616ac0bb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f16480015c0 tx=0x7f164802f000 comp rx=0 tx=0).stop 2024-01-16T15:15:14.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 -- 172.21.15.125:0/1582196615 shutdown_connections 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac076ea0 0x5616ac0bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac0765e0 0x5616ac0bb780 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 --2- 172.21.15.125:0/1582196615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac071ba0 0x5616ac09f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 -- 172.21.15.125:0/1582196615 >> 172.21.15.125:0/1582196615 conn(0x5616ac038c70 msgr2=0x5616ac0c2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 -- 172.21.15.125:0/1582196615 shutdown_connections 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.179+0000 7f1669dc5740 1 -- 172.21.15.125:0/1582196615 wait complete. 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.180+0000 7f1669dc5740 1 -- start start 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.180+0000 7f1669dc5740 1 Processor -- start 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac0c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.169 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ac127f00 con 0x5616ac076ea0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ac0c0a50 con 0x5616ac071ba0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616ac077c60 con 0x5616ac0765e0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1669dc5740 10 client.0 _ll_register_callbacks cb 0x7f1650007f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1665000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49218/0 (socket says 172.21.15.125:49218) 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1665000640 1 -- 172.21.15.125:0/542278372 learned_addr learned my addr 172.21.15.125:0/542278372 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f165fe00640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1665000640 1 -- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 msgr2=0x5616ac0c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.181+0000 7f1665000640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac0c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1665000640 1 -- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 msgr2=0x5616ac0c04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.170 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1665000640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0c04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1665000640 1 -- 172.21.15.125:0/542278372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1640003520 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1665000640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f1658083040 tx=0x7f165800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7979a41c5c264df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f164d600640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658074070 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1658054b60 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16580567f0 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1658054d20 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1669dc5740 10 client.9300 fetch_fsmap learned FSMap version 73 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5616abf9f170 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.182+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1658064020 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f165805c110 con 0x5616ac076ea0 2024-01-16T15:15:14.171 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 10 client.9300 fetch_fsmap finished waiting for FSMap version 73 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 10 client.9300 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 20 client.9300 populate_metadata read hostname 'smithi125' 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac126e20 0x5616ac127200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 4 client.9300 mds_command: new command op to 9162 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 --> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5616abcde6c0 con 0x5616ac126e20 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f165fe00640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac126e20 0x5616ac127200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f165fe00640 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac126e20 0x5616ac127200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7fd7a4f24e249b server_cookie=f8b68fc6d05c35c7 in_seq=0 out_seq=0 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.183+0000 7f164d600640 10 client.9300 ms_handle_connect on v2:172.21.15.125:6836/2099236429 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f164d600640 1 -- 172.21.15.125:0/542278372 <== mds.0 v2:172.21.15.125:6836/2099236429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5616abcde6c0 con 0x5616ac126e20 2024-01-16T15:15:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f164d600640 10 client.9300 handle_command_reply: tid=0 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 client.9300 shutdown 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 msgr2=0x5616ac116e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f1658083040 tx=0x7f165800df80 comp rx=0 tx=0).stop 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 shutdown_connections 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:6836/2099236429,v1:172.21.15.125:6837/2099236429] conn(0x5616ac126e20 0x5616ac127200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x5616ac076ea0 0x5616ac116e60 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5616ac0765e0 0x5616ac0c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 --2- 172.21.15.125:0/542278372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5616ac071ba0 0x5616ac0c04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f164d600640 0 client.9300 ms_handle_reset on v2:172.21.15.125:6836/2099236429 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 >> 172.21.15.125:0/542278372 conn(0x5616ac038c70 msgr2=0x5616ac06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.173 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 shutdown_connections 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 1 -- 172.21.15.125:0/542278372 wait complete. 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.184+0000 7f1669dc5740 20 client.9300 trim_cache size 0 max 16384 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "3c5ae53a-13b6-44c3-9861-4f92782fd96b", 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 9162, 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2024-01-16T15:15:14.174 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 6.6054590129999999, 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 73, 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 111, 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 111, 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 12.333251439 2024-01-16T15:15:14.175 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 msgr2=0x7f163806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 0x7f163806e8f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f16600695b0 tx=0x7f165000b040 comp rx=0 tx=0).stop 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 msgr2=0x7f166015f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f166006b630 tx=0x7f165800d900 comp rx=0 tx=0).stop 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 shutdown_connections 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f163806c430 0x7f163806e8f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f166015b1b0 0x7f166015f030 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.176 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1660158070 0x7f166015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 --2- 172.21.15.125:0/2332767294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400047c0 0x7f16601570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 >> 172.21.15.125:0/2332767294 conn(0x7f16600cf700 msgr2=0x7f166006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 shutdown_connections 2024-01-16T15:15:14.177 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.185+0000 7f1667000640 1 -- 172.21.15.125:0/2332767294 wait complete. 2024-01-16T15:15:14.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:14.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.395+0000 7f2ef2a00640 1 Processor -- start 2024-01-16T15:15:14.379 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.395+0000 7f2ef2a00640 1 -- start start 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2eec0ae5a0 0x7f2eec15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef2a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eec070c40 con 0x7f2eec0cf290 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eec06c470 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef2a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eec071450 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2eebe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2eec0ae5a0 0x7f2eec15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef0a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2eec0cf290 0x7f2eec0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51936/0 (socket says 172.21.15.125:51936) 2024-01-16T15:15:14.381 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.397+0000 7f2ef0a00640 1 -- 172.21.15.125:0/1628904604 learned_addr learned my addr 172.21.15.125:0/1628904604 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.398+0000 7f2eeb400640 1 -- 172.21.15.125:0/1628904604 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056070917 0 0) 0x7f2eec070c40 con 0x7f2eec0cf290 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.398+0000 7f2eebe00640 1 -- 172.21.15.125:0/1628904604 >> v1:172.21.15.137:6789/0 conn(0x7f2eec0b3ae0 legacy=0x7f2eec06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.398+0000 7f2eebe00640 1 -- 172.21.15.125:0/1628904604 >> v1:172.21.15.125:6789/0 conn(0x7f2eec0cf290 legacy=0x7f2eec0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.398+0000 7f2eebe00640 1 -- 172.21.15.125:0/1628904604 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec06c750 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.399+0000 7f2eebe00640 1 --2- 172.21.15.125:0/1628904604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec15e430 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ca76b1d5467a0d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.399+0000 7f2eeb400640 1 -- 172.21.15.125:0/1628904604 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc030610 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.399+0000 7f2eeb400640 1 -- 172.21.15.125:0/1628904604 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc030770 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.399+0000 7f2eeb400640 1 -- 172.21.15.125:0/1628904604 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc030b00 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 -- 172.21.15.125:0/1628904604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0ae5a0 msgr2=0x7f2eec15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/1628904604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec15e430 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02b960 comp rx=0 tx=0).stop 2024-01-16T15:15:14.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 -- 172.21.15.125:0/1628904604 shutdown_connections 2024-01-16T15:15:14.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/1628904604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec15e430 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 -- 172.21.15.125:0/1628904604 >> 172.21.15.125:0/1628904604 conn(0x7f2eec06a800 msgr2=0x7f2eec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 -- 172.21.15.125:0/1628904604 shutdown_connections 2024-01-16T15:15:14.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.400+0000 7f2ef2a00640 1 -- 172.21.15.125:0/1628904604 wait complete. 2024-01-16T15:15:14.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.402+0000 7f2ef2a00640 1 Processor -- start 2024-01-16T15:15:14.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.402+0000 7f2ef2a00640 1 -- start start 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec0cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eec0cf290 0x7f2eec157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec0c5550 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec0cec50 con 0x7f2eec0ae5a0 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ef2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec0cc9f0 con 0x7f2eec0cf290 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49238/0 (socket says 172.21.15.125:49238) 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 -- 172.21.15.125:0/3857553188 learned_addr learned my addr 172.21.15.125:0/3857553188 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2eebe00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec0cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 -- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eec0cf290 msgr2=0x7f2eec157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eec0cf290 0x7f2eec157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 -- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 msgr2=0x7f2eec0cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec0cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2ee3200640 1 -- 172.21.15.125:0/3857553188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc009400 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.404+0000 7f2eebe00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ee3200640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c430 tx=0x7f2ed400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4011790 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed40118f0 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4011ce0 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec0cccd0 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.405+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec0cd290 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.406+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec0c5700 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.407+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2ed401c460 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.408+0000 7f2ee8c00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 0x7f2ec406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.408+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed40889d0 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.408+0000 7f2eebe00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 0x7f2ec406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.409+0000 7f2eebe00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 0x7f2ec406e920 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2edc036000 tx=0x7f2edc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.412+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2eec0c5700 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.645+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2eec0675d0 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.646+0000 7f2ee8c00640 1 -- 172.21.15.125:0/3857553188 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2ed4054f40 con 0x7f2eec0b3ae0 2024-01-16T15:15:14.629 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 msgr2=0x7f2ec406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 0x7f2ec406e920 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2edc036000 tx=0x7f2edc03b040 comp rx=0 tx=0).stop 2024-01-16T15:15:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 msgr2=0x7f2eec0ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c430 tx=0x7f2ed400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 shutdown_connections 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eec0cf290 0x7f2eec157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2ec406c460 0x7f2ec406e920 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eec0b3ae0 0x7f2eec0ce710 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.648+0000 7f2ef2a00640 1 --2- 172.21.15.125:0/3857553188 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eec0ae5a0 0x7f2eec0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.649+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 >> 172.21.15.125:0/3857553188 conn(0x7f2eec06a800 msgr2=0x7f2eec0b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.649+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 shutdown_connections 2024-01-16T15:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:14.649+0000 7f2ef2a00640 1 -- 172.21.15.125:0/3857553188 wait complete. 2024-01-16T15:15:14.651 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:14.651 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:14.652 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:14.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:14.652 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2024-01-16T15:15:14.701 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:14.702 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 41534sec preferred_lft 41534sec 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:14.703 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a8b3:a6ff:fea3:1380/64 scope link 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether a6:77:c1:13:72:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::a477:c1ff:fe13:724f/64 scope link 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 62:8d:54:84:77:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:14.704 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::608d:54ff:fe84:77ec/64 scope link 2024-01-16T15:15:14.705 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:14.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:14.706 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:14.789 INFO:teuthology.orchestra.run.smithi125.stdout:1 2024-01-16T15:15:14.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:14.794 DEBUG:teuthology.orchestra.run.smithi125:> route 2024-01-16T15:15:14.846 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2024-01-16T15:15:14.847 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:14.847 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:14.847 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:14.847 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:14.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:14.848 DEBUG:teuthology.orchestra.run.smithi125:> ') 2024-01-16T15:15:15.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:15.031 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:15:15.083 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:15:15.084 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:15:15.085 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:15.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:15.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2024-01-16T15:15:15.179 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:15:15.179 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:15.180 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:15.233 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:15.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:15.233 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:15.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:15:15.373 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:15:15.428 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:15.429 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:15.430 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:15.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:15.432 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:15:15.480 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:15.481 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:15:15.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2024-01-16T15:15:15.566 DEBUG:teuthology.orchestra.run.smithi125:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/proc 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/sys 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/dev 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/security 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/shm 2024-01-16T15:15:15.608 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/pts 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/run 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/cgroup 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/pstore 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/ 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/selinux 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/hugepages 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/dev/mqueue 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/debug 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/tracing 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/sys/kernel/config 2024-01-16T15:15:15.609 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/user/1000 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:15.610 INFO:teuthology.orchestra.run.smithi125.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:15.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:15.611 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2024-01-16T15:15:15.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:2024-01-16T15:15:15.692+0000 7fb3dac0d480 -1 init, newargv = 0x5611d17bd9d0 newargc=15 2024-01-16T15:15:15.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[78084]: starting ceph client 2024-01-16T15:15:15.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[78084]: starting fuse 2024-01-16T15:15:15.695 INFO:teuthology.orchestra.run.smithi125.stdout:40 2024-01-16T15:15:15.695 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:15:15.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> import re 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> import os 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2024-01-16T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> return f 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.0")) 2024-01-16T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi125:> ' 2024-01-16T15:15:15.890 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.0.78084.asok 2024-01-16T15:15:15.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:15.907 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78084.asok 2024-01-16T15:15:15.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:15.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78084.asok status 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "0", 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:15:16.141 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "78084", 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 9315, 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 9315 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:15:16.142 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1771157144 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1771157144 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.9315 192.168.0.1:0/1771157144", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/1771157144", 2024-01-16T15:15:16.143 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 73, 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 111, 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2024-01-16T15:15:16.144 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:15:16.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:16.156 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2024-01-16T15:15:16.157 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:16.213 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2024-01-16T15:15:16.213 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:16.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:16.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:16.304 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:16.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.526+0000 7f4929c00640 1 Processor -- start 2024-01-16T15:15:16.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.526+0000 7f4929c00640 1 -- start start 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4929c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f49240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4929c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240cfc30 con 0x7f49240cf7d0 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4929c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240b1ee0 con 0x7f492415e3b0 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4929c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240b2780 con 0x7f4924143ae0 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4919e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f49240cf7d0 0x7f492415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51950/0 (socket says 172.21.15.125:51950) 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.527+0000 7f4919e00640 1 -- 172.21.15.125:0/4200228667 learned_addr learned my addr 172.21.15.125:0/4200228667 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922a00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f49240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682706937 0 0) 0x7f49240cfc30 con 0x7f49240cf7d0 2024-01-16T15:15:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f49240cf7d0 2024-01-16T15:15:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255846632 0 0) 0x7f4904003520 con 0x7f49240cf7d0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 msgr2=0x7f49240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f49240cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 >> v1:172.21.15.137:6789/0 conn(0x7f492415e3b0 legacy=0x7f49240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492415e990 con 0x7f49240cf7d0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f490c001670 con 0x7f49240cf7d0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 >> v1:172.21.15.125:6789/0 conn(0x7f49240cf7d0 legacy=0x7f492415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49040039a0 0x7f4904003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49040047c0 0x7f4904004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f4904005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49240b2780 con 0x7f49040047c0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.528+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490400ba30 con 0x7f49040039a0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490400de00 con 0x7f4924143ae0 2024-01-16T15:15:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49040047c0 0x7f4904004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4922a00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49040039a0 0x7f4904003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4923400640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f4904005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 -- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 msgr2=0x7f4904005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f4904005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 -- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49040039a0 msgr2=0x7f4904003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49040039a0 0x7f4904003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4919e00640 1 -- 172.21.15.125:0/4200228667 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904003520 con 0x7f49040047c0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4922a00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49040039a0 0x7f4904003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.529+0000 7f4923400640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4924143ae0 0x7f4904005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.530+0000 7f4919e00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49040047c0 0x7f4904004c40 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f490c007fd0 tx=0x7f49040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c97d12333fa955 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.530+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c005ad0 con 0x7f49040047c0 2024-01-16T15:15:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.530+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c005c30 con 0x7f49040047c0 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.530+0000 7f4922000640 1 -- 172.21.15.125:0/4200228667 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c02c850 con 0x7f49040047c0 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 -- 172.21.15.125:0/4200228667 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49040047c0 msgr2=0x7f4904004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 --2- 172.21.15.125:0/4200228667 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49040047c0 0x7f4904004c40 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f490c007fd0 tx=0x7f49040057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4923400640 1 -- 172.21.15.125:0/4200228667 reap_dead start 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 -- 172.21.15.125:0/4200228667 shutdown_connections 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 -- 172.21.15.125:0/4200228667 >> 172.21.15.125:0/4200228667 conn(0x7f492406a800 msgr2=0x7f492406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 -- 172.21.15.125:0/4200228667 shutdown_connections 2024-01-16T15:15:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.531+0000 7f4929c00640 1 -- 172.21.15.125:0/4200228667 wait complete. 2024-01-16T15:15:16.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.533+0000 7f4929c00640 1 Processor -- start 2024-01-16T15:15:16.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.533+0000 7f4929c00640 1 -- start start 2024-01-16T15:15:16.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 0x7f49240aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4924137fb0 0x7f49240b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492413a1c0 con 0x7f49240ce540 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49240cda60 con 0x7f4924137fb0 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4929c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240af120 con 0x7f49040047c0 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4919e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4919e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49266/0 (socket says 172.21.15.125:49266) 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.534+0000 7f4919e00640 1 -- 172.21.15.125:0/2574679649 learned_addr learned my addr 172.21.15.125:0/2574679649 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4922a00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 0x7f49240aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4923400640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4924137fb0 0x7f49240b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 -- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 msgr2=0x7f49240aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 0x7f49240aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 -- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4924137fb0 msgr2=0x7f49240b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4924137fb0 0x7f49240b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 -- 172.21.15.125:0/2574679649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240af400 con 0x7f49240ce540 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4922a00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 0x7f49240aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4919e00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f490c02c3f0 tx=0x7f490c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c03c040 con 0x7f49240ce540 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c02cc60 con 0x7f49240ce540 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c007070 con 0x7f49240ce540 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.535+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49240af6e0 con 0x7f49240ce540 2024-01-16T15:15:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.536+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49240afca0 con 0x7f49240ce540 2024-01-16T15:15:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.537+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49240675d0 con 0x7f49240ce540 2024-01-16T15:15:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.538+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f490c032070 con 0x7f49240ce540 2024-01-16T15:15:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.539+0000 7f491b400640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 0x7f48fc06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.539+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f490c0a9dd0 con 0x7f49240ce540 2024-01-16T15:15:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.539+0000 7f4922a00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 0x7f48fc06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.540+0000 7f4922a00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 0x7f48fc06e920 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4914009aa0 tx=0x7f4914008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.543+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f490c076690 con 0x7f49240ce540 2024-01-16T15:15:16.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.779+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f49241410d0 con 0x7f49240ce540 2024-01-16T15:15:16.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.780+0000 7f491b400640 1 -- 172.21.15.125:0/2574679649 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f490c079cb0 con 0x7f49240ce540 2024-01-16T15:15:16.763 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:16.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.782+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 msgr2=0x7f48fc06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.782+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 0x7f48fc06e920 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4914009aa0 tx=0x7f4914008040 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.782+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 msgr2=0x7f4924137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.782+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f490c02c3f0 tx=0x7f490c02e070 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 shutdown_connections 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4924137fb0 0x7f49240b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f48fc06c460 0x7f48fc06e920 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49240ce540 0x7f4924137a70 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 --2- 172.21.15.125:0/2574679649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49040047c0 0x7f49240aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 >> 172.21.15.125:0/2574679649 conn(0x7f492406a800 msgr2=0x7f49240702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 shutdown_connections 2024-01-16T15:15:16.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:16.783+0000 7f4929c00640 1 -- 172.21.15.125:0/2574679649 wait complete. 2024-01-16T15:15:16.784 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:16.784 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:16.785 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:16.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:16.785 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41521sec preferred_lft 41521sec 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.846 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::6028:adff:feab:700b/64 scope link 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 7e:ec:4e:3d:22:92 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::7cec:4eff:fe3d:2292/64 scope link 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 1a:8e:65:a3:f1:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::188e:65ff:fea3:f131/64 scope link 2024-01-16T15:15:16.847 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:16.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:16.849 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:16.941 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-16T15:15:16.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:16.946 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-16T15:15:17.000 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-16T15:15:17.000 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:17.000 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:17.000 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:17.001 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:17.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:17.001 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-16T15:15:17.001 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-16T15:15:17.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:17.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:17.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:17.002 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-16T15:15:17.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:17.180 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:15:17.232 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:15:17.232 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:15:17.233 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:17.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:17.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-16T15:15:17.329 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:15:17.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:17.329 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:17.381 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:17.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:17.382 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:17.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:15:17.530 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:17.582 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:15:17.582 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:15:17.583 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:17.584 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:17.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:17.586 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:15:17.637 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:17.637 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:15:17.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-16T15:15:17.719 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-16T15:15:17.760 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:17.761 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:17.762 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-16T15:15:17.763 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:17.763 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:17.763 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:17.763 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:17.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:17.764 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-16T15:15:17.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-16T15:15:17.844+0000 7f30bf235480 -1 init, newargv = 0x558cb96e4b50 newargc=15 2024-01-16T15:15:17.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47330]: starting ceph client 2024-01-16T15:15:17.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47330]: starting fuse 2024-01-16T15:15:17.846 INFO:teuthology.orchestra.run.smithi137.stdout:41 2024-01-16T15:15:17.846 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:15:17.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:17.846 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-16T15:15:17.846 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-16T15:15:17.847 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-16T15:15:17.848 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-16T15:15:17.849 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-16T15:15:18.033 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.47330.asok 2024-01-16T15:15:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-16T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:18.051 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47330.asok 2024-01-16T15:15:18.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:18.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47330.asok status 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-16T15:15:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "47330", 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 6808, 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 6808 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:15:18.283 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 540771741 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 540771741 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.6808 192.168.0.1:0/540771741", 2024-01-16T15:15:18.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/540771741", 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 73, 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 111, 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-16T15:15:18.285 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-16T15:15:18.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:18.297 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-16T15:15:18.297 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:18.351 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-16T15:15:18.351 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:18.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:18.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:18.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:15:18.666 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.682+0000 7f3cdb000640 1 Processor -- start 2024-01-16T15:15:18.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.682+0000 7f3cdb000640 1 -- start start 2024-01-16T15:15:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cdb000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 0x7f3cd4164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cdb000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40b1ee0 con 0x7f3cd4143ae0 2024-01-16T15:15:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cdb000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40b2780 con 0x7f3cd40cf9a0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cdb000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415e990 con 0x7f3cd415e3b0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cd3e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3cd4143ae0 0x7f3cd40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58210/0 (socket says 172.21.15.125:58210) 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cd3e00640 1 -- 172.21.15.125:0/32943441 learned_addr learned my addr 172.21.15.125:0/32943441 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.684+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282060508 0 0) 0x7f3cd40b1ee0 con 0x7f3cd4143ae0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd9000640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 0x7f3cd4164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4003520 con 0x7f3cd4143ae0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310472834 0 0) 0x7f3cb4003520 con 0x7f3cd4143ae0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 msgr2=0x7f3cd4164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 0x7f3cd4164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 >> v1:172.21.15.137:6789/0 conn(0x7f3cd40cf9a0 legacy=0x7f3cd415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40b2130 con 0x7f3cd4143ae0 2024-01-16T15:15:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3cc4001670 con 0x7f3cd4143ae0 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 >> v1:172.21.15.125:6789/0 conn(0x7f3cd4143ae0 legacy=0x7f3cd40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 0x7f3cb40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb4004d00 0x7f3cb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415e990 con 0x7f3cb4003f20 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb400ba60 con 0x7f3cb4004d00 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb400bc20 con 0x7f3cd415e3b0 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.685+0000 7f3cd3400640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd9000640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb4004d00 0x7f3cb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd3400640 1 -- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 msgr2=0x7f3cb40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd3400640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415e3b0 0x7f3cb40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd3400640 1 -- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb4004d00 msgr2=0x7f3cb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd3400640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cb4004d00 0x7f3cb4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.686+0000 7f3cd3400640 1 -- 172.21.15.125:0/32943441 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb4003520 con 0x7f3cb4003f20 2024-01-16T15:15:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.687+0000 7f3cd3400640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cb40047c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc001750 tx=0x7f3cb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6732aea7c407e504 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.687+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0052d0 con 0x7f3cb4003f20 2024-01-16T15:15:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.687+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc02e050 con 0x7f3cb4003f20 2024-01-16T15:15:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.687+0000 7f3cd2a00640 1 -- 172.21.15.125:0/32943441 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc003830 con 0x7f3cb4003f20 2024-01-16T15:15:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 -- 172.21.15.125:0/32943441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 msgr2=0x7f3cb40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 --2- 172.21.15.125:0/32943441 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cb40047c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc001750 tx=0x7f3cb40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cd9000640 1 -- 172.21.15.125:0/32943441 reap_dead start 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 -- 172.21.15.125:0/32943441 shutdown_connections 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 -- 172.21.15.125:0/32943441 >> 172.21.15.125:0/32943441 conn(0x7f3cd406a800 msgr2=0x7f3cd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 -- 172.21.15.125:0/32943441 shutdown_connections 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.688+0000 7f3cdb000640 1 -- 172.21.15.125:0/32943441 wait complete. 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.690+0000 7f3cdb000640 1 Processor -- start 2024-01-16T15:15:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.690+0000 7f3cdb000640 1 -- start start 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd40b0000 0x7f3cd40b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415c500 0x7f3cd415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd412f300 con 0x7f3cb4003f20 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40b1130 con 0x7f3cd40b0000 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cdb000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40b0b70 con 0x7f3cd415c500 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cd3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cd3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53146/0 (socket says 172.21.15.125:53146) 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.691+0000 7f3cd3e00640 1 -- 172.21.15.125:0/1182340438 learned_addr learned my addr 172.21.15.125:0/1182340438 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 -- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415c500 msgr2=0x7f3cd415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415c500 0x7f3cd415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 -- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd40b0000 msgr2=0x7f3cd40b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd40b0000 0x7f3cd40b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 -- 172.21.15.125:0/1182340438 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40b0e50 con 0x7f3cb4003f20 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cd3e00640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c6c0 tx=0x7f3cc400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40072d0 con 0x7f3cb4003f20 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4007430 con 0x7f3cb4003f20 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40047e0 con 0x7f3cb4003f20 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd4157a30 con 0x7f3cb4003f20 2024-01-16T15:15:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.692+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd41431b0 con 0x7f3cb4003f20 2024-01-16T15:15:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.693+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd406c3f0 con 0x7f3cb4003f20 2024-01-16T15:15:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.695+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f3cc4004940 con 0x7f3cb4003f20 2024-01-16T15:15:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.695+0000 7f3ccbe00640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 0x7f3cac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.696+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cc4087f00 con 0x7f3cb4003f20 2024-01-16T15:15:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.696+0000 7f3cd3400640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 0x7f3cac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.696+0000 7f3cd3400640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 0x7f3cac06e8f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc033000 tx=0x7f3cbc02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.700+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cc4054470 con 0x7f3cb4003f20 2024-01-16T15:15:18.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:18.929+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3cd40675d0 con 0x7f3cb4003f20 2024-01-16T15:15:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.666+0000 7f3ccbe00640 1 -- 172.21.15.125:0/1182340438 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3cc4018020 con 0x7f3cb4003f20 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 msgr2=0x7f3cac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 0x7f3cac06e8f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc033000 tx=0x7f3cbc02f000 comp rx=0 tx=0).stop 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 msgr2=0x7f3cd4142c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c6c0 tx=0x7f3cc400cb80 comp rx=0 tx=0).stop 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 shutdown_connections 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd415c500 0x7f3cd415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd40b0000 0x7f3cd40b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3cac06c430 0x7f3cac06e8f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 --2- 172.21.15.125:0/1182340438 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb4003f20 0x7f3cd4142c70 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 >> 172.21.15.125:0/1182340438 conn(0x7f3cd406a800 msgr2=0x7f3cd406c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.670+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 shutdown_connections 2024-01-16T15:15:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.671+0000 7f3cdb000640 1 -- 172.21.15.125:0/1182340438 wait complete. 2024-01-16T15:15:19.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:15:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.903+0000 7f4b77600640 1 Processor -- start 2024-01-16T15:15:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.904+0000 7f4b77600640 1 -- start start 2024-01-16T15:15:19.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.905+0000 7f4b77600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b70072230 0x7f4b70150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.905+0000 7f4b77600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 0x7f4b700b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.905+0000 7f4b77600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b70071190 con 0x7f4b700cf290 2024-01-16T15:15:19.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.905+0000 7f4b77600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b70072610 con 0x7f4b70072230 2024-01-16T15:15:19.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.905+0000 7f4b77600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70071470 con 0x7f4b7015da10 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.906+0000 7f4b75600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 0x7f4b700b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.906+0000 7f4b74c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b70072230 0x7f4b70150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.906+0000 7f4b75600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 0x7f4b700b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:47332/0 (socket says 172.21.15.125:47332) 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.906+0000 7f4b75600640 1 -- 172.21.15.125:0/3603937332 learned_addr learned my addr 172.21.15.125:0/3603937332 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.906+0000 7f4b67c00640 1 --1- 172.21.15.125:0/3603937332 >> v1:172.21.15.137:6789/0 conn(0x7f4b700cf290 0x7f4b7015d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:51976/0 (socket says 172.21.15.125:51976) 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.907+0000 7f4b74c00640 1 -- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 msgr2=0x7f4b700b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.907+0000 7f4b74c00640 1 --2- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 0x7f4b700b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.907+0000 7f4b74c00640 1 -- 172.21.15.125:0/3603937332 >> v1:172.21.15.137:6789/0 conn(0x7f4b700cf290 legacy=0x7f4b7015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.907+0000 7f4b74c00640 1 -- 172.21.15.125:0/3603937332 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70071750 con 0x7f4b70072230 2024-01-16T15:15:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.908+0000 7f4b74c00640 1 --2- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b70150760 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4b600097a0 tx=0x7f4b6002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8693d8a10e995720 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.909+0000 7f4b6fe00640 1 -- 172.21.15.125:0/3603937332 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60031830 con 0x7f4b70072230 2024-01-16T15:15:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.909+0000 7f4b6fe00640 1 -- 172.21.15.125:0/3603937332 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60031990 con 0x7f4b70072230 2024-01-16T15:15:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.909+0000 7f4b6fe00640 1 -- 172.21.15.125:0/3603937332 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60031d20 con 0x7f4b70072230 2024-01-16T15:15:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 -- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 msgr2=0x7f4b70150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 --2- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b70150760 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4b600097a0 tx=0x7f4b6002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 -- 172.21.15.125:0/3603937332 shutdown_connections 2024-01-16T15:15:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 --2- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7015da10 0x7f4b700b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 --2- 172.21.15.125:0/3603937332 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b70150760 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 -- 172.21.15.125:0/3603937332 >> 172.21.15.125:0/3603937332 conn(0x7f4b7006a800 msgr2=0x7f4b7006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 -- 172.21.15.125:0/3603937332 shutdown_connections 2024-01-16T15:15:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.910+0000 7f4b77600640 1 -- 172.21.15.125:0/3603937332 wait complete. 2024-01-16T15:15:19.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.912+0000 7f4b77600640 1 Processor -- start 2024-01-16T15:15:19.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.912+0000 7f4b77600640 1 -- start start 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.913+0000 7f4b77600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b701f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b77600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700cf290 0x7f4b701f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b77600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b77600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70145210 con 0x7f4b7015da10 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b77600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014bed0 con 0x7f4b70072230 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b77600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014c730 con 0x7f4b700cf290 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53162/0 (socket says 172.21.15.125:53162) 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 -- 172.21.15.125:0/870499232 learned_addr learned my addr 172.21.15.125:0/870499232 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b74c00640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b701f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b67c00640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700cf290 0x7f4b701f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 -- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700cf290 msgr2=0x7f4b701f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700cf290 0x7f4b701f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 -- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 msgr2=0x7f4b701f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b701f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.914+0000 7f4b75600640 1 -- 172.21.15.125:0/870499232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b60009400 con 0x7f4b7015da10 2024-01-16T15:15:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b75600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800e930 tx=0x7f4b6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800de00 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68010440 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7015a4c0 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680107f0 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.915+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7014c3c0 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.916+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7014c8e0 con 0x7f4b7015da10 2024-01-16T15:15:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.917+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4b68010950 con 0x7f4b7015da10 2024-01-16T15:15:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.918+0000 7f4b6d600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 0x7f4b4806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.918+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b68089200 con 0x7f4b7015da10 2024-01-16T15:15:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.918+0000 7f4b74c00640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 0x7f4b4806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.919+0000 7f4b74c00640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 0x7f4b4806e920 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4b60036000 tx=0x7f4b6003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:19.923+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b7014c8e0 con 0x7f4b7015da10 2024-01-16T15:15:20.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.141+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b70071d10 con 0x7f4b4806c460 2024-01-16T15:15:20.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.144+0000 7f4b6d600640 1 -- 172.21.15.125:0/870499232 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4b6003a420 con 0x7f4b4806c460 2024-01-16T15:15:20.132 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:15:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:15:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:15:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:15:20.132 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 msgr2=0x7f4b4806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 0x7f4b4806e920 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4b60036000 tx=0x7f4b6003c040 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 msgr2=0x7f4b70159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800e930 tx=0x7f4b6800edf0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 shutdown_connections 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4b4806c460 0x7f4b4806e920 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7015da10 0x7f4b70159920 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700cf290 0x7f4b701f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 --2- 172.21.15.125:0/870499232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b70072230 0x7f4b701f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 >> 172.21.15.125:0/870499232 conn(0x7f4b7006a800 msgr2=0x7f4b7014f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 shutdown_connections 2024-01-16T15:15:20.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.147+0000 7f4b77600640 1 -- 172.21.15.125:0/870499232 wait complete. 2024-01-16T15:15:20.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:20.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.375+0000 7fc0ae000640 1 Processor -- start 2024-01-16T15:15:20.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.375+0000 7fc0ae000640 1 -- start start 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0ae000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0a813e360 0x7fc0a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 0x7fc0a80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0ae000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a815c5f0 con 0x7fc0a813df80 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0ae000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a806c470 con 0x7fc0a813e360 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0ae000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a806c750 con 0x7fc0a815f9c0 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc09fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0a813e360 0x7fc0a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc09fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0a813e360 0x7fc0a80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53176/0 (socket says 172.21.15.125:53176) 2024-01-16T15:15:20.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc09fe00640 1 -- 172.21.15.125:0/2826544116 learned_addr learned my addr 172.21.15.125:0/2826544116 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.377+0000 7fc0a7e00640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 0x7fc0a80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.378+0000 7fc09fe00640 1 -- 172.21.15.125:0/2826544116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 msgr2=0x7fc0a80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.378+0000 7fc09fe00640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 0x7fc0a80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.378+0000 7fc09fe00640 1 -- 172.21.15.125:0/2826544116 >> v1:172.21.15.137:6789/0 conn(0x7fc0a813df80 legacy=0x7fc0a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.378+0000 7fc09fe00640 1 -- 172.21.15.125:0/2826544116 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a806ca30 con 0x7fc0a813e360 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.378+0000 7fc0a7e00640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 0x7fc0a80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:20.363 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.379+0000 7fc09fe00640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813e360 0x7fc0a80cf290 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc0a806ded0 tx=0x7fc09002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99074255730d55ba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.379+0000 7fc0a6a00640 1 -- 172.21.15.125:0/2826544116 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090031830 con 0x7fc0a813e360 2024-01-16T15:15:20.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.379+0000 7fc0a6a00640 1 -- 172.21.15.125:0/2826544116 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc090031990 con 0x7fc0a813e360 2024-01-16T15:15:20.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.379+0000 7fc0a6a00640 1 -- 172.21.15.125:0/2826544116 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090031d20 con 0x7fc0a813e360 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 -- 172.21.15.125:0/2826544116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813e360 msgr2=0x7fc0a80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813e360 0x7fc0a80cf290 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc0a806ded0 tx=0x7fc09002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 -- 172.21.15.125:0/2826544116 shutdown_connections 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a815f9c0 0x7fc0a80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 --2- 172.21.15.125:0/2826544116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813e360 0x7fc0a80cf290 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 -- 172.21.15.125:0/2826544116 >> 172.21.15.125:0/2826544116 conn(0x7fc0a806a800 msgr2=0x7fc0a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 -- 172.21.15.125:0/2826544116 shutdown_connections 2024-01-16T15:15:20.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.380+0000 7fc0ae000640 1 -- 172.21.15.125:0/2826544116 wait complete. 2024-01-16T15:15:20.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.382+0000 7fc0ae000640 1 Processor -- start 2024-01-16T15:15:20.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.382+0000 7fc0ae000640 1 -- start start 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a813e360 0x7fc0a813ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0a815f9c0 0x7fc0a815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a813c280 con 0x7fc0a813df80 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a815e280 con 0x7fc0a815f9c0 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.383+0000 7fc0ae000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a813b1e0 con 0x7fc0a813e360 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53188/0 (socket says 172.21.15.125:53188) 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 -- 172.21.15.125:0/1911648851 learned_addr learned my addr 172.21.15.125:0/1911648851 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7e00640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0a815f9c0 0x7fc0a815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc09fe00640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a813e360 0x7fc0a813ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 -- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a813e360 msgr2=0x7fc0a813ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a813e360 0x7fc0a813ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 -- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0a815f9c0 msgr2=0x7fc0a815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0a815f9c0 0x7fc0a815dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 -- 172.21.15.125:0/1911648851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0900094d0 con 0x7fc0a813df80 2024-01-16T15:15:20.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.384+0000 7fc0a7400640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fc09800b420 tx=0x7fc09800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.385+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098011830 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.385+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc098011990 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.385+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098011d40 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.385+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a813b4c0 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.385+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a813ba80 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.386+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a80675d0 con 0x7fc0a813df80 2024-01-16T15:15:20.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.387+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc09801b460 con 0x7fc0a813df80 2024-01-16T15:15:20.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.388+0000 7fc09f400640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 0x7fc08006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.389+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc098088630 con 0x7fc0a813df80 2024-01-16T15:15:20.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.389+0000 7fc09fe00640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 0x7fc08006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.389+0000 7fc09fe00640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 0x7fc08006e8f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc090036000 tx=0x7fc09003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.393+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc098054c00 con 0x7fc0a813df80 2024-01-16T15:15:20.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.585+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc0a80cc300 con 0x7fc0a813df80 2024-01-16T15:15:20.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.586+0000 7fc09f400640 1 -- 172.21.15.125:0/1911648851 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc098058220 con 0x7fc0a813df80 2024-01-16T15:15:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.588+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 msgr2=0x7fc08006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.588+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 0x7fc08006e8f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc090036000 tx=0x7fc09003c040 comp rx=0 tx=0).stop 2024-01-16T15:15:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.588+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 msgr2=0x7fc0a8152d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.588+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fc09800b420 tx=0x7fc09800b8e0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 shutdown_connections 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0a815f9c0 0x7fc0a815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0a813e360 0x7fc0a813ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc08006c430 0x7fc08006e8f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 --2- 172.21.15.125:0/1911648851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0a813df80 0x7fc0a8152d90 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 >> 172.21.15.125:0/1911648851 conn(0x7fc0a806a800 msgr2=0x7fc0a80af0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 shutdown_connections 2024-01-16T15:15:20.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.589+0000 7fc0ae000640 1 -- 172.21.15.125:0/1911648851 wait complete. 2024-01-16T15:15:20.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 30 2024-01-16T15:15:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.815+0000 7fcbf9600640 1 Processor -- start 2024-01-16T15:15:20.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.815+0000 7fcbf9600640 1 -- start start 2024-01-16T15:15:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcbf415e3b0 0x7fcbf4164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf9600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf40b1ee0 con 0x7fcbf4143ae0 2024-01-16T15:15:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf40b2780 con 0x7fcbf40cf9a0 2024-01-16T15:15:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf415e990 con 0x7fcbf415e3b0 2024-01-16T15:15:20.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcbf4143ae0 0x7fcbf40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58224/0 (socket says 172.21.15.125:58224) 2024-01-16T15:15:20.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf2a00640 1 -- 172.21.15.125:0/268269654 learned_addr learned my addr 172.21.15.125:0/268269654 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:20.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691547431 0 0) 0x7fcbf40b1ee0 con 0x7fcbf4143ae0 2024-01-16T15:15:20.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4003520 con 0x7fcbf4143ae0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.817+0000 7fcbf3400640 1 --2- 172.21.15.125:0/268269654 >> v2:172.21.15.137:3300/0 conn(0x7fcbf415e3b0 0x7fcbf4164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866076165 0 0) 0x7fcbd4003520 con 0x7fcbf4143ae0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 >> v2:172.21.15.137:3300/0 conn(0x7fcbf415e3b0 msgr2=0x7fcbf4164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 --2- 172.21.15.125:0/268269654 >> v2:172.21.15.137:3300/0 conn(0x7fcbf415e3b0 0x7fcbf4164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 >> v1:172.21.15.137:6789/0 conn(0x7fcbf40cf9a0 legacy=0x7fcbf415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40b2130 con 0x7fcbf4143ae0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcbe40019f0 con 0x7fcbf4143ae0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 >> v1:172.21.15.125:6789/0 conn(0x7fcbf4143ae0 legacy=0x7fcbf40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40039c0 0x7fcbd4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbd40047c0 0x7fcbd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 0x7fcbd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf415e990 con 0x7fcbd40051d0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400c120 con 0x7fcbd40047c0 2024-01-16T15:15:20.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400c2e0 con 0x7fcbd40039c0 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.818+0000 7fcbf3400640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 0x7fcbd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbe9e00640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbd40047c0 0x7fcbd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbf3400640 1 -- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40039c0 msgr2=0x7fcbd4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbf3400640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40039c0 0x7fcbd4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbf3400640 1 -- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbd40047c0 msgr2=0x7fcbd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbf3400640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbd40047c0 0x7fcbd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.819+0000 7fcbf3400640 1 -- 172.21.15.125:0/268269654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd4003520 con 0x7fcbd40051d0 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.820+0000 7fcbf3400640 1 -- 172.21.15.125:0/268269654 reap_dead start 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.820+0000 7fcbf3400640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 0x7fcbd400b980 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00ba30 tx=0x7fcbd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee5a05f8f73fab2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.820+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec004020 con 0x7fcbd40051d0 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.820+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec004180 con 0x7fcbd40051d0 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.820+0000 7fcbf2000640 1 -- 172.21.15.125:0/268269654 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec003610 con 0x7fcbd40051d0 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 -- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 msgr2=0x7fcbd400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 0x7fcbd400b980 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00ba30 tx=0x7fcbd40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 -- 172.21.15.125:0/268269654 shutdown_connections 2024-01-16T15:15:20.806 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 --2- 172.21.15.125:0/268269654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbd40051d0 0x7fcbd400b980 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 -- 172.21.15.125:0/268269654 >> 172.21.15.125:0/268269654 conn(0x7fcbf406a800 msgr2=0x7fcbf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 -- 172.21.15.125:0/268269654 shutdown_connections 2024-01-16T15:15:20.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.821+0000 7fcbf9600640 1 -- 172.21.15.125:0/268269654 wait complete. 2024-01-16T15:15:20.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.823+0000 7fcbf9600640 1 Processor -- start 2024-01-16T15:15:20.807 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.823+0000 7fcbf9600640 1 -- start start 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40051d0 0x7fcbf4140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbf415a4e0 0x7fcbf415a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf415aea0 con 0x7fcbf40c55a0 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40c5140 con 0x7fcbf415a4e0 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40c3e50 con 0x7fcbd40051d0 2024-01-16T15:15:20.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53204/0 (socket says 172.21.15.125:53204) 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 -- 172.21.15.125:0/2676827780 learned_addr learned my addr 172.21.15.125:0/2676827780 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbe9e00640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbf415a4e0 0x7fcbf415a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 -- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40051d0 msgr2=0x7fcbf4140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40051d0 0x7fcbf4140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 -- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbf415a4e0 msgr2=0x7fcbf415a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.824+0000 7fcbf3400640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbf415a4e0 0x7fcbf415a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbf3400640 1 -- 172.21.15.125:0/2676827780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40c5f90 con 0x7fcbf40c55a0 2024-01-16T15:15:20.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbf3400640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fcbec02bfd0 tx=0x7fcbec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec004500 con 0x7fcbf40c55a0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec032480 con 0x7fcbf40c55a0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec032610 con 0x7fcbf40c55a0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf40c40a0 con 0x7fcbf40c55a0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.825+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf40c4660 con 0x7fcbf40c55a0 2024-01-16T15:15:20.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.826+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf40675d0 con 0x7fcbf40c55a0 2024-01-16T15:15:20.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.827+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fcbec032770 con 0x7fcbf40c55a0 2024-01-16T15:15:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.828+0000 7fcbeb400640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 0x7fcbcc06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.828+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbec0aa4c0 con 0x7fcbf40c55a0 2024-01-16T15:15:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.828+0000 7fcbf2a00640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 0x7fcbcc06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.829+0000 7fcbf2a00640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 0x7fcbcc06e910 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcbf41f3aa0 tx=0x7fcbe40018c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.817 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:20.832+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbec076d80 con 0x7fcbf40c55a0 2024-01-16T15:15:21.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.022+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fcbf4067ce0 con 0x7fcbf40c55a0 2024-01-16T15:15:21.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.028+0000 7fcbeb400640 1 -- 172.21.15.125:0/2676827780 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v43)=0 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7fcbec0400c0 con 0x7fcbf40c55a0 2024-01-16T15:15:21.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.031+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 msgr2=0x7fcbcc06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.031+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 0x7fcbcc06e910 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcbf41f3aa0 tx=0x7fcbe40018c0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.031+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 msgr2=0x7fcbf40c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.016 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.031+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fcbec02bfd0 tx=0x7fcbec004730 comp rx=0 tx=0).stop 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 shutdown_connections 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcbcc06c450 0x7fcbcc06e910 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcbf40c55a0 0x7fcbf40c5a50 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbf415a4e0 0x7fcbf415a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 --2- 172.21.15.125:0/2676827780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbd40051d0 0x7fcbf4140e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 >> 172.21.15.125:0/2676827780 conn(0x7fcbf406a800 msgr2=0x7fcbf40ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 shutdown_connections 2024-01-16T15:15:21.017 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.032+0000 7fcbf9600640 1 -- 172.21.15.125:0/2676827780 wait complete. 2024-01-16T15:15:21.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:15:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.259+0000 7f4245e00640 1 Processor -- start 2024-01-16T15:15:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.259+0000 7f4245e00640 1 -- start start 2024-01-16T15:15:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f4245e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f42400ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f4245e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42400aede0 0x7f424006f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f4245e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4240071de0 con 0x7f42400d0b60 2024-01-16T15:15:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f4245e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400af270 con 0x7f42400aede0 2024-01-16T15:15:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f4245e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4240072030 con 0x7f424006a3e0 2024-01-16T15:15:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f423e000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f42400d0b60 0x7f424006a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58236/0 (socket says 172.21.15.125:58236) 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f423e000640 1 -- 172.21.15.125:0/3545265550 learned_addr learned my addr 172.21.15.125:0/3545265550 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.261+0000 7f423f400640 1 --2- 172.21.15.125:0/3545265550 >> v2:172.21.15.125:3300/0 conn(0x7f42400aede0 0x7f424006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423ea00640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f42400ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423d600640 1 -- 172.21.15.125:0/3545265550 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882269924 0 0) 0x7f4240071de0 con 0x7f42400d0b60 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423d600640 1 -- 172.21.15.125:0/3545265550 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003520 con 0x7f42400d0b60 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423f400640 1 -- 172.21.15.125:0/3545265550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 msgr2=0x7f42400ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423f400640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f42400ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423f400640 1 -- 172.21.15.125:0/3545265550 >> v1:172.21.15.125:6789/0 conn(0x7f42400d0b60 legacy=0x7f424006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423f400640 1 -- 172.21.15.125:0/3545265550 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240072310 con 0x7f42400aede0 2024-01-16T15:15:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.262+0000 7f423ea00640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f42400ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.263+0000 7f423f400640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f424006f080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f42380097a0 tx=0x7f423802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffef98ca1675a2b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f423d600640 1 -- 172.21.15.125:0/3545265550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423802d040 con 0x7f42400aede0 2024-01-16T15:15:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f423d600640 1 -- 172.21.15.125:0/3545265550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238031810 con 0x7f42400aede0 2024-01-16T15:15:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f423d600640 1 -- 172.21.15.125:0/3545265550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238031ba0 con 0x7f42400aede0 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 -- 172.21.15.125:0/3545265550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 msgr2=0x7f424006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f424006f080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f42380097a0 tx=0x7f423802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 -- 172.21.15.125:0/3545265550 shutdown_connections 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f424006f080 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 --2- 172.21.15.125:0/3545265550 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f42400ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 -- 172.21.15.125:0/3545265550 >> 172.21.15.125:0/3545265550 conn(0x7f42400cfef0 msgr2=0x7f42400d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 -- 172.21.15.125:0/3545265550 shutdown_connections 2024-01-16T15:15:21.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.264+0000 7f4245e00640 1 -- 172.21.15.125:0/3545265550 wait complete. 2024-01-16T15:15:21.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.266+0000 7f4245e00640 1 Processor -- start 2024-01-16T15:15:21.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.266+0000 7f4245e00640 1 -- start start 2024-01-16T15:15:21.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f424015acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42400d0b60 0x7f4240147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424015a1f0 con 0x7f42400aede0 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240159150 con 0x7f42400d0b60 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.267+0000 7f4245e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4240159430 con 0x7f424006a3e0 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53224/0 (socket says 172.21.15.125:53224) 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 -- 172.21.15.125:0/4000854679 learned_addr learned my addr 172.21.15.125:0/4000854679 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423ea00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f424015acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 -- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 msgr2=0x7f424015acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f424015acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 -- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42400d0b60 msgr2=0x7f4240147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423f400640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42400d0b60 0x7f4240147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42400d0b60 0x7f4240147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 -- 172.21.15.125:0/4000854679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238009400 con 0x7f42400aede0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.268+0000 7f423e000640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f422800c4d0 tx=0x7f422800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.269+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228005ce0 con 0x7f42400aede0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.269+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228005e40 con 0x7f42400aede0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.269+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228007270 con 0x7f42400aede0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.269+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4240159710 con 0x7f42400aede0 2024-01-16T15:15:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.269+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4240159cd0 con 0x7f42400aede0 2024-01-16T15:15:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.270+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240159ef0 con 0x7f42400aede0 2024-01-16T15:15:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.271+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f42280073d0 con 0x7f42400aede0 2024-01-16T15:15:21.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.272+0000 7f4237400640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 0x7f421806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.272+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4228087ed0 con 0x7f42400aede0 2024-01-16T15:15:21.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.272+0000 7f423ea00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 0x7f421806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.273+0000 7f423ea00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 0x7f421806e8f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4230003fd0 tx=0x7f4230003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.276+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4228054550 con 0x7f42400aede0 2024-01-16T15:15:21.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.468+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f42400675d0 con 0x7f42400aede0 2024-01-16T15:15:21.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.475+0000 7f4237400640 1 -- 172.21.15.125:0/4000854679 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v44)=0 v44) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4228017070 con 0x7f42400aede0 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 msgr2=0x7f421806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 0x7f421806e8f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4230003fd0 tx=0x7f4230003990 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 msgr2=0x7f4240158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f422800c4d0 tx=0x7f422800c990 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 shutdown_connections 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42400d0b60 0x7f4240147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f421806c430 0x7f421806e8f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42400aede0 0x7f4240158670 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 --2- 172.21.15.125:0/4000854679 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f424006a3e0 0x7f424015acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 >> 172.21.15.125:0/4000854679 conn(0x7f42400cfef0 msgr2=0x7f42400b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 shutdown_connections 2024-01-16T15:15:21.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.478+0000 7f4245e00640 1 -- 172.21.15.125:0/4000854679 wait complete. 2024-01-16T15:15:21.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:15:21.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.704+0000 7f6225c00640 1 Processor -- start 2024-01-16T15:15:21.688 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.704+0000 7f6225c00640 1 -- start start 2024-01-16T15:15:21.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.705+0000 7f6225c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f62200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f6225c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200cfc30 con 0x7f622015e3b0 2024-01-16T15:15:21.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f6225c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200b1ee0 con 0x7f62200cf7d0 2024-01-16T15:15:21.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f6225c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62200b2780 con 0x7f6220143ae0 2024-01-16T15:15:21.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f621f400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f622015e3b0 0x7f62200702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58246/0 (socket says 172.21.15.125:58246) 2024-01-16T15:15:21.691 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f621f400640 1 -- 172.21.15.125:0/428844273 learned_addr learned my addr 172.21.15.125:0/428844273 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f621ea00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f62200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.706+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345727750 0 0) 0x7f62200cfc30 con 0x7f622015e3b0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6200003520 con 0x7f622015e3b0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 181128898 0 0) 0x7f6200003520 con 0x7f622015e3b0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 msgr2=0x7f62200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f62200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 >> v1:172.21.15.137:6789/0 conn(0x7f62200cf7d0 legacy=0x7f622015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622015e990 con 0x7f622015e3b0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f62180019f0 con 0x7f622015e3b0 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 >> v1:172.21.15.125:6789/0 conn(0x7f622015e3b0 legacy=0x7f62200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.693 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62000039a0 0x7f6200003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62000047c0 0x7f6200004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f6200005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62200b2780 con 0x7f62000039a0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620000ba30 con 0x7f62000047c0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620000de00 con 0x7f6220143ae0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.707+0000 7f621ea00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62000039a0 0x7f6200003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621f400640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f6200005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f6217e00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62000047c0 0x7f6200004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621ea00640 1 -- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 msgr2=0x7f6200005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621ea00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6220143ae0 0x7f6200005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621ea00640 1 -- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62000047c0 msgr2=0x7f6200004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621ea00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62000047c0 0x7f6200004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f621ea00640 1 -- 172.21.15.125:0/428844273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200003520 con 0x7f62000039a0 2024-01-16T15:15:21.694 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.708+0000 7f6217e00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62000047c0 0x7f6200004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.709+0000 7f621ea00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62000039a0 0x7f6200003dc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f6210009a90 tx=0x7f62000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d422abed9003a53d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.709+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621002c680 con 0x7f62000039a0 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.709+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621002c7e0 con 0x7f62000039a0 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.709+0000 7f621e000640 1 -- 172.21.15.125:0/428844273 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621002cb70 con 0x7f62000039a0 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 -- 172.21.15.125:0/428844273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62000039a0 msgr2=0x7f6200003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 --2- 172.21.15.125:0/428844273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62000039a0 0x7f6200003dc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f6210009a90 tx=0x7f62000057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f621f400640 1 -- 172.21.15.125:0/428844273 reap_dead start 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 -- 172.21.15.125:0/428844273 shutdown_connections 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 -- 172.21.15.125:0/428844273 >> 172.21.15.125:0/428844273 conn(0x7f622006a800 msgr2=0x7f622006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 -- 172.21.15.125:0/428844273 shutdown_connections 2024-01-16T15:15:21.696 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.710+0000 7f6225c00640 1 -- 172.21.15.125:0/428844273 wait complete. 2024-01-16T15:15:21.697 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.712+0000 7f6225c00640 1 Processor -- start 2024-01-16T15:15:21.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.712+0000 7f6225c00640 1 -- start start 2024-01-16T15:15:21.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62000039a0 0x7f62201f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6220137460 0x7f62201378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.698 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62200af920 con 0x7f62200cdc70 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220137e20 con 0x7f6220137460 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6225c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6220139db0 con 0x7f62000039a0 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53254/0 (socket says 172.21.15.125:53254) 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 -- 172.21.15.125:0/1155406028 learned_addr learned my addr 172.21.15.125:0/1155406028 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f621ea00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62000039a0 0x7f62201f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 -- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62000039a0 msgr2=0x7f62201f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62000039a0 0x7f62201f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 -- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6220137460 msgr2=0x7f62201378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6220137460 0x7f62201378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.713+0000 7f6217e00640 1 -- 172.21.15.125:0/1155406028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62201394c0 con 0x7f62200cdc70 2024-01-16T15:15:21.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6217e00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f620800d7d0 tx=0x7f620800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208017070 con 0x7f62200cdc70 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208005070 con 0x7f62200cdc70 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208005440 con 0x7f62200cdc70 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62201f3150 con 0x7f62200cdc70 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.714+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6220139760 con 0x7f62200cdc70 2024-01-16T15:15:21.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.715+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6220139f40 con 0x7f62200cdc70 2024-01-16T15:15:21.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.716+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6208007030 con 0x7f62200cdc70 2024-01-16T15:15:21.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.717+0000 7f6216a00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 0x7f61f806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.717+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6208088390 con 0x7f62200cdc70 2024-01-16T15:15:21.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.717+0000 7f621ea00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 0x7f61f806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.718+0000 7f621ea00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 0x7f61f806e970 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6210036000 tx=0x7f6210030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.721+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6208054900 con 0x7f62200cdc70 2024-01-16T15:15:21.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.938+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f62200b2410 con 0x7f61f806c4b0 2024-01-16T15:15:21.930 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:15:21.946+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:21.930 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:15:21.946+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:21.930 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:15:21.946+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:21.930 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:15:21.946+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:21.930 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:15:21.946+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:21.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.989+0000 7f6216a00640 1 -- 172.21.15.125:0/1155406028 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6210004020 con 0x7f61f806c4b0 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 msgr2=0x7f61f806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 0x7f61f806e970 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6210036000 tx=0x7f6210030000 comp rx=0 tx=0).stop 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 msgr2=0x7f622013a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f620800d7d0 tx=0x7f620800dc90 comp rx=0 tx=0).stop 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 shutdown_connections 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6220137460 0x7f62201378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f61f806c4b0 0x7f61f806e970 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62200cdc70 0x7f622013a240 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 --2- 172.21.15.125:0/1155406028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62000039a0 0x7f62201f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 >> 172.21.15.125:0/1155406028 conn(0x7f622006a800 msgr2=0x7f622015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 shutdown_connections 2024-01-16T15:15:21.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:21.992+0000 7f6225c00640 1 -- 172.21.15.125:0/1155406028 wait complete. 2024-01-16T15:15:21.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:22.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.213+0000 7f7603e00640 1 Processor -- start 2024-01-16T15:15:22.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.213+0000 7f7603e00640 1 -- start start 2024-01-16T15:15:22.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7603e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f75fc13df80 0x7f75fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7603e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75fc0cf9a0 0x7f75fc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7603e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc06c470 con 0x7f75fc13e360 2024-01-16T15:15:22.200 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7603e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75fc06c750 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7603e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75fc06ca30 con 0x7f75fc13df80 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7601e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75fc0cf9a0 0x7f75fc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7600a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f75fc13e360 0x7f75fc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58260/0 (socket says 172.21.15.125:58260) 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.215+0000 7f7600a00640 1 -- 172.21.15.125:0/3224084350 learned_addr learned my addr 172.21.15.125:0/3224084350 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f7601400640 1 --2- 172.21.15.125:0/3224084350 >> v2:172.21.15.137:3300/0 conn(0x7f75fc13df80 0x7f75fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f7601e00640 1 -- 172.21.15.125:0/3224084350 >> v2:172.21.15.137:3300/0 conn(0x7f75fc13df80 msgr2=0x7f75fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f7601e00640 1 --2- 172.21.15.125:0/3224084350 >> v2:172.21.15.137:3300/0 conn(0x7f75fc13df80 0x7f75fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f7601e00640 1 -- 172.21.15.125:0/3224084350 >> v1:172.21.15.125:6789/0 conn(0x7f75fc13e360 legacy=0x7f75fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f7601e00640 1 -- 172.21.15.125:0/3224084350 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc06cd10 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.216+0000 7f75fbe00640 1 -- 172.21.15.125:0/3224084350 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836209176 0 0) 0x7f75fc06c470 con 0x7f75fc13e360 2024-01-16T15:15:22.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.217+0000 7f7601e00640 1 --2- 172.21.15.125:0/3224084350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cd840 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f75fc072770 tx=0x7f75f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f1564a36ec1dbb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.217+0000 7f75fbe00640 1 -- 172.21.15.125:0/3224084350 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4031830 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.217+0000 7f75fbe00640 1 -- 172.21.15.125:0/3224084350 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4031990 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.217+0000 7f75fbe00640 1 -- 172.21.15.125:0/3224084350 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4031d20 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 -- 172.21.15.125:0/3224084350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc0cf9a0 msgr2=0x7f75fc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 --2- 172.21.15.125:0/3224084350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cd840 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f75fc072770 tx=0x7f75f402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 -- 172.21.15.125:0/3224084350 shutdown_connections 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 --2- 172.21.15.125:0/3224084350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cd840 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 --2- 172.21.15.125:0/3224084350 >> v2:172.21.15.137:3300/0 conn(0x7f75fc13df80 0x7f75fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 -- 172.21.15.125:0/3224084350 >> 172.21.15.125:0/3224084350 conn(0x7f75fc06a800 msgr2=0x7f75fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 -- 172.21.15.125:0/3224084350 shutdown_connections 2024-01-16T15:15:22.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.218+0000 7f7603e00640 1 -- 172.21.15.125:0/3224084350 wait complete. 2024-01-16T15:15:22.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.220+0000 7f7603e00640 1 Processor -- start 2024-01-16T15:15:22.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.220+0000 7f7603e00640 1 -- start start 2024-01-16T15:15:22.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 0x7f75fc1f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.206 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1f3e70 con 0x7f75fc13e360 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc148360 con 0x7f75fc0cf9a0 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7603e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0aa3c0 con 0x7f75fc13df80 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7601e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7601e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53288/0 (socket says 172.21.15.125:53288) 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7601e00640 1 -- 172.21.15.125:0/3290121782 learned_addr learned my addr 172.21.15.125:0/3290121782 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.221+0000 7f7601400640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7600a00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 0x7f75fc1f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 -- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 msgr2=0x7f75fc1f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 0x7f75fc1f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 -- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc0cf9a0 msgr2=0x7f75fc0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 -- 172.21.15.125:0/3290121782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4009400 con 0x7f75fc13e360 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7600a00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 0x7f75fc1f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7601e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f75f402b910 tx=0x7f75f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f403a9e0 con 0x7f75fc13e360 2024-01-16T15:15:22.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f403ab60 con 0x7f75fc13e360 2024-01-16T15:15:22.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40443f0 con 0x7f75fc13e360 2024-01-16T15:15:22.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc1f35d0 con 0x7f75fc13e360 2024-01-16T15:15:22.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.222+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc1f3b90 con 0x7f75fc13e360 2024-01-16T15:15:22.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.224+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc1484d0 con 0x7f75fc13e360 2024-01-16T15:15:22.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.224+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f75f4044550 con 0x7f75fc13e360 2024-01-16T15:15:22.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.225+0000 7f75f9600640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 0x7f75d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.225+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75f40aade0 con 0x7f75fc13e360 2024-01-16T15:15:22.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.226+0000 7f7601400640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 0x7f75d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.226+0000 7f7601400640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 0x7f75d406e8f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009a60 tx=0x7f75ec003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.230+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75f4077470 con 0x7f75fc13e360 2024-01-16T15:15:22.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.447+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f75fc0ccef0 con 0x7f75d406c430 2024-01-16T15:15:22.441 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.456+0000 7f75f9600640 1 -- 172.21.15.125:0/3290121782 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f75fc0ccef0 con 0x7f75d406c430 2024-01-16T15:15:22.441 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 msgr2=0x7f75d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 0x7f75d406e8f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009a60 tx=0x7f75ec003990 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 msgr2=0x7f75fc0a9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f75f402b910 tx=0x7f75f40046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 shutdown_connections 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f75d406c430 0x7f75d406e8f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75fc13e360 0x7f75fc0a9e80 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75fc13df80 0x7f75fc1f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 --2- 172.21.15.125:0/3290121782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75fc0cf9a0 0x7f75fc0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.444 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 >> 172.21.15.125:0/3290121782 conn(0x7f75fc06a800 msgr2=0x7f75fc071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 shutdown_connections 2024-01-16T15:15:22.445 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.459+0000 7f7603e00640 1 -- 172.21.15.125:0/3290121782 wait complete. 2024-01-16T15:15:22.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.682+0000 7fcf35c00640 1 Processor -- start 2024-01-16T15:15:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.682+0000 7fcf35c00640 1 -- start start 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf35c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcf30143ae0 0x7fcf300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf35c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf35c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300cfc30 con 0x7fcf300cf7d0 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf35c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300b1ee0 con 0x7fcf30143ae0 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf35c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300b2780 con 0x7fcf3015e3b0 2024-01-16T15:15:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf27e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcf300cf7d0 0x7fcf3015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58266/0 (socket says 172.21.15.125:58266) 2024-01-16T15:15:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf27e00640 1 -- 172.21.15.125:0/1948559540 learned_addr learned my addr 172.21.15.125:0/1948559540 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.684+0000 7fcf2ea00640 1 --2- 172.21.15.125:0/1948559540 >> v2:172.21.15.137:3300/0 conn(0x7fcf30143ae0 0x7fcf300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2f400640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157729372 0 0) 0x7fcf300cfc30 con 0x7fcf300cf7d0 2024-01-16T15:15:22.670 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf10005730 con 0x7fcf300cf7d0 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737422781 0 0) 0x7fcf10005730 con 0x7fcf300cf7d0 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 msgr2=0x7fcf300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 >> v2:172.21.15.137:3300/0 conn(0x7fcf30143ae0 msgr2=0x7fcf300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 --2- 172.21.15.125:0/1948559540 >> v2:172.21.15.137:3300/0 conn(0x7fcf30143ae0 0x7fcf300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3015e990 con 0x7fcf300cf7d0 2024-01-16T15:15:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcf18001670 con 0x7fcf300cf7d0 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 >> v1:172.21.15.125:6789/0 conn(0x7fcf300cf7d0 legacy=0x7fcf3015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf10005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf10006100 0x7fcf100069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf10006f10 0x7fcf10007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300b1ee0 con 0x7fcf10006f10 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300b2780 con 0x7fcf10006100 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.685+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1000db90 con 0x7fcf3015e3b0 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf10006f10 0x7fcf10007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf27e00640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf10006100 0x7fcf100069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2ea00640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf10005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 -- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 msgr2=0x7fcf10005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf3015e3b0 0x7fcf10005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 -- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf10006100 msgr2=0x7fcf100069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf10006100 0x7fcf100069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf2f400640 1 -- 172.21.15.125:0/1948559540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf10005730 con 0x7fcf10006f10 2024-01-16T15:15:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.686+0000 7fcf27e00640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf10006100 0x7fcf100069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.687+0000 7fcf2f400640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf10006f10 0x7fcf10007390 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009fc0 tx=0x7fcf10007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1691914b03f76f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.687+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2802cc00 con 0x7fcf10006f10 2024-01-16T15:15:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.687+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2800f040 con 0x7fcf10006f10 2024-01-16T15:15:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.687+0000 7fcf2e000640 1 -- 172.21.15.125:0/1948559540 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28034440 con 0x7fcf10006f10 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 -- 172.21.15.125:0/1948559540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf10006f10 msgr2=0x7fcf10007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 --2- 172.21.15.125:0/1948559540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf10006f10 0x7fcf10007390 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009fc0 tx=0x7fcf10007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf2f400640 1 -- 172.21.15.125:0/1948559540 reap_dead start 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 -- 172.21.15.125:0/1948559540 shutdown_connections 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 -- 172.21.15.125:0/1948559540 >> 172.21.15.125:0/1948559540 conn(0x7fcf3006a800 msgr2=0x7fcf3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 -- 172.21.15.125:0/1948559540 shutdown_connections 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.688+0000 7fcf35c00640 1 -- 172.21.15.125:0/1948559540 wait complete. 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.690+0000 7fcf35c00640 1 Processor -- start 2024-01-16T15:15:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.690+0000 7fcf35c00640 1 -- start start 2024-01-16T15:15:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf10006f10 0x7fcf3013b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 0x7fcf30158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3013ef90 con 0x7fcf301596e0 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3013f270 con 0x7fcf30159c30 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf35c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301409c0 con 0x7fcf10006f10 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.691+0000 7fcf27e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53312/0 (socket says 172.21.15.125:53312) 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 -- 172.21.15.125:0/3803386320 learned_addr learned my addr 172.21.15.125:0/3803386320 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf2ea00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf10006f10 0x7fcf3013b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf2f400640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 0x7fcf30158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 -- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf10006f10 msgr2=0x7fcf3013b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf10006f10 0x7fcf3013b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 -- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 msgr2=0x7fcf30158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 0x7fcf30158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 -- 172.21.15.125:0/3803386320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30140ca0 con 0x7fcf301596e0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf2f400640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 0x7fcf30158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf27e00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800e870 tx=0x7fcf1800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800c8d0 con 0x7fcf301596e0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1800ca30 con 0x7fcf301596e0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800cda0 con 0x7fcf301596e0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.692+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf30140f80 con 0x7fcf301596e0 2024-01-16T15:15:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.693+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3013ec20 con 0x7fcf301596e0 2024-01-16T15:15:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.694+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf30140820 con 0x7fcf301596e0 2024-01-16T15:15:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.695+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fcf1801d460 con 0x7fcf301596e0 2024-01-16T15:15:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.696+0000 7fcf26a00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 0x7fcf0806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.696+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf18088ae0 con 0x7fcf301596e0 2024-01-16T15:15:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.696+0000 7fcf2ea00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 0x7fcf0806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.697+0000 7fcf2ea00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 0x7fcf0806e8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099d0 tx=0x7fcf20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.701+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf30140820 con 0x7fcf301596e0 2024-01-16T15:15:22.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.920+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf30140b30 con 0x7fcf0806c430 2024-01-16T15:15:22.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.931+0000 7fcf26a00640 1 -- 172.21.15.125:0/3803386320 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fcf30140b30 con 0x7fcf0806c430 2024-01-16T15:15:22.916 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.933+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 msgr2=0x7fcf0806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.933+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 0x7fcf0806e8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099d0 tx=0x7fcf20008040 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.933+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 msgr2=0x7fcf300b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.933+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800e870 tx=0x7fcf1800ed30 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 shutdown_connections 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf30159c30 0x7fcf30158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcf0806c430 0x7fcf0806e8f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf301596e0 0x7fcf300b0e30 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 --2- 172.21.15.125:0/3803386320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf10006f10 0x7fcf3013b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 >> 172.21.15.125:0/3803386320 conn(0x7fcf3006a800 msgr2=0x7fcf3015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 shutdown_connections 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:22.934+0000 7fcf35c00640 1 -- 172.21.15.125:0/3803386320 wait complete. 2024-01-16T15:15:22.936 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:15:22.936 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:15:22.936 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:22.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:22.985 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:36.879 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:15:36.879 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:15:36.879 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.7777 s, 77.9 MB/s 2024-01-16T15:15:36.880 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:15:36.881 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:36.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:36.935 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:42.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:42.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:47.799 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:15:47.800 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:15:47.800 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8035 s, 99.4 MB/s 2024-01-16T15:15:47.801 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:15:47.801 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:47.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.855 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:58.558 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records in 2024-01-16T15:15:58.559 INFO:teuthology.orchestra.run.smithi125.stderr:1024+0 records out 2024-01-16T15:15:58.559 INFO:teuthology.orchestra.run.smithi125.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.6447 s, 101 MB/s 2024-01-16T15:15:58.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:15:58.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.798+0000 7f5c13e00640 1 Processor -- start 2024-01-16T15:15:58.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.798+0000 7f5c13e00640 1 -- start start 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c13e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5c1415dca0 0x7f5c140cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 0x7f5c1406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c13e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c140ce370 con 0x7f5c140aeac0 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c13e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1415e920 con 0x7f5c1415dca0 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c13e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140cddb0 con 0x7f5c1415acc0 2024-01-16T15:15:58.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c12000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5c140aeac0 0x7f5c1415e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38232/0 (socket says 172.21.15.125:38232) 2024-01-16T15:15:58.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.800+0000 7f5c12000640 1 -- 172.21.15.125:0/995150735 learned_addr learned my addr 172.21.15.125:0/995150735 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:58.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c12a00640 1 --2- 172.21.15.125:0/995150735 >> v2:172.21.15.137:3300/0 conn(0x7f5c1415dca0 0x7f5c140cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404550062 0 0) 0x7f5c140ce370 con 0x7f5c140aeac0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c13400640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 0x7f5c1406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf4003520 con 0x7f5c140aeac0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1859890157 0 0) 0x7f5bf4003520 con 0x7f5c140aeac0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 msgr2=0x7f5c1406cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 0x7f5c1406cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 >> v2:172.21.15.137:3300/0 conn(0x7f5c1415dca0 msgr2=0x7f5c140cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 --2- 172.21.15.125:0/995150735 >> v2:172.21.15.137:3300/0 conn(0x7f5c1415dca0 0x7f5c140cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c140ce090 con 0x7f5c140aeac0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f5bfc0017f0 con 0x7f5c140aeac0 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 >> v1:172.21.15.125:6789/0 conn(0x7f5c140aeac0 legacy=0x7f5c1415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bf4003970 0x7f5bf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.801+0000 7f5c11600640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf40047c0 0x7f5bf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c11600640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 0x7f5bf4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1415e920 con 0x7f5bf4003970 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140cddb0 con 0x7f5bf40047c0 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf400ddd0 con 0x7f5c1415acc0 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bf4003970 0x7f5bf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12000640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf40047c0 0x7f5bf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 -- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 msgr2=0x7f5bf4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1415acc0 0x7f5bf4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 -- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf40047c0 msgr2=0x7f5bf4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf40047c0 0x7f5bf4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.789 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.802+0000 7f5c12a00640 1 -- 172.21.15.125:0/995150735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf4003520 con 0x7f5bf4003970 2024-01-16T15:15:58.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.803+0000 7f5c12a00640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bf4003970 0x7f5bf4003d90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba30 tx=0x7f5bf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3442d5944e9d7355 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.803+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04033650 con 0x7f5bf4003970 2024-01-16T15:15:58.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c040337b0 con 0x7f5bf4003970 2024-01-16T15:15:58.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c11600640 1 -- 172.21.15.125:0/995150735 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04033b40 con 0x7f5bf4003970 2024-01-16T15:15:58.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 -- 172.21.15.125:0/995150735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bf4003970 msgr2=0x7f5bf4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 --2- 172.21.15.125:0/995150735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bf4003970 0x7f5bf4003d90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba30 tx=0x7f5bf40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13400640 1 -- 172.21.15.125:0/995150735 reap_dead start 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 -- 172.21.15.125:0/995150735 shutdown_connections 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 -- 172.21.15.125:0/995150735 >> 172.21.15.125:0/995150735 conn(0x7f5c140cc1d0 msgr2=0x7f5c140cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 -- 172.21.15.125:0/995150735 shutdown_connections 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.804+0000 7f5c13e00640 1 -- 172.21.15.125:0/995150735 wait complete. 2024-01-16T15:15:58.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.807+0000 7f5c13e00640 1 Processor -- start 2024-01-16T15:15:58.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.807+0000 7f5c13e00640 1 -- start start 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf4003970 0x7f5c140adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c14159cf0 0x7f5c1414de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c141575b0 con 0x7f5c1414ee20 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14155da0 con 0x7f5bf4003970 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c13e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14157850 con 0x7f5c14159cf0 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c12000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c12000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34904/0 (socket says 172.21.15.125:34904) 2024-01-16T15:15:58.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.808+0000 7f5c12000640 1 -- 172.21.15.125:0/994417812 learned_addr learned my addr 172.21.15.125:0/994417812 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 -- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c14159cf0 msgr2=0x7f5c1414de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c14159cf0 0x7f5c1414de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 -- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf4003970 msgr2=0x7f5c140adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf4003970 0x7f5c140adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 -- 172.21.15.125:0/994417812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c14157b30 con 0x7f5c1414ee20 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c12000640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc001020 tx=0x7f5bfc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc007480 con 0x7f5c1414ee20 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc0075e0 con 0x7f5c1414ee20 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14155ab0 con 0x7f5c1414ee20 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc0059c0 con 0x7f5c1414ee20 2024-01-16T15:15:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.809+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1414f2a0 con 0x7f5c1414ee20 2024-01-16T15:15:58.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.811+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c14157d20 con 0x7f5c1414ee20 2024-01-16T15:15:58.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.811+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 42) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f5bfc005b20 con 0x7f5c1414ee20 2024-01-16T15:15:58.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.812+0000 7f5c0aa00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 0x7f5bec06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.812+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5bfc088cb0 con 0x7f5c1414ee20 2024-01-16T15:15:58.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.812+0000 7f5c12a00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 0x7f5bec06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.813+0000 7f5c12a00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 0x7f5bec06e9b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba00 tx=0x7f5c04032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:58.817+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bfc055200 con 0x7f5c1414ee20 2024-01-16T15:15:59.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.037+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c1406ab30 con 0x7f5bec06c4f0 2024-01-16T15:15:59.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.064+0000 7f5c0aa00640 1 -- 172.21.15.125:0/994417812 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c1406ab30 con 0x7f5bec06c4f0 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 msgr2=0x7f5bec06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 0x7f5bec06e9b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba00 tx=0x7f5c04032000 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 msgr2=0x7f5c14155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc001020 tx=0x7f5bfc00dbe0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 shutdown_connections 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c14159cf0 0x7f5c1414de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5bec06c4f0 0x7f5bec06e9b0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c1414ee20 0x7f5c14155570 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 --2- 172.21.15.125:0/994417812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bf4003970 0x7f5c140adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.066+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 >> 172.21.15.125:0/994417812 conn(0x7f5c140cc1d0 msgr2=0x7f5c1406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.067+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 shutdown_connections 2024-01-16T15:15:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.067+0000 7f5c13e00640 1 -- 172.21.15.125:0/994417812 wait complete. 2024-01-16T15:15:59.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.293+0000 7f0a27800640 1 Processor -- start 2024-01-16T15:15:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.293+0000 7f0a27800640 1 -- start start 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a27800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0a20143ae0 0x7f0a200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a27800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a2015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a27800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200cfc30 con 0x7f0a2015e1e0 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a27800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a200b1ee0 con 0x7f0a20143ae0 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a27800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200b2780 con 0x7f0a200cf7d0 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a24e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0a20143ae0 0x7f0a200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a25800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0a2015e1e0 0x7f0a2015a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38238/0 (socket says 172.21.15.125:38238) 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a25800640 1 -- 172.21.15.125:0/1269092115 learned_addr learned my addr 172.21.15.125:0/1269092115 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.295+0000 7f0a17c00640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a2015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.296+0000 7f0a24e00640 1 -- 172.21.15.125:0/1269092115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 msgr2=0x7f0a2015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.296+0000 7f0a24e00640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a2015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.296+0000 7f0a24e00640 1 -- 172.21.15.125:0/1269092115 >> v1:172.21.15.125:6789/0 conn(0x7f0a2015e1e0 legacy=0x7f0a2015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.296+0000 7f0a1fe00640 1 -- 172.21.15.125:0/1269092115 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291470974 0 0) 0x7f0a200cfc30 con 0x7f0a2015e1e0 2024-01-16T15:15:59.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.296+0000 7f0a24e00640 1 -- 172.21.15.125:0/1269092115 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2015e990 con 0x7f0a20143ae0 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.297+0000 7f0a24e00640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a200cf290 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f0a100097a0 tx=0x7f0a1002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efba1cca088b786d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.297+0000 7f0a1fe00640 1 -- 172.21.15.125:0/1269092115 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1002d040 con 0x7f0a20143ae0 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.297+0000 7f0a1fe00640 1 -- 172.21.15.125:0/1269092115 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a10031810 con 0x7f0a20143ae0 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.297+0000 7f0a1fe00640 1 -- 172.21.15.125:0/1269092115 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10031ba0 con 0x7f0a20143ae0 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 -- 172.21.15.125:0/1269092115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 msgr2=0x7f0a200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a200cf290 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f0a100097a0 tx=0x7f0a1002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 -- 172.21.15.125:0/1269092115 shutdown_connections 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a2015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 --2- 172.21.15.125:0/1269092115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a200cf290 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 -- 172.21.15.125:0/1269092115 >> 172.21.15.125:0/1269092115 conn(0x7f0a2006a800 msgr2=0x7f0a2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:59.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 -- 172.21.15.125:0/1269092115 shutdown_connections 2024-01-16T15:15:59.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.298+0000 7f0a27800640 1 -- 172.21.15.125:0/1269092115 wait complete. 2024-01-16T15:15:59.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.300+0000 7f0a27800640 1 Processor -- start 2024-01-16T15:15:59.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.300+0000 7f0a27800640 1 -- start start 2024-01-16T15:15:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a200ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a2015e1e0 0x7f0a200aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a201401c0 con 0x7f0a20143ae0 2024-01-16T15:15:59.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200ae5c0 con 0x7f0a2015e1e0 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a27800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a201f3a20 con 0x7f0a200cf7d0 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34928/0 (socket says 172.21.15.125:34928) 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 -- 172.21.15.125:0/2004705065 learned_addr learned my addr 172.21.15.125:0/2004705065 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 -- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 msgr2=0x7f0a200ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a200ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 -- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a2015e1e0 msgr2=0x7f0a200aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a2015e1e0 0x7f0a200aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.301+0000 7f0a17c00640 1 -- 172.21.15.125:0/2004705065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10009400 con 0x7f0a20143ae0 2024-01-16T15:15:59.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a17c00640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0a08007ef0 tx=0x7f0a0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08014620 con 0x7f0a20143ae0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08014780 con 0x7f0a20143ae0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08014b30 con 0x7f0a20143ae0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a201f3740 con 0x7f0a20143ae0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.302+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a200cc990 con 0x7f0a20143ae0 2024-01-16T15:15:59.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.303+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a200b2bf0 con 0x7f0a20143ae0 2024-01-16T15:15:59.289 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.304+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 42) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f0a08014c90 con 0x7f0a20143ae0 2024-01-16T15:15:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.305+0000 7f0a1d600640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 0x7f09f806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.305+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0a080894d0 con 0x7f0a20143ae0 2024-01-16T15:15:59.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.305+0000 7f0a24e00640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 0x7f09f806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:59.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.306+0000 7f0a24e00640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 0x7f09f806e980 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0a10036000 tx=0x7f0a10030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:59.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.309+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a200b2bf0 con 0x7f0a20143ae0 2024-01-16T15:15:59.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.528+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a2015e640 con 0x7f09f806c4c0 2024-01-16T15:15:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.534+0000 7f0a1d600640 1 -- 172.21.15.125:0/2004705065 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f0a10031a80 con 0x7f09f806c4c0 2024-01-16T15:15:59.519 INFO:teuthology.orchestra.run.smithi125.stdout:/volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 msgr2=0x7f09f806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 0x7f09f806e980 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0a10036000 tx=0x7f0a10030000 comp rx=0 tx=0).stop 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 msgr2=0x7f0a201f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0a08007ef0 tx=0x7f0a0800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 shutdown_connections 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a2015e1e0 0x7f0a200aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f09f806c4c0 0x7f09f806e980 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a20143ae0 0x7f0a201f3c90 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 --2- 172.21.15.125:0/2004705065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a200cf7d0 0x7f0a200ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 >> 172.21.15.125:0/2004705065 conn(0x7f0a2006a800 msgr2=0x7f0a2006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 shutdown_connections 2024-01-16T15:15:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:15:59.537+0000 7f0a27800640 1 -- 172.21.15.125:0/2004705065 wait complete. 2024-01-16T15:15:59.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:59.540 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520) 2024-01-16T15:15:59.593 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:15:59.593 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:15:59.593 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:15:59.594 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:16:01.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:01.595 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520) 2024-01-16T15:16:01.662 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:16:01.663 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:16:01.663 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:16:01.663 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:16:03.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:03.665 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520) 2024-01-16T15:16:03.727 INFO:teuthology.orchestra.run.smithi125.stdout:# file: volumes/_nogroup/sv1/1bd6727f-1bef-48fc-a7a7-0ba49701d520 2024-01-16T15:16:03.727 INFO:teuthology.orchestra.run.smithi125.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:16:03.727 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:16:03.728 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:16:03.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75491 2024-01-16T15:16:03.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.961+0000 7fa92b200640 1 Processor -- start 2024-01-16T15:16:03.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.962+0000 7fa92b200640 1 -- start start 2024-01-16T15:16:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.963+0000 7fa92b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.963+0000 7fa92b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa9240cf7d0 0x7fa92415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.963+0000 7fa92b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9240cfc30 con 0x7fa92415e1e0 2024-01-16T15:16:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.963+0000 7fa92b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240b1ee0 con 0x7fa9240cf7d0 2024-01-16T15:16:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.963+0000 7fa92b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240b2780 con 0x7fa924143ae0 2024-01-16T15:16:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.964+0000 7fa923e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.964+0000 7fa923e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:34434/0 (socket says 172.21.15.125:34434) 2024-01-16T15:16:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.964+0000 7fa923e00640 1 -- 172.21.15.125:0/3260712290 learned_addr learned my addr 172.21.15.125:0/3260712290 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.964+0000 7fa91b200640 1 --2- 172.21.15.125:0/3260712290 >> v2:172.21.15.137:3300/0 conn(0x7fa9240cf7d0 0x7fa92415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.965+0000 7fa923e00640 1 -- 172.21.15.125:0/3260712290 >> v2:172.21.15.137:3300/0 conn(0x7fa9240cf7d0 msgr2=0x7fa92415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.965+0000 7fa923e00640 1 --2- 172.21.15.125:0/3260712290 >> v2:172.21.15.137:3300/0 conn(0x7fa9240cf7d0 0x7fa92415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.965+0000 7fa923e00640 1 -- 172.21.15.125:0/3260712290 >> v1:172.21.15.137:6789/0 conn(0x7fa92415e1e0 legacy=0x7fa92415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.965+0000 7fa923e00640 1 -- 172.21.15.125:0/3260712290 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92415e990 con 0x7fa924143ae0 2024-01-16T15:16:03.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.966+0000 7fa923e00640 1 --2- 172.21.15.125:0/3260712290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa914009870 tx=0x7fa91402db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbf1132d50be722 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:03.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.967+0000 7fa923400640 1 -- 172.21.15.125:0/3260712290 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91402e940 con 0x7fa924143ae0 2024-01-16T15:16:03.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.967+0000 7fa923400640 1 -- 172.21.15.125:0/3260712290 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91402eaa0 con 0x7fa924143ae0 2024-01-16T15:16:03.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.967+0000 7fa923400640 1 -- 172.21.15.125:0/3260712290 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91402ee30 con 0x7fa924143ae0 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.967+0000 7fa92b200640 1 -- 172.21.15.125:0/3260712290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 msgr2=0x7fa9240cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.967+0000 7fa92b200640 1 --2- 172.21.15.125:0/3260712290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa914009870 tx=0x7fa91402db90 comp rx=0 tx=0).stop 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 -- 172.21.15.125:0/3260712290 shutdown_connections 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 --2- 172.21.15.125:0/3260712290 >> v2:172.21.15.137:3300/0 conn(0x7fa9240cf7d0 0x7fa92415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 --2- 172.21.15.125:0/3260712290 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa9240cf290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 -- 172.21.15.125:0/3260712290 >> 172.21.15.125:0/3260712290 conn(0x7fa92406a800 msgr2=0x7fa92406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 -- 172.21.15.125:0/3260712290 shutdown_connections 2024-01-16T15:16:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.968+0000 7fa92b200640 1 -- 172.21.15.125:0/3260712290 wait complete. 2024-01-16T15:16:03.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.969+0000 7fa92b200640 1 Processor -- start 2024-01-16T15:16:03.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 -- start start 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa924159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa92415e1e0 0x7fa924159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241f3b80 con 0x7fa9240cf7d0 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240acc10 con 0x7fa92415e1e0 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.970+0000 7fa92b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240acef0 con 0x7fa924143ae0 2024-01-16T15:16:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34940/0 (socket says 172.21.15.125:34940) 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 -- 172.21.15.125:0/962788505 learned_addr learned my addr 172.21.15.125:0/962788505 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa929200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa92415e1e0 0x7fa924159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 -- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 msgr2=0x7fa924159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa924159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 -- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa92415e1e0 msgr2=0x7fa924159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa92415e1e0 0x7fa924159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:03.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 -- 172.21.15.125:0/962788505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140094d0 con 0x7fa9240cf7d0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.971+0000 7fa923e00640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fa914000c00 tx=0x7fa9140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.972+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914040600 con 0x7fa9240cf7d0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.972+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914040760 con 0x7fa9240cf7d0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.972+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914040b10 con 0x7fa9240cf7d0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.972+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9240ad1d0 con 0x7fa9240cf7d0 2024-01-16T15:16:03.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.972+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9240afe40 con 0x7fa9240cf7d0 2024-01-16T15:16:03.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.973+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9240675d0 con 0x7fa9240cf7d0 2024-01-16T15:16:03.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.974+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 42) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa914040c70 con 0x7fa9240cf7d0 2024-01-16T15:16:03.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.975+0000 7fa920c00640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 0x7fa8fc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:03.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.975+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9140ab110 con 0x7fa9240cf7d0 2024-01-16T15:16:03.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.975+0000 7fa91b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 0x7fa8fc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:03.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.976+0000 7fa91b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 0x7fa8fc06e980 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa90c006a90 tx=0x7fa90c006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:03.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:03.980+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa914077b50 con 0x7fa9240cf7d0 2024-01-16T15:16:04.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.208+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75491", "target": ["mon-mgr", ""]}) v1 -- 0x7fa92415e640 con 0x7fa8fc06c4c0 2024-01-16T15:16:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.380+0000 7fa920c00640 1 -- 172.21.15.125:0/962788505 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa90c0096c0 con 0x7fa8fc06c4c0 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.383+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 msgr2=0x7fa8fc06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.383+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 0x7fa8fc06e980 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa90c006a90 tx=0x7fa90c006a20 comp rx=0 tx=0).stop 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.383+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 msgr2=0x7fa9240af900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.383+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fa914000c00 tx=0x7fa9140046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 shutdown_connections 2024-01-16T15:16:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa92415e1e0 0x7fa924159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa924143ae0 0x7fa924159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa8fc06c4c0 0x7fa8fc06e980 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 --2- 172.21.15.125:0/962788505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9240cf7d0 0x7fa9240af900 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 >> 172.21.15.125:0/962788505 conn(0x7fa92406a800 msgr2=0x7fa9240cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 shutdown_connections 2024-01-16T15:16:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.384+0000 7fa92b200640 1 -- 172.21.15.125:0/962788505 wait complete. 2024-01-16T15:16:04.372 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.387+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.372 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.387+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.372 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.387+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.372 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.387+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.372 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.387+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75492 2024-01-16T15:16:04.391 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.406+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.391 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.406+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.391 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.406+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.391 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.406+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.391 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:04.406+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.591+0000 7f57b8c00640 1 Processor -- start 2024-01-16T15:16:04.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.591+0000 7f57b8c00640 1 -- start start 2024-01-16T15:16:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.592+0000 7f57b8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 0x7f57b415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.592+0000 7f57b8c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57b40cf9a0 0x7f57b40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b8c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b406c470 con 0x7f57b413e360 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b8c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b406c750 con 0x7f57b40cf9a0 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b406ca30 con 0x7f57b413df80 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57b40cf9a0 0x7f57b40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57b40cf9a0 0x7f57b40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34968/0 (socket says 172.21.15.125:34968) 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 -- 172.21.15.125:0/1756614664 learned_addr learned my addr 172.21.15.125:0/1756614664 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b1e00640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 0x7f57b415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 -- 172.21.15.125:0/1756614664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 msgr2=0x7f57b415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 0x7f57b415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 -- 172.21.15.125:0/1756614664 >> v1:172.21.15.137:6789/0 conn(0x7f57b413e360 legacy=0x7f57b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b2800640 1 -- 172.21.15.125:0/1756614664 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b406cd10 con 0x7f57b40cf9a0 2024-01-16T15:16:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.593+0000 7f57b1e00640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 0x7f57b415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b2800640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b40cd840 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009e50 tx=0x7f57ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49cc841cfcf2411 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b1400640 1 -- 172.21.15.125:0/1756614664 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac02ca70 con 0x7f57b40cf9a0 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b1400640 1 -- 172.21.15.125:0/1756614664 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac02cbd0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b1400640 1 -- 172.21.15.125:0/1756614664 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0343f0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b8c00640 1 -- 172.21.15.125:0/1756614664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 msgr2=0x7f57b40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b40cd840 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009e50 tx=0x7f57ac02bed0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.595+0000 7f57b8c00640 1 -- 172.21.15.125:0/1756614664 shutdown_connections 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.596+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b40cd840 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.596+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1756614664 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413df80 0x7f57b415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.596+0000 7f57b8c00640 1 -- 172.21.15.125:0/1756614664 >> 172.21.15.125:0/1756614664 conn(0x7f57b406a800 msgr2=0x7f57b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.596+0000 7f57b8c00640 1 -- 172.21.15.125:0/1756614664 shutdown_connections 2024-01-16T15:16:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.596+0000 7f57b8c00640 1 -- 172.21.15.125:0/1756614664 wait complete. 2024-01-16T15:16:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.597+0000 7f57b8c00640 1 Processor -- start 2024-01-16T15:16:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.597+0000 7f57b8c00640 1 -- start start 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b413df80 0x7f57b4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413e360 0x7f57b40a9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4157610 con 0x7f57b40cf9a0 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b41578f0 con 0x7f57b413df80 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.598+0000 7f57b8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4157bd0 con 0x7f57b413e360 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34980/0 (socket says 172.21.15.125:34980) 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 -- 172.21.15.125:0/1257921890 learned_addr learned my addr 172.21.15.125:0/1257921890 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57abe00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b413df80 0x7f57b4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b2800640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413e360 0x7f57b40a9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 -- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413e360 msgr2=0x7f57b40a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413e360 0x7f57b40a9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 -- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b413df80 msgr2=0x7f57b4156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b413df80 0x7f57b4156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 -- 172.21.15.125:0/1257921890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac009b30 con 0x7f57b40cf9a0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.599+0000 7f57b1e00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c7f0 tx=0x7f57a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.600+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4007a30 con 0x7f57b40cf9a0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.600+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a4007b90 con 0x7f57b40cf9a0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.600+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b4157eb0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.600+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a400ce40 con 0x7f57b40cf9a0 2024-01-16T15:16:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.600+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b41586b0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.601+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b406d920 con 0x7f57b40cf9a0 2024-01-16T15:16:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.602+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 42) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f57a40113f0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.603+0000 7f57aa000640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 0x7f578c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.603+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57a4088cf0 con 0x7f57b40cf9a0 2024-01-16T15:16:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.603+0000 7f57abe00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 0x7f578c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:04.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.604+0000 7f57abe00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 0x7f578c06e980 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f57b41f4620 tx=0x7f579c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:04.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.607+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57a4055240 con 0x7f57b40cf9a0 2024-01-16T15:16:04.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.835+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75492", "target": ["mon-mgr", ""]}) v1 -- 0x7f57b415f9c0 con 0x7f578c06c4c0 2024-01-16T15:16:04.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.978+0000 7f57aa000640 1 -- 172.21.15.125:0/1257921890 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f579c007db0 con 0x7f578c06c4c0 2024-01-16T15:16:04.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 msgr2=0x7f578c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 0x7f578c06e980 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f57b41f4620 tx=0x7f579c009040 comp rx=0 tx=0).stop 2024-01-16T15:16:04.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 msgr2=0x7f57b4153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c7f0 tx=0x7f57a400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 shutdown_connections 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57b413e360 0x7f57b40a9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b413df80 0x7f57b4156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f578c06c4c0 0x7f578c06e980 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 --2- 172.21.15.125:0/1257921890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b40cf9a0 0x7f57b4153100 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.981+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 >> 172.21.15.125:0/1257921890 conn(0x7f57b406a800 msgr2=0x7f57b40b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:04.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.982+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 shutdown_connections 2024-01-16T15:16:04.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:04.982+0000 7f57b8c00640 1 -- 172.21.15.125:0/1257921890 wait complete. 2024-01-16T15:16:04.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75493 2024-01-16T15:16:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.211+0000 7ff584e00640 1 Processor -- start 2024-01-16T15:16:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.212+0000 7ff584e00640 1 -- start start 2024-01-16T15:16:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.213+0000 7ff584e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff5800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.213+0000 7ff584e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff58015f9c0 0x7ff5800cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.213+0000 7ff584e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58015c5f0 con 0x7ff58013df80 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.213+0000 7ff584e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58006c470 con 0x7ff58015f9c0 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.213+0000 7ff584e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58006c750 con 0x7ff58013e360 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff58013df80 0x7ff58015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38252/0 (socket says 172.21.15.125:38252) 2024-01-16T15:16:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57de00640 1 -- 172.21.15.125:0/1834395049 learned_addr learned my addr 172.21.15.125:0/1834395049 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:16:05.201 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671598932 0 0) 0x7ff58015c5f0 con 0x7ff58013df80 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff560003520 con 0x7ff58013df80 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff575200640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff5800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> v2:172.21.15.137:3300/0 conn(0x7ff58015f9c0 0x7ff5800cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200856000 0 0) 0x7ff560003520 con 0x7ff58013df80 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 msgr2=0x7ff5800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff5800cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 >> v2:172.21.15.137:3300/0 conn(0x7ff58015f9c0 msgr2=0x7ff5800cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 --2- 172.21.15.125:0/1834395049 >> v2:172.21.15.137:3300/0 conn(0x7ff58015f9c0 0x7ff5800cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58006ca30 con 0x7ff58013df80 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff570001670 con 0x7ff58013df80 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> v2:172.21.15.137:3300/0 conn(0x7ff58015f9c0 0x7ff5800cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.214+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 >> v1:172.21.15.125:6789/0 conn(0x7ff58013df80 legacy=0x7ff58015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff560003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560003fa0 0x7ff5600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff560004d00 0x7ff560005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58006c470 con 0x7ff560004d00 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58006c750 con 0x7ff560003fa0 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56000bb90 con 0x7ff58013e360 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff560004d00 0x7ff560005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57de00640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff560003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff575200640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560003fa0 0x7ff5600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 -- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 msgr2=0x7ff560003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff58013e360 0x7ff560003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 -- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560003fa0 msgr2=0x7ff5600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560003fa0 0x7ff5600047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.215+0000 7ff57e800640 1 -- 172.21.15.125:0/1834395049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560003520 con 0x7ff560004d00 2024-01-16T15:16:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.216+0000 7ff57e800640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff560004d00 0x7ff560005180 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c490 tx=0x7ff5600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5284713da32db9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578033660 con 0x7ff560004d00 2024-01-16T15:16:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5780337c0 con 0x7ff560004d00 2024-01-16T15:16:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff57d400640 1 -- 172.21.15.125:0/1834395049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578033b70 con 0x7ff560004d00 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 -- 172.21.15.125:0/1834395049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff560004d00 msgr2=0x7ff560005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 --2- 172.21.15.125:0/1834395049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff560004d00 0x7ff560005180 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c490 tx=0x7ff5600057f0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff57e800640 1 -- 172.21.15.125:0/1834395049 reap_dead start 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 -- 172.21.15.125:0/1834395049 shutdown_connections 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 -- 172.21.15.125:0/1834395049 >> 172.21.15.125:0/1834395049 conn(0x7ff58006a800 msgr2=0x7ff58006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 -- 172.21.15.125:0/1834395049 shutdown_connections 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.217+0000 7ff584e00640 1 -- 172.21.15.125:0/1834395049 wait complete. 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.219+0000 7ff584e00640 1 Processor -- start 2024-01-16T15:16:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.219+0000 7ff584e00640 1 -- start start 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 0x7ff58013d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff580157f90 0x7ff580158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800b36e0 con 0x7ff5800b2cf0 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58014e720 con 0x7ff560004d00 2024-01-16T15:16:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff584e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58013dd80 con 0x7ff580157f90 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.220+0000 7ff57e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34998/0 (socket says 172.21.15.125:34998) 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 -- 172.21.15.125:0/498583144 learned_addr learned my addr 172.21.15.125:0/498583144 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff575200640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff580157f90 0x7ff580158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57de00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 0x7ff58013d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 -- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff580157f90 msgr2=0x7ff580158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff580157f90 0x7ff580158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 -- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 msgr2=0x7ff58013d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 0x7ff58013d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 -- 172.21.15.125:0/498583144 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58014e970 con 0x7ff5800b2cf0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57de00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 0x7ff58013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.221+0000 7ff57e800640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c5b0 tx=0x7ff578004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.222+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578007590 con 0x7ff5800b2cf0 2024-01-16T15:16:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.222+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578033e70 con 0x7ff5800b2cf0 2024-01-16T15:16:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.222+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578045610 con 0x7ff5800b2cf0 2024-01-16T15:16:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.222+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58014ec50 con 0x7ff5800b2cf0 2024-01-16T15:16:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.222+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58014f210 con 0x7ff5800b2cf0 2024-01-16T15:16:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.223+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5800675d0 con 0x7ff5800b2cf0 2024-01-16T15:16:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.224+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 42) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff578031070 con 0x7ff5800b2cf0 2024-01-16T15:16:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.225+0000 7ff576a00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 0x7ff55806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.225+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5780aad00 con 0x7ff5800b2cf0 2024-01-16T15:16:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.225+0000 7ff57de00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 0x7ff55806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.226+0000 7ff57de00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 0x7ff55806e9b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5700017a0 tx=0x7ff570001c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:05.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.229+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff578077600 con 0x7ff5800b2cf0 2024-01-16T15:16:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.455+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75493", "target": ["mon-mgr", ""]}) v1 -- 0x7ff58006cff0 con 0x7ff55806c4f0 2024-01-16T15:16:05.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:05.479+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff5780779d0 con 0x7ff5800b2cf0 2024-01-16T15:16:07.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.041+0000 7ff576a00640 1 -- 172.21.15.125:0/498583144 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff57000f7c0 con 0x7ff55806c4f0 2024-01-16T15:16:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 msgr2=0x7ff55806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 0x7ff55806e9b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5700017a0 tx=0x7ff570001c80 comp rx=0 tx=0).stop 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 msgr2=0x7ff5800b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c5b0 tx=0x7ff578004ab0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 shutdown_connections 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff55806c4f0 0x7ff55806e9b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800b2cf0 0x7ff5800b31a0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff580157f90 0x7ff580158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 --2- 172.21.15.125:0/498583144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff560004d00 0x7ff58013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.046+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 >> 172.21.15.125:0/498583144 conn(0x7ff58006a800 msgr2=0x7ff5801536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:07.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.047+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 shutdown_connections 2024-01-16T15:16:07.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.047+0000 7ff584e00640 1 -- 172.21.15.125:0/498583144 wait complete. 2024-01-16T15:16:07.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75494 2024-01-16T15:16:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.273+0000 7fed5ac00640 1 Processor -- start 2024-01-16T15:16:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.273+0000 7fed5ac00640 1 -- start start 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed5ac00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fed54157c30 0x7fed540cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed5ac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed54071200 0x7fed5406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed5ac00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5415deb0 con 0x7fed541618f0 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed5ac00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5415ea30 con 0x7fed54157c30 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed5ac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5406d860 con 0x7fed54071200 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed53e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fed54157c30 0x7fed540cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed53e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fed54157c30 0x7fed540cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49670/0 (socket says 172.21.15.125:49670) 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed53e00640 1 -- 172.21.15.125:0/3758337269 learned_addr learned my addr 172.21.15.125:0/3758337269 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed53400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fed541618f0 0x7fed54070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45504/0 (socket says 172.21.15.125:45504) 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.275+0000 7fed58c00640 1 --2- 172.21.15.125:0/3758337269 >> v2:172.21.15.137:3300/0 conn(0x7fed54071200 0x7fed5406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.276+0000 7fed53e00640 1 -- 172.21.15.125:0/3758337269 >> v2:172.21.15.137:3300/0 conn(0x7fed54071200 msgr2=0x7fed5406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.276+0000 7fed53e00640 1 --2- 172.21.15.125:0/3758337269 >> v2:172.21.15.137:3300/0 conn(0x7fed54071200 0x7fed5406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.276+0000 7fed53e00640 1 -- 172.21.15.125:0/3758337269 >> v1:172.21.15.125:6789/0 conn(0x7fed541618f0 legacy=0x7fed54070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.276+0000 7fed53e00640 1 -- 172.21.15.125:0/3758337269 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5415e190 con 0x7fed54157c30 2024-01-16T15:16:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.276+0000 7fed52a00640 1 -- 172.21.15.125:0/3758337269 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502853495 0 0) 0x7fed5415deb0 con 0x7fed541618f0 2024-01-16T15:16:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.277+0000 7fed53e00640 1 --2- 172.21.15.125:0/3758337269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed540cd600 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fed540cbb60 tx=0x7fed4402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fefbf28aa2001bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.277+0000 7fed52a00640 1 -- 172.21.15.125:0/3758337269 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44030860 con 0x7fed54157c30 2024-01-16T15:16:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.277+0000 7fed52a00640 1 -- 172.21.15.125:0/3758337269 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed440309c0 con 0x7fed54157c30 2024-01-16T15:16:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.277+0000 7fed52a00640 1 -- 172.21.15.125:0/3758337269 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44030d50 con 0x7fed54157c30 2024-01-16T15:16:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 -- 172.21.15.125:0/3758337269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 msgr2=0x7fed540cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3758337269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed540cd600 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fed540cbb60 tx=0x7fed4402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 -- 172.21.15.125:0/3758337269 shutdown_connections 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3758337269 >> v2:172.21.15.137:3300/0 conn(0x7fed54071200 0x7fed5406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3758337269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed540cd600 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 -- 172.21.15.125:0/3758337269 >> 172.21.15.125:0/3758337269 conn(0x7fed540cf700 msgr2=0x7fed540cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 -- 172.21.15.125:0/3758337269 shutdown_connections 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.278+0000 7fed5ac00640 1 -- 172.21.15.125:0/3758337269 wait complete. 2024-01-16T15:16:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.279+0000 7fed5ac00640 1 Processor -- start 2024-01-16T15:16:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.280+0000 7fed5ac00640 1 -- start start 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 0x7fed5414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed541618f0 0x7fed540cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541f3cc0 con 0x7fed54157c30 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5415cb80 con 0x7fed54071200 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed5ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5415ce60 con 0x7fed541618f0 2024-01-16T15:16:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49676/0 (socket says 172.21.15.125:49676) 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 -- 172.21.15.125:0/3803427638 learned_addr learned my addr 172.21.15.125:0/3803427638 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53e00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 0x7fed5414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 -- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed541618f0 msgr2=0x7fed540cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed541618f0 0x7fed540cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 -- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 msgr2=0x7fed5414a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 0x7fed5414a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53400640 1 -- 172.21.15.125:0/3803427638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440094d0 con 0x7fed54157c30 2024-01-16T15:16:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.281+0000 7fed53e00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 0x7fed5414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed53400640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00c4d0 tx=0x7fed3c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0074e0 con 0x7fed54157c30 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c007660 con 0x7fed54157c30 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c004a20 con 0x7fed54157c30 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5415d140 con 0x7fed54157c30 2024-01-16T15:16:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.282+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5414a680 con 0x7fed54157c30 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.283+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed541f3e70 con 0x7fed54157c30 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.284+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fed3c004b80 con 0x7fed54157c30 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.285+0000 7fed4be00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 0x7fed2c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.285+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed3c087d70 con 0x7fed54157c30 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.285+0000 7fed53e00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 0x7fed2c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.286+0000 7fed53e00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 0x7fed2c06eaa0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed44036000 tx=0x7fed4403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.290+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed541f3e70 con 0x7fed54157c30 2024-01-16T15:16:07.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:07.518+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75494", "target": ["mon-mgr", ""]}) v1 -- 0x7fed5415e750 con 0x7fed2c06c5e0 2024-01-16T15:16:12.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.041+0000 7fed4be00640 1 -- 172.21.15.125:0/3803427638 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fed44030c30 con 0x7fed2c06c5e0 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 msgr2=0x7fed2c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 0x7fed2c06eaa0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed44036000 tx=0x7fed4403b040 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 msgr2=0x7fed5415c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00c4d0 tx=0x7fed3c00c990 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 shutdown_connections 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed541618f0 0x7fed540cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fed2c06c5e0 0x7fed2c06eaa0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed54157c30 0x7fed5415c0c0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 --2- 172.21.15.125:0/3803427638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed54071200 0x7fed5414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 >> 172.21.15.125:0/3803427638 conn(0x7fed540cf700 msgr2=0x7fed540b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.045+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 shutdown_connections 2024-01-16T15:16:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.046+0000 7fed5ac00640 1 -- 172.21.15.125:0/3803427638 wait complete. 2024-01-16T15:16:12.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75495 2024-01-16T15:16:12.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.268+0000 7f3b3b200640 1 Processor -- start 2024-01-16T15:16:12.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.269+0000 7f3b3b200640 1 -- start start 2024-01-16T15:16:12.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.270+0000 7f3b3b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b3415d610 0x7f3b340ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.270+0000 7f3b3b200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b340d1ef0 0x7f3b340b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.270+0000 7f3b3b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340b3400 con 0x7f3b340ced60 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.270+0000 7f3b3b200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340d1390 con 0x7f3b340d1ef0 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.270+0000 7f3b3b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340b27f0 con 0x7f3b3415d610 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b340d1ef0 0x7f3b340b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b2be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b340ced60 0x7f3b340d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45516/0 (socket says 172.21.15.125:45516) 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b340d1ef0 0x7f3b340b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49692/0 (socket says 172.21.15.125:49692) 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b2be00640 1 -- 172.21.15.125:0/1815048235 learned_addr learned my addr 172.21.15.125:0/1815048235 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b33e00640 1 --2- 172.21.15.125:0/1815048235 >> v2:172.21.15.137:3300/0 conn(0x7f3b3415d610 0x7f3b340ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 -- 172.21.15.125:0/1815048235 >> v2:172.21.15.137:3300/0 conn(0x7f3b3415d610 msgr2=0x7f3b340ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 --2- 172.21.15.125:0/1815048235 >> v2:172.21.15.137:3300/0 conn(0x7f3b3415d610 0x7f3b340ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 -- 172.21.15.125:0/1815048235 >> v1:172.21.15.125:6789/0 conn(0x7f3b340ced60 legacy=0x7f3b340d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b39200640 1 -- 172.21.15.125:0/1815048235 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340b2ad0 con 0x7f3b340d1ef0 2024-01-16T15:16:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.271+0000 7f3b33400640 1 -- 172.21.15.125:0/1815048235 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873951231 0 0) 0x7f3b340b3400 con 0x7f3b340ced60 2024-01-16T15:16:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.272+0000 7f3b39200640 1 --2- 172.21.15.125:0/1815048235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b340b3890 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3b340b5dc0 tx=0x7f3b2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d396f2a467cfba94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b33400640 1 -- 172.21.15.125:0/1815048235 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c02cc80 con 0x7f3b340d1ef0 2024-01-16T15:16:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b33400640 1 -- 172.21.15.125:0/1815048235 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c02cde0 con 0x7f3b340d1ef0 2024-01-16T15:16:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b33400640 1 -- 172.21.15.125:0/1815048235 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c034660 con 0x7f3b340d1ef0 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 -- 172.21.15.125:0/1815048235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 msgr2=0x7f3b340b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 --2- 172.21.15.125:0/1815048235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b340b3890 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3b340b5dc0 tx=0x7f3b2c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 -- 172.21.15.125:0/1815048235 shutdown_connections 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 --2- 172.21.15.125:0/1815048235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b340b3890 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 --2- 172.21.15.125:0/1815048235 >> v2:172.21.15.137:3300/0 conn(0x7f3b3415d610 0x7f3b340ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 -- 172.21.15.125:0/1815048235 >> 172.21.15.125:0/1815048235 conn(0x7f3b3409c540 msgr2=0x7f3b340d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 -- 172.21.15.125:0/1815048235 shutdown_connections 2024-01-16T15:16:12.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.273+0000 7f3b3b200640 1 -- 172.21.15.125:0/1815048235 wait complete. 2024-01-16T15:16:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.276+0000 7f3b3b200640 1 Processor -- start 2024-01-16T15:16:12.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.276+0000 7f3b3b200640 1 -- start start 2024-01-16T15:16:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b340ced60 0x7f3b3415fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3415d610 0x7f3b340b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b341600f0 con 0x7f3b340d1ef0 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b6a60 con 0x7f3b340ced60 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b3b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b6cb0 con 0x7f3b3415d610 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b2be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b2be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49702/0 (socket says 172.21.15.125:49702) 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.277+0000 7f3b2be00640 1 -- 172.21.15.125:0/151062818 learned_addr learned my addr 172.21.15.125:0/151062818 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b33e00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b340ced60 0x7f3b3415fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b39200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3415d610 0x7f3b340b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 -- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3415d610 msgr2=0x7f3b340b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3415d610 0x7f3b340b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 -- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b340ced60 msgr2=0x7f3b3415fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b340ced60 0x7f3b3415fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 -- 172.21.15.125:0/151062818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009a70 con 0x7f3b340d1ef0 2024-01-16T15:16:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b2be00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c4d0 tx=0x7f3b1c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.278+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0074e0 con 0x7f3b340d1ef0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.279+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c007660 con 0x7f3b340d1ef0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.279+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c004a20 con 0x7f3b340d1ef0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.279+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b340b6f90 con 0x7f3b340d1ef0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.279+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b340b7550 con 0x7f3b340d1ef0 2024-01-16T15:16:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.280+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b340675d0 con 0x7f3b340d1ef0 2024-01-16T15:16:12.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.281+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3b1c004b80 con 0x7f3b340d1ef0 2024-01-16T15:16:12.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.282+0000 7f3b30c00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 0x7f3b0c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:12.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.282+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3b1c087d70 con 0x7f3b340d1ef0 2024-01-16T15:16:12.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.282+0000 7f3b33e00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 0x7f3b0c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:12.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.283+0000 7f3b33e00640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 0x7f3b0c06eaa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3b24006fd0 tx=0x7f3b24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:12.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.286+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b1c054290 con 0x7f3b340d1ef0 2024-01-16T15:16:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:12.510+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75495", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b340b0120 con 0x7f3b0c06c5e0 2024-01-16T15:16:12.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:12.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:17.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.038+0000 7f3b30c00640 1 -- 172.21.15.125:0/151062818 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3b340b0120 con 0x7f3b0c06c5e0 2024-01-16T15:16:17.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 msgr2=0x7f3b0c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 0x7f3b0c06eaa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3b24006fd0 tx=0x7f3b24008040 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 msgr2=0x7f3b34152560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c4d0 tx=0x7f3b1c00c990 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 shutdown_connections 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b3415d610 0x7f3b340b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b0c06c5e0 0x7f3b0c06eaa0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340d1ef0 0x7f3b34152560 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 --2- 172.21.15.125:0/151062818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b340ced60 0x7f3b3415fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.041+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 >> 172.21.15.125:0/151062818 conn(0x7f3b3409c540 msgr2=0x7f3b3406b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:17.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.042+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 shutdown_connections 2024-01-16T15:16:17.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.042+0000 7f3b3b200640 1 -- 172.21.15.125:0/151062818 wait complete. 2024-01-16T15:16:17.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75496 2024-01-16T15:16:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.274+0000 7f1721000640 1 Processor -- start 2024-01-16T15:16:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.274+0000 7f1721000640 1 -- start start 2024-01-16T15:16:17.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1721000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f171c13df80 0x7f171c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1721000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f171c0cf9a0 0x7f171c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1721000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c06c470 con 0x7f171c13e360 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1721000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c06c750 con 0x7f171c13df80 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1721000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c06ca30 con 0x7f171c0cf9a0 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1719400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f171c13e360 0x7f171c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45266/0 (socket says 172.21.15.125:45266) 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1719400640 1 -- 172.21.15.125:0/3602471702 learned_addr learned my addr 172.21.15.125:0/3602471702 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.276+0000 7f1719e00640 1 --2- 172.21.15.125:0/3602471702 >> v2:172.21.15.125:3300/0 conn(0x7f171c13df80 0x7f171c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f171a800640 1 --2- 172.21.15.125:0/3602471702 >> v2:172.21.15.137:3300/0 conn(0x7f171c0cf9a0 0x7f171c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f1719e00640 1 -- 172.21.15.125:0/3602471702 >> v2:172.21.15.137:3300/0 conn(0x7f171c0cf9a0 msgr2=0x7f171c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f1719e00640 1 --2- 172.21.15.125:0/3602471702 >> v2:172.21.15.137:3300/0 conn(0x7f171c0cf9a0 0x7f171c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f1718a00640 1 -- 172.21.15.125:0/3602471702 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782080763 0 0) 0x7f171c06c470 con 0x7f171c13e360 2024-01-16T15:16:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f1719e00640 1 -- 172.21.15.125:0/3602471702 >> v1:172.21.15.125:6789/0 conn(0x7f171c13e360 legacy=0x7f171c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.277+0000 7f1719e00640 1 -- 172.21.15.125:0/3602471702 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c06cd10 con 0x7f171c13df80 2024-01-16T15:16:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.278+0000 7f1719e00640 1 --2- 172.21.15.125:0/3602471702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c13df80 0x7f171c15bee0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f171c0ccb10 tx=0x7f170c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f92ec4e8f7c117 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.278+0000 7f1718a00640 1 -- 172.21.15.125:0/3602471702 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c031830 con 0x7f171c13df80 2024-01-16T15:16:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.278+0000 7f1718a00640 1 -- 172.21.15.125:0/3602471702 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c031990 con 0x7f171c13df80 2024-01-16T15:16:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.278+0000 7f1718a00640 1 -- 172.21.15.125:0/3602471702 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c031d20 con 0x7f171c13df80 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 -- 172.21.15.125:0/3602471702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c13df80 msgr2=0x7f171c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 --2- 172.21.15.125:0/3602471702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c13df80 0x7f171c15bee0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f171c0ccb10 tx=0x7f170c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 -- 172.21.15.125:0/3602471702 shutdown_connections 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 --2- 172.21.15.125:0/3602471702 >> v2:172.21.15.137:3300/0 conn(0x7f171c0cf9a0 0x7f171c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 --2- 172.21.15.125:0/3602471702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c13df80 0x7f171c15bee0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 -- 172.21.15.125:0/3602471702 >> 172.21.15.125:0/3602471702 conn(0x7f171c06a800 msgr2=0x7f171c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 -- 172.21.15.125:0/3602471702 shutdown_connections 2024-01-16T15:16:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.279+0000 7f1721000640 1 -- 172.21.15.125:0/3602471702 wait complete. 2024-01-16T15:16:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.281+0000 7f1721000640 1 Processor -- start 2024-01-16T15:16:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.281+0000 7f1721000640 1 -- start start 2024-01-16T15:16:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f171c13df80 0x7f171c159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f171c13e360 0x7f171c14bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c14aa20 con 0x7f171c0cf9a0 2024-01-16T15:16:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c1f2f00 con 0x7f171c13e360 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1721000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c14ac30 con 0x7f171c13df80 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1719e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1719e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34328/0 (socket says 172.21.15.125:34328) 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.282+0000 7f1719e00640 1 -- 172.21.15.125:0/1715758843 learned_addr learned my addr 172.21.15.125:0/1715758843 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719400640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f171c13df80 0x7f171c159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f171a800640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f171c13e360 0x7f171c14bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 -- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f171c13df80 msgr2=0x7f171c159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f171c13df80 0x7f171c159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 -- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f171c13e360 msgr2=0x7f171c14bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f171c13e360 0x7f171c14bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 -- 172.21.15.125:0/1715758843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c009400 con 0x7f171c0cf9a0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1719e00640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f170c009890 tx=0x7f170c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.283+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0317d0 con 0x7f171c0cf9a0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.284+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c03ba70 con 0x7f171c0cf9a0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.284+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c03be20 con 0x7f171c0cf9a0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.284+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c14af10 con 0x7f171c0cf9a0 2024-01-16T15:16:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.284+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c14b4d0 con 0x7f171c0cf9a0 2024-01-16T15:16:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.285+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c0675d0 con 0x7f171c0cf9a0 2024-01-16T15:16:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.286+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f170c0453f0 con 0x7f171c0cf9a0 2024-01-16T15:16:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.287+0000 7f1712000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 0x7f16f406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.287+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f170c0aafe0 con 0x7f171c0cf9a0 2024-01-16T15:16:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.287+0000 7f1719400640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 0x7f16f406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.287+0000 7f1719400640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 0x7f16f406eaa0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b3d80 tx=0x7f1704007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.291+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f170c077620 con 0x7f171c0cf9a0 2024-01-16T15:16:17.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.521+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75496", "target": ["mon-mgr", ""]}) v1 -- 0x7f171c15f9c0 con 0x7f16f406c5e0 2024-01-16T15:16:17.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.658+0000 7f1712000640 1 -- 172.21.15.125:0/1715758843 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f17040076a0 con 0x7f16f406c5e0 2024-01-16T15:16:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 msgr2=0x7f16f406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 0x7f16f406eaa0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b3d80 tx=0x7f1704007b50 comp rx=0 tx=0).stop 2024-01-16T15:16:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 msgr2=0x7f171c159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f170c009890 tx=0x7f170c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 shutdown_connections 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f171c13e360 0x7f171c14bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f171c13df80 0x7f171c159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f16f406c5e0 0x7f16f406eaa0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 --2- 172.21.15.125:0/1715758843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f171c0cf9a0 0x7f171c159a30 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 >> 172.21.15.125:0/1715758843 conn(0x7f171c06a800 msgr2=0x7f171c15d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.661+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 shutdown_connections 2024-01-16T15:16:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.662+0000 7f1721000640 1 -- 172.21.15.125:0/1715758843 wait complete. 2024-01-16T15:16:17.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone75497 2024-01-16T15:16:17.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.892+0000 7fa0ab800640 1 Processor -- start 2024-01-16T15:16:17.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.892+0000 7fa0ab800640 1 -- start start 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.893+0000 7fa0ab800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.893+0000 7fa0ab800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0a413e360 0x7fa0a40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.893+0000 7fa0ab800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a406c470 con 0x7fa0a415f9c0 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.893+0000 7fa0ab800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a406c750 con 0x7fa0a413e360 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.893+0000 7fa0ab800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a406ca30 con 0x7fa0a413df80 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.894+0000 7fa0a8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.894+0000 7fa0a8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:57830/0 (socket says 172.21.15.125:57830) 2024-01-16T15:16:17.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.894+0000 7fa0a8e00640 1 -- 172.21.15.125:0/25630665 learned_addr learned my addr 172.21.15.125:0/25630665 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.894+0000 7fa0a9800640 1 --1- 172.21.15.125:0/25630665 >> v1:172.21.15.137:6789/0 conn(0x7fa0a415f9c0 0x7fa0a41408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:56924/0 (socket says 172.21.15.125:56924) 2024-01-16T15:16:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.894+0000 7fa09be00640 1 --2- 172.21.15.125:0/25630665 >> v2:172.21.15.137:3300/0 conn(0x7fa0a413e360 0x7fa0a40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.895+0000 7fa0a8e00640 1 -- 172.21.15.125:0/25630665 >> v2:172.21.15.137:3300/0 conn(0x7fa0a413e360 msgr2=0x7fa0a40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:17.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.895+0000 7fa0a8e00640 1 --2- 172.21.15.125:0/25630665 >> v2:172.21.15.137:3300/0 conn(0x7fa0a413e360 0x7fa0a40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.895+0000 7fa0a8e00640 1 -- 172.21.15.125:0/25630665 >> v1:172.21.15.137:6789/0 conn(0x7fa0a415f9c0 legacy=0x7fa0a41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.895+0000 7fa0a8e00640 1 -- 172.21.15.125:0/25630665 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a406cd10 con 0x7fa0a413df80 2024-01-16T15:16:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.896+0000 7fa0a8e00640 1 --2- 172.21.15.125:0/25630665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7f242c1a434628c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.897+0000 7fa0a3e00640 1 -- 172.21.15.125:0/25630665 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094030840 con 0x7fa0a413df80 2024-01-16T15:16:17.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.897+0000 7fa0a3e00640 1 -- 172.21.15.125:0/25630665 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0940309a0 con 0x7fa0a413df80 2024-01-16T15:16:17.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.897+0000 7fa0a3e00640 1 -- 172.21.15.125:0/25630665 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094030d30 con 0x7fa0a413df80 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.897+0000 7fa0ab800640 1 -- 172.21.15.125:0/25630665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 msgr2=0x7fa0a415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.897+0000 7fa0ab800640 1 --2- 172.21.15.125:0/25630665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09402b980 comp rx=0 tx=0).stop 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 -- 172.21.15.125:0/25630665 shutdown_connections 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 --2- 172.21.15.125:0/25630665 >> v2:172.21.15.137:3300/0 conn(0x7fa0a413e360 0x7fa0a40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 --2- 172.21.15.125:0/25630665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a413df80 0x7fa0a415bee0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 -- 172.21.15.125:0/25630665 >> 172.21.15.125:0/25630665 conn(0x7fa0a406a800 msgr2=0x7fa0a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 -- 172.21.15.125:0/25630665 shutdown_connections 2024-01-16T15:16:17.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.898+0000 7fa0ab800640 1 -- 172.21.15.125:0/25630665 wait complete. 2024-01-16T15:16:17.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.899+0000 7fa0ab800640 1 Processor -- start 2024-01-16T15:16:17.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.899+0000 7fa0ab800640 1 -- start start 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0a413e360 0x7fa0a4071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 0x7fa0a413cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a40b2a90 con 0x7fa0a413df80 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a414bd40 con 0x7fa0a413e360 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.900+0000 7fa0ab800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a413dd00 con 0x7fa0a415f9c0 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34330/0 (socket says 172.21.15.125:34330) 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 -- 172.21.15.125:0/2902895450 learned_addr learned my addr 172.21.15.125:0/2902895450 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa09be00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0a413e360 0x7fa0a4071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a9800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 0x7fa0a413cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 -- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 msgr2=0x7fa0a413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 0x7fa0a413cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 -- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0a413e360 msgr2=0x7fa0a4071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0a413e360 0x7fa0a4071ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 -- 172.21.15.125:0/2902895450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940094d0 con 0x7fa0a413df80 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a9800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 0x7fa0a413cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.901+0000 7fa0a8e00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa09402b9c0 tx=0x7fa094004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.902+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094039a90 con 0x7fa0a413df80 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.902+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa094039bf0 con 0x7fa0a413df80 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.902+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094043410 con 0x7fa0a413df80 2024-01-16T15:16:17.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.902+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a414bf90 con 0x7fa0a413df80 2024-01-16T15:16:17.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.902+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a40b26b0 con 0x7fa0a413df80 2024-01-16T15:16:17.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.903+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a40b2d00 con 0x7fa0a413df80 2024-01-16T15:16:17.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.904+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fa09402f390 con 0x7fa0a413df80 2024-01-16T15:16:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.905+0000 7fa0a1600640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 0x7fa07c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.905+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0940aa930 con 0x7fa0a413df80 2024-01-16T15:16:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.905+0000 7fa09be00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 0x7fa07c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:17.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.906+0000 7fa09be00640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 0x7fa07c06eaa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40af690 tx=0x7fa08c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:17.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:17.910+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa094077310 con 0x7fa0a413df80 2024-01-16T15:16:18.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.134+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone75497", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a40ce850 con 0x7fa07c06c5e0 2024-01-16T15:16:18.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.281+0000 7fa0a1600640 1 -- 172.21.15.125:0/2902895450 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa08c010840 con 0x7fa07c06c5e0 2024-01-16T15:16:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 msgr2=0x7fa07c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 0x7fa07c06eaa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40af690 tx=0x7fa08c008040 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 msgr2=0x7fa0a40b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa09402b9c0 tx=0x7fa094004730 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 shutdown_connections 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0a415f9c0 0x7fa0a413cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0a413e360 0x7fa0a4071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa07c06c5e0 0x7fa07c06eaa0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 --2- 172.21.15.125:0/2902895450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a413df80 0x7fa0a40b2eb0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 >> 172.21.15.125:0/2902895450 conn(0x7fa0a406a800 msgr2=0x7fa0a40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 shutdown_connections 2024-01-16T15:16:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.284+0000 7fa0ab800640 1 -- 172.21.15.125:0/2902895450 wait complete. 2024-01-16T15:16:18.291 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:16:18.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone75497 2024-01-16T15:16:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.518+0000 7f259d600640 1 Processor -- start 2024-01-16T15:16:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.518+0000 7f259d600640 1 -- start start 2024-01-16T15:16:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f259d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25980b3ae0 0x7f259815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f259d600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2598070c40 con 0x7f25980cf290 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f259d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259806c470 con 0x7f25980ae5a0 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f259d600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598071450 con 0x7f25980b3ae0 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f258fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25980b3ae0 0x7f259815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f2597400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f25980cf290 0x7f25980b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45272/0 (socket says 172.21.15.125:45272) 2024-01-16T15:16:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.520+0000 7f2597400640 1 -- 172.21.15.125:0/2139327302 learned_addr learned my addr 172.21.15.125:0/2139327302 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:16:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.521+0000 7f258fe00640 1 -- 172.21.15.125:0/2139327302 >> v1:172.21.15.137:6789/0 conn(0x7f25980ae5a0 legacy=0x7f259806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.521+0000 7f258fe00640 1 -- 172.21.15.125:0/2139327302 >> v1:172.21.15.125:6789/0 conn(0x7f25980cf290 legacy=0x7f25980b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.521+0000 7f258fe00640 1 -- 172.21.15.125:0/2139327302 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259806c750 con 0x7f25980b3ae0 2024-01-16T15:16:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.521+0000 7f2596000640 1 -- 172.21.15.125:0/2139327302 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469969523 0 0) 0x7f2598070c40 con 0x7f25980cf290 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.522+0000 7f258fe00640 1 --2- 172.21.15.125:0/2139327302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b3ae0 0x7f259815e430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f25800097a0 tx=0x7f258002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e6217aa5aa407 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.522+0000 7f2596000640 1 -- 172.21.15.125:0/2139327302 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580030610 con 0x7f25980b3ae0 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.522+0000 7f2596000640 1 -- 172.21.15.125:0/2139327302 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580030770 con 0x7f25980b3ae0 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.522+0000 7f2596000640 1 -- 172.21.15.125:0/2139327302 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580030b00 con 0x7f25980b3ae0 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 -- 172.21.15.125:0/2139327302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b3ae0 msgr2=0x7f259815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 --2- 172.21.15.125:0/2139327302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b3ae0 0x7f259815e430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f25800097a0 tx=0x7f258002b960 comp rx=0 tx=0).stop 2024-01-16T15:16:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 -- 172.21.15.125:0/2139327302 shutdown_connections 2024-01-16T15:16:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 --2- 172.21.15.125:0/2139327302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980b3ae0 0x7f259815e430 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 -- 172.21.15.125:0/2139327302 >> 172.21.15.125:0/2139327302 conn(0x7f259806a800 msgr2=0x7f259806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 -- 172.21.15.125:0/2139327302 shutdown_connections 2024-01-16T15:16:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.523+0000 7f259d600640 1 -- 172.21.15.125:0/2139327302 wait complete. 2024-01-16T15:16:18.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.524+0000 7f259d600640 1 Processor -- start 2024-01-16T15:16:18.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.525+0000 7f259d600640 1 -- start start 2024-01-16T15:16:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.525+0000 7f259d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980ae5a0 0x7f2598157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.525+0000 7f259d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25980b3ae0 0x7f25980cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.525+0000 7f259d600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f259d600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259815cd10 con 0x7f25980cf290 2024-01-16T15:16:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f259d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980ce8d0 con 0x7f25980b3ae0 2024-01-16T15:16:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f259d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980cce20 con 0x7f25980ae5a0 2024-01-16T15:16:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34342/0 (socket says 172.21.15.125:34342) 2024-01-16T15:16:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 -- 172.21.15.125:0/1321692179 learned_addr learned my addr 172.21.15.125:0/1321692179 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f258fe00640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25980b3ae0 0x7f25980cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 -- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980ae5a0 msgr2=0x7f2598157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2596a00640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980ae5a0 0x7f2598157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980ae5a0 0x7f2598157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 -- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25980b3ae0 msgr2=0x7f25980cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25980b3ae0 0x7f25980cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.526+0000 7f2597400640 1 -- 172.21.15.125:0/1321692179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580009400 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f2597400640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f259000c780 tx=0x7f259000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000ce90 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259000a640 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000a9d0 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259815c690 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.527+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259815c930 con 0x7f25980cf290 2024-01-16T15:16:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.528+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259815ce50 con 0x7f25980cf290 2024-01-16T15:16:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.529+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f259000ab30 con 0x7f25980cf290 2024-01-16T15:16:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.530+0000 7f258ea00640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 0x7f257006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.530+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2590088060 con 0x7f25980cf290 2024-01-16T15:16:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.530+0000 7f2596a00640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 0x7f257006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.531+0000 7f2596a00640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 0x7f257006eaa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2588003d90 tx=0x7f2588009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:18.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.534+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f259815ce50 con 0x7f25980cf290 2024-01-16T15:16:18.752 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.765+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone75497", "target": ["mon-mgr", ""]}) v1 -- 0x7f2598070550 con 0x7f257006c5e0 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.776+0000 7f258ea00640 1 -- 172.21.15.125:0/1321692179 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f2588003980 con 0x7f257006c5e0 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "status": { 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "pending", 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "source": { 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "volume": "cephfs", 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "subvolume": "sv1", 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "snapshot": "ss1" 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:16:18.763 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 msgr2=0x7f257006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 0x7f257006eaa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2588003d90 tx=0x7f2588009040 comp rx=0 tx=0).stop 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 msgr2=0x7f2598160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f259000c780 tx=0x7f259000cc40 comp rx=0 tx=0).stop 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 shutdown_connections 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f257006c5e0 0x7f257006eaa0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25980cf290 0x7f2598160f30 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25980b3ae0 0x7f25980cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.766 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 --2- 172.21.15.125:0/1321692179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25980ae5a0 0x7f2598157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 >> 172.21.15.125:0/1321692179 conn(0x7f259806a800 msgr2=0x7f259806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 shutdown_connections 2024-01-16T15:16:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:18.779+0000 7f259d600640 1 -- 172.21.15.125:0/1321692179 wait complete. 2024-01-16T15:16:18.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75491 2024-01-16T15:16:18.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.009+0000 7f59dd600640 1 Processor -- start 2024-01-16T15:16:18.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.009+0000 7f59dd600640 1 -- start start 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59dd600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f59d813cf10 0x7f59d813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59dd600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59d813fae0 0x7f59d80cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59dd600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80ce440 con 0x7f59d80cdc20 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59dd600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d813ff90 con 0x7f59d813fae0 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59dd600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80ce720 con 0x7f59d813cf10 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59d7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59d813fae0 0x7f59d80cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59d7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59d813fae0 0x7f59d80cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34346/0 (socket says 172.21.15.125:34346) 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59d7400640 1 -- 172.21.15.125:0/843858250 learned_addr learned my addr 172.21.15.125:0/843858250 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:18.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.011+0000 7f59d7e00640 1 --2- 172.21.15.125:0/843858250 >> v2:172.21.15.137:3300/0 conn(0x7f59d813cf10 0x7f59d813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:18.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.012+0000 7f59d7400640 1 -- 172.21.15.125:0/843858250 >> v2:172.21.15.137:3300/0 conn(0x7f59d813cf10 msgr2=0x7f59d813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:18.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.012+0000 7f59d7400640 1 --2- 172.21.15.125:0/843858250 >> v2:172.21.15.137:3300/0 conn(0x7f59d813cf10 0x7f59d813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:18.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.012+0000 7f59d7400640 1 -- 172.21.15.125:0/843858250 >> v1:172.21.15.137:6789/0 conn(0x7f59d80cdc20 legacy=0x7f59d815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:18.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.012+0000 7f59d7400640 1 -- 172.21.15.125:0/843858250 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80cea00 con 0x7f59d813fae0 2024-01-16T15:16:19.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.013+0000 7f59d7400640 1 --2- 172.21.15.125:0/843858250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813fae0 0x7f59d80cd6e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f59d806e390 tx=0x7f59c002e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d995288e38f4ccc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:19.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.013+0000 7f59d6a00640 1 -- 172.21.15.125:0/843858250 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0036440 con 0x7f59d813fae0 2024-01-16T15:16:19.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59d6a00640 1 -- 172.21.15.125:0/843858250 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c00365a0 con 0x7f59d813fae0 2024-01-16T15:16:19.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59d6a00640 1 -- 172.21.15.125:0/843858250 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0036930 con 0x7f59d813fae0 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 -- 172.21.15.125:0/843858250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813fae0 msgr2=0x7f59d80cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 --2- 172.21.15.125:0/843858250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813fae0 0x7f59d80cd6e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f59d806e390 tx=0x7f59c002e180 comp rx=0 tx=0).stop 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 -- 172.21.15.125:0/843858250 shutdown_connections 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 --2- 172.21.15.125:0/843858250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813fae0 0x7f59d80cd6e0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 --2- 172.21.15.125:0/843858250 >> v2:172.21.15.137:3300/0 conn(0x7f59d813cf10 0x7f59d813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 -- 172.21.15.125:0/843858250 >> 172.21.15.125:0/843858250 conn(0x7f59d806a7d0 msgr2=0x7f59d806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 -- 172.21.15.125:0/843858250 shutdown_connections 2024-01-16T15:16:19.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.014+0000 7f59dd600640 1 -- 172.21.15.125:0/843858250 wait complete. 2024-01-16T15:16:19.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.016+0000 7f59dd600640 1 Processor -- start 2024-01-16T15:16:19.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.016+0000 7f59dd600640 1 -- start start 2024-01-16T15:16:19.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80cdc20 0x7f59d81f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:19.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:19.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d813fae0 0x7f59d815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:19.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80acde0 con 0x7f59d813cf10 2024-01-16T15:16:19.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81f36a0 con 0x7f59d813fae0 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59dd600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8138ec0 con 0x7f59d80cdc20 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.017+0000 7f59d7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34352/0 (socket says 172.21.15.125:34352) 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 -- 172.21.15.125:0/2791111099 learned_addr learned my addr 172.21.15.125:0/2791111099 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7e00640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80cdc20 0x7f59d81f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 -- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80cdc20 msgr2=0x7f59d81f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80cdc20 0x7f59d81f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 -- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d813fae0 msgr2=0x7f59d815c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d813fae0 0x7f59d815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 -- 172.21.15.125:0/2791111099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c000b770 con 0x7f59d813cf10 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59d7400640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f59c0000c00 tx=0x7f59c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.018+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0036440 con 0x7f59d813cf10 2024-01-16T15:16:19.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.019+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0034900 con 0x7f59d813cf10 2024-01-16T15:16:19.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.019+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0034cb0 con 0x7f59d813cf10 2024-01-16T15:16:19.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.019+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d8138900 con 0x7f59d813cf10 2024-01-16T15:16:19.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.019+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80aca00 con 0x7f59d813cf10 2024-01-16T15:16:19.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.020+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8067450 con 0x7f59d813cf10 2024-01-16T15:16:19.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.021+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f59c004a3b0 con 0x7f59d813cf10 2024-01-16T15:16:19.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.021+0000 7f59cfe00640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 0x7f59b006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:19.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.022+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59c00ab290 con 0x7f59d813cf10 2024-01-16T15:16:19.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.022+0000 7f59d7e00640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 0x7f59b006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:19.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.023+0000 7f59d7e00640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 0x7f59b006eaa0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009ad0 tx=0x7f59c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:19.013 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.026+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59c00777b0 con 0x7f59d813cf10 2024-01-16T15:16:19.241 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:19.255+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75491", "target": ["mon-mgr", ""]}) v1 -- 0x7f59d80b2070 con 0x7f59b006c5e0 2024-01-16T15:16:36.888 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:36.901+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:36.888 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:36.901+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:36.888 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:36.901+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:36.888 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:36.901+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:36.889 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:36.901+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:37.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.102+0000 7f59cfe00640 1 -- 172.21.15.125:0/2791111099 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f59c8009800 con 0x7f59b006c5e0 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 msgr2=0x7f59b006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 0x7f59b006eaa0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009ad0 tx=0x7f59c8008040 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 msgr2=0x7f59d8137f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f59c0000c00 tx=0x7f59c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 shutdown_connections 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d813fae0 0x7f59d815c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f59b006c5e0 0x7f59b006eaa0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59d813cf10 0x7f59d8137f90 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 --2- 172.21.15.125:0/2791111099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80cdc20 0x7f59d81f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.105+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 >> 172.21.15.125:0/2791111099 conn(0x7f59d806a7d0 msgr2=0x7f59d80af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.106+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 shutdown_connections 2024-01-16T15:16:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:37.106+0000 7f59dd600640 1 -- 172.21.15.125:0/2791111099 wait complete. 2024-01-16T15:16:39.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75492 2024-01-16T15:16:39.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.351+0000 7f3e87200640 1 Processor -- start 2024-01-16T15:16:39.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.351+0000 7f3e87200640 1 -- start start 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.353+0000 7f3e87200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e880adf20 0x7f3e88070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.353+0000 7f3e87200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3e880ccb50 0x7f3e88143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.353+0000 7f3e87200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8815f4e0 con 0x7f3e8815bf10 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.353+0000 7f3e87200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8815f970 con 0x7f3e880adf20 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.353+0000 7f3e87200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e88071130 con 0x7f3e880ccb50 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e880adf20 0x7f3e88070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3e8815bf10 0x7f3e880cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59254/0 (socket says 172.21.15.125:59254) 2024-01-16T15:16:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85400640 1 -- 172.21.15.125:0/3342192935 learned_addr learned my addr 172.21.15.125:0/3342192935 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:16:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e86800640 1 --2- 172.21.15.125:0/3342192935 >> v2:172.21.15.137:3300/0 conn(0x7f3e880ccb50 0x7f3e88143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e84a00640 1 -- 172.21.15.125:0/3342192935 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345621375 0 0) 0x7f3e8815f4e0 con 0x7f3e8815bf10 2024-01-16T15:16:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e84a00640 1 -- 172.21.15.125:0/3342192935 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68003520 con 0x7f3e8815bf10 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85e00640 1 -- 172.21.15.125:0/3342192935 >> v2:172.21.15.137:3300/0 conn(0x7f3e880ccb50 msgr2=0x7f3e88143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85e00640 1 --2- 172.21.15.125:0/3342192935 >> v2:172.21.15.137:3300/0 conn(0x7f3e880ccb50 0x7f3e88143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85e00640 1 -- 172.21.15.125:0/3342192935 >> v1:172.21.15.125:6789/0 conn(0x7f3e8815bf10 legacy=0x7f3e880cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.354+0000 7f3e85e00640 1 -- 172.21.15.125:0/3342192935 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8815c390 con 0x7f3e880adf20 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.356+0000 7f3e85e00640 1 --2- 172.21.15.125:0/3342192935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e88070a30 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806ff20 tx=0x7f3e8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822cbe56065757ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.356+0000 7f3e84a00640 1 -- 172.21.15.125:0/3342192935 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8002e070 con 0x7f3e880adf20 2024-01-16T15:16:39.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.356+0000 7f3e84a00640 1 -- 172.21.15.125:0/3342192935 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e800315a0 con 0x7f3e880adf20 2024-01-16T15:16:39.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.356+0000 7f3e84a00640 1 -- 172.21.15.125:0/3342192935 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80031930 con 0x7f3e880adf20 2024-01-16T15:16:39.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 -- 172.21.15.125:0/3342192935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 msgr2=0x7f3e88070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 --2- 172.21.15.125:0/3342192935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e88070a30 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806ff20 tx=0x7f3e8002b840 comp rx=0 tx=0).stop 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 -- 172.21.15.125:0/3342192935 shutdown_connections 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 --2- 172.21.15.125:0/3342192935 >> v2:172.21.15.137:3300/0 conn(0x7f3e880ccb50 0x7f3e88143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 --2- 172.21.15.125:0/3342192935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e88070a30 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 -- 172.21.15.125:0/3342192935 >> 172.21.15.125:0/3342192935 conn(0x7f3e8806a7d0 msgr2=0x7f3e8806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 -- 172.21.15.125:0/3342192935 shutdown_connections 2024-01-16T15:16:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.357+0000 7f3e87200640 1 -- 172.21.15.125:0/3342192935 wait complete. 2024-01-16T15:16:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.359+0000 7f3e87200640 1 Processor -- start 2024-01-16T15:16:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.359+0000 7f3e87200640 1 -- start start 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e880ccb50 0x7f3e880afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e8815bf10 0x7f3e880b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e881396b0 con 0x7f3e880adf20 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880b1e90 con 0x7f3e8815bf10 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e87200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e88138ea0 con 0x7f3e880ccb50 2024-01-16T15:16:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e85e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e85e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52084/0 (socket says 172.21.15.125:52084) 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.360+0000 7f3e85e00640 1 -- 172.21.15.125:0/2575695086 learned_addr learned my addr 172.21.15.125:0/2575695086 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e86800640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e8815bf10 0x7f3e880b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85400640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e880ccb50 0x7f3e880afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 -- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e880ccb50 msgr2=0x7f3e880afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e880ccb50 0x7f3e880afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 -- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e8815bf10 msgr2=0x7f3e880b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e8815bf10 0x7f3e880b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 -- 172.21.15.125:0/2575695086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80009400 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e85e00640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009890 tx=0x7f3e800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.361+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80032840 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.362+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e800329c0 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.362+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80032d70 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.362+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e880b0300 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.362+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e881393d0 con 0x7f3e880adf20 2024-01-16T15:16:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.363+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e88067450 con 0x7f3e880adf20 2024-01-16T15:16:39.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.364+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 43) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3e800453f0 con 0x7f3e880adf20 2024-01-16T15:16:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.365+0000 7f3e7e000640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 0x7f3e6006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.365+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3e800aada0 con 0x7f3e880adf20 2024-01-16T15:16:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.365+0000 7f3e85400640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 0x7f3e6006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:39.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.366+0000 7f3e85400640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 0x7f3e6006eaa0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3e8815aee0 tx=0x7f3e70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:39.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.369+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e80077440 con 0x7f3e880adf20 2024-01-16T15:16:39.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.494+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 44) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3e80077810 con 0x7f3e880adf20 2024-01-16T15:16:39.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:39.604+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75492", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e8815f170 con 0x7f3e6006c5e0 2024-01-16T15:16:43.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:43.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:46.907 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:46.920+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:46.907 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:46.920+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:46.907 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:46.920+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:46.907 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:46.920+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:46.907 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:16:46.920+0000 7f0101400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:49.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:16:49.496+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 8 ==== mgrmap(e 45) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f3e800adf60 con 0x7f3e880adf20 2024-01-16T15:17:08.161 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:17:08.172+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:08.161 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:17:08.172+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:08.161 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:17:08.172+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:08.161 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:17:08.172+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:08.161 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:17:08.172+0000 7f0100a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:09.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:17:09.497+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 9 ==== mgrmap(e 46) v1 ==== 91426+0+0 (secure 0 0 0) 0x7f3e8004c5a0 con 0x7f3e880adf20 2024-01-16T15:17:13.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:13.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:29.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:17:29.503+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mon.0 v2:172.21.15.125:3300/0 10 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f3e8002fb20 con 0x7f3e880adf20 2024-01-16T15:17:43.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:43.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:13.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:13.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:37.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.338+0000 7f3e7e000640 1 -- 172.21.15.125:0/2575695086 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3e70001930 con 0x7f3e6006c5e0 2024-01-16T15:18:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 msgr2=0x7f3e6006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 0x7f3e6006eaa0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3e8815aee0 tx=0x7f3e70009040 comp rx=0 tx=0).stop 2024-01-16T15:18:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 msgr2=0x7f3e880af5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009890 tx=0x7f3e800046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 shutdown_connections 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e8815bf10 0x7f3e880b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e880ccb50 0x7f3e880afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e6006c5e0 0x7f3e6006eaa0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 --2- 172.21.15.125:0/2575695086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e880adf20 0x7f3e880af5a0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 >> 172.21.15.125:0/2575695086 conn(0x7f3e8806a7d0 msgr2=0x7f3e8806abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 shutdown_connections 2024-01-16T15:18:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:37.343+0000 7f3e87200640 1 -- 172.21.15.125:0/2575695086 wait complete. 2024-01-16T15:18:39.354 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75493 2024-01-16T15:18:39.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.586+0000 7f0e25000640 1 Processor -- start 2024-01-16T15:18:39.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.587+0000 7f0e25000640 1 -- start start 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.588+0000 7f0e25000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 0x7f0e200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.588+0000 7f0e25000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2015c5f0 con 0x7f0e2015f9c0 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.588+0000 7f0e25000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2006c470 con 0x7f0e2013df80 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.588+0000 7f0e25000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2006c750 con 0x7f0e2013e360 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.588+0000 7f0e1e800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0e2015f9c0 0x7f0e200cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53432/0 (socket says 172.21.15.125:53432) 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1e800640 1 -- 172.21.15.125:0/1787242986 learned_addr learned my addr 172.21.15.125:0/1787242986 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1d400640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 0x7f0e200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742305435 0 0) 0x7f0e2015c5f0 con 0x7f0e2015f9c0 2024-01-16T15:18:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e00003520 con 0x7f0e2015f9c0 2024-01-16T15:18:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345279081 0 0) 0x7f0e00003520 con 0x7f0e2015f9c0 2024-01-16T15:18:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 msgr2=0x7f0e200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 0x7f0e200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 >> v1:172.21.15.137:6789/0 conn(0x7f0e2013df80 legacy=0x7f0e2015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2006ca30 con 0x7f0e2015f9c0 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0e180019f0 con 0x7f0e2015f9c0 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 >> v1:172.21.15.125:6789/0 conn(0x7f0e2015f9c0 legacy=0x7f0e200cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 0x7f0e000039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.589+0000 7f0e1ca00640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e00003f70 0x7f0e000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1ca00640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2006c750 con 0x7f0e00004d00 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0000bac0 con 0x7f0e00003f70 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0000bc80 con 0x7f0e2013e360 2024-01-16T15:18:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 -- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 msgr2=0x7f0e000039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013e360 0x7f0e000039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 -- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e00003f70 msgr2=0x7f0e000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e00003f70 0x7f0e000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.590+0000 7f0e1e800640 1 -- 172.21.15.125:0/1787242986 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00003520 con 0x7f0e00004d00 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.591+0000 7f0e1e800640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e00005180 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800a760 tx=0x7f0e000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03dce7c1289e5d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18034630 con 0x7f0e00004d00 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1802f070 con 0x7f0e00004d00 2024-01-16T15:18:39.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e1ca00640 1 -- 172.21.15.125:0/1787242986 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18034a50 con 0x7f0e00004d00 2024-01-16T15:18:39.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 -- 172.21.15.125:0/1787242986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 msgr2=0x7f0e00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 --2- 172.21.15.125:0/1787242986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e00005180 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800a760 tx=0x7f0e000057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e1e800640 1 -- 172.21.15.125:0/1787242986 reap_dead start 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 -- 172.21.15.125:0/1787242986 shutdown_connections 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 -- 172.21.15.125:0/1787242986 >> 172.21.15.125:0/1787242986 conn(0x7f0e2006a800 msgr2=0x7f0e2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 -- 172.21.15.125:0/1787242986 shutdown_connections 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.592+0000 7f0e25000640 1 -- 172.21.15.125:0/1787242986 wait complete. 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.594+0000 7f0e25000640 1 Processor -- start 2024-01-16T15:18:39.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.594+0000 7f0e25000640 1 -- start start 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013d640 0x7f0e200b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e20156360 0x7f0e201567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e20156d20 con 0x7f0e00004d00 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200b3580 con 0x7f0e20156360 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e25000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200b4100 con 0x7f0e2013d640 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.595+0000 7f0e1de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57432/0 (socket says 172.21.15.125:57432) 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 -- 172.21.15.125:0/2025642798 learned_addr learned my addr 172.21.15.125:0/2025642798 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1e800640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e20156360 0x7f0e201567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1d400640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013d640 0x7f0e200b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 -- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013d640 msgr2=0x7f0e200b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013d640 0x7f0e200b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 -- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e20156360 msgr2=0x7f0e201567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e20156360 0x7f0e201567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 -- 172.21.15.125:0/2025642798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200b2720 con 0x7f0e00004d00 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e1de00640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000c430 tx=0x7f0e1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.596+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100052d0 con 0x7f0e00004d00 2024-01-16T15:18:39.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.597+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e10005430 con 0x7f0e00004d00 2024-01-16T15:18:39.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.597+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10003830 con 0x7f0e00004d00 2024-01-16T15:18:39.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.597+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e200b37d0 con 0x7f0e00004d00 2024-01-16T15:18:39.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.597+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e200b3d90 con 0x7f0e00004d00 2024-01-16T15:18:39.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.598+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e200675d0 con 0x7f0e00004d00 2024-01-16T15:18:39.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.599+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f0e10003990 con 0x7f0e00004d00 2024-01-16T15:18:39.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.600+0000 7f0e16000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 0x7f0df806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.600+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e10087de0 con 0x7f0e00004d00 2024-01-16T15:18:39.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.601+0000 7f0e1d400640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 0x7f0df806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.601+0000 7f0e1d400640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 0x7f0df806ebc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004770 tx=0x7f0e08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.605+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e10057d60 con 0x7f0e00004d00 2024-01-16T15:18:39.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:39.833+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75493", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e2006cff0 con 0x7f0df806c700 2024-01-16T15:18:42.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.265+0000 7f0e16000640 1 -- 172.21.15.125:0/2025642798 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0e080095c0 con 0x7f0df806c700 2024-01-16T15:18:42.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 msgr2=0x7f0df806ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 0x7f0df806ebc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004770 tx=0x7f0e08008040 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 msgr2=0x7f0e2013ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000c430 tx=0x7f0e1000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 shutdown_connections 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e20156360 0x7f0e201567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e2013d640 0x7f0e200b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0df806c700 0x7f0df806ebc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 --2- 172.21.15.125:0/2025642798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e00004d00 0x7f0e2013ba50 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.268+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 >> 172.21.15.125:0/2025642798 conn(0x7f0e2006a800 msgr2=0x7f0e200cdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.269+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 shutdown_connections 2024-01-16T15:18:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:42.269+0000 7f0e25000640 1 -- 172.21.15.125:0/2025642798 wait complete. 2024-01-16T15:18:43.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:43.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:44.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75494 2024-01-16T15:18:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.508+0000 7fa07f400640 1 Processor -- start 2024-01-16T15:18:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.508+0000 7fa07f400640 1 -- start start 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.509+0000 7fa07f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa0780b29c0 0x7fa0780d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.509+0000 7fa07f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0780d1bb0 0x7fa0780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.509+0000 7fa07f400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780d3f00 con 0x7fa078160920 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.509+0000 7fa07f400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0780d2040 con 0x7fa0780b29c0 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.509+0000 7fa07f400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa078160d00 con 0x7fa0780d1bb0 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa0780b29c0 0x7fa0780d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa0780b29c0 0x7fa0780d1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57436/0 (socket says 172.21.15.125:57436) 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 -- 172.21.15.125:0/1018833520 learned_addr learned my addr 172.21.15.125:0/1018833520 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:18:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07e000640 1 --1- 172.21.15.125:0/1018833520 >> v1:172.21.15.125:6789/0 conn(0x7fa078160920 0x7fa0780d0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53442/0 (socket says 172.21.15.125:53442) 2024-01-16T15:18:44.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 -- 172.21.15.125:0/1018833520 >> v2:172.21.15.137:3300/0 conn(0x7fa0780d1bb0 msgr2=0x7fa0780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:44.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 --2- 172.21.15.125:0/1018833520 >> v2:172.21.15.137:3300/0 conn(0x7fa0780d1bb0 0x7fa0780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 -- 172.21.15.125:0/1018833520 >> v1:172.21.15.125:6789/0 conn(0x7fa078160920 legacy=0x7fa0780d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.510+0000 7fa07d600640 1 -- 172.21.15.125:0/1018833520 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780cfb30 con 0x7fa0780b29c0 2024-01-16T15:18:44.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.511+0000 7fa07d600640 1 --2- 172.21.15.125:0/1018833520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0780b29c0 0x7fa0780d1670 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fa0780b2950 tx=0x7fa06002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279aabbe18988ad8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07cc00640 1 -- 172.21.15.125:0/1018833520 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060030860 con 0x7fa0780b29c0 2024-01-16T15:18:44.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07cc00640 1 -- 172.21.15.125:0/1018833520 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0600309c0 con 0x7fa0780b29c0 2024-01-16T15:18:44.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07cc00640 1 -- 172.21.15.125:0/1018833520 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060030d50 con 0x7fa0780b29c0 2024-01-16T15:18:44.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 -- 172.21.15.125:0/1018833520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0780b29c0 msgr2=0x7fa0780d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 --2- 172.21.15.125:0/1018833520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0780b29c0 0x7fa0780d1670 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fa0780b2950 tx=0x7fa06002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 -- 172.21.15.125:0/1018833520 shutdown_connections 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 --2- 172.21.15.125:0/1018833520 >> v2:172.21.15.137:3300/0 conn(0x7fa0780d1bb0 0x7fa0780ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 --2- 172.21.15.125:0/1018833520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0780b29c0 0x7fa0780d1670 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.512+0000 7fa07f400640 1 -- 172.21.15.125:0/1018833520 >> 172.21.15.125:0/1018833520 conn(0x7fa0780d3090 msgr2=0x7fa0780d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.513+0000 7fa07f400640 1 -- 172.21.15.125:0/1018833520 shutdown_connections 2024-01-16T15:18:44.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.513+0000 7fa07f400640 1 -- 172.21.15.125:0/1018833520 wait complete. 2024-01-16T15:18:44.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.514+0000 7fa07f400640 1 Processor -- start 2024-01-16T15:18:44.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.514+0000 7fa07f400640 1 -- start start 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0780b29c0 0x7fa0780cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0780d1bb0 0x7fa0781f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07815b520 con 0x7fa078160920 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07815bc80 con 0x7fa0780d1bb0 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.515+0000 7fa07f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780cc820 con 0x7fa0780b29c0 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57440/0 (socket says 172.21.15.125:57440) 2024-01-16T15:18:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 -- 172.21.15.125:0/504689475 learned_addr learned my addr 172.21.15.125:0/504689475 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07d600640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0780d1bb0 0x7fa0781f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07e000640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0780b29c0 0x7fa0780cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 -- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0780b29c0 msgr2=0x7fa0780cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0780b29c0 0x7fa0780cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 -- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0780d1bb0 msgr2=0x7fa0781f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0780d1bb0 0x7fa0781f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 -- 172.21.15.125:0/504689475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600094d0 con 0x7fa078160920 2024-01-16T15:18:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.516+0000 7fa07ea00640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fa07000b970 tx=0x7fa07000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.517+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07000cae0 con 0x7fa078160920 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.517+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07000cc60 con 0x7fa078160920 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.517+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070014440 con 0x7fa078160920 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.517+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07815a8a0 con 0x7fa078160920 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.517+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07815ae60 con 0x7fa078160920 2024-01-16T15:18:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.518+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0780b6760 con 0x7fa078160920 2024-01-16T15:18:44.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.519+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fa07001d3b0 con 0x7fa078160920 2024-01-16T15:18:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.520+0000 7fa076000640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 0x7fa05006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.520+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0700887b0 con 0x7fa078160920 2024-01-16T15:18:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.520+0000 7fa07e000640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 0x7fa05006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.521+0000 7fa07e000640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 0x7fa05006ebc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa068007fd0 tx=0x7fa068009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.525+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa070055070 con 0x7fa078160920 2024-01-16T15:18:44.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:44.756+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75494", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0780d03d0 con 0x7fa05006c700 2024-01-16T15:18:47.246 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.256+0000 7fa076000640 1 -- 172.21.15.125:0/504689475 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa068001210 con 0x7fa05006c700 2024-01-16T15:18:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 msgr2=0x7fa05006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 0x7fa05006ebc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa068007fd0 tx=0x7fa068009040 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 msgr2=0x7fa07815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fa07000b970 tx=0x7fa07000be30 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 shutdown_connections 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa05006c700 0x7fa05006ebc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa078160920 0x7fa07815cd70 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0780d1bb0 0x7fa0781f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 --2- 172.21.15.125:0/504689475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0780b29c0 0x7fa0780cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.259+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 >> 172.21.15.125:0/504689475 conn(0x7fa0780d3090 msgr2=0x7fa0780733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.260+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 shutdown_connections 2024-01-16T15:18:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:47.260+0000 7fa07f400640 1 -- 172.21.15.125:0/504689475 wait complete. 2024-01-16T15:18:49.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75495 2024-01-16T15:18:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.502+0000 7f843ae00640 1 Processor -- start 2024-01-16T15:18:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.502+0000 7f843ae00640 1 -- start start 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 0x7f8434070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843ae00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f843415bf10 0x7f84340cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843ae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843415f4e0 con 0x7f84340cc980 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843ae00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843415f970 con 0x7f843415bf10 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84340cce10 con 0x7f84340adf20 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843a400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f84340cc980 0x7f8434143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41740/0 (socket says 172.21.15.125:41740) 2024-01-16T15:18:49.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f843a400640 1 -- 172.21.15.125:0/66607010 learned_addr learned my addr 172.21.15.125:0/66607010 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:18:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.504+0000 7f8439000640 1 --2- 172.21.15.125:0/66607010 >> v2:172.21.15.137:3300/0 conn(0x7f843415bf10 0x7f84340cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8439a00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 0x7f8434070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496284186 0 0) 0x7f843415f4e0 con 0x7f84340cc980 2024-01-16T15:18:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8414003520 con 0x7f84340cc980 2024-01-16T15:18:49.496 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899062733 0 0) 0x7f8414003520 con 0x7f84340cc980 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 msgr2=0x7f8434070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 0x7f8434070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 >> v2:172.21.15.137:3300/0 conn(0x7f843415bf10 msgr2=0x7f84340cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 --2- 172.21.15.125:0/66607010 >> v2:172.21.15.137:3300/0 conn(0x7f843415bf10 0x7f84340cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434071130 con 0x7f84340cc980 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f842c0019f0 con 0x7f84340cc980 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 >> v1:172.21.15.125:6789/0 conn(0x7f84340cc980 legacy=0x7f8434143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8414003970 0x7f8414003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140047c0 0x7f8414004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 0x7f8414005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843415f970 con 0x7f84140047c0 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84340cce10 con 0x7f8414003970 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.505+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841400b980 con 0x7f84340adf20 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140047c0 0x7f8414004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439a00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8414003970 0x7f8414003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 -- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 msgr2=0x7f8414005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340adf20 0x7f8414005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 -- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8414003970 msgr2=0x7f8414003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8414003970 0x7f8414003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.506+0000 7f8439000640 1 -- 172.21.15.125:0/66607010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8414003520 con 0x7f84140047c0 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.507+0000 7f8439000640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140047c0 0x7f8414004c40 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f841c009af0 tx=0x7f84140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9759c19eae0b8df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.507+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c02c710 con 0x7f84140047c0 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.507+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c02c870 con 0x7f84140047c0 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.507+0000 7f8433e00640 1 -- 172.21.15.125:0/66607010 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c02cc00 con 0x7f84140047c0 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 -- 172.21.15.125:0/66607010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140047c0 msgr2=0x7f8414004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 --2- 172.21.15.125:0/66607010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140047c0 0x7f8414004c40 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f841c009af0 tx=0x7f84140057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843a400640 1 -- 172.21.15.125:0/66607010 reap_dead start 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 -- 172.21.15.125:0/66607010 shutdown_connections 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 -- 172.21.15.125:0/66607010 >> 172.21.15.125:0/66607010 conn(0x7f843406a7d0 msgr2=0x7f843406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 -- 172.21.15.125:0/66607010 shutdown_connections 2024-01-16T15:18:49.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.508+0000 7f843ae00640 1 -- 172.21.15.125:0/66607010 wait complete. 2024-01-16T15:18:49.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.510+0000 7f843ae00640 1 Processor -- start 2024-01-16T15:18:49.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.510+0000 7f843ae00640 1 -- start start 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340ca5f0 0x7f84340c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84340c8d00 0x7f8434141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84341420b0 con 0x7f84340ca9d0 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84340c92c0 con 0x7f84340c8d00 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f843ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84340c95a0 con 0x7f84340ca5f0 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f8439000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f8439000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39268/0 (socket says 172.21.15.125:39268) 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.511+0000 7f8439000640 1 -- 172.21.15.125:0/1445306522 learned_addr learned my addr 172.21.15.125:0/1445306522 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439a00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340ca5f0 0x7f84340c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f843a400640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84340c8d00 0x7f8434141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 -- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340ca5f0 msgr2=0x7f84340c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340ca5f0 0x7f84340c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 -- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84340c8d00 msgr2=0x7f8434141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84340c8d00 0x7f8434141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 -- 172.21.15.125:0/1445306522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84340c9880 con 0x7f84340ca9d0 2024-01-16T15:18:49.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8439000640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f841c037040 tx=0x7f841c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c003810 con 0x7f84340ca9d0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c02de90 con 0x7f84340ca9d0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c003bb0 con 0x7f84340ca9d0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.512+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84340c9b60 con 0x7f84340ca9d0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.513+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84340c5bc0 con 0x7f84340ca9d0 2024-01-16T15:18:49.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.514+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8434067450 con 0x7f84340ca9d0 2024-01-16T15:18:49.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.515+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f841c003d10 con 0x7f84340ca9d0 2024-01-16T15:18:49.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.515+0000 7f8431600640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 0x7f840c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.516+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f841c0aa6a0 con 0x7f84340ca9d0 2024-01-16T15:18:49.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.516+0000 7f8439a00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 0x7f840c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.517+0000 7f8439a00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 0x7f840c06ebc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8424009930 tx=0x7f8424008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.521+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f841c077080 con 0x7f84340ca9d0 2024-01-16T15:18:49.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:49.746+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75495", "target": ["mon-mgr", ""]}) v1 -- 0x7f843415ee90 con 0x7f840c06c700 2024-01-16T15:18:59.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:18:59.523+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f841c077450 con 0x7f84340ca9d0 2024-01-16T15:19:08.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.572+0000 7f8431600640 1 -- 172.21.15.125:0/1445306522 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f84240095c0 con 0x7f840c06c700 2024-01-16T15:19:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.575+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 msgr2=0x7f840c06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.575+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 0x7f840c06ebc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8424009930 tx=0x7f8424008040 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.575+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 msgr2=0x7f8434141630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.575+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f841c037040 tx=0x7f841c004730 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 shutdown_connections 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84340c8d00 0x7f8434141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f840c06c700 0x7f840c06ebc0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84340ca9d0 0x7f8434141630 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 --2- 172.21.15.125:0/1445306522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84340ca5f0 0x7f84340c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 >> 172.21.15.125:0/1445306522 conn(0x7f843406a7d0 msgr2=0x7f84340ad370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 shutdown_connections 2024-01-16T15:19:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:08.576+0000 7f843ae00640 1 -- 172.21.15.125:0/1445306522 wait complete. 2024-01-16T15:19:10.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75496 2024-01-16T15:19:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.829+0000 7f2a93e00640 1 Processor -- start 2024-01-16T15:19:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.829+0000 7f2a93e00640 1 -- start start 2024-01-16T15:19:10.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a93e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2a8c143ae0 0x7f2a8c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 0x7f2a8c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.822 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a93e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0cfc30 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a93e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b1ee0 con 0x7f2a8c143ae0 2024-01-16T15:19:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a93e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b2780 con 0x7f2a8c0cf7d0 2024-01-16T15:19:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a91e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2a8c15e1e0 0x7f2a8c15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34622/0 (socket says 172.21.15.125:34622) 2024-01-16T15:19:10.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a91e00640 1 -- 172.21.15.125:0/2002782193 learned_addr learned my addr 172.21.15.125:0/2002782193 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.831+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779404874 0 0) 0x7f2a8c0cfc30 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a6c003520 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a91400640 1 --2- 172.21.15.125:0/2002782193 >> v2:172.21.15.137:3300/0 conn(0x7f2a8c143ae0 0x7f2a8c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a90a00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 0x7f2a8c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1396645559 0 0) 0x7f2a6c003520 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 msgr2=0x7f2a8c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 0x7f2a8c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 >> v2:172.21.15.137:3300/0 conn(0x7f2a8c143ae0 msgr2=0x7f2a8c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 --2- 172.21.15.125:0/2002782193 >> v2:172.21.15.137:3300/0 conn(0x7f2a8c143ae0 0x7f2a8c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c15e990 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2a840019f0 con 0x7f2a8c15e1e0 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 >> v1:172.21.15.125:6789/0 conn(0x7f2a8c15e1e0 legacy=0x7f2a8c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c003970 0x7f2a6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 0x7f2a6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a6c004d00 0x7f2a6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b1ee0 con 0x7f2a6c004d00 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b2780 con 0x7f2a6c003970 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c00ba30 con 0x7f2a8c0cf7d0 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.832+0000 7f2a91e00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a6c004d00 0x7f2a6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91400640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c003970 0x7f2a6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91e00640 1 -- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 msgr2=0x7f2a6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91e00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c0cf7d0 0x7f2a6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91e00640 1 -- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c003970 msgr2=0x7f2a6c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91e00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c003970 0x7f2a6c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.833+0000 7f2a91e00640 1 -- 172.21.15.125:0/2002782193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c003520 con 0x7f2a6c004d00 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.834+0000 7f2a91e00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a6c004d00 0x7f2a6c005180 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400a860 tx=0x7f2a6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1495c6d4980a8db1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.834+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84034630 con 0x7f2a6c004d00 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.834+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8402f070 con 0x7f2a6c004d00 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.834+0000 7f2a8be00640 1 -- 172.21.15.125:0/2002782193 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840349d0 con 0x7f2a6c004d00 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 -- 172.21.15.125:0/2002782193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a6c004d00 msgr2=0x7f2a6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 --2- 172.21.15.125:0/2002782193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a6c004d00 0x7f2a6c005180 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400a860 tx=0x7f2a6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a91e00640 1 -- 172.21.15.125:0/2002782193 reap_dead start 2024-01-16T15:19:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 -- 172.21.15.125:0/2002782193 shutdown_connections 2024-01-16T15:19:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 -- 172.21.15.125:0/2002782193 >> 172.21.15.125:0/2002782193 conn(0x7f2a8c06a800 msgr2=0x7f2a8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 -- 172.21.15.125:0/2002782193 shutdown_connections 2024-01-16T15:19:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.835+0000 7f2a93e00640 1 -- 172.21.15.125:0/2002782193 wait complete. 2024-01-16T15:19:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.837+0000 7f2a93e00640 1 Processor -- start 2024-01-16T15:19:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.837+0000 7f2a93e00640 1 -- start start 2024-01-16T15:19:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c004d00 0x7f2a8c06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a8c139380 0x7f2a8c1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c1373d0 0x7f2a8c137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c137dc0 con 0x7f2a8c139380 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c159f10 con 0x7f2a6c004d00 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a93e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c13a450 con 0x7f2a8c1373d0 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a8c139380 0x7f2a8c1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a8c139380 0x7f2a8c1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58772/0 (socket says 172.21.15.125:58772) 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 -- 172.21.15.125:0/2016493522 learned_addr learned my addr 172.21.15.125:0/2016493522 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 -- 172.21.15.125:0/2016493522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c1373d0 msgr2=0x7f2a8c137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2a8c1373d0 0x7f2a8c137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 -- 172.21.15.125:0/2016493522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c004d00 msgr2=0x7f2a8c06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a6c004d00 0x7f2a8c06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.838+0000 7f2a90a00640 1 -- 172.21.15.125:0/2016493522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c139840 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a90a00640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a8c139380 0x7f2a8c1599d0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400d830 tx=0x7f2a7400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7400b640 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7400b7a0 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7400bb30 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a93e00640 1 -- 172.21.15.125:0/2016493522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c139b20 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.839+0000 7f2a93e00640 1 -- 172.21.15.125:0/2016493522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c13a0e0 con 0x7f2a8c139380 2024-01-16T15:19:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.840+0000 7f2a93e00640 1 -- 172.21.15.125:0/2016493522 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c0675d0 con 0x7f2a8c139380 2024-01-16T15:19:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.841+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2a74014410 con 0x7f2a8c139380 2024-01-16T15:19:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.842+0000 7f2a89600640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2a6406c5e0 0x7f2a6406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.842+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a74088e20 con 0x7f2a8c139380 2024-01-16T15:19:10.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.842+0000 7f2a91400640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2a6406c5e0 0x7f2a6406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.843+0000 7f2a91400640 1 --2- 172.21.15.125:0/2016493522 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2a6406c5e0 0x7f2a6406eaa0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c006a90 tx=0x7f2a7c006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.838 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:10.847+0000 7f2a89600640 1 -- 172.21.15.125:0/2016493522 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a74055740 con 0x7f2a8c139380 2024-01-16T15:19:11.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:19:11.076+0000 7f2a93e00640 1 -- 172.21.15.125:0/2016493522 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone75496", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a8c15e640 con 0x7f2a6406c5e0 2024-01-16T15:19:13.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:13.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:43.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:43.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:14.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:14.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:44.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:44.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-16T15:21:10.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:21:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.935+0000 7f3e78600640 1 Processor -- start 2024-01-16T15:21:10.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.936+0000 7f3e78600640 1 -- start start 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e78600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e700cf9a0 0x7f3e70070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e78600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e70070ea0 0x7f3e700cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e78600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e70071690 con 0x7f3e7014b2c0 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e78600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7014b6a0 con 0x7f3e700cf9a0 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e78600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70071970 con 0x7f3e70070ea0 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e75c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3e7014b2c0 0x7f3e700cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40360/0 (socket says 172.21.15.125:40360) 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e75c00640 1 -- 172.21.15.125:0/808731995 learned_addr learned my addr 172.21.15.125:0/808731995 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.937+0000 7f3e75200640 1 --2- 172.21.15.125:0/808731995 >> v2:172.21.15.125:3300/0 conn(0x7f3e700cf9a0 0x7f3e70070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:10.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.938+0000 7f3e75200640 1 -- 172.21.15.125:0/808731995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e70070ea0 msgr2=0x7f3e700cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:10.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.938+0000 7f3e75200640 1 --2- 172.21.15.125:0/808731995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e70070ea0 0x7f3e700cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.938+0000 7f3e75200640 1 -- 172.21.15.125:0/808731995 >> v1:172.21.15.125:6789/0 conn(0x7f3e7014b2c0 legacy=0x7f3e700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:10.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.938+0000 7f3e75200640 1 -- 172.21.15.125:0/808731995 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e70071c50 con 0x7f3e700cf9a0 2024-01-16T15:21:10.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.939+0000 7f3e75200640 1 --2- 172.21.15.125:0/808731995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e700cf9a0 0x7f3e70070960 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f29805a7ed87dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.939+0000 7f3e6fe00640 1 -- 172.21.15.125:0/808731995 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5802d040 con 0x7f3e700cf9a0 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.939+0000 7f3e6fe00640 1 -- 172.21.15.125:0/808731995 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58031810 con 0x7f3e700cf9a0 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.939+0000 7f3e6fe00640 1 -- 172.21.15.125:0/808731995 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58031ba0 con 0x7f3e700cf9a0 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 -- 172.21.15.125:0/808731995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e700cf9a0 msgr2=0x7f3e70070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 --2- 172.21.15.125:0/808731995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e700cf9a0 0x7f3e70070960 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 -- 172.21.15.125:0/808731995 shutdown_connections 2024-01-16T15:21:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 --2- 172.21.15.125:0/808731995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e70070ea0 0x7f3e700cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 --2- 172.21.15.125:0/808731995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e700cf9a0 0x7f3e70070960 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 -- 172.21.15.125:0/808731995 >> 172.21.15.125:0/808731995 conn(0x7f3e7006a800 msgr2=0x7f3e7006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 -- 172.21.15.125:0/808731995 shutdown_connections 2024-01-16T15:21:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.940+0000 7f3e78600640 1 -- 172.21.15.125:0/808731995 wait complete. 2024-01-16T15:21:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.942+0000 7f3e78600640 1 Processor -- start 2024-01-16T15:21:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.942+0000 7f3e78600640 1 -- start start 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e700cf9a0 0x7f3e70164830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7014b2c0 0x7f3e700cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70144480 con 0x7f3e70070ea0 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7015ef60 con 0x7f3e700cf9a0 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e78600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70143e70 con 0x7f3e7014b2c0 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e75c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e75c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33830/0 (socket says 172.21.15.125:33830) 2024-01-16T15:21:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.943+0000 7f3e75c00640 1 -- 172.21.15.125:0/2925237257 learned_addr learned my addr 172.21.15.125:0/2925237257 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 -- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7014b2c0 msgr2=0x7f3e700cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7014b2c0 0x7f3e700cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 -- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e700cf9a0 msgr2=0x7f3e70164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e700cf9a0 0x7f3e70164830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 -- 172.21.15.125:0/2925237257 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58009400 con 0x7f3e70070ea0 2024-01-16T15:21:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e75c00640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000c480 tx=0x7f3e6000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60005ce0 con 0x7f3e70070ea0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60005e40 con 0x7f3e70070ea0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e600072f0 con 0x7f3e70070ea0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e70162e60 con 0x7f3e70070ea0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.944+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e70164d70 con 0x7f3e70070ea0 2024-01-16T15:21:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.946+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e700675d0 con 0x7f3e70070ea0 2024-01-16T15:21:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.947+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3e60007510 con 0x7f3e70070ea0 2024-01-16T15:21:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.947+0000 7f3e6d600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 0x7f3e4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.948+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3e60088160 con 0x7f3e70070ea0 2024-01-16T15:21:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.948+0000 7f3e75200640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 0x7f3e4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.949+0000 7f3e75200640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 0x7f3e4806eaa0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3e58036000 tx=0x7f3e58030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:10.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:10.952+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e60054730 con 0x7f3e70070ea0 2024-01-16T15:21:11.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.165+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e70071330 con 0x7f3e4806c5e0 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.167+0000 7f3e6d600640 1 -- 172.21.15.125:0/2925237257 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3e58031610 con 0x7f3e4806c5e0 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "cephfs" 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:21:11.161 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:21:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 msgr2=0x7f3e4806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 0x7f3e4806eaa0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3e58036000 tx=0x7f3e58030000 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 msgr2=0x7f3e7013f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000c480 tx=0x7f3e6000c940 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 shutdown_connections 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7014b2c0 0x7f3e700cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e700cf9a0 0x7f3e70164830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4806c5e0 0x7f3e4806eaa0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 --2- 172.21.15.125:0/2925237257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e70070ea0 0x7f3e7013f840 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 >> 172.21.15.125:0/2925237257 conn(0x7f3e7006a800 msgr2=0x7f3e70148720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 shutdown_connections 2024-01-16T15:21:11.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.170+0000 7f3e78600640 1 -- 172.21.15.125:0/2925237257 wait complete. 2024-01-16T15:21:11.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:21:11.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.393+0000 7f441b000640 1 Processor -- start 2024-01-16T15:21:11.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.393+0000 7f441b000640 1 -- start start 2024-01-16T15:21:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f441b000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44140cf9a0 0x7f4414166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f441b000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441415c5f0 con 0x7f441413df80 2024-01-16T15:21:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f441b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441406c470 con 0x7f441413e360 2024-01-16T15:21:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f441b000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441406c750 con 0x7f44140cf9a0 2024-01-16T15:21:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f4413e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f441413df80 0x7f441415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40364/0 (socket says 172.21.15.125:40364) 2024-01-16T15:21:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f4413e00640 1 -- 172.21.15.125:0/834739992 learned_addr learned my addr 172.21.15.125:0/834739992 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f4419000640 1 --2- 172.21.15.125:0/834739992 >> v2:172.21.15.125:3300/0 conn(0x7f44140cf9a0 0x7f4414166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f4412a00640 1 -- 172.21.15.125:0/834739992 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775506006 0 0) 0x7f441415c5f0 con 0x7f441413df80 2024-01-16T15:21:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.395+0000 7f4412a00640 1 -- 172.21.15.125:0/834739992 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f4003520 con 0x7f441413df80 2024-01-16T15:21:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.396+0000 7f4419000640 1 -- 172.21.15.125:0/834739992 >> v1:172.21.15.137:6789/0 conn(0x7f441413e360 legacy=0x7f44140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.396+0000 7f4419000640 1 -- 172.21.15.125:0/834739992 >> v1:172.21.15.125:6789/0 conn(0x7f441413df80 legacy=0x7f441415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.396+0000 7f4419000640 1 -- 172.21.15.125:0/834739992 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441406ca30 con 0x7f44140cf9a0 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f4419000640 1 --2- 172.21.15.125:0/834739992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f4414166030 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f440c009d90 tx=0x7f440c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3374646570cacf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f4412a00640 1 -- 172.21.15.125:0/834739992 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c02ccf0 con 0x7f44140cf9a0 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f4412a00640 1 -- 172.21.15.125:0/834739992 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c02ce50 con 0x7f44140cf9a0 2024-01-16T15:21:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f4412a00640 1 -- 172.21.15.125:0/834739992 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c034640 con 0x7f44140cf9a0 2024-01-16T15:21:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f441b000640 1 -- 172.21.15.125:0/834739992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 msgr2=0x7f4414166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.397+0000 7f441b000640 1 --2- 172.21.15.125:0/834739992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f4414166030 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f440c009d90 tx=0x7f440c02bf40 comp rx=0 tx=0).stop 2024-01-16T15:21:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.398+0000 7f441b000640 1 -- 172.21.15.125:0/834739992 shutdown_connections 2024-01-16T15:21:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.398+0000 7f441b000640 1 --2- 172.21.15.125:0/834739992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f4414166030 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.398+0000 7f441b000640 1 -- 172.21.15.125:0/834739992 >> 172.21.15.125:0/834739992 conn(0x7f441406a800 msgr2=0x7f441406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.398+0000 7f441b000640 1 -- 172.21.15.125:0/834739992 shutdown_connections 2024-01-16T15:21:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.398+0000 7f441b000640 1 -- 172.21.15.125:0/834739992 wait complete. 2024-01-16T15:21:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.399+0000 7f441b000640 1 Processor -- start 2024-01-16T15:21:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.400+0000 7f441b000640 1 -- start start 2024-01-16T15:21:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f441413df80 0x7f44141f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f441413e360 0x7f4414071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141f6cf0 con 0x7f44140cf9a0 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141f6fd0 con 0x7f441413e360 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f441b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44141f72b0 con 0x7f441413df80 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33844/0 (socket says 172.21.15.125:33844) 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 -- 172.21.15.125:0/2823945738 learned_addr learned my addr 172.21.15.125:0/2823945738 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 -- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f441413df80 msgr2=0x7f44141f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f441413df80 0x7f44141f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 -- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f441413e360 msgr2=0x7f4414071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f441413e360 0x7f4414071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 -- 172.21.15.125:0/2823945738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c009a70 con 0x7f44140cf9a0 2024-01-16T15:21:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.401+0000 7f4413e00640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f440400c410 tx=0x7f440400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.402+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404005ce0 con 0x7f44140cf9a0 2024-01-16T15:21:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.402+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404005e40 con 0x7f44140cf9a0 2024-01-16T15:21:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.402+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44040072f0 con 0x7f44140cf9a0 2024-01-16T15:21:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.402+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44140a9cc0 con 0x7f44140cf9a0 2024-01-16T15:21:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.402+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441413dd10 con 0x7f44140cf9a0 2024-01-16T15:21:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.403+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44140aa1e0 con 0x7f44140cf9a0 2024-01-16T15:21:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.404+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f4404007510 con 0x7f44140cf9a0 2024-01-16T15:21:11.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.405+0000 7f440be00640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 0x7f43ec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.405+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4404088270 con 0x7f44140cf9a0 2024-01-16T15:21:11.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.405+0000 7f4413400640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 0x7f43ec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.406+0000 7f4413400640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 0x7f43ec06eaa0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000f60 tx=0x7f43fc0079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.409+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4404054890 con 0x7f44140cf9a0 2024-01-16T15:21:11.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.621+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f441406cff0 con 0x7f43ec06c5e0 2024-01-16T15:21:11.622 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:11.629+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:11.622 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:11.629+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:11.623 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:11.629+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:11.623 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:11.629+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:11.623 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:11.629+0000 7f00fb000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.643+0000 7f440be00640 1 -- 172.21.15.125:0/2823945738 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7f43fc005080 con 0x7f43ec06c5e0 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "sv1" 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75495" 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75497" 2024-01-16T15:21:11.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75492" 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75493" 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75496" 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75491" 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2024-01-16T15:21:11.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:11.639 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "subvol_clone75494" 2024-01-16T15:21:11.639 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:21:11.639 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 msgr2=0x7f43ec06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 0x7f43ec06eaa0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000f60 tx=0x7f43fc0079d0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 msgr2=0x7f441413d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f440400c410 tx=0x7f440400c8d0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 shutdown_connections 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.645+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f441413e360 0x7f4414071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f441413df80 0x7f44141f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f43ec06c5e0 0x7f43ec06eaa0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 --2- 172.21.15.125:0/2823945738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44140cf9a0 0x7f441413d7d0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 >> 172.21.15.125:0/2823945738 conn(0x7f441406a800 msgr2=0x7f4414159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 shutdown_connections 2024-01-16T15:21:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.646+0000 7f441b000640 1 -- 172.21.15.125:0/2823945738 wait complete. 2024-01-16T15:21:11.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:21:11.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.849+0000 7f3946a00640 1 Processor -- start 2024-01-16T15:21:11.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.850+0000 7f3946a00640 1 -- start start 2024-01-16T15:21:11.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f3946a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f39400d1c40 0x7f39400b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f3946a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b3400 con 0x7f394015d610 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f3946a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b27f0 con 0x7f39400d17e0 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f3946a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400d1390 con 0x7f39400d1c40 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f393fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f394015d610 0x7f39400ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40370/0 (socket says 172.21.15.125:40370) 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f393fe00640 1 -- 172.21.15.125:0/3977522444 learned_addr learned my addr 172.21.15.125:0/3977522444 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:11.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.851+0000 7f3944a00640 1 --2- 172.21.15.125:0/3977522444 >> v2:172.21.15.125:3300/0 conn(0x7f39400d1c40 0x7f39400b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.852+0000 7f3944a00640 1 -- 172.21.15.125:0/3977522444 >> v1:172.21.15.137:6789/0 conn(0x7f39400d17e0 legacy=0x7f39400b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:11.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.852+0000 7f3944a00640 1 -- 172.21.15.125:0/3977522444 >> v1:172.21.15.125:6789/0 conn(0x7f394015d610 legacy=0x7f39400ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.852+0000 7f3944a00640 1 -- 172.21.15.125:0/3977522444 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400b2ad0 con 0x7f39400d1c40 2024-01-16T15:21:11.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.853+0000 7f3944a00640 1 --2- 172.21.15.125:0/3977522444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39400d1c40 0x7f39400b3fa0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f393802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d16372ea6d5ee8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.853+0000 7f393f400640 1 -- 172.21.15.125:0/3977522444 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393802cc60 con 0x7f39400d1c40 2024-01-16T15:21:11.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.853+0000 7f393f400640 1 -- 172.21.15.125:0/3977522444 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393802cdc0 con 0x7f39400d1c40 2024-01-16T15:21:11.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.853+0000 7f393f400640 1 -- 172.21.15.125:0/3977522444 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938034660 con 0x7f39400d1c40 2024-01-16T15:21:11.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 -- 172.21.15.125:0/3977522444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39400d1c40 msgr2=0x7f39400b3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 --2- 172.21.15.125:0/3977522444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39400d1c40 0x7f39400b3fa0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f393802beb0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 -- 172.21.15.125:0/3977522444 shutdown_connections 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 --2- 172.21.15.125:0/3977522444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39400d1c40 0x7f39400b3fa0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 -- 172.21.15.125:0/3977522444 >> 172.21.15.125:0/3977522444 conn(0x7f394009c540 msgr2=0x7f39400d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 -- 172.21.15.125:0/3977522444 shutdown_connections 2024-01-16T15:21:11.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.854+0000 7f3946a00640 1 -- 172.21.15.125:0/3977522444 wait complete. 2024-01-16T15:21:11.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.856+0000 7f3946a00640 1 Processor -- start 2024-01-16T15:21:11.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.856+0000 7f3946a00640 1 -- start start 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39400d17e0 0x7f3940156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 0x7f394015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400b6050 con 0x7f394015d610 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400b62a0 con 0x7f39400d1c40 2024-01-16T15:21:11.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3946a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39400b6580 con 0x7f39400d17e0 2024-01-16T15:21:11.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33900/0 (socket says 172.21.15.125:33900) 2024-01-16T15:21:11.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 -- 172.21.15.125:0/3799611847 learned_addr learned my addr 172.21.15.125:0/3799611847 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 -- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39400d17e0 msgr2=0x7f3940156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39400d17e0 0x7f3940156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 -- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 msgr2=0x7f394015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3937e00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 0x7f394015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 0x7f394015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:11.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3944a00640 1 -- 172.21.15.125:0/3799611847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938009a70 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.857+0000 7f3937e00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 0x7f394015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f3944a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f3938004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938040450 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39380405d0 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39400b6860 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938040980 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.858+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39400ca3d0 con 0x7f394015d610 2024-01-16T15:21:11.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.859+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39400ca620 con 0x7f394015d610 2024-01-16T15:21:11.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.860+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3938040ae0 con 0x7f394015d610 2024-01-16T15:21:11.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.861+0000 7f393cc00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 0x7f391806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:11.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.861+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f39380ab700 con 0x7f394015d610 2024-01-16T15:21:11.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.861+0000 7f393fe00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 0x7f391806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:11.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.862+0000 7f393fe00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 0x7f391806eaa0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39300015f0 tx=0x7f3930007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:11.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:11.866+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3938077d40 con 0x7f394015d610 2024-01-16T15:21:12.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.087+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39400b6410 con 0x7f391806c5e0 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.109+0000 7f393cc00640 1 -- 172.21.15.125:0/3799611847 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f39400b6410 con 0x7f391806c5e0 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "ss1" 2024-01-16T15:21:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: } 2024-01-16T15:21:12.104 INFO:teuthology.orchestra.run.smithi125.stdout:] 2024-01-16T15:21:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 msgr2=0x7f391806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 0x7f391806eaa0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39300015f0 tx=0x7f3930007a40 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 msgr2=0x7f39400cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f3938004a40 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 shutdown_connections 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f391806c5e0 0x7f391806eaa0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f394015d610 0x7f39400cb1b0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39400d1c40 0x7f394015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 --2- 172.21.15.125:0/3799611847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39400d17e0 0x7f3940156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.112+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 >> 172.21.15.125:0/3799611847 conn(0x7f394009c540 msgr2=0x7f39400b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.113+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 shutdown_connections 2024-01-16T15:21:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.113+0000 7f3946a00640 1 -- 172.21.15.125:0/3799611847 wait complete. 2024-01-16T15:21:12.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:12.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.337+0000 7fe1dda00640 1 Processor -- start 2024-01-16T15:21:12.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.338+0000 7fe1dda00640 1 -- start start 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1dda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d813e360 0x7fe1d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1dda00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1dda00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d815c5f0 con 0x7fe1d813df80 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1dda00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d806c470 con 0x7fe1d815f9c0 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1dda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d806c750 con 0x7fe1d813e360 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33912/0 (socket says 172.21.15.125:33912) 2024-01-16T15:21:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 -- 172.21.15.125:0/1764163351 learned_addr learned my addr 172.21.15.125:0/1764163351 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 -- 172.21.15.125:0/1764163351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d813e360 msgr2=0x7fe1d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 --2- 172.21.15.125:0/1764163351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d813e360 0x7fe1d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.339+0000 7fe1d7400640 1 -- 172.21.15.125:0/1764163351 >> v1:172.21.15.137:6789/0 conn(0x7fe1d813df80 legacy=0x7fe1d815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.340+0000 7fe1d7400640 1 -- 172.21.15.125:0/1764163351 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d806ca30 con 0x7fe1d815f9c0 2024-01-16T15:21:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1d7400640 1 --2- 172.21.15.125:0/1764163351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009870 tx=0x7fe1d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fef5c44776a082 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1d5600640 1 -- 172.21.15.125:0/1764163351 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0031640 con 0x7fe1d815f9c0 2024-01-16T15:21:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1d5600640 1 -- 172.21.15.125:0/1764163351 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d00317a0 con 0x7fe1d815f9c0 2024-01-16T15:21:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1d5600640 1 -- 172.21.15.125:0/1764163351 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0031b30 con 0x7fe1d815f9c0 2024-01-16T15:21:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1dda00640 1 -- 172.21.15.125:0/1764163351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d815f9c0 msgr2=0x7fe1d80cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.341+0000 7fe1dda00640 1 --2- 172.21.15.125:0/1764163351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009870 tx=0x7fe1d002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 -- 172.21.15.125:0/1764163351 shutdown_connections 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 --2- 172.21.15.125:0/1764163351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d815f9c0 0x7fe1d80cf7d0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 --2- 172.21.15.125:0/1764163351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d813e360 0x7fe1d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 -- 172.21.15.125:0/1764163351 >> 172.21.15.125:0/1764163351 conn(0x7fe1d806a800 msgr2=0x7fe1d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 -- 172.21.15.125:0/1764163351 shutdown_connections 2024-01-16T15:21:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.342+0000 7fe1dda00640 1 -- 172.21.15.125:0/1764163351 wait complete. 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.343+0000 7fe1dda00640 1 Processor -- start 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.343+0000 7fe1dda00640 1 -- start start 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d813df80 0x7fe1d81f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d813e360 0x7fe1d814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d815f9c0 0x7fe1d81f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d81f3ce0 con 0x7fe1d813df80 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d81f4640 con 0x7fe1d813e360 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.344+0000 7fe1dda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d814b2b0 con 0x7fe1d815f9c0 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d813df80 0x7fe1d81f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d813df80 0x7fe1d81f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33924/0 (socket says 172.21.15.125:33924) 2024-01-16T15:21:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 -- 172.21.15.125:0/680775125 learned_addr learned my addr 172.21.15.125:0/680775125 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 -- 172.21.15.125:0/680775125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d815f9c0 msgr2=0x7fe1d81f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1d815f9c0 0x7fe1d81f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 -- 172.21.15.125:0/680775125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d813e360 msgr2=0x7fe1d814da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d813e360 0x7fe1d814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 -- 172.21.15.125:0/680775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00094d0 con 0x7fe1d813df80 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1d6a00640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1d813df80 0x7fe1d81f4100 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800bab0 tx=0x7fe1c800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.345+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800cc60 con 0x7fe1d813df80 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.346+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800ce00 con 0x7fe1d813df80 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.346+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8003bb0 con 0x7fe1d813df80 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.346+0000 7fe1dda00640 1 -- 172.21.15.125:0/680775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d814df60 con 0x7fe1d813df80 2024-01-16T15:21:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.346+0000 7fe1dda00640 1 -- 172.21.15.125:0/680775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d814afd0 con 0x7fe1d813df80 2024-01-16T15:21:12.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.347+0000 7fe1dda00640 1 -- 172.21.15.125:0/680775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d80675d0 con 0x7fe1d813df80 2024-01-16T15:21:12.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.348+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 48) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fe1c8003d10 con 0x7fe1d813df80 2024-01-16T15:21:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.348+0000 7fe1cea00640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe1b006c5e0 0x7fe1b006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.349+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe1c8088a10 con 0x7fe1d813df80 2024-01-16T15:21:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.349+0000 7fe1d6000640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe1b006c5e0 0x7fe1b006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.350+0000 7fe1d6000640 1 --2- 172.21.15.125:0/680775125 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe1b006c5e0 0x7fe1b006eaa0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81550e0 tx=0x7fe1c0006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.353+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1c8054f30 con 0x7fe1d813df80 2024-01-16T15:21:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.567+0000 7fe1dda00640 1 -- 172.21.15.125:0/680775125 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1d815eb10 con 0x7fe1b006c5e0 2024-01-16T15:21:12.573 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:12.580+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:12.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:12.580+0000 7fe1cea00640 1 -- 172.21.15.125:0/680775125 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe1d815eb10 con 0x7fe1b006c5e0 2024-01-16T15:21:12.574 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:14.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:14.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:14.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:14.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.912+0000 7f87af400640 1 Processor -- start 2024-01-16T15:21:14.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.912+0000 7f87af400640 1 -- start start 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a8150760 0x7f87a8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87a815d300 0x7f87a815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f87a80b2bd0 0x7f87a80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a815dc80 con 0x7f87a815d300 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a8071190 con 0x7f87a80b2bd0 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.914+0000 7f87af400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8071470 con 0x7f87a8150760 2024-01-16T15:21:14.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.915+0000 7f87a7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87a815d300 0x7f87a815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.915+0000 7f87a7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87a815d300 0x7f87a815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33940/0 (socket says 172.21.15.125:33940) 2024-01-16T15:21:14.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.915+0000 7f87a7e00640 1 -- 172.21.15.125:0/1771563634 learned_addr learned my addr 172.21.15.125:0/1771563634 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.915+0000 7f87ad400640 1 --2- 172.21.15.125:0/1771563634 >> v2:172.21.15.137:3300/0 conn(0x7f87a80b2bd0 0x7f87a80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.915+0000 7f87aca00640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a8150760 0x7f87a8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.916+0000 7f87a7e00640 1 -- 172.21.15.125:0/1771563634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a8150760 msgr2=0x7f87a8072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.916+0000 7f87a7e00640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a8150760 0x7f87a8072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.916+0000 7f87a7e00640 1 -- 172.21.15.125:0/1771563634 >> v2:172.21.15.137:3300/0 conn(0x7f87a80b2bd0 msgr2=0x7f87a80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.916+0000 7f87a7e00640 1 --2- 172.21.15.125:0/1771563634 >> v2:172.21.15.137:3300/0 conn(0x7f87a80b2bd0 0x7f87a80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.916+0000 7f87a7e00640 1 -- 172.21.15.125:0/1771563634 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8071750 con 0x7f87a815d300 2024-01-16T15:21:14.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.918+0000 7f87a7e00640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a815d300 0x7f87a815d740 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f87a80cbc10 tx=0x7f879002e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84952c2328b6fea8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.918+0000 7f87a7400640 1 -- 172.21.15.125:0/1771563634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879002ee80 con 0x7f87a815d300 2024-01-16T15:21:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.918+0000 7f87a7400640 1 -- 172.21.15.125:0/1771563634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790036440 con 0x7f87a815d300 2024-01-16T15:21:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.918+0000 7f87a7400640 1 -- 172.21.15.125:0/1771563634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87900367b0 con 0x7f87a815d300 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 -- 172.21.15.125:0/1771563634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a815d300 msgr2=0x7f87a815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a815d300 0x7f87a815d740 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f87a80cbc10 tx=0x7f879002e0d0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 -- 172.21.15.125:0/1771563634 shutdown_connections 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 --2- 172.21.15.125:0/1771563634 >> v2:172.21.15.137:3300/0 conn(0x7f87a80b2bd0 0x7f87a80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a815d300 0x7f87a815d740 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 --2- 172.21.15.125:0/1771563634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a8150760 0x7f87a8072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.920+0000 7f87af400640 1 -- 172.21.15.125:0/1771563634 >> 172.21.15.125:0/1771563634 conn(0x7f87a806a800 msgr2=0x7f87a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.921+0000 7f87af400640 1 -- 172.21.15.125:0/1771563634 shutdown_connections 2024-01-16T15:21:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.921+0000 7f87af400640 1 -- 172.21.15.125:0/1771563634 wait complete. 2024-01-16T15:21:14.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.924+0000 7f87af400640 1 Processor -- start 2024-01-16T15:21:14.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.924+0000 7f87af400640 1 -- start start 2024-01-16T15:21:14.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2bd0 0x7f87a80ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a8150760 0x7f87a81f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a815d300 0x7f87a80cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a81f2e60 con 0x7f87a8150760 2024-01-16T15:21:14.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a814a490 con 0x7f87a815d300 2024-01-16T15:21:14.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87af400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8155180 con 0x7f87a80b2bd0 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.926+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a8150760 0x7f87a81f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a8150760 0x7f87a81f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33944/0 (socket says 172.21.15.125:33944) 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 -- 172.21.15.125:0/265201535 learned_addr learned my addr 172.21.15.125:0/265201535 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87aca00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2bd0 0x7f87a80ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87ad400640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a815d300 0x7f87a80cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 -- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2bd0 msgr2=0x7f87a80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2bd0 0x7f87a80ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 -- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a815d300 msgr2=0x7f87a80cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a815d300 0x7f87a80cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.927+0000 7f87a7e00640 1 -- 172.21.15.125:0/265201535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879000b770 con 0x7f87a8150760 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.928+0000 7f87a7e00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87a8150760 0x7f87a81f2920 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f879002e110 tx=0x7f8790004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.928+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87900407f0 con 0x7f87a8150760 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.928+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790040950 con 0x7f87a8150760 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.928+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790040d00 con 0x7f87a8150760 2024-01-16T15:21:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.928+0000 7f87af400640 1 -- 172.21.15.125:0/265201535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a80cce40 con 0x7f87a8150760 2024-01-16T15:21:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.929+0000 7f87af400640 1 -- 172.21.15.125:0/265201535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a80cc340 con 0x7f87a8150760 2024-01-16T15:21:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.931+0000 7f87af400640 1 -- 172.21.15.125:0/265201535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a80675d0 con 0x7f87a8150760 2024-01-16T15:21:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.932+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f8790041480 con 0x7f87a8150760 2024-01-16T15:21:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.934+0000 7f87a4c00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878006c670 0x7f878006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.934+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f87900abd20 con 0x7f87a8150760 2024-01-16T15:21:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.935+0000 7f87aca00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878006c670 0x7f878006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:14.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.937+0000 7f87aca00640 1 --2- 172.21.15.125:0/265201535 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f878006c670 0x7f878006eb30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8798006fd0 tx=0x7f8798008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:14.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:14.940+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8790078330 con 0x7f87a8150760 2024-01-16T15:21:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:15.358+0000 7f87af400640 1 -- 172.21.15.125:0/265201535 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87a8150b40 con 0x7f878006c670 2024-01-16T15:21:15.356 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:15.363+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:15.363+0000 7f87a4c00640 1 -- 172.21.15.125:0/265201535 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f879800b200 con 0x7f878006c670 2024-01-16T15:21:15.357 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:17.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.598+0000 7f0c6fe00640 1 Processor -- start 2024-01-16T15:21:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.598+0000 7f0c6fe00640 1 -- start start 2024-01-16T15:21:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c6fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0c701447f0 0x7f0c7015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c6fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7015c5c0 con 0x7f0c7006dab0 2024-01-16T15:21:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c6fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7006d690 con 0x7f0c701447f0 2024-01-16T15:21:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c6fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700cebb0 con 0x7f0c7006c470 2024-01-16T15:21:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c67e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c67e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:47318/0 (socket says 172.21.15.125:47318) 2024-01-16T15:21:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.600+0000 7f0c67e00640 1 -- 172.21.15.125:0/3764425592 learned_addr learned my addr 172.21.15.125:0/3764425592 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.601+0000 7f0c6ea00640 1 --2- 172.21.15.125:0/3764425592 >> v2:172.21.15.137:3300/0 conn(0x7f0c701447f0 0x7f0c7015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.601+0000 7f0c67e00640 1 -- 172.21.15.125:0/3764425592 >> v2:172.21.15.137:3300/0 conn(0x7f0c701447f0 msgr2=0x7f0c7015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:17.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.601+0000 7f0c67e00640 1 --2- 172.21.15.125:0/3764425592 >> v2:172.21.15.137:3300/0 conn(0x7f0c701447f0 0x7f0c7015c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.601+0000 7f0c67e00640 1 -- 172.21.15.125:0/3764425592 >> v1:172.21.15.137:6789/0 conn(0x7f0c7006dab0 legacy=0x7f0c7015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:17.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.601+0000 7f0c67e00640 1 -- 172.21.15.125:0/3764425592 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7006ca10 con 0x7f0c7006c470 2024-01-16T15:21:17.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.603+0000 7f0c67e00640 1 --2- 172.21.15.125:0/3764425592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009870 tx=0x7f0c5802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef87814b947f417 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:17.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.603+0000 7f0c6e000640 1 -- 172.21.15.125:0/3764425592 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58031640 con 0x7f0c7006c470 2024-01-16T15:21:17.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.603+0000 7f0c6e000640 1 -- 172.21.15.125:0/3764425592 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c580317a0 con 0x7f0c7006c470 2024-01-16T15:21:17.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.603+0000 7f0c6e000640 1 -- 172.21.15.125:0/3764425592 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58031b30 con 0x7f0c7006c470 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 -- 172.21.15.125:0/3764425592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 msgr2=0x7f0c700cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 --2- 172.21.15.125:0/3764425592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009870 tx=0x7f0c5802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 -- 172.21.15.125:0/3764425592 shutdown_connections 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 --2- 172.21.15.125:0/3764425592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c7006c470 0x7f0c700cc5e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 --2- 172.21.15.125:0/3764425592 >> v2:172.21.15.137:3300/0 conn(0x7f0c701447f0 0x7f0c7015c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 -- 172.21.15.125:0/3764425592 >> 172.21.15.125:0/3764425592 conn(0x7f0c7006a3e0 msgr2=0x7f0c7006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:17.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 -- 172.21.15.125:0/3764425592 shutdown_connections 2024-01-16T15:21:17.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.604+0000 7f0c6fe00640 1 -- 172.21.15.125:0/3764425592 wait complete. 2024-01-16T15:21:17.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.607+0000 7f0c6fe00640 1 Processor -- start 2024-01-16T15:21:17.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.607+0000 7f0c6fe00640 1 -- start start 2024-01-16T15:21:17.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c7006c470 0x7f0c700b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c7006dab0 0x7f0c701392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c701447f0 0x7f0c70139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70138bb0 con 0x7f0c7006dab0 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700b00a0 con 0x7f0c7006c470 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c6fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700af2e0 con 0x7f0c701447f0 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c67e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c7006dab0 0x7f0c701392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c67e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c7006dab0 0x7f0c701392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47942/0 (socket says 172.21.15.125:47942) 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.608+0000 7f0c67e00640 1 -- 172.21.15.125:0/500514141 learned_addr learned my addr 172.21.15.125:0/500514141 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c6ea00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c7006c470 0x7f0c700b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c6f400640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c701447f0 0x7f0c70139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 -- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c701447f0 msgr2=0x7f0c70139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c701447f0 0x7f0c70139800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 -- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c7006c470 msgr2=0x7f0c700b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:17.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c7006c470 0x7f0c700b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:17.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 -- 172.21.15.125:0/500514141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580094d0 con 0x7f0c7006dab0 2024-01-16T15:21:17.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.609+0000 7f0c67e00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c7006dab0 0x7f0c701392c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f0c58000c00 tx=0x7f0c580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:17.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.610+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5803a930 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.610+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5803aa90 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.610+0000 7f0c6fe00640 1 -- 172.21.15.125:0/500514141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c700af5c0 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.610+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5803ae40 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.610+0000 7f0c6fe00640 1 -- 172.21.15.125:0/500514141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700afb80 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.611+0000 7f0c6fe00640 1 -- 172.21.15.125:0/500514141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c700b2ac0 con 0x7f0c7006dab0 2024-01-16T15:21:17.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.612+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f0c5802f050 con 0x7f0c7006dab0 2024-01-16T15:21:17.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.613+0000 7f0c66a00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c4806c670 0x7f0c4806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:17.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.613+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c580aaa40 con 0x7f0c7006dab0 2024-01-16T15:21:17.606 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.613+0000 7f0c6ea00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c4806c670 0x7f0c4806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:17.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.614+0000 7f0c6ea00640 1 --2- 172.21.15.125:0/500514141 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0c4806c670 0x7f0c4806eb30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009ad0 tx=0x7f0c60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:17.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.617+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c58076f50 con 0x7f0c7006dab0 2024-01-16T15:21:17.834 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.840+0000 7f0c6fe00640 1 -- 172.21.15.125:0/500514141 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c700ce710 con 0x7f0c4806c670 2024-01-16T15:21:17.842 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:17.849+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:17.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:17.849+0000 7f0c66a00640 1 -- 172.21.15.125:0/500514141 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0c700ce710 con 0x7f0c4806c670 2024-01-16T15:21:17.843 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:19.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:20.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.086+0000 7fc47a400640 1 Processor -- start 2024-01-16T15:21:20.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.086+0000 7fc47a400640 1 -- start start 2024-01-16T15:21:20.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc47a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc474146850 0x7fc4740715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc47a400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc474071b10 0x7fc4740ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc47a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474071f70 con 0x7fc4740adb00 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc47a400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47406d480 con 0x7fc474071b10 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc47a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740b3490 con 0x7fc474146850 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc46bc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc474071b10 0x7fc4740ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc479a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc4740adb00 0x7fc4740700f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59732/0 (socket says 172.21.15.125:59732) 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.088+0000 7fc479a00640 1 -- 172.21.15.125:0/2039357600 learned_addr learned my addr 172.21.15.125:0/2039357600 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc479000640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc474146850 0x7fc4740715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc46bc00640 1 -- 172.21.15.125:0/2039357600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc474146850 msgr2=0x7fc4740715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc46bc00640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc474146850 0x7fc4740715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc46bc00640 1 -- 172.21.15.125:0/2039357600 >> v1:172.21.15.125:6789/0 conn(0x7fc4740adb00 legacy=0x7fc4740700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc46bc00640 1 -- 172.21.15.125:0/2039357600 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740b3770 con 0x7fc474071b10 2024-01-16T15:21:20.083 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.089+0000 7fc473e00640 1 -- 172.21.15.125:0/2039357600 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474400147 0 0) 0x7fc474071f70 con 0x7fc4740adb00 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.090+0000 7fc46bc00640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4740ad5c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0097a0 tx=0x7fc45c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bd82006e52a87 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.090+0000 7fc473e00640 1 -- 172.21.15.125:0/2039357600 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c02d040 con 0x7fc474071b10 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.090+0000 7fc473e00640 1 -- 172.21.15.125:0/2039357600 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c031810 con 0x7fc474071b10 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.090+0000 7fc473e00640 1 -- 172.21.15.125:0/2039357600 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c031ba0 con 0x7fc474071b10 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 -- 172.21.15.125:0/2039357600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 msgr2=0x7fc4740ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4740ad5c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0097a0 tx=0x7fc45c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 -- 172.21.15.125:0/2039357600 shutdown_connections 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4740ad5c0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 --2- 172.21.15.125:0/2039357600 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc474146850 0x7fc4740715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 -- 172.21.15.125:0/2039357600 >> 172.21.15.125:0/2039357600 conn(0x7fc47406a3e0 msgr2=0x7fc47406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:20.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 -- 172.21.15.125:0/2039357600 shutdown_connections 2024-01-16T15:21:20.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.091+0000 7fc47a400640 1 -- 172.21.15.125:0/2039357600 wait complete. 2024-01-16T15:21:20.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.093+0000 7fc47a400640 1 Processor -- start 2024-01-16T15:21:20.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.093+0000 7fc47a400640 1 -- start start 2024-01-16T15:21:20.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4741f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4740adb00 0x7fc474160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc474146850 0x7fc47415e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47415fe30 con 0x7fc474071b10 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47415f1b0 con 0x7fc474146850 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc47a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47415f490 con 0x7fc4740adb00 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4741f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4741f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47950/0 (socket says 172.21.15.125:47950) 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 -- 172.21.15.125:0/1279944303 learned_addr learned my addr 172.21.15.125:0/1279944303 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc46bc00640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4740adb00 0x7fc474160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479a00640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc474146850 0x7fc47415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 -- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4740adb00 msgr2=0x7fc474160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4740adb00 0x7fc474160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 -- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc474146850 msgr2=0x7fc47415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc474146850 0x7fc47415e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc479000640 1 -- 172.21.15.125:0/1279944303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c009400 con 0x7fc474071b10 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.094+0000 7fc46bc00640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4740adb00 0x7fc474160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc479000640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc474071b10 0x7fc4741f23b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc46400d5f0 tx=0x7fc46400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4640125f0 con 0x7fc474071b10 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc464012750 con 0x7fc474071b10 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc47a400640 1 -- 172.21.15.125:0/1279944303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47415f770 con 0x7fc474071b10 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464012b00 con 0x7fc474071b10 2024-01-16T15:21:20.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.095+0000 7fc47a400640 1 -- 172.21.15.125:0/1279944303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47415eda0 con 0x7fc474071b10 2024-01-16T15:21:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.096+0000 7fc47a400640 1 -- 172.21.15.125:0/1279944303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc474067450 con 0x7fc474071b10 2024-01-16T15:21:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.097+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fc464012c60 con 0x7fc474071b10 2024-01-16T15:21:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.098+0000 7fc471600640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc44c06c670 0x7fc44c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.098+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc464089100 con 0x7fc474071b10 2024-01-16T15:21:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.098+0000 7fc46bc00640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc44c06c670 0x7fc44c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.099+0000 7fc46bc00640 1 --2- 172.21.15.125:0/1279944303 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc44c06c670 0x7fc44c06eb30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc45c036000 tx=0x7fc45c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.102+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc464055610 con 0x7fc474071b10 2024-01-16T15:21:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.326+0000 7fc47a400640 1 -- 172.21.15.125:0/1279944303 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc474146c30 con 0x7fc44c06c670 2024-01-16T15:21:20.327 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:20.333+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:20.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:20.334+0000 7fc471600640 1 -- 172.21.15.125:0/1279944303 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc45c03a420 con 0x7fc44c06c670 2024-01-16T15:21:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:20.328 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:22.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:22.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.566+0000 7f5e3d000640 1 Processor -- start 2024-01-16T15:21:22.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.567+0000 7f5e3d000640 1 -- start start 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e3815d610 0x7f5e380ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e3d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e380d1ef0 0x7f5e380b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e3d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e380b3400 con 0x7f5e380ced60 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e3d000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380d1390 con 0x7f5e380d1ef0 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380b27f0 con 0x7f5e3815d610 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e36800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e380d1ef0 0x7f5e380b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e35400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5e380ced60 0x7f5e380d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59744/0 (socket says 172.21.15.125:59744) 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.568+0000 7f5e35400640 1 -- 172.21.15.125:0/3294747245 learned_addr learned my addr 172.21.15.125:0/3294747245 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:22.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e35e00640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e3815d610 0x7f5e380ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e36800640 1 -- 172.21.15.125:0/3294747245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e3815d610 msgr2=0x7f5e380ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:22.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e36800640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e3815d610 0x7f5e380ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e36800640 1 -- 172.21.15.125:0/3294747245 >> v1:172.21.15.125:6789/0 conn(0x7f5e380ced60 legacy=0x7f5e380d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:22.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e36800640 1 -- 172.21.15.125:0/3294747245 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380b2ad0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.569+0000 7f5e34a00640 1 -- 172.21.15.125:0/3294747245 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647176208 0 0) 0x7f5e380b3400 con 0x7f5e380ced60 2024-01-16T15:21:22.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.571+0000 7f5e36800640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380b3890 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009d90 tx=0x7f5e3002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3466588f04e77f62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:22.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.571+0000 7f5e34a00640 1 -- 172.21.15.125:0/3294747245 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002cc50 con 0x7f5e380d1ef0 2024-01-16T15:21:22.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.571+0000 7f5e34a00640 1 -- 172.21.15.125:0/3294747245 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3002cdb0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.571+0000 7f5e34a00640 1 -- 172.21.15.125:0/3294747245 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30034660 con 0x7f5e380d1ef0 2024-01-16T15:21:22.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 -- 172.21.15.125:0/3294747245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 msgr2=0x7f5e380b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380b3890 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009d90 tx=0x7f5e3002bea0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 -- 172.21.15.125:0/3294747245 shutdown_connections 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380b3890 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 --2- 172.21.15.125:0/3294747245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e3815d610 0x7f5e380ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 -- 172.21.15.125:0/3294747245 >> 172.21.15.125:0/3294747245 conn(0x7f5e3809c540 msgr2=0x7f5e380d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 -- 172.21.15.125:0/3294747245 shutdown_connections 2024-01-16T15:21:22.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.572+0000 7f5e3d000640 1 -- 172.21.15.125:0/3294747245 wait complete. 2024-01-16T15:21:22.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.574+0000 7f5e3d000640 1 Processor -- start 2024-01-16T15:21:22.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.574+0000 7f5e3d000640 1 -- start start 2024-01-16T15:21:22.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.575+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e380ced60 0x7f5e3815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.575+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3815d610 0x7f5e380a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e3d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380b7d80 con 0x7f5e380d1ef0 2024-01-16T15:21:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e3d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380cb680 con 0x7f5e3815d610 2024-01-16T15:21:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380a0fe0 con 0x7f5e380ced60 2024-01-16T15:21:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47978/0 (socket says 172.21.15.125:47978) 2024-01-16T15:21:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 -- 172.21.15.125:0/639444480 learned_addr learned my addr 172.21.15.125:0/639444480 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:22.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e36800640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3815d610 0x7f5e380a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35e00640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e380ced60 0x7f5e3815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 -- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e380ced60 msgr2=0x7f5e3815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:22.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e380ced60 0x7f5e3815a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 -- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3815d610 msgr2=0x7f5e380a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.576+0000 7f5e35400640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3815d610 0x7f5e380a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e35400640 1 -- 172.21.15.125:0/639444480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30009a70 con 0x7f5e380d1ef0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e36800640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3815d610 0x7f5e380a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e35e00640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e380ced60 0x7f5e3815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e35400640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e380d1ef0 0x7f5e380cb140 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000c4d0 tx=0x7f5e2000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e200074e0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e20007660 con 0x7f5e380d1ef0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20004a20 con 0x7f5e380d1ef0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.577+0000 7f5e3d000640 1 -- 172.21.15.125:0/639444480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e380cb860 con 0x7f5e380d1ef0 2024-01-16T15:21:22.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.578+0000 7f5e3d000640 1 -- 172.21.15.125:0/639444480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e380b76c0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.579+0000 7f5e3d000640 1 -- 172.21.15.125:0/639444480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380b7ec0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.580+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f5e20004b80 con 0x7f5e380d1ef0 2024-01-16T15:21:22.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.581+0000 7f5e2e000640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e1006c670 0x7f5e1006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:22.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.581+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e20087d50 con 0x7f5e380d1ef0 2024-01-16T15:21:22.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.581+0000 7f5e35e00640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e1006c670 0x7f5e1006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:22.576 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.582+0000 7f5e35e00640 1 --2- 172.21.15.125:0/639444480 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5e1006c670 0x7f5e1006eb30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5e28006fd0 tx=0x7f5e28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:22.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.586+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e380b7ec0 con 0x7f5e380d1ef0 2024-01-16T15:21:22.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.815+0000 7f5e3d000640 1 -- 172.21.15.125:0/639444480 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e3806c120 con 0x7f5e1006c670 2024-01-16T15:21:22.819 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:22.825+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:22.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:22.826+0000 7f5e2e000640 1 -- 172.21.15.125:0/639444480 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5e3806c120 con 0x7f5e1006c670 2024-01-16T15:21:22.819 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:22.820 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:24.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:25.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.068+0000 7ff823200640 1 Processor -- start 2024-01-16T15:21:25.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.068+0000 7ff823200640 1 -- start start 2024-01-16T15:21:25.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff823200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff81c143ae0 0x7ff81c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff823200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 0x7ff81c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff823200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c0cfc30 con 0x7ff81c15e1e0 2024-01-16T15:21:25.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff823200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81c0b1ee0 con 0x7ff81c143ae0 2024-01-16T15:21:25.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff823200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81c0b2780 con 0x7ff81c0cf7d0 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff81be00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff81c143ae0 0x7ff81c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff821200640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff81c15e1e0 0x7ff81c15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59758/0 (socket says 172.21.15.125:59758) 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.070+0000 7ff821200640 1 -- 172.21.15.125:0/4152058985 learned_addr learned my addr 172.21.15.125:0/4152058985 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81be00640 1 -- 172.21.15.125:0/4152058985 >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 msgr2=0x7ff81c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81b400640 1 --2- 172.21.15.125:0/4152058985 >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 0x7ff81c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81be00640 1 --2- 172.21.15.125:0/4152058985 >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 0x7ff81c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81be00640 1 -- 172.21.15.125:0/4152058985 >> v1:172.21.15.125:6789/0 conn(0x7ff81c15e1e0 legacy=0x7ff81c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81aa00640 1 -- 172.21.15.125:0/4152058985 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588613942 0 0) 0x7ff81c0cfc30 con 0x7ff81c15e1e0 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81be00640 1 -- 172.21.15.125:0/4152058985 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c15e990 con 0x7ff81c143ae0 2024-01-16T15:21:25.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.071+0000 7ff81b400640 1 --2- 172.21.15.125:0/4152058985 >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 0x7ff81c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:25.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.072+0000 7ff81be00640 1 --2- 172.21.15.125:0/4152058985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c143ae0 0x7ff81c0cf290 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0cb830 tx=0x7ff80c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5824378d82e6d869 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:25.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.072+0000 7ff81aa00640 1 -- 172.21.15.125:0/4152058985 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c031830 con 0x7ff81c143ae0 2024-01-16T15:21:25.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.072+0000 7ff81aa00640 1 -- 172.21.15.125:0/4152058985 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80c031990 con 0x7ff81c143ae0 2024-01-16T15:21:25.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.072+0000 7ff81aa00640 1 -- 172.21.15.125:0/4152058985 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c031d20 con 0x7ff81c143ae0 2024-01-16T15:21:25.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 -- 172.21.15.125:0/4152058985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c143ae0 msgr2=0x7ff81c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:25.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 --2- 172.21.15.125:0/4152058985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c143ae0 0x7ff81c0cf290 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0cb830 tx=0x7ff80c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 -- 172.21.15.125:0/4152058985 shutdown_connections 2024-01-16T15:21:25.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 --2- 172.21.15.125:0/4152058985 >> v2:172.21.15.137:3300/0 conn(0x7ff81c0cf7d0 0x7ff81c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 --2- 172.21.15.125:0/4152058985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c143ae0 0x7ff81c0cf290 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 -- 172.21.15.125:0/4152058985 >> 172.21.15.125:0/4152058985 conn(0x7ff81c06a800 msgr2=0x7ff81c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:25.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 -- 172.21.15.125:0/4152058985 shutdown_connections 2024-01-16T15:21:25.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.073+0000 7ff823200640 1 -- 172.21.15.125:0/4152058985 wait complete. 2024-01-16T15:21:25.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.075+0000 7ff823200640 1 Processor -- start 2024-01-16T15:21:25.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.075+0000 7ff823200640 1 -- start start 2024-01-16T15:21:25.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff81c0cf7d0 0x7ff81c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81c143ae0 0x7ff81c0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c15e1e0 0x7ff81c0c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c0cc610 con 0x7ff81c15e1e0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c0cc860 con 0x7ff81c0cf7d0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff823200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c0ccb40 con 0x7ff81c143ae0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff821200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c15e1e0 0x7ff81c0c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff821200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c15e1e0 0x7ff81c0c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47994/0 (socket says 172.21.15.125:47994) 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.076+0000 7ff821200640 1 -- 172.21.15.125:0/41482520 learned_addr learned my addr 172.21.15.125:0/41482520 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff81be00640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff81c0cf7d0 0x7ff81c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff81b400640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81c143ae0 0x7ff81c0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 -- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81c143ae0 msgr2=0x7ff81c0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81c143ae0 0x7ff81c0c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 -- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff81c0cf7d0 msgr2=0x7ff81c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff81c0cf7d0 0x7ff81c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 -- 172.21.15.125:0/41482520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c009400 con 0x7ff81c15e1e0 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff821200640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff81c15e1e0 0x7ff81c0c4390 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff81400c7e0 tx=0x7ff81400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81400a640 con 0x7ff81c15e1e0 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81400a7c0 con 0x7ff81c15e1e0 2024-01-16T15:21:25.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81400ab70 con 0x7ff81c15e1e0 2024-01-16T15:21:25.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.077+0000 7ff823200640 1 -- 172.21.15.125:0/41482520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c0cce20 con 0x7ff81c15e1e0 2024-01-16T15:21:25.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.078+0000 7ff823200640 1 -- 172.21.15.125:0/41482520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c0b1370 con 0x7ff81c15e1e0 2024-01-16T15:21:25.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.079+0000 7ff823200640 1 -- 172.21.15.125:0/41482520 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c0b2bf0 con 0x7ff81c15e1e0 2024-01-16T15:21:25.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.080+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ff81400acd0 con 0x7ff81c15e1e0 2024-01-16T15:21:25.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.081+0000 7ff813e00640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff7f406c670 0x7ff7f406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:25.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.081+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff814088240 con 0x7ff81c15e1e0 2024-01-16T15:21:25.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.081+0000 7ff81be00640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff7f406c670 0x7ff7f406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:25.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.082+0000 7ff81be00640 1 --2- 172.21.15.125:0/41482520 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff7f406c670 0x7ff7f406eb30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff80c036000 tx=0x7ff80c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:25.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.085+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff81c0b2bf0 con 0x7ff81c15e1e0 2024-01-16T15:21:25.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.313+0000 7ff823200640 1 -- 172.21.15.125:0/41482520 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff81c06b1b0 con 0x7ff7f406c670 2024-01-16T15:21:25.320 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:25.326+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:25.320 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:25.327+0000 7ff813e00640 1 -- 172.21.15.125:0/41482520 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff81c06b1b0 con 0x7ff7f406c670 2024-01-16T15:21:25.321 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:25.321 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:27.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:27.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.567+0000 7f1980600640 1 Processor -- start 2024-01-16T15:21:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.567+0000 7f1980600640 1 -- start start 2024-01-16T15:21:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f1980600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f197813e360 0x7f19780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f1980600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197815c5f0 con 0x7f197815f9c0 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f1980600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197806c470 con 0x7f197813df80 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f1980600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197806c750 con 0x7f197813e360 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f197813e360 0x7f19780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f197813e360 0x7f19780cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54580/0 (socket says 172.21.15.125:54580) 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197e600640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f197815f9c0 0x7f19780cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49398/0 (socket says 172.21.15.125:49398) 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 -- 172.21.15.125:0/289636576 learned_addr learned my addr 172.21.15.125:0/289636576 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 -- 172.21.15.125:0/289636576 >> v1:172.21.15.137:6789/0 conn(0x7f197813df80 legacy=0x7f197815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 -- 172.21.15.125:0/289636576 >> v1:172.21.15.125:6789/0 conn(0x7f197815f9c0 legacy=0x7f19780cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.569+0000 7f197d200640 1 -- 172.21.15.125:0/289636576 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197806ca30 con 0x7f197813e360 2024-01-16T15:21:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f197d200640 1 --2- 172.21.15.125:0/289636576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19780cf290 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53933567fce30463 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f1977e00640 1 -- 172.21.15.125:0/289636576 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960030850 con 0x7f197813e360 2024-01-16T15:21:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f1977e00640 1 -- 172.21.15.125:0/289636576 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600309b0 con 0x7f197813e360 2024-01-16T15:21:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f1977e00640 1 -- 172.21.15.125:0/289636576 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960030d40 con 0x7f197813e360 2024-01-16T15:21:27.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f1980600640 1 -- 172.21.15.125:0/289636576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 msgr2=0x7f19780cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:27.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.571+0000 7f1980600640 1 --2- 172.21.15.125:0/289636576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19780cf290 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196002b990 comp rx=0 tx=0).stop 2024-01-16T15:21:27.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.572+0000 7f1980600640 1 -- 172.21.15.125:0/289636576 shutdown_connections 2024-01-16T15:21:27.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.572+0000 7f1980600640 1 --2- 172.21.15.125:0/289636576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19780cf290 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:27.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.572+0000 7f1980600640 1 -- 172.21.15.125:0/289636576 >> 172.21.15.125:0/289636576 conn(0x7f197806a800 msgr2=0x7f197806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:27.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.572+0000 7f1980600640 1 -- 172.21.15.125:0/289636576 shutdown_connections 2024-01-16T15:21:27.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.572+0000 7f1980600640 1 -- 172.21.15.125:0/289636576 wait complete. 2024-01-16T15:21:27.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.573+0000 7f1980600640 1 Processor -- start 2024-01-16T15:21:27.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.573+0000 7f1980600640 1 -- start start 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f197813df80 0x7f19780a85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19781532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f197815f9c0 0x7f19781f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1978158d30 con 0x7f197813e360 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19781537e0 con 0x7f197813df80 2024-01-16T15:21:27.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.574+0000 7f1980600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19781f1800 con 0x7f197815f9c0 2024-01-16T15:21:27.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19781532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:27.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19781532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54594/0 (socket says 172.21.15.125:54594) 2024-01-16T15:21:27.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 -- 172.21.15.125:0/3376576389 learned_addr learned my addr 172.21.15.125:0/3376576389 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197dc00640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f197813df80 0x7f19780a85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 -- 172.21.15.125:0/3376576389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f197815f9c0 msgr2=0x7f19781f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f197815f9c0 0x7f19781f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 -- 172.21.15.125:0/3376576389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f197813df80 msgr2=0x7f19780a85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f197813df80 0x7f19780a85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 -- 172.21.15.125:0/3376576389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600094d0 con 0x7f197813e360 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.575+0000 7f197d200640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f197813e360 0x7f19781532a0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1960000c00 tx=0x7f1960004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.576+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960030800 con 0x7f197813e360 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.576+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960043400 con 0x7f197813e360 2024-01-16T15:21:27.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.576+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600437b0 con 0x7f197813e360 2024-01-16T15:21:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.576+0000 7f1980600640 1 -- 172.21.15.125:0/3376576389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19781f1ae0 con 0x7f197813e360 2024-01-16T15:21:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.576+0000 7f1980600640 1 -- 172.21.15.125:0/3376576389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1978158f40 con 0x7f197813e360 2024-01-16T15:21:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.577+0000 7f1980600640 1 -- 172.21.15.125:0/3376576389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19780675d0 con 0x7f197813e360 2024-01-16T15:21:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.578+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f196002e070 con 0x7f197813e360 2024-01-16T15:21:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.579+0000 7f1975600640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f195006c670 0x7f195006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.579+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f196003e070 con 0x7f197813e360 2024-01-16T15:21:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.579+0000 7f197dc00640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f195006c670 0x7f195006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.580+0000 7f197dc00640 1 --2- 172.21.15.125:0/3376576389 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f195006c670 0x7f195006eb30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1968006fd0 tx=0x7f1968008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.584+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1960078260 con 0x7f197813e360 2024-01-16T15:21:27.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.797+0000 7f1980600640 1 -- 172.21.15.125:0/3376576389 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f197806cff0 con 0x7f195006c670 2024-01-16T15:21:27.799 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:27.806+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:27.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:27.806+0000 7f1975600640 1 -- 172.21.15.125:0/3376576389 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f196800c4f0 con 0x7f195006c670 2024-01-16T15:21:27.800 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:27.800 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:29.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.048+0000 7f5651000640 1 Processor -- start 2024-01-16T15:21:30.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.048+0000 7f5651000640 1 -- start start 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5651000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f564c0cf9a0 0x7f564c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5651000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f564c15e1e0 0x7f564c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5651000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c0b1ee0 con 0x7f564c143ae0 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5651000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0b2780 con 0x7f564c0cf9a0 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5651000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c15e990 con 0x7f564c15e1e0 2024-01-16T15:21:30.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f564c0cf9a0 0x7f564c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f564c0cf9a0 0x7f564c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54608/0 (socket says 172.21.15.125:54608) 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 -- 172.21.15.125:0/3907604824 learned_addr learned my addr 172.21.15.125:0/3907604824 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 -- 172.21.15.125:0/3907604824 >> v2:172.21.15.137:3300/0 conn(0x7f564c15e1e0 msgr2=0x7f564c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 --2- 172.21.15.125:0/3907604824 >> v2:172.21.15.137:3300/0 conn(0x7f564c15e1e0 0x7f564c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 -- 172.21.15.125:0/3907604824 >> v1:172.21.15.137:6789/0 conn(0x7f564c143ae0 legacy=0x7f564c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:30.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.050+0000 7f5649400640 1 -- 172.21.15.125:0/3907604824 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0b2130 con 0x7f564c0cf9a0 2024-01-16T15:21:30.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.052+0000 7f5649400640 1 --2- 172.21.15.125:0/3907604824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c0cf9a0 0x7f564c15dca0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f564c0ae9d0 tx=0x7f563402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f458dcc41e923b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:30.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.052+0000 7f5648a00640 1 -- 172.21.15.125:0/3907604824 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634031870 con 0x7f564c0cf9a0 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.052+0000 7f5648a00640 1 -- 172.21.15.125:0/3907604824 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56340319d0 con 0x7f564c0cf9a0 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.052+0000 7f5648a00640 1 -- 172.21.15.125:0/3907604824 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634031d60 con 0x7f564c0cf9a0 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 -- 172.21.15.125:0/3907604824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c0cf9a0 msgr2=0x7f564c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 --2- 172.21.15.125:0/3907604824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c0cf9a0 0x7f564c15dca0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f564c0ae9d0 tx=0x7f563402b910 comp rx=0 tx=0).stop 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 -- 172.21.15.125:0/3907604824 shutdown_connections 2024-01-16T15:21:30.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 --2- 172.21.15.125:0/3907604824 >> v2:172.21.15.137:3300/0 conn(0x7f564c15e1e0 0x7f564c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:30.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 --2- 172.21.15.125:0/3907604824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c0cf9a0 0x7f564c15dca0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:30.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 -- 172.21.15.125:0/3907604824 >> 172.21.15.125:0/3907604824 conn(0x7f564c06a800 msgr2=0x7f564c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:30.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 -- 172.21.15.125:0/3907604824 shutdown_connections 2024-01-16T15:21:30.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.053+0000 7f5651000640 1 -- 172.21.15.125:0/3907604824 wait complete. 2024-01-16T15:21:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.055+0000 7f5651000640 1 Processor -- start 2024-01-16T15:21:30.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.055+0000 7f5651000640 1 -- start start 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564c0cf9a0 0x7f564c0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c143ae0 0x7f564c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564c15e1e0 0x7f564c13b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c4510 con 0x7f564c143ae0 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c7930 con 0x7f564c0cf9a0 2024-01-16T15:21:30.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5651000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c7c10 con 0x7f564c15e1e0 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c143ae0 0x7f564c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c143ae0 0x7f564c0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54620/0 (socket says 172.21.15.125:54620) 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 -- 172.21.15.125:0/1536108542 learned_addr learned my addr 172.21.15.125:0/1536108542 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649e00640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564c0cf9a0 0x7f564c0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f564a800640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564c15e1e0 0x7f564c13b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:30.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 -- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564c15e1e0 msgr2=0x7f564c13b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564c15e1e0 0x7f564c13b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.057+0000 7f5649400640 1 -- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564c0cf9a0 msgr2=0x7f564c0c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5649400640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564c0cf9a0 0x7f564c0c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5649400640 1 -- 172.21.15.125:0/1536108542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340094d0 con 0x7f564c143ae0 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f564a800640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564c15e1e0 0x7f564c13b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:30.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5649400640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f564c143ae0 0x7f564c0cd8d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f56340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:30.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563403ad10 con 0x7f564c143ae0 2024-01-16T15:21:30.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563403ae70 con 0x7f564c143ae0 2024-01-16T15:21:30.058 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634044610 con 0x7f564c143ae0 2024-01-16T15:21:30.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5651000640 1 -- 172.21.15.125:0/1536108542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0c3e50 con 0x7f564c143ae0 2024-01-16T15:21:30.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.058+0000 7f5651000640 1 -- 172.21.15.125:0/1536108542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c0c7620 con 0x7f564c143ae0 2024-01-16T15:21:30.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.059+0000 7f5651000640 1 -- 172.21.15.125:0/1536108542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c0c4650 con 0x7f564c143ae0 2024-01-16T15:21:30.059 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.061+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f563404d450 con 0x7f564c143ae0 2024-01-16T15:21:30.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.061+0000 7f5642000640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f562406c670 0x7f562406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:30.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.062+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f563402d040 con 0x7f564c143ae0 2024-01-16T15:21:30.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.062+0000 7f5649e00640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f562406c670 0x7f562406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:30.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.063+0000 7f5649e00640 1 --2- 172.21.15.125:0/1536108542 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f562406c670 0x7f562406eb30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f5644001db0 tx=0x7f5644009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:30.060 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.066+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f564c0c4650 con 0x7f564c143ae0 2024-01-16T15:21:30.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.294+0000 7f5651000640 1 -- 172.21.15.125:0/1536108542 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f564c15e670 con 0x7f562406c670 2024-01-16T15:21:30.299 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:30.305+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:30.305+0000 7f5642000640 1 -- 172.21.15.125:0/1536108542 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f56440019a0 con 0x7f562406c670 2024-01-16T15:21:30.300 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:30.301 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:32.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:32.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.548+0000 7fc4c2000640 1 Processor -- start 2024-01-16T15:21:32.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.548+0000 7fc4c2000640 1 -- start start 2024-01-16T15:21:32.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4c2000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 0x7fc4bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4c2000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc06c470 con 0x7fc4bc15f9c0 2024-01-16T15:21:32.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4c2000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc06c750 con 0x7fc4bc13e360 2024-01-16T15:21:32.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4c2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc06ca30 con 0x7fc4bc13df80 2024-01-16T15:21:32.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.550+0000 7fc4bb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:32.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:57494/0 (socket says 172.21.15.125:57494) 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 -- 172.21.15.125:0/1591067269 learned_addr learned my addr 172.21.15.125:0/1591067269 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4b3e00640 1 --2- 172.21.15.125:0/1591067269 >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 0x7fc4bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 -- 172.21.15.125:0/1591067269 >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 msgr2=0x7fc4bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 --2- 172.21.15.125:0/1591067269 >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 0x7fc4bc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 -- 172.21.15.125:0/1591067269 >> v1:172.21.15.137:6789/0 conn(0x7fc4bc15f9c0 legacy=0x7fc4bc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4bb400640 1 -- 172.21.15.125:0/1591067269 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc06cd10 con 0x7fc4bc13df80 2024-01-16T15:21:32.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.551+0000 7fc4b3e00640 1 --2- 172.21.15.125:0/1591067269 >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 0x7fc4bc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:21:32.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.553+0000 7fc4bb400640 1 --2- 172.21.15.125:0/1591067269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009870 tx=0x7fc4ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b945c3c241765ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:32.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.554+0000 7fc4baa00640 1 -- 172.21.15.125:0/1591067269 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac030840 con 0x7fc4bc13df80 2024-01-16T15:21:32.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.554+0000 7fc4baa00640 1 -- 172.21.15.125:0/1591067269 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac0309a0 con 0x7fc4bc13df80 2024-01-16T15:21:32.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4baa00640 1 -- 172.21.15.125:0/1591067269 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac030d30 con 0x7fc4bc13df80 2024-01-16T15:21:32.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 -- 172.21.15.125:0/1591067269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 msgr2=0x7fc4bc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 --2- 172.21.15.125:0/1591067269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009870 tx=0x7fc4ac02b980 comp rx=0 tx=0).stop 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 -- 172.21.15.125:0/1591067269 shutdown_connections 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 --2- 172.21.15.125:0/1591067269 >> v2:172.21.15.137:3300/0 conn(0x7fc4bc13e360 0x7fc4bc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 --2- 172.21.15.125:0/1591067269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc13df80 0x7fc4bc15bee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 -- 172.21.15.125:0/1591067269 >> 172.21.15.125:0/1591067269 conn(0x7fc4bc06a800 msgr2=0x7fc4bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.555+0000 7fc4c2000640 1 -- 172.21.15.125:0/1591067269 shutdown_connections 2024-01-16T15:21:32.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.556+0000 7fc4c2000640 1 -- 172.21.15.125:0/1591067269 wait complete. 2024-01-16T15:21:32.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.557+0000 7fc4c2000640 1 Processor -- start 2024-01-16T15:21:32.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.557+0000 7fc4c2000640 1 -- start start 2024-01-16T15:21:32.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4bc13df80 0x7fc4bc0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4bc13e360 0x7fc4bc14d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc15f9c0 0x7fc4bc13d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b3880 con 0x7fc4bc13e360 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b3b60 con 0x7fc4bc13df80 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.558+0000 7fc4c2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b3e40 con 0x7fc4bc15f9c0 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4bc13e360 0x7fc4bc14d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4bc13e360 0x7fc4bc14d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54634/0 (socket says 172.21.15.125:54634) 2024-01-16T15:21:32.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 -- 172.21.15.125:0/295234658 learned_addr learned my addr 172.21.15.125:0/295234658 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4bb400640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4bc13df80 0x7fc4bc0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 -- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc15f9c0 msgr2=0x7fc4bc13d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4bc15f9c0 0x7fc4bc13d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 -- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4bc13df80 msgr2=0x7fc4bc0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4bc13df80 0x7fc4bc0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4b3e00640 1 -- 172.21.15.125:0/295234658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0094d0 con 0x7fc4bc13e360 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.559+0000 7fc4bb400640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4bc13df80 0x7fc4bc0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4b3e00640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4bc13e360 0x7fc4bc14d8b0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400e730 tx=0x7fc4a400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:32.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4004230 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4004390 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400c8f0 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4c2000640 1 -- 172.21.15.125:0/295234658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc151860 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.560+0000 7fc4c2000640 1 -- 172.21.15.125:0/295234658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc13da50 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.561+0000 7fc4c2000640 1 -- 172.21.15.125:0/295234658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc0675d0 con 0x7fc4bc13e360 2024-01-16T15:21:32.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.562+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fc4a400ca50 con 0x7fc4bc13e360 2024-01-16T15:21:32.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.563+0000 7fc4b3400640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc49406c670 0x7fc49406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:32.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.563+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc4a4088890 con 0x7fc4bc13e360 2024-01-16T15:21:32.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.563+0000 7fc4bb400640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc49406c670 0x7fc49406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:32.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.564+0000 7fc4bb400640 1 --2- 172.21.15.125:0/295234658 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc49406c670 0x7fc49406eb30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac036000 tx=0x7fc4ac03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:32.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.568+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4a4054d20 con 0x7fc4bc13e360 2024-01-16T15:21:32.783 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.789+0000 7fc4c2000640 1 -- 172.21.15.125:0/295234658 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4bc0ce850 con 0x7fc49406c670 2024-01-16T15:21:32.794 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:32.800+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:32.800+0000 7fc4b3400640 1 -- 172.21.15.125:0/295234658 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc4ac030ca0 con 0x7fc49406c670 2024-01-16T15:21:32.794 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:34.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.044+0000 7fb07da00640 1 Processor -- start 2024-01-16T15:21:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.045+0000 7fb07da00640 1 -- start start 2024-01-16T15:21:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.046+0000 7fb07da00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb0780cf9a0 0x7fb078166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.046+0000 7fb07da00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07815c5f0 con 0x7fb07813e360 2024-01-16T15:21:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.046+0000 7fb07da00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07806c470 con 0x7fb07813df80 2024-01-16T15:21:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.046+0000 7fb07da00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07806c750 con 0x7fb0780cf9a0 2024-01-16T15:21:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.046+0000 7fb077400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb0780cf9a0 0x7fb078166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.047+0000 7fb077400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb0780cf9a0 0x7fb078166030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54644/0 (socket says 172.21.15.125:54644) 2024-01-16T15:21:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.047+0000 7fb077400640 1 -- 172.21.15.125:0/4066421414 learned_addr learned my addr 172.21.15.125:0/4066421414 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.047+0000 7fb077400640 1 -- 172.21.15.125:0/4066421414 >> v1:172.21.15.137:6789/0 conn(0x7fb07813df80 legacy=0x7fb07815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.047+0000 7fb077400640 1 -- 172.21.15.125:0/4066421414 >> v1:172.21.15.125:6789/0 conn(0x7fb07813e360 legacy=0x7fb0780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.047+0000 7fb077400640 1 -- 172.21.15.125:0/4066421414 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07806ca30 con 0x7fb0780cf9a0 2024-01-16T15:21:35.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.048+0000 7fb077400640 1 --2- 172.21.15.125:0/4066421414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0780cf9a0 0x7fb078166030 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fb070009870 tx=0x7fb07002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa09f1293d4a6c75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:35.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.048+0000 7fb075600640 1 -- 172.21.15.125:0/4066421414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070030860 con 0x7fb0780cf9a0 2024-01-16T15:21:35.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.048+0000 7fb075600640 1 -- 172.21.15.125:0/4066421414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0700309c0 con 0x7fb0780cf9a0 2024-01-16T15:21:35.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.048+0000 7fb075600640 1 -- 172.21.15.125:0/4066421414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070030d50 con 0x7fb0780cf9a0 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 -- 172.21.15.125:0/4066421414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0780cf9a0 msgr2=0x7fb078166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 --2- 172.21.15.125:0/4066421414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0780cf9a0 0x7fb078166030 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fb070009870 tx=0x7fb07002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 -- 172.21.15.125:0/4066421414 shutdown_connections 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 --2- 172.21.15.125:0/4066421414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0780cf9a0 0x7fb078166030 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 -- 172.21.15.125:0/4066421414 >> 172.21.15.125:0/4066421414 conn(0x7fb07806a800 msgr2=0x7fb07806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 -- 172.21.15.125:0/4066421414 shutdown_connections 2024-01-16T15:21:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.049+0000 7fb07da00640 1 -- 172.21.15.125:0/4066421414 wait complete. 2024-01-16T15:21:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.051+0000 7fb07da00640 1 Processor -- start 2024-01-16T15:21:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.051+0000 7fb07da00640 1 -- start start 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0780cf9a0 0x7fb07814b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb07813df80 0x7fb07814da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07813e360 0x7fb078135520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780704d0 con 0x7fb07813e360 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07814dfb0 con 0x7fb0780cf9a0 2024-01-16T15:21:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb07da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07814bcf0 con 0x7fb07813df80 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb077400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07813e360 0x7fb078135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb077400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07813e360 0x7fb078135520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54652/0 (socket says 172.21.15.125:54652) 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.052+0000 7fb077400640 1 -- 172.21.15.125:0/847384287 learned_addr learned my addr 172.21.15.125:0/847384287 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb076a00640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0780cf9a0 0x7fb07814b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb076000640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb07813df80 0x7fb07814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 -- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb07813df80 msgr2=0x7fb07814da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb07813df80 0x7fb07814da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 -- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0780cf9a0 msgr2=0x7fb07814b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0780cf9a0 0x7fb07814b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 -- 172.21.15.125:0/847384287 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0700094d0 con 0x7fb07813e360 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb076a00640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0780cf9a0 0x7fb07814b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb076000640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb07813df80 0x7fb07814da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb077400640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb07813e360 0x7fb078135520 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fb070009870 tx=0x7fb070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070039d90 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb070043410 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb07da00640 1 -- 172.21.15.125:0/847384287 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07814bfd0 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.053+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0700437c0 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.054+0000 7fb07da00640 1 -- 172.21.15.125:0/847384287 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb078070a00 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.055+0000 7fb07da00640 1 -- 172.21.15.125:0/847384287 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07814c210 con 0x7fb07813e360 2024-01-16T15:21:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.056+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fb07004c3b0 con 0x7fb07813e360 2024-01-16T15:21:35.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.057+0000 7fb06ea00640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006c670 0x7fb05006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:35.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.057+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb0700ab390 con 0x7fb07813e360 2024-01-16T15:21:35.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.057+0000 7fb076a00640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006c670 0x7fb05006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:35.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.058+0000 7fb076a00640 1 --2- 172.21.15.125:0/847384287 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb05006c670 0x7fb05006eb30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb06800a830 tx=0x7fb068008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:35.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.062+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0700778a0 con 0x7fb07813e360 2024-01-16T15:21:35.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.286+0000 7fb07da00640 1 -- 172.21.15.125:0/847384287 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb07806cff0 con 0x7fb05006c670 2024-01-16T15:21:35.291 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:35.298+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:35.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:35.298+0000 7fb06ea00640 1 -- 172.21.15.125:0/847384287 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb06800adb0 con 0x7fb05006c670 2024-01-16T15:21:35.292 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:35.292 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:37.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.539+0000 7f5eff200640 1 Processor -- start 2024-01-16T15:21:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.539+0000 7f5eff200640 1 -- start start 2024-01-16T15:21:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.541+0000 7f5eff200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.541+0000 7f5eff200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5ef80cf9a0 0x7f5ef80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.541+0000 7f5eff200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef806c470 con 0x7f5ef813e360 2024-01-16T15:21:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.541+0000 7f5eff200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef806c750 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.541+0000 7f5eff200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef806ca30 con 0x7f5ef813df80 2024-01-16T15:21:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.542+0000 7f5ef7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.542+0000 7f5efd200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5ef80cf9a0 0x7f5ef80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.542+0000 7f5ef7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:53544/0 (socket says 172.21.15.125:53544) 2024-01-16T15:21:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.542+0000 7f5ef7e00640 1 -- 172.21.15.125:0/3183991492 learned_addr learned my addr 172.21.15.125:0/3183991492 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.543+0000 7f5efd200640 1 -- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 msgr2=0x7f5ef815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.543+0000 7f5efd200640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.543+0000 7f5efd200640 1 -- 172.21.15.125:0/3183991492 >> v1:172.21.15.137:6789/0 conn(0x7f5ef813e360 legacy=0x7f5ef80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.543+0000 7f5efd200640 1 -- 172.21.15.125:0/3183991492 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef806cd10 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.543+0000 7f5ef7e00640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:21:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.544+0000 7f5efd200640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef80cd840 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00097a0 tx=0x7f5ef002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c05ff62d36b985d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.544+0000 7f5ef6a00640 1 -- 172.21.15.125:0/3183991492 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef002d040 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.544+0000 7f5ef6a00640 1 -- 172.21.15.125:0/3183991492 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef0031810 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.544+0000 7f5ef6a00640 1 -- 172.21.15.125:0/3183991492 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0031ba0 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 -- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 msgr2=0x7f5ef80cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef80cd840 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00097a0 tx=0x7f5ef002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 -- 172.21.15.125:0/3183991492 shutdown_connections 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef80cd840 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 --2- 172.21.15.125:0/3183991492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813df80 0x7f5ef815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 -- 172.21.15.125:0/3183991492 >> 172.21.15.125:0/3183991492 conn(0x7f5ef806a800 msgr2=0x7f5ef806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 -- 172.21.15.125:0/3183991492 shutdown_connections 2024-01-16T15:21:37.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.545+0000 7f5eff200640 1 -- 172.21.15.125:0/3183991492 wait complete. 2024-01-16T15:21:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.547+0000 7f5eff200640 1 Processor -- start 2024-01-16T15:21:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.548+0000 7f5eff200640 1 -- start start 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef8157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ef813df80 0x7f5ef813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813e360 0x7f5ef81f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef8156280 con 0x7f5ef813df80 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef8156560 con 0x7f5ef80cf9a0 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5eff200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef8156840 con 0x7f5ef813e360 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5ef7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ef813df80 0x7f5ef813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5ef7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ef813df80 0x7f5ef813cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55932/0 (socket says 172.21.15.125:55932) 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5ef7400640 1 -- 172.21.15.125:0/4058029646 learned_addr learned my addr 172.21.15.125:0/4058029646 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5efd200640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813e360 0x7f5ef81f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.549+0000 7f5ef7e00640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef8157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 -- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813e360 msgr2=0x7f5ef81f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813e360 0x7f5ef81f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 -- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 msgr2=0x7f5ef8157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef8157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 -- 172.21.15.125:0/4058029646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009b30 con 0x7f5ef813df80 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5efd200640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ef813e360 0x7f5ef81f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7e00640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ef80cf9a0 0x7f5ef8157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5ef7400640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ef813df80 0x7f5ef813cea0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c450 tx=0x7f5ee000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00052d0 con 0x7f5ef813df80 2024-01-16T15:21:37.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0005430 con 0x7f5ef813df80 2024-01-16T15:21:37.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5eff200640 1 -- 172.21.15.125:0/4058029646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef0009400 con 0x7f5ef813df80 2024-01-16T15:21:37.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0012870 con 0x7f5ef813df80 2024-01-16T15:21:37.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.550+0000 7f5eff200640 1 -- 172.21.15.125:0/4058029646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef8156e00 con 0x7f5ef813df80 2024-01-16T15:21:37.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.551+0000 7f5eff200640 1 -- 172.21.15.125:0/4058029646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef80675d0 con 0x7f5ef813df80 2024-01-16T15:21:37.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.553+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f5ee00129d0 con 0x7f5ef813df80 2024-01-16T15:21:37.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.553+0000 7f5eefe00640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ed006c670 0x7f5ed006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:37.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.554+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ee00883b0 con 0x7f5ef813df80 2024-01-16T15:21:37.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.554+0000 7f5ef7e00640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ed006c670 0x7f5ed006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:37.549 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.554+0000 7f5ef7e00640 1 --2- 172.21.15.125:0/4058029646 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f5ed006c670 0x7f5ed006eb30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009fd0 tx=0x7f5ee800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:37.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.558+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ee0054c70 con 0x7f5ef813df80 2024-01-16T15:21:37.778 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.783+0000 7f5eff200640 1 -- 172.21.15.125:0/4058029646 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ef815f9c0 con 0x7f5ed006c670 2024-01-16T15:21:37.791 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:37.796+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:37.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:37.796+0000 7f5eefe00640 1 -- 172.21.15.125:0/4058029646 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ee80144e0 con 0x7f5ed006c670 2024-01-16T15:21:37.791 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:37.792 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:39.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:40.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.038+0000 7f3718a00640 1 Processor -- start 2024-01-16T15:21:40.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.038+0000 7f3718a00640 1 -- start start 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.039+0000 7f3718a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37141447f0 0x7f371415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3718a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f371406c470 0x7f37140cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3718a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406dab0 0x7f37140ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3718a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f371406d690 con 0x7f37141447f0 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3718a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37140cebb0 con 0x7f371406c470 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3718a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371415c5c0 con 0x7f371406dab0 2024-01-16T15:21:40.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37141447f0 0x7f371415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37141447f0 0x7f371415c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55948/0 (socket says 172.21.15.125:55948) 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 -- 172.21.15.125:0/1630283451 learned_addr learned my addr 172.21.15.125:0/1630283451 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 -- 172.21.15.125:0/1630283451 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406dab0 msgr2=0x7f37140ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 --2- 172.21.15.125:0/1630283451 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406dab0 0x7f37140ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 -- 172.21.15.125:0/1630283451 >> v2:172.21.15.137:3300/0 conn(0x7f371406c470 msgr2=0x7f37140cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 --2- 172.21.15.125:0/1630283451 >> v2:172.21.15.137:3300/0 conn(0x7f371406c470 0x7f37140cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.040+0000 7f3712800640 1 -- 172.21.15.125:0/1630283451 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371406ca10 con 0x7f37141447f0 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.041+0000 7f3712800640 1 --2- 172.21.15.125:0/1630283451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37141447f0 0x7f371415c080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f371406c900 tx=0x7f370402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9dbb65587d330a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.042+0000 7f3711e00640 1 -- 172.21.15.125:0/1630283451 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704031740 con 0x7f37141447f0 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.042+0000 7f3711e00640 1 -- 172.21.15.125:0/1630283451 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37040318a0 con 0x7f37141447f0 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.042+0000 7f3711e00640 1 -- 172.21.15.125:0/1630283451 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704031c30 con 0x7f37141447f0 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.042+0000 7f3718a00640 1 -- 172.21.15.125:0/1630283451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37141447f0 msgr2=0x7f371415c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.042+0000 7f3718a00640 1 --2- 172.21.15.125:0/1630283451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37141447f0 0x7f371415c080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f371406c900 tx=0x7f370402b7e0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 -- 172.21.15.125:0/1630283451 shutdown_connections 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 --2- 172.21.15.125:0/1630283451 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406dab0 0x7f37140ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 --2- 172.21.15.125:0/1630283451 >> v2:172.21.15.137:3300/0 conn(0x7f371406c470 0x7f37140cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 --2- 172.21.15.125:0/1630283451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37141447f0 0x7f371415c080 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 -- 172.21.15.125:0/1630283451 >> 172.21.15.125:0/1630283451 conn(0x7f371406a3e0 msgr2=0x7f371406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 -- 172.21.15.125:0/1630283451 shutdown_connections 2024-01-16T15:21:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.043+0000 7f3718a00640 1 -- 172.21.15.125:0/1630283451 wait complete. 2024-01-16T15:21:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.044+0000 7f3718a00640 1 Processor -- start 2024-01-16T15:21:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.045+0000 7f3718a00640 1 -- start start 2024-01-16T15:21:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406c470 0x7f37140b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f371406dab0 0x7f3714159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37141447f0 0x7f3714159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714159fa0 con 0x7f371406dab0 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37141f5fe0 con 0x7f37141447f0 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3718a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37141f6230 con 0x7f371406c470 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f371406dab0 0x7f3714159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f371406dab0 0x7f3714159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55952/0 (socket says 172.21.15.125:55952) 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 -- 172.21.15.125:0/1516193139 learned_addr learned my addr 172.21.15.125:0/1516193139 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3712800640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406c470 0x7f37140b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f3713200640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37141447f0 0x7f3714159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 -- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406c470 msgr2=0x7f37140b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406c470 0x7f37140b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 -- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37141447f0 msgr2=0x7f3714159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.046+0000 7f370be00640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37141447f0 0x7f3714159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f370be00640 1 -- 172.21.15.125:0/1516193139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37040094d0 con 0x7f371406dab0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f3713200640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37141447f0 0x7f3714159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f3712800640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f371406c470 0x7f37140b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f370be00640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f371406dab0 0x7f3714159520 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00b440 tx=0x7f36fc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc011890 con 0x7f371406dab0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36fc0119f0 con 0x7f371406dab0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f3718a00640 1 -- 172.21.15.125:0/1516193139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37141f6510 con 0x7f371406dab0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc011da0 con 0x7f371406dab0 2024-01-16T15:21:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.047+0000 7f3718a00640 1 -- 172.21.15.125:0/1516193139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37141f6ad0 con 0x7f371406dab0 2024-01-16T15:21:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.048+0000 7f3718a00640 1 -- 172.21.15.125:0/1516193139 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37141f6cf0 con 0x7f371406dab0 2024-01-16T15:21:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.049+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f36fc01a400 con 0x7f371406dab0 2024-01-16T15:21:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.050+0000 7f370aa00640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ec06c670 0x7f36ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.050+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f36fc088f00 con 0x7f371406dab0 2024-01-16T15:21:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.051+0000 7f3712800640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ec06c670 0x7f36ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.051+0000 7f3712800640 1 --2- 172.21.15.125:0/1516193139 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f36ec06c670 0x7f36ec06eb30 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3704035000 tx=0x7f3704008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:40.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.054+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36fc055500 con 0x7f371406dab0 2024-01-16T15:21:40.273 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.279+0000 7f3718a00640 1 -- 172.21.15.125:0/1516193139 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f371406d2b0 con 0x7f36ec06c670 2024-01-16T15:21:40.288 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:40.293+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:40.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:40.294+0000 7f370aa00640 1 -- 172.21.15.125:0/1516193139 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3704031cc0 con 0x7f36ec06c670 2024-01-16T15:21:40.289 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:40.289 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:42.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.678+0000 7fcfd6a00640 1 Processor -- start 2024-01-16T15:21:42.674 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.679+0000 7fcfd6a00640 1 -- start start 2024-01-16T15:21:42.675 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.680+0000 7fcfd6a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfd015e3b0 0x7fcfd0164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.680+0000 7fcfd6a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd00b1ee0 con 0x7fcfd0143ae0 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.680+0000 7fcfd6a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd00b2780 con 0x7fcfd00cf9a0 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.680+0000 7fcfd6a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd015e990 con 0x7fcfd015e3b0 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfcfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcfd0143ae0 0x7fcfd00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47564/0 (socket says 172.21.15.125:47564) 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfcfe00640 1 -- 172.21.15.125:0/3094813221 learned_addr learned my addr 172.21.15.125:0/3094813221 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfd4a00640 1 --2- 172.21.15.125:0/3094813221 >> v2:172.21.15.125:3300/0 conn(0x7fcfd015e3b0 0x7fcfd0164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.676 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfcf400640 1 -- 172.21.15.125:0/3094813221 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788313821 0 0) 0x7fcfd00b1ee0 con 0x7fcfd0143ae0 2024-01-16T15:21:42.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3094813221 >> v1:172.21.15.137:6789/0 conn(0x7fcfd00cf9a0 legacy=0x7fcfd015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:42.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3094813221 >> v1:172.21.15.125:6789/0 conn(0x7fcfd0143ae0 legacy=0x7fcfd00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.681+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3094813221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00b2130 con 0x7fcfd015e3b0 2024-01-16T15:21:42.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.682+0000 7fcfd4a00640 1 --2- 172.21.15.125:0/3094813221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd0164de0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009d90 tx=0x7fcfc802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925886481a500a78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.677 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfcf400640 1 -- 172.21.15.125:0/3094813221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802cc60 con 0x7fcfd015e3b0 2024-01-16T15:21:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfcf400640 1 -- 172.21.15.125:0/3094813221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc802cdc0 con 0x7fcfd015e3b0 2024-01-16T15:21:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfcf400640 1 -- 172.21.15.125:0/3094813221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8034660 con 0x7fcfd015e3b0 2024-01-16T15:21:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3094813221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 msgr2=0x7fcfd0164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 --2- 172.21.15.125:0/3094813221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd0164de0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009d90 tx=0x7fcfc802beb0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3094813221 shutdown_connections 2024-01-16T15:21:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 --2- 172.21.15.125:0/3094813221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd0164de0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3094813221 >> 172.21.15.125:0/3094813221 conn(0x7fcfd006a800 msgr2=0x7fcfd006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3094813221 shutdown_connections 2024-01-16T15:21:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.683+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3094813221 wait complete. 2024-01-16T15:21:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.685+0000 7fcfd6a00640 1 Processor -- start 2024-01-16T15:21:42.680 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.685+0000 7fcfd6a00640 1 -- start start 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfd00cf9a0 0x7fcfd0159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd0143ae0 0x7fcfd0157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd0142d10 con 0x7fcfd015e3b0 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd0142f80 con 0x7fcfd00cf9a0 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.686+0000 7fcfd6a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd013add0 con 0x7fcfd0143ae0 2024-01-16T15:21:42.682 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd013b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55960/0 (socket says 172.21.15.125:55960) 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3549948834 learned_addr learned my addr 172.21.15.125:0/3549948834 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfcfe00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfd00cf9a0 0x7fcfd0159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3549948834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd0143ae0 msgr2=0x7fcfd0157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd0143ae0 0x7fcfd0157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3549948834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfd00cf9a0 msgr2=0x7fcfd0159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfd00cf9a0 0x7fcfd0159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 -- 172.21.15.125:0/3549948834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8009a70 con 0x7fcfd015e3b0 2024-01-16T15:21:42.683 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.687+0000 7fcfd4a00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfd015e3b0 0x7fcfd013b5d0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009eb0 tx=0x7fcfc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.688+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803e610 con 0x7fcfd015e3b0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.688+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc803e770 con 0x7fcfd015e3b0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.688+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803eae0 con 0x7fcfd015e3b0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.688+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3549948834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd013b0b0 con 0x7fcfd015e3b0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.688+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3549948834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd0157b60 con 0x7fcfd015e3b0 2024-01-16T15:21:42.684 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.689+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3549948834 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd00675d0 con 0x7fcfd015e3b0 2024-01-16T15:21:42.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.690+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fcfc803b050 con 0x7fcfd015e3b0 2024-01-16T15:21:42.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.690+0000 7fcfccc00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcfa806c670 0x7fcfa806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.685 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.691+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcfc80ab380 con 0x7fcfd015e3b0 2024-01-16T15:21:42.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.691+0000 7fcfcfe00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcfa806c670 0x7fcfa806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.686 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.692+0000 7fcfcfe00640 1 --2- 172.21.15.125:0/3549948834 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcfa806c670 0x7fcfa806eb30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0000f60 tx=0x7fcfc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.690 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.695+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfc8077890 con 0x7fcfd015e3b0 2024-01-16T15:21:42.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.919+0000 7fcfd6a00640 1 -- 172.21.15.125:0/3549948834 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfd00cb820 con 0x7fcfa806c670 2024-01-16T15:21:42.923 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:42.928+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:42.928+0000 7fcfccc00640 1 -- 172.21.15.125:0/3549948834 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcfd00cb820 con 0x7fcfa806c670 2024-01-16T15:21:42.924 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:42.924 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:44.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:44.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:44.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:45.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.157+0000 7fdf5cc00640 1 Processor -- start 2024-01-16T15:21:45.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.157+0000 7fdf5cc00640 1 -- start start 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf5cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf580b3ae0 0x7fdf5815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf5cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 0x7fdf580cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf5cc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf58070c40 con 0x7fdf580ae5a0 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf5cc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf58071450 con 0x7fdf580b3ae0 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf5cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf580cfc60 con 0x7fdf580cf290 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf55400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf580b3ae0 0x7fdf5815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf55400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf580b3ae0 0x7fdf5815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55964/0 (socket says 172.21.15.125:55964) 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf55400640 1 -- 172.21.15.125:0/42491629 learned_addr learned my addr 172.21.15.125:0/42491629 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.159+0000 7fdf55e00640 1 --1- 172.21.15.125:0/42491629 >> v1:172.21.15.125:6789/0 conn(0x7fdf580ae5a0 0x7fdf5806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47568/0 (socket says 172.21.15.125:47568) 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf56800640 1 --2- 172.21.15.125:0/42491629 >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 0x7fdf580cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:45.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf55400640 1 -- 172.21.15.125:0/42491629 >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 msgr2=0x7fdf580cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:45.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf55400640 1 --2- 172.21.15.125:0/42491629 >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 0x7fdf580cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:45.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf55400640 1 -- 172.21.15.125:0/42491629 >> v1:172.21.15.125:6789/0 conn(0x7fdf580ae5a0 legacy=0x7fdf5806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:45.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf55400640 1 -- 172.21.15.125:0/42491629 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5806c470 con 0x7fdf580b3ae0 2024-01-16T15:21:45.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf56800640 1 --2- 172.21.15.125:0/42491629 >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 0x7fdf580cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:45.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.160+0000 7fdf54a00640 1 -- 172.21.15.125:0/42491629 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079261980 0 0) 0x7fdf58070c40 con 0x7fdf580ae5a0 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.161+0000 7fdf55400640 1 --2- 172.21.15.125:0/42491629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf5815e430 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fdf580b3f70 tx=0x7fdf4802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e60851c2c26cd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.161+0000 7fdf54a00640 1 -- 172.21.15.125:0/42491629 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802f850 con 0x7fdf580b3ae0 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.161+0000 7fdf54a00640 1 -- 172.21.15.125:0/42491629 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4802f9b0 con 0x7fdf580b3ae0 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.161+0000 7fdf54a00640 1 -- 172.21.15.125:0/42491629 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802fd40 con 0x7fdf580b3ae0 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 -- 172.21.15.125:0/42491629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 msgr2=0x7fdf5815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 --2- 172.21.15.125:0/42491629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf5815e430 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fdf580b3f70 tx=0x7fdf4802ba30 comp rx=0 tx=0).stop 2024-01-16T15:21:45.157 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 -- 172.21.15.125:0/42491629 shutdown_connections 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 --2- 172.21.15.125:0/42491629 >> v2:172.21.15.137:3300/0 conn(0x7fdf580cf290 0x7fdf580cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 --2- 172.21.15.125:0/42491629 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf5815e430 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 -- 172.21.15.125:0/42491629 >> 172.21.15.125:0/42491629 conn(0x7fdf5806a800 msgr2=0x7fdf5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 -- 172.21.15.125:0/42491629 shutdown_connections 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.162+0000 7fdf5cc00640 1 -- 172.21.15.125:0/42491629 wait complete. 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.164+0000 7fdf5cc00640 1 Processor -- start 2024-01-16T15:21:45.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.164+0000 7fdf5cc00640 1 -- start start 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf580ae5a0 0x7fdf580c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf580b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf580cf290 0x7fdf581f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf581f40e0 con 0x7fdf580b3ae0 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf580cdeb0 con 0x7fdf580cf290 2024-01-16T15:21:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf5cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf580c4180 con 0x7fdf580ae5a0 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf580b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf580b25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55980/0 (socket says 172.21.15.125:55980) 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 -- 172.21.15.125:0/998759431 learned_addr learned my addr 172.21.15.125:0/998759431 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 -- 172.21.15.125:0/998759431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf580ae5a0 msgr2=0x7fdf580c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf580ae5a0 0x7fdf580c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 -- 172.21.15.125:0/998759431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf580cf290 msgr2=0x7fdf581f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf580cf290 0x7fdf581f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:45.161 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.165+0000 7fdf55400640 1 -- 172.21.15.125:0/998759431 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf480094d0 con 0x7fdf580b3ae0 2024-01-16T15:21:45.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf55400640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf580b3ae0 0x7fdf580b25d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fdf4802ba70 tx=0x7fdf48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:45.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802f840 con 0x7fdf580b3ae0 2024-01-16T15:21:45.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48041400 con 0x7fdf580b3ae0 2024-01-16T15:21:45.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480417b0 con 0x7fdf580b3ae0 2024-01-16T15:21:45.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf5cc00640 1 -- 172.21.15.125:0/998759431 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf580ce150 con 0x7fdf580b3ae0 2024-01-16T15:21:45.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.166+0000 7fdf5cc00640 1 -- 172.21.15.125:0/998759431 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf580ce710 con 0x7fdf580b3ae0 2024-01-16T15:21:45.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.167+0000 7fdf5cc00640 1 -- 172.21.15.125:0/998759431 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf581f6080 con 0x7fdf580b3ae0 2024-01-16T15:21:45.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.168+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fdf48041910 con 0x7fdf580b3ae0 2024-01-16T15:21:45.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.169+0000 7fdf4e000640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf3006c6a0 0x7fdf3006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:45.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.169+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf4803c070 con 0x7fdf580b3ae0 2024-01-16T15:21:45.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.169+0000 7fdf55e00640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf3006c6a0 0x7fdf3006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:45.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.170+0000 7fdf55e00640 1 --2- 172.21.15.125:0/998759431 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf3006c6a0 0x7fdf3006eb60 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdf40007fd0 tx=0x7fdf40007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:45.168 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.174+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf581f6080 con 0x7fdf580b3ae0 2024-01-16T15:21:45.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.396+0000 7fdf5cc00640 1 -- 172.21.15.125:0/998759431 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf58071170 con 0x7fdf3006c6a0 2024-01-16T15:21:45.402 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:45.407+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:45.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:45.407+0000 7fdf4e000640 1 -- 172.21.15.125:0/998759431 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdf400070d0 con 0x7fdf3006c6a0 2024-01-16T15:21:45.402 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:45.402 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:47.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:47.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.652+0000 7f1bbdc00640 1 Processor -- start 2024-01-16T15:21:47.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.652+0000 7f1bbdc00640 1 -- start start 2024-01-16T15:21:47.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bbdc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bb815da10 0x7f1bb8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb80b2bd0 0x7f1bb80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bbdc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb8071190 con 0x7f1bb8150760 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bbdc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb8071470 con 0x7f1bb815da10 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bbdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8071750 con 0x7f1bb80b2bd0 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bb6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bb815da10 0x7f1bb8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bb6000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bb815da10 0x7f1bb8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57334/0 (socket says 172.21.15.125:57334) 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb6000640 1 -- 172.21.15.125:0/1463421099 learned_addr learned my addr 172.21.15.125:0/1463421099 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:47.649 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.653+0000 7f1bb6a00640 1 --1- 172.21.15.125:0/1463421099 >> v1:172.21.15.125:6789/0 conn(0x7f1bb8150760 0x7f1bb815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49588/0 (socket says 172.21.15.125:49588) 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb7400640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb80b2bd0 0x7f1bb80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb6000640 1 -- 172.21.15.125:0/1463421099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb80b2bd0 msgr2=0x7f1bb80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb6000640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb80b2bd0 0x7f1bb80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb6000640 1 -- 172.21.15.125:0/1463421099 >> v1:172.21.15.125:6789/0 conn(0x7f1bb8150760 legacy=0x7f1bb815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb6000640 1 -- 172.21.15.125:0/1463421099 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8071a30 con 0x7f1bb815da10 2024-01-16T15:21:47.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.654+0000 7f1bb5600640 1 -- 172.21.15.125:0/1463421099 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683989726 0 0) 0x7f1bb8071190 con 0x7f1bb8150760 2024-01-16T15:21:47.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.655+0000 7f1bb6000640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb815da10 0x7f1bb8072230 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009e50 tx=0x7f1ba802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553ac553b5f8644e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:47.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bb5600640 1 -- 172.21.15.125:0/1463421099 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba802cb00 con 0x7f1bb815da10 2024-01-16T15:21:47.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bb5600640 1 -- 172.21.15.125:0/1463421099 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba802cc60 con 0x7f1bb815da10 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bb5600640 1 -- 172.21.15.125:0/1463421099 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80343f0 con 0x7f1bb815da10 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 -- 172.21.15.125:0/1463421099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb815da10 msgr2=0x7f1bb8072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb815da10 0x7f1bb8072230 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009e50 tx=0x7f1ba802bf60 comp rx=0 tx=0).stop 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 -- 172.21.15.125:0/1463421099 shutdown_connections 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb80b2bd0 0x7f1bb80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 --2- 172.21.15.125:0/1463421099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb815da10 0x7f1bb8072230 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 -- 172.21.15.125:0/1463421099 >> 172.21.15.125:0/1463421099 conn(0x7f1bb806a800 msgr2=0x7f1bb806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 -- 172.21.15.125:0/1463421099 shutdown_connections 2024-01-16T15:21:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.656+0000 7f1bbdc00640 1 -- 172.21.15.125:0/1463421099 wait complete. 2024-01-16T15:21:47.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.658+0000 7f1bbdc00640 1 Processor -- start 2024-01-16T15:21:47.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.658+0000 7f1bbdc00640 1 -- start start 2024-01-16T15:21:47.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bb80b2bd0 0x7f1bb8155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb8150760 0x7f1bb814f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb815da10 0x7f1bb80cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8154e80 con 0x7f1bb8150760 2024-01-16T15:21:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8155160 con 0x7f1bb80b2bd0 2024-01-16T15:21:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bbdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8155440 con 0x7f1bb815da10 2024-01-16T15:21:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bb6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb8150760 0x7f1bb814f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bb6000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb8150760 0x7f1bb814f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57336/0 (socket says 172.21.15.125:57336) 2024-01-16T15:21:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.659+0000 7f1bb6000640 1 -- 172.21.15.125:0/3185303848 learned_addr learned my addr 172.21.15.125:0/3185303848 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:47.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6a00640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bb80b2bd0 0x7f1bb8155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 -- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb815da10 msgr2=0x7f1bb80cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:47.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb7400640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb815da10 0x7f1bb80cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb815da10 0x7f1bb80cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:47.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 -- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bb80b2bd0 msgr2=0x7f1bb8155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bb80b2bd0 0x7f1bb8155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 -- 172.21.15.125:0/3185303848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8009b30 con 0x7f1bb8150760 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6a00640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bb80b2bd0 0x7f1bb8155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb7400640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bb815da10 0x7f1bb80cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1bb6000640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb8150760 0x7f1bb814f7a0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8000f80 tx=0x7f1ba8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.660+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8040670 con 0x7f1bb8150760 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.661+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba8040810 con 0x7f1bb8150760 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.661+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8040bc0 con 0x7f1bb8150760 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.661+0000 7f1bbdc00640 1 -- 172.21.15.125:0/3185303848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb8155720 con 0x7f1bb8150760 2024-01-16T15:21:47.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.661+0000 7f1bbdc00640 1 -- 172.21.15.125:0/3185303848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb80cd090 con 0x7f1bb8150760 2024-01-16T15:21:47.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.662+0000 7f1bbdc00640 1 -- 172.21.15.125:0/3185303848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb80675d0 con 0x7f1bb8150760 2024-01-16T15:21:47.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.663+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f1ba8040d20 con 0x7f1bb8150760 2024-01-16T15:21:47.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.663+0000 7f1baea00640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9006c670 0x7f1b9006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:47.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.664+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1ba80abcb0 con 0x7f1bb8150760 2024-01-16T15:21:47.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.664+0000 7f1bb6a00640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9006c670 0x7f1b9006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:47.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.664+0000 7f1bb6a00640 1 --2- 172.21.15.125:0/3185303848 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9006c670 0x7f1b9006eb30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0000f60 tx=0x7f1ba0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.668+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ba80782e0 con 0x7f1bb8150760 2024-01-16T15:21:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.889+0000 7f1bbdc00640 1 -- 172.21.15.125:0/3185303848 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1bb8150b40 con 0x7f1b9006c670 2024-01-16T15:21:47.896 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:47.901+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:47.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:47.901+0000 7f1baea00640 1 -- 172.21.15.125:0/3185303848 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1ba00013d0 con 0x7f1b9006c670 2024-01-16T15:21:47.896 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:47.897 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:49.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:50.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.138+0000 7f92cb200640 1 Processor -- start 2024-01-16T15:21:50.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.138+0000 7f92cb200640 1 -- start start 2024-01-16T15:21:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.140+0000 7f92cb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 0x7f92c4070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.140+0000 7f92cb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 0x7f92c406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.140+0000 7f92cb200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c4071460 con 0x7f92c4157c30 2024-01-16T15:21:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.140+0000 7f92cb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c415ea30 con 0x7f92c4071030 2024-01-16T15:21:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.140+0000 7f92cb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c406d860 con 0x7f92c40cd600 2024-01-16T15:21:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.141+0000 7f92bb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 0x7f92c4070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.141+0000 7f92c3e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f92c4157c30 0x7f92c41618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:47354/0 (socket says 172.21.15.125:47354) 2024-01-16T15:21:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.141+0000 7f92c3e00640 1 -- 172.21.15.125:0/1323594362 learned_addr learned my addr 172.21.15.125:0/1323594362 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.141+0000 7f92c9200640 1 --2- 172.21.15.125:0/1323594362 >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 0x7f92c406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.142+0000 7f92bb200640 1 -- 172.21.15.125:0/1323594362 >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 msgr2=0x7f92c406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.142+0000 7f92bb200640 1 --2- 172.21.15.125:0/1323594362 >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 0x7f92c406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.142+0000 7f92bb200640 1 -- 172.21.15.125:0/1323594362 >> v1:172.21.15.137:6789/0 conn(0x7f92c4157c30 legacy=0x7f92c41618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.142+0000 7f92bb200640 1 -- 172.21.15.125:0/1323594362 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c415deb0 con 0x7f92c40cd600 2024-01-16T15:21:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.142+0000 7f92c9200640 1 --2- 172.21.15.125:0/1323594362 >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 0x7f92c406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:21:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.143+0000 7f92bb200640 1 --2- 172.21.15.125:0/1323594362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 0x7f92c4070af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0097a0 tx=0x7f92ac02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afa20892af7250 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.143+0000 7f92c3400640 1 -- 172.21.15.125:0/1323594362 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac02d040 con 0x7f92c40cd600 2024-01-16T15:21:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.143+0000 7f92c3400640 1 -- 172.21.15.125:0/1323594362 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac031810 con 0x7f92c40cd600 2024-01-16T15:21:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.143+0000 7f92c3400640 1 -- 172.21.15.125:0/1323594362 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac031ba0 con 0x7f92c40cd600 2024-01-16T15:21:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 -- 172.21.15.125:0/1323594362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 msgr2=0x7f92c4070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 --2- 172.21.15.125:0/1323594362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 0x7f92c4070af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0097a0 tx=0x7f92ac02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 -- 172.21.15.125:0/1323594362 shutdown_connections 2024-01-16T15:21:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 --2- 172.21.15.125:0/1323594362 >> v2:172.21.15.137:3300/0 conn(0x7f92c4071030 0x7f92c406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 --2- 172.21.15.125:0/1323594362 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c40cd600 0x7f92c4070af0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 -- 172.21.15.125:0/1323594362 >> 172.21.15.125:0/1323594362 conn(0x7f92c40cf700 msgr2=0x7f92c40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 -- 172.21.15.125:0/1323594362 shutdown_connections 2024-01-16T15:21:50.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.144+0000 7f92cb200640 1 -- 172.21.15.125:0/1323594362 wait complete. 2024-01-16T15:21:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.146+0000 7f92cb200640 1 Processor -- start 2024-01-16T15:21:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.146+0000 7f92cb200640 1 -- start start 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c4071030 0x7f92c40af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92c40cd600 0x7f92c40ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92c4157c30 0x7f92c40ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40afe40 con 0x7f92c40cd600 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40b0120 con 0x7f92c4157c30 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92cb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40b0400 con 0x7f92c4071030 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92bb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92c40cd600 0x7f92c40ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92bb200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92c40cd600 0x7f92c40ad390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57340/0 (socket says 172.21.15.125:57340) 2024-01-16T15:21:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92bb200640 1 -- 172.21.15.125:0/1714736107 learned_addr learned my addr 172.21.15.125:0/1714736107 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.147+0000 7f92c9200640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92c4157c30 0x7f92c40ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92c3e00640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c4071030 0x7f92c40af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 -- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c4071030 msgr2=0x7f92c40af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92c4071030 0x7f92c40af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 -- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92c4157c30 msgr2=0x7f92c40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92c4157c30 0x7f92c40ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 -- 172.21.15.125:0/1714736107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc009a70 con 0x7f92c40cd600 2024-01-16T15:21:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92bb200640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92c40cd600 0x7f92c40ad390 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098c0 tx=0x7f92ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac006e10 con 0x7f92c40cd600 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac03ad60 con 0x7f92c40cd600 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac044620 con 0x7f92c40cd600 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.148+0000 7f92cb200640 1 -- 172.21.15.125:0/1714736107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac009400 con 0x7f92c40cd600 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.149+0000 7f92cb200640 1 -- 172.21.15.125:0/1714736107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c40b09c0 con 0x7f92c40cd600 2024-01-16T15:21:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.150+0000 7f92cb200640 1 -- 172.21.15.125:0/1714736107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c40675d0 con 0x7f92c40cd600 2024-01-16T15:21:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.151+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f92ac04d440 con 0x7f92c40cd600 2024-01-16T15:21:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.152+0000 7f92c0c00640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f929c06c670 0x7f929c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.152+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f92ac0aaf30 con 0x7f92c40cd600 2024-01-16T15:21:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.152+0000 7f92c3e00640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f929c06c670 0x7f929c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.153+0000 7f92c3e00640 1 --2- 172.21.15.125:0/1714736107 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f929c06c670 0x7f929c06eb30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f92b4000f60 tx=0x7f92b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.156+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92ac0774a0 con 0x7f92c40cd600 2024-01-16T15:21:50.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.378+0000 7f92cb200640 1 -- 172.21.15.125:0/1714736107 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92c40cfd10 con 0x7f929c06c670 2024-01-16T15:21:50.385 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:50.390+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:50.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:50.391+0000 7f92c0c00640 1 -- 172.21.15.125:0/1714736107 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f92c40cfd10 con 0x7f929c06c670 2024-01-16T15:21:50.386 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:50.387 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:52.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:52.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.631+0000 7ff871e00640 1 Processor -- start 2024-01-16T15:21:52.626 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.631+0000 7ff871e00640 1 -- start start 2024-01-16T15:21:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff871e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff86c0b3ae0 0x7ff86c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff871e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff86c0cf290 0x7ff86c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff871e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c070c40 con 0x7ff86c0ae5a0 2024-01-16T15:21:52.628 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff871e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86c071450 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff871e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86c0cfc60 con 0x7ff86c0cf290 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff863e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff86c0b3ae0 0x7ff86c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff863e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff86c0b3ae0 0x7ff86c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57356/0 (socket says 172.21.15.125:57356) 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.633+0000 7ff863e00640 1 -- 172.21.15.125:0/1398326262 learned_addr learned my addr 172.21.15.125:0/1398326262 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.634+0000 7ff86b400640 1 --2- 172.21.15.125:0/1398326262 >> v2:172.21.15.137:3300/0 conn(0x7ff86c0cf290 0x7ff86c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.629 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.634+0000 7ff863e00640 1 -- 172.21.15.125:0/1398326262 >> v2:172.21.15.137:3300/0 conn(0x7ff86c0cf290 msgr2=0x7ff86c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.634+0000 7ff863e00640 1 --2- 172.21.15.125:0/1398326262 >> v2:172.21.15.137:3300/0 conn(0x7ff86c0cf290 0x7ff86c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.634+0000 7ff863e00640 1 -- 172.21.15.125:0/1398326262 >> v1:172.21.15.137:6789/0 conn(0x7ff86c0ae5a0 legacy=0x7ff86c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.634+0000 7ff863e00640 1 -- 172.21.15.125:0/1398326262 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c06c470 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.635+0000 7ff863e00640 1 --2- 172.21.15.125:0/1398326262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c15e430 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0b3f70 tx=0x7ff85402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915d4c47c3034577 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.635+0000 7ff86a000640 1 -- 172.21.15.125:0/1398326262 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854030860 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.635+0000 7ff86a000640 1 -- 172.21.15.125:0/1398326262 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8540309c0 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.635+0000 7ff86a000640 1 -- 172.21.15.125:0/1398326262 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854030d50 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 -- 172.21.15.125:0/1398326262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 msgr2=0x7ff86c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:52.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 --2- 172.21.15.125:0/1398326262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c15e430 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0b3f70 tx=0x7ff85402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 -- 172.21.15.125:0/1398326262 shutdown_connections 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 --2- 172.21.15.125:0/1398326262 >> v2:172.21.15.137:3300/0 conn(0x7ff86c0cf290 0x7ff86c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 --2- 172.21.15.125:0/1398326262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c15e430 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 -- 172.21.15.125:0/1398326262 >> 172.21.15.125:0/1398326262 conn(0x7ff86c06a800 msgr2=0x7ff86c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 -- 172.21.15.125:0/1398326262 shutdown_connections 2024-01-16T15:21:52.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.636+0000 7ff871e00640 1 -- 172.21.15.125:0/1398326262 wait complete. 2024-01-16T15:21:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.638+0000 7ff871e00640 1 Processor -- start 2024-01-16T15:21:52.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.638+0000 7ff871e00640 1 -- start start 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff86c0ae5a0 0x7ff86c0cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c0cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff86c0cf290 0x7ff86c0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0b0490 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0ce9a0 con 0x7ff86c0ae5a0 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff871e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0ccab0 con 0x7ff86c0cf290 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff863e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c0cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff863e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c0cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57368/0 (socket says 172.21.15.125:57368) 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.639+0000 7ff863e00640 1 -- 172.21.15.125:0/3147073908 learned_addr learned my addr 172.21.15.125:0/3147073908 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff86aa00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff86c0ae5a0 0x7ff86c0cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff86b400640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff86c0cf290 0x7ff86c0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 -- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff86c0cf290 msgr2=0x7ff86c0ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff86c0cf290 0x7ff86c0ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 -- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff86c0ae5a0 msgr2=0x7ff86c0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff86c0ae5a0 0x7ff86c0cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 -- 172.21.15.125:0/3147073908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540094d0 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863e00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff86c0b3ae0 0x7ff86c0cdf20 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7ff854000c00 tx=0x7ff854004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854039d60 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854044480 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854044830 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.640+0000 7ff871e00640 1 -- 172.21.15.125:0/3147073908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c0af880 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.641+0000 7ff871e00640 1 -- 172.21.15.125:0/3147073908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c0afe40 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.642+0000 7ff871e00640 1 -- 172.21.15.125:0/3147073908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c0675d0 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.643+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ff85404d3b0 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.643+0000 7ff863400640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff84406c6a0 0x7ff84406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.644+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff85403e070 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.644+0000 7ff86aa00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff84406c6a0 0x7ff84406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:52.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.644+0000 7ff86aa00640 1 --2- 172.21.15.125:0/3147073908 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff84406c6a0 0x7ff84406eb60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff85c004ea0 tx=0x7ff85c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.648+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff854077dd0 con 0x7ff86c0b3ae0 2024-01-16T15:21:52.867 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.872+0000 7ff871e00640 1 -- 172.21.15.125:0/3147073908 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff86c071170 con 0x7ff84406c6a0 2024-01-16T15:21:52.880 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:52.882+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:52.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:52.882+0000 7ff863400640 1 -- 172.21.15.125:0/3147073908 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff85c007db0 con 0x7ff84406c6a0 2024-01-16T15:21:52.881 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:52.881 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:54.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:55.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.122+0000 7f5086000640 1 Processor -- start 2024-01-16T15:21:55.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.123+0000 7f5086000640 1 -- start start 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.124+0000 7f5086000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f508015d610 0x7f50800ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.124+0000 7f5086000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50800d1ef0 0x7f50800b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.124+0000 7f5086000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800b3400 con 0x7f50800ced60 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.124+0000 7f5086000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800d1390 con 0x7f508015d610 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.124+0000 7f5086000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800b27f0 con 0x7f50800d1ef0 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f508015d610 0x7f50800ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507ea00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f50800ced60 0x7f50800d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49598/0 (socket says 172.21.15.125:49598) 2024-01-16T15:21:55.120 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507ea00640 1 -- 172.21.15.125:0/2712822339 learned_addr learned my addr 172.21.15.125:0/2712822339 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:55.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507f400640 1 -- 172.21.15.125:0/2712822339 >> v2:172.21.15.137:3300/0 conn(0x7f50800d1ef0 msgr2=0x7f50800b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:55.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507f400640 1 --2- 172.21.15.125:0/2712822339 >> v2:172.21.15.137:3300/0 conn(0x7f50800d1ef0 0x7f50800b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507f400640 1 -- 172.21.15.125:0/2712822339 >> v1:172.21.15.125:6789/0 conn(0x7f50800ced60 legacy=0x7f50800d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:55.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507f400640 1 -- 172.21.15.125:0/2712822339 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800b2ad0 con 0x7f508015d610 2024-01-16T15:21:55.121 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.125+0000 7f507e000640 1 -- 172.21.15.125:0/2712822339 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546227293 0 0) 0x7f50800b3400 con 0x7f50800ced60 2024-01-16T15:21:55.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.126+0000 7f507f400640 1 --2- 172.21.15.125:0/2712822339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f508015d610 0x7f50800ce820 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f508008ba80 tx=0x7f507002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680eb9f59d6c1472 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:55.122 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f507e000640 1 -- 172.21.15.125:0/2712822339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507002cc80 con 0x7f508015d610 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f507e000640 1 -- 172.21.15.125:0/2712822339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507002cde0 con 0x7f508015d610 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f507e000640 1 -- 172.21.15.125:0/2712822339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070034660 con 0x7f508015d610 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 -- 172.21.15.125:0/2712822339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f508015d610 msgr2=0x7f50800ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 --2- 172.21.15.125:0/2712822339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f508015d610 0x7f50800ce820 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f508008ba80 tx=0x7f507002bed0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 -- 172.21.15.125:0/2712822339 shutdown_connections 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 --2- 172.21.15.125:0/2712822339 >> v2:172.21.15.137:3300/0 conn(0x7f50800d1ef0 0x7f50800b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 --2- 172.21.15.125:0/2712822339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f508015d610 0x7f50800ce820 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 -- 172.21.15.125:0/2712822339 >> 172.21.15.125:0/2712822339 conn(0x7f508009c540 msgr2=0x7f50800d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 -- 172.21.15.125:0/2712822339 shutdown_connections 2024-01-16T15:21:55.123 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.127+0000 7f5086000640 1 -- 172.21.15.125:0/2712822339 wait complete. 2024-01-16T15:21:55.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.129+0000 7f5086000640 1 Processor -- start 2024-01-16T15:21:55.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.129+0000 7f5086000640 1 -- start start 2024-01-16T15:21:55.125 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50800ced60 0x7f508014ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50800d1ef0 0x7f508014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f508015d610 0x7f50800ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50800b7290 con 0x7f50800d1ef0 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50800b7e10 con 0x7f50800ced60 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f5086000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50800b74e0 con 0x7f508015d610 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.130+0000 7f507ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50800d1ef0 0x7f508014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50800d1ef0 0x7f508014e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57386/0 (socket says 172.21.15.125:57386) 2024-01-16T15:21:55.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 -- 172.21.15.125:0/539367776 learned_addr learned my addr 172.21.15.125:0/539367776 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:55.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507f400640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50800ced60 0x7f508014ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:55.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 -- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f508015d610 msgr2=0x7f50800ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:55.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f508015d610 0x7f50800ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 -- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50800ced60 msgr2=0x7f508014ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:55.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50800ced60 0x7f508014ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:55.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 -- 172.21.15.125:0/539367776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070009a70 con 0x7f50800d1ef0 2024-01-16T15:21:55.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507f400640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50800ced60 0x7f508014ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:55.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.131+0000 7f507ea00640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50800d1ef0 0x7f508014e5e0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f506800c500 tx=0x7f506800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:55.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.132+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068007560 con 0x7f50800d1ef0 2024-01-16T15:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.132+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50680047e0 con 0x7f50800d1ef0 2024-01-16T15:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.132+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068004b90 con 0x7f50800d1ef0 2024-01-16T15:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.132+0000 7f5086000640 1 -- 172.21.15.125:0/539367776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50800b77c0 con 0x7f50800d1ef0 2024-01-16T15:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.132+0000 7f5086000640 1 -- 172.21.15.125:0/539367776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508014eb20 con 0x7f50800d1ef0 2024-01-16T15:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.133+0000 7f5086000640 1 -- 172.21.15.125:0/539367776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50800675d0 con 0x7f50800d1ef0 2024-01-16T15:21:55.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.134+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f5068004cf0 con 0x7f50800d1ef0 2024-01-16T15:21:55.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.135+0000 7f5077400640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f505806c670 0x7f505806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:55.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.135+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5068087dd0 con 0x7f50800d1ef0 2024-01-16T15:21:55.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.135+0000 7f507f400640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f505806c670 0x7f505806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:55.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.136+0000 7f507f400640 1 --2- 172.21.15.125:0/539367776 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f505806c670 0x7f505806eb30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5070036000 tx=0x7f5070030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:55.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.139+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50680542e0 con 0x7f50800d1ef0 2024-01-16T15:21:55.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.364+0000 7f5086000640 1 -- 172.21.15.125:0/539367776 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50800c97d0 con 0x7f505806c670 2024-01-16T15:21:55.366 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:55.371+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:55.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:55.371+0000 7f5077400640 1 -- 172.21.15.125:0/539367776 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f50800c97d0 con 0x7f505806c670 2024-01-16T15:21:55.367 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:55.367 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:57.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:57.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.611+0000 7fccc2200640 1 Processor -- start 2024-01-16T15:21:57.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.611+0000 7fccc2200640 1 -- start start 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.612+0000 7fccc2200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fccbc13df80 0x7fccbc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.612+0000 7fccc2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 0x7fccbc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.612+0000 7fccc2200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc06c470 con 0x7fccbc13e360 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.612+0000 7fccc2200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc06c750 con 0x7fccbc13df80 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.612+0000 7fccc2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc06ca30 con 0x7fccbc0cf9a0 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccb2800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fccbc13e360 0x7fccbc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59636/0 (socket says 172.21.15.125:59636) 2024-01-16T15:21:57.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccb2800640 1 -- 172.21.15.125:0/2436925932 learned_addr learned my addr 172.21.15.125:0/2436925932 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccbb400640 1 --2- 172.21.15.125:0/2436925932 >> v2:172.21.15.137:3300/0 conn(0x7fccbc13df80 0x7fccbc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccbbe00640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 0x7fccbc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229288629 0 0) 0x7fccbc06c470 con 0x7fccbc13e360 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.613+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003520 con 0x7fccbc13e360 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3380596782 0 0) 0x7fcc9c003520 con 0x7fccbc13e360 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 msgr2=0x7fccbc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 0x7fccbc0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 >> v2:172.21.15.137:3300/0 conn(0x7fccbc13df80 msgr2=0x7fccbc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:57.611 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 --2- 172.21.15.125:0/2436925932 >> v2:172.21.15.137:3300/0 conn(0x7fccbc13df80 0x7fccbc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc06cd10 con 0x7fccbc13e360 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fcca4001670 con 0x7fccbc13e360 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 >> v1:172.21.15.125:6789/0 conn(0x7fccbc13e360 legacy=0x7fccbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c003970 0x7fcc9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 0x7fcc9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc9c004d00 0x7fcc9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc06c750 con 0x7fcc9c003970 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc06ca30 con 0x7fcc9c004d00 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00b980 con 0x7fccbc0cf9a0 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c003970 0x7fcc9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 -- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 msgr2=0x7fcc9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:57.612 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccbc0cf9a0 0x7fcc9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 -- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc9c004d00 msgr2=0x7fcc9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc9c004d00 0x7fcc9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.614+0000 7fccbb400640 1 -- 172.21.15.125:0/2436925932 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c003520 con 0x7fcc9c003970 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccbb400640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c003970 0x7fcc9c003d90 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fccac009af0 tx=0x7fcc9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee2f15ccf1e2648 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac02c710 con 0x7fcc9c003970 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac02c870 con 0x7fcc9c003970 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccbaa00640 1 -- 172.21.15.125:0/2436925932 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac02cc00 con 0x7fcc9c003970 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccc2200640 1 -- 172.21.15.125:0/2436925932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c003970 msgr2=0x7fcc9c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccc2200640 1 --2- 172.21.15.125:0/2436925932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c003970 0x7fcc9c003d90 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fccac009af0 tx=0x7fcc9c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccbbe00640 1 -- 172.21.15.125:0/2436925932 reap_dead start 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccc2200640 1 -- 172.21.15.125:0/2436925932 shutdown_connections 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.616+0000 7fccc2200640 1 -- 172.21.15.125:0/2436925932 >> 172.21.15.125:0/2436925932 conn(0x7fccbc06a800 msgr2=0x7fccbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:57.613 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.617+0000 7fccc2200640 1 -- 172.21.15.125:0/2436925932 shutdown_connections 2024-01-16T15:21:57.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.617+0000 7fccc2200640 1 -- 172.21.15.125:0/2436925932 wait complete. 2024-01-16T15:21:57.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.618+0000 7fccc2200640 1 Processor -- start 2024-01-16T15:21:57.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.618+0000 7fccc2200640 1 -- start start 2024-01-16T15:21:57.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc9c003970 0x7fccbc13d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccbc15a180 0x7fccbc15a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccbc0716c0 0x7fccbc071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc13de10 con 0x7fccbc0716c0 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc15ab40 con 0x7fccbc15a180 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.619+0000 7fccc2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc070d70 con 0x7fcc9c003970 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccbc0716c0 0x7fccbc071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccbc0716c0 0x7fccbc071b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37294/0 (socket says 172.21.15.125:37294) 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 -- 172.21.15.125:0/1557042426 learned_addr learned my addr 172.21.15.125:0/1557042426 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:21:57.617 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccb2800640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccbc15a180 0x7fccbc15a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 -- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc9c003970 msgr2=0x7fccbc13d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc9c003970 0x7fccbc13d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 -- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccbc15a180 msgr2=0x7fccbc15a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccbc15a180 0x7fccbc15a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 -- 172.21.15.125:0/1557042426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc070440 con 0x7fccbc0716c0 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccb2800640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccbc15a180 0x7fccbc15a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.620+0000 7fccbbe00640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccbc0716c0 0x7fccbc071b40 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0cfea0 tx=0x7fccb400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.621+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4010040 con 0x7fccbc0716c0 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.621+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb400ba80 con 0x7fccbc0716c0 2024-01-16T15:21:57.618 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.621+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400be50 con 0x7fccbc0716c0 2024-01-16T15:21:57.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.621+0000 7fccc2200640 1 -- 172.21.15.125:0/1557042426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc070720 con 0x7fccbc0716c0 2024-01-16T15:21:57.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.621+0000 7fccc2200640 1 -- 172.21.15.125:0/1557042426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc072080 con 0x7fccbc0716c0 2024-01-16T15:21:57.619 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.622+0000 7fccc2200640 1 -- 172.21.15.125:0/1557042426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc0675d0 con 0x7fccbc0716c0 2024-01-16T15:21:57.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.623+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fccb401e410 con 0x7fccbc0716c0 2024-01-16T15:21:57.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.624+0000 7fccb3e00640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9406c670 0x7fcc9406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:57.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.624+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fccb4088ea0 con 0x7fccbc0716c0 2024-01-16T15:21:57.620 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.624+0000 7fccbb400640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9406c670 0x7fcc9406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:57.621 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.625+0000 7fccbb400640 1 --2- 172.21.15.125:0/1557042426 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9406c670 0x7fcc9406eb30 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fccac036000 tx=0x7fccac02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:57.622 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.628+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccb40553b0 con 0x7fccbc0716c0 2024-01-16T15:21:57.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.851+0000 7fccc2200640 1 -- 172.21.15.125:0/1557042426 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccbc15f9c0 con 0x7fcc9406c670 2024-01-16T15:21:57.855 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:21:57.860+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:57.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:21:57.861+0000 7fccb3e00640 1 -- 172.21.15.125:0/1557042426 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fccac02c9e0 con 0x7fcc9406c670 2024-01-16T15:21:57.855 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:21:57.855 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:59.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.098+0000 7f04b4400640 1 Processor -- start 2024-01-16T15:22:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.098+0000 7f04b4400640 1 -- start start 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.099+0000 7f04b4400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f04ac143ae0 0x7f04ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.099+0000 7f04b4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f04ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.099+0000 7f04b4400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac0cfc30 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.099+0000 7f04b4400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac0b1ee0 con 0x7f04ac143ae0 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.099+0000 7f04b4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b2780 con 0x7f04ac15e3b0 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04b1000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f04ac0cf7d0 0x7f04ac15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59638/0 (socket says 172.21.15.125:59638) 2024-01-16T15:22:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04b1000640 1 -- 172.21.15.125:0/826922561 learned_addr learned my addr 172.21.15.125:0/826922561 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04b1a00640 1 --2- 172.21.15.125:0/826922561 >> v2:172.21.15.137:3300/0 conn(0x7f04ac143ae0 0x7f04ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954840839 0 0) 0x7f04ac0cfc30 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04b2400640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f04ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003520 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729843299 0 0) 0x7f048c003520 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 msgr2=0x7f04ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f04ac0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 >> v2:172.21.15.137:3300/0 conn(0x7f04ac143ae0 msgr2=0x7f04ac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.100+0000 7f04abe00640 1 --2- 172.21.15.125:0/826922561 >> v2:172.21.15.137:3300/0 conn(0x7f04ac143ae0 0x7f04ac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac15e990 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0494001670 con 0x7f04ac0cf7d0 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 >> v1:172.21.15.125:6789/0 conn(0x7f04ac0cf7d0 legacy=0x7f04ac15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f048c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048c003ef0 0x7f048c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048c004d00 0x7f048c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b1ee0 con 0x7f048c003ef0 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b2780 con 0x7f048c004d00 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c00dbe0 con 0x7f04ac15e3b0 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048c003ef0 0x7f048c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 -- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 msgr2=0x7f048c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1a00640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f048c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac15e3b0 0x7f048c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b2400640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048c004d00 0x7f048c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 -- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048c004d00 msgr2=0x7f048c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048c004d00 0x7f048c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b1000640 1 -- 172.21.15.125:0/826922561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c003520 con 0x7f048c003ef0 2024-01-16T15:22:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.101+0000 7f04b2400640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f048c004d00 0x7f048c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b1000640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048c003ef0 0x7f048c0047c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f049400c210 tx=0x7f048c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4b39fb606d352c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049402cae0 con 0x7f048c003ef0 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049402cc40 con 0x7f048c003ef0 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04abe00640 1 -- 172.21.15.125:0/826922561 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494005af0 con 0x7f048c003ef0 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b4400640 1 -- 172.21.15.125:0/826922561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048c003ef0 msgr2=0x7f048c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b4400640 1 --2- 172.21.15.125:0/826922561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048c003ef0 0x7f048c0047c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f049400c210 tx=0x7f048c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b2400640 1 -- 172.21.15.125:0/826922561 reap_dead start 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b4400640 1 -- 172.21.15.125:0/826922561 shutdown_connections 2024-01-16T15:22:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.103+0000 7f04b4400640 1 -- 172.21.15.125:0/826922561 >> 172.21.15.125:0/826922561 conn(0x7f04ac06a800 msgr2=0x7f04ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.104+0000 7f04b4400640 1 -- 172.21.15.125:0/826922561 shutdown_connections 2024-01-16T15:22:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.104+0000 7f04b4400640 1 -- 172.21.15.125:0/826922561 wait complete. 2024-01-16T15:22:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.105+0000 7f04b4400640 1 Processor -- start 2024-01-16T15:22:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.106+0000 7f04b4400640 1 -- start start 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.106+0000 7f04b4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048c003ef0 0x7f04ac13faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b4400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac15c880 0x7f04ac15cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b4400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac140d20 0x7f04ac1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b4400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac1f3000 con 0x7f04ac140d20 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b4400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac15d220 con 0x7f04ac15c880 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac142320 con 0x7f048c003ef0 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac140d20 0x7f04ac1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac140d20 0x7f04ac1411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37308/0 (socket says 172.21.15.125:37308) 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 -- 172.21.15.125:0/669752549 learned_addr learned my addr 172.21.15.125:0/669752549 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 -- 172.21.15.125:0/669752549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048c003ef0 msgr2=0x7f04ac13faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048c003ef0 0x7f04ac13faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 -- 172.21.15.125:0/669752549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac15c880 msgr2=0x7f04ac15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac15c880 0x7f04ac15cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 -- 172.21.15.125:0/669752549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac141a80 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.107+0000 7f04b2400640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac140d20 0x7f04ac1411a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f04ac071f10 tx=0x7f04a4009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.108+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4007cf0 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.108+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4007e70 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.108+0000 7f04b4400640 1 -- 172.21.15.125:0/669752549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac141d60 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.108+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4014620 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.108+0000 7f04b4400640 1 -- 172.21.15.125:0/669752549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1416e0 con 0x7f04ac140d20 2024-01-16T15:22:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.109+0000 7f04b4400640 1 -- 172.21.15.125:0/669752549 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac1f3140 con 0x7f04ac140d20 2024-01-16T15:22:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.110+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f04a401d3b0 con 0x7f04ac140d20 2024-01-16T15:22:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.111+0000 7f04a9600640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f048406c6a0 0x7f048406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.111+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f04a40887d0 con 0x7f04ac140d20 2024-01-16T15:22:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.111+0000 7f04b1a00640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f048406c6a0 0x7f048406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.112+0000 7f04b1a00640 1 --2- 172.21.15.125:0/669752549 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f048406c6a0 0x7f048406eb60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f049c0099d0 tx=0x7f049c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:00.109 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.115+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04ac1f3140 con 0x7f04ac140d20 2024-01-16T15:22:00.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.336+0000 7f04b4400640 1 -- 172.21.15.125:0/669752549 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04ac0b2410 con 0x7f048406c6a0 2024-01-16T15:22:00.343 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:00.348+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:00.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:00.348+0000 7f04a9600640 1 -- 172.21.15.125:0/669752549 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f049c009600 con 0x7f048406c6a0 2024-01-16T15:22:00.344 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:00.344 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:02.357 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:02.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.589+0000 7f11ba200640 1 Processor -- start 2024-01-16T15:22:02.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.589+0000 7f11ba200640 1 -- start start 2024-01-16T15:22:02.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11ba200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f11b40d0100 0x7f11b40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11ba200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40d0f40 0x7f11b40d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11ba200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b40d0a10 con 0x7f11b4160920 2024-01-16T15:22:02.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11ba200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b40cf080 con 0x7f11b40d0f40 2024-01-16T15:22:02.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11ba200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b40c6be0 con 0x7f11b40d0100 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11b9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40d0f40 0x7f11b40d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11b9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11b40d0f40 0x7f11b40d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37324/0 (socket says 172.21.15.125:37324) 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.591+0000 7f11b9800640 1 -- 172.21.15.125:0/636232312 learned_addr learned my addr 172.21.15.125:0/636232312 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.592+0000 7f11b9800640 1 -- 172.21.15.125:0/636232312 >> v2:172.21.15.137:3300/0 conn(0x7f11b40d0100 msgr2=0x7f11b40b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.592+0000 7f11b9800640 1 --2- 172.21.15.125:0/636232312 >> v2:172.21.15.137:3300/0 conn(0x7f11b40d0100 0x7f11b40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.592+0000 7f11b9800640 1 -- 172.21.15.125:0/636232312 >> v1:172.21.15.137:6789/0 conn(0x7f11b4160920 legacy=0x7f11b40cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:02.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.592+0000 7f11b9800640 1 -- 172.21.15.125:0/636232312 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40ce7e0 con 0x7f11b40d0f40 2024-01-16T15:22:02.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.593+0000 7f11b9800640 1 --2- 172.21.15.125:0/636232312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40d0f40 0x7f11b40d1390 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f11b408c1f0 tx=0x7f11ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e97c4270141167e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:02.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.593+0000 7f11b3e00640 1 -- 172.21.15.125:0/636232312 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac02ca80 con 0x7f11b40d0f40 2024-01-16T15:22:02.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.593+0000 7f11b3e00640 1 -- 172.21.15.125:0/636232312 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac02cbe0 con 0x7f11b40d0f40 2024-01-16T15:22:02.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.593+0000 7f11b3e00640 1 -- 172.21.15.125:0/636232312 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0343f0 con 0x7f11b40d0f40 2024-01-16T15:22:02.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 -- 172.21.15.125:0/636232312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40d0f40 msgr2=0x7f11b40d1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:02.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 --2- 172.21.15.125:0/636232312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40d0f40 0x7f11b40d1390 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f11b408c1f0 tx=0x7f11ac02bee0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 -- 172.21.15.125:0/636232312 shutdown_connections 2024-01-16T15:22:02.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 --2- 172.21.15.125:0/636232312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b40d0f40 0x7f11b40d1390 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 --2- 172.21.15.125:0/636232312 >> v2:172.21.15.137:3300/0 conn(0x7f11b40d0100 0x7f11b40b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 -- 172.21.15.125:0/636232312 >> 172.21.15.125:0/636232312 conn(0x7f11b40d3090 msgr2=0x7f11b40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:02.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 -- 172.21.15.125:0/636232312 shutdown_connections 2024-01-16T15:22:02.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.594+0000 7f11ba200640 1 -- 172.21.15.125:0/636232312 wait complete. 2024-01-16T15:22:02.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.596+0000 7f11ba200640 1 Processor -- start 2024-01-16T15:22:02.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.596+0000 7f11ba200640 1 -- start start 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b40d0100 0x7f11b40b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b40d0f40 0x7f11b406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b4160920 0x7f11b406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40cac10 con 0x7f11b4160920 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40ccc40 con 0x7f11b40d0100 2024-01-16T15:22:02.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11ba200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b41f2910 con 0x7f11b40d0f40 2024-01-16T15:22:02.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11b9800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b4160920 0x7f11b406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:02.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11b9800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b4160920 0x7f11b406b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37330/0 (socket says 172.21.15.125:37330) 2024-01-16T15:22:02.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11b9800640 1 -- 172.21.15.125:0/2838166569 learned_addr learned my addr 172.21.15.125:0/2838166569 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11b8e00640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b40d0100 0x7f11b40b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.597+0000 7f11abe00640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b40d0f40 0x7f11b406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 -- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b40d0f40 msgr2=0x7f11b406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11b40d0f40 0x7f11b406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 -- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b40d0100 msgr2=0x7f11b40b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b40d0100 0x7f11b40b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 -- 172.21.15.125:0/2838166569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac009b30 con 0x7f11b4160920 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b8e00640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11b40d0100 0x7f11b40b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b9800640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11b4160920 0x7f11b406b3f0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f11ac036040 tx=0x7f11ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac03e450 con 0x7f11b4160920 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac03e5d0 con 0x7f11b4160920 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac03e980 con 0x7f11b4160920 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.598+0000 7f11ba200640 1 -- 172.21.15.125:0/2838166569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b40ca2e0 con 0x7f11b4160920 2024-01-16T15:22:02.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.599+0000 7f11ba200640 1 -- 172.21.15.125:0/2838166569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b40ca8a0 con 0x7f11b4160920 2024-01-16T15:22:02.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.600+0000 7f11ba200640 1 -- 172.21.15.125:0/2838166569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b40caa30 con 0x7f11b4160920 2024-01-16T15:22:02.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.601+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f11ac02cd50 con 0x7f11b4160920 2024-01-16T15:22:02.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.601+0000 7f11b1600640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06c670 0x7f118c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:02.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.602+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f11ac0ab890 con 0x7f11b4160920 2024-01-16T15:22:02.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.602+0000 7f11b8e00640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06c670 0x7f118c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:02.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.603+0000 7f11b8e00640 1 --2- 172.21.15.125:0/2838166569 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f118c06c670 0x7f118c06eb30 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f11b415b9f0 tx=0x7f11a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:02.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.607+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11ac077da0 con 0x7f11b4160920 2024-01-16T15:22:02.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.829+0000 7f11ba200640 1 -- 172.21.15.125:0/2838166569 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11b4160d00 con 0x7f118c06c670 2024-01-16T15:22:02.830 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:02.836+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:02.836+0000 7f11b1600640 1 -- 172.21.15.125:0/2838166569 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f11a400ad80 con 0x7f118c06c670 2024-01-16T15:22:02.831 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:04.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:05.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.075+0000 7f90b9800640 1 Processor -- start 2024-01-16T15:22:05.069 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.075+0000 7f90b9800640 1 -- start start 2024-01-16T15:22:05.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b9800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f90b40d0b60 0x7f90b40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b9800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90b40aede0 0x7f90b406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b9800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4071de0 con 0x7f90b406a3e0 2024-01-16T15:22:05.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b9800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b40af270 con 0x7f90b40aede0 2024-01-16T15:22:05.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b9800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4072030 con 0x7f90b40d0b60 2024-01-16T15:22:05.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b2a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f90b406a3e0 0x7f90b406a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59654/0 (socket says 172.21.15.125:59654) 2024-01-16T15:22:05.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b2a00640 1 -- 172.21.15.125:0/154280679 learned_addr learned my addr 172.21.15.125:0/154280679 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:05.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b3400640 1 --2- 172.21.15.125:0/154280679 >> v2:172.21.15.125:3300/0 conn(0x7f90b40aede0 0x7f90b406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b2000640 1 --2- 172.21.15.125:0/154280679 >> v2:172.21.15.137:3300/0 conn(0x7f90b40d0b60 0x7f90b40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.077+0000 7f90b1600640 1 -- 172.21.15.125:0/154280679 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536079581 0 0) 0x7f90b4071de0 con 0x7f90b406a3e0 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.078+0000 7f90b1600640 1 -- 172.21.15.125:0/154280679 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094003520 con 0x7f90b406a3e0 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.078+0000 7f90b3400640 1 -- 172.21.15.125:0/154280679 >> v2:172.21.15.137:3300/0 conn(0x7f90b40d0b60 msgr2=0x7f90b40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.078+0000 7f90b3400640 1 --2- 172.21.15.125:0/154280679 >> v2:172.21.15.137:3300/0 conn(0x7f90b40d0b60 0x7f90b40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.078+0000 7f90b3400640 1 -- 172.21.15.125:0/154280679 >> v1:172.21.15.125:6789/0 conn(0x7f90b406a3e0 legacy=0x7f90b406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:05.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.078+0000 7f90b3400640 1 -- 172.21.15.125:0/154280679 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4072310 con 0x7f90b40aede0 2024-01-16T15:22:05.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b3400640 1 --2- 172.21.15.125:0/154280679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40aede0 0x7f90b406f080 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f90b40b6120 tx=0x7f90ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969ae2b4caca8d59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:05.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b1600640 1 -- 172.21.15.125:0/154280679 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02cc80 con 0x7f90b40aede0 2024-01-16T15:22:05.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b1600640 1 -- 172.21.15.125:0/154280679 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac02cde0 con 0x7f90b40aede0 2024-01-16T15:22:05.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b1600640 1 -- 172.21.15.125:0/154280679 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac034660 con 0x7f90b40aede0 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b9800640 1 -- 172.21.15.125:0/154280679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40aede0 msgr2=0x7f90b406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.079+0000 7f90b9800640 1 --2- 172.21.15.125:0/154280679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40aede0 0x7f90b406f080 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f90b40b6120 tx=0x7f90ac02bed0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 -- 172.21.15.125:0/154280679 shutdown_connections 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 --2- 172.21.15.125:0/154280679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40aede0 0x7f90b406f080 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 --2- 172.21.15.125:0/154280679 >> v2:172.21.15.137:3300/0 conn(0x7f90b40d0b60 0x7f90b40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 -- 172.21.15.125:0/154280679 >> 172.21.15.125:0/154280679 conn(0x7f90b40cfef0 msgr2=0x7f90b40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 -- 172.21.15.125:0/154280679 shutdown_connections 2024-01-16T15:22:05.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.080+0000 7f90b9800640 1 -- 172.21.15.125:0/154280679 wait complete. 2024-01-16T15:22:05.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.081+0000 7f90b9800640 1 Processor -- start 2024-01-16T15:22:05.076 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 -- start start 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90b406a3e0 0x7f90b40cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90b40aede0 0x7f90b40cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40d0b60 0x7f90b4153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40cf7f0 con 0x7f90b40d0b60 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40cf360 con 0x7f90b406a3e0 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.082+0000 7f90b9800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40ce750 con 0x7f90b40aede0 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40d0b60 0x7f90b4153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40d0b60 0x7f90b4153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37344/0 (socket says 172.21.15.125:37344) 2024-01-16T15:22:05.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 -- 172.21.15.125:0/1257972283 learned_addr learned my addr 172.21.15.125:0/1257972283 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 -- 172.21.15.125:0/1257972283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90b40aede0 msgr2=0x7f90b40cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90b40aede0 0x7f90b40cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 -- 172.21.15.125:0/1257972283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90b406a3e0 msgr2=0x7f90b40cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90b406a3e0 0x7f90b40cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 -- 172.21.15.125:0/1257972283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac009a70 con 0x7f90b40d0b60 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.083+0000 7f90b3400640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90b40d0b60 0x7f90b4153070 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f90ac037040 tx=0x7f90ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.084+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac040650 con 0x7f90b40d0b60 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.084+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac0407b0 con 0x7f90b40d0b60 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.084+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac040b60 con 0x7f90b40d0b60 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.084+0000 7f90b9800640 1 -- 172.21.15.125:0/1257972283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b40cea30 con 0x7f90b40d0b60 2024-01-16T15:22:05.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.084+0000 7f90b9800640 1 -- 172.21.15.125:0/1257972283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b40ceff0 con 0x7f90b40d0b60 2024-01-16T15:22:05.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.085+0000 7f90b9800640 1 -- 172.21.15.125:0/1257972283 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b40675d0 con 0x7f90b40d0b60 2024-01-16T15:22:05.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.086+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f90ac040cc0 con 0x7f90b40d0b60 2024-01-16T15:22:05.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.087+0000 7f90aaa00640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f908c06c670 0x7f908c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:05.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.087+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90ac0ab8c0 con 0x7f90b40d0b60 2024-01-16T15:22:05.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.087+0000 7f90b2a00640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f908c06c670 0x7f908c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:05.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.088+0000 7f90b2a00640 1 --2- 172.21.15.125:0/1257972283 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f908c06c670 0x7f908c06eb30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f90b406b360 tx=0x7f90a4007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.091+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90ac077e30 con 0x7f90b40d0b60 2024-01-16T15:22:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.318+0000 7f90b9800640 1 -- 172.21.15.125:0/1257972283 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90b406eb60 con 0x7f908c06c670 2024-01-16T15:22:05.320 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:05.326+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:05.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:05.326+0000 7f90aaa00640 1 -- 172.21.15.125:0/1257972283 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f90a40058b0 con 0x7f908c06c670 2024-01-16T15:22:05.321 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:05.321 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:07.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.568+0000 7feadfa00640 1 Processor -- start 2024-01-16T15:22:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.568+0000 7feadfa00640 1 -- start start 2024-01-16T15:22:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7feadfa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7feadfa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fead813e360 0x7fead80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7feadfa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead806c470 con 0x7fead815f9c0 2024-01-16T15:22:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7feadfa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead806c750 con 0x7fead813e360 2024-01-16T15:22:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7feadfa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead806ca30 con 0x7fead813df80 2024-01-16T15:22:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7fead7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fead813e360 0x7fead80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7fead7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fead813e360 0x7fead80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34986/0 (socket says 172.21.15.125:34986) 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.570+0000 7fead7e00640 1 -- 172.21.15.125:0/2652525366 learned_addr learned my addr 172.21.15.125:0/2652525366 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.571+0000 7feadd000640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.571+0000 7fead7e00640 1 -- 172.21.15.125:0/2652525366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 msgr2=0x7fead815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.571+0000 7fead7e00640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.571+0000 7fead7e00640 1 -- 172.21.15.125:0/2652525366 >> v1:172.21.15.137:6789/0 conn(0x7fead815f9c0 legacy=0x7fead81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.571+0000 7fead7e00640 1 -- 172.21.15.125:0/2652525366 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead806cd10 con 0x7fead813e360 2024-01-16T15:22:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.572+0000 7fead7e00640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead813e360 0x7fead80cf290 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7feac0009870 tx=0x7feac002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7152c55b9cec9b6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.572+0000 7fead7400640 1 -- 172.21.15.125:0/2652525366 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0030840 con 0x7fead813e360 2024-01-16T15:22:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.572+0000 7fead7400640 1 -- 172.21.15.125:0/2652525366 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac00309a0 con 0x7fead813e360 2024-01-16T15:22:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.572+0000 7fead7400640 1 -- 172.21.15.125:0/2652525366 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0030d30 con 0x7fead813e360 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 -- 172.21.15.125:0/2652525366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead813e360 msgr2=0x7fead80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead813e360 0x7fead80cf290 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7feac0009870 tx=0x7feac002b980 comp rx=0 tx=0).stop 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 -- 172.21.15.125:0/2652525366 shutdown_connections 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead813e360 0x7fead80cf290 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 --2- 172.21.15.125:0/2652525366 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 -- 172.21.15.125:0/2652525366 >> 172.21.15.125:0/2652525366 conn(0x7fead806a800 msgr2=0x7fead806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 -- 172.21.15.125:0/2652525366 shutdown_connections 2024-01-16T15:22:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.573+0000 7feadfa00640 1 -- 172.21.15.125:0/2652525366 wait complete. 2024-01-16T15:22:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.575+0000 7feadfa00640 1 Processor -- start 2024-01-16T15:22:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.575+0000 7feadfa00640 1 -- start start 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead8136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fead813e360 0x7fead81f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead815f9c0 0x7fead80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead81f6ca0 con 0x7fead815f9c0 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead81f6f80 con 0x7fead813e360 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadfa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead81f7260 con 0x7fead813df80 2024-01-16T15:22:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead815f9c0 0x7fead80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadda00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead815f9c0 0x7fead80708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34996/0 (socket says 172.21.15.125:34996) 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadda00640 1 -- 172.21.15.125:0/3162918297 learned_addr learned my addr 172.21.15.125:0/3162918297 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.576+0000 7feadd000640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead8136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7fead7e00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fead813e360 0x7fead81f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 -- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 msgr2=0x7fead8136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fead813df80 0x7fead8136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 -- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fead813e360 msgr2=0x7fead81f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fead813e360 0x7fead81f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 -- 172.21.15.125:0/3162918297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00094d0 con 0x7fead815f9c0 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7fead7e00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fead813e360 0x7fead81f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadda00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fead815f9c0 0x7fead80708b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fead000c7a0 tx=0x7fead000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0003b70 con 0x7fead815f9c0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead0003cf0 con 0x7fead815f9c0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0003ea0 con 0x7fead815f9c0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadfa00640 1 -- 172.21.15.125:0/3162918297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead813cb10 con 0x7fead815f9c0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.577+0000 7feadfa00640 1 -- 172.21.15.125:0/3162918297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead8136fc0 con 0x7fead815f9c0 2024-01-16T15:22:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.578+0000 7feadfa00640 1 -- 172.21.15.125:0/3162918297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead813d030 con 0x7fead815f9c0 2024-01-16T15:22:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.579+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fead000aa90 con 0x7fead815f9c0 2024-01-16T15:22:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.580+0000 7fead4c00640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feab006c670 0x7feab006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.580+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fead00886d0 con 0x7fead815f9c0 2024-01-16T15:22:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.581+0000 7feadd000640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feab006c670 0x7feab006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.581+0000 7feadd000640 1 --2- 172.21.15.125:0/3162918297 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feab006c670 0x7feab006eb30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feac8006fd0 tx=0x7feac8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.585+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fead813d030 con 0x7fead815f9c0 2024-01-16T15:22:07.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.807+0000 7feadfa00640 1 -- 172.21.15.125:0/3162918297 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fead80ce850 con 0x7feab006c670 2024-01-16T15:22:07.812 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:07.818+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:07.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:07.819+0000 7fead4c00640 1 -- 172.21.15.125:0/3162918297 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7feac800f860 con 0x7feab006c670 2024-01-16T15:22:07.813 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:07.814 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:09.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.067+0000 7f31fae00640 1 Processor -- start 2024-01-16T15:22:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.067+0000 7f31fae00640 1 -- start start 2024-01-16T15:22:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fae00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f31f413cda0 0x7f31f415da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fae00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31f40ce0c0 0x7f31f4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fae00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f40ce4f0 con 0x7f31f415dfa0 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fae00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f40700f0 con 0x7f31f40ce0c0 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fae00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f415e3e0 con 0x7f31f413cda0 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31fa400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31f40ce0c0 0x7f31f4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31f9000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f31f415dfa0 0x7f31f40cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42594/0 (socket says 172.21.15.125:42594) 2024-01-16T15:22:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.069+0000 7f31f9000640 1 -- 172.21.15.125:0/2370300070 learned_addr learned my addr 172.21.15.125:0/2370300070 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31f9a00640 1 --2- 172.21.15.125:0/2370300070 >> v2:172.21.15.137:3300/0 conn(0x7f31f413cda0 0x7f31f415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31fa400640 1 -- 172.21.15.125:0/2370300070 >> v2:172.21.15.137:3300/0 conn(0x7f31f413cda0 msgr2=0x7f31f415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31fa400640 1 --2- 172.21.15.125:0/2370300070 >> v2:172.21.15.137:3300/0 conn(0x7f31f413cda0 0x7f31f415da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31fa400640 1 -- 172.21.15.125:0/2370300070 >> v1:172.21.15.125:6789/0 conn(0x7f31f415dfa0 legacy=0x7f31f40cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31f3e00640 1 -- 172.21.15.125:0/2370300070 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972089199 0 0) 0x7f31f40ce4f0 con 0x7f31f415dfa0 2024-01-16T15:22:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.070+0000 7f31fa400640 1 -- 172.21.15.125:0/2370300070 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f40ce720 con 0x7f31f40ce0c0 2024-01-16T15:22:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.071+0000 7f31fa400640 1 --2- 172.21.15.125:0/2370300070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f4070b40 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f31f406df10 tx=0x7f31e402da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74516fe74bd2a2d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.071+0000 7f31f3e00640 1 -- 172.21.15.125:0/2370300070 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e4031870 con 0x7f31f40ce0c0 2024-01-16T15:22:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.071+0000 7f31f3e00640 1 -- 172.21.15.125:0/2370300070 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e40319d0 con 0x7f31f40ce0c0 2024-01-16T15:22:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.071+0000 7f31f3e00640 1 -- 172.21.15.125:0/2370300070 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e4031d60 con 0x7f31f40ce0c0 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 -- 172.21.15.125:0/2370300070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 msgr2=0x7f31f4070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 --2- 172.21.15.125:0/2370300070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f4070b40 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f31f406df10 tx=0x7f31e402da50 comp rx=0 tx=0).stop 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 -- 172.21.15.125:0/2370300070 shutdown_connections 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 --2- 172.21.15.125:0/2370300070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f4070b40 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 --2- 172.21.15.125:0/2370300070 >> v2:172.21.15.137:3300/0 conn(0x7f31f413cda0 0x7f31f415da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 -- 172.21.15.125:0/2370300070 >> 172.21.15.125:0/2370300070 conn(0x7f31f406a3e0 msgr2=0x7f31f406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 -- 172.21.15.125:0/2370300070 shutdown_connections 2024-01-16T15:22:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.072+0000 7f31fae00640 1 -- 172.21.15.125:0/2370300070 wait complete. 2024-01-16T15:22:10.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.074+0000 7f31fae00640 1 Processor -- start 2024-01-16T15:22:10.068 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.074+0000 7f31fae00640 1 -- start start 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f413b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f413cda0 0x7f31f4157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f415dfa0 0x7f31f415c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f4155eb0 con 0x7f31f40ce0c0 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f4154b30 con 0x7f31f415dfa0 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31fae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f4154e10 con 0x7f31f413cda0 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31f9a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f413b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31f9a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f413b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35026/0 (socket says 172.21.15.125:35026) 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.075+0000 7f31f9a00640 1 -- 172.21.15.125:0/3344980400 learned_addr learned my addr 172.21.15.125:0/3344980400 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 -- 172.21.15.125:0/3344980400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f413cda0 msgr2=0x7f31f4157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31f413cda0 0x7f31f4157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 -- 172.21.15.125:0/3344980400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f415dfa0 msgr2=0x7f31f415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31fa400640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f415dfa0 0x7f31f415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31f415dfa0 0x7f31f415c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 -- 172.21.15.125:0/3344980400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4009400 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f9a00640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31f40ce0c0 0x7f31f413b770 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f31ec007ef0 tx=0x7f31ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec014620 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec014780 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31fae00640 1 -- 172.21.15.125:0/3344980400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f41550f0 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.076+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec014b30 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.077+0000 7f31fae00640 1 -- 172.21.15.125:0/3344980400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f41556b0 con 0x7f31f40ce0c0 2024-01-16T15:22:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.078+0000 7f31fae00640 1 -- 172.21.15.125:0/3344980400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f4067450 con 0x7f31f40ce0c0 2024-01-16T15:22:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.079+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f31ec014c90 con 0x7f31f40ce0c0 2024-01-16T15:22:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.080+0000 7f31f1600640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c670 0x7f31cc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.080+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31ec089320 con 0x7f31f40ce0c0 2024-01-16T15:22:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.080+0000 7f31f9000640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c670 0x7f31cc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.081+0000 7f31f9000640 1 --2- 172.21.15.125:0/3344980400 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f31cc06c670 0x7f31cc06eb30 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f31dc007fd0 tx=0x7f31dc007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:10.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.084+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31ec055b50 con 0x7f31f40ce0c0 2024-01-16T15:22:10.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.310+0000 7f31fae00640 1 -- 172.21.15.125:0/3344980400 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31f4070620 con 0x7f31cc06c670 2024-01-16T15:22:10.314 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:10.320+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:10.315 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:10.320+0000 7f31f1600640 1 -- 172.21.15.125:0/3344980400 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f31dc0012b0 con 0x7f31cc06c670 2024-01-16T15:22:10.315 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:10.315 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:12.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.559+0000 7f3b7b200640 1 Processor -- start 2024-01-16T15:22:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.559+0000 7f3b7b200640 1 -- start start 2024-01-16T15:22:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.560+0000 7f3b7b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 0x7f3b7415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.560+0000 7f3b7b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b740cccf0 0x7f3b7415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.560+0000 7f3b7b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7415c5c0 con 0x7f3b7406c470 2024-01-16T15:22:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b7b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7406d690 con 0x7f3b740cccf0 2024-01-16T15:22:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b7b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740cebb0 con 0x7f3b741447f0 2024-01-16T15:22:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b79400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b7406c470 0x7f3b740cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42600/0 (socket says 172.21.15.125:42600) 2024-01-16T15:22:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b79400640 1 -- 172.21.15.125:0/1332663307 learned_addr learned my addr 172.21.15.125:0/1332663307 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b7a800640 1 --2- 172.21.15.125:0/1332663307 >> v2:172.21.15.137:3300/0 conn(0x7f3b740cccf0 0x7f3b7415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b79e00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 0x7f3b7415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555507820 0 0) 0x7f3b7415c5c0 con 0x7f3b7406c470 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.561+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b54003520 con 0x7f3b7406c470 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460998059 0 0) 0x7f3b54003520 con 0x7f3b7406c470 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 msgr2=0x7f3b7415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 0x7f3b7415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 >> v2:172.21.15.137:3300/0 conn(0x7f3b740cccf0 msgr2=0x7f3b7415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 --2- 172.21.15.125:0/1332663307 >> v2:172.21.15.137:3300/0 conn(0x7f3b740cccf0 0x7f3b7415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7406ca10 con 0x7f3b7406c470 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3b5c001670 con 0x7f3b7406c470 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 >> v1:172.21.15.125:6789/0 conn(0x7f3b7406c470 legacy=0x7f3b740cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b54003970 0x7f3b54003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b540047c0 0x7f3b54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 0x7f3b54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7406d690 con 0x7f3b54003970 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740cebb0 con 0x7f3b540047c0 2024-01-16T15:22:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5400ba00 con 0x7f3b741447f0 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.562+0000 7f3b79e00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b54003970 0x7f3b54003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.563+0000 7f3b79e00640 1 -- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 msgr2=0x7f3b54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.563+0000 7f3b79e00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b741447f0 0x7f3b54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.563+0000 7f3b79e00640 1 -- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b540047c0 msgr2=0x7f3b54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.563+0000 7f3b79e00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b540047c0 0x7f3b54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.563+0000 7f3b79e00640 1 -- 172.21.15.125:0/1332663307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54003520 con 0x7f3b54003970 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.564+0000 7f3b79e00640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b54003970 0x7f3b54003db0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009ac0 tx=0x7f3b540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340271d8c8f7ac60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.564+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6402c710 con 0x7f3b54003970 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.564+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6402c870 con 0x7f3b54003970 2024-01-16T15:22:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.564+0000 7f3b78a00640 1 -- 172.21.15.125:0/1332663307 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6402cc00 con 0x7f3b54003970 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 -- 172.21.15.125:0/1332663307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b54003970 msgr2=0x7f3b54003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 --2- 172.21.15.125:0/1332663307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b54003970 0x7f3b54003db0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009ac0 tx=0x7f3b540057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7a800640 1 -- 172.21.15.125:0/1332663307 reap_dead start 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 -- 172.21.15.125:0/1332663307 shutdown_connections 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 -- 172.21.15.125:0/1332663307 >> 172.21.15.125:0/1332663307 conn(0x7f3b7406a3e0 msgr2=0x7f3b7406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 -- 172.21.15.125:0/1332663307 shutdown_connections 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.565+0000 7f3b7b200640 1 -- 172.21.15.125:0/1332663307 wait complete. 2024-01-16T15:22:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.567+0000 7f3b7b200640 1 Processor -- start 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.567+0000 7f3b7b200640 1 -- start start 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b54003970 0x7f3b7413cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b7413cff0 0x7f3b7413e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b7413ee20 0x7f3b741f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740ad3c0 con 0x7f3b7413ee20 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7413a210 con 0x7f3b7413cff0 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740aca90 con 0x7f3b54003970 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b7413ee20 0x7f3b741f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b7413ee20 0x7f3b741f5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35032/0 (socket says 172.21.15.125:35032) 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b7a800640 1 -- 172.21.15.125:0/2044453485 learned_addr learned my addr 172.21.15.125:0/2044453485 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b79e00640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b54003970 0x7f3b7413cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.568+0000 7f3b79400640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b7413cff0 0x7f3b7413e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 -- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b54003970 msgr2=0x7f3b7413cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b54003970 0x7f3b7413cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 -- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b7413cff0 msgr2=0x7f3b7413e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b7413cff0 0x7f3b7413e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 -- 172.21.15.125:0/2044453485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740acd70 con 0x7f3b7413ee20 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b79e00640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b54003970 0x7f3b7413cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7a800640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b7413ee20 0x7f3b741f5cf0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f3b740cc4f0 tx=0x7f3b6c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c010040 con 0x7f3b7413ee20 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c00bb10 con 0x7f3b7413ee20 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00bea0 con 0x7f3b7413ee20 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7b200640 1 -- 172.21.15.125:0/2044453485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b740ad050 con 0x7f3b7413ee20 2024-01-16T15:22:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.569+0000 7f3b7b200640 1 -- 172.21.15.125:0/2044453485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b741f6230 con 0x7f3b7413ee20 2024-01-16T15:22:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.570+0000 7f3b7b200640 1 -- 172.21.15.125:0/2044453485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b740ad570 con 0x7f3b7413ee20 2024-01-16T15:22:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.572+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3b6c01e410 con 0x7f3b7413ee20 2024-01-16T15:22:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.573+0000 7f3b72000640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b4c06c6a0 0x7f3b4c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.573+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3b6c088df0 con 0x7f3b7413ee20 2024-01-16T15:22:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.573+0000 7f3b79e00640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b4c06c6a0 0x7f3b4c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.573+0000 7f3b79e00640 1 --2- 172.21.15.125:0/2044453485 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3b4c06c6a0 0x7f3b4c06eb60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3b64036000 tx=0x7f3b64030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:12.572 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.577+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b740ad570 con 0x7f3b7413ee20 2024-01-16T15:22:12.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.800+0000 7f3b7b200640 1 -- 172.21.15.125:0/2044453485 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b7406d2b0 con 0x7f3b4c06c6a0 2024-01-16T15:22:12.803 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:12.808+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:12.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:12.808+0000 7f3b72000640 1 -- 172.21.15.125:0/2044453485 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3b6402cb60 con 0x7f3b4c06c6a0 2024-01-16T15:22:12.803 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:12.803 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:14.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:14.844 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:14.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.044+0000 7f35b3e00640 1 Processor -- start 2024-01-16T15:22:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.044+0000 7f35b3e00640 1 -- start start 2024-01-16T15:22:15.040 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.046+0000 7f35b3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f35b41447f0 0x7f35b415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.046+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.046+0000 7f35b3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b415c5c0 con 0x7f35b406dab0 2024-01-16T15:22:15.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.046+0000 7f35b3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b406d690 con 0x7f35b41447f0 2024-01-16T15:22:15.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.046+0000 7f35b3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40cebb0 con 0x7f35b406c470 2024-01-16T15:22:15.041 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.047+0000 7f35b2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.047+0000 7f35b2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:48164/0 (socket says 172.21.15.125:48164) 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.047+0000 7f35b2000640 1 -- 172.21.15.125:0/2449571921 learned_addr learned my addr 172.21.15.125:0/2449571921 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.047+0000 7f35b2a00640 1 --2- 172.21.15.125:0/2449571921 >> v2:172.21.15.137:3300/0 conn(0x7f35b41447f0 0x7f35b415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.048+0000 7f35b2000640 1 -- 172.21.15.125:0/2449571921 >> v2:172.21.15.137:3300/0 conn(0x7f35b41447f0 msgr2=0x7f35b415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.048+0000 7f35b2000640 1 --2- 172.21.15.125:0/2449571921 >> v2:172.21.15.137:3300/0 conn(0x7f35b41447f0 0x7f35b415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.048+0000 7f35b2000640 1 -- 172.21.15.125:0/2449571921 >> v1:172.21.15.137:6789/0 conn(0x7f35b406dab0 legacy=0x7f35b415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:15.042 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.048+0000 7f35b2000640 1 -- 172.21.15.125:0/2449571921 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b406ca10 con 0x7f35b406c470 2024-01-16T15:22:15.043 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.049+0000 7f35b2000640 1 --2- 172.21.15.125:0/2449571921 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f359c009870 tx=0x7f359c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297cb70d4fb93e9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:15.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.049+0000 7f35b1600640 1 -- 172.21.15.125:0/2449571921 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c031640 con 0x7f35b406c470 2024-01-16T15:22:15.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.049+0000 7f35b1600640 1 -- 172.21.15.125:0/2449571921 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c0317a0 con 0x7f35b406c470 2024-01-16T15:22:15.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.049+0000 7f35b1600640 1 -- 172.21.15.125:0/2449571921 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c031b30 con 0x7f35b406c470 2024-01-16T15:22:15.044 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 -- 172.21.15.125:0/2449571921 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 msgr2=0x7f35b40cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 --2- 172.21.15.125:0/2449571921 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f359c009870 tx=0x7f359c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 -- 172.21.15.125:0/2449571921 shutdown_connections 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 --2- 172.21.15.125:0/2449571921 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b406c470 0x7f35b40cc5e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 --2- 172.21.15.125:0/2449571921 >> v2:172.21.15.137:3300/0 conn(0x7f35b41447f0 0x7f35b415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 -- 172.21.15.125:0/2449571921 >> 172.21.15.125:0/2449571921 conn(0x7f35b406a3e0 msgr2=0x7f35b406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 -- 172.21.15.125:0/2449571921 shutdown_connections 2024-01-16T15:22:15.045 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.050+0000 7f35b3e00640 1 -- 172.21.15.125:0/2449571921 wait complete. 2024-01-16T15:22:15.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.052+0000 7f35b3e00640 1 Processor -- start 2024-01-16T15:22:15.046 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.052+0000 7f35b3e00640 1 -- start start 2024-01-16T15:22:15.047 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b406c470 0x7f35b41385d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b406dab0 0x7f35b4142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b41447f0 0x7f35b4143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b413c090 con 0x7f35b406dab0 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b413aff0 con 0x7f35b406c470 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b413b2d0 con 0x7f35b41447f0 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b406dab0 0x7f35b4142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b2000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b406dab0 0x7f35b4142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35036/0 (socket says 172.21.15.125:35036) 2024-01-16T15:22:15.048 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.053+0000 7f35b2000640 1 -- 172.21.15.125:0/2733814653 learned_addr learned my addr 172.21.15.125:0/2733814653 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:15.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2a00640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b406c470 0x7f35b41385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b3400640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b41447f0 0x7f35b4143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.049 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 -- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b41447f0 msgr2=0x7f35b4143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b41447f0 0x7f35b4143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 -- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b406c470 msgr2=0x7f35b41385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b406c470 0x7f35b41385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 -- 172.21.15.125:0/2733814653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0094d0 con 0x7f35b406dab0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b3400640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b41447f0 0x7f35b4143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b2000640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b406dab0 0x7f35b4142df0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f359c000c00 tx=0x7f359c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c03ac60 con 0x7f35b406dab0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c03adc0 con 0x7f35b406dab0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c044630 con 0x7f35b406dab0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.054+0000 7f35b3e00640 1 -- 172.21.15.125:0/2733814653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b413b5b0 con 0x7f35b406dab0 2024-01-16T15:22:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.055+0000 7f35b3e00640 1 -- 172.21.15.125:0/2733814653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b413bb70 con 0x7f35b406dab0 2024-01-16T15:22:15.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.056+0000 7f35b3e00640 1 -- 172.21.15.125:0/2733814653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b413bd60 con 0x7f35b406dab0 2024-01-16T15:22:15.051 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.057+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f359c02f050 con 0x7f35b406dab0 2024-01-16T15:22:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.057+0000 7f35aaa00640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f358c06c6a0 0x7f358c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.058+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f359c0aa9d0 con 0x7f35b406dab0 2024-01-16T15:22:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.058+0000 7f35b2a00640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f358c06c6a0 0x7f358c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:15.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.058+0000 7f35b2a00640 1 --2- 172.21.15.125:0/2733814653 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f358c06c6a0 0x7f358c06eb60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f35a4006fd0 tx=0x7f35a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:15.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.062+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f359c076f40 con 0x7f35b406dab0 2024-01-16T15:22:15.277 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.282+0000 7f35b3e00640 1 -- 172.21.15.125:0/2733814653 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35b4142830 con 0x7f358c06c6a0 2024-01-16T15:22:15.285 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:15.291+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:15.285 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:15.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:15.291+0000 7f35aaa00640 1 -- 172.21.15.125:0/2733814653 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f35b4142830 con 0x7f358c06c6a0 2024-01-16T15:22:15.286 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:17.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.533+0000 7f0f42200640 1 Processor -- start 2024-01-16T15:22:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.534+0000 7f0f42200640 1 -- start start 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f42200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f3c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f42200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c06c470 con 0x7f0f3c13e360 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f42200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c06c750 con 0x7f0f3c0cf9a0 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f42200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c06ca30 con 0x7f0f3c13df80 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f32800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0f3c13e360 0x7f0f3c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44580/0 (socket says 172.21.15.125:44580) 2024-01-16T15:22:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.535+0000 7f0f32800640 1 -- 172.21.15.125:0/4185954101 learned_addr learned my addr 172.21.15.125:0/4185954101 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f3c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399406721 0 0) 0x7f0f3c06c470 con 0x7f0f3c13e360 2024-01-16T15:22:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c003520 con 0x7f0f3c13e360 2024-01-16T15:22:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 990293203 0 0) 0x7f0f1c003520 con 0x7f0f3c13e360 2024-01-16T15:22:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 msgr2=0x7f0f3c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f3c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 >> v1:172.21.15.137:6789/0 conn(0x7f0f3c0cf9a0 legacy=0x7f0f3c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c06cd10 con 0x7f0f3c13e360 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0f24001670 con 0x7f0f3c13e360 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 >> v1:172.21.15.125:6789/0 conn(0x7f0f3c13e360 legacy=0x7f0f3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c0039a0 0x7f0f1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1c004d00 0x7f0f1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c06ca30 con 0x7f0f1c004d00 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c00bb10 con 0x7f0f1c0039a0 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.536+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c00bca0 con 0x7f0f3c13df80 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1c004d00 0x7f0f1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f32800640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c0039a0 0x7f0f1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 -- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 msgr2=0x7f0f1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c13df80 0x7f0f1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 -- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c0039a0 msgr2=0x7f0f1c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c0039a0 0x7f0f1c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3be00640 1 -- 172.21.15.125:0/4185954101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c003520 con 0x7f0f1c004d00 2024-01-16T15:22:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.537+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c0039a0 0x7f0f1c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.538+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1c004d00 0x7f0f1c005180 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f0f34001e80 tx=0x7f0f1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680a2f074903dffc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.538+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34003b70 con 0x7f0f1c004d00 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f34003cd0 con 0x7f0f1c004d00 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f3aa00640 1 -- 172.21.15.125:0/4185954101 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34005310 con 0x7f0f1c004d00 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 -- 172.21.15.125:0/4185954101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1c004d00 msgr2=0x7f0f1c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 --2- 172.21.15.125:0/4185954101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1c004d00 0x7f0f1c005180 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f0f34001e80 tx=0x7f0f1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f3be00640 1 -- 172.21.15.125:0/4185954101 reap_dead start 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 -- 172.21.15.125:0/4185954101 shutdown_connections 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 -- 172.21.15.125:0/4185954101 >> 172.21.15.125:0/4185954101 conn(0x7f0f3c06a800 msgr2=0x7f0f3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 -- 172.21.15.125:0/4185954101 shutdown_connections 2024-01-16T15:22:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.539+0000 7f0f42200640 1 -- 172.21.15.125:0/4185954101 wait complete. 2024-01-16T15:22:17.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.541+0000 7f0f42200640 1 Processor -- start 2024-01-16T15:22:17.536 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.541+0000 7f0f42200640 1 -- start start 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c004d00 0x7f0f3c1589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c0c47d0 0x7f0f3c0a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f3c0a9ff0 0x7f0f3c0aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c158840 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0c4ed0 con 0x7f0f1c004d00 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f42200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0abff0 con 0x7f0f3c0c47d0 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f3c0a9ff0 0x7f0f3c0aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f3be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f3c0a9ff0 0x7f0f3c0aa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46214/0 (socket says 172.21.15.125:46214) 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.542+0000 7f0f3be00640 1 -- 172.21.15.125:0/4059792939 learned_addr learned my addr 172.21.15.125:0/4059792939 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:17.537 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c004d00 0x7f0f3c1589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.538 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f32800640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c0c47d0 0x7f0f3c0a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 -- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c0c47d0 msgr2=0x7f0f3c0a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3c0c47d0 0x7f0f3c0a9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 -- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c004d00 msgr2=0x7f0f3c1589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f1c004d00 0x7f0f3c1589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 -- 172.21.15.125:0/4059792939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c0c4c50 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f3be00640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f3c0a9ff0 0x7f0f3c0aa470 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f0f34005040 tx=0x7f0f340014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.543+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3403e070 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.544+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f34003e40 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.544+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34006d70 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.544+0000 7f0f42200640 1 -- 172.21.15.125:0/4059792939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c13dd70 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.539 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.544+0000 7f0f42200640 1 -- 172.21.15.125:0/4059792939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c0ac4e0 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.545+0000 7f0f42200640 1 -- 172.21.15.125:0/4059792939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c0675d0 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.540 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.546+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f0f34007030 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.546+0000 7f0f33e00640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1406c670 0x7f0f1406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:17.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.547+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f340a9d50 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.541 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.547+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1406c670 0x7f0f1406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:17.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.547+0000 7f0f3b400640 1 --2- 172.21.15.125:0/4059792939 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1406c670 0x7f0f1406eb30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c006fd0 tx=0x7f0f2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:17.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.551+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f34076310 con 0x7f0f3c0a9ff0 2024-01-16T15:22:17.773 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.778+0000 7f0f42200640 1 -- 172.21.15.125:0/4059792939 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f3c15bee0 con 0x7f0f1406c670 2024-01-16T15:22:17.780 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:17.785+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:17.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:17.786+0000 7f0f33e00640 1 -- 172.21.15.125:0/4059792939 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0f2c009e70 con 0x7f0f1406c670 2024-01-16T15:22:17.780 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:17.781 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:19.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:20.018 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.023+0000 7f3e7a600640 1 Processor -- start 2024-01-16T15:22:20.019 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.024+0000 7f3e7a600640 1 -- start start 2024-01-16T15:22:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.025+0000 7f3e7a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.025+0000 7f3e7a600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e74071070 0x7f3e740cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.025+0000 7f3e7a600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e740cfbf0 con 0x7f3e740cf7d0 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.025+0000 7f3e7a600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7414b6a0 con 0x7f3e74071070 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.025+0000 7f3e7a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74071690 con 0x7f3e7414b2c0 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e74071070 0x7f3e740cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e6be00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3e740cf7d0 0x7f3e74070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44594/0 (socket says 172.21.15.125:44594) 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e6be00640 1 -- 172.21.15.125:0/219489373 learned_addr learned my addr 172.21.15.125:0/219489373 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e74071070 0x7f3e740cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46222/0 (socket says 172.21.15.125:46222) 2024-01-16T15:22:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73400640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 -- 172.21.15.125:0/219489373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 msgr2=0x7f3e740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 -- 172.21.15.125:0/219489373 >> v1:172.21.15.125:6789/0 conn(0x7f3e740cf7d0 legacy=0x7f3e74070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e73e00640 1 -- 172.21.15.125:0/219489373 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74071970 con 0x7f3e74071070 2024-01-16T15:22:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.026+0000 7f3e72a00640 1 -- 172.21.15.125:0/219489373 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080049680 0 0) 0x7f3e740cfbf0 con 0x7f3e740cf7d0 2024-01-16T15:22:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.028+0000 7f3e73e00640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e740cc160 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009d90 tx=0x7f3e6c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bedbb9b39ba3258 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.028+0000 7f3e72a00640 1 -- 172.21.15.125:0/219489373 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c02cc50 con 0x7f3e74071070 2024-01-16T15:22:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.028+0000 7f3e72a00640 1 -- 172.21.15.125:0/219489373 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c02cdb0 con 0x7f3e74071070 2024-01-16T15:22:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.028+0000 7f3e72a00640 1 -- 172.21.15.125:0/219489373 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c034660 con 0x7f3e74071070 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 -- 172.21.15.125:0/219489373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 msgr2=0x7f3e740cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e740cc160 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009d90 tx=0x7f3e6c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 -- 172.21.15.125:0/219489373 shutdown_connections 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e740cc160 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 --2- 172.21.15.125:0/219489373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 -- 172.21.15.125:0/219489373 >> 172.21.15.125:0/219489373 conn(0x7f3e7406a800 msgr2=0x7f3e7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 -- 172.21.15.125:0/219489373 shutdown_connections 2024-01-16T15:22:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.029+0000 7f3e7a600640 1 -- 172.21.15.125:0/219489373 wait complete. 2024-01-16T15:22:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.031+0000 7f3e7a600640 1 Processor -- start 2024-01-16T15:22:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.031+0000 7f3e7a600640 1 -- start start 2024-01-16T15:22:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e74145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e740cf7d0 0x7f3e74161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e741623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e740ce9c0 con 0x7f3e74071070 2024-01-16T15:22:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7413ef20 con 0x7f3e740cf7d0 2024-01-16T15:22:20.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.032+0000 7f3e7a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e740ce400 con 0x7f3e7414b2c0 2024-01-16T15:22:20.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e74145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e74145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46232/0 (socket says 172.21.15.125:46232) 2024-01-16T15:22:20.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 -- 172.21.15.125:0/3395561718 learned_addr learned my addr 172.21.15.125:0/3395561718 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73e00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e741623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e6be00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e740cf7d0 0x7f3e74161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 -- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 msgr2=0x7f3e741623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e741623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 -- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e740cf7d0 msgr2=0x7f3e74161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e740cf7d0 0x7f3e74161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:20.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 -- 172.21.15.125:0/3395561718 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c009a70 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73e00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3e7414b2c0 0x7f3e741623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e6be00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e740cf7d0 0x7f3e74161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.033+0000 7f3e73400640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e74071070 0x7f3e74145540 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400d5f0 tx=0x7f3e6400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.034+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e640125f0 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.034+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e64012750 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.034+0000 7f3e7a600640 1 -- 172.21.15.125:0/3395561718 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e740ce6e0 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.034+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64012b00 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.034+0000 7f3e7a600640 1 -- 172.21.15.125:0/3395561718 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e741436c0 con 0x7f3e74071070 2024-01-16T15:22:20.030 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.035+0000 7f3e7a600640 1 -- 172.21.15.125:0/3395561718 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74143880 con 0x7f3e74071070 2024-01-16T15:22:20.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.036+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3e64012c60 con 0x7f3e74071070 2024-01-16T15:22:20.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.037+0000 7f3e6b400640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4c06c670 0x7f3e4c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:20.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.037+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3e640890b0 con 0x7f3e74071070 2024-01-16T15:22:20.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.037+0000 7f3e6be00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4c06c670 0x7f3e4c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:20.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.038+0000 7f3e6be00640 1 --2- 172.21.15.125:0/3395561718 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3e4c06c670 0x7f3e4c06eb30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3e741fa8b0 tx=0x7f3e5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:20.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.041+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e640555c0 con 0x7f3e74071070 2024-01-16T15:22:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.264+0000 7f3e7a600640 1 -- 172.21.15.125:0/3395561718 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e740cb960 con 0x7f3e4c06c670 2024-01-16T15:22:20.266 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:20.271+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:20.272+0000 7f3e6b400640 1 -- 172.21.15.125:0/3395561718 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3e5c0013d0 con 0x7f3e4c06c670 2024-01-16T15:22:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:22.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:22.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.508+0000 7fdf71a00640 1 Processor -- start 2024-01-16T15:22:22.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.509+0000 7fdf71a00640 1 -- start start 2024-01-16T15:22:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf71a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf6c15d610 0x7fdf6c0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf71a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf6c0ced60 0x7fdf6c0d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf71a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0d2170 con 0x7fdf6c0d1d20 2024-01-16T15:22:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf71a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c0d1390 con 0x7fdf6c15d610 2024-01-16T15:22:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf71a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b3400 con 0x7fdf6c0ced60 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf6aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf6c15d610 0x7fdf6c0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf6aa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf6c15d610 0x7fdf6c0ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46240/0 (socket says 172.21.15.125:46240) 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf6aa00640 1 -- 172.21.15.125:0/421195493 learned_addr learned my addr 172.21.15.125:0/421195493 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.510+0000 7fdf6b400640 1 --1- 172.21.15.125:0/421195493 >> v1:172.21.15.125:6789/0 conn(0x7fdf6c0d1d20 0x7fdf6c0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44604/0 (socket says 172.21.15.125:44604) 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf61e00640 1 --2- 172.21.15.125:0/421195493 >> v2:172.21.15.137:3300/0 conn(0x7fdf6c0ced60 0x7fdf6c0d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf6aa00640 1 -- 172.21.15.125:0/421195493 >> v2:172.21.15.137:3300/0 conn(0x7fdf6c0ced60 msgr2=0x7fdf6c0d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/421195493 >> v2:172.21.15.137:3300/0 conn(0x7fdf6c0ced60 0x7fdf6c0d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf6aa00640 1 -- 172.21.15.125:0/421195493 >> v1:172.21.15.125:6789/0 conn(0x7fdf6c0d1d20 legacy=0x7fdf6c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf6a000640 1 -- 172.21.15.125:0/421195493 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356034354 0 0) 0x7fdf64001490 con 0x7fdf6c0d1d20 2024-01-16T15:22:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.511+0000 7fdf6aa00640 1 -- 172.21.15.125:0/421195493 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0b27f0 con 0x7fdf6c15d610 2024-01-16T15:22:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.512+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/421195493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0ce820 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c08d4b0 tx=0x7fdf5c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b466687c8d24d7a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf6a000640 1 -- 172.21.15.125:0/421195493 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c030860 con 0x7fdf6c15d610 2024-01-16T15:22:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf6a000640 1 -- 172.21.15.125:0/421195493 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0309c0 con 0x7fdf6c15d610 2024-01-16T15:22:22.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf6a000640 1 -- 172.21.15.125:0/421195493 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c030d50 con 0x7fdf6c15d610 2024-01-16T15:22:22.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 -- 172.21.15.125:0/421195493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 msgr2=0x7fdf6c0ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:22.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 --2- 172.21.15.125:0/421195493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0ce820 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c08d4b0 tx=0x7fdf5c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 -- 172.21.15.125:0/421195493 shutdown_connections 2024-01-16T15:22:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 --2- 172.21.15.125:0/421195493 >> v2:172.21.15.137:3300/0 conn(0x7fdf6c0ced60 0x7fdf6c0d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 --2- 172.21.15.125:0/421195493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0ce820 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 -- 172.21.15.125:0/421195493 >> 172.21.15.125:0/421195493 conn(0x7fdf6c09c540 msgr2=0x7fdf6c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 -- 172.21.15.125:0/421195493 shutdown_connections 2024-01-16T15:22:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.513+0000 7fdf71a00640 1 -- 172.21.15.125:0/421195493 wait complete. 2024-01-16T15:22:22.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.515+0000 7fdf71a00640 1 Processor -- start 2024-01-16T15:22:22.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.515+0000 7fdf71a00640 1 -- start start 2024-01-16T15:22:22.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0ced60 0x7fdf6c1f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.512 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0d1d20 0x7fdf6c1f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b6b10 con 0x7fdf6c15d610 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0cb8c0 con 0x7fdf6c0d1d20 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.516+0000 7fdf71a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c1f6a60 con 0x7fdf6c0ced60 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46242/0 (socket says 172.21.15.125:46242) 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 -- 172.21.15.125:0/896218494 learned_addr learned my addr 172.21.15.125:0/896218494 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf61e00640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0d1d20 0x7fdf6c1f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0ced60 0x7fdf6c1f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 -- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0ced60 msgr2=0x7fdf6c1f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0ced60 0x7fdf6c1f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 -- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0d1d20 msgr2=0x7fdf6c1f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0d1d20 0x7fdf6c1f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 -- 172.21.15.125:0/896218494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0094d0 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0ced60 0x7fdf6c1f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.517+0000 7fdf6b400640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf6c15d610 0x7fdf6c0caca0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400c7f0 tx=0x7fdf6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.518+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400a5e0 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.518+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6400a760 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.518+0000 7fdf71a00640 1 -- 172.21.15.125:0/896218494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c0cb5e0 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.518+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400ab50 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.518+0000 7fdf71a00640 1 -- 172.21.15.125:0/896218494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c0b6730 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.519+0000 7fdf71a00640 1 -- 172.21.15.125:0/896218494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c0b62b0 con 0x7fdf6c15d610 2024-01-16T15:22:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.520+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fdf6400acb0 con 0x7fdf6c15d610 2024-01-16T15:22:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.521+0000 7fdf63400640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf4406c670 0x7fdf4406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.521+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf64088260 con 0x7fdf6c15d610 2024-01-16T15:22:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.521+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf4406c670 0x7fdf4406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.521+0000 7fdf6aa00640 1 --2- 172.21.15.125:0/896218494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdf4406c670 0x7fdf4406eb30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c036000 tx=0x7fdf5c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:22.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.525+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf6c0b62b0 con 0x7fdf6c15d610 2024-01-16T15:22:22.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.746+0000 7fdf71a00640 1 -- 172.21.15.125:0/896218494 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf6c0b3090 con 0x7fdf4406c670 2024-01-16T15:22:22.753 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:22.758+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:22.753 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:22.758+0000 7fdf63400640 1 -- 172.21.15.125:0/896218494 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdf5c030cc0 con 0x7fdf4406c670 2024-01-16T15:22:22.753 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:22.754 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:24.769 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:24.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:24.999+0000 7f0f3d000640 1 Processor -- start 2024-01-16T15:22:24.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:24.999+0000 7f0f3d000640 1 -- start start 2024-01-16T15:22:24.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f3d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0f380cf9a0 0x7f0f3815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f3d000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380b1ee0 con 0x7f0f38143ae0 2024-01-16T15:22:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f3d000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f380b2780 con 0x7f0f380cf9a0 2024-01-16T15:22:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3815e990 con 0x7f0f3815e1e0 2024-01-16T15:22:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f35e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0f38143ae0 0x7f0f380cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44608/0 (socket says 172.21.15.125:44608) 2024-01-16T15:22:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f35e00640 1 -- 172.21.15.125:0/800429987 learned_addr learned my addr 172.21.15.125:0/800429987 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f2d200640 1 --2- 172.21.15.125:0/800429987 >> v2:172.21.15.137:3300/0 conn(0x7f0f380cf9a0 0x7f0f3815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f36800640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050097386 0 0) 0x7f0f380b1ee0 con 0x7f0f38143ae0 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.001+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18005730 con 0x7f0f38143ae0 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754237419 0 0) 0x7f0f18005730 con 0x7f0f38143ae0 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 msgr2=0x7f0f380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 >> v2:172.21.15.137:3300/0 conn(0x7f0f380cf9a0 msgr2=0x7f0f3815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 --2- 172.21.15.125:0/800429987 >> v2:172.21.15.137:3300/0 conn(0x7f0f380cf9a0 0x7f0f3815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380b2130 con 0x7f0f38143ae0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f0f28001670 con 0x7f0f38143ae0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 >> v1:172.21.15.125:6789/0 conn(0x7f0f38143ae0 legacy=0x7f0f380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f18005b80 0x7f0f18005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f180069d0 0x7f0f18006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f18007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380b2780 con 0x7f0f180069d0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3815e990 con 0x7f0f18005b80 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800db90 con 0x7f0f3815e1e0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.002+0000 7f0f2d200640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f180069d0 0x7f0f18006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f35e00640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f18005b80 0x7f0f18005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f36800640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f18007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f2d200640 1 -- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 msgr2=0x7f0f18007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:25.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f2d200640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f3815e1e0 0x7f0f18007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:25.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f2d200640 1 -- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f18005b80 msgr2=0x7f0f18005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:25.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f2d200640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f18005b80 0x7f0f18005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:25.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.003+0000 7f0f2d200640 1 -- 172.21.15.125:0/800429987 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18005730 con 0x7f0f180069d0 2024-01-16T15:22:25.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.004+0000 7f0f2d200640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f180069d0 0x7f0f18006e50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009ac0 tx=0x7f0f18007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7640f1eeea94fda1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:25.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.004+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2002c710 con 0x7f0f180069d0 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.004+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2002c870 con 0x7f0f180069d0 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.004+0000 7f0f35400640 1 -- 172.21.15.125:0/800429987 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2002cc00 con 0x7f0f180069d0 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 -- 172.21.15.125:0/800429987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f180069d0 msgr2=0x7f0f18006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 --2- 172.21.15.125:0/800429987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f180069d0 0x7f0f18006e50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009ac0 tx=0x7f0f18007a00 comp rx=0 tx=0).stop 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f36800640 1 -- 172.21.15.125:0/800429987 reap_dead start 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 -- 172.21.15.125:0/800429987 shutdown_connections 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 -- 172.21.15.125:0/800429987 >> 172.21.15.125:0/800429987 conn(0x7f0f3806a800 msgr2=0x7f0f3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 -- 172.21.15.125:0/800429987 shutdown_connections 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.005+0000 7f0f3d000640 1 -- 172.21.15.125:0/800429987 wait complete. 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.007+0000 7f0f3d000640 1 Processor -- start 2024-01-16T15:22:25.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.007+0000 7f0f3d000640 1 -- start start 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f180069d0 0x7f0f381eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f380a8ed0 0x7f0f3813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f380cc610 0x7f0f380cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380ae790 con 0x7f0f380a8ed0 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380a9330 con 0x7f0f380cc610 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f3d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f381ef0c0 con 0x7f0f180069d0 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f380a8ed0 0x7f0f3813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f380a8ed0 0x7f0f3813d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46266/0 (socket says 172.21.15.125:46266) 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 -- 172.21.15.125:0/904634698 learned_addr learned my addr 172.21.15.125:0/904634698 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f36800640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f380cc610 0x7f0f380cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f35e00640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f180069d0 0x7f0f381eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 -- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f180069d0 msgr2=0x7f0f381eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0f180069d0 0x7f0f381eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 -- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f380cc610 msgr2=0x7f0f380cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f380cc610 0x7f0f380cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f2d200640 1 -- 172.21.15.125:0/904634698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f381ef3a0 con 0x7f0f380a8ed0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.008+0000 7f0f36800640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0f380cc610 0x7f0f380cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f2d200640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f380a8ed0 0x7f0f3813d860 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f0f2002c2b0 tx=0x7f0f20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20003810 con 0x7f0f380a8ed0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2002de90 con 0x7f0f380a8ed0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20003b70 con 0x7f0f380a8ed0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f3d000640 1 -- 172.21.15.125:0/904634698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f381ef680 con 0x7f0f380a8ed0 2024-01-16T15:22:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.009+0000 7f0f3d000640 1 -- 172.21.15.125:0/904634698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f381f2100 con 0x7f0f380a8ed0 2024-01-16T15:22:25.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.010+0000 7f0f3d000640 1 -- 172.21.15.125:0/904634698 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f381f2290 con 0x7f0f380a8ed0 2024-01-16T15:22:25.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.011+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f0f20003cd0 con 0x7f0f380a8ed0 2024-01-16T15:22:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.012+0000 7f0f2ea00640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1006c6a0 0x7f0f1006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.013+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f200aaa70 con 0x7f0f380a8ed0 2024-01-16T15:22:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.013+0000 7f0f35e00640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1006c6a0 0x7f0f1006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.013+0000 7f0f35e00640 1 --2- 172.21.15.125:0/904634698 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f0f1006c6a0 0x7f0f1006eb60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0f28001eb0 tx=0x7f0f28000fd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:25.012 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.017+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f20076fe0 con 0x7f0f380a8ed0 2024-01-16T15:22:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.235+0000 7f0f3d000640 1 -- 172.21.15.125:0/904634698 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f38143380 con 0x7f0f1006c6a0 2024-01-16T15:22:25.238 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:25.243+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:25.238 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:25.243+0000 7f0f2ea00640 1 -- 172.21.15.125:0/904634698 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0f38143380 con 0x7f0f1006c6a0 2024-01-16T15:22:25.238 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:25.239 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:27.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:27.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.481+0000 7fd20a400640 1 Processor -- start 2024-01-16T15:22:27.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.482+0000 7fd20a400640 1 -- start start 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd20a400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd204072230 0x7fd204150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd20a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd20415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd20a400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd204071190 con 0x7fd2040cf6d0 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd20a400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd204072610 con 0x7fd204072230 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd20a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd204071470 con 0x7fd2040cf290 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd203e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd2040cf6d0 0x7fd2040b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60030/0 (socket says 172.21.15.125:60030) 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd203e00640 1 -- 172.21.15.125:0/3988125310 learned_addr learned my addr 172.21.15.125:0/3988125310 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:27.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.483+0000 7fd203400640 1 --2- 172.21.15.125:0/3988125310 >> v2:172.21.15.125:3300/0 conn(0x7fd204072230 0x7fd204150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd1fa800640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd20415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd202a00640 1 -- 172.21.15.125:0/3988125310 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367304313 0 0) 0x7fd204071190 con 0x7fd2040cf6d0 2024-01-16T15:22:27.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd202a00640 1 -- 172.21.15.125:0/3988125310 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e4003520 con 0x7fd2040cf6d0 2024-01-16T15:22:27.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd203400640 1 -- 172.21.15.125:0/3988125310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 msgr2=0x7fd20415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:27.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd203400640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd20415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd203400640 1 -- 172.21.15.125:0/3988125310 >> v1:172.21.15.125:6789/0 conn(0x7fd2040cf6d0 legacy=0x7fd2040b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:27.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.484+0000 7fd203400640 1 -- 172.21.15.125:0/3988125310 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204071750 con 0x7fd204072230 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.485+0000 7fd203400640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204150760 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009d90 tx=0x7fd1f402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac2c8b969054f39 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.485+0000 7fd202a00640 1 -- 172.21.15.125:0/3988125310 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f402cc60 con 0x7fd204072230 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.485+0000 7fd202a00640 1 -- 172.21.15.125:0/3988125310 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f402cdc0 con 0x7fd204072230 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.485+0000 7fd202a00640 1 -- 172.21.15.125:0/3988125310 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4034660 con 0x7fd204072230 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 -- 172.21.15.125:0/3988125310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 msgr2=0x7fd204150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204150760 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009d90 tx=0x7fd1f402beb0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 -- 172.21.15.125:0/3988125310 shutdown_connections 2024-01-16T15:22:27.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd20415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 --2- 172.21.15.125:0/3988125310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204150760 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 -- 172.21.15.125:0/3988125310 >> 172.21.15.125:0/3988125310 conn(0x7fd20406a800 msgr2=0x7fd20406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:27.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 -- 172.21.15.125:0/3988125310 shutdown_connections 2024-01-16T15:22:27.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.486+0000 7fd20a400640 1 -- 172.21.15.125:0/3988125310 wait complete. 2024-01-16T15:22:27.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.488+0000 7fd20a400640 1 Processor -- start 2024-01-16T15:22:27.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.488+0000 7fd20a400640 1 -- start start 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd204158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2040cf6d0 0x7fd204158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd204159370 con 0x7fd204072230 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040ccb60 con 0x7fd2040cf6d0 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd20a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040ce7b0 con 0x7fd2040cf290 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd203400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd203400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40044/0 (socket says 172.21.15.125:40044) 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.489+0000 7fd203400640 1 -- 172.21.15.125:0/3232087929 learned_addr learned my addr 172.21.15.125:0/3232087929 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd1fa800640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd204158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203e00640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2040cf6d0 0x7fd204158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 -- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 msgr2=0x7fd204158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2040cf290 0x7fd204158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 -- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2040cf6d0 msgr2=0x7fd204158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2040cf6d0 0x7fd204158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 -- 172.21.15.125:0/3232087929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f4009a70 con 0x7fd204072230 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203e00640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2040cf6d0 0x7fd204158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:27.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd203400640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd204072230 0x7fd204157f60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009d90 tx=0x7fd1f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4040650 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f40407b0 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4040b60 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.490+0000 7fd20a400640 1 -- 172.21.15.125:0/3232087929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2041f2f10 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.491+0000 7fd20a400640 1 -- 172.21.15.125:0/3232087929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2041f31d0 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.492+0000 7fd20a400640 1 -- 172.21.15.125:0/3232087929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2040675d0 con 0x7fd204072230 2024-01-16T15:22:27.487 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.493+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fd1f4040cc0 con 0x7fd204072230 2024-01-16T15:22:27.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.493+0000 7fd1fbe00640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd1dc06c670 0x7fd1dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:27.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.493+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1f40aba40 con 0x7fd204072230 2024-01-16T15:22:27.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.494+0000 7fd1fa800640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd1dc06c670 0x7fd1dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:27.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.494+0000 7fd1fa800640 1 --2- 172.21.15.125:0/3232087929 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd1dc06c670 0x7fd1dc06eb30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd20415e050 tx=0x7fd1ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:27.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.498+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1f4077fb0 con 0x7fd204072230 2024-01-16T15:22:27.722 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.727+0000 7fd20a400640 1 -- 172.21.15.125:0/3232087929 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd204071d10 con 0x7fd1dc06c670 2024-01-16T15:22:27.729 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:27.734+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:27.729 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:27.735+0000 7fd1fbe00640 1 -- 172.21.15.125:0/3232087929 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd1ec009500 con 0x7fd1dc06c670 2024-01-16T15:22:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:27.730 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:29.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:29.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.977+0000 7f7108800640 1 Processor -- start 2024-01-16T15:22:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.978+0000 7f7108800640 1 -- start start 2024-01-16T15:22:29.974 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f710013df80 0x7f710015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013e360 0x7f71000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f710015f9c0 0x7f71000cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710006c470 con 0x7f710013df80 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710006c750 con 0x7f710015f9c0 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.979+0000 7f7108800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710006ca30 con 0x7f710013e360 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f710013df80 0x7f710015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f710013df80 0x7f710015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40048/0 (socket says 172.21.15.125:40048) 2024-01-16T15:22:29.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 -- 172.21.15.125:0/451225549 learned_addr learned my addr 172.21.15.125:0/451225549 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7106800640 1 --2- 172.21.15.125:0/451225549 >> v2:172.21.15.137:3300/0 conn(0x7f710015f9c0 0x7f71000cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105400640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013e360 0x7f71000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 -- 172.21.15.125:0/451225549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013e360 msgr2=0x7f71000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013e360 0x7f71000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 -- 172.21.15.125:0/451225549 >> v2:172.21.15.137:3300/0 conn(0x7f710015f9c0 msgr2=0x7f71000cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:29.976 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 --2- 172.21.15.125:0/451225549 >> v2:172.21.15.137:3300/0 conn(0x7f710015f9c0 0x7f71000cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.980+0000 7f7105e00640 1 -- 172.21.15.125:0/451225549 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710006cd10 con 0x7f710013df80 2024-01-16T15:22:29.977 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.982+0000 7f7105e00640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013df80 0x7f710015bee0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f710006e6d0 tx=0x7f70f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2175b7a6e93eb5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.982+0000 7f7104a00640 1 -- 172.21.15.125:0/451225549 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f002ca70 con 0x7f710013df80 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.982+0000 7f7104a00640 1 -- 172.21.15.125:0/451225549 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f002cbd0 con 0x7f710013df80 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.982+0000 7f7104a00640 1 -- 172.21.15.125:0/451225549 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f00343f0 con 0x7f710013df80 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 -- 172.21.15.125:0/451225549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013df80 msgr2=0x7f710015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013df80 0x7f710015bee0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f710006e6d0 tx=0x7f70f002bed0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 -- 172.21.15.125:0/451225549 shutdown_connections 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 --2- 172.21.15.125:0/451225549 >> v2:172.21.15.137:3300/0 conn(0x7f710015f9c0 0x7f71000cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013e360 0x7f71000cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.978 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 --2- 172.21.15.125:0/451225549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013df80 0x7f710015bee0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 -- 172.21.15.125:0/451225549 >> 172.21.15.125:0/451225549 conn(0x7f710006a800 msgr2=0x7f710006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:29.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 -- 172.21.15.125:0/451225549 shutdown_connections 2024-01-16T15:22:29.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.983+0000 7f7108800640 1 -- 172.21.15.125:0/451225549 wait complete. 2024-01-16T15:22:29.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.985+0000 7f7108800640 1 Processor -- start 2024-01-16T15:22:29.979 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.985+0000 7f7108800640 1 -- start start 2024-01-16T15:22:29.980 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013df80 0x7f7100143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.981 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013e360 0x7f71001f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f710015f9c0 0x7f7100155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71001572e0 con 0x7f710013e360 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7100156660 con 0x7f710015f9c0 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7108800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7100156940 con 0x7f710013df80 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.986+0000 7f7105400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013e360 0x7f71001f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013e360 0x7f71001f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40056/0 (socket says 172.21.15.125:40056) 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 -- 172.21.15.125:0/1099310977 learned_addr learned my addr 172.21.15.125:0/1099310977 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105e00640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013df80 0x7f7100143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 -- 172.21.15.125:0/1099310977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013df80 msgr2=0x7f7100143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f710013df80 0x7f7100143fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 -- 172.21.15.125:0/1099310977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f710015f9c0 msgr2=0x7f7100155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.982 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f710015f9c0 0x7f7100155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.983 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 -- 172.21.15.125:0/1099310977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0009b30 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.987+0000 7f7105400640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f710013e360 0x7f71001f0380 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f70e800d800 tx=0x7f70e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.988+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800b6c0 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.988+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800b840 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.988+0000 7f7108800640 1 -- 172.21.15.125:0/1099310977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7100156c20 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.988+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800bbf0 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.988+0000 7f7108800640 1 -- 172.21.15.125:0/1099310977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71001f08c0 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.989+0000 7f7108800640 1 -- 172.21.15.125:0/1099310977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7100156a80 con 0x7f710013e360 2024-01-16T15:22:29.984 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.990+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f70e800bd50 con 0x7f710013e360 2024-01-16T15:22:29.985 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.991+0000 7f70fe000640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70d806c580 0x7f70d806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.991+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f70e80890e0 con 0x7f710013e360 2024-01-16T15:22:29.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.991+0000 7f7105e00640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70d806c580 0x7f70d806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.986 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.992+0000 7f7105e00640 1 --2- 172.21.15.125:0/1099310977 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f70d806c580 0x7f70d806ea40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f70f0035000 tx=0x7f70f0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.990 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:29.995+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7100156a80 con 0x7f710013e360 2024-01-16T15:22:30.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:30.219+0000 7f7108800640 1 -- 172.21.15.125:0/1099310977 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f710006d410 con 0x7f70d806c580 2024-01-16T15:22:30.222 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:30.226+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:30.222 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:30.226+0000 7f70fe000640 1 -- 172.21.15.125:0/1099310977 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f710006d410 con 0x7f70d806c580 2024-01-16T15:22:30.223 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:30.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:32.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.470+0000 7f28db200640 1 Processor -- start 2024-01-16T15:22:32.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.470+0000 7f28db200640 1 -- start start 2024-01-16T15:22:32.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28db200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28dc146850 0x7f28dc0715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28db200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc0ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28db200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc06d480 con 0x7f28dc0adb00 2024-01-16T15:22:32.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28db200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc071fa0 con 0x7f28dc146850 2024-01-16T15:22:32.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28db200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0b3490 con 0x7f28dc071b10 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28d9e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28dc146850 0x7f28dc0715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28d9e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f28dc146850 0x7f28dc0715d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40062/0 (socket says 172.21.15.125:40062) 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28d9e00640 1 -- 172.21.15.125:0/4050111240 learned_addr learned my addr 172.21.15.125:0/4050111240 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.472+0000 7f28d3e00640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc0ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.473+0000 7f28d9e00640 1 -- 172.21.15.125:0/4050111240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 msgr2=0x7f28dc0ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.473+0000 7f28d9e00640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc0ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.473+0000 7f28d9e00640 1 -- 172.21.15.125:0/4050111240 >> v1:172.21.15.125:6789/0 conn(0x7f28dc0adb00 legacy=0x7f28dc0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:32.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.473+0000 7f28d9e00640 1 -- 172.21.15.125:0/4050111240 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0b3770 con 0x7f28dc146850 2024-01-16T15:22:32.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.474+0000 7f28d9e00640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc146850 0x7f28dc0715d0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009870 tx=0x7f28cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88473c00d84df0f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:32.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.474+0000 7f28d9400640 1 -- 172.21.15.125:0/4050111240 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc030840 con 0x7f28dc146850 2024-01-16T15:22:32.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.474+0000 7f28d9400640 1 -- 172.21.15.125:0/4050111240 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc0309a0 con 0x7f28dc146850 2024-01-16T15:22:32.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.474+0000 7f28d9400640 1 -- 172.21.15.125:0/4050111240 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc030d30 con 0x7f28dc146850 2024-01-16T15:22:32.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 -- 172.21.15.125:0/4050111240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc146850 msgr2=0x7f28dc0715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:32.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc146850 0x7f28dc0715d0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009870 tx=0x7f28cc02b980 comp rx=0 tx=0).stop 2024-01-16T15:22:32.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 -- 172.21.15.125:0/4050111240 shutdown_connections 2024-01-16T15:22:32.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc0ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 --2- 172.21.15.125:0/4050111240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc146850 0x7f28dc0715d0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 -- 172.21.15.125:0/4050111240 >> 172.21.15.125:0/4050111240 conn(0x7f28dc06a3e0 msgr2=0x7f28dc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 -- 172.21.15.125:0/4050111240 shutdown_connections 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.475+0000 7f28db200640 1 -- 172.21.15.125:0/4050111240 wait complete. 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.477+0000 7f28db200640 1 Processor -- start 2024-01-16T15:22:32.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.477+0000 7f28db200640 1 -- start start 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc12ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc0adb00 0x7f28dc12d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28dc146850 0x7f28dc15c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc12f1b0 con 0x7f28dc0adb00 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc12dbd0 con 0x7f28dc146850 2024-01-16T15:22:32.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28db200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc12deb0 con 0x7f28dc071b10 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc0adb00 0x7f28dc12d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc0adb00 0x7f28dc12d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40078/0 (socket says 172.21.15.125:40078) 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 -- 172.21.15.125:0/866518806 learned_addr learned my addr 172.21.15.125:0/866518806 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d9e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc12ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28da800640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28dc146850 0x7f28dc15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 -- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 msgr2=0x7f28dc12ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28dc071b10 0x7f28dc12ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 -- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28dc146850 msgr2=0x7f28dc15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28dc146850 0x7f28dc15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28d3e00640 1 -- 172.21.15.125:0/866518806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0094d0 con 0x7f28dc0adb00 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.478+0000 7f28da800640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28dc146850 0x7f28dc15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:32.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28d3e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28dc0adb00 0x7f28dc12d310 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f28c400d800 tx=0x7f28c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400b6c0 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c400b820 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400bbb0 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28db200640 1 -- 172.21.15.125:0/866518806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc12e190 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.479+0000 7f28db200640 1 -- 172.21.15.125:0/866518806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc12e750 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.480+0000 7f28db200640 1 -- 172.21.15.125:0/866518806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc12e910 con 0x7f28dc0adb00 2024-01-16T15:22:32.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.481+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f28c400bd10 con 0x7f28dc0adb00 2024-01-16T15:22:32.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.482+0000 7f28d2000640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f28b406c550 0x7f28b406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:32.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.482+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28c4089110 con 0x7f28dc0adb00 2024-01-16T15:22:32.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.482+0000 7f28d9e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f28b406c550 0x7f28b406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:32.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.483+0000 7f28d9e00640 1 --2- 172.21.15.125:0/866518806 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f28b406c550 0x7f28b406ea10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f28cc036000 tx=0x7f28cc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:32.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.487+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28c4055650 con 0x7f28dc0adb00 2024-01-16T15:22:32.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.707+0000 7f28db200640 1 -- 172.21.15.125:0/866518806 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28dc146c30 con 0x7f28b406c550 2024-01-16T15:22:32.716 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:32.720+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:32.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:32.720+0000 7f28d2000640 1 -- 172.21.15.125:0/866518806 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f28cc030ca0 con 0x7f28b406c550 2024-01-16T15:22:32.716 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:32.716 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:34.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:34.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.960+0000 7f3d57e00640 1 Processor -- start 2024-01-16T15:22:34.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.960+0000 7f3d57e00640 1 -- start start 2024-01-16T15:22:34.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.962+0000 7f3d57e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d581447f0 0x7f3d5815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.962+0000 7f3d57e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d580cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.962+0000 7f3d57e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5815c5c0 con 0x7f3d5806dab0 2024-01-16T15:22:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.962+0000 7f3d57e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5806d690 con 0x7f3d581447f0 2024-01-16T15:22:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.962+0000 7f3d57e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580cebb0 con 0x7f3d5806c470 2024-01-16T15:22:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.963+0000 7f3d4fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d580cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.963+0000 7f3d56a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d581447f0 0x7f3d5815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.963+0000 7f3d57400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f3d5806dab0 0x7f3d5815c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:52258/0 (socket says 172.21.15.125:52258) 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.963+0000 7f3d57400640 1 -- 172.21.15.125:0/1281363053 learned_addr learned my addr 172.21.15.125:0/1281363053 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.964+0000 7f3d56a00640 1 -- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 msgr2=0x7f3d580cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.964+0000 7f3d56a00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d580cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.964+0000 7f3d56a00640 1 -- 172.21.15.125:0/1281363053 >> v1:172.21.15.137:6789/0 conn(0x7f3d5806dab0 legacy=0x7f3d5815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.964+0000 7f3d56a00640 1 -- 172.21.15.125:0/1281363053 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5806ca10 con 0x7f3d581447f0 2024-01-16T15:22:34.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.964+0000 7f3d4fe00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d580cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:22:34.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.966+0000 7f3d56a00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d5815c080 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b56d36b354bbbc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.966+0000 7f3d56000640 1 -- 172.21.15.125:0/1281363053 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4802e070 con 0x7f3d581447f0 2024-01-16T15:22:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.966+0000 7f3d56000640 1 -- 172.21.15.125:0/1281363053 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d480315a0 con 0x7f3d581447f0 2024-01-16T15:22:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.966+0000 7f3d56000640 1 -- 172.21.15.125:0/1281363053 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48031930 con 0x7f3d581447f0 2024-01-16T15:22:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 -- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 msgr2=0x7f3d5815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d5815c080 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b840 comp rx=0 tx=0).stop 2024-01-16T15:22:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 -- 172.21.15.125:0/1281363053 shutdown_connections 2024-01-16T15:22:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d580cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 --2- 172.21.15.125:0/1281363053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d5815c080 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 -- 172.21.15.125:0/1281363053 >> 172.21.15.125:0/1281363053 conn(0x7f3d5806a3e0 msgr2=0x7f3d5806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 -- 172.21.15.125:0/1281363053 shutdown_connections 2024-01-16T15:22:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.967+0000 7f3d57e00640 1 -- 172.21.15.125:0/1281363053 wait complete. 2024-01-16T15:22:34.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.969+0000 7f3d57e00640 1 Processor -- start 2024-01-16T15:22:34.966 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.970+0000 7f3d57e00640 1 -- start start 2024-01-16T15:22:34.968 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d58139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d5806dab0 0x7f3d58141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d580a8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580aa410 con 0x7f3d5806dab0 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5813b330 con 0x7f3d581447f0 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58142540 con 0x7f3d5806c470 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d4fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d5806dab0 0x7f3d58141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d4fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d5806dab0 0x7f3d58141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40086/0 (socket says 172.21.15.125:40086) 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d4fe00640 1 -- 172.21.15.125:0/427817177 learned_addr learned my addr 172.21.15.125:0/427817177 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d56a00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d58139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.971+0000 7f3d57400640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d580a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 -- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 msgr2=0x7f3d58139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5806c470 0x7f3d58139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 -- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 msgr2=0x7f3d580a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d581447f0 0x7f3d580a8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 -- 172.21.15.125:0/427817177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40009400 con 0x7f3d5806dab0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4fe00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d5806dab0 0x7f3d58141a80 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e960 tx=0x7f3d4000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000ca00 con 0x7f3d5806dab0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4000cb60 con 0x7f3d5806dab0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d57e00640 1 -- 172.21.15.125:0/427817177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48009400 con 0x7f3d5806dab0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40013470 con 0x7f3d5806dab0 2024-01-16T15:22:34.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.972+0000 7f3d57e00640 1 -- 172.21.15.125:0/427817177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5813b540 con 0x7f3d5806dab0 2024-01-16T15:22:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.973+0000 7f3d57e00640 1 -- 172.21.15.125:0/427817177 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d58067450 con 0x7f3d5806dab0 2024-01-16T15:22:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.974+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3d400135d0 con 0x7f3d5806dab0 2024-01-16T15:22:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.975+0000 7f3d4ea00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d3006c580 0x7f3d3006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.975+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d40088a60 con 0x7f3d5806dab0 2024-01-16T15:22:34.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.976+0000 7f3d56a00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d3006c580 0x7f3d3006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.976+0000 7f3d56a00640 1 --2- 172.21.15.125:0/427817177 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3d3006c580 0x7f3d3006ea40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3d48036000 tx=0x7f3d4803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.975 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:34.980+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d40055320 con 0x7f3d5806dab0 2024-01-16T15:22:35.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:35.202+0000 7f3d57e00640 1 -- 172.21.15.125:0/427817177 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d5806d2b0 con 0x7f3d3006c580 2024-01-16T15:22:35.209 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:35.213+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:35.210 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:35.214+0000 7f3d4ea00640 1 -- 172.21.15.125:0/427817177 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3d48031ae0 con 0x7f3d3006c580 2024-01-16T15:22:35.210 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:35.210 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:37.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:37.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.456+0000 7f9eef800640 1 Processor -- start 2024-01-16T15:22:37.453 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.457+0000 7f9eef800640 1 -- start start 2024-01-16T15:22:37.454 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.458+0000 7f9eef800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ee8099a60 0x7f9ee8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.458+0000 7f9eef800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 0x7f9ee806ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.458+0000 7f9eef800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee80cfa70 con 0x7f9ee815e880 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.458+0000 7f9eef800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee809a3e0 con 0x7f9ee8099a60 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.458+0000 7f9eef800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80cff00 con 0x7f9ee806e7c0 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ee8099a60 0x7f9ee8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ee8099a60 0x7f9ee8099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54002/0 (socket says 172.21.15.125:54002) 2024-01-16T15:22:37.455 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 -- 172.21.15.125:0/3868744883 learned_addr learned my addr 172.21.15.125:0/3868744883 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9eed800640 1 --2- 172.21.15.125:0/3868744883 >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 0x7f9ee806ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 -- 172.21.15.125:0/3868744883 >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 msgr2=0x7f9ee806ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 --2- 172.21.15.125:0/3868744883 >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 0x7f9ee806ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 -- 172.21.15.125:0/3868744883 >> v1:172.21.15.125:6789/0 conn(0x7f9ee815e880 legacy=0x7f9ee806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9ee7e00640 1 -- 172.21.15.125:0/3868744883 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee806fc70 con 0x7f9ee8099a60 2024-01-16T15:22:37.456 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.459+0000 7f9eed800640 1 --2- 172.21.15.125:0/3868744883 >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 0x7f9ee806ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:37.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.460+0000 7f9ee7e00640 1 --2- 172.21.15.125:0/3868744883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee8099a60 0x7f9ee8099ea0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80b72c0 tx=0x7f9ed002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a7684e44c6b324 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:37.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9ee7400640 1 -- 172.21.15.125:0/3868744883 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed002f850 con 0x7f9ee8099a60 2024-01-16T15:22:37.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9ee7400640 1 -- 172.21.15.125:0/3868744883 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed002f9b0 con 0x7f9ee8099a60 2024-01-16T15:22:37.457 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9ee7400640 1 -- 172.21.15.125:0/3868744883 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed002fd40 con 0x7f9ee8099a60 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 -- 172.21.15.125:0/3868744883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee8099a60 msgr2=0x7f9ee8099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 --2- 172.21.15.125:0/3868744883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee8099a60 0x7f9ee8099ea0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80b72c0 tx=0x7f9ed002ba30 comp rx=0 tx=0).stop 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 -- 172.21.15.125:0/3868744883 shutdown_connections 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 --2- 172.21.15.125:0/3868744883 >> v2:172.21.15.137:3300/0 conn(0x7f9ee806e7c0 0x7f9ee806ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 --2- 172.21.15.125:0/3868744883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee8099a60 0x7f9ee8099ea0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 -- 172.21.15.125:0/3868744883 >> 172.21.15.125:0/3868744883 conn(0x7f9ee80b2d00 msgr2=0x7f9ee80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.461+0000 7f9eef800640 1 -- 172.21.15.125:0/3868744883 shutdown_connections 2024-01-16T15:22:37.458 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.462+0000 7f9eef800640 1 -- 172.21.15.125:0/3868744883 wait complete. 2024-01-16T15:22:37.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.463+0000 7f9eef800640 1 Processor -- start 2024-01-16T15:22:37.459 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.463+0000 7f9eef800640 1 -- start start 2024-01-16T15:22:37.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee806e7c0 0x7f9ee80b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ee8099a60 0x7f9ee815a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ee815e880 0x7f9ee80c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80b67b0 con 0x7f9ee806e7c0 2024-01-16T15:22:37.461 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80b5b30 con 0x7f9ee815e880 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.464+0000 7f9eef800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80b5e10 con 0x7f9ee8099a60 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee806e7c0 0x7f9ee80b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee806e7c0 0x7f9ee80b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54010/0 (socket says 172.21.15.125:54010) 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 -- 172.21.15.125:0/1174292998 learned_addr learned my addr 172.21.15.125:0/1174292998 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 -- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ee8099a60 msgr2=0x7f9ee815a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9ee7e00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ee8099a60 0x7f9ee815a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eed800640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ee815e880 0x7f9ee80c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ee8099a60 0x7f9ee815a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 -- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ee815e880 msgr2=0x7f9ee80c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ee815e880 0x7f9ee80c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.465+0000 7f9eece00640 1 -- 172.21.15.125:0/1174292998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed00094d0 con 0x7f9ee806e7c0 2024-01-16T15:22:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9eece00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ee806e7c0 0x7f9ee80b6bd0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800c400 tx=0x7f9ed800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80052d0 con 0x7f9ee806e7c0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed8005430 con 0x7f9ee806e7c0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8003830 con 0x7f9ee806e7c0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9eef800640 1 -- 172.21.15.125:0/1174292998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee80b60f0 con 0x7f9ee806e7c0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.466+0000 7f9eef800640 1 -- 172.21.15.125:0/1174292998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee80c7970 con 0x7f9ee806e7c0 2024-01-16T15:22:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.467+0000 7f9eef800640 1 -- 172.21.15.125:0/1174292998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee80b68f0 con 0x7f9ee806e7c0 2024-01-16T15:22:37.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.468+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9ed8003990 con 0x7f9ee806e7c0 2024-01-16T15:22:37.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.469+0000 7f9ee4c00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9ec006c550 0x7f9ec006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:37.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.469+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9ed8088110 con 0x7f9ee806e7c0 2024-01-16T15:22:37.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.469+0000 7f9ee7e00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9ec006c550 0x7f9ec006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:37.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.470+0000 7f9ee7e00640 1 --2- 172.21.15.125:0/1174292998 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f9ec006c550 0x7f9ec006ea10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0036000 tx=0x7f9ed0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:37.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.474+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ed8054650 con 0x7f9ee806e7c0 2024-01-16T15:22:37.692 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.696+0000 7f9eef800640 1 -- 172.21.15.125:0/1174292998 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ee80cf700 con 0x7f9ec006c550 2024-01-16T15:22:37.703 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:37.708+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:37.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:37.708+0000 7f9ee4c00640 1 -- 172.21.15.125:0/1174292998 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ed002fcb0 con 0x7f9ec006c550 2024-01-16T15:22:37.704 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:37.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:39.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:39.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.943+0000 7f4c38c00640 1 Processor -- start 2024-01-16T15:22:39.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.943+0000 7f4c38c00640 1 -- start start 2024-01-16T15:22:39.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c38c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4c34143ae0 0x7f4c340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c38c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c3415e3b0 0x7f4c340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c38c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340cfc30 con 0x7f4c340cf7d0 2024-01-16T15:22:39.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c38c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340b1ee0 con 0x7f4c34143ae0 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c38c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340b2780 con 0x7f4c3415e3b0 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c31e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4c34143ae0 0x7f4c340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c31e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4c34143ae0 0x7f4c340cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54014/0 (socket says 172.21.15.125:54014) 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c31e00640 1 -- 172.21.15.125:0/3403351794 learned_addr learned my addr 172.21.15.125:0/3403351794 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c32800640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c3415e3b0 0x7f4c340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.945+0000 7f4c31e00640 1 -- 172.21.15.125:0/3403351794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c3415e3b0 msgr2=0x7f4c340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.946+0000 7f4c31e00640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c3415e3b0 0x7f4c340702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.946+0000 7f4c31e00640 1 -- 172.21.15.125:0/3403351794 >> v1:172.21.15.125:6789/0 conn(0x7f4c340cf7d0 legacy=0x7f4c3415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.946+0000 7f4c31e00640 1 -- 172.21.15.125:0/3403351794 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3415e990 con 0x7f4c34143ae0 2024-01-16T15:22:39.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c31e00640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c34143ae0 0x7f4c340cf290 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f4c240098a0 tx=0x7f4c2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b1b41b2c8886aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c30a00640 1 -- 172.21.15.125:0/3403351794 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24030840 con 0x7f4c34143ae0 2024-01-16T15:22:39.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c30a00640 1 -- 172.21.15.125:0/3403351794 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c240309a0 con 0x7f4c34143ae0 2024-01-16T15:22:39.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c30a00640 1 -- 172.21.15.125:0/3403351794 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24030d30 con 0x7f4c34143ae0 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c38c00640 1 -- 172.21.15.125:0/3403351794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c34143ae0 msgr2=0x7f4c340cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.947+0000 7f4c38c00640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c34143ae0 0x7f4c340cf290 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f4c240098a0 tx=0x7f4c2402b980 comp rx=0 tx=0).stop 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 -- 172.21.15.125:0/3403351794 shutdown_connections 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c3415e3b0 0x7f4c340702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 --2- 172.21.15.125:0/3403351794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c34143ae0 0x7f4c340cf290 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 -- 172.21.15.125:0/3403351794 >> 172.21.15.125:0/3403351794 conn(0x7f4c3406a800 msgr2=0x7f4c3406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 -- 172.21.15.125:0/3403351794 shutdown_connections 2024-01-16T15:22:39.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.948+0000 7f4c38c00640 1 -- 172.21.15.125:0/3403351794 wait complete. 2024-01-16T15:22:39.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.949+0000 7f4c38c00640 1 Processor -- start 2024-01-16T15:22:39.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.949+0000 7f4c38c00640 1 -- start start 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c340cf7d0 0x7f4c34137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c34143ae0 0x7f4c3413d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c3415e3b0 0x7f4c3413d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34139dc0 con 0x7f4c3415e3b0 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34159db0 con 0x7f4c340cf7d0 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.950+0000 7f4c38c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3415a090 con 0x7f4c34143ae0 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c3415e3b0 0x7f4c3413d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.947 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c3415e3b0 0x7f4c3413d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54026/0 (socket says 172.21.15.125:54026) 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 -- 172.21.15.125:0/3300157973 learned_addr learned my addr 172.21.15.125:0/3300157973 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c31400640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c34143ae0 0x7f4c3413d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c31e00640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c340cf7d0 0x7f4c34137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 -- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c34143ae0 msgr2=0x7f4c3413d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c34143ae0 0x7f4c3413d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 -- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c340cf7d0 msgr2=0x7f4c34137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c340cf7d0 0x7f4c34137800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 -- 172.21.15.125:0/3300157973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240094d0 con 0x7f4c3415e3b0 2024-01-16T15:22:39.948 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.951+0000 7f4c32800640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c3415e3b0 0x7f4c3413d760 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c007e60 tx=0x7f4c2c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.952+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c014420 con 0x7f4c3415e3b0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.952+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c0145a0 con 0x7f4c3415e3b0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.952+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c014950 con 0x7f4c3415e3b0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.952+0000 7f4c38c00640 1 -- 172.21.15.125:0/3300157973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c34139770 con 0x7f4c3415e3b0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.952+0000 7f4c38c00640 1 -- 172.21.15.125:0/3300157973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c34137d40 con 0x7f4c3415e3b0 2024-01-16T15:22:39.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.953+0000 7f4c38c00640 1 -- 172.21.15.125:0/3300157973 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c34139f70 con 0x7f4c3415e3b0 2024-01-16T15:22:39.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.954+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4c2c014ab0 con 0x7f4c3415e3b0 2024-01-16T15:22:39.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.955+0000 7f4c2a000640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4c0c06c550 0x7f4c0c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.955+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c2c0894c0 con 0x7f4c3415e3b0 2024-01-16T15:22:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.955+0000 7f4c31e00640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4c0c06c550 0x7f4c0c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.956+0000 7f4c31e00640 1 --2- 172.21.15.125:0/3300157973 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f4c0c06c550 0x7f4c0c06ea10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4c24036000 tx=0x7f4c2403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:39.959+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c34139f70 con 0x7f4c3415e3b0 2024-01-16T15:22:40.178 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:40.182+0000 7f4c38c00640 1 -- 172.21.15.125:0/3300157973 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c340b2410 con 0x7f4c0c06c550 2024-01-16T15:22:40.186 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:40.190+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:40.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:40.191+0000 7f4c2a000640 1 -- 172.21.15.125:0/3300157973 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c24030ca0 con 0x7f4c0c06c550 2024-01-16T15:22:40.187 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:40.187 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:42.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:42.421 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.425+0000 7f272d200640 1 Processor -- start 2024-01-16T15:22:42.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.425+0000 7f272d200640 1 -- start start 2024-01-16T15:22:42.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f272815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f272813e360 0x7f27280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f272815f9c0 0x7f27280cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272806c470 con 0x7f272813e360 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272806c750 con 0x7f272815f9c0 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f272d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272806ca30 con 0x7f272813df80 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.427+0000 7f271d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f272813e360 0x7f27280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f272813e360 0x7f27280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54042/0 (socket says 172.21.15.125:54042) 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 -- 172.21.15.125:0/3946794248 learned_addr learned my addr 172.21.15.125:0/3946794248 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f2726800640 1 --2- 172.21.15.125:0/3946794248 >> v2:172.21.15.137:3300/0 conn(0x7f272815f9c0 0x7f27280cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f2725e00640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f272815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 -- 172.21.15.125:0/3946794248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 msgr2=0x7f272815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f272815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 -- 172.21.15.125:0/3946794248 >> v2:172.21.15.137:3300/0 conn(0x7f272815f9c0 msgr2=0x7f27280cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 --2- 172.21.15.125:0/3946794248 >> v2:172.21.15.137:3300/0 conn(0x7f272815f9c0 0x7f27280cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.428+0000 7f271d200640 1 -- 172.21.15.125:0/3946794248 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272806cd10 con 0x7f272813e360 2024-01-16T15:22:42.425 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.429+0000 7f271d200640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280cf290 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f272806e6d0 tx=0x7f271002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d949e0649d31fb92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:42.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.429+0000 7f2725400640 1 -- 172.21.15.125:0/3946794248 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710031870 con 0x7f272813e360 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.429+0000 7f2725400640 1 -- 172.21.15.125:0/3946794248 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100319d0 con 0x7f272813e360 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.429+0000 7f2725400640 1 -- 172.21.15.125:0/3946794248 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710031d60 con 0x7f272813e360 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 -- 172.21.15.125:0/3946794248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 msgr2=0x7f27280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280cf290 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f272806e6d0 tx=0x7f271002b910 comp rx=0 tx=0).stop 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 -- 172.21.15.125:0/3946794248 shutdown_connections 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 --2- 172.21.15.125:0/3946794248 >> v2:172.21.15.137:3300/0 conn(0x7f272815f9c0 0x7f27280cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280cf290 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 --2- 172.21.15.125:0/3946794248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f272815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 -- 172.21.15.125:0/3946794248 >> 172.21.15.125:0/3946794248 conn(0x7f272806a800 msgr2=0x7f272806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 -- 172.21.15.125:0/3946794248 shutdown_connections 2024-01-16T15:22:42.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.430+0000 7f272d200640 1 -- 172.21.15.125:0/3946794248 wait complete. 2024-01-16T15:22:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.432+0000 7f272d200640 1 Processor -- start 2024-01-16T15:22:42.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.432+0000 7f272d200640 1 -- start start 2024-01-16T15:22:42.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f27281346d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.430 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f272815f9c0 0x7f27280a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280b1ca0 con 0x7f272813e360 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272813cc90 con 0x7f272815f9c0 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f272d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280a9010 con 0x7f272813df80 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f271d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f271d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54046/0 (socket says 172.21.15.125:54046) 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.433+0000 7f271d200640 1 -- 172.21.15.125:0/661121570 learned_addr learned my addr 172.21.15.125:0/661121570 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f2725e00640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f27281346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f2726800640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f272815f9c0 0x7f27280a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 -- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 msgr2=0x7f27281346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f272813df80 0x7f27281346d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.431 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 -- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f272815f9c0 msgr2=0x7f27280a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f272815f9c0 0x7f27280a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 -- 172.21.15.125:0/661121570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27100094d0 con 0x7f272813e360 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.434+0000 7f271d200640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f272813e360 0x7f27280b2680 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f271002b950 tx=0x7f27100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.435+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271003ad30 con 0x7f272813e360 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.435+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100443f0 con 0x7f272813e360 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.435+0000 7f272d200640 1 -- 172.21.15.125:0/661121570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27280a92f0 con 0x7f272813e360 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.435+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100447a0 con 0x7f272813e360 2024-01-16T15:22:42.432 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.435+0000 7f272d200640 1 -- 172.21.15.125:0/661121570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27280b2160 con 0x7f272813e360 2024-01-16T15:22:42.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.436+0000 7f272d200640 1 -- 172.21.15.125:0/661121570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27280a9500 con 0x7f272813e360 2024-01-16T15:22:42.433 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.437+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f271004d450 con 0x7f272813e360 2024-01-16T15:22:42.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.438+0000 7f271ea00640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f270006c550 0x7f270006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:42.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.438+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f271002d040 con 0x7f272813e360 2024-01-16T15:22:42.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.438+0000 7f2725e00640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f270006c550 0x7f270006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:42.434 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.439+0000 7f2725e00640 1 --2- 172.21.15.125:0/661121570 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f270006c550 0x7f270006ea10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2718006fd0 tx=0x7f2718008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:42.438 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.443+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27100776e0 con 0x7f272813e360 2024-01-16T15:22:42.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.671+0000 7f272d200640 1 -- 172.21.15.125:0/661121570 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27281408d0 con 0x7f270006c550 2024-01-16T15:22:42.677 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:42.682+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:42.682+0000 7f271ea00640 1 -- 172.21.15.125:0/661121570 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f271800f860 con 0x7f270006c550 2024-01-16T15:22:42.678 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:42.678 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:44.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:44.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.923+0000 7f6d20c00640 1 Processor -- start 2024-01-16T15:22:44.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.924+0000 7f6d20c00640 1 -- start start 2024-01-16T15:22:44.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.925+0000 7f6d20c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d1c15d3c0 0x7f6d1c0d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.925+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.925+0000 7f6d20c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c0d12f0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.925+0000 7f6d20c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0ce5d0 con 0x7f6d1c15d3c0 2024-01-16T15:22:44.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.925+0000 7f6d20c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b23c0 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:56726/0 (socket says 172.21.15.125:56726) 2024-01-16T15:22:44.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 -- 172.21.15.125:0/2791282059 learned_addr learned my addr 172.21.15.125:0/2791282059 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:44.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1b400640 1 --2- 172.21.15.125:0/2791282059 >> v2:172.21.15.137:3300/0 conn(0x7f6d1c15d3c0 0x7f6d1c0d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 -- 172.21.15.125:0/2791282059 >> v2:172.21.15.137:3300/0 conn(0x7f6d1c15d3c0 msgr2=0x7f6d1c0d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:44.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 --2- 172.21.15.125:0/2791282059 >> v2:172.21.15.137:3300/0 conn(0x7f6d1c15d3c0 0x7f6d1c0d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 -- 172.21.15.125:0/2791282059 >> v1:172.21.15.137:6789/0 conn(0x7f6d1c0d0e90 legacy=0x7f6d1c0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.926+0000 7f6d1be00640 1 -- 172.21.15.125:0/2791282059 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c0d0800 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.923 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.928+0000 7f6d1be00640 1 --2- 172.21.15.125:0/2791282059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009e50 tx=0x7f6d1402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea037bfb6d774b60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.928+0000 7f6d1aa00640 1 -- 172.21.15.125:0/2791282059 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1402ca70 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.928+0000 7f6d1aa00640 1 -- 172.21.15.125:0/2791282059 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1402cbd0 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.928+0000 7f6d1aa00640 1 -- 172.21.15.125:0/2791282059 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d140343f0 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 -- 172.21.15.125:0/2791282059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 msgr2=0x7f6d1c0cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 --2- 172.21.15.125:0/2791282059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009e50 tx=0x7f6d1402bed0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 -- 172.21.15.125:0/2791282059 shutdown_connections 2024-01-16T15:22:44.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 --2- 172.21.15.125:0/2791282059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c0cee80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 --2- 172.21.15.125:0/2791282059 >> v2:172.21.15.137:3300/0 conn(0x7f6d1c15d3c0 0x7f6d1c0d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 -- 172.21.15.125:0/2791282059 >> 172.21.15.125:0/2791282059 conn(0x7f6d1c0d30b0 msgr2=0x7f6d1c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:44.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 -- 172.21.15.125:0/2791282059 shutdown_connections 2024-01-16T15:22:44.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.929+0000 7f6d20c00640 1 -- 172.21.15.125:0/2791282059 wait complete. 2024-01-16T15:22:44.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.931+0000 7f6d20c00640 1 Processor -- start 2024-01-16T15:22:44.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.931+0000 7f6d20c00640 1 -- start start 2024-01-16T15:22:44.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d1c0d0e90 0x7f6d1c0b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d1c15d3c0 0x7f6d1c0b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.929 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1593c0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b34e0 con 0x7f6d1c15d3c0 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d20c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b37c0 con 0x7f6d1c0ce9f0 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d13e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d1c0d0e90 0x7f6d1c0b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d13e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d1c0d0e90 0x7f6d1c0b8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54056/0 (socket says 172.21.15.125:54056) 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d13e00640 1 -- 172.21.15.125:0/2087048146 learned_addr learned my addr 172.21.15.125:0/2087048146 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.932+0000 7f6d1b400640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d1be00640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d1c15d3c0 0x7f6d1c0b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 -- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 msgr2=0x7f6d1c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1c0ce9f0 0x7f6d1c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 -- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d1c15d3c0 msgr2=0x7f6d1c0b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d1c15d3c0 0x7f6d1c0b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 -- 172.21.15.125:0/2087048146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d14009b30 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13e00640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d1c0d0e90 0x7f6d1c0b8120 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c400 tx=0x7f6d0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040052d0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d04005430 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003830 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d20c00640 1 -- 172.21.15.125:0/2087048146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c0b3aa0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.933+0000 7f6d20c00640 1 -- 172.21.15.125:0/2087048146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c0b5dd0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.934+0000 7f6d20c00640 1 -- 172.21.15.125:0/2087048146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c0b3fc0 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.931 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.936+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f6d04003990 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.936+0000 7f6d13400640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cf406c550 0x7f6cf406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.936+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6d04087f10 con 0x7f6d1c0d0e90 2024-01-16T15:22:44.932 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.937+0000 7f6d1b400640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cf406c550 0x7f6cf406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.933 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.937+0000 7f6d1b400640 1 --2- 172.21.15.125:0/2087048146 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f6cf406c550 0x7f6cf406ea10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c160c70 tx=0x7f6d0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.937 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:44.941+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d1c0b3fc0 con 0x7f6d1c0d0e90 2024-01-16T15:22:45.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:45.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:45.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:45.166+0000 7f6d20c00640 1 -- 172.21.15.125:0/2087048146 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d1c0cdbc0 con 0x7f6cf406c550 2024-01-16T15:22:45.174 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:45.178+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:45.174 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:45.178+0000 7f6d13400640 1 -- 172.21.15.125:0/2087048146 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6d0c0096a0 con 0x7f6cf406c550 2024-01-16T15:22:45.174 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:45.175 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:47.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:47.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.414+0000 7f08ed200640 1 Processor -- start 2024-01-16T15:22:47.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.414+0000 7f08ed200640 1 -- start start 2024-01-16T15:22:47.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08ed200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08ed200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08e8070ea0 0x7f08e80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08ed200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e8071690 con 0x7f08e814b2c0 2024-01-16T15:22:47.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08ed200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e814b6a0 con 0x7f08e8070ea0 2024-01-16T15:22:47.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08ed200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8071970 con 0x7f08e80cf9a0 2024-01-16T15:22:47.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.416+0000 7f08dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:51286/0 (socket says 172.21.15.125:51286) 2024-01-16T15:22:47.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08dfe00640 1 -- 172.21.15.125:0/3090571038 learned_addr learned my addr 172.21.15.125:0/3090571038 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:47.413 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08e6800640 1 --2- 172.21.15.125:0/3090571038 >> v2:172.21.15.137:3300/0 conn(0x7f08e8070ea0 0x7f08e80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08dfe00640 1 -- 172.21.15.125:0/3090571038 >> v2:172.21.15.137:3300/0 conn(0x7f08e8070ea0 msgr2=0x7f08e80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08dfe00640 1 --2- 172.21.15.125:0/3090571038 >> v2:172.21.15.137:3300/0 conn(0x7f08e8070ea0 0x7f08e80cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08dfe00640 1 -- 172.21.15.125:0/3090571038 >> v1:172.21.15.137:6789/0 conn(0x7f08e814b2c0 legacy=0x7f08e80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08dfe00640 1 -- 172.21.15.125:0/3090571038 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8071c50 con 0x7f08e80cf9a0 2024-01-16T15:22:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.417+0000 7f08e5400640 1 -- 172.21.15.125:0/3090571038 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959620741 0 0) 0x7f08e8071690 con 0x7f08e814b2c0 2024-01-16T15:22:47.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.419+0000 7f08dfe00640 1 --2- 172.21.15.125:0/3090571038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009870 tx=0x7f08d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad46de075a40ec1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:47.415 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.419+0000 7f08e5400640 1 -- 172.21.15.125:0/3090571038 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0030840 con 0x7f08e80cf9a0 2024-01-16T15:22:47.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08e5400640 1 -- 172.21.15.125:0/3090571038 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d00309a0 con 0x7f08e80cf9a0 2024-01-16T15:22:47.416 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08e5400640 1 -- 172.21.15.125:0/3090571038 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0030d30 con 0x7f08e80cf9a0 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 -- 172.21.15.125:0/3090571038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 msgr2=0x7f08e8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 --2- 172.21.15.125:0/3090571038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009870 tx=0x7f08d002b980 comp rx=0 tx=0).stop 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 -- 172.21.15.125:0/3090571038 shutdown_connections 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 --2- 172.21.15.125:0/3090571038 >> v2:172.21.15.137:3300/0 conn(0x7f08e8070ea0 0x7f08e80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 --2- 172.21.15.125:0/3090571038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8070960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 -- 172.21.15.125:0/3090571038 >> 172.21.15.125:0/3090571038 conn(0x7f08e806a800 msgr2=0x7f08e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 -- 172.21.15.125:0/3090571038 shutdown_connections 2024-01-16T15:22:47.417 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.420+0000 7f08ed200640 1 -- 172.21.15.125:0/3090571038 wait complete. 2024-01-16T15:22:47.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.422+0000 7f08ed200640 1 Processor -- start 2024-01-16T15:22:47.418 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.422+0000 7f08ed200640 1 -- start start 2024-01-16T15:22:47.419 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08e8070ea0 0x7f08e8164670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08e814b2c0 0x7f08e81611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8145ca0 con 0x7f08e814b2c0 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8163af0 con 0x7f08e8070ea0 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08ed200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8161f10 con 0x7f08e80cf9a0 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08e6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08e814b2c0 0x7f08e81611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08e6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08e814b2c0 0x7f08e81611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53214/0 (socket says 172.21.15.125:53214) 2024-01-16T15:22:47.420 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.423+0000 7f08e6800640 1 -- 172.21.15.125:0/1289667978 learned_addr learned my addr 172.21.15.125:0/1289667978 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08dfe00640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e5e00640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08e8070ea0 0x7f08e8164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 -- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 msgr2=0x7f08e8145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f08e80cf9a0 0x7f08e8145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 -- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08e8070ea0 msgr2=0x7f08e8164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08e8070ea0 0x7f08e8164670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 -- 172.21.15.125:0/1289667978 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d00094d0 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08e6800640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08e814b2c0 0x7f08e81611b0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f08e000efc0 tx=0x7f08e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.424+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0014480 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.425+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e000f040 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.425+0000 7f08ed200640 1 -- 172.21.15.125:0/1289667978 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e81621f0 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.425+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0014820 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.425+0000 7f08ed200640 1 -- 172.21.15.125:0/1289667978 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e81627b0 con 0x7f08e814b2c0 2024-01-16T15:22:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.426+0000 7f08ed200640 1 -- 172.21.15.125:0/1289667978 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8161750 con 0x7f08e814b2c0 2024-01-16T15:22:47.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.427+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f08e0014980 con 0x7f08e814b2c0 2024-01-16T15:22:47.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.428+0000 7f08de000640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c006c550 0x7f08c006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:47.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.428+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f08e0089760 con 0x7f08e814b2c0 2024-01-16T15:22:47.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.428+0000 7f08e5e00640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c006c550 0x7f08c006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:47.424 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.429+0000 7f08e5e00640 1 --2- 172.21.15.125:0/1289667978 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f08c006c550 0x7f08c006ea10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f08d8000f60 tx=0x7f08d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:47.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.432+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08e0055ca0 con 0x7f08e814b2c0 2024-01-16T15:22:47.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.656+0000 7f08ed200640 1 -- 172.21.15.125:0/1289667978 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08e8071330 con 0x7f08c006c550 2024-01-16T15:22:47.664 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:47.668+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:47.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:47.668+0000 7f08de000640 1 -- 172.21.15.125:0/1289667978 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f08d80050d0 con 0x7f08c006c550 2024-01-16T15:22:47.665 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:49.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:49.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.905+0000 7f729b200640 1 Processor -- start 2024-01-16T15:22:49.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.905+0000 7f729b200640 1 -- start start 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.905+0000 7f729b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 0x7f729c0d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.905+0000 7f729b200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0cf6a0 con 0x7f729c160c70 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f729b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0d1430 con 0x7f729c0c6be0 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f729b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0b4370 con 0x7f729c160890 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f7299400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f729c160c70 0x7f729c0d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48786/0 (socket says 172.21.15.125:48786) 2024-01-16T15:22:49.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f7299400640 1 -- 172.21.15.125:0/85073175 learned_addr learned my addr 172.21.15.125:0/85073175 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:49.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724746271 0 0) 0x7f729c0cf6a0 con 0x7f729c160c70 2024-01-16T15:22:49.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f7299e00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 0x7f729c0d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.906+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f727c003520 con 0x7f729c160c70 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262844066 0 0) 0x7f727c003520 con 0x7f729c160c70 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 msgr2=0x7f729c0d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 0x7f729c0d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 >> v1:172.21.15.137:6789/0 conn(0x7f729c0c6be0 legacy=0x7f729c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c0ce7c0 con 0x7f729c160c70 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f7284001670 con 0x7f729c160c70 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 >> v1:172.21.15.125:6789/0 conn(0x7f729c160c70 legacy=0x7f729c0d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c0039a0 0x7f727c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f727c0047c0 0x7f727c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 0x7f727c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0b4370 con 0x7f727c0039a0 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c00bb40 con 0x7f727c0047c0 2024-01-16T15:22:49.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c00bd00 con 0x7f729c160890 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.907+0000 7f7299e00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c0039a0 0x7f727c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299400640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f727c0047c0 0x7f727c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299e00640 1 -- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 msgr2=0x7f727c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299e00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c160890 0x7f727c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299e00640 1 -- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f727c0047c0 msgr2=0x7f727c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299e00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f727c0047c0 0x7f727c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.908+0000 7f7299e00640 1 -- 172.21.15.125:0/85073175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c003520 con 0x7f727c0039a0 2024-01-16T15:22:49.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.909+0000 7f7299e00640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c0039a0 0x7f727c003dc0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f728c009ac0 tx=0x7f727c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6327cd4d8791aedb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.909+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c02c710 con 0x7f727c0039a0 2024-01-16T15:22:49.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.909+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728c02c870 con 0x7f727c0039a0 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.909+0000 7f7298a00640 1 -- 172.21.15.125:0/85073175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c02cc00 con 0x7f727c0039a0 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 -- 172.21.15.125:0/85073175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c0039a0 msgr2=0x7f727c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 --2- 172.21.15.125:0/85073175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c0039a0 0x7f727c003dc0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f728c009ac0 tx=0x7f727c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729a800640 1 -- 172.21.15.125:0/85073175 reap_dead start 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 -- 172.21.15.125:0/85073175 shutdown_connections 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 -- 172.21.15.125:0/85073175 >> 172.21.15.125:0/85073175 conn(0x7f729c0d3090 msgr2=0x7f729c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 -- 172.21.15.125:0/85073175 shutdown_connections 2024-01-16T15:22:49.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.910+0000 7f729b200640 1 -- 172.21.15.125:0/85073175 wait complete. 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.912+0000 7f729b200640 1 Processor -- start 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.912+0000 7f729b200640 1 -- start start 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c1f26d0 0x7f729c06be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f729c06c380 0x7f729c14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f729c1503f0 0x7f729c0b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c151890 con 0x7f729c06c380 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c06c800 con 0x7f729c1503f0 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f729b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c151cb0 con 0x7f729c1f26d0 2024-01-16T15:22:49.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f7299400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f729c06c380 0x7f729c14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f7299400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f729c06c380 0x7f729c14feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53224/0 (socket says 172.21.15.125:53224) 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.913+0000 7f7299400640 1 -- 172.21.15.125:0/2636154157 learned_addr learned my addr 172.21.15.125:0/2636154157 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299e00640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c1f26d0 0x7f729c06be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 -- 172.21.15.125:0/2636154157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c1f26d0 msgr2=0x7f729c06be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f729c1f26d0 0x7f729c06be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 -- 172.21.15.125:0/2636154157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f729c1503f0 msgr2=0x7f729c0b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f729c1503f0 0x7f729c0b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 -- 172.21.15.125:0/2636154157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c150c10 con 0x7f729c06c380 2024-01-16T15:22:49.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7299400640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f729c06c380 0x7f729c14feb0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f728400dae0 tx=0x7f7284007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284004cb0 con 0x7f729c06c380 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7284004e10 con 0x7f729c06c380 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72840072d0 con 0x7f729c06c380 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.914+0000 7f729b200640 1 -- 172.21.15.125:0/2636154157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c150ef0 con 0x7f729c06c380 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.915+0000 7f729b200640 1 -- 172.21.15.125:0/2636154157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c1514b0 con 0x7f729c06c380 2024-01-16T15:22:49.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.916+0000 7f729b200640 1 -- 172.21.15.125:0/2636154157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c151670 con 0x7f729c06c380 2024-01-16T15:22:49.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.916+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7284007430 con 0x7f729c06c380 2024-01-16T15:22:49.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.918+0000 7f7292000640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f727406c580 0x7f727406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.918+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f72840884d0 con 0x7f729c06c380 2024-01-16T15:22:49.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.918+0000 7f7299e00640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f727406c580 0x7f727406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.919+0000 7f7299e00640 1 --2- 172.21.15.125:0/2636154157 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f727406c580 0x7f727406ea40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f728c036000 tx=0x7f728c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:49.922+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7284054a10 con 0x7f729c06c380 2024-01-16T15:22:50.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:50.146+0000 7f729b200640 1 -- 172.21.15.125:0/2636154157 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f729c0ced80 con 0x7f727406c580 2024-01-16T15:22:50.153 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:50.156+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:50.157+0000 7f7292000640 1 -- 172.21.15.125:0/2636154157 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f728c02c9e0 con 0x7f727406c580 2024-01-16T15:22:50.153 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:50.153 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:52.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:52.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.399+0000 7ff217e00640 1 Processor -- start 2024-01-16T15:22:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.399+0000 7ff217e00640 1 -- start start 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff217e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff217e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff21015e3b0 0x7ff2100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff217e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2100cfc30 con 0x7ff2100cf7d0 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff217e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100b1ee0 con 0x7ff21015e3b0 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff217e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100b2780 con 0x7ff210143ae0 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff215e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff21015e3b0 0x7ff2100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff215e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff21015e3b0 0x7ff2100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53238/0 (socket says 172.21.15.125:53238) 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.401+0000 7ff215e00640 1 -- 172.21.15.125:0/3406936352 learned_addr learned my addr 172.21.15.125:0/3406936352 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.402+0000 7ff215400640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.402+0000 7ff215e00640 1 -- 172.21.15.125:0/3406936352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 msgr2=0x7ff2100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.402+0000 7ff215e00640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.402+0000 7ff215e00640 1 -- 172.21.15.125:0/3406936352 >> v1:172.21.15.137:6789/0 conn(0x7ff2100cf7d0 legacy=0x7ff21015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:52.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.402+0000 7ff215e00640 1 -- 172.21.15.125:0/3406936352 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21015e990 con 0x7ff21015e3b0 2024-01-16T15:22:52.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.403+0000 7ff215e00640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff21015e3b0 0x7ff2100702c0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e02b429ef2e328 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:52.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.403+0000 7ff20fe00640 1 -- 172.21.15.125:0/3406936352 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20802ec80 con 0x7ff21015e3b0 2024-01-16T15:22:52.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.403+0000 7ff20fe00640 1 -- 172.21.15.125:0/3406936352 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20802ede0 con 0x7ff21015e3b0 2024-01-16T15:22:52.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.403+0000 7ff20fe00640 1 -- 172.21.15.125:0/3406936352 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208036610 con 0x7ff21015e3b0 2024-01-16T15:22:52.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 -- 172.21.15.125:0/3406936352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff21015e3b0 msgr2=0x7ff2100702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff21015e3b0 0x7ff2100702c0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20802e0e0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 -- 172.21.15.125:0/3406936352 shutdown_connections 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff21015e3b0 0x7ff2100702c0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 --2- 172.21.15.125:0/3406936352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 -- 172.21.15.125:0/3406936352 >> 172.21.15.125:0/3406936352 conn(0x7ff21006a800 msgr2=0x7ff21006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 -- 172.21.15.125:0/3406936352 shutdown_connections 2024-01-16T15:22:52.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.404+0000 7ff217e00640 1 -- 172.21.15.125:0/3406936352 wait complete. 2024-01-16T15:22:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.407+0000 7ff217e00640 1 Processor -- start 2024-01-16T15:22:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.407+0000 7ff217e00640 1 -- start start 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2100cf7d0 0x7ff2101f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff21015e3b0 0x7ff21015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100ac8d0 con 0x7ff2100cf7d0 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101f3b90 con 0x7ff21015e3b0 2024-01-16T15:22:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff217e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100acb70 con 0x7ff210143ae0 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff215400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2100cf7d0 0x7ff2101f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff215400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2100cf7d0 0x7ff2101f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53250/0 (socket says 172.21.15.125:53250) 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.408+0000 7ff215400640 1 -- 172.21.15.125:0/1324678092 learned_addr learned my addr 172.21.15.125:0/1324678092 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215e00640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff21015e3b0 0x7ff21015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 -- 172.21.15.125:0/1324678092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 msgr2=0x7ff2100cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff210143ae0 0x7ff2100cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 -- 172.21.15.125:0/1324678092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff21015e3b0 msgr2=0x7ff21015b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff21015e3b0 0x7ff21015b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 -- 172.21.15.125:0/1324678092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208009b30 con 0x7ff2100cf7d0 2024-01-16T15:22:52.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff215400640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2100cf7d0 0x7ff2101f3ed0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7ff20000d800 tx=0x7ff20000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.409+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000b6c0 con 0x7ff2100cf7d0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.410+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000b820 con 0x7ff2100cf7d0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.410+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000bbb0 con 0x7ff2100cf7d0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.410+0000 7ff217e00640 1 -- 172.21.15.125:0/1324678092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100ace50 con 0x7ff2100cf7d0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.410+0000 7ff217e00640 1 -- 172.21.15.125:0/1324678092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2100cb4e0 con 0x7ff2100cf7d0 2024-01-16T15:22:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.411+0000 7ff217e00640 1 -- 172.21.15.125:0/1324678092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2100675d0 con 0x7ff2100cf7d0 2024-01-16T15:22:52.407 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.412+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff20000bd10 con 0x7ff2100cf7d0 2024-01-16T15:22:52.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.413+0000 7ff20d600640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff1e806c550 0x7ff1e806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:52.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.413+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff200089130 con 0x7ff2100cf7d0 2024-01-16T15:22:52.408 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.413+0000 7ff214a00640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff1e806c550 0x7ff1e806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.414+0000 7ff214a00640 1 --2- 172.21.15.125:0/1324678092 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff1e806c550 0x7ff1e806ea10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff2100b0010 tx=0x7ff1f80079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:52.412 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.417+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff200055670 con 0x7ff2100cf7d0 2024-01-16T15:22:52.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.648+0000 7ff217e00640 1 -- 172.21.15.125:0/1324678092 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2100ceaa0 con 0x7ff1e806c550 2024-01-16T15:22:52.657 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:52.661+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:52.662+0000 7ff20d600640 1 -- 172.21.15.125:0/1324678092 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff2100ceaa0 con 0x7ff1e806c550 2024-01-16T15:22:52.657 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:54.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.905+0000 7f796fa00640 1 Processor -- start 2024-01-16T15:22:54.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.905+0000 7f796fa00640 1 -- start start 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.906+0000 7f796fa00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7968143ae0 0x7f79680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.906+0000 7f796fa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f79680cf7d0 0x7f796815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.906+0000 7f796fa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680cfc30 con 0x7f796815e1e0 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.906+0000 7f796fa00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680b1ee0 con 0x7f7968143ae0 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.906+0000 7f796fa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680b2780 con 0x7f79680cf7d0 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7968143ae0 0x7f79680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7968143ae0 0x7f79680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53264/0 (socket says 172.21.15.125:53264) 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 -- 172.21.15.125:0/2567034659 learned_addr learned my addr 172.21.15.125:0/2567034659 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 -- 172.21.15.125:0/2567034659 >> v2:172.21.15.137:3300/0 conn(0x7f79680cf7d0 msgr2=0x7f796815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 --2- 172.21.15.125:0/2567034659 >> v2:172.21.15.137:3300/0 conn(0x7f79680cf7d0 0x7f796815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 -- 172.21.15.125:0/2567034659 >> v1:172.21.15.137:6789/0 conn(0x7f796815e1e0 legacy=0x7f796815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.907+0000 7f796d000640 1 -- 172.21.15.125:0/2567034659 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796815e990 con 0x7f7968143ae0 2024-01-16T15:22:54.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.908+0000 7f796d000640 1 --2- 172.21.15.125:0/2567034659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7968143ae0 0x7f79680cf290 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f79680cb830 tx=0x7f795802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519a5dfd94b93e14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f7967e00640 1 -- 172.21.15.125:0/2567034659 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958031870 con 0x7f7968143ae0 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f7967e00640 1 -- 172.21.15.125:0/2567034659 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79580319d0 con 0x7f7968143ae0 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f7967e00640 1 -- 172.21.15.125:0/2567034659 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958031d60 con 0x7f7968143ae0 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 -- 172.21.15.125:0/2567034659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7968143ae0 msgr2=0x7f79680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 --2- 172.21.15.125:0/2567034659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7968143ae0 0x7f79680cf290 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f79680cb830 tx=0x7f795802b910 comp rx=0 tx=0).stop 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 -- 172.21.15.125:0/2567034659 shutdown_connections 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 --2- 172.21.15.125:0/2567034659 >> v2:172.21.15.137:3300/0 conn(0x7f79680cf7d0 0x7f796815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 --2- 172.21.15.125:0/2567034659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7968143ae0 0x7f79680cf290 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.909+0000 7f796fa00640 1 -- 172.21.15.125:0/2567034659 >> 172.21.15.125:0/2567034659 conn(0x7f796806a800 msgr2=0x7f796806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.910+0000 7f796fa00640 1 -- 172.21.15.125:0/2567034659 shutdown_connections 2024-01-16T15:22:54.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.910+0000 7f796fa00640 1 -- 172.21.15.125:0/2567034659 wait complete. 2024-01-16T15:22:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.911+0000 7f796fa00640 1 Processor -- start 2024-01-16T15:22:54.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.911+0000 7f796fa00640 1 -- start start 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79680cf7d0 0x7f7968142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7968143ae0 0x7f79680afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f796815e1e0 0x7f79680b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796813b5c0 con 0x7f79680cf7d0 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796813b310 con 0x7f796815e1e0 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796813be40 con 0x7f7968143ae0 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.912+0000 7f796d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79680cf7d0 0x7f7968142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:54.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79680cf7d0 0x7f7968142f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53274/0 (socket says 172.21.15.125:53274) 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 -- 172.21.15.125:0/2187036316 learned_addr learned my addr 172.21.15.125:0/2187036316 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 -- 172.21.15.125:0/2187036316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7968143ae0 msgr2=0x7f79680afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7968143ae0 0x7f79680afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 -- 172.21.15.125:0/2187036316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f796815e1e0 msgr2=0x7f79680b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f796815e1e0 0x7f79680b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 -- 172.21.15.125:0/2187036316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580094d0 con 0x7f79680cf7d0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f796d000640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79680cf7d0 0x7f7968142f50 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f7958000c00 tx=0x7f79580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.913+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795803ad10 con 0x7f79680cf7d0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.914+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795803ae90 con 0x7f79680cf7d0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.914+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958044630 con 0x7f79680cf7d0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.914+0000 7f796fa00640 1 -- 172.21.15.125:0/2187036316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7968140180 con 0x7f79680cf7d0 2024-01-16T15:22:54.910 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.914+0000 7f796fa00640 1 -- 172.21.15.125:0/2187036316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796813b7f0 con 0x7f79680cf7d0 2024-01-16T15:22:54.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.915+0000 7f796fa00640 1 -- 172.21.15.125:0/2187036316 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796806dcf0 con 0x7f79680cf7d0 2024-01-16T15:22:54.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.916+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f795804d450 con 0x7f79680cf7d0 2024-01-16T15:22:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.917+0000 7f7965600640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f794006c580 0x7f794006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.917+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f795802d040 con 0x7f79680cf7d0 2024-01-16T15:22:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.917+0000 7f795fc00640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f794006c580 0x7f794006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:54.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.918+0000 7f795fc00640 1 --2- 172.21.15.125:0/2187036316 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f794006c580 0x7f794006ea40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f7968067ea0 tx=0x7f7950006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:54.922+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7958077830 con 0x7f79680cf7d0 2024-01-16T15:22:55.140 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:55.145+0000 7f796fa00640 1 -- 172.21.15.125:0/2187036316 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f796815e670 con 0x7f794006c580 2024-01-16T15:22:55.152 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:55.157+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:55.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:55.157+0000 7f7965600640 1 -- 172.21.15.125:0/2187036316 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7950009280 con 0x7f794006c580 2024-01-16T15:22:55.153 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:55.153 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:57.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:57.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.398+0000 7f42dc800640 1 Processor -- start 2024-01-16T15:22:57.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.398+0000 7f42dc800640 1 -- start start 2024-01-16T15:22:57.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42dc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d413e360 0x7f42d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42dc800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d415f9c0 0x7f42d40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42dc800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d415c5f0 con 0x7f42d413df80 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42dc800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d406c470 con 0x7f42d415f9c0 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42dc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d406c750 con 0x7f42d413e360 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42d9e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f42d413df80 0x7f42d415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45012/0 (socket says 172.21.15.125:45012) 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42d9e00640 1 -- 172.21.15.125:0/3862832723 learned_addr learned my addr 172.21.15.125:0/3862832723 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.400+0000 7f42da800640 1 --2- 172.21.15.125:0/3862832723 >> v2:172.21.15.125:3300/0 conn(0x7f42d415f9c0 0x7f42d40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.401+0000 7f42d9400640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d413e360 0x7f42d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.401+0000 7f42da800640 1 -- 172.21.15.125:0/3862832723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d413e360 msgr2=0x7f42d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:57.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.401+0000 7f42da800640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d413e360 0x7f42d40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.401+0000 7f42da800640 1 -- 172.21.15.125:0/3862832723 >> v1:172.21.15.125:6789/0 conn(0x7f42d413df80 legacy=0x7f42d415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:57.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.401+0000 7f42da800640 1 -- 172.21.15.125:0/3862832723 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d406ca30 con 0x7f42d415f9c0 2024-01-16T15:22:57.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.402+0000 7f42da800640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415f9c0 0x7f42d40cf7d0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009d90 tx=0x7f42cc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0c3a2f314fe32d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:57.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.402+0000 7f42d8a00640 1 -- 172.21.15.125:0/3862832723 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc02cc50 con 0x7f42d415f9c0 2024-01-16T15:22:57.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42d8a00640 1 -- 172.21.15.125:0/3862832723 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc02cdb0 con 0x7f42d415f9c0 2024-01-16T15:22:57.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42d8a00640 1 -- 172.21.15.125:0/3862832723 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc034660 con 0x7f42d415f9c0 2024-01-16T15:22:57.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 -- 172.21.15.125:0/3862832723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415f9c0 msgr2=0x7f42d40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415f9c0 0x7f42d40cf7d0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009d90 tx=0x7f42cc02bea0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 -- 172.21.15.125:0/3862832723 shutdown_connections 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415f9c0 0x7f42d40cf7d0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 --2- 172.21.15.125:0/3862832723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d413e360 0x7f42d40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 -- 172.21.15.125:0/3862832723 >> 172.21.15.125:0/3862832723 conn(0x7f42d406a800 msgr2=0x7f42d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 -- 172.21.15.125:0/3862832723 shutdown_connections 2024-01-16T15:22:57.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.403+0000 7f42dc800640 1 -- 172.21.15.125:0/3862832723 wait complete. 2024-01-16T15:22:57.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.405+0000 7f42dc800640 1 Processor -- start 2024-01-16T15:22:57.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.405+0000 7f42dc800640 1 -- start start 2024-01-16T15:22:57.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d413df80 0x7f42d4152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d413e360 0x7f42d413b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415f9c0 0x7f42d40afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d413ad30 con 0x7f42d413e360 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d413d370 con 0x7f42d413df80 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42dc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4152e80 con 0x7f42d415f9c0 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42d9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d413e360 0x7f42d413b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42d9400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d413e360 0x7f42d413b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41170/0 (socket says 172.21.15.125:41170) 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.406+0000 7f42d9400640 1 -- 172.21.15.125:0/1760368363 learned_addr learned my addr 172.21.15.125:0/1760368363 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42da800640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415f9c0 0x7f42d40afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9e00640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d413df80 0x7f42d4152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 -- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415f9c0 msgr2=0x7f42d40afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415f9c0 0x7f42d40afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 -- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d413df80 msgr2=0x7f42d4152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d413df80 0x7f42d4152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 -- 172.21.15.125:0/1760368363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc009a70 con 0x7f42d413e360 2024-01-16T15:22:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9e00640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d413df80 0x7f42d4152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:57.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d9400640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d413e360 0x7f42d413b150 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00d800 tx=0x7f42bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.407+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00b6c0 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.408+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc00b820 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.408+0000 7f42dc800640 1 -- 172.21.15.125:0/1760368363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d413a390 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.408+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00bbb0 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.408+0000 7f42dc800640 1 -- 172.21.15.125:0/1760368363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d413a950 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.409+0000 7f42dc800640 1 -- 172.21.15.125:0/1760368363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d40675d0 con 0x7f42d413e360 2024-01-16T15:22:57.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.410+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f42bc00bd10 con 0x7f42d413e360 2024-01-16T15:22:57.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.410+0000 7f42d2000640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42ac06c550 0x7f42ac06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:57.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.411+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42bc089130 con 0x7f42d413e360 2024-01-16T15:22:57.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.411+0000 7f42d9e00640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42ac06c550 0x7f42ac06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:57.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.411+0000 7f42d9e00640 1 --2- 172.21.15.125:0/1760368363 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f42ac06c550 0x7f42ac06ea10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005880 tx=0x7f42c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:57.410 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.415+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42bc055670 con 0x7f42d413e360 2024-01-16T15:22:57.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.640+0000 7f42dc800640 1 -- 172.21.15.125:0/1760368363 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42d406cff0 con 0x7f42ac06c550 2024-01-16T15:22:57.649 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:22:57.654+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:57.650 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:57.654+0000 7f42d2000640 1 -- 172.21.15.125:0/1760368363 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f42c4001c40 con 0x7f42ac06c550 2024-01-16T15:22:57.650 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:22:57.650 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:59.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:59.875 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.880+0000 7f17ad800640 1 Processor -- start 2024-01-16T15:22:59.876 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.880+0000 7f17ad800640 1 -- start start 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.881+0000 7f17ad800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f17a8143ae0 0x7f17a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.881+0000 7f17ad800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f17a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.881+0000 7f17ad800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80cfc30 con 0x7f17a80cf7d0 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.881+0000 7f17ad800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80b1ee0 con 0x7f17a8143ae0 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.881+0000 7f17ad800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b2780 con 0x7f17a815e3b0 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f179de00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f17a80cf7d0 0x7f17a815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45016/0 (socket says 172.21.15.125:45016) 2024-01-16T15:22:59.877 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f179de00640 1 -- 172.21.15.125:0/836531930 learned_addr learned my addr 172.21.15.125:0/836531930 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:22:59.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264868686 0 0) 0x7f17a80cfc30 con 0x7f17a80cf7d0 2024-01-16T15:22:59.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1788003520 con 0x7f17a80cf7d0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a7400640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f17a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6a00640 1 --2- 172.21.15.125:0/836531930 >> v2:172.21.15.137:3300/0 conn(0x7f17a8143ae0 0x7f17a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233870865 0 0) 0x7f1788003520 con 0x7f17a80cf7d0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 msgr2=0x7f17a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f17a80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 >> v2:172.21.15.137:3300/0 conn(0x7f17a8143ae0 msgr2=0x7f17a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 --2- 172.21.15.125:0/836531930 >> v2:172.21.15.137:3300/0 conn(0x7f17a8143ae0 0x7f17a80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a815e990 con 0x7f17a80cf7d0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f1790001670 con 0x7f17a80cf7d0 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.882+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 >> v1:172.21.15.125:6789/0 conn(0x7f17a80cf7d0 legacy=0x7f17a815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f1788003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1788003fa0 0x7f17880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f1788005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b1ee0 con 0x7f1788004d00 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b2780 con 0x7f1788003fa0 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178800dc90 con 0x7f17a815e3b0 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f1788005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a6a00640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f1788003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f179de00640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1788003fa0 0x7f17880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 -- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 msgr2=0x7f1788003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a815e3b0 0x7f1788003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 -- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1788003fa0 msgr2=0x7f17880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1788003fa0 0x7f17880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.883+0000 7f17a7400640 1 -- 172.21.15.125:0/836531930 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788003520 con 0x7f1788004d00 2024-01-16T15:22:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.884+0000 7f17a7400640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f1788005180 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f17a0007360 tx=0x7f17880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a151a9456ecf6d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a002ccb0 con 0x7f1788004d00 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a002ce10 con 0x7f1788004d00 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17a6000640 1 -- 172.21.15.125:0/836531930 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0034660 con 0x7f1788004d00 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17ad800640 1 -- 172.21.15.125:0/836531930 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 msgr2=0x7f1788005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17ad800640 1 --2- 172.21.15.125:0/836531930 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f1788005180 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f17a0007360 tx=0x7f17880057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17a7400640 1 -- 172.21.15.125:0/836531930 reap_dead start 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17ad800640 1 -- 172.21.15.125:0/836531930 shutdown_connections 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17ad800640 1 -- 172.21.15.125:0/836531930 >> 172.21.15.125:0/836531930 conn(0x7f17a806a800 msgr2=0x7f17a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.885+0000 7f17ad800640 1 -- 172.21.15.125:0/836531930 shutdown_connections 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.886+0000 7f17ad800640 1 -- 172.21.15.125:0/836531930 wait complete. 2024-01-16T15:22:59.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.887+0000 7f17ad800640 1 Processor -- start 2024-01-16T15:22:59.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.887+0000 7f17ad800640 1 -- start start 2024-01-16T15:22:59.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f17a8141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a80adaf0 0x7f17a80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a80adf70 0x7f17a80ab830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80a9340 con 0x7f1788004d00 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80a9620 con 0x7f17a80adaf0 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.888+0000 7f17ad800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80a9900 con 0x7f17a80adf70 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f17a8141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f17a8141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41182/0 (socket says 172.21.15.125:41182) 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 -- 172.21.15.125:0/4113737881 learned_addr learned my addr 172.21.15.125:0/4113737881 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 -- 172.21.15.125:0/4113737881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a80adf70 msgr2=0x7f17a80ab830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a80adf70 0x7f17a80ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 -- 172.21.15.125:0/4113737881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a80adaf0 msgr2=0x7f17a80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a80adaf0 0x7f17a80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 -- 172.21.15.125:0/4113737881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80a9be0 con 0x7f1788004d00 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.889+0000 7f17a6a00640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1788004d00 0x7f17a8141df0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f179800f600 tx=0x7f179800fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.890+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798014870 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.890+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17980149d0 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.890+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798014d40 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.890+0000 7f17ad800640 1 -- 172.21.15.125:0/4113737881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a8142330 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.890+0000 7f17ad800640 1 -- 172.21.15.125:0/4113737881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a80ca710 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.891+0000 7f17ad800640 1 -- 172.21.15.125:0/4113737881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80675d0 con 0x7f1788004d00 2024-01-16T15:22:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.892+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1798015410 con 0x7f1788004d00 2024-01-16T15:22:59.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.893+0000 7f179f400640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f178006c550 0x7f178006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.893+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1798089230 con 0x7f1788004d00 2024-01-16T15:22:59.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.893+0000 7f179de00640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f178006c550 0x7f178006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.894+0000 7f179de00640 1 --2- 172.21.15.125:0/4113737881 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f178006c550 0x7f178006ea10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1790001eb0 tx=0x7f1790001d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:22:59.898+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1798055af0 con 0x7f1788004d00 2024-01-16T15:23:00.115 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:00.120+0000 7f17ad800640 1 -- 172.21.15.125:0/4113737881 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17a80b2410 con 0x7f178006c550 2024-01-16T15:23:00.125 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:00.130+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:00.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:00.130+0000 7f179f400640 1 -- 172.21.15.125:0/4113737881 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1790005050 con 0x7f178006c550 2024-01-16T15:23:00.126 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:00.126 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:02.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:02.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.369+0000 7fb317800640 1 Processor -- start 2024-01-16T15:23:02.365 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.369+0000 7fb317800640 1 -- start start 2024-01-16T15:23:02.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.370+0000 7fb317800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb31015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb317800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31006c470 con 0x7fb31013e360 2024-01-16T15:23:02.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb317800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31006c750 con 0x7fb3100cf9a0 2024-01-16T15:23:02.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb317800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31006ca30 con 0x7fb31013df80 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb30fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb31013e360 0x7fb3100cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45018/0 (socket says 172.21.15.125:45018) 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb30fe00640 1 -- 172.21.15.125:0/2700459372 learned_addr learned my addr 172.21.15.125:0/2700459372 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb314e00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb31015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162073148 0 0) 0x7fb31006c470 con 0x7fb31013e360 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.371+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f0003520 con 0x7fb31013e360 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479814039 0 0) 0x7fb2f0003520 con 0x7fb31013e360 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 msgr2=0x7fb31015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:02.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb31015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 >> v1:172.21.15.137:6789/0 conn(0x7fb3100cf9a0 legacy=0x7fb3100cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31006cd10 con 0x7fb31013e360 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fb2f8001670 con 0x7fb31013e360 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 >> v1:172.21.15.125:6789/0 conn(0x7fb31013e360 legacy=0x7fb3100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2f00039a0 0x7fb2f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb2f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0004d00 0x7fb2f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31006ca30 con 0x7fb2f00039a0 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f000ba60 con 0x7fb2f0004d00 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f000bc20 con 0x7fb31013df80 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.372+0000 7fb314e00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2f00039a0 0x7fb2f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb314e00640 1 -- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 msgr2=0x7fb2f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb30fe00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb2f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:02.369 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb314e00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb31013df80 0x7fb2f00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb314e00640 1 -- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0004d00 msgr2=0x7fb2f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:02.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb314e00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0004d00 0x7fb2f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.370 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.373+0000 7fb314e00640 1 -- 172.21.15.125:0/2700459372 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0003520 con 0x7fb2f00039a0 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.374+0000 7fb314e00640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2f00039a0 0x7fb2f0003dc0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fb300009a90 tx=0x7fb2f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5139e2c619c83926 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.374+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30002c680 con 0x7fb2f00039a0 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.374+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30002c7e0 con 0x7fb2f00039a0 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.374+0000 7fb30f400640 1 -- 172.21.15.125:0/2700459372 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30002cb70 con 0x7fb2f00039a0 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 -- 172.21.15.125:0/2700459372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2f00039a0 msgr2=0x7fb2f0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 --2- 172.21.15.125:0/2700459372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2f00039a0 0x7fb2f0003dc0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fb300009a90 tx=0x7fb2f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb315800640 1 -- 172.21.15.125:0/2700459372 reap_dead start 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 -- 172.21.15.125:0/2700459372 shutdown_connections 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 -- 172.21.15.125:0/2700459372 >> 172.21.15.125:0/2700459372 conn(0x7fb31006a800 msgr2=0x7fb31006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 -- 172.21.15.125:0/2700459372 shutdown_connections 2024-01-16T15:23:02.371 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.375+0000 7fb317800640 1 -- 172.21.15.125:0/2700459372 wait complete. 2024-01-16T15:23:02.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.376+0000 7fb317800640 1 Processor -- start 2024-01-16T15:23:02.372 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 -- start start 2024-01-16T15:23:02.373 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00039a0 0x7fb3100b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3100c9bf0 0x7fb3100cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3100cbb10 0x7fb3100c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100c4e20 con 0x7fb3100c9bf0 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.377+0000 7fb317800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100cbfc0 con 0x7fb3100cbb10 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb317800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31014bd70 con 0x7fb2f00039a0 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3100c9bf0 0x7fb3100cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3100c9bf0 0x7fb3100cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41198/0 (socket says 172.21.15.125:41198) 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 -- 172.21.15.125:0/1166080503 learned_addr learned my addr 172.21.15.125:0/1166080503 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 -- 172.21.15.125:0/1166080503 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00039a0 msgr2=0x7fb3100b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00039a0 0x7fb3100b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 -- 172.21.15.125:0/1166080503 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3100cbb10 msgr2=0x7fb3100c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3100cbb10 0x7fb3100c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:02.374 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.378+0000 7fb30fe00640 1 -- 172.21.15.125:0/1166080503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31014c050 con 0x7fb3100c9bf0 2024-01-16T15:23:02.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb30fe00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3100c9bf0 0x7fb3100cb5d0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800d810 tx=0x7fb2f800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:02.375 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80074e0 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8007640 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8004a00 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb317800640 1 -- 172.21.15.125:0/1166080503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31014c330 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.379+0000 7fb317800640 1 -- 172.21.15.125:0/1166080503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3100b34a0 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.380+0000 7fb317800640 1 -- 172.21.15.125:0/1166080503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31014c850 con 0x7fb3100c9bf0 2024-01-16T15:23:02.376 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.381+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb2f8004b60 con 0x7fb3100c9bf0 2024-01-16T15:23:02.377 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.382+0000 7fb30cc00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2e806c5a0 0x7fb2e806ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:02.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.382+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2f8088860 con 0x7fb3100c9bf0 2024-01-16T15:23:02.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.382+0000 7fb314e00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2e806c5a0 0x7fb2e806ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:02.378 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.383+0000 7fb314e00640 1 --2- 172.21.15.125:0/1166080503 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb2e806c5a0 0x7fb2e806ea60 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb300036000 tx=0x7fb30002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:02.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.387+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb31014c850 con 0x7fb3100c9bf0 2024-01-16T15:23:02.601 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.606+0000 7fb317800640 1 -- 172.21.15.125:0/1166080503 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3100cc390 con 0x7fb2e806c5a0 2024-01-16T15:23:02.609 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:02.614+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:02.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:02.614+0000 7fb30cc00640 1 -- 172.21.15.125:0/1166080503 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb3100cc390 con 0x7fb2e806c5a0 2024-01-16T15:23:02.610 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:02.610 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:04.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.850+0000 7fd52c400640 1 Processor -- start 2024-01-16T15:23:04.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.850+0000 7fd52c400640 1 -- start start 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.852+0000 7fd52c400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd5240b3ae0 0x7fd52415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.852+0000 7fd52c400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 0x7fd5240cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.852+0000 7fd52c400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd524070c40 con 0x7fd5240ae5a0 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.852+0000 7fd52c400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd524071450 con 0x7fd5240b3ae0 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.852+0000 7fd52c400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240cfc60 con 0x7fd5240cf290 2024-01-16T15:23:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd52a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 0x7fd5240cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd529000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd5240b3ae0 0x7fd52415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd529a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fd5240ae5a0 0x7fd52406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:38110/0 (socket says 172.21.15.125:38110) 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd529a00640 1 -- 172.21.15.125:0/436924627 learned_addr learned my addr 172.21.15.125:0/436924627 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd52a400640 1 -- 172.21.15.125:0/436924627 >> v2:172.21.15.137:3300/0 conn(0x7fd5240b3ae0 msgr2=0x7fd52415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd52a400640 1 --2- 172.21.15.125:0/436924627 >> v2:172.21.15.137:3300/0 conn(0x7fd5240b3ae0 0x7fd52415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd523e00640 1 -- 172.21.15.125:0/436924627 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374770778 0 0) 0x7fd524070c40 con 0x7fd5240ae5a0 2024-01-16T15:23:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd52a400640 1 -- 172.21.15.125:0/436924627 >> v1:172.21.15.137:6789/0 conn(0x7fd5240ae5a0 legacy=0x7fd52406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.853+0000 7fd52a400640 1 -- 172.21.15.125:0/436924627 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52406c470 con 0x7fd5240cf290 2024-01-16T15:23:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.855+0000 7fd52a400640 1 --2- 172.21.15.125:0/436924627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 0x7fd5240cf720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009d90 tx=0x7fd51c02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e709c8f0cc6d35 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.855+0000 7fd523e00640 1 -- 172.21.15.125:0/436924627 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c02cce0 con 0x7fd5240cf290 2024-01-16T15:23:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.855+0000 7fd523e00640 1 -- 172.21.15.125:0/436924627 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c02ce40 con 0x7fd5240cf290 2024-01-16T15:23:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.855+0000 7fd523e00640 1 -- 172.21.15.125:0/436924627 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c034660 con 0x7fd5240cf290 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 -- 172.21.15.125:0/436924627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 msgr2=0x7fd5240cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 --2- 172.21.15.125:0/436924627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 0x7fd5240cf720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009d90 tx=0x7fd51c02bf30 comp rx=0 tx=0).stop 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 -- 172.21.15.125:0/436924627 shutdown_connections 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 --2- 172.21.15.125:0/436924627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240cf290 0x7fd5240cf720 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 --2- 172.21.15.125:0/436924627 >> v2:172.21.15.137:3300/0 conn(0x7fd5240b3ae0 0x7fd52415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 -- 172.21.15.125:0/436924627 >> 172.21.15.125:0/436924627 conn(0x7fd52406a800 msgr2=0x7fd52406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 -- 172.21.15.125:0/436924627 shutdown_connections 2024-01-16T15:23:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.856+0000 7fd52c400640 1 -- 172.21.15.125:0/436924627 wait complete. 2024-01-16T15:23:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.858+0000 7fd52c400640 1 Processor -- start 2024-01-16T15:23:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.858+0000 7fd52c400640 1 -- start start 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5240ae5a0 0x7fd5241576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240b3ae0 0x7fd5241f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5240cf290 0x7fd5241f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5241f7210 con 0x7fd5240cf290 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd524151180 con 0x7fd5240ae5a0 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52c400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd524157de0 con 0x7fd5240b3ae0 2024-01-16T15:23:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5240cf290 0x7fd5241f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52a400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5240cf290 0x7fd5241f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41204/0 (socket says 172.21.15.125:41204) 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd52a400640 1 -- 172.21.15.125:0/2202561844 learned_addr learned my addr 172.21.15.125:0/2202561844 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.859+0000 7fd529a00640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5240ae5a0 0x7fd5241576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd529000640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240b3ae0 0x7fd5241f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 -- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240b3ae0 msgr2=0x7fd5241f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5240b3ae0 0x7fd5241f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 -- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5240ae5a0 msgr2=0x7fd5241576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5240ae5a0 0x7fd5241576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 -- 172.21.15.125:0/2202561844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c009a70 con 0x7fd5240cf290 2024-01-16T15:23:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52a400640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5240cf290 0x7fd5241f26d0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009d90 tx=0x7fd51c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c03d600 con 0x7fd5240cf290 2024-01-16T15:23:04.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c03d780 con 0x7fd5240cf290 2024-01-16T15:23:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd52c400640 1 -- 172.21.15.125:0/2202561844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5241580c0 con 0x7fd5240cf290 2024-01-16T15:23:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.860+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c03db30 con 0x7fd5240cf290 2024-01-16T15:23:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.861+0000 7fd52c400640 1 -- 172.21.15.125:0/2202561844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd524158680 con 0x7fd5240cf290 2024-01-16T15:23:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.862+0000 7fd52c400640 1 -- 172.21.15.125:0/2202561844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5241512f0 con 0x7fd5240cf290 2024-01-16T15:23:04.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.863+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd51c03b040 con 0x7fd5240cf290 2024-01-16T15:23:04.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.863+0000 7fd521600640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd4fc06c550 0x7fd4fc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.864+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd51c0aada0 con 0x7fd5240cf290 2024-01-16T15:23:04.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.864+0000 7fd529a00640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd4fc06c550 0x7fd4fc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.864+0000 7fd529a00640 1 --2- 172.21.15.125:0/2202561844 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd4fc06c550 0x7fd4fc06ea10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd514000f60 tx=0x7fd514007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:04.868+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd51c077660 con 0x7fd5240cf290 2024-01-16T15:23:05.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:05.090+0000 7fd52c400640 1 -- 172.21.15.125:0/2202561844 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd52415a630 con 0x7fd4fc06c550 2024-01-16T15:23:05.098 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:05.103+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:05.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:05.103+0000 7fd521600640 1 -- 172.21.15.125:0/2202561844 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd52415a630 con 0x7fd4fc06c550 2024-01-16T15:23:05.099 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:05.099 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:07.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:07.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.352+0000 7f8bf0a00640 1 Processor -- start 2024-01-16T15:23:07.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.352+0000 7f8bf0a00640 1 -- start start 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.353+0000 7f8bf0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.353+0000 7f8bf0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8be80ced60 0x7f8be80d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bf0a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be80b3400 con 0x7f8be80d1d20 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bf0a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be80d1390 con 0x7f8be80ced60 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bf0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80d21b0 con 0x7f8be815d610 2024-01-16T15:23:07.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8be80ced60 0x7f8be80d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8be80ced60 0x7f8be80d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60152/0 (socket says 172.21.15.125:60152) 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 -- 172.21.15.125:0/2609572748 learned_addr learned my addr 172.21.15.125:0/2609572748 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 -- 172.21.15.125:0/2609572748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 msgr2=0x7f8be80ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bee000640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 -- 172.21.15.125:0/2609572748 >> v1:172.21.15.125:6789/0 conn(0x7f8be80d1d20 legacy=0x7f8be80cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bed600640 1 -- 172.21.15.125:0/2609572748 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80b27f0 con 0x7f8be80ced60 2024-01-16T15:23:07.351 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.354+0000 7f8bee000640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.356+0000 7f8bed600640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80ced60 0x7f8be80d17e0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009870 tx=0x7f8bd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef1e43879b6ac4a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.356+0000 7f8becc00640 1 -- 172.21.15.125:0/2609572748 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0030840 con 0x7f8be80ced60 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.356+0000 7f8becc00640 1 -- 172.21.15.125:0/2609572748 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd00309a0 con 0x7f8be80ced60 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.356+0000 7f8becc00640 1 -- 172.21.15.125:0/2609572748 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0030d30 con 0x7f8be80ced60 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 -- 172.21.15.125:0/2609572748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80ced60 msgr2=0x7f8be80d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80ced60 0x7f8be80d17e0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009870 tx=0x7f8bd002b980 comp rx=0 tx=0).stop 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 -- 172.21.15.125:0/2609572748 shutdown_connections 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80ced60 0x7f8be80d17e0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 --2- 172.21.15.125:0/2609572748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 -- 172.21.15.125:0/2609572748 >> 172.21.15.125:0/2609572748 conn(0x7f8be809c540 msgr2=0x7f8be80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 -- 172.21.15.125:0/2609572748 shutdown_connections 2024-01-16T15:23:07.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.357+0000 7f8bf0a00640 1 -- 172.21.15.125:0/2609572748 wait complete. 2024-01-16T15:23:07.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.359+0000 7f8bf0a00640 1 Processor -- start 2024-01-16T15:23:07.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.359+0000 7f8bf0a00640 1 -- start start 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8be80ced60 0x7f8be80c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80d1d20 0x7f8be80cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80b9070 con 0x7f8be80d1d20 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80ca850 con 0x7f8be80ced60 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bf0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80cab30 con 0x7f8be815d610 2024-01-16T15:23:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80d1d20 0x7f8be80cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80d1d20 0x7f8be80cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60158/0 (socket says 172.21.15.125:60158) 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 -- 172.21.15.125:0/3195165820 learned_addr learned my addr 172.21.15.125:0/3195165820 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8beea00640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bee000640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8be80ced60 0x7f8be80c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 -- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 msgr2=0x7f8be80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8be815d610 0x7f8be80cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 -- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8be80ced60 msgr2=0x7f8be80c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8be80ced60 0x7f8be80c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bed600640 1 -- 172.21.15.125:0/3195165820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd00094d0 con 0x7f8be80d1d20 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.360+0000 7f8bee000640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8be80ced60 0x7f8be80c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:07.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8bed600640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8be80d1d20 0x7f8be80cb8f0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009870 tx=0x7f8bd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0039d60 con 0x7f8be80d1d20 2024-01-16T15:23:07.358 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0043400 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd00437b0 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8bf0a00640 1 -- 172.21.15.125:0/3195165820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be80cae10 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.361+0000 7f8bf0a00640 1 -- 172.21.15.125:0/3195165820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be80cb3d0 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.362+0000 7f8bf0a00640 1 -- 172.21.15.125:0/3195165820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be80b3770 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.363+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f8bd0043910 con 0x7f8be80d1d20 2024-01-16T15:23:07.359 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.364+0000 7f8be6000640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bc006c550 0x7f8bc006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.364+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8bd003e070 con 0x7f8be80d1d20 2024-01-16T15:23:07.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.364+0000 7f8bee000640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bc006c550 0x7f8bc006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.365+0000 7f8bee000640 1 --2- 172.21.15.125:0/3195165820 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8bc006c550 0x7f8bc006ea10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8006fd0 tx=0x7f8bd8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.364 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.369+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bd00782e0 con 0x7f8be80d1d20 2024-01-16T15:23:07.585 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.589+0000 7f8bf0a00640 1 -- 172.21.15.125:0/3195165820 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8be80b20f0 con 0x7f8bc006c550 2024-01-16T15:23:07.595 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:07.600+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:07.601+0000 7f8be6000640 1 -- 172.21.15.125:0/3195165820 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8be80b20f0 con 0x7f8bc006c550 2024-01-16T15:23:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:07.597 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:09.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:09.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.845+0000 7fb8ff200640 1 Processor -- start 2024-01-16T15:23:09.840 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.845+0000 7fb8ff200640 1 -- start start 2024-01-16T15:23:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.846+0000 7fb8ff200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb9001447f0 0x7fb90015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.846+0000 7fb8ff200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.846+0000 7fb8ff200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90015c5c0 con 0x7fb90006dab0 2024-01-16T15:23:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.846+0000 7fb8ff200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90006d690 con 0x7fb9001447f0 2024-01-16T15:23:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8ff200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000cebb0 con 0x7fb90006c470 2024-01-16T15:23:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8f7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8f7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:43720/0 (socket says 172.21.15.125:43720) 2024-01-16T15:23:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8fe800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb90006dab0 0x7fb90015c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:40992/0 (socket says 172.21.15.125:40992) 2024-01-16T15:23:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8fde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb9001447f0 0x7fb90015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.847+0000 7fb8f7e00640 1 -- 172.21.15.125:0/2364182085 learned_addr learned my addr 172.21.15.125:0/2364182085 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.848+0000 7fb8f7e00640 1 -- 172.21.15.125:0/2364182085 >> v2:172.21.15.137:3300/0 conn(0x7fb9001447f0 msgr2=0x7fb90015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.848+0000 7fb8f7e00640 1 --2- 172.21.15.125:0/2364182085 >> v2:172.21.15.137:3300/0 conn(0x7fb9001447f0 0x7fb90015c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.848+0000 7fb8f7e00640 1 -- 172.21.15.125:0/2364182085 >> v1:172.21.15.137:6789/0 conn(0x7fb90006dab0 legacy=0x7fb90015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.848+0000 7fb8f7e00640 1 -- 172.21.15.125:0/2364182085 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90006ca10 con 0x7fb90006c470 2024-01-16T15:23:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.848+0000 7fb8fd400640 1 -- 172.21.15.125:0/2364182085 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129116085 0 0) 0x7fb90015c5c0 con 0x7fb90006dab0 2024-01-16T15:23:09.845 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.850+0000 7fb8f7e00640 1 --2- 172.21.15.125:0/2364182085 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009870 tx=0x7fb8e802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bd3ab97bc24e56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.850+0000 7fb8fd400640 1 -- 172.21.15.125:0/2364182085 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8031640 con 0x7fb90006c470 2024-01-16T15:23:09.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.850+0000 7fb8fd400640 1 -- 172.21.15.125:0/2364182085 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e80317a0 con 0x7fb90006c470 2024-01-16T15:23:09.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.850+0000 7fb8fd400640 1 -- 172.21.15.125:0/2364182085 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8031b30 con 0x7fb90006c470 2024-01-16T15:23:09.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 -- 172.21.15.125:0/2364182085 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 msgr2=0x7fb9000cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 --2- 172.21.15.125:0/2364182085 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009870 tx=0x7fb8e802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 -- 172.21.15.125:0/2364182085 shutdown_connections 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 --2- 172.21.15.125:0/2364182085 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000cc5e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 --2- 172.21.15.125:0/2364182085 >> v2:172.21.15.137:3300/0 conn(0x7fb9001447f0 0x7fb90015c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 -- 172.21.15.125:0/2364182085 >> 172.21.15.125:0/2364182085 conn(0x7fb90006a3e0 msgr2=0x7fb90006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 -- 172.21.15.125:0/2364182085 shutdown_connections 2024-01-16T15:23:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.851+0000 7fb8ff200640 1 -- 172.21.15.125:0/2364182085 wait complete. 2024-01-16T15:23:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.853+0000 7fb8ff200640 1 Processor -- start 2024-01-16T15:23:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.853+0000 7fb8ff200640 1 -- start start 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb90006dab0 0x7fb90013ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9001447f0 0x7fb9000b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb900157140 con 0x7fb90006dab0 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000b0910 con 0x7fb9001447f0 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8ff200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb900139120 con 0x7fb90006c470 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8f7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb90006dab0 0x7fb90013ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8f7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb90006dab0 0x7fb90013ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60166/0 (socket says 172.21.15.125:60166) 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.854+0000 7fb8f7e00640 1 -- 172.21.15.125:0/1567814687 learned_addr learned my addr 172.21.15.125:0/1567814687 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8fde00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8fe800640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9001447f0 0x7fb9000b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 -- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 msgr2=0x7fb9000b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb90006c470 0x7fb9000b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 -- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9001447f0 msgr2=0x7fb9000b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9001447f0 0x7fb9000b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 -- 172.21.15.125:0/1567814687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e80094d0 con 0x7fb90006dab0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f7e00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb90006dab0 0x7fb90013ad70 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8034040 tx=0x7fb8e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.855+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8004230 con 0x7fb90006dab0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.856+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e80043b0 con 0x7fb90006dab0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.856+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8004520 con 0x7fb90006dab0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.856+0000 7fb8ff200640 1 -- 172.21.15.125:0/1567814687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb900139400 con 0x7fb90006dab0 2024-01-16T15:23:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.856+0000 7fb8ff200640 1 -- 172.21.15.125:0/1567814687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9001399c0 con 0x7fb90006dab0 2024-01-16T15:23:09.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.857+0000 7fb8ff200640 1 -- 172.21.15.125:0/1567814687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900139bb0 con 0x7fb90006dab0 2024-01-16T15:23:09.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.858+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb8e8042070 con 0x7fb90006dab0 2024-01-16T15:23:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.859+0000 7fb8f6000640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8d806c550 0x7fb8d806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.859+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8e80aa490 con 0x7fb90006dab0 2024-01-16T15:23:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.859+0000 7fb8fde00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8d806c550 0x7fb8d806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.859+0000 7fb8fde00640 1 --2- 172.21.15.125:0/1567814687 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fb8d806c550 0x7fb8d806ea10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009b40 tx=0x7fb8f0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:09.863+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8e8076d50 con 0x7fb90006dab0 2024-01-16T15:23:10.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:10.090+0000 7fb8ff200640 1 -- 172.21.15.125:0/1567814687 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb90006d2b0 con 0x7fb8d806c550 2024-01-16T15:23:10.096 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:10.101+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:10.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:10.101+0000 7fb8f6000640 1 -- 172.21.15.125:0/1567814687 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb8f0011a60 con 0x7fb8d806c550 2024-01-16T15:23:10.097 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:10.097 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:12.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:12.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.346+0000 7fc9a0600640 1 Processor -- start 2024-01-16T15:23:12.342 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.347+0000 7fc9a0600640 1 -- start start 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc9a0600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc9980d0ac0 0x7fc9980b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc9a0600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9980cea50 con 0x7fc998144d00 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc9a0600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9980cd180 con 0x7fc9980d0610 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc9a0600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9980d0ed0 con 0x7fc9980d0ac0 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc99dc00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc998144d00 0x7fc9980ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48330/0 (socket says 172.21.15.125:48330) 2024-01-16T15:23:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.348+0000 7fc99dc00640 1 -- 172.21.15.125:0/3276800054 learned_addr learned my addr 172.21.15.125:0/3276800054 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676750817 0 0) 0x7fc9980cea50 con 0x7fc998144d00 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc978003520 con 0x7fc998144d00 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc99e600640 1 --2- 172.21.15.125:0/3276800054 >> v2:172.21.15.137:3300/0 conn(0x7fc9980d0ac0 0x7fc9980b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690524058 0 0) 0x7fc978003520 con 0x7fc998144d00 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 >> v2:172.21.15.137:3300/0 conn(0x7fc9980d0ac0 msgr2=0x7fc9980b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 --2- 172.21.15.125:0/3276800054 >> v2:172.21.15.137:3300/0 conn(0x7fc9980d0ac0 0x7fc9980b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 >> v1:172.21.15.137:6789/0 conn(0x7fc9980d0610 legacy=0x7fc9980cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9980cd420 con 0x7fc998144d00 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.349+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc980001670 con 0x7fc998144d00 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 >> v1:172.21.15.125:6789/0 conn(0x7fc998144d00 legacy=0x7fc9980ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9780039c0 0x7fc978003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 0x7fc978004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780051d0 0x7fc97800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980d0ed0 con 0x7fc9780047c0 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97800c120 con 0x7fc9780051d0 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97800c2e0 con 0x7fc9780039c0 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 0x7fc978004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99e600640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780051d0 0x7fc97800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 -- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9780039c0 msgr2=0x7fc978003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9780039c0 0x7fc978003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 -- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780051d0 msgr2=0x7fc97800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780051d0 0x7fc97800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99e600640 1 -- 172.21.15.125:0/3276800054 reap_dead start 2024-01-16T15:23:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.350+0000 7fc99d200640 1 -- 172.21.15.125:0/3276800054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978003520 con 0x7fc9780047c0 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.351+0000 7fc99d200640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 0x7fc978004c40 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fc988001e50 tx=0x7fc9780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c520493980d0f1d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988003b70 con 0x7fc9780047c0 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc988003cd0 con 0x7fc9780047c0 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc997e00640 1 -- 172.21.15.125:0/3276800054 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988005310 con 0x7fc9780047c0 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 -- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 msgr2=0x7fc978004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 0x7fc978004c40 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fc988001e50 tx=0x7fc9780057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 -- 172.21.15.125:0/3276800054 shutdown_connections 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 --2- 172.21.15.125:0/3276800054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9780047c0 0x7fc978004c40 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 -- 172.21.15.125:0/3276800054 >> 172.21.15.125:0/3276800054 conn(0x7fc998066690 msgr2=0x7fc99806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 -- 172.21.15.125:0/3276800054 shutdown_connections 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.352+0000 7fc9a0600640 1 -- 172.21.15.125:0/3276800054 wait complete. 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.354+0000 7fc9a0600640 1 Processor -- start 2024-01-16T15:23:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.354+0000 7fc9a0600640 1 -- start start 2024-01-16T15:23:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780047c0 0x7fc998141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9980c7a30 0x7fc99815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9980c8140 0x7fc9980c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980c8b00 con 0x7fc9980c8140 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc998142080 con 0x7fc9780047c0 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.355+0000 7fc9a0600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99815ce20 con 0x7fc9980c7a30 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9980c8140 0x7fc9980c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9980c8140 0x7fc9980c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60172/0 (socket says 172.21.15.125:60172) 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 -- 172.21.15.125:0/81046781 learned_addr learned my addr 172.21.15.125:0/81046781 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99dc00640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780047c0 0x7fc998141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99d200640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9980c7a30 0x7fc99815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 -- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9980c7a30 msgr2=0x7fc99815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9980c7a30 0x7fc99815c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 -- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780047c0 msgr2=0x7fc998141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780047c0 0x7fc998141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 -- 172.21.15.125:0/81046781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99815d0f0 con 0x7fc9980c8140 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99dc00640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9780047c0 0x7fc998141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.356+0000 7fc99e600640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9980c8140 0x7fc9980c85c0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fc998155eb0 tx=0x7fc99000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.357+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990004230 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.357+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc990004390 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.357+0000 7fc9a0600640 1 -- 172.21.15.125:0/81046781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99815d3d0 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.357+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990003850 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.357+0000 7fc9a0600640 1 -- 172.21.15.125:0/81046781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99815d990 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.358+0000 7fc9a0600640 1 -- 172.21.15.125:0/81046781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9980ce870 con 0x7fc9980c8140 2024-01-16T15:23:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.359+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc9900039b0 con 0x7fc9980c8140 2024-01-16T15:23:12.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.360+0000 7fc995600640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc97006c580 0x7fc97006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.360+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc990088130 con 0x7fc9980c8140 2024-01-16T15:23:12.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.360+0000 7fc99dc00640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc97006c580 0x7fc97006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.361+0000 7fc99dc00640 1 --2- 172.21.15.125:0/81046781 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc97006c580 0x7fc97006ea40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc998067b00 tx=0x7fc980009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:12.360 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.364+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9900546d0 con 0x7fc9980c8140 2024-01-16T15:23:12.584 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.589+0000 7fc9a0600640 1 -- 172.21.15.125:0/81046781 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9980cf2c0 con 0x7fc97006c580 2024-01-16T15:23:12.600 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:23:12.604+0000 7f00fb000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:12.600 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:12.604+0000 7fc995600640 1 -- 172.21.15.125:0/81046781 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc9800070d0 con 0x7fc97006c580 2024-01-16T15:23:12.600 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:12.601 INFO:teuthology.orchestra.run.smithi125.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:12.721 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for ... ERROR 2024-01-16T15:23:12.721 INFO:tasks.cephfs_test_runner:ERROR 2024-01-16T15:23:12.721 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:23:12.722 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75496' 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:12.723 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1546.780s 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:12.724 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:23:12.725 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone75496' 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:12.726 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:12.727 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.974 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4825fb26b30e410f91a1aef68391b4fa Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:12.975 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-16T15:23:12.987 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-16T15:23:13.001 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-16T15:23:13.002 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:23:13.002 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:13.071 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:23:13.071 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:13.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:13.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.337+0000 7f983e400640 1 Processor -- start 2024-01-16T15:23:13.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.337+0000 7f983e400640 1 -- start start 2024-01-16T15:23:13.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f983e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f983813df80 0x7f983815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f983e400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983806c470 con 0x7f98380cf9a0 2024-01-16T15:23:13.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f983e400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983806c750 con 0x7f983813e360 2024-01-16T15:23:13.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f983e400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983806ca30 con 0x7f983813df80 2024-01-16T15:23:13.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f9837e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f98380cf9a0 0x7f98380cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48332/0 (socket says 172.21.15.125:48332) 2024-01-16T15:23:13.335 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f9837e00640 1 -- 172.21.15.125:0/1451239331 learned_addr learned my addr 172.21.15.125:0/1451239331 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.339+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892352498 0 0) 0x7f983806c470 con 0x7f98380cf9a0 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818003520 con 0x7f98380cf9a0 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9837400640 1 --2- 172.21.15.125:0/1451239331 >> v2:172.21.15.137:3300/0 conn(0x7f983813df80 0x7f983815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741064143 0 0) 0x7f9818003520 con 0x7f98380cf9a0 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 >> v2:172.21.15.137:3300/0 conn(0x7f983813df80 msgr2=0x7f983815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 --2- 172.21.15.125:0/1451239331 >> v2:172.21.15.137:3300/0 conn(0x7f983813df80 0x7f983815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 >> v1:172.21.15.137:6789/0 conn(0x7f983813e360 legacy=0x7f98380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983806cd10 con 0x7f98380cf9a0 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f98300019f0 con 0x7f98380cf9a0 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 >> v1:172.21.15.125:6789/0 conn(0x7f98380cf9a0 legacy=0x7f98380cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98180039c0 0x7f9818003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 0x7f9818004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180051d0 0x7f981800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983806ca30 con 0x7f98180047c0 2024-01-16T15:23:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981800c120 con 0x7f98180051d0 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981800c2e0 con 0x7f98180039c0 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f9837400640 1 --2- 172.21.15.125:0/1451239331 >> v2:172.21.15.137:3300/0 conn(0x7f983813df80 0x7f983815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.340+0000 7f982fe00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 0x7f9818004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f9837e00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180051d0 0x7f981800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f9837400640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98180039c0 0x7f9818003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f982fe00640 1 -- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98180039c0 msgr2=0x7f9818003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f982fe00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98180039c0 0x7f9818003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.339 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f982fe00640 1 -- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180051d0 msgr2=0x7f981800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f982fe00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180051d0 0x7f981800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f982fe00640 1 -- 172.21.15.125:0/1451239331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818003520 con 0x7f98180047c0 2024-01-16T15:23:13.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f9837e00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180051d0 0x7f981800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:13.340 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.341+0000 7f9837e00640 1 -- 172.21.15.125:0/1451239331 reap_dead start 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.342+0000 7f982fe00640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 0x7f9818004c40 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f9820001720 tx=0x7f98180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bf935459c72cc4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.342+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200052d0 con 0x7f98180047c0 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.342+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982002e050 con 0x7f98180047c0 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.342+0000 7f9836a00640 1 -- 172.21.15.125:0/1451239331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820003830 con 0x7f98180047c0 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 -- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 msgr2=0x7f9818004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 0x7f9818004c40 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f9820001720 tx=0x7f98180057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 -- 172.21.15.125:0/1451239331 shutdown_connections 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 --2- 172.21.15.125:0/1451239331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98180047c0 0x7f9818004c40 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 -- 172.21.15.125:0/1451239331 >> 172.21.15.125:0/1451239331 conn(0x7f983806a800 msgr2=0x7f983806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 -- 172.21.15.125:0/1451239331 shutdown_connections 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.343+0000 7f983e400640 1 -- 172.21.15.125:0/1451239331 wait complete. 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.345+0000 7f983e400640 1 Processor -- start 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.345+0000 7f983e400640 1 -- start start 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 0x7f983813d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9838158820 0x7f9838158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983815e260 con 0x7f983815d8a0 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98381592a0 con 0x7f98180047c0 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f983e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9838159580 con 0x7f9838158820 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60196/0 (socket says 172.21.15.125:60196) 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 -- 172.21.15.125:0/2036415021 learned_addr learned my addr 172.21.15.125:0/2036415021 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:13.345 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 0x7f983813d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 -- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9838158820 msgr2=0x7f9838158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9838158820 0x7f9838158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 -- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 msgr2=0x7f983813d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 0x7f983813d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837e00640 1 -- 172.21.15.125:0/2036415021 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838159860 con 0x7f983815d8a0 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.346+0000 7f9837400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 0x7f983813d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f9837e00640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f98380cfea0 tx=0x7f983000df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000b7e0 con 0x7f983815d8a0 2024-01-16T15:23:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983000b960 con 0x7f983815d8a0 2024-01-16T15:23:13.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000bd10 con 0x7f983815d8a0 2024-01-16T15:23:13.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9838159b40 con 0x7f983815d8a0 2024-01-16T15:23:13.347 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.347+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983813dd10 con 0x7f983815d8a0 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.348+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98380675d0 con 0x7f983815d8a0 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.349+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9830007070 con 0x7f983815d8a0 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.350+0000 7f982f400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 0x7f981006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.350+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9830088e70 con 0x7f983815d8a0 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.350+0000 7f9837400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 0x7f981006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.348 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.351+0000 7f9837400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 0x7f981006ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f982800ba40 tx=0x7f9828008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.350 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.354+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9830055320 con 0x7f983815d8a0 2024-01-16T15:23:13.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.550+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98380cec50 con 0x7f983815d8a0 2024-01-16T15:23:13.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.553+0000 7f982f400640 1 -- 172.21.15.125:0/2036415021 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f9830058940 con 0x7f983815d8a0 2024-01-16T15:23:13.549 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:13.549 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:28.491486+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 msgr2=0x7f981006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 0x7f981006ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f982800ba40 tx=0x7f9828008040 comp rx=0 tx=0).stop 2024-01-16T15:23:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 msgr2=0x7f983815dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f98380cfea0 tx=0x7f983000df60 comp rx=0 tx=0).stop 2024-01-16T15:23:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 shutdown_connections 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f981006c550 0x7f981006ea10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f983815d8a0 0x7f983815dd20 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9838158820 0x7f9838158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 --2- 172.21.15.125:0/2036415021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98180047c0 0x7f983813d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.555+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 >> 172.21.15.125:0/2036415021 conn(0x7f983806a800 msgr2=0x7f98380704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.556+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 shutdown_connections 2024-01-16T15:23:13.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.556+0000 7f983e400640 1 -- 172.21.15.125:0/2036415021 wait complete. 2024-01-16T15:23:13.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:04.806000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:06.026871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:13.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.784+0000 7fa209e00640 1 Processor -- start 2024-01-16T15:23:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.784+0000 7fa209e00640 1 -- start start 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa2040cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2040ce440 con 0x7fa2040cdc20 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2040ce720 con 0x7fa20413cf10 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040cea00 con 0x7fa20413fcb0 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa2040cdc20 0x7fa2040b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48348/0 (socket says 172.21.15.125:48348) 2024-01-16T15:23:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.786+0000 7fa209400640 1 -- 172.21.15.125:0/2914751502 learned_addr learned my addr 172.21.15.125:0/2914751502 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa2040cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799693693 0 0) 0x7fa2040ce440 con 0x7fa2040cdc20 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e4005730 con 0x7fa2040cdc20 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556159800 0 0) 0x7fa1e4005730 con 0x7fa2040cdc20 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 msgr2=0x7fa2040cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa2040cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 >> v1:172.21.15.137:6789/0 conn(0x7fa20413cf10 legacy=0x7fa20413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2040b1d90 con 0x7fa2040cdc20 2024-01-16T15:23:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fa1fc001670 con 0x7fa2040cdc20 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 >> v1:172.21.15.125:6789/0 conn(0x7fa2040cdc20 legacy=0x7fa2040b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1e4005bb0 0x7fa1e4005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa1e40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1e4006f10 0x7fa1e4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040cea00 con 0x7fa1e4005bb0 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e400dca0 con 0x7fa1e4006f10 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.787+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e400de60 con 0x7fa20413fcb0 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1e4005bb0 0x7fa1e4005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa1e40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa209400640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1e4006f10 0x7fa1e4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 -- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 msgr2=0x7fa1e40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa1e40069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 -- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1e4006f10 msgr2=0x7fa1e4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1e4006f10 0x7fa1e4007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa208a00640 1 -- 172.21.15.125:0/2914751502 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4005730 con 0x7fa1e4005bb0 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa20413fcb0 0x7fa1e40069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.788+0000 7fa209400640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1e4006f10 0x7fa1e4007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:13.788 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.789+0000 7fa208a00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1e4005bb0 0x7fa1e4005fd0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4001720 tx=0x7fa1e4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5519b26e3fbd7d55 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.789+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40052d0 con 0x7fa1e4005bb0 2024-01-16T15:23:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.789+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f402e050 con 0x7fa1e4005bb0 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.789+0000 7fa203e00640 1 -- 172.21.15.125:0/2914751502 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40035f0 con 0x7fa1e4005bb0 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 -- 172.21.15.125:0/2914751502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1e4005bb0 msgr2=0x7fa1e4005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 --2- 172.21.15.125:0/2914751502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1e4005bb0 0x7fa1e4005fd0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4001720 tx=0x7fa1e4007a00 comp rx=0 tx=0).stop 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209400640 1 -- 172.21.15.125:0/2914751502 reap_dead start 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 -- 172.21.15.125:0/2914751502 shutdown_connections 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 -- 172.21.15.125:0/2914751502 >> 172.21.15.125:0/2914751502 conn(0x7fa20406a7d0 msgr2=0x7fa20406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 -- 172.21.15.125:0/2914751502 shutdown_connections 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.790+0000 7fa209e00640 1 -- 172.21.15.125:0/2914751502 wait complete. 2024-01-16T15:23:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.792+0000 7fa209e00640 1 Processor -- start 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.792+0000 7fa209e00640 1 -- start start 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1e4005bb0 0x7fa2040ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa204151bb0 0x7fa204155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204070870 con 0x7fa204155d10 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040713f0 con 0x7fa204151bb0 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa209e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa204070ac0 con 0x7fa1e4005bb0 2024-01-16T15:23:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa1fbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa1fbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60214/0 (socket says 172.21.15.125:60214) 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.793+0000 7fa1fbc00640 1 -- 172.21.15.125:0/2595647448 learned_addr learned my addr 172.21.15.125:0/2595647448 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa208a00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1e4005bb0 0x7fa2040ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa209400640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa204151bb0 0x7fa204155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 -- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1e4005bb0 msgr2=0x7fa2040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1e4005bb0 0x7fa2040ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 -- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa204151bb0 msgr2=0x7fa204155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa204151bb0 0x7fa204155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 -- 172.21.15.125:0/2595647448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204070da0 con 0x7fa204155d10 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa1fbc00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00f380 tx=0x7fa1ec00f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec003b90 con 0x7fa204155d10 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec011040 con 0x7fa204155d10 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa204071080 con 0x7fa204155d10 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec004020 con 0x7fa204155d10 2024-01-16T15:23:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.794+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2040b0050 con 0x7fa204155d10 2024-01-16T15:23:13.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.795+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa204071580 con 0x7fa204155d10 2024-01-16T15:23:13.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.797+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fa1ec004180 con 0x7fa204155d10 2024-01-16T15:23:13.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.798+0000 7fa201600640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 0x7fa1dc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.798+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1ec0892d0 con 0x7fa204155d10 2024-01-16T15:23:13.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.798+0000 7fa208a00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 0x7fa1dc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.799+0000 7fa208a00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 0x7fa1dc06ea10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4033000 tx=0x7fa1f4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.802+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1ec055810 con 0x7fa204155d10 2024-01-16T15:23:13.988 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.992+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa204067450 con 0x7fa204155d10 2024-01-16T15:23:13.989 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.993+0000 7fa201600640 1 -- 172.21.15.125:0/2595647448 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1ec058e30 con 0x7fa204155d10 2024-01-16T15:23:13.989 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:23:13.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 msgr2=0x7fa1dc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.992 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 0x7fa1dc06ea10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4033000 tx=0x7fa1f4009030 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 msgr2=0x7fa2040afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00f380 tx=0x7fa1ec00f840 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 shutdown_connections 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa204151bb0 0x7fa204155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fa1dc06c550 0x7fa1dc06ea10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa204155d10 0x7fa2040afb10 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 --2- 172.21.15.125:0/2595647448 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa1e4005bb0 0x7fa2040ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 >> 172.21.15.125:0/2595647448 conn(0x7fa20406a7d0 msgr2=0x7fa20415a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 shutdown_connections 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:13.996+0000 7fa209e00640 1 -- 172.21.15.125:0/2595647448 wait complete. 2024-01-16T15:23:14.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:14.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.216+0000 7f980dc00640 1 Processor -- start 2024-01-16T15:23:14.212 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.217+0000 7f980dc00640 1 -- start start 2024-01-16T15:23:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.218+0000 7f980dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f9808070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.218+0000 7f980dc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f980815bf10 0x7f9808143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.218+0000 7f980dc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980815f4e0 con 0x7f98080adf20 2024-01-16T15:23:14.214 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.218+0000 7f980dc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980815f970 con 0x7f980815bf10 2024-01-16T15:23:14.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.218+0000 7f980dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980815c3c0 con 0x7f98080ccb50 2024-01-16T15:23:14.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f980815bf10 0x7f9808143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f980815bf10 0x7f9808143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60230/0 (socket says 172.21.15.125:60230) 2024-01-16T15:23:14.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 -- 172.21.15.125:0/3047721536 learned_addr learned my addr 172.21.15.125:0/3047721536 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:14.215 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f97ffe00640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f9808070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 -- 172.21.15.125:0/3047721536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 msgr2=0x7f9808070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:14.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f9808070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 -- 172.21.15.125:0/3047721536 >> v1:172.21.15.137:6789/0 conn(0x7f98080adf20 legacy=0x7f98080cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:14.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.219+0000 7f980d200640 1 -- 172.21.15.125:0/3047721536 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808071130 con 0x7f980815bf10 2024-01-16T15:23:14.216 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.220+0000 7f980d200640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f9808143d30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f9800009e50 tx=0x7f980002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ceaca73a19b318 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f9807400640 1 -- 172.21.15.125:0/3047721536 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980002c9e0 con 0x7f980815bf10 2024-01-16T15:23:14.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f9807400640 1 -- 172.21.15.125:0/3047721536 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980002cb40 con 0x7f980815bf10 2024-01-16T15:23:14.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f9807400640 1 -- 172.21.15.125:0/3047721536 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98000343f0 con 0x7f980815bf10 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 -- 172.21.15.125:0/3047721536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 msgr2=0x7f9808143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f9808143d30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f9800009e50 tx=0x7f980002be40 comp rx=0 tx=0).stop 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 -- 172.21.15.125:0/3047721536 shutdown_connections 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f9808143d30 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 --2- 172.21.15.125:0/3047721536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f9808070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.221+0000 7f980dc00640 1 -- 172.21.15.125:0/3047721536 >> 172.21.15.125:0/3047721536 conn(0x7f980806a7d0 msgr2=0x7f980806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.222+0000 7f980dc00640 1 -- 172.21.15.125:0/3047721536 shutdown_connections 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.222+0000 7f980dc00640 1 -- 172.21.15.125:0/3047721536 wait complete. 2024-01-16T15:23:14.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.223+0000 7f980dc00640 1 Processor -- start 2024-01-16T15:23:14.219 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.223+0000 7f980dc00640 1 -- start start 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98080adf20 0x7f980815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f98080b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980815ae50 con 0x7f980815bf10 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98081e71b0 con 0x7f98080adf20 2024-01-16T15:23:14.220 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.224+0000 7f980dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98081e7490 con 0x7f98080ccb50 2024-01-16T15:23:14.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60236/0 (socket says 172.21.15.125:60236) 2024-01-16T15:23:14.221 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 -- 172.21.15.125:0/1270337618 learned_addr learned my addr 172.21.15.125:0/1270337618 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f97ffe00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f98080b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f9807e00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98080adf20 0x7f980815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 -- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 msgr2=0x7f98080b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f98080b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 -- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98080adf20 msgr2=0x7f980815a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.223 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98080adf20 0x7f980815a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 -- 172.21.15.125:0/1270337618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800009b30 con 0x7f980815bf10 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.225+0000 7f980d200640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f9800009e50 tx=0x7f9800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.226+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800009510 con 0x7f980815bf10 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.226+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980003e450 con 0x7f980815bf10 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.226+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98081e7770 con 0x7f980815bf10 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.226+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980003e7e0 con 0x7f980815bf10 2024-01-16T15:23:14.224 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.226+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98081e7d30 con 0x7f980815bf10 2024-01-16T15:23:14.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.227+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9808067450 con 0x7f980815bf10 2024-01-16T15:23:14.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.228+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f980003e940 con 0x7f980815bf10 2024-01-16T15:23:14.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.229+0000 7f9804c00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 0x7f97e006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.229+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98000ab560 con 0x7f980815bf10 2024-01-16T15:23:14.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.229+0000 7f9807e00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 0x7f97e006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.230+0000 7f9807e00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 0x7f97e006ea10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f97f8004ea0 tx=0x7f97f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.233+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98000e1e90 con 0x7f980815bf10 2024-01-16T15:23:14.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.427+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9808070410 con 0x7f980815bf10 2024-01-16T15:23:14.423 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.428+0000 7f9804c00640 1 -- 172.21.15.125:0/1270337618 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9800043020 con 0x7f980815bf10 2024-01-16T15:23:14.423 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T15:23:14.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.430+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 msgr2=0x7f97e006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.430+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 0x7f97e006ea10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f97f8004ea0 tx=0x7f97f8009040 comp rx=0 tx=0).stop 2024-01-16T15:23:14.426 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.430+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 msgr2=0x7f98080b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.430+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f9800009e50 tx=0x7f9800004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 shutdown_connections 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97e006c550 0x7f97e006ea10 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980815bf10 0x7f98080b3420 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98080ccb50 0x7f98080b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 --2- 172.21.15.125:0/1270337618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98080adf20 0x7f980815a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 >> 172.21.15.125:0/1270337618 conn(0x7f980806a7d0 msgr2=0x7f980806d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:14.427 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 shutdown_connections 2024-01-16T15:23:14.428 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.431+0000 7f980dc00640 1 -- 172.21.15.125:0/1270337618 wait complete. 2024-01-16T15:23:14.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.647+0000 7f59a3e00640 1 Processor -- start 2024-01-16T15:23:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.647+0000 7f59a3e00640 1 -- start start 2024-01-16T15:23:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 0x7f599c150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f599c0cf290 0x7f599c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599c072610 con 0x7f599c0cf290 2024-01-16T15:23:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599c071190 con 0x7f599c072230 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.648+0000 7f59a3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c071470 con 0x7f599c0cf6d0 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f599c0cf290 0x7f599c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f599c0cf290 0x7f599c15d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60252/0 (socket says 172.21.15.125:60252) 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 -- 172.21.15.125:0/3204792093 learned_addr learned my addr 172.21.15.125:0/3204792093 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 -- 172.21.15.125:0/3204792093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 msgr2=0x7f599c15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a1400640 1 --2- 172.21.15.125:0/3204792093 >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 0x7f599c150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a1e00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 -- 172.21.15.125:0/3204792093 >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 msgr2=0x7f599c150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 --2- 172.21.15.125:0/3204792093 >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 0x7f599c150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a0a00640 1 -- 172.21.15.125:0/3204792093 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c071750 con 0x7f599c0cf290 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a1400640 1 --2- 172.21.15.125:0/3204792093 >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 0x7f599c150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.649+0000 7f59a1e00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f59a0a00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c15d300 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f599c06e420 tx=0x7f598402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea5979ef875883b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f599be00640 1 -- 172.21.15.125:0/3204792093 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984031870 con 0x7f599c0cf290 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f599be00640 1 -- 172.21.15.125:0/3204792093 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59840319d0 con 0x7f599c0cf290 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f599be00640 1 -- 172.21.15.125:0/3204792093 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984031d60 con 0x7f599c0cf290 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f59a3e00640 1 -- 172.21.15.125:0/3204792093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 msgr2=0x7f599c15d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.651+0000 7f59a3e00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c15d300 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f599c06e420 tx=0x7f598402b910 comp rx=0 tx=0).stop 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 -- 172.21.15.125:0/3204792093 shutdown_connections 2024-01-16T15:23:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 --2- 172.21.15.125:0/3204792093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c15d300 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 --2- 172.21.15.125:0/3204792093 >> v2:172.21.15.137:3300/0 conn(0x7f599c072230 0x7f599c150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 -- 172.21.15.125:0/3204792093 >> 172.21.15.125:0/3204792093 conn(0x7f599c06a800 msgr2=0x7f599c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 -- 172.21.15.125:0/3204792093 shutdown_connections 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.652+0000 7f59a3e00640 1 -- 172.21.15.125:0/3204792093 wait complete. 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.653+0000 7f59a3e00640 1 Processor -- start 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 -- start start 2024-01-16T15:23:14.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f599c072230 0x7f599c143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c1f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c14cdc0 con 0x7f599c0cf290 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c14d060 con 0x7f599c072230 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.654+0000 7f59a3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c14d340 con 0x7f599c0cf6d0 2024-01-16T15:23:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60254/0 (socket says 172.21.15.125:60254) 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 -- 172.21.15.125:0/1160288275 learned_addr learned my addr 172.21.15.125:0/1160288275 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a1400640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f599c072230 0x7f599c143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a1e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c1f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 -- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 msgr2=0x7f599c1f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c1f0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 -- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f599c072230 msgr2=0x7f599c143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f599c072230 0x7f599c143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.655+0000 7f59a0a00640 1 -- 172.21.15.125:0/1160288275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840094d0 con 0x7f599c0cf290 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f59a0a00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f59840097c0 tx=0x7f59840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984031820 con 0x7f599c0cf290 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59840443f0 con 0x7f599c0cf290 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c14d620 con 0x7f599c0cf290 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840447a0 con 0x7f599c0cf290 2024-01-16T15:23:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.656+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c143fb0 con 0x7f599c0cf290 2024-01-16T15:23:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.657+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c14db40 con 0x7f599c0cf290 2024-01-16T15:23:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.658+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f598404d450 con 0x7f599c0cf290 2024-01-16T15:23:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.659+0000 7f5999600640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 0x7f597406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.659+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f598402d040 con 0x7f599c0cf290 2024-01-16T15:23:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.659+0000 7f59a1400640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 0x7f597406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.660+0000 7f59a1400640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 0x7f597406ea40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f598c006b10 tx=0x7f598c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.659 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.664+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f599c14db40 con 0x7f599c0cf290 2024-01-16T15:23:14.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.854+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f599c14db40 con 0x7f599c0cf290 2024-01-16T15:23:14.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.855+0000 7f5999600640 1 -- 172.21.15.125:0/1160288275 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f599c14db40 con 0x7f599c0cf290 2024-01-16T15:23:14.850 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T15:23:14.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.857+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 msgr2=0x7f597406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.857+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 0x7f597406ea40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f598c006b10 tx=0x7f598c008040 comp rx=0 tx=0).stop 2024-01-16T15:23:14.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.857+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 msgr2=0x7f599c1f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.857+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f59840097c0 tx=0x7f59840046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 shutdown_connections 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f599c0cf6d0 0x7f599c1f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f597406c580 0x7f597406ea40 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f599c0cf290 0x7f599c1f0850 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 --2- 172.21.15.125:0/1160288275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f599c072230 0x7f599c143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 >> 172.21.15.125:0/1160288275 conn(0x7f599c06a800 msgr2=0x7f599c153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 shutdown_connections 2024-01-16T15:23:14.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:14.858+0000 7f59a3e00640 1 -- 172.21.15.125:0/1160288275 wait complete. 2024-01-16T15:23:14.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:15.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.088+0000 7fd807200640 1 Processor -- start 2024-01-16T15:23:15.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.088+0000 7fd807200640 1 -- start start 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd807200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8080b3f30 0x7fd8080d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd807200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8080d0a10 con 0x7fd808160920 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd807200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8080cf080 con 0x7fd8080d0100 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd807200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080c6be0 con 0x7fd8080b3f30 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd805e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd808160920 0x7fd8080cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48356/0 (socket says 172.21.15.125:48356) 2024-01-16T15:23:15.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd805e00640 1 -- 172.21.15.125:0/2781892775 learned_addr learned my addr 172.21.15.125:0/2781892775 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:15.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.090+0000 7fd806800640 1 --2- 172.21.15.125:0/2781892775 >> v2:172.21.15.125:3300/0 conn(0x7fd8080b3f30 0x7fd8080d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.091+0000 7fd806800640 1 -- 172.21.15.125:0/2781892775 >> v1:172.21.15.137:6789/0 conn(0x7fd8080d0100 legacy=0x7fd8080d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.091+0000 7fd806800640 1 -- 172.21.15.125:0/2781892775 >> v1:172.21.15.125:6789/0 conn(0x7fd808160920 legacy=0x7fd8080cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.091+0000 7fd806800640 1 -- 172.21.15.125:0/2781892775 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080ce7e0 con 0x7fd8080b3f30 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.092+0000 7fd806800640 1 --2- 172.21.15.125:0/2781892775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080b3f30 0x7fd8080d1650 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd800009d90 tx=0x7fd80002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32432d88e7b8fb75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.092+0000 7fd805400640 1 -- 172.21.15.125:0/2781892775 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80002cc60 con 0x7fd8080b3f30 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.092+0000 7fd805400640 1 -- 172.21.15.125:0/2781892775 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80002cdc0 con 0x7fd8080b3f30 2024-01-16T15:23:15.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.092+0000 7fd805400640 1 -- 172.21.15.125:0/2781892775 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800034660 con 0x7fd8080b3f30 2024-01-16T15:23:15.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 -- 172.21.15.125:0/2781892775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080b3f30 msgr2=0x7fd8080d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 --2- 172.21.15.125:0/2781892775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080b3f30 0x7fd8080d1650 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd800009d90 tx=0x7fd80002beb0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 -- 172.21.15.125:0/2781892775 shutdown_connections 2024-01-16T15:23:15.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 --2- 172.21.15.125:0/2781892775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080b3f30 0x7fd8080d1650 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 -- 172.21.15.125:0/2781892775 >> 172.21.15.125:0/2781892775 conn(0x7fd8080d3090 msgr2=0x7fd8080d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:15.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 -- 172.21.15.125:0/2781892775 shutdown_connections 2024-01-16T15:23:15.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.093+0000 7fd807200640 1 -- 172.21.15.125:0/2781892775 wait complete. 2024-01-16T15:23:15.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.095+0000 7fd807200640 1 Processor -- start 2024-01-16T15:23:15.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.095+0000 7fd807200640 1 -- start start 2024-01-16T15:23:15.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 0x7fd8080b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.093 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd808160920 0x7fd8080cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080b1380 con 0x7fd8080d0100 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080cb810 con 0x7fd808160920 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd807200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080b5df0 con 0x7fd8080b3f30 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.094 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60280/0 (socket says 172.21.15.125:60280) 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd7ffe00640 1 -- 172.21.15.125:0/2741634010 learned_addr learned my addr 172.21.15.125:0/2741634010 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd806800640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd808160920 0x7fd8080cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd805e00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 0x7fd8080b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.096+0000 7fd7ffe00640 1 -- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 msgr2=0x7fd8080b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7ffe00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 0x7fd8080b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7ffe00640 1 -- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd808160920 msgr2=0x7fd8080cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7ffe00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd808160920 0x7fd8080cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7ffe00640 1 -- 172.21.15.125:0/2741634010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009a70 con 0x7fd8080d0100 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd805e00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 0x7fd8080b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7ffe00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c430 tx=0x7fd7f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0011790 con 0x7fd8080d0100 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f00118f0 con 0x7fd8080d0100 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0011ce0 con 0x7fd8080d0100 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8080b60d0 con 0x7fd8080d0100 2024-01-16T15:23:15.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.097+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd808152780 con 0x7fd8080d0100 2024-01-16T15:23:15.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.098+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8080b65f0 con 0x7fd8080d0100 2024-01-16T15:23:15.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.099+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd7f001c460 con 0x7fd8080d0100 2024-01-16T15:23:15.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.100+0000 7fd7fe000640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 0x7fd7e006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.100+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7f0088b30 con 0x7fd8080d0100 2024-01-16T15:23:15.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.100+0000 7fd805e00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 0x7fd7e006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.101+0000 7fd805e00640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 0x7fd7e006ea10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005880 tx=0x7fd7f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.104+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8080b65f0 con 0x7fd8080d0100 2024-01-16T15:23:15.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:15.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:15.304 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.308+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd8080b65f0 con 0x7fd8080d0100 2024-01-16T15:23:15.305 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.309+0000 7fd7fe000640 1 -- 172.21.15.125:0/2741634010 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd8080b65f0 con 0x7fd8080d0100 2024-01-16T15:23:15.305 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T15:23:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.311+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 msgr2=0x7fd7e006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.311+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 0x7fd7e006ea10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005880 tx=0x7fd7f8009040 comp rx=0 tx=0).stop 2024-01-16T15:23:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 msgr2=0x7fd808152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c430 tx=0x7fd7f000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 shutdown_connections 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd808160920 0x7fd8080cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd7e006c550 0x7fd7e006ea10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8080d0100 0x7fd808152240 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 --2- 172.21.15.125:0/2741634010 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8080b3f30 0x7fd8080b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 >> 172.21.15.125:0/2741634010 conn(0x7fd8080d3090 msgr2=0x7fd80806c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 shutdown_connections 2024-01-16T15:23:15.310 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.312+0000 7fd807200640 1 -- 172.21.15.125:0/2741634010 wait complete. 2024-01-16T15:23:15.325 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:23:15.325 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:15.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:15.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.634+0000 7fcccc000640 1 Processor -- start 2024-01-16T15:23:15.630 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.634+0000 7fcccc000640 1 -- start start 2024-01-16T15:23:15.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.635+0000 7fcccc000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fccc413df80 0x7fccc415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fcccc000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc40cf9a0 0x7fccc40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fcccc000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc406c470 con 0x7fccc413e360 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fcccc000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc406c750 con 0x7fccc413df80 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fcccc000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc406ca30 con 0x7fccc40cf9a0 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fccc9600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fccc413df80 0x7fccc415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fccc8c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fccc413e360 0x7fccc40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48364/0 (socket says 172.21.15.125:48364) 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fccc8c00640 1 -- 172.21.15.125:0/229480777 learned_addr learned my addr 172.21.15.125:0/229480777 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:15.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fccc9600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fccc413df80 0x7fccc415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60284/0 (socket says 172.21.15.125:60284) 2024-01-16T15:23:15.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.636+0000 7fccca000640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc40cf9a0 0x7fccc40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.637+0000 7fccc9600640 1 -- 172.21.15.125:0/229480777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc40cf9a0 msgr2=0x7fccc40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:15.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.637+0000 7fccc9600640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc40cf9a0 0x7fccc40cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.637+0000 7fccc9600640 1 -- 172.21.15.125:0/229480777 >> v1:172.21.15.125:6789/0 conn(0x7fccc413e360 legacy=0x7fccc40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.637+0000 7fccc9600640 1 -- 172.21.15.125:0/229480777 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc406cd10 con 0x7fccc413df80 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.638+0000 7fccc9600640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc413df80 0x7fccc415bee0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fccb40097a0 tx=0x7fccb402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c9a75768ef8f86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fccc3e00640 1 -- 172.21.15.125:0/229480777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb402d040 con 0x7fccc413df80 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fccc3e00640 1 -- 172.21.15.125:0/229480777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4031810 con 0x7fccc413df80 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fccc3e00640 1 -- 172.21.15.125:0/229480777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4031ba0 con 0x7fccc413df80 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 -- 172.21.15.125:0/229480777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc413df80 msgr2=0x7fccc415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc413df80 0x7fccc415bee0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fccb40097a0 tx=0x7fccb402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 -- 172.21.15.125:0/229480777 shutdown_connections 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc40cf9a0 0x7fccc40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 --2- 172.21.15.125:0/229480777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc413df80 0x7fccc415bee0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 -- 172.21.15.125:0/229480777 >> 172.21.15.125:0/229480777 conn(0x7fccc406a800 msgr2=0x7fccc406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 -- 172.21.15.125:0/229480777 shutdown_connections 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.639+0000 7fcccc000640 1 -- 172.21.15.125:0/229480777 wait complete. 2024-01-16T15:23:15.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.641+0000 7fcccc000640 1 Processor -- start 2024-01-16T15:23:15.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.641+0000 7fcccc000640 1 -- start start 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 0x7fccc4159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 0x7fccc40a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4158790 con 0x7fccc40cf9a0 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41589a0 con 0x7fccc413df80 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fcccc000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4158c80 con 0x7fccc413e360 2024-01-16T15:23:15.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fccc9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fccc9600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60292/0 (socket says 172.21.15.125:60292) 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fccc9600640 1 -- 172.21.15.125:0/1903955458 learned_addr learned my addr 172.21.15.125:0/1903955458 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.642+0000 7fccca000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 0x7fccc40a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc8c00640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 0x7fccc4159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 -- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 msgr2=0x7fccc40a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 0x7fccc40a9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 -- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 msgr2=0x7fccc4159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 0x7fccc4159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 -- 172.21.15.125:0/1903955458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4009400 con 0x7fccc40cf9a0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccca000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 0x7fccc40a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc8c00640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 0x7fccc4159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc9600640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fccb40098c0 tx=0x7fccb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb403ad60 con 0x7fccc40cf9a0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4044400 con 0x7fccc40cf9a0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc4158f60 con 0x7fccc40cf9a0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40447b0 con 0x7fccc40cf9a0 2024-01-16T15:23:15.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.643+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4133b70 con 0x7fccc40cf9a0 2024-01-16T15:23:15.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.644+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc4159480 con 0x7fccc40cf9a0 2024-01-16T15:23:15.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.646+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fccb404d440 con 0x7fccc40cf9a0 2024-01-16T15:23:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.646+0000 7fccc1600640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 0x7fcc9c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.647+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fccb40aaf50 con 0x7fccc40cf9a0 2024-01-16T15:23:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.647+0000 7fccc8c00640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 0x7fcc9c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:15.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.647+0000 7fccc8c00640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 0x7fcc9c06ea10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fccc40b3f60 tx=0x7fccac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:15.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.651+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccc4159480 con 0x7fccc40cf9a0 2024-01-16T15:23:15.844 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.849+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc4159480 con 0x7fccc40cf9a0 2024-01-16T15:23:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.850+0000 7fccc1600640 1 -- 172.21.15.125:0/1903955458 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16668 (secure 0 0 0) 0x7fccc4159480 con 0x7fccc40cf9a0 2024-01-16T15:23:15.847 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:15.847 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:28.491486+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 msgr2=0x7fcc9c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 0x7fcc9c06ea10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fccc40b3f60 tx=0x7fccac009040 comp rx=0 tx=0).stop 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 msgr2=0x7fccc4133630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fccb40098c0 tx=0x7fccb40046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 shutdown_connections 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccc413e360 0x7fccc40a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccc413df80 0x7fccc4159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fcc9c06c550 0x7fcc9c06ea10 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 --2- 172.21.15.125:0/1903955458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccc40cf9a0 0x7fccc4133630 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 >> 172.21.15.125:0/1903955458 conn(0x7fccc406a800 msgr2=0x7fccc406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 shutdown_connections 2024-01-16T15:23:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:15.853+0000 7fcccc000640 1 -- 172.21.15.125:0/1903955458 wait complete. 2024-01-16T15:23:15.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:04.806000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:06.026871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:15.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:16.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.128+0000 7f33df200640 1 Processor -- start 2024-01-16T15:23:16.124 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.129+0000 7f33df200640 1 -- start start 2024-01-16T15:23:16.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.130+0000 7f33df200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f33d8160c90 0x7f33d80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.126 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.131+0000 7f33df200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d80c6c00 0x7f33d80cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.131+0000 7f33df200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d80cf6c0 con 0x7f33d81608b0 2024-01-16T15:23:16.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.131+0000 7f33df200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d80d1450 con 0x7f33d8160c90 2024-01-16T15:23:16.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.131+0000 7f33df200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d80b4390 con 0x7f33d80c6c00 2024-01-16T15:23:16.127 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.131+0000 7f33dd400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f33d8160c90 0x7f33d80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.132+0000 7f33dde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f33d81608b0 0x7f33d80d1690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48368/0 (socket says 172.21.15.125:48368) 2024-01-16T15:23:16.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.132+0000 7f33dde00640 1 -- 172.21.15.125:0/893644110 learned_addr learned my addr 172.21.15.125:0/893644110 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:16.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.132+0000 7f33de800640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d80c6c00 0x7f33d80cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.128 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.133+0000 7f33dd400640 1 -- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d80c6c00 msgr2=0x7f33d80cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:16.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.133+0000 7f33dd400640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d80c6c00 0x7f33d80cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.133+0000 7f33dd400640 1 -- 172.21.15.125:0/893644110 >> v1:172.21.15.125:6789/0 conn(0x7f33d81608b0 legacy=0x7f33d80d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.133+0000 7f33dca00640 1 -- 172.21.15.125:0/893644110 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713616999 0 0) 0x7f33d80cf6c0 con 0x7f33d81608b0 2024-01-16T15:23:16.129 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.133+0000 7f33dd400640 1 -- 172.21.15.125:0/893644110 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d80ce7e0 con 0x7f33d8160c90 2024-01-16T15:23:16.130 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.134+0000 7f33dd400640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d8160c90 0x7f33d80d3bd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f33c00097a0 tx=0x7f33c002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ddba7bd487cf79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.135+0000 7f33dca00640 1 -- 172.21.15.125:0/893644110 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c00316f0 con 0x7f33d8160c90 2024-01-16T15:23:16.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.135+0000 7f33dca00640 1 -- 172.21.15.125:0/893644110 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c0031850 con 0x7f33d8160c90 2024-01-16T15:23:16.131 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.135+0000 7f33dca00640 1 -- 172.21.15.125:0/893644110 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0031be0 con 0x7f33d8160c90 2024-01-16T15:23:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 -- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d8160c90 msgr2=0x7f33d80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d8160c90 0x7f33d80d3bd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f33c00097a0 tx=0x7f33c002b790 comp rx=0 tx=0).stop 2024-01-16T15:23:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 -- 172.21.15.125:0/893644110 shutdown_connections 2024-01-16T15:23:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d80c6c00 0x7f33d80cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 --2- 172.21.15.125:0/893644110 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d8160c90 0x7f33d80d3bd0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.137+0000 7f33df200640 1 -- 172.21.15.125:0/893644110 >> 172.21.15.125:0/893644110 conn(0x7f33d80d30b0 msgr2=0x7f33d80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:16.134 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.138+0000 7f33df200640 1 -- 172.21.15.125:0/893644110 shutdown_connections 2024-01-16T15:23:16.135 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.138+0000 7f33df200640 1 -- 172.21.15.125:0/893644110 wait complete. 2024-01-16T15:23:16.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.142+0000 7f33df200640 1 Processor -- start 2024-01-16T15:23:16.138 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.143+0000 7f33df200640 1 -- start start 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d80c6c00 0x7f33d80b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d81608b0 0x7f33d815e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d815d390 con 0x7f33d8160c90 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d815c780 con 0x7f33d80c6c00 2024-01-16T15:23:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.145+0000 7f33df200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d815ca60 con 0x7f33d81608b0 2024-01-16T15:23:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.146+0000 7f33de800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.146+0000 7f33de800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60306/0 (socket says 172.21.15.125:60306) 2024-01-16T15:23:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.146+0000 7f33de800640 1 -- 172.21.15.125:0/2476119666 learned_addr learned my addr 172.21.15.125:0/2476119666 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33dd400640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d81608b0 0x7f33d815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33de800640 1 -- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d81608b0 msgr2=0x7f33d815e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33de800640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d81608b0 0x7f33d815e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33de800640 1 -- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d80c6c00 msgr2=0x7f33d80b8670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:23:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33de800640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d80c6c00 0x7f33d80b8670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.147+0000 7f33de800640 1 -- 172.21.15.125:0/2476119666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009400 con 0x7f33d8160c90 2024-01-16T15:23:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.148+0000 7f33de800640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009fd0 tx=0x7f33d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.149+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0014420 con 0x7f33d8160c90 2024-01-16T15:23:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.149+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0014580 con 0x7f33d8160c90 2024-01-16T15:23:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.149+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0014930 con 0x7f33d8160c90 2024-01-16T15:23:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.151+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d815cd40 con 0x7f33d8160c90 2024-01-16T15:23:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.151+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d80b7a30 con 0x7f33d8160c90 2024-01-16T15:23:16.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.153+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d80b7bc0 con 0x7f33d8160c90 2024-01-16T15:23:16.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.157+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f33d0014a90 con 0x7f33d8160c90 2024-01-16T15:23:16.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.159+0000 7f33d6000640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 0x7f33b006eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.159+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f33d0089460 con 0x7f33d8160c90 2024-01-16T15:23:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.159+0000 7f33dde00640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 0x7f33b006eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.164+0000 7f33dde00640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 0x7f33b006eb10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f33c80011e0 tx=0x7f33c80078b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.167+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33d00559a0 con 0x7f33d8160c90 2024-01-16T15:23:16.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.510+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33d8067450 con 0x7f33d8160c90 2024-01-16T15:23:16.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.511+0000 7f33d6000640 1 -- 172.21.15.125:0/2476119666 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33d0058fc0 con 0x7f33d8160c90 2024-01-16T15:23:16.507 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:23:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 msgr2=0x7f33b006eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 0x7f33b006eb10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f33c80011e0 tx=0x7f33c80078b0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 msgr2=0x7f33d815e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009fd0 tx=0x7f33d000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 shutdown_connections 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f33b006c650 0x7f33b006eb10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33d8160c90 0x7f33d815e790 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33d81608b0 0x7f33d815e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 --2- 172.21.15.125:0/2476119666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33d80c6c00 0x7f33d80b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 >> 172.21.15.125:0/2476119666 conn(0x7f33d80d30b0 msgr2=0x7f33d80c7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 shutdown_connections 2024-01-16T15:23:16.511 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.514+0000 7f33df200640 1 -- 172.21.15.125:0/2476119666 wait complete. 2024-01-16T15:23:16.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:16.871 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.875+0000 7f9a1a800640 1 Processor -- start 2024-01-16T15:23:16.872 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.876+0000 7f9a1a800640 1 -- start start 2024-01-16T15:23:16.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.880+0000 7f9a1a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 0x7f9a14166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.880+0000 7f9a1a800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1415c5f0 con 0x7f9a1413df80 2024-01-16T15:23:16.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.880+0000 7f9a1a800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1406c470 con 0x7f9a1413e360 2024-01-16T15:23:16.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.880+0000 7f9a1a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1406c750 con 0x7f9a140cf9a0 2024-01-16T15:23:16.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.882+0000 7f9a0be00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f9a1413e360 0x7f9a140cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55018/0 (socket says 172.21.15.125:55018) 2024-01-16T15:23:16.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.882+0000 7f9a0be00640 1 -- 172.21.15.125:0/3843031902 learned_addr learned my addr 172.21.15.125:0/3843031902 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:16.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.882+0000 7f9a13e00640 1 --2- 172.21.15.125:0/3843031902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 0x7f9a14166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.883+0000 7f9a13e00640 1 -- 172.21.15.125:0/3843031902 >> v1:172.21.15.137:6789/0 conn(0x7f9a1413e360 legacy=0x7f9a140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.883+0000 7f9a13e00640 1 -- 172.21.15.125:0/3843031902 >> v1:172.21.15.125:6789/0 conn(0x7f9a1413df80 legacy=0x7f9a1415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.883+0000 7f9a12a00640 1 -- 172.21.15.125:0/3843031902 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085889064 0 0) 0x7f9a1406c470 con 0x7f9a1413e360 2024-01-16T15:23:16.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.883+0000 7f9a13e00640 1 -- 172.21.15.125:0/3843031902 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1406ca30 con 0x7f9a140cf9a0 2024-01-16T15:23:16.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.885+0000 7f9a13e00640 1 --2- 172.21.15.125:0/3843031902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 0x7f9a14166030 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009dc0 tx=0x7f9a0c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114c3285fde18021 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.886+0000 7f9a12a00640 1 -- 172.21.15.125:0/3843031902 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c02ca80 con 0x7f9a140cf9a0 2024-01-16T15:23:16.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.886+0000 7f9a12a00640 1 -- 172.21.15.125:0/3843031902 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c02cbe0 con 0x7f9a140cf9a0 2024-01-16T15:23:16.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.886+0000 7f9a12a00640 1 -- 172.21.15.125:0/3843031902 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c034440 con 0x7f9a140cf9a0 2024-01-16T15:23:16.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 -- 172.21.15.125:0/3843031902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 msgr2=0x7f9a14166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 --2- 172.21.15.125:0/3843031902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 0x7f9a14166030 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009dc0 tx=0x7f9a0c02bd50 comp rx=0 tx=0).stop 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 -- 172.21.15.125:0/3843031902 shutdown_connections 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 --2- 172.21.15.125:0/3843031902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a140cf9a0 0x7f9a14166030 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 -- 172.21.15.125:0/3843031902 >> 172.21.15.125:0/3843031902 conn(0x7f9a1406a800 msgr2=0x7f9a1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 -- 172.21.15.125:0/3843031902 shutdown_connections 2024-01-16T15:23:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.887+0000 7f9a1a800640 1 -- 172.21.15.125:0/3843031902 wait complete. 2024-01-16T15:23:16.884 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.889+0000 7f9a1a800640 1 Processor -- start 2024-01-16T15:23:16.885 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.889+0000 7f9a1a800640 1 -- start start 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 0x7f9a1414f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a1413e360 0x7f9a1413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414f540 con 0x7f9a1413df80 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414d8b0 con 0x7f9a140cf9a0 2024-01-16T15:23:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.890+0000 7f9a1a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414eef0 con 0x7f9a1413e360 2024-01-16T15:23:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a13400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 0x7f9a1414f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a0be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a13400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 0x7f9a1414f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.125:35450/0 (socket says 172.21.15.125:35450) 2024-01-16T15:23:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a0be00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36936/0 (socket says 172.21.15.125:36936) 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a13400640 1 -- 172.21.15.125:0/2587380243 learned_addr learned my addr 172.21.15.125:0/2587380243 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.891+0000 7f9a13e00640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a1413e360 0x7f9a1413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 -- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a1413e360 msgr2=0x7f9a1413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a1413e360 0x7f9a1413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 -- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 msgr2=0x7f9a1414f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 0x7f9a1414f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 -- 172.21.15.125:0/2587380243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c009a70 con 0x7f9a1413df80 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0be00640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c450 tx=0x7f99fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.892+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc0074e0 con 0x7f9a1413df80 2024-01-16T15:23:16.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.893+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc007640 con 0x7f9a1413df80 2024-01-16T15:23:16.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.893+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc004a20 con 0x7f9a1413df80 2024-01-16T15:23:16.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.893+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1413dd80 con 0x7f9a1413df80 2024-01-16T15:23:16.890 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.893+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1414f160 con 0x7f9a1413df80 2024-01-16T15:23:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.895+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1414f680 con 0x7f9a1413df80 2024-01-16T15:23:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.897+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f99fc004b80 con 0x7f9a1413df80 2024-01-16T15:23:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.899+0000 7f9a0b400640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 0x7f99ec06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.899+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f99fc088030 con 0x7f9a1413df80 2024-01-16T15:23:16.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.903+0000 7f9a13400640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 0x7f99ec06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.905+0000 7f9a13400640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 0x7f99ec06ea60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000f60 tx=0x7f9a04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:16.907+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a1414f680 con 0x7f9a1413df80 2024-01-16T15:23:17.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.148+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a1414f680 con 0x7f9a1413df80 2024-01-16T15:23:17.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.149+0000 7f9a0b400640 1 -- 172.21.15.125:0/2587380243 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a1414f680 con 0x7f9a1413df80 2024-01-16T15:23:17.145 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T15:23:17.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 msgr2=0x7f99ec06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 0x7f99ec06ea60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000f60 tx=0x7f9a04009040 comp rx=0 tx=0).stop 2024-01-16T15:23:17.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 msgr2=0x7f9a1414d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c450 tx=0x7f99fc00c910 comp rx=0 tx=0).stop 2024-01-16T15:23:17.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 shutdown_connections 2024-01-16T15:23:17.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a1413e360 0x7f9a1413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f99ec06c5a0 0x7f99ec06ea60 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a1413df80 0x7f9a1414d370 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.151+0000 7f9a1a800640 1 --2- 172.21.15.125:0/2587380243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a140cf9a0 0x7f9a1414f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.152+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 >> 172.21.15.125:0/2587380243 conn(0x7f9a1406a800 msgr2=0x7f9a1415e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.152+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 shutdown_connections 2024-01-16T15:23:17.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.152+0000 7f9a1a800640 1 -- 172.21.15.125:0/2587380243 wait complete. 2024-01-16T15:23:17.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:17.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.475+0000 7f05c2200640 1 Processor -- start 2024-01-16T15:23:17.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.475+0000 7f05c2200640 1 -- start start 2024-01-16T15:23:17.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.477+0000 7f05c2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc150760 0x7f05bc072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.477+0000 7f05c2200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f05bc0b32e0 0x7f05bc15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.477+0000 7f05c2200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc15dc80 con 0x7f05bc15d300 2024-01-16T15:23:17.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.477+0000 7f05c2200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0b3770 con 0x7f05bc0b32e0 2024-01-16T15:23:17.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.477+0000 7f05c2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc071190 con 0x7f05bc150760 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05b3e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f05bc15d300 0x7f05bc0b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.125:55020/0 (socket says 172.21.15.125:55020) 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f05bc0b32e0 0x7f05bc15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05b3e00640 1 -- 172.21.15.125:0/1313802036 learned_addr learned my addr 172.21.15.125:0/1313802036 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bb400640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc150760 0x7f05bc072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bbe00640 1 -- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc150760 msgr2=0x7f05bc072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bbe00640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc150760 0x7f05bc072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bbe00640 1 -- 172.21.15.125:0/1313802036 >> v1:172.21.15.137:6789/0 conn(0x7f05bc15d300 legacy=0x7f05bc0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.478+0000 7f05bbe00640 1 -- 172.21.15.125:0/1313802036 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc071470 con 0x7f05bc0b32e0 2024-01-16T15:23:17.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.480+0000 7f05bbe00640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc15d740 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d90 tx=0x7f05b402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=379a9b34301d7703 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.480+0000 7f05baa00640 1 -- 172.21.15.125:0/1313802036 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b402cc50 con 0x7f05bc0b32e0 2024-01-16T15:23:17.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05baa00640 1 -- 172.21.15.125:0/1313802036 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b402cdb0 con 0x7f05bc0b32e0 2024-01-16T15:23:17.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05baa00640 1 -- 172.21.15.125:0/1313802036 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4034660 con 0x7f05bc0b32e0 2024-01-16T15:23:17.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 -- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 msgr2=0x7f05bc15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc15d740 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d90 tx=0x7f05b402bea0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 -- 172.21.15.125:0/1313802036 shutdown_connections 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc15d740 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 --2- 172.21.15.125:0/1313802036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc150760 0x7f05bc072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.481+0000 7f05c2200640 1 -- 172.21.15.125:0/1313802036 >> 172.21.15.125:0/1313802036 conn(0x7f05bc06a800 msgr2=0x7f05bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.482+0000 7f05c2200640 1 -- 172.21.15.125:0/1313802036 shutdown_connections 2024-01-16T15:23:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.482+0000 7f05c2200640 1 -- 172.21.15.125:0/1313802036 wait complete. 2024-01-16T15:23:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.484+0000 7f05c2200640 1 Processor -- start 2024-01-16T15:23:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.484+0000 7f05c2200640 1 -- start start 2024-01-16T15:23:17.480 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc14ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc15d300 0x7f05bc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1578a0 con 0x7f05bc150760 2024-01-16T15:23:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0cdfe0 con 0x7f05bc0b32e0 2024-01-16T15:23:17.481 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05c2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0ce2c0 con 0x7f05bc15d300 2024-01-16T15:23:17.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05b3e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36950/0 (socket says 172.21.15.125:36950) 2024-01-16T15:23:17.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05b3e00640 1 -- 172.21.15.125:0/383623709 learned_addr learned my addr 172.21.15.125:0/383623709 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.485+0000 7f05bb400640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc14ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05bbe00640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc15d300 0x7f05bc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 -- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc15d300 msgr2=0x7f05bc06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc15d300 0x7f05bc06d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 -- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 msgr2=0x7f05bc14ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc14ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 -- 172.21.15.125:0/383623709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4009a70 con 0x7f05bc150760 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.486+0000 7f05b3e00640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c430 tx=0x7f05a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.483 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.487+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40064e0 con 0x7f05bc150760 2024-01-16T15:23:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.487+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4006640 con 0x7f05bc150760 2024-01-16T15:23:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.487+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40049e0 con 0x7f05bc150760 2024-01-16T15:23:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.487+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc157af0 con 0x7f05bc150760 2024-01-16T15:23:17.484 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.488+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc14d370 con 0x7f05bc150760 2024-01-16T15:23:17.485 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.489+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc14d530 con 0x7f05bc150760 2024-01-16T15:23:17.486 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.491+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f05a4004c00 con 0x7f05bc150760 2024-01-16T15:23:17.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.492+0000 7f05b3400640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 0x7f059406eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.493+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05a4088080 con 0x7f05bc150760 2024-01-16T15:23:17.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.494+0000 7f05bb400640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 0x7f059406eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.495+0000 7f05bb400640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 0x7f059406eae0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05ac006fd0 tx=0x7f05ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.500+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05a4054670 con 0x7f05bc150760 2024-01-16T15:23:17.847 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.851+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05bc0675d0 con 0x7f05bc150760 2024-01-16T15:23:17.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.853+0000 7f05b3400640 1 -- 172.21.15.125:0/383623709 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05a4017020 con 0x7f05bc150760 2024-01-16T15:23:17.849 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T15:23:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 msgr2=0x7f059406eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 0x7f059406eae0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05ac006fd0 tx=0x7f05ac008040 comp rx=0 tx=0).stop 2024-01-16T15:23:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 msgr2=0x7f05bc06cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c430 tx=0x7f05a400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 shutdown_connections 2024-01-16T15:23:17.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05bc15d300 0x7f05bc06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f059406c620 0x7f059406eae0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05bc150760 0x7f05bc06cd00 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 --2- 172.21.15.125:0/383623709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05bc0b32e0 0x7f05bc14ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 >> 172.21.15.125:0/383623709 conn(0x7f05bc06a800 msgr2=0x7f05bc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:17.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 shutdown_connections 2024-01-16T15:23:17.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:17.858+0000 7f05c2200640 1 -- 172.21.15.125:0/383623709 wait complete. 2024-01-16T15:23:17.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.074+0000 7fe53de00640 1 Processor -- start 2024-01-16T15:23:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.074+0000 7fe53de00640 1 -- start start 2024-01-16T15:23:18.071 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.075+0000 7fe53de00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe538143ae0 0x7fe5380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.075+0000 7fe53de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5380cf7d0 0x7fe53815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.075+0000 7fe53de00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5380cfc30 con 0x7fe53815e1e0 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.075+0000 7fe53de00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5380b1ee0 con 0x7fe538143ae0 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.075+0000 7fe53de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b2780 con 0x7fe5380cf7d0 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe538143ae0 0x7fe5380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe538143ae0 0x7fe5380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36956/0 (socket says 172.21.15.125:36956) 2024-01-16T15:23:18.072 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 -- 172.21.15.125:0/1425247071 learned_addr learned my addr 172.21.15.125:0/1425247071 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:18.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 -- 172.21.15.125:0/1425247071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5380cf7d0 msgr2=0x7fe53815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:18.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 --2- 172.21.15.125:0/1425247071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5380cf7d0 0x7fe53815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 -- 172.21.15.125:0/1425247071 >> v1:172.21.15.137:6789/0 conn(0x7fe53815e1e0 legacy=0x7fe53815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:18.073 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.076+0000 7fe536a00640 1 -- 172.21.15.125:0/1425247071 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53815e990 con 0x7fe538143ae0 2024-01-16T15:23:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.077+0000 7fe536a00640 1 --2- 172.21.15.125:0/1425247071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe5380cf290 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fe528009870 tx=0x7fe52802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1dccac0c59eca1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.077+0000 7fe535600640 1 -- 172.21.15.125:0/1425247071 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528031640 con 0x7fe538143ae0 2024-01-16T15:23:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.077+0000 7fe535600640 1 -- 172.21.15.125:0/1425247071 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5280317a0 con 0x7fe538143ae0 2024-01-16T15:23:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.077+0000 7fe535600640 1 -- 172.21.15.125:0/1425247071 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528031b30 con 0x7fe538143ae0 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 -- 172.21.15.125:0/1425247071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 msgr2=0x7fe5380cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 --2- 172.21.15.125:0/1425247071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe5380cf290 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fe528009870 tx=0x7fe52802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 -- 172.21.15.125:0/1425247071 shutdown_connections 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 --2- 172.21.15.125:0/1425247071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5380cf7d0 0x7fe53815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 --2- 172.21.15.125:0/1425247071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe5380cf290 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 -- 172.21.15.125:0/1425247071 >> 172.21.15.125:0/1425247071 conn(0x7fe53806a800 msgr2=0x7fe53806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 -- 172.21.15.125:0/1425247071 shutdown_connections 2024-01-16T15:23:18.075 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.078+0000 7fe53de00640 1 -- 172.21.15.125:0/1425247071 wait complete. 2024-01-16T15:23:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.080+0000 7fe53de00640 1 Processor -- start 2024-01-16T15:23:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.080+0000 7fe53de00640 1 -- start start 2024-01-16T15:23:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5380cf7d0 0x7fe5381598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.077 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe53815e1e0 0x7fe53806c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380a86f0 con 0x7fe538143ae0 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380ae850 con 0x7fe5380cf7d0 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe53de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380adfb0 con 0x7fe53815e1e0 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36970/0 (socket says 172.21.15.125:36970) 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 -- 172.21.15.125:0/1577174652 learned_addr learned my addr 172.21.15.125:0/1577174652 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 -- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe53815e1e0 msgr2=0x7fe53806c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe53815e1e0 0x7fe53806c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.078 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 -- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5380cf7d0 msgr2=0x7fe5381598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5380cf7d0 0x7fe5381598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.081+0000 7fe536000640 1 -- 172.21.15.125:0/1577174652 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280094d0 con 0x7fe538143ae0 2024-01-16T15:23:18.079 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe536000640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b4d0 tx=0x7fe52000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520010870 con 0x7fe538143ae0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5200109d0 con 0x7fe538143ae0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5380ae290 con 0x7fe538143ae0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520010d80 con 0x7fe538143ae0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.082+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe538159e20 con 0x7fe538143ae0 2024-01-16T15:23:18.080 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.083+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5380a84c0 con 0x7fe538143ae0 2024-01-16T15:23:18.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.084+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe52000f3c0 con 0x7fe538143ae0 2024-01-16T15:23:18.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.085+0000 7fe52f400640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 0x7fe51006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.081 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.085+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe520088710 con 0x7fe538143ae0 2024-01-16T15:23:18.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.085+0000 7fe536a00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 0x7fe51006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.086+0000 7fe536a00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 0x7fe51006ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe528036000 tx=0x7fe528030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:18.084 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.089+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5380a84c0 con 0x7fe538143ae0 2024-01-16T15:23:18.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.280+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe5380b2bf0 con 0x7fe538143ae0 2024-01-16T15:23:18.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.280+0000 7fe52f400640 1 -- 172.21.15.125:0/1577174652 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe5380b2bf0 con 0x7fe538143ae0 2024-01-16T15:23:18.276 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 msgr2=0x7fe51006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 0x7fe51006ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe528036000 tx=0x7fe528030000 comp rx=0 tx=0).stop 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 msgr2=0x7fe53806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b4d0 tx=0x7fe52000b990 comp rx=0 tx=0).stop 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 shutdown_connections 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe53815e1e0 0x7fe53806c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fe51006c550 0x7fe51006ea10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe538143ae0 0x7fe53806c470 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 --2- 172.21.15.125:0/1577174652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5380cf7d0 0x7fe5381598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 >> 172.21.15.125:0/1577174652 conn(0x7fe53806a800 msgr2=0x7fe53806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:18.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 shutdown_connections 2024-01-16T15:23:18.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:18.283+0000 7fe53de00640 1 -- 172.21.15.125:0/1577174652 wait complete. 2024-01-16T15:23:18.295 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2024-01-16T15:23:18.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:18.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:23:18.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi125.stderr:ceph-fuse[78084]: fuse finished with error 0 and tester_r 0 2024-01-16T15:23:18.972 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:23:18.972 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:23:18.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:18.973 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:23:19.022 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2024-01-16T15:23:19.022 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:19.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053843,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249392k,nr_inodes=812348,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498788k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:23:19.068 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:23:19.068 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:19.126 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:23:19.126 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:19.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:19.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.396+0000 7f9854a00640 1 Processor -- start 2024-01-16T15:23:19.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.396+0000 7f9854a00640 1 -- start start 2024-01-16T15:23:19.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.398+0000 7f9854a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f984c13df80 0x7f984c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.398+0000 7f9854a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.398+0000 7f9854a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c06c470 con 0x7f984c13e360 2024-01-16T15:23:19.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.398+0000 7f9854a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984c06c750 con 0x7f984c13df80 2024-01-16T15:23:19.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.398+0000 7f9854a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c06ca30 con 0x7f984c0cf9a0 2024-01-16T15:23:19.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.399+0000 7f9852a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.399+0000 7f9852000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f984c13df80 0x7f984c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.399+0000 7f9852a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:60780/0 (socket says 172.21.15.125:60780) 2024-01-16T15:23:19.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.399+0000 7f9852a00640 1 -- 172.21.15.125:0/1464768213 learned_addr learned my addr 172.21.15.125:0/1464768213 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.399+0000 7f9850c00640 1 -- 172.21.15.125:0/1464768213 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507412544 0 0) 0x7f984c06c470 con 0x7f984c13e360 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9850c00640 1 -- 172.21.15.125:0/1464768213 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c003520 con 0x7f984c13e360 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9852000640 1 -- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 msgr2=0x7f984c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9852000640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9852000640 1 -- 172.21.15.125:0/1464768213 >> v1:172.21.15.137:6789/0 conn(0x7f984c13e360 legacy=0x7f984c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9852000640 1 -- 172.21.15.125:0/1464768213 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c06cd10 con 0x7f984c13df80 2024-01-16T15:23:19.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.400+0000 7f9852a00640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:23:19.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.401+0000 7f9852000640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c15bee0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f98440097a0 tx=0x7f984402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0392d8212dacef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9850c00640 1 -- 172.21.15.125:0/1464768213 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984402d040 con 0x7f984c13df80 2024-01-16T15:23:19.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9850c00640 1 -- 172.21.15.125:0/1464768213 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844031810 con 0x7f984c13df80 2024-01-16T15:23:19.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9850c00640 1 -- 172.21.15.125:0/1464768213 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844031ba0 con 0x7f984c13df80 2024-01-16T15:23:19.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 -- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 msgr2=0x7f984c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c15bee0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f98440097a0 tx=0x7f984402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 -- 172.21.15.125:0/1464768213 shutdown_connections 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c0cf9a0 0x7f984c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 --2- 172.21.15.125:0/1464768213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c15bee0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 -- 172.21.15.125:0/1464768213 >> 172.21.15.125:0/1464768213 conn(0x7f984c06a800 msgr2=0x7f984c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 -- 172.21.15.125:0/1464768213 shutdown_connections 2024-01-16T15:23:19.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.402+0000 7f9854a00640 1 -- 172.21.15.125:0/1464768213 wait complete. 2024-01-16T15:23:19.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.404+0000 7f9854a00640 1 Processor -- start 2024-01-16T15:23:19.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.404+0000 7f9854a00640 1 -- start start 2024-01-16T15:23:19.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c13e360 0x7f984c138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c13ddf0 con 0x7f984c0cf9a0 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c071810 con 0x7f984c13df80 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9854a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c1392a0 con 0x7f984c13e360 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9852000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.405+0000 7f9852000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36978/0 (socket says 172.21.15.125:36978) 2024-01-16T15:23:19.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 -- 172.21.15.125:0/1555766348 learned_addr learned my addr 172.21.15.125:0/1555766348 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c13e360 0x7f984c138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9851600640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 -- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c13e360 msgr2=0x7f984c138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c13e360 0x7f984c138d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 -- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 msgr2=0x7f984c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c0712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 -- 172.21.15.125:0/1555766348 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340094d0 con 0x7f984c0cf9a0 2024-01-16T15:23:19.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f9852000640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f98440098c0 tx=0x7f98440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.406+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984403ad60 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.407+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844044400 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.407+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440445b0 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.407+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9844009400 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.407+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c14abc0 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.408+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c14aa20 con 0x7f984c0cf9a0 2024-01-16T15:23:19.405 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.409+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f984404d440 con 0x7f984c0cf9a0 2024-01-16T15:23:19.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.409+0000 7f984a000640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 0x7f982406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.410+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98440aae30 con 0x7f984c0cf9a0 2024-01-16T15:23:19.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.410+0000 7f9851600640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 0x7f982406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.410+0000 7f9851600640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 0x7f982406ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f984c15b2e0 tx=0x7f983c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.409 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.413+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f984c14aa20 con 0x7f984c0cf9a0 2024-01-16T15:23:19.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.591+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f984c14aa20 con 0x7f984c0cf9a0 2024-01-16T15:23:19.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.594+0000 7f984a000640 1 -- 172.21.15.125:0/1555766348 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f984c14aa20 con 0x7f984c0cf9a0 2024-01-16T15:23:19.590 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:19.590 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:28.491486+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:19.592 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 msgr2=0x7f982406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 0x7f982406ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f984c15b2e0 tx=0x7f983c009040 comp rx=0 tx=0).stop 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 msgr2=0x7f984c13d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f98440098c0 tx=0x7f98440046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 shutdown_connections 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f984c13e360 0x7f984c138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f984c13df80 0x7f984c0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f982406c550 0x7f982406ea10 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.593 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 --2- 172.21.15.125:0/1555766348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f984c0cf9a0 0x7f984c13d480 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 >> 172.21.15.125:0/1555766348 conn(0x7f984c06a800 msgr2=0x7f984c15d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:19.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 shutdown_connections 2024-01-16T15:23:19.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.597+0000 7f9854a00640 1 -- 172.21.15.125:0/1555766348 wait complete. 2024-01-16T15:23:19.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:04.806000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:06.026871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:19.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.817+0000 7f12f6200640 1 Processor -- start 2024-01-16T15:23:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.818+0000 7f12f6200640 1 -- start start 2024-01-16T15:23:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.819+0000 7f12f6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f00cd240 0x7f12f00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.819+0000 7f12f6200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12f009a170 0x7f12f00cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.819+0000 7f12f6200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f006cea0 con 0x7f12f006c6b0 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.819+0000 7f12f6200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f006cbc0 con 0x7f12f009a170 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.819+0000 7f12f6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f006f350 con 0x7f12f00cd240 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12f009a170 0x7f12f00cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12f009a170 0x7f12f00cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36980/0 (socket says 172.21.15.125:36980) 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12e7e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f12f006c6b0 0x7f12f0099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32860/0 (socket says 172.21.15.125:32860) 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 -- 172.21.15.125:0/1532963138 learned_addr learned my addr 172.21.15.125:0/1532963138 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12ef400640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f00cd240 0x7f12f00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 -- 172.21.15.125:0/1532963138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f00cd240 msgr2=0x7f12f00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f00cd240 0x7f12f00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 -- 172.21.15.125:0/1532963138 >> v1:172.21.15.125:6789/0 conn(0x7f12f006c6b0 legacy=0x7f12f0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.820+0000 7f12efe00640 1 -- 172.21.15.125:0/1532963138 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f006f630 con 0x7f12f009a170 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.821+0000 7f12efe00640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f009a170 0x7f12f00cf640 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e50 tx=0x7f12e802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6920a2a016a89654 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.821+0000 7f12eea00640 1 -- 172.21.15.125:0/1532963138 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e802cb00 con 0x7f12f009a170 2024-01-16T15:23:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12eea00640 1 -- 172.21.15.125:0/1532963138 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e802cc60 con 0x7f12f009a170 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12eea00640 1 -- 172.21.15.125:0/1532963138 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80343f0 con 0x7f12f009a170 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 -- 172.21.15.125:0/1532963138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f009a170 msgr2=0x7f12f00cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f009a170 0x7f12f00cf640 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e50 tx=0x7f12e802bf60 comp rx=0 tx=0).stop 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 -- 172.21.15.125:0/1532963138 shutdown_connections 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f009a170 0x7f12f00cf640 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 --2- 172.21.15.125:0/1532963138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f00cd240 0x7f12f00b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 -- 172.21.15.125:0/1532963138 >> 172.21.15.125:0/1532963138 conn(0x7f12f00b2d00 msgr2=0x7f12f00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 -- 172.21.15.125:0/1532963138 shutdown_connections 2024-01-16T15:23:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.822+0000 7f12f6200640 1 -- 172.21.15.125:0/1532963138 wait complete. 2024-01-16T15:23:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.825+0000 7f12f6200640 1 Processor -- start 2024-01-16T15:23:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.825+0000 7f12f6200640 1 -- start start 2024-01-16T15:23:19.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f009a170 0x7f12f00c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 0x7f12f00c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.823 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0151c80 con 0x7f12f006c6b0 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c6190 con 0x7f12f00cd240 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12f6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00b41e0 con 0x7f12f009a170 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12ef400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12ef400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36982/0 (socket says 172.21.15.125:36982) 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.826+0000 7f12ef400640 1 -- 172.21.15.125:0/2674789592 learned_addr learned my addr 172.21.15.125:0/2674789592 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12e7e00640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f009a170 0x7f12f00c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12efe00640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 0x7f12f00c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 -- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f009a170 msgr2=0x7f12f00c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f009a170 0x7f12f00c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 -- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 msgr2=0x7f12f00c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 0x7f12f00c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 -- 172.21.15.125:0/2674789592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8009b30 con 0x7f12f006c6b0 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12efe00640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 0x7f12f00c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12ef400640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d800 tx=0x7f12e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.827+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000b6c0 con 0x7f12f006c6b0 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.828+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e000b820 con 0x7f12f006c6b0 2024-01-16T15:23:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.828+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000bbb0 con 0x7f12f006c6b0 2024-01-16T15:23:19.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.828+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f00b44c0 con 0x7f12f006c6b0 2024-01-16T15:23:19.826 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.828+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f00b4a80 con 0x7f12f006c6b0 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.829+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f00b4c40 con 0x7f12f006c6b0 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.830+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f12e000bd10 con 0x7f12f006c6b0 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.831+0000 7f12e7400640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 0x7f12c806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.831+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12e0089060 con 0x7f12f006c6b0 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.831+0000 7f12e7e00640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 0x7f12c806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.827 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.832+0000 7f12e7e00640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 0x7f12c806ea40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f12f01f63c0 tx=0x7f12d8007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.830 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:19.835+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12e0055600 con 0x7f12f006c6b0 2024-01-16T15:23:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.026+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12f0073d90 con 0x7f12f006c6b0 2024-01-16T15:23:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.027+0000 7f12e7400640 1 -- 172.21.15.125:0/2674789592 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12e0058c20 con 0x7f12f006c6b0 2024-01-16T15:23:20.023 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 msgr2=0x7f12c806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 0x7f12c806ea40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f12f01f63c0 tx=0x7f12d8007c80 comp rx=0 tx=0).stop 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 msgr2=0x7f12f0151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d800 tx=0x7f12e000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 shutdown_connections 2024-01-16T15:23:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12f00cd240 0x7f12f00c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12f009a170 0x7f12f00c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f12c806c580 0x7f12c806ea40 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 --2- 172.21.15.125:0/2674789592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f006c6b0 0x7f12f0151740 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 >> 172.21.15.125:0/2674789592 conn(0x7f12f00b2d00 msgr2=0x7f12f00b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 shutdown_connections 2024-01-16T15:23:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.030+0000 7f12f6200640 1 -- 172.21.15.125:0/2674789592 wait complete. 2024-01-16T15:23:20.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.257+0000 7f2786000640 1 Processor -- start 2024-01-16T15:23:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.258+0000 7f2786000640 1 -- start start 2024-01-16T15:23:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.259+0000 7f2786000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2780160f80 0x7f2780161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.259+0000 7f2786000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f27800ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.259+0000 7f2786000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27801618a0 con 0x7f27800cfbe0 2024-01-16T15:23:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.259+0000 7f2786000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27800d0de0 con 0x7f2780160f80 2024-01-16T15:23:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.259+0000 7f2786000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27800d10c0 con 0x7f27800ce8f0 2024-01-16T15:23:20.257 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277fe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f27800cfbe0 0x7f27800d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32864/0 (socket says 172.21.15.125:32864) 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277fe00640 1 -- 172.21.15.125:0/2049654831 learned_addr learned my addr 172.21.15.125:0/2049654831 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277f400640 1 --2- 172.21.15.125:0/2049654831 >> v2:172.21.15.137:3300/0 conn(0x7f2780160f80 0x7f2780161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277ea00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f27800ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526511837 0 0) 0x7f27801618a0 con 0x7f27800cfbe0 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2760003520 con 0x7f27800cfbe0 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999718694 0 0) 0x7f2760003520 con 0x7f27800cfbe0 2024-01-16T15:23:20.258 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.260+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 msgr2=0x7f27800ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f27800ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 >> v2:172.21.15.137:3300/0 conn(0x7f2780160f80 msgr2=0x7f2780161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 --2- 172.21.15.125:0/2049654831 >> v2:172.21.15.137:3300/0 conn(0x7f2780160f80 0x7f2780161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800d13a0 con 0x7f27800cfbe0 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f2778001670 con 0x7f27800cfbe0 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 >> v1:172.21.15.125:6789/0 conn(0x7f27800cfbe0 legacy=0x7f27800d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f2760003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2760003ef0 0x7f27600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f2760005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800d0de0 con 0x7f2760004d00 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800d10c0 con 0x7f2760003ef0 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276000b980 con 0x7f27800ce8f0 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277fe00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f2760005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277f400640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f2760003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277ea00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2760003ef0 0x7f27600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277fe00640 1 -- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 msgr2=0x7f2760003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:20.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.261+0000 7f277fe00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27800ce8f0 0x7f2760003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.262+0000 7f277fe00640 1 -- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2760003ef0 msgr2=0x7f27600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:20.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.262+0000 7f277fe00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2760003ef0 0x7f27600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.262+0000 7f277fe00640 1 -- 172.21.15.125:0/2049654831 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760003520 con 0x7f2760004d00 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f277fe00640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f2760005180 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f277800c240 tx=0x7f27600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d429ffb0d1fee648 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277802cae0 con 0x7f2760004d00 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277802cc40 con 0x7f2760004d00 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f277e000640 1 -- 172.21.15.125:0/2049654831 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778005af0 con 0x7f2760004d00 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f2786000640 1 -- 172.21.15.125:0/2049654831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 msgr2=0x7f2760005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f2786000640 1 --2- 172.21.15.125:0/2049654831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f2760005180 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f277800c240 tx=0x7f27600057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.263+0000 7f277fe00640 1 -- 172.21.15.125:0/2049654831 reap_dead start 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.264+0000 7f2786000640 1 -- 172.21.15.125:0/2049654831 shutdown_connections 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.264+0000 7f2786000640 1 -- 172.21.15.125:0/2049654831 >> 172.21.15.125:0/2049654831 conn(0x7f278009c550 msgr2=0x7f27800d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.264+0000 7f2786000640 1 -- 172.21.15.125:0/2049654831 shutdown_connections 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.264+0000 7f2786000640 1 -- 172.21.15.125:0/2049654831 wait complete. 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.265+0000 7f2786000640 1 Processor -- start 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.266+0000 7f2786000640 1 -- start start 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.266+0000 7f2786000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.266+0000 7f2786000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27801600f0 0x7f278015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.266+0000 7f2786000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 0x7f278015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f2786000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278015bb30 con 0x7f2760004d00 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f2786000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800b8aa0 con 0x7f278015b170 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f2786000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2780160570 con 0x7f27801600f0 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37000/0 (socket says 172.21.15.125:37000) 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 -- 172.21.15.125:0/3505297933 learned_addr learned my addr 172.21.15.125:0/3505297933 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277ea00640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27801600f0 0x7f278015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277fe00640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 0x7f278015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 -- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27801600f0 msgr2=0x7f278015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27801600f0 0x7f278015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 -- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 msgr2=0x7f278015b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 0x7f278015b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277f400640 1 -- 172.21.15.125:0/3505297933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800b8f30 con 0x7f2760004d00 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.267+0000 7f277fe00640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 0x7f278015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f277f400640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f2770007f30 tx=0x7f2770011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000e030 con 0x7f2760004d00 2024-01-16T15:23:20.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770016470 con 0x7f2760004d00 2024-01-16T15:23:20.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770016820 con 0x7f2760004d00 2024-01-16T15:23:20.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27800b7e90 con 0x7f2760004d00 2024-01-16T15:23:20.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.268+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27800b8450 con 0x7f2760004d00 2024-01-16T15:23:20.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.269+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800675d0 con 0x7f2760004d00 2024-01-16T15:23:20.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.270+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2770016980 con 0x7f2760004d00 2024-01-16T15:23:20.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.271+0000 7f2777400640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 0x7f275806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.271+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f277000f070 con 0x7f2760004d00 2024-01-16T15:23:20.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.271+0000 7f277ea00640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 0x7f275806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.270 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.272+0000 7f277ea00640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 0x7f275806ea10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f27800cccd0 tx=0x7f2768008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.271 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.276+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2770056110 con 0x7f2760004d00 2024-01-16T15:23:20.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.469+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2780067b60 con 0x7f2760004d00 2024-01-16T15:23:20.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.470+0000 7f2777400640 1 -- 172.21.15.125:0/3505297933 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2770059730 con 0x7f2760004d00 2024-01-16T15:23:20.465 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T15:23:20.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.473+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 msgr2=0x7f275806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.473+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 0x7f275806ea10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f27800cccd0 tx=0x7f2768008040 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.473+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 msgr2=0x7f27800b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.473+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f2770007f30 tx=0x7f2770011660 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 shutdown_connections 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f278015b170 0x7f278015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27801600f0 0x7f278015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f275806c550 0x7f275806ea10 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 --2- 172.21.15.125:0/3505297933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2760004d00 0x7f27800b77e0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 >> 172.21.15.125:0/3505297933 conn(0x7f278009c550 msgr2=0x7f27800b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 shutdown_connections 2024-01-16T15:23:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.474+0000 7f2786000640 1 -- 172.21.15.125:0/3505297933 wait complete. 2024-01-16T15:23:20.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:20.699 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.703+0000 7f46e5200640 1 Processor -- start 2024-01-16T15:23:20.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.704+0000 7f46e5200640 1 -- start start 2024-01-16T15:23:20.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.705+0000 7f46e5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f46e0143ae0 0x7f46e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.705+0000 7f46e5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.705+0000 7f46e5200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00cfc30 con 0x7f46e015e1e0 2024-01-16T15:23:20.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.705+0000 7f46e5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e00b1ee0 con 0x7f46e0143ae0 2024-01-16T15:23:20.701 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.705+0000 7f46e5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00b2780 con 0x7f46e00cf7d0 2024-01-16T15:23:20.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46d5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46dde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f46e0143ae0 0x7f46e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46d5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:60862/0 (socket says 172.21.15.125:60862) 2024-01-16T15:23:20.702 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46d5200640 1 -- 172.21.15.125:0/164263065 learned_addr learned my addr 172.21.15.125:0/164263065 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46dde00640 1 -- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 msgr2=0x7f46e015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.706+0000 7f46dde00640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.707+0000 7f46dde00640 1 -- 172.21.15.125:0/164263065 >> v1:172.21.15.137:6789/0 conn(0x7f46e015e1e0 legacy=0x7f46e015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.707+0000 7f46dde00640 1 -- 172.21.15.125:0/164263065 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e015e990 con 0x7f46e0143ae0 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.707+0000 7f46d5200640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:23:20.703 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.708+0000 7f46dde00640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e0143ae0 0x7f46e00cf290 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009d90 tx=0x7f46d002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803621abc2e2cd21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46dd400640 1 -- 172.21.15.125:0/164263065 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d002cc60 con 0x7f46e0143ae0 2024-01-16T15:23:20.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46dd400640 1 -- 172.21.15.125:0/164263065 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d002cdc0 con 0x7f46e0143ae0 2024-01-16T15:23:20.704 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46dd400640 1 -- 172.21.15.125:0/164263065 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0034660 con 0x7f46e0143ae0 2024-01-16T15:23:20.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 -- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e0143ae0 msgr2=0x7f46e00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e0143ae0 0x7f46e00cf290 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009d90 tx=0x7f46d002beb0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 -- 172.21.15.125:0/164263065 shutdown_connections 2024-01-16T15:23:20.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e00cf7d0 0x7f46e015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 --2- 172.21.15.125:0/164263065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e0143ae0 0x7f46e00cf290 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 -- 172.21.15.125:0/164263065 >> 172.21.15.125:0/164263065 conn(0x7f46e006a800 msgr2=0x7f46e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:20.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.709+0000 7f46e5200640 1 -- 172.21.15.125:0/164263065 shutdown_connections 2024-01-16T15:23:20.706 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.710+0000 7f46e5200640 1 -- 172.21.15.125:0/164263065 wait complete. 2024-01-16T15:23:20.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.711+0000 7f46e5200640 1 Processor -- start 2024-01-16T15:23:20.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.711+0000 7f46e5200640 1 -- start start 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e015e1e0 0x7f46e01e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 0x7f46e015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e015c1d0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00cab00 con 0x7f46e015b7e0 2024-01-16T15:23:20.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.712+0000 7f46e5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01429a0 con 0x7f46e015e1e0 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37012/0 (socket says 172.21.15.125:37012) 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 -- 172.21.15.125:0/3003177219 learned_addr learned my addr 172.21.15.125:0/3003177219 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46de800640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 0x7f46e015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 -- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e015e1e0 msgr2=0x7f46e01e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e015e1e0 0x7f46e01e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 -- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 msgr2=0x7f46e015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 0x7f46e015bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46dde00640 1 -- 172.21.15.125:0/3003177219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c80094d0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.713+0000 7f46de800640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 0x7f46e015bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46dde00640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f46d002c440 tx=0x7f46d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0040650 con 0x7f46e00cf7d0 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d00407d0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0040b80 con 0x7f46e00cf7d0 2024-01-16T15:23:20.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d0009a70 con 0x7f46e00cf7d0 2024-01-16T15:23:20.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.714+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01e69b0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.715+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e00675d0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.716+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f46d0040ce0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.717+0000 7f46d6a00640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 0x7f46b806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.717+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f46d00ab8b0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.717+0000 7f46d5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 0x7f46b806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:20.714 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.718+0000 7f46d5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 0x7f46b806ea10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f46c8006f40 tx=0x7f46c8006e90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:20.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.721+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46d0077df0 con 0x7f46e00cf7d0 2024-01-16T15:23:20.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.912+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46e0072230 con 0x7f46e00cf7d0 2024-01-16T15:23:20.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.913+0000 7f46d6a00640 1 -- 172.21.15.125:0/3003177219 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46d0039020 con 0x7f46e00cf7d0 2024-01-16T15:23:20.909 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T15:23:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.915+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 msgr2=0x7f46b806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.915+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 0x7f46b806ea10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f46c8006f40 tx=0x7f46c8006e90 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.915+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 msgr2=0x7f46e0142460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.915+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f46d002c440 tx=0x7f46d0004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 shutdown_connections 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46e015b7e0 0x7f46e015bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46e015e1e0 0x7f46e01e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f46b806c550 0x7f46b806ea10 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 --2- 172.21.15.125:0/3003177219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e00cf7d0 0x7f46e0142460 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 >> 172.21.15.125:0/3003177219 conn(0x7f46e006a800 msgr2=0x7f46e0070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:20.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 shutdown_connections 2024-01-16T15:23:20.913 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:20.916+0000 7f46e5200640 1 -- 172.21.15.125:0/3003177219 wait complete. 2024-01-16T15:23:20.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.147+0000 7fc57d800640 1 Processor -- start 2024-01-16T15:23:21.143 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.147+0000 7fc57d800640 1 -- start start 2024-01-16T15:23:21.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.148+0000 7fc57d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc578143ae0 0x7fc5780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.148+0000 7fc57d800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc5780cf7d0 0x7fc57815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.148+0000 7fc57d800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5780cfc30 con 0x7fc57815e1e0 2024-01-16T15:23:21.144 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.148+0000 7fc57d800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5780b1ee0 con 0x7fc5780cf7d0 2024-01-16T15:23:21.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.148+0000 7fc57d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780b2780 con 0x7fc578143ae0 2024-01-16T15:23:21.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc577400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc57815e1e0 0x7fc57815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32870/0 (socket says 172.21.15.125:32870) 2024-01-16T15:23:21.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc577400640 1 -- 172.21.15.125:0/879457193 learned_addr learned my addr 172.21.15.125:0/879457193 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:21.145 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576000640 1 --2- 172.21.15.125:0/879457193 >> v2:172.21.15.125:3300/0 conn(0x7fc5780cf7d0 0x7fc57815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576a00640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc578143ae0 0x7fc5780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc575600640 1 -- 172.21.15.125:0/879457193 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302668745 0 0) 0x7fc5780cfc30 con 0x7fc57815e1e0 2024-01-16T15:23:21.146 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc575600640 1 -- 172.21.15.125:0/879457193 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc558003520 con 0x7fc57815e1e0 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576000640 1 -- 172.21.15.125:0/879457193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc578143ae0 msgr2=0x7fc5780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576000640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc578143ae0 0x7fc5780cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576000640 1 -- 172.21.15.125:0/879457193 >> v1:172.21.15.125:6789/0 conn(0x7fc57815e1e0 legacy=0x7fc57815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.149+0000 7fc576000640 1 -- 172.21.15.125:0/879457193 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57815e990 con 0x7fc5780cf7d0 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.150+0000 7fc576000640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5780cf7d0 0x7fc57815dca0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fc5600097a0 tx=0x7fc56002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ee2f932f33fb53 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc575600640 1 -- 172.21.15.125:0/879457193 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560031830 con 0x7fc5780cf7d0 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc575600640 1 -- 172.21.15.125:0/879457193 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560031990 con 0x7fc5780cf7d0 2024-01-16T15:23:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc575600640 1 -- 172.21.15.125:0/879457193 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560031d20 con 0x7fc5780cf7d0 2024-01-16T15:23:21.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 -- 172.21.15.125:0/879457193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5780cf7d0 msgr2=0x7fc57815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5780cf7d0 0x7fc57815dca0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fc5600097a0 tx=0x7fc56002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.148 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 -- 172.21.15.125:0/879457193 shutdown_connections 2024-01-16T15:23:21.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5780cf7d0 0x7fc57815dca0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 --2- 172.21.15.125:0/879457193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc578143ae0 0x7fc5780cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 -- 172.21.15.125:0/879457193 >> 172.21.15.125:0/879457193 conn(0x7fc57806a800 msgr2=0x7fc57806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:21.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 -- 172.21.15.125:0/879457193 shutdown_connections 2024-01-16T15:23:21.149 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.151+0000 7fc57d800640 1 -- 172.21.15.125:0/879457193 wait complete. 2024-01-16T15:23:21.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.153+0000 7fc57d800640 1 Processor -- start 2024-01-16T15:23:21.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.153+0000 7fc57d800640 1 -- start start 2024-01-16T15:23:21.151 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 0x7fc578156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc578143ae0 0x7fc578141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780cc610 con 0x7fc57815e1e0 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc578157fc0 con 0x7fc578143ae0 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.154+0000 7fc57d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781573b0 con 0x7fc5780cf7d0 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37016/0 (socket says 172.21.15.125:37016) 2024-01-16T15:23:21.152 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 -- 172.21.15.125:0/300374100 learned_addr learned my addr 172.21.15.125:0/300374100 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc576a00640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 0x7fc578156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc576000640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc578143ae0 0x7fc578141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 -- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 msgr2=0x7fc578156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 0x7fc578156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 -- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc578143ae0 msgr2=0x7fc578141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc578143ae0 0x7fc578141d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 -- 172.21.15.125:0/300374100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560009400 con 0x7fc57815e1e0 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc576a00640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 0x7fc578156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:21.153 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.155+0000 7fc577400640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fc57000caa0 tx=0x7fc57000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.156+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000ac00 con 0x7fc57815e1e0 2024-01-16T15:23:21.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.156+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57000ad60 con 0x7fc57815e1e0 2024-01-16T15:23:21.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.156+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc578157690 con 0x7fc57815e1e0 2024-01-16T15:23:21.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.156+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570012620 con 0x7fc57815e1e0 2024-01-16T15:23:21.154 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.156+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc578157c50 con 0x7fc57815e1e0 2024-01-16T15:23:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.157+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5780675d0 con 0x7fc57815e1e0 2024-01-16T15:23:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.158+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc570007070 con 0x7fc57815e1e0 2024-01-16T15:23:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.159+0000 7fc56ea00640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 0x7fc55006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.159+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc570088ae0 con 0x7fc57815e1e0 2024-01-16T15:23:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.159+0000 7fc576a00640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 0x7fc55006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.156 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.160+0000 7fc576a00640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 0x7fc55006ea40 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc568006fd0 tx=0x7fc568008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.158 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.163+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc570055080 con 0x7fc57815e1e0 2024-01-16T15:23:21.361 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.365+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc57813c880 con 0x7fc57815e1e0 2024-01-16T15:23:21.362 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.366+0000 7fc56ea00640 1 -- 172.21.15.125:0/300374100 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc570017020 con 0x7fc57815e1e0 2024-01-16T15:23:21.363 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T15:23:21.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 msgr2=0x7fc55006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 0x7fc55006ea40 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc568006fd0 tx=0x7fc568008040 comp rx=0 tx=0).stop 2024-01-16T15:23:21.366 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 msgr2=0x7fc578142260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fc57000caa0 tx=0x7fc57000cf60 comp rx=0 tx=0).stop 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 shutdown_connections 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc55006c580 0x7fc55006ea40 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc57815e1e0 0x7fc578142260 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc578143ae0 0x7fc578141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 --2- 172.21.15.125:0/300374100 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5780cf7d0 0x7fc578156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 >> 172.21.15.125:0/300374100 conn(0x7fc57806a800 msgr2=0x7fc57813d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:21.367 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 shutdown_connections 2024-01-16T15:23:21.368 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.369+0000 7fc57d800640 1 -- 172.21.15.125:0/300374100 wait complete. 2024-01-16T15:23:21.384 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:23:21.384 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:21.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:21.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.655+0000 7fdcc6800640 1 Processor -- start 2024-01-16T15:23:21.651 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.655+0000 7fdcc6800640 1 -- start start 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 0x7fdcc015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 0x7fdcc00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc006c470 con 0x7fdcc015f9c0 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc006c750 con 0x7fdcc013e360 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcc6800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc006ca30 con 0x7fdcc013df80 2024-01-16T15:23:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcbfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcbfe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37030/0 (socket says 172.21.15.125:37030) 2024-01-16T15:23:21.654 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.657+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2146193863 learned_addr learned my addr 172.21.15.125:0/2146193863 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbf400640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 0x7fdcc015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcb6800640 1 --2- 172.21.15.125:0/2146193863 >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 0x7fdcc00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2146193863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 msgr2=0x7fdcc015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 0x7fdcc015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2146193863 >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 msgr2=0x7fdcc00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2146193863 >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 0x7fdcc00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2146193863 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc006cd10 con 0x7fdcc015f9c0 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcbf400640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 0x7fdcc015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.658+0000 7fdcb6800640 1 --2- 172.21.15.125:0/2146193863 >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 0x7fdcc00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.659+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00cfed0 tx=0x7fdcb802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba65eb0188ba7f52 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.659+0000 7fdcbea00640 1 -- 172.21.15.125:0/2146193863 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8031880 con 0x7fdcc015f9c0 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.659+0000 7fdcbea00640 1 -- 172.21.15.125:0/2146193863 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb80319e0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.659+0000 7fdcbea00640 1 -- 172.21.15.125:0/2146193863 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8031d70 con 0x7fdcc015f9c0 2024-01-16T15:23:21.657 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 -- 172.21.15.125:0/2146193863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 msgr2=0x7fdcc00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00cfed0 tx=0x7fdcb802b920 comp rx=0 tx=0).stop 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 -- 172.21.15.125:0/2146193863 shutdown_connections 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00cf7d0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2146193863 >> v2:172.21.15.137:3300/0 conn(0x7fdcc013e360 0x7fdcc00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2146193863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013df80 0x7fdcc015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 -- 172.21.15.125:0/2146193863 >> 172.21.15.125:0/2146193863 conn(0x7fdcc006a800 msgr2=0x7fdcc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 -- 172.21.15.125:0/2146193863 shutdown_connections 2024-01-16T15:23:21.658 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.660+0000 7fdcc6800640 1 -- 172.21.15.125:0/2146193863 wait complete. 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.662+0000 7fdcc6800640 1 Processor -- start 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.662+0000 7fdcc6800640 1 -- start start 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdcc013df80 0x7fdcc00cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013e360 0x7fdcc00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01f6270 con 0x7fdcc015f9c0 2024-01-16T15:23:21.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01f64c0 con 0x7fdcc013df80 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.663+0000 7fdcc6800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01f67a0 con 0x7fdcc013e360 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37038/0 (socket says 172.21.15.125:37038) 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2458564176 learned_addr learned my addr 172.21.15.125:0/2458564176 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcb6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013e360 0x7fdcc00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbf400640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdcc013df80 0x7fdcc00cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013e360 msgr2=0x7fdcc00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013e360 0x7fdcc00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdcc013df80 msgr2=0x7fdcc00cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdcc013df80 0x7fdcc00cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.664+0000 7fdcbfe00640 1 -- 172.21.15.125:0/2458564176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80094d0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcbfe00640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdcb802b960 tx=0x7fdcb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8031810 con 0x7fdcc015f9c0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb80443f0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc01f6a80 con 0x7fdcc015f9c0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80447a0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.663 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.665+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc01f7040 con 0x7fdcc015f9c0 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.666+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc00702c0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.667+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fdcb8044900 con 0x7fdcc015f9c0 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.668+0000 7fdcb7e00640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 0x7fdc9806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.668+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdcb80ab440 con 0x7fdcc015f9c0 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.668+0000 7fdcbf400640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 0x7fdc9806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.669+0000 7fdcbf400640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 0x7fdc9806ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0007510 tx=0x7fdcb00074a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.667 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.672+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdcc00702c0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.857 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.861+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc00702c0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.864+0000 7fdcb7e00640 1 -- 172.21.15.125:0/2458564176 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16668 (secure 0 0 0) 0x7fdcc00702c0 con 0x7fdcc015f9c0 2024-01-16T15:23:21.860 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:21.860 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:28.491486+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 msgr2=0x7fdc9806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 0x7fdc9806ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0007510 tx=0x7fdcb00074a0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 msgr2=0x7fdcc00b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdcb802b960 tx=0x7fdcb80046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 shutdown_connections 2024-01-16T15:23:21.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdc9806c550 0x7fdc9806ea10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdcc015f9c0 0x7fdcc00b3310 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdcc013e360 0x7fdcc00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 --2- 172.21.15.125:0/2458564176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdcc013df80 0x7fdcc00cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 >> 172.21.15.125:0/2458564176 conn(0x7fdcc006a800 msgr2=0x7fdcc015e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 shutdown_connections 2024-01-16T15:23:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:21.867+0000 7fdcc6800640 1 -- 172.21.15.125:0/2458564176 wait complete. 2024-01-16T15:23:21.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:56:33.985088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:04.806000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:06.026871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:21.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:22.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.086+0000 7f5869800640 1 Processor -- start 2024-01-16T15:23:22.082 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.086+0000 7f5869800640 1 -- start start 2024-01-16T15:23:22.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5869800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f58640cf9a0 0x7f586415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5869800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58640b1ee0 con 0x7f586415e1e0 2024-01-16T15:23:22.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5869800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58640b2780 con 0x7f5864143ae0 2024-01-16T15:23:22.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5869800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586415e990 con 0x7f58640cf9a0 2024-01-16T15:23:22.086 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5863400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f586415e1e0 0x7f58640702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32874/0 (socket says 172.21.15.125:32874) 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5863400640 1 -- 172.21.15.125:0/3188843026 learned_addr learned my addr 172.21.15.125:0/3188843026 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5859e00640 1 --2- 172.21.15.125:0/3188843026 >> v2:172.21.15.137:3300/0 conn(0x7f58640cf9a0 0x7f586415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733057778 0 0) 0x7f58640b1ee0 con 0x7f586415e1e0 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.088+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5844003520 con 0x7f586415e1e0 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973715302 0 0) 0x7f5844003520 con 0x7f586415e1e0 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 >> v2:172.21.15.137:3300/0 conn(0x7f58640cf9a0 msgr2=0x7f586415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 --2- 172.21.15.125:0/3188843026 >> v2:172.21.15.137:3300/0 conn(0x7f58640cf9a0 0x7f586415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.087 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 >> v1:172.21.15.137:6789/0 conn(0x7f5864143ae0 legacy=0x7f58640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58640b2130 con 0x7f586415e1e0 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f585c0019f0 con 0x7f586415e1e0 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 >> v1:172.21.15.125:6789/0 conn(0x7f586415e1e0 legacy=0x7f58640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58440039c0 0x7f5844003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 0x7f5844004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5844005180 0x7f584400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586415e990 con 0x7f58440047c0 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584400c080 con 0x7f5844005180 2024-01-16T15:23:22.088 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584400c240 con 0x7f58440039c0 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.089+0000 7f5859e00640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 0x7f5844004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5862a00640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58440039c0 0x7f5844003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5863400640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5844005180 0x7f584400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5859e00640 1 -- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58440039c0 msgr2=0x7f5844003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5859e00640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58440039c0 0x7f5844003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5859e00640 1 -- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5844005180 msgr2=0x7f584400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5859e00640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5844005180 0x7f584400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5859e00640 1 -- 172.21.15.125:0/3188843026 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5844003520 con 0x7f58440047c0 2024-01-16T15:23:22.090 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.090+0000 7f5863400640 1 -- 172.21.15.125:0/3188843026 reap_dead start 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.091+0000 7f5859e00640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 0x7f5844004c40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f584c00bd60 tx=0x7f58440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60a1dba8c267373 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.091+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c0306e0 con 0x7f58440047c0 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.091+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c030840 con 0x7f58440047c0 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.091+0000 7f5862000640 1 -- 172.21.15.125:0/3188843026 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c030bd0 con 0x7f58440047c0 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 -- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 msgr2=0x7f5844004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 0x7f5844004c40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f584c00bd60 tx=0x7f58440057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 -- 172.21.15.125:0/3188843026 shutdown_connections 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 --2- 172.21.15.125:0/3188843026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440047c0 0x7f5844004c40 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 -- 172.21.15.125:0/3188843026 >> 172.21.15.125:0/3188843026 conn(0x7f586406a800 msgr2=0x7f586406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 -- 172.21.15.125:0/3188843026 shutdown_connections 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.092+0000 7f5869800640 1 -- 172.21.15.125:0/3188843026 wait complete. 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.093+0000 7f5869800640 1 Processor -- start 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.094+0000 7f5869800640 1 -- start start 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.094+0000 7f5869800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58440047c0 0x7f58641597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.094+0000 7f5869800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58640cd730 0x7f58640cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.094+0000 7f5869800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5869800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58640ce0f0 con 0x7f58640aadb0 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5869800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586415d2c0 con 0x7f58440047c0 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5869800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586415c220 con 0x7f58640cd730 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37046/0 (socket says 172.21.15.125:37046) 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 -- 172.21.15.125:0/3867354906 learned_addr learned my addr 172.21.15.125:0/3867354906 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:22.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 -- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58640cd730 msgr2=0x7f58640cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58640cd730 0x7f58640cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 -- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58440047c0 msgr2=0x7f58641597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5862a00640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58440047c0 0x7f58641597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58440047c0 0x7f58641597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 -- 172.21.15.125:0/3867354906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586415c500 con 0x7f58640aadb0 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.095+0000 7f5863400640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f58640cc3c0 tx=0x7f585c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.096+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c00ead0 con 0x7f58640aadb0 2024-01-16T15:23:22.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.096+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585c00ec30 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.096+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586415c7e0 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.096+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c014430 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.096+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586415cda0 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.097+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58640675d0 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.098+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f585c014590 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.099+0000 7f585b400640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 0x7f583c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.099+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f585c0888b0 con 0x7f58640aadb0 2024-01-16T15:23:22.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.100+0000 7f5862a00640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 0x7f583c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.100+0000 7f5862a00640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 0x7f583c06ea40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f58640695b0 tx=0x7f5854009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.104+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f585c054df0 con 0x7f58640aadb0 2024-01-16T15:23:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.279+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f586415cf60 con 0x7f58640aadb0 2024-01-16T15:23:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.280+0000 7f585b400640 1 -- 172.21.15.125:0/3867354906 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f585c058410 con 0x7f58640aadb0 2024-01-16T15:23:22.276 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 msgr2=0x7f583c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 0x7f583c06ea40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f58640695b0 tx=0x7f5854009040 comp rx=0 tx=0).stop 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 msgr2=0x7f58640ab260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f58640cc3c0 tx=0x7f585c007bc0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 shutdown_connections 2024-01-16T15:23:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f583c06c580 0x7f583c06ea40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58640aadb0 0x7f58640ab260 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58640cd730 0x7f58640cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 --2- 172.21.15.125:0/3867354906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58440047c0 0x7f58641597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 >> 172.21.15.125:0/3867354906 conn(0x7f586406a800 msgr2=0x7f58640cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 shutdown_connections 2024-01-16T15:23:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.283+0000 7f5869800640 1 -- 172.21.15.125:0/3867354906 wait complete. 2024-01-16T15:23:22.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.511+0000 7ff397e00640 1 Processor -- start 2024-01-16T15:23:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.511+0000 7ff397e00640 1 -- start start 2024-01-16T15:23:22.508 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff397e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff3980b35a0 0x7ff3980cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff397e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980ce440 con 0x7ff39813cf10 2024-01-16T15:23:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff397e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980ce720 con 0x7ff39813fcb0 2024-01-16T15:23:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff397e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3980cea00 con 0x7ff3980b35a0 2024-01-16T15:23:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff396a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff39813cf10 0x7ff39813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32876/0 (socket says 172.21.15.125:32876) 2024-01-16T15:23:22.509 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff396a00640 1 -- 172.21.15.125:0/4173581146 learned_addr learned my addr 172.21.15.125:0/4173581146 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453758713 0 0) 0x7ff3980ce440 con 0x7ff39813cf10 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff378003520 con 0x7ff39813cf10 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.513+0000 7ff397400640 1 --2- 172.21.15.125:0/4173581146 >> v2:172.21.15.137:3300/0 conn(0x7ff3980b35a0 0x7ff3980cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674052057 0 0) 0x7ff378003520 con 0x7ff39813cf10 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 >> v2:172.21.15.137:3300/0 conn(0x7ff3980b35a0 msgr2=0x7ff3980cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 --2- 172.21.15.125:0/4173581146 >> v2:172.21.15.137:3300/0 conn(0x7ff3980b35a0 0x7ff3980cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 >> v1:172.21.15.137:6789/0 conn(0x7ff39813fcb0 legacy=0x7ff3980cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.513 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980b1d90 con 0x7ff39813cf10 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7ff388001670 con 0x7ff39813cf10 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 >> v1:172.21.15.125:6789/0 conn(0x7ff39813cf10 legacy=0x7ff39813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3780047c0 0x7ff378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780051d0 0x7ff37800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980cea00 con 0x7ff3780039c0 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37800c120 con 0x7ff3780047c0 2024-01-16T15:23:22.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37800e4f0 con 0x7ff3780051d0 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff396a00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff38fe00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3780047c0 0x7ff378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.514+0000 7ff397400640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780051d0 0x7ff37800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff396a00640 1 -- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780051d0 msgr2=0x7ff37800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff396a00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780051d0 0x7ff37800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff396a00640 1 -- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3780047c0 msgr2=0x7ff378004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff396a00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3780047c0 0x7ff378004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff396a00640 1 -- 172.21.15.125:0/4173581146 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378003520 con 0x7ff3780039c0 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.515+0000 7ff397400640 1 -- 172.21.15.125:0/4173581146 reap_dead start 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff396a00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 0x7ff378003de0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7ff38800c0e0 tx=0x7ff3780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41a7e831e6860cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388005ad0 con 0x7ff3780039c0 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388005c50 con 0x7ff3780039c0 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff396000640 1 -- 172.21.15.125:0/4173581146 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38802c7c0 con 0x7ff3780039c0 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff397e00640 1 -- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 msgr2=0x7ff378003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.516+0000 7ff397e00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 0x7ff378003de0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7ff38800c0e0 tx=0x7ff3780057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.517+0000 7ff397e00640 1 -- 172.21.15.125:0/4173581146 shutdown_connections 2024-01-16T15:23:22.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.517+0000 7ff397e00640 1 --2- 172.21.15.125:0/4173581146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.517+0000 7ff397e00640 1 -- 172.21.15.125:0/4173581146 >> 172.21.15.125:0/4173581146 conn(0x7ff39806a7d0 msgr2=0x7ff39806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:22.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.517+0000 7ff397e00640 1 -- 172.21.15.125:0/4173581146 shutdown_connections 2024-01-16T15:23:22.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.517+0000 7ff397e00640 1 -- 172.21.15.125:0/4173581146 wait complete. 2024-01-16T15:23:22.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.518+0000 7ff397e00640 1 Processor -- start 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.518+0000 7ff397e00640 1 -- start start 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780039c0 0x7ff3981f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff398159490 0x7ff39813e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39814fbd0 con 0x7ff39813b750 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39814fe70 con 0x7ff398159490 2024-01-16T15:23:22.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.519+0000 7ff397e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff398150150 con 0x7ff3780039c0 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37070/0 (socket says 172.21.15.125:37070) 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 -- 172.21.15.125:0/2713624865 learned_addr learned my addr 172.21.15.125:0/2713624865 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 -- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780039c0 msgr2=0x7ff3981f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780039c0 0x7ff3981f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 -- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff398159490 msgr2=0x7ff39813e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff398159490 0x7ff39813e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 -- 172.21.15.125:0/2713624865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff398150430 con 0x7ff39813b750 2024-01-16T15:23:22.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397400640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7ff3980ae9b0 tx=0x7ff390009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390007db0 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39000f050 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390014650 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.520+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39813ed60 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.521+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39813bc00 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.522+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39813be80 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.523+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff3900147b0 con 0x7ff39813b750 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.524+0000 7ff38f400640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 0x7ff37006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.524+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff390089230 con 0x7ff39813b750 2024-01-16T15:23:22.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.524+0000 7ff396a00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 0x7ff37006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.525+0000 7ff396a00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 0x7ff37006ea40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff388034000 tx=0x7ff388039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.529+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff390055770 con 0x7ff39813b750 2024-01-16T15:23:22.716 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.720+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff398150670 con 0x7ff39813b750 2024-01-16T15:23:22.717 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.721+0000 7ff38f400640 1 -- 172.21.15.125:0/2713624865 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff398150670 con 0x7ff39813b750 2024-01-16T15:23:22.717 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2024-01-16T15:23:22.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 msgr2=0x7ff37006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 0x7ff37006ea40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff388034000 tx=0x7ff388039040 comp rx=0 tx=0).stop 2024-01-16T15:23:22.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 msgr2=0x7ff3981f5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7ff3980ae9b0 tx=0x7ff390009dc0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 shutdown_connections 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff37006c580 0x7ff37006ea40 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff39813b750 0x7ff3981f5ca0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff398159490 0x7ff39813e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 --2- 172.21.15.125:0/2713624865 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3780039c0 0x7ff3981f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.723+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 >> 172.21.15.125:0/2713624865 conn(0x7ff39806a7d0 msgr2=0x7ff39806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.724+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 shutdown_connections 2024-01-16T15:23:22.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.724+0000 7ff397e00640 1 -- 172.21.15.125:0/2713624865 wait complete. 2024-01-16T15:23:22.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:22.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.954+0000 7fddcbc00640 1 Processor -- start 2024-01-16T15:23:22.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.954+0000 7fddcbc00640 1 -- start start 2024-01-16T15:23:22.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.955+0000 7fddcbc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fddc4099a60 0x7fddc4099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.955+0000 7fddcbc00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40cfa70 con 0x7fddc406e7c0 2024-01-16T15:23:22.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.955+0000 7fddcbc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc409a3e0 con 0x7fddc415e880 2024-01-16T15:23:22.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.955+0000 7fddcbc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc40cff00 con 0x7fddc4099a60 2024-01-16T15:23:22.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.956+0000 7fddc9c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fddc406e7c0 0x7fddc406ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32892/0 (socket says 172.21.15.125:32892) 2024-01-16T15:23:22.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.956+0000 7fddc9c00640 1 -- 172.21.15.125:0/4016094858 learned_addr learned my addr 172.21.15.125:0/4016094858 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.956+0000 7fddc3e00640 1 --2- 172.21.15.125:0/4016094858 >> v2:172.21.15.137:3300/0 conn(0x7fddc4099a60 0x7fddc4099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.956+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015380419 0 0) 0x7fddc40cfa70 con 0x7fddc406e7c0 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.956+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003520 con 0x7fddc406e7c0 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346580236 0 0) 0x7fdda4003520 con 0x7fddc406e7c0 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 >> v2:172.21.15.137:3300/0 conn(0x7fddc4099a60 msgr2=0x7fddc4099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 --2- 172.21.15.125:0/4016094858 >> v2:172.21.15.137:3300/0 conn(0x7fddc4099a60 0x7fddc4099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.954 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 >> v1:172.21.15.137:6789/0 conn(0x7fddc415e880 legacy=0x7fddc406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc406fc70 con 0x7fddc406e7c0 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fddbc001670 con 0x7fddc406e7c0 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 >> v1:172.21.15.125:6789/0 conn(0x7fddc406e7c0 legacy=0x7fddc406ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fdda4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdda40047c0 0x7fdda4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdda4005180 0x7fdda400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40cff00 con 0x7fdda4003970 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda400c030 con 0x7fdda4005180 2024-01-16T15:23:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda400c1f0 con 0x7fdda40047c0 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9200640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fdda4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9c00640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdda4005180 0x7fdda400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9200640 1 -- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdda40047c0 msgr2=0x7fdda4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9200640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdda40047c0 0x7fdda4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9200640 1 -- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdda4005180 msgr2=0x7fdda400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.957+0000 7fddc9200640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdda4005180 0x7fdda400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.958+0000 7fddc9200640 1 -- 172.21.15.125:0/4016094858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda4003520 con 0x7fdda4003970 2024-01-16T15:23:22.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.958+0000 7fddc9c00640 1 -- 172.21.15.125:0/4016094858 reap_dead start 2024-01-16T15:23:22.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddc9200640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fdda4003d90 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fddac001750 tx=0x7fdda40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7257b6f66ffcc449 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0052d0 con 0x7fdda4003970 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac02e050 con 0x7fdda4003970 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddc3400640 1 -- 172.21.15.125:0/4016094858 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac003830 con 0x7fdda4003970 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddcbc00640 1 -- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 msgr2=0x7fdda4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.959+0000 7fddcbc00640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fdda4003d90 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fddac001750 tx=0x7fdda40057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.960+0000 7fddcbc00640 1 -- 172.21.15.125:0/4016094858 shutdown_connections 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.960+0000 7fddcbc00640 1 --2- 172.21.15.125:0/4016094858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fdda4003d90 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.960+0000 7fddcbc00640 1 -- 172.21.15.125:0/4016094858 >> 172.21.15.125:0/4016094858 conn(0x7fddc40b2d00 msgr2=0x7fddc40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:22.959 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.960+0000 7fddcbc00640 1 -- 172.21.15.125:0/4016094858 shutdown_connections 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.960+0000 7fddcbc00640 1 -- 172.21.15.125:0/4016094858 wait complete. 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.961+0000 7fddcbc00640 1 Processor -- start 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.962+0000 7fddcbc00640 1 -- start start 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.962+0000 7fddcbc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.962+0000 7fddcbc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddc40c9fd0 0x7fddc40cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.962+0000 7fddcbc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddc415b770 0x7fddc415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.962+0000 7fddcbc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc415c130 con 0x7fdda4003970 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddcbc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40c96d0 con 0x7fddc40c9fd0 2024-01-16T15:23:22.960 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddcbc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40c8820 con 0x7fddc415b770 2024-01-16T15:23:22.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37086/0 (socket says 172.21.15.125:37086) 2024-01-16T15:23:22.961 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 -- 172.21.15.125:0/2410518042 learned_addr learned my addr 172.21.15.125:0/2410518042 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:22.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 -- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddc415b770 msgr2=0x7fddc415bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddc415b770 0x7fddc415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 -- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddc40c9fd0 msgr2=0x7fddc40cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:22.962 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddc40c9fd0 0x7fddc40cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.963+0000 7fddc9200640 1 -- 172.21.15.125:0/2410518042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40c8ac0 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddc9200640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fddac02bc00 tx=0x7fddac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0076c0 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac007a30 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac006c20 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc40c8da0 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.964+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc40c9360 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.965+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc4073d70 con 0x7fdda4003970 2024-01-16T15:23:22.963 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.966+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fddac0035f0 con 0x7fdda4003970 2024-01-16T15:23:22.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.967+0000 7fddc0c00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 0x7fdd9c06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:22.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.967+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fddac0aa750 con 0x7fdda4003970 2024-01-16T15:23:22.964 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.967+0000 7fddc3e00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 0x7fdd9c06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:22.965 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.968+0000 7fddc3e00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 0x7fdd9c06ea60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fddc40c5290 tx=0x7fddb400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:22.967 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:22.972+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddac076c90 con 0x7fdda4003970 2024-01-16T15:23:23.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.163+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddc40675d0 con 0x7fdda4003970 2024-01-16T15:23:23.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.164+0000 7fddc0c00640 1 -- 172.21.15.125:0/2410518042 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddac041020 con 0x7fdda4003970 2024-01-16T15:23:23.160 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2024-01-16T15:23:23.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 msgr2=0x7fdd9c06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 0x7fdd9c06ea60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fddc40c5290 tx=0x7fddb400b040 comp rx=0 tx=0).stop 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 msgr2=0x7fddc40c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fddac02bc00 tx=0x7fddac001120 comp rx=0 tx=0).stop 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 shutdown_connections 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fddc415b770 0x7fddc415bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fddc40c9fd0 0x7fddc40cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fdd9c06c5a0 0x7fdd9c06ea60 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.163 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.166+0000 7fddcbc00640 1 --2- 172.21.15.125:0/2410518042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdda4003970 0x7fddc40c82e0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.167+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 >> 172.21.15.125:0/2410518042 conn(0x7fddc40b2d00 msgr2=0x7fddc406c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.167+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 shutdown_connections 2024-01-16T15:23:23.164 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.167+0000 7fddcbc00640 1 -- 172.21.15.125:0/2410518042 wait complete. 2024-01-16T15:23:23.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:23.382 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.386+0000 7f64fe000640 1 Processor -- start 2024-01-16T15:23:23.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.387+0000 7f64fe000640 1 -- start start 2024-01-16T15:23:23.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.388+0000 7f64fe000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64f8143ae0 0x7f64f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.388+0000 7f64fe000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.388+0000 7f64fe000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f80cfc30 con 0x7f64f80cf7d0 2024-01-16T15:23:23.384 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.388+0000 7f64fe000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f80b1ee0 con 0x7f64f8143ae0 2024-01-16T15:23:23.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.388+0000 7f64fe000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80b2780 con 0x7f64f815e3b0 2024-01-16T15:23:23.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64ee800640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64f80cf7d0 0x7f64f815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32898/0 (socket says 172.21.15.125:32898) 2024-01-16T15:23:23.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64ee800640 1 -- 172.21.15.125:0/1376352803 learned_addr learned my addr 172.21.15.125:0/1376352803 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:23.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f7400640 1 --2- 172.21.15.125:0/1376352803 >> v2:172.21.15.137:3300/0 conn(0x7f64f8143ae0 0x7f64f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f7e00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549646008 0 0) 0x7f64f80cfc30 con 0x7f64f80cf7d0 2024-01-16T15:23:23.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d8005730 con 0x7f64f80cf7d0 2024-01-16T15:23:23.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3062085999 0 0) 0x7f64d8005730 con 0x7f64f80cf7d0 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 msgr2=0x7f64f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64f80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 >> v2:172.21.15.137:3300/0 conn(0x7f64f8143ae0 msgr2=0x7f64f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 --2- 172.21.15.125:0/1376352803 >> v2:172.21.15.137:3300/0 conn(0x7f64f8143ae0 0x7f64f80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f815e990 con 0x7f64f80cf7d0 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.389+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f64e00019f0 con 0x7f64f80cf7d0 2024-01-16T15:23:23.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 >> v1:172.21.15.125:6789/0 conn(0x7f64f80cf7d0 legacy=0x7f64f815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8005b80 0x7f64d8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64d80069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d8006f10 0x7f64d8007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80b1ee0 con 0x7f64d8006f10 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80b2780 con 0x7f64d8005b80 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d800db90 con 0x7f64f815e3b0 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d8006f10 0x7f64d8007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64ee800640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64d80069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7400640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8005b80 0x7f64d8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 -- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 msgr2=0x7f64d80069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f815e3b0 0x7f64d80069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 -- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8005b80 msgr2=0x7f64d8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8005b80 0x7f64d8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7e00640 1 -- 172.21.15.125:0/1376352803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d8005730 con 0x7f64d8006f10 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.390+0000 7f64f7400640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8005b80 0x7f64d8005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:23.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.391+0000 7f64f7e00640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d8006f10 0x7f64d8007390 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009af0 tx=0x7f64d8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf1831693e82d2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f002c6b0 con 0x7f64d8006f10 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f002c810 con 0x7f64d8006f10 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64f6a00640 1 -- 172.21.15.125:0/1376352803 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f002cba0 con 0x7f64d8006f10 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64fe000640 1 -- 172.21.15.125:0/1376352803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d8006f10 msgr2=0x7f64d8007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64fe000640 1 --2- 172.21.15.125:0/1376352803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d8006f10 0x7f64d8007390 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009af0 tx=0x7f64d8007a00 comp rx=0 tx=0).stop 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64f7e00640 1 -- 172.21.15.125:0/1376352803 reap_dead start 2024-01-16T15:23:23.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64fe000640 1 -- 172.21.15.125:0/1376352803 shutdown_connections 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.392+0000 7f64fe000640 1 -- 172.21.15.125:0/1376352803 >> 172.21.15.125:0/1376352803 conn(0x7f64f806a800 msgr2=0x7f64f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.393+0000 7f64fe000640 1 -- 172.21.15.125:0/1376352803 shutdown_connections 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.393+0000 7f64fe000640 1 -- 172.21.15.125:0/1376352803 wait complete. 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.395+0000 7f64fe000640 1 Processor -- start 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.395+0000 7f64fe000640 1 -- start start 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 0x7f64f813e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f80cd850 0x7f64f815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f815d000 con 0x7f64f815cb80 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80cc7b0 con 0x7f64d8006f10 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.396+0000 7f64fe000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80cca90 con 0x7f64f80cd850 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37100/0 (socket says 172.21.15.125:37100) 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 -- 172.21.15.125:0/3931553497 learned_addr learned my addr 172.21.15.125:0/3931553497 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:23.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64ee800640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f80cd850 0x7f64f815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7400640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 0x7f64f813e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 -- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f80cd850 msgr2=0x7f64f815c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f80cd850 0x7f64f815c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 -- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 msgr2=0x7f64f813e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 0x7f64f813e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 -- 172.21.15.125:0/3931553497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f80ccd70 con 0x7f64f815cb80 2024-01-16T15:23:23.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7400640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 0x7f64f813e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.397+0000 7f64f7e00640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009af0 tx=0x7f64f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.398+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f002c6b0 con 0x7f64f815cb80 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.398+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f0003830 con 0x7f64f815cb80 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.398+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0003be0 con 0x7f64f815cb80 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.398+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f80cd050 con 0x7f64f815cb80 2024-01-16T15:23:23.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.398+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f813ee00 con 0x7f64f815cb80 2024-01-16T15:23:23.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.399+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f80675d0 con 0x7f64f815cb80 2024-01-16T15:23:23.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.400+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f64f0003d40 con 0x7f64f815cb80 2024-01-16T15:23:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.401+0000 7f64efe00640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 0x7f64d006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.401+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f64f00a9bd0 con 0x7f64f815cb80 2024-01-16T15:23:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.401+0000 7f64f7400640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 0x7f64d006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.402+0000 7f64f7400640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 0x7f64d006ea10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009b20 tx=0x7f64e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.405+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64f0076170 con 0x7f64f815cb80 2024-01-16T15:23:23.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.609+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64f80b0950 con 0x7f64f815cb80 2024-01-16T15:23:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.609+0000 7f64efe00640 1 -- 172.21.15.125:0/3931553497 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7f64f0079790 con 0x7f64f815cb80 2024-01-16T15:23:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:listed 0 entries 2024-01-16T15:23:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 msgr2=0x7f64d006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 0x7f64d006ea10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009b20 tx=0x7f64e8008040 comp rx=0 tx=0).stop 2024-01-16T15:23:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 msgr2=0x7f64f8142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009af0 tx=0x7f64f0004730 comp rx=0 tx=0).stop 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 shutdown_connections 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f64d006c550 0x7f64d006ea10 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f815cb80 0x7f64f8142a50 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64f80cd850 0x7f64f815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 --2- 172.21.15.125:0/3931553497 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64d8006f10 0x7f64f813e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 >> 172.21.15.125:0/3931553497 conn(0x7f64f806a800 msgr2=0x7f64f81417b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.612+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 shutdown_connections 2024-01-16T15:23:23.610 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:23.613+0000 7f64fe000640 1 -- 172.21.15.125:0/3931553497 wait complete. 2024-01-16T15:23:23.627 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-16T15:23:23.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:23.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:23:23.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47330]: fuse finished with error 0 and tester_r 0 2024-01-16T15:23:23.724 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:23:23.806 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:23:23.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:23.807 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:23:23.860 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-16T15:23:23.860 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:23.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:23:23.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:23.911 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2024-01-16T15:23:23.966 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:23:23.966 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:23:23.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:23.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:23:24.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:23:24.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.149 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2024-01-16T15:23:24.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.297 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-16T15:23:24.353 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:23:24.353 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:23:24.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:23:24.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:23:24.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:24.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-01-16T15:23:24.669 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T15:23:24.681 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T15:23:24.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:24.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.895+0000 7fc33c800640 1 Processor -- start 2024-01-16T15:23:24.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.895+0000 7fc33c800640 1 -- start start 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc33c800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc33413df80 0x7fc33415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc33c800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33406c470 con 0x7fc33413e360 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc33c800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33406c750 con 0x7fc3340cf9a0 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc33c800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33406ca30 con 0x7fc33413df80 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc339400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc33413e360 0x7fc3340cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32900/0 (socket says 172.21.15.125:32900) 2024-01-16T15:23:24.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc339400640 1 -- 172.21.15.125:0/2743222515 learned_addr learned my addr 172.21.15.125:0/2743222515 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.897+0000 7fc339e00640 1 --2- 172.21.15.125:0/2743222515 >> v2:172.21.15.137:3300/0 conn(0x7fc33413df80 0x7fc33415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455420799 0 0) 0x7fc33406c470 con 0x7fc33413e360 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314003520 con 0x7fc33413e360 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473174586 0 0) 0x7fc314003520 con 0x7fc33413e360 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 >> v2:172.21.15.137:3300/0 conn(0x7fc33413df80 msgr2=0x7fc33415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:24.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 --2- 172.21.15.125:0/2743222515 >> v2:172.21.15.137:3300/0 conn(0x7fc33413df80 0x7fc33415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 >> v1:172.21.15.137:6789/0 conn(0x7fc3340cf9a0 legacy=0x7fc3340cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33406cd10 con 0x7fc33413e360 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7fc31c001670 con 0x7fc33413e360 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 >> v1:172.21.15.125:6789/0 conn(0x7fc33413e360 legacy=0x7fc3340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314003970 0x7fc314003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3140047c0 0x7fc314004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 0x7fc31400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33406ca30 con 0x7fc314005180 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31400c020 con 0x7fc314003970 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.898+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31400c1e0 con 0x7fc3140047c0 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 0x7fc31400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc339e00640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314003970 0x7fc314003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 -- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3140047c0 msgr2=0x7fc314004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc339400640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3140047c0 0x7fc314004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3140047c0 0x7fc314004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 -- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314003970 msgr2=0x7fc314003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:24.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314003970 0x7fc314003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc33a800640 1 -- 172.21.15.125:0/2743222515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314003520 con 0x7fc314005180 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.899+0000 7fc339400640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3140047c0 0x7fc314004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.900+0000 7fc33a800640 1 -- 172.21.15.125:0/2743222515 reap_dead start 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.900+0000 7fc33a800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 0x7fc31400b980 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fc32c001e20 tx=0x7fc3140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2847a18c4fb909 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.900+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c003b70 con 0x7fc314005180 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.900+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c003cd0 con 0x7fc314005180 2024-01-16T15:23:24.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.900+0000 7fc338a00640 1 -- 172.21.15.125:0/2743222515 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c005310 con 0x7fc314005180 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 -- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 msgr2=0x7fc31400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 0x7fc31400b980 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fc32c001e20 tx=0x7fc3140057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 -- 172.21.15.125:0/2743222515 shutdown_connections 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 --2- 172.21.15.125:0/2743222515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc314005180 0x7fc31400b980 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 -- 172.21.15.125:0/2743222515 >> 172.21.15.125:0/2743222515 conn(0x7fc33406a800 msgr2=0x7fc33406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:24.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 -- 172.21.15.125:0/2743222515 shutdown_connections 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.901+0000 7fc33c800640 1 -- 172.21.15.125:0/2743222515 wait complete. 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.903+0000 7fc33c800640 1 Processor -- start 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.903+0000 7fc33c800640 1 -- start start 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 0x7fc3341528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc33414ed90 0x7fc33414f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33413dc40 con 0x7fc334071e80 2024-01-16T15:23:24.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33414f750 con 0x7fc314005180 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc33c800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc334152e10 con 0x7fc33414ed90 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc339400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc339400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37108/0 (socket says 172.21.15.125:37108) 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.904+0000 7fc339400640 1 -- 172.21.15.125:0/416702635 learned_addr learned my addr 172.21.15.125:0/416702635 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339e00640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 0x7fc3341528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 -- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc33414ed90 msgr2=0x7fc33414f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc33414ed90 0x7fc33414f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 -- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 msgr2=0x7fc3341528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:24.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 0x7fc3341528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 -- 172.21.15.125:0/416702635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3341530d0 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339e00640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 0x7fc3341528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc339400640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00d7c0 tx=0x7fc31c00dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0074e0 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31c007640 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3341533b0 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0049e0 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.905+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc334153970 con 0x7fc334071e80 2024-01-16T15:23:24.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.907+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3340675d0 con 0x7fc334071e80 2024-01-16T15:23:24.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.907+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc31c004b40 con 0x7fc334071e80 2024-01-16T15:23:24.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.908+0000 7fc332000640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 0x7fc30c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:24.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.908+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc31c088860 con 0x7fc334071e80 2024-01-16T15:23:24.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.908+0000 7fc339e00640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 0x7fc30c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:24.908 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.909+0000 7fc339e00640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 0x7fc30c06ea10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc324009980 tx=0x7fc324008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:24.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:24.912+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc31c054e00 con 0x7fc334071e80 2024-01-16T15:23:25.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.100+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3340cf290 con 0x7fc30c06c550 2024-01-16T15:23:25.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.114+0000 7fc332000640 1 -- 172.21.15.125:0/416702635 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289727 (secure 0 0 0) 0x7fc324012950 con 0x7fc30c06c550 2024-01-16T15:23:25.115 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:25.116 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:23:25.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.119+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 msgr2=0x7fc30c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.119+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 0x7fc30c06ea10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc324009980 tx=0x7fc324008040 comp rx=0 tx=0).stop 2024-01-16T15:23:25.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.119+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 msgr2=0x7fc334158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.118 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.119+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00d7c0 tx=0x7fc31c00dc80 comp rx=0 tx=0).stop 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 shutdown_connections 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc33414ed90 0x7fc33414f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fc30c06c550 0x7fc30c06ea10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc334071e80 0x7fc334158470 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 --2- 172.21.15.125:0/416702635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc314005180 0x7fc3341528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 >> 172.21.15.125:0/416702635 conn(0x7fc33406a800 msgr2=0x7fc3340cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 shutdown_connections 2024-01-16T15:23:25.119 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.120+0000 7fc33c800640 1 -- 172.21.15.125:0/416702635 wait complete. 2024-01-16T15:23:25.136 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":911,"stamp":"2024-01-16T15:23:24.539302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444256947,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2244,"num_write_kb":6295328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4674,"ondisk_log_size":4674,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13135212,"kb_used_data":12597548,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":741839508,"statfs":{"total":773094113280,"available":759643656192,"internally_reserved":0,"allocated":12899889152,"data_stored":12893378371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781103,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":273,"num_write_kb":1048621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003181"},"pg_stats":[{"pgid":"17.3f","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449797+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:23:18.449797+0000","last_peered":"2024-01-16T15:23:18.449797+0000","last_clean":"2024-01-16T15:23:18.449797+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:23:18.449797+0000","last_undegraded":"2024-01-16T15:23:18.449797+0000","last_fullsized":"2024-01-16T15:23:18.449797+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:15.148113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'69","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449144+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:23:18.449144+0000","last_peered":"2024-01-16T15:23:18.449144+0000","last_clean":"2024-01-16T15:23:18.449144+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:18.449144+0000","last_undegraded":"2024-01-16T15:23:18.449144+0000","last_fullsized":"2024-01-16T15:23:18.449144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'67","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460230+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:23:18.460230+0000","last_peered":"2024-01-16T15:23:18.460230+0000","last_clean":"2024-01-16T15:23:18.460230+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:18.460230+0000","last_undegraded":"2024-01-16T15:23:18.460230+0000","last_fullsized":"2024-01-16T15:23:18.460230+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458132+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458132+0000","last_peered":"2024-01-16T15:23:18.458132+0000","last_clean":"2024-01-16T15:23:18.458132+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458132+0000","last_undegraded":"2024-01-16T15:23:18.458132+0000","last_fullsized":"2024-01-16T15:23:18.458132+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:02:11.876145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'60","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440343+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:23:18.440343+0000","last_peered":"2024-01-16T15:23:18.440343+0000","last_clean":"2024-01-16T15:23:18.440343+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:18.440343+0000","last_undegraded":"2024-01-16T15:23:18.440343+0000","last_fullsized":"2024-01-16T15:23:18.440343+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:46.608210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'73","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448167+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:23:18.448167+0000","last_peered":"2024-01-16T15:23:18.448167+0000","last_clean":"2024-01-16T15:23:18.448167+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:18.448167+0000","last_undegraded":"2024-01-16T15:23:18.448167+0000","last_fullsized":"2024-01-16T15:23:18.448167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:00.714461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'66","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446647+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:23:18.446647+0000","last_peered":"2024-01-16T15:23:18.446647+0000","last_clean":"2024-01-16T15:23:18.446647+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:23:18.446647+0000","last_undegraded":"2024-01-16T15:23:18.446647+0000","last_fullsized":"2024-01-16T15:23:18.446647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'54","reported_seq":81,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459294+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:23:18.459294+0000","last_peered":"2024-01-16T15:23:18.459294+0000","last_clean":"2024-01-16T15:23:18.459294+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:18.459294+0000","last_undegraded":"2024-01-16T15:23:18.459294+0000","last_fullsized":"2024-01-16T15:23:18.459294+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'79","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456461+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:23:18.456461+0000","last_peered":"2024-01-16T15:23:18.456461+0000","last_clean":"2024-01-16T15:23:18.456461+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:18.456461+0000","last_undegraded":"2024-01-16T15:23:18.456461+0000","last_fullsized":"2024-01-16T15:23:18.456461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:25.063859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446098+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:23:18.446098+0000","last_peered":"2024-01-16T15:23:18.446098+0000","last_clean":"2024-01-16T15:23:18.446098+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:23:18.446098+0000","last_undegraded":"2024-01-16T15:23:18.446098+0000","last_fullsized":"2024-01-16T15:23:18.446098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:45.318654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446325+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.446325+0000","last_peered":"2024-01-16T15:23:18.446325+0000","last_clean":"2024-01-16T15:23:18.446325+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.446325+0000","last_undegraded":"2024-01-16T15:23:18.446325+0000","last_fullsized":"2024-01-16T15:23:18.446325+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448637+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.448637+0000","last_peered":"2024-01-16T15:23:18.448637+0000","last_clean":"2024-01-16T15:23:18.448637+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.448637+0000","last_undegraded":"2024-01-16T15:23:18.448637+0000","last_fullsized":"2024-01-16T15:23:18.448637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:32.928994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'57","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449865+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:23:18.449865+0000","last_peered":"2024-01-16T15:23:18.449865+0000","last_clean":"2024-01-16T15:23:18.449865+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:18.449865+0000","last_undegraded":"2024-01-16T15:23:18.449865+0000","last_fullsized":"2024-01-16T15:23:18.449865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'65","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454924+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:23:18.454924+0000","last_peered":"2024-01-16T15:23:18.454924+0000","last_clean":"2024-01-16T15:23:18.454924+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:18.454924+0000","last_undegraded":"2024-01-16T15:23:18.454924+0000","last_fullsized":"2024-01-16T15:23:18.454924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'54","reported_seq":77,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446988+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:23:18.446988+0000","last_peered":"2024-01-16T15:23:18.446988+0000","last_clean":"2024-01-16T15:23:18.446988+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:18.446988+0000","last_undegraded":"2024-01-16T15:23:18.446988+0000","last_fullsized":"2024-01-16T15:23:18.446988+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:28.316281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'58","reported_seq":76,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458781+0000","last_change":"2024-01-16T15:15:28.509366+0000","last_active":"2024-01-16T15:23:18.458781+0000","last_peered":"2024-01-16T15:23:18.458781+0000","last_clean":"2024-01-16T15:23:18.458781+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:23:18.458781+0000","last_undegraded":"2024-01-16T15:23:18.458781+0000","last_fullsized":"2024-01-16T15:23:18.458781+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:03:48.894250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'49","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447621+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:23:18.447621+0000","last_peered":"2024-01-16T15:23:18.447621+0000","last_clean":"2024-01-16T15:23:18.447621+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:18.447621+0000","last_undegraded":"2024-01-16T15:23:18.447621+0000","last_fullsized":"2024-01-16T15:23:18.447621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'72","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455262+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:23:18.455262+0000","last_peered":"2024-01-16T15:23:18.455262+0000","last_clean":"2024-01-16T15:23:18.455262+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:18.455262+0000","last_undegraded":"2024-01-16T15:23:18.455262+0000","last_fullsized":"2024-01-16T15:23:18.455262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'71","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456798+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:23:18.456798+0000","last_peered":"2024-01-16T15:23:18.456798+0000","last_clean":"2024-01-16T15:23:18.456798+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:18.456798+0000","last_undegraded":"2024-01-16T15:23:18.456798+0000","last_fullsized":"2024-01-16T15:23:18.456798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'67","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453469+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:23:18.453469+0000","last_peered":"2024-01-16T15:23:18.453469+0000","last_clean":"2024-01-16T15:23:18.453469+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:18.453469+0000","last_undegraded":"2024-01-16T15:23:18.453469+0000","last_fullsized":"2024-01-16T15:23:18.453469+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'73","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454699+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:23:18.454699+0000","last_peered":"2024-01-16T15:23:18.454699+0000","last_clean":"2024-01-16T15:23:18.454699+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:18.454699+0000","last_undegraded":"2024-01-16T15:23:18.454699+0000","last_fullsized":"2024-01-16T15:23:18.454699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'64","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453747+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:23:18.453747+0000","last_peered":"2024-01-16T15:23:18.453747+0000","last_clean":"2024-01-16T15:23:18.453747+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:18.453747+0000","last_undegraded":"2024-01-16T15:23:18.453747+0000","last_fullsized":"2024-01-16T15:23:18.453747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'57","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454432+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:23:18.454432+0000","last_peered":"2024-01-16T15:23:18.454432+0000","last_clean":"2024-01-16T15:23:18.454432+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:18.454432+0000","last_undegraded":"2024-01-16T15:23:18.454432+0000","last_fullsized":"2024-01-16T15:23:18.454432+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'70","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459248+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:23:18.459248+0000","last_peered":"2024-01-16T15:23:18.459248+0000","last_clean":"2024-01-16T15:23:18.459248+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:18.459248+0000","last_undegraded":"2024-01-16T15:23:18.459248+0000","last_fullsized":"2024-01-16T15:23:18.459248+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'71","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454209+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:23:18.454209+0000","last_peered":"2024-01-16T15:23:18.454209+0000","last_clean":"2024-01-16T15:23:18.454209+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:18.454209+0000","last_undegraded":"2024-01-16T15:23:18.454209+0000","last_fullsized":"2024-01-16T15:23:18.454209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'81","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:06.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'64","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.442089+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:23:18.442089+0000","last_peered":"2024-01-16T15:23:18.442089+0000","last_clean":"2024-01-16T15:23:18.442089+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:18.442089+0000","last_undegraded":"2024-01-16T15:23:18.442089+0000","last_fullsized":"2024-01-16T15:23:18.442089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:21:39.656818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'43","reported_seq":64,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455446+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:23:18.455446+0000","last_peered":"2024-01-16T15:23:18.455446+0000","last_clean":"2024-01-16T15:23:18.455446+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:23:18.455446+0000","last_undegraded":"2024-01-16T15:23:18.455446+0000","last_fullsized":"2024-01-16T15:23:18.455446+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456111+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:18.456111+0000","last_peered":"2024-01-16T15:23:18.456111+0000","last_clean":"2024-01-16T15:23:18.456111+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:18.456111+0000","last_undegraded":"2024-01-16T15:23:18.456111+0000","last_fullsized":"2024-01-16T15:23:18.456111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'76","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.445031+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:23:18.445031+0000","last_peered":"2024-01-16T15:23:18.445031+0000","last_clean":"2024-01-16T15:23:18.445031+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:23:18.445031+0000","last_undegraded":"2024-01-16T15:23:18.445031+0000","last_fullsized":"2024-01-16T15:23:18.445031+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'81","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453047+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:23:18.453047+0000","last_peered":"2024-01-16T15:23:18.453047+0000","last_clean":"2024-01-16T15:23:18.453047+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:18.453047+0000","last_undegraded":"2024-01-16T15:23:18.453047+0000","last_fullsized":"2024-01-16T15:23:18.453047+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:50.738078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'63","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447640+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:23:18.447640+0000","last_peered":"2024-01-16T15:23:18.447640+0000","last_clean":"2024-01-16T15:23:18.447640+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:18.447640+0000","last_undegraded":"2024-01-16T15:23:18.447640+0000","last_fullsized":"2024-01-16T15:23:18.447640+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:08.692285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'62","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.434993+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:23:18.434993+0000","last_peered":"2024-01-16T15:23:18.434993+0000","last_clean":"2024-01-16T15:23:18.434993+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:18.434993+0000","last_undegraded":"2024-01-16T15:23:18.434993+0000","last_fullsized":"2024-01-16T15:23:18.434993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:12.183950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.419085+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:19:19.419085+0000","last_peered":"2024-01-16T15:19:19.419085+0000","last_clean":"2024-01-16T15:19:19.419085+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:19:19.419085+0000","last_undegraded":"2024-01-16T15:19:19.419085+0000","last_fullsized":"2024-01-16T15:19:19.419085+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'54","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455581+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:23:18.455581+0000","last_peered":"2024-01-16T15:23:18.455581+0000","last_clean":"2024-01-16T15:23:18.455581+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:18.455581+0000","last_undegraded":"2024-01-16T15:23:18.455581+0000","last_fullsized":"2024-01-16T15:23:18.455581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.370582+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:21:09.370582+0000","last_peered":"2024-01-16T15:21:09.370582+0000","last_clean":"2024-01-16T15:21:09.370582+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:21:09.370582+0000","last_undegraded":"2024-01-16T15:21:09.370582+0000","last_fullsized":"2024-01-16T15:21:09.370582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'71","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447180+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:23:18.447180+0000","last_peered":"2024-01-16T15:23:18.447180+0000","last_clean":"2024-01-16T15:23:18.447180+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:18.447180+0000","last_undegraded":"2024-01-16T15:23:18.447180+0000","last_fullsized":"2024-01-16T15:23:18.447180+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:48.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705826+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:20:09.705826+0000","last_peered":"2024-01-16T15:20:09.705826+0000","last_clean":"2024-01-16T15:20:09.705826+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:20:09.705826+0000","last_undegraded":"2024-01-16T15:20:09.705826+0000","last_fullsized":"2024-01-16T15:20:09.705826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:22.852738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'68","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459017+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:23:18.459017+0000","last_peered":"2024-01-16T15:23:18.459017+0000","last_clean":"2024-01-16T15:23:18.459017+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:23:18.459017+0000","last_undegraded":"2024-01-16T15:23:18.459017+0000","last_fullsized":"2024-01-16T15:23:18.459017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705800+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:20:09.705800+0000","last_peered":"2024-01-16T15:20:09.705800+0000","last_clean":"2024-01-16T15:20:09.705800+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:20:09.705800+0000","last_undegraded":"2024-01-16T15:20:09.705800+0000","last_fullsized":"2024-01-16T15:20:09.705800+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:07.677232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'72","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457243+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:23:18.457243+0000","last_peered":"2024-01-16T15:23:18.457243+0000","last_clean":"2024-01-16T15:23:18.457243+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:23:18.457243+0000","last_undegraded":"2024-01-16T15:23:18.457243+0000","last_fullsized":"2024-01-16T15:23:18.457243+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:08.656641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.427723+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:20:09.427723+0000","last_peered":"2024-01-16T15:20:09.427723+0000","last_clean":"2024-01-16T15:20:09.427723+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:20:09.427723+0000","last_undegraded":"2024-01-16T15:20:09.427723+0000","last_fullsized":"2024-01-16T15:20:09.427723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'72","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454108+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:23:18.454108+0000","last_peered":"2024-01-16T15:23:18.454108+0000","last_clean":"2024-01-16T15:23:18.454108+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:23:18.454108+0000","last_undegraded":"2024-01-16T15:23:18.454108+0000","last_fullsized":"2024-01-16T15:23:18.454108+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.709208+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:20:09.709208+0000","last_peered":"2024-01-16T15:20:09.709208+0000","last_clean":"2024-01-16T15:20:09.709208+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:20:09.709208+0000","last_undegraded":"2024-01-16T15:20:09.709208+0000","last_fullsized":"2024-01-16T15:20:09.709208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'73","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444105+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:23:18.444105+0000","last_peered":"2024-01-16T15:23:18.444105+0000","last_clean":"2024-01-16T15:23:18.444105+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:23:18.444105+0000","last_undegraded":"2024-01-16T15:23:18.444105+0000","last_fullsized":"2024-01-16T15:23:18.444105+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:19.703629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.548197+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:19:59.548197+0000","last_peered":"2024-01-16T15:19:59.548197+0000","last_clean":"2024-01-16T15:19:59.548197+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:19:59.548197+0000","last_undegraded":"2024-01-16T15:19:59.548197+0000","last_fullsized":"2024-01-16T15:19:59.548197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'73","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.443778+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:23:18.443778+0000","last_peered":"2024-01-16T15:23:18.443778+0000","last_clean":"2024-01-16T15:23:18.443778+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:23:18.443778+0000","last_undegraded":"2024-01-16T15:23:18.443778+0000","last_fullsized":"2024-01-16T15:23:18.443778+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.786104+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:19:59.786104+0000","last_peered":"2024-01-16T15:19:59.786104+0000","last_clean":"2024-01-16T15:19:59.786104+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:19:59.786104+0000","last_undegraded":"2024-01-16T15:19:59.786104+0000","last_fullsized":"2024-01-16T15:19:59.786104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:43.538921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'58","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448671+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:23:18.448671+0000","last_peered":"2024-01-16T15:23:18.448671+0000","last_clean":"2024-01-16T15:23:18.448671+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:23:18.448671+0000","last_undegraded":"2024-01-16T15:23:18.448671+0000","last_fullsized":"2024-01-16T15:23:18.448671+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.704056+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:19:59.704056+0000","last_peered":"2024-01-16T15:19:59.704056+0000","last_clean":"2024-01-16T15:19:59.704056+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:19:59.704056+0000","last_undegraded":"2024-01-16T15:19:59.704056+0000","last_fullsized":"2024-01-16T15:19:59.704056+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:27.301228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'61","reported_seq":85,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451772+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:23:18.451772+0000","last_peered":"2024-01-16T15:23:18.451772+0000","last_clean":"2024-01-16T15:23:18.451772+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:18.451772+0000","last_undegraded":"2024-01-16T15:23:18.451772+0000","last_fullsized":"2024-01-16T15:23:18.451772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.707473+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:19:59.707473+0000","last_peered":"2024-01-16T15:19:59.707473+0000","last_clean":"2024-01-16T15:19:59.707473+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:19:59.707473+0000","last_undegraded":"2024-01-16T15:19:59.707473+0000","last_fullsized":"2024-01-16T15:19:59.707473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'62","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'31","reported_seq":48,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.209386+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:21:14.209386+0000","last_peered":"2024-01-16T15:21:14.209386+0000","last_clean":"2024-01-16T15:21:14.209386+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:21:14.209386+0000","last_undegraded":"2024-01-16T15:21:14.209386+0000","last_fullsized":"2024-01-16T15:21:14.209386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'69","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453483+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:23:18.453483+0000","last_peered":"2024-01-16T15:23:18.453483+0000","last_clean":"2024-01-16T15:23:18.453483+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:23:18.453483+0000","last_undegraded":"2024-01-16T15:23:18.453483+0000","last_fullsized":"2024-01-16T15:23:18.453483+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'319","reported_seq":336,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.401133+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:23:18.401133+0000","last_peered":"2024-01-16T15:23:18.401133+0000","last_clean":"2024-01-16T15:23:18.401133+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:18.401133+0000","last_undegraded":"2024-01-16T15:23:18.401133+0000","last_fullsized":"2024-01-16T15:23:18.401133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:32.472941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1209133,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":318,"num_write_kb":1331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'54","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440515+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:23:18.440515+0000","last_peered":"2024-01-16T15:23:18.440515+0000","last_clean":"2024-01-16T15:23:18.440515+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:18.440515+0000","last_undegraded":"2024-01-16T15:23:18.440515+0000","last_fullsized":"2024-01-16T15:23:18.440515+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:50.060535+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:19:50.060535+0000","last_peered":"2024-01-16T15:19:50.060535+0000","last_clean":"2024-01-16T15:19:50.060535+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:19:50.060535+0000","last_undegraded":"2024-01-16T15:19:50.060535+0000","last_fullsized":"2024-01-16T15:19:50.060535+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:40.359437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'72","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.452704+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:23:18.452704+0000","last_peered":"2024-01-16T15:23:18.452704+0000","last_clean":"2024-01-16T15:23:18.452704+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:18.452704+0000","last_undegraded":"2024-01-16T15:23:18.452704+0000","last_fullsized":"2024-01-16T15:23:18.452704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.562805+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:19:49.562805+0000","last_peered":"2024-01-16T15:19:49.562805+0000","last_clean":"2024-01-16T15:19:49.562805+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:19:49.562805+0000","last_undegraded":"2024-01-16T15:19:49.562805+0000","last_fullsized":"2024-01-16T15:19:49.562805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'63","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456290+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:23:18.456290+0000","last_peered":"2024-01-16T15:23:18.456290+0000","last_clean":"2024-01-16T15:23:18.456290+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:18.456290+0000","last_undegraded":"2024-01-16T15:23:18.456290+0000","last_fullsized":"2024-01-16T15:23:18.456290+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":25,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.702173+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:19:49.702173+0000","last_peered":"2024-01-16T15:19:49.702173+0000","last_clean":"2024-01-16T15:19:49.702173+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:19:49.702173+0000","last_undegraded":"2024-01-16T15:19:49.702173+0000","last_fullsized":"2024-01-16T15:19:49.702173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:11.168897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'46","reported_seq":68,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448040+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:23:18.448040+0000","last_peered":"2024-01-16T15:23:18.448040+0000","last_clean":"2024-01-16T15:23:18.448040+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:18.448040+0000","last_undegraded":"2024-01-16T15:23:18.448040+0000","last_fullsized":"2024-01-16T15:23:18.448040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:21.652051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.782153+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:19:39.782153+0000","last_peered":"2024-01-16T15:19:39.782153+0000","last_clean":"2024-01-16T15:19:39.782153+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:19:39.782153+0000","last_undegraded":"2024-01-16T15:19:39.782153+0000","last_fullsized":"2024-01-16T15:19:39.782153+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:53.963781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'61","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451100+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:23:18.451100+0000","last_peered":"2024-01-16T15:23:18.451100+0000","last_clean":"2024-01-16T15:23:18.451100+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:18.451100+0000","last_undegraded":"2024-01-16T15:23:18.451100+0000","last_fullsized":"2024-01-16T15:23:18.451100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:45.884405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.700531+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:19:39.700531+0000","last_peered":"2024-01-16T15:19:39.700531+0000","last_clean":"2024-01-16T15:19:39.700531+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:19:39.700531+0000","last_undegraded":"2024-01-16T15:19:39.700531+0000","last_fullsized":"2024-01-16T15:19:39.700531+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:59.699408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455552+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:23:18.455552+0000","last_peered":"2024-01-16T15:23:18.455552+0000","last_clean":"2024-01-16T15:23:18.455552+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:18.455552+0000","last_undegraded":"2024-01-16T15:23:18.455552+0000","last_fullsized":"2024-01-16T15:23:18.455552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.561225+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:19:39.561225+0000","last_peered":"2024-01-16T15:19:39.561225+0000","last_clean":"2024-01-16T15:19:39.561225+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:19:39.561225+0000","last_undegraded":"2024-01-16T15:19:39.561225+0000","last_fullsized":"2024-01-16T15:19:39.561225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'62","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460777+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:23:18.460777+0000","last_peered":"2024-01-16T15:23:18.460777+0000","last_clean":"2024-01-16T15:23:18.460777+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:23:18.460777+0000","last_undegraded":"2024-01-16T15:23:18.460777+0000","last_fullsized":"2024-01-16T15:23:18.460777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.422539+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:19:39.422539+0000","last_peered":"2024-01-16T15:19:39.422539+0000","last_clean":"2024-01-16T15:19:39.422539+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:19:39.422539+0000","last_undegraded":"2024-01-16T15:19:39.422539+0000","last_fullsized":"2024-01-16T15:19:39.422539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'56","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455067+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:23:18.455067+0000","last_peered":"2024-01-16T15:23:18.455067+0000","last_clean":"2024-01-16T15:23:18.455067+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:18.455067+0000","last_undegraded":"2024-01-16T15:23:18.455067+0000","last_fullsized":"2024-01-16T15:23:18.455067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:40.058585+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:19:40.058585+0000","last_peered":"2024-01-16T15:19:40.058585+0000","last_clean":"2024-01-16T15:19:40.058585+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:19:40.058585+0000","last_undegraded":"2024-01-16T15:19:40.058585+0000","last_fullsized":"2024-01-16T15:19:40.058585+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457134+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:18.457134+0000","last_peered":"2024-01-16T15:23:18.457134+0000","last_clean":"2024-01-16T15:23:18.457134+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:18.457134+0000","last_undegraded":"2024-01-16T15:23:18.457134+0000","last_fullsized":"2024-01-16T15:23:18.457134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.557222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.698458+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:19:29.698458+0000","last_peered":"2024-01-16T15:19:29.698458+0000","last_clean":"2024-01-16T15:19:29.698458+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:19:29.698458+0000","last_undegraded":"2024-01-16T15:19:29.698458+0000","last_fullsized":"2024-01-16T15:19:29.698458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'65","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460089+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:23:18.460089+0000","last_peered":"2024-01-16T15:23:18.460089+0000","last_clean":"2024-01-16T15:23:18.460089+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:18.460089+0000","last_undegraded":"2024-01-16T15:23:18.460089+0000","last_fullsized":"2024-01-16T15:23:18.460089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:30.056556+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:19:30.056556+0000","last_peered":"2024-01-16T15:19:30.056556+0000","last_clean":"2024-01-16T15:19:30.056556+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:19:30.056556+0000","last_undegraded":"2024-01-16T15:19:30.056556+0000","last_fullsized":"2024-01-16T15:19:30.056556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'57","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456066+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:23:18.456066+0000","last_peered":"2024-01-16T15:23:18.456066+0000","last_clean":"2024-01-16T15:23:18.456066+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:18.456066+0000","last_undegraded":"2024-01-16T15:23:18.456066+0000","last_fullsized":"2024-01-16T15:23:18.456066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.780012+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:19:29.780012+0000","last_peered":"2024-01-16T15:19:29.780012+0000","last_clean":"2024-01-16T15:19:29.780012+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:19:29.780012+0000","last_undegraded":"2024-01-16T15:19:29.780012+0000","last_fullsized":"2024-01-16T15:19:29.780012+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:02.498301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'53","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451025+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:23:18.451025+0000","last_peered":"2024-01-16T15:23:18.451025+0000","last_clean":"2024-01-16T15:23:18.451025+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:18.451025+0000","last_undegraded":"2024-01-16T15:23:18.451025+0000","last_fullsized":"2024-01-16T15:23:18.451025+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.701583+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:19:29.701583+0000","last_peered":"2024-01-16T15:19:29.701583+0000","last_clean":"2024-01-16T15:19:29.701583+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:19:29.701583+0000","last_undegraded":"2024-01-16T15:19:29.701583+0000","last_fullsized":"2024-01-16T15:19:29.701583+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457653+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:23:18.457653+0000","last_peered":"2024-01-16T15:23:18.457653+0000","last_clean":"2024-01-16T15:23:18.457653+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:18.457653+0000","last_undegraded":"2024-01-16T15:23:18.457653+0000","last_fullsized":"2024-01-16T15:23:18.457653+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:40.476189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'30","reported_seq":47,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:31.869661+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:22:31.869661+0000","last_peered":"2024-01-16T15:22:31.869661+0000","last_clean":"2024-01-16T15:22:31.869661+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:22:31.869661+0000","last_undegraded":"2024-01-16T15:22:31.869661+0000","last_fullsized":"2024-01-16T15:22:31.869661+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451801+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:18.451801+0000","last_peered":"2024-01-16T15:23:18.451801+0000","last_clean":"2024-01-16T15:23:18.451801+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:18.451801+0000","last_undegraded":"2024-01-16T15:23:18.451801+0000","last_fullsized":"2024-01-16T15:23:18.451801+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:29.732315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.778111+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:19:19.778111+0000","last_peered":"2024-01-16T15:19:19.778111+0000","last_clean":"2024-01-16T15:19:19.778111+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:19:19.778111+0000","last_undegraded":"2024-01-16T15:19:19.778111+0000","last_fullsized":"2024-01-16T15:19:19.778111+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:41.055270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'59","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454509+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:23:18.454509+0000","last_peered":"2024-01-16T15:23:18.454509+0000","last_clean":"2024-01-16T15:23:18.454509+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:18.454509+0000","last_undegraded":"2024-01-16T15:23:18.454509+0000","last_fullsized":"2024-01-16T15:23:18.454509+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.699352+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:19:19.699352+0000","last_peered":"2024-01-16T15:19:19.699352+0000","last_clean":"2024-01-16T15:19:19.699352+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:19:19.699352+0000","last_undegraded":"2024-01-16T15:19:19.699352+0000","last_fullsized":"2024-01-16T15:19:19.699352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'63","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449103+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:23:18.449103+0000","last_peered":"2024-01-16T15:23:18.449103+0000","last_clean":"2024-01-16T15:23:18.449103+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:23:18.449103+0000","last_undegraded":"2024-01-16T15:23:18.449103+0000","last_fullsized":"2024-01-16T15:23:18.449103+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.418924+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:19:19.418924+0000","last_peered":"2024-01-16T15:19:19.418924+0000","last_clean":"2024-01-16T15:19:19.418924+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:19:19.418924+0000","last_undegraded":"2024-01-16T15:19:19.418924+0000","last_fullsized":"2024-01-16T15:19:19.418924+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455227+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:23:18.455227+0000","last_peered":"2024-01-16T15:23:18.455227+0000","last_clean":"2024-01-16T15:23:18.455227+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:18.455227+0000","last_undegraded":"2024-01-16T15:23:18.455227+0000","last_fullsized":"2024-01-16T15:23:18.455227+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.540697+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:19:19.540697+0000","last_peered":"2024-01-16T15:19:19.540697+0000","last_clean":"2024-01-16T15:19:19.540697+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:19:19.540697+0000","last_undegraded":"2024-01-16T15:19:19.540697+0000","last_fullsized":"2024-01-16T15:19:19.540697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'68","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457418+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:23:18.457418+0000","last_peered":"2024-01-16T15:23:18.457418+0000","last_clean":"2024-01-16T15:23:18.457418+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:18.457418+0000","last_undegraded":"2024-01-16T15:23:18.457418+0000","last_fullsized":"2024-01-16T15:23:18.457418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'33","reported_seq":82,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.200536+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:21:14.200536+0000","last_peered":"2024-01-16T15:21:14.200536+0000","last_clean":"2024-01-16T15:21:14.200536+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:21:14.200536+0000","last_undegraded":"2024-01-16T15:21:14.200536+0000","last_fullsized":"2024-01-16T15:21:14.200536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444662+0000","last_change":"2024-01-16T15:15:28.574045+0000","last_active":"2024-01-16T15:23:18.444662+0000","last_peered":"2024-01-16T15:23:18.444662+0000","last_clean":"2024-01-16T15:23:18.444662+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:23:18.444662+0000","last_undegraded":"2024-01-16T15:23:18.444662+0000","last_fullsized":"2024-01-16T15:23:18.444662+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.801252+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:21:09.801252+0000","last_peered":"2024-01-16T15:21:09.801252+0000","last_clean":"2024-01-16T15:21:09.801252+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:21:09.801252+0000","last_undegraded":"2024-01-16T15:21:09.801252+0000","last_fullsized":"2024-01-16T15:21:09.801252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:10.684617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'49","reported_seq":78,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458455+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:23:18.458455+0000","last_peered":"2024-01-16T15:23:18.458455+0000","last_clean":"2024-01-16T15:23:18.458455+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:18.458455+0000","last_undegraded":"2024-01-16T15:23:18.458455+0000","last_fullsized":"2024-01-16T15:23:18.458455+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.424236+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:19:49.424236+0000","last_peered":"2024-01-16T15:19:49.424236+0000","last_clean":"2024-01-16T15:19:49.424236+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:19:49.424236+0000","last_undegraded":"2024-01-16T15:19:49.424236+0000","last_fullsized":"2024-01-16T15:19:49.424236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":271,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.364834+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:20:39.364834+0000","last_peered":"2024-01-16T15:20:39.364834+0000","last_clean":"2024-01-16T15:20:39.364834+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:20:39.364834+0000","last_undegraded":"2024-01-16T15:20:39.364834+0000","last_fullsized":"2024-01-16T15:20:39.364834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4112,"ondisk_log_size":4112,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1213575,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":481,"num_write_kb":1418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2490368,"data_stored":2427150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.151,"15min":1.135},"min":{"1min":0.89800000000000002,"5min":0.66000000000000003,"15min":0.60199999999999998},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.86299999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.5580000000000001,"5min":2.0619999999999998,"15min":5.9619999999999997},"last":1.5329999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1950000000000001,"15min":1.256},"min":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.56200000000000006},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.024},{"interface":"front","average":{"1min":1.359,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.73099999999999998},"max":{"1min":1.956,"5min":1.956,"15min":6.2690000000000001},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.24,"15min":1.228},"min":{"1min":0.80900000000000005,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.7310000000000001,"5min":2.0070000000000001,"15min":3.7290000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.244,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.5449999999999999,"5min":1.8129999999999999,"15min":3.3919999999999999},"last":1.105}]},{"osd":3,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3240000000000001,"15min":1.3660000000000001},"min":{"1min":0.97999999999999998,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.649,"5min":1.9279999999999999,"15min":12.331},"last":1.151},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.3640000000000001},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.645},"last":0.90400000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":0.49199999999999999},"max":{"1min":1.837,"5min":1.837,"15min":6.1799999999999997},"last":1.643},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3240000000000001,"15min":1.3169999999999999},"min":{"1min":0.94699999999999995,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.7410000000000001,"5min":1.9510000000000001,"15min":2.9809999999999999},"last":1.28}]},{"osd":5,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.89000000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.508,"5min":2.508,"15min":2.593},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.502,"15min":1.528},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.54700000000000004},"max":{"1min":2.7090000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.7549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.484,"15min":1.4990000000000001},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.823,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.5640000000000001,"15min":1.585},"min":{"1min":1.0940000000000001,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":4.7649999999999997},"last":1.79}]}]},{"osd":6,"up_from":13,"seq":55834575174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.391},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.355}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.4770000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.208}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":0.85799999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.54}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.26},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.512}]}]},{"osd":5,"up_from":13,"seq":55834575174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1759999999999999,"15min":1.1870000000000001},"min":{"1min":0.754,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":7.6529999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.27,"5min":1.294,"15min":1.2589999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.49299999999999999},"max":{"1min":1.843,"5min":1.8480000000000001,"15min":8.7880000000000003},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.22,"15min":1.216},"min":{"1min":0.86599999999999999,"5min":0.72599999999999998,"15min":0.28899999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":3.835},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.3129999999999999,"15min":1.2889999999999999},"min":{"1min":1.0389999999999999,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.96,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":1.1739999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3009999999999999,"15min":1.329},"min":{"1min":0.745,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.375,"15min":1.3620000000000001},"min":{"1min":1.0169999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.49}]},{"osd":3,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.337},"min":{"1min":0.96199999999999997,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.7130000000000001,"5min":1.742,"15min":3.9319999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.45,"15min":1.4470000000000001},"min":{"1min":0.91800000000000004,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":16.158000000000001},"last":1.401}]},{"osd":4,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.27,"15min":1.2350000000000001},"min":{"1min":0.94099999999999995,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":0.996},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.492,"15min":1.496},"min":{"1min":1.2150000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.4450000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.393,"15min":1.3939999999999999},"min":{"1min":0.995,"5min":0.78600000000000003,"15min":0.46800000000000003},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.431,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":1.052,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":6.093},"last":1.1060000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.4370000000000001,"15min":1.52},"min":{"1min":1.216,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":20.488},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.5289999999999999,"15min":1.51},"min":{"1min":0.88300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.444,"5min":2.444,"15min":5.0209999999999999},"last":1.5600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1928112,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92443728,"statfs":{"total":96636764160,"available":94662377472,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.385,"15min":1.3919999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.73,"5min":1.73,"15min":17.315999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4319999999999999,"15min":1.431},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.60399999999999998},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":17.279},"last":1.163}]},{"osd":1,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.498},"max":{"1min":1.7509999999999999,"5min":1.839,"15min":6.1849999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1970000000000001,"15min":1.173},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.6080000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":0.87}]},{"osd":2,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.204,"15min":1.196},"min":{"1min":0.93400000000000005,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":3.859},"last":1.109},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.383,"15min":1.367},"min":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.53500000000000003},"max":{"1min":1.6970000000000001,"5min":1.8120000000000001,"15min":6.8179999999999996},"last":1.0489999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.264,"15min":1.306},"min":{"1min":0.89700000000000002,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.204},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3740000000000001,"15min":1.3700000000000001},"min":{"1min":1.022,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.6890000000000001,"5min":1.712,"15min":4.7930000000000001},"last":1.246}]},{"osd":5,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2829999999999999,"15min":1.3009999999999999},"min":{"1min":0.91600000000000004,"5min":0.69999999999999996,"15min":0.48499999999999999},"max":{"1min":1.5780000000000001,"5min":1.8999999999999999,"15min":2.597},"last":1.54},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.395},"min":{"1min":0.92300000000000004,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.8580000000000001,"5min":1.8660000000000001,"15min":17.140999999999998},"last":1.609}]},{"osd":6,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3600000000000001,"15min":1.365},"min":{"1min":0.97199999999999998,"5min":0.88400000000000001,"15min":0.57099999999999995},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":4.1879999999999997},"last":1.379},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.488,"15min":1.5329999999999999},"min":{"1min":1.0820000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.746,"5min":1.946,"15min":12.324999999999999},"last":1.6699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.444,"15min":1.464},"min":{"1min":1.0549999999999999,"5min":0.99199999999999999,"15min":0.53300000000000003},"max":{"1min":1.968,"5min":1.968,"15min":4.9539999999999997},"last":1.694},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5640000000000001,"15min":1.5489999999999999},"min":{"1min":1.2689999999999999,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.782,"5min":2.0699999999999998,"15min":4.6390000000000002},"last":1.5249999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575174,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818788,"kb_used_data":1751580,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92553052,"statfs":{"total":96636764160,"available":94774325248,"internally_reserved":0,"allocated":1793617920,"data_stored":1792786434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.4059999999999999,"5min":1.6240000000000001,"15min":2.27},"last":0.52700000000000002},{"interface":"front","average":{"1min":1,"5min":1.1080000000000001,"15min":1.0609999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.4079999999999999,"5min":6.6740000000000004,"15min":7.375},"last":0.66100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.95299999999999996,"15min":0.98499999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.44900000000000001},"max":{"1min":1.3049999999999999,"5min":1.6839999999999999,"15min":3.1150000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1160000000000001,"15min":1.1100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38500000000000001},"max":{"1min":1.498,"5min":2.7410000000000001,"15min":8.5449999999999999},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1479999999999999,"15min":1.1830000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46100000000000002},"max":{"1min":1.724,"5min":1.8169999999999999,"15min":7.2809999999999997},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.623,"5min":1.986,"15min":3.4279999999999999},"last":1.3129999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.68799999999999994},"max":{"1min":1.6739999999999999,"5min":2.0030000000000001,"15min":7.4950000000000001},"last":2.1989999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.401,"15min":1.423},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.622},"max":{"1min":1.591,"5min":1.9810000000000001,"15min":9.1289999999999996},"last":1.52}]},{"osd":5,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.361,"15min":1.365},"min":{"1min":0.755,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.643,"5min":2.085,"15min":5.0099999999999998},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.377,"15min":1.431},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.60799999999999998},"max":{"1min":1.706,"5min":1.7430000000000001,"15min":11.255000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.399,"15min":1.4790000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.752},"max":{"1min":1.7450000000000001,"5min":2.0569999999999999,"15min":7.4560000000000004},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.4299999999999999,"15min":1.5149999999999999},"min":{"1min":1.034,"5min":1.0209999999999999,"15min":0.67000000000000004},"max":{"1min":1.7769999999999999,"5min":2.2519999999999998,"15min":12.064},"last":1.26}]},{"osd":7,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4650000000000001,"15min":1.5189999999999999},"min":{"1min":1.038,"5min":1.038,"15min":0.73099999999999998},"max":{"1min":1.6890000000000001,"5min":1.9370000000000001,"15min":7.5339999999999998},"last":1.571},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.474,"15min":1.5349999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.7050000000000001,"5min":2.1760000000000002,"15min":10.804},"last":1.839}]}]},{"osd":2,"up_from":13,"seq":55834575173,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89700000000000002,"15min":0.97899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.314},"max":{"1min":1.3300000000000001,"5min":1.5569999999999999,"15min":24.155000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.0629999999999999,"15min":1.0449999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":1.76,"15min":2.952},"last":0.91500000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.0649999999999999,"15min":1.1180000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.4450000000000001,"5min":1.6970000000000001,"15min":16.776},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.1040000000000001,"15min":1.226},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.46500000000000002},"max":{"1min":1.468,"5min":1.6479999999999999,"15min":25.186},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0569999999999999,"15min":1.1919999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.3240000000000001,"5min":1.788,"15min":37.470999999999997},"last":1.032},{"interface":"front","average":{"1min":1.159,"5min":1.2270000000000001,"15min":1.244},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":6.5549999999999997},"last":1.2230000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.349,"15min":1.5920000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":2.0019999999999998,"5min":2.0680000000000001,"15min":25.062000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.448},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.746},"max":{"1min":1.919,"5min":1.919,"15min":37.307000000000002},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":2.206,"5min":2.206,"15min":6.8849999999999998},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.4139999999999999,"15min":1.538},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.7649999999999999,"5min":1.8839999999999999,"15min":25.141999999999999},"last":2.1139999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3839999999999999,"15min":1.5269999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.81999999999999995},"max":{"1min":1.8300000000000001,"5min":2.0720000000000001,"15min":37.363},"last":1.986},{"interface":"front","average":{"1min":1.478,"5min":1.488,"15min":1.5149999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":6.9630000000000001},"last":1.887}]},{"osd":7,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.498,"15min":1.5860000000000001},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.66100000000000003},"max":{"1min":2.125,"5min":2.125,"15min":25.106999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.4730000000000001,"15min":1.599},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.78400000000000003},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":37.402999999999999},"last":1.649}]}]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.92900000000000005},"min":{"1min":0.625,"5min":0.38900000000000001,"15min":0.36399999999999999},"max":{"1min":1.196,"5min":1.4370000000000001,"15min":14.106999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.478,"5min":1.4990000000000001,"15min":4.3719999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.29799999999999999},"max":{"1min":1.5920000000000001,"5min":3.5409999999999999,"15min":4.8789999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.085,"15min":1.149},"min":{"1min":0.628,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.484,"5min":1.663,"15min":14.055},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2150000000000001,"15min":1.3620000000000001},"min":{"1min":0.86399999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.51,"5min":1.6439999999999999,"15min":25.611000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.60299999999999998,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":1.7090000000000001,"5min":1.952,"15min":4.6699999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.232,"15min":1.254},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.52500000000000002},"max":{"1min":1.472,"5min":3.77,"15min":4.423},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.861,"5min":1.861,"15min":14.161},"last":0.89700000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.5509999999999999},"min":{"1min":0.84999999999999998,"5min":0.60099999999999998,"15min":0.58699999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4239999999999999,"15min":1.5189999999999999},"min":{"1min":1.093,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.9359999999999999,"5min":3.8090000000000002,"15min":16.433},"last":1.095}]},{"osd":6,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.423,"15min":1.496},"min":{"1min":0.93600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":14.212},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.462},"min":{"1min":1.0449999999999999,"5min":0.92800000000000005,"15min":0.63},"max":{"1min":2.0270000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":0.91400000000000003}]},{"osd":7,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4419999999999999,"15min":1.591},"min":{"1min":1.2509999999999999,"5min":0.82499999999999996,"15min":0.75700000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":19.747},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.456,"15min":1.46},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.64100000000000001},"max":{"1min":1.7869999999999999,"5min":3.726,"15min":6.0229999999999997},"last":0.96199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575173,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048736,"kb_used_data":1981528,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323104,"statfs":{"total":96636764160,"available":94538858496,"internally_reserved":0,"allocated":2029084672,"data_stored":2028256257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.82999999999999996,"15min":0.879},"min":{"1min":0.65600000000000003,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.4339999999999999,"5min":1.601,"15min":10.683999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.056,"15min":1.0289999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.41799999999999998},"max":{"1min":1.476,"5min":1.72,"15min":6.2199999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":17.260000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.1160000000000001,"15min":1.1120000000000001},"min":{"1min":0.65100000000000002,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.784,"5min":1.784,"15min":6.4260000000000002},"last":0.60799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.089,"15min":1.0880000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.4730000000000001,"5min":1.706,"15min":3.4670000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.337,"15min":1.2809999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.8400000000000001,"5min":2.2549999999999999,"15min":2.319},"last":1.0389999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.333,"15min":1.393},"min":{"1min":0.76000000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":9.6080000000000005},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.306,"15min":1.2989999999999999},"min":{"1min":1.0009999999999999,"5min":0.83799999999999997,"15min":0.64400000000000002},"max":{"1min":1.5840000000000001,"5min":1.75,"15min":3.6829999999999998},"last":4.0810000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4319999999999999,"15min":1.415},"min":{"1min":1.052,"5min":1.052,"15min":0.624},"max":{"1min":2.222,"5min":2.6890000000000001,"15min":3.1259999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.502,"15min":1.4990000000000001},"min":{"1min":1.2190000000000001,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.2730000000000001,"5min":2.7160000000000002,"15min":6.5430000000000001},"last":1.0129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.395},"min":{"1min":1.008,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":2.0710000000000002,"5min":2.1789999999999998,"15min":11.462},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.454,"15min":1.4570000000000001},"min":{"1min":1.032,"5min":0.92600000000000005,"15min":0.73199999999999998},"max":{"1min":2.093,"5min":2.5680000000000001,"15min":7.1849999999999996},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.49,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.1299999999999999,"5min":2.5099999999999998,"15min":6.9969999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.5069999999999999,"15min":1.478},"min":{"1min":1.1020000000000001,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":2.1669999999999998,"5min":2.633,"15min":3.746},"last":3.976}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1228800,"data_stored":1209245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1224704,"data_stored":1209203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:25.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:25.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.393+0000 7f1997400640 1 Processor -- start 2024-01-16T15:23:25.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.394+0000 7f1997400640 1 -- start start 2024-01-16T15:23:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1997400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19900cd240 0x7f19900b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1997400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 0x7f19900cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1997400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199006cea0 con 0x7f199006c6b0 2024-01-16T15:23:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1997400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199006cbc0 con 0x7f19900cd240 2024-01-16T15:23:25.392 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1997400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199006f350 con 0x7f199009a170 2024-01-16T15:23:25.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1994a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19900cd240 0x7f19900b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1994a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19900cd240 0x7f19900b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37124/0 (socket says 172.21.15.125:37124) 2024-01-16T15:23:25.393 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.395+0000 7f1994a00640 1 -- 172.21.15.125:0/142500300 learned_addr learned my addr 172.21.15.125:0/142500300 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1995400640 1 --2- 172.21.15.125:0/142500300 >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 0x7f19900cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1994a00640 1 -- 172.21.15.125:0/142500300 >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 msgr2=0x7f19900cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1994a00640 1 --2- 172.21.15.125:0/142500300 >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 0x7f19900cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1994a00640 1 -- 172.21.15.125:0/142500300 >> v1:172.21.15.125:6789/0 conn(0x7f199006c6b0 legacy=0x7f1990099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1994a00640 1 -- 172.21.15.125:0/142500300 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199006f630 con 0x7f19900cd240 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.396+0000 7f1995400640 1 --2- 172.21.15.125:0/142500300 >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 0x7f19900cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:25.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.397+0000 7f1994a00640 1 --2- 172.21.15.125:0/142500300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19900cd240 0x7f19900b3820 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f19900cb7f0 tx=0x7f198002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d4e5c02d1d0616 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.397+0000 7f198f400640 1 -- 172.21.15.125:0/142500300 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1980030870 con 0x7f19900cd240 2024-01-16T15:23:25.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f198f400640 1 -- 172.21.15.125:0/142500300 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19800309d0 con 0x7f19900cd240 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f198f400640 1 -- 172.21.15.125:0/142500300 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1980030d60 con 0x7f19900cd240 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 -- 172.21.15.125:0/142500300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19900cd240 msgr2=0x7f19900b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 --2- 172.21.15.125:0/142500300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19900cd240 0x7f19900b3820 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f19900cb7f0 tx=0x7f198002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 -- 172.21.15.125:0/142500300 shutdown_connections 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 --2- 172.21.15.125:0/142500300 >> v2:172.21.15.137:3300/0 conn(0x7f199009a170 0x7f19900cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 --2- 172.21.15.125:0/142500300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19900cd240 0x7f19900b3820 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 -- 172.21.15.125:0/142500300 >> 172.21.15.125:0/142500300 conn(0x7f19900b2d00 msgr2=0x7f19900b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:25.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 -- 172.21.15.125:0/142500300 shutdown_connections 2024-01-16T15:23:25.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.398+0000 7f1997400640 1 -- 172.21.15.125:0/142500300 wait complete. 2024-01-16T15:23:25.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.400+0000 7f1997400640 1 Processor -- start 2024-01-16T15:23:25.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.400+0000 7f1997400640 1 -- start start 2024-01-16T15:23:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f199009a170 0x7f19900c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19900cd240 0x7f19900b6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19901f7360 con 0x7f199006c6b0 2024-01-16T15:23:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199015ad10 con 0x7f19900cd240 2024-01-16T15:23:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1997400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19900b4200 con 0x7f199009a170 2024-01-16T15:23:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1994a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1994a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37134/0 (socket says 172.21.15.125:37134) 2024-01-16T15:23:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.401+0000 7f1994a00640 1 -- 172.21.15.125:0/679382843 learned_addr learned my addr 172.21.15.125:0/679382843 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:25.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 -- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f199009a170 msgr2=0x7f19900c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.400 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f198fe00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f199009a170 0x7f19900c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f199009a170 0x7f19900c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 -- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19900cd240 msgr2=0x7f19900b6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1995400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19900cd240 0x7f19900b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19900cd240 0x7f19900b6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 -- 172.21.15.125:0/679382843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800094d0 con 0x7f199006c6b0 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1994a00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f198002b9f0 tx=0x7f1980004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1980030860 con 0x7f199006c6b0 2024-01-16T15:23:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1980039d90 con 0x7f199006c6b0 2024-01-16T15:23:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1980043650 con 0x7f199006c6b0 2024-01-16T15:23:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.402+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19900b44e0 con 0x7f199006c6b0 2024-01-16T15:23:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.403+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19900b4aa0 con 0x7f199006c6b0 2024-01-16T15:23:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.404+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1990073d90 con 0x7f199006c6b0 2024-01-16T15:23:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.405+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f198002f3c0 con 0x7f199006c6b0 2024-01-16T15:23:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.405+0000 7f198cc00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 0x7f196806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.406+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19800aaff0 con 0x7f199006c6b0 2024-01-16T15:23:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.406+0000 7f198fe00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 0x7f196806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.406+0000 7f198fe00640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 0x7f196806ea10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f199009f2d0 tx=0x7f1978009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.406 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.410+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1980077590 con 0x7f199006c6b0 2024-01-16T15:23:25.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.598+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19900675d0 con 0x7f196806c550 2024-01-16T15:23:25.607 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.610+0000 7f198cc00640 1 -- 172.21.15.125:0/679382843 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289727 (secure 0 0 0) 0x7f19900675d0 con 0x7f196806c550 2024-01-16T15:23:25.611 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:25.613 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:23:25.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 msgr2=0x7f196806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 0x7f196806ea10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f199009f2d0 tx=0x7f1978009040 comp rx=0 tx=0).stop 2024-01-16T15:23:25.614 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 msgr2=0x7f199015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f198002b9f0 tx=0x7f1980004730 comp rx=0 tx=0).stop 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 shutdown_connections 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19900cd240 0x7f19900b6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f199009a170 0x7f19900c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f196806c550 0x7f196806ea10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 --2- 172.21.15.125:0/679382843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f199006c6b0 0x7f199015a7d0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 >> 172.21.15.125:0/679382843 conn(0x7f19900b2d00 msgr2=0x7f19900b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:25.615 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 shutdown_connections 2024-01-16T15:23:25.616 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.617+0000 7f1997400640 1 -- 172.21.15.125:0/679382843 wait complete. 2024-01-16T15:23:25.633 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":911,"stamp":"2024-01-16T15:23:24.539302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444256947,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2244,"num_write_kb":6295328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4674,"ondisk_log_size":4674,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13135212,"kb_used_data":12597548,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":741839508,"statfs":{"total":773094113280,"available":759643656192,"internally_reserved":0,"allocated":12899889152,"data_stored":12893378371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781103,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":273,"num_write_kb":1048621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003181"},"pg_stats":[{"pgid":"17.3f","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449797+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:23:18.449797+0000","last_peered":"2024-01-16T15:23:18.449797+0000","last_clean":"2024-01-16T15:23:18.449797+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:23:18.449797+0000","last_undegraded":"2024-01-16T15:23:18.449797+0000","last_fullsized":"2024-01-16T15:23:18.449797+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:15.148113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'69","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449144+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:23:18.449144+0000","last_peered":"2024-01-16T15:23:18.449144+0000","last_clean":"2024-01-16T15:23:18.449144+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:18.449144+0000","last_undegraded":"2024-01-16T15:23:18.449144+0000","last_fullsized":"2024-01-16T15:23:18.449144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'67","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460230+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:23:18.460230+0000","last_peered":"2024-01-16T15:23:18.460230+0000","last_clean":"2024-01-16T15:23:18.460230+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:18.460230+0000","last_undegraded":"2024-01-16T15:23:18.460230+0000","last_fullsized":"2024-01-16T15:23:18.460230+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458132+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458132+0000","last_peered":"2024-01-16T15:23:18.458132+0000","last_clean":"2024-01-16T15:23:18.458132+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458132+0000","last_undegraded":"2024-01-16T15:23:18.458132+0000","last_fullsized":"2024-01-16T15:23:18.458132+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:02:11.876145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'60","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440343+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:23:18.440343+0000","last_peered":"2024-01-16T15:23:18.440343+0000","last_clean":"2024-01-16T15:23:18.440343+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:18.440343+0000","last_undegraded":"2024-01-16T15:23:18.440343+0000","last_fullsized":"2024-01-16T15:23:18.440343+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:46.608210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'73","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448167+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:23:18.448167+0000","last_peered":"2024-01-16T15:23:18.448167+0000","last_clean":"2024-01-16T15:23:18.448167+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:18.448167+0000","last_undegraded":"2024-01-16T15:23:18.448167+0000","last_fullsized":"2024-01-16T15:23:18.448167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:00.714461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'66","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446647+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:23:18.446647+0000","last_peered":"2024-01-16T15:23:18.446647+0000","last_clean":"2024-01-16T15:23:18.446647+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:23:18.446647+0000","last_undegraded":"2024-01-16T15:23:18.446647+0000","last_fullsized":"2024-01-16T15:23:18.446647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'54","reported_seq":81,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459294+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:23:18.459294+0000","last_peered":"2024-01-16T15:23:18.459294+0000","last_clean":"2024-01-16T15:23:18.459294+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:18.459294+0000","last_undegraded":"2024-01-16T15:23:18.459294+0000","last_fullsized":"2024-01-16T15:23:18.459294+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'79","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456461+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:23:18.456461+0000","last_peered":"2024-01-16T15:23:18.456461+0000","last_clean":"2024-01-16T15:23:18.456461+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:18.456461+0000","last_undegraded":"2024-01-16T15:23:18.456461+0000","last_fullsized":"2024-01-16T15:23:18.456461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:25.063859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446098+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:23:18.446098+0000","last_peered":"2024-01-16T15:23:18.446098+0000","last_clean":"2024-01-16T15:23:18.446098+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:23:18.446098+0000","last_undegraded":"2024-01-16T15:23:18.446098+0000","last_fullsized":"2024-01-16T15:23:18.446098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:45.318654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446325+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.446325+0000","last_peered":"2024-01-16T15:23:18.446325+0000","last_clean":"2024-01-16T15:23:18.446325+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.446325+0000","last_undegraded":"2024-01-16T15:23:18.446325+0000","last_fullsized":"2024-01-16T15:23:18.446325+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448637+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.448637+0000","last_peered":"2024-01-16T15:23:18.448637+0000","last_clean":"2024-01-16T15:23:18.448637+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.448637+0000","last_undegraded":"2024-01-16T15:23:18.448637+0000","last_fullsized":"2024-01-16T15:23:18.448637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:32.928994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'57","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449865+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:23:18.449865+0000","last_peered":"2024-01-16T15:23:18.449865+0000","last_clean":"2024-01-16T15:23:18.449865+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:18.449865+0000","last_undegraded":"2024-01-16T15:23:18.449865+0000","last_fullsized":"2024-01-16T15:23:18.449865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'65","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454924+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:23:18.454924+0000","last_peered":"2024-01-16T15:23:18.454924+0000","last_clean":"2024-01-16T15:23:18.454924+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:18.454924+0000","last_undegraded":"2024-01-16T15:23:18.454924+0000","last_fullsized":"2024-01-16T15:23:18.454924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'54","reported_seq":77,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446988+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:23:18.446988+0000","last_peered":"2024-01-16T15:23:18.446988+0000","last_clean":"2024-01-16T15:23:18.446988+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:18.446988+0000","last_undegraded":"2024-01-16T15:23:18.446988+0000","last_fullsized":"2024-01-16T15:23:18.446988+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:28.316281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'58","reported_seq":76,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458781+0000","last_change":"2024-01-16T15:15:28.509366+0000","last_active":"2024-01-16T15:23:18.458781+0000","last_peered":"2024-01-16T15:23:18.458781+0000","last_clean":"2024-01-16T15:23:18.458781+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:23:18.458781+0000","last_undegraded":"2024-01-16T15:23:18.458781+0000","last_fullsized":"2024-01-16T15:23:18.458781+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:03:48.894250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'49","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447621+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:23:18.447621+0000","last_peered":"2024-01-16T15:23:18.447621+0000","last_clean":"2024-01-16T15:23:18.447621+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:18.447621+0000","last_undegraded":"2024-01-16T15:23:18.447621+0000","last_fullsized":"2024-01-16T15:23:18.447621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'72","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455262+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:23:18.455262+0000","last_peered":"2024-01-16T15:23:18.455262+0000","last_clean":"2024-01-16T15:23:18.455262+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:18.455262+0000","last_undegraded":"2024-01-16T15:23:18.455262+0000","last_fullsized":"2024-01-16T15:23:18.455262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'71","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456798+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:23:18.456798+0000","last_peered":"2024-01-16T15:23:18.456798+0000","last_clean":"2024-01-16T15:23:18.456798+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:18.456798+0000","last_undegraded":"2024-01-16T15:23:18.456798+0000","last_fullsized":"2024-01-16T15:23:18.456798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'67","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453469+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:23:18.453469+0000","last_peered":"2024-01-16T15:23:18.453469+0000","last_clean":"2024-01-16T15:23:18.453469+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:18.453469+0000","last_undegraded":"2024-01-16T15:23:18.453469+0000","last_fullsized":"2024-01-16T15:23:18.453469+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'73","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454699+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:23:18.454699+0000","last_peered":"2024-01-16T15:23:18.454699+0000","last_clean":"2024-01-16T15:23:18.454699+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:18.454699+0000","last_undegraded":"2024-01-16T15:23:18.454699+0000","last_fullsized":"2024-01-16T15:23:18.454699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'64","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453747+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:23:18.453747+0000","last_peered":"2024-01-16T15:23:18.453747+0000","last_clean":"2024-01-16T15:23:18.453747+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:18.453747+0000","last_undegraded":"2024-01-16T15:23:18.453747+0000","last_fullsized":"2024-01-16T15:23:18.453747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'57","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454432+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:23:18.454432+0000","last_peered":"2024-01-16T15:23:18.454432+0000","last_clean":"2024-01-16T15:23:18.454432+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:18.454432+0000","last_undegraded":"2024-01-16T15:23:18.454432+0000","last_fullsized":"2024-01-16T15:23:18.454432+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'70","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459248+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:23:18.459248+0000","last_peered":"2024-01-16T15:23:18.459248+0000","last_clean":"2024-01-16T15:23:18.459248+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:18.459248+0000","last_undegraded":"2024-01-16T15:23:18.459248+0000","last_fullsized":"2024-01-16T15:23:18.459248+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'71","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454209+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:23:18.454209+0000","last_peered":"2024-01-16T15:23:18.454209+0000","last_clean":"2024-01-16T15:23:18.454209+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:18.454209+0000","last_undegraded":"2024-01-16T15:23:18.454209+0000","last_fullsized":"2024-01-16T15:23:18.454209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'81","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:06.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'64","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.442089+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:23:18.442089+0000","last_peered":"2024-01-16T15:23:18.442089+0000","last_clean":"2024-01-16T15:23:18.442089+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:18.442089+0000","last_undegraded":"2024-01-16T15:23:18.442089+0000","last_fullsized":"2024-01-16T15:23:18.442089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:21:39.656818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'43","reported_seq":64,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455446+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:23:18.455446+0000","last_peered":"2024-01-16T15:23:18.455446+0000","last_clean":"2024-01-16T15:23:18.455446+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:23:18.455446+0000","last_undegraded":"2024-01-16T15:23:18.455446+0000","last_fullsized":"2024-01-16T15:23:18.455446+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456111+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:18.456111+0000","last_peered":"2024-01-16T15:23:18.456111+0000","last_clean":"2024-01-16T15:23:18.456111+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:18.456111+0000","last_undegraded":"2024-01-16T15:23:18.456111+0000","last_fullsized":"2024-01-16T15:23:18.456111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'76","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.445031+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:23:18.445031+0000","last_peered":"2024-01-16T15:23:18.445031+0000","last_clean":"2024-01-16T15:23:18.445031+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:23:18.445031+0000","last_undegraded":"2024-01-16T15:23:18.445031+0000","last_fullsized":"2024-01-16T15:23:18.445031+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'81","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453047+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:23:18.453047+0000","last_peered":"2024-01-16T15:23:18.453047+0000","last_clean":"2024-01-16T15:23:18.453047+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:18.453047+0000","last_undegraded":"2024-01-16T15:23:18.453047+0000","last_fullsized":"2024-01-16T15:23:18.453047+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:50.738078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'63","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447640+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:23:18.447640+0000","last_peered":"2024-01-16T15:23:18.447640+0000","last_clean":"2024-01-16T15:23:18.447640+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:18.447640+0000","last_undegraded":"2024-01-16T15:23:18.447640+0000","last_fullsized":"2024-01-16T15:23:18.447640+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:08.692285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'62","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.434993+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:23:18.434993+0000","last_peered":"2024-01-16T15:23:18.434993+0000","last_clean":"2024-01-16T15:23:18.434993+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:18.434993+0000","last_undegraded":"2024-01-16T15:23:18.434993+0000","last_fullsized":"2024-01-16T15:23:18.434993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:12.183950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.419085+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:19:19.419085+0000","last_peered":"2024-01-16T15:19:19.419085+0000","last_clean":"2024-01-16T15:19:19.419085+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:19:19.419085+0000","last_undegraded":"2024-01-16T15:19:19.419085+0000","last_fullsized":"2024-01-16T15:19:19.419085+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'54","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455581+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:23:18.455581+0000","last_peered":"2024-01-16T15:23:18.455581+0000","last_clean":"2024-01-16T15:23:18.455581+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:18.455581+0000","last_undegraded":"2024-01-16T15:23:18.455581+0000","last_fullsized":"2024-01-16T15:23:18.455581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.370582+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:21:09.370582+0000","last_peered":"2024-01-16T15:21:09.370582+0000","last_clean":"2024-01-16T15:21:09.370582+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:21:09.370582+0000","last_undegraded":"2024-01-16T15:21:09.370582+0000","last_fullsized":"2024-01-16T15:21:09.370582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'71","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447180+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:23:18.447180+0000","last_peered":"2024-01-16T15:23:18.447180+0000","last_clean":"2024-01-16T15:23:18.447180+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:18.447180+0000","last_undegraded":"2024-01-16T15:23:18.447180+0000","last_fullsized":"2024-01-16T15:23:18.447180+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:48.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705826+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:20:09.705826+0000","last_peered":"2024-01-16T15:20:09.705826+0000","last_clean":"2024-01-16T15:20:09.705826+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:20:09.705826+0000","last_undegraded":"2024-01-16T15:20:09.705826+0000","last_fullsized":"2024-01-16T15:20:09.705826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:22.852738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'68","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459017+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:23:18.459017+0000","last_peered":"2024-01-16T15:23:18.459017+0000","last_clean":"2024-01-16T15:23:18.459017+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:23:18.459017+0000","last_undegraded":"2024-01-16T15:23:18.459017+0000","last_fullsized":"2024-01-16T15:23:18.459017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705800+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:20:09.705800+0000","last_peered":"2024-01-16T15:20:09.705800+0000","last_clean":"2024-01-16T15:20:09.705800+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:20:09.705800+0000","last_undegraded":"2024-01-16T15:20:09.705800+0000","last_fullsized":"2024-01-16T15:20:09.705800+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:07.677232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'72","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457243+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:23:18.457243+0000","last_peered":"2024-01-16T15:23:18.457243+0000","last_clean":"2024-01-16T15:23:18.457243+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:23:18.457243+0000","last_undegraded":"2024-01-16T15:23:18.457243+0000","last_fullsized":"2024-01-16T15:23:18.457243+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:08.656641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.427723+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:20:09.427723+0000","last_peered":"2024-01-16T15:20:09.427723+0000","last_clean":"2024-01-16T15:20:09.427723+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:20:09.427723+0000","last_undegraded":"2024-01-16T15:20:09.427723+0000","last_fullsized":"2024-01-16T15:20:09.427723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'72","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454108+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:23:18.454108+0000","last_peered":"2024-01-16T15:23:18.454108+0000","last_clean":"2024-01-16T15:23:18.454108+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:23:18.454108+0000","last_undegraded":"2024-01-16T15:23:18.454108+0000","last_fullsized":"2024-01-16T15:23:18.454108+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.709208+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:20:09.709208+0000","last_peered":"2024-01-16T15:20:09.709208+0000","last_clean":"2024-01-16T15:20:09.709208+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:20:09.709208+0000","last_undegraded":"2024-01-16T15:20:09.709208+0000","last_fullsized":"2024-01-16T15:20:09.709208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'73","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444105+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:23:18.444105+0000","last_peered":"2024-01-16T15:23:18.444105+0000","last_clean":"2024-01-16T15:23:18.444105+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:23:18.444105+0000","last_undegraded":"2024-01-16T15:23:18.444105+0000","last_fullsized":"2024-01-16T15:23:18.444105+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:19.703629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.548197+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:19:59.548197+0000","last_peered":"2024-01-16T15:19:59.548197+0000","last_clean":"2024-01-16T15:19:59.548197+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:19:59.548197+0000","last_undegraded":"2024-01-16T15:19:59.548197+0000","last_fullsized":"2024-01-16T15:19:59.548197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'73","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.443778+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:23:18.443778+0000","last_peered":"2024-01-16T15:23:18.443778+0000","last_clean":"2024-01-16T15:23:18.443778+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:23:18.443778+0000","last_undegraded":"2024-01-16T15:23:18.443778+0000","last_fullsized":"2024-01-16T15:23:18.443778+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.786104+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:19:59.786104+0000","last_peered":"2024-01-16T15:19:59.786104+0000","last_clean":"2024-01-16T15:19:59.786104+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:19:59.786104+0000","last_undegraded":"2024-01-16T15:19:59.786104+0000","last_fullsized":"2024-01-16T15:19:59.786104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:43.538921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'58","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448671+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:23:18.448671+0000","last_peered":"2024-01-16T15:23:18.448671+0000","last_clean":"2024-01-16T15:23:18.448671+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:23:18.448671+0000","last_undegraded":"2024-01-16T15:23:18.448671+0000","last_fullsized":"2024-01-16T15:23:18.448671+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.704056+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:19:59.704056+0000","last_peered":"2024-01-16T15:19:59.704056+0000","last_clean":"2024-01-16T15:19:59.704056+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:19:59.704056+0000","last_undegraded":"2024-01-16T15:19:59.704056+0000","last_fullsized":"2024-01-16T15:19:59.704056+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:27.301228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'61","reported_seq":85,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451772+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:23:18.451772+0000","last_peered":"2024-01-16T15:23:18.451772+0000","last_clean":"2024-01-16T15:23:18.451772+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:18.451772+0000","last_undegraded":"2024-01-16T15:23:18.451772+0000","last_fullsized":"2024-01-16T15:23:18.451772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.707473+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:19:59.707473+0000","last_peered":"2024-01-16T15:19:59.707473+0000","last_clean":"2024-01-16T15:19:59.707473+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:19:59.707473+0000","last_undegraded":"2024-01-16T15:19:59.707473+0000","last_fullsized":"2024-01-16T15:19:59.707473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'62","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'31","reported_seq":48,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.209386+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:21:14.209386+0000","last_peered":"2024-01-16T15:21:14.209386+0000","last_clean":"2024-01-16T15:21:14.209386+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:21:14.209386+0000","last_undegraded":"2024-01-16T15:21:14.209386+0000","last_fullsized":"2024-01-16T15:21:14.209386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'69","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453483+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:23:18.453483+0000","last_peered":"2024-01-16T15:23:18.453483+0000","last_clean":"2024-01-16T15:23:18.453483+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:23:18.453483+0000","last_undegraded":"2024-01-16T15:23:18.453483+0000","last_fullsized":"2024-01-16T15:23:18.453483+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'319","reported_seq":336,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.401133+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:23:18.401133+0000","last_peered":"2024-01-16T15:23:18.401133+0000","last_clean":"2024-01-16T15:23:18.401133+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:18.401133+0000","last_undegraded":"2024-01-16T15:23:18.401133+0000","last_fullsized":"2024-01-16T15:23:18.401133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:32.472941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1209133,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":318,"num_write_kb":1331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'54","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440515+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:23:18.440515+0000","last_peered":"2024-01-16T15:23:18.440515+0000","last_clean":"2024-01-16T15:23:18.440515+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:18.440515+0000","last_undegraded":"2024-01-16T15:23:18.440515+0000","last_fullsized":"2024-01-16T15:23:18.440515+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:50.060535+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:19:50.060535+0000","last_peered":"2024-01-16T15:19:50.060535+0000","last_clean":"2024-01-16T15:19:50.060535+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:19:50.060535+0000","last_undegraded":"2024-01-16T15:19:50.060535+0000","last_fullsized":"2024-01-16T15:19:50.060535+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:40.359437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'72","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.452704+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:23:18.452704+0000","last_peered":"2024-01-16T15:23:18.452704+0000","last_clean":"2024-01-16T15:23:18.452704+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:18.452704+0000","last_undegraded":"2024-01-16T15:23:18.452704+0000","last_fullsized":"2024-01-16T15:23:18.452704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.562805+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:19:49.562805+0000","last_peered":"2024-01-16T15:19:49.562805+0000","last_clean":"2024-01-16T15:19:49.562805+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:19:49.562805+0000","last_undegraded":"2024-01-16T15:19:49.562805+0000","last_fullsized":"2024-01-16T15:19:49.562805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'63","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456290+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:23:18.456290+0000","last_peered":"2024-01-16T15:23:18.456290+0000","last_clean":"2024-01-16T15:23:18.456290+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:18.456290+0000","last_undegraded":"2024-01-16T15:23:18.456290+0000","last_fullsized":"2024-01-16T15:23:18.456290+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":25,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.702173+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:19:49.702173+0000","last_peered":"2024-01-16T15:19:49.702173+0000","last_clean":"2024-01-16T15:19:49.702173+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:19:49.702173+0000","last_undegraded":"2024-01-16T15:19:49.702173+0000","last_fullsized":"2024-01-16T15:19:49.702173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:11.168897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'46","reported_seq":68,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448040+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:23:18.448040+0000","last_peered":"2024-01-16T15:23:18.448040+0000","last_clean":"2024-01-16T15:23:18.448040+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:18.448040+0000","last_undegraded":"2024-01-16T15:23:18.448040+0000","last_fullsized":"2024-01-16T15:23:18.448040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:21.652051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.782153+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:19:39.782153+0000","last_peered":"2024-01-16T15:19:39.782153+0000","last_clean":"2024-01-16T15:19:39.782153+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:19:39.782153+0000","last_undegraded":"2024-01-16T15:19:39.782153+0000","last_fullsized":"2024-01-16T15:19:39.782153+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:53.963781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'61","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451100+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:23:18.451100+0000","last_peered":"2024-01-16T15:23:18.451100+0000","last_clean":"2024-01-16T15:23:18.451100+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:18.451100+0000","last_undegraded":"2024-01-16T15:23:18.451100+0000","last_fullsized":"2024-01-16T15:23:18.451100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:45.884405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.700531+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:19:39.700531+0000","last_peered":"2024-01-16T15:19:39.700531+0000","last_clean":"2024-01-16T15:19:39.700531+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:19:39.700531+0000","last_undegraded":"2024-01-16T15:19:39.700531+0000","last_fullsized":"2024-01-16T15:19:39.700531+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:59.699408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455552+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:23:18.455552+0000","last_peered":"2024-01-16T15:23:18.455552+0000","last_clean":"2024-01-16T15:23:18.455552+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:18.455552+0000","last_undegraded":"2024-01-16T15:23:18.455552+0000","last_fullsized":"2024-01-16T15:23:18.455552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.561225+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:19:39.561225+0000","last_peered":"2024-01-16T15:19:39.561225+0000","last_clean":"2024-01-16T15:19:39.561225+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:19:39.561225+0000","last_undegraded":"2024-01-16T15:19:39.561225+0000","last_fullsized":"2024-01-16T15:19:39.561225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'62","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460777+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:23:18.460777+0000","last_peered":"2024-01-16T15:23:18.460777+0000","last_clean":"2024-01-16T15:23:18.460777+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:23:18.460777+0000","last_undegraded":"2024-01-16T15:23:18.460777+0000","last_fullsized":"2024-01-16T15:23:18.460777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.422539+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:19:39.422539+0000","last_peered":"2024-01-16T15:19:39.422539+0000","last_clean":"2024-01-16T15:19:39.422539+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:19:39.422539+0000","last_undegraded":"2024-01-16T15:19:39.422539+0000","last_fullsized":"2024-01-16T15:19:39.422539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'56","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455067+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:23:18.455067+0000","last_peered":"2024-01-16T15:23:18.455067+0000","last_clean":"2024-01-16T15:23:18.455067+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:18.455067+0000","last_undegraded":"2024-01-16T15:23:18.455067+0000","last_fullsized":"2024-01-16T15:23:18.455067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:40.058585+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:19:40.058585+0000","last_peered":"2024-01-16T15:19:40.058585+0000","last_clean":"2024-01-16T15:19:40.058585+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:19:40.058585+0000","last_undegraded":"2024-01-16T15:19:40.058585+0000","last_fullsized":"2024-01-16T15:19:40.058585+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457134+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:18.457134+0000","last_peered":"2024-01-16T15:23:18.457134+0000","last_clean":"2024-01-16T15:23:18.457134+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:18.457134+0000","last_undegraded":"2024-01-16T15:23:18.457134+0000","last_fullsized":"2024-01-16T15:23:18.457134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.557222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.698458+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:19:29.698458+0000","last_peered":"2024-01-16T15:19:29.698458+0000","last_clean":"2024-01-16T15:19:29.698458+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:19:29.698458+0000","last_undegraded":"2024-01-16T15:19:29.698458+0000","last_fullsized":"2024-01-16T15:19:29.698458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'65","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460089+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:23:18.460089+0000","last_peered":"2024-01-16T15:23:18.460089+0000","last_clean":"2024-01-16T15:23:18.460089+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:18.460089+0000","last_undegraded":"2024-01-16T15:23:18.460089+0000","last_fullsized":"2024-01-16T15:23:18.460089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:30.056556+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:19:30.056556+0000","last_peered":"2024-01-16T15:19:30.056556+0000","last_clean":"2024-01-16T15:19:30.056556+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:19:30.056556+0000","last_undegraded":"2024-01-16T15:19:30.056556+0000","last_fullsized":"2024-01-16T15:19:30.056556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'57","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456066+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:23:18.456066+0000","last_peered":"2024-01-16T15:23:18.456066+0000","last_clean":"2024-01-16T15:23:18.456066+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:18.456066+0000","last_undegraded":"2024-01-16T15:23:18.456066+0000","last_fullsized":"2024-01-16T15:23:18.456066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.780012+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:19:29.780012+0000","last_peered":"2024-01-16T15:19:29.780012+0000","last_clean":"2024-01-16T15:19:29.780012+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:19:29.780012+0000","last_undegraded":"2024-01-16T15:19:29.780012+0000","last_fullsized":"2024-01-16T15:19:29.780012+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:02.498301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'53","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451025+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:23:18.451025+0000","last_peered":"2024-01-16T15:23:18.451025+0000","last_clean":"2024-01-16T15:23:18.451025+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:18.451025+0000","last_undegraded":"2024-01-16T15:23:18.451025+0000","last_fullsized":"2024-01-16T15:23:18.451025+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.701583+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:19:29.701583+0000","last_peered":"2024-01-16T15:19:29.701583+0000","last_clean":"2024-01-16T15:19:29.701583+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:19:29.701583+0000","last_undegraded":"2024-01-16T15:19:29.701583+0000","last_fullsized":"2024-01-16T15:19:29.701583+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457653+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:23:18.457653+0000","last_peered":"2024-01-16T15:23:18.457653+0000","last_clean":"2024-01-16T15:23:18.457653+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:18.457653+0000","last_undegraded":"2024-01-16T15:23:18.457653+0000","last_fullsized":"2024-01-16T15:23:18.457653+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:40.476189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'30","reported_seq":47,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:31.869661+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:22:31.869661+0000","last_peered":"2024-01-16T15:22:31.869661+0000","last_clean":"2024-01-16T15:22:31.869661+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:22:31.869661+0000","last_undegraded":"2024-01-16T15:22:31.869661+0000","last_fullsized":"2024-01-16T15:22:31.869661+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451801+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:18.451801+0000","last_peered":"2024-01-16T15:23:18.451801+0000","last_clean":"2024-01-16T15:23:18.451801+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:18.451801+0000","last_undegraded":"2024-01-16T15:23:18.451801+0000","last_fullsized":"2024-01-16T15:23:18.451801+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:29.732315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.778111+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:19:19.778111+0000","last_peered":"2024-01-16T15:19:19.778111+0000","last_clean":"2024-01-16T15:19:19.778111+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:19:19.778111+0000","last_undegraded":"2024-01-16T15:19:19.778111+0000","last_fullsized":"2024-01-16T15:19:19.778111+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:41.055270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'59","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454509+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:23:18.454509+0000","last_peered":"2024-01-16T15:23:18.454509+0000","last_clean":"2024-01-16T15:23:18.454509+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:18.454509+0000","last_undegraded":"2024-01-16T15:23:18.454509+0000","last_fullsized":"2024-01-16T15:23:18.454509+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.699352+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:19:19.699352+0000","last_peered":"2024-01-16T15:19:19.699352+0000","last_clean":"2024-01-16T15:19:19.699352+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:19:19.699352+0000","last_undegraded":"2024-01-16T15:19:19.699352+0000","last_fullsized":"2024-01-16T15:19:19.699352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'63","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449103+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:23:18.449103+0000","last_peered":"2024-01-16T15:23:18.449103+0000","last_clean":"2024-01-16T15:23:18.449103+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:23:18.449103+0000","last_undegraded":"2024-01-16T15:23:18.449103+0000","last_fullsized":"2024-01-16T15:23:18.449103+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.418924+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:19:19.418924+0000","last_peered":"2024-01-16T15:19:19.418924+0000","last_clean":"2024-01-16T15:19:19.418924+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:19:19.418924+0000","last_undegraded":"2024-01-16T15:19:19.418924+0000","last_fullsized":"2024-01-16T15:19:19.418924+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455227+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:23:18.455227+0000","last_peered":"2024-01-16T15:23:18.455227+0000","last_clean":"2024-01-16T15:23:18.455227+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:18.455227+0000","last_undegraded":"2024-01-16T15:23:18.455227+0000","last_fullsized":"2024-01-16T15:23:18.455227+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.540697+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:19:19.540697+0000","last_peered":"2024-01-16T15:19:19.540697+0000","last_clean":"2024-01-16T15:19:19.540697+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:19:19.540697+0000","last_undegraded":"2024-01-16T15:19:19.540697+0000","last_fullsized":"2024-01-16T15:19:19.540697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'68","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457418+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:23:18.457418+0000","last_peered":"2024-01-16T15:23:18.457418+0000","last_clean":"2024-01-16T15:23:18.457418+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:18.457418+0000","last_undegraded":"2024-01-16T15:23:18.457418+0000","last_fullsized":"2024-01-16T15:23:18.457418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'33","reported_seq":82,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.200536+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:21:14.200536+0000","last_peered":"2024-01-16T15:21:14.200536+0000","last_clean":"2024-01-16T15:21:14.200536+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:21:14.200536+0000","last_undegraded":"2024-01-16T15:21:14.200536+0000","last_fullsized":"2024-01-16T15:21:14.200536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444662+0000","last_change":"2024-01-16T15:15:28.574045+0000","last_active":"2024-01-16T15:23:18.444662+0000","last_peered":"2024-01-16T15:23:18.444662+0000","last_clean":"2024-01-16T15:23:18.444662+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:23:18.444662+0000","last_undegraded":"2024-01-16T15:23:18.444662+0000","last_fullsized":"2024-01-16T15:23:18.444662+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.801252+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:21:09.801252+0000","last_peered":"2024-01-16T15:21:09.801252+0000","last_clean":"2024-01-16T15:21:09.801252+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:21:09.801252+0000","last_undegraded":"2024-01-16T15:21:09.801252+0000","last_fullsized":"2024-01-16T15:21:09.801252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:10.684617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'49","reported_seq":78,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458455+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:23:18.458455+0000","last_peered":"2024-01-16T15:23:18.458455+0000","last_clean":"2024-01-16T15:23:18.458455+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:18.458455+0000","last_undegraded":"2024-01-16T15:23:18.458455+0000","last_fullsized":"2024-01-16T15:23:18.458455+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.424236+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:19:49.424236+0000","last_peered":"2024-01-16T15:19:49.424236+0000","last_clean":"2024-01-16T15:19:49.424236+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:19:49.424236+0000","last_undegraded":"2024-01-16T15:19:49.424236+0000","last_fullsized":"2024-01-16T15:19:49.424236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":271,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.364834+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:20:39.364834+0000","last_peered":"2024-01-16T15:20:39.364834+0000","last_clean":"2024-01-16T15:20:39.364834+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:20:39.364834+0000","last_undegraded":"2024-01-16T15:20:39.364834+0000","last_fullsized":"2024-01-16T15:20:39.364834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4112,"ondisk_log_size":4112,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1213575,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":481,"num_write_kb":1418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2490368,"data_stored":2427150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.151,"15min":1.135},"min":{"1min":0.89800000000000002,"5min":0.66000000000000003,"15min":0.60199999999999998},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.86299999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.5580000000000001,"5min":2.0619999999999998,"15min":5.9619999999999997},"last":1.5329999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1950000000000001,"15min":1.256},"min":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.56200000000000006},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.024},{"interface":"front","average":{"1min":1.359,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.73099999999999998},"max":{"1min":1.956,"5min":1.956,"15min":6.2690000000000001},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.24,"15min":1.228},"min":{"1min":0.80900000000000005,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.7310000000000001,"5min":2.0070000000000001,"15min":3.7290000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.244,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.5449999999999999,"5min":1.8129999999999999,"15min":3.3919999999999999},"last":1.105}]},{"osd":3,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3240000000000001,"15min":1.3660000000000001},"min":{"1min":0.97999999999999998,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.649,"5min":1.9279999999999999,"15min":12.331},"last":1.151},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.3640000000000001},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.645},"last":0.90400000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":0.49199999999999999},"max":{"1min":1.837,"5min":1.837,"15min":6.1799999999999997},"last":1.643},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3240000000000001,"15min":1.3169999999999999},"min":{"1min":0.94699999999999995,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.7410000000000001,"5min":1.9510000000000001,"15min":2.9809999999999999},"last":1.28}]},{"osd":5,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.89000000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.508,"5min":2.508,"15min":2.593},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.502,"15min":1.528},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.54700000000000004},"max":{"1min":2.7090000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.7549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.484,"15min":1.4990000000000001},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.823,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.5640000000000001,"15min":1.585},"min":{"1min":1.0940000000000001,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":4.7649999999999997},"last":1.79}]}]},{"osd":6,"up_from":13,"seq":55834575174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.391},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.355}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.4770000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.208}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":0.85799999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.54}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.26},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.512}]}]},{"osd":5,"up_from":13,"seq":55834575174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1759999999999999,"15min":1.1870000000000001},"min":{"1min":0.754,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":7.6529999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.27,"5min":1.294,"15min":1.2589999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.49299999999999999},"max":{"1min":1.843,"5min":1.8480000000000001,"15min":8.7880000000000003},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.22,"15min":1.216},"min":{"1min":0.86599999999999999,"5min":0.72599999999999998,"15min":0.28899999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":3.835},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.3129999999999999,"15min":1.2889999999999999},"min":{"1min":1.0389999999999999,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.96,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":1.1739999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3009999999999999,"15min":1.329},"min":{"1min":0.745,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.375,"15min":1.3620000000000001},"min":{"1min":1.0169999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.49}]},{"osd":3,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.337},"min":{"1min":0.96199999999999997,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.7130000000000001,"5min":1.742,"15min":3.9319999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.45,"15min":1.4470000000000001},"min":{"1min":0.91800000000000004,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":16.158000000000001},"last":1.401}]},{"osd":4,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.27,"15min":1.2350000000000001},"min":{"1min":0.94099999999999995,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":0.996},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.492,"15min":1.496},"min":{"1min":1.2150000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.4450000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.393,"15min":1.3939999999999999},"min":{"1min":0.995,"5min":0.78600000000000003,"15min":0.46800000000000003},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.431,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":1.052,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":6.093},"last":1.1060000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.4370000000000001,"15min":1.52},"min":{"1min":1.216,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":20.488},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.5289999999999999,"15min":1.51},"min":{"1min":0.88300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.444,"5min":2.444,"15min":5.0209999999999999},"last":1.5600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1928112,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92443728,"statfs":{"total":96636764160,"available":94662377472,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.385,"15min":1.3919999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.73,"5min":1.73,"15min":17.315999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4319999999999999,"15min":1.431},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.60399999999999998},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":17.279},"last":1.163}]},{"osd":1,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.498},"max":{"1min":1.7509999999999999,"5min":1.839,"15min":6.1849999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1970000000000001,"15min":1.173},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.6080000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":0.87}]},{"osd":2,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.204,"15min":1.196},"min":{"1min":0.93400000000000005,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":3.859},"last":1.109},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.383,"15min":1.367},"min":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.53500000000000003},"max":{"1min":1.6970000000000001,"5min":1.8120000000000001,"15min":6.8179999999999996},"last":1.0489999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.264,"15min":1.306},"min":{"1min":0.89700000000000002,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.204},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3740000000000001,"15min":1.3700000000000001},"min":{"1min":1.022,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.6890000000000001,"5min":1.712,"15min":4.7930000000000001},"last":1.246}]},{"osd":5,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2829999999999999,"15min":1.3009999999999999},"min":{"1min":0.91600000000000004,"5min":0.69999999999999996,"15min":0.48499999999999999},"max":{"1min":1.5780000000000001,"5min":1.8999999999999999,"15min":2.597},"last":1.54},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.395},"min":{"1min":0.92300000000000004,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.8580000000000001,"5min":1.8660000000000001,"15min":17.140999999999998},"last":1.609}]},{"osd":6,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3600000000000001,"15min":1.365},"min":{"1min":0.97199999999999998,"5min":0.88400000000000001,"15min":0.57099999999999995},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":4.1879999999999997},"last":1.379},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.488,"15min":1.5329999999999999},"min":{"1min":1.0820000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.746,"5min":1.946,"15min":12.324999999999999},"last":1.6699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.444,"15min":1.464},"min":{"1min":1.0549999999999999,"5min":0.99199999999999999,"15min":0.53300000000000003},"max":{"1min":1.968,"5min":1.968,"15min":4.9539999999999997},"last":1.694},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5640000000000001,"15min":1.5489999999999999},"min":{"1min":1.2689999999999999,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.782,"5min":2.0699999999999998,"15min":4.6390000000000002},"last":1.5249999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575174,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818788,"kb_used_data":1751580,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92553052,"statfs":{"total":96636764160,"available":94774325248,"internally_reserved":0,"allocated":1793617920,"data_stored":1792786434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.4059999999999999,"5min":1.6240000000000001,"15min":2.27},"last":0.52700000000000002},{"interface":"front","average":{"1min":1,"5min":1.1080000000000001,"15min":1.0609999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.4079999999999999,"5min":6.6740000000000004,"15min":7.375},"last":0.66100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.95299999999999996,"15min":0.98499999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.44900000000000001},"max":{"1min":1.3049999999999999,"5min":1.6839999999999999,"15min":3.1150000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1160000000000001,"15min":1.1100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38500000000000001},"max":{"1min":1.498,"5min":2.7410000000000001,"15min":8.5449999999999999},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1479999999999999,"15min":1.1830000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46100000000000002},"max":{"1min":1.724,"5min":1.8169999999999999,"15min":7.2809999999999997},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.623,"5min":1.986,"15min":3.4279999999999999},"last":1.3129999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.68799999999999994},"max":{"1min":1.6739999999999999,"5min":2.0030000000000001,"15min":7.4950000000000001},"last":2.1989999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.401,"15min":1.423},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.622},"max":{"1min":1.591,"5min":1.9810000000000001,"15min":9.1289999999999996},"last":1.52}]},{"osd":5,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.361,"15min":1.365},"min":{"1min":0.755,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.643,"5min":2.085,"15min":5.0099999999999998},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.377,"15min":1.431},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.60799999999999998},"max":{"1min":1.706,"5min":1.7430000000000001,"15min":11.255000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.399,"15min":1.4790000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.752},"max":{"1min":1.7450000000000001,"5min":2.0569999999999999,"15min":7.4560000000000004},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.4299999999999999,"15min":1.5149999999999999},"min":{"1min":1.034,"5min":1.0209999999999999,"15min":0.67000000000000004},"max":{"1min":1.7769999999999999,"5min":2.2519999999999998,"15min":12.064},"last":1.26}]},{"osd":7,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4650000000000001,"15min":1.5189999999999999},"min":{"1min":1.038,"5min":1.038,"15min":0.73099999999999998},"max":{"1min":1.6890000000000001,"5min":1.9370000000000001,"15min":7.5339999999999998},"last":1.571},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.474,"15min":1.5349999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.7050000000000001,"5min":2.1760000000000002,"15min":10.804},"last":1.839}]}]},{"osd":2,"up_from":13,"seq":55834575173,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89700000000000002,"15min":0.97899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.314},"max":{"1min":1.3300000000000001,"5min":1.5569999999999999,"15min":24.155000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.0629999999999999,"15min":1.0449999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":1.76,"15min":2.952},"last":0.91500000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.0649999999999999,"15min":1.1180000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.4450000000000001,"5min":1.6970000000000001,"15min":16.776},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.1040000000000001,"15min":1.226},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.46500000000000002},"max":{"1min":1.468,"5min":1.6479999999999999,"15min":25.186},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0569999999999999,"15min":1.1919999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.3240000000000001,"5min":1.788,"15min":37.470999999999997},"last":1.032},{"interface":"front","average":{"1min":1.159,"5min":1.2270000000000001,"15min":1.244},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":6.5549999999999997},"last":1.2230000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.349,"15min":1.5920000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":2.0019999999999998,"5min":2.0680000000000001,"15min":25.062000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.448},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.746},"max":{"1min":1.919,"5min":1.919,"15min":37.307000000000002},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":2.206,"5min":2.206,"15min":6.8849999999999998},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.4139999999999999,"15min":1.538},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.7649999999999999,"5min":1.8839999999999999,"15min":25.141999999999999},"last":2.1139999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3839999999999999,"15min":1.5269999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.81999999999999995},"max":{"1min":1.8300000000000001,"5min":2.0720000000000001,"15min":37.363},"last":1.986},{"interface":"front","average":{"1min":1.478,"5min":1.488,"15min":1.5149999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":6.9630000000000001},"last":1.887}]},{"osd":7,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.498,"15min":1.5860000000000001},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.66100000000000003},"max":{"1min":2.125,"5min":2.125,"15min":25.106999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.4730000000000001,"15min":1.599},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.78400000000000003},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":37.402999999999999},"last":1.649}]}]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.92900000000000005},"min":{"1min":0.625,"5min":0.38900000000000001,"15min":0.36399999999999999},"max":{"1min":1.196,"5min":1.4370000000000001,"15min":14.106999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.478,"5min":1.4990000000000001,"15min":4.3719999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.29799999999999999},"max":{"1min":1.5920000000000001,"5min":3.5409999999999999,"15min":4.8789999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.085,"15min":1.149},"min":{"1min":0.628,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.484,"5min":1.663,"15min":14.055},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2150000000000001,"15min":1.3620000000000001},"min":{"1min":0.86399999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.51,"5min":1.6439999999999999,"15min":25.611000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.60299999999999998,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":1.7090000000000001,"5min":1.952,"15min":4.6699999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.232,"15min":1.254},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.52500000000000002},"max":{"1min":1.472,"5min":3.77,"15min":4.423},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.861,"5min":1.861,"15min":14.161},"last":0.89700000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.5509999999999999},"min":{"1min":0.84999999999999998,"5min":0.60099999999999998,"15min":0.58699999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4239999999999999,"15min":1.5189999999999999},"min":{"1min":1.093,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.9359999999999999,"5min":3.8090000000000002,"15min":16.433},"last":1.095}]},{"osd":6,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.423,"15min":1.496},"min":{"1min":0.93600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":14.212},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.462},"min":{"1min":1.0449999999999999,"5min":0.92800000000000005,"15min":0.63},"max":{"1min":2.0270000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":0.91400000000000003}]},{"osd":7,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4419999999999999,"15min":1.591},"min":{"1min":1.2509999999999999,"5min":0.82499999999999996,"15min":0.75700000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":19.747},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.456,"15min":1.46},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.64100000000000001},"max":{"1min":1.7869999999999999,"5min":3.726,"15min":6.0229999999999997},"last":0.96199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575173,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048736,"kb_used_data":1981528,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323104,"statfs":{"total":96636764160,"available":94538858496,"internally_reserved":0,"allocated":2029084672,"data_stored":2028256257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.82999999999999996,"15min":0.879},"min":{"1min":0.65600000000000003,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.4339999999999999,"5min":1.601,"15min":10.683999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.056,"15min":1.0289999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.41799999999999998},"max":{"1min":1.476,"5min":1.72,"15min":6.2199999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":17.260000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.1160000000000001,"15min":1.1120000000000001},"min":{"1min":0.65100000000000002,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.784,"5min":1.784,"15min":6.4260000000000002},"last":0.60799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.089,"15min":1.0880000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.4730000000000001,"5min":1.706,"15min":3.4670000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.337,"15min":1.2809999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.8400000000000001,"5min":2.2549999999999999,"15min":2.319},"last":1.0389999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.333,"15min":1.393},"min":{"1min":0.76000000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":9.6080000000000005},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.306,"15min":1.2989999999999999},"min":{"1min":1.0009999999999999,"5min":0.83799999999999997,"15min":0.64400000000000002},"max":{"1min":1.5840000000000001,"5min":1.75,"15min":3.6829999999999998},"last":4.0810000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4319999999999999,"15min":1.415},"min":{"1min":1.052,"5min":1.052,"15min":0.624},"max":{"1min":2.222,"5min":2.6890000000000001,"15min":3.1259999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.502,"15min":1.4990000000000001},"min":{"1min":1.2190000000000001,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.2730000000000001,"5min":2.7160000000000002,"15min":6.5430000000000001},"last":1.0129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.395},"min":{"1min":1.008,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":2.0710000000000002,"5min":2.1789999999999998,"15min":11.462},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.454,"15min":1.4570000000000001},"min":{"1min":1.032,"5min":0.92600000000000005,"15min":0.73199999999999998},"max":{"1min":2.093,"5min":2.5680000000000001,"15min":7.1849999999999996},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.49,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.1299999999999999,"5min":2.5099999999999998,"15min":6.9969999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.5069999999999999,"15min":1.478},"min":{"1min":1.1020000000000001,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":2.1669999999999998,"5min":2.633,"15min":3.746},"last":3.976}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1228800,"data_stored":1209245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1224704,"data_stored":1209203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:25.638 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T15:23:25.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:25.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.890+0000 7f654cc00640 1 Processor -- start 2024-01-16T15:23:25.887 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.890+0000 7f654cc00640 1 -- start start 2024-01-16T15:23:25.888 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.891+0000 7f654cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 0x7f654815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f654cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f654813e360 0x7f65480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f654cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654815f9c0 0x7f65480cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f654cc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654806c470 con 0x7f654813e360 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f654cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654806c750 con 0x7f654813df80 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f654cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654806ca30 con 0x7f654815f9c0 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f654813e360 0x7f65480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f654813e360 0x7f65480cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37150/0 (socket says 172.21.15.125:37150) 2024-01-16T15:23:25.889 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 -- 172.21.15.125:0/2785572278 learned_addr learned my addr 172.21.15.125:0/2785572278 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 -- 172.21.15.125:0/2785572278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654815f9c0 msgr2=0x7f65480cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 --2- 172.21.15.125:0/2785572278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654815f9c0 0x7f65480cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 -- 172.21.15.125:0/2785572278 >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 msgr2=0x7f654815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f6545e00640 1 --2- 172.21.15.125:0/2785572278 >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 0x7f654815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.892+0000 7f653fe00640 1 --2- 172.21.15.125:0/2785572278 >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 0x7f654815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.893+0000 7f653fe00640 1 -- 172.21.15.125:0/2785572278 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654806cd10 con 0x7f654813e360 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.893+0000 7f6545e00640 1 --2- 172.21.15.125:0/2785572278 >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 0x7f654815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:25.891 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.894+0000 7f653fe00640 1 --2- 172.21.15.125:0/2785572278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654813e360 0x7f65480cf290 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f6548072770 tx=0x7f653002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26982189015ef5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.894+0000 7f6545400640 1 -- 172.21.15.125:0/2785572278 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530031870 con 0x7f654813e360 2024-01-16T15:23:25.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.894+0000 7f6545400640 1 -- 172.21.15.125:0/2785572278 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65300319d0 con 0x7f654813e360 2024-01-16T15:23:25.892 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.894+0000 7f6545400640 1 -- 172.21.15.125:0/2785572278 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530031d60 con 0x7f654813e360 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 -- 172.21.15.125:0/2785572278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654813e360 msgr2=0x7f65480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 --2- 172.21.15.125:0/2785572278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654813e360 0x7f65480cf290 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f6548072770 tx=0x7f653002b910 comp rx=0 tx=0).stop 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 -- 172.21.15.125:0/2785572278 shutdown_connections 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 --2- 172.21.15.125:0/2785572278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654815f9c0 0x7f65480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 --2- 172.21.15.125:0/2785572278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654813e360 0x7f65480cf290 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 --2- 172.21.15.125:0/2785572278 >> v2:172.21.15.137:3300/0 conn(0x7f654813df80 0x7f654815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 -- 172.21.15.125:0/2785572278 >> 172.21.15.125:0/2785572278 conn(0x7f654806a800 msgr2=0x7f654806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 -- 172.21.15.125:0/2785572278 shutdown_connections 2024-01-16T15:23:25.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.895+0000 7f654cc00640 1 -- 172.21.15.125:0/2785572278 wait complete. 2024-01-16T15:23:25.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.897+0000 7f654cc00640 1 Processor -- start 2024-01-16T15:23:25.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.897+0000 7f654cc00640 1 -- start start 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654813df80 0x7f654815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f654813e360 0x7f654813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480a9490 con 0x7f654815f9c0 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480a91f0 con 0x7f654813e360 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f654cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548159450 con 0x7f654813df80 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f6546800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f6546800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37152/0 (socket says 172.21.15.125:37152) 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f6546800640 1 -- 172.21.15.125:0/604866331 learned_addr learned my addr 172.21.15.125:0/604866331 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.898+0000 7f653fe00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f654813e360 0x7f654813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6545e00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654813df80 0x7f654815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 -- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654813df80 msgr2=0x7f654815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654813df80 0x7f654815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 -- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f654813e360 msgr2=0x7f654813cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f654813e360 0x7f654813cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:25.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 -- 172.21.15.125:0/604866331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300094d0 con 0x7f654815f9c0 2024-01-16T15:23:25.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f6546800640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f654000b9e0 tx=0x7f654000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654000cb20 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654000cc80 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540014680 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6548159730 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.899+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6548159cf0 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.901+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6548159ee0 con 0x7f654815f9c0 2024-01-16T15:23:25.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.901+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f654001d3b0 con 0x7f654815f9c0 2024-01-16T15:23:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.902+0000 7f653e000640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 0x7f652006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.903+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6540088ab0 con 0x7f654815f9c0 2024-01-16T15:23:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.903+0000 7f6545e00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 0x7f652006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:25.900 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.903+0000 7f6545e00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 0x7f652006ea10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f653800a950 tx=0x7f6538008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:25.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:25.907+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6540055050 con 0x7f654815f9c0 2024-01-16T15:23:26.089 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.092+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65481408d0 con 0x7f652006c550 2024-01-16T15:23:26.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.106+0000 7f653e000640 1 -- 172.21.15.125:0/604866331 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289727 (secure 0 0 0) 0x7f653800c050 con 0x7f652006c550 2024-01-16T15:23:26.104 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:26.108 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:23:26.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.112+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 msgr2=0x7f652006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.110 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.112+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 0x7f652006ea10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f653800a950 tx=0x7f6538008040 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 msgr2=0x7f65480a8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f654000b9e0 tx=0x7f654000bea0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 shutdown_connections 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f652006c550 0x7f652006ea10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f654815f9c0 0x7f65480a8cb0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f654813e360 0x7f654813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 --2- 172.21.15.125:0/604866331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f654813df80 0x7f654815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.111 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 >> 172.21.15.125:0/604866331 conn(0x7f654806a800 msgr2=0x7f65480cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:26.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 shutdown_connections 2024-01-16T15:23:26.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.113+0000 7f654cc00640 1 -- 172.21.15.125:0/604866331 wait complete. 2024-01-16T15:23:26.130 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":911,"stamp":"2024-01-16T15:23:24.539302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444256947,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2244,"num_write_kb":6295328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4674,"ondisk_log_size":4674,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13135212,"kb_used_data":12597548,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":741839508,"statfs":{"total":773094113280,"available":759643656192,"internally_reserved":0,"allocated":12899889152,"data_stored":12893378371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781103,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":273,"num_write_kb":1048621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003181"},"pg_stats":[{"pgid":"17.3f","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449797+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:23:18.449797+0000","last_peered":"2024-01-16T15:23:18.449797+0000","last_clean":"2024-01-16T15:23:18.449797+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:23:18.449797+0000","last_undegraded":"2024-01-16T15:23:18.449797+0000","last_fullsized":"2024-01-16T15:23:18.449797+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:15.148113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'69","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449144+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:23:18.449144+0000","last_peered":"2024-01-16T15:23:18.449144+0000","last_clean":"2024-01-16T15:23:18.449144+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:18.449144+0000","last_undegraded":"2024-01-16T15:23:18.449144+0000","last_fullsized":"2024-01-16T15:23:18.449144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'67","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460230+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:23:18.460230+0000","last_peered":"2024-01-16T15:23:18.460230+0000","last_clean":"2024-01-16T15:23:18.460230+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:18.460230+0000","last_undegraded":"2024-01-16T15:23:18.460230+0000","last_fullsized":"2024-01-16T15:23:18.460230+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458132+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458132+0000","last_peered":"2024-01-16T15:23:18.458132+0000","last_clean":"2024-01-16T15:23:18.458132+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458132+0000","last_undegraded":"2024-01-16T15:23:18.458132+0000","last_fullsized":"2024-01-16T15:23:18.458132+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:02:11.876145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'60","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440343+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:23:18.440343+0000","last_peered":"2024-01-16T15:23:18.440343+0000","last_clean":"2024-01-16T15:23:18.440343+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:18.440343+0000","last_undegraded":"2024-01-16T15:23:18.440343+0000","last_fullsized":"2024-01-16T15:23:18.440343+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:46.608210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'73","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448167+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:23:18.448167+0000","last_peered":"2024-01-16T15:23:18.448167+0000","last_clean":"2024-01-16T15:23:18.448167+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:18.448167+0000","last_undegraded":"2024-01-16T15:23:18.448167+0000","last_fullsized":"2024-01-16T15:23:18.448167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:00.714461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'66","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446647+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:23:18.446647+0000","last_peered":"2024-01-16T15:23:18.446647+0000","last_clean":"2024-01-16T15:23:18.446647+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:23:18.446647+0000","last_undegraded":"2024-01-16T15:23:18.446647+0000","last_fullsized":"2024-01-16T15:23:18.446647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'54","reported_seq":81,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459294+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:23:18.459294+0000","last_peered":"2024-01-16T15:23:18.459294+0000","last_clean":"2024-01-16T15:23:18.459294+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:18.459294+0000","last_undegraded":"2024-01-16T15:23:18.459294+0000","last_fullsized":"2024-01-16T15:23:18.459294+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'79","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456461+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:23:18.456461+0000","last_peered":"2024-01-16T15:23:18.456461+0000","last_clean":"2024-01-16T15:23:18.456461+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:18.456461+0000","last_undegraded":"2024-01-16T15:23:18.456461+0000","last_fullsized":"2024-01-16T15:23:18.456461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:25.063859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446098+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:23:18.446098+0000","last_peered":"2024-01-16T15:23:18.446098+0000","last_clean":"2024-01-16T15:23:18.446098+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:23:18.446098+0000","last_undegraded":"2024-01-16T15:23:18.446098+0000","last_fullsized":"2024-01-16T15:23:18.446098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:29:45.318654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446325+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.446325+0000","last_peered":"2024-01-16T15:23:18.446325+0000","last_clean":"2024-01-16T15:23:18.446325+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.446325+0000","last_undegraded":"2024-01-16T15:23:18.446325+0000","last_fullsized":"2024-01-16T15:23:18.446325+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448637+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.448637+0000","last_peered":"2024-01-16T15:23:18.448637+0000","last_clean":"2024-01-16T15:23:18.448637+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.448637+0000","last_undegraded":"2024-01-16T15:23:18.448637+0000","last_fullsized":"2024-01-16T15:23:18.448637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:32.928994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'57","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449865+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:23:18.449865+0000","last_peered":"2024-01-16T15:23:18.449865+0000","last_clean":"2024-01-16T15:23:18.449865+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:18.449865+0000","last_undegraded":"2024-01-16T15:23:18.449865+0000","last_fullsized":"2024-01-16T15:23:18.449865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'65","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454924+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:23:18.454924+0000","last_peered":"2024-01-16T15:23:18.454924+0000","last_clean":"2024-01-16T15:23:18.454924+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:18.454924+0000","last_undegraded":"2024-01-16T15:23:18.454924+0000","last_fullsized":"2024-01-16T15:23:18.454924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'54","reported_seq":77,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446988+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:23:18.446988+0000","last_peered":"2024-01-16T15:23:18.446988+0000","last_clean":"2024-01-16T15:23:18.446988+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:18.446988+0000","last_undegraded":"2024-01-16T15:23:18.446988+0000","last_fullsized":"2024-01-16T15:23:18.446988+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:28.316281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'58","reported_seq":76,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458781+0000","last_change":"2024-01-16T15:15:28.509366+0000","last_active":"2024-01-16T15:23:18.458781+0000","last_peered":"2024-01-16T15:23:18.458781+0000","last_clean":"2024-01-16T15:23:18.458781+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:23:18.458781+0000","last_undegraded":"2024-01-16T15:23:18.458781+0000","last_fullsized":"2024-01-16T15:23:18.458781+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:03:48.894250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'49","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447621+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:23:18.447621+0000","last_peered":"2024-01-16T15:23:18.447621+0000","last_clean":"2024-01-16T15:23:18.447621+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:18.447621+0000","last_undegraded":"2024-01-16T15:23:18.447621+0000","last_fullsized":"2024-01-16T15:23:18.447621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'72","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455262+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:23:18.455262+0000","last_peered":"2024-01-16T15:23:18.455262+0000","last_clean":"2024-01-16T15:23:18.455262+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:18.455262+0000","last_undegraded":"2024-01-16T15:23:18.455262+0000","last_fullsized":"2024-01-16T15:23:18.455262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'71","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456798+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:23:18.456798+0000","last_peered":"2024-01-16T15:23:18.456798+0000","last_clean":"2024-01-16T15:23:18.456798+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:18.456798+0000","last_undegraded":"2024-01-16T15:23:18.456798+0000","last_fullsized":"2024-01-16T15:23:18.456798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'67","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453469+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:23:18.453469+0000","last_peered":"2024-01-16T15:23:18.453469+0000","last_clean":"2024-01-16T15:23:18.453469+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:18.453469+0000","last_undegraded":"2024-01-16T15:23:18.453469+0000","last_fullsized":"2024-01-16T15:23:18.453469+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'73","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454699+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:23:18.454699+0000","last_peered":"2024-01-16T15:23:18.454699+0000","last_clean":"2024-01-16T15:23:18.454699+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:18.454699+0000","last_undegraded":"2024-01-16T15:23:18.454699+0000","last_fullsized":"2024-01-16T15:23:18.454699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'64","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453747+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:23:18.453747+0000","last_peered":"2024-01-16T15:23:18.453747+0000","last_clean":"2024-01-16T15:23:18.453747+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:18.453747+0000","last_undegraded":"2024-01-16T15:23:18.453747+0000","last_fullsized":"2024-01-16T15:23:18.453747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'57","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454432+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:23:18.454432+0000","last_peered":"2024-01-16T15:23:18.454432+0000","last_clean":"2024-01-16T15:23:18.454432+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:18.454432+0000","last_undegraded":"2024-01-16T15:23:18.454432+0000","last_fullsized":"2024-01-16T15:23:18.454432+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'70","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459248+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:23:18.459248+0000","last_peered":"2024-01-16T15:23:18.459248+0000","last_clean":"2024-01-16T15:23:18.459248+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:18.459248+0000","last_undegraded":"2024-01-16T15:23:18.459248+0000","last_fullsized":"2024-01-16T15:23:18.459248+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'71","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454209+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:23:18.454209+0000","last_peered":"2024-01-16T15:23:18.454209+0000","last_clean":"2024-01-16T15:23:18.454209+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:18.454209+0000","last_undegraded":"2024-01-16T15:23:18.454209+0000","last_fullsized":"2024-01-16T15:23:18.454209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'81","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:06.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'64","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.442089+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:23:18.442089+0000","last_peered":"2024-01-16T15:23:18.442089+0000","last_clean":"2024-01-16T15:23:18.442089+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:18.442089+0000","last_undegraded":"2024-01-16T15:23:18.442089+0000","last_fullsized":"2024-01-16T15:23:18.442089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:21:39.656818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'43","reported_seq":64,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455446+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:23:18.455446+0000","last_peered":"2024-01-16T15:23:18.455446+0000","last_clean":"2024-01-16T15:23:18.455446+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:23:18.455446+0000","last_undegraded":"2024-01-16T15:23:18.455446+0000","last_fullsized":"2024-01-16T15:23:18.455446+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456111+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:18.456111+0000","last_peered":"2024-01-16T15:23:18.456111+0000","last_clean":"2024-01-16T15:23:18.456111+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:18.456111+0000","last_undegraded":"2024-01-16T15:23:18.456111+0000","last_fullsized":"2024-01-16T15:23:18.456111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'76","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.445031+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:23:18.445031+0000","last_peered":"2024-01-16T15:23:18.445031+0000","last_clean":"2024-01-16T15:23:18.445031+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:23:18.445031+0000","last_undegraded":"2024-01-16T15:23:18.445031+0000","last_fullsized":"2024-01-16T15:23:18.445031+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'81","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453047+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:23:18.453047+0000","last_peered":"2024-01-16T15:23:18.453047+0000","last_clean":"2024-01-16T15:23:18.453047+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:18.453047+0000","last_undegraded":"2024-01-16T15:23:18.453047+0000","last_fullsized":"2024-01-16T15:23:18.453047+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:50.738078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'63","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447640+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:23:18.447640+0000","last_peered":"2024-01-16T15:23:18.447640+0000","last_clean":"2024-01-16T15:23:18.447640+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:18.447640+0000","last_undegraded":"2024-01-16T15:23:18.447640+0000","last_fullsized":"2024-01-16T15:23:18.447640+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:08.692285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'62","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.434993+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:23:18.434993+0000","last_peered":"2024-01-16T15:23:18.434993+0000","last_clean":"2024-01-16T15:23:18.434993+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:18.434993+0000","last_undegraded":"2024-01-16T15:23:18.434993+0000","last_fullsized":"2024-01-16T15:23:18.434993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:12.183950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.419085+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:19:19.419085+0000","last_peered":"2024-01-16T15:19:19.419085+0000","last_clean":"2024-01-16T15:19:19.419085+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:19:19.419085+0000","last_undegraded":"2024-01-16T15:19:19.419085+0000","last_fullsized":"2024-01-16T15:19:19.419085+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'54","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455581+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:23:18.455581+0000","last_peered":"2024-01-16T15:23:18.455581+0000","last_clean":"2024-01-16T15:23:18.455581+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:18.455581+0000","last_undegraded":"2024-01-16T15:23:18.455581+0000","last_fullsized":"2024-01-16T15:23:18.455581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.370582+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:21:09.370582+0000","last_peered":"2024-01-16T15:21:09.370582+0000","last_clean":"2024-01-16T15:21:09.370582+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:21:09.370582+0000","last_undegraded":"2024-01-16T15:21:09.370582+0000","last_fullsized":"2024-01-16T15:21:09.370582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'71","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447180+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:23:18.447180+0000","last_peered":"2024-01-16T15:23:18.447180+0000","last_clean":"2024-01-16T15:23:18.447180+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:18.447180+0000","last_undegraded":"2024-01-16T15:23:18.447180+0000","last_fullsized":"2024-01-16T15:23:18.447180+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:48.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705826+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:20:09.705826+0000","last_peered":"2024-01-16T15:20:09.705826+0000","last_clean":"2024-01-16T15:20:09.705826+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:20:09.705826+0000","last_undegraded":"2024-01-16T15:20:09.705826+0000","last_fullsized":"2024-01-16T15:20:09.705826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:22.852738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'68","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459017+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:23:18.459017+0000","last_peered":"2024-01-16T15:23:18.459017+0000","last_clean":"2024-01-16T15:23:18.459017+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:23:18.459017+0000","last_undegraded":"2024-01-16T15:23:18.459017+0000","last_fullsized":"2024-01-16T15:23:18.459017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.705800+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:20:09.705800+0000","last_peered":"2024-01-16T15:20:09.705800+0000","last_clean":"2024-01-16T15:20:09.705800+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:20:09.705800+0000","last_undegraded":"2024-01-16T15:20:09.705800+0000","last_fullsized":"2024-01-16T15:20:09.705800+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:07.677232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'72","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457243+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:23:18.457243+0000","last_peered":"2024-01-16T15:23:18.457243+0000","last_clean":"2024-01-16T15:23:18.457243+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:23:18.457243+0000","last_undegraded":"2024-01-16T15:23:18.457243+0000","last_fullsized":"2024-01-16T15:23:18.457243+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:08.656641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.427723+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:20:09.427723+0000","last_peered":"2024-01-16T15:20:09.427723+0000","last_clean":"2024-01-16T15:20:09.427723+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:20:09.427723+0000","last_undegraded":"2024-01-16T15:20:09.427723+0000","last_fullsized":"2024-01-16T15:20:09.427723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'72","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454108+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:23:18.454108+0000","last_peered":"2024-01-16T15:23:18.454108+0000","last_clean":"2024-01-16T15:23:18.454108+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:23:18.454108+0000","last_undegraded":"2024-01-16T15:23:18.454108+0000","last_fullsized":"2024-01-16T15:23:18.454108+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.709208+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:20:09.709208+0000","last_peered":"2024-01-16T15:20:09.709208+0000","last_clean":"2024-01-16T15:20:09.709208+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:20:09.709208+0000","last_undegraded":"2024-01-16T15:20:09.709208+0000","last_fullsized":"2024-01-16T15:20:09.709208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'73","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444105+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:23:18.444105+0000","last_peered":"2024-01-16T15:23:18.444105+0000","last_clean":"2024-01-16T15:23:18.444105+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:23:18.444105+0000","last_undegraded":"2024-01-16T15:23:18.444105+0000","last_fullsized":"2024-01-16T15:23:18.444105+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:19.703629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.548197+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:19:59.548197+0000","last_peered":"2024-01-16T15:19:59.548197+0000","last_clean":"2024-01-16T15:19:59.548197+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:19:59.548197+0000","last_undegraded":"2024-01-16T15:19:59.548197+0000","last_fullsized":"2024-01-16T15:19:59.548197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'73","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.443778+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:23:18.443778+0000","last_peered":"2024-01-16T15:23:18.443778+0000","last_clean":"2024-01-16T15:23:18.443778+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:23:18.443778+0000","last_undegraded":"2024-01-16T15:23:18.443778+0000","last_fullsized":"2024-01-16T15:23:18.443778+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.786104+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:19:59.786104+0000","last_peered":"2024-01-16T15:19:59.786104+0000","last_clean":"2024-01-16T15:19:59.786104+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:19:59.786104+0000","last_undegraded":"2024-01-16T15:19:59.786104+0000","last_fullsized":"2024-01-16T15:19:59.786104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:43.538921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'58","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448671+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:23:18.448671+0000","last_peered":"2024-01-16T15:23:18.448671+0000","last_clean":"2024-01-16T15:23:18.448671+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:23:18.448671+0000","last_undegraded":"2024-01-16T15:23:18.448671+0000","last_fullsized":"2024-01-16T15:23:18.448671+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.704056+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:19:59.704056+0000","last_peered":"2024-01-16T15:19:59.704056+0000","last_clean":"2024-01-16T15:19:59.704056+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:19:59.704056+0000","last_undegraded":"2024-01-16T15:19:59.704056+0000","last_fullsized":"2024-01-16T15:19:59.704056+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:27.301228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'61","reported_seq":85,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451772+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:23:18.451772+0000","last_peered":"2024-01-16T15:23:18.451772+0000","last_clean":"2024-01-16T15:23:18.451772+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:18.451772+0000","last_undegraded":"2024-01-16T15:23:18.451772+0000","last_fullsized":"2024-01-16T15:23:18.451772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.707473+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:19:59.707473+0000","last_peered":"2024-01-16T15:19:59.707473+0000","last_clean":"2024-01-16T15:19:59.707473+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:19:59.707473+0000","last_undegraded":"2024-01-16T15:19:59.707473+0000","last_fullsized":"2024-01-16T15:19:59.707473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'62","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'31","reported_seq":48,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.209386+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:21:14.209386+0000","last_peered":"2024-01-16T15:21:14.209386+0000","last_clean":"2024-01-16T15:21:14.209386+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:21:14.209386+0000","last_undegraded":"2024-01-16T15:21:14.209386+0000","last_fullsized":"2024-01-16T15:21:14.209386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.821106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'69","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453483+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:23:18.453483+0000","last_peered":"2024-01-16T15:23:18.453483+0000","last_clean":"2024-01-16T15:23:18.453483+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:23:18.453483+0000","last_undegraded":"2024-01-16T15:23:18.453483+0000","last_fullsized":"2024-01-16T15:23:18.453483+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'319","reported_seq":336,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.401133+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:23:18.401133+0000","last_peered":"2024-01-16T15:23:18.401133+0000","last_clean":"2024-01-16T15:23:18.401133+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:18.401133+0000","last_undegraded":"2024-01-16T15:23:18.401133+0000","last_fullsized":"2024-01-16T15:23:18.401133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:32.472941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1209133,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":318,"num_write_kb":1331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'54","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440515+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:23:18.440515+0000","last_peered":"2024-01-16T15:23:18.440515+0000","last_clean":"2024-01-16T15:23:18.440515+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:18.440515+0000","last_undegraded":"2024-01-16T15:23:18.440515+0000","last_fullsized":"2024-01-16T15:23:18.440515+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:50.060535+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:19:50.060535+0000","last_peered":"2024-01-16T15:19:50.060535+0000","last_clean":"2024-01-16T15:19:50.060535+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:19:50.060535+0000","last_undegraded":"2024-01-16T15:19:50.060535+0000","last_fullsized":"2024-01-16T15:19:50.060535+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:40.359437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'72","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.452704+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:23:18.452704+0000","last_peered":"2024-01-16T15:23:18.452704+0000","last_clean":"2024-01-16T15:23:18.452704+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:18.452704+0000","last_undegraded":"2024-01-16T15:23:18.452704+0000","last_fullsized":"2024-01-16T15:23:18.452704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.562805+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:19:49.562805+0000","last_peered":"2024-01-16T15:19:49.562805+0000","last_clean":"2024-01-16T15:19:49.562805+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:19:49.562805+0000","last_undegraded":"2024-01-16T15:19:49.562805+0000","last_fullsized":"2024-01-16T15:19:49.562805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'63","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456290+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:23:18.456290+0000","last_peered":"2024-01-16T15:23:18.456290+0000","last_clean":"2024-01-16T15:23:18.456290+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:18.456290+0000","last_undegraded":"2024-01-16T15:23:18.456290+0000","last_fullsized":"2024-01-16T15:23:18.456290+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":25,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.702173+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:19:49.702173+0000","last_peered":"2024-01-16T15:19:49.702173+0000","last_clean":"2024-01-16T15:19:49.702173+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:19:49.702173+0000","last_undegraded":"2024-01-16T15:19:49.702173+0000","last_fullsized":"2024-01-16T15:19:49.702173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:11.168897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'46","reported_seq":68,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448040+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:23:18.448040+0000","last_peered":"2024-01-16T15:23:18.448040+0000","last_clean":"2024-01-16T15:23:18.448040+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:18.448040+0000","last_undegraded":"2024-01-16T15:23:18.448040+0000","last_fullsized":"2024-01-16T15:23:18.448040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:21.652051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.782153+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:19:39.782153+0000","last_peered":"2024-01-16T15:19:39.782153+0000","last_clean":"2024-01-16T15:19:39.782153+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:19:39.782153+0000","last_undegraded":"2024-01-16T15:19:39.782153+0000","last_fullsized":"2024-01-16T15:19:39.782153+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:53.963781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'61","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451100+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:23:18.451100+0000","last_peered":"2024-01-16T15:23:18.451100+0000","last_clean":"2024-01-16T15:23:18.451100+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:18.451100+0000","last_undegraded":"2024-01-16T15:23:18.451100+0000","last_fullsized":"2024-01-16T15:23:18.451100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:45.884405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.700531+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:19:39.700531+0000","last_peered":"2024-01-16T15:19:39.700531+0000","last_clean":"2024-01-16T15:19:39.700531+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:19:39.700531+0000","last_undegraded":"2024-01-16T15:19:39.700531+0000","last_fullsized":"2024-01-16T15:19:39.700531+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:59.699408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455552+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:23:18.455552+0000","last_peered":"2024-01-16T15:23:18.455552+0000","last_clean":"2024-01-16T15:23:18.455552+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:18.455552+0000","last_undegraded":"2024-01-16T15:23:18.455552+0000","last_fullsized":"2024-01-16T15:23:18.455552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.561225+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:19:39.561225+0000","last_peered":"2024-01-16T15:19:39.561225+0000","last_clean":"2024-01-16T15:19:39.561225+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:19:39.561225+0000","last_undegraded":"2024-01-16T15:19:39.561225+0000","last_fullsized":"2024-01-16T15:19:39.561225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'62","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460777+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:23:18.460777+0000","last_peered":"2024-01-16T15:23:18.460777+0000","last_clean":"2024-01-16T15:23:18.460777+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:23:18.460777+0000","last_undegraded":"2024-01-16T15:23:18.460777+0000","last_fullsized":"2024-01-16T15:23:18.460777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.422539+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:19:39.422539+0000","last_peered":"2024-01-16T15:19:39.422539+0000","last_clean":"2024-01-16T15:19:39.422539+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:19:39.422539+0000","last_undegraded":"2024-01-16T15:19:39.422539+0000","last_fullsized":"2024-01-16T15:19:39.422539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'56","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455067+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:23:18.455067+0000","last_peered":"2024-01-16T15:23:18.455067+0000","last_clean":"2024-01-16T15:23:18.455067+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:18.455067+0000","last_undegraded":"2024-01-16T15:23:18.455067+0000","last_fullsized":"2024-01-16T15:23:18.455067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'3","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:40.058585+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:19:40.058585+0000","last_peered":"2024-01-16T15:19:40.058585+0000","last_clean":"2024-01-16T15:19:40.058585+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:19:40.058585+0000","last_undegraded":"2024-01-16T15:19:40.058585+0000","last_fullsized":"2024-01-16T15:19:40.058585+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457134+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:18.457134+0000","last_peered":"2024-01-16T15:23:18.457134+0000","last_clean":"2024-01-16T15:23:18.457134+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:18.457134+0000","last_undegraded":"2024-01-16T15:23:18.457134+0000","last_fullsized":"2024-01-16T15:23:18.457134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.557222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.698458+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:19:29.698458+0000","last_peered":"2024-01-16T15:19:29.698458+0000","last_clean":"2024-01-16T15:19:29.698458+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:19:29.698458+0000","last_undegraded":"2024-01-16T15:19:29.698458+0000","last_fullsized":"2024-01-16T15:19:29.698458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:54.307591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'65","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460089+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:23:18.460089+0000","last_peered":"2024-01-16T15:23:18.460089+0000","last_clean":"2024-01-16T15:23:18.460089+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:18.460089+0000","last_undegraded":"2024-01-16T15:23:18.460089+0000","last_fullsized":"2024-01-16T15:23:18.460089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:30.056556+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:19:30.056556+0000","last_peered":"2024-01-16T15:19:30.056556+0000","last_clean":"2024-01-16T15:19:30.056556+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:19:30.056556+0000","last_undegraded":"2024-01-16T15:19:30.056556+0000","last_fullsized":"2024-01-16T15:19:30.056556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'57","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456066+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:23:18.456066+0000","last_peered":"2024-01-16T15:23:18.456066+0000","last_clean":"2024-01-16T15:23:18.456066+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:18.456066+0000","last_undegraded":"2024-01-16T15:23:18.456066+0000","last_fullsized":"2024-01-16T15:23:18.456066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.780012+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:19:29.780012+0000","last_peered":"2024-01-16T15:19:29.780012+0000","last_clean":"2024-01-16T15:19:29.780012+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:19:29.780012+0000","last_undegraded":"2024-01-16T15:19:29.780012+0000","last_fullsized":"2024-01-16T15:19:29.780012+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:02.498301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'53","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451025+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:23:18.451025+0000","last_peered":"2024-01-16T15:23:18.451025+0000","last_clean":"2024-01-16T15:23:18.451025+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:18.451025+0000","last_undegraded":"2024-01-16T15:23:18.451025+0000","last_fullsized":"2024-01-16T15:23:18.451025+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.701583+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:19:29.701583+0000","last_peered":"2024-01-16T15:19:29.701583+0000","last_clean":"2024-01-16T15:19:29.701583+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:19:29.701583+0000","last_undegraded":"2024-01-16T15:19:29.701583+0000","last_fullsized":"2024-01-16T15:19:29.701583+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457653+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:23:18.457653+0000","last_peered":"2024-01-16T15:23:18.457653+0000","last_clean":"2024-01-16T15:23:18.457653+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:18.457653+0000","last_undegraded":"2024-01-16T15:23:18.457653+0000","last_fullsized":"2024-01-16T15:23:18.457653+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:40.476189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'30","reported_seq":47,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:31.869661+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:22:31.869661+0000","last_peered":"2024-01-16T15:22:31.869661+0000","last_clean":"2024-01-16T15:22:31.869661+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:22:31.869661+0000","last_undegraded":"2024-01-16T15:22:31.869661+0000","last_fullsized":"2024-01-16T15:22:31.869661+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451801+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:18.451801+0000","last_peered":"2024-01-16T15:23:18.451801+0000","last_clean":"2024-01-16T15:23:18.451801+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:18.451801+0000","last_undegraded":"2024-01-16T15:23:18.451801+0000","last_fullsized":"2024-01-16T15:23:18.451801+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:29.732315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.778111+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:19:19.778111+0000","last_peered":"2024-01-16T15:19:19.778111+0000","last_clean":"2024-01-16T15:19:19.778111+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:19:19.778111+0000","last_undegraded":"2024-01-16T15:19:19.778111+0000","last_fullsized":"2024-01-16T15:19:19.778111+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:41.055270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'59","reported_seq":86,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454509+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:23:18.454509+0000","last_peered":"2024-01-16T15:23:18.454509+0000","last_clean":"2024-01-16T15:23:18.454509+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:18.454509+0000","last_undegraded":"2024-01-16T15:23:18.454509+0000","last_fullsized":"2024-01-16T15:23:18.454509+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.699352+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:19:19.699352+0000","last_peered":"2024-01-16T15:19:19.699352+0000","last_clean":"2024-01-16T15:19:19.699352+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:19:19.699352+0000","last_undegraded":"2024-01-16T15:19:19.699352+0000","last_fullsized":"2024-01-16T15:19:19.699352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'63","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449103+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:23:18.449103+0000","last_peered":"2024-01-16T15:23:18.449103+0000","last_clean":"2024-01-16T15:23:18.449103+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:23:18.449103+0000","last_undegraded":"2024-01-16T15:23:18.449103+0000","last_fullsized":"2024-01-16T15:23:18.449103+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.418924+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:19:19.418924+0000","last_peered":"2024-01-16T15:19:19.418924+0000","last_clean":"2024-01-16T15:19:19.418924+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:19:19.418924+0000","last_undegraded":"2024-01-16T15:19:19.418924+0000","last_fullsized":"2024-01-16T15:19:19.418924+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'66","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455227+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:23:18.455227+0000","last_peered":"2024-01-16T15:23:18.455227+0000","last_clean":"2024-01-16T15:23:18.455227+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:18.455227+0000","last_undegraded":"2024-01-16T15:23:18.455227+0000","last_fullsized":"2024-01-16T15:23:18.455227+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:19.540697+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:19:19.540697+0000","last_peered":"2024-01-16T15:19:19.540697+0000","last_clean":"2024-01-16T15:19:19.540697+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:19:19.540697+0000","last_undegraded":"2024-01-16T15:19:19.540697+0000","last_fullsized":"2024-01-16T15:19:19.540697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'68","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457418+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:23:18.457418+0000","last_peered":"2024-01-16T15:23:18.457418+0000","last_clean":"2024-01-16T15:23:18.457418+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:18.457418+0000","last_undegraded":"2024-01-16T15:23:18.457418+0000","last_fullsized":"2024-01-16T15:23:18.457418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'33","reported_seq":82,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:14.200536+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:21:14.200536+0000","last_peered":"2024-01-16T15:21:14.200536+0000","last_clean":"2024-01-16T15:21:14.200536+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:21:14.200536+0000","last_undegraded":"2024-01-16T15:21:14.200536+0000","last_fullsized":"2024-01-16T15:21:14.200536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:38.360191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444662+0000","last_change":"2024-01-16T15:15:28.574045+0000","last_active":"2024-01-16T15:23:18.444662+0000","last_peered":"2024-01-16T15:23:18.444662+0000","last_clean":"2024-01-16T15:23:18.444662+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:23:18.444662+0000","last_undegraded":"2024-01-16T15:23:18.444662+0000","last_fullsized":"2024-01-16T15:23:18.444662+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.801252+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:21:09.801252+0000","last_peered":"2024-01-16T15:21:09.801252+0000","last_clean":"2024-01-16T15:21:09.801252+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:21:09.801252+0000","last_undegraded":"2024-01-16T15:21:09.801252+0000","last_fullsized":"2024-01-16T15:21:09.801252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:10.684617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'49","reported_seq":78,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458455+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:23:18.458455+0000","last_peered":"2024-01-16T15:23:18.458455+0000","last_clean":"2024-01-16T15:23:18.458455+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:18.458455+0000","last_undegraded":"2024-01-16T15:23:18.458455+0000","last_fullsized":"2024-01-16T15:23:18.458455+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.424236+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:19:49.424236+0000","last_peered":"2024-01-16T15:19:49.424236+0000","last_clean":"2024-01-16T15:19:49.424236+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:19:49.424236+0000","last_undegraded":"2024-01-16T15:19:49.424236+0000","last_fullsized":"2024-01-16T15:19:49.424236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":271,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.364834+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:20:39.364834+0000","last_peered":"2024-01-16T15:20:39.364834+0000","last_clean":"2024-01-16T15:20:39.364834+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:20:39.364834+0000","last_undegraded":"2024-01-16T15:20:39.364834+0000","last_fullsized":"2024-01-16T15:20:39.364834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4112,"ondisk_log_size":4112,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1213575,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":481,"num_write_kb":1418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2490368,"data_stored":2427150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.151,"15min":1.135},"min":{"1min":0.89800000000000002,"5min":0.66000000000000003,"15min":0.60199999999999998},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.86299999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.5580000000000001,"5min":2.0619999999999998,"15min":5.9619999999999997},"last":1.5329999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1950000000000001,"15min":1.256},"min":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.56200000000000006},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.024},{"interface":"front","average":{"1min":1.359,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.73099999999999998},"max":{"1min":1.956,"5min":1.956,"15min":6.2690000000000001},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.24,"15min":1.228},"min":{"1min":0.80900000000000005,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.7310000000000001,"5min":2.0070000000000001,"15min":3.7290000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.244,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.5449999999999999,"5min":1.8129999999999999,"15min":3.3919999999999999},"last":1.105}]},{"osd":3,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3240000000000001,"15min":1.3660000000000001},"min":{"1min":0.97999999999999998,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.649,"5min":1.9279999999999999,"15min":12.331},"last":1.151},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.3640000000000001},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.645},"last":0.90400000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":0.49199999999999999},"max":{"1min":1.837,"5min":1.837,"15min":6.1799999999999997},"last":1.643},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3240000000000001,"15min":1.3169999999999999},"min":{"1min":0.94699999999999995,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.7410000000000001,"5min":1.9510000000000001,"15min":2.9809999999999999},"last":1.28}]},{"osd":5,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.89000000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.508,"5min":2.508,"15min":2.593},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.502,"15min":1.528},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.54700000000000004},"max":{"1min":2.7090000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.7549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.484,"15min":1.4990000000000001},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.823,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.5640000000000001,"15min":1.585},"min":{"1min":1.0940000000000001,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":4.7649999999999997},"last":1.79}]}]},{"osd":6,"up_from":13,"seq":55834575174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.391},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.355}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.4770000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.208}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":0.85799999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.54}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.26},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.512}]}]},{"osd":5,"up_from":13,"seq":55834575174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1759999999999999,"15min":1.1870000000000001},"min":{"1min":0.754,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":7.6529999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.27,"5min":1.294,"15min":1.2589999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.49299999999999999},"max":{"1min":1.843,"5min":1.8480000000000001,"15min":8.7880000000000003},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.22,"15min":1.216},"min":{"1min":0.86599999999999999,"5min":0.72599999999999998,"15min":0.28899999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":3.835},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.3129999999999999,"15min":1.2889999999999999},"min":{"1min":1.0389999999999999,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.96,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":1.1739999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3009999999999999,"15min":1.329},"min":{"1min":0.745,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.375,"15min":1.3620000000000001},"min":{"1min":1.0169999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.49}]},{"osd":3,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.337},"min":{"1min":0.96199999999999997,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.7130000000000001,"5min":1.742,"15min":3.9319999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.45,"15min":1.4470000000000001},"min":{"1min":0.91800000000000004,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":16.158000000000001},"last":1.401}]},{"osd":4,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.27,"15min":1.2350000000000001},"min":{"1min":0.94099999999999995,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":0.996},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.492,"15min":1.496},"min":{"1min":1.2150000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.4450000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.393,"15min":1.3939999999999999},"min":{"1min":0.995,"5min":0.78600000000000003,"15min":0.46800000000000003},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.431,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":1.052,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":6.093},"last":1.1060000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:22:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.4370000000000001,"15min":1.52},"min":{"1min":1.216,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":20.488},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.5289999999999999,"15min":1.51},"min":{"1min":0.88300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.444,"5min":2.444,"15min":5.0209999999999999},"last":1.5600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1928112,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92443728,"statfs":{"total":96636764160,"available":94662377472,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.385,"15min":1.3919999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.73,"5min":1.73,"15min":17.315999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4319999999999999,"15min":1.431},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.60399999999999998},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":17.279},"last":1.163}]},{"osd":1,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.498},"max":{"1min":1.7509999999999999,"5min":1.839,"15min":6.1849999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1970000000000001,"15min":1.173},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.6080000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":0.87}]},{"osd":2,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.204,"15min":1.196},"min":{"1min":0.93400000000000005,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":3.859},"last":1.109},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.383,"15min":1.367},"min":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.53500000000000003},"max":{"1min":1.6970000000000001,"5min":1.8120000000000001,"15min":6.8179999999999996},"last":1.0489999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.264,"15min":1.306},"min":{"1min":0.89700000000000002,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.204},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3740000000000001,"15min":1.3700000000000001},"min":{"1min":1.022,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.6890000000000001,"5min":1.712,"15min":4.7930000000000001},"last":1.246}]},{"osd":5,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2829999999999999,"15min":1.3009999999999999},"min":{"1min":0.91600000000000004,"5min":0.69999999999999996,"15min":0.48499999999999999},"max":{"1min":1.5780000000000001,"5min":1.8999999999999999,"15min":2.597},"last":1.54},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.395},"min":{"1min":0.92300000000000004,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.8580000000000001,"5min":1.8660000000000001,"15min":17.140999999999998},"last":1.609}]},{"osd":6,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3600000000000001,"15min":1.365},"min":{"1min":0.97199999999999998,"5min":0.88400000000000001,"15min":0.57099999999999995},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":4.1879999999999997},"last":1.379},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.488,"15min":1.5329999999999999},"min":{"1min":1.0820000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.746,"5min":1.946,"15min":12.324999999999999},"last":1.6699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.444,"15min":1.464},"min":{"1min":1.0549999999999999,"5min":0.99199999999999999,"15min":0.53300000000000003},"max":{"1min":1.968,"5min":1.968,"15min":4.9539999999999997},"last":1.694},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5640000000000001,"15min":1.5489999999999999},"min":{"1min":1.2689999999999999,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.782,"5min":2.0699999999999998,"15min":4.6390000000000002},"last":1.5249999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575174,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818788,"kb_used_data":1751580,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92553052,"statfs":{"total":96636764160,"available":94774325248,"internally_reserved":0,"allocated":1793617920,"data_stored":1792786434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.4059999999999999,"5min":1.6240000000000001,"15min":2.27},"last":0.52700000000000002},{"interface":"front","average":{"1min":1,"5min":1.1080000000000001,"15min":1.0609999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.4079999999999999,"5min":6.6740000000000004,"15min":7.375},"last":0.66100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.95299999999999996,"15min":0.98499999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.44900000000000001},"max":{"1min":1.3049999999999999,"5min":1.6839999999999999,"15min":3.1150000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1160000000000001,"15min":1.1100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38500000000000001},"max":{"1min":1.498,"5min":2.7410000000000001,"15min":8.5449999999999999},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1479999999999999,"15min":1.1830000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46100000000000002},"max":{"1min":1.724,"5min":1.8169999999999999,"15min":7.2809999999999997},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.623,"5min":1.986,"15min":3.4279999999999999},"last":1.3129999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.68799999999999994},"max":{"1min":1.6739999999999999,"5min":2.0030000000000001,"15min":7.4950000000000001},"last":2.1989999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.401,"15min":1.423},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.622},"max":{"1min":1.591,"5min":1.9810000000000001,"15min":9.1289999999999996},"last":1.52}]},{"osd":5,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.361,"15min":1.365},"min":{"1min":0.755,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.643,"5min":2.085,"15min":5.0099999999999998},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.377,"15min":1.431},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.60799999999999998},"max":{"1min":1.706,"5min":1.7430000000000001,"15min":11.255000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.399,"15min":1.4790000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.752},"max":{"1min":1.7450000000000001,"5min":2.0569999999999999,"15min":7.4560000000000004},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.4299999999999999,"15min":1.5149999999999999},"min":{"1min":1.034,"5min":1.0209999999999999,"15min":0.67000000000000004},"max":{"1min":1.7769999999999999,"5min":2.2519999999999998,"15min":12.064},"last":1.26}]},{"osd":7,"last update":"Tue Jan 16 15:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4650000000000001,"15min":1.5189999999999999},"min":{"1min":1.038,"5min":1.038,"15min":0.73099999999999998},"max":{"1min":1.6890000000000001,"5min":1.9370000000000001,"15min":7.5339999999999998},"last":1.571},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.474,"15min":1.5349999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.7050000000000001,"5min":2.1760000000000002,"15min":10.804},"last":1.839}]}]},{"osd":2,"up_from":13,"seq":55834575173,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89700000000000002,"15min":0.97899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.314},"max":{"1min":1.3300000000000001,"5min":1.5569999999999999,"15min":24.155000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.0629999999999999,"15min":1.0449999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":1.76,"15min":2.952},"last":0.91500000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.0649999999999999,"15min":1.1180000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.4450000000000001,"5min":1.6970000000000001,"15min":16.776},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.1040000000000001,"15min":1.226},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.46500000000000002},"max":{"1min":1.468,"5min":1.6479999999999999,"15min":25.186},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0569999999999999,"15min":1.1919999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.3240000000000001,"5min":1.788,"15min":37.470999999999997},"last":1.032},{"interface":"front","average":{"1min":1.159,"5min":1.2270000000000001,"15min":1.244},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":6.5549999999999997},"last":1.2230000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.349,"15min":1.5920000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":2.0019999999999998,"5min":2.0680000000000001,"15min":25.062000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.448},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.746},"max":{"1min":1.919,"5min":1.919,"15min":37.307000000000002},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":2.206,"5min":2.206,"15min":6.8849999999999998},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.4139999999999999,"15min":1.538},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.7649999999999999,"5min":1.8839999999999999,"15min":25.141999999999999},"last":2.1139999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3839999999999999,"15min":1.5269999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.81999999999999995},"max":{"1min":1.8300000000000001,"5min":2.0720000000000001,"15min":37.363},"last":1.986},{"interface":"front","average":{"1min":1.478,"5min":1.488,"15min":1.5149999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":6.9630000000000001},"last":1.887}]},{"osd":7,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.498,"15min":1.5860000000000001},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.66100000000000003},"max":{"1min":2.125,"5min":2.125,"15min":25.106999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.4730000000000001,"15min":1.599},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.78400000000000003},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":37.402999999999999},"last":1.649}]}]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.92900000000000005},"min":{"1min":0.625,"5min":0.38900000000000001,"15min":0.36399999999999999},"max":{"1min":1.196,"5min":1.4370000000000001,"15min":14.106999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.478,"5min":1.4990000000000001,"15min":4.3719999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.29799999999999999},"max":{"1min":1.5920000000000001,"5min":3.5409999999999999,"15min":4.8789999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.085,"15min":1.149},"min":{"1min":0.628,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.484,"5min":1.663,"15min":14.055},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2150000000000001,"15min":1.3620000000000001},"min":{"1min":0.86399999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.51,"5min":1.6439999999999999,"15min":25.611000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.60299999999999998,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":1.7090000000000001,"5min":1.952,"15min":4.6699999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.232,"15min":1.254},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.52500000000000002},"max":{"1min":1.472,"5min":3.77,"15min":4.423},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.861,"5min":1.861,"15min":14.161},"last":0.89700000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.5509999999999999},"min":{"1min":0.84999999999999998,"5min":0.60099999999999998,"15min":0.58699999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4239999999999999,"15min":1.5189999999999999},"min":{"1min":1.093,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.9359999999999999,"5min":3.8090000000000002,"15min":16.433},"last":1.095}]},{"osd":6,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.423,"15min":1.496},"min":{"1min":0.93600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":14.212},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.462},"min":{"1min":1.0449999999999999,"5min":0.92800000000000005,"15min":0.63},"max":{"1min":2.0270000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":0.91400000000000003}]},{"osd":7,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4419999999999999,"15min":1.591},"min":{"1min":1.2509999999999999,"5min":0.82499999999999996,"15min":0.75700000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":19.747},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.456,"15min":1.46},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.64100000000000001},"max":{"1min":1.7869999999999999,"5min":3.726,"15min":6.0229999999999997},"last":0.96199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575173,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048736,"kb_used_data":1981528,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323104,"statfs":{"total":96636764160,"available":94538858496,"internally_reserved":0,"allocated":2029084672,"data_stored":2028256257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.82999999999999996,"15min":0.879},"min":{"1min":0.65600000000000003,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.4339999999999999,"5min":1.601,"15min":10.683999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.056,"15min":1.0289999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.41799999999999998},"max":{"1min":1.476,"5min":1.72,"15min":6.2199999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":17.260000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.1160000000000001,"15min":1.1120000000000001},"min":{"1min":0.65100000000000002,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.784,"5min":1.784,"15min":6.4260000000000002},"last":0.60799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.089,"15min":1.0880000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.4730000000000001,"5min":1.706,"15min":3.4670000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.337,"15min":1.2809999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.8400000000000001,"5min":2.2549999999999999,"15min":2.319},"last":1.0389999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.333,"15min":1.393},"min":{"1min":0.76000000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":9.6080000000000005},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.306,"15min":1.2989999999999999},"min":{"1min":1.0009999999999999,"5min":0.83799999999999997,"15min":0.64400000000000002},"max":{"1min":1.5840000000000001,"5min":1.75,"15min":3.6829999999999998},"last":4.0810000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4319999999999999,"15min":1.415},"min":{"1min":1.052,"5min":1.052,"15min":0.624},"max":{"1min":2.222,"5min":2.6890000000000001,"15min":3.1259999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.502,"15min":1.4990000000000001},"min":{"1min":1.2190000000000001,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.2730000000000001,"5min":2.7160000000000002,"15min":6.5430000000000001},"last":1.0129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.395},"min":{"1min":1.008,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":2.0710000000000002,"5min":2.1789999999999998,"15min":11.462},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.454,"15min":1.4570000000000001},"min":{"1min":1.032,"5min":0.92600000000000005,"15min":0.73199999999999998},"max":{"1min":2.093,"5min":2.5680000000000001,"15min":7.1849999999999996},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.49,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.1299999999999999,"5min":2.5099999999999998,"15min":6.9969999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.5069999999999999,"15min":1.478},"min":{"1min":1.1020000000000001,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":2.1669999999999998,"5min":2.633,"15min":3.746},"last":3.976}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1228800,"data_stored":1209245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1224704,"data_stored":1209203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:26.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:26.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.386+0000 7f527ec00640 1 Processor -- start 2024-01-16T15:23:26.383 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.386+0000 7f527ec00640 1 -- start start 2024-01-16T15:23:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.388+0000 7f527ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 0x7f52780ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.388+0000 7f527ec00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52780aede0 0x7f527806f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.388+0000 7f527ec00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5278071de0 con 0x7f527806a3e0 2024-01-16T15:23:26.385 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.388+0000 7f527ec00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52780af270 con 0x7f52780aede0 2024-01-16T15:23:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.388+0000 7f527ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5278072030 con 0x7f52780d0b60 2024-01-16T15:23:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f5277e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f527806a3e0 0x7f527806a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32908/0 (socket says 172.21.15.125:32908) 2024-01-16T15:23:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f5277e00640 1 -- 172.21.15.125:0/3124995179 learned_addr learned my addr 172.21.15.125:0/3124995179 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:26.386 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f527cc00640 1 --2- 172.21.15.125:0/3124995179 >> v2:172.21.15.125:3300/0 conn(0x7f52780aede0 0x7f527806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.387 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f5277400640 1 -- 172.21.15.125:0/3124995179 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873190530 0 0) 0x7f5278071de0 con 0x7f527806a3e0 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f5277400640 1 -- 172.21.15.125:0/3124995179 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5258003520 con 0x7f527806a3e0 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f526f200640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 0x7f52780ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f527cc00640 1 -- 172.21.15.125:0/3124995179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 msgr2=0x7f52780ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f527cc00640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 0x7f52780ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f527cc00640 1 -- 172.21.15.125:0/3124995179 >> v1:172.21.15.125:6789/0 conn(0x7f527806a3e0 legacy=0x7f527806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f527cc00640 1 -- 172.21.15.125:0/3124995179 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278072310 con 0x7f52780aede0 2024-01-16T15:23:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.389+0000 7f526f200640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 0x7f52780ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:26.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.390+0000 7f527cc00640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52780aede0 0x7f527806f080 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f52700097a0 tx=0x7f527002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16b9c06d9f57234 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.391+0000 7f5277400640 1 -- 172.21.15.125:0/3124995179 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527002d040 con 0x7f52780aede0 2024-01-16T15:23:26.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.391+0000 7f5277400640 1 -- 172.21.15.125:0/3124995179 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270031810 con 0x7f52780aede0 2024-01-16T15:23:26.389 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.391+0000 7f5277400640 1 -- 172.21.15.125:0/3124995179 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270031b80 con 0x7f52780aede0 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.391+0000 7f527ec00640 1 -- 172.21.15.125:0/3124995179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52780aede0 msgr2=0x7f527806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.391+0000 7f527ec00640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52780aede0 0x7f527806f080 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f52700097a0 tx=0x7f527002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 -- 172.21.15.125:0/3124995179 shutdown_connections 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52780aede0 0x7f527806f080 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 --2- 172.21.15.125:0/3124995179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780d0b60 0x7f52780ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 -- 172.21.15.125:0/3124995179 >> 172.21.15.125:0/3124995179 conn(0x7f52780cfef0 msgr2=0x7f52780d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:26.390 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 -- 172.21.15.125:0/3124995179 shutdown_connections 2024-01-16T15:23:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.392+0000 7f527ec00640 1 -- 172.21.15.125:0/3124995179 wait complete. 2024-01-16T15:23:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.394+0000 7f527ec00640 1 Processor -- start 2024-01-16T15:23:26.391 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.394+0000 7f527ec00640 1 -- start start 2024-01-16T15:23:26.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780aede0 0x7f5278140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52780d0b60 0x7f52780ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52781597d0 con 0x7f527806a3e0 2024-01-16T15:23:26.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278158bc0 con 0x7f52780d0b60 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.395+0000 7f527ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5278158ea0 con 0x7f52780aede0 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37166/0 (socket says 172.21.15.125:37166) 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 -- 172.21.15.125:0/1663359206 learned_addr learned my addr 172.21.15.125:0/1663359206 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f527cc00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52780d0b60 0x7f52780ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f526f200640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780aede0 0x7f5278140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 -- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780aede0 msgr2=0x7f5278140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780aede0 0x7f5278140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 -- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52780d0b60 msgr2=0x7f52780ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52780d0b60 0x7f52780ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.396+0000 7f5277e00640 1 -- 172.21.15.125:0/1663359206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270009400 con 0x7f527806a3e0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f5277e00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f526800caa0 tx=0x7f526800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526800ac00 con 0x7f527806a3e0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526800ad60 con 0x7f527806a3e0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5278159180 con 0x7f527806a3e0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268012620 con 0x7f527806a3e0 2024-01-16T15:23:26.396 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.397+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527815b3b0 con 0x7f527806a3e0 2024-01-16T15:23:26.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.399+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52780675d0 con 0x7f527806a3e0 2024-01-16T15:23:26.397 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.401+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5268007070 con 0x7f527806a3e0 2024-01-16T15:23:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.402+0000 7f5274c00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 0x7f525006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.402+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5268088ad0 con 0x7f527806a3e0 2024-01-16T15:23:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.402+0000 7f526f200640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 0x7f525006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.403+0000 7f526f200640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 0x7f525006ea10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f52780cfbe0 tx=0x7f5260008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.404 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.407+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5268055070 con 0x7f527806a3e0 2024-01-16T15:23:26.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.600+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527806e390 con 0x7f527806a3e0 2024-01-16T15:23:26.599 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.603+0000 7f5274c00640 1 -- 172.21.15.125:0/1663359206 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f5268017070 con 0x7f527806a3e0 2024-01-16T15:23:26.600 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:26.600 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"fsid":"3c5ae53a-13b6-44c3-9861-4f92782fd96b","created":"2024-01-16T14:56:20.924760+0000","modified":"2024-01-16T15:15:28.491486+0000","last_up_change":"2024-01-16T14:56:33.200328+0000","last_in_change":"2024-01-16T14:56:24.407089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:56:33.985088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:04.806000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:06.026871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f237602d-0b37-40be-8dae-8c522579f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6801","nonce":3892726325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6803","nonce":3892726325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6807","nonce":3892726325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":3892726325},{"type":"v1","addr":"172.21.15.125:6805","nonce":3892726325}]},"public_addr":"172.21.15.125:6801/3892726325","cluster_addr":"172.21.15.125:6803/3892726325","heartbeat_back_addr":"172.21.15.125:6807/3892726325","heartbeat_front_addr":"172.21.15.125:6805/3892726325","state":["exists","up"]},{"osd":1,"uuid":"c57d8db5-afc1-4ae9-8e25-5dbc99da5467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6809","nonce":2568561981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6812","nonce":2568561981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6819","nonce":2568561981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":2568561981},{"type":"v1","addr":"172.21.15.125:6815","nonce":2568561981}]},"public_addr":"172.21.15.125:6809/2568561981","cluster_addr":"172.21.15.125:6812/2568561981","heartbeat_back_addr":"172.21.15.125:6819/2568561981","heartbeat_front_addr":"172.21.15.125:6815/2568561981","state":["exists","up"]},{"osd":2,"uuid":"6ba40f64-5a36-4738-9543-ef1f8b4f4bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6813","nonce":1951203061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6818","nonce":1951203061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6827","nonce":1951203061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":1951203061},{"type":"v1","addr":"172.21.15.125:6822","nonce":1951203061}]},"public_addr":"172.21.15.125:6813/1951203061","cluster_addr":"172.21.15.125:6818/1951203061","heartbeat_back_addr":"172.21.15.125:6827/1951203061","heartbeat_front_addr":"172.21.15.125:6822/1951203061","state":["exists","up"]},{"osd":3,"uuid":"41d61215-6fd0-43de-a7b5-c4ac99339e19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6823","nonce":397977967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6826","nonce":397977967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6831","nonce":397977967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":397977967},{"type":"v1","addr":"172.21.15.125:6829","nonce":397977967}]},"public_addr":"172.21.15.125:6823/397977967","cluster_addr":"172.21.15.125:6826/397977967","heartbeat_back_addr":"172.21.15.125:6831/397977967","heartbeat_front_addr":"172.21.15.125:6829/397977967","state":["exists","up"]},{"osd":4,"uuid":"9394c828-c510-4367-be90-fa386b887393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6809","nonce":1439911452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6811","nonce":1439911452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6815","nonce":1439911452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1439911452},{"type":"v1","addr":"172.21.15.137:6813","nonce":1439911452}]},"public_addr":"172.21.15.137:6809/1439911452","cluster_addr":"172.21.15.137:6811/1439911452","heartbeat_back_addr":"172.21.15.137:6815/1439911452","heartbeat_front_addr":"172.21.15.137:6813/1439911452","state":["exists","up"]},{"osd":5,"uuid":"ae8630f8-7c9f-43fe-ba8b-47066f53b1a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6801","nonce":2884121150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6803","nonce":2884121150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6807","nonce":2884121150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2884121150},{"type":"v1","addr":"172.21.15.137:6805","nonce":2884121150}]},"public_addr":"172.21.15.137:6801/2884121150","cluster_addr":"172.21.15.137:6803/2884121150","heartbeat_back_addr":"172.21.15.137:6807/2884121150","heartbeat_front_addr":"172.21.15.137:6805/2884121150","state":["exists","up"]},{"osd":6,"uuid":"567e5abc-1b15-4cc2-b7b3-97e37c2cdc5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6818","nonce":1720012579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6822","nonce":1720012579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6830","nonce":1720012579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1720012579},{"type":"v1","addr":"172.21.15.137:6826","nonce":1720012579}]},"public_addr":"172.21.15.137:6818/1720012579","cluster_addr":"172.21.15.137:6822/1720012579","heartbeat_back_addr":"172.21.15.137:6830/1720012579","heartbeat_front_addr":"172.21.15.137:6826/1720012579","state":["exists","up"]},{"osd":7,"uuid":"2defd956-d6b8-499c-a7cf-877615a2e0bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6817","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6819","nonce":1873161633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6821","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6823","nonce":1873161633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6829","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6831","nonce":1873161633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6825","nonce":1873161633},{"type":"v1","addr":"172.21.15.137:6827","nonce":1873161633}]},"public_addr":"172.21.15.137:6819/1873161633","cluster_addr":"172.21.15.137:6823/1873161633","heartbeat_back_addr":"172.21.15.137:6831/1873161633","heartbeat_front_addr":"172.21.15.137:6827/1873161633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.671870+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.708547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.458739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.774474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.742510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.904850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.696849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:29.816119+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:26.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 msgr2=0x7f525006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 0x7f525006ea10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f52780cfbe0 tx=0x7f5260008040 comp rx=0 tx=0).stop 2024-01-16T15:23:26.602 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 msgr2=0x7f527815ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f526800caa0 tx=0x7f526800cf60 comp rx=0 tx=0).stop 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 shutdown_connections 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52780d0b60 0x7f52780ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52780aede0 0x7f5278140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f525006c550 0x7f525006ea10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 --2- 172.21.15.125:0/1663359206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f527806a3e0 0x7f527815ae70 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.605+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 >> 172.21.15.125:0/1663359206 conn(0x7f52780cfef0 msgr2=0x7f527815c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.606+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 shutdown_connections 2024-01-16T15:23:26.604 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:26.606+0000 7f527ec00640 1 -- 172.21.15.125:0/1663359206 wait complete. 2024-01-16T15:23:27.622 INFO:tasks.ceph:Scrubbing osd.0 2024-01-16T15:23:27.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:27.848 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.851+0000 7f1bc3c00640 1 Processor -- start 2024-01-16T15:23:27.849 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.852+0000 7f1bc3c00640 1 -- start start 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.853+0000 7f1bc3c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bbc13df80 0x7f1bbc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.853+0000 7f1bc3c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.853+0000 7f1bc3c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc06c470 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.853+0000 7f1bc3c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bbc06c750 con 0x7f1bbc13df80 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.853+0000 7f1bc3c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc06ca30 con 0x7f1bbc13e360 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bbc13df80 0x7f1bbc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.851 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1bbc13df80 0x7f1bbc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41220/0 (socket says 172.21.15.125:41220) 2024-01-16T15:23:27.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 -- 172.21.15.125:0/1533893783 learned_addr learned my addr 172.21.15.125:0/1533893783 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:27.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 -- 172.21.15.125:0/1533893783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 msgr2=0x7f1bbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:27.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 --2- 172.21.15.125:0/1533893783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 -- 172.21.15.125:0/1533893783 >> v1:172.21.15.137:6789/0 conn(0x7f1bbc15f9c0 legacy=0x7f1bbc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:27.852 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.854+0000 7f1bc1200640 1 -- 172.21.15.125:0/1533893783 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc06cd10 con 0x7f1bbc13df80 2024-01-16T15:23:27.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.855+0000 7f1bc1200640 1 --2- 172.21.15.125:0/1533893783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15bee0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009870 tx=0x7f1bac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72ac3a4347d38a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:27.853 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.855+0000 7f1bbbe00640 1 -- 172.21.15.125:0/1533893783 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac031640 con 0x7f1bbc13df80 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.856+0000 7f1bbbe00640 1 -- 172.21.15.125:0/1533893783 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0317a0 con 0x7f1bbc13df80 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.856+0000 7f1bbbe00640 1 -- 172.21.15.125:0/1533893783 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac031b30 con 0x7f1bbc13df80 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.856+0000 7f1bc3c00640 1 -- 172.21.15.125:0/1533893783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc13df80 msgr2=0x7f1bbc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.856+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/1533893783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15bee0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009870 tx=0x7f1bac02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 -- 172.21.15.125:0/1533893783 shutdown_connections 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/1533893783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/1533893783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15bee0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.854 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 -- 172.21.15.125:0/1533893783 >> 172.21.15.125:0/1533893783 conn(0x7f1bbc06a800 msgr2=0x7f1bbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:27.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 -- 172.21.15.125:0/1533893783 shutdown_connections 2024-01-16T15:23:27.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.857+0000 7f1bc3c00640 1 -- 172.21.15.125:0/1533893783 wait complete. 2024-01-16T15:23:27.855 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.859+0000 7f1bc3c00640 1 Processor -- start 2024-01-16T15:23:27.856 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.859+0000 7f1bc3c00640 1 -- start start 2024-01-16T15:23:27.858 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc1f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 0x7f1bbc0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc156e20 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc157090 con 0x7f1bbc13df80 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.860+0000 7f1bc3c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc157370 con 0x7f1bbc13e360 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 0x7f1bbc0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 0x7f1bbc0b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41236/0 (socket says 172.21.15.125:41236) 2024-01-16T15:23:27.859 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 -- 172.21.15.125:0/3828090509 learned_addr learned my addr 172.21.15.125:0/3828090509 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1200640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bb3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc1f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 -- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 msgr2=0x7f1bbc1f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bbc13e360 0x7f1bbc1f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 -- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bbc13df80 msgr2=0x7f1bbc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bbc13df80 0x7f1bbc15a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.861+0000 7f1bc1c00640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac0094d0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.860 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bc1c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 0x7f1bbc0b0bc0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c780 tx=0x7f1bb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:27.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4003d80 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb400ce90 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc157650 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40052b0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.861 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.862+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc15aad0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.863+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1bbc15acf0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.864+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1bb4005410 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.862 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bb9600640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9406c550 0x7f1b9406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1bb40883d0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bb9600640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f1b940747b0 0x7f1b94076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b940772c0 con 0x7f1b940747b0 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bb4088ff0 con 0x7f1bbc15f9c0 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.865+0000 7f1bc1200640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9406c550 0x7f1b9406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.863 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.866+0000 7f1bb3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f1b940747b0 0x7f1b94076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:27.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.866+0000 7f1bc1200640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9406c550 0x7f1b9406ea10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1bac036000 tx=0x7f1bac03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:27.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.866+0000 7f1bb3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f1b940747b0 0x7f1b94076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:27.864 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.867+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== osd.0 v2:172.21.15.125:6800/3892726325 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1b940772c0 con 0x7f1b940747b0 2024-01-16T15:23:27.878 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.881+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 --> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1bbc0ce850 con 0x7f1b940747b0 2024-01-16T15:23:27.879 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.882+0000 7f1bb9600640 1 -- 172.21.15.125:0/3828090509 <== osd.0 v2:172.21.15.125:6800/3892726325 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1ba400a920 con 0x7f1b940747b0 2024-01-16T15:23:27.879 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:27.879 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:27.879 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:27.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.882+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f1b940747b0 msgr2=0x7f1b94076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.880 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.882+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6800/3892726325,v1:172.21.15.125:6801/3892726325] conn(0x7f1b940747b0 0x7f1b94076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9406c550 msgr2=0x7f1b9406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1b9406c550 0x7f1b9406ea10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1bac036000 tx=0x7f1bac03c040 comp rx=0 tx=0).stop 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 msgr2=0x7f1bbc0b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 --2- 172.21.15.125:0/3828090509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bbc15f9c0 0x7f1bbc0b0bc0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c780 tx=0x7f1bb400cc40 comp rx=0 tx=0).stop 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc1c00640 1 -- 172.21.15.125:0/3828090509 reap_dead start 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 shutdown_connections 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 >> 172.21.15.125:0/3828090509 conn(0x7f1bbc06a800 msgr2=0x7f1bbc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:27.881 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 shutdown_connections 2024-01-16T15:23:27.882 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:27.883+0000 7f1bc3c00640 1 -- 172.21.15.125:0/3828090509 wait complete. 2024-01-16T15:23:27.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-16T15:23:28.095 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.098+0000 7ff0e5000640 1 Processor -- start 2024-01-16T15:23:28.096 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.099+0000 7ff0e5000640 1 -- start start 2024-01-16T15:23:28.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.100+0000 7ff0e5000640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0e00cf9a0 0x7ff0e015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.100+0000 7ff0e5000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e0070c40 con 0x7ff0e00ae5a0 2024-01-16T15:23:28.097 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.100+0000 7ff0e5000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e006c470 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.100+0000 7ff0e5000640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e0071450 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.100+0000 7ff0dde00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff0e00ae5a0 0x7ff0e006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50192/0 (socket says 172.21.15.125:50192) 2024-01-16T15:23:28.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0dde00640 1 -- 172.21.15.125:0/1418510172 learned_addr learned my addr 172.21.15.125:0/1418510172 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:28.098 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0de800640 1 --2- 172.21.15.125:0/1418510172 >> v2:172.21.15.125:3300/0 conn(0x7ff0e00cf9a0 0x7ff0e015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0dca00640 1 -- 172.21.15.125:0/1418510172 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656435009 0 0) 0x7ff0e0070c40 con 0x7ff0e00ae5a0 2024-01-16T15:23:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0de800640 1 -- 172.21.15.125:0/1418510172 >> v1:172.21.15.137:6789/0 conn(0x7ff0e00b3ae0 legacy=0x7ff0e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0de800640 1 -- 172.21.15.125:0/1418510172 >> v1:172.21.15.125:6789/0 conn(0x7ff0e00ae5a0 legacy=0x7ff0e006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.101+0000 7ff0de800640 1 -- 172.21.15.125:0/1418510172 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e006c750 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.102+0000 7ff0de800640 1 --2- 172.21.15.125:0/1418510172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015e430 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0976262031d58a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.102+0000 7ff0dca00640 1 -- 172.21.15.125:0/1418510172 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d802ccf0 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0dca00640 1 -- 172.21.15.125:0/1418510172 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d802ce50 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0dca00640 1 -- 172.21.15.125:0/1418510172 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8034660 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 -- 172.21.15.125:0/1418510172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00cf9a0 msgr2=0x7ff0e015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 --2- 172.21.15.125:0/1418510172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015e430 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d802bf40 comp rx=0 tx=0).stop 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 -- 172.21.15.125:0/1418510172 shutdown_connections 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 --2- 172.21.15.125:0/1418510172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015e430 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 -- 172.21.15.125:0/1418510172 >> 172.21.15.125:0/1418510172 conn(0x7ff0e006a800 msgr2=0x7ff0e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 -- 172.21.15.125:0/1418510172 shutdown_connections 2024-01-16T15:23:28.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.103+0000 7ff0e5000640 1 -- 172.21.15.125:0/1418510172 wait complete. 2024-01-16T15:23:28.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.105+0000 7ff0e5000640 1 Processor -- start 2024-01-16T15:23:28.102 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.105+0000 7ff0e5000640 1 -- start start 2024-01-16T15:23:28.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e00ae5a0 0x7ff0e01f5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00cecb0 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e01f5d40 con 0x7ff0e00cf9a0 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0e5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00ce0a0 con 0x7ff0e00ae5a0 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41250/0 (socket says 172.21.15.125:41250) 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.106+0000 7ff0dd400640 1 -- 172.21.15.125:0/3711115494 learned_addr learned my addr 172.21.15.125:0/3711115494 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0de800640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 -- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e00ae5a0 msgr2=0x7ff0e01f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e00ae5a0 0x7ff0e01f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 -- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0e00cf9a0 msgr2=0x7ff0e015fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 -- 172.21.15.125:0/3711115494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d8009a70 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0dd400640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c430 tx=0x7ff0c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8011790 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c80118f0 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8011ce0 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.107+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e00ce380 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.108+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e00ce940 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.109+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e00675d0 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.110+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff0c801c460 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.107 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.111+0000 7ff0d6000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 0x7ff0b806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.111+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff0c8088a70 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.111+0000 7ff0dde00640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 0x7ff0b806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.108 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.112+0000 7ff0dde00640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 0x7ff0b806ea40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0007fd0 tx=0x7ff0d0007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.112 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.116+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0c8055010 con 0x7ff0e00b3ae0 2024-01-16T15:23:28.302 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.305+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0e0070550 con 0x7ff0b806c580 2024-01-16T15:23:28.303 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.306+0000 7ff0d6000640 1 -- 172.21.15.125:0/3711115494 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff0d00070d0 con 0x7ff0b806c580 2024-01-16T15:23:28.303 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 0 to deep-scrub 2024-01-16T15:23:28.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 msgr2=0x7ff0b806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 0x7ff0b806ea40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0007fd0 tx=0x7ff0d0007920 comp rx=0 tx=0).stop 2024-01-16T15:23:28.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 msgr2=0x7ff0e015e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c430 tx=0x7ff0c800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.306 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 shutdown_connections 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0e00cf9a0 0x7ff0e015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0b806c580 0x7ff0b806ea40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e00b3ae0 0x7ff0e015e150 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 --2- 172.21.15.125:0/3711115494 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff0e00ae5a0 0x7ff0e01f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 >> 172.21.15.125:0/3711115494 conn(0x7ff0e006a800 msgr2=0x7ff0e00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 shutdown_connections 2024-01-16T15:23:28.307 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.309+0000 7ff0e5000640 1 -- 172.21.15.125:0/3711115494 wait complete. 2024-01-16T15:23:28.326 INFO:tasks.ceph:Scrubbing osd.1 2024-01-16T15:23:28.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:28.542 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.545+0000 7f37f7400640 1 Processor -- start 2024-01-16T15:23:28.543 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.545+0000 7f37f7400640 1 -- start start 2024-01-16T15:23:28.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f009a170 0x7f37f00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37f006c6b0 0x7f37f00cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.544 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f7400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f006cea0 con 0x7f37f00cd240 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f7400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f006cbc0 con 0x7f37f006c6b0 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f006f350 con 0x7f37f009a170 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f5400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37f006c6b0 0x7f37f00cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f5400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37f006c6b0 0x7f37f00cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41262/0 (socket says 172.21.15.125:41262) 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37f5400640 1 -- 172.21.15.125:0/2104624137 learned_addr learned my addr 172.21.15.125:0/2104624137 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.547+0000 7f37efe00640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f009a170 0x7f37f00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.548+0000 7f37f5400640 1 -- 172.21.15.125:0/2104624137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f009a170 msgr2=0x7f37f00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.548+0000 7f37f5400640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f009a170 0x7f37f00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.548+0000 7f37f5400640 1 -- 172.21.15.125:0/2104624137 >> v1:172.21.15.137:6789/0 conn(0x7f37f00cd240 legacy=0x7f37f0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:28.546 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.548+0000 7f37f5400640 1 -- 172.21.15.125:0/2104624137 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f006f630 con 0x7f37f006c6b0 2024-01-16T15:23:28.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.549+0000 7f37f5400640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00cf640 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009870 tx=0x7f37e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb682f385c4b5370 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.549+0000 7f37ef400640 1 -- 172.21.15.125:0/2104624137 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8030840 con 0x7f37f006c6b0 2024-01-16T15:23:28.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.549+0000 7f37ef400640 1 -- 172.21.15.125:0/2104624137 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e80309a0 con 0x7f37f006c6b0 2024-01-16T15:23:28.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.549+0000 7f37ef400640 1 -- 172.21.15.125:0/2104624137 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8030d30 con 0x7f37f006c6b0 2024-01-16T15:23:28.547 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 -- 172.21.15.125:0/2104624137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 msgr2=0x7f37f00cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00cf640 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009870 tx=0x7f37e802b980 comp rx=0 tx=0).stop 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 -- 172.21.15.125:0/2104624137 shutdown_connections 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00cf640 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 --2- 172.21.15.125:0/2104624137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f009a170 0x7f37f00b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 -- 172.21.15.125:0/2104624137 >> 172.21.15.125:0/2104624137 conn(0x7f37f00b2d00 msgr2=0x7f37f00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 -- 172.21.15.125:0/2104624137 shutdown_connections 2024-01-16T15:23:28.548 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.550+0000 7f37f7400640 1 -- 172.21.15.125:0/2104624137 wait complete. 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.552+0000 7f37f7400640 1 Processor -- start 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.552+0000 7f37f7400640 1 -- start start 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37f009a170 0x7f37f00c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f00cd240 0x7f37f00c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f015ad00 con 0x7f37f006c6b0 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00c4dc0 con 0x7f37f009a170 2024-01-16T15:23:28.550 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f7400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00c50a0 con 0x7f37f00cd240 2024-01-16T15:23:28.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41274/0 (socket says 172.21.15.125:41274) 2024-01-16T15:23:28.551 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 -- 172.21.15.125:0/3920596922 learned_addr learned my addr 172.21.15.125:0/3920596922 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:28.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37efe00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37f009a170 0x7f37f00c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 -- 172.21.15.125:0/3920596922 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f00cd240 msgr2=0x7f37f00c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.552 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f37f00cd240 0x7f37f00c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 -- 172.21.15.125:0/3920596922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37f009a170 msgr2=0x7f37f00c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37f009a170 0x7f37f00c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.553+0000 7f37f4a00640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e80094d0 con 0x7f37f006c6b0 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37f4a00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00c7480 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c450 tx=0x7f37e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00052d0 con 0x7f37f006c6b0 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e0005430 con 0x7f37f006c6b0 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0012870 con 0x7f37f006c6b0 2024-01-16T15:23:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f00c5380 con 0x7f37f006c6b0 2024-01-16T15:23:28.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.554+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f00c5940 con 0x7f37f006c6b0 2024-01-16T15:23:28.554 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.555+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f37f00cd620 con 0x7f37f006c6b0 2024-01-16T15:23:28.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.556+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f37e00129d0 con 0x7f37f006c6b0 2024-01-16T15:23:28.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37ecc00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37c806c550 0x7f37c806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.555 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37e0089100 con 0x7f37f006c6b0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37efe00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37c806c550 0x7f37c806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37ecc00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7f37c80747b0 0x7f37c8076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37c80772c0 con 0x7f37c80747b0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.557+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37e0089820 con 0x7f37f006c6b0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.558+0000 7f37f5400640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7f37c80747b0 0x7f37c8076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.558+0000 7f37efe00640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37c806c550 0x7f37c806ea10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f37f0070270 tx=0x7f37d800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.556 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.558+0000 7f37f5400640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7f37c80747b0 0x7f37c8076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.557 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.559+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== osd.1 v2:172.21.15.125:6808/2568561981 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f37c80772c0 con 0x7f37c80747b0 2024-01-16T15:23:28.570 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.573+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 --> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f37f00675d0 con 0x7f37c80747b0 2024-01-16T15:23:28.571 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37ecc00640 1 -- 172.21.15.125:0/3920596922 <== osd.1 v2:172.21.15.125:6808/2568561981 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f37f00675d0 con 0x7f37c80747b0 2024-01-16T15:23:28.571 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:28.571 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:28.571 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7f37c80747b0 msgr2=0x7f37c8076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6808/2568561981,v1:172.21.15.125:6809/2568561981] conn(0x7f37c80747b0 0x7f37c8076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37c806c550 msgr2=0x7f37c806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f37c806c550 0x7f37c806ea10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f37f0070270 tx=0x7f37d800b040 comp rx=0 tx=0).stop 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 msgr2=0x7f37f00c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.574+0000 7f37f7400640 1 --2- 172.21.15.125:0/3920596922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37f006c6b0 0x7f37f00c7480 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c450 tx=0x7f37e000c910 comp rx=0 tx=0).stop 2024-01-16T15:23:28.573 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.575+0000 7f37f5400640 1 -- 172.21.15.125:0/3920596922 reap_dead start 2024-01-16T15:23:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.575+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 shutdown_connections 2024-01-16T15:23:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.575+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 >> 172.21.15.125:0/3920596922 conn(0x7f37f00b2d00 msgr2=0x7f37f00b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.575+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 shutdown_connections 2024-01-16T15:23:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.575+0000 7f37f7400640 1 -- 172.21.15.125:0/3920596922 wait complete. 2024-01-16T15:23:28.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-16T15:23:28.797 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.800+0000 7f0097c00640 1 Processor -- start 2024-01-16T15:23:28.798 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.800+0000 7f0097c00640 1 -- start start 2024-01-16T15:23:28.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0097c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009013df80 0x7f009015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0097c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009006c470 con 0x7f00900cf9a0 2024-01-16T15:23:28.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0097c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009006c750 con 0x7f009013e360 2024-01-16T15:23:28.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0097c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009006ca30 con 0x7f009013df80 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0095200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009013df80 0x7f009015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0095c00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f00900cf9a0 0x7f00900cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50202/0 (socket says 172.21.15.125:50202) 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0095c00640 1 -- 172.21.15.125:0/1249961572 learned_addr learned my addr 172.21.15.125:0/1249961572 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0095200640 1 -- 172.21.15.125:0/1249961572 >> v1:172.21.15.137:6789/0 conn(0x7f009013e360 legacy=0x7f00900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f0095200640 1 -- 172.21.15.125:0/1249961572 >> v1:172.21.15.125:6789/0 conn(0x7f00900cf9a0 legacy=0x7f00900cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.803+0000 7f0095200640 1 -- 172.21.15.125:0/1249961572 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009006cd10 con 0x7f009013df80 2024-01-16T15:23:28.802 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.802+0000 7f008f400640 1 -- 172.21.15.125:0/1249961572 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403392356 0 0) 0x7f009006c470 con 0x7f00900cf9a0 2024-01-16T15:23:28.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f0095200640 1 --2- 172.21.15.125:0/1249961572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013df80 0x7f009015bee0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f00800097a0 tx=0x7f008002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f1ccd7e7b87b1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f008f400640 1 -- 172.21.15.125:0/1249961572 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080031830 con 0x7f009013df80 2024-01-16T15:23:28.803 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f008f400640 1 -- 172.21.15.125:0/1249961572 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0080031990 con 0x7f009013df80 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f008f400640 1 -- 172.21.15.125:0/1249961572 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080031d20 con 0x7f009013df80 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f0097c00640 1 -- 172.21.15.125:0/1249961572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013df80 msgr2=0x7f009015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.804+0000 7f0097c00640 1 --2- 172.21.15.125:0/1249961572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013df80 0x7f009015bee0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f00800097a0 tx=0x7f008002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.805+0000 7f0097c00640 1 -- 172.21.15.125:0/1249961572 shutdown_connections 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.805+0000 7f0097c00640 1 --2- 172.21.15.125:0/1249961572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013df80 0x7f009015bee0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.804 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.805+0000 7f0097c00640 1 -- 172.21.15.125:0/1249961572 >> 172.21.15.125:0/1249961572 conn(0x7f009006a800 msgr2=0x7f009006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:28.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.805+0000 7f0097c00640 1 -- 172.21.15.125:0/1249961572 shutdown_connections 2024-01-16T15:23:28.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.805+0000 7f0097c00640 1 -- 172.21.15.125:0/1249961572 wait complete. 2024-01-16T15:23:28.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.806+0000 7f0097c00640 1 Processor -- start 2024-01-16T15:23:28.805 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 -- start start 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00900cf9a0 0x7f009013d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 0x7f00900b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00901e8c80 con 0x7f009013e360 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00901549a0 con 0x7f009013df80 2024-01-16T15:23:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.807+0000 7f0097c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00901e8000 con 0x7f00900cf9a0 2024-01-16T15:23:28.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41286/0 (socket says 172.21.15.125:41286) 2024-01-16T15:23:28.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 -- 172.21.15.125:0/3884450660 learned_addr learned my addr 172.21.15.125:0/3884450660 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:28.809 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 -- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00900cf9a0 msgr2=0x7f009013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00900cf9a0 0x7f009013d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 -- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 msgr2=0x7f00900b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f008fe00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 0x7f00900b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 0x7f00900b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f0095c00640 1 -- 172.21.15.125:0/3884450660 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080009400 con 0x7f009013e360 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.808+0000 7f008fe00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 0x7f00900b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:28.810 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f0095c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f008800c6d0 tx=0x7f008800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088004d60 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008800a640 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00901e82e0 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008800aa70 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.809+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00901e88a0 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.810+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00900675d0 con 0x7f009013e360 2024-01-16T15:23:28.811 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.811+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f00880070d0 con 0x7f009013e360 2024-01-16T15:23:28.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.812+0000 7f008cc00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 0x7f006806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.812 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.812+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f00880884e0 con 0x7f009013e360 2024-01-16T15:23:28.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.812+0000 7f0095200640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 0x7f006806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.813 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.813+0000 7f0095200640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 0x7f006806ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0080036000 tx=0x7f0080030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.814 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:28.817+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00880549f0 con 0x7f009013e360 2024-01-16T15:23:29.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.006+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f009015f9c0 con 0x7f006806c550 2024-01-16T15:23:29.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.007+0000 7f008cc00640 1 -- 172.21.15.125:0/3884450660 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0080031b00 con 0x7f006806c550 2024-01-16T15:23:29.005 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 1 to deep-scrub 2024-01-16T15:23:29.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 msgr2=0x7f006806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 0x7f006806ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0080036000 tx=0x7f0080030000 comp rx=0 tx=0).stop 2024-01-16T15:23:29.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 msgr2=0x7f00900b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f008800c6d0 tx=0x7f008800cb90 comp rx=0 tx=0).stop 2024-01-16T15:23:29.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 shutdown_connections 2024-01-16T15:23:29.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f006806c550 0x7f006806ea10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f009013e360 0x7f00900b3dc0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009013df80 0x7f00900b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 --2- 172.21.15.125:0/3884450660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00900cf9a0 0x7f009013d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 >> 172.21.15.125:0/3884450660 conn(0x7f009006a800 msgr2=0x7f009015a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 shutdown_connections 2024-01-16T15:23:29.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.010+0000 7f0097c00640 1 -- 172.21.15.125:0/3884450660 wait complete. 2024-01-16T15:23:29.027 INFO:tasks.ceph:Scrubbing osd.2 2024-01-16T15:23:29.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:29.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.251+0000 7febc2200640 1 Processor -- start 2024-01-16T15:23:29.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.251+0000 7febc2200640 1 -- start start 2024-01-16T15:23:29.249 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc143c40 0x7febbc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc2200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7febbc0ccc10 0x7febbc06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc2200640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc070e50 con 0x7febbc0717d0 2024-01-16T15:23:29.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc2200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc0723c0 con 0x7febbc0ccc10 2024-01-16T15:23:29.250 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc144020 con 0x7febbc143c40 2024-01-16T15:23:29.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc1800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7febbc0ccc10 0x7febbc06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febbbe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7febbc0717d0 0x7febbc0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50218/0 (socket says 172.21.15.125:50218) 2024-01-16T15:23:29.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febbbe00640 1 -- 172.21.15.125:0/2232807221 learned_addr learned my addr 172.21.15.125:0/2232807221 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:29.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.253+0000 7febc0e00640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc143c40 0x7febbc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.251 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.254+0000 7febc1800640 1 -- 172.21.15.125:0/2232807221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc143c40 msgr2=0x7febbc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.254+0000 7febc1800640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc143c40 0x7febbc071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.254+0000 7febc1800640 1 -- 172.21.15.125:0/2232807221 >> v1:172.21.15.125:6789/0 conn(0x7febbc0717d0 legacy=0x7febbc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.254+0000 7febc1800640 1 -- 172.21.15.125:0/2232807221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc071e00 con 0x7febbc0ccc10 2024-01-16T15:23:29.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.254+0000 7febbb400640 1 -- 172.21.15.125:0/2232807221 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607052388 0 0) 0x7febbc070e50 con 0x7febbc0717d0 2024-01-16T15:23:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.255+0000 7febc1800640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc0ccc10 0x7febbc06d160 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7febb4009d90 tx=0x7febb402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0290c3b2d597d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.255+0000 7febbb400640 1 -- 172.21.15.125:0/2232807221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb402cbc0 con 0x7febbc0ccc10 2024-01-16T15:23:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.255+0000 7febbb400640 1 -- 172.21.15.125:0/2232807221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb402cd20 con 0x7febbc0ccc10 2024-01-16T15:23:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.255+0000 7febbb400640 1 -- 172.21.15.125:0/2232807221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4034660 con 0x7febbc0ccc10 2024-01-16T15:23:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 -- 172.21.15.125:0/2232807221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc0ccc10 msgr2=0x7febbc06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc0ccc10 0x7febbc06d160 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7febb4009d90 tx=0x7febb402be10 comp rx=0 tx=0).stop 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 -- 172.21.15.125:0/2232807221 shutdown_connections 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc0ccc10 0x7febbc06d160 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 --2- 172.21.15.125:0/2232807221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc143c40 0x7febbc071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 -- 172.21.15.125:0/2232807221 >> 172.21.15.125:0/2232807221 conn(0x7febbc06a800 msgr2=0x7febbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 -- 172.21.15.125:0/2232807221 shutdown_connections 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.256+0000 7febc2200640 1 -- 172.21.15.125:0/2232807221 wait complete. 2024-01-16T15:23:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.258+0000 7febc2200640 1 Processor -- start 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.258+0000 7febc2200640 1 -- start start 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febbc0717d0 0x7febbc141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc0ccc10 0x7febbc13e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 0x7febbc15a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc0ac550 con 0x7febbc143c40 2024-01-16T15:23:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc13e6a0 con 0x7febbc0717d0 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc15acf0 con 0x7febbc0ccc10 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 0x7febbc15a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc1800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 0x7febbc15a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41304/0 (socket says 172.21.15.125:41304) 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.259+0000 7febc1800640 1 -- 172.21.15.125:0/240284351 learned_addr learned my addr 172.21.15.125:0/240284351 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc0e00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febbc0717d0 0x7febbc141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 -- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc0ccc10 msgr2=0x7febbc13e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febbbe00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc0ccc10 0x7febbc13e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc0ccc10 0x7febbc13e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 -- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febbc0717d0 msgr2=0x7febbc141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febbc0717d0 0x7febbc141c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb4009a70 con 0x7febbc143c40 2024-01-16T15:23:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc0e00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febbc0717d0 0x7febbc141c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febbbe00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7febbc0ccc10 0x7febbc13e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc1800640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 0x7febbc15a7b0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7febb4009d90 tx=0x7febb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4040650 con 0x7febbc143c40 2024-01-16T15:23:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb402ce90 con 0x7febbc143c40 2024-01-16T15:23:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4040a10 con 0x7febbc143c40 2024-01-16T15:23:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.260+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc142270 con 0x7febbc143c40 2024-01-16T15:23:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.261+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc142830 con 0x7febbc143c40 2024-01-16T15:23:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.261+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7febbc06cc40 con 0x7febbc143c40 2024-01-16T15:23:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.263+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7febb4040b70 con 0x7febbc143c40 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.263+0000 7febb8c00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feb9406c550 0x7feb9406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7febb40ab8b0 con 0x7febbc143c40 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febc0e00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feb9406c550 0x7feb9406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febb8c00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7feb940747b0 0x7feb94076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb940772c0 con 0x7feb940747b0 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7febb40ac4d0 con 0x7febbc143c40 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.264+0000 7febbbe00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7feb940747b0 0x7feb94076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.265+0000 7febc0e00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feb9406c550 0x7feb9406ea10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7febac006fd0 tx=0x7febac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.265+0000 7febbbe00640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7feb940747b0 0x7feb94076bf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.266+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== osd.2 v2:172.21.15.125:6810/1951203061 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7feb940772c0 con 0x7feb940747b0 2024-01-16T15:23:29.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.280+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 --> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7febbc15b4c0 con 0x7feb940747b0 2024-01-16T15:23:29.278 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.280+0000 7febb8c00640 1 -- 172.21.15.125:0/240284351 <== osd.2 v2:172.21.15.125:6810/1951203061 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7feba40018d0 con 0x7feb940747b0 2024-01-16T15:23:29.279 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:29.279 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:29.279 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:29.280 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7feb940747b0 msgr2=0x7feb94076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6810/1951203061,v1:172.21.15.125:6813/1951203061] conn(0x7feb940747b0 0x7feb94076bf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feb9406c550 msgr2=0x7feb9406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7feb9406c550 0x7feb9406ea10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7febac006fd0 tx=0x7febac008040 comp rx=0 tx=0).stop 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 msgr2=0x7febbc15a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 --2- 172.21.15.125:0/240284351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febbc143c40 0x7febbc15a7b0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7febb4009d90 tx=0x7febb4004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc1800640 1 -- 172.21.15.125:0/240284351 reap_dead start 2024-01-16T15:23:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 shutdown_connections 2024-01-16T15:23:29.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 >> 172.21.15.125:0/240284351 conn(0x7febbc06a800 msgr2=0x7febbc0aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 shutdown_connections 2024-01-16T15:23:29.282 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.281+0000 7febc2200640 1 -- 172.21.15.125:0/240284351 wait complete. 2024-01-16T15:23:29.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-16T15:23:29.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.503+0000 7f45c5e00640 1 Processor -- start 2024-01-16T15:23:29.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.503+0000 7f45c5e00640 1 -- start start 2024-01-16T15:23:29.503 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45c013cda0 0x7f45c015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 0x7f45c0070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c00ce4f0 con 0x7f45c015dfa0 2024-01-16T15:23:29.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c00700f0 con 0x7f45c013cda0 2024-01-16T15:23:29.504 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c015e3e0 con 0x7f45c00ce0c0 2024-01-16T15:23:29.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c4a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45c013cda0 0x7f45c015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45bfe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f45c015dfa0 0x7f45c00cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50228/0 (socket says 172.21.15.125:50228) 2024-01-16T15:23:29.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45bfe00640 1 -- 172.21.15.125:0/2631652834 learned_addr learned my addr 172.21.15.125:0/2631652834 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:29.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.505+0000 7f45c5400640 1 --2- 172.21.15.125:0/2631652834 >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 0x7f45c0070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45bf400640 1 -- 172.21.15.125:0/2631652834 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841743318 0 0) 0x7f45c00ce4f0 con 0x7f45c015dfa0 2024-01-16T15:23:29.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45bf400640 1 -- 172.21.15.125:0/2631652834 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a0003520 con 0x7f45c015dfa0 2024-01-16T15:23:29.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45c4a00640 1 -- 172.21.15.125:0/2631652834 >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 msgr2=0x7f45c0070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.506 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45c4a00640 1 --2- 172.21.15.125:0/2631652834 >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 0x7f45c0070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45c4a00640 1 -- 172.21.15.125:0/2631652834 >> v1:172.21.15.125:6789/0 conn(0x7f45c015dfa0 legacy=0x7f45c00cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45c4a00640 1 -- 172.21.15.125:0/2631652834 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00ce720 con 0x7f45c013cda0 2024-01-16T15:23:29.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.506+0000 7f45c5400640 1 --2- 172.21.15.125:0/2631652834 >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 0x7f45c0070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.507 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.507+0000 7f45c4a00640 1 --2- 172.21.15.125:0/2631652834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c013cda0 0x7f45c015da60 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f45c00aea90 tx=0x7f45b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39927936985eed5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.507+0000 7f45bf400640 1 -- 172.21.15.125:0/2631652834 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b002e070 con 0x7f45c013cda0 2024-01-16T15:23:29.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.507+0000 7f45bf400640 1 -- 172.21.15.125:0/2631652834 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b00315a0 con 0x7f45c013cda0 2024-01-16T15:23:29.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.507+0000 7f45bf400640 1 -- 172.21.15.125:0/2631652834 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0031930 con 0x7f45c013cda0 2024-01-16T15:23:29.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 -- 172.21.15.125:0/2631652834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c013cda0 msgr2=0x7f45c015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 --2- 172.21.15.125:0/2631652834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c013cda0 0x7f45c015da60 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f45c00aea90 tx=0x7f45b002b840 comp rx=0 tx=0).stop 2024-01-16T15:23:29.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 -- 172.21.15.125:0/2631652834 shutdown_connections 2024-01-16T15:23:29.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 --2- 172.21.15.125:0/2631652834 >> v2:172.21.15.137:3300/0 conn(0x7f45c00ce0c0 0x7f45c0070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 --2- 172.21.15.125:0/2631652834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c013cda0 0x7f45c015da60 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 -- 172.21.15.125:0/2631652834 >> 172.21.15.125:0/2631652834 conn(0x7f45c006a3e0 msgr2=0x7f45c006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.516 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 -- 172.21.15.125:0/2631652834 shutdown_connections 2024-01-16T15:23:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.508+0000 7f45c5e00640 1 -- 172.21.15.125:0/2631652834 wait complete. 2024-01-16T15:23:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.510+0000 7f45c5e00640 1 Processor -- start 2024-01-16T15:23:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.510+0000 7f45c5e00640 1 -- start start 2024-01-16T15:23:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45c00ce0c0 0x7f45c013d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45c013cda0 0x7f45c0133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c0135840 con 0x7f45c015dfa0 2024-01-16T15:23:29.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c01354a0 con 0x7f45c013cda0 2024-01-16T15:23:29.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c013f140 con 0x7f45c00ce0c0 2024-01-16T15:23:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41316/0 (socket says 172.21.15.125:41316) 2024-01-16T15:23:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 -- 172.21.15.125:0/1288026860 learned_addr learned my addr 172.21.15.125:0/1288026860 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 -- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45c00ce0c0 msgr2=0x7f45c013d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45c00ce0c0 0x7f45c013d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45bfe00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45c013cda0 0x7f45c0133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 -- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45c013cda0 msgr2=0x7f45c0133ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45c013cda0 0x7f45c0133ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.511+0000 7f45c5400640 1 -- 172.21.15.125:0/1288026860 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0009400 con 0x7f45c015dfa0 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45c5400640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f45b800cd50 tx=0x7f45b800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b800f680 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b800f800 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c013f420 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b800fbb0 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.512+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c0133ff0 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.513+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c0067450 con 0x7f45c015dfa0 2024-01-16T15:23:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.514+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f45b800b040 con 0x7f45c015dfa0 2024-01-16T15:23:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.515+0000 7f45bcc00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 0x7f459806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.515+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f45b8089150 con 0x7f45c015dfa0 2024-01-16T15:23:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.515+0000 7f45c4a00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 0x7f459806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.516+0000 7f45c4a00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 0x7f459806ea10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f45b0036000 tx=0x7f45b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.520+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45b8055a70 con 0x7f45c015dfa0 2024-01-16T15:23:29.708 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.711+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f45c00cd760 con 0x7f459806c550 2024-01-16T15:23:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 2 to deep-scrub 2024-01-16T15:23:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.712+0000 7f45bcc00640 1 -- 172.21.15.125:0/1288026860 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f45c00cd760 con 0x7f459806c550 2024-01-16T15:23:29.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 msgr2=0x7f459806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 0x7f459806ea10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f45b0036000 tx=0x7f45b0030000 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 msgr2=0x7f45c0159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f45b800cd50 tx=0x7f45b800a650 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 shutdown_connections 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f459806c550 0x7f459806ea10 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45c015dfa0 0x7f45c0159450 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45c013cda0 0x7f45c0133ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 --2- 172.21.15.125:0/1288026860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45c00ce0c0 0x7f45c013d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.714+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 >> 172.21.15.125:0/1288026860 conn(0x7f45c006a3e0 msgr2=0x7f45c015d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.715+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 shutdown_connections 2024-01-16T15:23:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.715+0000 7f45c5e00640 1 -- 172.21.15.125:0/1288026860 wait complete. 2024-01-16T15:23:29.730 INFO:tasks.ceph:Scrubbing osd.3 2024-01-16T15:23:29.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:29.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.942+0000 7f13f0000640 1 Processor -- start 2024-01-16T15:23:29.939 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.942+0000 7f13f0000640 1 -- start start 2024-01-16T15:23:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.943+0000 7f13f0000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f13e80cd240 0x7f13e80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13f0000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13e8099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13f0000640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e806cea0 con 0x7f13e8099fa0 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13f0000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e809a430 con 0x7f13e80cd240 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13f0000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e806cbc0 con 0x7f13e806c6b0 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13ee000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f13e8099fa0 0x7f13e806b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50232/0 (socket says 172.21.15.125:50232) 2024-01-16T15:23:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13ee000640 1 -- 172.21.15.125:0/1739560058 learned_addr learned my addr 172.21.15.125:0/1739560058 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:29.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13ecc00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13e8099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.943 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13ed600640 1 --2- 172.21.15.125:0/1739560058 >> v2:172.21.15.137:3300/0 conn(0x7f13e80cd240 0x7f13e80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.944+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570678262 0 0) 0x7f13e806cea0 con 0x7f13e8099fa0 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13e8099fa0 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728091867 0 0) 0x7f13c8003520 con 0x7f13e8099fa0 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 msgr2=0x7f13e8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13e8099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 >> v2:172.21.15.137:3300/0 conn(0x7f13e80cd240 msgr2=0x7f13e80b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 --2- 172.21.15.125:0/1739560058 >> v2:172.21.15.137:3300/0 conn(0x7f13e80cd240 0x7f13e80b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e806f350 con 0x7f13e8099fa0 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f13e00019f0 con 0x7f13e8099fa0 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 >> v1:172.21.15.125:6789/0 conn(0x7f13e8099fa0 legacy=0x7f13e806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e809a430 con 0x7f13c80047c0 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e806cbc0 con 0x7f13c8003970 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c800b980 con 0x7f13e806c6b0 2024-01-16T15:23:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.945+0000 7f13ecc00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ed600640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ee000640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ecc00640 1 -- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 msgr2=0x7f13c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ecc00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e806c6b0 0x7f13c8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ecc00640 1 -- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13c8003970 msgr2=0x7f13c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ecc00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.946+0000 7f13ecc00640 1 -- 172.21.15.125:0/1739560058 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8003520 con 0x7f13c80047c0 2024-01-16T15:23:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.947+0000 7f13ecc00640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009a90 tx=0x7f13c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248a2dd0456356e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.947+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d002c6b0 con 0x7f13c80047c0 2024-01-16T15:23:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.947+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d002c810 con 0x7f13c80047c0 2024-01-16T15:23:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.947+0000 7f13e7e00640 1 -- 172.21.15.125:0/1739560058 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d002cba0 con 0x7f13c80047c0 2024-01-16T15:23:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 -- 172.21.15.125:0/1739560058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 msgr2=0x7f13c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 --2- 172.21.15.125:0/1739560058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009a90 tx=0x7f13c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13ee000640 1 -- 172.21.15.125:0/1739560058 reap_dead start 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 -- 172.21.15.125:0/1739560058 shutdown_connections 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 -- 172.21.15.125:0/1739560058 >> 172.21.15.125:0/1739560058 conn(0x7f13e80b2d00 msgr2=0x7f13e80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 -- 172.21.15.125:0/1739560058 shutdown_connections 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.948+0000 7f13f0000640 1 -- 172.21.15.125:0/1739560058 wait complete. 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.950+0000 7f13f0000640 1 Processor -- start 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.950+0000 7f13f0000640 1 -- start start 2024-01-16T15:23:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13e80c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13e80b6660 0x7f13e80b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e8152270 0x7f13e81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80b7020 con 0x7f13c80047c0 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80c7f70 con 0x7f13e80b6660 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.951+0000 7f13f0000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80c76d0 con 0x7f13e8152270 2024-01-16T15:23:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13e80c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13e80c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41334/0 (socket says 172.21.15.125:41334) 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 -- 172.21.15.125:0/2049712017 learned_addr learned my addr 172.21.15.125:0/2049712017 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ee000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e8152270 0x7f13e81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ecc00640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13e80b6660 0x7f13e80b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 -- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e8152270 msgr2=0x7f13e81526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e8152270 0x7f13e81526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 -- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13e80b6660 msgr2=0x7f13e80b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13e80b6660 0x7f13e80b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e80c79b0 con 0x7f13c80047c0 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ecc00640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13e80b6660 0x7f13e80b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ee000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13e8152270 0x7f13e81526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.952+0000 7f13ed600640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13e80c6ea0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f13d80116d0 tx=0x7f13d8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.953+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d800f9c0 con 0x7f13c80047c0 2024-01-16T15:23:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.953+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d800fb20 con 0x7f13c80047c0 2024-01-16T15:23:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.953+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e80c7c90 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.953+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d800fe90 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.953+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e80c73e0 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.954+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f13e80cd620 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.956+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f13d8015460 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13e5600640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13c006c580 0x7f13c006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f13d80890a0 con 0x7f13c80047c0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13ecc00640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13c006c580 0x7f13c006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13e5600640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f13c00747e0 0x7f13c0076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13c00772f0 con 0x7f13c00747e0 2024-01-16T15:23:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13d808eb60 con 0x7f13c80047c0 2024-01-16T15:23:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.957+0000 7f13ee000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f13c00747e0 0x7f13c0076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.958+0000 7f13ecc00640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13c006c580 0x7f13c006ea40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f13d0036000 tx=0x7f13d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.957 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.958+0000 7f13ee000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f13c00747e0 0x7f13c0076c20 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.958 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.959+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== osd.3 v2:172.21.15.125:6820/397977967 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f13c00772f0 con 0x7f13c00747e0 2024-01-16T15:23:29.970 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.973+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 --> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f13e80d0f30 con 0x7f13c00747e0 2024-01-16T15:23:29.971 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13e5600640 1 -- 172.21.15.125:0/2049712017 <== osd.3 v2:172.21.15.125:6820/397977967 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f13e000d040 con 0x7f13c00747e0 2024-01-16T15:23:29.971 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:29.971 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:29.971 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:29.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f13c00747e0 msgr2=0x7f13c0076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6820/397977967,v1:172.21.15.125:6823/397977967] conn(0x7f13c00747e0 0x7f13c0076c20 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.972 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13c006c580 msgr2=0x7f13c006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f13c006c580 0x7f13c006ea40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f13d0036000 tx=0x7f13d0030000 comp rx=0 tx=0).stop 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 msgr2=0x7f13e80c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13f0000640 1 --2- 172.21.15.125:0/2049712017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13c80047c0 0x7f13e80c6ea0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f13d80116d0 tx=0x7f13d8011b90 comp rx=0 tx=0).stop 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.974+0000 7f13ee000640 1 -- 172.21.15.125:0/2049712017 reap_dead start 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.975+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 shutdown_connections 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.975+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 >> 172.21.15.125:0/2049712017 conn(0x7f13e80b2d00 msgr2=0x7f13e80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.975+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 shutdown_connections 2024-01-16T15:23:29.973 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:29.975+0000 7f13f0000640 1 -- 172.21.15.125:0/2049712017 wait complete. 2024-01-16T15:23:29.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-16T15:23:30.186 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.189+0000 7f3c46400640 1 Processor -- start 2024-01-16T15:23:30.187 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.189+0000 7f3c46400640 1 -- start start 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c46400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3c4015e170 0x7f3c400cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c46400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c400cdef0 0x7f3c40070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c46400640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c400700f0 con 0x7f3c4013cda0 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c46400640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c400ce720 con 0x7f3c400cdef0 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c46400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c400cea00 con 0x7f3c4015e170 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c45a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c400cdef0 0x7f3c40070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.188 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c45000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3c4013cda0 0x7f3c4015da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50242/0 (socket says 172.21.15.125:50242) 2024-01-16T15:23:30.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c45000640 1 -- 172.21.15.125:0/810803462 learned_addr learned my addr 172.21.15.125:0/810803462 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:30.189 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c45a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c400cdef0 0x7f3c40070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41344/0 (socket says 172.21.15.125:41344) 2024-01-16T15:23:30.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.191+0000 7f3c3fe00640 1 --2- 172.21.15.125:0/810803462 >> v2:172.21.15.137:3300/0 conn(0x7f3c4015e170 0x7f3c400cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c3f400640 1 -- 172.21.15.125:0/810803462 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749829508 0 0) 0x7f3c400700f0 con 0x7f3c4013cda0 2024-01-16T15:23:30.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c3f400640 1 -- 172.21.15.125:0/810803462 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c20005730 con 0x7f3c4013cda0 2024-01-16T15:23:30.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c45a00640 1 -- 172.21.15.125:0/810803462 >> v2:172.21.15.137:3300/0 conn(0x7f3c4015e170 msgr2=0x7f3c400cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:30.190 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c45a00640 1 --2- 172.21.15.125:0/810803462 >> v2:172.21.15.137:3300/0 conn(0x7f3c4015e170 0x7f3c400cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c45a00640 1 -- 172.21.15.125:0/810803462 >> v1:172.21.15.125:6789/0 conn(0x7f3c4013cda0 legacy=0x7f3c4015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.192+0000 7f3c45a00640 1 -- 172.21.15.125:0/810803462 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40070340 con 0x7f3c400cdef0 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.193+0000 7f3c45a00640 1 --2- 172.21.15.125:0/810803462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40070b40 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f3c40157fd0 tx=0x7f3c3802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e9d3e0270534ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.193+0000 7f3c3f400640 1 -- 172.21.15.125:0/810803462 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3802cbe0 con 0x7f3c400cdef0 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.193+0000 7f3c3f400640 1 -- 172.21.15.125:0/810803462 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3802cd40 con 0x7f3c400cdef0 2024-01-16T15:23:30.191 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.193+0000 7f3c3f400640 1 -- 172.21.15.125:0/810803462 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38034660 con 0x7f3c400cdef0 2024-01-16T15:23:30.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 -- 172.21.15.125:0/810803462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 msgr2=0x7f3c40070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.192 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 --2- 172.21.15.125:0/810803462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40070b40 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f3c40157fd0 tx=0x7f3c3802be30 comp rx=0 tx=0).stop 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 -- 172.21.15.125:0/810803462 shutdown_connections 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 --2- 172.21.15.125:0/810803462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40070b40 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 --2- 172.21.15.125:0/810803462 >> v2:172.21.15.137:3300/0 conn(0x7f3c4015e170 0x7f3c400cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 -- 172.21.15.125:0/810803462 >> 172.21.15.125:0/810803462 conn(0x7f3c4006a3e0 msgr2=0x7f3c4006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 -- 172.21.15.125:0/810803462 shutdown_connections 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.194+0000 7f3c46400640 1 -- 172.21.15.125:0/810803462 wait complete. 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.196+0000 7f3c46400640 1 Processor -- start 2024-01-16T15:23:30.193 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.196+0000 7f3c46400640 1 -- start start 2024-01-16T15:23:30.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c4013cda0 0x7f3c40152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.195 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c4015e170 0x7f3c40152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c401515f0 con 0x7f3c400cdef0 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c401518d0 con 0x7f3c4013cda0 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c46400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40151bb0 con 0x7f3c4015e170 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41346/0 (socket says 172.21.15.125:41346) 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 -- 172.21.15.125:0/3404416906 learned_addr learned my addr 172.21.15.125:0/3404416906 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c3fe00640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c4013cda0 0x7f3c40152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 -- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c4015e170 msgr2=0x7f3c40152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45a00640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c4015e170 0x7f3c40152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c4015e170 0x7f3c40152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 -- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c4013cda0 msgr2=0x7f3c40152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c4013cda0 0x7f3c40152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.197+0000 7f3c45000640 1 -- 172.21.15.125:0/3404416906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38009a70 con 0x7f3c400cdef0 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c45000640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000c410 tx=0x7f3c3000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30005ce0 con 0x7f3c400cdef0 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30005e40 con 0x7f3c400cdef0 2024-01-16T15:23:30.197 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300072f0 con 0x7f3c400cdef0 2024-01-16T15:23:30.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c40151e90 con 0x7f3c400cdef0 2024-01-16T15:23:30.198 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.198+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c40153a20 con 0x7f3c400cdef0 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.199+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c401523b0 con 0x7f3c400cdef0 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.200+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3c30007510 con 0x7f3c400cdef0 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.201+0000 7f3c3cc00640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 0x7f3c1806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.201+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c300881e0 con 0x7f3c400cdef0 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.201+0000 7f3c3fe00640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 0x7f3c1806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.202+0000 7f3c3fe00640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 0x7f3c1806ea10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3c401f5fd0 tx=0x7f3c28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.202 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.206+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c401523b0 con 0x7f3c400cdef0 2024-01-16T15:23:30.394 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.397+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c40151a40 con 0x7f3c1806c550 2024-01-16T15:23:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.398+0000 7f3c3cc00640 1 -- 172.21.15.125:0/3404416906 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3c40151a40 con 0x7f3c1806c550 2024-01-16T15:23:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 3 to deep-scrub 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 msgr2=0x7f3c1806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 0x7f3c1806ea10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3c401f5fd0 tx=0x7f3c28008040 comp rx=0 tx=0).stop 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 msgr2=0x7f3c40157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000c410 tx=0x7f3c3000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 shutdown_connections 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c4015e170 0x7f3c40152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c4013cda0 0x7f3c40152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3c1806c550 0x7f3c1806ea10 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 --2- 172.21.15.125:0/3404416906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c400cdef0 0x7f3c40157720 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 >> 172.21.15.125:0/3404416906 conn(0x7f3c4006a3e0 msgr2=0x7f3c400712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 shutdown_connections 2024-01-16T15:23:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.401+0000 7f3c46400640 1 -- 172.21.15.125:0/3404416906 wait complete. 2024-01-16T15:23:30.416 INFO:tasks.ceph:Scrubbing osd.4 2024-01-16T15:23:30.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:30.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.634+0000 7f8db7e00640 1 Processor -- start 2024-01-16T15:23:30.631 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.634+0000 7f8db7e00640 1 -- start start 2024-01-16T15:23:30.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.635+0000 7f8db7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8db8071410 0x7f8db8146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.632 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8db8071f00 0x7f8db8166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db80b2e10 con 0x7f8db806c7c0 2024-01-16T15:23:30.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db806cc50 con 0x7f8db8071f00 2024-01-16T15:23:30.633 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db806cec0 con 0x7f8db8071410 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8db8071f00 0x7f8db8166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8dafe00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8db806c7c0 0x7f8db80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50256/0 (socket says 172.21.15.125:50256) 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db7400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8db8071f00 0x7f8db8166cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41358/0 (socket says 172.21.15.125:41358) 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8dafe00640 1 -- 172.21.15.125:0/1314804126 learned_addr learned my addr 172.21.15.125:0/1314804126 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.636+0000 7f8db6a00640 1 --2- 172.21.15.125:0/1314804126 >> v2:172.21.15.137:3300/0 conn(0x7f8db8071410 0x7f8db8146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.634 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.637+0000 7f8db7400640 1 -- 172.21.15.125:0/1314804126 >> v2:172.21.15.137:3300/0 conn(0x7f8db8071410 msgr2=0x7f8db8146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:30.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.637+0000 7f8db7400640 1 --2- 172.21.15.125:0/1314804126 >> v2:172.21.15.137:3300/0 conn(0x7f8db8071410 0x7f8db8146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.637+0000 7f8db7400640 1 -- 172.21.15.125:0/1314804126 >> v1:172.21.15.125:6789/0 conn(0x7f8db806c7c0 legacy=0x7f8db80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.637+0000 7f8db7400640 1 -- 172.21.15.125:0/1314804126 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db806d1a0 con 0x7f8db8071f00 2024-01-16T15:23:30.635 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.637+0000 7f8db6000640 1 -- 172.21.15.125:0/1314804126 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475341065 0 0) 0x7f8db80b2e10 con 0x7f8db806c7c0 2024-01-16T15:23:30.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.638+0000 7f8db7400640 1 --2- 172.21.15.125:0/1314804126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db8071f00 0x7f8db8166cf0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f8db80ad5f0 tx=0x7f8db002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32be843ca8bde656 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.638+0000 7f8db6000640 1 -- 172.21.15.125:0/1314804126 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db002cc80 con 0x7f8db8071f00 2024-01-16T15:23:30.636 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.638+0000 7f8db6000640 1 -- 172.21.15.125:0/1314804126 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db002cde0 con 0x7f8db8071f00 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.638+0000 7f8db6000640 1 -- 172.21.15.125:0/1314804126 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0034660 con 0x7f8db8071f00 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 -- 172.21.15.125:0/1314804126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db8071f00 msgr2=0x7f8db8166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1314804126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db8071f00 0x7f8db8166cf0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f8db80ad5f0 tx=0x7f8db002bed0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 -- 172.21.15.125:0/1314804126 shutdown_connections 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1314804126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db8071f00 0x7f8db8166cf0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1314804126 >> v2:172.21.15.137:3300/0 conn(0x7f8db8071410 0x7f8db8146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 -- 172.21.15.125:0/1314804126 >> 172.21.15.125:0/1314804126 conn(0x7f8db806a800 msgr2=0x7f8db806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 -- 172.21.15.125:0/1314804126 shutdown_connections 2024-01-16T15:23:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.639+0000 7f8db7e00640 1 -- 172.21.15.125:0/1314804126 wait complete. 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.641+0000 7f8db7e00640 1 Processor -- start 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.641+0000 7f8db7e00640 1 -- start start 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 0x7f8db8142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db8071410 0x7f8db81e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db8071f00 0x7f8db813e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db81e6070 con 0x7f8db806c7c0 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db81e6490 con 0x7f8db8071f00 2024-01-16T15:23:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db81e53f0 con 0x7f8db8071410 2024-01-16T15:23:30.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 0x7f8db8142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 0x7f8db8142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41372/0 (socket says 172.21.15.125:41372) 2024-01-16T15:23:30.640 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 -- 172.21.15.125:0/1489356893 learned_addr learned my addr 172.21.15.125:0/1489356893 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:30.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 -- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db8071410 msgr2=0x7f8db81e4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8db8071410 0x7f8db81e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 -- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db8071f00 msgr2=0x7f8db813e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8db8071f00 0x7f8db813e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.641 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.642+0000 7f8db6a00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0009a70 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8db6a00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 0x7f8db8142690 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8da800d800 tx=0x7f8da800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da800b6c0 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da800b820 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da800bbb0 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db81e56d0 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.643+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db81e5c90 con 0x7f8db806c7c0 2024-01-16T15:23:30.642 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.644+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8db806da40 con 0x7f8db806c7c0 2024-01-16T15:23:30.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.645+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f8da800bd10 con 0x7f8db806c7c0 2024-01-16T15:23:30.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8daea00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d9006c550 0x7f8d9006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.643 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8da8089100 con 0x7f8db806c7c0 2024-01-16T15:23:30.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8daea00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7f8d900747b0 0x7f8d90076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d900772c0 con 0x7f8d900747b0 2024-01-16T15:23:30.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8da8089d20 con 0x7f8db806c7c0 2024-01-16T15:23:30.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.646+0000 7f8dafe00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d9006c550 0x7f8d9006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.644 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.647+0000 7f8db7400640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7f8d900747b0 0x7f8d90076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.647+0000 7f8dafe00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d9006c550 0x7f8d9006ea10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8da0007fd0 tx=0x7f8da0007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.645 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.648+0000 7f8db7400640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7f8d900747b0 0x7f8d90076bf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.646 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.650+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== osd.4 v2:172.21.15.137:6808/1439911452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8d900772c0 con 0x7f8d900747b0 2024-01-16T15:23:30.661 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.664+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 --> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8db806d760 con 0x7f8d900747b0 2024-01-16T15:23:30.662 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.665+0000 7f8daea00640 1 -- 172.21.15.125:0/1489356893 <== osd.4 v2:172.21.15.137:6808/1439911452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8db0009450 con 0x7f8d900747b0 2024-01-16T15:23:30.662 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:30.662 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:30.663 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7f8d900747b0 msgr2=0x7f8d90076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.137:6808/1439911452,v1:172.21.15.137:6809/1439911452] conn(0x7f8d900747b0 0x7f8d90076bf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d9006c550 msgr2=0x7f8d9006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f8d9006c550 0x7f8d9006ea10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8da0007fd0 tx=0x7f8da0007920 comp rx=0 tx=0).stop 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 msgr2=0x7f8db8142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.664 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 --2- 172.21.15.125:0/1489356893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8db806c7c0 0x7f8db8142690 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8da800d800 tx=0x7f8da800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7400640 1 -- 172.21.15.125:0/1489356893 reap_dead start 2024-01-16T15:23:30.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 shutdown_connections 2024-01-16T15:23:30.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 >> 172.21.15.125:0/1489356893 conn(0x7f8db806a800 msgr2=0x7f8db80b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:30.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 shutdown_connections 2024-01-16T15:23:30.665 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.666+0000 7f8db7e00640 1 -- 172.21.15.125:0/1489356893 wait complete. 2024-01-16T15:23:30.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-16T15:23:30.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.895+0000 7ff118800640 1 Processor -- start 2024-01-16T15:23:30.893 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.896+0000 7ff118800640 1 -- start start 2024-01-16T15:23:30.894 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff118800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1100cf9a0 0x7ff11015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff118800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff11015e1e0 0x7ff1100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff118800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100b1ee0 con 0x7ff110143ae0 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff118800640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1100b2780 con 0x7ff11015e1e0 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff118800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11015e990 con 0x7ff1100cf9a0 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.897+0000 7ff116800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff11015e1e0 0x7ff1100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff11015e1e0 0x7ff1100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41378/0 (socket says 172.21.15.125:41378) 2024-01-16T15:23:30.895 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 -- 172.21.15.125:0/1067606458 learned_addr learned my addr 172.21.15.125:0/1067606458 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:30.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff115400640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1100cf9a0 0x7ff11015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 -- 172.21.15.125:0/1067606458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1100cf9a0 msgr2=0x7ff11015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:30.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1100cf9a0 0x7ff11015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 -- 172.21.15.125:0/1067606458 >> v1:172.21.15.137:6789/0 conn(0x7ff110143ae0 legacy=0x7ff1100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:30.896 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.898+0000 7ff116800640 1 -- 172.21.15.125:0/1067606458 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100b2130 con 0x7ff11015e1e0 2024-01-16T15:23:30.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.899+0000 7ff116800640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff11015e1e0 0x7ff1100702c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff108009870 tx=0x7ff10802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3637c1803459042 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.899+0000 7ff114a00640 1 -- 172.21.15.125:0/1067606458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108031640 con 0x7ff11015e1e0 2024-01-16T15:23:30.897 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.899+0000 7ff114a00640 1 -- 172.21.15.125:0/1067606458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1080317a0 con 0x7ff11015e1e0 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.899+0000 7ff114a00640 1 -- 172.21.15.125:0/1067606458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108031b30 con 0x7ff11015e1e0 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 -- 172.21.15.125:0/1067606458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff11015e1e0 msgr2=0x7ff1100702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff11015e1e0 0x7ff1100702c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff108009870 tx=0x7ff10802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 -- 172.21.15.125:0/1067606458 shutdown_connections 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff11015e1e0 0x7ff1100702c0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 --2- 172.21.15.125:0/1067606458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1100cf9a0 0x7ff11015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 -- 172.21.15.125:0/1067606458 >> 172.21.15.125:0/1067606458 conn(0x7ff11006a800 msgr2=0x7ff11006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 -- 172.21.15.125:0/1067606458 shutdown_connections 2024-01-16T15:23:30.898 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.900+0000 7ff118800640 1 -- 172.21.15.125:0/1067606458 wait complete. 2024-01-16T15:23:30.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.902+0000 7ff118800640 1 Processor -- start 2024-01-16T15:23:30.899 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.903+0000 7ff118800640 1 -- start start 2024-01-16T15:23:30.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.902 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 0x7ff1100cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff11015e1e0 0x7ff1100afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11013ea80 con 0x7ff1100cf9a0 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11013f590 con 0x7ff110143ae0 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff118800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11013f870 con 0x7ff11015e1e0 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff115e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff115e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41394/0 (socket says 172.21.15.125:41394) 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff115e00640 1 -- 172.21.15.125:0/350850654 learned_addr learned my addr 172.21.15.125:0/350850654 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:30.903 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.904+0000 7ff116800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff11015e1e0 0x7ff1100afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115400640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 0x7ff1100cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 -- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff11015e1e0 msgr2=0x7ff1100afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff11015e1e0 0x7ff1100afba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 -- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 msgr2=0x7ff1100cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 0x7ff1100cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 -- 172.21.15.125:0/350850654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080094d0 con 0x7ff1100cf9a0 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115400640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 0x7ff1100cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:30.904 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff115e00640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c7c0 tx=0x7ff10000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10000aa50 con 0x7ff1100cf9a0 2024-01-16T15:23:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10000abb0 con 0x7ff1100cf9a0 2024-01-16T15:23:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100012420 con 0x7ff1100cf9a0 2024-01-16T15:23:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11013fb50 con 0x7ff1100cf9a0 2024-01-16T15:23:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.905+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff110140110 con 0x7ff1100cf9a0 2024-01-16T15:23:30.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.907+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1100675d0 con 0x7ff1100cf9a0 2024-01-16T15:23:30.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.907+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff100012580 con 0x7ff1100cf9a0 2024-01-16T15:23:30.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.908+0000 7ff10e000640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 0x7ff0e806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:30.906 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.909+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff100088b00 con 0x7ff1100cf9a0 2024-01-16T15:23:30.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.909+0000 7ff115400640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 0x7ff0e806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:30.907 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.909+0000 7ff115400640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 0x7ff0e806ea10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff11006caf0 tx=0x7ff0f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:30.909 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:30.913+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff100055040 con 0x7ff1100cf9a0 2024-01-16T15:23:31.100 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.104+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff11015e640 con 0x7ff0e806c550 2024-01-16T15:23:31.101 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.105+0000 7ff10e000640 1 -- 172.21.15.125:0/350850654 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff0f8009e10 con 0x7ff0e806c550 2024-01-16T15:23:31.101 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 4 to deep-scrub 2024-01-16T15:23:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 msgr2=0x7ff0e806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 0x7ff0e806ea10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff11006caf0 tx=0x7ff0f8008040 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 msgr2=0x7ff11013e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c7c0 tx=0x7ff10000cc80 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 shutdown_connections 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff11015e1e0 0x7ff1100afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff110143ae0 0x7ff1100cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff0e806c550 0x7ff0e806ea10 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 --2- 172.21.15.125:0/350850654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100cf9a0 0x7ff11013e1b0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 >> 172.21.15.125:0/350850654 conn(0x7ff11006a800 msgr2=0x7ff11015ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 shutdown_connections 2024-01-16T15:23:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.107+0000 7ff118800640 1 -- 172.21.15.125:0/350850654 wait complete. 2024-01-16T15:23:31.123 INFO:tasks.ceph:Scrubbing osd.5 2024-01-16T15:23:31.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.324+0000 7f75a7200640 1 Processor -- start 2024-01-16T15:23:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.324+0000 7f75a7200640 1 -- start start 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 0x7f75a015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75a013e360 0x7f75a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 0x7f75a00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a006c470 con 0x7f75a013e360 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a006c750 con 0x7f75a015f9c0 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a006ca30 con 0x7f75a013df80 2024-01-16T15:23:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f759f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75a013e360 0x7f75a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f759f400640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75a013e360 0x7f75a00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41402/0 (socket says 172.21.15.125:41402) 2024-01-16T15:23:31.324 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f759f400640 1 -- 172.21.15.125:0/3914597368 learned_addr learned my addr 172.21.15.125:0/3914597368 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:31.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f759fe00640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 0x7f75a015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.326+0000 7f75a5200640 1 --2- 172.21.15.125:0/3914597368 >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 0x7f75a00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759f400640 1 -- 172.21.15.125:0/3914597368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 msgr2=0x7f75a015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759f400640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 0x7f75a015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.325 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759f400640 1 -- 172.21.15.125:0/3914597368 >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 msgr2=0x7f75a00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759f400640 1 --2- 172.21.15.125:0/3914597368 >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 0x7f75a00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759f400640 1 -- 172.21.15.125:0/3914597368 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a006cd10 con 0x7f75a013e360 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f759fe00640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 0x7f75a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.327+0000 7f75a5200640 1 --2- 172.21.15.125:0/3914597368 >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 0x7f75a00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.328+0000 7f759f400640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013e360 0x7f75a00cf290 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f75a006e6d0 tx=0x7f758802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bdbf5f77c87d37 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.328+0000 7f759ea00640 1 -- 172.21.15.125:0/3914597368 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588031860 con 0x7f75a013e360 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.328+0000 7f759ea00640 1 -- 172.21.15.125:0/3914597368 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880319c0 con 0x7f75a013e360 2024-01-16T15:23:31.326 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.328+0000 7f759ea00640 1 -- 172.21.15.125:0/3914597368 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588031d50 con 0x7f75a013e360 2024-01-16T15:23:31.327 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 -- 172.21.15.125:0/3914597368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013e360 msgr2=0x7f75a00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013e360 0x7f75a00cf290 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f75a006e6d0 tx=0x7f758802b900 comp rx=0 tx=0).stop 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 -- 172.21.15.125:0/3914597368 shutdown_connections 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 --2- 172.21.15.125:0/3914597368 >> v2:172.21.15.137:3300/0 conn(0x7f75a015f9c0 0x7f75a00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013e360 0x7f75a00cf290 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 --2- 172.21.15.125:0/3914597368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a013df80 0x7f75a015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 -- 172.21.15.125:0/3914597368 >> 172.21.15.125:0/3914597368 conn(0x7f75a006a800 msgr2=0x7f75a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.328 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 -- 172.21.15.125:0/3914597368 shutdown_connections 2024-01-16T15:23:31.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.329+0000 7f75a7200640 1 -- 172.21.15.125:0/3914597368 wait complete. 2024-01-16T15:23:31.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.331+0000 7f75a7200640 1 Processor -- start 2024-01-16T15:23:31.329 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.331+0000 7f75a7200640 1 -- start start 2024-01-16T15:23:31.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 0x7f75a014fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.331 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75a013e360 0x7f75a0137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a015f9c0 0x7f75a013e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a01517e0 con 0x7f75a013df80 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0153410 con 0x7f75a013e360 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0150850 con 0x7f75a015f9c0 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 0x7f75a014fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 0x7f75a014fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41416/0 (socket says 172.21.15.125:41416) 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 -- 172.21.15.125:0/2624722803 learned_addr learned my addr 172.21.15.125:0/2624722803 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:31.332 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f75a5200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a015f9c0 0x7f75a013e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 -- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a015f9c0 msgr2=0x7f75a013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759f400640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75a013e360 0x7f75a0137ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75a015f9c0 0x7f75a013e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.332+0000 7f759fe00640 1 -- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75a013e360 msgr2=0x7f75a0137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f759fe00640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75a013e360 0x7f75a0137ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f759fe00640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880094d0 con 0x7f75a013df80 2024-01-16T15:23:31.333 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f759f400640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75a013e360 0x7f75a0137ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f759fe00640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 0x7f75a014fc90 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f759000c7f0 tx=0x7f759000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590007820 con 0x7f75a013df80 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590007980 con 0x7f75a013df80 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590007cf0 con 0x7f75a013df80 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a0150570 con 0x7f75a013df80 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.333+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a0153030 con 0x7f75a013df80 2024-01-16T15:23:31.334 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.334+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f75a01408d0 con 0x7f75a013df80 2024-01-16T15:23:31.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.336+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f75900113f0 con 0x7f75a013df80 2024-01-16T15:23:31.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.336+0000 7f7597e00640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f757806c550 0x7f757806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f75900888c0 con 0x7f75a013df80 2024-01-16T15:23:31.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f759f400640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f757806c550 0x7f757806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.336 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f7597e00640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7f75780747b0 0x7f7578076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75780772c0 con 0x7f75780747b0 2024-01-16T15:23:31.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f759008eab0 con 0x7f75a013df80 2024-01-16T15:23:31.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f759f400640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f757806c550 0x7f757806ea10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7588036000 tx=0x7f758803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.337+0000 7f75a5200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7f75780747b0 0x7f7578076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.337 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.339+0000 7f75a5200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7f75780747b0 0x7f7578076bf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.338 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.341+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== osd.5 v2:172.21.15.137:6800/2884121150 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f75780772c0 con 0x7f75780747b0 2024-01-16T15:23:31.352 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.355+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 --> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f75a0153680 con 0x7f75780747b0 2024-01-16T15:23:31.353 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.356+0000 7f7597e00640 1 -- 172.21.15.125:0/2624722803 <== osd.5 v2:172.21.15.137:6800/2884121150 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f75a0153680 con 0x7f75780747b0 2024-01-16T15:23:31.354 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:31.354 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:31.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7f75780747b0 msgr2=0x7f7578076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.137:6800/2884121150,v1:172.21.15.137:6801/2884121150] conn(0x7f75780747b0 0x7f7578076bf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.355 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f757806c550 msgr2=0x7f757806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f757806c550 0x7f757806ea10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7588036000 tx=0x7f758803c040 comp rx=0 tx=0).stop 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 msgr2=0x7f75a014fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a7200640 1 --2- 172.21.15.125:0/2624722803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a013df80 0x7f75a014fc90 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f759000c7f0 tx=0x7f759000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.357+0000 7f75a5200640 1 -- 172.21.15.125:0/2624722803 reap_dead start 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.358+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 shutdown_connections 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.358+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 >> 172.21.15.125:0/2624722803 conn(0x7f75a006a800 msgr2=0x7f75a00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.356 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.358+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 shutdown_connections 2024-01-16T15:23:31.357 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.358+0000 7f75a7200640 1 -- 172.21.15.125:0/2624722803 wait complete. 2024-01-16T15:23:31.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-16T15:23:31.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.584+0000 7f3f5a800640 1 Processor -- start 2024-01-16T15:23:31.581 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.584+0000 7f3f5a800640 1 -- start start 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f5a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f5415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f5a800640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5406c470 con 0x7f3f540cf9a0 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f5a800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5406c750 con 0x7f3f5413e360 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f5a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5406ca30 con 0x7f3f5413df80 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f53e00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3f540cf9a0 0x7f3f540cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50258/0 (socket says 172.21.15.125:50258) 2024-01-16T15:23:31.583 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f53e00640 1 -- 172.21.15.125:0/3022910638 learned_addr learned my addr 172.21.15.125:0/3022910638 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:31.586 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f53400640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f5415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880471905 0 0) 0x7f3f5406c470 con 0x7f3f540cf9a0 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.586+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f34003520 con 0x7f3f540cf9a0 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157483877 0 0) 0x7f3f34003520 con 0x7f3f540cf9a0 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 msgr2=0x7f3f5415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f5415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 >> v1:172.21.15.137:6789/0 conn(0x7f3f5413e360 legacy=0x7f3f540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5406cd10 con 0x7f3f540cf9a0 2024-01-16T15:23:31.587 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2458260829 0 0) 0x7f3f4c0019f0 con 0x7f3f540cf9a0 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 >> v1:172.21.15.125:6789/0 conn(0x7f3f540cf9a0 legacy=0x7f3f540cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340039a0 0x7f3f34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f340047c0 0x7f3f34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5406ca30 con 0x7f3f340047c0 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3400ba60 con 0x7f3f340039a0 2024-01-16T15:23:31.588 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3400bc20 con 0x7f3f5413df80 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f52a00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f340047c0 0x7f3f34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.587+0000 7f3f53400640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340039a0 0x7f3f34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f53e00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f52a00640 1 -- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 msgr2=0x7f3f34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f52a00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f52a00640 1 -- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340039a0 msgr2=0x7f3f34003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f52a00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340039a0 0x7f3f34003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.589 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f52a00640 1 -- 172.21.15.125:0/3022910638 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34003520 con 0x7f3f340047c0 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.588+0000 7f3f53e00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f5413df80 0x7f3f34005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f52a00640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f340047c0 0x7f3f34004c40 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c001720 tx=0x7f3f340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182a6ad230ecf2ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0052d0 con 0x7f3f340047c0 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c02e050 con 0x7f3f340047c0 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f52000640 1 -- 172.21.15.125:0/3022910638 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0035f0 con 0x7f3f340047c0 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f5a800640 1 -- 172.21.15.125:0/3022910638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f340047c0 msgr2=0x7f3f34004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.589+0000 7f3f5a800640 1 --2- 172.21.15.125:0/3022910638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f340047c0 0x7f3f34004c40 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c001720 tx=0x7f3f340057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.590 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.590+0000 7f3f53e00640 1 -- 172.21.15.125:0/3022910638 reap_dead start 2024-01-16T15:23:31.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.590+0000 7f3f5a800640 1 -- 172.21.15.125:0/3022910638 shutdown_connections 2024-01-16T15:23:31.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.590+0000 7f3f5a800640 1 -- 172.21.15.125:0/3022910638 >> 172.21.15.125:0/3022910638 conn(0x7f3f5406a800 msgr2=0x7f3f5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.590+0000 7f3f5a800640 1 -- 172.21.15.125:0/3022910638 shutdown_connections 2024-01-16T15:23:31.591 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.590+0000 7f3f5a800640 1 -- 172.21.15.125:0/3022910638 wait complete. 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.591+0000 7f3f5a800640 1 Processor -- start 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.592+0000 7f3f5a800640 1 -- start start 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.592+0000 7f3f5a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340047c0 0x7f3f541514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.592+0000 7f3f5a800640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f5a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f54152e30 0x7f3f541532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f5a800640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540705d0 con 0x7f3f5413d730 2024-01-16T15:23:31.594 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f5a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540702c0 con 0x7f3f340047c0 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f5a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540ad820 con 0x7f3f54152e30 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41440/0 (socket says 172.21.15.125:41440) 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 -- 172.21.15.125:0/761314104 learned_addr learned my addr 172.21.15.125:0/761314104 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 -- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f54152e30 msgr2=0x7f3f541532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f54152e30 0x7f3f541532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 -- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340047c0 msgr2=0x7f3f541514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.595 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340047c0 0x7f3f541514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.593+0000 7f3f52a00640 1 -- 172.21.15.125:0/761314104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540adac0 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f52a00640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c008fd0 tx=0x7f3f3c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0076c0 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c007a30 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f540adda0 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c006c00 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.594+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f541537f0 con 0x7f3f5413d730 2024-01-16T15:23:31.596 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.595+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54070840 con 0x7f3f5413d730 2024-01-16T15:23:31.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.596+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3f3c0035f0 con 0x7f3f5413d730 2024-01-16T15:23:31.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.597+0000 7f3f4b400640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 0x7f3f2c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.597+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3f3c0aa610 con 0x7f3f5413d730 2024-01-16T15:23:31.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.597+0000 7f3f53400640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 0x7f3f2c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.597 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.598+0000 7f3f53400640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 0x7f3f2c06ea10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009910 tx=0x7f3f44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.598 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.602+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f3c076ed0 con 0x7f3f5413d730 2024-01-16T15:23:31.795 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.799+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f5415bee0 con 0x7f3f2c06c550 2024-01-16T15:23:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.800+0000 7f3f4b400640 1 -- 172.21.15.125:0/761314104 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3f44011860 con 0x7f3f2c06c550 2024-01-16T15:23:31.796 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 5 to deep-scrub 2024-01-16T15:23:31.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 msgr2=0x7f3f2c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.799 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 0x7f3f2c06ea10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009910 tx=0x7f3f44008040 comp rx=0 tx=0).stop 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 msgr2=0x7f3f540ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c008fd0 tx=0x7f3f3c001120 comp rx=0 tx=0).stop 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 shutdown_connections 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f54152e30 0x7f3f541532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f3f2c06c550 0x7f3f2c06ea10 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5413d730 0x7f3f540ad2e0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.800 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 --2- 172.21.15.125:0/761314104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f340047c0 0x7f3f541514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.802+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 >> 172.21.15.125:0/761314104 conn(0x7f3f5406a800 msgr2=0x7f3f540cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.803+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 shutdown_connections 2024-01-16T15:23:31.801 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:31.803+0000 7f3f5a800640 1 -- 172.21.15.125:0/761314104 wait complete. 2024-01-16T15:23:31.816 INFO:tasks.ceph:Scrubbing osd.6 2024-01-16T15:23:31.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:32.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.024+0000 7f980ca00640 1 Processor -- start 2024-01-16T15:23:32.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.025+0000 7f980ca00640 1 -- start start 2024-01-16T15:23:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 0x7f98040d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980ca00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9804160e10 0x7f98040d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980ca00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98040d4160 con 0x7f98040cfcf0 2024-01-16T15:23:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980ca00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040ceff0 con 0x7f9804160e10 2024-01-16T15:23:32.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980ca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040cf410 con 0x7f98040d3910 2024-01-16T15:23:32.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980a000640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f98040cfcf0 0x7f9804160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50268/0 (socket says 172.21.15.125:50268) 2024-01-16T15:23:32.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980a000640 1 -- 172.21.15.125:0/3100950758 learned_addr learned my addr 172.21.15.125:0/3100950758 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:23:32.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.026+0000 7f980aa00640 1 --2- 172.21.15.125:0/3100950758 >> v2:172.21.15.125:3300/0 conn(0x7f9804160e10 0x7f98040d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f9809600640 1 --2- 172.21.15.125:0/3100950758 >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 0x7f98040d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f980aa00640 1 -- 172.21.15.125:0/3100950758 >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 msgr2=0x7f98040d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f980aa00640 1 --2- 172.21.15.125:0/3100950758 >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 0x7f98040d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f980aa00640 1 -- 172.21.15.125:0/3100950758 >> v1:172.21.15.125:6789/0 conn(0x7f98040cfcf0 legacy=0x7f9804160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f980aa00640 1 -- 172.21.15.125:0/3100950758 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040b2420 con 0x7f9804160e10 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f9808c00640 1 -- 172.21.15.125:0/3100950758 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616793416 0 0) 0x7f98040d4160 con 0x7f98040cfcf0 2024-01-16T15:23:32.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.027+0000 7f9809600640 1 --2- 172.21.15.125:0/3100950758 >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 0x7f98040d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.028+0000 7f980aa00640 1 --2- 172.21.15.125:0/3100950758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040d1640 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f980408b640 tx=0x7f97fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a802427cb8afa703 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.028+0000 7f9808c00640 1 -- 172.21.15.125:0/3100950758 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc02e070 con 0x7f9804160e10 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.028+0000 7f9808c00640 1 -- 172.21.15.125:0/3100950758 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc0315a0 con 0x7f9804160e10 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.028+0000 7f9808c00640 1 -- 172.21.15.125:0/3100950758 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc031930 con 0x7f9804160e10 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 -- 172.21.15.125:0/3100950758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 msgr2=0x7f98040d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 --2- 172.21.15.125:0/3100950758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040d1640 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f980408b640 tx=0x7f97fc02b840 comp rx=0 tx=0).stop 2024-01-16T15:23:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 -- 172.21.15.125:0/3100950758 shutdown_connections 2024-01-16T15:23:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 --2- 172.21.15.125:0/3100950758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040d1640 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 --2- 172.21.15.125:0/3100950758 >> v2:172.21.15.137:3300/0 conn(0x7f98040d3910 0x7f98040d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 -- 172.21.15.125:0/3100950758 >> 172.21.15.125:0/3100950758 conn(0x7f980409c530 msgr2=0x7f98040d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 -- 172.21.15.125:0/3100950758 shutdown_connections 2024-01-16T15:23:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.029+0000 7f980ca00640 1 -- 172.21.15.125:0/3100950758 wait complete. 2024-01-16T15:23:32.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.032+0000 7f980ca00640 1 Processor -- start 2024-01-16T15:23:32.029 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.032+0000 7f980ca00640 1 -- start start 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98040cfcf0 0x7f9804154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98040d3910 0x7f98040c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9804154c80 con 0x7f9804160e10 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040c75a0 con 0x7f98040d3910 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98040c7880 con 0x7f98040cfcf0 2024-01-16T15:23:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980aa00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41458/0 (socket says 172.21.15.125:41458) 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.033+0000 7f980aa00640 1 -- 172.21.15.125:0/3360124578 learned_addr learned my addr 172.21.15.125:0/3360124578 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980a000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98040cfcf0 0x7f9804154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f9809600640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98040d3910 0x7f98040c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 -- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98040cfcf0 msgr2=0x7f9804154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98040cfcf0 0x7f9804154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 -- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98040d3910 msgr2=0x7f98040c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98040d3910 0x7f98040c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc009400 con 0x7f9804160e10 2024-01-16T15:23:32.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980a000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98040cfcf0 0x7f9804154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:23:32.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f980aa00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040b6b40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f97fc034040 tx=0x7f97fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.034+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc032840 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.035+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc0329c0 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.035+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc032d70 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.035+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040c7b60 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.035+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98040c83d0 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.036+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98040cec10 con 0x7f9804160e10 2024-01-16T15:23:32.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.037+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f97fc03b3f0 con 0x7f9804160e10 2024-01-16T15:23:32.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f9802000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97dc06c550 0x7f97dc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f97fc0aac30 con 0x7f9804160e10 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f980a000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97dc06c550 0x7f97dc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f9802000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7f97dc0747b0 0x7f97dc076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97dc0772c0 con 0x7f97dc0747b0 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.038+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97fc0ab550 con 0x7f9804160e10 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.039+0000 7f9809600640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7f97dc0747b0 0x7f97dc076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.039+0000 7f980a000640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97dc06c550 0x7f97dc06ea10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f97f40012e0 tx=0x7f97f4005b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.037 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.040+0000 7f9809600640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7f97dc0747b0 0x7f97dc076bf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.038 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.042+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== osd.6 v2:172.21.15.137:6816/1720012579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f97dc0772c0 con 0x7f97dc0747b0 2024-01-16T15:23:32.053 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.056+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 --> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f98040d3e80 con 0x7f97dc0747b0 2024-01-16T15:23:32.054 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.057+0000 7f9802000640 1 -- 172.21.15.125:0/3360124578 <== osd.6 v2:172.21.15.137:6816/1720012579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f97ec00a660 con 0x7f97dc0747b0 2024-01-16T15:23:32.054 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:32.054 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:32.055 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:32.055 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7f97dc0747b0 msgr2=0x7f97dc076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.137:6816/1720012579,v1:172.21.15.137:6818/1720012579] conn(0x7f97dc0747b0 0x7f97dc076bf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97dc06c550 msgr2=0x7f97dc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f97dc06c550 0x7f97dc06ea10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f97f40012e0 tx=0x7f97f4005b90 comp rx=0 tx=0).stop 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 msgr2=0x7f98040b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 --2- 172.21.15.125:0/3360124578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9804160e10 0x7f98040b6b40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f97fc034040 tx=0x7f97fc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980aa00640 1 -- 172.21.15.125:0/3360124578 reap_dead start 2024-01-16T15:23:32.056 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 shutdown_connections 2024-01-16T15:23:32.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 >> 172.21.15.125:0/3360124578 conn(0x7f980409c530 msgr2=0x7f98040d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 shutdown_connections 2024-01-16T15:23:32.057 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.058+0000 7f980ca00640 1 -- 172.21.15.125:0/3360124578 wait complete. 2024-01-16T15:23:32.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-16T15:23:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.284+0000 7ff53a200640 1 Processor -- start 2024-01-16T15:23:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.284+0000 7ff53a200640 1 -- start start 2024-01-16T15:23:32.283 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.286+0000 7ff53a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 0x7ff5340cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff53a200640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff53406a3e0 0x7ff53406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff53a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5340ce980 con 0x7ff5340b28a0 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff53a200640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff534144eb0 con 0x7ff53406a3e0 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff53a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53406ad30 con 0x7ff534144090 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff53406a3e0 0x7ff53406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff53406a3e0 0x7ff53406a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41462/0 (socket says 172.21.15.125:41462) 2024-01-16T15:23:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 -- 172.21.15.125:0/4259985975 learned_addr learned my addr 172.21.15.125:0/4259985975 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533400640 1 --2- 172.21.15.125:0/4259985975 >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 0x7ff5340cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 -- 172.21.15.125:0/4259985975 >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 msgr2=0x7ff5340cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 --2- 172.21.15.125:0/4259985975 >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 0x7ff5340cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 -- 172.21.15.125:0/4259985975 >> v1:172.21.15.137:6789/0 conn(0x7ff5340b28a0 legacy=0x7ff534144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:32.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533e00640 1 -- 172.21.15.125:0/4259985975 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5340cebb0 con 0x7ff53406a3e0 2024-01-16T15:23:32.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.287+0000 7ff533400640 1 --2- 172.21.15.125:0/4259985975 >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 0x7ff5340cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:32.287 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff533e00640 1 --2- 172.21.15.125:0/4259985975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff53406a7f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7ff53406d060 tx=0x7ff52c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcd456547da772 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff532000640 1 -- 172.21.15.125:0/4259985975 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c031870 con 0x7ff53406a3e0 2024-01-16T15:23:32.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff532000640 1 -- 172.21.15.125:0/4259985975 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c0319d0 con 0x7ff53406a3e0 2024-01-16T15:23:32.288 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff532000640 1 -- 172.21.15.125:0/4259985975 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c031d60 con 0x7ff53406a3e0 2024-01-16T15:23:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff53a200640 1 -- 172.21.15.125:0/4259985975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 msgr2=0x7ff53406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.289+0000 7ff53a200640 1 --2- 172.21.15.125:0/4259985975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff53406a7f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7ff53406d060 tx=0x7ff52c02b910 comp rx=0 tx=0).stop 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 -- 172.21.15.125:0/4259985975 shutdown_connections 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 --2- 172.21.15.125:0/4259985975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff53406a7f0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 --2- 172.21.15.125:0/4259985975 >> v2:172.21.15.137:3300/0 conn(0x7ff534144090 0x7ff5340cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 -- 172.21.15.125:0/4259985975 >> 172.21.15.125:0/4259985975 conn(0x7ff5340b1c60 msgr2=0x7ff5340b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 -- 172.21.15.125:0/4259985975 shutdown_connections 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.290+0000 7ff53a200640 1 -- 172.21.15.125:0/4259985975 wait complete. 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.291+0000 7ff53a200640 1 Processor -- start 2024-01-16T15:23:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 -- start start 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff5340b28a0 0x7ff53413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff534144090 0x7ff5341426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5341403a0 con 0x7ff53406a3e0 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53413e410 con 0x7ff534144090 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.292+0000 7ff53a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53413e6f0 con 0x7ff5340b28a0 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41476/0 (socket says 172.21.15.125:41476) 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 -- 172.21.15.125:0/1926208024 learned_addr learned my addr 172.21.15.125:0/1926208024 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff532a00640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff5340b28a0 0x7ff53413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 -- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff5340b28a0 msgr2=0x7ff53413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff5340b28a0 0x7ff53413fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 -- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff534144090 msgr2=0x7ff5341426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533e00640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff534144090 0x7ff5341426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff534144090 0x7ff5341426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 -- 172.21.15.125:0/1926208024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c0094d0 con 0x7ff53406a3e0 2024-01-16T15:23:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.293+0000 7ff533400640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cf60 tx=0x7ff52400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.294+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52400d840 con 0x7ff53406a3e0 2024-01-16T15:23:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.294+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52400d9a0 con 0x7ff53406a3e0 2024-01-16T15:23:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.294+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52400dd30 con 0x7ff53406a3e0 2024-01-16T15:23:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.294+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53413e9d0 con 0x7ff53406a3e0 2024-01-16T15:23:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.294+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53413ef90 con 0x7ff53406a3e0 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.295+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53413f1b0 con 0x7ff53406a3e0 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.296+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff524015440 con 0x7ff53406a3e0 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.297+0000 7ff52b400640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 0x7ff50c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.297+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5240894b0 con 0x7ff53406a3e0 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.297+0000 7ff532a00640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 0x7ff50c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.298+0000 7ff532a00640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 0x7ff50c06ea10 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff53415b520 tx=0x7ff51c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.298 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.302+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff524055b00 con 0x7ff53406a3e0 2024-01-16T15:23:32.490 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.493+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5340adc80 con 0x7ff50c06c550 2024-01-16T15:23:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.494+0000 7ff52b400640 1 -- 172.21.15.125:0/1926208024 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff5340adc80 con 0x7ff50c06c550 2024-01-16T15:23:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 6 to deep-scrub 2024-01-16T15:23:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 msgr2=0x7ff50c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 0x7ff50c06ea10 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff53415b520 tx=0x7ff51c009040 comp rx=0 tx=0).stop 2024-01-16T15:23:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 msgr2=0x7ff5340b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cf60 tx=0x7ff52400cf90 comp rx=0 tx=0).stop 2024-01-16T15:23:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 shutdown_connections 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff534144090 0x7ff5341426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff5340b28a0 0x7ff53413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7ff50c06c550 0x7ff50c06ea10 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 --2- 172.21.15.125:0/1926208024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff53406a3e0 0x7ff5340b4c30 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 >> 172.21.15.125:0/1926208024 conn(0x7ff5340b1c60 msgr2=0x7ff53406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 shutdown_connections 2024-01-16T15:23:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.497+0000 7ff53a200640 1 -- 172.21.15.125:0/1926208024 wait complete. 2024-01-16T15:23:32.513 INFO:tasks.ceph:Scrubbing osd.7 2024-01-16T15:23:32.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:32.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.732+0000 7fd4ade00640 1 Processor -- start 2024-01-16T15:23:32.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.732+0000 7fd4ade00640 1 -- start start 2024-01-16T15:23:32.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.733+0000 7fd4ade00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4a80cf9a0 0x7fd4a815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.733+0000 7fd4ade00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80b1ee0 con 0x7fd4a8143ae0 2024-01-16T15:23:32.734 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd4ade00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80b2780 con 0x7fd4a815e1e0 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd4ade00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a815e990 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4a80cf9a0 0x7fd4a815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4a80cf9a0 0x7fd4a815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41480/0 (socket says 172.21.15.125:41480) 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 -- 172.21.15.125:0/1407199218 learned_addr learned my addr 172.21.15.125:0/1407199218 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd4a6a00640 1 --1- 172.21.15.125:0/1407199218 >> v1:172.21.15.125:6789/0 conn(0x7fd4a8143ae0 0x7fd4a80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50282/0 (socket says 172.21.15.125:50282) 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 -- 172.21.15.125:0/1407199218 >> v1:172.21.15.137:6789/0 conn(0x7fd4a815e1e0 legacy=0x7fd4a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 -- 172.21.15.125:0/1407199218 >> v1:172.21.15.125:6789/0 conn(0x7fd4a8143ae0 legacy=0x7fd4a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.735 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd49fe00640 1 -- 172.21.15.125:0/1407199218 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80b2130 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.734+0000 7fd4a6000640 1 -- 172.21.15.125:0/1407199218 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701373772 0 0) 0x7fd4a80b1ee0 con 0x7fd4a8143ae0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.735+0000 7fd49fe00640 1 --2- 172.21.15.125:0/1407199218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a815dca0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fd490009870 tx=0x7fd49002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433893d4620182d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4a6000640 1 -- 172.21.15.125:0/1407199218 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490030860 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4a6000640 1 -- 172.21.15.125:0/1407199218 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4900309c0 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4a6000640 1 -- 172.21.15.125:0/1407199218 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490030d50 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 -- 172.21.15.125:0/1407199218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 msgr2=0x7fd4a815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 --2- 172.21.15.125:0/1407199218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a815dca0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fd490009870 tx=0x7fd49002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.736 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 -- 172.21.15.125:0/1407199218 shutdown_connections 2024-01-16T15:23:32.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 --2- 172.21.15.125:0/1407199218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a815dca0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 -- 172.21.15.125:0/1407199218 >> 172.21.15.125:0/1407199218 conn(0x7fd4a806a800 msgr2=0x7fd4a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 -- 172.21.15.125:0/1407199218 shutdown_connections 2024-01-16T15:23:32.737 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.736+0000 7fd4ade00640 1 -- 172.21.15.125:0/1407199218 wait complete. 2024-01-16T15:23:32.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.738+0000 7fd4ade00640 1 Processor -- start 2024-01-16T15:23:32.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.738+0000 7fd4ade00640 1 -- start start 2024-01-16T15:23:32.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a8159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.739 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd4a8143ae0 0x7fd4a80cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4a815e1e0 0x7fd4a80aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80ad0b0 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80cdbb0 con 0x7fd4a815e1e0 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4ade00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80cde90 con 0x7fd4a8143ae0 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a8159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.739+0000 7fd4a6a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a8159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41492/0 (socket says 172.21.15.125:41492) 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 -- 172.21.15.125:0/866903821 learned_addr learned my addr 172.21.15.125:0/866903821 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:32.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd49fe00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd4a8143ae0 0x7fd4a80cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a7400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4a815e1e0 0x7fd4a80aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 -- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd4a8143ae0 msgr2=0x7fd4a80cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd4a8143ae0 0x7fd4a80cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 -- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4a815e1e0 msgr2=0x7fd4a80aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4a815e1e0 0x7fd4a80aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900094d0 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.741 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a7400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4a815e1e0 0x7fd4a80aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd4a6a00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a8159890 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c4d0 tx=0x7fd49800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.740+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980074e0 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.741+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498007660 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.741+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498004a20 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.741+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a80ce170 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.741+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a813ec40 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.742 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.742+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4a815e670 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.743+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd498004b80 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49f400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd48006c550 0x7fd48006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd498087c90 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49fe00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd48006c550 0x7fd48006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49f400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fd4800747b0 0x7fd480076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4800772c0 con 0x7fd4800747b0 2024-01-16T15:23:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.744+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd498088840 con 0x7fd4a80cf9a0 2024-01-16T15:23:32.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.745+0000 7fd49fe00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd48006c550 0x7fd48006ea10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd490036000 tx=0x7fd49003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.745+0000 7fd4a7400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fd4800747b0 0x7fd480076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:32.745 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.746+0000 7fd4a7400640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fd4800747b0 0x7fd480076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:32.746 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.748+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== osd.7 v2:172.21.15.137:6817/1873161633 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd4800772c0 con 0x7fd4800747b0 2024-01-16T15:23:32.759 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.762+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 --> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd4a80acb30 con 0x7fd4800747b0 2024-01-16T15:23:32.760 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.763+0000 7fd49f400640 1 -- 172.21.15.125:0/866903821 <== osd.7 v2:172.21.15.137:6817/1873161633 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd4a80acb30 con 0x7fd4800747b0 2024-01-16T15:23:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2024-01-16T15:23:32.761 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:} 2024-01-16T15:23:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fd4800747b0 msgr2=0x7fd480076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.137:6817/1873161633,v1:172.21.15.137:6819/1873161633] conn(0x7fd4800747b0 0x7fd480076bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd48006c550 msgr2=0x7fd48006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7fd48006c550 0x7fd48006ea10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd490036000 tx=0x7fd49003b040 comp rx=0 tx=0).stop 2024-01-16T15:23:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 msgr2=0x7fd4a8159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 --2- 172.21.15.125:0/866903821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4a80cf9a0 0x7fd4a8159890 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c4d0 tx=0x7fd49800c990 comp rx=0 tx=0).stop 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4a7400640 1 -- 172.21.15.125:0/866903821 reap_dead start 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 shutdown_connections 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 >> 172.21.15.125:0/866903821 conn(0x7fd4a806a800 msgr2=0x7fd4a8142d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 shutdown_connections 2024-01-16T15:23:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:32.764+0000 7fd4ade00640 1 -- 172.21.15.125:0/866903821 wait complete. 2024-01-16T15:23:32.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-16T15:23:32.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.000+0000 7f12c9e00640 1 Processor -- start 2024-01-16T15:23:32.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.000+0000 7f12c9e00640 1 -- start start 2024-01-16T15:23:32.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12c40cf9f0 0x7f12c40cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:32.999 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9e00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c40d0a10 con 0x7f12c4161010 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9e00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c40d03b0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40d1f50 con 0x7f12c4160920 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12c40cf9f0 0x7f12c40cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c3e00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12c40cf9f0 0x7f12c40cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41502/0 (socket says 172.21.15.125:41502) 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c3e00640 1 -- 172.21.15.125:0/320179175 learned_addr learned my addr 172.21.15.125:0/320179175 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:33.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c9400640 1 --1- 172.21.15.125:0/320179175 >> v1:172.21.15.125:6789/0 conn(0x7f12c4161010 0x7f12c41614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50290/0 (socket says 172.21.15.125:50290) 2024-01-16T15:23:33.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.002+0000 7f12c8a00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.003+0000 7f12c3e00640 1 -- 172.21.15.125:0/320179175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 msgr2=0x7f12c40b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.003+0000 7f12c3e00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c40b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.003+0000 7f12c3e00640 1 -- 172.21.15.125:0/320179175 >> v1:172.21.15.125:6789/0 conn(0x7f12c4161010 legacy=0x7f12c41614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.003+0000 7f12c3e00640 1 -- 172.21.15.125:0/320179175 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40cf080 con 0x7f12c40cf9f0 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.003+0000 7f12c3400640 1 -- 172.21.15.125:0/320179175 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137159590 0 0) 0x7f12c40d0a10 con 0x7f12c4161010 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.004+0000 7f12c3e00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c40cfe70 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009870 tx=0x7f12ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b79163b478ab83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.005+0000 7f12c3400640 1 -- 172.21.15.125:0/320179175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac030840 con 0x7f12c40cf9f0 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.005+0000 7f12c3400640 1 -- 172.21.15.125:0/320179175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac0309a0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.005+0000 7f12c3400640 1 -- 172.21.15.125:0/320179175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac030d30 con 0x7f12c40cf9f0 2024-01-16T15:23:33.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.005+0000 7f12c9e00640 1 -- 172.21.15.125:0/320179175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 msgr2=0x7f12c40cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.005+0000 7f12c9e00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c40cfe70 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009870 tx=0x7f12ac02b980 comp rx=0 tx=0).stop 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 -- 172.21.15.125:0/320179175 shutdown_connections 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c40cfe70 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 --2- 172.21.15.125:0/320179175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c40b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 -- 172.21.15.125:0/320179175 >> 172.21.15.125:0/320179175 conn(0x7f12c40d3090 msgr2=0x7f12c40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 -- 172.21.15.125:0/320179175 shutdown_connections 2024-01-16T15:23:33.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.006+0000 7f12c9e00640 1 -- 172.21.15.125:0/320179175 wait complete. 2024-01-16T15:23:33.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.008+0000 7f12c9e00640 1 Processor -- start 2024-01-16T15:23:33.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.008+0000 7f12c9e00640 1 -- start start 2024-01-16T15:23:33.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.006 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12c4161010 0x7f12c40ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40cb290 con 0x7f12c40cf9f0 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40cbb50 con 0x7f12c4161010 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40cacd0 con 0x7f12c4160920 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41518/0 (socket says 172.21.15.125:41518) 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 -- 172.21.15.125:0/1507697920 learned_addr learned my addr 172.21.15.125:0/1507697920 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c3e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.007 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 -- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 msgr2=0x7f12c415e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c9400640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12c4161010 0x7f12c40ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c415e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 -- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12c4161010 msgr2=0x7f12c40ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.009+0000 7f12c8a00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12c4161010 0x7f12c40ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c8a00640 1 -- 172.21.15.125:0/1507697920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac0094d0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c8a00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9b0 tx=0x7f12b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4009670 con 0x7f12c40cf9f0 2024-01-16T15:23:33.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b400c9a0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.009 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400cd50 con 0x7f12c40cf9f0 2024-01-16T15:23:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c40cafb0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.010+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c40b8950 con 0x7f12c40cf9f0 2024-01-16T15:23:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.012+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4067450 con 0x7f12c40cf9f0 2024-01-16T15:23:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.013+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f12b4013470 con 0x7f12c40cf9f0 2024-01-16T15:23:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.014+0000 7f12c0c00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 0x7f129c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.014+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12b40889c0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.014+0000 7f12c3e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 0x7f129c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.011 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.015+0000 7f12c3e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 0x7f129c06ea40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f12ac036000 tx=0x7f12ac03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.015 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.019+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12b40552e0 con 0x7f12c40cf9f0 2024-01-16T15:23:33.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.251+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f12c40ceac0 con 0x7f129c06c580 2024-01-16T15:23:33.248 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.252+0000 7f12c0c00640 1 -- 172.21.15.125:0/1507697920 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f12ac030b90 con 0x7f129c06c580 2024-01-16T15:23:33.248 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 7 to deep-scrub 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 msgr2=0x7f129c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 0x7f129c06ea40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f12ac036000 tx=0x7f12ac03b040 comp rx=0 tx=0).stop 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 msgr2=0x7f12c414f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9b0 tx=0x7f12b400ee70 comp rx=0 tx=0).stop 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 shutdown_connections 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12c4161010 0x7f12c40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f12c4160920 0x7f12c415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f129c06c580 0x7f129c06ea40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 --2- 172.21.15.125:0/1507697920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12c40cf9f0 0x7f12c414f810 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.254+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 >> 172.21.15.125:0/1507697920 conn(0x7f12c40d3090 msgr2=0x7f12c406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:33.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.255+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 shutdown_connections 2024-01-16T15:23:33.253 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.255+0000 7f12c9e00640 1 -- 172.21.15.125:0/1507697920 wait complete. 2024-01-16T15:23:33.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:33.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.466+0000 7f1c0e600640 1 Processor -- start 2024-01-16T15:23:33.463 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.466+0000 7f1c0e600640 1 -- start start 2024-01-16T15:23:33.464 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.467+0000 7f1c0e600640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1c080cf9a0 0x7f1c0815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.467+0000 7f1c0e600640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080b1ee0 con 0x7f1c0815e1e0 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.467+0000 7f1c0e600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080b2780 con 0x7f1c08143ae0 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.467+0000 7f1c0e600640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0815e990 con 0x7f1c080cf9a0 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.468+0000 7f1c06a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1c080cf9a0 0x7f1c0815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.468+0000 7f1c06a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1c080cf9a0 0x7f1c0815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41526/0 (socket says 172.21.15.125:41526) 2024-01-16T15:23:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.468+0000 7f1c06a00640 1 -- 172.21.15.125:0/4045277791 learned_addr learned my addr 172.21.15.125:0/4045277791 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:33.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.468+0000 7f1c06a00640 1 -- 172.21.15.125:0/4045277791 >> v1:172.21.15.137:6789/0 conn(0x7f1c08143ae0 legacy=0x7f1c080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.468+0000 7f1c06a00640 1 -- 172.21.15.125:0/4045277791 >> v1:172.21.15.125:6789/0 conn(0x7f1c0815e1e0 legacy=0x7f1c080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.469+0000 7f1c06a00640 1 -- 172.21.15.125:0/4045277791 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080b2130 con 0x7f1c080cf9a0 2024-01-16T15:23:33.466 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.469+0000 7f1c06000640 1 -- 172.21.15.125:0/4045277791 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849604255 0 0) 0x7f1c080b1ee0 con 0x7f1c0815e1e0 2024-01-16T15:23:33.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.470+0000 7f1c06a00640 1 --2- 172.21.15.125:0/4045277791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c080cf9a0 0x7f1c0815dca0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000bb10 tx=0x7f1bf002e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9041ea80ee102bce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.470+0000 7f1c06000640 1 -- 172.21.15.125:0/4045277791 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0036440 con 0x7f1c080cf9a0 2024-01-16T15:23:33.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.470+0000 7f1c06000640 1 -- 172.21.15.125:0/4045277791 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf00365a0 con 0x7f1c080cf9a0 2024-01-16T15:23:33.467 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.470+0000 7f1c06000640 1 -- 172.21.15.125:0/4045277791 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0036930 con 0x7f1c080cf9a0 2024-01-16T15:23:33.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 -- 172.21.15.125:0/4045277791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c080cf9a0 msgr2=0x7f1c0815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.468 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 --2- 172.21.15.125:0/4045277791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c080cf9a0 0x7f1c0815dca0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000bb10 tx=0x7f1bf002e200 comp rx=0 tx=0).stop 2024-01-16T15:23:33.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 -- 172.21.15.125:0/4045277791 shutdown_connections 2024-01-16T15:23:33.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 --2- 172.21.15.125:0/4045277791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c080cf9a0 0x7f1c0815dca0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 -- 172.21.15.125:0/4045277791 >> 172.21.15.125:0/4045277791 conn(0x7f1c0806a800 msgr2=0x7f1c0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:33.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 -- 172.21.15.125:0/4045277791 shutdown_connections 2024-01-16T15:23:33.469 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.471+0000 7f1c0e600640 1 -- 172.21.15.125:0/4045277791 wait complete. 2024-01-16T15:23:33.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.473+0000 7f1c0e600640 1 Processor -- start 2024-01-16T15:23:33.470 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.474+0000 7f1c0e600640 1 -- start start 2024-01-16T15:23:33.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.474+0000 7f1c0e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 0x7f1c081e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.472 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.474+0000 7f1c0e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c08143ae0 0x7f1c081e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c0e600640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c0e600640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081e5320 con 0x7f1c0815e1e0 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c0e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081e4000 con 0x7f1c080cf9a0 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c0e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081e4280 con 0x7f1c08143ae0 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41538/0 (socket says 172.21.15.125:41538) 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 -- 172.21.15.125:0/3322763082 learned_addr learned my addr 172.21.15.125:0/3322763082 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:33.473 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c06a00640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c08143ae0 0x7f1c081e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07400640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 0x7f1c081e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 -- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c08143ae0 msgr2=0x7f1c081e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c08143ae0 0x7f1c081e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 -- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 msgr2=0x7f1c081e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 0x7f1c081e68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07e00640 1 -- 172.21.15.125:0/3322763082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf000b770 con 0x7f1c0815e1e0 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.475+0000 7f1c07400640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 0x7f1c081e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:33.474 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1c07e00640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c710 tx=0x7f1c0000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00003da0 con 0x7f1c0815e1e0 2024-01-16T15:23:33.475 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000ce20 con 0x7f1c0815e1e0 2024-01-16T15:23:33.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c081e4560 con 0x7f1c0815e1e0 2024-01-16T15:23:33.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00005290 con 0x7f1c0815e1e0 2024-01-16T15:23:33.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.476+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c081e4b20 con 0x7f1c0815e1e0 2024-01-16T15:23:33.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.478+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c080675d0 con 0x7f1c0815e1e0 2024-01-16T15:23:33.476 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.479+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1c000053f0 con 0x7f1c0815e1e0 2024-01-16T15:23:33.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.479+0000 7f1bff400640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 0x7f1be006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.480+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c00088360 con 0x7f1c0815e1e0 2024-01-16T15:23:33.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.480+0000 7f1c07400640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 0x7f1be006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.477 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.481+0000 7f1c07400640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 0x7f1be006ea10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1c081e7f60 tx=0x7f1bf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:33.482 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.485+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c00054900 con 0x7f1c0815e1e0 2024-01-16T15:23:33.689 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.692+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c0815e670 con 0x7f1be006c550 2024-01-16T15:23:33.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.714+0000 7f1bff400640 1 -- 172.21.15.125:0/3322763082 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287852 (secure 0 0 0) 0x7f1bf8001cd0 con 0x7f1be006c550 2024-01-16T15:23:33.715 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:33.717 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.721+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 msgr2=0x7f1be006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.721+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 0x7f1be006ea10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1c081e7f60 tx=0x7f1bf8009040 comp rx=0 tx=0).stop 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.721+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 msgr2=0x7f1c080b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.721+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c710 tx=0x7f1c0000cbd0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 shutdown_connections 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f1be006c550 0x7f1be006ea10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.719 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c0815e1e0 0x7f1c080b0e30 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c08143ae0 0x7f1c081e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 --2- 172.21.15.125:0/3322763082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c080cf9a0 0x7f1c081e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 >> 172.21.15.125:0/3322763082 conn(0x7f1c0806a800 msgr2=0x7f1c0806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:33.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 shutdown_connections 2024-01-16T15:23:33.720 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:33.722+0000 7f1c0e600640 1 -- 172.21.15.125:0/3322763082 wait complete. 2024-01-16T15:23:33.738 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":915,"stamp":"2024-01-16T15:23:32.542117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444268918,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2250,"num_write_kb":6295341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4678,"ondisk_log_size":4678,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13135228,"kb_used_data":12597564,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":741839492,"statfs":{"total":773094113280,"available":759643639808,"internally_reserved":0,"allocated":12899905536,"data_stored":12893402313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11971,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"17.3f","version":"113'70","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.309297+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:23:28.309297+0000","last_peered":"2024-01-16T15:23:28.309297+0000","last_clean":"2024-01-16T15:23:28.309297+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:23:28.309297+0000","last_undegraded":"2024-01-16T15:23:28.309297+0000","last_fullsized":"2024-01-16T15:23:28.309297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'69","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.450714+0000","last_change":"2024-01-16T15:15:07.588972+0000","last_active":"2024-01-16T15:23:18.450714+0000","last_peered":"2024-01-16T15:23:18.450714+0000","last_clean":"2024-01-16T15:23:18.450714+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:18.450714+0000","last_undegraded":"2024-01-16T15:23:18.450714+0000","last_fullsized":"2024-01-16T15:23:18.450714+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'67","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460853+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:23:18.460853+0000","last_peered":"2024-01-16T15:23:18.460853+0000","last_clean":"2024-01-16T15:23:18.460853+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:18.460853+0000","last_undegraded":"2024-01-16T15:23:18.460853+0000","last_fullsized":"2024-01-16T15:23:18.460853+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:44.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458250+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458250+0000","last_peered":"2024-01-16T15:23:18.458250+0000","last_clean":"2024-01-16T15:23:18.458250+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458250+0000","last_undegraded":"2024-01-16T15:23:18.458250+0000","last_fullsized":"2024-01-16T15:23:18.458250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'60","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440467+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:23:18.440467+0000","last_peered":"2024-01-16T15:23:18.440467+0000","last_clean":"2024-01-16T15:23:18.440467+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:18.440467+0000","last_undegraded":"2024-01-16T15:23:18.440467+0000","last_fullsized":"2024-01-16T15:23:18.440467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'73","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448268+0000","last_change":"2024-01-16T15:15:07.584220+0000","last_active":"2024-01-16T15:23:18.448268+0000","last_peered":"2024-01-16T15:23:18.448268+0000","last_clean":"2024-01-16T15:23:18.448268+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:18.448268+0000","last_undegraded":"2024-01-16T15:23:18.448268+0000","last_fullsized":"2024-01-16T15:23:18.448268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'66","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446647+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:23:18.446647+0000","last_peered":"2024-01-16T15:23:18.446647+0000","last_clean":"2024-01-16T15:23:18.446647+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:23:18.446647+0000","last_undegraded":"2024-01-16T15:23:18.446647+0000","last_fullsized":"2024-01-16T15:23:18.446647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'54","reported_seq":81,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459294+0000","last_change":"2024-01-16T15:15:07.582391+0000","last_active":"2024-01-16T15:23:18.459294+0000","last_peered":"2024-01-16T15:23:18.459294+0000","last_clean":"2024-01-16T15:23:18.459294+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:18.459294+0000","last_undegraded":"2024-01-16T15:23:18.459294+0000","last_fullsized":"2024-01-16T15:23:18.459294+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'79","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456769+0000","last_change":"2024-01-16T15:15:07.597782+0000","last_active":"2024-01-16T15:23:18.456769+0000","last_peered":"2024-01-16T15:23:18.456769+0000","last_clean":"2024-01-16T15:23:18.456769+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:18.456769+0000","last_undegraded":"2024-01-16T15:23:18.456769+0000","last_fullsized":"2024-01-16T15:23:18.456769+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'66","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448403+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:23:18.448403+0000","last_peered":"2024-01-16T15:23:18.448403+0000","last_clean":"2024-01-16T15:23:18.448403+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:23:18.448403+0000","last_undegraded":"2024-01-16T15:23:18.448403+0000","last_fullsized":"2024-01-16T15:23:18.448403+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446325+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.446325+0000","last_peered":"2024-01-16T15:23:18.446325+0000","last_clean":"2024-01-16T15:23:18.446325+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.446325+0000","last_undegraded":"2024-01-16T15:23:18.446325+0000","last_fullsized":"2024-01-16T15:23:18.446325+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449095+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.449095+0000","last_peered":"2024-01-16T15:23:18.449095+0000","last_clean":"2024-01-16T15:23:18.449095+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.449095+0000","last_undegraded":"2024-01-16T15:23:18.449095+0000","last_fullsized":"2024-01-16T15:23:18.449095+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'57","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449865+0000","last_change":"2024-01-16T15:15:07.580530+0000","last_active":"2024-01-16T15:23:18.449865+0000","last_peered":"2024-01-16T15:23:18.449865+0000","last_clean":"2024-01-16T15:23:18.449865+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:18.449865+0000","last_undegraded":"2024-01-16T15:23:18.449865+0000","last_fullsized":"2024-01-16T15:23:18.449865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'65","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454924+0000","last_change":"2024-01-16T15:15:07.590172+0000","last_active":"2024-01-16T15:23:18.454924+0000","last_peered":"2024-01-16T15:23:18.454924+0000","last_clean":"2024-01-16T15:23:18.454924+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:18.454924+0000","last_undegraded":"2024-01-16T15:23:18.454924+0000","last_fullsized":"2024-01-16T15:23:18.454924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:36.729507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'54","reported_seq":78,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447117+0000","last_change":"2024-01-16T15:15:07.586446+0000","last_active":"2024-01-16T15:23:18.447117+0000","last_peered":"2024-01-16T15:23:18.447117+0000","last_clean":"2024-01-16T15:23:18.447117+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:18.447117+0000","last_undegraded":"2024-01-16T15:23:18.447117+0000","last_fullsized":"2024-01-16T15:23:18.447117+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'58","reported_seq":77,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458944+0000","last_change":"2024-01-16T15:15:28.509366+0000","last_active":"2024-01-16T15:23:18.458944+0000","last_peered":"2024-01-16T15:23:18.458944+0000","last_clean":"2024-01-16T15:23:18.458944+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:23:18.458944+0000","last_undegraded":"2024-01-16T15:23:18.458944+0000","last_fullsized":"2024-01-16T15:23:18.458944+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'49","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447621+0000","last_change":"2024-01-16T15:15:07.595007+0000","last_active":"2024-01-16T15:23:18.447621+0000","last_peered":"2024-01-16T15:23:18.447621+0000","last_clean":"2024-01-16T15:23:18.447621+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:18.447621+0000","last_undegraded":"2024-01-16T15:23:18.447621+0000","last_fullsized":"2024-01-16T15:23:18.447621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'72","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455262+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:23:18.455262+0000","last_peered":"2024-01-16T15:23:18.455262+0000","last_clean":"2024-01-16T15:23:18.455262+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:18.455262+0000","last_undegraded":"2024-01-16T15:23:18.455262+0000","last_fullsized":"2024-01-16T15:23:18.455262+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.313949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'71","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456798+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:23:18.456798+0000","last_peered":"2024-01-16T15:23:18.456798+0000","last_clean":"2024-01-16T15:23:18.456798+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:18.456798+0000","last_undegraded":"2024-01-16T15:23:18.456798+0000","last_fullsized":"2024-01-16T15:23:18.456798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:30.836159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'67","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453469+0000","last_change":"2024-01-16T15:15:07.574649+0000","last_active":"2024-01-16T15:23:18.453469+0000","last_peered":"2024-01-16T15:23:18.453469+0000","last_clean":"2024-01-16T15:23:18.453469+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:18.453469+0000","last_undegraded":"2024-01-16T15:23:18.453469+0000","last_fullsized":"2024-01-16T15:23:18.453469+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'73","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454699+0000","last_change":"2024-01-16T15:15:07.594094+0000","last_active":"2024-01-16T15:23:18.454699+0000","last_peered":"2024-01-16T15:23:18.454699+0000","last_clean":"2024-01-16T15:23:18.454699+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:18.454699+0000","last_undegraded":"2024-01-16T15:23:18.454699+0000","last_fullsized":"2024-01-16T15:23:18.454699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'64","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453747+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:23:18.453747+0000","last_peered":"2024-01-16T15:23:18.453747+0000","last_clean":"2024-01-16T15:23:18.453747+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:18.453747+0000","last_undegraded":"2024-01-16T15:23:18.453747+0000","last_fullsized":"2024-01-16T15:23:18.453747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:18.912933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'57","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454432+0000","last_change":"2024-01-16T15:15:07.573047+0000","last_active":"2024-01-16T15:23:18.454432+0000","last_peered":"2024-01-16T15:23:18.454432+0000","last_clean":"2024-01-16T15:23:18.454432+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:18.454432+0000","last_undegraded":"2024-01-16T15:23:18.454432+0000","last_fullsized":"2024-01-16T15:23:18.454432+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:31.248088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'70","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459248+0000","last_change":"2024-01-16T15:15:07.579472+0000","last_active":"2024-01-16T15:23:18.459248+0000","last_peered":"2024-01-16T15:23:18.459248+0000","last_clean":"2024-01-16T15:23:18.459248+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:18.459248+0000","last_undegraded":"2024-01-16T15:23:18.459248+0000","last_fullsized":"2024-01-16T15:23:18.459248+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:05:24.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'71","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454209+0000","last_change":"2024-01-16T15:15:07.593001+0000","last_active":"2024-01-16T15:23:18.454209+0000","last_peered":"2024-01-16T15:23:18.454209+0000","last_clean":"2024-01-16T15:23:18.454209+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:18.454209+0000","last_undegraded":"2024-01-16T15:23:18.454209+0000","last_fullsized":"2024-01-16T15:23:18.454209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'81","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454550+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:23:18.454550+0000","last_peered":"2024-01-16T15:23:18.454550+0000","last_clean":"2024-01-16T15:23:18.454550+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:23:18.454550+0000","last_undegraded":"2024-01-16T15:23:18.454550+0000","last_fullsized":"2024-01-16T15:23:18.454550+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'64","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.450817+0000","last_change":"2024-01-16T15:15:07.593726+0000","last_active":"2024-01-16T15:23:18.450817+0000","last_peered":"2024-01-16T15:23:18.450817+0000","last_clean":"2024-01-16T15:23:18.450817+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:18.450817+0000","last_undegraded":"2024-01-16T15:23:18.450817+0000","last_fullsized":"2024-01-16T15:23:18.450817+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'43","reported_seq":64,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455446+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:23:18.455446+0000","last_peered":"2024-01-16T15:23:18.455446+0000","last_clean":"2024-01-16T15:23:18.455446+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:23:18.455446+0000","last_undegraded":"2024-01-16T15:23:18.455446+0000","last_fullsized":"2024-01-16T15:23:18.455446+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456321+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:18.456321+0000","last_peered":"2024-01-16T15:23:18.456321+0000","last_clean":"2024-01-16T15:23:18.456321+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:18.456321+0000","last_undegraded":"2024-01-16T15:23:18.456321+0000","last_fullsized":"2024-01-16T15:23:18.456321+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:47.759998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'76","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.445140+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:23:18.445140+0000","last_peered":"2024-01-16T15:23:18.445140+0000","last_clean":"2024-01-16T15:23:18.445140+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:23:18.445140+0000","last_undegraded":"2024-01-16T15:23:18.445140+0000","last_fullsized":"2024-01-16T15:23:18.445140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:31.396223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'81","reported_seq":118,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308899+0000","last_change":"2024-01-16T15:15:07.594221+0000","last_active":"2024-01-16T15:23:28.308899+0000","last_peered":"2024-01-16T15:23:28.308899+0000","last_clean":"2024-01-16T15:23:28.308899+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:28.308899+0000","last_undegraded":"2024-01-16T15:23:28.308899+0000","last_fullsized":"2024-01-16T15:23:28.308899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'63","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.009544+0000","last_change":"2024-01-16T15:15:07.585313+0000","last_active":"2024-01-16T15:23:29.009544+0000","last_peered":"2024-01-16T15:23:29.009544+0000","last_clean":"2024-01-16T15:23:29.009544+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:29.009544+0000","last_undegraded":"2024-01-16T15:23:29.009544+0000","last_fullsized":"2024-01-16T15:23:29.009544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'62","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446245+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:23:18.446245+0000","last_peered":"2024-01-16T15:23:18.446245+0000","last_clean":"2024-01-16T15:23:18.446245+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:18.446245+0000","last_undegraded":"2024-01-16T15:23:18.446245+0000","last_fullsized":"2024-01-16T15:23:18.446245+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":22,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:24.440876+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:21:24.440876+0000","last_peered":"2024-01-16T15:21:24.440876+0000","last_clean":"2024-01-16T15:21:24.440876+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:21:24.440876+0000","last_undegraded":"2024-01-16T15:21:24.440876+0000","last_fullsized":"2024-01-16T15:21:24.440876+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:45.868245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'54","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455690+0000","last_change":"2024-01-16T15:15:07.588649+0000","last_active":"2024-01-16T15:23:18.455690+0000","last_peered":"2024-01-16T15:23:18.455690+0000","last_clean":"2024-01-16T15:23:18.455690+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:18.455690+0000","last_undegraded":"2024-01-16T15:23:18.455690+0000","last_fullsized":"2024-01-16T15:23:18.455690+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:36.780302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:09.370582+0000","last_change":"2024-01-16T15:15:06.559048+0000","last_active":"2024-01-16T15:21:09.370582+0000","last_peered":"2024-01-16T15:21:09.370582+0000","last_clean":"2024-01-16T15:21:09.370582+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:21:09.370582+0000","last_undegraded":"2024-01-16T15:21:09.370582+0000","last_fullsized":"2024-01-16T15:21:09.370582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'71","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449254+0000","last_change":"2024-01-16T15:15:07.596414+0000","last_active":"2024-01-16T15:23:18.449254+0000","last_peered":"2024-01-16T15:23:18.449254+0000","last_clean":"2024-01-16T15:23:18.449254+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:18.449254+0000","last_undegraded":"2024-01-16T15:23:18.449254+0000","last_fullsized":"2024-01-16T15:23:18.449254+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'3","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:39.732865+0000","last_change":"2024-01-16T15:15:06.558462+0000","last_active":"2024-01-16T15:22:39.732865+0000","last_peered":"2024-01-16T15:22:39.732865+0000","last_clean":"2024-01-16T15:22:39.732865+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:22:39.732865+0000","last_undegraded":"2024-01-16T15:22:39.732865+0000","last_fullsized":"2024-01-16T15:22:39.732865+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'68","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.459017+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:23:18.459017+0000","last_peered":"2024-01-16T15:23:18.459017+0000","last_clean":"2024-01-16T15:23:18.459017+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:23:18.459017+0000","last_undegraded":"2024-01-16T15:23:18.459017+0000","last_fullsized":"2024-01-16T15:23:18.459017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:41:44.553974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'2","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:39.732818+0000","last_change":"2024-01-16T15:15:06.557679+0000","last_active":"2024-01-16T15:22:39.732818+0000","last_peered":"2024-01-16T15:22:39.732818+0000","last_clean":"2024-01-16T15:22:39.732818+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:22:39.732818+0000","last_undegraded":"2024-01-16T15:22:39.732818+0000","last_fullsized":"2024-01-16T15:22:39.732818+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'72","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458004+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:23:18.458004+0000","last_peered":"2024-01-16T15:23:18.458004+0000","last_clean":"2024-01-16T15:23:18.458004+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:23:18.458004+0000","last_undegraded":"2024-01-16T15:23:18.458004+0000","last_fullsized":"2024-01-16T15:23:18.458004+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.427723+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:20:09.427723+0000","last_peered":"2024-01-16T15:20:09.427723+0000","last_clean":"2024-01-16T15:20:09.427723+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:20:09.427723+0000","last_undegraded":"2024-01-16T15:20:09.427723+0000","last_fullsized":"2024-01-16T15:20:09.427723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:58.319334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'72","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454108+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:23:18.454108+0000","last_peered":"2024-01-16T15:23:18.454108+0000","last_clean":"2024-01-16T15:23:18.454108+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:23:18.454108+0000","last_undegraded":"2024-01-16T15:23:18.454108+0000","last_fullsized":"2024-01-16T15:23:18.454108+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.709208+0000","last_change":"2024-01-16T15:15:06.556349+0000","last_active":"2024-01-16T15:20:09.709208+0000","last_peered":"2024-01-16T15:20:09.709208+0000","last_clean":"2024-01-16T15:20:09.709208+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:20:09.709208+0000","last_undegraded":"2024-01-16T15:20:09.709208+0000","last_fullsized":"2024-01-16T15:20:09.709208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:30.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'73","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.450054+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:23:18.450054+0000","last_peered":"2024-01-16T15:23:18.450054+0000","last_clean":"2024-01-16T15:23:18.450054+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:23:18.450054+0000","last_undegraded":"2024-01-16T15:23:18.450054+0000","last_fullsized":"2024-01-16T15:23:18.450054+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.548197+0000","last_change":"2024-01-16T15:15:06.556429+0000","last_active":"2024-01-16T15:19:59.548197+0000","last_peered":"2024-01-16T15:19:59.548197+0000","last_clean":"2024-01-16T15:19:59.548197+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:19:59.548197+0000","last_undegraded":"2024-01-16T15:19:59.548197+0000","last_fullsized":"2024-01-16T15:19:59.548197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:35.765248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'73","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.443778+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:23:18.443778+0000","last_peered":"2024-01-16T15:23:18.443778+0000","last_clean":"2024-01-16T15:23:18.443778+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:23:18.443778+0000","last_undegraded":"2024-01-16T15:23:18.443778+0000","last_fullsized":"2024-01-16T15:23:18.443778+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:52.948874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.816419+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:22:24.816419+0000","last_peered":"2024-01-16T15:22:24.816419+0000","last_clean":"2024-01-16T15:22:24.816419+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:22:24.816419+0000","last_undegraded":"2024-01-16T15:22:24.816419+0000","last_fullsized":"2024-01-16T15:22:24.816419+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'58","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448671+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:23:18.448671+0000","last_peered":"2024-01-16T15:23:18.448671+0000","last_clean":"2024-01-16T15:23:18.448671+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:23:18.448671+0000","last_undegraded":"2024-01-16T15:23:18.448671+0000","last_fullsized":"2024-01-16T15:23:18.448671+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:06.520921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.730006+0000","last_change":"2024-01-16T15:15:06.563564+0000","last_active":"2024-01-16T15:22:24.730006+0000","last_peered":"2024-01-16T15:22:24.730006+0000","last_clean":"2024-01-16T15:22:24.730006+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:22:24.730006+0000","last_undegraded":"2024-01-16T15:22:24.730006+0000","last_fullsized":"2024-01-16T15:22:24.730006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'61","reported_seq":85,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451772+0000","last_change":"2024-01-16T15:15:07.599186+0000","last_active":"2024-01-16T15:23:18.451772+0000","last_peered":"2024-01-16T15:23:18.451772+0000","last_clean":"2024-01-16T15:23:18.451772+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:18.451772+0000","last_undegraded":"2024-01-16T15:23:18.451772+0000","last_fullsized":"2024-01-16T15:23:18.451772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:07.675081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.707473+0000","last_change":"2024-01-16T15:15:06.562108+0000","last_active":"2024-01-16T15:19:59.707473+0000","last_peered":"2024-01-16T15:19:59.707473+0000","last_clean":"2024-01-16T15:19:59.707473+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:19:59.707473+0000","last_undegraded":"2024-01-16T15:19:59.707473+0000","last_fullsized":"2024-01-16T15:19:59.707473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:45.361792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'62","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454199+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:23:18.454199+0000","last_peered":"2024-01-16T15:23:18.454199+0000","last_clean":"2024-01-16T15:23:18.454199+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:23:18.454199+0000","last_undegraded":"2024-01-16T15:23:18.454199+0000","last_fullsized":"2024-01-16T15:23:18.454199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.433437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'31","reported_seq":49,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.816519+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:22:24.816519+0000","last_peered":"2024-01-16T15:22:24.816519+0000","last_clean":"2024-01-16T15:22:24.816519+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:22:24.816519+0000","last_undegraded":"2024-01-16T15:22:24.816519+0000","last_fullsized":"2024-01-16T15:22:24.816519+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'69","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453483+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:23:18.453483+0000","last_peered":"2024-01-16T15:23:18.453483+0000","last_clean":"2024-01-16T15:23:18.453483+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:23:18.453483+0000","last_undegraded":"2024-01-16T15:23:18.453483+0000","last_fullsized":"2024-01-16T15:23:18.453483+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'321","reported_seq":339,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:23.729120+0000","last_change":"2024-01-16T15:15:06.560918+0000","last_active":"2024-01-16T15:23:23.729120+0000","last_peered":"2024-01-16T15:23:23.729120+0000","last_clean":"2024-01-16T15:23:23.729120+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:23.729120+0000","last_undegraded":"2024-01-16T15:23:23.729120+0000","last_fullsized":"2024-01-16T15:23:23.729120+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1221104,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'54","reported_seq":79,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440515+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:23:18.440515+0000","last_peered":"2024-01-16T15:23:18.440515+0000","last_clean":"2024-01-16T15:23:18.440515+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:18.440515+0000","last_undegraded":"2024-01-16T15:23:18.440515+0000","last_fullsized":"2024-01-16T15:23:18.440515+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:01.213173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'2","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:10.087124+0000","last_change":"2024-01-16T15:15:06.557798+0000","last_active":"2024-01-16T15:22:10.087124+0000","last_peered":"2024-01-16T15:22:10.087124+0000","last_clean":"2024-01-16T15:22:10.087124+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:22:10.087124+0000","last_undegraded":"2024-01-16T15:22:10.087124+0000","last_fullsized":"2024-01-16T15:22:10.087124+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'72","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.452704+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:23:18.452704+0000","last_peered":"2024-01-16T15:23:18.452704+0000","last_clean":"2024-01-16T15:23:18.452704+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:18.452704+0000","last_undegraded":"2024-01-16T15:23:18.452704+0000","last_fullsized":"2024-01-16T15:23:18.452704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.562805+0000","last_change":"2024-01-16T15:15:06.561153+0000","last_active":"2024-01-16T15:19:49.562805+0000","last_peered":"2024-01-16T15:19:49.562805+0000","last_clean":"2024-01-16T15:19:49.562805+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:19:49.562805+0000","last_undegraded":"2024-01-16T15:19:49.562805+0000","last_fullsized":"2024-01-16T15:19:49.562805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:13.036652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'63","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456290+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:23:18.456290+0000","last_peered":"2024-01-16T15:23:18.456290+0000","last_clean":"2024-01-16T15:23:18.456290+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:18.456290+0000","last_undegraded":"2024-01-16T15:23:18.456290+0000","last_fullsized":"2024-01-16T15:23:18.456290+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:18.842216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'8","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:09.727133+0000","last_change":"2024-01-16T15:15:06.548911+0000","last_active":"2024-01-16T15:22:09.727133+0000","last_peered":"2024-01-16T15:22:09.727133+0000","last_clean":"2024-01-16T15:22:09.727133+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:22:09.727133+0000","last_undegraded":"2024-01-16T15:22:09.727133+0000","last_fullsized":"2024-01-16T15:22:09.727133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'46","reported_seq":69,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448231+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:23:18.448231+0000","last_peered":"2024-01-16T15:23:18.448231+0000","last_clean":"2024-01-16T15:23:18.448231+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:18.448231+0000","last_undegraded":"2024-01-16T15:23:18.448231+0000","last_fullsized":"2024-01-16T15:23:18.448231+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:54.811032+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:21:54.811032+0000","last_peered":"2024-01-16T15:21:54.811032+0000","last_clean":"2024-01-16T15:21:54.811032+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:21:54.811032+0000","last_undegraded":"2024-01-16T15:21:54.811032+0000","last_fullsized":"2024-01-16T15:21:54.811032+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'61","reported_seq":94,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451512+0000","last_change":"2024-01-16T15:15:07.599583+0000","last_active":"2024-01-16T15:23:18.451512+0000","last_peered":"2024-01-16T15:23:18.451512+0000","last_clean":"2024-01-16T15:23:18.451512+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:18.451512+0000","last_undegraded":"2024-01-16T15:23:18.451512+0000","last_fullsized":"2024-01-16T15:23:18.451512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'2","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:54.724753+0000","last_change":"2024-01-16T15:15:06.554926+0000","last_active":"2024-01-16T15:21:54.724753+0000","last_peered":"2024-01-16T15:21:54.724753+0000","last_clean":"2024-01-16T15:21:54.724753+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:21:54.724753+0000","last_undegraded":"2024-01-16T15:21:54.724753+0000","last_fullsized":"2024-01-16T15:21:54.724753+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'61","reported_seq":91,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455552+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:23:18.455552+0000","last_peered":"2024-01-16T15:23:18.455552+0000","last_clean":"2024-01-16T15:23:18.455552+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:18.455552+0000","last_undegraded":"2024-01-16T15:23:18.455552+0000","last_fullsized":"2024-01-16T15:23:18.455552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:38.025065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'1","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.561225+0000","last_change":"2024-01-16T15:15:06.558256+0000","last_active":"2024-01-16T15:19:39.561225+0000","last_peered":"2024-01-16T15:19:39.561225+0000","last_clean":"2024-01-16T15:19:39.561225+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:19:39.561225+0000","last_undegraded":"2024-01-16T15:19:39.561225+0000","last_fullsized":"2024-01-16T15:19:39.561225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:51.933821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'62","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460777+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:23:18.460777+0000","last_peered":"2024-01-16T15:23:18.460777+0000","last_clean":"2024-01-16T15:23:18.460777+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:23:18.460777+0000","last_undegraded":"2024-01-16T15:23:18.460777+0000","last_fullsized":"2024-01-16T15:23:18.460777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:03.513354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:39.422539+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:19:39.422539+0000","last_peered":"2024-01-16T15:19:39.422539+0000","last_clean":"2024-01-16T15:19:39.422539+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:19:39.422539+0000","last_undegraded":"2024-01-16T15:19:39.422539+0000","last_fullsized":"2024-01-16T15:19:39.422539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.185415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'56","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455067+0000","last_change":"2024-01-16T15:15:07.582344+0000","last_active":"2024-01-16T15:23:18.455067+0000","last_peered":"2024-01-16T15:23:18.455067+0000","last_clean":"2024-01-16T15:23:18.455067+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:18.455067+0000","last_undegraded":"2024-01-16T15:23:18.455067+0000","last_fullsized":"2024-01-16T15:23:18.455067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:04.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'3","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:55.085490+0000","last_change":"2024-01-16T15:15:06.561693+0000","last_active":"2024-01-16T15:21:55.085490+0000","last_peered":"2024-01-16T15:21:55.085490+0000","last_clean":"2024-01-16T15:21:55.085490+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:21:55.085490+0000","last_undegraded":"2024-01-16T15:21:55.085490+0000","last_fullsized":"2024-01-16T15:21:55.085490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457265+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:18.457265+0000","last_peered":"2024-01-16T15:23:18.457265+0000","last_clean":"2024-01-16T15:23:18.457265+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:18.457265+0000","last_undegraded":"2024-01-16T15:23:18.457265+0000","last_fullsized":"2024-01-16T15:23:18.457265+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:39.721686+0000","last_change":"2024-01-16T15:15:06.562089+0000","last_active":"2024-01-16T15:21:39.721686+0000","last_peered":"2024-01-16T15:21:39.721686+0000","last_clean":"2024-01-16T15:21:39.721686+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:21:39.721686+0000","last_undegraded":"2024-01-16T15:21:39.721686+0000","last_fullsized":"2024-01-16T15:21:39.721686+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'65","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460089+0000","last_change":"2024-01-16T15:15:07.576331+0000","last_active":"2024-01-16T15:23:18.460089+0000","last_peered":"2024-01-16T15:23:18.460089+0000","last_clean":"2024-01-16T15:23:18.460089+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:18.460089+0000","last_undegraded":"2024-01-16T15:23:18.460089+0000","last_fullsized":"2024-01-16T15:23:18.460089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:51.939051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:40.082963+0000","last_change":"2024-01-16T15:15:06.560462+0000","last_active":"2024-01-16T15:21:40.082963+0000","last_peered":"2024-01-16T15:21:40.082963+0000","last_clean":"2024-01-16T15:21:40.082963+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:21:40.082963+0000","last_undegraded":"2024-01-16T15:21:40.082963+0000","last_fullsized":"2024-01-16T15:21:40.082963+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'57","reported_seq":83,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456066+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:23:18.456066+0000","last_peered":"2024-01-16T15:23:18.456066+0000","last_clean":"2024-01-16T15:23:18.456066+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:18.456066+0000","last_undegraded":"2024-01-16T15:23:18.456066+0000","last_fullsized":"2024-01-16T15:23:18.456066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:27.598966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:39.807156+0000","last_change":"2024-01-16T15:15:06.564599+0000","last_active":"2024-01-16T15:21:39.807156+0000","last_peered":"2024-01-16T15:21:39.807156+0000","last_clean":"2024-01-16T15:21:39.807156+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:21:39.807156+0000","last_undegraded":"2024-01-16T15:21:39.807156+0000","last_fullsized":"2024-01-16T15:21:39.807156+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'53","reported_seq":75,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.451025+0000","last_change":"2024-01-16T15:15:07.599124+0000","last_active":"2024-01-16T15:23:18.451025+0000","last_peered":"2024-01-16T15:23:18.451025+0000","last_clean":"2024-01-16T15:23:18.451025+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:18.451025+0000","last_undegraded":"2024-01-16T15:23:18.451025+0000","last_fullsized":"2024-01-16T15:23:18.451025+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:25.587391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:29.701583+0000","last_change":"2024-01-16T15:15:06.565533+0000","last_active":"2024-01-16T15:19:29.701583+0000","last_peered":"2024-01-16T15:19:29.701583+0000","last_clean":"2024-01-16T15:19:29.701583+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:19:29.701583+0000","last_undegraded":"2024-01-16T15:19:29.701583+0000","last_fullsized":"2024-01-16T15:19:29.701583+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:46.744945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'61","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457751+0000","last_change":"2024-01-16T15:15:07.592351+0000","last_active":"2024-01-16T15:23:18.457751+0000","last_peered":"2024-01-16T15:23:18.457751+0000","last_clean":"2024-01-16T15:23:18.457751+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:18.457751+0000","last_undegraded":"2024-01-16T15:23:18.457751+0000","last_fullsized":"2024-01-16T15:23:18.457751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'32","reported_seq":49,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:21.871035+0000","last_change":"2024-01-16T15:15:06.559863+0000","last_active":"2024-01-16T15:23:21.871035+0000","last_peered":"2024-01-16T15:23:21.871035+0000","last_clean":"2024-01-16T15:23:21.871035+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:23:21.871035+0000","last_undegraded":"2024-01-16T15:23:21.871035+0000","last_fullsized":"2024-01-16T15:23:21.871035+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:05.505867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308852+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:28.308852+0000","last_peered":"2024-01-16T15:23:28.308852+0000","last_clean":"2024-01-16T15:23:28.308852+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:28.308852+0000","last_undegraded":"2024-01-16T15:23:28.308852+0000","last_fullsized":"2024-01-16T15:23:28.308852+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308451+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:23:28.308451+0000","last_peered":"2024-01-16T15:23:28.308451+0000","last_clean":"2024-01-16T15:23:28.308451+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:23:28.308451+0000","last_undegraded":"2024-01-16T15:23:28.308451+0000","last_fullsized":"2024-01-16T15:23:28.308451+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'59","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455114+0000","last_change":"2024-01-16T15:15:07.592439+0000","last_active":"2024-01-16T15:23:18.455114+0000","last_peered":"2024-01-16T15:23:18.455114+0000","last_clean":"2024-01-16T15:23:18.455114+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:18.455114+0000","last_undegraded":"2024-01-16T15:23:18.455114+0000","last_fullsized":"2024-01-16T15:23:18.455114+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:55.117906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:24.723016+0000","last_change":"2024-01-16T15:15:06.561322+0000","last_active":"2024-01-16T15:21:24.723016+0000","last_peered":"2024-01-16T15:21:24.723016+0000","last_clean":"2024-01-16T15:21:24.723016+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:21:24.723016+0000","last_undegraded":"2024-01-16T15:21:24.723016+0000","last_fullsized":"2024-01-16T15:21:24.723016+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:08.898773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'63","reported_seq":88,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453619+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:23:18.453619+0000","last_peered":"2024-01-16T15:23:18.453619+0000","last_clean":"2024-01-16T15:23:18.453619+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:23:18.453619+0000","last_undegraded":"2024-01-16T15:23:18.453619+0000","last_fullsized":"2024-01-16T15:23:18.453619+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.913826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:24.440751+0000","last_change":"2024-01-16T15:15:06.558734+0000","last_active":"2024-01-16T15:21:24.440751+0000","last_peered":"2024-01-16T15:21:24.440751+0000","last_clean":"2024-01-16T15:21:24.440751+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:21:24.440751+0000","last_undegraded":"2024-01-16T15:21:24.440751+0000","last_fullsized":"2024-01-16T15:21:24.440751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:13.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'66","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455563+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:23:18.455563+0000","last_peered":"2024-01-16T15:23:18.455563+0000","last_clean":"2024-01-16T15:23:18.455563+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:18.455563+0000","last_undegraded":"2024-01-16T15:23:18.455563+0000","last_fullsized":"2024-01-16T15:23:18.455563+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:24.569589+0000","last_change":"2024-01-16T15:15:06.559344+0000","last_active":"2024-01-16T15:21:24.569589+0000","last_peered":"2024-01-16T15:21:24.569589+0000","last_clean":"2024-01-16T15:21:24.569589+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:21:24.569589+0000","last_undegraded":"2024-01-16T15:21:24.569589+0000","last_fullsized":"2024-01-16T15:21:24.569589+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:06.660028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'68","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457418+0000","last_change":"2024-01-16T15:15:07.589807+0000","last_active":"2024-01-16T15:23:18.457418+0000","last_peered":"2024-01-16T15:23:18.457418+0000","last_clean":"2024-01-16T15:23:18.457418+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:18.457418+0000","last_undegraded":"2024-01-16T15:23:18.457418+0000","last_fullsized":"2024-01-16T15:23:18.457418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:46.268498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'33","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.713828+0000","last_change":"2024-01-16T15:15:06.557860+0000","last_active":"2024-01-16T15:23:29.713828+0000","last_peered":"2024-01-16T15:23:29.713828+0000","last_clean":"2024-01-16T15:23:29.713828+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:23:29.713828+0000","last_undegraded":"2024-01-16T15:23:29.713828+0000","last_fullsized":"2024-01-16T15:23:29.713828+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'70","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.444662+0000","last_change":"2024-01-16T15:15:28.574045+0000","last_active":"2024-01-16T15:23:18.444662+0000","last_peered":"2024-01-16T15:23:18.444662+0000","last_clean":"2024-01-16T15:23:18.444662+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:23:18.444662+0000","last_undegraded":"2024-01-16T15:23:18.444662+0000","last_fullsized":"2024-01-16T15:23:18.444662+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:46.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:09.823522+0000","last_change":"2024-01-16T15:15:06.562655+0000","last_active":"2024-01-16T15:23:09.823522+0000","last_peered":"2024-01-16T15:23:09.823522+0000","last_clean":"2024-01-16T15:23:09.823522+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:23:09.823522+0000","last_undegraded":"2024-01-16T15:23:09.823522+0000","last_fullsized":"2024-01-16T15:23:09.823522+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'49","reported_seq":78,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458455+0000","last_change":"2024-01-16T15:15:07.575694+0000","last_active":"2024-01-16T15:23:18.458455+0000","last_peered":"2024-01-16T15:23:18.458455+0000","last_clean":"2024-01-16T15:23:18.458455+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:18.458455+0000","last_undegraded":"2024-01-16T15:23:18.458455+0000","last_fullsized":"2024-01-16T15:23:18.458455+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:23.575739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.424236+0000","last_change":"2024-01-16T15:15:06.558658+0000","last_active":"2024-01-16T15:19:49.424236+0000","last_peered":"2024-01-16T15:19:49.424236+0000","last_clean":"2024-01-16T15:19:49.424236+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:19:49.424236+0000","last_undegraded":"2024-01-16T15:19:49.424236+0000","last_fullsized":"2024-01-16T15:19:49.424236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:32.653896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":271,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.364834+0000","last_change":"2024-01-16T14:56:35.668532+0000","last_active":"2024-01-16T15:20:39.364834+0000","last_peered":"2024-01-16T15:20:39.364834+0000","last_clean":"2024-01-16T15:20:39.364834+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:20:39.364834+0000","last_undegraded":"2024-01-16T15:20:39.364834+0000","last_fullsized":"2024-01-16T15:20:39.364834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:56:34.224878+0000","last_clean_scrub_stamp":"2024-01-16T14:56:34.224878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:48.754033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4112,"ondisk_log_size":4112,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1225546,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":487,"num_write_kb":1431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2506752,"data_stored":2451092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575175,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.151,"15min":1.135},"min":{"1min":0.89800000000000002,"5min":0.66000000000000003,"15min":0.60199999999999998},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.86299999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.5580000000000001,"5min":2.0619999999999998,"15min":5.9619999999999997},"last":1.0529999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1950000000000001,"15min":1.256},"min":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.56200000000000006},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.359,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.73099999999999998},"max":{"1min":1.956,"5min":1.956,"15min":6.2690000000000001},"last":1.167}]},{"osd":2,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.24,"15min":1.228},"min":{"1min":0.80900000000000005,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.7310000000000001,"5min":2.0070000000000001,"15min":3.7290000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.28,"15min":1.2809999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.5449999999999999,"5min":1.8129999999999999,"15min":3.3919999999999999},"last":1.0980000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3240000000000001,"15min":1.3660000000000001},"min":{"1min":0.97999999999999998,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.649,"5min":1.9279999999999999,"15min":12.331},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.3640000000000001},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.645},"last":0.99399999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":0.49199999999999999},"max":{"1min":1.837,"5min":1.837,"15min":6.1799999999999997},"last":1.363},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3240000000000001,"15min":1.3169999999999999},"min":{"1min":0.94699999999999995,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.7410000000000001,"5min":1.9510000000000001,"15min":2.9809999999999999},"last":1.508}]},{"osd":5,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.89000000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.508,"5min":2.508,"15min":2.593},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.502,"15min":1.528},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.54700000000000004},"max":{"1min":2.7090000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.458}]},{"osd":6,"last update":"Tue Jan 16 15:22:36 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.484,"15min":1.4990000000000001},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.823,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.593,"5min":1.5640000000000001,"15min":1.585},"min":{"1min":1.0940000000000001,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":4.7649999999999997},"last":1.4850000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575176,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.236}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.3}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.135},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.403}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":1.7}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.679}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.353},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.6240000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575175,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.179,"15min":1.159},"min":{"1min":0.628,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":1.9419999999999999,"5min":2.2040000000000002,"15min":3.7130000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.49299999999999999},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":8.7880000000000003},"last":0.76500000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":2.1389999999999998,"15min":3.835},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.292,"5min":1.3120000000000001,"15min":1.298},"min":{"1min":0.67400000000000004,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.95,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":0.94299999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.292,"15min":1.329},"min":{"1min":0.96299999999999997,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":1.6120000000000001,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.101},{"interface":"front","average":{"1min":1.351,"5min":1.373,"15min":1.3720000000000001},"min":{"1min":1.0209999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.673,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.159}]},{"osd":3,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.325,"15min":1.3400000000000001},"min":{"1min":0.86599999999999999,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.6719999999999999,"5min":1.742,"15min":3.9319999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.86699999999999999,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":16.158000000000001},"last":1.0620000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.278,"15min":1.238},"min":{"1min":0.91000000000000003,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":1.8009999999999999,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":1.488},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.488,"15min":1.4910000000000001},"min":{"1min":1.0920000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.887,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.4490000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":1.117,"5min":0.878,"15min":0.46800000000000003},"max":{"1min":1.857,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.89100000000000001,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.351,"5min":2.351,"15min":6.093},"last":1.329}]},{"osd":7,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4430000000000001,"15min":1.462},"min":{"1min":1.079,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.145,"5min":2.145,"15min":3.8809999999999998},"last":1.29},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.556,"15min":1.522},"min":{"1min":1.141,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.456,"5min":2.456,"15min":5.0209999999999999},"last":1.3939999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575175,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1928112,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92443728,"statfs":{"total":96636764160,"available":94662377472,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.3680000000000001},"min":{"1min":0.628,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.6579999999999999,"5min":1.73,"15min":17.315999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.395,"15min":1.427},"min":{"1min":0.753,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.7,"5min":1.7949999999999999,"15min":17.279},"last":1.5429999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.226,"15min":1.266},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.498},"max":{"1min":1.3720000000000001,"5min":1.839,"15min":6.1849999999999996},"last":0.997},{"interface":"front","average":{"1min":1.29,"5min":1.2230000000000001,"15min":1.1830000000000001},"min":{"1min":0.87,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":1.107}]},{"osd":2,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1990000000000001,"15min":1.1870000000000001},"min":{"1min":0.90500000000000003,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.6479999999999999,"5min":1.6830000000000001,"15min":2.1299999999999999},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.365,"15min":1.3460000000000001},"min":{"1min":1.0429999999999999,"5min":0.88600000000000001,"15min":0.75800000000000001},"max":{"1min":1.645,"5min":1.8120000000000001,"15min":2.4780000000000002},"last":1.288}]},{"osd":3,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.266,"15min":1.3069999999999999},"min":{"1min":0.96499999999999997,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.667,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3740000000000001,"15min":1.3560000000000001},"min":{"1min":1.056,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.77,"5min":1.77,"15min":4.7930000000000001},"last":1.488}]},{"osd":5,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.276,"15min":1.3049999999999999},"min":{"1min":0.98599999999999999,"5min":0.69999999999999996,"15min":0.63},"max":{"1min":1.54,"5min":1.8999999999999999,"15min":2.597},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.335,"15min":1.333},"min":{"1min":1.0069999999999999,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.7929999999999999,"5min":1.8660000000000001,"15min":4.8520000000000003},"last":1.347}]},{"osd":6,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":1.004,"5min":0.89700000000000002,"15min":0.57099999999999995},"max":{"1min":1.8420000000000001,"5min":2.0230000000000001,"15min":2.3479999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4810000000000001,"15min":1.52},"min":{"1min":1.0940000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7529999999999999,"5min":1.946,"15min":12.324999999999999},"last":1.6299999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4610000000000001,"15min":1.466},"min":{"1min":1.0860000000000001,"5min":1.032,"15min":0.53600000000000003},"max":{"1min":1.839,"5min":1.968,"15min":4.9539999999999997},"last":1.587},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.5649999999999999,"15min":1.534},"min":{"1min":1.296,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.903,"5min":1.9039999999999999,"15min":2.2599999999999998},"last":1.798}]}]},{"osd":3,"up_from":13,"seq":55834575176,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818796,"kb_used_data":1751588,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92553044,"statfs":{"total":96636764160,"available":94774317056,"internally_reserved":0,"allocated":1793626112,"data_stored":1792798405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94499999999999995,"15min":0.91600000000000004},"min":{"1min":0.34599999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.48,"5min":1.6240000000000001,"15min":1.925},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.097,"15min":1.03},"min":{"1min":0.45100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.377,"5min":6.6740000000000004,"15min":6.6740000000000004},"last":0.85199999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.93500000000000005,"15min":0.98599999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.44900000000000001},"max":{"1min":1.3140000000000001,"5min":1.6220000000000001,"15min":3.1150000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.105,"15min":1.077},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.38500000000000001},"max":{"1min":1.385,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":0.70199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1240000000000001,"15min":1.1319999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5609999999999999,"5min":1.8169999999999999,"15min":3.0670000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1919999999999999,"15min":1.204},"min":{"1min":0.55000000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.605,"5min":1.986,"15min":3.4279999999999999},"last":0.68700000000000006}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3540000000000001,"15min":1.343},"min":{"1min":0.84699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.5049999999999999},"last":1.514},{"interface":"front","average":{"1min":1.367,"5min":1.3859999999999999,"15min":1.391},"min":{"1min":1.0920000000000001,"5min":0.85499999999999998,"15min":0.64000000000000001},"max":{"1min":1.718,"5min":1.9810000000000001,"15min":3.8250000000000002},"last":1.163}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3480000000000001,"15min":1.3440000000000001},"min":{"1min":0.88300000000000001,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.6240000000000001,"5min":2.085,"15min":2.3540000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.365,"15min":1.3939999999999999},"min":{"1min":0.80500000000000005,"5min":0.79500000000000004,"15min":0.71299999999999997},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.907},"last":1.1850000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3939999999999999,"15min":1.4379999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.8560000000000001,"5min":2.0569999999999999,"15min":6.4480000000000004},"last":1.27},{"interface":"front","average":{"1min":1.454,"5min":1.4259999999999999,"15min":1.4770000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.81299999999999994},"max":{"1min":1.7869999999999999,"5min":2.1539999999999999,"15min":3.9039999999999999},"last":1.3879999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.73099999999999998},"max":{"1min":1.778,"5min":1.9370000000000001,"15min":3.7850000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.482,"15min":1.5169999999999999},"min":{"1min":1.0429999999999999,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.891,"5min":2.085,"15min":3.9590000000000001},"last":1.236}]}]},{"osd":2,"up_from":13,"seq":55834575175,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89700000000000002,"15min":0.97899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.314},"max":{"1min":1.3300000000000001,"5min":1.5569999999999999,"15min":24.155000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.0629999999999999,"15min":1.0449999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":1.76,"15min":2.952},"last":0.49099999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.0649999999999999,"15min":1.1180000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.4450000000000001,"5min":1.6970000000000001,"15min":16.776},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.1040000000000001,"15min":1.226},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.46500000000000002},"max":{"1min":1.468,"5min":1.6479999999999999,"15min":25.186},"last":0.436}]},{"osd":3,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0569999999999999,"15min":1.1919999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.3240000000000001,"5min":1.788,"15min":37.470999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.159,"5min":1.2270000000000001,"15min":1.244},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":6.5549999999999997},"last":0.67900000000000005}]},{"osd":4,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.349,"15min":1.5920000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":2.0019999999999998,"5min":2.0680000000000001,"15min":25.062000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.448},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.746},"max":{"1min":1.919,"5min":1.919,"15min":37.307000000000002},"last":0.75}]},{"osd":5,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":2.206,"5min":2.206,"15min":6.8849999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.4139999999999999,"15min":1.538},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.7649999999999999,"5min":1.8839999999999999,"15min":25.141999999999999},"last":0.74099999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3839999999999999,"15min":1.5269999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.81999999999999995},"max":{"1min":1.8300000000000001,"5min":2.0720000000000001,"15min":37.363},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.478,"5min":1.488,"15min":1.5149999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":6.9630000000000001},"last":1.022}]},{"osd":7,"last update":"Tue Jan 16 15:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.498,"15min":1.5860000000000001},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.66100000000000003},"max":{"1min":2.125,"5min":2.125,"15min":25.106999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.409,"5min":1.4730000000000001,"15min":1.599},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.78400000000000003},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":37.402999999999999},"last":0.86199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575176,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.92900000000000005},"min":{"1min":0.625,"5min":0.38900000000000001,"15min":0.36399999999999999},"max":{"1min":1.196,"5min":1.4370000000000001,"15min":14.106999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.478,"5min":1.4990000000000001,"15min":4.3719999999999999},"last":1.504}]},{"osd":2,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.29799999999999999},"max":{"1min":1.5920000000000001,"5min":3.5409999999999999,"15min":4.8789999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.052,"5min":1.085,"15min":1.149},"min":{"1min":0.628,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.484,"5min":1.663,"15min":14.055},"last":1.1719999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2150000000000001,"15min":1.3620000000000001},"min":{"1min":0.86399999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.51,"5min":1.6439999999999999,"15min":25.611000000000001},"last":1.403},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.60299999999999998,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":1.7090000000000001,"5min":1.952,"15min":4.6699999999999999},"last":1.107}]},{"osd":4,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.232,"15min":1.254},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.52500000000000002},"max":{"1min":1.472,"5min":3.77,"15min":4.423},"last":1.462},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.861,"5min":1.861,"15min":14.161},"last":1.355}]},{"osd":5,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.5509999999999999},"min":{"1min":0.84999999999999998,"5min":0.60099999999999998,"15min":0.58699999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4239999999999999,"15min":1.5189999999999999},"min":{"1min":1.093,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.9359999999999999,"5min":3.8090000000000002,"15min":16.433},"last":1.5669999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.423,"15min":1.496},"min":{"1min":0.93600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":14.212},"last":1.95},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.462},"min":{"1min":1.0449999999999999,"5min":0.92800000000000005,"15min":0.63},"max":{"1min":2.0270000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":1.238}]},{"osd":7,"last update":"Tue Jan 16 15:22:27 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4419999999999999,"15min":1.591},"min":{"1min":1.2509999999999999,"5min":0.82499999999999996,"15min":0.75700000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":19.747},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.456,"15min":1.46},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.64100000000000001},"max":{"1min":1.7869999999999999,"5min":3.726,"15min":6.0229999999999997},"last":1.6040000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575175,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048744,"kb_used_data":1981536,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323096,"statfs":{"total":96636764160,"available":94538850304,"internally_reserved":0,"allocated":2029092864,"data_stored":2028268228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.83499999999999996,"15min":0.84199999999999997},"min":{"1min":0.621,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.3240000000000001,"5min":1.601,"15min":1.921},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0449999999999999,"15min":1.01},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.41799999999999998},"max":{"1min":1.653,"5min":1.72,"15min":4.4059999999999997},"last":0.88200000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0109999999999999,"15min":1.0649999999999999},"min":{"1min":0.80200000000000005,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":1.6220000000000001,"15min":17.260000000000002},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.115,"15min":1.0980000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.492,"5min":1.784,"15min":4.492},"last":1.21}]},{"osd":3,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1180000000000001,"15min":1.087},"min":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.538,"5min":1.706,"15min":3.4670000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.325,"15min":1.2869999999999999},"min":{"1min":0.83799999999999997,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.5840000000000001,"5min":1.8400000000000001,"15min":2.319},"last":1.5089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.337,"15min":1.3460000000000001},"min":{"1min":1.0860000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":4.2889999999999997},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3360000000000001,"15min":1.3100000000000001},"min":{"1min":0.93000000000000005,"5min":0.83799999999999997,"15min":0.67800000000000005},"max":{"1min":4.0810000000000004,"5min":4.0810000000000004,"15min":4.0810000000000004},"last":1.5529999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4259999999999999,"15min":1.4159999999999999},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.624},"max":{"1min":2.1850000000000001,"5min":2.222,"15min":3.1259999999999999},"last":1.353},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4950000000000001,"15min":1.486},"min":{"1min":1.0129999999999999,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.1389999999999998,"5min":2.2730000000000001,"15min":6.5430000000000001},"last":1.627}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.397,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":11.462},"last":1.468},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4419999999999999,"15min":1.4570000000000001},"min":{"1min":1.0189999999999999,"5min":0.92600000000000005,"15min":0.80100000000000005},"max":{"1min":1.992,"5min":2.093,"15min":7.1849999999999996},"last":1.593}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.466,"15min":1.502},"min":{"1min":0.86799999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.9399999999999999,"5min":2.1299999999999999,"15min":5.2450000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.514,"15min":1.4750000000000001},"min":{"1min":1.123,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.2729999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":1221216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1232896,"data_stored":1221174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:33.783 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.783 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.784 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.785 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.786 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.787 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.788 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.789 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.790 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.791 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.792 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 16.10 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.793 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.794 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 16.9 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.795 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.796 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.797 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.798 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.799 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.799 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-16T14:56:34.224878+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=14, tm_min=56, tm_sec=34, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:33.799 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:23:45.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:45.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:53.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:54.020 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.024+0000 7f7e3ea00640 1 Processor -- start 2024-01-16T15:23:54.021 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.024+0000 7f7e3ea00640 1 -- start start 2024-01-16T15:23:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e3ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 0x7f7e380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e3ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e3ea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e380cfc30 con 0x7f7e3815e1e0 2024-01-16T15:23:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e3ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e380b1ee0 con 0x7f7e38143ae0 2024-01-16T15:23:54.022 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e3ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380b2780 con 0x7f7e380cf7d0 2024-01-16T15:23:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e37e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 0x7f7e380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e2fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e2fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.125:43454/0 (socket says 172.21.15.125:43454) 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.026+0000 7f7e2fe00640 1 -- 172.21.15.125:0/1225010899 learned_addr learned my addr 172.21.15.125:0/1225010899 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.027+0000 7f7e2fe00640 1 -- 172.21.15.125:0/1225010899 >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 msgr2=0x7f7e380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.027+0000 7f7e2fe00640 1 --2- 172.21.15.125:0/1225010899 >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 0x7f7e380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.027+0000 7f7e2fe00640 1 -- 172.21.15.125:0/1225010899 >> v1:172.21.15.137:6789/0 conn(0x7f7e3815e1e0 legacy=0x7f7e3815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.027+0000 7f7e2fe00640 1 -- 172.21.15.125:0/1225010899 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3815e990 con 0x7f7e380cf7d0 2024-01-16T15:23:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.027+0000 7f7e37e00640 1 --2- 172.21.15.125:0/1225010899 >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 0x7f7e380cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:23:54.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.028+0000 7f7e2fe00640 1 --2- 172.21.15.125:0/1225010899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e2002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e3c3d1590d40a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.025 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.029+0000 7f7e37400640 1 -- 172.21.15.125:0/1225010899 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20030840 con 0x7f7e380cf7d0 2024-01-16T15:23:54.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.029+0000 7f7e37400640 1 -- 172.21.15.125:0/1225010899 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e200309a0 con 0x7f7e380cf7d0 2024-01-16T15:23:54.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.029+0000 7f7e37400640 1 -- 172.21.15.125:0/1225010899 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20030d30 con 0x7f7e380cf7d0 2024-01-16T15:23:54.026 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 -- 172.21.15.125:0/1225010899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 msgr2=0x7f7e3815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/1225010899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e2002b980 comp rx=0 tx=0).stop 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 -- 172.21.15.125:0/1225010899 shutdown_connections 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/1225010899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e3815dca0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/1225010899 >> v2:172.21.15.137:3300/0 conn(0x7f7e38143ae0 0x7f7e380cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 -- 172.21.15.125:0/1225010899 >> 172.21.15.125:0/1225010899 conn(0x7f7e3806a800 msgr2=0x7f7e3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 -- 172.21.15.125:0/1225010899 shutdown_connections 2024-01-16T15:23:54.027 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.030+0000 7f7e3ea00640 1 -- 172.21.15.125:0/1225010899 wait complete. 2024-01-16T15:23:54.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.032+0000 7f7e3ea00640 1 Processor -- start 2024-01-16T15:23:54.028 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.032+0000 7f7e3ea00640 1 -- start start 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e380aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e38143ae0 0x7f7e38159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3813a430 con 0x7f7e3815e1e0 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e38139a90 con 0x7f7e38143ae0 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e38139d70 con 0x7f7e380cf7d0 2024-01-16T15:23:54.031 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ca00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47376/0 (socket says 172.21.15.125:47376) 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.033+0000 7f7e3ca00640 1 -- 172.21.15.125:0/3119966887 learned_addr learned my addr 172.21.15.125:0/3119966887 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e2fe00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e38143ae0 0x7f7e38159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e37e00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e380aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 -- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 msgr2=0x7f7e380aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e380aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 -- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e38143ae0 msgr2=0x7f7e38159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.032 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e38143ae0 0x7f7e38159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 -- 172.21.15.125:0/3119966887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e28009400 con 0x7f7e3815e1e0 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.034+0000 7f7e3ca00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c6b0 tx=0x7f7e3000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.035+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30003b70 con 0x7f7e3815e1e0 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.035+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e30003cf0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.035+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e200094d0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.033 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.035+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30003ea0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.034 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.035+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e381396b0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.035 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.036+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e38139bd0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.037+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7e30005310 con 0x7f7e3815e1e0 2024-01-16T15:23:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.038+0000 7f7e34c00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 0x7f7e1006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.038+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e30088850 con 0x7f7e3815e1e0 2024-01-16T15:23:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.038+0000 7f7e37e00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 0x7f7e1006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.039+0000 7f7e37e00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 0x7f7e1006ea10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009cd0 tx=0x7f7e28009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.039 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.043+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e38139bd0 con 0x7f7e3815e1e0 2024-01-16T15:23:54.229 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.233+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e38071150 con 0x7f7e1006c550 2024-01-16T15:23:54.252 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.255+0000 7f7e34c00640 1 -- 172.21.15.125:0/3119966887 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288626 (secure 0 0 0) 0x7f7e38071150 con 0x7f7e1006c550 2024-01-16T15:23:54.256 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:23:54.258 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 msgr2=0x7f7e1006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 0x7f7e1006ea10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009cd0 tx=0x7f7e28009190 comp rx=0 tx=0).stop 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 msgr2=0x7f7e38163690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c6b0 tx=0x7f7e3000cb70 comp rx=0 tx=0).stop 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 shutdown_connections 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f7e1006c550 0x7f7e1006ea10 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e3815e1e0 0x7f7e38163690 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.260 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e38143ae0 0x7f7e38159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 --2- 172.21.15.125:0/3119966887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e380cf7d0 0x7f7e380aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.262+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 >> 172.21.15.125:0/3119966887 conn(0x7f7e3806a800 msgr2=0x7f7e3806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.263+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 shutdown_connections 2024-01-16T15:23:54.261 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:23:54.263+0000 7f7e3ea00640 1 -- 172.21.15.125:0/3119966887 wait complete. 2024-01-16T15:23:54.277 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":925,"stamp":"2024-01-16T15:23:52.549723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444268918,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2250,"num_write_kb":6295341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5260,"ondisk_log_size":5260,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13139388,"kb_used_data":12597564,"kb_used_omap":12,"kb_used_meta":541747,"kb_avail":741835332,"statfs":{"total":773094113280,"available":759639379968,"internally_reserved":0,"allocated":12899905536,"data_stored":12893402313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":554749488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004941"},"pg_stats":[{"pgid":"17.3f","version":"113'70","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.309297+0000","last_change":"2024-01-16T15:15:07.594505+0000","last_active":"2024-01-16T15:23:28.309297+0000","last_peered":"2024-01-16T15:23:28.309297+0000","last_clean":"2024-01-16T15:23:28.309297+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:23:28.309297+0000","last_undegraded":"2024-01-16T15:23:28.309297+0000","last_fullsized":"2024-01-16T15:23:28.309297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'91","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.213438+0000","last_change":"2024-01-16T15:23:42.213497+0000","last_active":"2024-01-16T15:23:42.213438+0000","last_peered":"2024-01-16T15:23:42.213438+0000","last_clean":"2024-01-16T15:23:42.213438+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:42.213438+0000","last_undegraded":"2024-01-16T15:23:42.213438+0000","last_fullsized":"2024-01-16T15:23:42.213438+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'91","last_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_deep_scrub":"113'91","last_deep_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.213253+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:45.051080+0000","scrub_duration":0.29806467199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'67","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.398386+0000","last_change":"2024-01-16T15:15:07.581381+0000","last_active":"2024-01-16T15:23:29.398386+0000","last_peered":"2024-01-16T15:23:29.398386+0000","last_clean":"2024-01-16T15:23:29.398386+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:29.398386+0000","last_undegraded":"2024-01-16T15:23:29.398386+0000","last_fullsized":"2024-01-16T15:23:29.398386+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458250+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458250+0000","last_peered":"2024-01-16T15:23:18.458250+0000","last_clean":"2024-01-16T15:23:18.458250+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458250+0000","last_undegraded":"2024-01-16T15:23:18.458250+0000","last_fullsized":"2024-01-16T15:23:18.458250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'60","reported_seq":87,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.440467+0000","last_change":"2024-01-16T15:15:07.585027+0000","last_active":"2024-01-16T15:23:18.440467+0000","last_peered":"2024-01-16T15:23:18.440467+0000","last_clean":"2024-01-16T15:23:18.440467+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:18.440467+0000","last_undegraded":"2024-01-16T15:23:18.440467+0000","last_fullsized":"2024-01-16T15:23:18.440467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'100","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.880459+0000","last_change":"2024-01-16T15:23:43.880600+0000","last_active":"2024-01-16T15:23:43.880459+0000","last_peered":"2024-01-16T15:23:43.880459+0000","last_clean":"2024-01-16T15:23:43.880459+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:43.880459+0000","last_undegraded":"2024-01-16T15:23:43.880459+0000","last_fullsized":"2024-01-16T15:23:43.880459+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.880280+0000","objects_scrubbed":31,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:51.072560+0000","scrub_duration":0.388654414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'66","reported_seq":88,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446808+0000","last_change":"2024-01-16T15:15:07.588043+0000","last_active":"2024-01-16T15:23:18.446808+0000","last_peered":"2024-01-16T15:23:18.446808+0000","last_clean":"2024-01-16T15:23:18.446808+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:23:18.446808+0000","last_undegraded":"2024-01-16T15:23:18.446808+0000","last_fullsized":"2024-01-16T15:23:18.446808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'71","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.992771+0000","last_change":"2024-01-16T15:23:39.992859+0000","last_active":"2024-01-16T15:23:39.992771+0000","last_peered":"2024-01-16T15:23:39.992771+0000","last_clean":"2024-01-16T15:23:39.992771+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:39.992771+0000","last_undegraded":"2024-01-16T15:23:39.992771+0000","last_fullsized":"2024-01-16T15:23:39.992771+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_deep_scrub":"113'71","last_deep_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.992591+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:54.989201+0000","scrub_duration":0.23396198900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'100","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.211670+0000","last_change":"2024-01-16T15:23:33.211722+0000","last_active":"2024-01-16T15:23:33.211670+0000","last_peered":"2024-01-16T15:23:33.211670+0000","last_clean":"2024-01-16T15:23:33.211670+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:33.211670+0000","last_undegraded":"2024-01-16T15:23:33.211670+0000","last_fullsized":"2024-01-16T15:23:33.211670+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.211507+0000","objects_scrubbed":27,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:47.572037+0000","scrub_duration":0.235911915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'66","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448403+0000","last_change":"2024-01-16T15:15:07.592509+0000","last_active":"2024-01-16T15:23:18.448403+0000","last_peered":"2024-01-16T15:23:18.448403+0000","last_clean":"2024-01-16T15:23:18.448403+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:23:18.448403+0000","last_undegraded":"2024-01-16T15:23:18.448403+0000","last_fullsized":"2024-01-16T15:23:18.448403+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447016+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.447016+0000","last_peered":"2024-01-16T15:23:18.447016+0000","last_clean":"2024-01-16T15:23:18.447016+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.447016+0000","last_undegraded":"2024-01-16T15:23:18.447016+0000","last_fullsized":"2024-01-16T15:23:18.447016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449095+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.449095+0000","last_peered":"2024-01-16T15:23:18.449095+0000","last_clean":"2024-01-16T15:23:18.449095+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.449095+0000","last_undegraded":"2024-01-16T15:23:18.449095+0000","last_fullsized":"2024-01-16T15:23:18.449095+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'78","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.086907+0000","last_change":"2024-01-16T15:23:48.087015+0000","last_active":"2024-01-16T15:23:48.086907+0000","last_peered":"2024-01-16T15:23:48.086907+0000","last_clean":"2024-01-16T15:23:48.086907+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:48.086907+0000","last_undegraded":"2024-01-16T15:23:48.086907+0000","last_fullsized":"2024-01-16T15:23:48.086907+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.086739+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:49.934440+0000","scrub_duration":0.26635034800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'88","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.116864+0000","last_change":"2024-01-16T15:23:44.116922+0000","last_active":"2024-01-16T15:23:44.116864+0000","last_peered":"2024-01-16T15:23:44.116864+0000","last_clean":"2024-01-16T15:23:44.116864+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:44.116864+0000","last_undegraded":"2024-01-16T15:23:44.116864+0000","last_fullsized":"2024-01-16T15:23:44.116864+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.116705+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:20.435799+0000","scrub_duration":0.35551533600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'64","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.593097+0000","last_change":"2024-01-16T15:23:33.593181+0000","last_active":"2024-01-16T15:23:33.593097+0000","last_peered":"2024-01-16T15:23:33.593097+0000","last_clean":"2024-01-16T15:23:33.593097+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:33.593097+0000","last_undegraded":"2024-01-16T15:23:33.593097+0000","last_fullsized":"2024-01-16T15:23:33.593097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'64","last_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_deep_scrub":"113'64","last_deep_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.592968+0000","objects_scrubbed":14,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:45.001245+0000","scrub_duration":0.13121580099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'58","reported_seq":77,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458944+0000","last_change":"2024-01-16T15:15:28.509366+0000","last_active":"2024-01-16T15:23:18.458944+0000","last_peered":"2024-01-16T15:23:18.458944+0000","last_clean":"2024-01-16T15:23:18.458944+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:23:18.458944+0000","last_undegraded":"2024-01-16T15:23:18.458944+0000","last_fullsized":"2024-01-16T15:23:18.458944+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'66","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.091684+0000","last_change":"2024-01-16T15:23:49.091768+0000","last_active":"2024-01-16T15:23:49.091684+0000","last_peered":"2024-01-16T15:23:49.091684+0000","last_clean":"2024-01-16T15:23:49.091684+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:49.091684+0000","last_undegraded":"2024-01-16T15:23:49.091684+0000","last_fullsized":"2024-01-16T15:23:49.091684+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'66","last_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_deep_scrub":"113'66","last_deep_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.091510+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:04.088120+0000","scrub_duration":0.232593777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'72","reported_seq":114,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455373+0000","last_change":"2024-01-16T15:15:07.591828+0000","last_active":"2024-01-16T15:23:18.455373+0000","last_peered":"2024-01-16T15:23:18.455373+0000","last_clean":"2024-01-16T15:23:18.455373+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:18.455373+0000","last_undegraded":"2024-01-16T15:23:18.455373+0000","last_fullsized":"2024-01-16T15:23:18.455373+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'71","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.460363+0000","last_change":"2024-01-16T15:15:07.577895+0000","last_active":"2024-01-16T15:23:18.460363+0000","last_peered":"2024-01-16T15:23:18.460363+0000","last_clean":"2024-01-16T15:23:18.460363+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:18.460363+0000","last_undegraded":"2024-01-16T15:23:18.460363+0000","last_fullsized":"2024-01-16T15:23:18.460363+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'87","reported_seq":124,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.217923+0000","last_change":"2024-01-16T15:23:43.218013+0000","last_active":"2024-01-16T15:23:43.217923+0000","last_peered":"2024-01-16T15:23:43.217923+0000","last_clean":"2024-01-16T15:23:43.217923+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:43.217923+0000","last_undegraded":"2024-01-16T15:23:43.217923+0000","last_fullsized":"2024-01-16T15:23:43.217923+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'87","last_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_deep_scrub":"113'87","last_deep_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.217792+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.208703+0000","scrub_duration":0.29447676900000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'103","reported_seq":151,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.212264+0000","last_change":"2024-01-16T15:23:48.212338+0000","last_active":"2024-01-16T15:23:48.212264+0000","last_peered":"2024-01-16T15:23:48.212264+0000","last_clean":"2024-01-16T15:23:48.212264+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:48.212264+0000","last_undegraded":"2024-01-16T15:23:48.212264+0000","last_fullsized":"2024-01-16T15:23:48.212264+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'103","last_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_deep_scrub":"113'103","last_deep_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.212080+0000","objects_scrubbed":30,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:31.387018+0000","scrub_duration":0.36970662999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'64","reported_seq":88,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453877+0000","last_change":"2024-01-16T15:15:07.595387+0000","last_active":"2024-01-16T15:23:18.453877+0000","last_peered":"2024-01-16T15:23:18.453877+0000","last_clean":"2024-01-16T15:23:18.453877+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:18.453877+0000","last_undegraded":"2024-01-16T15:23:18.453877+0000","last_fullsized":"2024-01-16T15:23:18.453877+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'79","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.249793+0000","last_change":"2024-01-16T15:23:37.249926+0000","last_active":"2024-01-16T15:23:37.249793+0000","last_peered":"2024-01-16T15:23:37.249793+0000","last_clean":"2024-01-16T15:23:37.249793+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:37.249793+0000","last_undegraded":"2024-01-16T15:23:37.249793+0000","last_fullsized":"2024-01-16T15:23:37.249793+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.249557+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:23.718419+0000","scrub_duration":0.30745207200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'90","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.124563+0000","last_change":"2024-01-16T15:23:47.124563+0000","last_active":"2024-01-16T15:23:47.124563+0000","last_peered":"2024-01-16T15:23:47.124563+0000","last_clean":"2024-01-16T15:23:47.124563+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:47.124563+0000","last_undegraded":"2024-01-16T15:23:47.124563+0000","last_fullsized":"2024-01-16T15:23:47.124563+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'90","last_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_deep_scrub":"113'90","last_deep_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.124399+0000","objects_scrubbed":24,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:02.121008+0000","scrub_duration":0.271690178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'96","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.160049+0000","last_change":"2024-01-16T15:23:47.160049+0000","last_active":"2024-01-16T15:23:47.160049+0000","last_peered":"2024-01-16T15:23:47.160049+0000","last_clean":"2024-01-16T15:23:47.160049+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:47.160049+0000","last_undegraded":"2024-01-16T15:23:47.160049+0000","last_fullsized":"2024-01-16T15:23:47.160049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.159896+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:22.125663+0000","scrub_duration":0.29641904899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'81","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308973+0000","last_change":"2024-01-16T15:15:07.593145+0000","last_active":"2024-01-16T15:23:28.308973+0000","last_peered":"2024-01-16T15:23:28.308973+0000","last_clean":"2024-01-16T15:23:28.308973+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:23:28.308973+0000","last_undegraded":"2024-01-16T15:23:28.308973+0000","last_fullsized":"2024-01-16T15:23:28.308973+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'86","reported_seq":122,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.783319+0000","last_change":"2024-01-16T15:23:44.783319+0000","last_active":"2024-01-16T15:23:44.783319+0000","last_peered":"2024-01-16T15:23:44.783319+0000","last_clean":"2024-01-16T15:23:44.783319+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:44.783319+0000","last_undegraded":"2024-01-16T15:23:44.783319+0000","last_fullsized":"2024-01-16T15:23:44.783319+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'86","last_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_deep_scrub":"113'86","last_deep_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.783151+0000","objects_scrubbed":24,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.138136+0000","scrub_duration":0.29777104900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'43","reported_seq":66,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.400945+0000","last_change":"2024-01-16T15:15:07.601384+0000","last_active":"2024-01-16T15:23:30.400945+0000","last_peered":"2024-01-16T15:23:30.400945+0000","last_clean":"2024-01-16T15:23:30.400945+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:23:30.400945+0000","last_undegraded":"2024-01-16T15:23:30.400945+0000","last_fullsized":"2024-01-16T15:23:30.400945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":88,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.592883+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:29.592883+0000","last_peered":"2024-01-16T15:23:29.592883+0000","last_clean":"2024-01-16T15:23:29.592883+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:29.592883+0000","last_undegraded":"2024-01-16T15:23:29.592883+0000","last_fullsized":"2024-01-16T15:23:29.592883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'76","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.463312+0000","last_change":"2024-01-16T15:15:07.593782+0000","last_active":"2024-01-16T15:23:29.463312+0000","last_peered":"2024-01-16T15:23:29.463312+0000","last_clean":"2024-01-16T15:23:29.463312+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:23:29.463312+0000","last_undegraded":"2024-01-16T15:23:29.463312+0000","last_fullsized":"2024-01-16T15:23:29.463312+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'106","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.282980+0000","last_change":"2024-01-16T15:23:34.283133+0000","last_active":"2024-01-16T15:23:34.282980+0000","last_peered":"2024-01-16T15:23:34.282980+0000","last_clean":"2024-01-16T15:23:34.282980+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:34.282980+0000","last_undegraded":"2024-01-16T15:23:34.282980+0000","last_fullsized":"2024-01-16T15:23:34.282980+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'106","last_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_deep_scrub":"113'106","last_deep_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.282804+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:08.354546+0000","scrub_duration":0.30496924800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'82","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.785613+0000","last_change":"2024-01-16T15:23:42.785799+0000","last_active":"2024-01-16T15:23:42.785613+0000","last_peered":"2024-01-16T15:23:42.785613+0000","last_clean":"2024-01-16T15:23:42.785613+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:42.785613+0000","last_undegraded":"2024-01-16T15:23:42.785613+0000","last_fullsized":"2024-01-16T15:23:42.785613+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'82","last_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_deep_scrub":"113'82","last_deep_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.785476+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:05.117297+0000","scrub_duration":0.269243968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'62","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.446245+0000","last_change":"2024-01-16T15:15:07.588100+0000","last_active":"2024-01-16T15:23:18.446245+0000","last_peered":"2024-01-16T15:23:18.446245+0000","last_clean":"2024-01-16T15:23:18.446245+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:18.446245+0000","last_undegraded":"2024-01-16T15:23:18.446245+0000","last_fullsized":"2024-01-16T15:23:18.446245+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'4","reported_seq":23,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.463425+0000","last_change":"2024-01-16T15:15:06.560295+0000","last_active":"2024-01-16T15:23:29.463425+0000","last_peered":"2024-01-16T15:23:29.463425+0000","last_clean":"2024-01-16T15:23:29.463425+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:23:29.463425+0000","last_undegraded":"2024-01-16T15:23:29.463425+0000","last_fullsized":"2024-01-16T15:23:29.463425+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'73","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.352331+0000","last_change":"2024-01-16T15:23:49.352425+0000","last_active":"2024-01-16T15:23:49.352331+0000","last_peered":"2024-01-16T15:23:49.352331+0000","last_clean":"2024-01-16T15:23:49.352331+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:49.352331+0000","last_undegraded":"2024-01-16T15:23:49.352331+0000","last_fullsized":"2024-01-16T15:23:49.352331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'73","last_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_deep_scrub":"113'73","last_deep_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.352155+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:06:37.591834+0000","scrub_duration":0.23652007799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.752037+0000","last_change":"2024-01-16T15:23:36.752093+0000","last_active":"2024-01-16T15:23:36.752037+0000","last_peered":"2024-01-16T15:23:36.752037+0000","last_clean":"2024-01-16T15:23:36.752037+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:23:36.752037+0000","last_undegraded":"2024-01-16T15:23:36.752037+0000","last_fullsized":"2024-01-16T15:23:36.752037+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.751793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:04.897513+0000","scrub_duration":0.0034892019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'96","reported_seq":140,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.838614+0000","last_change":"2024-01-16T15:23:47.838729+0000","last_active":"2024-01-16T15:23:47.838614+0000","last_peered":"2024-01-16T15:23:47.838614+0000","last_clean":"2024-01-16T15:23:47.838614+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:47.838614+0000","last_undegraded":"2024-01-16T15:23:47.838614+0000","last_fullsized":"2024-01-16T15:23:47.838614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.838456+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:01.112625+0000","scrub_duration":0.318651024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.965524+0000","last_change":"2024-01-16T15:23:33.965582+0000","last_active":"2024-01-16T15:23:33.965524+0000","last_peered":"2024-01-16T15:23:33.965524+0000","last_clean":"2024-01-16T15:23:33.965524+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:23:33.965524+0000","last_undegraded":"2024-01-16T15:23:33.965524+0000","last_fullsized":"2024-01-16T15:23:33.965524+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.965351+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:58.186239+0000","scrub_duration":0.0074428489999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'68","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.461121+0000","last_change":"2024-01-16T15:15:07.581293+0000","last_active":"2024-01-16T15:23:18.461121+0000","last_peered":"2024-01-16T15:23:18.461121+0000","last_clean":"2024-01-16T15:23:18.461121+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:23:18.461121+0000","last_undegraded":"2024-01-16T15:23:18.461121+0000","last_fullsized":"2024-01-16T15:23:18.461121+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'3","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.098394+0000","last_change":"2024-01-16T15:23:49.098456+0000","last_active":"2024-01-16T15:23:49.098394+0000","last_peered":"2024-01-16T15:23:49.098394+0000","last_clean":"2024-01-16T15:23:49.098394+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:23:49.098394+0000","last_undegraded":"2024-01-16T15:23:49.098394+0000","last_fullsized":"2024-01-16T15:23:49.098394+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.098224+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:34.500097+0000","scrub_duration":0.0069822110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'72","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458004+0000","last_change":"2024-01-16T15:15:07.597705+0000","last_active":"2024-01-16T15:23:18.458004+0000","last_peered":"2024-01-16T15:23:18.458004+0000","last_clean":"2024-01-16T15:23:18.458004+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:23:18.458004+0000","last_undegraded":"2024-01-16T15:23:18.458004+0000","last_fullsized":"2024-01-16T15:23:18.458004+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:39.454494+0000","last_change":"2024-01-16T15:15:06.566848+0000","last_active":"2024-01-16T15:22:39.454494+0000","last_peered":"2024-01-16T15:22:39.454494+0000","last_clean":"2024-01-16T15:22:39.454494+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:22:39.454494+0000","last_undegraded":"2024-01-16T15:22:39.454494+0000","last_fullsized":"2024-01-16T15:22:39.454494+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'72","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454232+0000","last_change":"2024-01-16T15:15:07.591653+0000","last_active":"2024-01-16T15:23:18.454232+0000","last_peered":"2024-01-16T15:23:18.454232+0000","last_clean":"2024-01-16T15:23:18.454232+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:23:18.454232+0000","last_undegraded":"2024-01-16T15:23:18.454232+0000","last_fullsized":"2024-01-16T15:23:18.454232+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.864743+0000","last_change":"2024-01-16T15:23:31.864779+0000","last_active":"2024-01-16T15:23:31.864743+0000","last_peered":"2024-01-16T15:23:31.864743+0000","last_clean":"2024-01-16T15:23:31.864743+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:23:31.864743+0000","last_undegraded":"2024-01-16T15:23:31.864743+0000","last_fullsized":"2024-01-16T15:23:31.864743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.864652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:25.530956+0000","scrub_duration":0.00281139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'73","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.450054+0000","last_change":"2024-01-16T15:15:07.595810+0000","last_active":"2024-01-16T15:23:18.450054+0000","last_peered":"2024-01-16T15:23:18.450054+0000","last_clean":"2024-01-16T15:23:18.450054+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:23:18.450054+0000","last_undegraded":"2024-01-16T15:23:18.450054+0000","last_fullsized":"2024-01-16T15:23:18.450054+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.896191+0000","last_change":"2024-01-16T15:23:35.896238+0000","last_active":"2024-01-16T15:23:35.896191+0000","last_peered":"2024-01-16T15:23:35.896191+0000","last_clean":"2024-01-16T15:23:35.896191+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:23:35.896191+0000","last_undegraded":"2024-01-16T15:23:35.896191+0000","last_fullsized":"2024-01-16T15:23:35.896191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.896004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:41.978043+0000","scrub_duration":0.003846173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'73","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.445725+0000","last_change":"2024-01-16T15:15:07.594243+0000","last_active":"2024-01-16T15:23:18.445725+0000","last_peered":"2024-01-16T15:23:18.445725+0000","last_clean":"2024-01-16T15:23:18.445725+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:23:18.445725+0000","last_undegraded":"2024-01-16T15:23:18.445725+0000","last_fullsized":"2024-01-16T15:23:18.445725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.816419+0000","last_change":"2024-01-16T15:15:06.558856+0000","last_active":"2024-01-16T15:22:24.816419+0000","last_peered":"2024-01-16T15:22:24.816419+0000","last_clean":"2024-01-16T15:22:24.816419+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:22:24.816419+0000","last_undegraded":"2024-01-16T15:22:24.816419+0000","last_fullsized":"2024-01-16T15:22:24.816419+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'58","reported_seq":80,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448782+0000","last_change":"2024-01-16T15:15:07.586283+0000","last_active":"2024-01-16T15:23:18.448782+0000","last_peered":"2024-01-16T15:23:18.448782+0000","last_clean":"2024-01-16T15:23:18.448782+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:23:18.448782+0000","last_undegraded":"2024-01-16T15:23:18.448782+0000","last_fullsized":"2024-01-16T15:23:18.448782+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.081149+0000","last_change":"2024-01-16T15:23:42.081324+0000","last_active":"2024-01-16T15:23:42.081149+0000","last_peered":"2024-01-16T15:23:42.081149+0000","last_clean":"2024-01-16T15:23:42.081149+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:23:42.081149+0000","last_undegraded":"2024-01-16T15:23:42.081149+0000","last_fullsized":"2024-01-16T15:23:42.081149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.080895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:49.273175+0000","scrub_duration":0.0037055339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'78","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.281779+0000","last_change":"2024-01-16T15:23:47.281905+0000","last_active":"2024-01-16T15:23:47.281779+0000","last_peered":"2024-01-16T15:23:47.281779+0000","last_clean":"2024-01-16T15:23:47.281779+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:47.281779+0000","last_undegraded":"2024-01-16T15:23:47.281779+0000","last_fullsized":"2024-01-16T15:23:47.281779+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.281611+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:50.119438+0000","scrub_duration":0.19022307299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.832184+0000","last_change":"2024-01-16T15:23:35.832238+0000","last_active":"2024-01-16T15:23:35.832184+0000","last_peered":"2024-01-16T15:23:35.832184+0000","last_clean":"2024-01-16T15:23:35.832184+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:23:35.832184+0000","last_undegraded":"2024-01-16T15:23:35.832184+0000","last_fullsized":"2024-01-16T15:23:35.832184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.831980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:16.763641+0000","scrub_duration":0.004520446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'62","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.454309+0000","last_change":"2024-01-16T15:15:07.590407+0000","last_active":"2024-01-16T15:23:18.454309+0000","last_peered":"2024-01-16T15:23:18.454309+0000","last_clean":"2024-01-16T15:23:18.454309+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:23:18.454309+0000","last_undegraded":"2024-01-16T15:23:18.454309+0000","last_fullsized":"2024-01-16T15:23:18.454309+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'31","reported_seq":49,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.816519+0000","last_change":"2024-01-16T15:15:06.564172+0000","last_active":"2024-01-16T15:22:24.816519+0000","last_peered":"2024-01-16T15:22:24.816519+0000","last_clean":"2024-01-16T15:22:24.816519+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:22:24.816519+0000","last_undegraded":"2024-01-16T15:22:24.816519+0000","last_fullsized":"2024-01-16T15:22:24.816519+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'69","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453608+0000","last_change":"2024-01-16T15:15:07.585396+0000","last_active":"2024-01-16T15:23:18.453608+0000","last_peered":"2024-01-16T15:23:18.453608+0000","last_clean":"2024-01-16T15:23:18.453608+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:23:18.453608+0000","last_undegraded":"2024-01-16T15:23:18.453608+0000","last_fullsized":"2024-01-16T15:23:18.453608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'323","reported_seq":349,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.951958+0000","last_change":"2024-01-16T15:23:34.952024+0000","last_active":"2024-01-16T15:23:34.951958+0000","last_peered":"2024-01-16T15:23:34.951958+0000","last_clean":"2024-01-16T15:23:34.951958+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:34.951958+0000","last_undegraded":"2024-01-16T15:23:34.951958+0000","last_fullsized":"2024-01-16T15:23:34.951958+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'323","last_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_deep_scrub":"113'323","last_deep_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.951788+0000","objects_scrubbed":4,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:08.012160+0000","scrub_duration":0.019282695999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1221104,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'54","reported_seq":80,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.442859+0000","last_change":"2024-01-16T15:15:07.586356+0000","last_active":"2024-01-16T15:23:18.442859+0000","last_peered":"2024-01-16T15:23:18.442859+0000","last_clean":"2024-01-16T15:23:18.442859+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:18.442859+0000","last_undegraded":"2024-01-16T15:23:18.442859+0000","last_fullsized":"2024-01-16T15:23:18.442859+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:32.491378+0000","last_change":"2024-01-16T15:23:32.491489+0000","last_active":"2024-01-16T15:23:32.491378+0000","last_peered":"2024-01-16T15:23:32.491378+0000","last_clean":"2024-01-16T15:23:32.491378+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:23:32.491378+0000","last_undegraded":"2024-01-16T15:23:32.491378+0000","last_fullsized":"2024-01-16T15:23:32.491378+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_clean_scrub_stamp":"2024-01-16T15:23:32.491197+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:41.739634+0000","scrub_duration":0.0079106109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'72","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.453159+0000","last_change":"2024-01-16T15:15:07.586149+0000","last_active":"2024-01-16T15:23:18.453159+0000","last_peered":"2024-01-16T15:23:18.453159+0000","last_clean":"2024-01-16T15:23:18.453159+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:18.453159+0000","last_undegraded":"2024-01-16T15:23:18.453159+0000","last_fullsized":"2024-01-16T15:23:18.453159+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.223333+0000","last_change":"2024-01-16T15:23:34.223383+0000","last_active":"2024-01-16T15:23:34.223333+0000","last_peered":"2024-01-16T15:23:34.223333+0000","last_clean":"2024-01-16T15:23:34.223333+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:23:34.223333+0000","last_undegraded":"2024-01-16T15:23:34.223333+0000","last_fullsized":"2024-01-16T15:23:34.223333+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.223138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:21.983148+0000","scrub_duration":0.003763078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'63","reported_seq":90,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.456395+0000","last_change":"2024-01-16T15:15:07.587887+0000","last_active":"2024-01-16T15:23:18.456395+0000","last_peered":"2024-01-16T15:23:18.456395+0000","last_clean":"2024-01-16T15:23:18.456395+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:18.456395+0000","last_undegraded":"2024-01-16T15:23:18.456395+0000","last_fullsized":"2024-01-16T15:23:18.456395+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'12","reported_seq":38,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.078941+0000","last_change":"2024-01-16T15:23:45.079051+0000","last_active":"2024-01-16T15:23:45.078941+0000","last_peered":"2024-01-16T15:23:45.078941+0000","last_clean":"2024-01-16T15:23:45.078941+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:23:45.078941+0000","last_undegraded":"2024-01-16T15:23:45.078941+0000","last_fullsized":"2024-01-16T15:23:45.078941+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'12","last_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_deep_scrub":"113'12","last_deep_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.078685+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.433669+0000","scrub_duration":0.010660358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'46","reported_seq":69,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.448231+0000","last_change":"2024-01-16T15:15:07.587056+0000","last_active":"2024-01-16T15:23:18.448231+0000","last_peered":"2024-01-16T15:23:18.448231+0000","last_clean":"2024-01-16T15:23:18.448231+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:18.448231+0000","last_undegraded":"2024-01-16T15:23:18.448231+0000","last_fullsized":"2024-01-16T15:23:18.448231+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'1","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:54.811032+0000","last_change":"2024-01-16T15:15:06.561877+0000","last_active":"2024-01-16T15:21:54.811032+0000","last_peered":"2024-01-16T15:21:54.811032+0000","last_clean":"2024-01-16T15:21:54.811032+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:21:54.811032+0000","last_undegraded":"2024-01-16T15:21:54.811032+0000","last_fullsized":"2024-01-16T15:21:54.811032+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'85","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.819083+0000","last_change":"2024-01-16T15:23:49.819161+0000","last_active":"2024-01-16T15:23:49.819083+0000","last_peered":"2024-01-16T15:23:49.819083+0000","last_clean":"2024-01-16T15:23:49.819083+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:49.819083+0000","last_undegraded":"2024-01-16T15:23:49.819083+0000","last_fullsized":"2024-01-16T15:23:49.819083+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'85","last_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_deep_scrub":"113'85","last_deep_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.818959+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:41.716928+0000","scrub_duration":0.28833247000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'3","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.044270+0000","last_change":"2024-01-16T15:23:37.044424+0000","last_active":"2024-01-16T15:23:37.044270+0000","last_peered":"2024-01-16T15:23:37.044270+0000","last_clean":"2024-01-16T15:23:37.044270+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:23:37.044270+0000","last_undegraded":"2024-01-16T15:23:37.044270+0000","last_fullsized":"2024-01-16T15:23:37.044270+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.044057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:58.743730+0000","scrub_duration":0.0080528830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'61","reported_seq":92,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.455657+0000","last_change":"2024-01-16T15:15:07.593281+0000","last_active":"2024-01-16T15:23:18.455657+0000","last_peered":"2024-01-16T15:23:18.455657+0000","last_clean":"2024-01-16T15:23:18.455657+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:18.455657+0000","last_undegraded":"2024-01-16T15:23:18.455657+0000","last_fullsized":"2024-01-16T15:23:18.455657+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'2","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.204623+0000","last_change":"2024-01-16T15:23:36.204678+0000","last_active":"2024-01-16T15:23:36.204623+0000","last_peered":"2024-01-16T15:23:36.204623+0000","last_clean":"2024-01-16T15:23:36.204623+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:23:36.204623+0000","last_undegraded":"2024-01-16T15:23:36.204623+0000","last_fullsized":"2024-01-16T15:23:36.204623+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.204448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:33.212894+0000","scrub_duration":0.0074182859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'62","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.461030+0000","last_change":"2024-01-16T15:15:07.585813+0000","last_active":"2024-01-16T15:23:18.461030+0000","last_peered":"2024-01-16T15:23:18.461030+0000","last_clean":"2024-01-16T15:23:18.461030+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:23:18.461030+0000","last_undegraded":"2024-01-16T15:23:18.461030+0000","last_fullsized":"2024-01-16T15:23:18.461030+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:21:54.446447+0000","last_change":"2024-01-16T15:15:06.562133+0000","last_active":"2024-01-16T15:21:54.446447+0000","last_peered":"2024-01-16T15:21:54.446447+0000","last_clean":"2024-01-16T15:21:54.446447+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:21:54.446447+0000","last_undegraded":"2024-01-16T15:21:54.446447+0000","last_fullsized":"2024-01-16T15:21:54.446447+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'80","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.087001+0000","last_change":"2024-01-16T15:23:45.087001+0000","last_active":"2024-01-16T15:23:45.087001+0000","last_peered":"2024-01-16T15:23:45.087001+0000","last_clean":"2024-01-16T15:23:45.087001+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:45.087001+0000","last_undegraded":"2024-01-16T15:23:45.087001+0000","last_fullsized":"2024-01-16T15:23:45.087001+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'80","last_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_deep_scrub":"113'80","last_deep_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.086751+0000","objects_scrubbed":26,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:06.380394+0000","scrub_duration":0.32344294400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.492389+0000","last_change":"2024-01-16T15:23:31.492439+0000","last_active":"2024-01-16T15:23:31.492389+0000","last_peered":"2024-01-16T15:23:31.492389+0000","last_clean":"2024-01-16T15:23:31.492389+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:23:31.492389+0000","last_undegraded":"2024-01-16T15:23:31.492389+0000","last_fullsized":"2024-01-16T15:23:31.492389+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.492299+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:57.929823+0000","scrub_duration":0.0056823760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.457265+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:18.457265+0000","last_peered":"2024-01-16T15:23:18.457265+0000","last_clean":"2024-01-16T15:23:18.457265+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:18.457265+0000","last_undegraded":"2024-01-16T15:23:18.457265+0000","last_fullsized":"2024-01-16T15:23:18.457265+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.044075+0000","last_change":"2024-01-16T15:23:38.044075+0000","last_active":"2024-01-16T15:23:38.044075+0000","last_peered":"2024-01-16T15:23:38.044075+0000","last_clean":"2024-01-16T15:23:38.044075+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:23:38.044075+0000","last_undegraded":"2024-01-16T15:23:38.044075+0000","last_fullsized":"2024-01-16T15:23:38.044075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.043846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:00.375667+0000","scrub_duration":0.0035534389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'84","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:46.072906+0000","last_change":"2024-01-16T15:23:46.072906+0000","last_active":"2024-01-16T15:23:46.072906+0000","last_peered":"2024-01-16T15:23:46.072906+0000","last_clean":"2024-01-16T15:23:46.072906+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:46.072906+0000","last_undegraded":"2024-01-16T15:23:46.072906+0000","last_fullsized":"2024-01-16T15:23:46.072906+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_clean_scrub_stamp":"2024-01-16T15:23:46.072694+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:53:22.491262+0000","scrub_duration":0.27693848900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.532350+0000","last_change":"2024-01-16T15:23:30.532350+0000","last_active":"2024-01-16T15:23:30.532350+0000","last_peered":"2024-01-16T15:23:30.532350+0000","last_clean":"2024-01-16T15:23:30.532350+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:23:30.532350+0000","last_undegraded":"2024-01-16T15:23:30.532350+0000","last_fullsized":"2024-01-16T15:23:30.532350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.532200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:36.853744+0000","scrub_duration":0.0022174130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'57","reported_seq":85,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.400678+0000","last_change":"2024-01-16T15:15:07.595538+0000","last_active":"2024-01-16T15:23:30.400678+0000","last_peered":"2024-01-16T15:23:30.400678+0000","last_clean":"2024-01-16T15:23:30.400678+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:30.400678+0000","last_undegraded":"2024-01-16T15:23:30.400678+0000","last_fullsized":"2024-01-16T15:23:30.400678+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.996817+0000","last_change":"2024-01-16T15:23:47.996817+0000","last_active":"2024-01-16T15:23:47.996817+0000","last_peered":"2024-01-16T15:23:47.996817+0000","last_clean":"2024-01-16T15:23:47.996817+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:23:47.996817+0000","last_undegraded":"2024-01-16T15:23:47.996817+0000","last_fullsized":"2024-01-16T15:23:47.996817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.996601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:48.743854+0000","scrub_duration":0.0036748639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'67","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.245238+0000","last_change":"2024-01-16T15:23:45.245238+0000","last_active":"2024-01-16T15:23:45.245238+0000","last_peered":"2024-01-16T15:23:45.245238+0000","last_clean":"2024-01-16T15:23:45.245238+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:45.245238+0000","last_undegraded":"2024-01-16T15:23:45.245238+0000","last_fullsized":"2024-01-16T15:23:45.245238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'67","last_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_deep_scrub":"113'67","last_deep_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.244953+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:26.176614+0000","scrub_duration":0.199400785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'3","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.817445+0000","last_change":"2024-01-16T15:23:38.817445+0000","last_active":"2024-01-16T15:23:38.817445+0000","last_peered":"2024-01-16T15:23:38.817445+0000","last_clean":"2024-01-16T15:23:38.817445+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:23:38.817445+0000","last_undegraded":"2024-01-16T15:23:38.817445+0000","last_fullsized":"2024-01-16T15:23:38.817445+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.817208+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:41.655035+0000","scrub_duration":0.0067620090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'83","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.194713+0000","last_change":"2024-01-16T15:23:30.194713+0000","last_active":"2024-01-16T15:23:30.194713+0000","last_peered":"2024-01-16T15:23:30.194713+0000","last_clean":"2024-01-16T15:23:30.194713+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:30.194713+0000","last_undegraded":"2024-01-16T15:23:30.194713+0000","last_fullsized":"2024-01-16T15:23:30.194713+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'83","last_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_deep_scrub":"113'83","last_deep_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.194521+0000","objects_scrubbed":24,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:25.708546+0000","scrub_duration":0.24677183899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'32","reported_seq":59,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.130401+0000","last_change":"2024-01-16T15:23:39.130401+0000","last_active":"2024-01-16T15:23:39.130401+0000","last_peered":"2024-01-16T15:23:39.130401+0000","last_clean":"2024-01-16T15:23:39.130401+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:23:39.130401+0000","last_undegraded":"2024-01-16T15:23:39.130401+0000","last_fullsized":"2024-01-16T15:23:39.130401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'32","last_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_deep_scrub":"113'32","last_deep_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.130166+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:57.838306+0000","scrub_duration":0.0043416110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705,"num_omap_keys":13,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308852+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:28.308852+0000","last_peered":"2024-01-16T15:23:28.308852+0000","last_clean":"2024-01-16T15:23:28.308852+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:28.308852+0000","last_undegraded":"2024-01-16T15:23:28.308852+0000","last_fullsized":"2024-01-16T15:23:28.308852+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308451+0000","last_change":"2024-01-16T15:15:06.564146+0000","last_active":"2024-01-16T15:23:28.308451+0000","last_peered":"2024-01-16T15:23:28.308451+0000","last_clean":"2024-01-16T15:23:28.308451+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:23:28.308451+0000","last_undegraded":"2024-01-16T15:23:28.308451+0000","last_fullsized":"2024-01-16T15:23:28.308451+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:05.508220+0000","last_clean_scrub_stamp":"2024-01-16T15:15:05.508220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'79","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.416185+0000","last_change":"2024-01-16T15:23:43.416246+0000","last_active":"2024-01-16T15:23:43.416185+0000","last_peered":"2024-01-16T15:23:43.416185+0000","last_clean":"2024-01-16T15:23:43.416185+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:43.416185+0000","last_undegraded":"2024-01-16T15:23:43.416185+0000","last_fullsized":"2024-01-16T15:23:43.416185+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.416030+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:24.455972+0000","scrub_duration":0.30055066899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.851113+0000","last_change":"2024-01-16T15:23:39.851159+0000","last_active":"2024-01-16T15:23:39.851113+0000","last_peered":"2024-01-16T15:23:39.851113+0000","last_clean":"2024-01-16T15:23:39.851113+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:23:39.851113+0000","last_undegraded":"2024-01-16T15:23:39.851113+0000","last_fullsized":"2024-01-16T15:23:39.851113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.850958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:29.841870+0000","scrub_duration":0.003185556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'63","reported_seq":89,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.592827+0000","last_change":"2024-01-16T15:15:07.590858+0000","last_active":"2024-01-16T15:23:29.592827+0000","last_peered":"2024-01-16T15:23:29.592827+0000","last_clean":"2024-01-16T15:23:29.592827+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:23:29.592827+0000","last_undegraded":"2024-01-16T15:23:29.592827+0000","last_fullsized":"2024-01-16T15:23:29.592827+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.074149+0000","last_change":"2024-01-16T15:23:41.074229+0000","last_active":"2024-01-16T15:23:41.074149+0000","last_peered":"2024-01-16T15:23:41.074149+0000","last_clean":"2024-01-16T15:23:41.074149+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:23:41.074149+0000","last_undegraded":"2024-01-16T15:23:41.074149+0000","last_fullsized":"2024-01-16T15:23:41.074149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.073835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:56.198760+0000","scrub_duration":0.004400836,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'66","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.592921+0000","last_change":"2024-01-16T15:15:07.576911+0000","last_active":"2024-01-16T15:23:29.592921+0000","last_peered":"2024-01-16T15:23:29.592921+0000","last_clean":"2024-01-16T15:23:29.592921+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:29.592921+0000","last_undegraded":"2024-01-16T15:23:29.592921+0000","last_fullsized":"2024-01-16T15:23:29.592921+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.837299+0000","last_change":"2024-01-16T15:23:33.837346+0000","last_active":"2024-01-16T15:23:33.837299+0000","last_peered":"2024-01-16T15:23:33.837299+0000","last_clean":"2024-01-16T15:23:33.837299+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:23:33.837299+0000","last_undegraded":"2024-01-16T15:23:33.837299+0000","last_fullsized":"2024-01-16T15:23:33.837299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.837111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:32.726849+0000","scrub_duration":0.0035917940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'95","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.161310+0000","last_change":"2024-01-16T15:23:47.161339+0000","last_active":"2024-01-16T15:23:47.161310+0000","last_peered":"2024-01-16T15:23:47.161310+0000","last_clean":"2024-01-16T15:23:47.161310+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:47.161310+0000","last_undegraded":"2024-01-16T15:23:47.161310+0000","last_fullsized":"2024-01-16T15:23:47.161310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'95","last_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_deep_scrub":"113'95","last_deep_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.161227+0000","objects_scrubbed":29,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:50.112996+0000","scrub_duration":0.32985411999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'34","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.113907+0000","last_change":"2024-01-16T15:23:47.114004+0000","last_active":"2024-01-16T15:23:47.113907+0000","last_peered":"2024-01-16T15:23:47.113907+0000","last_clean":"2024-01-16T15:23:47.113907+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:23:47.113907+0000","last_undegraded":"2024-01-16T15:23:47.113907+0000","last_fullsized":"2024-01-16T15:23:47.113907+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'34","last_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_deep_scrub":"113'34","last_deep_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.113654+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:00.387823+0000","scrub_duration":0.0063658849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'70","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.802082+0000","last_change":"2024-01-16T15:15:28.574045+0000","last_active":"2024-01-16T15:23:31.802082+0000","last_peered":"2024-01-16T15:23:31.802082+0000","last_clean":"2024-01-16T15:23:31.802082+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:23:31.802082+0000","last_undegraded":"2024-01-16T15:23:31.802082+0000","last_fullsized":"2024-01-16T15:23:31.802082+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.978255+0000","last_change":"2024-01-16T15:23:39.978309+0000","last_active":"2024-01-16T15:23:39.978255+0000","last_peered":"2024-01-16T15:23:39.978255+0000","last_clean":"2024-01-16T15:23:39.978255+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:23:39.978255+0000","last_undegraded":"2024-01-16T15:23:39.978255+0000","last_fullsized":"2024-01-16T15:23:39.978255+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.977987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:08.833788+0000","scrub_duration":0.0043159469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'62","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.078519+0000","last_change":"2024-01-16T15:23:41.078597+0000","last_active":"2024-01-16T15:23:41.078519+0000","last_peered":"2024-01-16T15:23:41.078519+0000","last_clean":"2024-01-16T15:23:41.078519+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:41.078519+0000","last_undegraded":"2024-01-16T15:23:41.078519+0000","last_fullsized":"2024-01-16T15:23:41.078519+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'62","last_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_deep_scrub":"113'62","last_deep_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.078271+0000","objects_scrubbed":17,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:52.687411+0000","scrub_duration":0.197239254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.012004+0000","last_change":"2024-01-16T15:23:37.012089+0000","last_active":"2024-01-16T15:23:37.012004+0000","last_peered":"2024-01-16T15:23:37.012004+0000","last_clean":"2024-01-16T15:23:37.012004+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:23:37.012004+0000","last_undegraded":"2024-01-16T15:23:37.012004+0000","last_fullsized":"2024-01-16T15:23:37.012004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.011669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:27.915725+0000","scrub_duration":0.0040625560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"113'119","reported_seq":282,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.845903+0000","last_change":"2024-01-16T15:23:34.845995+0000","last_active":"2024-01-16T15:23:34.845903+0000","last_peered":"2024-01-16T15:23:34.845903+0000","last_clean":"2024-01-16T15:23:34.845903+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:23:34.845903+0000","last_undegraded":"2024-01-16T15:23:34.845903+0000","last_fullsized":"2024-01-16T15:23:34.845903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'119","last_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_deep_scrub":"113'119","last_deep_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.845733+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:46.454874+0000","scrub_duration":0.058578549000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4677,"ondisk_log_size":4677,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1225546,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":487,"num_write_kb":1431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2506752,"data_stored":2451092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.151,"15min":1.143},"min":{"1min":0.755,"5min":0.66000000000000003,"15min":0.628},"max":{"1min":1.927,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.198,"15min":1.2190000000000001},"min":{"1min":0.623,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.004,"5min":2.004,"15min":5.9619999999999997},"last":0.877}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1830000000000001,"15min":1.2549999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.69699999999999995},"max":{"1min":1.6699999999999999,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7430000000000001,"5min":1.956,"15min":6.2690000000000001},"last":1.3680000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.21,"15min":1.2090000000000001},"min":{"1min":0.67000000000000004,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.5640000000000001,"5min":2.0070000000000001,"15min":2.5910000000000002},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2709999999999999,"15min":1.2689999999999999},"min":{"1min":0.95199999999999996,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.879,"5min":1.879,"15min":2.6920000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2829999999999999,"15min":1.3380000000000001},"min":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.4990000000000001,"5min":1.9279999999999999,"15min":12.331},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.373,"15min":1.3560000000000001},"min":{"1min":0.90400000000000003,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":1.954,"5min":2.0649999999999999,"15min":2.645},"last":1.145}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3220000000000001,"15min":1.3560000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.49199999999999999},"max":{"1min":2.109,"5min":2.109,"15min":6.1799999999999997},"last":1.073},{"interface":"front","average":{"1min":1.431,"5min":1.321,"15min":1.319},"min":{"1min":1.0780000000000001,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":2.4159999999999999},"last":1.1080000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.397,"15min":1.401},"min":{"1min":0.88900000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.4350000000000001,"5min":2.508,"15min":2.593},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4930000000000001,"15min":1.532},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.54700000000000004},"max":{"1min":2.2050000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.2809999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.464,"15min":1.496},"min":{"1min":0.78800000000000003,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.988,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.621,"5min":1.5649999999999999,"15min":1.5860000000000001},"min":{"1min":1.032,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":4.7649999999999997},"last":1.3120000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575180,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.591},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.6990000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":1.097},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.1619999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":1.615},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.403},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.468}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":1.5129999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.78},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.343}]}]},{"osd":5,"up_from":13,"seq":55834575179,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.179,"15min":1.159},"min":{"1min":0.628,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":1.9419999999999999,"5min":2.2040000000000002,"15min":3.7130000000000001},"last":0.629},{"interface":"front","average":{"1min":1.161,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.49299999999999999},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":8.7880000000000003},"last":0.70099999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":2.1389999999999998,"15min":3.835},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.292,"5min":1.3120000000000001,"15min":1.298},"min":{"1min":0.67400000000000004,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.95,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":0.93799999999999994}]},{"osd":2,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.292,"15min":1.329},"min":{"1min":0.96299999999999997,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":1.6120000000000001,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.373,"15min":1.3720000000000001},"min":{"1min":1.0209999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.673,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.044}]},{"osd":3,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.325,"15min":1.3400000000000001},"min":{"1min":0.86599999999999999,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.6719999999999999,"5min":1.742,"15min":3.9319999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.86699999999999999,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":16.158000000000001},"last":0.83699999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.278,"15min":1.238},"min":{"1min":0.91000000000000003,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":1.8009999999999999,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.488,"15min":1.4910000000000001},"min":{"1min":1.0920000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.887,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":0.50700000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":1.117,"5min":0.878,"15min":0.46800000000000003},"max":{"1min":1.857,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.363,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.89100000000000001,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.351,"5min":2.351,"15min":6.093},"last":0.55600000000000005}]},{"osd":7,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4430000000000001,"15min":1.462},"min":{"1min":1.079,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.145,"5min":2.145,"15min":3.8809999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.556,"15min":1.522},"min":{"1min":1.141,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.456,"5min":2.456,"15min":5.0209999999999999},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1928112,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92443728,"statfs":{"total":96636764160,"available":94662377472,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.3680000000000001},"min":{"1min":0.628,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.6579999999999999,"5min":1.73,"15min":17.315999999999999},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.395,"15min":1.427},"min":{"1min":0.753,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.7,"5min":1.7949999999999999,"15min":17.279},"last":1.522}]},{"osd":1,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.226,"15min":1.266},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.498},"max":{"1min":1.3720000000000001,"5min":1.839,"15min":6.1849999999999996},"last":1.252},{"interface":"front","average":{"1min":1.29,"5min":1.2230000000000001,"15min":1.1830000000000001},"min":{"1min":0.87,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":1.157}]},{"osd":2,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1990000000000001,"15min":1.1870000000000001},"min":{"1min":0.90500000000000003,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.6479999999999999,"5min":1.6830000000000001,"15min":2.1299999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.365,"15min":1.3460000000000001},"min":{"1min":1.0429999999999999,"5min":0.88600000000000001,"15min":0.75800000000000001},"max":{"1min":1.645,"5min":1.8120000000000001,"15min":2.4780000000000002},"last":1.627}]},{"osd":3,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.266,"15min":1.3069999999999999},"min":{"1min":0.96499999999999997,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.667,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3740000000000001,"15min":1.3560000000000001},"min":{"1min":1.056,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.77,"5min":1.77,"15min":4.7930000000000001},"last":1.3759999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.276,"15min":1.3049999999999999},"min":{"1min":0.98599999999999999,"5min":0.69999999999999996,"15min":0.63},"max":{"1min":1.54,"5min":1.8999999999999999,"15min":2.597},"last":1.409},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.335,"15min":1.333},"min":{"1min":1.0069999999999999,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.7929999999999999,"5min":1.8660000000000001,"15min":4.8520000000000003},"last":1.2370000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":1.004,"5min":0.89700000000000002,"15min":0.57099999999999995},"max":{"1min":1.8420000000000001,"5min":2.0230000000000001,"15min":2.3479999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4810000000000001,"15min":1.52},"min":{"1min":1.0940000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7529999999999999,"5min":1.946,"15min":12.324999999999999},"last":1.0940000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4610000000000001,"15min":1.466},"min":{"1min":1.0860000000000001,"5min":1.032,"15min":0.53600000000000003},"max":{"1min":1.839,"5min":1.968,"15min":4.9539999999999997},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.5649999999999999,"15min":1.534},"min":{"1min":1.296,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.903,"5min":1.9039999999999999,"15min":2.2599999999999998},"last":1.5900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575180,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1822956,"kb_used_data":1751588,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92548884,"statfs":{"total":96636764160,"available":94770057216,"internally_reserved":0,"allocated":1793626112,"data_stored":1792798405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94499999999999995,"15min":0.91600000000000004},"min":{"1min":0.34599999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.48,"5min":1.6240000000000001,"15min":1.925},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.097,"15min":1.03},"min":{"1min":0.45100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.377,"5min":6.6740000000000004,"15min":6.6740000000000004},"last":0.84099999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.93500000000000005,"15min":0.98599999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.44900000000000001},"max":{"1min":1.3140000000000001,"5min":1.6220000000000001,"15min":3.1150000000000002},"last":1.014},{"interface":"front","average":{"1min":1.056,"5min":1.105,"15min":1.077},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.38500000000000001},"max":{"1min":1.385,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.1539999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1240000000000001,"15min":1.1319999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5609999999999999,"5min":1.8169999999999999,"15min":3.0670000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1919999999999999,"15min":1.204},"min":{"1min":0.55000000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.605,"5min":1.986,"15min":3.4279999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3540000000000001,"15min":1.343},"min":{"1min":0.84699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.5049999999999999},"last":1.052},{"interface":"front","average":{"1min":1.367,"5min":1.3859999999999999,"15min":1.391},"min":{"1min":1.0920000000000001,"5min":0.85499999999999998,"15min":0.64000000000000001},"max":{"1min":1.718,"5min":1.9810000000000001,"15min":3.8250000000000002},"last":1.389}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3480000000000001,"15min":1.3440000000000001},"min":{"1min":0.88300000000000001,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.6240000000000001,"5min":2.085,"15min":2.3540000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.365,"15min":1.3939999999999999},"min":{"1min":0.80500000000000005,"5min":0.79500000000000004,"15min":0.71299999999999997},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.907},"last":1.486}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3939999999999999,"15min":1.4379999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.8560000000000001,"5min":2.0569999999999999,"15min":6.4480000000000004},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4259999999999999,"15min":1.4770000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.81299999999999994},"max":{"1min":1.7869999999999999,"5min":2.1539999999999999,"15min":3.9039999999999999},"last":1.127}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.73099999999999998},"max":{"1min":1.778,"5min":1.9370000000000001,"15min":3.7850000000000001},"last":1.411},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.482,"15min":1.5169999999999999},"min":{"1min":1.0429999999999999,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.891,"5min":2.085,"15min":3.9590000000000001},"last":1.214}]}]},{"osd":2,"up_from":13,"seq":55834575179,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.90000000000000002,"15min":0.88900000000000001},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.314},"max":{"1min":1.603,"5min":1.603,"15min":2.8919999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.024,"15min":1.036},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.42299999999999999},"max":{"1min":1.4350000000000001,"5min":1.76,"15min":2.952},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.034,"15min":1.0569999999999999},"min":{"1min":0.58799999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.5820000000000001,"5min":1.605,"15min":1.823},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.127},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3600000000000001,"5min":1.6479999999999999,"15min":4.734},"last":1.153}]},{"osd":3,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.028,"15min":1.0509999999999999},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.403,"5min":1.788,"15min":1.788},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":0.63700000000000001,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.5660000000000001,"5min":1.8120000000000001,"15min":2.3220000000000001},"last":1.417}]},{"osd":4,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.335,"15min":1.4770000000000001},"min":{"1min":0.84699999999999998,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":1.5660000000000001,"5min":2.0680000000000001,"15min":16.681999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.292,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.746},"max":{"1min":1.544,"5min":1.919,"15min":1.964},"last":1.2410000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.97199999999999998,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":1.879,"5min":2.206,"15min":2.3799999999999999},"last":1.083},{"interface":"front","average":{"1min":1.373,"5min":1.4039999999999999,"15min":1.431},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":4.8659999999999997},"last":1.452}]},{"osd":6,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.373,"15min":1.3879999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.986,"5min":2.0720000000000001,"15min":2.1539999999999999},"last":1.373},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4710000000000001,"15min":1.484},"min":{"1min":1.022,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":1.887,"5min":2.1080000000000001,"15min":6.2599999999999998},"last":1.341}]},{"osd":7,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.472,"15min":1.49},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.70099999999999996},"max":{"1min":1.7010000000000001,"5min":2.125,"15min":3.7530000000000001},"last":1.611},{"interface":"front","average":{"1min":1.341,"5min":1.4379999999999999,"15min":1.45},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.78400000000000003},"max":{"1min":1.6950000000000001,"5min":2.0670000000000002,"15min":2.294},"last":1.5649999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.91200000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.399,"5min":1.4330000000000001,"15min":14.106999999999999},"last":0.754},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.03,"15min":1.0569999999999999},"min":{"1min":0.47899999999999998,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.504,"5min":1.504,"15min":4.3719999999999999},"last":1.2070000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.96299999999999997,"15min":0.96499999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.29799999999999999},"max":{"1min":1.2709999999999999,"5min":3.5409999999999999,"15min":3.6309999999999998},"last":1.264},{"interface":"front","average":{"1min":0.995,"5min":1.0740000000000001,"15min":1.1339999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.3720000000000001,"5min":1.663,"15min":14.055},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.22,"15min":1.266},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.48,"5min":1.6439999999999999,"15min":13.967000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1060000000000001,"15min":1.135},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5840000000000001,"5min":1.952,"15min":4.6699999999999999},"last":1.083}]},{"osd":4,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.228,"15min":1.244},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52500000000000002},"max":{"1min":1.698,"5min":3.77,"15min":4.3799999999999999},"last":1.605},{"interface":"front","average":{"1min":1.379,"5min":1.3540000000000001,"15min":1.4650000000000001},"min":{"1min":0.89700000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":14.161},"last":1.748}]},{"osd":5,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.5369999999999999},"min":{"1min":1.0289999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7450000000000001,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.413,"15min":1.508},"min":{"1min":1.079,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.8460000000000001,"5min":3.8090000000000002,"15min":16.433},"last":1.776}]},{"osd":6,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4390000000000001,"15min":1.4930000000000001},"min":{"1min":1.1100000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":14.212},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3979999999999999,"15min":1.444},"min":{"1min":0.872,"5min":0.872,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":1.71}]},{"osd":7,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.431,"15min":1.5209999999999999},"min":{"1min":0.94099999999999995,"5min":0.82499999999999996,"15min":0.78100000000000003},"max":{"1min":1.8460000000000001,"5min":2.1930000000000001,"15min":12.590999999999999},"last":1.375},{"interface":"front","average":{"1min":1.373,"5min":1.4530000000000001,"15min":1.454},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.64100000000000001},"max":{"1min":1.7829999999999999,"5min":3.726,"15min":6.0229999999999997},"last":1.409}]}]},{"osd":0,"up_from":13,"seq":55834575179,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048744,"kb_used_data":1981536,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323096,"statfs":{"total":96636764160,"available":94538850304,"internally_reserved":0,"allocated":2029092864,"data_stored":2028268228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.83499999999999996,"15min":0.84199999999999997},"min":{"1min":0.621,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.3240000000000001,"5min":1.601,"15min":1.921},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0449999999999999,"15min":1.01},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.41799999999999998},"max":{"1min":1.653,"5min":1.72,"15min":4.4059999999999997},"last":0.72699999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0109999999999999,"15min":1.0649999999999999},"min":{"1min":0.80200000000000005,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":1.6220000000000001,"15min":17.260000000000002},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.115,"15min":1.0980000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.492,"5min":1.784,"15min":4.492},"last":1.238}]},{"osd":3,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1180000000000001,"15min":1.087},"min":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.538,"5min":1.706,"15min":3.4670000000000001},"last":1.026},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.325,"15min":1.2869999999999999},"min":{"1min":0.83799999999999997,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.5840000000000001,"5min":1.8400000000000001,"15min":2.319},"last":1.546}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.337,"15min":1.3460000000000001},"min":{"1min":1.0860000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":4.2889999999999997},"last":1.504},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3360000000000001,"15min":1.3100000000000001},"min":{"1min":0.93000000000000005,"5min":0.83799999999999997,"15min":0.67800000000000005},"max":{"1min":4.0810000000000004,"5min":4.0810000000000004,"15min":4.0810000000000004},"last":1.181}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4259999999999999,"15min":1.4159999999999999},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.624},"max":{"1min":2.1850000000000001,"5min":2.222,"15min":3.1259999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4950000000000001,"15min":1.486},"min":{"1min":1.0129999999999999,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.1389999999999998,"5min":2.2730000000000001,"15min":6.5430000000000001},"last":1.0589999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.397,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":11.462},"last":1.341},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4419999999999999,"15min":1.4570000000000001},"min":{"1min":1.0189999999999999,"5min":0.92600000000000005,"15min":0.80100000000000005},"max":{"1min":1.992,"5min":2.093,"15min":7.1849999999999996},"last":1.1120000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.466,"15min":1.502},"min":{"1min":0.86799999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.9399999999999999,"5min":2.1299999999999999,"15min":5.2450000000000001},"last":1.409},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.514,"15min":1.4750000000000001},"min":{"1min":1.123,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.472}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":1221216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1232896,"data_stored":1221174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:54.282 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.283 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.283 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.283 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.283 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.283 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.284 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.284 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.284 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.284 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.284 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.285 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.285 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.285 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.285 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.285 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.286 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.287 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.287 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.287 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.287 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.287 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.288 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.289 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.290 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.290 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:15:05.508220+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=5, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.290 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.290 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.290 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:23:54.291 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:24:14.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:24:14.514 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.518+0000 7f83b9c00640 1 Processor -- start 2024-01-16T15:24:14.515 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.518+0000 7f83b9c00640 1 -- start start 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.519+0000 7f83b9c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83b4072230 0x7f83b4150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b9c00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b4071190 con 0x7f83b415da10 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b9c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b4071470 con 0x7f83b40cf290 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b9c00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b4072610 con 0x7f83b4072230 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b2a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83b4072230 0x7f83b4150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b3400640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f83b415da10 0x7f83b40b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56160/0 (socket says 172.21.15.125:56160) 2024-01-16T15:24:14.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b3400640 1 -- 172.21.15.125:0/1280536711 learned_addr learned my addr 172.21.15.125:0/1280536711 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:24:14.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b2a00640 1 -- 172.21.15.125:0/1280536711 >> v1:172.21.15.137:6789/0 conn(0x7f83b40cf290 legacy=0x7f83b415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:14.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b2a00640 1 -- 172.21.15.125:0/1280536711 >> v1:172.21.15.125:6789/0 conn(0x7f83b415da10 legacy=0x7f83b40b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b2000640 1 -- 172.21.15.125:0/1280536711 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171559808 0 0) 0x7f83b4071190 con 0x7f83b415da10 2024-01-16T15:24:14.518 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.520+0000 7f83b2a00640 1 -- 172.21.15.125:0/1280536711 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4071750 con 0x7f83b4072230 2024-01-16T15:24:14.519 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b2a00640 1 --2- 172.21.15.125:0/1280536711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4072230 0x7f83b4150760 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f83a40097a0 tx=0x7f83a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78165e0c28c5f76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b2000640 1 -- 172.21.15.125:0/1280536711 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4031830 con 0x7f83b4072230 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b2000640 1 -- 172.21.15.125:0/1280536711 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4031990 con 0x7f83b4072230 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b2000640 1 -- 172.21.15.125:0/1280536711 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4031d20 con 0x7f83b4072230 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b9c00640 1 -- 172.21.15.125:0/1280536711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4072230 msgr2=0x7f83b4150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.522+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1280536711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4072230 0x7f83b4150760 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f83a40097a0 tx=0x7f83a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.523+0000 7f83b9c00640 1 -- 172.21.15.125:0/1280536711 shutdown_connections 2024-01-16T15:24:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.523+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1280536711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4072230 0x7f83b4150760 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.523+0000 7f83b9c00640 1 -- 172.21.15.125:0/1280536711 >> 172.21.15.125:0/1280536711 conn(0x7f83b406a800 msgr2=0x7f83b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.523+0000 7f83b9c00640 1 -- 172.21.15.125:0/1280536711 shutdown_connections 2024-01-16T15:24:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.523+0000 7f83b9c00640 1 -- 172.21.15.125:0/1280536711 wait complete. 2024-01-16T15:24:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.525+0000 7f83b9c00640 1 Processor -- start 2024-01-16T15:24:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.525+0000 7f83b9c00640 1 -- start start 2024-01-16T15:24:14.523 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83b4072230 0x7f83b414f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83b415da10 0x7f83b41587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41ed050 con 0x7f83b40cf290 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cc120 con 0x7f83b415da10 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83b9c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41e9ec0 con 0x7f83b4072230 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83a9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83a9e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46632/0 (socket says 172.21.15.125:46632) 2024-01-16T15:24:14.524 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.526+0000 7f83a9e00640 1 -- 172.21.15.125:0/1688060895 learned_addr learned my addr 172.21.15.125:0/1688060895 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83b2a00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83b4072230 0x7f83b414f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83b3400640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83b415da10 0x7f83b41587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 -- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83b4072230 msgr2=0x7f83b414f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83b4072230 0x7f83b414f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 -- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83b415da10 msgr2=0x7f83b41587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83b415da10 0x7f83b41587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 -- 172.21.15.125:0/1688060895 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4009400 con 0x7f83b40cf290 2024-01-16T15:24:14.525 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83a9e00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c430 tx=0x7f839c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:14.526 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c011790 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c0118f0 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.527+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41ea1a0 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.528+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c011ca0 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.528+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b40cc3c0 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.529+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b41ea6c0 con 0x7f83b40cf290 2024-01-16T15:24:14.527 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.530+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f839c01c460 con 0x7f83b40cf290 2024-01-16T15:24:14.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.531+0000 7f83ab400640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 0x7f838c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:14.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.531+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f839c088a70 con 0x7f83b40cf290 2024-01-16T15:24:14.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.531+0000 7f83b2a00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 0x7f838c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:14.528 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.532+0000 7f83b2a00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 0x7f838c06ea10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f83a4036000 tx=0x7f83a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:14.532 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.536+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83b41ea6c0 con 0x7f83b40cf290 2024-01-16T15:24:14.718 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.722+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83b4071d10 con 0x7f838c06c550 2024-01-16T15:24:14.740 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.743+0000 7f83ab400640 1 -- 172.21.15.125:0/1688060895 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291050 (secure 0 0 0) 0x7f83a4045410 con 0x7f838c06c550 2024-01-16T15:24:14.744 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:24:14.745 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:24:14.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 msgr2=0x7f838c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 0x7f838c06ea10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f83a4036000 tx=0x7f83a403c040 comp rx=0 tx=0).stop 2024-01-16T15:24:14.747 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 msgr2=0x7f83b4158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c430 tx=0x7f839c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 shutdown_connections 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83b415da10 0x7f83b41587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f838c06c550 0x7f838c06ea10 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b40cf290 0x7f83b4158270 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 --2- 172.21.15.125:0/1688060895 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83b4072230 0x7f83b414f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:14.748 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 >> 172.21.15.125:0/1688060895 conn(0x7f83b406a800 msgr2=0x7f83b406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:14.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 shutdown_connections 2024-01-16T15:24:14.749 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:14.748+0000 7f83b9c00640 1 -- 172.21.15.125:0/1688060895 wait complete. 2024-01-16T15:24:14.763 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":936,"stamp":"2024-01-16T15:24:14.558688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444268918,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2250,"num_write_kb":6295341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6013,"ondisk_log_size":6013,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13143548,"kb_used_data":12597564,"kb_used_omap":12,"kb_used_meta":545907,"kb_avail":741831172,"statfs":{"total":773094113280,"available":759635120128,"internally_reserved":0,"allocated":12899905536,"data_stored":12893402313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":559009328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004657"},"pg_stats":[{"pgid":"17.3f","version":"113'98","reported_seq":138,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.383941+0000","last_change":"2024-01-16T15:24:02.384001+0000","last_active":"2024-01-16T15:24:02.383941+0000","last_peered":"2024-01-16T15:24:02.383941+0000","last_clean":"2024-01-16T15:24:02.383941+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:24:02.383941+0000","last_undegraded":"2024-01-16T15:24:02.383941+0000","last_fullsized":"2024-01-16T15:24:02.383941+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'98","last_scrub_stamp":"2024-01-16T15:24:02.383764+0000","last_deep_scrub":"113'98","last_deep_scrub_stamp":"2024-01-16T15:24:02.383764+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.383764+0000","objects_scrubbed":28,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:21.709126+0000","scrub_duration":0.323488779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'91","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.213438+0000","last_change":"2024-01-16T15:23:42.213497+0000","last_active":"2024-01-16T15:23:42.213438+0000","last_peered":"2024-01-16T15:23:42.213438+0000","last_clean":"2024-01-16T15:23:42.213438+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:42.213438+0000","last_undegraded":"2024-01-16T15:23:42.213438+0000","last_fullsized":"2024-01-16T15:23:42.213438+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'91","last_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_deep_scrub":"113'91","last_deep_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.213253+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:45.051080+0000","scrub_duration":0.29806467199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'94","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:58.326078+0000","last_change":"2024-01-16T15:23:58.326121+0000","last_active":"2024-01-16T15:23:58.326078+0000","last_peered":"2024-01-16T15:23:58.326078+0000","last_clean":"2024-01-16T15:23:58.326078+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:58.326078+0000","last_undegraded":"2024-01-16T15:23:58.326078+0000","last_fullsized":"2024-01-16T15:23:58.326078+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'94","last_scrub_stamp":"2024-01-16T15:23:58.325934+0000","last_deep_scrub":"113'94","last_deep_scrub_stamp":"2024-01-16T15:23:58.325934+0000","last_clean_scrub_stamp":"2024-01-16T15:23:58.325934+0000","objects_scrubbed":27,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:49:22.071877+0000","scrub_duration":0.32196777599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'66","reported_seq":95,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.458250+0000","last_change":"2024-01-16T15:15:07.599765+0000","last_active":"2024-01-16T15:23:18.458250+0000","last_peered":"2024-01-16T15:23:18.458250+0000","last_clean":"2024-01-16T15:23:18.458250+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:23:18.458250+0000","last_undegraded":"2024-01-16T15:23:18.458250+0000","last_fullsized":"2024-01-16T15:23:18.458250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'85","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.463762+0000","last_change":"2024-01-16T15:23:57.463810+0000","last_active":"2024-01-16T15:23:57.463762+0000","last_peered":"2024-01-16T15:23:57.463762+0000","last_clean":"2024-01-16T15:23:57.463762+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:57.463762+0000","last_undegraded":"2024-01-16T15:23:57.463762+0000","last_fullsized":"2024-01-16T15:23:57.463762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'85","last_scrub_stamp":"2024-01-16T15:23:57.463577+0000","last_deep_scrub":"113'85","last_deep_scrub_stamp":"2024-01-16T15:23:57.463577+0000","last_clean_scrub_stamp":"2024-01-16T15:23:57.463577+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:44.314486+0000","scrub_duration":0.28701645999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'100","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.880459+0000","last_change":"2024-01-16T15:23:43.880600+0000","last_active":"2024-01-16T15:23:43.880459+0000","last_peered":"2024-01-16T15:23:43.880459+0000","last_clean":"2024-01-16T15:23:43.880459+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:43.880459+0000","last_undegraded":"2024-01-16T15:23:43.880459+0000","last_fullsized":"2024-01-16T15:23:43.880459+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.880280+0000","objects_scrubbed":31,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:51.072560+0000","scrub_duration":0.388654414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'84","reported_seq":114,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.465385+0000","last_change":"2024-01-16T15:24:02.465465+0000","last_active":"2024-01-16T15:24:02.465385+0000","last_peered":"2024-01-16T15:24:02.465385+0000","last_clean":"2024-01-16T15:24:02.465385+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:24:02.465385+0000","last_undegraded":"2024-01-16T15:24:02.465385+0000","last_fullsized":"2024-01-16T15:24:02.465385+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:24:02.465203+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:24:02.465203+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.465203+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:52.456114+0000","scrub_duration":0.229053322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'71","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.992771+0000","last_change":"2024-01-16T15:23:39.992859+0000","last_active":"2024-01-16T15:23:39.992771+0000","last_peered":"2024-01-16T15:23:39.992771+0000","last_clean":"2024-01-16T15:23:39.992771+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:39.992771+0000","last_undegraded":"2024-01-16T15:23:39.992771+0000","last_fullsized":"2024-01-16T15:23:39.992771+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_deep_scrub":"113'71","last_deep_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.992591+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:54.989201+0000","scrub_duration":0.23396198900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'100","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.211670+0000","last_change":"2024-01-16T15:23:33.211722+0000","last_active":"2024-01-16T15:23:33.211670+0000","last_peered":"2024-01-16T15:23:33.211670+0000","last_clean":"2024-01-16T15:23:33.211670+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:33.211670+0000","last_undegraded":"2024-01-16T15:23:33.211670+0000","last_fullsized":"2024-01-16T15:23:33.211670+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.211507+0000","objects_scrubbed":27,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:47.572037+0000","scrub_duration":0.235911915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'88","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.390744+0000","last_change":"2024-01-16T15:24:08.390835+0000","last_active":"2024-01-16T15:24:08.390744+0000","last_peered":"2024-01-16T15:24:08.390744+0000","last_clean":"2024-01-16T15:24:08.390744+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:24:08.390744+0000","last_undegraded":"2024-01-16T15:24:08.390744+0000","last_fullsized":"2024-01-16T15:24:08.390744+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:24:08.390604+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:24:08.390604+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.390604+0000","objects_scrubbed":22,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:22.535744+0000","scrub_duration":0.297321159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'70","reported_seq":105,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.447016+0000","last_change":"2024-01-16T15:15:07.591675+0000","last_active":"2024-01-16T15:23:18.447016+0000","last_peered":"2024-01-16T15:23:18.447016+0000","last_clean":"2024-01-16T15:23:18.447016+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:23:18.447016+0000","last_undegraded":"2024-01-16T15:23:18.447016+0000","last_fullsized":"2024-01-16T15:23:18.447016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'59","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:18.449095+0000","last_change":"2024-01-16T15:15:07.593468+0000","last_active":"2024-01-16T15:23:18.449095+0000","last_peered":"2024-01-16T15:23:18.449095+0000","last_clean":"2024-01-16T15:23:18.449095+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:23:18.449095+0000","last_undegraded":"2024-01-16T15:23:18.449095+0000","last_fullsized":"2024-01-16T15:23:18.449095+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'78","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.086907+0000","last_change":"2024-01-16T15:23:48.087015+0000","last_active":"2024-01-16T15:23:48.086907+0000","last_peered":"2024-01-16T15:23:48.086907+0000","last_clean":"2024-01-16T15:23:48.086907+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:48.086907+0000","last_undegraded":"2024-01-16T15:23:48.086907+0000","last_fullsized":"2024-01-16T15:23:48.086907+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.086739+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:49.934440+0000","scrub_duration":0.26635034800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'88","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.116864+0000","last_change":"2024-01-16T15:23:44.116922+0000","last_active":"2024-01-16T15:23:44.116864+0000","last_peered":"2024-01-16T15:23:44.116864+0000","last_clean":"2024-01-16T15:23:44.116864+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:44.116864+0000","last_undegraded":"2024-01-16T15:23:44.116864+0000","last_fullsized":"2024-01-16T15:23:44.116864+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.116705+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:20.435799+0000","scrub_duration":0.35551533600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'64","reported_seq":96,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.593097+0000","last_change":"2024-01-16T15:23:33.593181+0000","last_active":"2024-01-16T15:23:33.593097+0000","last_peered":"2024-01-16T15:23:33.593097+0000","last_clean":"2024-01-16T15:23:33.593097+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:33.593097+0000","last_undegraded":"2024-01-16T15:23:33.593097+0000","last_fullsized":"2024-01-16T15:23:33.593097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'64","last_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_deep_scrub":"113'64","last_deep_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.592968+0000","objects_scrubbed":14,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:45.001245+0000","scrub_duration":0.13121580099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'77","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.460923+0000","last_change":"2024-01-16T15:24:00.461021+0000","last_active":"2024-01-16T15:24:00.460923+0000","last_peered":"2024-01-16T15:24:00.460923+0000","last_clean":"2024-01-16T15:24:00.460923+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:24:00.460923+0000","last_undegraded":"2024-01-16T15:24:00.460923+0000","last_fullsized":"2024-01-16T15:24:00.460923+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'77","last_scrub_stamp":"2024-01-16T15:24:00.460742+0000","last_deep_scrub":"113'77","last_deep_scrub_stamp":"2024-01-16T15:24:00.460742+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.460742+0000","objects_scrubbed":19,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:21.783247+0000","scrub_duration":0.22164234999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'66","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.091684+0000","last_change":"2024-01-16T15:23:49.091768+0000","last_active":"2024-01-16T15:23:49.091684+0000","last_peered":"2024-01-16T15:23:49.091684+0000","last_clean":"2024-01-16T15:23:49.091684+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:49.091684+0000","last_undegraded":"2024-01-16T15:23:49.091684+0000","last_fullsized":"2024-01-16T15:23:49.091684+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'66","last_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_deep_scrub":"113'66","last_deep_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.091510+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:04.088120+0000","scrub_duration":0.232593777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'107","reported_seq":158,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.575880+0000","last_change":"2024-01-16T15:23:52.575949+0000","last_active":"2024-01-16T15:23:52.575880+0000","last_peered":"2024-01-16T15:23:52.575880+0000","last_clean":"2024-01-16T15:23:52.575880+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:52.575880+0000","last_undegraded":"2024-01-16T15:23:52.575880+0000","last_fullsized":"2024-01-16T15:23:52.575880+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'107","last_scrub_stamp":"2024-01-16T15:23:52.575711+0000","last_deep_scrub":"113'107","last_deep_scrub_stamp":"2024-01-16T15:23:52.575711+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.575711+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:20.721431+0000","scrub_duration":0.40777005500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'100","reported_seq":136,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.312479+0000","last_change":"2024-01-16T15:23:59.312479+0000","last_active":"2024-01-16T15:23:59.312479+0000","last_peered":"2024-01-16T15:23:59.312479+0000","last_clean":"2024-01-16T15:23:59.312479+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:59.312479+0000","last_undegraded":"2024-01-16T15:23:59.312479+0000","last_fullsized":"2024-01-16T15:23:59.312479+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:59.312256+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:59.312256+0000","last_clean_scrub_stamp":"2024-01-16T15:23:59.312256+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:33.383998+0000","scrub_duration":0.31913148600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'87","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.218013+0000","last_change":"2024-01-16T15:23:43.218013+0000","last_active":"2024-01-16T15:23:43.218013+0000","last_peered":"2024-01-16T15:23:43.218013+0000","last_clean":"2024-01-16T15:23:43.218013+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:43.218013+0000","last_undegraded":"2024-01-16T15:23:43.218013+0000","last_fullsized":"2024-01-16T15:23:43.218013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'87","last_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_deep_scrub":"113'87","last_deep_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.217792+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.208703+0000","scrub_duration":0.29447676900000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'103","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.212338+0000","last_change":"2024-01-16T15:23:48.212338+0000","last_active":"2024-01-16T15:23:48.212338+0000","last_peered":"2024-01-16T15:23:48.212338+0000","last_clean":"2024-01-16T15:23:48.212338+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:48.212338+0000","last_undegraded":"2024-01-16T15:23:48.212338+0000","last_fullsized":"2024-01-16T15:23:48.212338+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'103","last_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_deep_scrub":"113'103","last_deep_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.212080+0000","objects_scrubbed":30,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:31.387018+0000","scrub_duration":0.36970662999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'90","reported_seq":124,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:53.116762+0000","last_change":"2024-01-16T15:23:53.116762+0000","last_active":"2024-01-16T15:23:53.116762+0000","last_peered":"2024-01-16T15:23:53.116762+0000","last_clean":"2024-01-16T15:23:53.116762+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:53.116762+0000","last_undegraded":"2024-01-16T15:23:53.116762+0000","last_fullsized":"2024-01-16T15:23:53.116762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'90","last_scrub_stamp":"2024-01-16T15:23:53.116433+0000","last_deep_scrub":"113'90","last_deep_scrub_stamp":"2024-01-16T15:23:53.116433+0000","last_clean_scrub_stamp":"2024-01-16T15:23:53.116433+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:15.298285+0000","scrub_duration":0.29584653500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'79","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.249926+0000","last_change":"2024-01-16T15:23:37.249926+0000","last_active":"2024-01-16T15:23:37.249926+0000","last_peered":"2024-01-16T15:23:37.249926+0000","last_clean":"2024-01-16T15:23:37.249926+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:37.249926+0000","last_undegraded":"2024-01-16T15:23:37.249926+0000","last_fullsized":"2024-01-16T15:23:37.249926+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.249557+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:23.718419+0000","scrub_duration":0.30745207200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'90","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.124563+0000","last_change":"2024-01-16T15:23:47.124563+0000","last_active":"2024-01-16T15:23:47.124563+0000","last_peered":"2024-01-16T15:23:47.124563+0000","last_clean":"2024-01-16T15:23:47.124563+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:47.124563+0000","last_undegraded":"2024-01-16T15:23:47.124563+0000","last_fullsized":"2024-01-16T15:23:47.124563+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'90","last_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_deep_scrub":"113'90","last_deep_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.124399+0000","objects_scrubbed":24,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:02.121008+0000","scrub_duration":0.271690178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'96","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.160049+0000","last_change":"2024-01-16T15:23:47.160049+0000","last_active":"2024-01-16T15:23:47.160049+0000","last_peered":"2024-01-16T15:23:47.160049+0000","last_clean":"2024-01-16T15:23:47.160049+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:47.160049+0000","last_undegraded":"2024-01-16T15:23:47.160049+0000","last_fullsized":"2024-01-16T15:23:47.160049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.159896+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:22.125663+0000","scrub_duration":0.29641904899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'107","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.460144+0000","last_change":"2024-01-16T15:24:06.460201+0000","last_active":"2024-01-16T15:24:06.460144+0000","last_peered":"2024-01-16T15:24:06.460144+0000","last_clean":"2024-01-16T15:24:06.460144+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:24:06.460144+0000","last_undegraded":"2024-01-16T15:24:06.460144+0000","last_fullsized":"2024-01-16T15:24:06.460144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'107","last_scrub_stamp":"2024-01-16T15:24:06.459972+0000","last_deep_scrub":"113'107","last_deep_scrub_stamp":"2024-01-16T15:24:06.459972+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.459972+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:24.029836+0000","scrub_duration":0.33540472199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'86","reported_seq":122,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.783319+0000","last_change":"2024-01-16T15:23:44.783319+0000","last_active":"2024-01-16T15:23:44.783319+0000","last_peered":"2024-01-16T15:23:44.783319+0000","last_clean":"2024-01-16T15:23:44.783319+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:44.783319+0000","last_undegraded":"2024-01-16T15:23:44.783319+0000","last_fullsized":"2024-01-16T15:23:44.783319+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'86","last_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_deep_scrub":"113'86","last_deep_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.783151+0000","objects_scrubbed":24,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.138136+0000","scrub_duration":0.29777104900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'62","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.068006+0000","last_change":"2024-01-16T15:24:01.068122+0000","last_active":"2024-01-16T15:24:01.068006+0000","last_peered":"2024-01-16T15:24:01.068006+0000","last_clean":"2024-01-16T15:24:01.068006+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:24:01.068006+0000","last_undegraded":"2024-01-16T15:24:01.068006+0000","last_fullsized":"2024-01-16T15:24:01.068006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'62","last_scrub_stamp":"2024-01-16T15:24:01.067805+0000","last_deep_scrub":"113'62","last_deep_scrub_stamp":"2024-01-16T15:24:01.067805+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.067805+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:02.835098+0000","scrub_duration":0.23695159800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'57","reported_seq":88,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:29.592883+0000","last_change":"2024-01-16T15:15:07.576715+0000","last_active":"2024-01-16T15:23:29.592883+0000","last_peered":"2024-01-16T15:23:29.592883+0000","last_clean":"2024-01-16T15:23:29.592883+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:23:29.592883+0000","last_undegraded":"2024-01-16T15:23:29.592883+0000","last_fullsized":"2024-01-16T15:23:29.592883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'104","reported_seq":147,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.487580+0000","last_change":"2024-01-16T15:24:04.487638+0000","last_active":"2024-01-16T15:24:04.487580+0000","last_peered":"2024-01-16T15:24:04.487580+0000","last_clean":"2024-01-16T15:24:04.487580+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:24:04.487580+0000","last_undegraded":"2024-01-16T15:24:04.487580+0000","last_fullsized":"2024-01-16T15:24:04.487580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'104","last_scrub_stamp":"2024-01-16T15:24:04.487387+0000","last_deep_scrub":"113'104","last_deep_scrub_stamp":"2024-01-16T15:24:04.487387+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.487387+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:47.662326+0000","scrub_duration":0.30441336299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'106","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.282980+0000","last_change":"2024-01-16T15:23:34.283133+0000","last_active":"2024-01-16T15:23:34.282980+0000","last_peered":"2024-01-16T15:23:34.282980+0000","last_clean":"2024-01-16T15:23:34.282980+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:34.282980+0000","last_undegraded":"2024-01-16T15:23:34.282980+0000","last_fullsized":"2024-01-16T15:23:34.282980+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'106","last_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_deep_scrub":"113'106","last_deep_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.282804+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:08.354546+0000","scrub_duration":0.30496924800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'82","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.785613+0000","last_change":"2024-01-16T15:23:42.785799+0000","last_active":"2024-01-16T15:23:42.785613+0000","last_peered":"2024-01-16T15:23:42.785613+0000","last_clean":"2024-01-16T15:23:42.785613+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:42.785613+0000","last_undegraded":"2024-01-16T15:23:42.785613+0000","last_fullsized":"2024-01-16T15:23:42.785613+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'82","last_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_deep_scrub":"113'82","last_deep_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.785476+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:05.117297+0000","scrub_duration":0.269243968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'88","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:58.804502+0000","last_change":"2024-01-16T15:23:58.804545+0000","last_active":"2024-01-16T15:23:58.804502+0000","last_peered":"2024-01-16T15:23:58.804502+0000","last_clean":"2024-01-16T15:23:58.804502+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:58.804502+0000","last_undegraded":"2024-01-16T15:23:58.804502+0000","last_fullsized":"2024-01-16T15:23:58.804502+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:23:58.804380+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:23:58.804380+0000","last_clean_scrub_stamp":"2024-01-16T15:23:58.804380+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:45.655289+0000","scrub_duration":0.30971083900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'6","reported_seq":33,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.264808+0000","last_change":"2024-01-16T15:23:59.264869+0000","last_active":"2024-01-16T15:23:59.264808+0000","last_peered":"2024-01-16T15:23:59.264808+0000","last_clean":"2024-01-16T15:23:59.264808+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:23:59.264808+0000","last_undegraded":"2024-01-16T15:23:59.264808+0000","last_fullsized":"2024-01-16T15:23:59.264808+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-01-16T15:23:59.264614+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-01-16T15:23:59.264614+0000","last_clean_scrub_stamp":"2024-01-16T15:23:59.264614+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:25:05.701638+0000","scrub_duration":0.0087135949999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'73","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.352331+0000","last_change":"2024-01-16T15:23:49.352425+0000","last_active":"2024-01-16T15:23:49.352331+0000","last_peered":"2024-01-16T15:23:49.352331+0000","last_clean":"2024-01-16T15:23:49.352331+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:49.352331+0000","last_undegraded":"2024-01-16T15:23:49.352331+0000","last_fullsized":"2024-01-16T15:23:49.352331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'73","last_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_deep_scrub":"113'73","last_deep_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.352155+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:06:37.591834+0000","scrub_duration":0.23652007799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.752037+0000","last_change":"2024-01-16T15:23:36.752093+0000","last_active":"2024-01-16T15:23:36.752037+0000","last_peered":"2024-01-16T15:23:36.752037+0000","last_clean":"2024-01-16T15:23:36.752037+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:23:36.752037+0000","last_undegraded":"2024-01-16T15:23:36.752037+0000","last_fullsized":"2024-01-16T15:23:36.752037+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.751793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:04.897513+0000","scrub_duration":0.0034892019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'96","reported_seq":140,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.838614+0000","last_change":"2024-01-16T15:23:47.838729+0000","last_active":"2024-01-16T15:23:47.838614+0000","last_peered":"2024-01-16T15:23:47.838614+0000","last_clean":"2024-01-16T15:23:47.838614+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:47.838614+0000","last_undegraded":"2024-01-16T15:23:47.838614+0000","last_fullsized":"2024-01-16T15:23:47.838614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.838456+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:01.112625+0000","scrub_duration":0.318651024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.965524+0000","last_change":"2024-01-16T15:23:33.965582+0000","last_active":"2024-01-16T15:23:33.965524+0000","last_peered":"2024-01-16T15:23:33.965524+0000","last_clean":"2024-01-16T15:23:33.965524+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:23:33.965524+0000","last_undegraded":"2024-01-16T15:23:33.965524+0000","last_fullsized":"2024-01-16T15:23:33.965524+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.965351+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:58.186239+0000","scrub_duration":0.0074428489999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'89","reported_seq":135,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.285933+0000","last_change":"2024-01-16T15:24:00.286123+0000","last_active":"2024-01-16T15:24:00.285933+0000","last_peered":"2024-01-16T15:24:00.285933+0000","last_clean":"2024-01-16T15:24:00.285933+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:24:00.285933+0000","last_undegraded":"2024-01-16T15:24:00.285933+0000","last_fullsized":"2024-01-16T15:24:00.285933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:24:00.285640+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:24:00.285640+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.285640+0000","objects_scrubbed":21,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:33.346013+0000","scrub_duration":0.25024972699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'3","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.098394+0000","last_change":"2024-01-16T15:23:49.098456+0000","last_active":"2024-01-16T15:23:49.098394+0000","last_peered":"2024-01-16T15:23:49.098394+0000","last_clean":"2024-01-16T15:23:49.098394+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:23:49.098394+0000","last_undegraded":"2024-01-16T15:23:49.098394+0000","last_fullsized":"2024-01-16T15:23:49.098394+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.098224+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:34.500097+0000","scrub_duration":0.0069822110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'95","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.383122+0000","last_change":"2024-01-16T15:24:05.383213+0000","last_active":"2024-01-16T15:24:05.383122+0000","last_peered":"2024-01-16T15:24:05.383122+0000","last_clean":"2024-01-16T15:24:05.383122+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:24:05.383122+0000","last_undegraded":"2024-01-16T15:24:05.383122+0000","last_fullsized":"2024-01-16T15:24:05.383122+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'95","last_scrub_stamp":"2024-01-16T15:24:05.383025+0000","last_deep_scrub":"113'95","last_deep_scrub_stamp":"2024-01-16T15:24:05.383025+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.383025+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:09.216054+0000","scrub_duration":0.29273136199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:54.176509+0000","last_change":"2024-01-16T15:23:54.176563+0000","last_active":"2024-01-16T15:23:54.176509+0000","last_peered":"2024-01-16T15:23:54.176509+0000","last_clean":"2024-01-16T15:23:54.176509+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:23:54.176509+0000","last_undegraded":"2024-01-16T15:23:54.176509+0000","last_fullsized":"2024-01-16T15:23:54.176509+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:54.176314+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:54.176314+0000","last_clean_scrub_stamp":"2024-01-16T15:23:54.176314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:35.216255+0000","scrub_duration":0.0044258329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'97","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.126742+0000","last_change":"2024-01-16T15:24:04.126802+0000","last_active":"2024-01-16T15:24:04.126742+0000","last_peered":"2024-01-16T15:24:04.126742+0000","last_clean":"2024-01-16T15:24:04.126742+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:24:04.126742+0000","last_undegraded":"2024-01-16T15:24:04.126742+0000","last_fullsized":"2024-01-16T15:24:04.126742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'97","last_scrub_stamp":"2024-01-16T15:24:04.126566+0000","last_deep_scrub":"113'97","last_deep_scrub_stamp":"2024-01-16T15:24:04.126566+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.126566+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:25.214136+0000","scrub_duration":0.30058973999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.864743+0000","last_change":"2024-01-16T15:23:31.864779+0000","last_active":"2024-01-16T15:23:31.864743+0000","last_peered":"2024-01-16T15:23:31.864743+0000","last_clean":"2024-01-16T15:23:31.864743+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:23:31.864743+0000","last_undegraded":"2024-01-16T15:23:31.864743+0000","last_fullsized":"2024-01-16T15:23:31.864743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.864652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:25.530956+0000","scrub_duration":0.00281139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'96","reported_seq":136,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.339744+0000","last_change":"2024-01-16T15:24:01.339827+0000","last_active":"2024-01-16T15:24:01.339744+0000","last_peered":"2024-01-16T15:24:01.339744+0000","last_clean":"2024-01-16T15:24:01.339744+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:24:01.339744+0000","last_undegraded":"2024-01-16T15:24:01.339744+0000","last_fullsized":"2024-01-16T15:24:01.339744+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:24:01.339583+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:24:01.339583+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.339583+0000","objects_scrubbed":23,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:38.682058+0000","scrub_duration":0.27514720100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.896191+0000","last_change":"2024-01-16T15:23:35.896238+0000","last_active":"2024-01-16T15:23:35.896191+0000","last_peered":"2024-01-16T15:23:35.896191+0000","last_clean":"2024-01-16T15:23:35.896191+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:23:35.896191+0000","last_undegraded":"2024-01-16T15:23:35.896191+0000","last_fullsized":"2024-01-16T15:23:35.896191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.896004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:41.978043+0000","scrub_duration":0.003846173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'101","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.579448+0000","last_change":"2024-01-16T15:24:03.579511+0000","last_active":"2024-01-16T15:24:03.579448+0000","last_peered":"2024-01-16T15:24:03.579448+0000","last_clean":"2024-01-16T15:24:03.579448+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:24:03.579448+0000","last_undegraded":"2024-01-16T15:24:03.579448+0000","last_fullsized":"2024-01-16T15:24:03.579448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'101","last_scrub_stamp":"2024-01-16T15:24:03.579285+0000","last_deep_scrub":"113'101","last_deep_scrub_stamp":"2024-01-16T15:24:03.579285+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.579285+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:38.545052+0000","scrub_duration":0.35447070899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.039548+0000","last_change":"2024-01-16T15:23:52.039594+0000","last_active":"2024-01-16T15:23:52.039548+0000","last_peered":"2024-01-16T15:23:52.039548+0000","last_clean":"2024-01-16T15:23:52.039548+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:23:52.039548+0000","last_undegraded":"2024-01-16T15:23:52.039548+0000","last_fullsized":"2024-01-16T15:23:52.039548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:52.039362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:52.039362+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.039362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:03.966788+0000","scrub_duration":0.0035075739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'76","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.418427+0000","last_change":"2024-01-16T15:24:08.418515+0000","last_active":"2024-01-16T15:24:08.418427+0000","last_peered":"2024-01-16T15:24:08.418427+0000","last_clean":"2024-01-16T15:24:08.418427+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:24:08.418427+0000","last_undegraded":"2024-01-16T15:24:08.418427+0000","last_fullsized":"2024-01-16T15:24:08.418427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'76","last_scrub_stamp":"2024-01-16T15:24:08.418256+0000","last_deep_scrub":"113'76","last_deep_scrub_stamp":"2024-01-16T15:24:08.418256+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.418256+0000","objects_scrubbed":18,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:44.737350+0000","scrub_duration":0.238497493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.081149+0000","last_change":"2024-01-16T15:23:42.081324+0000","last_active":"2024-01-16T15:23:42.081149+0000","last_peered":"2024-01-16T15:23:42.081149+0000","last_clean":"2024-01-16T15:23:42.081149+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:23:42.081149+0000","last_undegraded":"2024-01-16T15:23:42.081149+0000","last_fullsized":"2024-01-16T15:23:42.081149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.080895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:49.273175+0000","scrub_duration":0.0037055339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'78","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.281779+0000","last_change":"2024-01-16T15:23:47.281905+0000","last_active":"2024-01-16T15:23:47.281779+0000","last_peered":"2024-01-16T15:23:47.281779+0000","last_clean":"2024-01-16T15:23:47.281779+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:47.281779+0000","last_undegraded":"2024-01-16T15:23:47.281779+0000","last_fullsized":"2024-01-16T15:23:47.281779+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.281611+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:50.119438+0000","scrub_duration":0.19022307299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.832184+0000","last_change":"2024-01-16T15:23:35.832238+0000","last_active":"2024-01-16T15:23:35.832184+0000","last_peered":"2024-01-16T15:23:35.832184+0000","last_clean":"2024-01-16T15:23:35.832184+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:23:35.832184+0000","last_undegraded":"2024-01-16T15:23:35.832184+0000","last_fullsized":"2024-01-16T15:23:35.832184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.831980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:16.763641+0000","scrub_duration":0.004520446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'88","reported_seq":130,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.143739+0000","last_change":"2024-01-16T15:24:00.143798+0000","last_active":"2024-01-16T15:24:00.143739+0000","last_peered":"2024-01-16T15:24:00.143739+0000","last_clean":"2024-01-16T15:24:00.143739+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:24:00.143739+0000","last_undegraded":"2024-01-16T15:24:00.143739+0000","last_fullsized":"2024-01-16T15:24:00.143739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:24:00.143536+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:24:00.143536+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.143536+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:55.657562+0000","scrub_duration":0.29741432699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'33","reported_seq":59,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:54.078478+0000","last_change":"2024-01-16T15:23:54.078515+0000","last_active":"2024-01-16T15:23:54.078478+0000","last_peered":"2024-01-16T15:23:54.078478+0000","last_clean":"2024-01-16T15:23:54.078478+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:23:54.078478+0000","last_undegraded":"2024-01-16T15:23:54.078478+0000","last_fullsized":"2024-01-16T15:23:54.078478+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2024-01-16T15:23:54.078388+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2024-01-16T15:23:54.078388+0000","last_clean_scrub_stamp":"2024-01-16T15:23:54.078388+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:31.766859+0000","scrub_duration":0.0055228780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'96","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.496888+0000","last_change":"2024-01-16T15:24:06.496928+0000","last_active":"2024-01-16T15:24:06.496888+0000","last_peered":"2024-01-16T15:24:06.496888+0000","last_clean":"2024-01-16T15:24:06.496888+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:24:06.496888+0000","last_undegraded":"2024-01-16T15:24:06.496888+0000","last_fullsized":"2024-01-16T15:24:06.496888+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:24:06.496760+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:24:06.496760+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.496760+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:21.493369+0000","scrub_duration":0.35910446899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'323","reported_seq":349,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.951958+0000","last_change":"2024-01-16T15:23:34.952024+0000","last_active":"2024-01-16T15:23:34.951958+0000","last_peered":"2024-01-16T15:23:34.951958+0000","last_clean":"2024-01-16T15:23:34.951958+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:23:34.951958+0000","last_undegraded":"2024-01-16T15:23:34.951958+0000","last_fullsized":"2024-01-16T15:23:34.951958+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'323","last_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_deep_scrub":"113'323","last_deep_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.951788+0000","objects_scrubbed":4,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:08.012160+0000","scrub_duration":0.019282695999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1221104,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'75","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.087596+0000","last_change":"2024-01-16T15:23:52.087701+0000","last_active":"2024-01-16T15:23:52.087596+0000","last_peered":"2024-01-16T15:23:52.087596+0000","last_clean":"2024-01-16T15:23:52.087596+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:52.087596+0000","last_undegraded":"2024-01-16T15:23:52.087596+0000","last_fullsized":"2024-01-16T15:23:52.087596+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'75","last_scrub_stamp":"2024-01-16T15:23:52.087437+0000","last_deep_scrub":"113'75","last_deep_scrub_stamp":"2024-01-16T15:23:52.087437+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.087437+0000","objects_scrubbed":21,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:13.381080+0000","scrub_duration":0.22988671899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:32.491378+0000","last_change":"2024-01-16T15:23:32.491489+0000","last_active":"2024-01-16T15:23:32.491378+0000","last_peered":"2024-01-16T15:23:32.491378+0000","last_clean":"2024-01-16T15:23:32.491378+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:23:32.491378+0000","last_undegraded":"2024-01-16T15:23:32.491378+0000","last_fullsized":"2024-01-16T15:23:32.491378+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_clean_scrub_stamp":"2024-01-16T15:23:32.491197+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:41.739634+0000","scrub_duration":0.0079106109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'98","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:56.080417+0000","last_change":"2024-01-16T15:23:56.080477+0000","last_active":"2024-01-16T15:23:56.080417+0000","last_peered":"2024-01-16T15:23:56.080417+0000","last_clean":"2024-01-16T15:23:56.080417+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:56.080417+0000","last_undegraded":"2024-01-16T15:23:56.080417+0000","last_fullsized":"2024-01-16T15:23:56.080417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'98","last_scrub_stamp":"2024-01-16T15:23:56.080232+0000","last_deep_scrub":"113'98","last_deep_scrub_stamp":"2024-01-16T15:23:56.080232+0000","last_clean_scrub_stamp":"2024-01-16T15:23:56.080232+0000","objects_scrubbed":26,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:59.032001+0000","scrub_duration":0.28766982299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.223333+0000","last_change":"2024-01-16T15:23:34.223383+0000","last_active":"2024-01-16T15:23:34.223333+0000","last_peered":"2024-01-16T15:23:34.223333+0000","last_clean":"2024-01-16T15:23:34.223333+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:23:34.223333+0000","last_undegraded":"2024-01-16T15:23:34.223333+0000","last_fullsized":"2024-01-16T15:23:34.223333+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.223138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:21.983148+0000","scrub_duration":0.003763078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'89","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.172463+0000","last_change":"2024-01-16T15:23:50.172521+0000","last_active":"2024-01-16T15:23:50.172463+0000","last_peered":"2024-01-16T15:23:50.172463+0000","last_clean":"2024-01-16T15:23:50.172463+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:50.172463+0000","last_undegraded":"2024-01-16T15:23:50.172463+0000","last_fullsized":"2024-01-16T15:23:50.172463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:23:50.172295+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:23:50.172295+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.172295+0000","objects_scrubbed":26,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:20.561902+0000","scrub_duration":0.32687721199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'12","reported_seq":38,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.078941+0000","last_change":"2024-01-16T15:23:45.079051+0000","last_active":"2024-01-16T15:23:45.078941+0000","last_peered":"2024-01-16T15:23:45.078941+0000","last_clean":"2024-01-16T15:23:45.078941+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:23:45.078941+0000","last_undegraded":"2024-01-16T15:23:45.078941+0000","last_fullsized":"2024-01-16T15:23:45.078941+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'12","last_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_deep_scrub":"113'12","last_deep_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.078685+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.433669+0000","scrub_duration":0.010660358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'61","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:53.287936+0000","last_change":"2024-01-16T15:23:53.287936+0000","last_active":"2024-01-16T15:23:53.287936+0000","last_peered":"2024-01-16T15:23:53.287936+0000","last_clean":"2024-01-16T15:23:53.287936+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:53.287936+0000","last_undegraded":"2024-01-16T15:23:53.287936+0000","last_fullsized":"2024-01-16T15:23:53.287936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'61","last_scrub_stamp":"2024-01-16T15:23:53.287681+0000","last_deep_scrub":"113'61","last_deep_scrub_stamp":"2024-01-16T15:23:53.287681+0000","last_clean_scrub_stamp":"2024-01-16T15:23:53.287681+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:33:33.193702+0000","scrub_duration":0.19822701700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'2","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:56.062414+0000","last_change":"2024-01-16T15:23:56.062414+0000","last_active":"2024-01-16T15:23:56.062414+0000","last_peered":"2024-01-16T15:23:56.062414+0000","last_clean":"2024-01-16T15:23:56.062414+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:23:56.062414+0000","last_undegraded":"2024-01-16T15:23:56.062414+0000","last_fullsized":"2024-01-16T15:23:56.062414+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-01-16T15:23:56.062035+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-01-16T15:23:56.062035+0000","last_clean_scrub_stamp":"2024-01-16T15:23:56.062035+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:22:13.546196+0000","scrub_duration":0.0072074310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'85","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.819161+0000","last_change":"2024-01-16T15:23:49.819161+0000","last_active":"2024-01-16T15:23:49.819161+0000","last_peered":"2024-01-16T15:23:49.819161+0000","last_clean":"2024-01-16T15:23:49.819161+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:49.819161+0000","last_undegraded":"2024-01-16T15:23:49.819161+0000","last_fullsized":"2024-01-16T15:23:49.819161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'85","last_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_deep_scrub":"113'85","last_deep_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.818959+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:41.716928+0000","scrub_duration":0.28833247000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'3","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.044424+0000","last_change":"2024-01-16T15:23:37.044424+0000","last_active":"2024-01-16T15:23:37.044424+0000","last_peered":"2024-01-16T15:23:37.044424+0000","last_clean":"2024-01-16T15:23:37.044424+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:23:37.044424+0000","last_undegraded":"2024-01-16T15:23:37.044424+0000","last_fullsized":"2024-01-16T15:23:37.044424+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.044057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:58.743730+0000","scrub_duration":0.0080528830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'75","reported_seq":115,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:51.021247+0000","last_change":"2024-01-16T15:23:51.021247+0000","last_active":"2024-01-16T15:23:51.021247+0000","last_peered":"2024-01-16T15:23:51.021247+0000","last_clean":"2024-01-16T15:23:51.021247+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:51.021247+0000","last_undegraded":"2024-01-16T15:23:51.021247+0000","last_fullsized":"2024-01-16T15:23:51.021247+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'75","last_scrub_stamp":"2024-01-16T15:23:51.020999+0000","last_deep_scrub":"113'75","last_deep_scrub_stamp":"2024-01-16T15:23:51.020999+0000","last_clean_scrub_stamp":"2024-01-16T15:23:51.020999+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:46.535025+0000","scrub_duration":0.151350661,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'2","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.204678+0000","last_change":"2024-01-16T15:23:36.204678+0000","last_active":"2024-01-16T15:23:36.204678+0000","last_peered":"2024-01-16T15:23:36.204678+0000","last_clean":"2024-01-16T15:23:36.204678+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:23:36.204678+0000","last_undegraded":"2024-01-16T15:23:36.204678+0000","last_fullsized":"2024-01-16T15:23:36.204678+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.204448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:33.212894+0000","scrub_duration":0.0074182859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'89","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.418610+0000","last_change":"2024-01-16T15:24:01.418610+0000","last_active":"2024-01-16T15:24:01.418610+0000","last_peered":"2024-01-16T15:24:01.418610+0000","last_clean":"2024-01-16T15:24:01.418610+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:24:01.418610+0000","last_undegraded":"2024-01-16T15:24:01.418610+0000","last_fullsized":"2024-01-16T15:24:01.418610+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:24:01.418314+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:24:01.418314+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.418314+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:50.120896+0000","scrub_duration":0.35096044399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.200689+0000","last_change":"2024-01-16T15:23:50.200689+0000","last_active":"2024-01-16T15:23:50.200689+0000","last_peered":"2024-01-16T15:23:50.200689+0000","last_clean":"2024-01-16T15:23:50.200689+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:23:50.200689+0000","last_undegraded":"2024-01-16T15:23:50.200689+0000","last_fullsized":"2024-01-16T15:23:50.200689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:50.200547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:50.200547+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.200547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:08.908687+0000","scrub_duration":0.0031404839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'80","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.087001+0000","last_change":"2024-01-16T15:23:45.087001+0000","last_active":"2024-01-16T15:23:45.087001+0000","last_peered":"2024-01-16T15:23:45.087001+0000","last_clean":"2024-01-16T15:23:45.087001+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:45.087001+0000","last_undegraded":"2024-01-16T15:23:45.087001+0000","last_fullsized":"2024-01-16T15:23:45.087001+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'80","last_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_deep_scrub":"113'80","last_deep_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.086751+0000","objects_scrubbed":26,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:06.380394+0000","scrub_duration":0.32344294400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'4","reported_seq":31,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.492439+0000","last_change":"2024-01-16T15:23:31.492439+0000","last_active":"2024-01-16T15:23:31.492439+0000","last_peered":"2024-01-16T15:23:31.492439+0000","last_clean":"2024-01-16T15:23:31.492439+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:23:31.492439+0000","last_undegraded":"2024-01-16T15:23:31.492439+0000","last_fullsized":"2024-01-16T15:23:31.492439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.492299+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:57.929823+0000","scrub_duration":0.0056823760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'70","reported_seq":102,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308873+0000","last_change":"2024-01-16T15:15:07.590187+0000","last_active":"2024-01-16T15:23:28.308873+0000","last_peered":"2024-01-16T15:23:28.308873+0000","last_clean":"2024-01-16T15:23:28.308873+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:23:28.308873+0000","last_undegraded":"2024-01-16T15:23:28.308873+0000","last_fullsized":"2024-01-16T15:23:28.308873+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.044075+0000","last_change":"2024-01-16T15:23:38.044075+0000","last_active":"2024-01-16T15:23:38.044075+0000","last_peered":"2024-01-16T15:23:38.044075+0000","last_clean":"2024-01-16T15:23:38.044075+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:23:38.044075+0000","last_undegraded":"2024-01-16T15:23:38.044075+0000","last_fullsized":"2024-01-16T15:23:38.044075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.043846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:00.375667+0000","scrub_duration":0.0035534389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'84","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:46.072906+0000","last_change":"2024-01-16T15:23:46.072906+0000","last_active":"2024-01-16T15:23:46.072906+0000","last_peered":"2024-01-16T15:23:46.072906+0000","last_clean":"2024-01-16T15:23:46.072906+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:46.072906+0000","last_undegraded":"2024-01-16T15:23:46.072906+0000","last_fullsized":"2024-01-16T15:23:46.072906+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_clean_scrub_stamp":"2024-01-16T15:23:46.072694+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:53:22.491262+0000","scrub_duration":0.27693848900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.532350+0000","last_change":"2024-01-16T15:23:30.532350+0000","last_active":"2024-01-16T15:23:30.532350+0000","last_peered":"2024-01-16T15:23:30.532350+0000","last_clean":"2024-01-16T15:23:30.532350+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:23:30.532350+0000","last_undegraded":"2024-01-16T15:23:30.532350+0000","last_fullsized":"2024-01-16T15:23:30.532350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.532200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:36.853744+0000","scrub_duration":0.0022174130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'77","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.010592+0000","last_change":"2024-01-16T15:23:57.010680+0000","last_active":"2024-01-16T15:23:57.010592+0000","last_peered":"2024-01-16T15:23:57.010592+0000","last_clean":"2024-01-16T15:23:57.010592+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:57.010592+0000","last_undegraded":"2024-01-16T15:23:57.010592+0000","last_fullsized":"2024-01-16T15:23:57.010592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'77","last_scrub_stamp":"2024-01-16T15:23:57.010482+0000","last_deep_scrub":"113'77","last_deep_scrub_stamp":"2024-01-16T15:23:57.010482+0000","last_clean_scrub_stamp":"2024-01-16T15:23:57.010482+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:58.858184+0000","scrub_duration":0.234133231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.996817+0000","last_change":"2024-01-16T15:23:47.996817+0000","last_active":"2024-01-16T15:23:47.996817+0000","last_peered":"2024-01-16T15:23:47.996817+0000","last_clean":"2024-01-16T15:23:47.996817+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:23:47.996817+0000","last_undegraded":"2024-01-16T15:23:47.996817+0000","last_fullsized":"2024-01-16T15:23:47.996817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.996601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:48.743854+0000","scrub_duration":0.0036748639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'67","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.245238+0000","last_change":"2024-01-16T15:23:45.245238+0000","last_active":"2024-01-16T15:23:45.245238+0000","last_peered":"2024-01-16T15:23:45.245238+0000","last_clean":"2024-01-16T15:23:45.245238+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:45.245238+0000","last_undegraded":"2024-01-16T15:23:45.245238+0000","last_fullsized":"2024-01-16T15:23:45.245238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'67","last_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_deep_scrub":"113'67","last_deep_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.244953+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:26.176614+0000","scrub_duration":0.199400785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'3","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.817445+0000","last_change":"2024-01-16T15:23:38.817445+0000","last_active":"2024-01-16T15:23:38.817445+0000","last_peered":"2024-01-16T15:23:38.817445+0000","last_clean":"2024-01-16T15:23:38.817445+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:23:38.817445+0000","last_undegraded":"2024-01-16T15:23:38.817445+0000","last_fullsized":"2024-01-16T15:23:38.817445+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.817208+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:41.655035+0000","scrub_duration":0.0067620090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'83","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.194713+0000","last_change":"2024-01-16T15:23:30.194713+0000","last_active":"2024-01-16T15:23:30.194713+0000","last_peered":"2024-01-16T15:23:30.194713+0000","last_clean":"2024-01-16T15:23:30.194713+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:30.194713+0000","last_undegraded":"2024-01-16T15:23:30.194713+0000","last_fullsized":"2024-01-16T15:23:30.194713+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'83","last_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_deep_scrub":"113'83","last_deep_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.194521+0000","objects_scrubbed":24,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:25.708546+0000","scrub_duration":0.24677183899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'32","reported_seq":59,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.130401+0000","last_change":"2024-01-16T15:23:39.130401+0000","last_active":"2024-01-16T15:23:39.130401+0000","last_peered":"2024-01-16T15:23:39.130401+0000","last_clean":"2024-01-16T15:23:39.130401+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:23:39.130401+0000","last_undegraded":"2024-01-16T15:23:39.130401+0000","last_fullsized":"2024-01-16T15:23:39.130401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'32","last_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_deep_scrub":"113'32","last_deep_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.130166+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:57.838306+0000","scrub_duration":0.0043416110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705,"num_omap_keys":13,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'69","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:28.308852+0000","last_change":"2024-01-16T15:15:07.603613+0000","last_active":"2024-01-16T15:23:28.308852+0000","last_peered":"2024-01-16T15:23:28.308852+0000","last_clean":"2024-01-16T15:23:28.308852+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:23:28.308852+0000","last_undegraded":"2024-01-16T15:23:28.308852+0000","last_fullsized":"2024-01-16T15:23:28.308852+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:06.523273+0000","last_clean_scrub_stamp":"2024-01-16T15:15:06.523273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.997068+0000","last_change":"2024-01-16T15:23:49.997120+0000","last_active":"2024-01-16T15:23:49.997068+0000","last_peered":"2024-01-16T15:23:49.997068+0000","last_clean":"2024-01-16T15:23:49.997068+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:23:49.997068+0000","last_undegraded":"2024-01-16T15:23:49.997068+0000","last_fullsized":"2024-01-16T15:23:49.997068+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:49.996799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:49.996799+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.996799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:12.037675+0000","scrub_duration":0.0049679720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'79","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.416185+0000","last_change":"2024-01-16T15:23:43.416246+0000","last_active":"2024-01-16T15:23:43.416185+0000","last_peered":"2024-01-16T15:23:43.416185+0000","last_clean":"2024-01-16T15:23:43.416185+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:43.416185+0000","last_undegraded":"2024-01-16T15:23:43.416185+0000","last_fullsized":"2024-01-16T15:23:43.416185+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.416030+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:24.455972+0000","scrub_duration":0.30055066899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.851113+0000","last_change":"2024-01-16T15:23:39.851159+0000","last_active":"2024-01-16T15:23:39.851113+0000","last_peered":"2024-01-16T15:23:39.851113+0000","last_clean":"2024-01-16T15:23:39.851113+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:23:39.851113+0000","last_undegraded":"2024-01-16T15:23:39.851113+0000","last_fullsized":"2024-01-16T15:23:39.851113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.850958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:29.841870+0000","scrub_duration":0.003185556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'84","reported_seq":118,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.117719+0000","last_change":"2024-01-16T15:24:06.117806+0000","last_active":"2024-01-16T15:24:06.117719+0000","last_peered":"2024-01-16T15:24:06.117719+0000","last_clean":"2024-01-16T15:24:06.117719+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:24:06.117719+0000","last_undegraded":"2024-01-16T15:24:06.117719+0000","last_fullsized":"2024-01-16T15:24:06.117719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:24:06.117538+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:24:06.117538+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.117538+0000","objects_scrubbed":21,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:45.913228+0000","scrub_duration":0.26484186300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.074149+0000","last_change":"2024-01-16T15:23:41.074229+0000","last_active":"2024-01-16T15:23:41.074149+0000","last_peered":"2024-01-16T15:23:41.074149+0000","last_clean":"2024-01-16T15:23:41.074149+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:23:41.074149+0000","last_undegraded":"2024-01-16T15:23:41.074149+0000","last_fullsized":"2024-01-16T15:23:41.074149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.073835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:56.198760+0000","scrub_duration":0.004400836,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'93","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.157245+0000","last_change":"2024-01-16T15:23:50.157313+0000","last_active":"2024-01-16T15:23:50.157245+0000","last_peered":"2024-01-16T15:23:50.157245+0000","last_clean":"2024-01-16T15:23:50.157245+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:50.157245+0000","last_undegraded":"2024-01-16T15:23:50.157245+0000","last_fullsized":"2024-01-16T15:23:50.157245+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'93","last_scrub_stamp":"2024-01-16T15:23:50.157078+0000","last_deep_scrub":"113'93","last_deep_scrub_stamp":"2024-01-16T15:23:50.157078+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.157078+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:20.546685+0000","scrub_duration":0.333928897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.837299+0000","last_change":"2024-01-16T15:23:33.837346+0000","last_active":"2024-01-16T15:23:33.837299+0000","last_peered":"2024-01-16T15:23:33.837299+0000","last_clean":"2024-01-16T15:23:33.837299+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:23:33.837299+0000","last_undegraded":"2024-01-16T15:23:33.837299+0000","last_fullsized":"2024-01-16T15:23:33.837299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.837111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:32.726849+0000","scrub_duration":0.0035917940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'95","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.161310+0000","last_change":"2024-01-16T15:23:47.161339+0000","last_active":"2024-01-16T15:23:47.161310+0000","last_peered":"2024-01-16T15:23:47.161310+0000","last_clean":"2024-01-16T15:23:47.161310+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:47.161310+0000","last_undegraded":"2024-01-16T15:23:47.161310+0000","last_fullsized":"2024-01-16T15:23:47.161310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'95","last_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_deep_scrub":"113'95","last_deep_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.161227+0000","objects_scrubbed":29,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:50.112996+0000","scrub_duration":0.32985411999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'34","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.113907+0000","last_change":"2024-01-16T15:23:47.114004+0000","last_active":"2024-01-16T15:23:47.113907+0000","last_peered":"2024-01-16T15:23:47.113907+0000","last_clean":"2024-01-16T15:23:47.113907+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:23:47.113907+0000","last_undegraded":"2024-01-16T15:23:47.113907+0000","last_fullsized":"2024-01-16T15:23:47.113907+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'34","last_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_deep_scrub":"113'34","last_deep_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.113654+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:00.387823+0000","scrub_duration":0.0063658849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'102","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.569176+0000","last_change":"2024-01-16T15:24:09.569241+0000","last_active":"2024-01-16T15:24:09.569176+0000","last_peered":"2024-01-16T15:24:09.569176+0000","last_clean":"2024-01-16T15:24:09.569176+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:24:09.569176+0000","last_undegraded":"2024-01-16T15:24:09.569176+0000","last_fullsized":"2024-01-16T15:24:09.569176+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'102","last_scrub_stamp":"2024-01-16T15:24:09.569013+0000","last_deep_scrub":"113'102","last_deep_scrub_stamp":"2024-01-16T15:24:09.569013+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.569013+0000","objects_scrubbed":32,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:30.862656+0000","scrub_duration":0.35738116199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.978255+0000","last_change":"2024-01-16T15:23:39.978309+0000","last_active":"2024-01-16T15:23:39.978255+0000","last_peered":"2024-01-16T15:23:39.978255+0000","last_clean":"2024-01-16T15:23:39.978255+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:23:39.978255+0000","last_undegraded":"2024-01-16T15:23:39.978255+0000","last_fullsized":"2024-01-16T15:23:39.978255+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.977987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:08.833788+0000","scrub_duration":0.0043159469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'62","reported_seq":100,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.078519+0000","last_change":"2024-01-16T15:23:41.078597+0000","last_active":"2024-01-16T15:23:41.078519+0000","last_peered":"2024-01-16T15:23:41.078519+0000","last_clean":"2024-01-16T15:23:41.078519+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:41.078519+0000","last_undegraded":"2024-01-16T15:23:41.078519+0000","last_fullsized":"2024-01-16T15:23:41.078519+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'62","last_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_deep_scrub":"113'62","last_deep_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.078271+0000","objects_scrubbed":17,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:52.687411+0000","scrub_duration":0.197239254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.012004+0000","last_change":"2024-01-16T15:23:37.012089+0000","last_active":"2024-01-16T15:23:37.012004+0000","last_peered":"2024-01-16T15:23:37.012004+0000","last_clean":"2024-01-16T15:23:37.012004+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:23:37.012004+0000","last_undegraded":"2024-01-16T15:23:37.012004+0000","last_fullsized":"2024-01-16T15:23:37.012004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.011669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:27.915725+0000","scrub_duration":0.0040625560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"113'119","reported_seq":282,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.845903+0000","last_change":"2024-01-16T15:23:34.845995+0000","last_active":"2024-01-16T15:23:34.845903+0000","last_peered":"2024-01-16T15:23:34.845903+0000","last_clean":"2024-01-16T15:23:34.845903+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:23:34.845903+0000","last_undegraded":"2024-01-16T15:23:34.845903+0000","last_fullsized":"2024-01-16T15:23:34.845903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'119","last_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_deep_scrub":"113'119","last_deep_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.845733+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:46.454874+0000","scrub_duration":0.058578549000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5425,"ondisk_log_size":5425,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1225546,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":487,"num_write_kb":1431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2506752,"data_stored":2451092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":469,"ondisk_log_size":469,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575184,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.151,"15min":1.143},"min":{"1min":0.755,"5min":0.66000000000000003,"15min":0.628},"max":{"1min":1.927,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":1.151},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.198,"15min":1.2190000000000001},"min":{"1min":0.623,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.004,"5min":2.004,"15min":5.9619999999999997},"last":1.4119999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1830000000000001,"15min":1.2549999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.69699999999999995},"max":{"1min":1.6699999999999999,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7430000000000001,"5min":1.956,"15min":6.2690000000000001},"last":1.0800000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.21,"15min":1.2090000000000001},"min":{"1min":0.67000000000000004,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.5640000000000001,"5min":2.0070000000000001,"15min":2.5910000000000002},"last":1.272},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2709999999999999,"15min":1.2689999999999999},"min":{"1min":0.95199999999999996,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.879,"5min":1.879,"15min":2.6920000000000002},"last":0.96399999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2829999999999999,"15min":1.3380000000000001},"min":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.4990000000000001,"5min":1.9279999999999999,"15min":12.331},"last":1.131},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.373,"15min":1.3560000000000001},"min":{"1min":0.90400000000000003,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":1.954,"5min":2.0649999999999999,"15min":2.645},"last":1.4590000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3220000000000001,"15min":1.3560000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.49199999999999999},"max":{"1min":2.109,"5min":2.109,"15min":6.1799999999999997},"last":1.605},{"interface":"front","average":{"1min":1.431,"5min":1.321,"15min":1.319},"min":{"1min":1.0780000000000001,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":2.4159999999999999},"last":1.0389999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.397,"15min":1.401},"min":{"1min":0.88900000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.4350000000000001,"5min":2.508,"15min":2.593},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4930000000000001,"15min":1.532},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.54700000000000004},"max":{"1min":2.2050000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.228}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.464,"15min":1.496},"min":{"1min":0.78800000000000003,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.988,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.5649999999999999,"15min":1.5860000000000001},"min":{"1min":1.032,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":4.7649999999999997},"last":1.5640000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.276,"15min":1.2430000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.631,"5min":1.6679999999999999,"15min":2.6970000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.3280000000000001,"15min":1.3009999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.27}]},{"osd":1,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.89300000000000002,"5min":0.80200000000000005,"15min":0.64700000000000002},"max":{"1min":1.3109999999999999,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":1.077},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.242,"15min":1.22},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5960000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.028}]},{"osd":2,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.331,"15min":1.296},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.628},"max":{"1min":1.6910000000000001,"5min":5.7439999999999998,"15min":5.7439999999999998},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.294,"5min":1.385,"15min":1.3580000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69799999999999995},"max":{"1min":1.675,"5min":5.5869999999999997,"15min":5.5869999999999997},"last":1.357}]},{"osd":3,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3480000000000001,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.88500000000000001,"15min":0.77800000000000002},"max":{"1min":1.661,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.502},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.6419999999999999,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.4770000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.4199999999999999,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.6639999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":1.228},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3839999999999999},"min":{"1min":0.85799999999999998,"5min":0.80100000000000005,"15min":0.48899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8440000000000001,"15min":6.7809999999999997},"last":1.3959999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.52300000000000002},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":3.2029999999999998},"last":1.587},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.5209999999999999},"min":{"1min":0.97499999999999998,"5min":0.77000000000000002,"15min":0.66600000000000004},"max":{"1min":1.77,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.554}]},{"osd":7,"last update":"Tue Jan 16 15:23:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5009999999999999,"15min":1.5840000000000001},"min":{"1min":0.97799999999999998,"5min":0.96399999999999997,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":2.4390000000000001,"15min":24.675000000000001},"last":1.758},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.51,"15min":1.5209999999999999},"min":{"1min":1.3129999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":1.952,"5min":4.6239999999999997,"15min":5.2469999999999999},"last":1.698}]}]},{"osd":5,"up_from":13,"seq":55834575184,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.179,"15min":1.159},"min":{"1min":0.628,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":1.9419999999999999,"5min":2.2040000000000002,"15min":3.7130000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.161,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.49299999999999999},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":8.7880000000000003},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":2.1389999999999998,"15min":3.835},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.292,"5min":1.3120000000000001,"15min":1.298},"min":{"1min":0.67400000000000004,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":1.95,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":1.181}]},{"osd":2,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.292,"15min":1.329},"min":{"1min":0.96299999999999997,"5min":0.69699999999999995,"15min":0.44500000000000001},"max":{"1min":1.6120000000000001,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.119},{"interface":"front","average":{"1min":1.351,"5min":1.373,"15min":1.3720000000000001},"min":{"1min":1.0209999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.673,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.6439999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.325,"15min":1.3400000000000001},"min":{"1min":0.86599999999999999,"5min":0.75600000000000001,"15min":0.38200000000000001},"max":{"1min":1.6719999999999999,"5min":1.742,"15min":3.9319999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.86699999999999999,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":16.158000000000001},"last":1.5449999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.278,"15min":1.238},"min":{"1min":0.91000000000000003,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":1.8009999999999999,"5min":2.1240000000000001,"15min":6.1660000000000004},"last":1.319},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.488,"15min":1.4910000000000001},"min":{"1min":1.0920000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.887,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.282}]},{"osd":6,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":1.117,"5min":0.878,"15min":0.46800000000000003},"max":{"1min":1.857,"5min":2.3679999999999999,"15min":7.6669999999999998},"last":1.071},{"interface":"front","average":{"1min":1.363,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.89100000000000001,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":2.351,"5min":2.351,"15min":6.093},"last":1.4330000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:23:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4430000000000001,"15min":1.462},"min":{"1min":1.079,"5min":0.54700000000000004,"15min":0.41499999999999998},"max":{"1min":2.145,"5min":2.145,"15min":3.8809999999999998},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.556,"15min":1.522},"min":{"1min":1.141,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.456,"5min":2.456,"15min":5.0209999999999999},"last":1.7350000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575183,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932272,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439568,"statfs":{"total":96636764160,"available":94658117632,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.3680000000000001},"min":{"1min":0.628,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.6579999999999999,"5min":1.73,"15min":17.315999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.395,"15min":1.427},"min":{"1min":0.753,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.7,"5min":1.7949999999999999,"15min":17.279},"last":1.252}]},{"osd":1,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.226,"15min":1.266},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.498},"max":{"1min":1.3720000000000001,"5min":1.839,"15min":6.1849999999999996},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.2230000000000001,"15min":1.1830000000000001},"min":{"1min":0.87,"5min":0.80000000000000004,"15min":0.53700000000000003},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":1.0760000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1990000000000001,"15min":1.1870000000000001},"min":{"1min":0.90500000000000003,"5min":0.88300000000000001,"15min":0.5},"max":{"1min":1.6479999999999999,"5min":1.6830000000000001,"15min":2.1299999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.365,"15min":1.3460000000000001},"min":{"1min":1.0429999999999999,"5min":0.88600000000000001,"15min":0.75800000000000001},"max":{"1min":1.645,"5min":1.8120000000000001,"15min":2.4780000000000002},"last":1.157}]},{"osd":3,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.266,"15min":1.3069999999999999},"min":{"1min":0.96499999999999997,"5min":0.85699999999999998,"15min":0.58199999999999996},"max":{"1min":1.667,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3740000000000001,"15min":1.3560000000000001},"min":{"1min":1.056,"5min":0.95999999999999996,"15min":0.65300000000000002},"max":{"1min":1.77,"5min":1.77,"15min":4.7930000000000001},"last":1.3540000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.276,"15min":1.3049999999999999},"min":{"1min":0.98599999999999999,"5min":0.69999999999999996,"15min":0.63},"max":{"1min":1.54,"5min":1.8999999999999999,"15min":2.597},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.335,"15min":1.333},"min":{"1min":1.0069999999999999,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.7929999999999999,"5min":1.8660000000000001,"15min":4.8520000000000003},"last":0.97099999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":1.004,"5min":0.89700000000000002,"15min":0.57099999999999995},"max":{"1min":1.8420000000000001,"5min":2.0230000000000001,"15min":2.3479999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4810000000000001,"15min":1.52},"min":{"1min":1.0940000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7529999999999999,"5min":1.946,"15min":12.324999999999999},"last":1.3839999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:24 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4610000000000001,"15min":1.466},"min":{"1min":1.0860000000000001,"5min":1.032,"15min":0.53600000000000003},"max":{"1min":1.839,"5min":1.968,"15min":4.9539999999999997},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.5649999999999999,"15min":1.534},"min":{"1min":1.296,"5min":1.0760000000000001,"15min":0.64700000000000002},"max":{"1min":1.903,"5min":1.9039999999999999,"15min":2.2599999999999998},"last":1.4039999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575184,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1822956,"kb_used_data":1751588,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92548884,"statfs":{"total":96636764160,"available":94770057216,"internally_reserved":0,"allocated":1793626112,"data_stored":1792798405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94499999999999995,"15min":0.91600000000000004},"min":{"1min":0.34599999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.48,"5min":1.6240000000000001,"15min":1.925},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.097,"15min":1.03},"min":{"1min":0.45100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.377,"5min":6.6740000000000004,"15min":6.6740000000000004},"last":0.72199999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.93500000000000005,"15min":0.98599999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.44900000000000001},"max":{"1min":1.3140000000000001,"5min":1.6220000000000001,"15min":3.1150000000000002},"last":0.872},{"interface":"front","average":{"1min":1.056,"5min":1.105,"15min":1.077},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.38500000000000001},"max":{"1min":1.385,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.286}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1240000000000001,"15min":1.1319999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5609999999999999,"5min":1.8169999999999999,"15min":3.0670000000000002},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1919999999999999,"15min":1.204},"min":{"1min":0.55000000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.605,"5min":1.986,"15min":3.4279999999999999},"last":1.4099999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3540000000000001,"15min":1.343},"min":{"1min":0.84699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.5049999999999999},"last":1.244},{"interface":"front","average":{"1min":1.367,"5min":1.3859999999999999,"15min":1.391},"min":{"1min":1.0920000000000001,"5min":0.85499999999999998,"15min":0.64000000000000001},"max":{"1min":1.718,"5min":1.9810000000000001,"15min":3.8250000000000002},"last":1.4950000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3480000000000001,"15min":1.3440000000000001},"min":{"1min":0.88300000000000001,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.6240000000000001,"5min":2.085,"15min":2.3540000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.365,"15min":1.3939999999999999},"min":{"1min":0.80500000000000005,"5min":0.79500000000000004,"15min":0.71299999999999997},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.907},"last":1.45}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3939999999999999,"15min":1.4379999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.8560000000000001,"5min":2.0569999999999999,"15min":6.4480000000000004},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4259999999999999,"15min":1.4770000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.81299999999999994},"max":{"1min":1.7869999999999999,"5min":2.1539999999999999,"15min":3.9039999999999999},"last":1.1779999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.73099999999999998},"max":{"1min":1.778,"5min":1.9370000000000001,"15min":3.7850000000000001},"last":1.349},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.482,"15min":1.5169999999999999},"min":{"1min":1.0429999999999999,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":1.891,"5min":2.085,"15min":3.9590000000000001},"last":1.3220000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575183,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.90000000000000002,"15min":0.88900000000000001},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.314},"max":{"1min":1.603,"5min":1.603,"15min":2.8919999999999999},"last":1.05},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.024,"15min":1.036},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.42299999999999999},"max":{"1min":1.4350000000000001,"5min":1.76,"15min":2.952},"last":0.85199999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.034,"15min":1.0569999999999999},"min":{"1min":0.58799999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.5820000000000001,"5min":1.605,"15min":1.823},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.127},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3600000000000001,"5min":1.6479999999999999,"15min":4.734},"last":0.97899999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.028,"15min":1.0509999999999999},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.403,"5min":1.788,"15min":1.788},"last":1.238},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":0.63700000000000001,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.5660000000000001,"5min":1.8120000000000001,"15min":2.3220000000000001},"last":1.3620000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.335,"15min":1.4770000000000001},"min":{"1min":0.84699999999999998,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":1.5660000000000001,"5min":2.0680000000000001,"15min":16.681999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.292,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.746},"max":{"1min":1.544,"5min":1.919,"15min":1.964},"last":1.448}]},{"osd":5,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.97199999999999998,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":1.879,"5min":2.206,"15min":2.3799999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.4039999999999999,"15min":1.431},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":4.8659999999999997},"last":1.1359999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.373,"15min":1.3879999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.986,"5min":2.0720000000000001,"15min":2.1539999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4710000000000001,"15min":1.484},"min":{"1min":1.022,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":1.887,"5min":2.1080000000000001,"15min":6.2599999999999998},"last":1.492}]},{"osd":7,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.472,"15min":1.49},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.70099999999999996},"max":{"1min":1.7010000000000001,"5min":2.125,"15min":3.7530000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.4379999999999999,"15min":1.45},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.78400000000000003},"max":{"1min":1.6950000000000001,"5min":2.0670000000000002,"15min":2.294},"last":1.526}]}]},{"osd":1,"up_from":13,"seq":55834575184,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1403888,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92967952,"statfs":{"total":96636764160,"available":95199182848,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.91200000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.399,"5min":1.4330000000000001,"15min":14.106999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.03,"15min":1.0569999999999999},"min":{"1min":0.47899999999999998,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.504,"5min":1.504,"15min":4.3719999999999999},"last":1.117}]},{"osd":2,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.96299999999999997,"15min":0.96499999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.29799999999999999},"max":{"1min":1.2709999999999999,"5min":3.5409999999999999,"15min":3.6309999999999998},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.995,"5min":1.0740000000000001,"15min":1.1339999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.3720000000000001,"5min":1.663,"15min":14.055},"last":0.89400000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.22,"15min":1.266},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.48,"5min":1.6439999999999999,"15min":13.967000000000001},"last":0.997},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1060000000000001,"15min":1.135},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5840000000000001,"5min":1.952,"15min":4.6699999999999999},"last":1.042}]},{"osd":4,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.228,"15min":1.244},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52500000000000002},"max":{"1min":1.698,"5min":3.77,"15min":4.3799999999999999},"last":1.242},{"interface":"front","average":{"1min":1.379,"5min":1.3540000000000001,"15min":1.4650000000000001},"min":{"1min":0.89700000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":14.161},"last":1.091}]},{"osd":5,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.5369999999999999},"min":{"1min":1.0289999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7450000000000001,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.413,"15min":1.508},"min":{"1min":1.079,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.8460000000000001,"5min":3.8090000000000002,"15min":16.433},"last":1.367}]},{"osd":6,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4390000000000001,"15min":1.4930000000000001},"min":{"1min":1.1100000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":14.212},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3979999999999999,"15min":1.444},"min":{"1min":0.872,"5min":0.872,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":1.2}]},{"osd":7,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.431,"15min":1.5209999999999999},"min":{"1min":0.94099999999999995,"5min":0.82499999999999996,"15min":0.78100000000000003},"max":{"1min":1.8460000000000001,"5min":2.1930000000000001,"15min":12.590999999999999},"last":1.333},{"interface":"front","average":{"1min":1.373,"5min":1.4530000000000001,"15min":1.454},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.64100000000000001},"max":{"1min":1.7829999999999999,"5min":3.726,"15min":6.0229999999999997},"last":1.4239999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575183,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048744,"kb_used_data":1981536,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323096,"statfs":{"total":96636764160,"available":94538850304,"internally_reserved":0,"allocated":2029092864,"data_stored":2028268228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.83499999999999996,"15min":0.84199999999999997},"min":{"1min":0.621,"5min":0.46899999999999997,"15min":0.28799999999999998},"max":{"1min":1.3240000000000001,"5min":1.601,"15min":1.921},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0449999999999999,"15min":1.01},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.41799999999999998},"max":{"1min":1.653,"5min":1.72,"15min":4.4059999999999997},"last":1.2629999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0109999999999999,"15min":1.0649999999999999},"min":{"1min":0.80200000000000005,"5min":0.60599999999999998,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":1.6220000000000001,"15min":17.260000000000002},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.115,"15min":1.0980000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.33600000000000002},"max":{"1min":1.492,"5min":1.784,"15min":4.492},"last":1.5169999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1180000000000001,"15min":1.087},"min":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.33100000000000002},"max":{"1min":1.538,"5min":1.706,"15min":3.4670000000000001},"last":1.429},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.325,"15min":1.2869999999999999},"min":{"1min":0.83799999999999997,"5min":0.63600000000000001,"15min":0.41699999999999998},"max":{"1min":1.5840000000000001,"5min":1.8400000000000001,"15min":2.319},"last":1.466}]},{"osd":4,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.337,"15min":1.3460000000000001},"min":{"1min":1.0860000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":4.2889999999999997},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3360000000000001,"15min":1.3100000000000001},"min":{"1min":0.93000000000000005,"5min":0.83799999999999997,"15min":0.67800000000000005},"max":{"1min":4.0810000000000004,"5min":4.0810000000000004,"15min":4.0810000000000004},"last":1.595}]},{"osd":5,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4259999999999999,"15min":1.4159999999999999},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.624},"max":{"1min":2.1850000000000001,"5min":2.222,"15min":3.1259999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4950000000000001,"15min":1.486},"min":{"1min":1.0129999999999999,"5min":0.66800000000000004,"15min":0.60699999999999998},"max":{"1min":2.1389999999999998,"5min":2.2730000000000001,"15min":6.5430000000000001},"last":1.6850000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.397,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.91500000000000004,"15min":0.65700000000000003},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":11.462},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4419999999999999,"15min":1.4570000000000001},"min":{"1min":1.0189999999999999,"5min":0.92600000000000005,"15min":0.80100000000000005},"max":{"1min":1.992,"5min":2.093,"15min":7.1849999999999996},"last":1.5589999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.466,"15min":1.502},"min":{"1min":0.86799999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.9399999999999999,"5min":2.1299999999999999,"15min":5.2450000000000001},"last":2.181},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.514,"15min":1.4750000000000001},"min":{"1min":1.123,"5min":0.95299999999999996,"15min":0.68799999999999994},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.6619999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":1221216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1232896,"data_stored":1221174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:24:14.768 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.768 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.768 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.769 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.770 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.770 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:15:06.523273+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=6, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:14.770 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:24:15.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:15.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:34.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:24:34.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.993+0000 7f2b3cc00640 1 Processor -- start 2024-01-16T15:24:34.991 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.993+0000 7f2b3cc00640 1 -- start start 2024-01-16T15:24:34.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813df80 0x7f2b3815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:34.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2b3813e360 0x7f2b380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:34.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b3815f9c0 0x7f2b380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:34.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3806c470 con 0x7f2b3815f9c0 2024-01-16T15:24:34.993 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3806c750 con 0x7f2b3813e360 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.994+0000 7f2b3cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3806ca30 con 0x7f2b3813df80 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b3815f9c0 0x7f2b380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b3815f9c0 0x7f2b380cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57898/0 (socket says 172.21.15.125:57898) 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 -- 172.21.15.125:0/2816385499 learned_addr learned my addr 172.21.15.125:0/2816385499 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b35400640 1 --2- 172.21.15.125:0/2816385499 >> v2:172.21.15.137:3300/0 conn(0x7f2b3813e360 0x7f2b380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b35e00640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813df80 0x7f2b3815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 -- 172.21.15.125:0/2816385499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813df80 msgr2=0x7f2b3815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:34.994 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813df80 0x7f2b3815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 -- 172.21.15.125:0/2816385499 >> v2:172.21.15.137:3300/0 conn(0x7f2b3813e360 msgr2=0x7f2b380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:34.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 --2- 172.21.15.125:0/2816385499 >> v2:172.21.15.137:3300/0 conn(0x7f2b3813e360 0x7f2b380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.995 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.995+0000 7f2b36800640 1 -- 172.21.15.125:0/2816385499 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3806cd10 con 0x7f2b3815f9c0 2024-01-16T15:24:34.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.996+0000 7f2b36800640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3815f9c0 0x7f2b380cf7d0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f2b380cfed0 tx=0x7f2b3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d0fc144b67c1d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:34.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b34a00640 1 -- 172.21.15.125:0/2816385499 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3002ca70 con 0x7f2b3815f9c0 2024-01-16T15:24:34.996 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b34a00640 1 -- 172.21.15.125:0/2816385499 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3002cbd0 con 0x7f2b3815f9c0 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b34a00640 1 -- 172.21.15.125:0/2816385499 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b300343f0 con 0x7f2b3815f9c0 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2816385499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3815f9c0 msgr2=0x7f2b380cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3815f9c0 0x7f2b380cf7d0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f2b380cfed0 tx=0x7f2b3002bed0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2816385499 shutdown_connections 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3815f9c0 0x7f2b380cf7d0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2816385499 >> v2:172.21.15.137:3300/0 conn(0x7f2b3813e360 0x7f2b380cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2816385499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813df80 0x7f2b3815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:34.997 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.997+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2816385499 >> 172.21.15.125:0/2816385499 conn(0x7f2b3806a800 msgr2=0x7f2b3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:34.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.998+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2816385499 shutdown_connections 2024-01-16T15:24:34.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:34.998+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2816385499 wait complete. 2024-01-16T15:24:34.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.000+0000 7f2b3cc00640 1 Processor -- start 2024-01-16T15:24:34.998 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.000+0000 7f2b3cc00640 1 -- start start 2024-01-16T15:24:35.000 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813e360 0x7f2b38157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 0x7f2b38158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380ae340 con 0x7f2b3813df80 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380b2cf0 con 0x7f2b3815f9c0 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b3cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381f0c80 con 0x7f2b3813e360 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b35e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.001 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.001+0000 7f2b35e00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57908/0 (socket says 172.21.15.125:57908) 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 -- 172.21.15.125:0/2856261852 learned_addr learned my addr 172.21.15.125:0/2856261852 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35400640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813e360 0x7f2b38157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b36800640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 0x7f2b38158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 -- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813e360 msgr2=0x7f2b38157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813e360 0x7f2b38157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 -- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 msgr2=0x7f2b38158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 0x7f2b38158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.002 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 -- 172.21.15.125:0/2856261852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b30009b30 con 0x7f2b3813df80 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b36800640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 0x7f2b38158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.002+0000 7f2b35e00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800c7f0 tx=0x7f2b2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.003+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28004220 con 0x7f2b3813df80 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.003+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b28004380 con 0x7f2b3813df80 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.003+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28007840 con 0x7f2b3813df80 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.003+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b380b2f90 con 0x7f2b3813df80 2024-01-16T15:24:35.003 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.003+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b380b3550 con 0x7f2b3813df80 2024-01-16T15:24:35.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.004+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b380675d0 con 0x7f2b3813df80 2024-01-16T15:24:35.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.005+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 51) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2b28007bb0 con 0x7f2b3813df80 2024-01-16T15:24:35.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.006+0000 7f2b2e000640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 0x7f2b1006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.004 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.006+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b28088a50 con 0x7f2b3813df80 2024-01-16T15:24:35.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.006+0000 7f2b35400640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 0x7f2b1006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.005 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.007+0000 7f2b35400640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 0x7f2b1006e980 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2b38155b20 tx=0x7f2b20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.010 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.012+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b28054fa0 con 0x7f2b3813df80 2024-01-16T15:24:35.196 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.198+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 --> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b381408d0 con 0x7f2b1006c4c0 2024-01-16T15:24:35.217 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.220+0000 7f2b2e000640 1 -- 172.21.15.125:0/2856261852 <== mgr.4104 v2:172.21.15.125:6832/2670128691 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291366 (secure 0 0 0) 0x7f2b2000c050 con 0x7f2b1006c4c0 2024-01-16T15:24:35.222 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:24:35.223 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2024-01-16T15:24:35.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 msgr2=0x7f2b1006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 0x7f2b1006e980 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2b38155b20 tx=0x7f2b20008040 comp rx=0 tx=0).stop 2024-01-16T15:24:35.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 msgr2=0x7f2b380ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.225 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800c7f0 tx=0x7f2b2800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 shutdown_connections 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b3815f9c0 0x7f2b38158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b3813e360 0x7f2b38157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2b1006c4c0 0x7f2b1006e980 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 --2- 172.21.15.125:0/2856261852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b3813df80 0x7f2b380ade00 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.225+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 >> 172.21.15.125:0/2856261852 conn(0x7f2b3806a800 msgr2=0x7f2b3813ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.226+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 shutdown_connections 2024-01-16T15:24:35.226 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.226+0000 7f2b3cc00640 1 -- 172.21.15.125:0/2856261852 wait complete. 2024-01-16T15:24:35.242 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":946,"stamp":"2024-01-16T15:24:34.566650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444269059,"num_objects":1571,"num_object_clones":1,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":1,"num_read":946,"num_read_kb":3145890,"num_write":2252,"num_write_kb":6295343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6168,"ondisk_log_size":6168,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13147708,"kb_used_data":12597564,"kb_used_omap":12,"kb_used_meta":550067,"kb_avail":741827012,"statfs":{"total":773094113280,"available":759630860288,"internally_reserved":0,"allocated":12899905536,"data_stored":12893402595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563269168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004423"},"pg_stats":[{"pgid":"17.3f","version":"113'98","reported_seq":138,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.383941+0000","last_change":"2024-01-16T15:24:02.384001+0000","last_active":"2024-01-16T15:24:02.383941+0000","last_peered":"2024-01-16T15:24:02.383941+0000","last_clean":"2024-01-16T15:24:02.383941+0000","last_became_active":"2024-01-16T15:15:07.593529+0000","last_became_peered":"2024-01-16T15:15:07.593529+0000","last_unstale":"2024-01-16T15:24:02.383941+0000","last_undegraded":"2024-01-16T15:24:02.383941+0000","last_fullsized":"2024-01-16T15:24:02.383941+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'98","last_scrub_stamp":"2024-01-16T15:24:02.383764+0000","last_deep_scrub":"113'98","last_deep_scrub_stamp":"2024-01-16T15:24:02.383764+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.383764+0000","objects_scrubbed":28,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:21.709126+0000","scrub_duration":0.323488779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"113'91","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.213438+0000","last_change":"2024-01-16T15:23:42.213497+0000","last_active":"2024-01-16T15:23:42.213438+0000","last_peered":"2024-01-16T15:23:42.213438+0000","last_clean":"2024-01-16T15:23:42.213438+0000","last_became_active":"2024-01-16T15:15:07.588405+0000","last_became_peered":"2024-01-16T15:15:07.588405+0000","last_unstale":"2024-01-16T15:23:42.213438+0000","last_undegraded":"2024-01-16T15:23:42.213438+0000","last_fullsized":"2024-01-16T15:23:42.213438+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'91","last_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_deep_scrub":"113'91","last_deep_scrub_stamp":"2024-01-16T15:23:42.213253+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.213253+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:45.051080+0000","scrub_duration":0.29806467199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"113'94","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:58.326078+0000","last_change":"2024-01-16T15:23:58.326121+0000","last_active":"2024-01-16T15:23:58.326078+0000","last_peered":"2024-01-16T15:23:58.326078+0000","last_clean":"2024-01-16T15:23:58.326078+0000","last_became_active":"2024-01-16T15:15:07.580953+0000","last_became_peered":"2024-01-16T15:15:07.580953+0000","last_unstale":"2024-01-16T15:23:58.326078+0000","last_undegraded":"2024-01-16T15:23:58.326078+0000","last_fullsized":"2024-01-16T15:23:58.326078+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'94","last_scrub_stamp":"2024-01-16T15:23:58.325934+0000","last_deep_scrub":"113'94","last_deep_scrub_stamp":"2024-01-16T15:23:58.325934+0000","last_clean_scrub_stamp":"2024-01-16T15:23:58.325934+0000","objects_scrubbed":27,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:49:22.071877+0000","scrub_duration":0.32196777599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"113'91","reported_seq":129,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:13.472340+0000","last_change":"2024-01-16T15:24:13.472396+0000","last_active":"2024-01-16T15:24:13.472340+0000","last_peered":"2024-01-16T15:24:13.472340+0000","last_clean":"2024-01-16T15:24:13.472340+0000","last_became_active":"2024-01-16T15:15:07.599463+0000","last_became_peered":"2024-01-16T15:15:07.599463+0000","last_unstale":"2024-01-16T15:24:13.472340+0000","last_undegraded":"2024-01-16T15:24:13.472340+0000","last_fullsized":"2024-01-16T15:24:13.472340+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'91","last_scrub_stamp":"2024-01-16T15:24:13.472163+0000","last_deep_scrub":"113'91","last_deep_scrub_stamp":"2024-01-16T15:24:13.472163+0000","last_clean_scrub_stamp":"2024-01-16T15:24:13.472163+0000","objects_scrubbed":25,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:54.898474+0000","scrub_duration":0.29273706799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"113'85","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.463762+0000","last_change":"2024-01-16T15:23:57.463810+0000","last_active":"2024-01-16T15:23:57.463762+0000","last_peered":"2024-01-16T15:23:57.463762+0000","last_clean":"2024-01-16T15:23:57.463762+0000","last_became_active":"2024-01-16T15:15:07.584724+0000","last_became_peered":"2024-01-16T15:15:07.584724+0000","last_unstale":"2024-01-16T15:23:57.463762+0000","last_undegraded":"2024-01-16T15:23:57.463762+0000","last_fullsized":"2024-01-16T15:23:57.463762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'85","last_scrub_stamp":"2024-01-16T15:23:57.463577+0000","last_deep_scrub":"113'85","last_deep_scrub_stamp":"2024-01-16T15:23:57.463577+0000","last_clean_scrub_stamp":"2024-01-16T15:23:57.463577+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:44.314486+0000","scrub_duration":0.28701645999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"113'100","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.880459+0000","last_change":"2024-01-16T15:23:43.880600+0000","last_active":"2024-01-16T15:23:43.880459+0000","last_peered":"2024-01-16T15:23:43.880459+0000","last_clean":"2024-01-16T15:23:43.880459+0000","last_became_active":"2024-01-16T15:15:07.583959+0000","last_became_peered":"2024-01-16T15:15:07.583959+0000","last_unstale":"2024-01-16T15:23:43.880459+0000","last_undegraded":"2024-01-16T15:23:43.880459+0000","last_fullsized":"2024-01-16T15:23:43.880459+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:43.880280+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.880280+0000","objects_scrubbed":31,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:51.072560+0000","scrub_duration":0.388654414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"113'84","reported_seq":114,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.465385+0000","last_change":"2024-01-16T15:24:02.465465+0000","last_active":"2024-01-16T15:24:02.465385+0000","last_peered":"2024-01-16T15:24:02.465385+0000","last_clean":"2024-01-16T15:24:02.465385+0000","last_became_active":"2024-01-16T15:15:07.587712+0000","last_became_peered":"2024-01-16T15:15:07.587712+0000","last_unstale":"2024-01-16T15:24:02.465385+0000","last_undegraded":"2024-01-16T15:24:02.465385+0000","last_fullsized":"2024-01-16T15:24:02.465385+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:24:02.465203+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:24:02.465203+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.465203+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:52.456114+0000","scrub_duration":0.229053322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"113'71","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.992771+0000","last_change":"2024-01-16T15:23:39.992859+0000","last_active":"2024-01-16T15:23:39.992771+0000","last_peered":"2024-01-16T15:23:39.992771+0000","last_clean":"2024-01-16T15:23:39.992771+0000","last_became_active":"2024-01-16T15:15:07.582093+0000","last_became_peered":"2024-01-16T15:15:07.582093+0000","last_unstale":"2024-01-16T15:23:39.992771+0000","last_undegraded":"2024-01-16T15:23:39.992771+0000","last_fullsized":"2024-01-16T15:23:39.992771+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_deep_scrub":"113'71","last_deep_scrub_stamp":"2024-01-16T15:23:39.992591+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.992591+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:54.989201+0000","scrub_duration":0.23396198900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"113'100","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.211670+0000","last_change":"2024-01-16T15:23:33.211722+0000","last_active":"2024-01-16T15:23:33.211670+0000","last_peered":"2024-01-16T15:23:33.211670+0000","last_clean":"2024-01-16T15:23:33.211670+0000","last_became_active":"2024-01-16T15:15:07.597510+0000","last_became_peered":"2024-01-16T15:15:07.597510+0000","last_unstale":"2024-01-16T15:23:33.211670+0000","last_undegraded":"2024-01-16T15:23:33.211670+0000","last_fullsized":"2024-01-16T15:23:33.211670+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:33.211507+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.211507+0000","objects_scrubbed":27,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:47.572037+0000","scrub_duration":0.235911915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"113'88","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.390744+0000","last_change":"2024-01-16T15:24:08.390835+0000","last_active":"2024-01-16T15:24:08.390744+0000","last_peered":"2024-01-16T15:24:08.390744+0000","last_clean":"2024-01-16T15:24:08.390744+0000","last_became_active":"2024-01-16T15:15:07.590969+0000","last_became_peered":"2024-01-16T15:15:07.590969+0000","last_unstale":"2024-01-16T15:24:08.390744+0000","last_undegraded":"2024-01-16T15:24:08.390744+0000","last_fullsized":"2024-01-16T15:24:08.390744+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:24:08.390604+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:24:08.390604+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.390604+0000","objects_scrubbed":22,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:22.535744+0000","scrub_duration":0.297321159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"113'97","reported_seq":141,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.321751+0000","last_change":"2024-01-16T15:24:11.321804+0000","last_active":"2024-01-16T15:24:11.321751+0000","last_peered":"2024-01-16T15:24:11.321751+0000","last_clean":"2024-01-16T15:24:11.321751+0000","last_became_active":"2024-01-16T15:15:07.591396+0000","last_became_peered":"2024-01-16T15:15:07.591396+0000","last_unstale":"2024-01-16T15:24:11.321751+0000","last_undegraded":"2024-01-16T15:24:11.321751+0000","last_fullsized":"2024-01-16T15:24:11.321751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'97","last_scrub_stamp":"2024-01-16T15:24:11.321551+0000","last_deep_scrub":"113'97","last_deep_scrub_stamp":"2024-01-16T15:24:11.321551+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.321551+0000","objects_scrubbed":27,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:13.088844+0000","scrub_duration":0.33547377299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"113'84","reported_seq":131,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.468357+0000","last_change":"2024-01-16T15:24:11.468461+0000","last_active":"2024-01-16T15:24:11.468357+0000","last_peered":"2024-01-16T15:24:11.468357+0000","last_clean":"2024-01-16T15:24:11.468357+0000","last_became_active":"2024-01-16T15:15:07.593214+0000","last_became_peered":"2024-01-16T15:15:07.593214+0000","last_unstale":"2024-01-16T15:24:11.468357+0000","last_undegraded":"2024-01-16T15:24:11.468357+0000","last_fullsized":"2024-01-16T15:24:11.468357+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:24:11.468168+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:24:11.468168+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.468168+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:03.462338+0000","scrub_duration":0.30833007099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"113'78","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.086907+0000","last_change":"2024-01-16T15:23:48.087015+0000","last_active":"2024-01-16T15:23:48.086907+0000","last_peered":"2024-01-16T15:23:48.086907+0000","last_clean":"2024-01-16T15:23:48.086907+0000","last_became_active":"2024-01-16T15:15:07.578895+0000","last_became_peered":"2024-01-16T15:15:07.578895+0000","last_unstale":"2024-01-16T15:23:48.086907+0000","last_undegraded":"2024-01-16T15:23:48.086907+0000","last_fullsized":"2024-01-16T15:23:48.086907+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:48.086739+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.086739+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:49.934440+0000","scrub_duration":0.26635034800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"113'88","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.116922+0000","last_change":"2024-01-16T15:23:44.116922+0000","last_active":"2024-01-16T15:23:44.116922+0000","last_peered":"2024-01-16T15:23:44.116922+0000","last_clean":"2024-01-16T15:23:44.116922+0000","last_became_active":"2024-01-16T15:15:07.589882+0000","last_became_peered":"2024-01-16T15:15:07.589882+0000","last_unstale":"2024-01-16T15:23:44.116922+0000","last_undegraded":"2024-01-16T15:23:44.116922+0000","last_fullsized":"2024-01-16T15:23:44.116922+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:23:44.116705+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.116705+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:20.435799+0000","scrub_duration":0.35551533600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"113'64","reported_seq":97,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.593181+0000","last_change":"2024-01-16T15:23:33.593181+0000","last_active":"2024-01-16T15:23:33.593181+0000","last_peered":"2024-01-16T15:23:33.593181+0000","last_clean":"2024-01-16T15:23:33.593181+0000","last_became_active":"2024-01-16T15:15:07.586195+0000","last_became_peered":"2024-01-16T15:15:07.586195+0000","last_unstale":"2024-01-16T15:23:33.593181+0000","last_undegraded":"2024-01-16T15:23:33.593181+0000","last_fullsized":"2024-01-16T15:23:33.593181+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'64","last_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_deep_scrub":"113'64","last_deep_scrub_stamp":"2024-01-16T15:23:33.592968+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.592968+0000","objects_scrubbed":14,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:45.001245+0000","scrub_duration":0.13121580099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"113'77","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.460923+0000","last_change":"2024-01-16T15:24:00.461021+0000","last_active":"2024-01-16T15:24:00.460923+0000","last_peered":"2024-01-16T15:24:00.460923+0000","last_clean":"2024-01-16T15:24:00.460923+0000","last_became_active":"2024-01-16T15:15:28.509010+0000","last_became_peered":"2024-01-16T15:15:28.509010+0000","last_unstale":"2024-01-16T15:24:00.460923+0000","last_undegraded":"2024-01-16T15:24:00.460923+0000","last_fullsized":"2024-01-16T15:24:00.460923+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'77","last_scrub_stamp":"2024-01-16T15:24:00.460742+0000","last_deep_scrub":"113'77","last_deep_scrub_stamp":"2024-01-16T15:24:00.460742+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.460742+0000","objects_scrubbed":19,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:21.783247+0000","scrub_duration":0.22164234999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.2f","version":"113'66","reported_seq":102,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.091768+0000","last_change":"2024-01-16T15:23:49.091768+0000","last_active":"2024-01-16T15:23:49.091768+0000","last_peered":"2024-01-16T15:23:49.091768+0000","last_clean":"2024-01-16T15:23:49.091768+0000","last_became_active":"2024-01-16T15:15:07.593560+0000","last_became_peered":"2024-01-16T15:15:07.593560+0000","last_unstale":"2024-01-16T15:23:49.091768+0000","last_undegraded":"2024-01-16T15:23:49.091768+0000","last_fullsized":"2024-01-16T15:23:49.091768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'66","last_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_deep_scrub":"113'66","last_deep_scrub_stamp":"2024-01-16T15:23:49.091510+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.091510+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:04.088120+0000","scrub_duration":0.232593777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"113'107","reported_seq":159,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.575949+0000","last_change":"2024-01-16T15:23:52.575949+0000","last_active":"2024-01-16T15:23:52.575949+0000","last_peered":"2024-01-16T15:23:52.575949+0000","last_clean":"2024-01-16T15:23:52.575949+0000","last_became_active":"2024-01-16T15:15:07.591543+0000","last_became_peered":"2024-01-16T15:15:07.591543+0000","last_unstale":"2024-01-16T15:23:52.575949+0000","last_undegraded":"2024-01-16T15:23:52.575949+0000","last_fullsized":"2024-01-16T15:23:52.575949+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'107","last_scrub_stamp":"2024-01-16T15:23:52.575711+0000","last_deep_scrub":"113'107","last_deep_scrub_stamp":"2024-01-16T15:23:52.575711+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.575711+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:20.721431+0000","scrub_duration":0.40777005500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"113'100","reported_seq":136,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.312479+0000","last_change":"2024-01-16T15:23:59.312479+0000","last_active":"2024-01-16T15:23:59.312479+0000","last_peered":"2024-01-16T15:23:59.312479+0000","last_clean":"2024-01-16T15:23:59.312479+0000","last_became_active":"2024-01-16T15:15:07.576676+0000","last_became_peered":"2024-01-16T15:15:07.576676+0000","last_unstale":"2024-01-16T15:23:59.312479+0000","last_undegraded":"2024-01-16T15:23:59.312479+0000","last_fullsized":"2024-01-16T15:23:59.312479+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'100","last_scrub_stamp":"2024-01-16T15:23:59.312256+0000","last_deep_scrub":"113'100","last_deep_scrub_stamp":"2024-01-16T15:23:59.312256+0000","last_clean_scrub_stamp":"2024-01-16T15:23:59.312256+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:33.383998+0000","scrub_duration":0.31913148600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"113'87","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.218013+0000","last_change":"2024-01-16T15:23:43.218013+0000","last_active":"2024-01-16T15:23:43.218013+0000","last_peered":"2024-01-16T15:23:43.218013+0000","last_clean":"2024-01-16T15:23:43.218013+0000","last_became_active":"2024-01-16T15:15:07.574218+0000","last_became_peered":"2024-01-16T15:15:07.574218+0000","last_unstale":"2024-01-16T15:23:43.218013+0000","last_undegraded":"2024-01-16T15:23:43.218013+0000","last_fullsized":"2024-01-16T15:23:43.218013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'87","last_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_deep_scrub":"113'87","last_deep_scrub_stamp":"2024-01-16T15:23:43.217792+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.217792+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.208703+0000","scrub_duration":0.29447676900000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"113'103","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:48.212338+0000","last_change":"2024-01-16T15:23:48.212338+0000","last_active":"2024-01-16T15:23:48.212338+0000","last_peered":"2024-01-16T15:23:48.212338+0000","last_clean":"2024-01-16T15:23:48.212338+0000","last_became_active":"2024-01-16T15:15:07.593780+0000","last_became_peered":"2024-01-16T15:15:07.593780+0000","last_unstale":"2024-01-16T15:23:48.212338+0000","last_undegraded":"2024-01-16T15:23:48.212338+0000","last_fullsized":"2024-01-16T15:23:48.212338+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'103","last_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_deep_scrub":"113'103","last_deep_scrub_stamp":"2024-01-16T15:23:48.212080+0000","last_clean_scrub_stamp":"2024-01-16T15:23:48.212080+0000","objects_scrubbed":30,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:31.387018+0000","scrub_duration":0.36970662999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"113'90","reported_seq":124,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:53.116762+0000","last_change":"2024-01-16T15:23:53.116762+0000","last_active":"2024-01-16T15:23:53.116762+0000","last_peered":"2024-01-16T15:23:53.116762+0000","last_clean":"2024-01-16T15:23:53.116762+0000","last_became_active":"2024-01-16T15:15:07.595135+0000","last_became_peered":"2024-01-16T15:15:07.595135+0000","last_unstale":"2024-01-16T15:23:53.116762+0000","last_undegraded":"2024-01-16T15:23:53.116762+0000","last_fullsized":"2024-01-16T15:23:53.116762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'90","last_scrub_stamp":"2024-01-16T15:23:53.116433+0000","last_deep_scrub":"113'90","last_deep_scrub_stamp":"2024-01-16T15:23:53.116433+0000","last_clean_scrub_stamp":"2024-01-16T15:23:53.116433+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:15.298285+0000","scrub_duration":0.29584653500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"113'79","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.249926+0000","last_change":"2024-01-16T15:23:37.249926+0000","last_active":"2024-01-16T15:23:37.249926+0000","last_peered":"2024-01-16T15:23:37.249926+0000","last_clean":"2024-01-16T15:23:37.249926+0000","last_became_active":"2024-01-16T15:15:07.572594+0000","last_became_peered":"2024-01-16T15:15:07.572594+0000","last_unstale":"2024-01-16T15:23:37.249926+0000","last_undegraded":"2024-01-16T15:23:37.249926+0000","last_fullsized":"2024-01-16T15:23:37.249926+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:37.249557+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.249557+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:23.718419+0000","scrub_duration":0.30745207200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"113'90","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.124563+0000","last_change":"2024-01-16T15:23:47.124563+0000","last_active":"2024-01-16T15:23:47.124563+0000","last_peered":"2024-01-16T15:23:47.124563+0000","last_clean":"2024-01-16T15:23:47.124563+0000","last_became_active":"2024-01-16T15:15:07.579140+0000","last_became_peered":"2024-01-16T15:15:07.579140+0000","last_unstale":"2024-01-16T15:23:47.124563+0000","last_undegraded":"2024-01-16T15:23:47.124563+0000","last_fullsized":"2024-01-16T15:23:47.124563+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'90","last_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_deep_scrub":"113'90","last_deep_scrub_stamp":"2024-01-16T15:23:47.124399+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.124399+0000","objects_scrubbed":24,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:02.121008+0000","scrub_duration":0.271690178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"113'96","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.160049+0000","last_change":"2024-01-16T15:23:47.160049+0000","last_active":"2024-01-16T15:23:47.160049+0000","last_peered":"2024-01-16T15:23:47.160049+0000","last_clean":"2024-01-16T15:23:47.160049+0000","last_became_active":"2024-01-16T15:15:07.592706+0000","last_became_peered":"2024-01-16T15:15:07.592706+0000","last_unstale":"2024-01-16T15:23:47.160049+0000","last_undegraded":"2024-01-16T15:23:47.160049+0000","last_fullsized":"2024-01-16T15:23:47.160049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.159896+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.159896+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:22.125663+0000","scrub_duration":0.29641904899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"113'107","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.460144+0000","last_change":"2024-01-16T15:24:06.460201+0000","last_active":"2024-01-16T15:24:06.460144+0000","last_peered":"2024-01-16T15:24:06.460144+0000","last_clean":"2024-01-16T15:24:06.460144+0000","last_became_active":"2024-01-16T15:15:07.592892+0000","last_became_peered":"2024-01-16T15:15:07.592892+0000","last_unstale":"2024-01-16T15:24:06.460144+0000","last_undegraded":"2024-01-16T15:24:06.460144+0000","last_fullsized":"2024-01-16T15:24:06.460144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'107","last_scrub_stamp":"2024-01-16T15:24:06.459972+0000","last_deep_scrub":"113'107","last_deep_scrub_stamp":"2024-01-16T15:24:06.459972+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.459972+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:24.029836+0000","scrub_duration":0.33540472199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"113'86","reported_seq":122,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:44.783319+0000","last_change":"2024-01-16T15:23:44.783319+0000","last_active":"2024-01-16T15:23:44.783319+0000","last_peered":"2024-01-16T15:23:44.783319+0000","last_clean":"2024-01-16T15:23:44.783319+0000","last_became_active":"2024-01-16T15:15:07.593438+0000","last_became_peered":"2024-01-16T15:15:07.593438+0000","last_unstale":"2024-01-16T15:23:44.783319+0000","last_undegraded":"2024-01-16T15:23:44.783319+0000","last_fullsized":"2024-01-16T15:23:44.783319+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'86","last_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_deep_scrub":"113'86","last_deep_scrub_stamp":"2024-01-16T15:23:44.783151+0000","last_clean_scrub_stamp":"2024-01-16T15:23:44.783151+0000","objects_scrubbed":24,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.138136+0000","scrub_duration":0.29777104900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"113'62","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.068006+0000","last_change":"2024-01-16T15:24:01.068122+0000","last_active":"2024-01-16T15:24:01.068006+0000","last_peered":"2024-01-16T15:24:01.068006+0000","last_clean":"2024-01-16T15:24:01.068006+0000","last_became_active":"2024-01-16T15:15:07.600921+0000","last_became_peered":"2024-01-16T15:15:07.600921+0000","last_unstale":"2024-01-16T15:24:01.068006+0000","last_undegraded":"2024-01-16T15:24:01.068006+0000","last_fullsized":"2024-01-16T15:24:01.068006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'62","last_scrub_stamp":"2024-01-16T15:24:01.067805+0000","last_deep_scrub":"113'62","last_deep_scrub_stamp":"2024-01-16T15:24:01.067805+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.067805+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:02.835098+0000","scrub_duration":0.23695159800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"113'77","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.203581+0000","last_change":"2024-01-16T15:24:10.203646+0000","last_active":"2024-01-16T15:24:10.203581+0000","last_peered":"2024-01-16T15:24:10.203581+0000","last_clean":"2024-01-16T15:24:10.203581+0000","last_became_active":"2024-01-16T15:15:07.576002+0000","last_became_peered":"2024-01-16T15:15:07.576002+0000","last_unstale":"2024-01-16T15:24:10.203581+0000","last_undegraded":"2024-01-16T15:24:10.203581+0000","last_fullsized":"2024-01-16T15:24:10.203581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'77","last_scrub_stamp":"2024-01-16T15:24:10.203486+0000","last_deep_scrub":"113'77","last_deep_scrub_stamp":"2024-01-16T15:24:10.203486+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.203486+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:51:05.717511+0000","scrub_duration":0.25092224200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"113'104","reported_seq":147,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.487580+0000","last_change":"2024-01-16T15:24:04.487638+0000","last_active":"2024-01-16T15:24:04.487580+0000","last_peered":"2024-01-16T15:24:04.487580+0000","last_clean":"2024-01-16T15:24:04.487580+0000","last_became_active":"2024-01-16T15:15:07.590639+0000","last_became_peered":"2024-01-16T15:15:07.590639+0000","last_unstale":"2024-01-16T15:24:04.487580+0000","last_undegraded":"2024-01-16T15:24:04.487580+0000","last_fullsized":"2024-01-16T15:24:04.487580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'104","last_scrub_stamp":"2024-01-16T15:24:04.487387+0000","last_deep_scrub":"113'104","last_deep_scrub_stamp":"2024-01-16T15:24:04.487387+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.487387+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:47.662326+0000","scrub_duration":0.30441336299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"113'106","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.282980+0000","last_change":"2024-01-16T15:23:34.283133+0000","last_active":"2024-01-16T15:23:34.282980+0000","last_peered":"2024-01-16T15:23:34.282980+0000","last_clean":"2024-01-16T15:23:34.282980+0000","last_became_active":"2024-01-16T15:15:07.592522+0000","last_became_peered":"2024-01-16T15:15:07.592522+0000","last_unstale":"2024-01-16T15:23:34.282980+0000","last_undegraded":"2024-01-16T15:23:34.282980+0000","last_fullsized":"2024-01-16T15:23:34.282980+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'106","last_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_deep_scrub":"113'106","last_deep_scrub_stamp":"2024-01-16T15:23:34.282804+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.282804+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:08.354546+0000","scrub_duration":0.30496924800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"113'82","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.785613+0000","last_change":"2024-01-16T15:23:42.785799+0000","last_active":"2024-01-16T15:23:42.785613+0000","last_peered":"2024-01-16T15:23:42.785613+0000","last_clean":"2024-01-16T15:23:42.785613+0000","last_became_active":"2024-01-16T15:15:07.585040+0000","last_became_peered":"2024-01-16T15:15:07.585040+0000","last_unstale":"2024-01-16T15:23:42.785613+0000","last_undegraded":"2024-01-16T15:23:42.785613+0000","last_fullsized":"2024-01-16T15:23:42.785613+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'82","last_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_deep_scrub":"113'82","last_deep_scrub_stamp":"2024-01-16T15:23:42.785476+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.785476+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:05.117297+0000","scrub_duration":0.269243968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"113'88","reported_seq":125,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:58.804502+0000","last_change":"2024-01-16T15:23:58.804545+0000","last_active":"2024-01-16T15:23:58.804502+0000","last_peered":"2024-01-16T15:23:58.804502+0000","last_clean":"2024-01-16T15:23:58.804502+0000","last_became_active":"2024-01-16T15:15:07.587531+0000","last_became_peered":"2024-01-16T15:15:07.587531+0000","last_unstale":"2024-01-16T15:23:58.804502+0000","last_undegraded":"2024-01-16T15:23:58.804502+0000","last_fullsized":"2024-01-16T15:23:58.804502+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:23:58.804380+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:23:58.804380+0000","last_clean_scrub_stamp":"2024-01-16T15:23:58.804380+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:45.655289+0000","scrub_duration":0.30971083900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'6","reported_seq":33,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.264808+0000","last_change":"2024-01-16T15:23:59.264869+0000","last_active":"2024-01-16T15:23:59.264808+0000","last_peered":"2024-01-16T15:23:59.264808+0000","last_clean":"2024-01-16T15:23:59.264808+0000","last_became_active":"2024-01-16T15:15:06.559976+0000","last_became_peered":"2024-01-16T15:15:06.559976+0000","last_unstale":"2024-01-16T15:23:59.264808+0000","last_undegraded":"2024-01-16T15:23:59.264808+0000","last_fullsized":"2024-01-16T15:23:59.264808+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-01-16T15:23:59.264614+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-01-16T15:23:59.264614+0000","last_clean_scrub_stamp":"2024-01-16T15:23:59.264614+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:25:05.701638+0000","scrub_duration":0.0087135949999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"113'73","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.352331+0000","last_change":"2024-01-16T15:23:49.352425+0000","last_active":"2024-01-16T15:23:49.352331+0000","last_peered":"2024-01-16T15:23:49.352331+0000","last_clean":"2024-01-16T15:23:49.352331+0000","last_became_active":"2024-01-16T15:15:07.587660+0000","last_became_peered":"2024-01-16T15:15:07.587660+0000","last_unstale":"2024-01-16T15:23:49.352331+0000","last_undegraded":"2024-01-16T15:23:49.352331+0000","last_fullsized":"2024-01-16T15:23:49.352331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'73","last_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_deep_scrub":"113'73","last_deep_scrub_stamp":"2024-01-16T15:23:49.352155+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.352155+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:06:37.591834+0000","scrub_duration":0.23652007799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"110'1","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.752037+0000","last_change":"2024-01-16T15:23:36.752093+0000","last_active":"2024-01-16T15:23:36.752037+0000","last_peered":"2024-01-16T15:23:36.752037+0000","last_clean":"2024-01-16T15:23:36.752037+0000","last_became_active":"2024-01-16T15:15:06.558509+0000","last_became_peered":"2024-01-16T15:15:06.558509+0000","last_unstale":"2024-01-16T15:23:36.752037+0000","last_undegraded":"2024-01-16T15:23:36.752037+0000","last_fullsized":"2024-01-16T15:23:36.752037+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:36.751793+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.751793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:04.897513+0000","scrub_duration":0.0034892019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"113'96","reported_seq":140,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.838614+0000","last_change":"2024-01-16T15:23:47.838729+0000","last_active":"2024-01-16T15:23:47.838614+0000","last_peered":"2024-01-16T15:23:47.838614+0000","last_clean":"2024-01-16T15:23:47.838614+0000","last_became_active":"2024-01-16T15:15:07.595460+0000","last_became_peered":"2024-01-16T15:15:07.595460+0000","last_unstale":"2024-01-16T15:23:47.838614+0000","last_undegraded":"2024-01-16T15:23:47.838614+0000","last_fullsized":"2024-01-16T15:23:47.838614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:23:47.838456+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.838456+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:01.112625+0000","scrub_duration":0.318651024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'4","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.965524+0000","last_change":"2024-01-16T15:23:33.965582+0000","last_active":"2024-01-16T15:23:33.965524+0000","last_peered":"2024-01-16T15:23:33.965524+0000","last_clean":"2024-01-16T15:23:33.965524+0000","last_became_active":"2024-01-16T15:15:06.558167+0000","last_became_peered":"2024-01-16T15:15:06.558167+0000","last_unstale":"2024-01-16T15:23:33.965524+0000","last_undegraded":"2024-01-16T15:23:33.965524+0000","last_fullsized":"2024-01-16T15:23:33.965524+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:33.965351+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.965351+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:58.186239+0000","scrub_duration":0.0074428489999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"113'89","reported_seq":135,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.285933+0000","last_change":"2024-01-16T15:24:00.286123+0000","last_active":"2024-01-16T15:24:00.285933+0000","last_peered":"2024-01-16T15:24:00.285933+0000","last_clean":"2024-01-16T15:24:00.285933+0000","last_became_active":"2024-01-16T15:15:07.576934+0000","last_became_peered":"2024-01-16T15:15:07.576934+0000","last_unstale":"2024-01-16T15:24:00.285933+0000","last_undegraded":"2024-01-16T15:24:00.285933+0000","last_fullsized":"2024-01-16T15:24:00.285933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:24:00.285640+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:24:00.285640+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.285640+0000","objects_scrubbed":21,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:33.346013+0000","scrub_duration":0.25024972699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'3","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.098394+0000","last_change":"2024-01-16T15:23:49.098456+0000","last_active":"2024-01-16T15:23:49.098394+0000","last_peered":"2024-01-16T15:23:49.098394+0000","last_clean":"2024-01-16T15:23:49.098394+0000","last_became_active":"2024-01-16T15:15:06.557384+0000","last_became_peered":"2024-01-16T15:15:06.557384+0000","last_unstale":"2024-01-16T15:23:49.098394+0000","last_undegraded":"2024-01-16T15:23:49.098394+0000","last_fullsized":"2024-01-16T15:23:49.098394+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:49.098224+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.098224+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:34.500097+0000","scrub_duration":0.0069822110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"113'95","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.383122+0000","last_change":"2024-01-16T15:24:05.383213+0000","last_active":"2024-01-16T15:24:05.383122+0000","last_peered":"2024-01-16T15:24:05.383122+0000","last_clean":"2024-01-16T15:24:05.383122+0000","last_became_active":"2024-01-16T15:15:07.597436+0000","last_became_peered":"2024-01-16T15:15:07.597436+0000","last_unstale":"2024-01-16T15:24:05.383122+0000","last_undegraded":"2024-01-16T15:24:05.383122+0000","last_fullsized":"2024-01-16T15:24:05.383122+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'95","last_scrub_stamp":"2024-01-16T15:24:05.383025+0000","last_deep_scrub":"113'95","last_deep_scrub_stamp":"2024-01-16T15:24:05.383025+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.383025+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:09.216054+0000","scrub_duration":0.29273136199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:54.176509+0000","last_change":"2024-01-16T15:23:54.176563+0000","last_active":"2024-01-16T15:23:54.176509+0000","last_peered":"2024-01-16T15:23:54.176509+0000","last_clean":"2024-01-16T15:23:54.176509+0000","last_became_active":"2024-01-16T15:15:06.566541+0000","last_became_peered":"2024-01-16T15:15:06.566541+0000","last_unstale":"2024-01-16T15:23:54.176509+0000","last_undegraded":"2024-01-16T15:23:54.176509+0000","last_fullsized":"2024-01-16T15:23:54.176509+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:54.176314+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:54.176314+0000","last_clean_scrub_stamp":"2024-01-16T15:23:54.176314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:35.216255+0000","scrub_duration":0.0044258329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"113'97","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.126742+0000","last_change":"2024-01-16T15:24:04.126802+0000","last_active":"2024-01-16T15:24:04.126742+0000","last_peered":"2024-01-16T15:24:04.126742+0000","last_clean":"2024-01-16T15:24:04.126742+0000","last_became_active":"2024-01-16T15:15:07.591328+0000","last_became_peered":"2024-01-16T15:15:07.591328+0000","last_unstale":"2024-01-16T15:24:04.126742+0000","last_undegraded":"2024-01-16T15:24:04.126742+0000","last_fullsized":"2024-01-16T15:24:04.126742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'97","last_scrub_stamp":"2024-01-16T15:24:04.126566+0000","last_deep_scrub":"113'97","last_deep_scrub_stamp":"2024-01-16T15:24:04.126566+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.126566+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:25.214136+0000","scrub_duration":0.30058973999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"110'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.864743+0000","last_change":"2024-01-16T15:23:31.864779+0000","last_active":"2024-01-16T15:23:31.864743+0000","last_peered":"2024-01-16T15:23:31.864743+0000","last_clean":"2024-01-16T15:23:31.864743+0000","last_became_active":"2024-01-16T15:15:06.556051+0000","last_became_peered":"2024-01-16T15:15:06.556051+0000","last_unstale":"2024-01-16T15:23:31.864743+0000","last_undegraded":"2024-01-16T15:23:31.864743+0000","last_fullsized":"2024-01-16T15:23:31.864743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-16T15:23:31.864652+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.864652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:25.530956+0000","scrub_duration":0.00281139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"113'96","reported_seq":136,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.339744+0000","last_change":"2024-01-16T15:24:01.339827+0000","last_active":"2024-01-16T15:24:01.339744+0000","last_peered":"2024-01-16T15:24:01.339744+0000","last_clean":"2024-01-16T15:24:01.339744+0000","last_became_active":"2024-01-16T15:15:07.592285+0000","last_became_peered":"2024-01-16T15:15:07.592285+0000","last_unstale":"2024-01-16T15:24:01.339744+0000","last_undegraded":"2024-01-16T15:24:01.339744+0000","last_fullsized":"2024-01-16T15:24:01.339744+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:24:01.339583+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:24:01.339583+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.339583+0000","objects_scrubbed":23,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:38.682058+0000","scrub_duration":0.27514720100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.896191+0000","last_change":"2024-01-16T15:23:35.896238+0000","last_active":"2024-01-16T15:23:35.896191+0000","last_peered":"2024-01-16T15:23:35.896191+0000","last_clean":"2024-01-16T15:23:35.896191+0000","last_became_active":"2024-01-16T15:15:06.556147+0000","last_became_peered":"2024-01-16T15:15:06.556147+0000","last_unstale":"2024-01-16T15:23:35.896191+0000","last_undegraded":"2024-01-16T15:23:35.896191+0000","last_fullsized":"2024-01-16T15:23:35.896191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.896004+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.896004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:41.978043+0000","scrub_duration":0.003846173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"113'101","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.579448+0000","last_change":"2024-01-16T15:24:03.579511+0000","last_active":"2024-01-16T15:24:03.579448+0000","last_peered":"2024-01-16T15:24:03.579448+0000","last_clean":"2024-01-16T15:24:03.579448+0000","last_became_active":"2024-01-16T15:15:07.593941+0000","last_became_peered":"2024-01-16T15:15:07.593941+0000","last_unstale":"2024-01-16T15:24:03.579448+0000","last_undegraded":"2024-01-16T15:24:03.579448+0000","last_fullsized":"2024-01-16T15:24:03.579448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'101","last_scrub_stamp":"2024-01-16T15:24:03.579285+0000","last_deep_scrub":"113'101","last_deep_scrub_stamp":"2024-01-16T15:24:03.579285+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.579285+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:38.545052+0000","scrub_duration":0.35447070899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.039548+0000","last_change":"2024-01-16T15:23:52.039594+0000","last_active":"2024-01-16T15:23:52.039548+0000","last_peered":"2024-01-16T15:23:52.039548+0000","last_clean":"2024-01-16T15:23:52.039548+0000","last_became_active":"2024-01-16T15:15:06.558367+0000","last_became_peered":"2024-01-16T15:15:06.558367+0000","last_unstale":"2024-01-16T15:23:52.039548+0000","last_undegraded":"2024-01-16T15:23:52.039548+0000","last_fullsized":"2024-01-16T15:23:52.039548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:52.039362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:52.039362+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.039362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:03.966788+0000","scrub_duration":0.0035075739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"113'76","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.418427+0000","last_change":"2024-01-16T15:24:08.418515+0000","last_active":"2024-01-16T15:24:08.418427+0000","last_peered":"2024-01-16T15:24:08.418427+0000","last_clean":"2024-01-16T15:24:08.418427+0000","last_became_active":"2024-01-16T15:15:07.585930+0000","last_became_peered":"2024-01-16T15:15:07.585930+0000","last_unstale":"2024-01-16T15:24:08.418427+0000","last_undegraded":"2024-01-16T15:24:08.418427+0000","last_fullsized":"2024-01-16T15:24:08.418427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'76","last_scrub_stamp":"2024-01-16T15:24:08.418256+0000","last_deep_scrub":"113'76","last_deep_scrub_stamp":"2024-01-16T15:24:08.418256+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.418256+0000","objects_scrubbed":18,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:44.737350+0000","scrub_duration":0.238497493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"113'1","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:42.081149+0000","last_change":"2024-01-16T15:23:42.081324+0000","last_active":"2024-01-16T15:23:42.081149+0000","last_peered":"2024-01-16T15:23:42.081149+0000","last_clean":"2024-01-16T15:23:42.081149+0000","last_became_active":"2024-01-16T15:15:06.563155+0000","last_became_peered":"2024-01-16T15:15:06.563155+0000","last_unstale":"2024-01-16T15:23:42.081149+0000","last_undegraded":"2024-01-16T15:23:42.081149+0000","last_fullsized":"2024-01-16T15:23:42.081149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-16T15:23:42.080895+0000","last_clean_scrub_stamp":"2024-01-16T15:23:42.080895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:49.273175+0000","scrub_duration":0.0037055339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"113'78","reported_seq":111,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.281779+0000","last_change":"2024-01-16T15:23:47.281905+0000","last_active":"2024-01-16T15:23:47.281779+0000","last_peered":"2024-01-16T15:23:47.281779+0000","last_clean":"2024-01-16T15:23:47.281779+0000","last_became_active":"2024-01-16T15:15:07.598865+0000","last_became_peered":"2024-01-16T15:15:07.598865+0000","last_unstale":"2024-01-16T15:23:47.281779+0000","last_undegraded":"2024-01-16T15:23:47.281779+0000","last_fullsized":"2024-01-16T15:23:47.281779+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'78","last_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_deep_scrub":"113'78","last_deep_scrub_stamp":"2024-01-16T15:23:47.281611+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.281611+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:50.119438+0000","scrub_duration":0.19022307299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:35.832184+0000","last_change":"2024-01-16T15:23:35.832238+0000","last_active":"2024-01-16T15:23:35.832184+0000","last_peered":"2024-01-16T15:23:35.832184+0000","last_clean":"2024-01-16T15:23:35.832184+0000","last_became_active":"2024-01-16T15:15:06.561837+0000","last_became_peered":"2024-01-16T15:15:06.561837+0000","last_unstale":"2024-01-16T15:23:35.832184+0000","last_undegraded":"2024-01-16T15:23:35.832184+0000","last_fullsized":"2024-01-16T15:23:35.832184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:35.831980+0000","last_clean_scrub_stamp":"2024-01-16T15:23:35.831980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:16.763641+0000","scrub_duration":0.004520446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"113'88","reported_seq":131,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.143798+0000","last_change":"2024-01-16T15:24:00.143798+0000","last_active":"2024-01-16T15:24:00.143798+0000","last_peered":"2024-01-16T15:24:00.143798+0000","last_clean":"2024-01-16T15:24:00.143798+0000","last_became_active":"2024-01-16T15:15:07.589602+0000","last_became_peered":"2024-01-16T15:15:07.589602+0000","last_unstale":"2024-01-16T15:24:00.143798+0000","last_undegraded":"2024-01-16T15:24:00.143798+0000","last_fullsized":"2024-01-16T15:24:00.143798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'88","last_scrub_stamp":"2024-01-16T15:24:00.143536+0000","last_deep_scrub":"113'88","last_deep_scrub_stamp":"2024-01-16T15:24:00.143536+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.143536+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:55.657562+0000","scrub_duration":0.29741432699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'33","reported_seq":59,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:54.078478+0000","last_change":"2024-01-16T15:23:54.078515+0000","last_active":"2024-01-16T15:23:54.078478+0000","last_peered":"2024-01-16T15:23:54.078478+0000","last_clean":"2024-01-16T15:23:54.078478+0000","last_became_active":"2024-01-16T15:15:06.563842+0000","last_became_peered":"2024-01-16T15:15:06.563842+0000","last_unstale":"2024-01-16T15:23:54.078478+0000","last_undegraded":"2024-01-16T15:23:54.078478+0000","last_fullsized":"2024-01-16T15:23:54.078478+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2024-01-16T15:23:54.078388+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2024-01-16T15:23:54.078388+0000","last_clean_scrub_stamp":"2024-01-16T15:23:54.078388+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:31.766859+0000","scrub_duration":0.0055228780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"113'96","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.496888+0000","last_change":"2024-01-16T15:24:06.496928+0000","last_active":"2024-01-16T15:24:06.496888+0000","last_peered":"2024-01-16T15:24:06.496888+0000","last_clean":"2024-01-16T15:24:06.496888+0000","last_became_active":"2024-01-16T15:15:07.584870+0000","last_became_peered":"2024-01-16T15:15:07.584870+0000","last_unstale":"2024-01-16T15:24:06.496888+0000","last_undegraded":"2024-01-16T15:24:06.496888+0000","last_fullsized":"2024-01-16T15:24:06.496888+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:24:06.496760+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:24:06.496760+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.496760+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:21.493369+0000","scrub_duration":0.35910446899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'325","reported_seq":352,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.688473+0000","last_change":"2024-01-16T15:23:34.952024+0000","last_active":"2024-01-16T15:24:28.688473+0000","last_peered":"2024-01-16T15:24:28.688473+0000","last_clean":"2024-01-16T15:24:28.688473+0000","last_became_active":"2024-01-16T15:15:06.560303+0000","last_became_peered":"2024-01-16T15:15:06.560303+0000","last_unstale":"2024-01-16T15:24:28.688473+0000","last_undegraded":"2024-01-16T15:24:28.688473+0000","last_fullsized":"2024-01-16T15:24:28.688473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'323","last_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_deep_scrub":"113'323","last_deep_scrub_stamp":"2024-01-16T15:23:34.951788+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.951788+0000","objects_scrubbed":4,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:08.012160+0000","scrub_duration":0.019282695999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1221245,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":322,"num_write_kb":1346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"113'75","reported_seq":110,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.087701+0000","last_change":"2024-01-16T15:23:52.087701+0000","last_active":"2024-01-16T15:23:52.087701+0000","last_peered":"2024-01-16T15:23:52.087701+0000","last_clean":"2024-01-16T15:23:52.087701+0000","last_became_active":"2024-01-16T15:15:07.586002+0000","last_became_peered":"2024-01-16T15:15:07.586002+0000","last_unstale":"2024-01-16T15:23:52.087701+0000","last_undegraded":"2024-01-16T15:23:52.087701+0000","last_fullsized":"2024-01-16T15:23:52.087701+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'75","last_scrub_stamp":"2024-01-16T15:23:52.087437+0000","last_deep_scrub":"113'75","last_deep_scrub_stamp":"2024-01-16T15:23:52.087437+0000","last_clean_scrub_stamp":"2024-01-16T15:23:52.087437+0000","objects_scrubbed":21,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:13.381080+0000","scrub_duration":0.22988671899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'4","reported_seq":31,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:32.491489+0000","last_change":"2024-01-16T15:23:32.491489+0000","last_active":"2024-01-16T15:23:32.491489+0000","last_peered":"2024-01-16T15:23:32.491489+0000","last_clean":"2024-01-16T15:23:32.491489+0000","last_became_active":"2024-01-16T15:15:06.556879+0000","last_became_peered":"2024-01-16T15:15:06.556879+0000","last_unstale":"2024-01-16T15:23:32.491489+0000","last_undegraded":"2024-01-16T15:23:32.491489+0000","last_fullsized":"2024-01-16T15:23:32.491489+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:32.491197+0000","last_clean_scrub_stamp":"2024-01-16T15:23:32.491197+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:41.739634+0000","scrub_duration":0.0079106109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"113'98","reported_seq":135,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:56.080477+0000","last_change":"2024-01-16T15:23:56.080477+0000","last_active":"2024-01-16T15:23:56.080477+0000","last_peered":"2024-01-16T15:23:56.080477+0000","last_clean":"2024-01-16T15:23:56.080477+0000","last_became_active":"2024-01-16T15:15:07.585322+0000","last_became_peered":"2024-01-16T15:15:07.585322+0000","last_unstale":"2024-01-16T15:23:56.080477+0000","last_undegraded":"2024-01-16T15:23:56.080477+0000","last_fullsized":"2024-01-16T15:23:56.080477+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'98","last_scrub_stamp":"2024-01-16T15:23:56.080232+0000","last_deep_scrub":"113'98","last_deep_scrub_stamp":"2024-01-16T15:23:56.080232+0000","last_clean_scrub_stamp":"2024-01-16T15:23:56.080232+0000","objects_scrubbed":26,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:59.032001+0000","scrub_duration":0.28766982299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.223383+0000","last_change":"2024-01-16T15:23:34.223383+0000","last_active":"2024-01-16T15:23:34.223383+0000","last_peered":"2024-01-16T15:23:34.223383+0000","last_clean":"2024-01-16T15:23:34.223383+0000","last_became_active":"2024-01-16T15:15:06.559986+0000","last_became_peered":"2024-01-16T15:15:06.559986+0000","last_unstale":"2024-01-16T15:23:34.223383+0000","last_undegraded":"2024-01-16T15:23:34.223383+0000","last_fullsized":"2024-01-16T15:23:34.223383+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:34.223138+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.223138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:21.983148+0000","scrub_duration":0.003763078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"113'89","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.172521+0000","last_change":"2024-01-16T15:23:50.172521+0000","last_active":"2024-01-16T15:23:50.172521+0000","last_peered":"2024-01-16T15:23:50.172521+0000","last_clean":"2024-01-16T15:23:50.172521+0000","last_became_active":"2024-01-16T15:15:07.587658+0000","last_became_peered":"2024-01-16T15:15:07.587658+0000","last_unstale":"2024-01-16T15:23:50.172521+0000","last_undegraded":"2024-01-16T15:23:50.172521+0000","last_fullsized":"2024-01-16T15:23:50.172521+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:23:50.172295+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:23:50.172295+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.172295+0000","objects_scrubbed":26,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:20.561902+0000","scrub_duration":0.32687721199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'12","reported_seq":39,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.079051+0000","last_change":"2024-01-16T15:23:45.079051+0000","last_active":"2024-01-16T15:23:45.079051+0000","last_peered":"2024-01-16T15:23:45.079051+0000","last_clean":"2024-01-16T15:23:45.079051+0000","last_became_active":"2024-01-16T15:15:06.548615+0000","last_became_peered":"2024-01-16T15:15:06.548615+0000","last_unstale":"2024-01-16T15:23:45.079051+0000","last_undegraded":"2024-01-16T15:23:45.079051+0000","last_fullsized":"2024-01-16T15:23:45.079051+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'12","last_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_deep_scrub":"113'12","last_deep_scrub_stamp":"2024-01-16T15:23:45.078685+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.078685+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:45.433669+0000","scrub_duration":0.010660358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"113'61","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:53.287936+0000","last_change":"2024-01-16T15:23:53.287936+0000","last_active":"2024-01-16T15:23:53.287936+0000","last_peered":"2024-01-16T15:23:53.287936+0000","last_clean":"2024-01-16T15:23:53.287936+0000","last_became_active":"2024-01-16T15:15:07.586800+0000","last_became_peered":"2024-01-16T15:15:07.586800+0000","last_unstale":"2024-01-16T15:23:53.287936+0000","last_undegraded":"2024-01-16T15:23:53.287936+0000","last_fullsized":"2024-01-16T15:23:53.287936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'61","last_scrub_stamp":"2024-01-16T15:23:53.287681+0000","last_deep_scrub":"113'61","last_deep_scrub_stamp":"2024-01-16T15:23:53.287681+0000","last_clean_scrub_stamp":"2024-01-16T15:23:53.287681+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:33:33.193702+0000","scrub_duration":0.19822701700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'2","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:56.062414+0000","last_change":"2024-01-16T15:23:56.062414+0000","last_active":"2024-01-16T15:23:56.062414+0000","last_peered":"2024-01-16T15:23:56.062414+0000","last_clean":"2024-01-16T15:23:56.062414+0000","last_became_active":"2024-01-16T15:15:06.561591+0000","last_became_peered":"2024-01-16T15:15:06.561591+0000","last_unstale":"2024-01-16T15:23:56.062414+0000","last_undegraded":"2024-01-16T15:23:56.062414+0000","last_fullsized":"2024-01-16T15:23:56.062414+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-01-16T15:23:56.062035+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-01-16T15:23:56.062035+0000","last_clean_scrub_stamp":"2024-01-16T15:23:56.062035+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:22:13.546196+0000","scrub_duration":0.0072074310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"113'85","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.819161+0000","last_change":"2024-01-16T15:23:49.819161+0000","last_active":"2024-01-16T15:23:49.819161+0000","last_peered":"2024-01-16T15:23:49.819161+0000","last_clean":"2024-01-16T15:23:49.819161+0000","last_became_active":"2024-01-16T15:15:07.599327+0000","last_became_peered":"2024-01-16T15:15:07.599327+0000","last_unstale":"2024-01-16T15:23:49.819161+0000","last_undegraded":"2024-01-16T15:23:49.819161+0000","last_fullsized":"2024-01-16T15:23:49.819161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'85","last_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_deep_scrub":"113'85","last_deep_scrub_stamp":"2024-01-16T15:23:49.818959+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.818959+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:41.716928+0000","scrub_duration":0.28833247000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'3","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.044424+0000","last_change":"2024-01-16T15:23:37.044424+0000","last_active":"2024-01-16T15:23:37.044424+0000","last_peered":"2024-01-16T15:23:37.044424+0000","last_clean":"2024-01-16T15:23:37.044424+0000","last_became_active":"2024-01-16T15:15:06.554470+0000","last_became_peered":"2024-01-16T15:15:06.554470+0000","last_unstale":"2024-01-16T15:23:37.044424+0000","last_undegraded":"2024-01-16T15:23:37.044424+0000","last_fullsized":"2024-01-16T15:23:37.044424+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:37.044057+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.044057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:58.743730+0000","scrub_duration":0.0080528830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"113'75","reported_seq":115,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:51.021247+0000","last_change":"2024-01-16T15:23:51.021247+0000","last_active":"2024-01-16T15:23:51.021247+0000","last_peered":"2024-01-16T15:23:51.021247+0000","last_clean":"2024-01-16T15:23:51.021247+0000","last_became_active":"2024-01-16T15:15:07.592994+0000","last_became_peered":"2024-01-16T15:15:07.592994+0000","last_unstale":"2024-01-16T15:23:51.021247+0000","last_undegraded":"2024-01-16T15:23:51.021247+0000","last_fullsized":"2024-01-16T15:23:51.021247+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'75","last_scrub_stamp":"2024-01-16T15:23:51.020999+0000","last_deep_scrub":"113'75","last_deep_scrub_stamp":"2024-01-16T15:23:51.020999+0000","last_clean_scrub_stamp":"2024-01-16T15:23:51.020999+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:46.535025+0000","scrub_duration":0.151350661,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'2","reported_seq":29,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:36.204678+0000","last_change":"2024-01-16T15:23:36.204678+0000","last_active":"2024-01-16T15:23:36.204678+0000","last_peered":"2024-01-16T15:23:36.204678+0000","last_clean":"2024-01-16T15:23:36.204678+0000","last_became_active":"2024-01-16T15:15:06.557578+0000","last_became_peered":"2024-01-16T15:15:06.557578+0000","last_unstale":"2024-01-16T15:23:36.204678+0000","last_undegraded":"2024-01-16T15:23:36.204678+0000","last_fullsized":"2024-01-16T15:23:36.204678+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-01-16T15:23:36.204448+0000","last_clean_scrub_stamp":"2024-01-16T15:23:36.204448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:33.212894+0000","scrub_duration":0.0074182859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"113'89","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.418610+0000","last_change":"2024-01-16T15:24:01.418610+0000","last_active":"2024-01-16T15:24:01.418610+0000","last_peered":"2024-01-16T15:24:01.418610+0000","last_clean":"2024-01-16T15:24:01.418610+0000","last_became_active":"2024-01-16T15:15:07.585542+0000","last_became_peered":"2024-01-16T15:15:07.585542+0000","last_unstale":"2024-01-16T15:24:01.418610+0000","last_undegraded":"2024-01-16T15:24:01.418610+0000","last_fullsized":"2024-01-16T15:24:01.418610+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'89","last_scrub_stamp":"2024-01-16T15:24:01.418314+0000","last_deep_scrub":"113'89","last_deep_scrub_stamp":"2024-01-16T15:24:01.418314+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.418314+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:50.120896+0000","scrub_duration":0.35096044399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.200689+0000","last_change":"2024-01-16T15:23:50.200689+0000","last_active":"2024-01-16T15:23:50.200689+0000","last_peered":"2024-01-16T15:23:50.200689+0000","last_clean":"2024-01-16T15:23:50.200689+0000","last_became_active":"2024-01-16T15:15:06.561421+0000","last_became_peered":"2024-01-16T15:15:06.561421+0000","last_unstale":"2024-01-16T15:23:50.200689+0000","last_undegraded":"2024-01-16T15:23:50.200689+0000","last_fullsized":"2024-01-16T15:23:50.200689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:50.200547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:50.200547+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.200547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:08.908687+0000","scrub_duration":0.0031404839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"113'80","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.087001+0000","last_change":"2024-01-16T15:23:45.087001+0000","last_active":"2024-01-16T15:23:45.087001+0000","last_peered":"2024-01-16T15:23:45.087001+0000","last_clean":"2024-01-16T15:23:45.087001+0000","last_became_active":"2024-01-16T15:15:07.580864+0000","last_became_peered":"2024-01-16T15:15:07.580864+0000","last_unstale":"2024-01-16T15:23:45.087001+0000","last_undegraded":"2024-01-16T15:23:45.087001+0000","last_fullsized":"2024-01-16T15:23:45.087001+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'80","last_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_deep_scrub":"113'80","last_deep_scrub_stamp":"2024-01-16T15:23:45.086751+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.086751+0000","objects_scrubbed":26,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:06.380394+0000","scrub_duration":0.32344294400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'4","reported_seq":31,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:31.492439+0000","last_change":"2024-01-16T15:23:31.492439+0000","last_active":"2024-01-16T15:23:31.492439+0000","last_peered":"2024-01-16T15:23:31.492439+0000","last_clean":"2024-01-16T15:23:31.492439+0000","last_became_active":"2024-01-16T15:15:06.561401+0000","last_became_peered":"2024-01-16T15:15:06.561401+0000","last_unstale":"2024-01-16T15:23:31.492439+0000","last_undegraded":"2024-01-16T15:23:31.492439+0000","last_fullsized":"2024-01-16T15:23:31.492439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4","last_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_deep_scrub":"113'4","last_deep_scrub_stamp":"2024-01-16T15:23:31.492299+0000","last_clean_scrub_stamp":"2024-01-16T15:23:31.492299+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:57.929823+0000","scrub_duration":0.0056823760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"113'96","reported_seq":137,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.431332+0000","last_change":"2024-01-16T15:24:10.431396+0000","last_active":"2024-01-16T15:24:10.431332+0000","last_peered":"2024-01-16T15:24:10.431332+0000","last_clean":"2024-01-16T15:24:10.431332+0000","last_became_active":"2024-01-16T15:15:07.586460+0000","last_became_peered":"2024-01-16T15:15:07.586460+0000","last_unstale":"2024-01-16T15:24:10.431332+0000","last_undegraded":"2024-01-16T15:24:10.431332+0000","last_fullsized":"2024-01-16T15:24:10.431332+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'96","last_scrub_stamp":"2024-01-16T15:24:10.431092+0000","last_deep_scrub":"113'96","last_deep_scrub_stamp":"2024-01-16T15:24:10.431092+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.431092+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:12.679729+0000","scrub_duration":0.30352710999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.044075+0000","last_change":"2024-01-16T15:23:38.044075+0000","last_active":"2024-01-16T15:23:38.044075+0000","last_peered":"2024-01-16T15:23:38.044075+0000","last_clean":"2024-01-16T15:23:38.044075+0000","last_became_active":"2024-01-16T15:15:06.561827+0000","last_became_peered":"2024-01-16T15:15:06.561827+0000","last_unstale":"2024-01-16T15:23:38.044075+0000","last_undegraded":"2024-01-16T15:23:38.044075+0000","last_fullsized":"2024-01-16T15:23:38.044075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:38.043846+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.043846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:00.375667+0000","scrub_duration":0.0035534389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"113'84","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:46.072906+0000","last_change":"2024-01-16T15:23:46.072906+0000","last_active":"2024-01-16T15:23:46.072906+0000","last_peered":"2024-01-16T15:23:46.072906+0000","last_clean":"2024-01-16T15:23:46.072906+0000","last_became_active":"2024-01-16T15:15:07.575946+0000","last_became_peered":"2024-01-16T15:15:07.575946+0000","last_unstale":"2024-01-16T15:23:46.072906+0000","last_undegraded":"2024-01-16T15:23:46.072906+0000","last_fullsized":"2024-01-16T15:23:46.072906+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:23:46.072694+0000","last_clean_scrub_stamp":"2024-01-16T15:23:46.072694+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:53:22.491262+0000","scrub_duration":0.27693848900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.532350+0000","last_change":"2024-01-16T15:23:30.532350+0000","last_active":"2024-01-16T15:23:30.532350+0000","last_peered":"2024-01-16T15:23:30.532350+0000","last_clean":"2024-01-16T15:23:30.532350+0000","last_became_active":"2024-01-16T15:15:06.560187+0000","last_became_peered":"2024-01-16T15:15:06.560187+0000","last_unstale":"2024-01-16T15:23:30.532350+0000","last_undegraded":"2024-01-16T15:23:30.532350+0000","last_fullsized":"2024-01-16T15:23:30.532350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:30.532200+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.532200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:36.853744+0000","scrub_duration":0.0022174130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"113'77","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.010592+0000","last_change":"2024-01-16T15:23:57.010680+0000","last_active":"2024-01-16T15:23:57.010592+0000","last_peered":"2024-01-16T15:23:57.010592+0000","last_clean":"2024-01-16T15:23:57.010592+0000","last_became_active":"2024-01-16T15:15:07.595288+0000","last_became_peered":"2024-01-16T15:15:07.595288+0000","last_unstale":"2024-01-16T15:23:57.010592+0000","last_undegraded":"2024-01-16T15:23:57.010592+0000","last_fullsized":"2024-01-16T15:23:57.010592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'77","last_scrub_stamp":"2024-01-16T15:23:57.010482+0000","last_deep_scrub":"113'77","last_deep_scrub_stamp":"2024-01-16T15:23:57.010482+0000","last_clean_scrub_stamp":"2024-01-16T15:23:57.010482+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:58.858184+0000","scrub_duration":0.234133231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.996817+0000","last_change":"2024-01-16T15:23:47.996817+0000","last_active":"2024-01-16T15:23:47.996817+0000","last_peered":"2024-01-16T15:23:47.996817+0000","last_clean":"2024-01-16T15:23:47.996817+0000","last_became_active":"2024-01-16T15:15:06.563960+0000","last_became_peered":"2024-01-16T15:15:06.563960+0000","last_unstale":"2024-01-16T15:23:47.996817+0000","last_undegraded":"2024-01-16T15:23:47.996817+0000","last_fullsized":"2024-01-16T15:23:47.996817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:47.996601+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.996601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:48.743854+0000","scrub_duration":0.0036748639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"113'67","reported_seq":99,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.245238+0000","last_change":"2024-01-16T15:23:45.245238+0000","last_active":"2024-01-16T15:23:45.245238+0000","last_peered":"2024-01-16T15:23:45.245238+0000","last_clean":"2024-01-16T15:23:45.245238+0000","last_became_active":"2024-01-16T15:15:07.598817+0000","last_became_peered":"2024-01-16T15:15:07.598817+0000","last_unstale":"2024-01-16T15:23:45.245238+0000","last_undegraded":"2024-01-16T15:23:45.245238+0000","last_fullsized":"2024-01-16T15:23:45.245238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'67","last_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_deep_scrub":"113'67","last_deep_scrub_stamp":"2024-01-16T15:23:45.244953+0000","last_clean_scrub_stamp":"2024-01-16T15:23:45.244953+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:26.176614+0000","scrub_duration":0.199400785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'3","reported_seq":30,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:38.817445+0000","last_change":"2024-01-16T15:23:38.817445+0000","last_active":"2024-01-16T15:23:38.817445+0000","last_peered":"2024-01-16T15:23:38.817445+0000","last_clean":"2024-01-16T15:23:38.817445+0000","last_became_active":"2024-01-16T15:15:06.565188+0000","last_became_peered":"2024-01-16T15:15:06.565188+0000","last_unstale":"2024-01-16T15:23:38.817445+0000","last_undegraded":"2024-01-16T15:23:38.817445+0000","last_fullsized":"2024-01-16T15:23:38.817445+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3","last_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_deep_scrub":"113'3","last_deep_scrub_stamp":"2024-01-16T15:23:38.817208+0000","last_clean_scrub_stamp":"2024-01-16T15:23:38.817208+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:41.655035+0000","scrub_duration":0.0067620090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"113'83","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.194713+0000","last_change":"2024-01-16T15:23:30.194713+0000","last_active":"2024-01-16T15:23:30.194713+0000","last_peered":"2024-01-16T15:23:30.194713+0000","last_clean":"2024-01-16T15:23:30.194713+0000","last_became_active":"2024-01-16T15:15:07.591679+0000","last_became_peered":"2024-01-16T15:15:07.591679+0000","last_unstale":"2024-01-16T15:23:30.194713+0000","last_undegraded":"2024-01-16T15:23:30.194713+0000","last_fullsized":"2024-01-16T15:23:30.194713+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'83","last_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_deep_scrub":"113'83","last_deep_scrub_stamp":"2024-01-16T15:23:30.194521+0000","last_clean_scrub_stamp":"2024-01-16T15:23:30.194521+0000","objects_scrubbed":24,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:25.708546+0000","scrub_duration":0.24677183899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'32","reported_seq":59,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.130401+0000","last_change":"2024-01-16T15:23:39.130401+0000","last_active":"2024-01-16T15:23:39.130401+0000","last_peered":"2024-01-16T15:23:39.130401+0000","last_clean":"2024-01-16T15:23:39.130401+0000","last_became_active":"2024-01-16T15:15:06.559518+0000","last_became_peered":"2024-01-16T15:15:06.559518+0000","last_unstale":"2024-01-16T15:23:39.130401+0000","last_undegraded":"2024-01-16T15:23:39.130401+0000","last_fullsized":"2024-01-16T15:23:39.130401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'32","last_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_deep_scrub":"113'32","last_deep_scrub_stamp":"2024-01-16T15:23:39.130166+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.130166+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:57.838306+0000","scrub_duration":0.0043416110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705,"num_omap_keys":13,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"113'99","reported_seq":146,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:12.545776+0000","last_change":"2024-01-16T15:24:12.545841+0000","last_active":"2024-01-16T15:24:12.545776+0000","last_peered":"2024-01-16T15:24:12.545776+0000","last_clean":"2024-01-16T15:24:12.545776+0000","last_became_active":"2024-01-16T15:15:07.600468+0000","last_became_peered":"2024-01-16T15:15:07.600468+0000","last_unstale":"2024-01-16T15:24:12.545776+0000","last_undegraded":"2024-01-16T15:24:12.545776+0000","last_fullsized":"2024-01-16T15:24:12.545776+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'99","last_scrub_stamp":"2024-01-16T15:24:12.545585+0000","last_deep_scrub":"113'99","last_deep_scrub_stamp":"2024-01-16T15:24:12.545585+0000","last_clean_scrub_stamp":"2024-01-16T15:24:12.545585+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:27:16.283445+0000","scrub_duration":0.397784313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.997068+0000","last_change":"2024-01-16T15:23:49.997120+0000","last_active":"2024-01-16T15:23:49.997068+0000","last_peered":"2024-01-16T15:23:49.997068+0000","last_clean":"2024-01-16T15:23:49.997068+0000","last_became_active":"2024-01-16T15:15:06.562261+0000","last_became_peered":"2024-01-16T15:15:06.562261+0000","last_unstale":"2024-01-16T15:23:49.997068+0000","last_undegraded":"2024-01-16T15:23:49.997068+0000","last_fullsized":"2024-01-16T15:23:49.997068+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:49.996799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:49.996799+0000","last_clean_scrub_stamp":"2024-01-16T15:23:49.996799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:12.037675+0000","scrub_duration":0.0049679720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"113'79","reported_seq":117,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:43.416185+0000","last_change":"2024-01-16T15:23:43.416246+0000","last_active":"2024-01-16T15:23:43.416185+0000","last_peered":"2024-01-16T15:23:43.416185+0000","last_clean":"2024-01-16T15:23:43.416185+0000","last_became_active":"2024-01-16T15:15:07.591756+0000","last_became_peered":"2024-01-16T15:15:07.591756+0000","last_unstale":"2024-01-16T15:23:43.416185+0000","last_undegraded":"2024-01-16T15:23:43.416185+0000","last_fullsized":"2024-01-16T15:23:43.416185+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_deep_scrub":"113'79","last_deep_scrub_stamp":"2024-01-16T15:23:43.416030+0000","last_clean_scrub_stamp":"2024-01-16T15:23:43.416030+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:24.455972+0000","scrub_duration":0.30055066899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.851113+0000","last_change":"2024-01-16T15:23:39.851159+0000","last_active":"2024-01-16T15:23:39.851113+0000","last_peered":"2024-01-16T15:23:39.851113+0000","last_clean":"2024-01-16T15:23:39.851113+0000","last_became_active":"2024-01-16T15:15:06.559800+0000","last_became_peered":"2024-01-16T15:15:06.559800+0000","last_unstale":"2024-01-16T15:23:39.851113+0000","last_undegraded":"2024-01-16T15:23:39.851113+0000","last_fullsized":"2024-01-16T15:23:39.851113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.850958+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.850958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:29.841870+0000","scrub_duration":0.003185556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"113'84","reported_seq":118,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.117719+0000","last_change":"2024-01-16T15:24:06.117806+0000","last_active":"2024-01-16T15:24:06.117719+0000","last_peered":"2024-01-16T15:24:06.117719+0000","last_clean":"2024-01-16T15:24:06.117719+0000","last_became_active":"2024-01-16T15:15:07.586345+0000","last_became_peered":"2024-01-16T15:15:07.586345+0000","last_unstale":"2024-01-16T15:24:06.117719+0000","last_undegraded":"2024-01-16T15:24:06.117719+0000","last_fullsized":"2024-01-16T15:24:06.117719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-01-16T15:24:06.117538+0000","last_deep_scrub":"113'84","last_deep_scrub_stamp":"2024-01-16T15:24:06.117538+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.117538+0000","objects_scrubbed":21,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:45.913228+0000","scrub_duration":0.26484186300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.074149+0000","last_change":"2024-01-16T15:23:41.074229+0000","last_active":"2024-01-16T15:23:41.074149+0000","last_peered":"2024-01-16T15:23:41.074149+0000","last_clean":"2024-01-16T15:23:41.074149+0000","last_became_active":"2024-01-16T15:15:06.558462+0000","last_became_peered":"2024-01-16T15:15:06.558462+0000","last_unstale":"2024-01-16T15:23:41.074149+0000","last_undegraded":"2024-01-16T15:23:41.074149+0000","last_fullsized":"2024-01-16T15:23:41.074149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:41.073835+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.073835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:56.198760+0000","scrub_duration":0.004400836,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"113'93","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.157245+0000","last_change":"2024-01-16T15:23:50.157313+0000","last_active":"2024-01-16T15:23:50.157245+0000","last_peered":"2024-01-16T15:23:50.157245+0000","last_clean":"2024-01-16T15:23:50.157245+0000","last_became_active":"2024-01-16T15:15:07.576394+0000","last_became_peered":"2024-01-16T15:15:07.576394+0000","last_unstale":"2024-01-16T15:23:50.157245+0000","last_undegraded":"2024-01-16T15:23:50.157245+0000","last_fullsized":"2024-01-16T15:23:50.157245+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'93","last_scrub_stamp":"2024-01-16T15:23:50.157078+0000","last_deep_scrub":"113'93","last_deep_scrub_stamp":"2024-01-16T15:23:50.157078+0000","last_clean_scrub_stamp":"2024-01-16T15:23:50.157078+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:20.546685+0000","scrub_duration":0.333928897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:33.837299+0000","last_change":"2024-01-16T15:23:33.837346+0000","last_active":"2024-01-16T15:23:33.837299+0000","last_peered":"2024-01-16T15:23:33.837299+0000","last_clean":"2024-01-16T15:23:33.837299+0000","last_became_active":"2024-01-16T15:15:06.558892+0000","last_became_peered":"2024-01-16T15:15:06.558892+0000","last_unstale":"2024-01-16T15:23:33.837299+0000","last_undegraded":"2024-01-16T15:23:33.837299+0000","last_fullsized":"2024-01-16T15:23:33.837299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:33.837111+0000","last_clean_scrub_stamp":"2024-01-16T15:23:33.837111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:32.726849+0000","scrub_duration":0.0035917940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"113'95","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.161310+0000","last_change":"2024-01-16T15:23:47.161339+0000","last_active":"2024-01-16T15:23:47.161310+0000","last_peered":"2024-01-16T15:23:47.161310+0000","last_clean":"2024-01-16T15:23:47.161310+0000","last_became_active":"2024-01-16T15:15:07.589350+0000","last_became_peered":"2024-01-16T15:15:07.589350+0000","last_unstale":"2024-01-16T15:23:47.161310+0000","last_undegraded":"2024-01-16T15:23:47.161310+0000","last_fullsized":"2024-01-16T15:23:47.161310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'95","last_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_deep_scrub":"113'95","last_deep_scrub_stamp":"2024-01-16T15:23:47.161227+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.161227+0000","objects_scrubbed":29,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:50.112996+0000","scrub_duration":0.32985411999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"113'34","reported_seq":93,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.113907+0000","last_change":"2024-01-16T15:23:47.114004+0000","last_active":"2024-01-16T15:23:47.113907+0000","last_peered":"2024-01-16T15:23:47.113907+0000","last_clean":"2024-01-16T15:23:47.113907+0000","last_became_active":"2024-01-16T15:15:06.557566+0000","last_became_peered":"2024-01-16T15:15:06.557566+0000","last_unstale":"2024-01-16T15:23:47.113907+0000","last_undegraded":"2024-01-16T15:23:47.113907+0000","last_fullsized":"2024-01-16T15:23:47.113907+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'34","last_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_deep_scrub":"113'34","last_deep_scrub_stamp":"2024-01-16T15:23:47.113654+0000","last_clean_scrub_stamp":"2024-01-16T15:23:47.113654+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:00.387823+0000","scrub_duration":0.0063658849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3644,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"113'102","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.569176+0000","last_change":"2024-01-16T15:24:09.569241+0000","last_active":"2024-01-16T15:24:09.569176+0000","last_peered":"2024-01-16T15:24:09.569176+0000","last_clean":"2024-01-16T15:24:09.569176+0000","last_became_active":"2024-01-16T15:15:28.523994+0000","last_became_peered":"2024-01-16T15:15:28.523994+0000","last_unstale":"2024-01-16T15:24:09.569176+0000","last_undegraded":"2024-01-16T15:24:09.569176+0000","last_fullsized":"2024-01-16T15:24:09.569176+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'102","last_scrub_stamp":"2024-01-16T15:24:09.569013+0000","last_deep_scrub":"113'102","last_deep_scrub_stamp":"2024-01-16T15:24:09.569013+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.569013+0000","objects_scrubbed":32,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:30.862656+0000","scrub_duration":0.35738116199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.978255+0000","last_change":"2024-01-16T15:23:39.978309+0000","last_active":"2024-01-16T15:23:39.978255+0000","last_peered":"2024-01-16T15:23:39.978255+0000","last_clean":"2024-01-16T15:23:39.978255+0000","last_became_active":"2024-01-16T15:15:06.562371+0000","last_became_peered":"2024-01-16T15:15:06.562371+0000","last_unstale":"2024-01-16T15:23:39.978255+0000","last_undegraded":"2024-01-16T15:23:39.978255+0000","last_fullsized":"2024-01-16T15:23:39.978255+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:39.977987+0000","last_clean_scrub_stamp":"2024-01-16T15:23:39.977987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:08.833788+0000","scrub_duration":0.0043159469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"113'62","reported_seq":101,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:41.078597+0000","last_change":"2024-01-16T15:23:41.078597+0000","last_active":"2024-01-16T15:23:41.078597+0000","last_peered":"2024-01-16T15:23:41.078597+0000","last_clean":"2024-01-16T15:23:41.078597+0000","last_became_active":"2024-01-16T15:15:07.575362+0000","last_became_peered":"2024-01-16T15:15:07.575362+0000","last_unstale":"2024-01-16T15:23:41.078597+0000","last_undegraded":"2024-01-16T15:23:41.078597+0000","last_fullsized":"2024-01-16T15:23:41.078597+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'62","last_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_deep_scrub":"113'62","last_deep_scrub_stamp":"2024-01-16T15:23:41.078271+0000","last_clean_scrub_stamp":"2024-01-16T15:23:41.078271+0000","objects_scrubbed":17,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:52.687411+0000","scrub_duration":0.197239254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:37.012089+0000","last_change":"2024-01-16T15:23:37.012089+0000","last_active":"2024-01-16T15:23:37.012089+0000","last_peered":"2024-01-16T15:23:37.012089+0000","last_clean":"2024-01-16T15:23:37.012089+0000","last_became_active":"2024-01-16T15:15:06.558327+0000","last_became_peered":"2024-01-16T15:15:06.558327+0000","last_unstale":"2024-01-16T15:23:37.012089+0000","last_undegraded":"2024-01-16T15:23:37.012089+0000","last_fullsized":"2024-01-16T15:23:37.012089+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:23:37.011669+0000","last_clean_scrub_stamp":"2024-01-16T15:23:37.011669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:27.915725+0000","scrub_duration":0.0040625560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"113'119","reported_seq":282,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:23:34.845903+0000","last_change":"2024-01-16T15:23:34.845995+0000","last_active":"2024-01-16T15:23:34.845903+0000","last_peered":"2024-01-16T15:23:34.845903+0000","last_clean":"2024-01-16T15:23:34.845903+0000","last_became_active":"2024-01-16T14:56:35.665270+0000","last_became_peered":"2024-01-16T14:56:35.665270+0000","last_unstale":"2024-01-16T15:23:34.845903+0000","last_undegraded":"2024-01-16T15:23:34.845903+0000","last_fullsized":"2024-01-16T15:23:34.845903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'119","last_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_deep_scrub":"113'119","last_deep_scrub_stamp":"2024-01-16T15:23:34.845733+0000","last_clean_scrub_stamp":"2024-01-16T15:23:34.845733+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:46.454874+0000","scrub_duration":0.058578549000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442453004,"num_objects":1546,"num_object_clones":1,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":1,"num_read":798,"num_read_kb":3145758,"num_write":1604,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884975616,"data_stored":12884906008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5578,"ondisk_log_size":5578,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1225687,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":489,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5705,"num_omap_keys":23,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2506752,"data_stored":2451374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575188,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2092604,"kb_used_data":2025396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92279236,"statfs":{"total":96636764160,"available":94493937664,"internally_reserved":0,"allocated":2074005504,"data_stored":2073186049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.151,"15min":1.143},"min":{"1min":0.755,"5min":0.66000000000000003,"15min":0.628},"max":{"1min":1.927,"5min":2.0259999999999998,"15min":3.3969999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.198,"15min":1.2190000000000001},"min":{"1min":0.623,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.004,"5min":2.004,"15min":5.9619999999999997},"last":1.5029999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1830000000000001,"15min":1.2549999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.69699999999999995},"max":{"1min":1.6699999999999999,"5min":2.0840000000000001,"15min":8.1029999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7430000000000001,"5min":1.956,"15min":6.2690000000000001},"last":1.4490000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.21,"15min":1.2090000000000001},"min":{"1min":0.67000000000000004,"5min":0.629,"15min":0.44600000000000001},"max":{"1min":1.5640000000000001,"5min":2.0070000000000001,"15min":2.5910000000000002},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2709999999999999,"15min":1.2689999999999999},"min":{"1min":0.95199999999999996,"5min":0.82199999999999995,"15min":0.70099999999999996},"max":{"1min":1.879,"5min":1.879,"15min":2.6920000000000002},"last":1.0269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2829999999999999,"15min":1.3380000000000001},"min":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.65900000000000003},"max":{"1min":1.4990000000000001,"5min":1.9279999999999999,"15min":12.331},"last":1.145},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.373,"15min":1.3560000000000001},"min":{"1min":0.90400000000000003,"5min":0.75700000000000001,"15min":0.55500000000000005},"max":{"1min":1.954,"5min":2.0649999999999999,"15min":2.645},"last":1.2050000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3220000000000001,"15min":1.3560000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.49199999999999999},"max":{"1min":2.109,"5min":2.109,"15min":6.1799999999999997},"last":1.302},{"interface":"front","average":{"1min":1.431,"5min":1.321,"15min":1.319},"min":{"1min":1.0780000000000001,"5min":0.68700000000000006,"15min":0.45800000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":2.4159999999999999},"last":1.266}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.397,"15min":1.401},"min":{"1min":0.88900000000000001,"5min":0.73899999999999999,"15min":0.60299999999999998},"max":{"1min":2.4350000000000001,"5min":2.508,"15min":2.593},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4930000000000001,"15min":1.532},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.54700000000000004},"max":{"1min":2.2050000000000001,"5min":2.8780000000000001,"15min":6.3559999999999999},"last":1.55}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.464,"15min":1.496},"min":{"1min":0.78800000000000003,"5min":0.745,"15min":0.59899999999999998},"max":{"1min":1.988,"5min":2.0609999999999999,"15min":3.7639999999999998},"last":1.335},{"interface":"front","average":{"1min":1.621,"5min":1.5649999999999999,"15min":1.5860000000000001},"min":{"1min":1.032,"5min":0.93000000000000005,"15min":0.57799999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":4.7649999999999997},"last":1.476}]}]},{"osd":6,"up_from":13,"seq":55834575188,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1469360,"kb_used_data":1406312,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92902480,"statfs":{"total":96636764160,"available":95132139520,"internally_reserved":0,"allocated":1440063488,"data_stored":1439254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2989999999999999,"15min":1.242},"min":{"1min":1.014,"5min":0.64800000000000002,"15min":0.61899999999999999},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":2.6970000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3300000000000001,"15min":1.304},"min":{"1min":0.80700000000000005,"5min":0.69199999999999995,"15min":0.64600000000000002},"max":{"1min":1.823,"5min":2.0720000000000001,"15min":2.2000000000000002},"last":1.214}]},{"osd":1,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.141,"15min":1.155},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.64700000000000002},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":4.7110000000000003},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.2430000000000001,"15min":1.218},"min":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.639,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.421}]},{"osd":2,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.236,"15min":1.29},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.72699999999999998},"max":{"1min":1.7170000000000001,"5min":1.74,"15min":5.7439999999999998},"last":1.304},{"interface":"front","average":{"1min":1.321,"5min":1.337,"15min":1.359},"min":{"1min":0.80000000000000004,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7130000000000001,"5min":1.869,"15min":5.5869999999999997},"last":1.6160000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.355,"15min":1.345},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.77800000000000002},"max":{"1min":1.621,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.46},{"interface":"front","average":{"1min":1.482,"5min":1.4319999999999999,"15min":1.417},"min":{"1min":1.123,"5min":0.94799999999999995,"15min":0.753},"max":{"1min":1.837,"5min":2.2320000000000002,"15min":2.9300000000000002},"last":1.105}]},{"osd":4,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.417,"15min":1.381},"min":{"1min":0.94499999999999995,"5min":0.79500000000000004,"15min":0.627},"max":{"1min":1.9299999999999999,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.3480000000000001,"15min":1.3959999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.60299999999999998},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":6.7809999999999997},"last":1.0589999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.337,"15min":1.327},"min":{"1min":0.754,"5min":0.754,"15min":0.55300000000000005},"max":{"1min":1.847,"5min":2.012,"15min":3.2029999999999998},"last":1.161},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.538,"15min":1.5269999999999999},"min":{"1min":1.1679999999999999,"5min":0.95699999999999996,"15min":0.77000000000000002},"max":{"1min":2.0819999999999999,"5min":2.1949999999999998,"15min":3.1579999999999999},"last":1.258}]},{"osd":7,"last update":"Tue Jan 16 15:24:18 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.528,"15min":1.5900000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.64700000000000002},"max":{"1min":2.2559999999999998,"5min":2.4390000000000001,"15min":24.675000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.4970000000000001,"15min":1.53},"min":{"1min":1.0229999999999999,"5min":0.97899999999999998,"15min":0.63900000000000001},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":5.2469999999999999},"last":1.371}]}]},{"osd":5,"up_from":13,"seq":55834575188,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191024,"kb_used_data":1115496,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93180816,"statfs":{"total":96636764160,"available":95417155584,"internally_reserved":0,"allocated":1142267904,"data_stored":1141459740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1719999999999999,"15min":1.145},"min":{"1min":0.629,"5min":0.59699999999999998,"15min":0.33600000000000002},"max":{"1min":1.7629999999999999,"5min":2.2040000000000002,"15min":3.7130000000000001},"last":1.7629999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.228,"15min":1.24},"min":{"1min":0.70099999999999996,"5min":0.57999999999999996,"15min":0.49299999999999999},"max":{"1min":1.569,"5min":1.8620000000000001,"15min":8.7880000000000003},"last":1.5129999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.179,"15min":1.21},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.28899999999999998},"max":{"1min":1.5029999999999999,"5min":2.1389999999999998,"15min":3.835},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3100000000000001,"15min":1.2809999999999999},"min":{"1min":0.91200000000000003,"5min":0.61799999999999999,"15min":0.42599999999999999},"max":{"1min":2.0329999999999999,"5min":2.0830000000000002,"15min":5.3940000000000001},"last":1.26}]},{"osd":2,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.272,"15min":1.3149999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.44500000000000001},"max":{"1min":1.6799999999999999,"5min":2.2669999999999999,"15min":5.2110000000000003},"last":1.468},{"interface":"front","average":{"1min":1.401,"5min":1.3779999999999999,"15min":1.361},"min":{"1min":0.76100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.865,"5min":2.0680000000000001,"15min":5.3070000000000004},"last":1.4510000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.3069999999999999,"15min":1.3360000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.38200000000000001},"max":{"1min":1.6970000000000001,"5min":1.742,"15min":3.9319999999999999},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4319999999999999,"15min":1.4399999999999999},"min":{"1min":0.83699999999999997,"5min":0.71799999999999997,"15min":0.53800000000000003},"max":{"1min":1.7889999999999999,"5min":2.3900000000000001,"15min":16.158000000000001},"last":1.399}]},{"osd":4,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2829999999999999,"15min":1.2350000000000001},"min":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.46700000000000003},"max":{"1min":1.679,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.123},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4830000000000001,"15min":1.496},"min":{"1min":0.50700000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.8759999999999999,"5min":2.3180000000000001,"15min":3.9889999999999999},"last":1.6599999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.4039999999999999,"15min":1.3640000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.46800000000000003},"max":{"1min":1.639,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.325,"15min":1.2989999999999999},"min":{"1min":0.55600000000000005,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":1.6479999999999999,"5min":2.351,"15min":2.351},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.4430000000000001,"15min":1.4650000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.41499999999999998},"max":{"1min":1.9039999999999999,"5min":2.145,"15min":3.8809999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.5489999999999999,"15min":1.5169999999999999},"min":{"1min":0.65700000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.9299999999999999,"5min":2.456,"15min":3.2509999999999999},"last":1.087}]}]},{"osd":4,"up_from":13,"seq":55834575187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932272,"kb_used_data":1865064,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439568,"statfs":{"total":96636764160,"available":94658117632,"internally_reserved":0,"allocated":1909825536,"data_stored":1909017209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3540000000000001,"15min":1.3660000000000001},"min":{"1min":0.85699999999999998,"5min":0.60899999999999999,"15min":0.57899999999999996},"max":{"1min":1.829,"5min":1.829,"15min":17.315999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":1.1719999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":17.279},"last":1.347}]},{"osd":1,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.228,"15min":1.2589999999999999},"min":{"1min":0.997,"5min":0.77800000000000002,"15min":0.498},"max":{"1min":1.5800000000000001,"5min":1.839,"15min":6.1849999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2090000000000001,"15min":1.177},"min":{"1min":0.91000000000000003,"5min":0.80000000000000004,"15min":0.64600000000000002},"max":{"1min":1.5489999999999999,"5min":1.9770000000000001,"15min":4.2809999999999997},"last":1.4590000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.93500000000000005,"5min":0.88300000000000001,"15min":0.55800000000000005},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":2.1299999999999999},"last":1.304},{"interface":"front","average":{"1min":1.333,"5min":1.363,"15min":1.343},"min":{"1min":1.0800000000000001,"5min":0.88600000000000001,"15min":0.75800000000000001},"max":{"1min":1.627,"5min":1.8120000000000001,"15min":2.012},"last":0.96699999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.262,"15min":1.3009999999999999},"min":{"1min":1.05,"5min":0.85699999999999998,"15min":0.66100000000000003},"max":{"1min":1.4770000000000001,"5min":1.6859999999999999,"15min":9.8350000000000009},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.3919999999999999,"15min":1.3680000000000001},"min":{"1min":1.0800000000000001,"5min":0.95999999999999996,"15min":0.75},"max":{"1min":1.754,"5min":1.77,"15min":4.7930000000000001},"last":1.486}]},{"osd":5,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.2949999999999999,"15min":1.3029999999999999},"min":{"1min":1.0169999999999999,"5min":0.69999999999999996,"15min":0.63},"max":{"1min":1.7949999999999999,"5min":1.8999999999999999,"15min":2.2549999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3440000000000001,"15min":1.339},"min":{"1min":0.97099999999999997,"5min":0.88400000000000001,"15min":0.47599999999999998},"max":{"1min":1.6699999999999999,"5min":1.8660000000000001,"15min":4.8520000000000003},"last":1.3839999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4330000000000001,"15min":1.3700000000000001},"min":{"1min":1.1439999999999999,"5min":0.96199999999999997,"15min":0.59399999999999997},"max":{"1min":2.0179999999999998,"5min":2.0230000000000001,"15min":2.3479999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.494,"15min":1.484},"min":{"1min":1.0940000000000001,"5min":0.92800000000000005,"15min":0.55800000000000005},"max":{"1min":1.9019999999999999,"5min":1.946,"15min":2.3889999999999998},"last":1.2529999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4770000000000001,"15min":1.466},"min":{"1min":1.1779999999999999,"5min":1.054,"15min":0.53600000000000003},"max":{"1min":1.873,"5min":1.968,"15min":4.9539999999999997},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.593,"15min":1.5489999999999999},"min":{"1min":1.4039999999999999,"5min":1.2609999999999999,"15min":0.64700000000000002},"max":{"1min":1.976,"5min":1.976,"15min":2.2599999999999998},"last":1.4259999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575188,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1822956,"kb_used_data":1751588,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92548884,"statfs":{"total":96636764160,"available":94770057216,"internally_reserved":0,"allocated":1793626112,"data_stored":1792798546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.94999999999999996,"15min":0.92000000000000004},"min":{"1min":0.54400000000000004,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5189999999999999,"5min":1.6240000000000001,"15min":1.925},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0620000000000001,"15min":1.026},"min":{"1min":0.65300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.355,"5min":6.6740000000000004,"15min":6.6740000000000004},"last":1.129}]},{"osd":1,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.96299999999999997,"15min":0.98399999999999999},"min":{"1min":0.63800000000000001,"5min":0.5,"15min":0.44900000000000001},"max":{"1min":1.716,"5min":1.716,"15min":3.1150000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.046,"5min":1.0960000000000001,"15min":1.0760000000000001},"min":{"1min":0.48599999999999999,"5min":0.42099999999999999,"15min":0.38500000000000001},"max":{"1min":1.399,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.1850000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.129,"15min":1.133},"min":{"1min":0.73399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5449999999999999,"5min":1.8169999999999999,"15min":3.0670000000000002},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1990000000000001,"15min":1.1950000000000001},"min":{"1min":0.68700000000000006,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.7410000000000001,"5min":1.986,"15min":3.4279999999999999},"last":1.071}]},{"osd":4,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3560000000000001,"15min":1.3440000000000001},"min":{"1min":1.0029999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.7390000000000001,"5min":2.1989999999999998,"15min":3.5049999999999999},"last":1.248},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.1359999999999999,"5min":0.85499999999999998,"15min":0.64000000000000001},"max":{"1min":1.99,"5min":1.99,"15min":3.8250000000000002},"last":1.228}]},{"osd":5,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.373,"15min":1.349},"min":{"1min":1.2,"5min":0.755,"15min":0.70999999999999996},"max":{"1min":1.6930000000000001,"5min":2.085,"15min":2.3540000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3959999999999999,"15min":1.399},"min":{"1min":1.1439999999999999,"5min":0.79500000000000004,"15min":0.71299999999999997},"max":{"1min":1.839,"5min":1.839,"15min":2.907},"last":1.482}]},{"osd":6,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.90400000000000003,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":6.4480000000000004},"last":1.405},{"interface":"front","average":{"1min":1.47,"5min":1.4399999999999999,"15min":1.4710000000000001},"min":{"1min":1.04,"5min":0.82299999999999995,"15min":0.81299999999999994},"max":{"1min":2.254,"5min":2.254,"15min":3.9039999999999999},"last":1.1639999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.468,"15min":1.474},"min":{"1min":1.0609999999999999,"5min":0.89700000000000002,"15min":0.73099999999999998},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":3.7850000000000001},"last":1.323},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5,"15min":1.512},"min":{"1min":1.214,"5min":0.79200000000000004,"15min":0.59899999999999998},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":3.9590000000000001},"last":1.3660000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575187,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1182700,"kb_used_data":1115492,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93189140,"statfs":{"total":96636764160,"available":95425679360,"internally_reserved":0,"allocated":1142263808,"data_stored":1141462711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.90000000000000002,"15min":0.88900000000000001},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.314},"max":{"1min":1.603,"5min":1.603,"15min":2.8919999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.024,"15min":1.036},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.42299999999999999},"max":{"1min":1.4350000000000001,"5min":1.76,"15min":2.952},"last":1.2470000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.034,"15min":1.0569999999999999},"min":{"1min":0.58799999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.5820000000000001,"5min":1.605,"15min":1.823},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.127},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3600000000000001,"5min":1.6479999999999999,"15min":4.734},"last":1.016}]},{"osd":3,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.028,"15min":1.0509999999999999},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.48499999999999999},"max":{"1min":1.403,"5min":1.788,"15min":1.788},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":0.63700000000000001,"5min":0.57599999999999996,"15min":0.51400000000000001},"max":{"1min":1.5660000000000001,"5min":1.8120000000000001,"15min":2.3220000000000001},"last":1.3029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.335,"15min":1.4770000000000001},"min":{"1min":0.84699999999999998,"5min":0.76000000000000001,"15min":0.65200000000000002},"max":{"1min":1.5660000000000001,"5min":2.0680000000000001,"15min":16.681999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.292,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.746},"max":{"1min":1.544,"5min":1.919,"15min":1.964},"last":1.181}]},{"osd":5,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.97199999999999998,"5min":0.745,"15min":0.72799999999999998},"max":{"1min":1.879,"5min":2.206,"15min":2.3799999999999999},"last":1.206},{"interface":"front","average":{"1min":1.373,"5min":1.4039999999999999,"15min":1.431},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":4.8659999999999997},"last":1.452}]},{"osd":6,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.373,"15min":1.3879999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.986,"5min":2.0720000000000001,"15min":2.1539999999999999},"last":1.383},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4710000000000001,"15min":1.484},"min":{"1min":1.022,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":1.887,"5min":2.1080000000000001,"15min":6.2599999999999998},"last":1.4970000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:23:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.472,"15min":1.49},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.70099999999999996},"max":{"1min":1.7010000000000001,"5min":2.125,"15min":3.7530000000000001},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.4379999999999999,"15min":1.45},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.78400000000000003},"max":{"1min":1.6950000000000001,"5min":2.0670000000000002,"15min":2.294},"last":1.117}]}]},{"osd":1,"up_from":13,"seq":55834575188,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1408048,"kb_used_data":1336680,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92963792,"statfs":{"total":96636764160,"available":95194923008,"internally_reserved":0,"allocated":1368760320,"data_stored":1367955059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.91200000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.399,"5min":1.4330000000000001,"15min":14.106999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.03,"15min":1.0569999999999999},"min":{"1min":0.47899999999999998,"5min":0.42599999999999999,"15min":0.35899999999999999},"max":{"1min":1.504,"5min":1.504,"15min":4.3719999999999999},"last":1.119}]},{"osd":2,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.96299999999999997,"15min":0.96499999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.29799999999999999},"max":{"1min":1.2709999999999999,"5min":3.5409999999999999,"15min":3.6309999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.995,"5min":1.0740000000000001,"15min":1.1339999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.436},"max":{"1min":1.3720000000000001,"5min":1.663,"15min":14.055},"last":1.012}]},{"osd":3,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.22,"15min":1.266},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.48,"5min":1.6439999999999999,"15min":13.967000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1060000000000001,"15min":1.135},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5840000000000001,"5min":1.952,"15min":4.6699999999999999},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.228,"15min":1.244},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52500000000000002},"max":{"1min":1.698,"5min":3.77,"15min":4.3799999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.3540000000000001,"15min":1.4650000000000001},"min":{"1min":0.89700000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":14.161},"last":1.169}]},{"osd":5,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.5369999999999999},"min":{"1min":1.0289999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7450000000000001,"5min":1.7789999999999999,"15min":17.739999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.413,"15min":1.508},"min":{"1min":1.079,"5min":1.024,"15min":0.56100000000000005},"max":{"1min":1.8460000000000001,"5min":3.8090000000000002,"15min":16.433},"last":0.96199999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4390000000000001,"15min":1.4930000000000001},"min":{"1min":1.1100000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":14.212},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3979999999999999,"15min":1.444},"min":{"1min":0.872,"5min":0.872,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":2.2570000000000001,"15min":6.5350000000000001},"last":1.397}]},{"osd":7,"last update":"Tue Jan 16 15:23:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.431,"15min":1.5209999999999999},"min":{"1min":0.94099999999999995,"5min":0.82499999999999996,"15min":0.78100000000000003},"max":{"1min":1.8460000000000001,"5min":2.1930000000000001,"15min":12.590999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.4530000000000001,"15min":1.454},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.64100000000000001},"max":{"1min":1.7829999999999999,"5min":3.726,"15min":6.0229999999999997},"last":1.2110000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575187,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2048744,"kb_used_data":1981536,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92323096,"statfs":{"total":96636764160,"available":94538850304,"internally_reserved":0,"allocated":2029092864,"data_stored":2028268369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.84499999999999997,"15min":0.84399999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28799999999999998},"max":{"1min":1.552,"5min":1.552,"15min":1.921},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0389999999999999,"15min":1.0129999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.41799999999999998},"max":{"1min":1.6659999999999999,"5min":1.72,"15min":4.4059999999999997},"last":0.98899999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.012,"15min":1.0580000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":17.260000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1200000000000001,"15min":1.097},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.33600000000000002},"max":{"1min":1.571,"5min":1.784,"15min":4.492},"last":1.0880000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.137,"15min":1.0780000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.33100000000000002},"max":{"1min":1.732,"5min":1.732,"15min":2.2789999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.3280000000000001,"15min":1.292},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":1.7529999999999999,"5min":1.8400000000000001,"15min":2.319},"last":1.3220000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.341,"15min":1.339},"min":{"1min":0.875,"5min":0.76000000000000001,"15min":0.60799999999999998},"max":{"1min":1.6699999999999999,"5min":2.0459999999999998,"15min":4.2889999999999997},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3320000000000001,"15min":1.2909999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.629,"5min":4.0810000000000004,"15min":4.0810000000000004},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4370000000000001,"15min":1.4179999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.624},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":3.1259999999999999},"last":1.8069999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4950000000000001,"15min":1.486},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.66800000000000004},"max":{"1min":2.1890000000000001,"5min":2.2730000000000001,"15min":6.5430000000000001},"last":1.675}]},{"osd":6,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.391,"15min":1.3839999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.65700000000000003},"max":{"1min":2.0270000000000001,"5min":4.0430000000000001,"15min":11.462},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.464,"15min":1.4550000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0779999999999998,"5min":2.093,"15min":7.1849999999999996},"last":1.881}]},{"osd":7,"last update":"Tue Jan 16 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.48,"15min":1.4930000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.56899999999999995},"max":{"1min":2.181,"5min":2.181,"15min":5.2450000000000001},"last":1.847},{"interface":"front","average":{"1min":1.476,"5min":1.5049999999999999,"15min":1.4610000000000001},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.68799999999999994},"max":{"1min":2.21,"5min":3.976,"15min":3.976},"last":1.5800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":1221357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1232896,"data_stored":1221315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025861120,"data_stored":2025849476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790988288,"data_stored":1790969044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140854784,"data_stored":1140850949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:24:35.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T15:24:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.491+0000 7f94b0a00640 1 Processor -- start 2024-01-16T15:24:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.491+0000 7f94b0a00640 1 -- start start 2024-01-16T15:24:35.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94b0a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94ac071e40 0x7f94ac0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94b0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.491 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94b0a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac15e140 con 0x7f94ac143810 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94b0a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac143bf0 con 0x7f94ac071e40 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94b0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0722f0 con 0x7f94ac15d770 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94a2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94ac071e40 0x7f94ac0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94a2800640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94ac071e40 0x7f94ac0ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57924/0 (socket says 172.21.15.125:57924) 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94a2800640 1 -- 172.21.15.125:0/3541745266 learned_addr learned my addr 172.21.15.125:0/3541745266 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94ab400640 1 --1- 172.21.15.125:0/3541745266 >> v1:172.21.15.125:6789/0 conn(0x7f94ac143810 0x7f94ac15e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52568/0 (socket says 172.21.15.125:52568) 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.493+0000 7f94abe00640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.494+0000 7f94a2800640 1 -- 172.21.15.125:0/3541745266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 msgr2=0x7f94ac15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.494+0000 7f94a2800640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac15dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.494+0000 7f94a2800640 1 -- 172.21.15.125:0/3541745266 >> v1:172.21.15.125:6789/0 conn(0x7f94ac143810 legacy=0x7f94ac15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.494+0000 7f94a2800640 1 -- 172.21.15.125:0/3541745266 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac06c730 con 0x7f94ac071e40 2024-01-16T15:24:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.494+0000 7f94aaa00640 1 -- 172.21.15.125:0/3541745266 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406195631 0 0) 0x7f94ac15e140 con 0x7f94ac143810 2024-01-16T15:24:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.495+0000 7f94a2800640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac0ae170 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f9494009870 tx=0x7f949402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa80cad65123dd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.495+0000 7f94aaa00640 1 -- 172.21.15.125:0/3541745266 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940317c0 con 0x7f94ac071e40 2024-01-16T15:24:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.495+0000 7f94aaa00640 1 -- 172.21.15.125:0/3541745266 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9494031920 con 0x7f94ac071e40 2024-01-16T15:24:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.495+0000 7f94aaa00640 1 -- 172.21.15.125:0/3541745266 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494031cb0 con 0x7f94ac071e40 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 -- 172.21.15.125:0/3541745266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 msgr2=0x7f94ac0ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac0ae170 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f9494009870 tx=0x7f949402b860 comp rx=0 tx=0).stop 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 -- 172.21.15.125:0/3541745266 shutdown_connections 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac15dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3541745266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac0ae170 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 -- 172.21.15.125:0/3541745266 >> 172.21.15.125:0/3541745266 conn(0x7f94ac06a7d0 msgr2=0x7f94ac06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 -- 172.21.15.125:0/3541745266 shutdown_connections 2024-01-16T15:24:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.496+0000 7f94b0a00640 1 -- 172.21.15.125:0/3541745266 wait complete. 2024-01-16T15:24:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.498+0000 7f94b0a00640 1 Processor -- start 2024-01-16T15:24:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.498+0000 7f94b0a00640 1 -- start start 2024-01-16T15:24:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 0x7f94ac0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0aea00 con 0x7f94ac071e40 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0af540 con 0x7f94ac143810 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94b0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0b2e50 con 0x7f94ac15d770 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57934/0 (socket says 172.21.15.125:57934) 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 -- 172.21.15.125:0/3396992063 learned_addr learned my addr 172.21.15.125:0/3396992063 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94abe00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94a2800640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 0x7f94ac0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 -- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 msgr2=0x7f94ac06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac06d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 -- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 msgr2=0x7f94ac0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 0x7f94ac0b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94ab400640 1 -- 172.21.15.125:0/3396992063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940094d0 con 0x7f94ac071e40 2024-01-16T15:24:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.499+0000 7f94a2800640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 0x7f94ac0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94ab400640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c4d0 tx=0x7f949c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0074e0 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c007660 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c004a20 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac0aec10 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.500+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac0af1d0 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.501+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac067450 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.502+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 51) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f949c004b80 con 0x7f94ac071e40 2024-01-16T15:24:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.503+0000 7f94a3e00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 0x7f948406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.503+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f949c087ce0 con 0x7f94ac071e40 2024-01-16T15:24:35.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.503+0000 7f94a2800640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 0x7f948406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.502 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.504+0000 7f94a2800640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 0x7f948406e980 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9494036000 tx=0x7f949403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.505 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.507+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f949c054290 con 0x7f94ac071e40 2024-01-16T15:24:35.700 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.703+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f94ac0b1e20 con 0x7f94ac071e40 2024-01-16T15:24:35.705 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.707+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v45)=0 v45) v1 ==== 123+0+0 (secure 0 0 0) 0x7f949c018020 con 0x7f94ac071e40 2024-01-16T15:24:35.707 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.709+0000 7f94a3e00640 1 -- 172.21.15.125:0/3396992063 <== mon.0 v2:172.21.15.125:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f949c054290 con 0x7f94ac071e40 2024-01-16T15:24:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.711+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 msgr2=0x7f948406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.711+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 0x7f948406e980 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9494036000 tx=0x7f949403d040 comp rx=0 tx=0).stop 2024-01-16T15:24:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 msgr2=0x7f94ac146920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c4d0 tx=0x7f949c00c990 comp rx=0 tx=0).stop 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 shutdown_connections 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f94ac15d770 0x7f94ac06d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94ac143810 0x7f94ac0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f948406c4c0 0x7f948406e980 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 --2- 172.21.15.125:0/3396992063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94ac071e40 0x7f94ac146920 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 >> 172.21.15.125:0/3396992063 conn(0x7f94ac06a7d0 msgr2=0x7f94ac06abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 shutdown_connections 2024-01-16T15:24:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.712+0000 7f94b0a00640 1 -- 172.21.15.125:0/3396992063 wait complete. 2024-01-16T15:24:35.730 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:24:35.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:24:35.730 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:24:35.732+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:24:35.731 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:24:35.732+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:24:35.731 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:24:35.732+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:24:35.731 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:24:35.732+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:24:35.731 INFO:tasks.ceph.mgr.y.smithi125.stderr:2024-01-16T15:24:35.732+0000 7f0101e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:24:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.916+0000 7f2ff7a00640 1 Processor -- start 2024-01-16T15:24:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.916+0000 7f2ff7a00640 1 -- start start 2024-01-16T15:24:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.917+0000 7f2ff7a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2ff0143ae0 0x7f2ff00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.917+0000 7f2ff7a00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff00cf7d0 0x7f2ff015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.917+0000 7f2ff7a00640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00cfc30 con 0x7f2ff015e1e0 2024-01-16T15:24:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.917+0000 7f2ff7a00640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00b1ee0 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.917+0000 7f2ff7a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00b2780 con 0x7f2ff0143ae0 2024-01-16T15:24:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.918+0000 7f2fe7c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff00cf7d0 0x7f2ff015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.918+0000 7f2ff5000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2ff0143ae0 0x7f2ff00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.918+0000 7f2ff5a00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2ff015e1e0 0x7f2ff015a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52582/0 (socket says 172.21.15.125:52582) 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.918+0000 7f2fe7c00640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff00cf7d0 0x7f2ff015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57958/0 (socket says 172.21.15.125:57958) 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.918+0000 7f2ff5a00640 1 -- 172.21.15.125:0/3260401280 learned_addr learned my addr 172.21.15.125:0/3260401280 (peer_addr_for_me v1:172.21.15.125:0/0) 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.919+0000 7f2fe7c00640 1 -- 172.21.15.125:0/3260401280 >> v2:172.21.15.137:3300/0 conn(0x7f2ff0143ae0 msgr2=0x7f2ff00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.919+0000 7f2fe7c00640 1 --2- 172.21.15.125:0/3260401280 >> v2:172.21.15.137:3300/0 conn(0x7f2ff0143ae0 0x7f2ff00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.919+0000 7f2fe7c00640 1 -- 172.21.15.125:0/3260401280 >> v1:172.21.15.125:6789/0 conn(0x7f2ff015e1e0 legacy=0x7f2ff015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.919+0000 7f2fe7c00640 1 -- 172.21.15.125:0/3260401280 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff015e990 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2fe7c00640 1 --2- 172.21.15.125:0/3260401280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff015dca0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00cfea0 tx=0x7f2fd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28196302953624a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2fefe00640 1 -- 172.21.15.125:0/3260401280 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8031830 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2fefe00640 1 -- 172.21.15.125:0/3260401280 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2fd8031990 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2fefe00640 1 -- 172.21.15.125:0/3260401280 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8031d40 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.920 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2ff7a00640 1 -- 172.21.15.125:0/3260401280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff00cf7d0 msgr2=0x7f2ff015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.920+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/3260401280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff015dca0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00cfea0 tx=0x7f2fd802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 -- 172.21.15.125:0/3260401280 shutdown_connections 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/3260401280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff015dca0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/3260401280 >> v2:172.21.15.137:3300/0 conn(0x7f2ff0143ae0 0x7f2ff00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 -- 172.21.15.125:0/3260401280 >> 172.21.15.125:0/3260401280 conn(0x7f2ff006a800 msgr2=0x7f2ff006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 -- 172.21.15.125:0/3260401280 shutdown_connections 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.921+0000 7f2ff7a00640 1 -- 172.21.15.125:0/3260401280 wait complete. 2024-01-16T15:24:35.921 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.922+0000 7f2ff7a00640 1 Processor -- start 2024-01-16T15:24:35.922 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.922+0000 7f2ff7a00640 1 -- start start 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff013e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff0143ae0 0x7f2ff0138f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff013d6c0 con 0x7f2ff015e1e0 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff013d9a0 con 0x7f2ff00cf7d0 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.923+0000 7f2ff7a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff013dc80 con 0x7f2ff0143ae0 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.924 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57966/0 (socket says 172.21.15.125:57966) 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 -- 172.21.15.125:0/2894464798 learned_addr learned my addr 172.21.15.125:0/2894464798 (peer_addr_for_me v2:172.21.15.125:0/0) 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 -- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff0143ae0 msgr2=0x7f2ff0138f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff0143ae0 0x7f2ff0138f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 -- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00cf7d0 msgr2=0x7f2ff013e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5000640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff013e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.925 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff013e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 -- 172.21.15.125:0/2894464798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009400 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2ff5a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cbc0 tx=0x7f2fe800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.924+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800ad40 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.925+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2fe8012400 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.925+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800aea0 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.925+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff013df60 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.925+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff0139470 con 0x7f2ff015e1e0 2024-01-16T15:24:35.926 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.926+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff013e480 con 0x7f2ff015e1e0 2024-01-16T15:24:35.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.927+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 51) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2fe8007070 con 0x7f2ff015e1e0 2024-01-16T15:24:35.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.928+0000 7f2fed600640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 0x7f2fc806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:35.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.928+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2fe8088b60 con 0x7f2ff015e1e0 2024-01-16T15:24:35.927 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.928+0000 7f2ff5000640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 0x7f2fc806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:35.928 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.929+0000 7f2ff5000640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 0x7f2fc806e980 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009a10 tx=0x7f2fe0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:35.930 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:35.932+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ff013e480 con 0x7f2ff015e1e0 2024-01-16T15:24:36.160 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.162+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff013e480 con 0x7f2ff015e1e0 2024-01-16T15:24:36.162 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.164+0000 7f2fed600640 1 -- 172.21.15.125:0/2894464798 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f2ff013e480 con 0x7f2ff015e1e0 2024-01-16T15:24:36.162 INFO:teuthology.orchestra.run.smithi125.stdout: 2024-01-16T15:24:36.162 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6833/1181037452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":1181037452},{"type":"v1","addr":"172.21.15.137:6833","nonce":1181037452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/1991371319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1991371319},{"type":"v1","addr":"172.21.15.137:6835","nonce":1991371319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":9156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/13212755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":13212755},{"type":"v1","addr":"172.21.15.125:6835","nonce":13212755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:06.982766+0000","modified":"2024-01-16T15:15:10.072697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9162":{"gid":9162,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/2099236429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":2099236429},{"type":"v1","addr":"172.21.15.125:6837","nonce":2099236429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:24:36.162 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 74 2024-01-16T15:24:36.165 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 msgr2=0x7f2fc806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 0x7f2fc806e980 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009a10 tx=0x7f2fe0008040 comp rx=0 tx=0).stop 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 msgr2=0x7f2ff013cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cbc0 tx=0x7f2fe800a430 comp rx=0 tx=0).stop 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 shutdown_connections 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:6832/2670128691,v1:172.21.15.125:6833/2670128691] conn(0x7f2fc806c4c0 0x7f2fc806e980 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff015e1e0 0x7f2ff013cd50 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff0143ae0 0x7f2ff0138f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:36.166 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 --2- 172.21.15.125:0/2894464798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00cf7d0 0x7f2ff013e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:36.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 >> 172.21.15.125:0/2894464798 conn(0x7f2ff006a800 msgr2=0x7f2ff015ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:36.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 shutdown_connections 2024-01-16T15:24:36.167 INFO:teuthology.orchestra.run.smithi125.stderr:2024-01-16T15:24:36.167+0000 7f2ff7a00640 1 -- 172.21.15.125:0/2894464798 wait complete. 2024-01-16T15:24:36.184 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-16T15:24:36.184 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T15:24:36.184 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:24:36.184 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:36.202 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:24:36.203 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:24:36.203 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:36.281 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:24:36.281 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:24:36.281 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:36.305 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:24:36.305 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:24:36.306 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:36.375 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:24:36.375 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T15:24:36.376 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T15:24:36.376 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:42.478 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T15:24:42.478 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T15:24:42.479 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:43.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-16T15:24:45.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:45.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:48.581 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T15:24:48.581 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T15:24:48.582 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:48.661 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T15:24:48.661 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-16T15:24:48.661 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:49.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-16T15:24:49.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-16T15:24:54.764 INFO:tasks.ceph.osd.3:Stopped 2024-01-16T15:24:54.765 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-16T15:24:54.765 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:54.841 INFO:tasks.ceph.osd.4:Stopped 2024-01-16T15:24:54.842 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-16T15:24:54.842 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:54.920 INFO:tasks.ceph.osd.5:Stopped 2024-01-16T15:24:54.921 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-16T15:24:54.921 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:55.014 INFO:tasks.ceph.osd.6:Stopped 2024-01-16T15:24:55.014 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-16T15:24:55.014 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:24:55.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2024-01-16T15:25:01.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-16T15:25:01.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-01-16T15:25:01.117 INFO:tasks.ceph.osd.7:Stopped 2024-01-16T15:25:01.117 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T15:25:01.117 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-16T15:25:01.117 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:06.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-16T15:25:06.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-16T15:25:07.219 INFO:tasks.ceph.mgr.y:Stopped 2024-01-16T15:25:07.219 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T15:25:07.220 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.313 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T15:25:07.313 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T15:25:07.314 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T15:25:07.314 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.354 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T15:25:07.354 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-16T15:25:07.354 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.382 INFO:tasks.ceph.mon.b:Stopped 2024-01-16T15:25:07.383 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-16T15:25:07.383 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.452 INFO:tasks.ceph.mon.c:Stopped 2024-01-16T15:25:07.453 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T15:25:07.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-16T15:25:07.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:25:07.561 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T15:25:07.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:25:07.758 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T15:25:07.938 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:25:07.939 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T15:25:08.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:25:08.121 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-16T15:25:08.286 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:25:08.286 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-16T15:25:08.505 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:25:08.505 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-16T15:25:08.673 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:25:08.674 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-16T15:25:08.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:25:08.861 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-16T15:25:09.053 INFO:tasks.ceph:Archiving mon data... 2024-01-16T15:25:09.055 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/data/mon.a.tgz 2024-01-16T15:25:09.055 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2024-01-16T15:25:09.113 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.xEKNUWIzMO 2024-01-16T15:25:09.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.xEKNUWIzMO 2024-01-16T15:25:09.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.xEKNUWIzMO 2024-01-16T15:25:09.582 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.xEKNUWIzMO is 1MB 2024-01-16T15:25:09.710 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.xEKNUWIzMO 2024-01-16T15:25:09.766 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/data/mon.b.tgz 2024-01-16T15:25:09.766 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-16T15:25:09.823 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.DoIg3wNQ1h 2024-01-16T15:25:09.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DoIg3wNQ1h 2024-01-16T15:25:10.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.DoIg3wNQ1h 2024-01-16T15:25:10.304 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.DoIg3wNQ1h is 1MB 2024-01-16T15:25:10.423 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.DoIg3wNQ1h 2024-01-16T15:25:10.480 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/data/mon.c.tgz 2024-01-16T15:25:10.480 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-16T15:25:10.533 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.AEv0YbyMPX 2024-01-16T15:25:10.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.AEv0YbyMPX 2024-01-16T15:25:10.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.AEv0YbyMPX 2024-01-16T15:25:10.984 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.AEv0YbyMPX is 1MB 2024-01-16T15:25:11.103 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.AEv0YbyMPX 2024-01-16T15:25:11.159 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T15:25:11.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:25:11.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:25:11.317 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:25:11.375 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:25:11.375 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T15:25:11.377 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi125/crash 2024-01-16T15:25:11.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:25:11.469 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi137/crash 2024-01-16T15:25:11.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:25:11.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-01-16T15:25:11.566 INFO:tasks.ceph:Shutting down logrotate 2024-01-16T15:25:11.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:11.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:11.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:11.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:11.907 INFO:tasks.ceph:Compressing logs... 2024-01-16T15:25:11.907 DEBUG:teuthology.orchestra.run.smithi125:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:25:11.915 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:25:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73175.log 2024-01-16T15:25:12.000 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68964.log 2024-01-16T15:25:12.001 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51895.log 2024-01-16T15:25:12.001 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73175.log.gz 2024-01-16T15:25:12.002 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54444.log 2024-01-16T15:25:12.002 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68964.log.gz 2024-01-16T15:25:12.003 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51300.log 2024-01-16T15:25:12.003 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51895.log.gz 2024-01-16T15:25:12.004 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54444.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54444.log.gz 2024-01-16T15:25:12.004 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78392.log 2024-01-16T15:25:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74343.log 2024-01-16T15:25:12.006 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51300.log.gz 2024-01-16T15:25:12.006 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49279.log 2024-01-16T15:25:12.007 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78392.log.gz 2024-01-16T15:25:12.007 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65822.log 2024-01-16T15:25:12.008 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74343.log.gz 2024-01-16T15:25:12.008 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-16T15:25:12.009 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43278.log 2024-01-16T15:25:12.009 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49279.log.gz 2024-01-16T15:25:12.011 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43620.log 2024-01-16T15:25:12.011 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36307.log 2024-01-16T15:25:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2024-01-16T15:25:12.012 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.65822.log.gz 2024-01-16T15:25:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69249.log 2024-01-16T15:25:12.012 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43278.log.gz 2024-01-16T15:25:12.012 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74691.log 2024-01-16T15:25:12.014 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2024-01-16T15:25:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64796.log 2024-01-16T15:25:12.014 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69249.log.gz 2024-01-16T15:25:12.015 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47239.log 2024-01-16T15:25:12.015 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74691.log.gz 2024-01-16T15:25:12.015 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.43620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42954.log 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36307.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36307.log.gz 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.1.43620.log.gz 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46572.log 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-16T15:25:12.016 INFO:teuthology.orchestra.run.smithi137.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.1.42954.log.gz 2024-01-16T15:25:12.017 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.46572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36107.log 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68393.log 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64796.log.gz 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65486.log 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47239.log.gz 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68141.log 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68393.log.gz 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65486.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T15:25:12.019 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65486.log.gz 2024-01-16T15:25:12.022 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42734.log 2024-01-16T15:25:12.022 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68141.log.gz 2024-01-16T15:25:12.022 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75672.log 2024-01-16T15:25:12.022 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53351.log 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42734.log.gz 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36267.log 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36107.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36107.log.gz 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.1.46572.log.gz 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47330.log 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:25:12.023 INFO:teuthology.orchestra.run.smithi137.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36267.log.gz 2024-01-16T15:25:12.025 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36147.log 2024-01-16T15:25:12.025 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.47330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44351.log 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50403.log 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75672.log.gz 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59526.log 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53351.log.gz 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54086.log 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50403.log.gz 2024-01-16T15:25:12.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67343.log 2024-01-16T15:25:12.028 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59526.log.gz 2024-01-16T15:25:12.030 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41693.log 2024-01-16T15:25:12.030 INFO:teuthology.orchestra.run.smithi137.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36147.log.gz 2024-01-16T15:25:12.030 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-16T15:25:12.030 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.41693.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-16T15:25:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54086.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54086.log.gz 2024-01-16T15:25:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64712.log 2024-01-16T15:25:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67343.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67343.log.gz 2024-01-16T15:25:12.031 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81796.log 2024-01-16T15:25:12.032 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64712.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64712.log.gz 2024-01-16T15:25:12.032 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60671.log 2024-01-16T15:25:12.032 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83170.log 2024-01-16T15:25:12.032 INFO:teuthology.orchestra.run.smithi125.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81796.log.gz 2024-01-16T15:25:12.033 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83506.log 2024-01-16T15:25:12.034 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60671.log.gz 2024-01-16T15:25:12.034 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60629.log 2024-01-16T15:25:12.034 INFO:teuthology.orchestra.run.smithi125.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.83170.log.gz 2024-01-16T15:25:12.034 INFO:teuthology.orchestra.run.smithi137.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.1.41693.log.gz 2024-01-16T15:25:12.035 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.7.log: 95.3% -- replaced with /var/log/ceph/ceph-client.1.44351.log.gz 2024-01-16T15:25:12.035 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42213.log 2024-01-16T15:25:12.035 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.5.log: 96.7% -- replaced with /var/log/ceph/ceph-client.1.47330.log.gz 2024-01-16T15:25:12.035 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45115.log 2024-01-16T15:25:12.035 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.3.42213.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T15:25:12.037 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.45115.log: 91.6% -- replaced with /var/log/ceph/ceph-client.3.42213.log.gz 2024-01-16T15:25:12.037 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45904.log 2024-01-16T15:25:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73567.log 2024-01-16T15:25:12.038 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83506.log.gz 2024-01-16T15:25:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73693.log 2024-01-16T15:25:12.038 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60629.log.gz 2024-01-16T15:25:12.038 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58683.log 2024-01-16T15:25:12.039 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73567.log.gz 2024-01-16T15:25:12.039 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65192.log 2024-01-16T15:25:12.039 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73693.log.gz 2024-01-16T15:25:12.041 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83380.log 2024-01-16T15:25:12.042 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58683.log.gz 2024-01-16T15:25:12.042 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81040.log 2024-01-16T15:25:12.042 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65192.log.gz 2024-01-16T15:25:12.042 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47197.log 2024-01-16T15:25:12.042 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83380.log.gz 2024-01-16T15:25:12.043 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.1.45904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36227.log 2024-01-16T15:25:12.043 INFO:teuthology.orchestra.run.smithi137.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.1.45115.log.gz 2024-01-16T15:25:12.043 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36227.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:25:12.044 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36187.log 2024-01-16T15:25:12.044 INFO:teuthology.orchestra.run.smithi137.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.1.45904.log.gz 2024-01-16T15:25:12.045 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59413.log 2024-01-16T15:25:12.045 INFO:teuthology.orchestra.run.smithi125.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.81040.log.gz 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47197.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47197.log.gz -5 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59221.log 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42401.log 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59413.log.gz 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56520.log 2024-01-16T15:25:12.046 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59221.log.gz 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77597.log 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42401.log.gz 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56520.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56520.log.gz 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61620.log 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66389.log 2024-01-16T15:25:12.049 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77597.log.gz 2024-01-16T15:25:12.051 INFO:teuthology.orchestra.run.smithi137.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-16T15:25:12.051 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.36227.log.gz 2024-01-16T15:25:12.051 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36187.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36187.log.gz 2024-01-16T15:25:12.051 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36027.log 2024-01-16T15:25:12.051 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.36027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36067.log 2024-01-16T15:25:12.053 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2024-01-16T15:25:12.053 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61620.log.gz 2024-01-16T15:25:12.053 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82077.log 2024-01-16T15:25:12.053 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66389.log.gz 2024-01-16T15:25:12.054 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64922.log 2024-01-16T15:25:12.054 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2024-01-16T15:25:12.054 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78867.log 2024-01-16T15:25:12.054 INFO:teuthology.orchestra.run.smithi125.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.82077.log.gz 2024-01-16T15:25:12.055 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36067.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36067.log.gz 2024-01-16T15:25:12.055 INFO:teuthology.orchestra.run.smithi137.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:25:12.056 INFO:teuthology.orchestra.run.smithi137.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36027.log.gz 2024-01-16T15:25:12.056 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-16T15:25:12.057 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79017.log 2024-01-16T15:25:12.057 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64922.log.gz 2024-01-16T15:25:12.057 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76913.log 2024-01-16T15:25:12.057 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78867.log.gz 2024-01-16T15:25:12.058 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66004.log 2024-01-16T15:25:12.058 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79017.log.gz 2024-01-16T15:25:12.058 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57609.log 2024-01-16T15:25:12.058 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76913.log.gz 2024-01-16T15:25:12.059 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-16T15:25:12.060 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55286.log 88.1% 2024-01-16T15:25:12.060 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66004.log.gz 2024-01-16T15:25:12.060 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79539.log 2024-01-16T15:25:12.060 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57609.log.gz 2024-01-16T15:25:12.060 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72792.log 2024-01-16T15:25:12.062 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mds.d.log: 91.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-16T15:25:12.062 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T15:25:12.062 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55286.log.gz 2024-01-16T15:25:12.063 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62908.log 2024-01-16T15:25:12.063 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79539.log.gz 2024-01-16T15:25:12.063 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52198.log 2024-01-16T15:25:12.063 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72792.log.gz 2024-01-16T15:25:12.064 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42410.log 2024-01-16T15:25:12.064 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62908.log.gz 2024-01-16T15:25:12.065 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61133.log 2024-01-16T15:25:12.065 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52198.log.gz 2024-01-16T15:25:12.065 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43446.log 2024-01-16T15:25:12.066 INFO:teuthology.orchestra.run.smithi125.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42410.log.gz 2024-01-16T15:25:12.066 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63768.log 2024-01-16T15:25:12.067 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61133.log.gz 2024-01-16T15:25:12.067 INFO:teuthology.orchestra.run.smithi137.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-16T15:25:12.067 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43446.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.57567.log 2024-01-16T15:25:12.068 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43446.log.gz 2024-01-16T15:25:12.069 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63768.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63768.log.gz 2024-01-16T15:25:12.069 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47669.log 2024-01-16T15:25:12.069 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53863.log 2024-01-16T15:25:12.070 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57567.log.gz 2024-01-16T15:25:12.071 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-01-16T15:25:12.071 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47669.log.gz 2024-01-16T15:25:12.071 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.53863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43062.log 2024-01-16T15:25:12.073 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41976.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz -5 2024-01-16T15:25:12.073 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58867.log 2024-01-16T15:25:12.074 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72876.log 2024-01-16T15:25:12.074 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43062.log.gz 2024-01-16T15:25:12.075 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63461.log 2024-01-16T15:25:12.075 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58867.log.gz 2024-01-16T15:25:12.076 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80735.log 2024-01-16T15:25:12.076 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72876.log.gz 2024-01-16T15:25:12.077 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.63461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67469.log 2024-01-16T15:25:12.078 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80735.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.80735.log.gz 2024-01-16T15:25:12.078 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72044.log 2024-01-16T15:25:12.079 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2024-01-16T15:25:12.080 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67469.log.gz 2024-01-16T15:25:12.081 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71607.log 2024-01-16T15:25:12.081 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72044.log.gz 2024-01-16T15:25:12.082 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48089.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2024-01-16T15:25:12.082 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47081.log 2024-01-16T15:25:12.084 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48173.log 2024-01-16T15:25:12.084 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71607.log.gz 2024-01-16T15:25:12.085 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81993.log 2024-01-16T15:25:12.085 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47081.log.gz 2024-01-16T15:25:12.086 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48173.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48173.log.gz 2024-01-16T15:25:12.087 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59095.log 2024-01-16T15:25:12.087 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45010.log 2024-01-16T15:25:12.088 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81993.log.gz 2024-01-16T15:25:12.089 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80915.log 2024-01-16T15:25:12.089 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59095.log.gz 2024-01-16T15:25:12.090 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.2.45010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60755.log 2024-01-16T15:25:12.091 INFO:teuthology.orchestra.run.smithi125.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.2.45010.log.gz 2024-01-16T15:25:12.092 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53141.log 2024-01-16T15:25:12.092 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.80915.log.gz 2024-01-16T15:25:12.093 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60755.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.60755.log.gz -- 2024-01-16T15:25:12.094 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.53099.log 2024-01-16T15:25:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53141.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53141.log.gz 2024-01-16T15:25:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68645.log 2024-01-16T15:25:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83212.log 2024-01-16T15:25:12.096 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53099.log.gz 2024-01-16T15:25:12.097 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53267.log 2024-01-16T15:25:12.098 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68645.log.gz 2024-01-16T15:25:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83212.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83212.log.gz 2024-01-16T15:25:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79245.log 2024-01-16T15:25:12.100 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72750.log 2024-01-16T15:25:12.100 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53267.log.gz 2024-01-16T15:25:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81837.log 2024-01-16T15:25:12.102 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79245.log.gz 2024-01-16T15:25:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63642.log 2024-01-16T15:25:12.103 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72750.log.gz 2024-01-16T15:25:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52707.log 2024-01-16T15:25:12.104 INFO:teuthology.orchestra.run.smithi125.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.81837.log.gz 2024-01-16T15:25:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63822.log 2024-01-16T15:25:12.106 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63642.log.gz 2024-01-16T15:25:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52707.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52707.log.gz 2024-01-16T15:25:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55202.log 2024-01-16T15:25:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63822.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63822.log.gz 2024-01-16T15:25:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83758.log 2024-01-16T15:25:12.110 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46905.log 2024-01-16T15:25:12.110 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55202.log.gz 2024-01-16T15:25:12.111 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83758.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83758.log.gz 2024-01-16T15:25:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83002.log 2024-01-16T15:25:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76738.log 2024-01-16T15:25:12.113 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46905.log.gz 2024-01-16T15:25:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83002.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83002.log.gz 2024-01-16T15:25:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72405.log 2024-01-16T15:25:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76738.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76738.log.gz 2024-01-16T15:25:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49795.log 2024-01-16T15:25:12.116 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60122.log 2024-01-16T15:25:12.117 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72405.log.gz 2024-01-16T15:25:12.118 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64142.log 2024-01-16T15:25:12.118 INFO:teuthology.orchestra.run.smithi125.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.49795.log.gz 2024-01-16T15:25:12.119 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65276.log 2024-01-16T15:25:12.119 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60122.log.gz 2024-01-16T15:25:12.120 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51384.log 2024-01-16T15:25:12.120 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64142.log.gz 2024-01-16T15:25:12.122 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65276.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65276.log.gz 2024-01-16T15:25:12.122 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51733.log 2024-01-16T15:25:12.123 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51384.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51384.log.gz 2024-01-16T15:25:12.123 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47281.log 2024-01-16T15:25:12.124 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48509.log 2024-01-16T15:25:12.125 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51733.log.gz 2024-01-16T15:25:12.126 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42746.log 2024-01-16T15:25:12.126 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47281.log.gz 2024-01-16T15:25:12.127 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58909.log 2024-01-16T15:25:12.128 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48509.log.gz 2024-01-16T15:25:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59053.log 2024-01-16T15:25:12.129 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42746.log.gz 2024-01-16T15:25:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70378.log 2024-01-16T15:25:12.130 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58909.log.gz 2024-01-16T15:25:12.131 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59053.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.59053.log.gz -- 2024-01-16T15:25:12.131 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.73133.log 2024-01-16T15:25:12.133 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70378.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70378.log.gz 2024-01-16T15:25:12.133 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71978.log 2024-01-16T15:25:12.134 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56073.log 2024-01-16T15:25:12.134 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73133.log.gz 2024-01-16T15:25:12.135 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58759.log 2024-01-16T15:25:12.135 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71978.log.gz 2024-01-16T15:25:12.136 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56073.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56073.log.gz 2024-01-16T15:25:12.137 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68603.log 2024-01-16T15:25:12.137 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2024-01-16T15:25:12.138 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58759.log.gz 2024-01-16T15:25:12.139 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60587.log 2024-01-16T15:25:12.139 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68603.log.gz 2024-01-16T15:25:12.141 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48341.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2024-01-16T15:25:12.141 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2024-01-16T15:25:12.141 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69948.log 2024-01-16T15:25:12.142 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60587.log.gz 2024-01-16T15:25:12.143 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41766.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2024-01-16T15:25:12.143 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79059.log 2024-01-16T15:25:12.144 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69948.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69948.log.gz 2024-01-16T15:25:12.144 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78307.log 2024-01-16T15:25:12.145 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52461.log 2024-01-16T15:25:12.146 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79059.log.gz 2024-01-16T15:25:12.147 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67973.log 2024-01-16T15:25:12.147 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78307.log.gz 2024-01-16T15:25:12.148 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2024-01-16T15:25:12.149 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52461.log.gz 2024-01-16T15:25:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67973.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.67973.log.gz 2024-01-16T15:25:12.150 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69006.log 2024-01-16T15:25:12.151 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56352.log 2024-01-16T15:25:12.151 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2024-01-16T15:25:12.152 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69006.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69006.log.gz 2024-01-16T15:25:12.153 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75588.log 2024-01-16T15:25:12.154 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56352.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56352.log.gz 2024-01-16T15:25:12.154 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57759.log 2024-01-16T15:25:12.155 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68844.log 2024-01-16T15:25:12.155 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75588.log.gz 2024-01-16T15:25:12.156 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57759.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57759.log.gz 2024-01-16T15:25:12.156 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59947.log 2024-01-16T15:25:12.157 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2024-01-16T15:25:12.158 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68844.log.gz 2024-01-16T15:25:12.159 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72522.log 2024-01-16T15:25:12.159 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59947.log.gz 2024-01-16T15:25:12.160 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72447.log 2024-01-16T15:25:12.160 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2024-01-16T15:25:12.161 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62031.log 2024-01-16T15:25:12.162 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72522.log.gz 2024-01-16T15:25:12.163 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72447.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72447.log.gz 2024-01-16T15:25:12.163 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67805.log 2024-01-16T15:25:12.164 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62031.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62031.log.gz 2024-01-16T15:25:12.164 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83086.log 2024-01-16T15:25:12.165 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60545.log 2024-01-16T15:25:12.166 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67805.log.gz 2024-01-16T15:25:12.167 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64208.log 2024-01-16T15:25:12.167 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83086.log.gz 2024-01-16T15:25:12.168 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60545.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.60545.log.gz 2024-01-16T15:25:12.168 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64838.log 2024-01-16T15:25:12.169 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79837.log 2024-01-16T15:25:12.170 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64208.log.gz 2024-01-16T15:25:12.171 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73651.log 2024-01-16T15:25:12.171 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64838.log.gz 2024-01-16T15:25:12.172 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79837.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.79837.log.gz 2024-01-16T15:25:12.172 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60080.log 2024-01-16T15:25:12.173 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54712.log 2024-01-16T15:25:12.173 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73651.log.gz 2024-01-16T15:25:12.174 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59371.log 2024-01-16T15:25:12.175 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60080.log.gz 2024-01-16T15:25:12.176 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80102.log 2024-01-16T15:25:12.176 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54712.log.gz 2024-01-16T15:25:12.177 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80571.log 2024-01-16T15:25:12.177 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59371.log.gz 2024-01-16T15:25:12.178 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69860.log 2024-01-16T15:25:12.178 INFO:teuthology.orchestra.run.smithi125.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.80102.log.gz 2024-01-16T15:25:12.180 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80571.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.80571.log.gz 2024-01-16T15:25:12.180 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42406.log 2024-01-16T15:25:12.181 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69860.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69860.log.gz 2024-01-16T15:25:12.181 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47039.log 2024-01-16T15:25:12.182 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75095.log 2024-01-16T15:25:12.183 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42406.log.gz 2024-01-16T15:25:12.184 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65780.log 2024-01-16T15:25:12.184 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47039.log.gz 2024-01-16T15:25:12.185 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75095.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75095.log.gz 2024-01-16T15:25:12.185 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65402.log 2024-01-16T15:25:12.186 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61458.log 2024-01-16T15:25:12.186 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65780.log.gz 2024-01-16T15:25:12.188 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53477.log 2024-01-16T15:25:12.188 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65402.log.gz 2024-01-16T15:25:12.189 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65528.log 2024-01-16T15:25:12.189 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61458.log.gz 2024-01-16T15:25:12.190 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45950.log 2024-01-16T15:25:12.190 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53477.log.gz 2024-01-16T15:25:12.191 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64436.log 2024-01-16T15:25:12.192 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65528.log.gz 2024-01-16T15:25:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45950.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45950.log.gz 2024-01-16T15:25:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80061.log 2024-01-16T15:25:12.194 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-01-16T15:25:12.194 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64436.log.gz 2024-01-16T15:25:12.195 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58036.log 2024-01-16T15:25:12.196 INFO:teuthology.orchestra.run.smithi125.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.80061.log.gz 2024-01-16T15:25:12.197 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50501.log 2024-01-16T15:25:12.197 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2024-01-16T15:25:12.198 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.58036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58969.log 2024-01-16T15:25:12.200 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81081.log 2024-01-16T15:25:12.200 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50501.log.gz 2024-01-16T15:25:12.201 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81327.log 2024-01-16T15:25:12.201 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58969.log.gz 2024-01-16T15:25:12.202 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42409.log 2024-01-16T15:25:12.203 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81081.log.gz 2024-01-16T15:25:12.204 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-01-16T15:25:12.204 INFO:teuthology.orchestra.run.smithi125.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81327.log.gz 2024-01-16T15:25:12.206 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70486.log 2024-01-16T15:25:12.206 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-01-16T15:25:12.207 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68802.log 2024-01-16T15:25:12.207 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-01-16T15:25:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71810.log 2024-01-16T15:25:12.208 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70486.log.gz 2024-01-16T15:25:12.209 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60443.log 2024-01-16T15:25:12.210 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68802.log.gz 2024-01-16T15:25:12.211 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63684.log 2024-01-16T15:25:12.211 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71810.log.gz 2024-01-16T15:25:12.212 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50361.log 2024-01-16T15:25:12.212 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60443.log.gz 2024-01-16T15:25:12.213 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71936.log 2024-01-16T15:25:12.214 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63684.log.gz 2024-01-16T15:25:12.215 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62498.log 2024-01-16T15:25:12.215 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50361.log.gz 2024-01-16T15:25:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57203.log 2024-01-16T15:25:12.216 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71936.log.gz 2024-01-16T15:25:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58599.log 2024-01-16T15:25:12.218 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62498.log.gz 2024-01-16T15:25:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67385.log 2024-01-16T15:25:12.219 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57203.log.gz 2024-01-16T15:25:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76612.log 2024-01-16T15:25:12.220 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58599.log.gz 2024-01-16T15:25:12.222 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77699.log 2024-01-16T15:25:12.222 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67385.log.gz 2024-01-16T15:25:12.223 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54604.log 2024-01-16T15:25:12.223 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76612.log.gz 2024-01-16T15:25:12.224 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43003.log 2024-01-16T15:25:12.224 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.77699.log.gz 2024-01-16T15:25:12.225 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45446.log 2024-01-16T15:25:12.226 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54604.log.gz 2024-01-16T15:25:12.227 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64292.log 2024-01-16T15:25:12.227 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43003.log.gz 2024-01-16T15:25:12.228 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66058.log 2024-01-16T15:25:12.228 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45446.log.gz 2024-01-16T15:25:12.230 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52114.log 2024-01-16T15:25:12.230 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64292.log.gz 2024-01-16T15:25:12.231 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73609.log 2024-01-16T15:25:12.231 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66058.log.gz 2024-01-16T15:25:12.232 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76528.log 2024-01-16T15:25:12.232 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52114.log.gz 2024-01-16T15:25:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79203.log 2024-01-16T15:25:12.234 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73609.log.gz 2024-01-16T15:25:12.235 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73481.log 2024-01-16T15:25:12.235 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76528.log.gz 2024-01-16T15:25:12.236 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52622.log 2024-01-16T15:25:12.236 INFO:teuthology.orchestra.run.smithi125.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79203.log.gz 2024-01-16T15:25:12.238 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-01-16T15:25:12.238 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73481.log.gz 2024-01-16T15:25:12.239 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59137.log 2024-01-16T15:25:12.239 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52622.log.gz 2024-01-16T15:25:12.240 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55370.log 2024-01-16T15:25:12.240 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-01-16T15:25:12.242 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77207.log 2024-01-16T15:25:12.242 INFO:teuthology.orchestra.run.smithi125.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59137.log.gz 2024-01-16T15:25:12.243 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-01-16T15:25:12.243 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55370.log.gz 2024-01-16T15:25:12.244 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80348.log 2024-01-16T15:25:12.244 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77207.log.gz 2024-01-16T15:25:12.246 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54044.log 2024-01-16T15:25:12.246 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-01-16T15:25:12.247 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81591.log 2024-01-16T15:25:12.247 INFO:teuthology.orchestra.run.smithi125.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.80348.log.gz 2024-01-16T15:25:12.248 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64562.log 2024-01-16T15:25:12.248 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54044.log.gz 2024-01-16T15:25:12.250 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47365.log 2024-01-16T15:25:12.250 INFO:teuthology.orchestra.run.smithi125.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.81591.log.gz 2024-01-16T15:25:12.251 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47507.log 2024-01-16T15:25:12.251 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64562.log.gz 2024-01-16T15:25:12.252 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67847.log 2024-01-16T15:25:12.252 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47365.log.gz 2024-01-16T15:25:12.254 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62540.log 2024-01-16T15:25:12.254 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47507.log.gz 2024-01-16T15:25:12.255 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62806.log 2024-01-16T15:25:12.255 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67847.log.gz 2024-01-16T15:25:12.256 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2024-01-16T15:25:12.257 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62540.log.gz 2024-01-16T15:25:12.258 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80266.log 2024-01-16T15:25:12.258 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62806.log.gz 2024-01-16T15:25:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81878.log 2024-01-16T15:25:12.259 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2024-01-16T15:25:12.260 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75840.log 2024-01-16T15:25:12.261 INFO:teuthology.orchestra.run.smithi125.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80266.log.gz 2024-01-16T15:25:12.261 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49457.log 2024-01-16T15:25:12.262 INFO:teuthology.orchestra.run.smithi125.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81878.log.gz 2024-01-16T15:25:12.263 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76402.log 2024-01-16T15:25:12.263 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75840.log.gz 2024-01-16T15:25:12.265 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75630.log 2024-01-16T15:25:12.265 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49457.log.gz 2024-01-16T15:25:12.266 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76327.log 2024-01-16T15:25:12.266 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76402.log.gz 2024-01-16T15:25:12.267 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66143.log 2024-01-16T15:25:12.267 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75630.log.gz 2024-01-16T15:25:12.268 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70168.log 2024-01-16T15:25:12.269 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76327.log.gz 2024-01-16T15:25:12.269 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81204.log 2024-01-16T15:25:12.270 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66143.log.gz 2024-01-16T15:25:12.271 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55556.log 2024-01-16T15:25:12.271 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70168.log.gz 2024-01-16T15:25:12.272 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68435.log 2024-01-16T15:25:12.272 INFO:teuthology.orchestra.run.smithi125.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81204.log.gz 2024-01-16T15:25:12.274 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43320.log 2024-01-16T15:25:12.274 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55556.log.gz 2024-01-16T15:25:12.275 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43104.log 2024-01-16T15:25:12.275 INFO:teuthology.orchestra.run.smithi125.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68435.log.gz 2024-01-16T15:25:12.276 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2024-01-16T15:25:12.276 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43320.log.gz 2024-01-16T15:25:12.277 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70294.log 2024-01-16T15:25:12.278 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43104.log.gz 2024-01-16T15:25:12.279 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65006.log 2024-01-16T15:25:12.279 INFO:teuthology.orchestra.run.smithi125.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2024-01-16T15:25:12.280 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52664.log 2024-01-16T15:25:12.280 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70294.log.gz 2024-01-16T15:25:12.282 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63726.log 2024-01-16T15:25:12.282 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65006.log.gz 2024-01-16T15:25:12.283 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2024-01-16T15:25:12.283 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52664.log.gz 2024-01-16T15:25:12.284 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73777.log 2024-01-16T15:25:12.284 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63726.log.gz 2024-01-16T15:25:12.286 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75732.log 2024-01-16T15:25:12.286 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2024-01-16T15:25:12.287 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58217.log 2024-01-16T15:25:12.287 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73777.log.gz 2024-01-16T15:25:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75179.log 2024-01-16T15:25:12.288 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75732.log.gz 2024-01-16T15:25:12.290 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67763.log 2024-01-16T15:25:12.290 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58217.log.gz 2024-01-16T15:25:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66287.log 2024-01-16T15:25:12.291 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75179.log.gz 2024-01-16T15:25:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55664.log 2024-01-16T15:25:12.292 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67763.log.gz 2024-01-16T15:25:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48941.log 2024-01-16T15:25:12.294 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66287.log.gz 2024-01-16T15:25:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46523.log 2024-01-16T15:25:12.295 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55664.log.gz 2024-01-16T15:25:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76081.log 2024-01-16T15:25:12.296 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48941.log.gz 2024-01-16T15:25:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71541.log 2024-01-16T15:25:12.297 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46523.log.gz 2024-01-16T15:25:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46811.log 2024-01-16T15:25:12.299 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76081.log.gz 2024-01-16T15:25:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45488.log 2024-01-16T15:25:12.300 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71541.log.gz 2024-01-16T15:25:12.301 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70082.log 2024-01-16T15:25:12.301 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46811.log.gz 2024-01-16T15:25:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64066.log 2024-01-16T15:25:12.302 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45488.log.gz 2024-01-16T15:25:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72971.log 2024-01-16T15:25:12.304 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70082.log.gz 2024-01-16T15:25:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52072.log 2024-01-16T15:25:12.306 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64066.log.gz 2024-01-16T15:25:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2024-01-16T15:25:12.307 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72971.log.gz 2024-01-16T15:25:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66101.log 2024-01-16T15:25:12.308 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52072.log.gz 2024-01-16T15:25:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82625.log 2024-01-16T15:25:12.309 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2024-01-16T15:25:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68561.log 2024-01-16T15:25:12.311 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66101.log.gz 2024-01-16T15:25:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60359.log 2024-01-16T15:25:12.312 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82625.log.gz 2024-01-16T15:25:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78084.log 2024-01-16T15:25:12.314 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68561.log.gz 2024-01-16T15:25:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-01-16T15:25:12.314 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60359.log.gz 2024-01-16T15:25:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.78084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51006.log 2024-01-16T15:25:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62584.log 2024-01-16T15:25:12.317 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-01-16T15:25:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2024-01-16T15:25:12.318 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51006.log.gz 2024-01-16T15:25:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75137.log 2024-01-16T15:25:12.321 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62584.log.gz 2024-01-16T15:25:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51937.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2024-01-16T15:25:12.322 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-01-16T15:25:12.322 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75137.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75137.log.gz 2024-01-16T15:25:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74595.log 2024-01-16T15:25:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60038.log 2024-01-16T15:25:12.324 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-01-16T15:25:12.325 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74595.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74595.log.gz 2024-01-16T15:25:12.327 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.60038.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54853.log 2024-01-16T15:25:12.327 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60038.log.gz 2024-01-16T15:25:12.327 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65738.log 2024-01-16T15:25:12.329 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74511.log 2024-01-16T15:25:12.329 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54853.log.gz 2024-01-16T15:25:12.330 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65738.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65738.log.gz 2024-01-16T15:25:12.331 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2024-01-16T15:25:12.332 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74511.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74511.log.gz 2024-01-16T15:25:12.332 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80694.log 2024-01-16T15:25:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52541.log 2024-01-16T15:25:12.334 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2024-01-16T15:25:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80694.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.80694.log.gz 2024-01-16T15:25:12.335 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71112.log 2024-01-16T15:25:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52541.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52541.log.gz 2024-01-16T15:25:12.337 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44604.log 2024-01-16T15:25:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45908.log 2024-01-16T15:25:12.338 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71112.log.gz 2024-01-16T15:25:12.339 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43102.log 2024-01-16T15:25:12.340 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44604.log.gz 2024-01-16T15:25:12.341 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81673.log 2024-01-16T15:25:12.341 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45908.log.gz 2024-01-16T15:25:12.342 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46607.log 2024-01-16T15:25:12.342 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43102.log.gz 2024-01-16T15:25:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77513.log 2024-01-16T15:25:12.344 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81673.log.gz 2024-01-16T15:25:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46607.log: gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46607.log.gz 2024-01-16T15:25:12.345 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43614.log 2024-01-16T15:25:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77513.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77513.log.gz 2024-01-16T15:25:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66797.log 2024-01-16T15:25:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43614.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43614.log.gz 2024-01-16T15:25:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59722.log 2024-01-16T15:25:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68704.log 2024-01-16T15:25:12.349 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66797.log.gz 2024-01-16T15:25:12.351 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71423.log 2024-01-16T15:25:12.351 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59722.log.gz 2024-01-16T15:25:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68704.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:25:12.352 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68704.log.gz 2024-01-16T15:25:12.353 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71423.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71423.log.gz 2024-01-16T15:25:12.354 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79299.log 2024-01-16T15:25:12.355 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56783.log 2024-01-16T15:25:12.357 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79299.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79299.log.gz 2024-01-16T15:25:12.357 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69417.log 2024-01-16T15:25:12.359 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56783.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56783.log.gz 2024-01-16T15:25:12.360 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69417.log: gzip 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:25:12.360 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.73927.log 2024-01-16T15:25:12.360 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69417.log.gz 2024-01-16T15:25:12.361 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68183.log 2024-01-16T15:25:12.362 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73927.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73927.log.gz 2024-01-16T15:25:12.363 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51258.log 2024-01-16T15:25:12.364 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67511.log 2024-01-16T15:25:12.364 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68183.log.gz 2024-01-16T15:25:12.365 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51258.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51258.log.gz 2024-01-16T15:25:12.366 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64880.log 2024-01-16T15:25:12.366 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67511.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67511.log.gz 2024-01-16T15:25:12.367 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66839.log 2024-01-16T15:25:12.368 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60275.log 2024-01-16T15:25:12.368 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64880.log.gz 2024-01-16T15:25:12.369 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66839.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66839.log.gz 2024-01-16T15:25:12.370 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71894.log 2024-01-16T15:25:12.372 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52835.log 2024-01-16T15:25:12.372 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60275.log.gz 2024-01-16T15:25:12.373 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71894.log: gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71894.log.gz 2024-01-16T15:25:12.373 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.80448.log 2024-01-16T15:25:12.374 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52835.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52835.log.gz 2024-01-16T15:25:12.374 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52324.log 2024-01-16T15:25:12.376 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65863.log 2024-01-16T15:25:12.376 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.80448.log.gz 2024-01-16T15:25:12.377 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75445.log 2024-01-16T15:25:12.377 INFO:teuthology.orchestra.run.smithi125.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52324.log.gz 2024-01-16T15:25:12.379 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74869.log 2024-01-16T15:25:12.379 INFO:teuthology.orchestra.run.smithi125.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.65863.log.gz 2024-01-16T15:25:12.380 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57525.log 2024-01-16T15:25:12.380 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75445.log.gz 2024-01-16T15:25:12.381 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53309.log 2024-01-16T15:25:12.381 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74869.log.gz 2024-01-16T15:25:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68267.log 2024-01-16T15:25:12.383 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57525.log.gz 2024-01-16T15:25:12.384 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65360.log 2024-01-16T15:25:12.384 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53309.log.gz 2024-01-16T15:25:12.385 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83632.log 2024-01-16T15:25:12.385 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68267.log.gz 2024-01-16T15:25:12.387 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57651.log 2024-01-16T15:25:12.387 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65360.log.gz 2024-01-16T15:25:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64670.log 2024-01-16T15:25:12.388 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83632.log.gz 2024-01-16T15:25:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-01-16T15:25:12.389 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57651.log.gz 2024-01-16T15:25:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64616.log 2024-01-16T15:25:12.391 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64670.log.gz 2024-01-16T15:25:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60164.log 2024-01-16T15:25:12.392 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-01-16T15:25:12.393 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46034.log 2024-01-16T15:25:12.393 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64616.log.gz 2024-01-16T15:25:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61704.log 2024-01-16T15:25:12.395 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60164.log.gz 2024-01-16T15:25:12.396 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61091.log 2024-01-16T15:25:12.396 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46034.log.gz 2024-01-16T15:25:12.397 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61704.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61704.log.gz 2024-01-16T15:25:12.397 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45824.log 2024-01-16T15:25:12.399 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61091.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61091.log.gz 2024-01-16T15:25:12.399 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55118.log 2024-01-16T15:25:12.401 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45824.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45824.log.gz 2024-01-16T15:25:12.401 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.54242.log 2024-01-16T15:25:12.402 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55118.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80307.log 2024-01-16T15:25:12.402 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55118.log.gz 2024-01-16T15:25:12.403 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54242.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54242.log.gz 2024-01-16T15:25:12.404 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53225.log 2024-01-16T15:25:12.404 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80307.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80307.log.gz 2024-01-16T15:25:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48551.log 2024-01-16T15:25:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68886.log 2024-01-16T15:25:12.406 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53225.log.gz 2024-01-16T15:25:12.408 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48551.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48551.log.gz 2024-01-16T15:25:12.408 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-01-16T15:25:12.409 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68886.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68886.log.gz 2024-01-16T15:25:12.410 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67679.log 2024-01-16T15:25:12.411 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71852.log 2024-01-16T15:25:12.411 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-01-16T15:25:12.412 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67679.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67679.log.gz 2024-01-16T15:25:12.413 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5/var/log/ceph/ceph-client.admin.71852.log: --verbose -- /var/log/ceph/ceph-client.admin.46118.log 2024-01-16T15:25:12.413 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71852.log.gz 2024-01-16T15:25:12.415 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45404.log 2024-01-16T15:25:12.416 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67091.log 2024-01-16T15:25:12.416 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46118.log.gz 2024-01-16T15:25:12.417 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45404.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45404.log.gz 2024-01-16T15:25:12.418 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67091.log: /var/log/ceph/ceph-client.admin.69716.log 2024-01-16T15:25:12.418 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67091.log.gz 2024-01-16T15:25:12.419 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75391.log 2024-01-16T15:25:12.421 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61301.log 2024-01-16T15:25:12.421 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69716.log.gz 2024-01-16T15:25:12.422 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54796.log 2024-01-16T15:25:12.422 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75391.log.gz 2024-01-16T15:25:12.423 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52282.log 2024-01-16T15:25:12.424 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61301.log.gz 2024-01-16T15:25:12.425 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54796.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54796.log.gz 2024-01-16T15:25:12.426 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.52282.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60797.log 2024-01-16T15:25:12.426 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2024-01-16T15:25:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62290.log 2024-01-16T15:25:12.429 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2024-01-16T15:25:12.429 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60797.log.gz 2024-01-16T15:25:12.430 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69333.log 2024-01-16T15:25:12.430 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62290.log.gz 2024-01-16T15:25:12.431 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72103.log 2024-01-16T15:25:12.431 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2024-01-16T15:25:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66659.log 2024-01-16T15:25:12.433 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69333.log.gz 2024-01-16T15:25:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72103.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80184.log 2024-01-16T15:25:12.434 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72103.log.gz 2024-01-16T15:25:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84070.log 2024-01-16T15:25:12.436 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66659.log.gz 2024-01-16T15:25:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80184.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.80184.log.gz 2024-01-16T15:25:12.437 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.49501.log 2024-01-16T15:25:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.84070.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84070.log.gz 2024-01-16T15:25:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80958.log 2024-01-16T15:25:12.440 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68225.log 2024-01-16T15:25:12.440 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2024-01-16T15:25:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80958.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80958.log.gz 2024-01-16T15:25:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55874.log 2024-01-16T15:25:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68225.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68225.log.gz 2024-01-16T15:25:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54128.log 2024-01-16T15:25:12.444 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66245.log 2024-01-16T15:25:12.444 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55874.log.gz 2024-01-16T15:25:12.446 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54128.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54128.log.gz 2024-01-16T15:25:12.446 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64250.log 2024-01-16T15:25:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66245.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66245.log.gz 2024-01-16T15:25:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58259.log 2024-01-16T15:25:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83548.log 2024-01-16T15:25:12.449 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64250.log.gz 2024-01-16T15:25:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77036.log 2024-01-16T15:25:12.450 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58259.log.gz 2024-01-16T15:25:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83548.log: gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83548.log.gz 2024-01-16T15:25:12.452 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.62764.log 2024-01-16T15:25:12.452 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77036.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77036.log.gz 2024-01-16T15:25:12.453 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82709.log 2024-01-16T15:25:12.454 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61049.log 2024-01-16T15:25:12.454 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62764.log.gz 2024-01-16T15:25:12.455 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67427.log 2024-01-16T15:25:12.456 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82709.log.gz 2024-01-16T15:25:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42228.log 2024-01-16T15:25:12.457 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61049.log.gz 2024-01-16T15:25:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67427.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67427.log.gz 2024-01-16T15:25:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61821.log 2024-01-16T15:25:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42228.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2024-01-16T15:25:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82960.log 2024-01-16T15:25:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66701.log 2024-01-16T15:25:12.462 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61821.log.gz 2024-01-16T15:25:12.463 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82960.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82960.log.gz 2024-01-16T15:25:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66701.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66701.log.gz 2024-01-16T15:25:12.465 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43782.log 2024-01-16T15:25:12.466 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77555.log 2024-01-16T15:25:12.468 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71070.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43782.log.gz 2024-01-16T15:25:12.468 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:12.469 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77555.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77555.log.gz 2024-01-16T15:25:12.469 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-01-16T15:25:12.470 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80020.log 2024-01-16T15:25:12.471 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71070.log.gz 2024-01-16T15:25:12.472 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2024-01-16T15:25:12.472 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-01-16T15:25:12.473 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79583.log 2024-01-16T15:25:12.473 INFO:teuthology.orchestra.run.smithi125.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.80020.log.gz 2024-01-16T15:25:12.474 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76654.log 2024-01-16T15:25:12.475 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2024-01-16T15:25:12.476 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75487.log 2024-01-16T15:25:12.476 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79583.log.gz 2024-01-16T15:25:12.477 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55748.log 2024-01-16T15:25:12.477 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76654.log.gz 2024-01-16T15:25:12.478 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62073.log 2024-01-16T15:25:12.479 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75487.log.gz 2024-01-16T15:25:12.480 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65904.log 2024-01-16T15:25:12.480 INFO:teuthology.orchestra.run.smithi125.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55748.log.gz 2024-01-16T15:25:12.481 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76570.log 2024-01-16T15:25:12.481 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62073.log.gz 2024-01-16T15:25:12.483 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65904.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65904.log.gz 2024-01-16T15:25:12.483 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49025.log 2024-01-16T15:25:12.484 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76570.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76570.log.gz 2024-01-16T15:25:12.484 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2024-01-16T15:25:12.486 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83464.log 2024-01-16T15:25:12.486 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49025.log.gz 2024-01-16T15:25:12.487 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51132.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2024-01-16T15:25:12.488 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83464.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83464.log.gz 2024-01-16T15:25:12.489 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78265.log 2024-01-16T15:25:12.490 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-01-16T15:25:12.492 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2024-01-16T15:25:12.492 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78265.log.gz 2024-01-16T15:25:12.493 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42102.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-01-16T15:25:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42745.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2024-01-16T15:25:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61217.log 2024-01-16T15:25:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78349.log 2024-01-16T15:25:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63600.log 2024-01-16T15:25:12.498 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61217.log.gz 2024-01-16T15:25:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2024-01-16T15:25:12.499 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78349.log.gz 2024-01-16T15:25:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49585.log 2024-01-16T15:25:12.500 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63600.log.gz 2024-01-16T15:25:12.502 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48425.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2024-01-16T15:25:12.502 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.75774.log 2024-01-16T15:25:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49585.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49585.log.gz 2024-01-16T15:25:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64024.log 2024-01-16T15:25:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42404.log 2024-01-16T15:25:12.505 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75774.log.gz 2024-01-16T15:25:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82835.log 2024-01-16T15:25:12.506 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64024.log.gz 2024-01-16T15:25:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42404.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42404.log.gz 2024-01-16T15:25:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47549.log 2024-01-16T15:25:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82835.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82835.log.gz 2024-01-16T15:25:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77385.log 2024-01-16T15:25:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53057.log 2024-01-16T15:25:12.510 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47549.log.gz 2024-01-16T15:25:12.512 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77385.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77385.log.gz 2024-01-16T15:25:12.512 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69998.log 2024-01-16T15:25:12.513 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53057.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53057.log.gz 2024-01-16T15:25:12.513 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80612.log 2024-01-16T15:25:12.515 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69998.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69998.log.gz 2024-01-16T15:25:12.515 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.65945.log 2024-01-16T15:25:12.516 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80612.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80612.log.gz 2024-01-16T15:25:12.516 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T15:25:12.518 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70210.log 2024-01-16T15:25:12.518 INFO:teuthology.orchestra.run.smithi125.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.65945.log.gz 2024-01-16T15:25:12.524 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68351.log 2024-01-16T15:25:12.524 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70210.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70210.log.gz 2024-01-16T15:25:12.524 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80858.log 2024-01-16T15:25:12.524 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68351.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68351.log.gz 2024-01-16T15:25:12.524 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60839.log 2024-01-16T15:25:12.525 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80858.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80858.log.gz 2024-01-16T15:25:12.526 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60839.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60839.log.gz 2024-01-16T15:25:12.526 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68057.log 2024-01-16T15:25:12.528 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36092.log 2024-01-16T15:25:12.529 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68057.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68057.log.gz 2024-01-16T15:25:12.530 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74649.log 2024-01-16T15:25:12.532 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.36092.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36092.log.gz 2024-01-16T15:25:12.532 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log/var/log/ceph/ceph-client.admin.74649.log: 2024-01-16T15:25:12.533 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74649.log.gz 2024-01-16T15:25:12.533 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62848.log 2024-01-16T15:25:12.535 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47591.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2024-01-16T15:25:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70040.log/var/log/ceph/ceph-client.admin.62848.log: 2024-01-16T15:25:12.536 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62848.log.gz 2024-01-16T15:25:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49235.log 2024-01-16T15:25:12.539 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70040.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70040.log.gz 2024-01-16T15:25:12.539 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5/var/log/ceph/ceph-client.admin.49235.log: --verbose -- /var/log/ceph/ceph-client.admin.65150.log 2024-01-16T15:25:12.540 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49235.log.gz 2024-01-16T15:25:12.541 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74469.log 2024-01-16T15:25:12.542 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65150.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65150.log.gz 2024-01-16T15:25:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59806.log 2024-01-16T15:25:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74469.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74469.log.gz 2024-01-16T15:25:12.545 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83800.log 2024-01-16T15:25:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59806.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59806.log.gz 2024-01-16T15:25:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.83800.log: /var/log/ceph/ceph-client.admin.43656.log 2024-01-16T15:25:12.547 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83800.log.gz 2024-01-16T15:25:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76123.log 2024-01-16T15:25:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43656.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43656.log.gz 2024-01-16T15:25:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69291.log 2024-01-16T15:25:12.552 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76123.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76123.log.gz 2024-01-16T15:25:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50661.log 2024-01-16T15:25:12.554 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69291.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-01-16T15:25:12.554 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50661.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50661.log.gz 2024-01-16T15:25:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55160.log 2024-01-16T15:25:12.557 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-01-16T15:25:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66431.log 2024-01-16T15:25:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55160.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55160.log.gz 2024-01-16T15:25:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65444.log 2024-01-16T15:25:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66431.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66431.log.gz 2024-01-16T15:25:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78572.log/var/log/ceph/ceph-client.admin.65444.log: 2024-01-16T15:25:12.562 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65444.log.gz 2024-01-16T15:25:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58301.log 2024-01-16T15:25:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78572.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78572.log.gz 2024-01-16T15:25:12.566 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62115.log/var/log/ceph/ceph-client.admin.58301.log: 2024-01-16T15:25:12.566 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58301.log.gz 2024-01-16T15:25:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79119.log 2024-01-16T15:25:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62115.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62115.log.gz 2024-01-16T15:25:12.569 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70986.log 2024-01-16T15:25:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79119.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79119.log.gz 2024-01-16T15:25:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66881.log 2024-01-16T15:25:12.572 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70986.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70986.log.gz 2024-01-16T15:25:12.573 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56115.log/var/log/ceph/ceph-client.admin.66881.log: 2024-01-16T15:25:12.574 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66881.log.gz 2024-01-16T15:25:12.575 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69800.log 2024-01-16T15:25:12.576 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56115.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56115.log.gz 2024-01-16T15:25:12.577 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69592.log 2024-01-16T15:25:12.577 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69800.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69800.log.gz 2024-01-16T15:25:12.578 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45530.log 2024-01-16T15:25:12.580 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69592.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69592.log.gz 2024-01-16T15:25:12.580 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80653.log 2024-01-16T15:25:12.581 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45530.log.gz 2024-01-16T15:25:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69375.log 2024-01-16T15:25:12.584 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80653.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80653.log.gz 2024-01-16T15:25:12.584 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58641.log 2024-01-16T15:25:12.586 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69375.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69375.log.gz 2024-01-16T15:25:12.586 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.58641.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62992.log 2024-01-16T15:25:12.587 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58641.log.gz 2024-01-16T15:25:12.588 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78223.log 2024-01-16T15:25:12.589 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62992.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62992.log.gz 2024-01-16T15:25:12.590 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45362.log 2024-01-16T15:25:12.591 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78223.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78223.log.gz 2024-01-16T15:25:12.592 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57071.log 2024-01-16T15:25:12.593 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45362.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45362.log.gz 2024-01-16T15:25:12.594 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50277.log 2024-01-16T15:25:12.594 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57071.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57071.log.gz 2024-01-16T15:25:12.595 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67889.log 2024-01-16T15:25:12.596 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50277.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50277.log.gz 2024-01-16T15:25:12.598 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67889.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67889.log.gz 2024-01-16T15:25:12.598 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46769.log 2024-01-16T15:25:12.600 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46853.log 2024-01-16T15:25:12.601 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46769.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46769.log.gz 2024-01-16T15:25:12.602 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2024-01-16T15:25:12.604 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46853.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46853.log.gz 2024-01-16T15:25:12.604 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45740.log 2024-01-16T15:25:12.604 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42744.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2024-01-16T15:25:12.605 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51174.log 2024-01-16T15:25:12.606 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45740.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45740.log.gz 2024-01-16T15:25:12.608 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56436.log 2024-01-16T15:25:12.608 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51174.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51174.log.gz 2024-01-16T15:25:12.609 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57381.log 2024-01-16T15:25:12.610 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56436.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56436.log.gz 2024-01-16T15:25:12.611 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43068.log 2024-01-16T15:25:12.612 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57381.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57381.log.gz 2024-01-16T15:25:12.613 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76696.log 2024-01-16T15:25:12.614 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43068.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43068.log.gz 2024-01-16T15:25:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77343.log 2024-01-16T15:25:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76696.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76696.log.gz 2024-01-16T15:25:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61947.log 2024-01-16T15:25:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77343.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77343.log.gz 2024-01-16T15:25:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62456.log 2024-01-16T15:25:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61947.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61947.log.gz 2024-01-16T15:25:12.621 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62456.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62456.log.gz 2024-01-16T15:25:12.622 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72243.log 2024-01-16T15:25:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73735.log 2024-01-16T15:25:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72243.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72243.log.gz 2024-01-16T15:25:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55328.log 2024-01-16T15:25:12.626 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73735.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73735.log.gz 2024-01-16T15:25:12.627 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46955.log 2024-01-16T15:25:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55328.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55328.log.gz 2024-01-16T15:25:12.629 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65318.log 2024-01-16T15:25:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46955.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46955.log.gz 2024-01-16T15:25:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82583.log 2024-01-16T15:25:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65318.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65318.log.gz 2024-01-16T15:25:12.632 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53183.log 2024-01-16T15:25:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82583.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82583.log.gz 2024-01-16T15:25:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56193.log 2024-01-16T15:25:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53183.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53183.log.gz 2024-01-16T15:25:12.637 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56193.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56193.log.gz 2024-01-16T15:25:12.638 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46565.log 2024-01-16T15:25:12.639 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2024-01-16T15:25:12.641 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46565.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46565.log.gz 2024-01-16T15:25:12.641 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81122.log 2024-01-16T15:25:12.642 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48005.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2024-01-16T15:25:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59680.log 2024-01-16T15:25:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81122.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.81122.log.gz 2024-01-16T15:25:12.645 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67259.log 2024-01-16T15:25:12.646 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59680.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59680.log.gz 2024-01-16T15:25:12.647 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63034.log 2024-01-16T15:25:12.647 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67259.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67259.log.gz 2024-01-16T15:25:12.648 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59595.log 2024-01-16T15:25:12.649 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63034.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63034.log.gz 2024-01-16T15:25:12.650 INFO:teuthology.orchestra.run.smithi137.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-16T15:25:12.651 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61259.log 2024-01-16T15:25:12.651 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59595.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59595.log.gz 2024-01-16T15:25:12.652 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41636.log 2024-01-16T15:25:12.653 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61259.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61259.log.gz 2024-01-16T15:25:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72834.log/var/log/ceph/ceph-client.admin.41636.log: 2024-01-16T15:25:12.655 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41636.log.gz 2024-01-16T15:25:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56478.log 2024-01-16T15:25:12.657 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72834.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72834.log.gz 2024-01-16T15:25:12.658 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61905.log 2024-01-16T15:25:12.659 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56478.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56478.log.gz 2024-01-16T15:25:12.659 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55832.log 2024-01-16T15:25:12.661 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61905.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61905.log.gz 2024-01-16T15:25:12.662 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56646.log/var/log/ceph/ceph-client.admin.55832.log: 2024-01-16T15:25:12.662 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55832.log.gz 2024-01-16T15:25:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47323.log 2024-01-16T15:25:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56646.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56646.log.gz 2024-01-16T15:25:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.47323.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50319.log 2024-01-16T15:25:12.666 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47323.log.gz 2024-01-16T15:25:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79497.log 2024-01-16T15:25:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50319.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50319.log.gz 2024-01-16T15:25:12.669 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69904.log/var/log/ceph/ceph-client.admin.79497.log: 2024-01-16T15:25:12.670 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79497.log.gz 2024-01-16T15:25:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82195.log 2024-01-16T15:25:12.673 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69904.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69904.log.gz 2024-01-16T15:25:12.673 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42392.log 2024-01-16T15:25:12.673 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82195.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82195.log.gz 2024-01-16T15:25:12.674 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57113.log 2024-01-16T15:25:12.675 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42392.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42392.log.gz 2024-01-16T15:25:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52929.log 2024-01-16T15:25:12.677 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57113.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57113.log.gz 2024-01-16T15:25:12.678 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82279.log 2024-01-16T15:25:12.679 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52929.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52929.log.gz 2024-01-16T15:25:12.681 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2024-01-16T15:25:12.681 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82279.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82279.log.gz 2024-01-16T15:25:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-01-16T15:25:12.683 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49543.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2024-01-16T15:25:12.684 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29764.log/var/log/ceph/ceph-client.admin.41552.log: 2024-01-16T15:25:12.685 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-01-16T15:25:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69048.log 2024-01-16T15:25:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.tmp-client.admin.29764.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29764.log.gz 2024-01-16T15:25:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71208.log 2024-01-16T15:25:12.689 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69048.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69048.log.gz 2024-01-16T15:25:12.689 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69165.log 2024-01-16T15:25:12.690 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71208.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71208.log.gz 2024-01-16T15:25:12.691 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-01-16T15:25:12.692 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69165.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69165.log.gz 2024-01-16T15:25:12.693 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76833.log 2024-01-16T15:25:12.693 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47711.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-01-16T15:25:12.694 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52240.log 2024-01-16T15:25:12.695 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76833.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76833.log.gz 2024-01-16T15:25:12.696 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35836.log 2024-01-16T15:25:12.697 INFO:teuthology.orchestra.run.smithi137.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-16T15:25:12.697 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52240.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52240.log.gz 2024-01-16T15:25:12.698 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-01-16T15:25:12.699 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.35836.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35836.log.gz 2024-01-16T15:25:12.700 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49373.log 2024-01-16T15:25:12.701 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41850.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-01-16T15:25:12.702 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49373.log: /var/log/ceph/ceph-client.admin.60401.log 2024-01-16T15:25:12.702 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49373.log.gz 2024-01-16T15:25:12.703 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82119.log 2024-01-16T15:25:12.705 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60401.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60401.log.gz 2024-01-16T15:25:12.705 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67049.log 2024-01-16T15:25:12.706 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82119.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82119.log.gz 2024-01-16T15:25:12.707 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82035.log 2024-01-16T15:25:12.708 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67049.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67049.log.gz 2024-01-16T15:25:12.709 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74911.log 2024-01-16T15:25:12.710 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82035.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82035.log.gz 2024-01-16T15:25:12.711 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67595.log 2024-01-16T15:25:12.712 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74911.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74911.log.gz 2024-01-16T15:25:12.713 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56986.log/var/log/ceph/ceph-client.admin.67595.log: 2024-01-16T15:25:12.714 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67595.log.gz 2024-01-16T15:25:12.714 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66329.log 2024-01-16T15:25:12.716 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56986.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56986.log.gz 2024-01-16T15:25:12.717 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42739.log 2024-01-16T15:25:12.717 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66329.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66329.log.gz 2024-01-16T15:25:12.718 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43530.log 2024-01-16T15:25:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42739.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42739.log.gz 2024-01-16T15:25:12.721 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43530.log: /var/log/ceph/ceph-client.admin.51216.log 2024-01-16T15:25:12.721 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43530.log.gz 2024-01-16T15:25:12.722 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77079.log 2024-01-16T15:25:12.723 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51216.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51216.log.gz 2024-01-16T15:25:12.724 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.77079.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48383.log 2024-01-16T15:25:12.725 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77079.log.gz 2024-01-16T15:25:12.726 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79896.log 2024-01-16T15:25:12.728 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48383.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48383.log.gz 2024-01-16T15:25:12.728 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67553.log/var/log/ceph/ceph-client.admin.79896.log: 2024-01-16T15:25:12.728 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79896.log.gz 2024-01-16T15:25:12.730 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-16T15:25:12.731 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67553.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67553.log.gz 2024-01-16T15:25:12.731 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61863.log 2024-01-16T15:25:12.734 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.61863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58825.log 2024-01-16T15:25:12.735 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61863.log.gz 2024-01-16T15:25:12.737 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69512.log 2024-01-16T15:25:12.739 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58825.log.gz 2024-01-16T15:25:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76285.log 2024-01-16T15:25:12.742 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69512.log.gz 2024-01-16T15:25:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55933.log 2024-01-16T15:25:12.746 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76285.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76285.log.gz 2024-01-16T15:25:12.749 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56235.log 2024-01-16T15:25:12.750 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55933.log.gz 2024-01-16T15:25:12.751 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83296.log 2024-01-16T15:25:12.753 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56235.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56235.log.gz 2024-01-16T15:25:12.756 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47465.log 2024-01-16T15:25:12.759 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47465.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83296.log.gz 2024-01-16T15:25:12.760 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81368.log 2024-01-16T15:25:12.763 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81368.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71709.log 2024-01-16T15:25:12.763 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47465.log.gz 2024-01-16T15:25:12.764 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81368.log.gz 2024-01-16T15:25:12.766 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41808.log 2024-01-16T15:25:12.770 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41808.log: /var/log/ceph/ceph-client.admin.71709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51576.log 2024-01-16T15:25:12.772 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71709.log.gz 2024-01-16T15:25:12.772 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41808.log.gz 2024-01-16T15:25:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81714.log 2024-01-16T15:25:12.777 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81714.log: /var/log/ceph/ceph-client.admin.51576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64352.log 2024-01-16T15:25:12.779 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51576.log.gz 2024-01-16T15:25:12.779 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81714.log.gz 2024-01-16T15:25:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76444.log 2024-01-16T15:25:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64352.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64352.log.gz 2024-01-16T15:25:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68519.log 2024-01-16T15:25:12.787 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68519.log: /var/log/ceph/ceph-client.admin.76444.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48735.log 2024-01-16T15:25:12.787 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76444.log.gz 2024-01-16T15:25:12.788 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68519.log.gz 2024-01-16T15:25:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50445.log 2024-01-16T15:25:12.794 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50445.log: /var/log/ceph/ceph-client.admin.48735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2024-01-16T15:25:12.796 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48735.log.gz 2024-01-16T15:25:12.796 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50445.log.gz 2024-01-16T15:25:12.798 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77429.log 2024-01-16T15:25:12.801 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77429.log: /var/log/ceph/ceph-client.admin.41722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46160.log 2024-01-16T15:25:12.804 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2024-01-16T15:25:12.804 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77429.log.gz 2024-01-16T15:25:12.804 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80530.log 2024-01-16T15:25:12.805 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46160.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46160.log.gz 2024-01-16T15:25:12.807 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2024-01-16T15:25:12.810 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41934.log: /var/log/ceph/ceph-client.admin.80530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48467.log 2024-01-16T15:25:12.812 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72606.log 2024-01-16T15:25:12.812 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80530.log.gz 2024-01-16T15:25:12.814 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72606.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2024-01-16T15:25:12.815 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72606.log.gz 2024-01-16T15:25:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76165.log 2024-01-16T15:25:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56277.log 2024-01-16T15:25:12.820 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82751.log 2024-01-16T15:25:12.820 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56277.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56277.log.gz 2024-01-16T15:25:12.821 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48467.log.gz 2024-01-16T15:25:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80776.log 2024-01-16T15:25:12.826 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82751.log: 88.8% 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74427.log 2024-01-16T15:25:12.827 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80776.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80776.log.gz 2024-01-16T15:25:12.828 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82751.log.gz 2024-01-16T15:25:12.828 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76165.log.gz 2024-01-16T15:25:12.828 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82363.log 2024-01-16T15:25:12.830 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42399.log 2024-01-16T15:25:12.834 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6%/var/log/ceph/ceph-client.admin.42399.log: -- replaced with /var/log/ceph/ceph-client.admin.74427.log.gz 2024-01-16T15:25:12.834 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73051.log 2024-01-16T15:25:12.835 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82363.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82363.log.gz 2024-01-16T15:25:12.836 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42399.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83044.log 2024-01-16T15:25:12.837 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73051.log: 2024-01-16T15:25:12.840 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83044.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73051.log.gz 2024-01-16T15:25:12.842 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83044.log.gz 2024-01-16T15:25:12.842 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74553.log 2024-01-16T15:25:12.843 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48131.log 2024-01-16T15:25:12.845 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51635.log 2024-01-16T15:25:12.846 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48131.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74553.log.gz 2024-01-16T15:25:12.848 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56604.log 2024-01-16T15:25:12.849 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51635.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51635.log.gz 2024-01-16T15:25:12.849 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48131.log.gz 2024-01-16T15:25:12.851 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58557.log 2024-01-16T15:25:12.852 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56604.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56604.log.gz 2024-01-16T15:25:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62372.log 2024-01-16T15:25:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58557.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58557.log.gz 2024-01-16T15:25:12.858 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62372.log: gzip 88.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.49627.log 2024-01-16T15:25:12.858 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62372.log.gz 2024-01-16T15:25:12.859 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59638.log 2024-01-16T15:25:12.861 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50745.log 2024-01-16T15:25:12.863 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2024-01-16T15:25:12.864 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82793.log 2024-01-16T15:25:12.867 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50745.log: 89.0%/var/log/ceph/ceph-client.admin.82793.log: -- replaced with /var/log/ceph/ceph-client.admin.59638.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71028.log 2024-01-16T15:25:12.867 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:12.868 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50745.log.gz 2024-01-16T15:25:12.869 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82793.log.gz 2024-01-16T15:25:12.870 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47423.log 2024-01-16T15:25:12.874 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71028.log: /var/log/ceph/ceph-client.admin.47423.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71028.log.gz 2024-01-16T15:25:12.875 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64964.log 2024-01-16T15:25:12.875 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47423.log.gz 2024-01-16T15:25:12.876 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65570.log 2024-01-16T15:25:12.878 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56863.log 2024-01-16T15:25:12.879 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65570.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65570.log.gz 2024-01-16T15:25:12.880 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64964.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64964.log.gz 2024-01-16T15:25:12.881 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59457.log 2024-01-16T15:25:12.884 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72285.log 2024-01-16T15:25:12.885 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59457.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59457.log.gz 2024-01-16T15:25:12.886 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56863.log.gz 2024-01-16T15:25:12.888 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.72285.log: /var/log/ceph/ceph-client.admin.59905.log 2024-01-16T15:25:12.889 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72285.log.gz 2024-01-16T15:25:12.891 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59905.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59905.log.gz 2024-01-16T15:25:12.892 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73301.log 2024-01-16T15:25:12.895 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43824.log 2024-01-16T15:25:12.895 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73301.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73301.log.gz 2024-01-16T15:25:12.895 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64520.log 2024-01-16T15:25:12.898 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55598.log 2024-01-16T15:25:12.899 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64520.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64520.log.gz 2024-01-16T15:25:12.904 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43824.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59329.log 2024-01-16T15:25:12.904 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55598.log: /var/log/ceph/ceph-client.admin.59329.log: 2024-01-16T15:25:12.905 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55598.log.gz 2024-01-16T15:25:12.906 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59329.log.gz 2024-01-16T15:25:12.906 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70126.log 2024-01-16T15:25:12.907 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75349.log 2024-01-16T15:25:12.909 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63184.log 2024-01-16T15:25:12.910 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75349.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75349.log.gz 2024-01-16T15:25:12.911 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70126.log.gz 2024-01-16T15:25:12.913 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2024-01-16T15:25:12.914 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63184.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63184.log.gz 2024-01-16T15:25:12.916 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64394.log 2024-01-16T15:25:12.916 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51817.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2024-01-16T15:25:12.918 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83338.log 2024-01-16T15:25:12.920 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64394.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64394.log.gz 2024-01-16T15:25:12.921 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64754.log 2024-01-16T15:25:12.922 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83338.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83338.log.gz 2024-01-16T15:25:12.925 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64754.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64754.log.gz 2024-01-16T15:25:12.926 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82541.log 2024-01-16T15:25:12.928 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43866.log 2024-01-16T15:25:12.929 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82541.log.gz 2024-01-16T15:25:12.931 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79699.log 2024-01-16T15:25:12.935 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43866.log: /var/log/ceph/ceph-client.admin.79699.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59764.log 2024-01-16T15:25:12.935 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43866.log.gz 2024-01-16T15:25:12.936 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79699.log.gz 2024-01-16T15:25:12.938 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62414.log 2024-01-16T15:25:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59764.log: /var/log/ceph/ceph-client.admin.62414.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66557.log 2024-01-16T15:25:12.942 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59764.log.gz 2024-01-16T15:25:12.944 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62414.log.gz 2024-01-16T15:25:12.945 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2024-01-16T15:25:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66557.log: /var/log/ceph/ceph-client.admin.48635.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66557.log.gzgzip 2024-01-16T15:25:12.950 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-01-16T15:25:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42144.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2024-01-16T15:25:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45278.log 2024-01-16T15:25:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.71649.log 2024-01-16T15:25:12.957 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45278.log: -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-01-16T15:25:12.958 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45278.log.gz 2024-01-16T15:25:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45572.log 2024-01-16T15:25:12.964 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45572.log: /var/log/ceph/ceph-client.admin.71649.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81163.log 2024-01-16T15:25:12.964 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71649.log.gz 2024-01-16T15:25:12.965 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45572.log.gz 2024-01-16T15:25:12.966 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51468.log 2024-01-16T15:25:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51468.log: /var/log/ceph/ceph-client.admin.81163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2024-01-16T15:25:12.971 INFO:teuthology.orchestra.run.smithi125.stderr: 87.5% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81163.log.gz 2024-01-16T15:25:12.971 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51468.log.gz 2024-01-16T15:25:12.972 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78975.log 2024-01-16T15:25:12.976 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78975.log: /var/log/ceph/ceph-client.admin.43362.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45656.log 2024-01-16T15:25:12.976 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2024-01-16T15:25:12.977 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78975.log.gz 2024-01-16T15:25:12.979 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51342.log 2024-01-16T15:25:12.982 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51342.log: /var/log/ceph/ceph-client.admin.45656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60206.log 2024-01-16T15:25:12.985 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45656.log.gz 2024-01-16T15:25:12.985 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51342.log.gz 2024-01-16T15:25:12.986 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55064.log 2024-01-16T15:25:12.989 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55064.log: /var/log/ceph/ceph-client.admin.60206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82237.log 2024-01-16T15:25:12.991 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60206.log.gz 2024-01-16T15:25:12.992 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55064.log.gz 2024-01-16T15:25:12.993 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73389.log 2024-01-16T15:25:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73389.log: /var/log/ceph/ceph-client.admin.82237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45114.log 2024-01-16T15:25:12.998 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82237.log.gz 2024-01-16T15:25:12.999 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73389.log.gz 2024-01-16T15:25:13.000 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43572.log 2024-01-16T15:25:13.004 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43572.log: /var/log/ceph/ceph-client.admin.45114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61779.log 2024-01-16T15:25:13.006 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45114.log.gz 2024-01-16T15:25:13.006 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43572.log.gz 2024-01-16T15:25:13.008 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59275.log 2024-01-16T15:25:13.009 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65048.log 2024-01-16T15:25:13.010 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61779.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61779.log.gz 2024-01-16T15:25:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68099.log 2024-01-16T15:25:13.012 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65048.log: 88.4% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65048.log.gz 2024-01-16T15:25:13.015 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68099.log: -- replaced with /var/log/ceph/ceph-client.admin.59275.log.gz 2024-01-16T15:25:13.015 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80143.log 2024-01-16T15:25:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80143.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68099.log.gz 2024-01-16T15:25:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-16T15:25:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2024-01-16T15:25:13.022 INFO:teuthology.orchestra.run.smithi125.stderr: 86.6%/var/log/ceph/ceph-mgr.y.log: -- replaced with /var/log/ceph/ceph-client.admin.80143.log.gz 2024-01-16T15:25:13.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48047.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz -5 2024-01-16T15:25:13.027 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69758.log 2024-01-16T15:25:13.030 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2024-01-16T15:25:13.032 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 2024-01-16T15:25:13.033 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69758.log.gz 2024-01-16T15:25:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41386.log: /var/log/ceph/ceph-client.admin.47753.log: 89.1% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-01-16T15:25:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84016.log 2024-01-16T15:25:13.039 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83976.log 2024-01-16T15:25:13.039 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.84016.log: -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-01-16T15:25:13.040 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84016.log.gz 2024-01-16T15:25:13.044 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67931.log 2024-01-16T15:25:13.045 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83976.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83976.log.gz 2024-01-16T15:25:13.048 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70336.log 2024-01-16T15:25:13.048 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67931.log.gz 2024-01-16T15:25:13.051 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52791.log 2024-01-16T15:25:13.052 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70336.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70336.log.gz 2024-01-16T15:25:13.058 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52791.log: -- /var/log/ceph/ceph-client.admin.71154.log 2024-01-16T15:25:13.059 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52791.log.gz 2024-01-16T15:25:13.061 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75924.log 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71154.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83254.log 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71154.log.gz 2024-01-16T15:25:13.066 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75924.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75924.log.gz 2024-01-16T15:25:13.067 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68015.log 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-01-16T15:25:13.071 INFO:teuthology.orchestra.run.smithi125.stderr: 87.1%/var/log/ceph/ceph-client.admin.68015.log: -- replaced with /var/log/ceph/ceph-client.admin.83254.log.gz 2024-01-16T15:25:13.072 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68015.log.gz 2024-01-16T15:25:13.074 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65696.log 2024-01-16T15:25:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-01-16T15:25:13.076 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-01-16T15:25:13.081 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55514.log 2024-01-16T15:25:13.081 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65696.log.gz 2024-01-16T15:25:13.082 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42741.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2024-01-16T15:25:13.085 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55514.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55514.log.gz 2024-01-16T15:25:13.086 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66965.log 2024-01-16T15:25:13.087 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70252.log 2024-01-16T15:25:13.089 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66965.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66965.log.gz 2024-01-16T15:25:13.091 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57483.log 2024-01-16T15:25:13.093 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70252.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70252.log.gz 2024-01-16T15:25:13.094 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47837.log 2024-01-16T15:25:13.095 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57483.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57483.log.gz 2024-01-16T15:25:13.100 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66743.log 2024-01-16T15:25:13.100 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47837.log.gz 2024-01-16T15:25:13.101 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2024-01-16T15:25:13.102 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66743.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66743.log.gz 2024-01-16T15:25:13.106 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79741.log 2024-01-16T15:25:13.107 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54402.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2024-01-16T15:25:13.110 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78476.log 2024-01-16T15:25:13.110 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79741.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79741.log.gz 2024-01-16T15:25:13.115 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78476.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78476.log.gz 2024-01-16T15:25:13.116 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79979.log 2024-01-16T15:25:13.121 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80817.log 2024-01-16T15:25:13.122 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79979.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.79979.log.gz 2024-01-16T15:25:13.125 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67175.log 2024-01-16T15:25:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80817.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.80817.log.gz 2024-01-16T15:25:13.128 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49109.log 2024-01-16T15:25:13.130 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67175.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67175.log.gz 2024-01-16T15:25:13.134 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49109.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49109.log.gz 2024-01-16T15:25:13.135 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43488.log 2024-01-16T15:25:13.138 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77121.log 2024-01-16T15:25:13.140 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81755.log 2024-01-16T15:25:13.141 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77121.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77121.log.gz 2024-01-16T15:25:13.143 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43488.log.gz 2024-01-16T15:25:13.145 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81951.log 2024-01-16T15:25:13.146 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81755.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.81755.log.gz 2024-01-16T15:25:13.149 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81951.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81951.log.gz 2024-01-16T15:25:13.151 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45782.log 2024-01-16T15:25:13.153 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62720.log 2024-01-16T15:25:13.156 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50703.log 2024-01-16T15:25:13.158 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62720.log: /var/log/ceph/ceph-client.admin.45782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62720.log.gz 2024-01-16T15:25:13.158 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45782.log.gz 2024-01-16T15:25:13.159 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74204.log 2024-01-16T15:25:13.160 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50703.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50703.log.gz 2024-01-16T15:25:13.163 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61360.log 2024-01-16T15:25:13.166 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.74204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60493.log 2024-01-16T15:25:13.167 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61360.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61360.log.gz 2024-01-16T15:25:13.168 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80407.log 2024-01-16T15:25:13.169 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60493.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60493.log.gz 2024-01-16T15:25:13.173 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80407.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.80407.log.gz 2024-01-16T15:25:13.174 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54754.log 2024-01-16T15:25:13.177 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55454.log 2024-01-16T15:25:13.179 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54754.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54754.log.gz 2024-01-16T15:25:13.181 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61500.log 2024-01-16T15:25:13.182 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55454.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55454.log.gz 2024-01-16T15:25:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66515.log 2024-01-16T15:25:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61500.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61500.log.gz 2024-01-16T15:25:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66515.log.gz 2024-01-16T15:25:13.192 INFO:teuthology.orchestra.run.smithi125.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.53863.log.gz 2024-01-16T15:25:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2024-01-16T15:25:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49415.log 2024-01-16T15:25:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45698.log: gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz -- 2024-01-16T15:25:13.199 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.68477.log 2024-01-16T15:25:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49415.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49415.log.gz 2024-01-16T15:25:13.204 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72564.log 2024-01-16T15:25:13.205 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68477.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68477.log.gz 2024-01-16T15:25:13.209 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83716.log 2024-01-16T15:25:13.210 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72564.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72564.log.gz 2024-01-16T15:25:13.211 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45866.log 2024-01-16T15:25:13.212 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83716.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83716.log.gz 2024-01-16T15:25:13.215 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45866.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45866.log.gz 2024-01-16T15:25:13.216 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72363.log 2024-01-16T15:25:13.220 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56394.log 2024-01-16T15:25:13.221 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72363.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72363.log.gz 2024-01-16T15:25:13.224 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83422.log 2024-01-16T15:25:13.226 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56394.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56394.log.gz 2024-01-16T15:25:13.230 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78731.log 2024-01-16T15:25:13.231 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83422.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83422.log.gz 2024-01-16T15:25:13.234 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61542.log 2024-01-16T15:25:13.236 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78731.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78731.log.gz 2024-01-16T15:25:13.238 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61542.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61542.log.gz 2024-01-16T15:25:13.240 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78933.log 2024-01-16T15:25:13.244 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70944.log 2024-01-16T15:25:13.246 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78933.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78933.log.gz 2024-01-16T15:25:13.246 INFO:teuthology.orchestra.run.smithi125.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.63461.log.gz 2024-01-16T15:25:13.249 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57693.log 2024-01-16T15:25:13.250 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46727.log 2024-01-16T15:25:13.252 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8%/var/log/ceph/ceph-client.admin.57693.log: -- replaced with /var/log/ceph/ceph-client.admin.70944.log.gz 2024-01-16T15:25:13.254 INFO:teuthology.orchestra.run.smithi125.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57693.log.gz 2024-01-16T15:25:13.254 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83674.log 2024-01-16T15:25:13.256 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46727.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46727.log.gz 2024-01-16T15:25:13.257 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57029.log 2024-01-16T15:25:13.259 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83674.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83674.log.gz 2024-01-16T15:25:13.263 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57029.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57029.log.gz 2024-01-16T15:25:13.263 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T15:25:13.266 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77163.log 2024-01-16T15:25:13.270 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55790.log 2024-01-16T15:25:13.272 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78530.log 2024-01-16T15:25:13.273 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77163.log.gz 2024-01-16T15:25:13.277 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78530.log: /var/log/ceph/ceph-client.admin.55790.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78530.log.gz 2024-01-16T15:25:13.277 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.55790.log.gz 2024-01-16T15:25:13.278 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62678.log 2024-01-16T15:25:13.283 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51979.log 2024-01-16T15:25:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62678.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62678.log.gz 2024-01-16T15:25:13.285 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52879.log 2024-01-16T15:25:13.286 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51979.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51979.log.gz 2024-01-16T15:25:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-01-16T15:25:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52879.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52879.log.gz 2024-01-16T15:25:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42947.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-01-16T15:25:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63076.log 2024-01-16T15:25:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42738.log 2024-01-16T15:25:13.298 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63076.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63076.log.gz 2024-01-16T15:25:13.302 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82499.log 2024-01-16T15:25:13.303 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42738.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42738.log.gz 2024-01-16T15:25:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55244.log 2024-01-16T15:25:13.307 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82499.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82499.log.gz 2024-01-16T15:25:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55244.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55244.log.gz 2024-01-16T15:25:13.312 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75882.log 2024-01-16T15:25:13.313 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49669.log 2024-01-16T15:25:13.316 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56944.log 2024-01-16T15:25:13.320 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75882.log.gz 2024-01-16T15:25:13.320 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57158.log 2024-01-16T15:25:13.320 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49669.log.gz 2024-01-16T15:25:13.321 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56944.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56944.log.gz 2024-01-16T15:25:13.324 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49861.log 2024-01-16T15:25:13.325 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57158.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57158.log.gz 2024-01-16T15:25:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72648.log 2024-01-16T15:25:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49861.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49861.log.gz 2024-01-16T15:25:13.329 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78434.log 2024-01-16T15:25:13.331 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72648.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72648.log.gz 2024-01-16T15:25:13.333 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49753.log 2024-01-16T15:25:13.333 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78434.log.gz 2024-01-16T15:25:13.335 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55706.log 2024-01-16T15:25:13.337 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49753.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2024-01-16T15:25:13.341 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53435.log 2024-01-16T15:25:13.343 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43404.log 2024-01-16T15:25:13.343 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55706.log.gz 2024-01-16T15:25:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53435.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2024-01-16T15:25:13.346 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-01-16T15:25:13.347 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43404.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43404.log.gz 2024-01-16T15:25:13.347 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75529.log 2024-01-16T15:25:13.349 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41594.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-01-16T15:25:13.351 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-01-16T15:25:13.352 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75529.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75529.log.gz 2024-01-16T15:25:13.353 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74987.log 2024-01-16T15:25:13.355 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41467.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-01-16T15:25:13.356 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54486.log 2024-01-16T15:25:13.358 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74987.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74987.log.gz 2024-01-16T15:25:13.363 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54486.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54486.log.gz 2024-01-16T15:25:13.364 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72201.log 2024-01-16T15:25:13.368 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49193.log 2024-01-16T15:25:13.369 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72201.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72201.log.gz 2024-01-16T15:25:13.370 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75053.log 2024-01-16T15:25:13.371 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49193.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49193.log.gz 2024-01-16T15:25:13.374 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46649.log 2024-01-16T15:25:13.375 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75053.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75053.log.gz 2024-01-16T15:25:13.376 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55412.log 2024-01-16T15:25:13.377 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46649.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46649.log.gz 2024-01-16T15:25:13.381 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55412.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55412.log.gz 2024-01-16T15:25:13.383 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45614.log 2024-01-16T15:25:13.386 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52749.log 2024-01-16T15:25:13.387 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45614.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45614.log.gz 2024-01-16T15:25:13.389 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54895.log 2024-01-16T15:25:13.390 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52749.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52749.log.gz 2024-01-16T15:25:13.391 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45320.log 2024-01-16T15:25:13.393 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54895.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54895.log.gz 2024-01-16T15:25:13.397 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80225.log 2024-01-16T15:25:13.398 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45320.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45320.log.gz 2024-01-16T15:25:13.399 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:25:13.401 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80225.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.80225.log.gz 2024-01-16T15:25:13.406 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61989.log 2024-01-16T15:25:13.409 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56688.log 2024-01-16T15:25:13.410 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61989.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2024-01-16T15:25:13.411 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70902.log 2024-01-16T15:25:13.412 INFO:teuthology.orchestra.run.smithi125.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:25:13.413 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56688.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56688.log.gz 2024-01-16T15:25:13.419 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70902.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70902.log.gz 2024-01-16T15:25:13.421 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58397.log 2024-01-16T15:25:13.424 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42186.log 2024-01-16T15:25:13.425 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58397.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58397.log.gz 2024-01-16T15:25:13.427 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76994.log 2024-01-16T15:25:13.428 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42186.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42186.log.gz 2024-01-16T15:25:13.429 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-01-16T15:25:13.434 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43698.log 2024-01-16T15:25:13.435 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76994.log.gz 2024-01-16T15:25:13.436 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.41678.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-01-16T15:25:13.438 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81409.log 2024-01-16T15:25:13.440 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43698.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43698.log.gz 2024-01-16T15:25:13.441 INFO:teuthology.orchestra.run.smithi125.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.58036.log.gz 2024-01-16T15:25:13.442 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53586.log 2024-01-16T15:25:13.442 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81409.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.81409.log.gz 2024-01-16T15:25:13.444 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53586.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T15:25:13.448 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.2.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82321.log 2024-01-16T15:25:13.448 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53586.log.gz 2024-01-16T15:25:13.450 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80999.log 2024-01-16T15:25:13.453 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82321.log: /var/log/ceph/ceph-client.admin.80999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81632.log 2024-01-16T15:25:13.453 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82321.log.gz 2024-01-16T15:25:13.457 INFO:teuthology.orchestra.run.smithi125.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.80999.log.gz 2024-01-16T15:25:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81632.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.81632.log.gz 2024-01-16T15:25:13.459 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71751.log 2024-01-16T15:25:13.463 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73861.log 2024-01-16T15:25:13.464 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71751.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71751.log.gz 2024-01-16T15:25:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62628.log 2024-01-16T15:25:13.469 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73861.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.73861.log.gz 2024-01-16T15:25:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68309.log 2024-01-16T15:25:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62628.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62628.log.gz 2024-01-16T15:25:13.476 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66185.log 2024-01-16T15:25:13.476 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68309.log.gz 2024-01-16T15:25:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-16T15:25:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66185.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66185.log.gz 2024-01-16T15:25:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81286.log 2024-01-16T15:25:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61175.log 2024-01-16T15:25:13.492 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81286.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.81286.log.gz 2024-01-16T15:25:13.495 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75233.log 2024-01-16T15:25:13.496 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61175.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61175.log.gz 2024-01-16T15:25:13.497 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56031.log 2024-01-16T15:25:13.499 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75233.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75233.log.gz 2024-01-16T15:25:13.504 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53520.log 2024-01-16T15:25:13.505 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56031.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56031.log.gz 2024-01-16T15:25:13.507 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61662.log 2024-01-16T15:25:13.509 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53520.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.53520.log.gz 2024-01-16T15:25:13.511 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82457.log 2024-01-16T15:25:13.512 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61662.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61662.log.gz 2024-01-16T15:25:13.514 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82457.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82457.log.gz 2024-01-16T15:25:13.515 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60947.log 2024-01-16T15:25:13.517 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43908.log 2024-01-16T15:25:13.518 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60947.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60947.log.gz 2024-01-16T15:25:13.519 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78773.log 2024-01-16T15:25:13.521 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43908.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.43908.log.gz 2024-01-16T15:25:13.521 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75983.log 2024-01-16T15:25:13.522 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.78773.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78773.log.gz 2024-01-16T15:25:13.525 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.75983.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75983.log.gz 2024-01-16T15:25:13.527 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70763.log 2024-01-16T15:25:13.530 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46997.log 2024-01-16T15:25:13.534 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.70763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57297.log 2024-01-16T15:25:13.536 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46997.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46997.log.gz 2024-01-16T15:25:13.539 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57297.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57297.log.gz 2024-01-16T15:25:13.540 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83873.log 2024-01-16T15:25:13.542 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63982.log 2024-01-16T15:25:13.546 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83873.log: /var/log/ceph/ceph-client.admin.63982.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83873.log.gz 2024-01-16T15:25:13.548 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81468.log 2024-01-16T15:25:13.549 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63982.log.gz 2024-01-16T15:25:13.551 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81245.log 96.3% 2024-01-16T15:25:13.551 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.0.78084.log.gz 2024-01-16T15:25:13.556 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81468.log: gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.81468.log.gz 2024-01-16T15:25:13.556 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.48839.log 2024-01-16T15:25:13.560 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81245.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65654.log 2024-01-16T15:25:13.561 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48839.log: -- replaced with /var/log/ceph/ceph-client.admin.81245.log.gz 2024-01-16T15:25:13.561 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48839.log.gz 2024-01-16T15:25:13.563 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79938.log 2024-01-16T15:25:13.566 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65654.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44430.log 2024-01-16T15:25:13.567 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79938.log: -- replaced with /var/log/ceph/ceph-client.admin.65654.log.gz 2024-01-16T15:25:13.568 INFO:teuthology.orchestra.run.smithi125.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79938.log.gz 2024-01-16T15:25:13.571 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62950.log 2024-01-16T15:25:13.573 INFO:teuthology.orchestra.run.smithi125.stderr: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48881.log 2024-01-16T15:25:13.574 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62950.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62950.log.gz 2024-01-16T15:25:13.575 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.0.44430.log.gz 2024-01-16T15:25:13.576 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65234.log 2024-01-16T15:25:13.578 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70420.log 2024-01-16T15:25:13.579 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65234.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65234.log.gz 2024-01-16T15:25:13.580 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2024-01-16T15:25:13.583 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43099.log 2024-01-16T15:25:13.583 INFO:teuthology.orchestra.run.smithi125.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.70420.log.gz 2024-01-16T15:25:13.588 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2024-01-16T15:25:13.589 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43099.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43099.log.gz 2024-01-16T15:25:13.594 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47963.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz 2024-01-16T15:25:13.596 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43058.log 2024-01-16T15:25:13.600 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76243.log 2024-01-16T15:25:13.601 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43058.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43058.log.gz 2024-01-16T15:25:13.603 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50856.log 2024-01-16T15:25:13.607 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65612.log 2024-01-16T15:25:13.607 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76243.log.gz 2024-01-16T15:25:13.608 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50856.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2024-01-16T15:25:13.609 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-01-16T15:25:13.613 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-01-16T15:25:13.613 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65612.log.gz 2024-01-16T15:25:13.617 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.36220.log: 88.4%gzip/var/log/ceph/ceph-client.admin.41290.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.73345.log 2024-01-16T15:25:13.618 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-01-16T15:25:13.619 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-01-16T15:25:13.621 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48983.log 2024-01-16T15:25:13.622 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73345.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73345.log.gz 2024-01-16T15:25:13.623 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79795.log 2024-01-16T15:25:13.626 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57339.log 2024-01-16T15:25:13.627 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48983.log.gz 2024-01-16T15:25:13.629 INFO:teuthology.orchestra.run.smithi125.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79795.log.gz 2024-01-16T15:25:13.633 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.57339.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57339.log.gz 2024-01-16T15:25:13.633 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79437.log 2024-01-16T15:25:13.635 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73523.log 2024-01-16T15:25:13.640 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79437.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81509.log 2024-01-16T15:25:13.640 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79437.log.gz 2024-01-16T15:25:13.642 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73523.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73523.log.gz 2024-01-16T15:25:13.642 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66923.log 2024-01-16T15:25:13.646 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81509.log: gzip -5 --verbose 87.0%/var/log/ceph/ceph-client.admin.66923.log: -- replaced with /var/log/ceph/ceph-client.admin.81509.log.gz 2024-01-16T15:25:13.646 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.51775.log 2024-01-16T15:25:13.650 INFO:teuthology.orchestra.run.smithi125.stderr: 87.3%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.66923.log.gz --verbose -- /var/log/ceph/ceph-client.admin.73819.log 2024-01-16T15:25:13.650 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:13.651 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51775.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51775.log.gz 2024-01-16T15:25:13.654 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52971.log 2024-01-16T15:25:13.656 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73819.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73819.log.gz 2024-01-16T15:25:13.658 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73259.log 2024-01-16T15:25:13.658 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52971.log.gz 2024-01-16T15:25:13.661 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57253.log 2024-01-16T15:25:13.662 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73259.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73259.log.gz 2024-01-16T15:25:13.665 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57253.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57253.log.gz 2024-01-16T15:25:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51048.log 2024-01-16T15:25:13.671 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67007.log 2024-01-16T15:25:13.672 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51048.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51048.log.gz 2024-01-16T15:25:13.674 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42649.log 2024-01-16T15:25:13.675 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67007.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67007.log.gz 2024-01-16T15:25:13.676 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48257.log 2024-01-16T15:25:13.678 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42649.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2024-01-16T15:25:13.680 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48257.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48257.log.gz 2024-01-16T15:25:13.682 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48215.log 2024-01-16T15:25:13.686 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47123.log 2024-01-16T15:25:13.688 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48215.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48215.log.gz 2024-01-16T15:25:13.689 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53393.log 2024-01-16T15:25:13.691 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47123.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47123.log.gz 2024-01-16T15:25:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54956.log 2024-01-16T15:25:13.694 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53393.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53393.log.gz 2024-01-16T15:25:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67637.log 2024-01-16T15:25:13.698 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54956.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54956.log.gz 2024-01-16T15:25:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71465.log 2024-01-16T15:25:13.700 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67637.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67637.log.gz 2024-01-16T15:25:13.702 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72690.log 2024-01-16T15:25:13.703 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71465.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71465.log.gz 2024-01-16T15:25:13.705 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.72690.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72690.log.gz 2024-01-16T15:25:13.706 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65108.log 2024-01-16T15:25:13.710 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83590.log 2024-01-16T15:25:13.712 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65108.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65108.log.gz 2024-01-16T15:25:13.715 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64478.log 2024-01-16T15:25:13.717 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83590.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83590.log.gz 2024-01-16T15:25:13.719 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69123.log 2024-01-16T15:25:13.720 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64478.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64478.log.gz 2024-01-16T15:25:13.725 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69123.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60713.log 2024-01-16T15:25:13.726 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69123.log.gz 2024-01-16T15:25:13.731 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79161.log 2024-01-16T15:25:13.732 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60713.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60713.log.gz 2024-01-16T15:25:13.733 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83916.log 2024-01-16T15:25:13.735 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.79161.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79161.log.gz 2024-01-16T15:25:13.738 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66473.log 2024-01-16T15:25:13.739 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83916.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83916.log.gz 2024-01-16T15:25:13.739 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73217.log 2024-01-16T15:25:13.743 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66473.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66473.log.gz 2024-01-16T15:25:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58343.log 2024-01-16T15:25:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73217.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73217.log.gz 2024-01-16T15:25:13.746 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69207.log 2024-01-16T15:25:13.747 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58343.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58343.log.gz 2024-01-16T15:25:13.751 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69207.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69207.log.gz 2024-01-16T15:25:13.753 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55022.log 2024-01-16T15:25:13.756 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50964.log 2024-01-16T15:25:13.757 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55022.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55022.log.gz 2024-01-16T15:25:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67301.log 2024-01-16T15:25:13.760 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50964.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50964.log.gz 2024-01-16T15:25:13.761 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56562.log 2024-01-16T15:25:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67301.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67301.log.gz 2024-01-16T15:25:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52156.log 2024-01-16T15:25:13.767 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56562.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56562.log.gz 2024-01-16T15:25:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77655.log 2024-01-16T15:25:13.769 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52156.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52156.log.gz 2024-01-16T15:25:13.773 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48593.log 2024-01-16T15:25:13.773 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77655.log.gz 2024-01-16T15:25:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49711.log 2024-01-16T15:25:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48593.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48593.log.gz 2024-01-16T15:25:13.781 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54670.log 2024-01-16T15:25:13.783 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49711.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49711.log.gz 2024-01-16T15:25:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77301.log 2024-01-16T15:25:13.785 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54670.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54670.log.gz 2024-01-16T15:25:13.788 INFO:teuthology.orchestra.run.smithi125.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51426.log 2024-01-16T15:25:13.788 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77301.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77301.log.gz 2024-01-16T15:25:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57441.log 2024-01-16T15:25:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42402.log 2024-01-16T15:25:13.796 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51426.log: /var/log/ceph/ceph-client.admin.57441.log: 88.8% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51426.log.gz 2024-01-16T15:25:13.796 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57441.log.gz 2024-01-16T15:25:13.797 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.42402.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49151.log 2024-01-16T15:25:13.798 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42402.log.gz 2024-01-16T15:25:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54528.log 2024-01-16T15:25:13.805 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49151.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67133.log 2024-01-16T15:25:13.805 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49151.log.gz 2024-01-16T15:25:13.806 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54528.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54528.log.gz 2024-01-16T15:25:13.811 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58175.log 2024-01-16T15:25:13.811 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67133.log.gz 2024-01-16T15:25:13.813 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49323.log 2024-01-16T15:25:13.814 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58175.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58175.log.gz 2024-01-16T15:25:13.818 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63118.log 2024-01-16T15:25:13.819 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49323.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49323.log.gz 2024-01-16T15:25:13.823 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76486.log 2024-01-16T15:25:13.825 INFO:teuthology.orchestra.run.smithi125.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.63118.log.gz 2024-01-16T15:25:13.827 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2024-01-16T15:25:13.828 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.76486.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76486.log.gz 2024-01-16T15:25:13.831 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59996.log 2024-01-16T15:25:13.832 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46423.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2024-01-16T15:25:13.832 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74385.log 2024-01-16T15:25:13.834 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59996.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59996.log.gz 2024-01-16T15:25:13.836 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.74385.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74385.log.gz 2024-01-16T15:25:13.838 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71250.log 2024-01-16T15:25:13.841 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77251.log 2024-01-16T15:25:13.842 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71250.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71250.log.gz 2024-01-16T15:25:13.848 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77471.log 2024-01-16T15:25:13.848 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77251.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77251.log.gz 2024-01-16T15:25:13.850 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59011.log 2024-01-16T15:25:13.851 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77471.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77471.log.gz 2024-01-16T15:25:13.851 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51090.log 2024-01-16T15:25:13.853 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59011.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59011.log.gz 2024-01-16T15:25:13.854 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62210.log 2024-01-16T15:25:13.855 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51090.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51090.log.gz 2024-01-16T15:25:13.858 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45992.log 2024-01-16T15:25:13.859 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62210.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62210.log.gz 2024-01-16T15:25:13.862 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45992.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45992.log.gz 2024-01-16T15:25:13.863 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81550.log 2024-01-16T15:25:13.868 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54002.log 2024-01-16T15:25:13.869 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.81550.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.81550.log.gz 2024-01-16T15:25:13.872 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67217.log 2024-01-16T15:25:13.874 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54002.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54002.log.gz 2024-01-16T15:25:13.878 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51510.log 2024-01-16T15:25:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67217.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67217.log.gz 2024-01-16T15:25:13.884 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51510.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51510.log.gz 2024-01-16T15:25:13.886 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61007.log 2024-01-16T15:25:13.888 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73439.log 2024-01-16T15:25:13.890 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61007.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61007.log.gz 2024-01-16T15:25:13.892 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60881.log 2024-01-16T15:25:13.893 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.73439.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73439.log.gz 2024-01-16T15:25:13.896 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83128.log 2024-01-16T15:25:13.898 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60881.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60881.log.gz 2024-01-16T15:25:13.902 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43740.log 2024-01-16T15:25:13.903 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.83128.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83128.log.gz 2024-01-16T15:25:13.903 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-16T15:25:13.905 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43740.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43740.log.gz 2024-01-16T15:25:13.910 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69674.log 2024-01-16T15:25:13.912 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80489.log 2024-01-16T15:25:13.914 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69674.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69674.log.gz 2024-01-16T15:25:13.916 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82667.log 2024-01-16T15:25:13.917 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.80489.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.80489.log.gz 2024-01-16T15:25:13.918 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46481.log 2024-01-16T15:25:13.920 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.82667.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.82667.log.gz 2024-01-16T15:25:13.922 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66617.log 2024-01-16T15:25:13.924 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46481.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46481.log.gz 2024-01-16T15:25:13.926 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77807.log 2024-01-16T15:25:13.928 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66617.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66617.log.gz 2024-01-16T15:25:13.931 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77741.log 2024-01-16T15:25:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77807.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77807.log.gz 2024-01-16T15:25:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.77741.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.77741.log.gz 2024-01-16T15:25:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50138.log 2024-01-16T15:25:13.942 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48299.log 2024-01-16T15:25:13.946 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.50138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60317.log 2024-01-16T15:25:13.947 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48299.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48299.log.gz 2024-01-16T15:25:13.950 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-01-16T15:25:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60317.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60317.log.gz 2024-01-16T15:25:13.955 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47879.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-01-16T15:25:14.223 INFO:teuthology.orchestra.run.smithi137.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-16T15:25:14.447 INFO:teuthology.orchestra.run.smithi137.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-16T15:25:14.553 INFO:teuthology.orchestra.run.smithi125.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.74204.log.gz 2024-01-16T15:25:14.768 INFO:teuthology.orchestra.run.smithi137.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-16T15:25:14.782 INFO:teuthology.orchestra.run.smithi137.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-16T15:25:14.787 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-16T15:25:14.787 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m2.819s 2024-01-16T15:25:14.787 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m10.589s 2024-01-16T15:25:14.787 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m1.049s 2024-01-16T15:25:14.922 INFO:teuthology.orchestra.run.smithi125.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.70763.log.gz 2024-01-16T15:25:15.034 INFO:teuthology.orchestra.run.smithi125.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-16T15:25:15.335 INFO:teuthology.orchestra.run.smithi125.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.50138.log.gz 2024-01-16T15:25:16.453 INFO:teuthology.orchestra.run.smithi125.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T15:25:16.963 INFO:teuthology.orchestra.run.smithi125.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-16T15:25:17.015 INFO:teuthology.orchestra.run.smithi125.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T15:25:17.053 INFO:teuthology.orchestra.run.smithi125.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T15:25:17.247 INFO:teuthology.orchestra.run.smithi125.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-16T15:25:18.785 INFO:teuthology.orchestra.run.smithi125.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-16T15:25:18.789 INFO:teuthology.orchestra.run.smithi125.stderr: 2024-01-16T15:25:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:real 0m6.830s 2024-01-16T15:25:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:user 0m31.786s 2024-01-16T15:25:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:sys 0m4.940s 2024-01-16T15:25:18.791 INFO:tasks.ceph:Archiving logs... 2024-01-16T15:25:18.791 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi125/log 2024-01-16T15:25:18.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:25:22.721 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi137/log 2024-01-16T15:25:22.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:25:24.228 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T15:25:24.238 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T15:25:24.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:25:24.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:25:24.400 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T15:25:24.409 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T15:25:24.409 DEBUG:teuthology.orchestra.run.smithi125:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:25:24.417 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:25:24.464 INFO:teuthology.orchestra.run.smithi125.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:25:24.472 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:=============================================================================== 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2401us[+2376us] +/- 105ms 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +183us[ +159us] +/- 78ms 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -1811us[-1835us] +/- 73ms 2024-01-16T15:25:24.474 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:25:24.481 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:25:24.481 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-01-16T15:25:24.482 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2373us[+2335us] +/- 105ms 2024-01-16T15:25:24.482 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +161us[ +123us] +/- 78ms 2024-01-16T15:25:24.482 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -1840us[-1878us] +/- 73ms 2024-01-16T15:25:24.482 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:25:24.483 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T15:25:24.491 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T15:25:24.492 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T15:25:24.500 DEBUG:teuthology.orchestra.run.smithi125:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:25:24.701 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:25:24.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:25:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:25.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:25:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:25.123 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T15:25:25.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T15:25:25.142 INFO:teuthology.task.internal:Duration was 2241.073104 seconds 2024-01-16T15:25:25.143 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T15:25:25.151 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T15:25:25.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:25:25.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:25:25.301 INFO:teuthology.orchestra.run.smithi125.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:25:25.325 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:25:25.814 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T15:25:25.814 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi125.front.sepia.ceph.com 2024-01-16T15:25:25.814 DEBUG:teuthology.orchestra.run.smithi125:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:25:25.880 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-01-16T15:25:25.881 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:25:25.953 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T15:25:25.954 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:25:25.961 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:25:26.055 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T15:25:26.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:25:26.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:25:26.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T15:25:26.288 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T15:25:26.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:25:26.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:25:26.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T15:25:26.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:26.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:26.610 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = core 2024-01-16T15:25:26.612 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-01-16T15:25:26.663 DEBUG:teuthology.orchestra.run.smithi125:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:26.710 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:26.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T15:25:26.772 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T15:25:26.772 DEBUG:teuthology.misc:Transferring archived files from smithi125:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi125 2024-01-16T15:25:26.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:25:26.880 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191/remote/smithi137 2024-01-16T15:25:26.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:25:26.987 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T15:25:26.987 DEBUG:teuthology.orchestra.run.smithi125:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:25:26.996 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:25:27.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T15:25:27.062 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T15:25:27.062 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T15:25:27.072 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T15:25:27.072 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:25:27.079 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:25:27.126 INFO:teuthology.orchestra.run.smithi125.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:25 /home/ubuntu/cephtest 2024-01-16T15:25:27.135 INFO:teuthology.orchestra.run.smithi137.stdout: 394426 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:25 /home/ubuntu/cephtest 2024-01-16T15:25:27.139 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T15:25:27.149 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T15:25:27.168 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T15:25:27.196 DEBUG:teuthology.nuke:shortname: smithi125 2024-01-16T15:25:27.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:25:27.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:00.677787', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9/3G22L2LTGLeUmjReflqE9a8+S32eThZv9VfVV2VqP8LRTiHfJ25gh1NiimO9Lj/a+n3nB8QQhzq3wH1G6n0='} 2024-01-16T15:25:27.228 DEBUG:teuthology.nuke:shortname: smithi137 2024-01-16T15:25:27.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:25:27.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:37:50-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518191', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:00.680535', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvVVxBXj5PaYLksTxs/EpEzdXB4QGafjrHYSGvsfdMHzNM22qCplleZggqUAVamp+v+Yeyed3RnOuoQ4eGOA+M='} 2024-01-16T15:25:27.281 INFO:teuthology.orchestra.console.smithi125:Power off 2024-01-16T15:25:27.281 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:25:27.338 INFO:teuthology.orchestra.console.smithi137:Power off 2024-01-16T15:25:27.338 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:25:27.354 DEBUG:teuthology.orchestra.console.smithi125:power off output: Chassis Power Control: Down/Off 2024-01-16T15:25:27.354 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:27.363 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-01-16T15:25:27.364 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:27.375 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2024-01-16T15:25:27.385 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-16T15:25:31.377 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:31.385 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:31.487 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2024-01-16T15:25:31.495 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-16T15:25:35.492 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:35.501 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:35.603 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2024-01-16T15:25:35.611 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-01-16T15:25:35.704 INFO:teuthology.orchestra.console.smithi125:Power off completed 2024-01-16T15:25:35.712 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-01-16T15:25:35.817 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 2241.0731041431427 failure_reason: 'Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4825fb26b30e410f91a1aef68391b4fa status: fail success: false 2024-01-16T15:25:35.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T15:25:35.906 INFO:teuthology.run:FAIL